CN114550236B - Training method, device, equipment and storage medium for image recognition and model thereof - Google Patents

Abstract

The disclosure provides a training method, device, equipment and storage medium for image recognition and a model thereof, relates to the technical field of artificial intelligence, in particular to the technical field of deep learning and computer vision, and can be applied to scenes such as face recognition and face image recognition. The image recognition method comprises the following steps: carrying out feature extraction processing on an image to obtain local features of the image, wherein the local features are used for expressing features in an area of the image; acquiring global features of the image, wherein the global features are used for expressing inter-region features of the image; and acquiring an image recognition result of the image based on the local feature and the global feature. The present disclosure can improve image recognition accuracy.

Description

Image recognition and its model training method, device, equipment and storage medium

technical field

The present disclosure relates to the field of artificial intelligence technology, specifically the field of deep learning and computer vision technology, which can be applied to scenarios such as face recognition and face image recognition, and in particular relates to a training method, device, equipment, and training method for image recognition and its model. storage medium.

Background technique

Face recognition is a biometric technology for identification based on human facial feature information. Usually, a face recognition model can be used to recognize an input face image to obtain a face recognition result.

Contents of the invention

The disclosure provides a training method, device, equipment and storage medium for image recognition and its model.

According to one aspect of the present disclosure, there is provided an image recognition method, including: performing feature extraction processing on the image to obtain local features of the image, the local features are used to express the features in the region of the image; obtaining the The global feature of the image, the global feature is used to express the inter-regional feature of the image; based on the local feature and the global feature, an image recognition result of the image is obtained.

According to another aspect of the present disclosure, a method for training an image recognition model is provided, including: using an initial image recognition model, performing feature extraction processing on an input image sample to obtain local features of the image sample, the The local feature is used to express the intra-regional feature of the image sample; the global feature of the image sample is obtained, and the global feature is used to express the inter-regional feature of the image sample; based on the local feature and the global feature, Acquiring the predicted recognition result of the image sample; constructing a loss function based on the predicted recognition result and the real recognition result corresponding to the image sample; based on the loss function, adjusting the parameters of the initial image recognition model to generate The final image recognition model.

According to another aspect of the present disclosure, an image recognition device is provided, including: a first acquisition module, configured to perform feature extraction processing on an image to acquire local features of the image, and the local features are used to express the Intra-regional features of the image; a second acquisition module, configured to acquire global features of the image, the global features used to express inter-regional features of the image; a recognition module, configured based on the local features and the global feature, to obtain the image recognition result of the image.

According to another aspect of the present disclosure, an image recognition model training device is provided, including: a first acquisition module, configured to use an initial image recognition model to perform feature extraction processing on input image samples to acquire the image A local feature of the sample, the local feature is used to express the feature in the region of the image sample; a second acquisition module is used to obtain the global feature of the image sample, the global feature is used to express the region of the image sample Inter-features; a prediction module, used to obtain a predicted recognition result of the image sample based on the local feature and the global feature; a building module, used to obtain a real recognition result corresponding to the predicted recognition result and the image sample based on the local feature and the global feature , constructing a loss function; a generating module, configured to adjust parameters of the initial image recognition model based on the loss function, so as to generate a final image recognition model.

According to another aspect of the present disclosure, there is provided an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; Executable instructions, the instructions are executed by the at least one processor, so that the at least one processor can perform the method according to any one of the above aspects.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of the above-mentioned aspects. method.

According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of the above aspects.

According to the technical solution of the present disclosure, the image recognition accuracy can be improved.

It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood through the following description.

Description of drawings

The accompanying drawings are used to better understand the present solution, and do not constitute a limitation to the present disclosure. in:

FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure;

FIG. 2 is a schematic diagram according to a second embodiment of the present disclosure;

Fig. 3 is a schematic diagram according to a third embodiment of the present disclosure;

FIG. 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

FIG. 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

FIG. 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

FIG. 7 is a schematic diagram according to a seventh embodiment of the present disclosure;

FIG. 8 is a schematic diagram according to an eighth embodiment of the present disclosure;

FIG. 9 is a schematic diagram of an electronic device used to implement the image recognition method or the image recognition model training method of the embodiment of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure. This embodiment provides an image recognition method, the method comprising:

101. Perform feature extraction processing on an image to acquire local features of the image, where the local features are used to express intra-regional features of each region of the image.

102. Acquire global features of the image, where the global features are used to express inter-regional features of the respective regions.

103. Acquire an image recognition result of the image based on the local feature and the global feature.

Wherein, the executor of this embodiment may be an image recognition device, the image recognition device may be located in an electronic device, the electronic device may be a user terminal or a server, and the user terminal may include: a personal computer (PersonalComputer, PC), a mobile device, a smart home Devices, smart home devices, wearable devices, etc. Mobile devices include mobile phones, laptops, tablets, etc. Smart home devices include smart speakers, smart TVs, etc. Wearable devices include smart watches, smart glasses, etc. The server may be a local server or a cloud server.

Specifically, taking the image recognition as face recognition as an example, as shown in FIG. The face acquisition device (such as a camera) on the mobile device collects face images, and then, if the mobile device itself has face recognition capabilities, it can use the processor of the mobile device to perform face recognition on the collected face images to obtain Face recognition results. Alternatively, the APP 201 can send the face image to the server 202, and the server 202 can be located in a local server or in the cloud. The APP 201 and the server 202 may use a communication network for data transmission. The server 202 can perform face recognition on the received face image to obtain a face recognition result, and feed back the face recognition result to the APP 201 .

In the technical solution of this disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved are all in compliance with relevant laws and regulations, and do not violate public order and good customs.

Wherein, during image recognition, feature extraction processing may be performed on the image first to obtain image features of the image, and the image features may specifically be a feature map (feature map). In order to distinguish, since the image is processed here as a whole, the features extracted here can be considered as in-region features, which can be called local features. When the feature is a feature map, the local feature can be called a local feature map. Correspondingly, if a set of features can express the inter-regional features of different regions of the image, this feature can be called a global feature.

Taking the image as a face image and the local feature as a local feature map as an example, as shown in Figure 2, the face image can be input into a convolutional neural network (Convolutional Neural Networks, CNN), and the convolutional neural network is used to process the face image The feature extraction process is performed, and the output of the convolutional neural network is the local feature map corresponding to the input face image. The network structure of CNN can be, for example, VGG, or RetNet, etc.

FIG. 2 takes face recognition at the server as an example. It can be understood that face recognition may also be performed locally at the user terminal. Among them, the CNN used by the server to extract the local feature map can be called a face recognition model, so that the face recognition model can be used to obtain the local feature map, and then perform subsequent processing on the local feature map to obtain the face recognition result.

In related technologies, after obtaining a local feature map of a face image, the local feature map is generally directly used to obtain a face recognition result. That is, the face image feature is only a local feature, and since it only reflects local information, the accuracy of this face recognition method is insufficient.

In this embodiment, not only the local features of the image are obtained, but also the global features of the image are obtained, wherein the global features are used to express inter-regional features of the image. Since the local feature only considers the features in the region, it can be considered as local information, and the global feature considers the inter-regional features, which can be considered as global information. Therefore, the image features in this embodiment include local features and global features, reflecting the characteristics of the image to be recognized. Local information and global information improve the expression ability of image features, which in turn can improve the accuracy of image recognition.

FIG. 3 is a schematic diagram according to a third embodiment of the present disclosure. This embodiment provides an image recognition method. This embodiment takes face recognition as an example, and uses a face recognition model to obtain a local feature map of a face image as an example. The method of this embodiment includes:

301. Using a face recognition model, perform feature extraction processing on an input face image to output a local feature map of the face image.

Wherein, as shown in FIG. 4 , the CNN model may be used to perform feature extraction processing on the face image to obtain a local feature map of the face image.

The dimensionality of the local feature map F is (W, H, C).

Among them, W, H, and C are all positive integers, C is the number of channels, such as 128, W and H are the number of pixels contained in the width and height of the local feature map on each channel, for example, W= H=112, that is, on each channel, the width and height of the local feature map are both 112 pixels.

The image recognition method provided by the embodiments of the present disclosure can be applied to various image recognition scenarios, such as face recognition scenarios, animal and plant species recognition, and the like.

Taking the face recognition scene as an example, the image is a face image, and performing feature extraction processing on the image to obtain a local feature map of the image includes: using a face recognition model to input the face image Perform feature extraction processing to output the local feature map of the face image.

That is, when obtaining the local feature map, a face recognition model can be used to obtain it, and the face recognition model can be a CNN model, for example, VGG, RestNet, etc.

Since the face recognition model is usually a deep neural network model, such as the above-mentioned CNN model, the CNN model has the advantages of high accuracy, so that a more accurate local feature map can be extracted.

302. Based on the local feature map, acquire a block feature map.

Wherein, regional block processing may be performed on the local feature map to obtain multiple image blocks; based on the multiple image blocks, a block feature map is determined, and the block feature map includes the multiple image blocks.

Referring to FIG. 4 , after the local feature map F is obtained, the local feature map F can be divided into regions. Wherein, the size of each image block can be set in advance, assuming that the size of each image block is represented by w*h, and both w and h are positive integers, therefore, the number of image blocks N=(W//w)*( H//h).

Wherein, w and h are preset values, for example, w=h=2; * represents multiplication;

//Indicates divisibility. When setting w and h, you can choose a value that can be divisible by W and H.

Assuming W=H=112, w=h=2, it can be divided into N=56*56 image blocks.

After obtaining multiple image blocks, the block feature map T can be obtained based on the multiple image blocks.

Referring to Fig. 4, the dimension of block feature map T is (N, P, C), wherein, N is the number of image blocks, P is the number of pixels of each image block, that is, P=w*h, C is the number of channels. Based on the above example, N=56*56, P=2*2, C=128.

When the block feature map is obtained based on the local feature map, it can be processed for each channel, so that the local feature map and the block feature map have the same number of channels. When processing for each channel, the local feature map of W*H can be divided into multiple image blocks of size w*h, that is, N=(W//w)*(H//h) can be obtained image blocks, each image block includes w*h pixels, for each image block, these w*h pixels can be converted into w*h dimensional vectors, for example, each image block is h rows w columns The matrix can be arranged according to the order of the first row, the second row to the hth row, and each row includes w elements, which are converted into w*h dimensional vectors, so as to obtain P corresponding to each image block dimensional vector, since there are N image blocks, for each channel, an N*P dimensional matrix or image can be obtained. The N*P-dimensional image of C channels can form a block feature map with dimensions (N, P, C).

303. Obtain a region similarity matrix based on the block feature map.

Wherein, the region similarity matrix is used to indicate the similarity among the plurality of image blocks, that is, the inter-regional similarity of each region. As shown in Figure 4, the dimension of the region similarity matrix M is (N, N).

Wherein, the determining the region similarity matrix based on the block feature map may include:

Performing a first shape conversion (reshape) process on the block feature map to obtain a first matrix; performing a second shape reshape process on the block feature map to obtain a second matrix; combining the first matrix and The product of the second matrix is used as the region similarity matrix; wherein, the number of rows of the first matrix and the number of columns of the first matrix are both the number of the plurality of image blocks.

Specifically, the first matrix can be expressed as: reshape(T, [N, P*C]);

The second matrix can be expressed as: reshape(T, [P*C, N]);

Taking reshape(T, [N, P*C]) refers to converting the block feature map T with dimensions (N, P, C) into a matrix with dimensions (N, P*C), that is, the behavior N, column is a matrix of P*C. Regarding the shape conversion, it can be realized by related technologies. For example, for each row of T, the element values of the pixel points of each channel can be spliced to obtain P*C elements of each row of the first matrix, and similar processing can be performed by row After that, the first matrix with N rows and P*C columns can be obtained. The acquisition process of the second matrix is similar to the acquisition process of the first matrix, so that the second matrix whose rows are P*C and whose columns are N can be acquired.

After the first matrix and the second matrix are obtained, the product of the first matrix and the second matrix can be used as the area similarity matrix. Expressed as:

M = reshape(T, [N, P*C])*reshape(T, [P*C, N]).

By performing two kinds of shape conversion processing on the block feature map, the first matrix with row N and the second matrix with column N can be obtained respectively, and N is the number of image blocks. Therefore, based on the first matrix and the second matrix The regional similarity between two image blocks in multiple image blocks can be calculated, and the regional similarity matrix is composed of the regional similarity between two image blocks, so that the regional similarity matrix can express the connection between different regions.

304. Obtain a weighted feature map based on the region similarity matrix and the block feature map.

Among them, as shown in Figure 4, after obtaining the regional similarity matrix, weighted calculations can be performed on the regional similarity matrix and the block feature map to obtain the weighted feature map B, the dimension of the weighted feature map B and the dimension of the block feature map T Consistent, that is, the dimension of the weighted feature map B is (N, P, C).

Specifically, the region similarity matrix may be normalized to obtain a normalized region similarity matrix; the normalized region similarity matrix and the block feature map The product of is used as a weighted feature map.

Among them, the normalization can use the softmax function, and the calculation formula of the weighted feature map can be expressed as:

B＝∑softmax(M)*T

Among them, M is the region similarity matrix, T is the block feature map, and B is the weighted feature map.

softmax is a softmax function, and ∑ is a summation function.

The normalized M is an N*N matrix. When summing, for each image block, that is, for each row (or column) of the normalized M, the element values of the row can be added After that, it is used as the weighting coefficient of the image block corresponding to the row. When multiplying, for each channel of T, the weighted coefficient after row-wise summation can be used to multiply the elements of the row (a total of N rows) on the channel of T (a total of P elements in each row), Thus, B with dimensions (N, P, C) can be obtained.

Since M contains similarity information between different regions, B obtained based on M contains information between regions, that is, B contains rich global information and establishes connections between different regions.

By obtaining the weighted feature map based on the region similarity matrix and the block feature map, the weighted feature map containing global information can be obtained, so that the global feature map can be obtained based on the weighted feature map.

305. Acquire a global feature map based on the weighted feature map.

Among them, the shape conversion (reshape) processing can be performed on the weighted feature map, and the weighted feature map after the shape conversion process is used as the global feature map, that is, the weighted feature map B is converted into a feature map whose dimension is consistent with the dimension of the local feature map , and use this feature map as the global feature map.

The dimensionality of the global feature map is (W, H, C).

Since the global feature map is obtained after shape transformation of the weighted feature map, the weighted feature map contains rich global information. Therefore, the global feature map also contains rich global information, which can improve the expressive ability of image features and improve the image quality. recognition accuracy.

In addition, the global feature map is obtained based on the local feature map, specifically, the local feature map is obtained by region-blocking, and the global feature map can be obtained accurately and efficiently.

306. Acquire a fusion feature map based on the local feature map and the global feature map.

Wherein, the local feature map and the global feature map may be merged according to channel dimensions, and the merged feature map may be used as the fusion feature map.

For example, see Figure 4. The dimensions of the local feature map and the global feature map are both (W, H, C). After merging according to the channel dimension, a fusion feature map A with a dimension of (W, H, 2*C) can be obtained. .

After that, the fusion feature map can be used as the final image feature, and image recognition can be performed based on the final image feature.

Compared with the general scheme in which image features only include local features, in this embodiment, image features are combined with local features and global features, thereby improving the expressive ability of image features and improving the accuracy of image recognition.

307. Acquire a face recognition result of the face image based on the fused feature map.

Wherein, after the fusion feature map corresponding to the face image is obtained, the fusion feature map can be converted into a feature vector to be identified; The vector similarity among them; based on the vector similarity, determine the user to which the face image belongs.

Among them, as shown in Figure 4, the dimension of the fusion feature map A is (W, H, C), which can be flattened into a vector of W*H*C dimension, which can be called the feature vector to be identified .

Flattening can be achieved using various related technologies. For example, for each channel, it can be arranged in rows as W*H-dimensional vectors, and then the W*H-dimensional vectors corresponding to each channel in the C channels can be arranged as W* A vector of H*C dimensions.

The feature vectors of each of the multiple users may be pre-stored in the database, and the feature vectors of each user may be called candidate feature vectors. For example, there are 10,000 candidate feature vectors of users in the database.

The vector similarity between the feature vector to be identified and each candidate feature vector can be calculated, wherein the vector similarity calculation methods in various related technologies can be used, for example, the cosine similarity between vectors can be calculated. Afterwards, the face recognition result can be obtained based on the vector similarity, for example, the user corresponding to the candidate feature vector with the highest vector similarity is used as the user to which the recognized face image belongs.

By determining the image recognition result based on the similarity between vectors, the image recognition result can be obtained easily and quickly.

In this embodiment, the local feature map of the face image is obtained by using the face recognition model, and the local feature map is subjected to regional block processing to obtain the block feature map, the global feature map is obtained based on the block feature map, and the local feature map The image and the global feature map are fused to obtain the fused feature map, and the face recognition result is obtained based on the fused feature map. In the process of face recognition, the local information and global information of the face image can be combined to improve the image characteristics of the face image. expression ability, thereby improving the accuracy of face recognition.

The image recognition process is described above, and as described above, an image recognition model can be used to obtain a local feature map. For the training process of the image recognition model, please refer to the following embodiments.

FIG. 5 is a schematic diagram according to a fifth embodiment of the present disclosure. This embodiment provides a training method for an image recognition model, the method comprising:

501. Using an initial image recognition model, perform feature extraction processing on an input image sample to obtain local features of the image sample, where the local features are used to express features within a region of the image sample.

502. Acquire a global feature of the image sample, where the global feature is used to express an inter-region feature of the image sample.

503. Acquire a predicted recognition result of the image sample based on the local feature and the global feature.

504. Construct a loss function based on the predicted recognition result and the real recognition result corresponding to the image sample.

505. Adjust parameters of the initial image recognition model based on the loss function to generate a final image recognition model.

Wherein, the executor of this embodiment may be a training device for an image recognition model, the training device for an image recognition model may be located in an electronic device, the electronic device may be a user terminal or a server, and the user terminal may include: a personal computer (Personal Computer, PC ), mobile devices, smart home devices, smart home devices, wearable devices, etc. Mobile devices include mobile phones, laptops, tablets, etc. Smart home devices include smart speakers, smart TVs, etc. Wearable devices include Smart watches, smart glasses, etc. The server may be a local server or a cloud server.

Wherein, the images used in the model training stage may be called image samples, and the image samples may be obtained from existing datasets, such as ImageNet.

In addition, the image recognition results of the image samples can also be marked in advance by manual labeling or other methods, for example, the user ID of the user to which the image sample belongs, etc., and the pre-labeled image recognition results can be called the real recognition results corresponding to the image samples. result.

Similar to the model application stage (that is, the above-mentioned image recognition process), after the image sample is processed by the image recognition model, an image recognition result can be obtained, and the image recognition result can be called a predictive recognition result.

The model parameters of the initial image recognition model can be manually set, or the initial image recognition model can use an existing pre-trained model.

For the process from image samples to predicted recognition results, the following embodiments may be included. For specific content, reference may be made to the above-mentioned embodiments on the image recognition process, which will not be described in detail here.

In this embodiment, the predicted recognition result can be obtained based on the global features and local features, so that not only the local information of the image sample, but also the global information of the image sample can be considered. Therefore, the accuracy of the predicted recognition result can be improved, and the image The accuracy of the recognition model.

In some embodiments, the local feature is a local feature map, the global feature is a global feature map, and the obtaining the global feature of the image includes: performing regional block processing on the local feature map to obtain multiple image blocks; based on the plurality of image blocks, determine a block feature map, the block feature map includes the plurality of image blocks; based on the block feature map, determine a region similarity matrix, and the regions are similar The degree matrix is used to indicate the similarity between the plurality of image blocks; the global feature map is obtained based on the region similarity matrix and the block feature map.

The global feature map is obtained based on the local feature map, specifically, it is obtained based on the regional block of the local feature map, and the global feature map can be obtained accurately and efficiently. In addition, since the regional similarity matrix can express inter-regional information, a global feature map that can express inter-regional information can be obtained based on the regional similarity matrix.

In some embodiments, the determining the region similarity matrix based on the block feature map includes: performing a first shape conversion process on the block feature map to obtain a first matrix; performing a second shape conversion process to obtain a second matrix; using the product of the first matrix and the second matrix as the region similarity matrix; wherein, the number of rows of the first matrix is the same as the number of rows of the second matrix The number of columns in a matrix is the number of the plurality of image blocks.

By performing two kinds of shape conversion processing on the block feature map, the first matrix with row N and the second matrix with column N can be obtained respectively, and N is the number of image blocks. Therefore, based on the first matrix and the second matrix The regional similarity between two image blocks in the plurality of image blocks can be calculated, and the regional similarity matrix between the two image blocks is formed.

In some embodiments, the obtaining the global feature map based on the region similarity matrix and the block feature map includes: performing normalization processing on the region similarity matrix to obtain a normalized The regional similarity matrix after the normalization process; the product of the normalized regional similarity matrix and the block feature map is used as a weighted feature map; shape conversion processing is performed on the weighted feature map to obtain the described A global feature map, wherein the dimension of the global feature map is consistent with the dimension of the local feature map.

By obtaining weighted feature maps based on the region similarity matrix and block feature maps, weighted feature maps containing global information can be obtained. Since the global feature map is obtained after shape transformation of the weighted feature map, the weighted feature map contains rich global information. Therefore, the global feature map also contains rich global information, which can improve the expressive ability of image features and improve the image quality. recognition accuracy.

In some embodiments, the local feature is a local feature map, the global feature is a global feature map, and the obtaining the predicted recognition result of the image sample based on the local feature and the global feature includes: performing fusion processing on the local feature map and the global feature map to obtain a fusion feature map; based on the fusion feature map, obtaining a prediction recognition result of the image sample.

Compared with the general scheme in which image features only include local features, in this embodiment, image features are combined with local features and global features, thereby improving the expressive ability of image features and improving the accuracy of image recognition.

In some embodiments, the image sample is a face image sample, and the obtaining the predicted recognition result of the image sample based on the fusion feature map includes: converting the fusion feature map into a feature vector to be recognized; determining the The vector similarity between the feature vector to be identified and the candidate feature vectors of multiple users pre-stored; based on the vector similarity, determine the user information of the user to which the face image sample belongs, and use the The above user information is used as the predicted identification result.

Wherein, the user information may be a user identifier, or a probability value that the user is a correct identification result, and the like. Taking the probability value as an example, the real recognition result is used to indicate whether the user is the correct recognition result. The real recognition result can be 1 (correct recognition result) or 0 (not the correct recognition result), while the predicted recognition result is generally 0 to 1. value.

By determining the predicted recognition result based on the similarity between vectors, the predicted recognition result can be obtained easily and quickly.

After obtaining the predicted recognition results and the real recognition results, the loss function can be constructed.

For example, see Figure 6. After the face image sample is processed by convolution and area division, the fusion feature map can be obtained. Based on the fusion feature map, the recognition result can be predicted, and the loss function can be constructed based on the predicted recognition result and the real recognition result. The loss function is, for example, a classification loss function. Specifically, the classification loss function may be a cross-entropy function.

After the loss function is obtained, the model parameters of the image recognition model can be adjusted based on the loss function, wherein the model parameters can be adjusted by using a back propagation (Back Propagation, BP) algorithm or the like. The model parameter adjustment process can set a maximum number of iterations, and when the number of adjustments of the model parameters reaches the maximum number of iterations, the model corresponding to the model parameters at this time is used as the final image recognition model.

FIG. 7 is a schematic diagram of a seventh embodiment of the present disclosure. This embodiment provides an image recognition device, and the device 700 includes: a first acquisition module 701 , a second acquisition module 702 and an identification module 703 .

The first acquisition module 701 is used to perform feature extraction processing on the image to acquire the local features of the image, and the local features are used to express the features in the region of the image; the second acquisition module 702 is used to acquire the features of the image A global feature, the global feature is used to express the inter-regional feature of the image; the recognition module 703 is used to obtain an image recognition result of the image based on the local feature and the global feature.

In this embodiment, not only the local features of the image are obtained, but also the global features of the image are obtained, wherein the global features are used to express inter-regional features of the image. Since the local feature only considers the features in the region, it can be considered as local information, and the global feature considers the inter-regional features, which can be considered as global information. Therefore, the image features in this embodiment include local features and global features, reflecting the characteristics of the image to be recognized. Local information and global information improve the expression ability of image features, which in turn can improve the accuracy of image recognition.

In some embodiments, the local feature is a local feature map, the global feature is a global feature map, and the second acquisition module 702 is further configured to: perform regional block processing on the local feature map to obtain multiple An image block; based on the plurality of image blocks, determine a block feature map, the block feature map includes the plurality of image blocks; based on the block feature map, determine a region similarity matrix, the region similarity The matrix is used to indicate the similarity between the plurality of image blocks; the global feature map is obtained based on the region similarity matrix and the block feature map.

Since the regional similarity matrix can express inter-regional information, a global feature map that can express inter-regional information can be obtained based on the regional similarity matrix.

In some embodiments, the second obtaining module 702 is further configured to: perform a first shape transformation process on the block feature map to obtain a first matrix; perform a second shape transformation process on the block feature map, to obtain a second matrix; the product of the first matrix and the second matrix is used as the area similarity matrix; wherein, the number of rows of the first matrix and the number of columns of the first matrix are both The number of the plurality of image blocks.

By performing two kinds of shape conversion processing on the block feature map, the first matrix with row N and the second matrix with column N can be obtained respectively, and N is the number of image blocks. Therefore, based on the first matrix and the second matrix The regional similarity between two image blocks in multiple image blocks can be calculated, and the regional similarity matrix is composed of the regional similarity between two image blocks, so that the regional similarity matrix can express the connection between different regions.

In some embodiments, the second obtaining module 702 is further configured to: perform normalization processing on the region similarity matrix to obtain a normalized region similarity matrix; The product of the regional similarity matrix and the block feature map is used as a weighted feature map; the weighted feature map is subjected to shape conversion processing to obtain the global feature map, wherein the dimension of the global feature map is the same as the The dimensions of the local feature maps described above are consistent.

By obtaining weighted feature maps based on the region similarity matrix and block feature maps, weighted feature maps containing global information can be obtained. Since the global feature map is obtained after shape transformation of the weighted feature map, the weighted feature map contains rich global information. Therefore, the global feature map also contains rich global information, which can improve the expressive ability of image features and improve the image quality. recognition accuracy.

In some embodiments, the local feature is a local feature map, the global feature is a global feature map, and the identification module 703 is further configured to: perform fusion processing on the local feature map and the global feature map to obtain Fusion feature map; based on the fusion feature map, an image recognition result of the image is acquired.

Compared with the general scheme in which image features only include local features, in this embodiment, image features are combined with local features and global features, thereby improving the expressive ability of image features and improving the accuracy of image recognition.

In some embodiments, the image is a face image, and the identification module 703 is further configured to: convert the fusion feature map into a feature vector to be identified; determine the feature vector to be identified and the candidate features of multiple pre-stored users The vector similarity between each candidate feature vector in the vector; based on the vector similarity, determine the user to which the face image belongs.

By determining the image recognition result based on the similarity between vectors, the image recognition result can be obtained easily and quickly.

In some embodiments, the first acquisition module 701 is further configured to: use an image recognition model to perform feature extraction processing on the input image, so as to output local features of the image.

Since the face recognition model is usually a deep neural network model, such as the above-mentioned CNN model, the CNN model has the advantages of high accuracy, so that more accurate local features can be extracted.

8 is a schematic diagram of the eighth embodiment of the present disclosure. This embodiment provides a training device for an image recognition model. The device 800 includes: a first acquisition module 801, a second acquisition module 802, a prediction module 803, a construction module 804 and Generate module 805 .

The first acquisition module 801 is configured to use an initial image recognition model to perform feature extraction processing on the input image sample, so as to acquire local features of the image sample, and the local features are used to express regional features of the image sample; The second obtaining module 802 is used to obtain the global feature of the image sample, and the global feature is used to express the inter-regional feature of the image sample; the prediction module 803 is used to obtain the global feature based on the local feature and the global feature. The predicted recognition result of the image sample; the construction module 804 is used to construct a loss function based on the predicted recognition result and the real recognition result corresponding to the image sample; the generation module 805 is used to adjust the initial The parameters of the image recognition model to generate the final image recognition model.

In this embodiment, the predicted recognition result can be obtained based on the global features and local features, so that not only the local information of the image sample, but also the global information of the image sample can be considered. Therefore, the accuracy of the predicted recognition result can be improved, and the image The accuracy of the recognition model.

In some embodiments, the local feature is a local feature map, the global feature is a global feature map, and the second acquisition module 802 is further configured to: perform regional block processing on the local feature map to obtain multiple An image block; based on the plurality of image blocks, determine a block feature map, the block feature map includes the plurality of image blocks; based on the block feature map, determine a region similarity matrix, the region similarity The matrix is used to indicate the similarity between the plurality of image blocks; the global feature map is obtained based on the region similarity matrix and the block feature map.

Since the regional similarity matrix can express inter-regional information, a global feature map that can express inter-regional information can be obtained based on the regional similarity matrix.

In some embodiments, the second acquisition module 802 is further configured to: perform a first shape conversion process on the block feature map to obtain a first matrix; perform a second shape conversion process on the block feature map, to obtain a second matrix; the product of the first matrix and the second matrix is used as the area similarity matrix; wherein, the number of rows of the first matrix and the number of columns of the first matrix are both The number of the plurality of image blocks.

By performing two kinds of shape conversion processing on the block feature map, the first matrix with row N and the second matrix with column N can be obtained respectively, and N is the number of image blocks. Therefore, based on the first matrix and the second matrix The regional similarity between two image blocks in the plurality of image blocks can be calculated, and the regional similarity matrix between the two image blocks is formed.

In some embodiments, the second obtaining module 802 is further configured to: perform normalization processing on the region similarity matrix to obtain a normalized region similarity matrix; The product of the regional similarity matrix and the block feature map is used as a weighted feature map; the weighted feature map is subjected to shape conversion processing to obtain the global feature map, wherein the dimension of the global feature map is the same as the The dimensions of the local feature maps described above are consistent.

By obtaining weighted feature maps based on the region similarity matrix and block feature maps, weighted feature maps containing global information can be obtained. Since the global feature map is obtained after shape transformation of the weighted feature map, the weighted feature map contains rich global information. Therefore, the global feature map also contains rich global information, which can improve the expressive ability of image features and improve the image quality. recognition accuracy.

In some embodiments, the local feature is a local feature map, the global feature is a global feature map, and the prediction module 803 is further configured to: perform fusion processing on the local feature map and the global feature map to obtain Fusion feature map; based on the fusion feature map, obtain the predicted recognition result of the image sample.

Compared with the general scheme in which image features only include local features, in this embodiment, image features are combined with local features and global features, thereby improving the expressive ability of image features and improving the accuracy of image recognition.

In some embodiments, the image sample is a face image sample, and the prediction module 803 is further configured to: convert the fusion feature map into a feature vector to be identified; determine the feature vector to be identified and the pre-stored multiple user The vector similarity between each candidate feature vector in the candidate feature vectors; based on the vector similarity, determine the user information of the user to which the face image sample belongs, and use the user information as the predicted recognition result.

By determining the predicted recognition result based on the similarity between vectors, the predicted recognition result can be obtained easily and quickly.

It can be understood that the "first" and "second" in the embodiments of the present disclosure are only used for distinction, and do not indicate the level of importance, sequence of time, and the like.

It can be understood that, in the embodiments of the present disclosure, the same or similar content in different embodiments may refer to each other.

In the technical solution of this disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved are all in compliance with relevant laws and regulations, and do not violate public order and good customs.

According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

FIG. 9 shows a schematic block diagram of an example electronic device 900 that may be used to implement embodiments of the present disclosure. Electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 9 , an electronic device 900 includes a computing unit 901, which can perform calculations according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. Various appropriate actions and processes are performed. In the RAM 903, various programs and data necessary for the operation of the electronic device 900 can also be stored. The computing unit 901 , ROM 902 , and RAM 903 are connected to each other through a bus 904 . An input/output (I/O) interface 905 is also connected to the bus 904 .

Multiple components in the electronic device 900 are connected to the I/O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 909 allows the electronic device 900 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

The computing unit 901 may be various general-purpose and/or special-purpose processing components having processing and computing capabilities. Some examples of computing units 901 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The calculation unit 901 executes various methods and processes described above, such as an image recognition method or an image recognition model training method. For example, in some embodiments, the image recognition method or the training method of the image recognition model may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908 . In some embodiments, part or all of the computer program may be loaded and/or installed on the electronic device 900 via the ROM 902 and/or the communication unit 909 . When the computer program is loaded into RAM 903 and executed by computing unit 901, one or more steps of the image recognition method or image recognition model training method described above can be executed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute an image recognition method or an image recognition model training method in any other appropriate manner (for example, by means of firmware).

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOC), Complex Programmable Logic Device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor Can be special-purpose or general-purpose programmable processor, can receive data and instruction from storage system, at least one input device, and at least one output device, and transmit data and instruction to this storage system, this at least one input device, and this at least one output device an output device.

Program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special purpose computer, or other programmable data processing devices, so that the program codes, when executed by the processor or controller, make the functions/functions specified in the flow diagrams and/or block diagrams Action is implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide for interaction with the user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user. ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, speech input or, tactile input) to receive input from the user.

The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., as a a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN) and the Internet.

A computer system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the problem of traditional physical host and VPS service ("Virtual Private Server", or "VPS") Among them, there are defects such as difficult management and weak business scalability. The server can also be a server of a distributed system, or a server combined with a blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, each step described in the present disclosure may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in the present disclosure can be achieved, no limitation is imposed herein.

The specific implementation manners described above do not limit the protection scope of the present disclosure. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims (20)

1. An image recognition method, comprising:

carrying out feature extraction processing on an image to obtain local features of the image, wherein the local features are used for expressing features in an area of the image;

acquiring global features of the image, wherein the global features are used for expressing inter-region features of the image;

acquiring an image recognition result of the image based on the local feature and the global feature;

The local feature is a local feature map, the global feature is a global feature map, and the acquiring the global feature of the image includes:

performing regional blocking processing on the local feature map to obtain a plurality of image blocks;

determining a block feature map based on the plurality of image blocks, the block feature map comprising the plurality of image blocks;

determining a region similarity matrix based on the block feature map, wherein the region similarity matrix is used for indicating the similarity among the plurality of image blocks; wherein, based on the block feature map, determining a region similarity matrix includes: performing first shape conversion processing on the block feature map to obtain a first matrix; performing second shape conversion processing on the block feature map to obtain a second matrix; taking the product of the first matrix and the second matrix as the area similarity matrix; the number of rows of the first matrix and the number of columns of the second matrix are the number of the plurality of image blocks;

acquiring the global feature map based on the region similarity matrix and the block feature map; the dimension of the global feature map is consistent with the dimension of the local feature map, the global feature map is obtained by performing shape conversion on a weighted feature map, and the weighted feature map is obtained by performing weighted calculation on the block feature map and the region similarity matrix.

2. The method according to claim 1, wherein the weighted feature map is obtained by performing weighted calculation on the block feature map and the region similarity matrix, and specifically includes:

normalizing the region similarity matrix to obtain a normalized region similarity matrix;

and taking the product of the normalized area similarity matrix and the block feature map as a weighted feature map.

3. The method of claim 1, wherein the local feature is a local feature map and the global feature is a global feature map, the acquiring an image recognition result of the image based on the local feature and the global feature comprising:

carrying out fusion processing on the local feature map and the global feature map to obtain a fusion feature map;

and acquiring an image recognition result of the image based on the fusion feature map.

4. A method according to claim 3, wherein the image is a face image, and the acquiring an image recognition result of the image based on the fused feature map includes:

converting the fusion feature map into feature vectors to be identified;

determining the vector similarity between the feature vector to be identified and each candidate feature vector in the pre-stored candidate feature vectors of a plurality of users;

And determining the user to which the face image belongs based on the vector similarity.

5. A method according to any one of claims 1-3, wherein said subjecting the image to a feature extraction process to obtain local features of the image comprises:

and adopting an image recognition model to perform feature extraction processing on the input image so as to output local features of the image.

6. A training method of an image recognition model, comprising:

performing feature extraction processing on an input image sample by adopting an initial image recognition model to obtain local features of the image sample, wherein the local features are used for expressing features in a region of the image sample;

acquiring global features of the image sample, wherein the global features are used for expressing inter-region features of the image sample;

based on the local features and the global features, obtaining a prediction recognition result of the image sample;

constructing a loss function based on the predicted recognition result and a real recognition result corresponding to the image sample;

based on the loss function, adjusting parameters of the initial image recognition model to generate a final image recognition model;

The local feature is a local feature map, the global feature is a global feature map, and the acquiring the global feature of the image includes:

performing regional blocking processing on the local feature map to obtain a plurality of image blocks;

determining a block feature map based on the plurality of image blocks, the block feature map comprising the plurality of image blocks;

determining a region similarity matrix based on the block feature map, wherein the region similarity matrix is used for indicating the similarity among the plurality of image blocks; wherein, based on the block feature map, determining a region similarity matrix includes: performing first shape conversion processing on the block feature map to obtain a first matrix; performing second shape conversion processing on the block feature map to obtain a second matrix; taking the product of the first matrix and the second matrix as the area similarity matrix;

the number of rows of the first matrix and the number of columns of the second matrix are the number of the plurality of image blocks;

acquiring the global feature map based on the region similarity matrix and the block feature map; the dimension of the global feature map is consistent with the dimension of the local feature map, the global feature map is obtained by performing shape conversion on a weighted feature map, and the weighted feature map is obtained by performing weighted calculation on the block feature map and the region similarity matrix.

7. The method of claim 6, wherein the weighted feature map is obtained by performing weighted calculation on the block feature map and the region similarity matrix, and specifically includes:

normalizing the region similarity matrix to obtain a normalized region similarity matrix;

and taking the product of the normalized area similarity matrix and the block feature map as a weighted feature map.

8. The method according to any one of claims 6-7, wherein the local feature is a local feature map and the global feature is a global feature map, the obtaining a predictive recognition result of the image sample based on the local feature and the global feature comprises:

carrying out fusion processing on the local feature map and the global feature map to obtain a fusion feature map;

and acquiring a prediction recognition result of the image sample based on the fusion feature map.

9. The method of claim 8, wherein the image sample is a face image sample, the obtaining a predicted recognition result of the image sample based on the fused feature map comprises:

converting the fusion feature map into feature vectors to be identified;

Determining the vector similarity between the feature vector to be identified and each candidate feature vector in the pre-stored candidate feature vectors of a plurality of users;

and determining user information of a user to which the face image sample belongs based on the vector similarity, and taking the user information as the prediction recognition result.

10. An image recognition apparatus comprising:

the first acquisition module is used for carrying out feature extraction processing on the image so as to acquire local features of the image, wherein the local features are used for expressing the features in the region of the image;

the second acquisition module is used for acquiring global features of the image, wherein the global features are used for expressing inter-region features of the image;

the identification module is used for acquiring an image identification result of the image based on the local feature and the global feature;

wherein the local feature is a local feature map, the global feature is a global feature map, and the second obtaining module is further configured to:

performing regional blocking processing on the local feature map to obtain a plurality of image blocks;

determining a block feature map based on the plurality of image blocks, the block feature map comprising the plurality of image blocks;

Determining a region similarity matrix based on the block feature map, wherein the region similarity matrix is used for indicating the similarity among the plurality of image blocks;

acquiring the global feature map based on the region similarity matrix and the block feature map; the dimension of the global feature map is consistent with the dimension of the local feature map, the global feature map is obtained by performing shape conversion on a weighted feature map, and the weighted feature map is obtained by performing weighted calculation on the block feature map and the region similarity matrix;

wherein the second acquisition module is further configured to:

performing first shape conversion processing on the block feature map to obtain a first matrix;

performing second shape conversion processing on the block feature map to obtain a second matrix;

taking the product of the first matrix and the second matrix as the area similarity matrix;

the number of rows of the first matrix and the number of columns of the second matrix are the number of the plurality of image blocks.

11. The apparatus of claim 10, wherein the second acquisition module is further to:

normalizing the region similarity matrix to obtain a normalized region similarity matrix;

And taking the product of the normalized area similarity matrix and the block feature map as a weighted feature map.

12. The apparatus of claim 10, wherein the local feature is a local feature map and the global feature is a global feature map, the identification module further to:

carrying out fusion processing on the local feature map and the global feature map to obtain a fusion feature map;

and acquiring an image recognition result of the image based on the fusion feature map.

13. The apparatus of claim 12, wherein the image is a face image, the recognition module further to:

converting the fusion feature map into feature vectors to be identified;

determining the vector similarity between the feature vector to be identified and each candidate feature vector in the pre-stored candidate feature vectors of a plurality of users;

and determining the user to which the face image belongs based on the vector similarity.

14. The apparatus of any of claims 10-13, wherein the first acquisition module is further to:

and adopting an image recognition model to perform feature extraction processing on the input image so as to output local features of the image.

15. A training device for an image recognition model, comprising:

the first acquisition module is used for carrying out feature extraction processing on an input image sample by adopting an initial image recognition model so as to acquire local features of the image sample, wherein the local features are used for expressing the features in the region of the image sample;

the second acquisition module is used for acquiring global features of the image sample, wherein the global features are used for expressing inter-region features of the image sample;

the prediction module is used for acquiring a prediction recognition result of the image sample based on the local feature and the global feature;

the construction module is used for constructing a loss function based on the prediction recognition result and the real recognition result corresponding to the image sample;

the generation module is used for adjusting parameters of the initial image recognition model based on the loss function so as to generate a final image recognition model;

wherein the local feature is a local feature map, the global feature is a global feature map, and the second obtaining module is further configured to:

performing regional blocking processing on the local feature map to obtain a plurality of image blocks;

determining a block feature map based on the plurality of image blocks, the block feature map comprising the plurality of image blocks;

Determining a region similarity matrix based on the block feature map, wherein the region similarity matrix is used for indicating the similarity among the plurality of image blocks;

acquiring the global feature map based on the region similarity matrix and the block feature map; the dimension of the global feature map is consistent with the dimension of the local feature map, the global feature map is obtained by performing shape conversion on a weighted feature map, and the weighted feature map is obtained by performing weighted calculation on the block feature map and the region similarity matrix;

wherein the second acquisition module is further configured to:

performing first shape conversion processing on the block feature map to obtain a first matrix;

performing second shape conversion processing on the block feature map to obtain a second matrix;

taking the product of the first matrix and the second matrix as the area similarity matrix;

the number of rows of the first matrix and the number of columns of the second matrix are the number of the plurality of image blocks.

16. The apparatus of claim 15, wherein the second acquisition module is further to:

normalizing the region similarity matrix to obtain a normalized region similarity matrix;

And taking the product of the normalized area similarity matrix and the block feature map as a weighted feature map.

17. The apparatus of any of claims 15-16, wherein the local feature is a local feature map and the global feature is a global feature map, the prediction module further to:

carrying out fusion processing on the local feature map and the global feature map to obtain a fusion feature map;

and acquiring a prediction recognition result of the image sample based on the fusion feature map.

18. The apparatus of claim 17, wherein the image sample is a face image sample, the prediction module further to:

converting the fusion feature map into feature vectors to be identified;

determining the vector similarity between the feature vector to be identified and each candidate feature vector in the pre-stored candidate feature vectors of a plurality of users;

and determining user information of a user to which the face image sample belongs based on the vector similarity, and taking the user information as the prediction recognition result.

19. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-9.