WO2020125229A1 - Feature fusion method and apparatus, and electronic device and storage medium - Google Patents

Abstract

A feature fusion method, comprising: an electronic device obtains a plurality of images of a target object (S11); the electronic device inputs the plurality of images into a pre-trained image quality model to obtain the M-dimensional image quality of each image (S12); the electronic device inputs the plurality of images into a pre-trained feature recognition model to obtain N-dimensional image features of each image (S13); and the electronic device performs weighted summation on the M-dimensional image quality and the N-dimensional image features to obtain fusion features (S14), wherein M is a positive integer, N is a positive integer, when M=1, N≥2, and when M≥2, M=N. Also provided are a feature fusion apparatus, an electronic device and a storage medium. According to the method, the image recognition effect can be improved.

Description

Feature fusion method, device, electronic equipment and storage medium

This application requires priority to be submitted to the Chinese Patent Office on December 20, 2018, with the application number 201811565889.2 and the invention titled "Feature Fusion Method, Device, Electronic Equipment, and Storage Media", the entire contents of which are incorporated by reference In this application.

Technical field

The present invention relates to the field of image processing technology, and in particular, to a feature fusion method, device, electronic device, and storage medium.

Background technique

In video recognition scenes, it is common to take multiple snapshots of the same target to obtain multiple images, and extract features from each image separately. Because of the large number of images captured for the same target, and the captured images are different in size, lighting, posture, occlusion, expression, etc., this makes it impossible to use a single image feature for image recognition All parts of the image are recognized, and the effect of image recognition is poor.

Summary of the invention

In view of the above, it is necessary to provide a feature fusion method, device, electronic device, and storage medium, which can improve the effect of image recognition.

A first aspect of the present invention provides a feature fusion method. The method includes:

Acquire multiple images of the target object;

Input the multiple images into a pre-trained image quality model to obtain the M-dimensional image quality of each image;

Input the multiple images into a pre-trained feature recognition model to obtain N-dimensional image features of each image;

Weighting and summing the M-dimensional image quality and the N-dimensional image feature to obtain a fusion feature; wherein, M is a positive integer, the N is a positive integer, when M=1, N≥2 When M≥2, M=N.

In a possible implementation manner, the acquiring multiple images of the target object includes:

Capture multiple images of the target object from the video of the target object; or

Acquire multiple images of the target object taken at different times.

In a possible implementation, when the M=1, the weighted summation of the M-dimensional image quality and the N-dimensional image feature to obtain the fusion feature includes:

For the jth image, the image quality of the jth image is multiplied by the i-dimensional image feature of the jth image to obtain the i-dimensional sub-feature of the jth image;

Summing the i-dimensional sub-features of multiple images to obtain the i-dimensional fusion feature;

Among them, i is a positive integer, j is a positive integer, and 1≤i≤N, j≥1.

In a possible implementation manner, when the M≥2 and M=N, the weighted summation of the M-dimensional image quality and the N-dimensional image feature to obtain the fusion feature includes:

For the jth image, the i-dimensional image quality of the j-th image is multiplied by the i-dimensional image feature of the j-th image to obtain the i-th dimension of the j-th image Sub-feature

Summing the i-dimensional sub-features of multiple images to obtain the i-dimensional feature sum;

Summing the i-dimensional image qualities of multiple images to obtain the image quality sum;

Dividing the feature sum of the i-th dimension by the sum of the image quality to obtain the fusion feature of the i-th dimension;

Among them, i is a positive integer, j is a positive integer, and 1≤i≤N, j≥1.

In a possible implementation manner, the method further includes:

Perform normalization processing on the fusion feature to obtain a final feature.

In a possible implementation manner, before acquiring multiple images of the target object, the method further includes:

Acquiring multiple sample images of the object to be trained, and acquiring multiple standard images of the object to be trained;

Input the multiple sample images and the multiple standard images into a pre-trained feature recognition model to obtain image features of the object to be trained;

Input the plurality of sample images and the plurality of standard images into a preset training model to obtain the image quality of the object to be trained;

Calculate the fusion characteristics of the object to be trained according to the image characteristics of the object to be trained and the image quality of the object to be trained;

Input the fusion feature of the object to be trained into a preset loss function to obtain a loss value;

According to the loss value, use a back propagation algorithm to update the parameters of the training model;

If the loss value of the loss function reaches a convergence state, it is determined that the training model after updating the parameters is a trained image quality model.

In a possible implementation manner, the input order of the multiple images of the target object and the number of images have no effect on the fusion feature.

A second aspect of the present invention provides a feature fusion device. The device includes:

The acquisition module is used to acquire multiple images of the target object;

The input module is used to input the multiple images into a pre-trained image quality model to obtain the M-dimensional image quality of each image;

The input module is further configured to input the multiple images into a pre-trained feature recognition model to obtain N-dimensional image features of each image;

A calculation module, configured to perform weighted summation of the M-dimensional image quality and the N-dimensional image feature to obtain a fusion feature; wherein, M is a positive integer, the N is a positive integer, and when M=1 , N≥2, when M≥2, M=N.

A third aspect of the present invention provides an electronic device. The electronic device includes a processor and a memory. The processor is used to implement the feature fusion method when executing a computer program stored in the memory.

A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored on the computer-readable storage medium. The computer program is executed by a processor to implement the feature fusion method.

From the above technical solution, in the present invention, multiple images of the target object can be obtained first, the multiple images are input into a pre-trained image quality model to obtain the M-dimensional image quality of each image, and the Multiple images are input into a pre-trained feature recognition model to obtain N-dimensional image features of each image. Further, the M-dimensional image quality and the N-dimensional image features may be weighted and summed to obtain fusion features ; Where M is a positive integer and N is a positive integer, when M=1, N≥2, and when M≥2, M=N. It can be seen that in the present invention, after acquiring multiple images of the target object, the image quality and image features of the multiple images can be extracted through the image quality model and the feature recognition model, and then, the image quality and image features of all images are processed The weighted sum can obtain the fusion feature. Since the fusion feature is obtained by fusing multiple image features of multiple images and multiple image qualities, the fusion feature can include all the features of the target object, relative to As far as single image features are concerned, the fusion feature makes up for the lack of certain image features of the target object in the single image feature. When performing image recognition, the fusion feature can be used to identify the image in all directions, which can improve the image Recognition effect, recognition accuracy is higher.

BRIEF DESCRIPTION

In order to more clearly explain the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only This is an embodiment of the present invention. For those of ordinary skill in the art, without paying any creative labor, other drawings may be obtained according to the provided drawings.

FIG. 1 is a flowchart of a preferred embodiment of a feature fusion method disclosed in the present invention.

2 is a functional block diagram of a preferred embodiment of a feature fusion device disclosed in the present invention.

3 is a schematic structural diagram of an electronic device according to a preferred embodiment of a method for implementing feature fusion according to the present invention.

detailed description

Please refer to FIG. 1, which is a flowchart of a preferred embodiment of a feature fusion method disclosed in the present invention. According to different requirements, the sequence of steps in the flowchart can be changed, and some steps can be omitted.

S11. The electronic device acquires multiple images of the target object.

Generally, when shooting the same object (such as a target object), due to differences in lighting, motion blur, noise, expression, posture, and debris blocking, whether it is a static photo or a dynamic video, the difference is The images of time are all different. That is to say, the photos taken at the first time and the photos taken at the second time may be different, and the two images of the same object captured in the same video may also be different.

In the embodiment of the present invention, the electronic device may obtain multiple images of the target object that needs to be image-recognized in various ways. Among them, any two of the images have different characteristics.

Specifically, the acquiring multiple images of the target object includes:

Capture multiple images of the target object from the video of the target object; or

Acquire multiple images of the target object taken at different times.

In this optional embodiment, the target object may be photographed to obtain a video of the target object, and further, multiple images of the target object at different times may be captured from the video; or

The target object may be photographed multiple times at different times to obtain multiple images.

S12. The electronic device inputs the multiple images into a pre-trained image quality model to obtain the M-dimensional image quality of each image.

The image quality may be M-dimensional, and M is a positive integer. When M=1, that is, the quality of the image is 1 dimensional, using 1 value to indicate the quality of the image, when M ≥ 2, the quality of the image is multi-dimensional, using multiple values (such as 1 * M values) The quality of the image in M dimensions is good or bad.

In the embodiments of the present invention, the image quality model can be used to measure the quality of the image. By inputting multiple images of the target object into a pre-trained image quality model, the M-dimensional image quality of each image can be obtained. There is no requirement for the input order and number of images of the multiple images, that is, the images of the target object can be input to the pre-trained image quality model in any order, and at the same time, the number of images of the target object can be Any number.

Among them, the training process of the 1-dimensional image quality model is as follows:

11) Acquire a first image set and a second image set, the first image set includes multiple standard images, the second image set includes at least one sample image, and each sample in the at least one sample image The image and at least one standard image of the plurality of standard images contain the same elements;

12) Determine the similarity between each sample image in the at least one sample image and each standard image in the multiple standard images;

13) Determine the quality score of each sample image according to the similarity;

14) Input each sample image and the quality score into the model to be trained to obtain an image quality model.

Specifically, a correspondence relationship between the sample image and the quality score of the sample image may be established, and each sample image and the corresponding quality score are used as training samples. Then input the training samples into the model to be trained (such as the training model based on deep learning) to obtain the image quality model. Among them, the obtained trained image quality model can be used to obtain the 1-dimensional quality score of any image.

Among them, the training process of the multi-dimensional image quality model is as follows:

21) Acquire multiple sample images of the object to be trained, and acquire multiple standard images of the object to be trained;

22) Input the multiple sample images and the multiple standard images into a pre-trained feature recognition model to obtain image features of the object to be trained;

23) Input the multiple sample images and the multiple standard images into a preset training model to obtain the image quality of the object to be trained;

24) Calculate the fusion characteristics of the object to be trained according to the image characteristics of the object to be trained and the image quality of the object to be trained;

25) Input the fusion feature of the object to be trained into a preset loss function to obtain a loss value;

26) According to the loss value, use the back propagation algorithm to update the parameters of the training model;

27) If the loss value of the loss function reaches a convergence state, determine that the training model after updating the parameters is a trained image quality model.

Wherein, the sample image may be an image that is randomly taken from the object to be trained or intercepted from a video, and the standard image may be an ID photo of the object to be trained. The feature recognition model is pre-trained and will not be repeated here. A training model is preset first, and the parameters in the training model are all preset, and then the training model is trained to update the parameters to obtain an image quality model.

Wherein, the acquired multiple sample images and the multiple standard images can be mixed together and divided into several parts. The fusion features formed by different sets of image sets can be constructed according to the conventional recognition feature training scheme, such as SoftMax (ie normalized exponential function), Contrastive (ie contrast loss function) and Triplet (ie error function) and other losses Function, of course, can also build a unique loss function. Among them, if SoftMax or Contrastive is used, the multiple sample images and the multiple standard images after mixing need to be divided into n parts, n≥2, each part contains m images, m≥1, if used Triplet, the multiple sample images and the multiple standard images after mixing need to be divided into n parts, and n≥3 is required.

Further, the fusion feature of the object to be trained can be input to a preset loss function, a loss value is calculated, and according to the loss value, a parameter of the training model is updated using a back propagation method, and the above training steps are repeated Continue training. If the loss value of the loss function reaches the convergence state, you can determine that the training model with updated parameters is the trained image quality model. For example, if the loss value is stable at a small value, or if the loss value fluctuates slightly within a certain range, it can be determined that the loss value has reached a convergence state, and the training can be ended to obtain a trained image quality model.

S13. The electronic device inputs the multiple images into a pre-trained feature recognition model to obtain N-dimensional image features of each image.

There is no requirement on the input order and the number of images of the multiple images, that is, the images of the target object can be input to the pre-trained feature recognition model in any order, and at the same time, the number of images of the target object can be Any number.

S14. The electronic device performs weighted summation of the M-dimensional image quality and the N-dimensional image feature to obtain a fusion feature.

Wherein, M is a positive integer, and N is a positive integer, when M=1, N≥2, and when M≥2, M=N.

In the embodiment of the present invention, there are two ways to perform weighted summation on the M-dimensional image quality and the N-dimensional image features to obtain fusion features.

As an optional implementation manner, when M=1, the weighted summation of the M-dimensional image quality and the N-dimensional image feature to obtain a fusion feature includes:

For the jth image, the image quality of the jth image is multiplied by the i-dimensional image feature of the jth image to obtain the i-dimensional sub-feature of the jth image;

Summing the i-dimensional sub-features of multiple images to obtain the i-dimensional fusion feature;

Among them, i is a positive integer, j is a positive integer, and 1≤i≤N, j≥1.

f _i为第i维的融合特征，f _j,i为第j张图像的第i维图像特征，q _j为第j张图像的图像质量，K为图像的数量，K≥2，1≤i≤N，j≥1。 The above steps can be described by a first formula, and the first formula is:

f _i is the fusion feature of the i-th dimension, f _j,i is the i-dimensional image feature of the j-th image, q _j is the image quality of the j-th image, K is the number of images, K≥2, 1≤i ≤N, j≥1.

In this optional embodiment, the quality of each image of the target object is one-dimensional, and for each image, the image quality of the image and the image characteristics of the image may be multiplied, After that, all the multiplied results are summed to obtain the fusion feature of the target object.

Among them, in the entire process of using the first formula to calculate the fusion feature, the calculation amount is small and the calculation is simple. The final fusion feature can improve the image recognition effect, but it is also easy to add the wrong information to the fusion feature .

For example, assuming that the image feature is 3-dimensional, and there are 2 images of the same target object, the image feature of the first image is [0.5,0.3,0.2], where the image feature 0.2 of the third dimension is wrong, 1 The dimensional image quality is 0.5, the multi-dimensional image quality is [1.0,1.0,0.0], and the image feature of the second image is [0.2,0.3,0.5], where the image feature of the first dimension is 0.2 is wrong, 1 dimension The image quality is 0.5, and the multi-dimensional image quality is [0.0, 1.0, 1.0]. When using the 1D quality feature fusion scheme, according to the first formula, the 3rd feature of the first image and the 1st feature of the second image will be multiplied by the quality 0.5 to form the final feature, because of its own It is wrong and will introduce the error into the fused features.

Optionally, the fusion feature needs to be normalized to obtain the final feature.

Among them, for the fusion features obtained by the 1D image quality, a third formula is also needed to

Wherein, using the third formula to process the fusion feature, the modulus of the fusion feature can be 1, which reduces the impact on the overall fusion feature.

As an optional implementation manner, when M≥2 and M=N, the weighted summation of the M-dimensional image quality and the N-dimensional image feature to obtain the fusion feature includes:

For the jth image, the i-dimensional image quality of the j-th image is multiplied by the i-dimensional image feature of the j-th image to obtain the i-th dimension of the j-th image Sub-feature

Summing the i-dimensional sub-features of multiple images to obtain the i-dimensional feature sum;

Summing the i-dimensional image qualities of multiple images to obtain the image quality sum;

Dividing the feature sum of the i-th dimension by the sum of the image quality to obtain the fusion feature of the i-th dimension;

Among them, i is a positive integer, j is a positive integer, and 1≤i≤N, j≥1.

The above steps can be described by a second formula, and the second formula is:

f _i is the fusion feature of the i-th dimension, f _j,i is the i-dimensional image feature of the _j-th image, q _j,i is the i-dimensional image quality of the j-th image, K is the number of images, K≥ 2, 1≤i≤N, j≥1.

In this optional embodiment, the quality of each image of the target object is multi-dimensional, for each of the images, the image quality of the image and the image characteristics of the image may be multiplied, and then , After summing all the multiplied results, and finally, dividing the summation result by the sum of the image quality of the image, that is, normalizing the image quality of the image, which is conducive to reducing the image The multi-dimensional quality has an effect on the final fusion feature, which makes the final fusion feature more reasonable.

In the whole process of using the second formula to calculate the fusion feature, the amount of calculation is large and the calculation is complicated. However, the final fusion feature will not be doped with wrong information. At the same time, the final fusion feature can improve the image The effect of identification.

For example, assuming that the image feature is 3-dimensional, and there are 2 images of the same target object, the image feature of the first image is [0.5,0.3,0.2], where the image feature 0.2 of the third dimension is wrong, 1 The dimensional image quality is 0.5, the multi-dimensional image quality is [1.0,1.0,0.0], and the image feature of the second image is [0.2,0.3,0.5], where the image feature of the first dimension is 0.2 is wrong, 1 dimension The image quality is 0.5, and the multi-dimensional image quality is [0.0, 1.0, 1.0]. When using a multi-dimensional quality feature fusion scheme, according to the second formula, the 3rd feature of the first image and the 1st feature of the second image will be multiplied by the quality 0.0, so it will not cause the final feature influences.

Optionally, the fusion feature needs to be normalized to obtain the final feature.

Among them, for the fusion features obtained by the multi-dimensional image quality, a third formula is also needed for the fusion

Wherein, the fusion feature obtained by multi-dimensional image quality is processed by using the third formula, so that the modulus of the fusion feature is 1, and the influence on the overall fusion feature is reduced.

In the method flow described in FIG. 1, multiple images of a target object can be obtained first, and the multiple images are input into a pre-trained image quality model to obtain the M-dimensional image quality of each image, and the The multiple images are input into a pre-trained feature recognition model to obtain the N-dimensional image features of each image. Further, the M-dimensional image quality and the N-dimensional image features may be weighted and summed to obtain fusion Features; wherein, M is a positive integer, the N is a positive integer, when the M = 1, N ≥ 2, when the M ≥ 2, M = N. It can be seen that in the present invention, after acquiring multiple images of the target object, the image quality and image features of the multiple images can be extracted through the image quality model and the feature recognition model, and then, the image quality and image features of all images are processed The weighted sum can obtain the fusion feature. Since the fusion feature is obtained by fusing multiple image features of multiple images and multiple image qualities, the fusion feature can include all the features of the target object, relative to As far as single image features are concerned, the fusion feature makes up for the lack of certain image features of the target object in the single image feature. When performing image recognition, the fusion feature can be used to identify the image in all directions, which can improve the image Recognition effect, recognition accuracy is higher.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited to this. For those of ordinary skill in the art, without departing from the inventive concept of the present invention, they can also make Improvement, but these all belong to the protection scope of the present invention.

Please refer to FIG. 2, which is a functional block diagram of a preferred embodiment of a feature fusion device disclosed in the present invention.

In some embodiments, the feature fusion device runs in an electronic device. The feature fusion device may include multiple functional modules composed of program code segments. The program codes of each program segment in the feature fusion device may be stored in a memory and executed by at least one processor to perform some or all of the steps in the feature fusion method described in FIG. 1, for details, refer to FIG. 1 Relevant descriptions in will not be repeated here.

In this embodiment, the feature fusion device may be divided into multiple functional modules according to the functions it performs. The functional module may include: an acquisition module 201, an input module 202, and a calculation module 203. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can perform fixed functions, and are stored in a memory. In some embodiments, the functions of each module will be described in detail in subsequent embodiments.

The feature fusion device includes:

The first acquisition module 201 is used to acquire multiple images of the target object;

The input module 202 is used to input the multiple images into a pre-trained image quality model to obtain the M-dimensional image quality of each image;

The input module 202 is further configured to input the multiple images into a pre-trained feature recognition model to obtain N-dimensional image features of each image;

The calculation module 203 is configured to perform weighted summation of the M-dimensional image quality and the N-dimensional image feature to obtain a fusion feature; wherein, M is a positive integer, and N is a positive integer, when the M= When 1, N≥2, and when M≥2, M=N.

Optionally, the manner in which the first acquiring module 201 acquires multiple images of the target object is specifically:

Capture multiple images of the target object from the video of the target object; or

Acquire multiple images of the target object taken at different times.

Optionally, the calculation module 203 performs weighted summation of the quality of the M-dimensional image and the feature of the N-dimensional image to obtain a fusion feature specifically as follows:

For the jth image, the image quality of the jth image is multiplied by the i-dimensional image feature of the jth image to obtain the i-dimensional sub-feature of the jth image;

Summing the i-dimensional sub-features of multiple images to obtain the i-dimensional fusion feature;

Among them, i is a positive integer, j is a positive integer, and 1≤i≤N, j≥1.

Optionally, the calculation module 203 performs weighted summation of the quality of the M-dimensional image and the feature of the N-dimensional image to obtain a fusion feature specifically as follows:

For the jth image, the i-dimensional image quality of the j-th image is multiplied by the i-dimensional image feature of the j-th image to obtain the i-th dimension of the j-th image Sub-feature

Summing the i-dimensional sub-features of multiple images to obtain the i-dimensional feature sum;

Summing the i-dimensional image qualities of multiple images to obtain the image quality sum;

Dividing the feature sum of the i-th dimension by the sum of the image quality to obtain the fusion feature of the i-th dimension;

Among them, i is a positive integer, j is a positive integer, and 1≤i≤N, j≥1.

Optionally, the feature fusion device further includes:

The processing module is used for normalizing the fusion feature to obtain the final feature.

Optionally, the feature fusion device further includes:

A second acquisition module, configured to acquire multiple sample images of the object to be trained, and acquire multiple standard images of the object to be trained;

The input module 202 is further configured to input the multiple sample images and the multiple standard images into a pre-trained feature recognition model to obtain image features of the object to be trained;

The input module 202 is further configured to input the multiple sample images and the multiple standard images into a preset training model to obtain the image quality of the object to be trained;

The calculation module 203 is further configured to calculate the fusion characteristics of the object to be trained according to the image characteristics of the object to be trained and the image quality of the object to be trained;

The input module 202 is further configured to input the fusion feature of the object to be trained into a preset loss function to obtain a loss value;

The update module is used to update the parameters of the training model based on the loss value using a back propagation algorithm;

The determining module is used to determine that the training model after updating the parameters is a trained image quality model if the loss value of the loss function reaches a convergence state.

Optionally, the input order of the multiple images of the target object and the number of images have no effect on the fusion feature.

In the feature fusion device described in FIG. 2, multiple images of the target object can be acquired first, and the multiple images are input into a pre-trained image quality model to obtain the M-dimensional image quality of each image, and The multiple images are input into a pre-trained feature recognition model to obtain N-dimensional image features of each image. Further, the quality of the M-dimensional image and the N-dimensional image features may be weighted and summed to obtain Fusion feature; wherein, M is a positive integer, and N is a positive integer, when M=1, N≥2, and when M≥2, M=N. It can be seen that in the present invention, after acquiring multiple images of the target object, the image quality and image features of the multiple images can be extracted through the image quality model and the feature recognition model, and then, the image quality and image features of all images are processed The weighted sum can obtain the fusion feature. Since the fusion feature is obtained by fusing multiple image features of multiple images and multiple image qualities, the fusion feature can include all the features of the target object, relative to As far as single image features are concerned, the fusion feature makes up for the lack of certain image features of the target object in the single image feature. When performing image recognition, the fusion feature can be used to identify the image in all directions, which can improve the image Recognition effect, recognition accuracy is higher.

As shown in FIG. 3, FIG. 3 is a schematic structural diagram of an electronic device of a preferred embodiment of a method for implementing feature fusion according to the present invention. The electronic device 3 includes a memory 31, at least one processor 32, a computer program 33 stored in the memory 31 and executable on the at least one processor 32, and at least one communication bus 34.

Those skilled in the art may understand that the schematic diagram shown in FIG. 3 is only an example of the electronic device 3, and does not constitute a limitation on the electronic device 3, and may include more or less components than the illustration, or a combination Certain components, or different components, for example, the electronic device 3 may further include input and output devices, network access devices, and the like.

The electronic device 3 also includes, but is not limited to, any electronic product that can interact with a user through a keyboard, a mouse, a remote control, a touchpad, or a voice control device, such as a personal computer, tablet computer, smart phone, Personal digital assistant (Personal Digital Assistant, PDA), game console, interactive network TV (Internet Protocol, IPTV), smart wearable device, etc. The network where the electronic device 3 is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (Virtual Private Network, VPN), etc.

The at least one processor 32 may be a central processing unit (Central Processing Unit, CPU), or may be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC ), ready-made programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 32 may be a microprocessor or the processor 32 may also be any conventional processor, etc. The processor 32 is the control center of the electronic device 3, and uses various interfaces and lines to connect the entire electronic device 3 The various parts.

The memory 31 may be used to store the computer program 33 and/or module/unit, and the processor 32 executes or executes the computer program and/or module/unit stored in the memory 31, and calls the stored in the memory The data in 31 realizes various functions of the electronic device 3. The memory 31 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, application programs required by at least one function (such as a sound playback function, an image playback function, etc.), etc.; the storage data area may Data (such as audio data, phone book, etc.) created according to the use of the electronic device 3 is stored. In addition, the memory 31 may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart) Media, Card (SMC), and a secure digital (SD) Card, flash card (Flash), at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.

With reference to FIG. 1, the memory 31 in the electronic device 3 stores multiple instructions to implement a feature fusion method, and the processor 32 can execute the multiple instructions to achieve:

Acquire multiple images of the target object;

Input the multiple images into a pre-trained image quality model to obtain the M-dimensional image quality of each image;

Input the multiple images into a pre-trained feature recognition model to obtain N-dimensional image features of each image;

Weighting and summing the M-dimensional image quality and the N-dimensional image feature to obtain a fusion feature; wherein, M is a positive integer, the N is a positive integer, and when M=1, N≥2 When M≥2, M=N.

In an optional implementation manner, the acquiring multiple images of the target object includes:

Capture multiple images of the target object from the video of the target object; or

Acquire multiple images of the target object taken at different times.

In an optional embodiment, when the M=1, the weighted summation of the M-dimensional image quality and the N-dimensional image feature to obtain the fusion feature includes:

For the jth image, the image quality of the jth image is multiplied by the i-dimensional image feature of the jth image to obtain the i-dimensional sub-feature of the jth image;

Summing the i-dimensional sub-features of multiple images to obtain the i-dimensional fusion feature;

Among them, i is a positive integer, j is a positive integer, and 1≤i≤N, j≥1.

In an optional implementation manner, when M≥2 and M=N, the weighted summation of the M-dimensional image quality and the N-dimensional image features to obtain the fusion feature includes:

For the jth image, the i-dimensional image quality of the j-th image is multiplied by the i-dimensional image feature of the j-th image to obtain the i-th dimension of the j-th image Sub-feature

Summing the i-dimensional sub-features of multiple images to obtain the i-dimensional feature sum;

Summing the i-dimensional image qualities of multiple images to obtain the image quality sum;

Dividing the feature sum of the i-th dimension by the sum of the image quality to obtain the fusion feature of the i-th dimension;

Among them, i is a positive integer, j is a positive integer, and 1≤i≤N, j≥1.

In an optional embodiment, the processor 32 can execute the multiple instructions to implement:

Perform normalization processing on the fusion feature to obtain a final feature.

In an optional implementation manner, before acquiring multiple images of the target object, the processor 32 may execute the multiple instructions to implement:

Acquiring multiple sample images of the object to be trained, and acquiring multiple standard images of the object to be trained;

Input the multiple sample images and the multiple standard images into a pre-trained feature recognition model to obtain image features of the object to be trained;

Input the plurality of sample images and the plurality of standard images into a preset training model to obtain the image quality of the object to be trained;

Calculate the fusion characteristics of the object to be trained according to the image characteristics of the object to be trained and the image quality of the object to be trained;

Input the fusion feature of the object to be trained into a preset loss function to obtain a loss value;

According to the loss value, use a back propagation algorithm to update the parameters of the training model;

If the loss value of the loss function reaches a convergence state, it is determined that the training model after updating the parameters is a trained image quality model.

In an optional embodiment, the input sequence of the multiple images of the target object and the number of images have no effect on the fusion feature.

Specifically, for the specific implementation method of the above instruction by the processor 32, reference may be made to the description of relevant steps in the embodiment corresponding to FIG. 1, and details are not described herein.

In the electronic device 3 described in FIG. 3, multiple images of the target object can be acquired first, and the multiple images are input into a pre-trained image quality model to obtain the M-dimensional image quality of each image, and The multiple images are input into a pre-trained feature recognition model to obtain N-dimensional image features of each image. Further, the quality of the M-dimensional image and the N-dimensional image features may be weighted and summed to obtain Fusion feature; wherein, M is a positive integer, and N is a positive integer, when M=1, N≥2, and when M≥2, M=N. It can be seen that in the present invention, after acquiring multiple images of the target object, the image quality and image features of the multiple images can be extracted through the image quality model and the feature recognition model, and then, the image quality and image features of all images are processed The weighted sum can obtain the fusion feature. Since the fusion feature is obtained by fusing multiple image features of multiple images and multiple image qualities, the fusion feature can include all the features of the target object, relative to As far as single image features are concerned, the fusion feature makes up for the lack of certain image features of the target object in the single image feature. When performing image recognition, the fusion feature can be used to identify the image in all directions, which can improve the image Recognition effect, recognition accuracy is higher.

If the integrated module/unit of the electronic device 3 is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the present invention can realize all or part of the processes in the methods of the above embodiments, and can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the program is executed by the processor, the steps of the foregoing method embodiments may be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in a source code form, an object code form, an executable file, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disc, a computer memory, and a read-only memory (ROM, Read-Only Memory) , Random Access Memory (RAM, Random Access Memory), electrical carrier signals, telecommunications signals and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in jurisdictions. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media Does not include electrical carrier signals and telecommunications signals.

Claims (10)

A feature fusion method, characterized in that the method includes: Acquire multiple images of the target object; Input the multiple images into a pre-trained image quality model to obtain the M-dimensional image quality of each image; Input the multiple images into a pre-trained feature recognition model to obtain N-dimensional image features of each image; Weighting and summing the M-dimensional image quality and the N-dimensional image feature to obtain a fusion feature; wherein, M is a positive integer, the N is a positive integer, and when M=1, N≥2 When M≥2, M=N. The method according to claim 1, wherein the acquiring multiple images of the target object comprises: Capture multiple images of the target object from the video of the target object; or Acquire multiple images of the target object taken at different times. The method according to claim 1, wherein when M=1, the weighted summation of the M-dimensional image quality and the N-dimensional image feature to obtain a fusion feature includes: For the jth image, the image quality of the jth image is multiplied by the i-dimensional image feature of the jth image to obtain the i-dimensional sub-feature of the jth image; Summing the i-dimensional sub-features of multiple images to obtain the i-dimensional fusion feature; Among them, i is a positive integer, j is a positive integer, and 1≤i≤N, j≥1. The method according to claim 1, wherein when M≥2 and M=N, the weighted sum of the M-dimensional image quality and the N-dimensional image feature to obtain the fusion feature includes : For the jth image, the i-dimensional image quality of the j-th image is multiplied by the i-dimensional image feature of the j-th image to obtain the i-th dimension of the j-th image Sub-feature Summing the i-dimensional sub-features of multiple images to obtain the i-dimensional feature sum; Summing the i-dimensional image qualities of multiple images to obtain the image quality sum; Dividing the feature sum of the i-th dimension by the sum of the image quality to obtain the fusion feature of the i-th dimension; Among them, i is a positive integer, j is a positive integer, and 1≤i≤N, j≥1. The method according to any one of claims 1 to 4, wherein the method further comprises: Perform normalization processing on the fusion feature to obtain a final feature. The method according to any one of claims 1 to 4, wherein before the acquiring multiple images of the target object, the method further comprises: Acquiring multiple sample images of the object to be trained, and acquiring multiple standard images of the object to be trained; Input the multiple sample images and the multiple standard images into a pre-trained feature recognition model to obtain image features of the object to be trained; Input the plurality of sample images and the plurality of standard images into a preset training model to obtain the image quality of the object to be trained; Calculate the fusion characteristics of the object to be trained according to the image characteristics of the object to be trained and the image quality of the object to be trained; Input the fusion feature of the object to be trained into a preset loss function to obtain a loss value; According to the loss value, use a back propagation algorithm to update the parameters of the training model; If the loss value of the loss function reaches a convergence state, it is determined that the training model after updating the parameters is a trained image quality model. The method according to any one of claims 1 to 4, wherein the input order of the plurality of images of the target object and the number of images have no effect on the fusion feature. A feature fusion device, characterized in that the feature fusion device includes: The acquisition module is used to acquire multiple images of the target object; The input module is used to input the multiple images into a pre-trained image quality model to obtain the M-dimensional image quality of each image; The input module is further configured to input the multiple images into a pre-trained feature recognition model to obtain N-dimensional image features of each image; A calculation module, configured to perform weighted summation of the M-dimensional image quality and the N-dimensional image feature to obtain a fusion feature; wherein, M is a positive integer, the N is a positive integer, and when M=1 , N≥2, when M≥2, M=N. An electronic device, characterized in that the electronic device includes a processor and a memory, and the processor is used to execute a computer program stored in the memory to implement the feature fusion method according to any one of claims 1 to 7. A computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the feature fusion according to any one of claims 1 to 7 is realized method.

Images