CN104462489B - A kind of cross-module state search method based on Deep model - Google Patents

Abstract

The present invention proposes a cross-modal retrieval method based on a deep model, which includes: using a feature extraction method to obtain the target retrieval modality and the low-level expression vector of each retrieved modality in the retrieval database; the target retrieval modality The low-level expression vectors are respectively associated with the low-level expression vectors of each retrieved modality in the retrieval library, and the high-level expression vectors of the target retrieval modality and each of the retrieved modality are obtained by stacking the corresponding Restricted Boltzmann Machine Corr‑RBMs deep model. The high-level expression vector of the retrieved modality; using the high-level expression vector of the target retrieval modality and the high-level expression vector of each retrieved modality in the retrieval library to calculate the distance between the target retrieval modality and each retrieved modality in the retrieval library; Determining at least one retrieved modality closest to the target retrieval modality in the retrieval library as an object matching the target retrieval modality.

Description

A Deep Model-Based Cross-Modal Retrieval Method

technical field

The invention relates to multimedia retrieval technology, in particular to a cross-modal retrieval method based on a deep model.

Background technique

In recent years, the development of the Internet has resulted in an explosive growth of multimodal data. For example, products on e-commerce websites usually contain main text, short text descriptions, and related pictures; pictures shared on social networking sites are usually accompanied by tagged descriptors; pictures and video information contained in some online news are more than simple Text reports are more attractive, and the rapid growth of multimodal data has brought about a huge demand for cross-modal retrieval.

Different from traditional unimodal retrieval, cross-modal retrieval pays more attention to the relationship between different modalities. Therefore, the cross-modal retrieval problem contains two challenges: one is that the data from different modalities have completely different statistical properties, which makes it difficult to directly obtain the association relationship of different modal data; The extracted features usually have high-dimensional characteristics and the size of the data set is very large, which makes efficient retrieval difficult to achieve.

Contents of the invention

In view of this, the present invention provides a cross-modal retrieval method based on a deep model, which uses a deep model to solve the processing problem of cross-modal data, so that the cross-modal data processed by the deep model can efficiently perform distance calculation, So as to get better search results. The technical scheme that the present invention proposes is:

A deep model-based cross-modal retrieval method comprising:

Using the feature extraction method to obtain the low-level expression vector of the target retrieval modality and each retrieved modality in the retrieval database;

The low-level expression vectors of the target retrieval modality are respectively connected with the low-level expression vectors of each retrieved modality in the retrieval library, and the target retrieval modality is obtained by stacking the corresponding Restricted Boltzmann Machine Corr-RBMs deep model The high-level expression vector of the state and the high-level expression vector of each retrieved modality in the retrieval library;

calculating the distance between the target retrieval modality and each retrieved modality in the retrieval library by using the high-level expression vector of the target retrieval modality and the high-level expression vector of each retrieved modality in the retrieval library;

Determining at least one retrieved modality closest to the target retrieval modality in the retrieval library as an object matching the target retrieval modality.

To sum up, the technical solution of the present invention proposes a deep model-based cross-modal retrieval method. For the low-level representation obtained by feature extraction of cross-modal raw data, the corresponding Restricted Boltzmann Machine (Corr -RBM, Correspondence Restricted Boltzmann Machine)'s Corr-RBMs deep model processing, to obtain the low-dimensional high-level expression of cross-modal data in the same representation space, and then perform distance calculation on the low-dimensional high-level expression of cross-modal data, according to the distance Confirm the search results.

Description of drawings

Fig. 1 is the flowchart of technical scheme of the present invention;

Fig. 2 is the neural network structure diagram of Corr-RBMs deep layer model of the present invention;

Fig. 3 is a Corr-RBM model neural network structural diagram of the present invention;

Fig. 4 is a structural diagram of a restricted Boltzmann machine RBM model;

Fig. 5 is the method flow chart that determines Θ according to objective function F;

Fig. 6 is a flowchart of an embodiment of the present invention.

Detailed ways

In order to solve the cross-modal retrieval problem, the present invention proposes a cross-modal retrieval method based on the Corr-RBMs deep model. The flow chart of the technical solution of the present invention is shown in Figure 1, including the following steps:

Step 101: Obtain the low-level expression vectors of the target retrieval modality and any retrieved modality in the retrieval database by feature extraction method.

In this step, in order to retrieve objects that match the target retrieval modality in the retrieval database, firstly, the low-level expression vectors of the target retrieval modality and any retrieved modality in the retrieval database are needed. The low-level expression vectors obtained by the feature extraction method are generally The dimensionality is high, and the elements of the low-level expression vectors of different modalities are different, and generally cannot be directly used for retrieval operations.

Step 102: The low-level expression vector of the target retrieval modality is separately connected with the low-level expression vector of each retrieved modality in the retrieval library, and the high-level expression vector of the target retrieval modality is obtained by stacking the corresponding Restricted Boltzmann Machine Corr-RBMs deep model Expression vectors and high-level expression vectors for each retrieved modality in the retrieval library.

In this step, the low-level expression vector of the target retrieval modality is combined with the low-level expression vector of each retrieved modality in the retrieval library, and the target is obtained by stacking the corresponding Restricted Boltzmann Machine Corr-RBMs deep model The high-level representation vector of the retrieval modality and the high-level representation vector of each retrieved modality in the retrieval library. The high-level expression vector of the target retrieval modality obtained through the Corr-RBMs deep model and the high-level expression vector of each retrieved modality in the retrieval database have the characteristics of low dimensionality and consistent spatial elements, and can perform retrieval operations efficiently.

Step 103: Using the high-level expression vector of the target retrieval modality and the high-level expression vector of each retrieved modality in the retrieval database to calculate the distance between the target retrieval modality and any retrieved modality in the retrieval database.

Specifically, the Euclidean distance can be used to represent the distance between the target retrieval modality and each retrieved modality in the retrieval library.

Step 104: Determine at least one retrieved modality closest to the target retrieval modality in the retrieval database as an object matching the target retrieval modality.

In this step, the distance between each retrieved modality in the retrieval database and the target retrieval modality is sorted, and at least one retrieved modality closest to the target retrieval modality is selected as an object matching the target retrieval modality.

The present invention proposes a method for cross-modal retrieval using the Corr-RBMs deep model of the stacked Corr-RBM, and Fig. 2 is a neural network structure diagram of the Corr-RBMs deep model neural network of the stacked Corr-RBM of the present invention, as shown in Fig. 2 , the Corr-RBMs deep model is stacked by at least two layers of Corr-RBM models, and the Corr-RBMs deep model can obtain the high-level expression of the original data of two different modalities from the low-level expression of the original data of two different modalities; each The neural network structure diagram of the layered Corr-RBM model is shown in Figure 3. The Corr-RBM model is established on the basis of the RBM of the restricted Boltzmann machine, and Figure 4 is the neural network structure diagram of the restricted Boltzmann machine , the RBM model, Corr-RBM model and Corr-RBMs deep model are introduced in detail below.

(1) RBM model:

Fig. 4 is the neural network structure diagram of RBM. As shown in Fig. 4, the visible layer V of RBM includes m neural units v ₁ ~ v _m , and the bias of each neuron unit v _i is b _i . There is no connection; the hidden layer H contains s neuron units h ₁ ～h _s , the bias of each neuron unit h _j is c _j , and there is no connection between the neuron units in the visible layer; the neuron unit v _i in the visible layer is connected to the neuron unit in the hidden layer The connection weight of h _j is w _ij . For ease of understanding, only part of the connection weights between visible layer neural units and hidden layer neural units are drawn in FIG. 4 .

RBM has an undirected graph structure, with a Logistic activation function δ(x)=1/(1+exp(-x)), then the joint probability distribution of the visible layer V and hidden layer H neural units is:

Among them, Z is a normalization constant, and E(v,h) is an energy function defined by different configurations of visible layer neural units and hidden layer neural units of RBM. According to different configurations of visible layer neural units and hidden layer neural units, E(v,h) has different representations, that is, as long as the configuration of the visible layer neuron unit and the hidden layer neuron unit of the RBM are determined, there will be a corresponding energy function, which will not be described in detail here.

The bias b _i of the visible layer neuron unit v _i of RBM, the bias c _j of the hidden layer neuron unit h _j , the connection weight w _ij of the visible layer neuron unit v _i and the hidden layer neuron unit h _j can be learned by comparing The divergence estimation algorithm is obtained, compared with the divergence estimation algorithm, which is a relatively mature prior art, and will not be introduced in detail here.

(2) The corresponding restricted Boltzmann machine Corr-RBM model:

Fig. 3 is the structural diagram of Corr-RBM model of the present invention, as shown in Fig. 3, Corr-RBM model comprises the first mode RBM and the second mode RBM, and the first mode RBM and the second mode RBM comprise the same The number m of neuron units in the visible layer and the same number s of neuron units in the hidden layer, and there is a correlation constraint between the hidden layer of the first modality RBM and the second modality RBM.

It is assumed that Θ represents the parameter set of the Corr-RBM model, that is, Θ={W ^I , C ^I , B ^I , W ^T , C ^T , B ^T }, where the superscript I represents the first mode, and the superscript T represents the first mode. Two modes, specifically, W ^I is the connection weight parameter set between each visible layer neuron unit and hidden layer neuron unit of the first mode RBM, and C ^I is the bias of the visible layer neuron unit of the first mode RBM Parameter set, B ^I is the hidden layer neuron unit bias parameter set of the first modality RBM, W ^T is the connection weight parameter set between each visible layer neuron unit and the hidden layer neuron unit of the second modality RBM, C ^T is the visible layer neuron unit bias parameter set of the second modality RBM, and B ^T is the hidden layer neuron unit bias parameter set of the second modality RBM.

The parameter set Θ of the Corr-RBM model is determined by the following parameter learning algorithm:

The objective function F is defined according to the following principle: the parameter set Θ of the Corr-RBM model can minimize the distance between the first modality and the second modality in the shared representation space, and minimize the distance between the first modality and the second modality Negative log-likelihood function. The objective function F is F=l _D +αl _I +βl _T , that is, Θ is the set of parameters that minimizes F.

in,

Among them, l _D is the distance between the first mode and the second mode in the nesting space, l _I is the negative log-likelihood function of the first mode, and l _T is the negative log-likelihood of the second mode function, α and β are constants, α∈(0,1), β∈(0,1); f _I (·) is the mapping function from the visible layer to the hidden layer of the first modality RBM, and f _T (·) is The mapping function from the second modality RBM visible layer to the hidden layer; p _I (·) is the joint probability distribution of the first modality RBM visible layer and the hidden layer neural unit, p _T (·) is the second modality RBM visible layer and the joint probability distribution of hidden layer neural units, ||·|| is a two-norm mapping.

In order to determine Θ according to the objective function F, an alternate iterative optimization process can be used. First, the two likelihood functions l _I and l _T are updated using the contrastive divergence estimation algorithm, and then the gradient descent method is used to update l _D . The convergence can be On the verification set, use cross-modal retrieval to detect, specifically, Fig. 5 is a flowchart of determining Θ according to the objective function F, including the following steps:

Step 501: Utilize the contrastive divergence estimation algorithm to update the parameters of the first mode RBM.

The set of connection weight parameters between the visible layer neuron unit and the hidden layer neuron unit of the first mode RBM Visible layer neural unit bias and hidden layer neurons bias Expressed uniformly by θ ^I , updated according to the formula θ ^I ←θ ^I +τ·α·△θ ^I , where τ is the learning rate, τ∈(0,1); α∈(0,1); and,

Among them, <·> _data is the mathematical expectation under the empirical distribution, <·> _model is the mathematical expectation under the model distribution;

Step 502: Utilize the contrast divergence estimation algorithm to update the parameters of the second mode RBM.

A set of connection weight parameters between the visible layer neuron unit and the hidden layer neuron unit of the second modality RBM Visible layer neural unit bias and hidden layer neurons bias Expressed uniformly by θ ^T , and updated according to the formula θ ^T ←θ ^T +τ·β·△θ ^T , where β∈(0,1); and,

Step 503: Using gradient descent method to update the distance between the first mode and the second mode in the nesting space.

Specifically, the distance l _D between the first mode and the second mode in the nesting space is updated using the method of gradient descent according to the following formula:

Wherein, δ'(·)=δ(·)(1-δ(·)), and δ(·) is the Logistic activation function δ(x)=1/(1+exp(-x)).

Step 504: Repeat steps 501-503 until the algorithm converges.

The parameter set Θ of the Corr-RBM model can be obtained by the above method.

(3) Corr-RBMs deep model

Figure 2 is a neural network structure diagram of the Corr-RBMs deep model. Including the first modal Corr-RBMs and the second modal Corr-RBMs, the first modal Corr-RBMs deals with the low-level expression of the target retrieval modality, and the second modal Corr-RBMs deals with any retrieved modality in the retrieval library low-level expression.

The input of the first modality RBM visible layer neural unit of the underlying Corr-RBM is the low-level expression of the first modality obtained by feature extraction from the raw data of the first modality, and the second modality RBM visible layer neural unit of the underlying Corr-RBM The input is the low-level representation of the first modality obtained through feature extraction from the raw data of the second modality, and the low-level representation obtained from the raw data through specific extraction is a prior art, which will not be described in detail here.

The first RBM hidden layer of the top Corr-RBM outputs a high-level representation of the first modality, and the second RBM hidden layer of the top Corr-RBM outputs a high-level representation of the second modality.

In order to make the object, technical solution and advantages of the present invention more clearly, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

In this embodiment, it is assumed that the retrieval database includes N retrieved modalities, and the technical solution of the present invention is described by taking the retrieval of an object related to the picture P in the retrieval database as an example. Figure 6 is a flow chart of this embodiment, as shown in Figure 6 , including the following steps:

Step 601: Obtain the low-level representation of each retrieved modality in the retrieval database and the low-level representation of the picture P by using a feature extraction method.

In this step, the modal types of the retrieved modalities in the retrieval database are not limited. They may be image modalities, text modalities, or voice modalities. The original data of different modalities currently have relatively mature Feature extraction methods, for example, MPEG-7 and Gist descriptors can be used for feature extraction for image modality, feature extraction can be done for text modality using bag-of-words model, etc. The process of low-level expression of the state is described in detail.

Step 602: The low-level expression of the picture P and the low-level expression of each retrieved modality in the retrieval database are respectively processed through the Corr-RBMs deep model to obtain the high-level expression of the picture P and the high-level expression of each retrieved modality in the retrieval database , and then use the high-level expression of the picture P and the high-level expression of each retrieved modality in the retrieval database to perform Euclidean distance calculation, and calculate the Euclidean distance between the picture P and each retrieved modality in the retrieval database.

In this step, any retrieved modality and picture P in the retrieval library are taken as a combination, and the low-level expression of the retrieved modality in the combination and the low-level expression of the picture P are processed through the Corr-RBMs deep model to obtain the The high-level representation of the retrieved modality, the high-level representation of the picture P, and then calculate the Euclidean distance between the picture P and the retrieved modality according to the Euclidean distance calculation formula.

Generally, for two points t and y in n-dimensional Euclidean space, the formula for calculating their distance d is Calculate the Euclidean distance between the picture P and any retrieved modality.

Step 603: Sort according to the Euclidean distance between the picture P and each retrieved modality in the retrieval library from low to high, and select the top K retrieved modalities as the retrieval result output.

In this embodiment, the low-level expression of the image modality and the low-level expression of each retrieved modality in the retrieval library are processed through the Corr-RBMs deep model to obtain their respective high-level expressions, and then the Euclidean distance calculation can be efficiently obtained by using the high-level expressions Search Results.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (3)

1. A cross-modal retrieval method based on a deep model, wherein the deep model is a stack-up corresponding restricted Boltzmann machine (Corr-RBMs) deep model, and the method comprises the following steps:

respectively obtaining a target retrieval mode and a low-level expression vector of each retrieved mode in a retrieval library by using a feature extraction method;

the low-level expression vector of the target retrieval mode is respectively matched with the low-level expression vector of each retrieved mode in the retrieval library, and the high-level expression vector of the target retrieval mode and the high-level expression vector of each retrieved mode in the retrieval library are obtained by stacking corresponding limited Boltzmann machine Corr-RBMs deep models;

calculating the distance between the target retrieval modality and each retrieved modality in the retrieval library by using the high-level expression vector of the target retrieval modality and the high-level expression vector of each retrieved modality in the retrieval library;

determining at least one retrieved modality in the retrieval library which is closest to the target retrieval modality as an object matched with the target retrieval modality;

wherein,

the deep layer models of the Corr-RBMs are formed by stacking at least two layers of corresponding limited Boltzmann machine Corr-RBMs, each deep layer model of the Corr-RBMs comprises first-mode Corr-RBMs and second-mode Corr-RBMs, the first-mode Corr-RBMs process the low-level expression vectors of the target retrieval modes, and the second-mode Corr-RBMs process the low-level expression vectors of any retrieved mode in the retrieval library;

the Corr-RBM includes a first-mode restricted boltzmann machine RBM and a second-mode restricted boltzmann machine RBM, the first-mode RBM and the second-mode RBM include the same number of visible layer neural units and the same number of hidden layer neural units, and a hidden layer of the first-mode RBM and the second-mode RBM has a dependency constraint therebetween.

2. The method of claim 1, further comprising:

configuration parameters theta = { W of Corr-RBM ^I ,C ^I ,B ^I ,W ^T ,C ^T ,B ^T Wherein superscript I denotes a first modality, superscript T denotes a second modality, in particular W ^I A set of connection weight parameters between the visible layer neural units and the hidden layer neural units of the RBM of the first modality, C ^I Set of visible layer neural unit bias parameters for RBM of first modality, B ^I Hidden layer neural cell bias parameter set, W, for first modality RBM ^T A set of connection weight parameters between the visible layer neural units and the hidden layer neural units of the RBM of the second modality, C ^T Set of visible layer neural unit bias parameters for RBM of second modality, B ^T A hidden layer neural unit bias parameter set of a second mode RBM;

the configuration parameters theta of the corresponding restricted Boltzmann machine Corr-RBM are order objective functionsMinimum configuration parameters, and

wherein,the distance between the first mode and the second mode on the nesting space,is a negative log-likelihood function of the first modality,a negative log-likelihood function for the second modality; α and β are constants, and α ∈ (0,1), β ∈ (0,1); f. of _I (. Is a first modality RBM visible layer to hidden layer mapping function, f _T () and a second modality RBM visible layer to hidden layer mapping function; p is a radical of _I (.) a joint probability distribution of the RBM visible layer and hidden layer neural units in the first modality, p _T () a joint probability distribution of the visible layer and hidden layer neural units for the second modality RBM; | | · | | is a two-norm mapping; v refers to a visible unit in the RBM, corresponding to a visible variable; m is the number of modal samples.

3. The method according to claim 2, wherein the algorithm for determining Θ from the objective function F is:

A. set of connection weight parameters between visible layer neural units and hidden layer neural units of first modality RBMVisible layer nerve cellIs offset fromAnd hidden layer nerve cellIs offset fromBy theta ^I Expressed uniformly according to the formula theta ^I ←θ ^I +τ·α·△θ ^I Updating, wherein tau is a learning rate and tau e (0,1); α ∈ (0,1);and,

wherein,<·> _data for the mathematical expectations under the empirical distribution,<·> _model is a mathematical expectation under a model distribution;

B. set of connection weight parameters between visible layer neural units and hidden layer neural units of second modality RBMVisible layer nerve cellIs offset fromAnd hidden layer nerve unitIs offset fromBy theta ^T Expressed uniformly according to the formula theta ^T ←θ ^T +τ·β·△θ ^T Updating, wherein the beta epsilon (0,1);and,

C. updating using a gradient descent method according to the following formula

Wherein δ' (·) = δ (·) (1- δ (·)), and δ (·) is a Logistic activation function δ (x) = 1/(1 + exp (-x));

and repeating the steps A to C until the algorithm converges.