CN103440269B - A kind of video data search method based on study mutually - Google Patents

Abstract

The invention relates to a video data retrieval method based on mutual learning, which is characterized in that: calculating the similarity matrix of different types of video data features, and using the similarity matrix to calculate the Laplacian matrix; calculating the Laplacian of different types of video data The eigenvalues and eigenvectors of the matrix are the eigenvectors corresponding to the first M largest eigenvalues in the Laplacian matrix; the similarity matrix of different types of video data eigenvectors is calculated, and the corresponding elements of the similarity matrix of the eigenvectors are compared Multiply the learning matrix; multiply the learning matrix and the corresponding elements of the similarity matrix of each feature to obtain the learned similarity matrix; use the learned similarity matrix to sort the video data, and count the first few sorted In the video data, the number of video data belonging to the same category as the query target video data is used to obtain the corresponding retrieval accuracy. In the method of the invention, the retrieval accuracy rate is greatly improved compared with that before learning.

Description

A Video Data Retrieval Method Based on Mutual Learning

technical field

The invention relates to a video data retrieval method based on mutual learning, which can be applied to the retrieval of different types of video data.

Background technique

The rapid development of Internet technology and digital photography technology makes more and more video data on the network, and video data retrieval has become a hot and difficult problem in multimedia technology. Scholars at home and abroad have proposed a variety of features to retrieve video data. Most of these features are based on the color, texture, and shape of video data, collectively referred to as underlying features. In recent years, some scholars have proposed brain function features based on human brain cognition. Compared with the bottom-level features, the retrieval accuracy has been improved, but we found that different types of features reflect different characteristics of video data. If the advantages of these features can be combined, the retrieval accuracy will be greatly improved.

Contents of the invention

technical problem to be solved

In order to avoid the deficiencies of the prior art, the present invention proposes a video data retrieval method based on mutual learning, so that the underlying features and brain function features learn each other's advantages, and then used in video data retrieval, the results show that after The features learned from each other can greatly improve the accuracy of retrieval.

Technical solutions

A video data retrieval method based on mutual learning, characterized in that the steps are as follows:

Step 1. Calculate the similarity matrix W 1 of the features X ₁ , X ₂ ,...,X _N of N video data and the similarity matrix W ₂ of the features Y ₁ , Y ₂ ,...,Y _{N :}

use $w_{i,j}^1=\exp \frac{{(x_i-x_j)}^T\×(x_i-x_j)}{{\mathrm{\σ}}^2}$ Calculate the similarity matrix W ₁ ;

use $w_{i,j}^2=\exp \frac{{(Y_i-Y_j)}^T\×(Y_i-Y_j)}{{\mathrm{\σ}}^2}$ Calculate the similarity matrix W ₂ ;

Among them, X ₁ , X ₂ ,..., X _N represent the first feature of the 1st, 2nd and N video data; Y ₁ , Y ₂ ,..., Y _N represent the 1st, 2nd and N The second feature of the video data; Indicates the element in row i, column j of matrix W ₁ ; Represents the elements of the i-th row and the j-th column of the matrix W ₂ ; X _i , X _j represent the first feature of the i-th and j-th video data; Y _i , Y _j represent the i-th and j-th video data Two kinds of features; exp means taking exponents; i, j=1,2,...,N; N>0;σ>0, which is a constant; superscript T means vector transposition;

Step 2: Take advantage of Calculate the Laplacian matrix L _{1 of W 1 ;} use Calculate the Laplacian matrix L _{2 of W 2 ;}

where D1 represents _a diagonal matrix whose elements $d_{i,j}^1=\left\{\begin{array}{cc}\underset{t=1}{\overset{N}{\mathrm{\Σ}}}{w}_{i,t}^1& i=j\\ 0& i\≠j\end{array}\right.;$ t=1,2,...,N; Represents the elements of row i and column t of matrix W ₁ ; D ₂ represents a diagonal matrix whose elements $d_{i,j}^2=\left\{\begin{array}{cc}\underset{t=1}{\overset{N}{\mathrm{\Σ}}}{w}_{i,t}^2& i=j\\ 0& i\≠j\end{array}\right.;$ t=1,2,...,N; Represents the elements of the i-th row and the t-column of the matrix W ₂ ;

Step 3: Calculate the eigenvalues and eigenvectors of the Laplacian matrices L ₁ and L ₂ , and then select the eigenvectors U ₁ , U ₂ ,..., U _M and V ₁ corresponding to the first M largest eigenvalues respectively ,V ₂ ,...,V _M ; among them, M≥1 means a constant; U ₁ , U ₂ ,..., U _M means a feature vector of size N×1 belonging to L ₁ ; V ₁ , V ₂ ,...,V _M represents the feature vector of size N×1 belonging to L ₂ ;

Step 4: Use eigenvectors U ₁ , U ₂ ,..., U _M and V ₁ , V ₂ ,..., V _M to construct matrix P=[U ₁ U ₂ ... U _M ] and Q=[ V ₁ V ₂ ... V _M ]; calculate the similarity matrix S ₁ of [K ₁ K ₂ ... K _N ] ^T and the similarity matrix S ₂ of [L ₁ L ₂ ... L _N ] ^T ,

The formula for calculating the elements of S ₁ is ${\mathrm{the\; s}}_{i,j}^1=\exp \frac{{(K_i-K_j)}^T\×(K_i-K_j)}{{\mathrm{\σ}}^2};$

The element calculation formula of S ₂ is ${\mathrm{the\; s}}_{i,j}^2=\exp \frac{{(L_i-L_j)}^T\×(L_i-L_j)}{{\mathrm{\σ}}^2};$

Among them, K ₁ , K ₂ ,...,K _N represent elements of the 1st, 2nd,...,N rows of the matrix P; L ₁ , L ₂ ,...,L _N represent the 1st, 2nd,...,N elements of the matrix Q 2,...,N rows of elements;

Step ₅ : Multiply the corresponding elements _of the similarity matrix S1 and S2 to obtain the learning matrix S;

Step 6: Multiply the similarity matrix W ₁ and the corresponding elements of the learning matrix S to obtain the learned similarity matrix E ₁ , multiply the similarity matrix W ₂ and the corresponding elements of the learning matrix S to obtain the learned similarity matrix _E2 ;

Step 7: Use the formula r=β(I-λE ₁ ) ^-1 T and f=β(I-λE ₂ ) ^-1 T to calculate the score vectors r and f of N video data after two kinds of feature learning, and set N A video data is arranged according to the score size from high to low, and the sorted video data is obtained;

Among them, r=(r ₁ ,r ₂ ,...,r _N ) represents the score vector after retrieval of the first feature of N video data, and r ₁ ,r ₂ ,...,r _N represents the first ,2,...,N video data scores; f=(f ₁ ,f ₂ ,...,f _N ) represents the score vector after retrieval of the second feature of N video data; f ₁ , f ₂ ,...,f _N represent the scores of the 1st, 2nd,...,N video data; β=1-λ represents a constant; λ>0 represents a constant; T=[t ₁ ,..., t _N ] ^T represents the query vector during retrieval, t _i =1 indicates that the i-th video data is the query target video data, otherwise t _i =0.

After step 7, the number C of video data belonging to the same category as the query target video data in the first Q sorted video data is counted, and the retrieval accuracy rate A=C/Q is calculated.

Beneficial effect

A kind of video data retrieval method based on mutual learning that the present invention proposes, at first, calculate the similarity matrix of different kinds of video data features, and utilize the similarity matrix to calculate Laplacian matrix; Secondly, calculate different kinds of video data Laplacian The eigenvalues and eigenvectors of the Laplacian matrices, respectively find out the eigenvectors corresponding to the first M largest eigenvalues in these Laplacian matrices; third, calculate the similarity matrix of the eigenvectors of different types of video data, and divide the eigenvectors The learning matrix is obtained by multiplying the corresponding elements of the similarity matrix of each feature; fourth, the learning matrix is multiplied by the corresponding elements of the similarity matrix of each feature in the first step to obtain the learned similarity matrix; fifth, for each Query target video data, use the similarity matrix after learning to calculate the score of each video data, and arrange the video data according to the score from high to low, and count the video data consistent with the query target video data in the first few video data Quantity to calculate the retrieval accuracy.

The method proposed by the invention can allow different types of video data features to learn each other's advantages, and compared with before learning, the accuracy of video data retrieval is greatly improved.

Description of drawings

Fig. 1: basic flowchart of the inventive method

Fig. 2: the retrieval accuracy rate of the method of the present invention

detailed description

Now in conjunction with embodiment, accompanying drawing, the present invention will be further described:

The hardware environment used for implementation is: AMDAthlon64×25000+ computer, 2GB memory, 256M graphics card, and the running software environment is: Matlab2009a and WindowsXP. We have realized the method that the present invention proposes with Matlab software.

The present invention is specifically implemented as follows:

The flow chart of the present invention is as shown in accompanying drawing 1. The 1256 video data used for retrieval include three categories, namely: 561 sports video data, 364 weather forecast video data and 331 advertising video data. The two features are brain function features and underlying features. The specific steps are as follows:

1. Calculate the similarity matrix W ₁ of the features X ₁ , X ₂ ,...,X _N of N video data and the similarity matrix W ₂ , W ₁ of the features Y ₁ , Y ₂ ,...,Y _N The element calculation formula of is:

Calculate the matrix W ₂ in the same way, and its element calculation formula is $w_{i,j}^2=\exp \frac{{(Y_i-Y_j)}^T\×(Y_i-Y_j)}{{\mathrm{\σ}}^2};$

Among them, X ₁ , X ₂ ,..., X _N represent the first feature of the 1st, 2nd and N video data; Y ₁ , Y ₂ ,..., Y _N represent the 1st, 2nd and N The second feature of the video data; Indicates the element in row i, column j of matrix W ₁ ; Represents the elements of the i-th row and the j-th column of the matrix W ₂ ; X _i , X _j represent the first feature of the i-th and j-th video data; Y _i , Y _j represent the i-th and j-th video data Two kinds of features; exp means taking exponents; i, j=1,2,...,N; N=1256; superscript T means vector transposition; σ=8×10 ^-6 ;

2. Use the formula Calculate the Laplacian matrix L ₁ of W ₁ , and calculate in the same way where _D1 represents the diagonal matrix,

its elements $d_{i,j}^1=\left\{\begin{array}{cc}\underset{t=1}{\overset{N}{\mathrm{\Σ}}}{w}_{i,t}^1& i=j\\ 0& i\≠j\end{array}\right.;$ t=1,2,...,N; Represents the elements of row i and column t of matrix W ₁ ; D ₂ represents a diagonal matrix whose elements $d_{i,j}^2=\left\{\begin{array}{cc}\underset{t=1}{\overset{N}{\mathrm{\Σ}}}{w}_{i,t}^2& i=j\\ 0& i\≠j\end{array}\right.;$ t=1,2,...,N; Represents the elements of the i-th row and the t-column of the matrix W ₂ ;

3. Calculate the eigenvalues and eigenvectors of the Laplacian matrices L ₁ and L ₂ , and select the eigenvectors U ₁ , U ₂ ,..., U _M and V ₁ , V ₂ corresponding to the first M largest eigenvalues ,...,V _M ;

Among them, M≥1 represents a constant; U ₁ , U ₂ ,..., U _M represent the eigenvectors of size N×1 belonging to L ₁ ; V ₁ , V ₂ ,..., V _M represent those belonging to L ₂ An eigenvector of size N×1;

4. Utilize eigenvectors U ₁ , U ₂ ,..., U _M and V ₁ , V ₂ ,..., V _M to construct matrix P=[U ₁ U ₂ ... U _M ] and Q=[V ₁ V ₂ ... V _M ]; calculate the similarity matrix S ₁ of [K ₁ K ₂ ... K _N ] ^T and the similarity matrix S ₂ of [L ₁ L ₂ ... L _N ] ^T , S The formula for calculating the elements of ₁ is:

Calculate S ₂ in the same way, the element calculation formula of S ₂ is ${\mathrm{the\; s}}_{i,j}^2=\exp \frac{{(L_i-L_j)}^T\×(L_i-L_j)}{{\mathrm{\σ}}^2};$

Among them, K ₁ , K ₂ ,...,K _N represent elements of the 1st, 2nd,...,N rows of the matrix P; L ₁ , L ₂ ,...,L _N represent the 1st, 2nd,...,N elements of the matrix Q 2,...,N rows of elements;

5. Multiply the corresponding elements of the similarity matrix S ₁ and S ₂ to obtain the learning matrix S.

6. Multiply the similarity matrix W ₁ and the corresponding elements of the learning matrix S to obtain the learned similarity matrix E ₁ , multiply the similarity matrix W ₂ and the corresponding elements of the learning matrix S to obtain the learned similarity matrix E ₂

7. Use the formula r=β(I-λE ₁ ) ^-1 T and f=β(I-λE ₂ ) ^-1 T to calculate the score vectors r and f after learning the two features of N video data, and combine N The video data is arranged according to the scores from high to low, and the sorted video data is obtained.

Among them, r=(r ₁ ,r ₂ ,...,r _N ) represents the score vector after retrieval of the first feature of N video data, and r ₁ ,r ₂ ,...,r _N represents the first ,2,...,N video data scores; f=(f ₁ ,f ₂ ,...,f _N ) represents the score vector after retrieval of the second feature of N video data; f ₁ , f ₂ ,...,f _N represent the scores of the 1st, 2nd,...,N video data; β=1-λ represents a constant; λ=0.99; T=[t ₁ ,...,t _N ] T represents the query vector when retrieving, t _i =1 means that the i-th video data is the query target video data, otherwise t _i =0;

8. Count the number C of video data belonging to the same category as the query target video data among the first Q sorted video data, and calculate the retrieval accuracy rate A=C/Q.

Use this algorithm to retrieve video data, and use each of the 1256 video data as the query target video data for a retrieval, and count and query target videos in the first 5, 10, 15 and 20 sorted video data respectively The percentage of video data whose data belongs to the same category. The average retrieval accuracy of 1256 videos is obtained by averaging the percentages obtained from the query of 1256 videos. As shown in Figure 2. As a comparison, we also use the brain function features and underlying features to search separately. The retrieval process does not carry out mutual learning, and the retrieval accuracy obtained is also shown in Figure 2. It can be seen from the figure that the brain function features after learning The retrieval accuracy of the low-level and low-level features has been greatly improved compared with that before learning. Among them, the brain function features have improved by 19.8% compared with before learning, and the underlying features have increased by 27.5% compared with before learning.

Claims (2)

1. A video data retrieval method based on mutual learning, characterized in that the steps are as follows: Step 1. Calculate the similarity matrix W 1 of the features X ₁ , X ₂ ,...,X _N of N video data and the similarity matrix W ₂ of the features Y ₁ , Y ₂ ,...,Y _{N :} use Calculate the similarity matrix W ₁ ; use Calculate the similarity matrix W ₂ ; Among them, X ₁ , X ₂ ,...,X _N represent the first feature of the 1st, 2nd...N video data; Y ₁ , Y ₂ ,...,Y _N represent the 1st, 2...N The second feature of the video data, the first feature and the second feature are brain function features and underlying features respectively; Indicates the element in row i, column j of matrix W ₁ ; Represents the elements of the i-th row and the j-th column of the matrix W ₂ ; X _i , X _j represent the first feature of the i-th and j-th video data; Y _i , Y _j represent the i-th and j-th video data Two kinds of features; exp means taking exponents; i, j=1,2,...,N; N>0;σ>0, which is a constant; superscript T means vector transposition; Step 2: Take advantage of Calculate the Laplacian matrix L _{1 of W 1 ;} use Calculate the Laplacian matrix L _{2 of W 2 ;} where D1 represents _a diagonal matrix whose elements Represents the elements of the i-th row and column t of the matrix W ₁ ; D ₂ represents the diagonal matrix, whose elements Represents the elements of the i-th row and the t-column of the matrix W ₂ ; Step 3: Calculate the eigenvalues and eigenvectors of the Laplacian matrices L ₁ and L ₂ , and then select the eigenvectors U ₁ , U ₂ ,..., U _M and V ₁ corresponding to the first M largest eigenvalues respectively ,V ₂ ,...,V _M ; among them, M≥1 means a constant; U ₁ , U ₂ ,..., U _M means a feature vector of size N×1 belonging to L ₁ ; V ₁ , V ₂ ,...,V _M represents the feature vector of size N×1 belonging to L ₂ ; Step 4: Use eigenvectors U ₁ , U ₂ ,..., U _M and V ₁ , V ₂ ,..., V _M to construct matrix P=[U ₁ U ₂ ... U _M ] and Q=[ V ₁ V ₂ ... V _M ]; calculate the similarity matrix S ₁ of [K ₁ K ₂ ... K _N ] ^T and the similarity matrix S ₂ of [L ₁ L ₂ ... L _N ] ^T , The formula for calculating the elements of S ₁ is The element calculation formula of S ₂ is Among them, K ₁ , K ₂ ,...,K _N represent elements of the 1st, 2nd,...,N rows of the matrix P; L ₁ , L ₂ ,...,L _N represent the 1st, 2nd,...,N elements of the matrix Q 2,...,N rows of elements; Step ₅ : Multiply the corresponding elements _of the similarity matrix S1 and S2 to obtain the learning matrix S; Step 6: Multiply the similarity matrix W ₁ and the corresponding elements of the learning matrix S to obtain the learned similarity matrix E ₁ , multiply the similarity matrix W ₂ and the corresponding elements of the learning matrix S to obtain the learned similarity matrix _E2 ; Step 7: Use the formula r=β(I-λE ₁ ) ^-1 T and f=β(I-λE ₂ ) ^-1 T to calculate the score vectors r and f of N video data after two kinds of feature learning, and set N A video data is arranged according to the score size from high to low, and the sorted video data is obtained; Among them, r=(r ₁ ,r ₂ ,...,r _N ) represents the score vector after retrieval of the first feature of N video data, and r ₁ ,r ₂ ,...,r _N represents the first ,2,...,N video data scores; f=(f ₁ ,f ₂ ,...,f _N ) represents the score vector after retrieval of the second feature of N video data; f ₁ , f ₂ ,...,f _N represent the scores of the 1st, 2nd,...,N video data; β=1-λ represents a constant; λ>0 represents a constant; T=[t ₁ ,..., t _N ] ^T represents the query vector during retrieval, t _i =1 indicates that the i-th video data is the query target video data, otherwise t _i =0. 2. the video data retrieval method based on mutual learning according to claim 1 is characterized in that: after step 7, adopt the video data quantity C that belongs to the same class with the query target video data in the video data after the first Q ordering of statistics , calculate the retrieval accuracy rate A=C/Q.