CN111079601A - Video content description method, system and device based on multi-mode attention mechanism - Google Patents

Abstract

The invention belongs to the field of computer vision and natural language processing, and particularly relates to a video content description method, a system and a device based on a multi-mode attention mechanism, aiming at solving the problem that the accuracy of a generated description statement is low due to the fact that the video content description method only considers video characteristics and ignores high-level semantic attribute information. The method comprises the following steps: acquiring a video frame sequence of a video to be described; extracting multi-modal feature vectors of the video frame sequence, constructing the multi-modal feature vector sequence, and obtaining feature representations corresponding to the modal feature vector sequences through a recurrent neural network; obtaining semantic attribute vectors corresponding to the feature representations through a semantic attribute detection network; and expressing the concatenated vector and semantic attribute vector based on the features corresponding to each modal feature vector sequence, and obtaining a description statement of the video to be described through an LSTM network based on an attention mechanism. The invention integrates visual features and high-level semantic attributes, and improves the accuracy of generating the video description sentences.

Description

Video content description method, system and device based on multimodal attention mechanism

technical field

The invention belongs to the fields of computer vision and natural language processing, and in particular relates to a video content description method, system and device based on a multimodal attention mechanism.

Background technique

Artificial intelligence can be roughly divided into two research directions: perceptual intelligence and cognitive intelligence. The research progress of perceptual intelligence is rapid, such as image classification and natural language translation, but the development speed of cognitive intelligence is limited, such as seeing pictures and speaking, visual description, etc. Combining natural language and computer vision research is beneficial to build a bridge of communication between humans and machines, and promote the research of cognitive intelligence.

Video content description is different from label-based coarse-grained visual understanding tasks such as video classification and object detection. Instead, it needs to describe the video content with a smooth and accurate sentence. This requires not only recognizing objects in the video, but also understanding the interrelationships between objects in the video. At the same time, due to the variety of video content description styles, such as the abstract description of the scene, the description of the relationship between various objects, the description of the behavior and motion of objects in the video, etc., which will bring great challenges to the research of video content description. Traditional video content description algorithms mainly use language template-based methods or retrieval-based methods. The methods based on language templates, due to the limitation of fixed language templates, can only generate sentences with a single form and lack of flexibility. The retrieval-based method relies too much on the size of the retrieved video database. When there is no video in the database similar to the video to be described, the generated description sentence will have a large deviation from the video content. At the same time, these two methods require complex preprocessing of the video in the early stage, and the language sequence part of the back-end is not optimized enough, resulting in poor quality of the generated sentences.

With the advancement of deep learning technology, codec-based sequence learning models have made breakthroughs in video content description problems. The present invention is also based on the codec model. Such methods do not require complex processing procedures for video in the early stage, and directly implement end-to-end training through the network, and can directly learn the mapping relationship from video to language from a large amount of training data, thereby Produce video descriptions with more precise content, diverse forms, and flexible syntax.

The key to the problem of video content description is the extraction of video features. Since different modal information in the video can assist each other, encoding the multi-modal information of the video helps to mine more semantic information. At the same time, since the general video content description algorithm only considers the video features and ignores the video high-level semantic attribute information, in order to improve the quality of the generated description sentence, the present invention also discusses how to extract the high-level semantic attributes and apply the semantic attributes to the video content description task. . The invention also analyzes and studies the problem of insufficient optimization of the language generation part at the decoder side. Most of the current video content description algorithms use maximum likelihood to model language sequences, and use cross entropy loss for training optimization, which will bring There are two obvious defects: one is the exposure bias problem. When the model is training, the input of the decoder at each moment comes from the real word in the training set, while when the model is tested, the input at each moment comes from the word predicted at the previous moment. If one of the word predictions is not accurate enough, the error may be passed down, resulting in worse and worse quality of the words generated later. The second is the problem of inconsistent training indicators and evaluation criteria. In the training phase, the cross-entropy loss function is used to maximize the posterior probability, while in the evaluation phase, objective evaluation criteria such as BLEU, METEOR, and CIDER are used. This inconsistency makes the model unable to fully evaluate the video content. The evaluation metrics generated by the description are fully optimized.

SUMMARY OF THE INVENTION

In order to solve the above problem in the prior art, that is, in order to solve the problem that the existing video content description method only considers the video features but ignores the video high-level semantic attribute information, resulting in the low accuracy of the generated description sentence, the first aspect of the present invention, A video content description method based on a multimodal attention mechanism is proposed, which includes:

Step S100, obtaining the video frame sequence of the video to be described as the input sequence;

Step S200, extracting the multimodal feature vector of the input sequence, constructing a multimodal feature vector sequence, and obtaining the feature representation corresponding to each modal feature vector sequence through a recurrent neural network; the multimodal feature vector sequence includes video Frame feature vector sequence, optical flow frame feature vector sequence, video segment feature vector sequence;

Step S300, based on the feature representation corresponding to each modal feature vector sequence, obtain the semantic attribute vector corresponding to each feature representation through a semantic attribute detection network respectively;

Step S400, concatenate the feature representations corresponding to each modal feature vector sequence to obtain an initial coding vector; based on the initial coding vector and the semantic attribute vector corresponding to each feature representation, obtain the LSTM network based on the attention mechanism. The description sentence of the video to be described;

in,

The semantic attribute detection network is constructed based on a multi-layer perceptron, and is trained based on training samples, where the training samples include feature representation samples and corresponding semantic attribute vector labels.

In some preferred embodiments, in step S200, "extract the multimodal feature vector of the input sequence, and construct a multimodal feature vector sequence", and the method is:

Perform feature extraction on each frame of RGB image in the input sequence based on a deep residual network to obtain a video frame feature vector sequence;

Based on the input sequence, the optical flow sequence is obtained through the Lucas-Kanade algorithm; the feature extraction is performed on the optical flow sequence through the deep residual network to obtain the optical flow frame feature vector sequence;

The input sequence is equally divided into T segments, and feature vectors of each segment are extracted respectively through a three-dimensional convolutional deep neural network to obtain a sequence of video segment feature vectors.

In some preferred embodiments, the training method of the semantic attribute detection network is:

Obtain a training data set, the training data set includes a video and a corresponding description sentence;

Extract the words describing the sentences in the training data set, and sort them according to the frequency of occurrence, and select the first K words as high-level semantic attribute vectors; according to whether the description sentences contain the high-level semantic attribute vectors, obtain the real semantic attribute vectors of the video Label;

Obtain the feature representation corresponding to the multimodal feature vector sequence of the video in the training data set;

The semantic attribute detection network is trained based on the feature representation and the true semantic attribute vector labels.

In some preferred embodiments, the loss function loss ₁ of the semantic attribute detection network in the training process is:

Among them, N is the number of description sentences in the training data set, K is the dimension of the predicted semantic attribute vector label output by the semantic attribute detection network, s _ik is the predicted semantic attribute vector label output by the semantic attribute detection network, and y _ik is the real semantic attribute Vector label, i, k are subscripts, α is the weight, W ^encoder is the set of all weight matrix and bias matrix parameters of the recurrent neural network and the semantic attribute detection network.

In some preferred embodiments, in step S400, "based on the initial encoding vector and the semantic attribute vector corresponding to each feature representation, the sentence description of the video to be described is obtained through the LSTM network based on the attention mechanism", and the method is as follows: :

The semantic attribute vector corresponding to each feature representation is weighted by the attention mechanism to obtain the multi-modal semantic attribute vector;

Based on the initial encoding vector and the multimodal semantic attribute vector, a sentence description of the video to be described is generated through an LSTM network.

In some preferred embodiments, the LSTM network based on the attention mechanism adopts a factorization method to calculate the weight matrix in the training process.

In a second aspect of the present invention, a video content description system based on a multimodal attention mechanism is proposed, the system includes an acquisition module, a feature extraction module, a semantic attribute detection module, and a video description generation module;

The obtaining module is configured to obtain the video frame sequence of the video to be described as the input sequence;

The extraction feature representation module is configured to extract the multimodal feature vector of the input sequence, construct a multimodal feature vector sequence, and obtain the feature representation corresponding to each modal feature vector sequence through a recurrent neural network; the multimodal feature vector sequence The state feature vector sequence includes video frame feature vector sequence, optical flow frame feature vector sequence, and video segment feature vector sequence;

The semantic attribute detection module is configured to obtain a semantic attribute vector corresponding to each feature representation through a semantic attribute detection network based on the feature representation corresponding to each modal feature vector sequence;

The generating video description module is configured to concatenate the feature representations corresponding to each modal feature vector sequence to obtain an initial coding vector; based on the initial coding vector and the semantic attribute vector corresponding to each feature representation, the attention mechanism The LSTM network obtains the description sentence of the video to be described;

in,

The semantic attribute detection network is constructed based on a multi-layer perceptron, and is trained based on training samples, where the training samples include feature representation samples and corresponding semantic attribute vector labels.

In a third aspect of the present invention, a storage device is provided, in which a plurality of programs are stored, and the program applications are loaded and executed by a processor to implement the above-mentioned video content description method based on a multimodal attention mechanism.

In a fourth aspect of the present invention, a processing device is proposed, including a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store multiple programs; the programs are adapted to be loaded by the processor And execute to realize the above-mentioned video content description method based on multimodal attention mechanism.

Beneficial effects of the present invention:

The invention integrates visual features and high-level semantic attributes, and improves the accuracy of generating video description sentences. Starting from multimodal information, the invention adopts the method of combining video frames, optical flow frames and video segments to obtain video feature vector sequences, and simultaneously detects and generates video semantic attribute vector labels. In order to obtain more effective visual features and semantic attributes, the auxiliary classification loss and LSTM network loss in the semantic attribute vector label generation stage are simultaneously optimized to ensure the contextual relationship in the sentence. In the decoding stage, an attention mechanism algorithm combining semantic attributes is proposed, the semantic attribute vector is integrated into the traditional RNN weight matrix, and the attention mechanism is used to pay attention to specific semantic attributes at every moment of generating sentence words. Improved the accuracy of video content descriptions.

Description of drawings

Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments taken with reference to the following drawings.

1 is a schematic flowchart of a video content description method based on a multimodal attention mechanism according to an embodiment of the present invention;

2 is a schematic diagram of the framework of a video content description system based on a multimodal attention mechanism according to an embodiment of the present invention;

3 is a schematic diagram of a training process of a video content description method based on a multimodal attention mechanism according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a network structure of a semantic attribute detection network according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not All examples. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the related invention, but not to limit the invention. In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict.

The video content description method based on the multimodal attention mechanism of the present invention, as shown in Figure 1, includes the following steps:

Step S100, obtaining the video frame sequence of the video to be described as the input sequence;

Step S200, extracting the multimodal feature vector of the input sequence, constructing a multimodal feature vector sequence, and obtaining the feature representation corresponding to each modal feature vector sequence through a recurrent neural network; the multimodal feature vector sequence includes video Frame feature vector sequence, optical flow frame feature vector sequence, video segment feature vector sequence;

Step S300, based on the feature representation corresponding to each modal feature vector sequence, obtain the semantic attribute vector corresponding to each feature representation through a semantic attribute detection network respectively;

Step S400, concatenate the feature representations corresponding to each modal feature vector sequence to obtain an initial coding vector; based on the initial coding vector and the semantic attribute vector corresponding to each feature representation, obtain the LSTM network based on the attention mechanism. The description sentence of the video to be described;

in,

The semantic attribute detection network is constructed based on a multi-layer perceptron, and is trained based on training samples, where the training samples include feature representation samples and corresponding semantic attribute vector labels.

In order to describe the video content description method based on the multimodal attention mechanism of the present invention more clearly, each step in an embodiment of the method of the present invention is described in detail below with reference to the accompanying drawings.

The programming language for the specific operation of the method of the present invention is not limited, and the method of the present invention can be implemented by writing in any language. The invention adopts a 4-card Titan Xp GPU server with 12G-byte memory, and uses Python language to compile a working program of a video content description method based on a multimodal attention mechanism, thereby realizing the method of the present invention. The specific implementation steps are as follows:

Step S100, acquiring a video frame sequence of the video to be described as an input sequence.

In this embodiment, the video to be described may be a video shot in real time. For example, in an intelligent monitoring and behavior analysis scenario, the video shot in real time by a camera needs to be described. In this case, the video to be described may be a video shot in real time by a camera; or The video to be described can be a video obtained from the network. For example, in a video content preview scenario, the video obtained from the network needs to be described in natural language to enable users to preview the video content. In this case, the video to be described can be obtained from the network. The video that needs to be previewed; or, the video to be described can be a locally stored video. For example, in a video classification storage scenario, the video needs to be described and classified and stored according to the description information. At this time, the video to be described can be stored locally. Videos that need to be classified and stored. Based on the acquired video to be described, a sequence of video frames is extracted as input.

Step S200, extracting the multimodal feature vector of the input sequence, constructing a multimodal feature vector sequence, and obtaining the feature representation corresponding to each modal feature vector sequence through a recurrent neural network; the multimodal feature vector sequence includes video Frame feature vector sequence, optical flow frame feature vector sequence, video segment feature vector sequence.

In this example, multi-modal video features are extracted from the video to be described, which are video frames, optical flow frames and video clips respectively. Specific steps are as follows:

Step S201, using a pre-trained deep residual network to perform feature extraction on each frame of the video frame sequence of the video to be described, using the output of the i-th layer of the network as the feature representation of the frame, and obtaining a video frame feature vector sequence

The video frame feature vector sequence is input into the recurrent neural network LSTM in sequence, and the hidden state h _t at the last moment of the network is used as the feature representation of the video frame feature vector of the video, denoted as v _f .

Step S202, the video frame sequence of the video to be described generates the optical flow sequence of the video through the Lucas-Kanade algorithm, the pre-trained deep residual network performs feature extraction on each frame, and the output of the i-th layer of the network is used as the frame. The feature representation of the optical flow frame feature vector sequence is obtained

The sequence of optical flow frame feature vectors is input into the recurrent neural network LSTM in sequence, and the hidden state h _t at the last moment of the network is used as the feature representation of the optical flow frame of the video, denoted as v _o

Step S203, the video frame sequence of the video to be described is evenly divided into T segments, each segment uses a three-dimensional convolutional deep neural network to perform feature extraction, and the output of the i-th layer of the network is used as the feature representation for the t-th segment of the video to obtain the video segment feature vector. sequence

The sequence of feature vectors of video segments is input into the recurrent neural network LSTM in sequence, and the hidden state h _t at the last moment of the network is used as the feature representation of the video segment of the video, denoted as _vc .

The above steps extract the process of multi-modal feature representation of video, as shown in Figure 3, input Video (video), which is divided into video frame (Frame), optical flow frame (Optical flow) and video clip (Video clip), where video Frame output is Frame Feature (feature representation of video frame feature vector), that is, static feature, video clip output is C3D Feature (3D-CNN feature of video clip), and optical flow frame output is Motion Feature (dynamic feature). The other steps in Figure 3 are described below.

Step S300 , based on the feature representation corresponding to each modal feature vector sequence, obtain a semantic attribute vector corresponding to each feature representation through a semantic attribute detection network.

In this embodiment, the training of the semantic attribute detection network is introduced first, and then the semantic attribute vector corresponding to each feature representation obtained through the semantic attribute detection network is introduced.

The semantic attribute detection network is constructed based on a multi-layer perceptron. The structure is shown in Figure 4, including an input layer (InputLayer), a hidden layer (Hidden Layer), and an output layer (Output layer). The input is a video (InputVideo) and the corresponding The description sentence is "A Small child id playing the guitar (a small child is playing the guitar)", through the recurrent neural network LSTM, the multi-modal feature vector sequence (v _i1 ,v _i2 ,...,v _in ) is obtained, through The semantic attribute detection network outputs semantic attribute vectors s _i1 ,s _i2 ,...,s _iK . The specific training process of the semantic attribute detection network is as follows:

Step A301: Obtain a training data set, where the training data set includes videos and corresponding description sentences.

Step A302, extracting the words describing the sentences in the training data set, and sorting them according to the frequency of occurrence, and selecting the first K words as high-level semantic attribute vectors; according to whether the description sentences include the high-level semantic attribute vectors, obtain the real video of the video. Semantic attribute vector labels.

Extract the words that describe the sentences in the training data set, sort the words according to the frequency of word occurrence, remove the function words, and then select the K words with the highest occurrence probability as the high-level semantic attribute value

Assuming that the training data set has N sentences, y _i =[y _i1 , y _i1 ,...y _il ,...y _iK ] is the true semantic attribute vector label of the ith video. Wherein, if the description sentence corresponding to the video i contains the attribute word l, then y _il =1; otherwise, y _il =0.

Step A303: Obtain the feature representation corresponding to the multimodal feature vector sequence of the video in the training data set by using the method of step S200.

Step A304: Train the semantic attribute detection network based on the feature representation and the real semantic attribute vector label.

Let v _i ∈ {v _f , v _o , v _c } denote the feature representation learned from the ith video, and the training sample at this time is {v _i , y _i }. In the present invention, a semantic attribute detection network based on a multi-layer perceptron is used to learn the function f(·):R ^m →R ^K , which is expressed as mapping an m-dimensional space to K-dimension, where R ^m is m The real number space of dimensions, R ^K is the same, m is the dimension represented by the input feature, K is the dimension of the output semantic attribute vector, this dimension is equal to the number of semantic attribute values extracted above (the dimension of the high-level semantic attribute vector), and more The layer perceptron output vector s _i =[s _i1 ,...,s _iK ] is the predicted semantic attribute vector label of the ith video. The classification loss function loss ₁ of the semantic attribute detection network is shown in formula (1):

Among them, W ^encoder represents the set of weight matrix and bias matrix parameters of the recurrent neural network and semantic attribute detection network, α is the weight, s _i =α(f(vi ₎ ) learns to obtain a K-dimensional vector, and α( ) represents The sigmoid function, f( ) is implemented by a multilayer perceptron.

After the semantic attribute detection network is trained, in the actual application process, based on the feature representation corresponding to each modal feature vector sequence, the semantic attribute vector corresponding to each feature representation is obtained through the semantic attribute detection network. The Multimodal Semantic Detector module in Figure 3.

Step S400, concatenate the feature representations corresponding to each modal feature vector sequence to obtain an initial coding vector; based on the initial coding vector and the semantic attribute vector corresponding to each feature representation, obtain the LSTM network based on the attention mechanism. A description sentence for the video to be described.

In this embodiment, {v _f , v _o , v _c } is concatenated as the initial coding vector v, such as the Concatenation module in FIG. 3 . Attention Fusion is based on the attention module.

The following first introduces the training process of the LSTM network based on the attention mechanism, and then introduces the method of obtaining the description sentences of the video to be described through the LSTM network based on the attention mechanism.

When the LSTM network based on the attention mechanism is trained, the input description sentence is "A Small child idplaying the guitar". The specific training process is as follows:

When the output is a sentence, using the LSTM network as the decoder can capture the long-term dependencies of the sentence. Assuming that the word input at the current moment is w _t , the hidden state of the LSTM network at the last moment is h _t-1 , and the memory state of the cell at the last moment is c _t-1 , then the update rule of the LSTM network at time t is as formula (2) (3)(4)(5)(6)(7)(8):

i _t =σ(W _i w _t +U _hi h _t-1 +z) (2)

f _t =σ(W _f w _t +U _hf h _t-1 +z) (3)

o _t =σ(W _o w _t +U _ho h _t-1 +z) (4)

h _t =o _t ⊙tanh(c _t ) (7)

z=1(t=1)·Cv (8)

分别表示t时刻输入门、遗忘门、输出门、记忆单元和压缩输入的状态，tanh(·)表示双曲正切函数，1(t＝1)为指示函数，初始编码向量v在LSTM的初始时刻作为输入，z表示在t＝1初始时刻将视频向量作为输入。为了简化，上述公式中的偏置项均被省略。Use * to represent a subscript in the above formula {i,f,o,c}, where W _* , U _h* and C are weight matrices, i _t ,f _t ,o _t ,c _t ,

respectively represent the state of the input gate, forget gate, output gate, memory unit and compressed input at time t, tanh( ) represents the hyperbolic tangent function, 1(t=1) is the indicator function, and the initial encoding vector v is at the initial moment of LSTM As input, z represents the video vector as input at the initial time t=1. For simplicity, the bias terms in the above formulas are omitted.

其中n_h是隐藏单元数目，n_x是词嵌入向量的维数，则W_*(S_t)/U_h*(S_t)的表达式如公式(9)(10)所示：To better utilize auxiliary information from multiple modal semantic attributes, we propose an attention mechanism combined with semantic attributes to calculate weight matrices W _* and U _h* , extending each weight matrix of traditional LSTM to be related to K attributes A collection of relevant weight matrices for mining the meaning of individual words to generate final descriptive sentences. Replace the initial weight matrix W _* /U _h* with W _* (S _t )/U _h* (S _t ), where S _t ∈ R ^K is a multimodal semantic attribute vector that changes dynamically over time. In particular, define two weight matrices

Where n _h is the number of hidden units, and n _x is the dimension of the word embedding vector, then the expression of W _* (S _t )/U _h* (S _t ) is shown in formula (9) (10):

where W _τ [k], U _τ [k] are denoted as the kth 2D slice of the weight matrices W _τ and U _τ , respectively, which are associated with the probability value S _t [k], which _is the multimodal The kth element of the semantic attribute vector S _t of the state. Since W _τ is a three-dimensional vector, a 2D slice refers to a slice of W _τ , which is a two-dimensional vector.

The calculation process of S _t is shown in formula (11) (12) (13):

e _ti =w ^T tanh(W _a h _t-1 +U _a s _i ) (13)

Among them, l=3 represents the learned three semantic attribute vectors {s _f , s _o , s _c }, and the attention weight a _ti reflects the importance of generating the i-th semantic attribute in the video at the current moment. It can be seen that for different time steps t, the semantic attributes S _t are different, which makes the model selectively focus on different semantic attribute parts in the video each time a word is generated, j represents the subscript, and e _ti represents Unregularized attention weights, w ^T denotes the transformation matrix.

In the training process of the LSTM network based on the attention mechanism, it is equivalent to jointly training K LSTMs. The parameter amount of the network is proportional to the K value. When the K value is large, the network can hardly complete the training. The following factorization is adopted method, as shown in formula (14) (15):

W _* (S _t )=W _a ·diag(W _b S _t )·W _c (14)

U _h* (S _t )=U _a ·diag(U _b S _t ) ·U _c (15)

和

n_f表示因子分解的相关超参数。in,

and

n _f denotes the relevant hyperparameters for factorization.

Why the parameter quantity of the factorization network is greatly reduced, and the problem that the parameter quantity of the original network is proportional to the K value is avoided. The following is an analysis of the parameter quantity of the network. In formulas (9) and (10), the total parameter quantity is K·n _h ·(n _x +n _h ), and it can be considered that the parameter quantity is proportional to K. In formulas (14) and (15), the parameter quantity of the W _* (S _t ) formula is n _f ·(n _h +K+n _x ), and the parameter quantity of the U _h* (S _t ) formula is n _f ·(2n _{h )} +K), the sum of the two parameters is n _f ·(3n _h +2K+n _x ). When specifying n _f =n _h , for larger values of K, n _f ·(3n _h +2K+n _x ) is much smaller than K ·n _h ·(n _x +n _h ).

Bring the factorized formulas (14) (15) into the LSTM network update rule to obtain formulas (16) (17) (18):

表示融入语义属性向量的输入，

表示融入语义属性向量的隐藏状态。同理可证，f_t,o_t,c_t的表达式和上面公式相似。where ⊙ denotes the element-wise multiplication operator, and for each element value in S _t , the parameter matrices W _a and U _a are shared, which can effectively capture language patterns common in videos, while the diagonal matrix diag( W _b S _t ) and diag(U _b S _t ) consider the specific semantic attribute parts in different videos,

represents the input into the semantic attribute vector,

Represents the hidden state incorporated into the semantic attribute vector. It can be proved by the same reason that the expressions of f _t , o _t , and c _t are similar to the above formulas.

It can be seen from the above formulas that after the network is fully trained, it can not only effectively capture the common language pattern part in the video, but also focus on the specific semantic attribute part in the video. It avoids the problem that the parameter amount of the original network is proportional to the K value.

In the process of generating video description sentences, greedy search is used first, and the words output at each moment are shown in formula (19):

w _t =softmax(Wh _t ) (19)

Among them, W is the transformation matrix.

Therefore, the loss function loss ₂ of the generated sentence of the network is designed, as shown in formula (20):

loss ₂ =-logP(Y|v,s _f ,s _c ,s _o )=-∑logP(w _t |w _0～t-1 ) (20)

Among them, Y={w ₁ , w ₂ , ...... w _N }, represents a sentence composed of N words, and w _{0 to t-1} are words generated before time t.

The classification loss loss ₁ for generating high-level semantic attributes and the loss loss ₂ for generating description sentences are added together and optimized at the same time, which can ensure the context relationship in the sentence. Based on the obtained loss values, the network is trained using a back-propagation algorithm. As shown in Figure 3, the Classification Loss (classification loss) module and the Captioning Loss (sentence loss) module are added to obtain the Total Loss (total loss or global loss) module.

After the training of the LSTM network based on the attention mechanism is completed, based on the initial encoding vector and the corresponding semantic attribute vector of each feature representation, the description sentence of the video to be described is obtained through the LSTM network based on the attention mechanism.

A video content description system based on a multimodal attention mechanism according to the second embodiment of the present invention, as shown in FIG. 2 , includes: an acquisition module 100 , an extraction feature representation module 200 , a semantic attribute detection module 300 , and a video description generation module 400;

The obtaining module 100 is configured to obtain the video frame sequence of the video to be described as the input sequence;

The extraction feature representation module 200 is configured to extract the multimodal feature vector of the input sequence, construct a multimodal feature vector sequence, and obtain the feature representation corresponding to each modal feature vector sequence through a recurrent neural network; The modal feature vector sequence includes a video frame feature vector sequence, an optical flow frame feature vector sequence, and a video segment feature vector sequence;

The semantic attribute detection module 300 is configured to obtain a semantic attribute vector corresponding to each feature representation through a semantic attribute detection network based on the feature representation corresponding to each modal feature vector sequence;

The generating video description module 400 is configured to concatenate feature representations corresponding to each modal feature vector sequence to obtain an initial coding vector; The LSTM network of the mechanism obtains the description sentence of the video to be described;

in,

The semantic attribute detection network is constructed based on a multi-layer perceptron, and is trained based on training samples, where the training samples include feature representation samples and corresponding semantic attribute vector labels.

Those skilled in the technical field can clearly understand that, for the convenience and brevity of description, for the specific working process and related description of the system described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here.

It should be noted that, the video content description system based on the multi-modal attention mechanism provided by the above embodiments is only illustrated by the division of the above functional modules. That is, the modules or steps in the embodiments of the present invention are decomposed or combined. For example, the modules in the above-mentioned embodiments can be combined into one module, or can be further split into multiple sub-modules, so as to complete all the above descriptions. or some functions. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing each module or step, and should not be regarded as an improper limitation of the present invention.

A storage device according to a third embodiment of the present invention stores a plurality of programs, and the programs are suitable for being loaded by a processor and implementing the above-mentioned video content description method based on a multimodal attention mechanism.

A processing device according to a fourth embodiment of the present invention includes a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store multiple programs; the programs are adapted to be loaded and executed by the processor In order to realize the above-mentioned video content description method based on multimodal attention mechanism.

Those skilled in the technical field can clearly understand that the undescribed convenience and brevity are not described. Repeat.

Those skilled in the art should be aware that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software or a combination of the two, and the programs corresponding to the software modules and method steps Can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or as known in the art in any other form of storage medium. In order to clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been described generally in terms of functionality in the foregoing description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods of implementing the described functionality for each particular application, but such implementations should not be considered beyond the scope of the present invention.

The terms "first," "second," etc. are used to distinguish between similar objects, and are not used to describe or indicate a particular order or sequence.

The term "comprising" or any other similar term is intended to encompass a non-exclusive inclusion such that a process, method, article or device/means comprising a list of elements includes not only those elements but also other elements not expressly listed, or Also included are elements inherent to these processes, methods, articles or devices/devices.

So far, the technical solutions of the present invention have been described with reference to the preferred embodiments shown in the accompanying drawings, however, those skilled in the art can easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims (9)

1. a video content description method based on multimodal attention mechanism, is characterized in that, this method comprises: Step S100, obtaining the video frame sequence of the video to be described as the input sequence; Step S200, extracting the multimodal feature vector of the input sequence, constructing a multimodal feature vector sequence, and obtaining the feature representation corresponding to each modal feature vector sequence through a recurrent neural network; the multimodal feature vector sequence includes video Frame feature vector sequence, optical flow frame feature vector sequence, video segment feature vector sequence; Step S300, based on the feature representation corresponding to each modal feature vector sequence, obtain the semantic attribute vector corresponding to each feature representation through a semantic attribute detection network respectively; Step S400, concatenate the feature representations corresponding to each modal feature vector sequence to obtain an initial coding vector; based on the initial coding vector and the semantic attribute vector corresponding to each feature representation, obtain the LSTM network based on the attention mechanism. The description sentence of the video to be described; in, The semantic attribute detection network is constructed based on a multi-layer perceptron, and is trained based on training samples, where the training samples include feature representation samples and corresponding semantic attribute vector labels. 2. The video content description method based on a multimodal attention mechanism according to claim 1, wherein in step S200, "extract the multimodal feature vector of the input sequence, and construct a multimodal feature vector sequence" , the method is: Perform feature extraction on each frame of RGB image in the input sequence based on a deep residual network to obtain a video frame feature vector sequence; Based on the input sequence, the optical flow sequence is obtained through the Lucas-Kanade algorithm; the feature extraction is performed on the optical flow sequence through the deep residual network to obtain the optical flow frame feature vector sequence; The input sequence is equally divided into T segments, and feature vectors of each segment are extracted respectively through a three-dimensional convolutional deep neural network to obtain a sequence of video segment feature vectors. 3. the video content description method based on multimodal attention mechanism according to claim 1, is characterized in that, its training method of described semantic attribute detection network is: Obtain a training data set, the training data set includes a video and a corresponding description sentence; Extract the words describing the sentences in the training data set, and sort them according to the frequency of occurrence, and select the first K words as high-level semantic attribute vectors; according to whether the description sentences contain the high-level semantic attribute vectors, obtain the real semantic attribute vectors of the video Label; Obtain the feature representation corresponding to the multimodal feature vector sequence of the video in the training data set; The semantic attribute detection network is trained based on the feature representation and the true semantic attribute vector labels. 4. The video content description method based on multi-modal attention mechanism according to claim 3, is characterized in that, in the training process of described semantic attribute detection network, its loss function loss ₁ is:

Among them, N is the number of description sentences in the training data set, K is the dimension of the predicted semantic attribute vector label output by the semantic attribute detection network, s _ik is the predicted semantic attribute vector label output by the semantic attribute detection network, and y _ik is the real semantic attribute Vector label, i, k are subscripts, α is the weight, W ^encoder is the set of all weight matrix and bias matrix parameters of the recurrent neural network and the semantic attribute detection network. 5. The video content description method based on a multimodal attention mechanism according to claim 1, wherein in step S400 "based on the initial coding vector, each feature represents the corresponding semantic attribute vector, The LSTM network of the mechanism obtains the sentence description of the video to be described", and the method is: The semantic attribute vector corresponding to each feature representation is weighted by the attention mechanism to obtain the multi-modal semantic attribute vector; Based on the initial encoding vector and the multimodal semantic attribute vector, a sentence description of the video to be described is generated through an LSTM network. 6. The video content description method based on a multimodal attention mechanism according to claim 1, wherein the LSTM network based on the attention mechanism adopts the method of factoring to carry out the calculation of the weight matrix in the training process . 7. A video content description system based on a multimodal attention mechanism, characterized in that the system comprises an acquisition module, an extraction feature representation module, a semantic attribute detection module, and a generated video description module; The obtaining module is configured to obtain the video frame sequence of the video to be described as the input sequence; The extraction feature representation module is configured to extract the multimodal feature vector of the input sequence, construct a multimodal feature vector sequence, and obtain the feature representation corresponding to each modal feature vector sequence through a recurrent neural network; the multimodal feature vector sequence The state feature vector sequence includes video frame feature vector sequence, optical flow frame feature vector sequence, and video segment feature vector sequence; The semantic attribute detection module is configured to obtain a semantic attribute vector corresponding to each feature representation through a semantic attribute detection network based on the feature representation corresponding to each modal feature vector sequence; The generating video description module is configured to concatenate the feature representations corresponding to each modal feature vector sequence to obtain an initial coding vector; based on the initial coding vector and the semantic attribute vector corresponding to each feature representation, the attention mechanism The LSTM network obtains the description sentence of the video to be described; in, The semantic attribute detection network is constructed based on a multi-layer perceptron, and is trained based on training samples, where the training samples include feature representation samples and corresponding semantic attribute vector labels. 8. A storage device, wherein a plurality of programs are stored, wherein the program application is loaded and executed by a processor to realize the video based on the multimodal attention mechanism of any one of claims 1-6 Content description method. 9. A processing device, comprising a processor and a storage device; the processor is adapted to execute each program; the storage device is adapted to store a plurality of programs; characterized in that the program is adapted to be loaded and executed by the processor to The method for describing video content based on a multimodal attention mechanism according to any one of claims 1 to 6 is implemented.

Images