CN111988673A - Video description statement generation method and related equipment - Google Patents

Abstract

An embodiment of the present application provides a method and related equipment for generating a video description sentence, the method includes: obtaining a syntactic feature vector of a target paradigm example sentence; determining the syntax of the video description sentence to be generated according to the syntactic feature vector, and obtaining syntax information determining the semantics of the to-be-generated video description sentence corresponding to the syntax according to the syntax information and the video semantic feature vector of the target video, and obtaining semantic information; generating the video description sentence of the target video according to the semantic information. Therefore, it is possible to generate video description sentences with different syntactic structures by selecting different target example sentences, and solve the problem of single syntax of video description sentences.

Description

Video description sentence generation method and related equipment

technical field

The present application relates to the technical field of artificial intelligence, and in particular, to a method and related equipment for generating video description sentences.

Background technique

Video Captioning refers to generating sentences for a given video that can be used to describe the content in the video, and the generated sentences are called video description sentences. By generating the video description sentence for the video, it is convenient for the user to quickly know the content of the video only through the video description sentence without watching the video. In the related art, the generated video description sentences have the problem of single syntax.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a method and related device for generating a video description sentence, thereby solving the problem of single syntax of the video description sentence at least to a certain extent.

Other features and advantages of the present application will become apparent from the following detailed description, or be learned in part by practice of the present application.

According to an aspect of the embodiments of the present application, a method for generating a video description sentence is provided, the method includes: obtaining a syntactic feature vector of a target paradigm example sentence; determining the syntax of the video description sentence to be generated according to the syntactic feature vector, and obtaining Syntax information; determine the semantics of the to-be-generated video description sentence corresponding to the syntax according to the syntax information and the video semantic feature vector of the target video, and obtain semantic information; generate the video description sentence of the target video according to the semantic information .

According to an aspect of the embodiments of the present application, there is provided an apparatus for generating a video description sentence, the apparatus comprising: an obtaining module for obtaining a syntactic feature vector of a target paradigm example sentence; a syntax determining module for obtaining the syntactic feature according to the syntactic feature The vector determines the syntax of the video description sentence to be generated, and obtains syntax information; the semantic determination module is used for determining the semantics of the video description sentence to be generated corresponding to the syntax according to the syntax information and the video semantic feature vector of the target video, and obtains Semantic information; a video description sentence determination module, configured to generate a video description sentence of the target video according to the semantic information.

In some embodiments of the present application, the syntax determination module is configured to: generate a first latent vector according to the syntax feature vector by the first neural network included in the description generation model, where the first latent vector is used to indicate the Syntax information, the description generation model further includes a second neural network cascaded with the first neural network, the first neural network and the second neural network are gated based recurrent neural networks.

In this embodiment, the semantic determination module is configured to: generate a second latent vector by the second neural network according to the first latent vector and the video semantic feature vector, where the second latent vector is used to indicate the describe semantic information.

In some embodiments of the present application, the video description sentence determination module is configured to: determine a word vector at time t according to a second latent vector generated by the second neural network at time t; the video description sentence.

In this embodiment, the syntax determination module includes a first latent vector generation unit, which is configured to be generated by the first neural network according to the syntax feature vector, the word vector at time t-1, and the result obtained by the first neural network. Generate the first hidden vector at time t-1, and output the first hidden vector at time t.

In this embodiment, the semantic determination module includes a second latent vector generating unit, which is configured to generate the second latent vector by the second neural network according to the video semantic feature vector, the first latent vector at time t, and the second latent vector. The second latent vector at time t-1 generated by the neural network outputs the second latent vector at time t.

In some embodiments of the present application, the first latent vector generating unit includes: a first soft attention weighting unit, configured to perform soft attention weighting on the syntactic feature vector according to the first latent vector at the time t-1 , the target syntax feature vector corresponding to time t is obtained. The first splicing unit is used for splicing the target syntax feature vector corresponding to time t and the word vector at time t-1 to obtain a first splicing vector corresponding to time t. The first output unit is configured to use the first splicing vector corresponding to time t as an input by the first neural network, and output the first latent vector corresponding to time t.

In some embodiments of the present application, the first neural network includes a first input gate, a first forgetting gate, and a first output gate, and the first output unit includes: a first forgetting gate vector calculation unit for The first forgetting gate calculates the first forgetting gate vector at time t according to the first splicing vector corresponding to time t. and a first input gate vector calculation unit, configured to calculate the first input gate vector at time t by the first input gate according to the first splicing vector corresponding to time t. The first cell unit vector calculation unit is used to calculate the first forget gate vector at the time t, the first input gate vector at the time t, the first unit vector at the time t and the corresponding t of the first neural network. The first cell unit vector at time -1 is calculated to obtain the first cell unit vector at time t, and the first unit vector at time t is obtained by performing hyperbolic tangent calculation according to the first splicing vector corresponding to time t. The first hidden vector calculation unit is used to calculate the first hidden vector at time t according to the first cell unit vector at time t and the first output gate vector at time t. The first output gate vector at time t is Calculated by the first output gate according to the first splicing vector corresponding to time t.

In some embodiments of the present application, the syntax determination module further includes: a first normalization unit, configured to compare the first input gate vector, the first forget gate vector, and the first output in the first neural network The gate vector and the first unit vector are respectively normalized; the first transformation unit, according to the first offset vector and the first scaling vector, respectively normalizes the normalized first input gate vector, the first forget gate vector, the first The output gate vector and the first unit vector are transformed to obtain the target first input gate vector, the target first forget gate vector, the target first output gate vector and the target first unit vector, and the first offset vector is the A multi-layer perceptron is output according to the target syntax feature vector corresponding to time t, the first scaling vector is output by the second multi-layer perceptron according to the target syntax feature vector corresponding to time t, the The first multilayer perceptron is independent of the second multilayer perceptron.

In this embodiment, the first cell unit vector calculation unit is further configured to: according to the target first forget gate vector, the target first input gate vector, the target first unit vector and the first time t-1 The cell unit vector is calculated to obtain the first cell unit vector at time t.

In this embodiment, the first hidden vector calculation unit is further configured to: calculate and obtain the first hidden vector at time t according to the first cell unit vector at time t and the target output gate vector.

In some embodiments of the present application, the second latent vector generating unit includes: a second soft attention weighting unit, configured to perform soft attention on the video semantic feature vector according to the second latent vector at the time t-1 Force weighting to obtain the target video semantic vector corresponding to time t. The second splicing unit is used for splicing the target video semantic vector corresponding to time t with the first latent vector at time t to obtain a second splicing vector corresponding to time t. The second output unit is configured to use the second splicing vector corresponding to time t as input by the second neural network, and output the second latent vector corresponding to time t.

In some embodiments of the present application, the second neural network includes a second input gate, a second forgetting gate, and a second output gate, and the second output unit includes: a second forgetting gate vector calculation unit for The second forgetting gate calculates the second forgetting gate vector at time t according to the second splicing vector corresponding to time t; and the second input gate vector calculation unit is used for the second input gate according to the The second splicing vector at time t is calculated to obtain the second input gate vector at time t. The second cell unit vector calculation unit is configured to calculate the second forget gate vector at the time t, the second input gate vector at the time t, the second unit vector at the time t, and the t corresponding to the second neural network. The second cell unit vector at time -1 is calculated to obtain the second cell unit vector at time t, and the second unit vector at time t is obtained by performing hyperbolic tangent calculation according to the second splicing vector corresponding to time t; The second hidden vector calculation unit is configured to calculate the second latent vector at time t according to the second cell unit vector at time t and the second output gate vector at time t, where the second output gate vector at time t is Calculated by the second output gate according to the second splicing vector corresponding to time t.

In some embodiments of the present application, the semantic determination module further includes: a second normalization unit, configured to compare the second input gate vector, the second forget gate vector, and the second output gate vector in the second neural network and the second unit vector are normalized separately. The second transformation unit is configured to perform the normalized second input gate vector, the second forget gate vector, the second output gate vector and the second unit vector respectively according to the second offset vector and the second scaling vector. Transform to obtain the target second input gate vector, the target second forget gate vector, the target second output gate vector and the target second unit vector, and the second offset vector is the third multi-layer perceptron according to the corresponding t The second scaling vector is output by the fourth multi-layer perceptron according to the target video semantic vector corresponding to time t, and the third multi-layer perceptron is the same as the fourth multi-layer perceptron. Multilayer perceptrons are independent.

In this embodiment, the second cell unit vector calculation unit is further configured to: according to the target second forget gate vector, the target second input gate vector, the target second unit vector and the second time t-1 Calculate the cell unit vector to obtain the second cell unit vector at time t;

In this embodiment, the second latent vector calculation unit is further configured to: calculate and obtain the second latent vector at time t according to the second cell unit vector at time t and the target second output gate vector.

In some embodiments of the present application, the apparatus for generating video description sentences further includes: a training data acquisition module, configured to acquire training data, where the training data includes several sample videos and sample video description sentences corresponding to the sample videos. a semantic feature extraction module, used for performing semantic feature extraction on the sample video, to obtain a sample video semantic feature vector of the sample video; and a syntactic feature extraction module, used for syntactically extracting the sample video description sentences corresponding to the sample video Feature extraction, to obtain a sample syntax feature vector of the sample video description sentence. The first syntax loss determination module is configured to output the first latent vector sequence by the first neural network according to the sample syntax feature vector, and calculate the first syntax loss through the first latent vector sequence; the first semantic loss A determination module, used for outputting a second latent vector sequence by the second neural network according to the first latent vector sequence and the sample video semantic feature vector of the sample video, and calculating the first semantics through the second latent vector sequence loss. a first target loss calculation module, configured to calculate and obtain a first target loss according to the first syntactic loss and the first semantic loss; a first adjustment module, configured to adjust the description generation based on the first target loss parameters of the model.

In some embodiments of the present application, the first syntax loss determination module includes: a syntax tree prediction unit, configured to predict and obtain a syntax tree for the sample description sentence according to the first latent vector sequence through a sixth neural network, where The sixth neural network is a gated-based recurrent neural network; the first syntactic loss calculation unit is used to calculate the first syntactic loss according to the predicted syntax tree and the actual syntax tree of the sample description sentence .

In some embodiments of the present application, the first semantic loss determination module includes: a first description sentence output unit, configured to output a first description for the sample video according to the second latent vector sequence through a fifth multilayer perceptron statement. A first semantic loss calculation unit, configured to calculate and obtain the first semantic loss according to the first description sentence and the sample video description sentence corresponding to the sample video.

In some embodiments of the present application, the apparatus for generating a video description sentence further includes: a first sample syntax feature vector acquisition module, configured to obtain a sample syntax feature vector of a sample sentence, where the sample sentence includes a sample example sentence and a sample video The corresponding sample video description sentence; the second syntax loss calculation module is used for outputting a first latent vector sequence by the first neural network according to the sample syntax feature vector of the sample sentence, and the first hidden vector sequence corresponding to the sample sentence is passed. The vector sequence calculates the second syntactic loss; the second semantic loss calculation module is configured to output the second hidden vector by the second neural network according to the sample semantic feature vector of the sample sentence and the first latent vector sequence corresponding to the sample sentence. vector sequence, the sample semantic feature vector is obtained by performing semantic feature extraction on the sample sentence, and the second semantic loss is calculated through the second latent vector sequence corresponding to the sample sentence; the second target loss calculation module is used for A second target loss is calculated according to the second syntactic loss and the second semantic loss; and a second adjustment module is configured to adjust the parameters of the description generation model based on the second target loss.

In some embodiments of the present application, the training data further includes several sample exemplary sentences, and the apparatus for generating a video description sentence further includes: a second sample syntax feature vector acquisition module, configured to acquire the sample syntax of the sample exemplary sentences feature vector; the first hidden vector sequence output module is used for outputting the first latent vector sequence by the first neural network according to the sample syntax feature vector of the sample example sentence; the second latent vector sequence output module is used for the output by the The second neural network outputs a second latent vector sequence according to the first latent vector sequence corresponding to the sample model example sentence and the sample video semantic feature vector of the sample video; the second description sentence determination module is used for according to the sample corresponding to the sample. The second latent vector sequence of the video determines the second description sentence; the third target loss calculation module is used to calculate the third target loss according to the syntax tree corresponding to the sample example sentence and the syntax tree corresponding to the second description sentence; A third adjustment module, configured to adjust the parameters of the description generation model based on the third target loss.

In some embodiments of the present application, the obtaining module includes: a character feature vector obtaining unit, configured to obtain character feature vectors of characters included in each word in the target example sentence, where the character feature vectors are obtained by encoding characters The 3rd latent vector output unit is used for outputting the 3rd latent vector corresponding to each character according to the character feature vector of each character by the described 3rd neural network; Average calculation unit is used for each in the described target model example sentence word, according to the average calculation of the third hidden vector corresponding to each character in the word, to obtain the feature vector of the word; the fourth hidden vector output unit is used by the fourth neural network according to each word in the target example sentence The feature vector of , outputs a fourth latent vector, the fourth latent vector is used as the syntactic feature vector, and the third neural network and the fourth neural network are gated-based recurrent neural networks.

In some embodiments of the present application, the device for generating a video description sentence further includes: a video frame sequence acquisition module for acquiring a video frame sequence obtained by dividing the target video into frames; a semantic extraction module for The convolutional neural network performs semantic extraction on each video frame in the video frame sequence to obtain the semantic vector of each video frame; the fifth hidden vector output module is used for according to the video frame sequence through the fifth neural network. The semantic vector of each video frame outputs a fifth latent vector, and the fifth latent vector is used as the video semantic feature vector, and the fifth neural network is a gated-based recurrent neural network.

According to an aspect of the embodiments of the present application, an electronic device is provided, including: a processor; and a memory, where computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the above-mentioned Methods.

According to an aspect of the embodiments of the present application, there is provided a computer-readable storage medium having computer-readable instructions stored thereon, and when the computer-readable instructions are executed by a processor, the above method is implemented.

In the technical solutions provided by some embodiments of the present application, first, based on the syntactic feature vector of the target paradigm example sentence, syntactic information for guiding the syntactic structure of the video description sentence to be generated is obtained, and then based on the syntactic information and the video semantic feature of the target video The vector determines that the video description sentence to be generated corresponds to the semantic information of the syntactic structure indicated by the syntactic feature vector, and finally generates a video description sentence for the target video according to the semantic information, which not only ensures that the generated video description sentence is similar to the syntactic structure of the target example sentence, Furthermore, it is ensured that the generated video description sentence is semantically related to the target video, that is, the video description sentence can accurately describe the content in the target video.

Since the syntax of the generated video description sentence is controlled by the syntactic feature vector of the target sample sentence, then, for the same target video, if the target sample sentence with different syntactic structure is selected to constrain the syntactic structure of the video description sentence, different syntaxes can be generated. Structured video description sentences, thus, video description sentences with different syntactic structures can be generated for the same target video by changing the target example sentence, so as to realize the generation of diverse video description sentences for the target video, effectively solving the problem of video description in the prior art. Describe the problem of single syntax of the statement.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the present application.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description serve to explain the principles of the application. Obviously, the drawings in the following description are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort. In the attached image:

FIG. 1 shows a schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied;

2 is a flowchart of a method for generating a video description sentence according to an embodiment of the present application;

Figure 3 shows a schematic diagram of a long short-term memory neural network;

4 is a schematic diagram of a video description sentence generated under different target paradigm sentences for the same target video according to a specific embodiment;

5 is a flow chart of outputting a first latent vector according to an embodiment;

6 is a flow chart of outputting a second latent vector according to an embodiment;

7 is a flowchart illustrating training a description generation model according to an embodiment;

8 is a flowchart illustrating training a description generation model according to another embodiment;

9 is a flowchart illustrating training a description generation model according to another embodiment;

10 is a schematic diagram of generating a video description sentence according to an embodiment;

11 is a block diagram of an apparatus for generating video description sentences according to an embodiment;

FIG. 12 shows a schematic structural diagram of a computer system suitable for implementing the electronic device according to the embodiment of the present application.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments, however, can be embodied in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical solutions of the present application may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be employed. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the present application.

The block diagrams shown in the figures are merely functional entities and do not necessarily necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices entity.

The flowcharts shown in the figures are only exemplary illustrations and do not necessarily include all contents and operations/steps, nor do they have to be performed in the order described. For example, some operations/steps can be decomposed, and some operations/steps can be combined or partially combined, so the actual execution order may be changed according to the actual situation.

Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing (NLP) technology, and machine learning/deep learning.

FIG. 1 shows a schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied.

As shown in FIG. 1 , the system architecture may include terminal devices (one or more of a smart phone 101 , a tablet computer 102 and a portable computer 103 as shown in FIG. 1 , of course, a desktop computer, etc.), a network 104 and server 105. The network 104 is the medium used to provide the communication link between the terminal device and the server 105 . The network 104 may include various connection types, such as wired communication links, wireless communication links, and the like.

It should be understood that the numbers of terminal devices, networks and servers in FIG. 1 are merely illustrative. There can be any number of terminal devices, networks and servers according to implementation needs. For example, the server 105 may be a server cluster composed of multiple servers, or the like.

In an embodiment of the present application, the server may acquire the target example sentence and the target video uploaded on the terminal device, and then perform syntactic feature extraction on the target example sentence to obtain the syntactic feature vector of the target example sentence, and perform a video on the target video. Semantic extraction, to obtain the video semantic feature vector of the target video.

In an embodiment of the present application, the server can also store a set of example sentences, and the server can receive the example sentence selection instruction sent by the terminal device, and determine the target example sentence according to the example example sentence selection instruction, and then carry out the target example sentence. Syntax feature extraction.

Of course, in other embodiments, the server may also store several videos for the user to select, that is, the server may receive the video selection instruction sent by the terminal device, and use the video selected by the video selection instruction as the target of the video description sentence to be generated. video.

In one embodiment of the present application, after obtaining the syntactic feature vector of the target paradigm example sentence and the video semantic feature vector of the target video, the server generates a video description sentence for the target video based on the syntactic feature vector and the video semantic feature vector, so that the generated On the one hand, the video description sentence is the same or similar to the syntax of the target example sentence, and it is guaranteed that the semantics of the video description sentence is related to the video content in the target video, that is, related to the semantics of the target video.

In an embodiment of the present application, after generating the video description sentence for the target video, the server feeds back the generated video description sentence to the terminal device, so that the terminal device presents the video description sentence to the user.

It should be noted that, the method for generating a video description sentence provided by the embodiment of the present application is generally executed by the server 105 , and accordingly, a device for generating a video description sentence is generally set in the server 105 . However, in other embodiments of the present application, the terminal device may also have similar functions to the server, so as to execute the method for generating video description sentences provided by the embodiments of the present application.

The implementation details of the technical solutions of the embodiments of the present application are described in detail below:

FIG. 2 shows a flowchart of a method for generating a video description sentence according to an embodiment of the present application. The method for generating a video description sentence can be executed by a device with a computing processing function, such as the method shown in FIG. 1 . server 105 to execute. Referring to FIG. 2, the method for generating a video description sentence includes at least steps 210 to 240, and the details are as follows:

Step 210: Obtain the syntactic feature vector of the target paradigm example sentence.

The target paradigm sentence refers to the paradigm sentence used to constrain the syntactic structure of the video description sentence to be generated.

In some embodiments of the present application, a sample sentence set may be pre-built, so that the user selects a sample sentence from the sample sentence set as the target sample sentence, so as to constrain the syntactic structure of the video description sentence to be generated. Certainly, in other embodiments, the target example sentence may also be a sample sentence uploaded by the user through the terminal device.

The syntactic feature vector of the target paradigm example sentence is used to describe the syntactic structure of the target paradigm sentence sentence, which reflects the dependencies between words and syntactic structure information (such as subject-predicate-object, definite form complement, etc.) in the target paradigm sentence sentence.

The syntactic feature vector of the target paradigm example sentence can be obtained by syntactic analysis of the target paradigm sentence sentence. Among them, syntactic analysis can use syntactic structure analysis (also known as phrase structure analysis, component syntax analysis), dependency relationship analysis (also known as dependency syntax analysis, dependency analysis), and deep grammar and syntax analysis.

In some embodiments of the present application, a syntactic analysis tool may be used to perform a syntactic analysis on the target paradigm example sentence, and then a syntactic feature vector may be generated according to the syntactic analysis result. Syntax analysis tools such as StanfordCoreNLP, HanLP, SpaCy, FudanNLP. The syntactic analysis result may be a component syntax tree generated for the target paradigm example sentence, and the syntactic feature vector of the target paradigm sentence sentence is obtained by serializing the component syntax tree.

In some embodiments of the present application, syntactic feature extraction may be performed through a two-level cascaded gating-based recurrent neural network to obtain a syntactic feature vector of the target paradigm sentence.

Specifically, the character feature vector of the characters included in each word in the target example sentence is obtained first, and the character feature vector is obtained by encoding the character; then, the third neural network outputs the first corresponding to each character according to the character feature vector of each character. Three hidden vectors; then for each word in the target example sentence, the average calculation is carried out according to the third hidden vector corresponding to each character in the word to obtain the feature vector of the word; The feature vector of the word obtains the fourth latent vector sequence, the fourth latent vector sequence is used as the syntactic feature vector, and the third neural network and the fourth neural network are gated-based recurrent neural networks.

The gated-based recurrent neural network may be a Long Short Term Memory Network (LSTM) or a Gated Recurrent Unit (GRU).

GRU is a variant of LSTM, in which the improvement of GRU over LSTM includes: merging the forget gate and the input gate into one gate, that is, the update gate, and the other gate is called the reset gate; it is not used as an internal state in LSTM The division of the cell unit vector and the latent vector as the external state directly adds a linear dependency between the current state of the network (h _t ) and the state of the network at the previous moment (h _t-1 ).

In the following, the above-mentioned process of generating a syntactic feature vector for the target example sentence sentence will be described by taking that the third neural network and the fourth neural network are both long and short-term memory networks as an example. Before the specific description, it is necessary to explain the structure of the long-short-term memory neural network and the processing procedures involved in it.

Figure 3 shows a schematic diagram of a long-short-term memory neural network. As shown in Figure 3, at any time t (assuming it is time t), there are three inputs to the long-short-term memory network: the input vector x _t at time t, the previous time The output latent vector h _t-1 and the cell unit vector c _t-1 at the previous moment, wherein the cell unit vector is used to reflect the state of the cell unit at the corresponding moment, and the hidden vector is used as the output of the LSTM at the corresponding moment.

As shown in Figure 3, LSTM includes a forget gate, an input gate and an output gate. The forget gate determines how much of the cell unit vector c _t-1 at the previous moment is retained to the current cell unit vector c _t ; the input gate determines how much How much of the input vector x _t at the current moment is saved to the cell unit vector _ct at the current moment; the output gate is used to control how much of the cell unit vector ct _{is output to the current output h t of} the LSTM.

In LSTM, the hidden vector and cell unit vector at the corresponding moment are determined based on the calculation of the forget gate, input gate and output gate. For the convenience of description, the vector directly obtained by the calculation of the input gate in LSTM is called the input gate vector. The vector calculated by the output gate is called the output gate vector, and the vector calculated by the forget gate is called the forget gate vector.

For any time _t , the calculation process of forget gate vector ft , input gate vector it , unit vector _gt , cell unit vector _ct , output gate vector _ot , and hidden vector h _t corresponding to time _t is as follows.

Among them, the forget gate vector f _t is:

f _t =σ(W _f ·[h _t-1 , x _t ]+b _f ), (1)

Among them, σ is the sigmoid function, and its value range is (0, 1); W _f is the weight matrix of the forget gate, [h _t-1 , x _t ] represents the splicing of two vectors, and b _f is the bias of the forget gate Set terms, W _f and b _f can be determined by training.

The input gate vector i _t is:

i _t =σ(W _i ·[h _t-1 , x _t ]+ _bi ), (2)

_Wi is the weight matrix of the input gate, _bi is the bias term of the input gate, _Wi and _bi are determined by training.

In LSTM, the calculation of unit vector is also involved. The unit vector is used to describe the current input. The unit vector g _t is:

g _t =tanh(W _c ·[h _t-1 , x _t ]+b _c ), (3)

Among them, tanh represents the hyperbolic tangent function, W _c is the weight matrix, b _c is the bias term, and W _c and b _c can be determined through training.

The cell vector g _t is used to calculate the cell cell vector c _t , and the cell cell vector c _t is:

表示按元素乘。Among them, the symbol

Represents element-wise multiplication.

The output gate vector o _t is:

o _t =σ(W _o ·[h _t-1 , x _t ]+b _o ), (5)

The hidden vector ht is:

其输入到第三神经网络LSTMc中，其在第三神经网络LSTMc网络中的计算过程可以描述为：Continue back to the process of generating syntactic feature vectors for the target paradigm example sentences. For example, for the word "apple", its character sequence is: "apple", first input the character feature vector of each character in the character sequence into the third neural network LSTMc, and obtain the third hidden value corresponding to each character. vector. Specifically, it is assumed that the character feature vector of the l-th character in the n-th word in the target paradigm example sentence is

It is input into the third neural network LSTMc, and its calculation process in the third neural network LSTMc network can be described as:

为第三神经网络针对第n个词中第l-1个字符所输出的隐向量，

为第三神经网络针对第n个词中第l-1个字符所得到的细胞单元向量，

为第三神经网络针对第n个词中第1个字符所得到的细胞单元向量.

为第三神经网络第n个词中第1个字符所输出的隐向量，为便于区分，将第三神经网络所输出的隐向量称为第三隐向量。in,

is the latent vector output by the third neural network for the l-1th character in the nth word,

is the cell unit vector obtained by the third neural network for the l-1th character in the nth word,

is the cell unit vector obtained by the third neural network for the first character in the nth word.

is the latent vector output by the first character in the nth word of the third neural network. For the convenience of distinction, the latent vector output by the third neural network is called the third latent vector.

After the third latent vector corresponding to each character is obtained, average calculation is performed according to the third latent vector corresponding to each character in the nth word, and the obtained average vector is used as the feature vector w _n of the nth word:

Finally, the feature vectors corresponding to each word in the target example sentence are sequentially input into the fourth neural network LSTM ^w , where the processing process of the feature vector w _n of the nth word in the fourth neural network can be described as:

为第四神经网络针对第n个词的特征向量所输出的隐向量；

为第四神经网络针对第n个词的特征向量所输出的细胞单元向量；

为第四神经网络针对第n-1个词的特征向量所输出的隐向量；

为第四神经网络针对第n-1个词的特征向量所输出的细胞单元向量；为便于区分，将第四神经网络所输出的隐向量称为第四隐向量。in,

is the latent vector output by the fourth neural network for the feature vector of the nth word;

is the cell unit vector output by the fourth neural network for the feature vector of the nth word;

is the latent vector output by the fourth neural network for the feature vector of the n-1th word;

is the cell unit vector output by the fourth neural network for the feature vector of the n-1th word; for the convenience of distinction, the latent vector output by the fourth neural network is called the fourth latent vector.

该第四隐向量序列H^s作为目标范例句的句法特征向量，用于控制所要生成的视频描述语句的句法。Thus, the fourth hidden vector sequence is obtained by combining the fourth hidden vectors output at each moment

The fourth latent vector sequence H ^s is used as the syntactic feature vector of the target paradigm example sentence to control the syntax of the video description sentence to be generated.

Through the above process, the feature vector corresponding to the word is obtained based on the character-level coding for each word in the target sample sentence. feature.

Please continue to refer to FIG. 2, step 220, determine the syntax of the video description sentence to be generated according to the syntax feature vector, and obtain the syntax information.

For the generation of video description sentences, it is predicted and outputted by words, and the syntactic components of the words outputted at each moment in the video description sentences are different. For example, if the target example sentence is a subject-predicate-object structure, the corresponding words are output in the order of subject, predicate, and object, so as to form a video description sentence.

Since the syntactic feature vector of the target example sentence describes the dependencies and syntactic structure information between words in the target example sentence, therefore, according to the syntactic feature vector of the target example sentence, the syntactic structure of the target example sentence can be correspondingly determined. The syntactic structure of the sentence is used as the syntactic structure of the video description sentence to be generated. Based on this, the sentence components to be output at each moment are guided by the syntactic information.

That is to say, the syntactic information is used to indicate the sentence components of the video description sentence to be output at each moment, such as subject, predicate, object, attribute, adverbial and so on. Since the syntactic structure of the video description sentence to be generated is the same as the syntactic structure of the target example sentence, the sentence components to be output at each moment are correspondingly determined according to the syntactic structure indicated by the syntactic feature vector, so as to ensure the output video The syntactic structure of the descriptive sentence is consistent with the syntactic structure of the target paradigm.

Step 230: Determine the semantics of the to-be-generated video description sentence corresponding to the syntax according to the syntax information and the video semantic feature vector of the target video, and obtain semantic information.

The target video does not specifically refer to a certain video, but generally refers to the video that needs to generate a video description sentence.

The video semantic feature vector of the target video is used to describe the content of the video, and the content of the video can be understood as the semantics of the video. The content in the video may include objects in the video (eg, characters, animals, objects, scenes, etc.), behaviors of the objects, and the like.

For an object in a video, object recognition and determination can be performed based on video frames in the video; for behavior of an object in a video, action recognition of the object can be performed based on several consecutive video frames to determine the behavior of the object.

In some embodiments of the present application, in order to obtain the video semantic feature vector of the target video, the target video can be divided into frames first, and object recognition, action recognition, etc. can be performed based on the image features in each video frame, so as to generate the target video. Video semantic feature vector for the video.

In some embodiments of the present application, the video semantic feature vector of the target video can be extracted through the following process. Specifically, the video frame sequence obtained by dividing the target video into frames is obtained first; then, the semantic extraction of each video frame in the video frame sequence is performed through the convolutional neural network to obtain the semantic vector of each video frame; The neural network outputs the fifth latent vector sequence according to the semantic vector of each video frame in the video frame sequence, and the fifth latent vector sequence is used as the video semantic feature vector. The fifth neural network is a gated-based recurrent neural network.

Wherein, the latent vector output by the fifth neural network is called the fifth latent vector, and the fifth latent vector sequence is obtained by combining the fifth latent vectors output by the fifth neural network corresponding to each video frame.

It is assumed that the semantic vectors of each video frame are combined into a video semantic sequence V=[v ₁ , . . . , vm , _{. . . , v M ]} , where vm is the semantic vector of the _mth video frame.

其中，第m帧视频帧的语义向量v_m在第五神经网络LSTM^v中的处理过程可描述为：Then, the video semantic sequence is input into the fifth neural network (for example, the long short-term memory network LSTMv) for encoding, and the feature sequence containing the video context is obtained

Among them, the processing process of the semantic vector v _m of the mth video frame in the fifth neural network LSTM ^v can be described as:

为第五神经网络LSTM^v针对第m-1帧视频帧的语义向量所输出的隐向量；

为第五神经网络LSTM^v针对m-1帧视频帧的语义向量所输出的细胞单元向量；

为第五神经网络LSTM^v针对第m帧视频帧的语义向量所输出的隐向量；

为第五神经网络LSTM^v针对m帧视频帧的语义向量所输出的细胞单元向量；为便于区分，将第五神经网络LSTM^v所输出的隐向量称为第五隐向量。in,

is the latent vector output by the fifth neural network LSTM ^v for the semantic vector of the m-1th video frame;

is the cell unit vector output by the fifth neural network LSTM ^v for the semantic vector of the m-1 video frame;

is the latent vector output by the fifth neural network LSTM ^v for the semantic vector of the mth video frame;

is the cell unit vector output by the fifth neural network LSTM ^v for the semantic vector of m video frames; for the convenience of distinction, the latent vector output by the fifth neural network LSTM ^v is called the fifth latent vector.

The fifth latent vectors output by the fifth neural network LSTM ^v for each video frame in the target video are combined according to the sequence order of the video frames, that is, the fifth latent vector sequence is obtained.

As described above, the syntax information is used to indicate the sentence components to be output at each moment. Therefore, in order to ensure that the output video description sentence accurately expresses the content in the video, the semantic information of each sentence component is obtained by combining the video semantic feature vector and syntactic information of the target video to assign semantics to each sentence component.

It can be understood that if the syntactic structure of the target example sentence changes, the sentence components that need to be output in sequence at each moment also change correspondingly. Therefore, the semantic information output for the target video is controlled by the syntax limited by the target example sentence. structured.

Step 240: Generate a video description sentence of the target video according to the semantic information.

In some embodiments of the present application, after the semantic information corresponding to each syntactic component is obtained, words are predicted according to the semantic information, so that the words predicted at each moment are sequentially combined to obtain a video description sentence of the target video.

In some embodiments of the present application, a vocabulary is pre-deployed for word prediction. Based on the obtained semantic information, predict the probability that each word in the vocabulary corresponds to the semantic information, and then determine the word corresponding to the semantic information according to the predicted probability, for example, use the word with the highest probability as the word corresponding to the semantic information .

Through the solution of the present application, first, based on the syntactic feature vector of the target paradigm example sentence, the syntactic information used to guide the syntactic structure of the video description sentence to be generated is obtained, and then according to the syntactic information and the video semantic feature vector of the target video, the corresponding video description sentence to be generated is determined. Based on the semantic information of the syntactic structure indicated by the syntactic feature vector, finally generate a video description sentence for the target video according to the semantic information, which not only ensures that the generated video description sentence has the same syntactic structure as the target example sentence, but also ensures that the generated video description sentence The sentence is related to the semantics of the target video, that is, the video description sentence can accurately describe the content in the target video.

It can be understood that, for the same target video, if target example sentences with different syntactic structures are selected to constrain the syntactic structure of the video description sentences, then video description sentences with different syntactic structures can be generated. The same target video generates video description sentences with different syntactic structures, so as to generate diverse video description sentences for the target video.

Please refer to FIG. 4 , for the same target video, if the target example sentence is "Aerial view of a group of sheeps on a grass field.", then based on the target example sentence, the method of the present application generates a video description for the target video The sentence is "Cooking video of a recipe with ingredients in a glassbowl."; and if the target sample sentence is "Female patient watching TV and remoting control inhand in hospital bed", then the target video is the target video according to the method of this application. The generated video description sentence is "Woman cook slicing egg and mixing salad in bow in kitchentable"; if the target sample sentence is "Water splashes when a coin dropped in a glass", then based on the target sample sentence according to the method of the present application The video description sentence generated by the target video is "Egg scatters whena knife cut at board".

In some embodiments of the present application, the processes of step 220 and step 230 are respectively implemented by a gated-based recurrent neural network.

In this embodiment, step 220 includes: generating a first latent vector according to the syntactic feature vector by the first neural network included in the description generation model, where the first latent vector is used to indicate syntactic information, and the description generation model further includes a second neural network , the first neural network and the second neural network are gated based recurrent neural networks.

In some embodiments of the present application, the first hidden vector at time t is output by the first neural network according to the syntactic feature vector, the word vector at time t-1, and the first hidden vector at time t-1 generated by the first neural network. vector.

Specifically, the first latent vector may be generated through the process of steps 510-530 as shown in FIG. 5 . The specific instructions are as follows:

Step 510: Perform soft attention weighting on the syntactic feature vector according to the first latent vector at time t-1 to obtain a target syntactic feature vector corresponding to time t.

Step 520, splicing the target syntax feature vector corresponding to time t and the word vector at time t-1 to obtain a first splicing vector corresponding to time t.

Step 530, the first neural network uses the first splicing vector corresponding to time t as an input, and outputs the first latent vector corresponding to time t.

Soft attention weighting, also known as soft attention mechanism, performs re-weighted aggregation calculation on the remaining information by selectively ignoring part of the information.

Continuing with the above example, the above steps 510-530 are described by taking the fifth neural network as a long-short-term memory neural network.

则通过该t-1时刻的第一隐向量

对目标范例句的句法特征向量为

进行软注意力加权可以描述为：Assume that the first hidden vector output by the first neural network at time t-1 is

Then through the first hidden vector at the time t-1

The syntactic feature vector of the target paradigm example sentence is

Performing soft attention weighting can be described as:

为对应于t时刻的目标句法特征向量。in,

is the target syntax feature vector corresponding to time t.

与t-1时刻的词向量e_t-1进行拼接，所得到对应于t时刻的第一拼接向量为

The target syntactic feature vector at time t

Splicing with the word vector e _t- 1 at time t-1, the obtained first splicing vector corresponding to time t is

作为对应于t时刻第一神经网络LSTM^syn的输入向量，由该第一神经网络LSTM^syn输出对应于t时刻的第一隐向量

该过程可以描述为：Then, the first splice vector corresponding to time t is

As the input vector corresponding to the first neural network LSTM ^syn at time t, the first hidden vector corresponding to time t is output by the first neural network LSTM ^syn

The process can be described as:

为第一神经网络LSTM^syn中t时刻的细胞单元向量；

为第一神经网络LSTM^syn中t时刻的第一隐向量；

为第一神经网络LSTM^syn中t-1时刻的细胞单元向量。in,

is the cell unit vector at time t in the first neural network LSTM ^syn ;

is the first hidden vector at time t in the first neural network LSTM ^syn ;

is the cell unit vector at time t-1 in the first neural network LSTM ^syn .

When the first neural network is a long-short-term memory neural network LSTM ^syn , its specific structure can be referred to as shown in FIG. 3 . The first neural network includes a first input gate, a first forgetting gate and a first output gate, and the above step 530 includes: calculating the first forgetting gate at time t by the first forgetting gate according to the first splicing vector corresponding to time t vector; and by the first input gate according to the first splicing vector corresponding to the time t to obtain the first input gate vector at time t; then according to the first forget gate vector at time t, the first input gate vector at time t, t The first unit vector at time and the first cell unit vector at time t-1 corresponding to the first neural network are calculated to obtain the first cell unit vector at time t, and the first unit vector at time t is based on the first unit vector corresponding to time t. The splicing vector is calculated by hyperbolic tangent; finally, the first hidden vector at time t is calculated according to the first cell unit vector at time t and the first output gate vector at time t, and the first output gate vector at time t is given by The first output gate is calculated according to the first splice vector corresponding to time t.

In this embodiment, the first forget gate vector refers to the forget gate vector in the first neural network. Similarly, the first input gate vector, the first output gate vector, and the first cell unit vector refer to the first neural network respectively. The input gate vector, output gate vector, and cell unit vector in .

Wherein, for the calculation of the first input gate vector, the first output gate vector, the first cell unit vector, the first cell vector and the first hidden vector at time t, please refer to the above formulas (1)-(6) and no longer here. Repeat.

In some embodiments of the present application, the second neural network based on the gated recurrent neural network implements the process of step 230 as follows: the second neural network generates a second latent vector according to the first latent vector and the video semantic feature vector, and the second latent vector is generated by the second neural network. Binary latent vectors are used to indicate semantic information.

In some embodiments of the present application, the second neural network outputs the second latent vector at time t according to the video semantic feature vector, the first latent vector at time t, and the second latent vector at time t-1 generated by the second neural network. hidden vector.

In the case where the second neural network is a long-short-term memory neural network, the process of generating the second latent vector at time t may include steps 610-630 as shown in FIG. 6 below. The specific instructions are as follows:

Step 610: Perform soft attention weighting on the video semantic feature vector according to the second latent vector at time t-1 to obtain a target video semantic vector corresponding to time t.

Step 620, splicing the target video semantic vector corresponding to time t with the first latent vector at time t to obtain a second splicing vector corresponding to time t.

Step 630, the second neural network uses the second splicing vector corresponding to time t as an input, and outputs the second latent vector corresponding to time t.

通过t-1时刻第二神经网络的第二隐向量

对视频语义向量

进行软注意加权可以描述为：Continuing the above example, the video semantic vector of the target video is

Passing through the second latent vector of the second neural network at time t-1

video semantic vector

Performing soft attention weighting can be described as:

为对应于t时刻的目标视频语义向量。in,

is the target video semantic vector corresponding to time t.

与t-1时刻第一神经网络所输出的第一隐向量

进行拼接，得到对应于t时刻的第二拼接向量

Then, the target video semantic vector at time t is

and the first hidden vector output by the first neural network at time t-1

Perform splicing to obtain the second splicing vector corresponding to time t

作为t时刻第二神经网络LSTM^sem的输入向量，由第二神经网络LSTM^sem对应输出t时刻的第二隐向量，该过程可描述为：The second splice vector corresponding to time t

As the input vector of the second neural network LSTM ^sem at time t, the second latent vector at time t is correspondingly output by the second neural network LSTM ^sem . The process can be described as:

为第二神经网络LSTM^sem中t时刻的细胞单元向量；

为第二神经网络LSTM^sem中t时刻的第一隐向量；

为第二神经网络LSTM^sem中t-1时刻的细胞单元向量。in,

is the cell unit vector at time t in the second neural network LSTM ^sem ;

is the first hidden vector at time t in the second neural network LSTM ^sem ;

is the cell unit vector at time t-1 in the second neural network LSTM ^sem .

In the case where the second neural network is a long-short-term memory neural network, the second neural network includes a second input gate, a second forgetting gate, and a second output gate. In this embodiment, step 630 includes: The forgetting gate calculates the second forgetting gate vector at time t according to the second splicing vector corresponding to time t; and calculates the second input gate vector at time t by the second input gate according to the second splicing vector corresponding to time t; According to the second forgetting gate vector at time t, the second input gate vector at time t, the second unit vector at time t and the second cell unit vector at time t-1 corresponding to the second neural network, the second time at time t is calculated. Cell unit vector, the second unit vector at time t is calculated from the hyperbolic tangent of the second splicing vector corresponding to time t; calculated from the second cell unit vector at time t and the second output gate vector at time t The second latent vector at time t and the second output gate vector at time t are calculated by the second output gate according to the second splicing vector corresponding to time t.

In this embodiment, the second forget gate vector refers to the forget gate vector in the second neural network. Similarly, the second input gate vector, the second output gate vector, and the second cell unit vector refer to the second neural network respectively. The input gate vector, output gate vector, and cell unit vector in .

Among them, the calculation of the second input gate vector, the second output gate vector, the second cell unit vector, the second unit vector and the second hidden vector at time t refer to the above formulas (1)-(6), and here Repeat.

After obtaining the second latent vector output by the second neural network at time t, determine the word vector at time t according to the second latent vector generated by the second neural network at time t; generate video description sentences according to the word vector output at each time .

Among them, the word vector is used as the vector code of the word to be output, and the word vector at time t is predicted according to the second latent vector generated by the second neural network at time t, so that the words corresponding to the word vector predicted at each time are combined. , that is, the video description sentence of the target video is obtained.

In a model with a multi-layer neural network, the output of the previous layer is the input of the next layer. Since the neural network needs to go through calculation processes such as linear transformation and activation function, the input of the next layer may have a higher value range. big difference. If the distribution of the input of a certain layer of neural network changes, its parameters need to be re-learned, a phenomenon called internal covariate shift.

Therefore, in order to avoid the phenomenon of internal covariate shift in the description generative model, the first output gate vector, the first forget gate vector, the first output gate vector and the first unit vector in the first neural network, and the first output gate vector in the first neural network. The second output gate vector, the second forget gate vector, the second output gate vector and the second unit vector in the two neural networks are respectively subjected to a conditional layer normalization (CLN) operation.

Among them, the conditional layer normalization operation is defined as:

Among them, x is the variable to be normalized by the conditional layer, μ(x) is the mean value of the variable x, σ(x) is the standard deviation of the variable x; f _γ (y) is the conditional The vector y is the input and output vector (assumed to be the first vector), and f _β (y) is the output vector (assumed to be the second vector) of another multilayer perceptron with the condition vector y as the input. The multilayer perceptron that outputs the first vector is independent of the multilayer perceptron that outputs the second vector.

It can be seen from the above that in order to realize the normalization operation of the conditional layer, the variables need to be normalized first, and then the first vector is used as the scaling vector to scale and transform the normalized variables; the second vector is used as the scaling vector. Offset vector, used to offset the normalized variable.

The conditional layer normalization operation is performed on the first output gate vector, the first forget gate vector, the first output gate vector and the first unit vector in the first neural network. , the first forgetting gate vector, the first output gate vector and the first unit vector are normalized respectively; then the normalized first input gate vector, the first The forget gate vector, the first output gate vector and the first unit vector are transformed to obtain the target first input gate vector, the target first forget gate vector, the target first output gate vector and the target first unit vector, and the first offset vector is output by the first multi-layer perceptron according to the target syntax feature vector corresponding to time t, and the first scaling vector is output by the second multi-layer perceptron according to the target syntax feature vector corresponding to time t. Independent of the second multilayer perceptron.

In this embodiment, the calculation is performed according to the first forget gate vector at time t, the first input gate vector at time t, the first unit vector at time t, and the first cell unit vector at time t-1 corresponding to the first neural network. Obtaining the first cell unit vector at time t includes: calculating the first cell unit vector at time t according to the target first forget gate vector, the target first input gate vector, the target first unit vector and the first cell unit vector at time t-1. Cell unit vector.

In this embodiment, calculating the first latent vector at time t according to the first cell unit vector at time t and the first output gate vector at time t includes: according to the first cell unit vector at time t and the target output gate vector Calculate the first hidden vector at time t.

For the conditional layer normalization operation of the first output gate vector, the first forget gate vector, the first output gate vector and the first unit vector in the first neural network, the above condition vector y is the calculated value corresponding to time t. The target syntax feature vector.

The above conditional layer normalization operation for the first output gate vector, the first forget gate vector, the first output gate vector and the first unit vector in the first neural network can be expressed as:

和W_i ^syn均为权重矩阵，b^syn为偏置项，该

W_i ^syn、b^syn通过训练确定。上式(16)等号左边的f_t ^syn，

分别为进行条件层归一化操作后所得到的目标第一遗忘门向量、目标第一输入门向量、目标第一输出门向量和目标第一单元向量。in,

and Wi _syn are weight matrices, b ^{syn is} the bias term, the

_Wisyn and ^bsyn are determined by training ^. f _t ^syn on the left side of the equal sign of the above formula (16),

are the target first forget gate vector, the target first input gate vector, the target first output gate vector and the target first unit vector obtained after the normalization operation of the conditional layer is performed, respectively.

的计算，以及分别按照下述公式(18)进行t时刻的第一隐向量

的计算。After obtaining the target first forget gate vector, target first input gate vector, target first output gate vector and target first unit vector corresponding to time t, the target first forget gate vector, target first input gate vector, The target first output gate vector and the target first unit vector participate in the calculation of the first latent vector and the first cell unit vector in the long short-term memory neural network. Specifically, according to the following formula (17), the first cell unit vector at time t is calculated respectively.

The calculation of , and the first hidden vector at time t is performed according to the following formula (18) respectively

calculation.

The process of normalizing the conditional layer for the second input gate vector, the second forgetting gate vector, the second output gate vector and the second unit vector in the second neural network includes: The input gate vector, the second forget gate vector, the second output gate vector and the second unit vector are respectively normalized; then, the normalized second input gate is respectively normalized according to the second offset vector and the second scaling vector The vector, the second forget gate vector, the second output gate vector and the second unit vector are transformed to obtain the target second input gate vector, the target second forget gate vector, the target second output gate vector and the target second unit vector. The second offset vector is output by the third multi-layer perceptron according to the target video semantic vector corresponding to time t, the second scaling vector is output by the fourth multi-layer perceptron according to the target video semantic vector corresponding to time t, and the third The multilayer perceptron is independent of the fourth multilayer perceptron.

In this embodiment, the calculation is performed according to the second forget gate vector at time t, the second input gate vector at time t, the second unit vector at time t, and the second cell unit vector at time t-1 corresponding to the second neural network. Obtaining the second cell unit vector at time t includes: calculating the second cell unit vector at time t according to the target second forget gate vector, the target second input gate vector, the target second unit vector and the second cell unit vector at time t-1. Cell unit vector.

In this embodiment, calculating the second latent vector at time t according to the second cell unit vector at time t and the second output gate vector at time t includes: according to the second cell unit vector at time t and the target second output The gate vector is calculated to obtain the second hidden vector at time t.

The calculation process involved in the conditional layer normalization operation of the second input gate vector, the second forget gate vector, the second output gate vector and the second unit vector in the second neural network is similar to the calculation process in the first neural network, For details, reference may be made to the above-mentioned calculation process for the normalization operation of the conditional layer in the first neural network, which will not be repeated here.

The gated-based recurrent neural network adjusts the structure of the network on the basis of the simple recurrent neural network, and adds a gate control mechanism to control the transmission of information in the neural network. The gating mechanism can be used to control how much information in the memory unit needs to be retained, how much needs to be discarded, how much new state information needs to be saved to the memory unit, etc. dependencies without the vanishing and exploding gradients.

Moreover, the gating-based RNN retains the characteristics of the simple RNN, that is, the processing capability of the data stream with dependencies, such as the syntactic features of the data streams with dependencies, such as target paradigm sentences, target videos, and target paradigm sentences vector, the video semantic feature vector of the target video, etc., and, due to the addition of a gating mechanism, a relatively long-span dependency can be learned without gradient disappearance and gradient explosion.

It is worth mentioning that although the first neural network and the second neural network are used as long-short-term memory neural networks to describe the solution of the present application, it is not limited to using long-short-term memory neural networks to realize the solution of the present application. The scheme of the present application can also be implemented by using a gated recurrent network. For the specific process, refer to the above-mentioned process of implementing the scheme by using a long-short-term memory neural network.

In some embodiments of the present application, in order to ensure the accuracy of the video description sentences predicted and output by the description generation model, the description generation model also needs to be trained. The specific training process may include steps 710- as shown in FIG. 7 . 760 process. The specific description is as follows:

Step 710: Acquire training data, where the training data includes several sample videos and sample video description sentences corresponding to the sample videos.

Among them, the training data can use the existing video description to generate the data sets MSRVTT and ActivityNet. Of course, in other embodiments, training data can also be constructed as required.

Step 720: Extract semantic features of the sample video to obtain a sample video semantic feature vector of the sample video; and perform syntactic feature extraction on the sample video description sentence corresponding to the sample video to obtain a sample syntactic feature vector of the sample video description sentence.

Among them, the process of extracting the semantic features of the sample video to obtain the semantic feature vector of the sample video can be realized by using the convolutional neural network and the fifth neural network in the above.

The syntactic feature extraction performed on the sample description sentence to obtain the sample syntactic feature vector can be realized by using the third neural network and the fourth neural network in the above.

Step 730, the first neural network outputs the first latent vector sequence according to the sample syntax feature vector, and calculates the first syntax loss by using the first latent vector sequence.

In some embodiments of the present application, in order to calculate the first syntactic loss by using the first sequence of latent vectors, the sixth neural network is first used to predict the syntax tree for the sample description sentence according to the sequence of the first latent vectors, and the sixth neural network is based on A gated recurrent neural network; then the first syntactic loss is calculated based on the predicted syntax tree and the actual syntax tree of the sample description sentence.

为：In some embodiments of the present application, the description generation model may be trained syntactically with a negative log-likelihood loss function. Define the first syntactic loss function

for:

Wherein, P(C ^syn |H ^syn ; V, C) is expressed as the syntax tree H ^syn and the sample description sentence obtained according to the first hidden vector sequence prediction obtained from the sample video description sentence C corresponding to the sample video V and the sample video V The probability that the similarity of the actual syntax tree C ^syn of C satisfies the first preset condition.

The first preset condition may be set according to the first syntax tree similarity threshold, for example, if the similarity between the predicted syntax tree and the actual syntax tree of the sample description sentence is greater than or equal to the first syntax tree similarity threshold, it is considered that the first preset condition is met.

Thus, the first syntactic loss for the sample video and the sample video description sentence corresponding to the sample video is calculated according to the above-mentioned first syntactic loss function.

Step 740, the second neural network outputs the second latent vector sequence according to the first latent vector sequence and the sample video semantic feature vector of the sample video, and calculates the first semantic loss by using the second latent vector sequence.

In some embodiments of the present application, in order to calculate the first semantic loss through the second latent vector sequence, the fifth multilayer perceptron first outputs the first description sentence for the sample video according to the second latent vector sequence; then according to the first description The sentence and the sample video description sentence corresponding to the sample video are calculated to obtain the first semantic loss.

为：In some embodiments of the present application, semantically supervised training is performed on the description generation model by calculating a negative log-likelihood loss function, wherein a first semantic loss function is defined

for:

Wherein, P(C|H ^sem ; V, C) is based on the sample video V and the video description sentence C corresponding to the sample video V. The semantic similarity between the first description sentence and the sample description sentence satisfies the second preset condition. The probability.

The second preset condition may be set according to a first semantic similarity threshold. For example, if the predicted semantic similarity between the first description sentence and the sample description sentence is greater than or equal to the first semantic similarity threshold, then deemed to satisfy the second preset condition.

It can be understood that in order to calculate the semantic similarity between the first description sentence and the sample description sentence, it is necessary to construct the semantic vector of the first description sentence and the semantic vector of the sample description sentence respectively. Therefore, according to the semantic vector of the first description sentence The similarity calculation is performed with the semantic vector of the sample description sentence, and the semantic similarity is correspondingly obtained.

Thus, the first semantic loss for the sample video and the sample video description sentence corresponding to the sample video is calculated according to the above-mentioned first semantic loss function.

Step 750: Calculate the first target loss according to the first syntactic loss and the first semantic loss.

In some embodiments of the present application, the first syntactic loss function and the first semantic loss may be weighted, and the weighted result may be used as the first target loss function.

In a specific embodiment, the first syntactic loss function and the first semantic loss function are added, and the added sum is used as the first target loss function, that is, the first target loss function L _v,c is:

The first target loss can be correspondingly obtained by bringing in the first syntactic loss calculated in the above step 740 and the first semantic loss calculated in the step 750 .

Step 760: Adjust the parameters describing the generative model based on the first target loss.

Thus, the parameters describing the generative model are adjusted according to the calculated first target loss until the first target loss function converges.

In the training process, due to the limited training data, the performance of the description generation model may be limited. Therefore, in order to avoid this situation, the following Figures 8 and 9 are introduced on the basis of the training method corresponding to Figure 7. The description generation model is further trained to assist the training method shown in FIG. 7 .

In some embodiments of the present application, as shown in FIG. 8 , the method further includes:

Step 810: Obtain a sample syntax feature vector of a sample sentence, where the sample sentence includes a sample example sentence and a sample video description sentence corresponding to the sample video.

Wherein, the syntactic feature vector of the sample sentence can be realized by using the third neural network and the fourth neural network above, and the specific process can be referred to the above description, which will not be repeated here.

Step 820: The first neural network outputs a first latent vector sequence according to the sample syntactic feature vector of the sample sentence, and calculates the second syntactic loss by using the first latent vector sequence corresponding to the sample sentence.

In some embodiments of the present application, in order to calculate the second syntactic loss by using the first latent vector sequence corresponding to the sample sentence, the sixth neural network is first used to predict the syntax tree for the sample sentence according to the first latent vector sequence corresponding to the sample sentence, The sixth neural network is a gated-based recurrent neural network; the second syntactic loss is then calculated according to the predicted syntax tree and the actual syntax tree of the sample sentence.

为：In some embodiments of the present application, the description generation model is syntactically supervised through a negative log-likelihood loss function. Define the second syntactic loss function

for:

Wherein, P(S ^syn |H ^syn ; S, S) indicates that the similarity between the syntax tree H ^syn predicted by the first latent vector sequence obtained from the sample sentence S and the actual syntax tree S ^syn of the sample sentence S satisfies the third preset conditional probability.

The third preset condition may be set according to the second syntax tree similarity threshold, for example, if the similarity between the syntax tree predicted for the sample sentence and the actual syntax tree of the sample sentence is greater than or equal to the second syntax tree The similarity threshold is deemed to satisfy the third preset condition.

Thus, the second syntactic losses for the sample sentences are calculated respectively according to the above-mentioned second syntactic loss function.

Step 830, the second neural network outputs the second latent vector sequence according to the sample semantic feature vector of the sample sentence and the first latent vector sequence corresponding to the sample sentence. The sample semantic feature vector is obtained by performing semantic feature extraction on the sample sentence, and is obtained by The second semantic loss is calculated for the second latent vector sequence corresponding to the sample sentence.

In some embodiments of the present application, a sentence semantic encoding module may be constructed in advance to perform semantic feature extraction on sample sentences. Specifically, for a sample sentence, first use the glove word vector to encode each word in the sample sentence, and then input the encoded word vector sequence into a long and short-term memory network, and the output of the long and short-term memory network is the sample. The sample semantic feature vector of the sentence.

In some embodiments of the present application, in order to calculate the second semantic loss based on the second latent vector sequence corresponding to the sample sentence, a multilayer perceptron is first used to output the third description for the sample sentence according to the second latent vector sequence corresponding to the sample sentence sentence; and then calculate the second semantic loss according to the third description sentence and the sample sentence.

为：In some embodiments of the present application, semantically supervised training is performed on the description generation model by calculating a negative log-likelihood loss function, wherein a second semantic loss function is defined

for:

Wherein, P(S|H ^sem ; S, S) is the probability that the semantic similarity between the third description sentence H ^sem and the sample sentence S predicted based on the sample sentence S satisfies the fourth preset condition.

The fourth preset condition may be set according to the second semantic similarity threshold. For example, if the predicted semantic similarity between the third description sentence and the sample sentence is greater than or equal to the second semantic similarity threshold, then the to satisfy the fourth preset condition.

Similarly, the semantic similarity calculation is performed according to the semantic vector of the third description sentence and the semantic vector of the sample sentence.

可以为样本语句对应计算得到第二语义损失。Thus, according to the above-mentioned second semantic loss function

The second semantic loss can be calculated for the sample sentence correspondence.

Step 840: Calculate and obtain a second target loss according to the second syntactic loss and the second semantic loss.

In some embodiments of the present application, the second syntactic loss function and the second semantic loss function are added, and the addition result is used as the second objective loss function, that is, the second objective loss function L _{s, s} is:

Therefore, on the basis of calculating the second syntactic loss and the second syntactic loss, the second target loss for the sample sentence is calculated according to the above formula (24).

Step 850: Adjust the parameters describing the generative model based on the second target loss.

In some embodiments of the present application, the training data also includes several sample example sentences, as shown in FIG. 9 , which also includes:

Step 910: Obtain a sample syntactic feature vector of the sample model sentence sentence.

Step 920, the first neural network outputs the first latent vector sequence according to the sample syntax feature vector of the sample example sentence.

Step 930, outputting a second latent vector sequence by the second neural network according to the first latent vector sequence corresponding to the sample paradigm sentence sentence and the sample video semantic feature vector of the sample video.

Step 940: Determine the second description sentence according to the second latent vector sequence corresponding to the sample video.

In some embodiments of the present application, a multilayer perceptron may be used to output a second description sentence for the sample video according to the second latent vector sequence of the sample video.

Step 950: Calculate and obtain a third target loss according to the syntax tree corresponding to the sample example sentence and the syntax tree corresponding to the second description sentence.

In some embodiments of the present application, the syntax tree of the sample example sentence and the syntax tree of the second description sentence can be obtained by a syntax analysis tool, such as the tools listed above; A recurrent neural network is used to determine the syntactic tree of the sample exemplar sentences and the second descriptive sentence.

为：In some embodiments of the present application, a third objective loss function is defined

for:

Wherein, P(E ^syn | H ^syn ; V, E) represents the similarity between the syntax tree H ^syn of the second description sentence obtained based on the sample video V and the sample paradigm sentence E and the syntax tree E ^syn of the sample paradigm sentence E The probability of meeting the pre-conditions of the item.

Wherein, the fifth preset condition may be set according to the third syntax tree similarity threshold, for example, if the similarity between the syntax tree of the second description sentence and the syntax tree of the sample example sentence is greater than or equal to the similarity of the third syntax tree threshold, it is considered that the fifth preset condition is met.

Therefore, the third target loss for the sample video and the sample example sentence is correspondingly calculated according to the above-mentioned third target loss function.

Step 960: Adjust the parameters describing the generative model based on the third target loss.

In some embodiments of the present application, the description generation model can be trained in combination with the three training methods shown in FIGS. 7-9 . In this case, the total loss function L of the description generation model can be defined as the first objective loss function , the sum of the second objective loss function and the third objective loss function, namely:

Of course, in other embodiments, based on actual needs, the description generation model may be trained only in combination with the training methods in FIGS. 7 and 8 , or only in combination with the training methods in FIGS. 7 and 9 .

FIG. 10 is a schematic diagram of generating a video description sentence according to an embodiment. As shown in FIG. 10 , first, the video semantic feature vector of the input video is extracted by the video semantic encoding module, and the example is obtained by extracting the sentence syntax encoding module of the hierarchical structure. Then, the description generation model outputs the video description sentence for the input video according to the video semantic feature vector of the input video and the syntactic feature vector of the example sentence.

Specifically, as shown in Figure 10, the video semantic coding module includes Convolutional Neural Networks (CNN) and Long Short-Term Memory Neural Network (LSTM). Feature extraction to obtain the semantic vector of each video frame, and then input the semantic vector of each video frame into the long-short-term memory neural network according to the time sequence of the video frame, and the long-short-term memory neural network outputs the hidden vector for each video, and then, The video semantic feature vector of the input video is obtained by combining the latent vectors corresponding to each video frame.

The hierarchical structure of sentence syntax coding module includes two layers of long-term and short-term memory neural networks, namely LSTM ^c and LSTM ^w , where the long-term and short-term memory neural network LSTM ^c is used to perform the character feature vector of each character in each word in the example sentence. encoding, the hidden vector output by the long short-term memory neural network LSTM ^c for the character; then, for each word in the example sentence, average calculation is performed according to the hidden vector of each character in the word, and the feature vector of the word is obtained; then, The long-short-term memory neural network LSTM ^w outputs the latent vector sequence according to the feature vector of each word in the exemplary sentence, and the output latent vector sequence is used as the syntactic feature vector of the exemplary sentence.

The description generation model includes a cascaded two-layer long and short-term memory neural network, namely LSTM ^syn (the first neural network) and LSTM ^sem (the second neural network), where the long and short-term memory neural network LSTM ^syn is used to control the video to be generated. The syntax of the description sentence, the long short-term memory neural network LSTM ^sem is used to give semantic meaning to the video description sentence. Specifically, the syntactic feature vector of the example sentence is input into the LSTM ^syn , the hidden vector output by the LSTM ^syn is used as the input of the LSTM ^sem , and the hidden vector output by the LSTM ^syn is used to syntactically guide the video description sentence to be generated; then; , LSTM ^sem outputs the hidden vector according to the hidden vector output by LSTM ^syn and the video semantic feature vector of the input video; finally, the word vector is determined according to the hidden vector output by LSTM ^sem , and then the video is generated according to the word corresponding to the determined word vector. Descriptive sentences are used to ensure that the generated video description sentences can describe the content of the video and are syntactically similar to the example sentences.

As shown in Figure 10, the exemplary sentence is "Aerial view of a group of sheeps on a grassfield.", and the video description sentence generated for the input video based on the exemplary sentence is "Cooking video of arecipe with ingredients in a glass bowl".

In some embodiments of the present application, after the video description sentence is generated, it can also be output through a voice technology.

The apparatus embodiments of the present application are introduced below, which can be used to execute the methods in the foregoing embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the above-mentioned method embodiments of the present application.

The present application provides an apparatus 1100 for generating video description sentences. The apparatus 1100 for generating video description sentences can be configured on the server shown in FIG. 1 . As shown in FIG. 11 , the apparatus 1100 for generating video description sentences includes:

The obtaining module 1110 is configured to obtain the syntactic feature vector of the target paradigm example sentence.

The syntax determination module 1120 is configured to determine the syntax of the video description sentence to be generated according to the syntax feature vector to obtain syntax information.

The semantic determination module 1130 is configured to determine the semantic corresponding to the syntax of the video description sentence to be generated according to the syntax information and the video semantic feature vector of the target video, and obtain the semantic information.

The video description sentence determining module 1140 is configured to generate a video description sentence of the target video according to the semantic information.

In some embodiments of the present application, the syntax determination module is configured to: generate a first latent vector according to the syntactic feature vector by the first neural network included in the description generation model, where the first latent vector is used to indicate syntax information, and the description generation model Also included is a second neural network cascaded with the first neural network, the first neural network and the second neural network being gated-based recurrent neural networks.

In this embodiment, the semantic determination module is configured to: generate a second latent vector by the second neural network according to the first latent vector and the video semantic feature vector, and the second latent vector is used to indicate semantic information.

In some embodiments of the present application, the video description sentence determination module is configured to: determine a word vector at time t according to a second latent vector generated by the second neural network at time t; generate a video description according to the word vector output at each time statement.

In this embodiment, the syntax determination module includes a first latent vector generation unit, which is configured to be generated by the first neural network according to the syntax feature vector, the word vector at time t-1, and the time t-1 generated by the first neural network. The first hidden vector, which outputs the first hidden vector at time t.

In this embodiment, the semantic determination module includes a second latent vector generating unit, which is configured to generate time t-1 by the second neural network according to the video semantic feature vector, the first latent vector at time t, and the second neural network The second latent vector of , outputs the second latent vector at time t.

In some embodiments of the present application, the first latent vector generating unit includes: a first soft attention weighting unit, configured to perform soft attention weighting on the syntactic feature vector according to the first latent vector at time t-1, to obtain a value corresponding to The target syntactic feature vector at time t. The first splicing unit is used for splicing the target syntax feature vector corresponding to time t and the word vector at time t-1 to obtain a first splicing vector corresponding to time t. The first output unit is configured to use the first splicing vector corresponding to time t as input by the first neural network, and output the first latent vector corresponding to time t.

In some embodiments of the present application, the first neural network includes a first input gate, a first forgetting gate, and a first output gate, and the first output unit includes: a first forgetting gate vector calculation unit, which is used for calculating the first forgetting gate by the first forgetting gate. The first forget gate vector at time t is calculated according to the first splicing vector corresponding to time t. and a first input gate vector calculation unit, configured to calculate the first input gate vector at time t by the first input gate according to the first splicing vector corresponding to time t. The first cell unit vector calculation unit is used to calculate the first forget gate vector at time t, the first input gate vector at time t, the first unit vector at time t, and the first neural network corresponding to time t-1. The cell unit vector is calculated to obtain the first cell unit vector at time t, and the first unit vector at time t is obtained by performing hyperbolic tangent calculation according to the first splicing vector corresponding to time t. The first hidden vector calculation unit is used to calculate the first hidden vector at time t according to the first cell unit vector at time t and the first output gate vector at time t, and the first output gate vector at time t is calculated by the first output gate vector. The gate is computed from the first splice vector corresponding to time t.

In some embodiments of the present application, the syntax determination module further includes: a first normalization unit, configured to compare the first input gate vector, the first forget gate vector, the first output gate vector, and the first input gate vector in the first neural network. The one-unit vectors are respectively normalized; the first transformation unit, according to the first offset vector and the first scaling vector, respectively normalizes the normalized first input gate vector, the first forget gate vector, the first output gate vector and the The first unit vector is transformed to obtain the target first input gate vector, the target first forget gate vector, the target first output gate vector and the target first unit vector, and the first offset vector is the first multi-layer perceptron according to the corresponding The first scaling vector is output by the target syntax feature vector at time t, and the first scaling vector is output by the second multi-layer perceptron according to the target syntax feature vector corresponding to time t. The first multi-layer perceptron is independent from the second multi-layer perceptron.

In this embodiment, the first cell unit vector calculation unit is further configured to: calculate according to the target first forget gate vector, the target first input gate vector, the target first unit vector and the first cell unit vector at time t-1 Obtain the first cell unit vector at time t;

In this embodiment, the first hidden vector calculation unit is further configured to: calculate and obtain the first hidden vector at time t according to the first cell unit vector at time t and the target output gate vector.

In some embodiments of the present application, the second latent vector generating unit includes: a second soft attention weighting unit, configured to perform soft attention weighting on the video semantic feature vector according to the second latent vector at time t-1, to obtain The target video semantic vector corresponding to time t. The second splicing unit is used for splicing the target video semantic vector corresponding to time t with the first latent vector at time t to obtain a second splicing vector corresponding to time t. The second output unit is used for the second neural network to take the second splicing vector corresponding to time t as an input, and to output the second latent vector corresponding to time t.

In some embodiments of the present application, the second neural network includes a second input gate, a second forgetting gate, and a second output gate, and the second output unit includes: a second forgetting gate vector computing unit, configured to generate a second forgetting gate by the second forgetting gate The second forget gate vector at time t is obtained by calculating according to the second splicing vector corresponding to time t. and a second input gate vector calculation unit, configured to calculate the second input gate vector at time t by the second input gate according to the second splicing vector corresponding to time t. The second cell unit vector calculation unit is used to calculate the second forget gate vector at time t, the second input gate vector at time t, the second unit vector at time t, and the second neural network corresponding to time t-1. The cell unit vector is calculated to obtain the second cell unit vector at time t, and the second unit vector at time t is obtained by performing hyperbolic tangent calculation according to the second splicing vector corresponding to time t. The second hidden vector calculation unit is used to calculate the second hidden vector at time t according to the second cell unit vector at time t and the second output gate vector at time t, and the second output gate vector at time t is obtained by the second output gate vector The gate is computed from the second splice vector corresponding to time t.

In some embodiments of the present application, the semantic determination module further includes: a second normalization unit, configured to compare the second input gate vector, the second forget gate vector, the second output gate vector and the first gate vector in the second neural network. Two-element vectors are normalized separately. The second transformation unit is configured to transform the normalized second input gate vector, the second forget gate vector, the second output gate vector and the second unit vector respectively according to the second offset vector and the second scaling vector, Obtain the target second input gate vector, the target second forget gate vector, the target second output gate vector and the target second unit vector, and the second offset vector is the third multilayer perceptron according to the target video semantic vector corresponding to time t. In the output, the second scaling vector is output by the fourth multi-layer perceptron according to the target video semantic vector corresponding to time t, and the third multi-layer perceptron is independent from the fourth multi-layer perceptron.

In this embodiment, the second cell unit vector calculation unit is further configured to: calculate according to the target second forget gate vector, the target second input gate vector, the target second cell vector and the second cell unit vector at time t-1 Obtain the second cell unit vector at time t;

In this embodiment, the second hidden vector calculation unit is further configured to: calculate and obtain the second hidden vector at time t according to the second cell unit vector at time t and the target second output gate vector.

In some embodiments of the present application, the apparatus for generating video description sentences further includes: a training data acquisition module, configured to acquire training data, where the training data includes several sample videos and sample video description sentences corresponding to the sample videos. The semantic feature extraction module is used to extract the semantic features of the sample video to obtain the sample video semantic feature vector of the sample video; and the syntactic feature extraction module is used to extract the syntactic features of the sample video description sentences corresponding to the sample video to obtain the sample video. A sample syntactic feature vector describing the sentence. The first syntax loss determining module is used for outputting the first latent vector sequence by the first neural network according to the sample syntax feature vector, and calculating the first syntax loss by using the first latent vector sequence. The first semantic loss determination module is configured to output a second latent vector sequence by the second neural network according to the first latent vector sequence and the sample video semantic feature vector of the sample video, and calculate the first semantic loss by using the second latent vector sequence. The first target loss calculation module is configured to calculate and obtain the first target loss according to the first syntactic loss and the first semantic loss. The first adjustment module is configured to adjust the parameters describing the generation model based on the first target loss.

In some embodiments of the present application, the first syntax loss determination module includes: a syntax tree prediction unit, configured to predict and obtain a syntax tree for the sample description sentence according to the first latent vector sequence through the sixth neural network, and the sixth neural network is Gated based recurrent neural network. The first syntax loss calculation unit is configured to calculate and obtain the first syntax loss according to the predicted syntax tree and the actual syntax tree of the sample description sentence.

In some embodiments of the present application, the first semantic loss determination module includes: a first description sentence output unit, configured to output the first description sentence for the sample video according to the second latent vector sequence through the fifth multilayer perceptron. The first semantic loss calculation unit is configured to calculate and obtain the first semantic loss according to the first description sentence and the sample video description sentence corresponding to the sample video.

In some embodiments of the present application, the apparatus for generating a video description sentence further includes: a first sample syntax feature vector acquisition module, configured to obtain a sample syntax feature vector of the sample sentence, where the sample sentence includes a sample sentence corresponding to a sample sentence and a sample video Sample video description sentences. The second syntactic loss calculation module is used for outputting the first latent vector sequence by the first neural network according to the sample syntactic feature vector of the sample sentence, and calculating the second syntactic loss through the first latent vector sequence corresponding to the sample sentence. The second semantic loss calculation module is used for the second neural network to output the second latent vector sequence according to the sample semantic feature vector of the sample sentence and the first latent vector sequence corresponding to the sample sentence. The sample semantic feature vector is the semantic feature of the sample sentence. The obtained result is extracted, and the second semantic loss is calculated through the second latent vector sequence corresponding to the sample sentence. The second target loss calculation module is configured to calculate and obtain the second target loss according to the second syntactic loss and the second semantic loss. The second adjustment module is configured to adjust the parameters describing the generation model based on the second target loss.

In some embodiments of the present application, the training data further includes several sample exemplary sentences, and the apparatus for generating a video description sentence further includes: a second sample syntactic feature vector acquisition module, configured to acquire the sample syntactic feature vectors of the sample exemplary sentences. The first latent vector sequence output module is used for outputting the first latent vector sequence by the first neural network according to the sample syntactic feature vector of the sample example sentence. The second latent vector sequence output module is used for outputting the second latent vector sequence by the second neural network according to the first latent vector sequence corresponding to the sample example sentence and the sample video semantic feature vector of the sample video. The second description sentence determining module is configured to determine the second description sentence according to the second latent vector sequence corresponding to the sample video. The third target loss calculation module is configured to calculate and obtain the third target loss according to the syntax tree corresponding to the sample example sentence and the syntax tree corresponding to the second description sentence. The third adjustment module is configured to adjust the parameters describing the generation model based on the third target loss.

In some embodiments of the present application, the syntactic feature vector corresponding to the target paradigm example sentence is obtained through a syntactic model, the syntactic model includes a cascaded third neural network and a fourth neural network, and the third neural network and the fourth neural network are gate-based controlled recurrent neural network. In this embodiment, the obtaining module includes: a character feature vector obtaining unit, configured to obtain character feature vectors of characters included in each word in the target paradigm example sentence, where the character feature vectors are obtained by encoding the characters. The third latent vector output unit is used for outputting the third latent vector corresponding to each character by the third neural network according to the character feature vector of each character. The average calculation unit is used to perform average calculation for each word in the target example sentence according to the third latent vector corresponding to each character in the word to obtain the feature vector of the word. The fourth latent vector output unit is used for outputting the fourth latent vector by the fourth neural network according to the feature vector of each word in the target example sentence sentence, and the fourth latent vector is used as the syntactic feature vector.

In some embodiments of the present application, a video semantic feature vector corresponding to the target video is obtained through a video semantic model, the video semantic model includes a cascaded convolutional neural network and a fifth neural network, and the fifth neural network is a gated-based loop The neural network, and the apparatus for generating video description sentences further include: a video frame sequence acquisition module, configured to acquire a video frame sequence obtained by dividing the target video into frames. The semantic extraction module is used to perform semantic extraction on each video frame in the video frame sequence through the convolutional neural network, and obtain the semantic vector of each video frame. The fifth latent vector output module is used for outputting the fifth latent vector according to the semantic vector of each video frame in the video frame sequence through the fifth neural network, and the fifth latent vector is used as the video semantic feature vector.

FIG. 12 shows a schematic structural diagram of a computer system suitable for implementing the electronic device according to the embodiment of the present application.

It should be noted that the computer system 1200 of the electronic device shown in FIG. 12 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

As shown in FIG. 12, the computer system 1200 includes a central processing unit (Central Processing Unit, CPU) 1201, which can be loaded into a random device according to a program stored in a read-only memory (Read-Only Memory, ROM) 1202 or from a storage part 1208 A program in a memory (Random Access Memory, RAM) 1203 is accessed to perform various appropriate actions and processes, for example, the methods in the above embodiments are performed. In the RAM 1203, various programs and data necessary for system operation are also stored. The CPU 1201 , the ROM 1202 , and the RAM 1203 are connected to each other through a bus 1204 . An Input/Output (I/O) interface 1205 is also connected to the bus 1204 .

The following components are connected to the I/O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc. ; a storage section 1208 including a hard disk and the like; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication section 1209 performs communication processing via a network such as the Internet. Drivers 1210 are also connected to I/O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 1210 as needed so that a computer program read therefrom is installed into the storage section 1208 as needed.

In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network through the communication portion 1209, and/or installed from the removable medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, various functions defined in the system of the present application are executed.

It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Erasable Programmable Read Only Memory (EPROM), flash memory, optical fiber, portable Compact Disc Read-Only Memory (CD-ROM), optical storage device, magnetic storage device, or any suitable of the above The combination. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Wherein, each block in the flowchart or block diagram may represent a module, program segment, or part of code, and the above-mentioned module, program segment, or part of code contains one or more executables for realizing the specified logical function instruction. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

The units involved in the embodiments of the present application may be implemented in software or hardware, and the described units may also be provided in a processor. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances.

In one aspect of the embodiments of the present application, an electronic device is provided, the electronic device includes: a processor; and a memory, where computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, any one of the foregoing implementations is implemented The generation method of the video description sentence in the example.

As another aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium may be included in the electronic device described in the above-mentioned embodiments; in electronic equipment. Computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by a processor, implement the method in any of the above embodiments.

It should be noted that although several modules or units of the apparatus for action performance are mentioned in the above detailed description, this division is not mandatory. Indeed, according to embodiments of the present application, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above may be further divided into multiple modules or units to be embodied.

Those skilled in the art can easily understand from the description of the above embodiments that the exemplary embodiments described herein may be implemented by software, or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present application may be embodied in the form of software products, and the software products may be stored in a non-volatile storage medium (which may be CD-ROM, U disk, mobile hard disk, etc.) or on the network , which includes several instructions to cause a computing device (which may be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiment of the present application.

Other embodiments of the present application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses or adaptations of this application that follow the general principles of this application and include common knowledge or conventional techniques in the technical field not disclosed in this application .

It is to be understood that the present application is not limited to the precise structures described above and illustrated in the accompanying drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims (17)

1. A method for generating a video description sentence, the method comprising:

obtaining a syntactic characteristic vector of a target example sentence;

determining the syntax of a video description sentence to be generated according to the syntax feature vector to obtain syntax information;

determining the semantics of the video description sentence to be generated corresponding to the syntax according to the syntax information and the video semantic feature vector of the target video to obtain semantic information;

and generating a video description statement of the target video according to the semantic information.

2. The method of claim 1, wherein determining the syntax of the video description sentence to be generated according to the syntax feature vector, and obtaining syntax information comprises:

generating a first hidden vector according to the syntactic characteristic vector by a first neural network contained in a description generation model, wherein the first hidden vector is used for indicating the syntactic information, the description generation model further comprises a second neural network, and the first neural network and the second neural network are gate-based cyclic neural networks;

determining the semantics of the to-be-generated video description sentence corresponding to the syntax according to the syntax information and the video semantic feature vector of the target video to obtain semantic information, wherein the semantic information comprises:

generating, by the second neural network, a second hidden vector from the first hidden vector and the video semantic feature vector, the second hidden vector being used to indicate the semantic information.

3. The method of claim 2, wherein the generating a video description sentence of the target video according to the semantic information comprises:

determining a word vector at the t moment according to a second hidden vector generated by the second neural network at the t moment;

generating the video description sentence according to the word vector output at each moment;

generating a first hidden vector by a first neural network contained in a description generation model according to the syntactic feature vector, wherein the first hidden vector comprises:

outputting, by the first neural network, a first hidden vector at a time t according to the syntactic feature vector, the word vector at the time t-1 and the first hidden vector at the time t-1 generated by the first neural network;

generating, by the second neural network, a second hidden vector from the first hidden vector and the video semantic feature vector, comprising:

and outputting the second hidden vector at the t moment by the second neural network according to the video semantic feature vector, the first hidden vector at the t moment and the second hidden vector at the t-1 moment generated by the second neural network.

4. The method of claim 3, wherein outputting, by the first neural network, the first hidden vector at time t from the syntactic feature vector, the word vector at time t-1, and the first hidden vector at time t-1 generated by the first neural network comprises:

carrying out soft attention weighting on the syntactic characteristic vector according to the first implicit vector at the time t-1 to obtain a target syntactic characteristic vector corresponding to the time t;

splicing the target syntactic characteristic vector corresponding to the time t with the word vector at the time t-1 to obtain a first spliced vector corresponding to the time t;

and correspondingly outputting a first implicit vector at the time t by using the first splicing vector corresponding to the time t as an input of the first neural network.

5. The method of claim 4, wherein the first neural network comprises a first input gate, a first forgetting gate, and a first output gate, wherein the outputting, by the first neural network, the first stitched vector corresponding to time t as an input and the first hidden vector corresponding to time t as an output comprises:

calculating by the first forgetting gate according to the first splicing vector corresponding to the time t to obtain a first forgetting gate vector at the time t; calculating to obtain a first input gate vector at the t moment by the first input gate according to the first splicing vector corresponding to the t moment;

calculating to obtain a first cell unit vector at the time t according to the first forgetting gate vector at the time t, the first input gate vector at the time t, the first cell unit vector at the time t and the first cell unit vector at the time t-1 corresponding to the first neural network, wherein the first cell unit vector at the time t is obtained by performing hyperbolic tangent calculation according to the first splicing vector corresponding to the time t;

and calculating to obtain a first implicit vector at the time t according to the first cell unit vector at the time t and a first output gate vector at the time t, wherein the first output gate vector at the time t is calculated by the first output gate according to the first splicing vector corresponding to the time t.

6. The method of claim 5, wherein before the calculating the first cell unit vector at time t according to the first forgetting gate vector at time t, the first input gate vector at time t, the first unit vector at time t, and the first cell unit vector at time t-1 corresponding to the first neural network, the method further comprises:

respectively normalizing a first input gate vector, a first forgetting gate vector, a first output gate vector and a first unit vector in the first neural network;

respectively transforming the normalized first input gate vector, the normalized first forgetting gate vector, the normalized first output gate vector and the normalized first unit vector according to a first offset vector and a first scaling vector to obtain a target first input gate vector, a target first forgetting gate vector, a target first output gate vector and a target first unit vector, wherein the first offset vector is output by a first multilayer perceptron according to the target syntactic feature vector corresponding to the moment t, the first scaling vector is output by a second multilayer perceptron according to the target syntactic feature vector corresponding to the moment t, and the first multilayer perceptron and the second multilayer perceptron are independent;

the calculating according to the first forgetting gate vector at the time t, the first input gate vector at the time t, the first unit vector at the time t and the first cell unit vector at the time t-1 corresponding to the first neural network to obtain the first cell unit vector at the time t includes:

calculating to obtain a first cell unit vector at the t moment according to the target first forgetting gate vector, the target first input gate vector, the target first cell unit vector and the first cell unit vector at the t-1 moment;

the calculating to obtain the first implicit vector at the time t according to the first cell unit vector at the time t and the first output gate vector at the time t includes:

and calculating to obtain a first implicit vector at the time t according to the first cell unit vector at the time t and the target output gate vector.

7. The method of claim 3, wherein outputting, by the second neural network, the second hidden vector at time t from the video semantic feature vector, the first hidden vector at time t, and the second hidden vector at time t-1 generated by the second neural network comprises:

carrying out soft attention weighting on the video semantic feature vector according to the second hidden vector at the time t-1 to obtain a target video semantic vector corresponding to the time t;

splicing the target video semantic vector corresponding to the time t with the first hidden vector corresponding to the time t to obtain a second spliced vector corresponding to the time t;

and correspondingly outputting a second implicit vector at the t moment by the second neural network by taking the second splicing vector corresponding to the t moment as an input.

8. The method of claim 7, wherein the second neural network comprises a second input gate, a second forgetting gate, and a second output gate, wherein the outputting, by the second neural network, the second stitched vector corresponding to time t as an input and the second implicit vector corresponding to time t as an output comprises:

calculating by the second forgetting gate according to the second splicing vector corresponding to the time t to obtain a second forgetting gate vector at the time t; and calculating a second input gate vector at the time t by the second input gate according to the second splicing vector corresponding to the time t;

calculating a second cell unit vector at the time t according to the second forgetting gate vector at the time t, the second input gate vector at the time t, the second cell unit vector at the time t and the second cell unit vector at the time t-1 corresponding to the second neural network, wherein the second cell unit vector at the time t is obtained by performing hyperbolic tangent calculation according to the second splicing vector corresponding to the time t;

and calculating to obtain a second implicit vector at the time t according to the second cell unit vector at the time t and a second output gate vector at the time t, wherein the second output gate vector at the time t is calculated by the second output gate according to the second splicing vector corresponding to the time t.

9. The method of claim 8, wherein before the calculating the second cell unit vector at time t according to the second forgetting gate vector at time t, the second input gate vector at time t, the second cell unit vector at time t, and the second cell unit vector at time t-1 corresponding to the second neural network, the method further comprises:

respectively normalizing a second input gate vector, a second forgetting gate vector, a second output gate vector and a second unit vector in the second neural network;

respectively transforming the normalized second input gate vector, the normalized second forgetting gate vector, the normalized second output gate vector and the normalized second unit vector according to a second offset vector and a second scaling vector to obtain a target second input gate vector, a target second forgetting gate vector, a target second output gate vector and a target second unit vector, wherein the second offset vector is output by a third multilayer perceptron according to the target video semantic vector corresponding to the moment t, the second scaling vector is output by a fourth multilayer perceptron according to the target video semantic vector corresponding to the moment t, and the third multilayer perceptron and the fourth multilayer perceptron are independent;

the calculating according to the second forgetting gate vector at the time t, the second input gate vector at the time t, the second unit vector at the time t and the second unit vector at the time t-1 corresponding to the second neural network to obtain the second cell unit vector at the time t includes:

calculating to obtain a second cell unit vector at the t moment according to the target second forgetting gate vector, the target second input gate vector, the target second cell unit vector and the second cell unit vector at the t-1 moment;

the calculating to obtain a second implicit vector at the time t according to the second cell unit vector at the time t and a second output gate vector at the time t includes:

and calculating to obtain a second implicit vector at the time t according to the second cell unit vector at the time t and the target second output gate vector.

10. The method of claim 2, further comprising:

acquiring training data, wherein the training data comprises a plurality of sample videos and sample video description sentences corresponding to the sample videos;

semantic feature extraction is carried out on the sample video to obtain a sample video semantic feature vector of the sample video; carrying out syntactic feature extraction on a sample video description statement corresponding to the sample video to obtain a sample syntactic feature vector of the sample video description statement;

outputting, by the first neural network, a first sequence of hidden vectors from the sample syntactic feature vectors, a first syntactic loss being calculated from the first sequence of hidden vectors;

outputting a second hidden vector sequence by the second neural network according to the first hidden vector sequence and the sample video semantic feature vector of the sample video, and calculating a first semantic loss through the second hidden vector sequence;

calculating to obtain a first target loss according to the first syntax loss and the first semantic loss;

adjusting parameters of the description generative model based on the first target loss.

11. The method of claim 10, wherein said computing a first syntactic loss by said first sequence of hidden vectors comprises:

predicting a syntax tree for the sample description statement according to the first hidden vector sequence through a sixth neural network, wherein the sixth neural network is a gate-controlled cyclic neural network;

and calculating the first syntax loss according to the predicted syntax tree and the actual syntax tree of the sample description statement.

12. The method of claim 10, wherein said calculating a first semantic loss through said second sequence of hidden vectors comprises:

outputting a first description statement for the sample video according to the second hidden vector sequence by a fifth multilayer perceptron;

and calculating to obtain the first semantic loss according to the first description statement and the sample video description statement corresponding to the sample video.

13. The method of claim 10, further comprising:

obtaining a sample syntax feature vector of a sample statement, wherein the sample statement comprises a sample example sentence and a sample video description statement corresponding to a sample video;

outputting a first hidden vector sequence by the first neural network according to the sample syntax feature vector of the sample statement, and calculating a second syntax loss through the first hidden vector sequence corresponding to the sample statement;

outputting a second hidden vector sequence by the second neural network according to a sample semantic feature vector of the sample statement and the first hidden vector sequence corresponding to the sample statement, wherein the sample semantic feature vector is obtained by performing semantic feature extraction on the sample statement, and calculating a second semantic loss through the second hidden vector sequence corresponding to the sample statement;

calculating to obtain a second target loss according to the second syntax loss and the second semantic loss;

adjusting parameters of the description generative model based on the second target loss.

14. The method of claim 10 or 13, wherein the training data further comprises a number of sample example sentences, the method further comprising:

obtaining a sample syntax feature vector of the sample example sentence;

outputting, by the first neural network, a first hidden vector sequence according to the sample syntactic feature vector of the sample example sentence;

outputting, by the second neural network, a second hidden vector sequence according to the first hidden vector sequence corresponding to the sample example sentence and a sample video semantic feature vector of the sample video;

determining a second description statement from a second sequence of hidden vectors corresponding to the sample video;

calculating to obtain a third target loss according to the syntax tree corresponding to the sample example sentence and the syntax tree corresponding to the second description sentence;

adjusting parameters of the description generative model based on the third target loss.

15. The method of claim 1, wherein the obtaining of the syntactic feature vector of the target example sentence comprises:

acquiring character feature vectors of characters included by each word in the target example sentence, wherein the character feature vectors are obtained by coding characters;

outputting a third implicit vector corresponding to each character by a third neural network according to the character feature vector of each character;

aiming at each word in the target example sentence, carrying out average calculation according to a third hidden vector corresponding to each character in the word to obtain a feature vector of the word;

outputting a fourth hidden vector sequence by a fourth neural network according to the feature vector of each word in the target example sentence, wherein the fourth hidden vector sequence is used as the syntactic feature vector, and the third neural network and the fourth neural network are gate-controlled-based cyclic neural networks.

16. The method according to claim 1, wherein before determining the semantic meaning of the video description sentence to be generated corresponding to the syntax according to the syntax information and the video semantic feature vector of the target video, and obtaining the semantic information, the method further comprises:

acquiring a video frame sequence obtained by framing the target video;

semantic extraction is carried out on each video frame in the video frame sequence through a convolutional neural network to obtain a semantic vector of each video frame;

outputting a fifth hidden vector sequence according to the semantic vector of each video frame in the video frame sequence through a fifth neural network, wherein the fifth hidden vector sequence is used as the video semantic feature vector, and the fifth neural network is a gate-controlled cyclic neural network.

17. An apparatus for generating a video description sentence, the apparatus comprising:

the obtaining module is used for obtaining the syntactic characteristic vector of the target example sentence;

the syntax determining module is used for determining the syntax of the video description sentence to be generated according to the syntax feature vector to obtain syntax information;

the semantic determining module is used for determining the semantic of the video description sentence to be generated corresponding to the syntax according to the syntax information and the video semantic feature vector of the target video to obtain semantic information;

and the video description statement determining module is used for generating the video description statement of the target video according to the semantic information.

Images