WO2025026012A1 - Video retrieval method - Google Patents

Abstract

Embodiments of the present description provide a video retrieval method. The video retrieval method comprises: obtaining a retrieval text and at least one candidate video; inputting the retrieval text and a target candidate video into a video matching model to obtain a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, the matching weight is determined based on a first matching result and a second matching result between the retrieval text and the target candidate video, the first matching result is used for representing the matching degree between the retrieval text and each target object in the target candidate video, and the second matching result is used for representing the matching degree between the retrieval text and video semantics of the target candidate video; and determining at least one target video among the at least one candidate video on the basis of the matching weight corresponding to each candidate video. Local content and overall content of the candidate video are analyzed to improve the accuracy of text-based video retrieval.

Description

Video Retrieval Methods

This application claims the priority of the Chinese patent application filed with the China Patent Office on August 1, 2023, with application number 202310961671.3 and application name “Video Retrieval Method”, all contents of which are incorporated by reference in this application.

Technical Field

The embodiments of the present specification relate to the field of data processing technology, and in particular to a video retrieval method.

Background Art

With the development of various video platforms, people usually spend their free time watching videos. Due to different personal viewing needs, if users are not interested in the videos recommended by the video platform, they will conduct independent searches based on text to obtain videos. In current practical applications, searching for semantically related videos based on the natural language input by users often requires appropriate matching modeling between video and text data. Or, it is necessary to extract features and perform feature fusion by training multiple single-modal pre-trained models.

However, since videos contain rich visual elements, text descriptions may only correspond to part of the video content, lacking the perception of the overall and local content of the video, making the accuracy of video retrieval insufficient to meet user needs and reducing the user experience. Therefore, a method is urgently needed to solve the above problems.

Summary of the invention

In view of this, an embodiment of this specification provides a video retrieval method. One or more embodiments of this specification also relate to a video retrieval device, a computing device, a computer-readable storage medium and a computer program to solve the technical defects existing in the prior art.

According to a first aspect of an embodiment of this specification, a video retrieval method is provided, including:

Obtaining a search text and at least one candidate video;

The search text and the target candidate video are input into the video matching model to obtain the a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the search text and the target candidate video, wherein the first matching result is used to characterize the matching degree between the search text and each target object in the target candidate video, and the second matching result is used to characterize the matching degree between the search text and the video semantics of the target candidate video;

At least one target video is determined from the at least one candidate video based on the matching weights corresponding to the candidate videos.

According to a second aspect of the embodiments of this specification, a video retrieval device is provided, including:

An acquisition module, configured to acquire a search text and at least one candidate video;

An input module is configured to input the search text and the target candidate video into a video matching model, and obtain a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the search text and the target candidate video, wherein the first matching result is used to characterize the matching degree between the search text and each target object in the target candidate video, and the second matching result is used to characterize the matching degree between the search text and the video semantics of the target candidate video;

The determination module is configured to determine at least one target video from the at least one candidate video based on the matching weight corresponding to each candidate video.

According to a third aspect of an embodiment of this specification, a computing device is provided, including:

Memory and processor;

The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above-mentioned video retrieval method are implemented.

According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned video retrieval method are implemented.

According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned video retrieval method.

The video retrieval method provided in this specification includes: obtaining a retrieval text and at least one candidate video; inputting the search text and the target candidate video into a video matching model to obtain a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the search text and the target candidate video, the first matching result being used to characterize the matching degree between the search text and each target object in the target candidate video, and the second matching result being used to characterize the matching degree between the search text and the video semantics of the target candidate video; based on the matching weight corresponding to each candidate video, determining at least one target video in the at least one candidate video.

In one embodiment of the present specification, the first matching result between the search text and the target candidate video is calculated to determine the matching degree between the search text and each target object in the target candidate video, and the second matching result between the search text and the target candidate video is calculated to determine the matching degree between the search text and the video semantics of the target candidate video. Then, the matching weight of the target candidate video is determined by combining the first matching result and the second matching result, and the accuracy of determining the matching weight of the target candidate video is improved by analyzing the local content and the overall content of the target candidate video; then, the target video is determined in each candidate video according to the matching weight corresponding to each candidate video, thereby improving the accuracy of text-based video retrieval.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is an architecture diagram of a video retrieval system provided by an embodiment of the present specification;

FIG2 is a flow chart of a video retrieval method provided by an embodiment of the present specification;

FIG3 is a schematic diagram of a model architecture of a video matching model provided by an embodiment of the present specification;

FIG4a is a schematic diagram of an interactive interface of a video retrieval method provided by an embodiment of this specification;

FIG4b is a schematic diagram of an interactive interface of another video retrieval method provided by an embodiment of the present specification;

FIG5 is a flow chart of a processing process of a video retrieval method provided by an embodiment of the present specification;

FIG6 is a schematic diagram of the structure of a video retrieval device provided by an embodiment of this specification;

FIG. 7 is a structural block diagram of a computing device provided by an embodiment of the present specification.

DETAILED DESCRIPTION

Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and/or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and corresponding operation entrances shall be provided for users to choose to authorize or refuse.

The large model in one or more embodiments of this specification specifically refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, or even hundreds of billions of model parameters. The large model can also be called a foundation model/foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as large-scale language models (LLM), multi-modal pre-training models, etc.

In actual application, large models only need a small number of samples to fine-tune the pre-trained model and can be applied to different tasks. Large models can be widely used in natural language processing (NLP), computer vision and other fields. It is applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the large model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

First, the terms involved in one or more embodiments of this specification are explained.

Video text retrieval: Traditional image retrieval tasks are only applicable to searches between images, and their application scenarios are limited. Video text retrieval can achieve cross-modal search between natural language and video, and its model is generally trained based on a large amount of video-text data pairs.

CLIP: is a multimodal pre-training model. It can process images and texts simultaneously and combine the two for tasks such as classification, retrieval, and generation. During the training process, CLIP uses a large amount of image and text data. Its pre-training goal is to learn to align the representations of text and images so that similar text and images are closer in the embedding space. To achieve this goal, CLIP uses a contrastive learning method to maximize the cosine similarity of similar text and images, while minimizing the cosine similarity of dissimilar text and images, thereby training an embedding space that can align data of different modalities.

Transformer: A model that mainly uses self-attention to obtain long-term dependencies in a sequence. Transformer contains an encoder and a decoder. The encoder is used to generate a representation of the input sequence. The decoder is used to generate a representation of the target sequence. Both the encoder and the decoder are composed of multiple layers of self-attention and feedforward neural networks. In addition, the self-attention mechanism in Transformer is a mechanism that calculates the representation of each element in the sequence, which can help the model focus on the part of the input sequence related to the current position and generate an importance weight vector. For example, when a sentence is input, the meaning of each word in the sentence is related to the other words in the sentence.

Prototype: A representative representation vector. For example, an event can be represented by an event-level vector (prototype).

Understanding multimodal information is an important way for humans to perceive the world. As a basic task in multimodal understanding, video text retrieval has attracted great interest from researchers with the rapid development of short video platforms. Video text retrieval aims to search for semantically related videos based on natural language input by users, which requires appropriate matching modeling between video and text data. However, the inherent modal difference phenomenon increases the difficulty of associating multimodal data. To address this problem, in practical applications, multiple unimodal pre-trained models are usually used to extract features, and then metric learning strategies are used to strengthen modal alignment in the joint space. However, there are large differences in the initial distributions of multiple unimodal offline features, which inevitably brings challenges to feature fusion and affects the retrieval results.

In addition, since videos contain rich visual elements, text descriptions may only correspond to part of the video content and lack the perception of the overall and local content of the video, making the accuracy of video retrieval insufficient to meet user needs and reducing the user experience.

In this specification, a video retrieval method is provided. This specification also relates to a video retrieval system, a video retrieval device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

Referring to FIG. 1 , FIG. 1 shows an architecture diagram of a video retrieval system provided by an embodiment of the present specification. The video retrieval system may include a client 100 and a server 200;

The client 100 is used to send a video retrieval instruction to the server 200;

The server 200 is used to receive a video search instruction sent by the client 100; obtain a search text and at least one candidate video in response to the video search instruction; input the search text and the target candidate video into a video matching model to obtain a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the search text and the target candidate video, wherein the first matching result is used to characterize the matching degree between the search text and each target object in the target candidate video, and the second matching result is used to characterize the matching degree between the search text and the video semantics of the target candidate video; determine at least one target video in the at least one candidate video based on the matching weight corresponding to each candidate video; and send at least one target video to the client 100;

The client 100 is further configured to receive at least one target video sent by the server 200 .

By applying the scheme of the embodiments of the present specification, a user can trigger a video retrieval instruction through a client and send the video retrieval instruction to a server. After receiving the video retrieval instruction, the server obtains the retrieval text carried in the video retrieval instruction and obtains multiple candidate videos from a preset database, inputs the retrieval text and the candidate videos into a video matching model to determine a target video associated with the retrieval text among multiple candidate videos, and feeds the target video back to the client.

The video retrieval system may include multiple clients 100 and a server 200, wherein the client 100 may be referred to as a client-side device and the server 200 may be referred to as a cloud-side device. A communication connection may be established between multiple clients 100 through the server 200. In the video retrieval scenario, the server 200 is used to provide video retrieval services between multiple clients 100. Multiple clients 100 may serve as a sender or a receiver, respectively, and achieve communication through the server 200.

The user can interact with the server 200 through the client 100 to receive other client 100, or sends data to other clients 100, etc. In the video retrieval scenario, the user may publish a data stream to the server 200 through the client 100, and the server 200 determines at least one target video according to the data stream and pushes the at least one target video to other clients that establish communication.

The client 100 and the server 200 are connected via a network. The network provides a medium for a communication link between the client 100 and the server 200. The network may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, etc. before being released to the server 200.

The client 100 may be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language5, the fifth edition of Hypertext Markup Language) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The client 100 may be based on the software development kit (SDK, Software Development Kit) of the corresponding service provided by the server 200, such as based on the real-time communication (RTC, Real Time Communication) SDK development and acquisition, etc. The client 100 may be deployed in an electronic device, and may need to rely on the device to run or some APPs in the device to run, etc. For example, the electronic device may have a display screen and support information browsing, etc., such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications may also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers for background training that support models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

It is worth noting that the video retrieval method provided in the embodiments of this specification is generally executed by the server, but in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the video retrieval method provided in the embodiments of this specification. The client and the server execute together.

Referring to FIG. 2 , FIG. 2 shows a flow chart of a video retrieval method provided by an embodiment of the present specification, which specifically includes the following steps:

Step 202: Obtain a search text and at least one candidate video.

In actual applications, various short video platforms are gradually favored by the public, and people tend to kill time by refreshing and watching videos. However, in the process of watching videos, there may be cases where users are not interested in the video content recommended by the short video platform. In this case, users tend to search for videos in the search bar.

In this scenario, although each short video platform will make corresponding video recommendations based on the text entered by the user, there may also be a problem that the videos recommended by the short video platform do not match the text entered by the user's search or the matching degree is low, thereby reducing the user's viewing experience.

The video retrieval method provided in an embodiment of the present specification can recommend videos matching the retrieval text to the user based on the retrieval text input by the user.

The search text refers to the text used to search for videos, and the search text may be a keyword, a phrase, a sentence, or a text paragraph, etc. The candidate video refers to a video stored in a preset database, which is video data for searching videos based on the search text.

For example, if it is necessary to retrieve videos related to "cooking noodles" from database A, then "cooking noodles" is the search text, and each video stored in database A is a candidate video.

Specifically, when a user needs to search for videos, the user can manually enter the search text to be searched in the search bar, or input the search text by voice, thereby generating a video search instruction. The client sends the video search instruction to the server. After receiving the video search instruction, the server obtains the search text carried in the video search instruction and obtains the candidate videos in the preset database.

For example, if a user wants to obtain videos related to "cooking noodles", he can enter the word "cooking noodles" or text related to "cooking noodles" in the search bar of the client. Then, the client sends the video retrieval instruction generated for "cooking noodles" to the server. After receiving the video retrieval instruction, the server obtains the retrieval text "cooking noodles" carried in the video retrieval instruction and obtains multiple candidate videos from the preset database.

After obtaining the search text and multiple candidate videos, the search text and the multiple candidate videos can be matched to determine the matching degree between the search text and the multiple candidate videos, and based on the matching degree between the search text and the multiple candidate videos, determine the video that the user needs to retrieve.

In order to improve the retrieval efficiency of video text retrieval, the video matching model can be used to The search text and each candidate video are processed.

Step 204: input the search text and the target candidate video into a video matching model to obtain a matching weight corresponding to the target candidate video output by the video matching model.

The target candidate video refers to any one of the candidate videos. The video matching model is used to determine the matching weight between the search text and each candidate video. The matching weight is used to characterize the matching degree between the search text and each candidate video, which can be expressed by a numerical value, such as 0.5, 0.8, or 50%, 80%, etc.

Specifically, the acquired search text and target candidate video are input into the video matching model, and the matching weight corresponding to the target candidate video output by the video matching model can be obtained.

In a specific implementation provided in this specification, the video matching model includes a coding layer, a first matching layer, and a second matching layer;

Inputting the search text and the target candidate video into a video matching model, and obtaining a matching weight corresponding to the target candidate video output by the video matching model, comprises:

Inputting the search text and the target candidate video into the coding layer to obtain a text feature sequence of the search text and a target video frame block feature sequence of the target candidate video;

Inputting the text feature sequence and the target video frame block feature sequence into the first matching layer, obtaining a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence, and determining a first matching result according to the phrase feature sequence and the target object feature sequence;

The target object feature sequence, the text feature sequence and the first matching result are input into the second matching layer, a second matching result is determined according to the target object feature sequence and the text feature sequence, and a matching weight corresponding to the target candidate video is determined based on the first matching result and the second matching result.

Among them, the encoding layer is used to encode the search text and each candidate video to obtain a text feature sequence corresponding to the search text; the first matching layer is used to determine the first matching result between the search text and each candidate video, and the first matching result is used to characterize the matching degree between the search text and each target object in each candidate video, and the target object may refer to a person, an object, etc. in the video frame; the second matching layer is used to determine the second matching result between the search text and each candidate video, and the second matching result is used to characterize the matching degree between the search text and the video semantics of each candidate video.

Specifically, the retrieval text and the target candidate video are input into the video matching model. First, the retrieval text and the target candidate video need to be input into the encoding layer of the video matching model, and the retrieval text and the target candidate video are encoded respectively to obtain the text feature sequence corresponding to the retrieval text and the target video frame block feature sequence corresponding to the target candidate video.

The text feature sequence refers to a feature sequence composed of text feature vectors corresponding to the search text, and the text feature vectors include word feature vectors and identifier feature vectors; the target video frame block feature sequence refers to a feature sequence composed of target video frame block feature vectors corresponding to the target candidate video, and the target video frame block feature vectors include key frame block feature vectors and global frame feature vectors.

Since the retrieved text and the target candidate video belong to data of different modalities, the retrieved text and the target candidate video can be encoded using a text encoder and a video encoder respectively.

In a specific implementation provided in this specification, the encoding layer includes a text encoder and a video encoder;

Inputting the search text and the target candidate video into a video matching model to obtain a text feature sequence of the search text and a target video frame block feature sequence of the target candidate video, including:

Inputting the search text into the text encoder to obtain a text feature sequence of the search text;

The target candidate video is input into the video encoder to obtain a target video frame block feature sequence of the target candidate video.

The text encoder refers to an encoder used to encode the retrieved text, and the video encoder refers to an encoder used to encode the candidate video.

The search text and the target candidate video are input into the encoding layer of the video matching model, that is, the search text is input into the text encoder, and the target candidate video is input into the video encoder, thereby obtaining the text feature sequence corresponding to the search text and the target video frame block feature sequence corresponding to the target candidate video.

Specifically, in a specific implementation provided in this specification, obtaining the text feature sequence of the search text includes:

Identify the search text and obtain words in the search text;

Encode each word to obtain the word feature vector corresponding to each word;

Obtaining preset text identifiers, encoding each preset text identifier, and obtaining an identifier feature vector corresponding to each preset text identifier;

Based on each word feature vector and each identifier feature vector, a text feature sequence of the search text is obtained.

Preset text identifiers refer to identifiers that are pre-set to determine the start and end positions of the searched text, including the start identifier and the end identifier. [start] and [end]. The identifier feature vector is the vector representation of the start identifier and the end identifier, where the identifier feature vector corresponding to the end identifier is also the global feature vector of the search text.

The search text is segmented and recognized to obtain each word in the search text, and each word is encoded to obtain a word feature vector corresponding to each word, a preset text identifier is obtained, the preset identifier is encoded, an identifier feature vector corresponding to each preset identifier is obtained, and a text feature sequence is constructed based on each word feature vector and each identifier feature vector.

For example, for the retrieval text _Yi , the text feature sequence obtained based on the above method can be expressed as Among them, y _S is the starting identifier feature vector, is the word feature vector, y _E is the end identifier feature vector, and is also the global feature vector of the retrieved text _Yi , M is the number of words, and D is the feature dimension, which is usually set to 512 dimensions. This manual does not limit the feature dimension of the feature vector.

In a specific implementation manner provided in this specification, obtaining a target video frame block feature sequence of the target candidate video includes:

Sampling a preset number of key frames from the target candidate video;

Segmenting a target key frame to obtain at least one video frame block corresponding to the target key frame, wherein the target key frame is any one of the preset number of key frames;

Encoding each video frame block to obtain a key frame block feature sequence of the target key frame;

Obtaining a preset classification embedding symbol, encoding the preset classification embedding symbol, and obtaining a global frame feature vector corresponding to the preset classification embedding symbol;

Based on the key frame block feature sequence of each key frame and each global frame feature vector, a target video frame block feature sequence of the target candidate video is obtained.

The preset number refers to the number of video frames that need to be sampled in advance; the key frame refers to the video frame sampled from the target candidate video; the key frame block feature sequence refers to the feature sequence composed of the key frame block feature vectors corresponding to the key frame.

The preset classification embedding character refers to a special classification embedding character that is pre-set and inserted before the key frame. It is used in the classification task and can be specifically represented by [CLS]. The global frame feature vector refers to the vector representation of the preset classification embedding character.

A preset number of key frames are obtained, and a preset number of key frames are sampled from the target candidate video based on the preset number of key frames, and each key frame obtained by sampling is segmented to obtain multiple video frame blocks of each key frame, and then each video frame block is encoded to obtain the key frame corresponding to each video frame block. The frame block feature vector is obtained, and a key frame block feature sequence corresponding to each key frame is generated. A preset classification embedding symbol is obtained, and the preset classification embedding symbol is encoded to obtain a global frame feature vector corresponding to the preset classification embedding symbol. Based on each key frame block feature sequence and each global frame feature vector, a target video frame block feature sequence is constructed.

For example, for the target candidate video _Xi , the target video frame block feature sequence obtained based on the above method can be expressed as in, is the global frame feature vector, is the key frame block feature vector of the lth frame, is the key frame block feature sequence of the lth frame, K is the number of video frame blocks, L is the preset number, D is the feature dimension, usually set to 512 dimensions, and this specification does not limit the feature dimension of the feature vector.

Furthermore, after respectively encoding the retrieval text and the target candidate video and obtaining the text feature sequence corresponding to the retrieval text and the target video frame block feature sequence corresponding to the target candidate video, the text feature sequence and the target video frame block feature sequence are input into the first matching layer of the video matching model to obtain the phrase feature sequence corresponding to the text feature sequence and the target object feature sequence corresponding to the target video frame block feature sequence, so that the first matching result can be determined based on the phrase feature sequence and the target object feature sequence.

In a specific implementation provided in this specification, the first matching layer includes a spatial prototype generator and a target phrase matcher;

Inputting the text feature sequence and the target video frame block feature sequence into the first matching layer, obtaining a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence, and determining a first matching result according to the phrase feature sequence and the target object feature sequence, including:

Inputting the text feature sequence and the target video frame block feature sequence into the spatial prototype generator to obtain a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence;

The phrase feature sequence and the target object feature sequence are input into the target phrase matcher to obtain a first matching result.

The spatial prototype generator is used to aggregate each text feature vector in the text feature sequence into a phrase feature vector, and to aggregate each target video frame block feature vector in the target video frame block feature sequence into a target object feature vector.

The target phrase matcher is used to perform feature matching between each target object feature vector and each phrase feature vector.

Phrase feature sequence refers to the feature vectors of each phrase aggregated from the text feature sequence. The target object feature sequence refers to the feature sequence composed of the feature vectors of each target object aggregated from the target video frame block feature sequence.

Specifically, the text feature sequence and the target video frame block feature sequence are input into the spatial prototype generator, and the phrase feature sequence corresponding to the text feature sequence and the target object feature sequence corresponding to the target video frame block feature sequence can be obtained. Then, the obtained phrase feature sequence and target object feature sequence are input into the target phrase matcher, and the first matching result for characterizing the matching degree between the search text and each target object in the target candidate video can be obtained. Specifically, the implementation method of obtaining the phrase feature sequence is as follows:

In a specific implementation provided in this specification, obtaining a phrase feature sequence corresponding to the text feature sequence includes:

Determining a predicted text weight of a target text feature vector, wherein the target text feature vector is any one of the text feature vectors in the text feature sequence;

Generate a target phrase feature vector based on the target text feature vector and the predicted text weight;

Based on each target phrase feature vector, a phrase feature sequence is obtained.

Since there may be redundant feature vectors in each text feature vector, not all text feature vectors will be aggregated during the process of aggregating the text feature vectors. In order to screen the valid text feature vectors in each text feature vector, the weight corresponding to each text feature vector can be predicted, and the text feature vectors can be aggregated according to the weight corresponding to each text feature vector to generate a phrase feature sequence.

The target text feature vector refers to any one of the text feature vectors in the text feature sequence. The predicted text weight is used to characterize the importance of the text feature vector in obtaining effective information. _Np is the number of phrase feature vectors.

Specifically, any text feature vector is selected as the target text feature vector in the text feature sequence, the predicted text weight of the target text feature vector is determined, and the target phrase feature vector is generated based on the target text feature vector and the predicted text weight of the target text feature vector, and then, a phrase feature sequence is formed based on the generated target phrase feature vectors. Among them, the phrase feature sequence can be expressed as The generation process of phrase feature sequence can be expressed as

Furthermore, the method for obtaining the target object feature sequence is as follows:

In a specific implementation manner provided in this specification, the target video frame block characteristics are obtained. The target object feature sequence corresponding to the feature sequence includes:

Determine a prediction frame weight of a first frame block feature vector, wherein the first frame block feature vector is any one of the target video frame block feature vectors in the target video frame block feature sequence;

generating a first object feature vector based on the first frame block feature vector and the predicted frame weight;

Based on each first object feature vector, a target object feature sequence is obtained.

Since redundant feature vectors may also exist in the feature vectors of each target video frame block, which may even interfere with cross-modal alignment, not all target video frame block feature vectors will be aggregated during the process of aggregating the feature vectors of each target video frame block. In order to screen out the valid target video frame block feature vectors in the feature vectors of each target video frame block, the weights corresponding to the feature vectors of each target video frame block can be predicted. According to the weights corresponding to the feature vectors of each target video frame block, the feature vectors of each target video frame block are filtered and aggregated to generate a target object feature sequence.

The first frame block feature vector refers to any one of the target video frame block feature vectors in the target video frame block feature sequence. The predicted frame weight is used to characterize the importance of the target video frame block feature vector in obtaining effective information. It is represented by, and _No is the number of feature vectors of the target object.

Specifically, any target video frame block feature vector is selected as the first frame block feature vector in the target video frame block feature sequence, the predicted frame weight of the first frame block feature vector is determined, and the first object feature vector is generated based on the first frame block feature vector and the predicted frame weight of the first frame block feature vector, and then, the target object feature sequence is formed based on the generated first object feature vectors. The target object feature sequence can be expressed as The generation process of the target object feature sequence can be expressed as

In this way, redundant text feature vectors and redundant target video frame block feature vectors can be filtered, and a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence can be aggregated to improve the accuracy of determining the target video.

After the phrase feature sequence and the target object feature sequence are generated, the phrase feature sequence and the target object feature sequence may be input into a target phrase matcher to obtain a first matching result.

In a specific implementation manner provided in this specification, obtaining a first matching result includes:

Selecting a feature vector of an object to be processed from the target object feature sequence;

Calculating a first similarity between the feature vector of the object to be processed and each phrase feature vector in the phrase feature sequence, and calculating a second similarity between the feature vector of the object to be processed and each target video frame block feature vector in the target video frame block feature sequence;

Determine a target first similarity among each first similarity, and determine a target second similarity among each second similarity;

Determining an initial matching result of the feature vector of the object to be processed based on the first target similarity and the second target similarity;

Based on the initial matching results of the feature vectors of the objects to be processed, a first matching result is generated.

The feature vector of the object to be processed refers to any one of the feature vectors of the target object in the feature sequence of the target object. The first target similarity specifically refers to the maximum similarity among the first similarities, and the second target similarity specifically refers to the maximum similarity among the second similarities. The initial matching result refers to the matching result between the feature vector of the object to be processed and the feature vectors of each phrase, and is used to determine the phrase feature vector with the maximum similarity to the feature vector of the object to be processed among the feature vectors of each phrase.

Specifically, after generating the phrase feature sequence and the target object feature sequence, any target object feature vector is selected in the target object feature sequence for processing, that is, as the object feature vector to be processed, and the first similarity between the object feature vector to be processed and each phrase feature vector in the phrase feature sequence, and the second similarity between the object feature vector to be processed and each target video frame block feature vector in the target video frame block feature sequence are calculated, and the maximum first similarity is determined as the target first similarity in the first similarity between the object feature vector to be processed and each phrase feature vector in the phrase feature sequence, and the maximum second similarity is determined as the target second similarity in the second similarity between the object feature vector to be processed and each target video frame block feature vector in the target video frame block feature sequence, and the final similarity of the object feature vector to be processed is determined according to the target first similarity and the target second similarity, and the similarity is determined as the initial matching result of the object feature vector to be processed. Further, based on the same method as above, the initial matching results of each target object feature vector in the target object feature sequence are calculated, and the initial matching results are directly added or weighted and added as the first matching result.

By calculating the first matching result between the search text and the target candidate video, the matching degree between the search text and each target object in the target candidate video can be obtained.

Further, after obtaining the first matching result between the search text and the target candidate video, the target object feature sequence, the text feature sequence and the first matching result are input into the second matching layer to obtain the second matching result between the search text and the target candidate video, so that the matching result corresponding to the target candidate video can be determined according to the first matching result and the second matching result. Assign weights.

In a specific implementation provided in this specification, the second matching layer includes a temporal prototype generator and a semantic matcher;

Inputting the target object feature sequence, the text feature sequence, and the first matching result into the second matching layer, and determining a second matching result according to the target object feature sequence and the text feature sequence, comprising:

Inputting the target object feature sequence into the temporal prototype generator to obtain a semantic feature sequence corresponding to the target object feature sequence;

A global feature vector is determined in the text feature sequence, and the semantic feature sequence and the global feature vector are input into the semantic matcher to obtain a second matching result.

The temporal prototype generator is used to aggregate the target object feature vectors in the target object feature sequence into a semantic feature vector. The semantic feature sequence refers to a feature sequence composed of the semantic feature vectors aggregated from the target object feature sequence. The semantic matcher is used to perform feature matching between the global feature vector and the semantic feature vectors.

Specifically, the target object feature sequence is input into the temporal prototype generator to obtain a semantic feature sequence corresponding to the target object feature sequence, and then a global feature vector is determined in the text feature vector, and the global feature vector and the semantic feature sequence are input into the semantic matcher to obtain a second matching result between the search text and the target candidate video. Furthermore, the implementation method of obtaining the semantic feature sequence corresponding to the target object feature sequence is as follows:

In a specific implementation provided in this specification, obtaining a semantic feature sequence corresponding to the target object feature sequence includes:

Decoding the target object feature sequence to obtain a key frame feature sequence;

Determining the correlation relationship between the key frame feature vectors in the key frame feature sequence;

Based on the association relationship, generating at least one semantic feature vector;

Based on each semantic feature vector, a semantic feature sequence is obtained.

A frame decoder is provided in the temporal prototype generator, which can decode each target object feature vector to generate a key frame feature vector at the frame level, and further, can generate a semantic feature vector based on each key frame feature vector.

Among them, the key frame feature sequence refers to the feature sequence composed of the key frame feature vectors obtained by decoding the target object feature vector. The key frame feature sequence can be expressed as The semantic feature sequence refers to a feature sequence composed of semantic feature vectors generated by the interaction of each key frame feature vector in the key frame feature sequence. The semantic feature sequence can be Expressed as Among them, _Ne is the number of semantic feature vectors.

Specifically, a frame decoder is used to decode each target object feature vector in the target object feature sequence, and the spatial relationship between each target object feature vector is analyzed to obtain each key frame feature vector to form a corresponding key frame feature sequence. The association relationship between each key frame feature vector in the key frame feature sequence is determined, and at least one semantic feature vector is generated according to each association relationship, and a semantic feature sequence is generated based on the at least one semantic feature vector.

Furthermore, the attention mechanism can be used to analyze the spatial relationship between the feature vectors of each target object. Specifically, the frame query feature vector _Qf is randomly initialized, each target object feature vector _Po is linearly transformed, and the feature vectors after linear transformation are used as the frame key feature vector _Ko and the frame value feature vector _Vo , respectively. Through mask attention calculation, the spatial relationship between the feature vectors of each target object is obtained, and based on the spatial relationship between the feature vectors of each target object, a key frame feature sequence is generated.

Furthermore, the mask attention calculation can be implemented by the following formula (1):

Among them, _Pf is the key frame feature vector, _Qf is the frame query feature vector, _Ko is the frame key feature vector, _Vo is the frame value feature vector, and _Mf is the attention mask, which can be specifically expressed as softmax(.) is a normalized exponential function.

Similarly, the attention mechanism can be used to dynamically analyze the semantic relationship between the feature vectors of each key frame. Specifically, the semantic query feature vector _Qe is randomly initialized, each key frame feature vector _Pf is linearly transformed, and the feature vectors after linear transformation are respectively used as the semantic key feature vector _Kf and the semantic value feature vector _Vf . Through dynamic attention calculation, the association relationship between the feature vectors of each key frame is obtained, and based on the association relationship between the feature vectors of each key frame, a semantic feature sequence is generated. Among them, the number of semantic feature vectors can be customized based on the actual application situation, for example, the number of semantic feature vectors can be set to 2, 3, etc.

Dynamic attention calculation can be achieved by the following formula (2):

Among them, _Pe is the semantic feature vector, _Qe is the semantic query feature vector, _Kf is the semantic key feature vector, _Vf is the semantic value feature vector, and softmax(.) is the normalized exponential function.

Through the above method, the spatial relationship between the feature vectors of each target object is analyzed to generate a key frame feature sequence, and the semantic relationship between the feature vectors of each key frame is analyzed to generate a semantic feature sequence, so that the video semantic diversity of the target candidate video can be known. In the subsequent process of retrieving the target video, the target video can be retrieved in combination with the video semantics of the candidate video, which can improve the accuracy of retrieving the target video.

After the semantic feature sequence is generated, the search text needs to be matched with the video semantics of the target candidate video to obtain a second matching result between the search text and the target candidate video.

In a specific implementation manner provided in this specification, obtaining a second matching result includes:

Determining the semantic similarity between the global feature vector and each semantic feature vector in the semantic feature sequence;

Based on the semantic similarity of each semantic feature vector, a second matching result between the search text and the target candidate video is determined.

Specifically, the semantic similarity between the global feature vector and each semantic feature vector in the semantic feature sequence is calculated, the maximum semantic similarity is determined among the semantic similarities between the global feature vector and each semantic feature vector, and the maximum semantic similarity is determined as the second matching result.

By calculating the second matching result between the search text and the target candidate video, the matching degree between the search text and the video semantics of the target candidate video can be known.

After obtaining a first matching result and a second matching result between the search text and the target candidate video, a matching weight of the target candidate video is determined based on the first matching result and the second matching result.

Specifically, the first matching result and the second matching result may be weighted and calculated, and the calculation result may be determined as the matching weight of the target candidate video. The specific calculation method may refer to the following formula (3):
s＝s _es +βs _op (3)

Wherein, s is the matching weight, s _es is the second matching result, s _op is the first matching result, and β is the spatial matching factor. β can be manually adjusted based on actual application conditions, and the specific numerical setting is not limited in this specification.

The video retrieval method provided in this specification determines the matching degree between the retrieval text and each target object in the target candidate video by calculating the first matching result between the retrieval text and the target candidate video, and determines the matching degree between the retrieval text and the video semantics of the target candidate video by calculating the second matching result between the retrieval text and the target candidate video. Then, the matching weight of the target candidate video is determined by combining the first matching result and the second matching result, and the accuracy of determining the matching weight of the target candidate video is improved by analyzing the local content and the overall content of the target candidate video.

Referring to FIG. 3, FIG. 3 shows a video Schematic diagram of the model architecture of the matching model. As shown in Figure 3, the video matching model includes an encoding layer, a first matching layer, and a second matching layer. The encoding layer includes a video encoder and a text encoder, the first matching layer includes a spatial prototype generator and a target phrase matcher, and the second matching layer includes a temporal prototype generator and a semantic matcher.

Furthermore, in the actual application process, the search text is input into the text encoder of the encoding layer to obtain the text feature sequence corresponding to the search text, and the target candidate video is input into the video encoder of the encoding layer to obtain the target video frame block feature sequence corresponding to the target candidate video; the text feature sequence and the target video frame block feature sequence are input into the spatial prototype generator of the first matching layer to obtain the phrase feature sequence corresponding to the text feature sequence and the target object feature sequence corresponding to the target video frame block feature sequence; the phrase feature sequence and the target object feature sequence are input into the target phrase matcher of the first matching layer to obtain the first matching result; further, the target object feature sequence is input into the temporal prototype generator of the second matching layer to obtain the semantic feature sequence corresponding to the target object feature sequence; the global feature vector and the semantic feature sequence in the text feature sequence are input into the semantic matcher of the second matching layer to obtain the second matching result; finally, the second matching layer calculates the matching weight of the target candidate video based on the first matching result and the second matching result. By combining the analysis of the local content and the overall content of the target candidate video, the accuracy of determining the matching weight of the target candidate video is improved.

Furthermore, in order to improve the accuracy of the video matching model, the video matching model is trained by the following method:

Obtaining training data sample pairs and matching weight labels corresponding to the training data sample pairs;

Inputting the training data sample pairs into the video matching model to obtain predicted matching weights;

Calculating a model loss value of the video matching model according to the matching weight label and the predicted matching weight;

The model parameters of the video matching model are adjusted according to the model loss value, and the layout generation model is continuously trained until a training stop condition is reached.

Among them, the training data sample pair refers to the text-video data pair obtained in the training data sample pair set, which is the training sample of the video matching model, including the positive training data sample pair and the negative training data sample pair; the training data sample pair set refers to the text content in the collected input speech or input text, and the video retrieved according to the text content, which is composed of the text-video data pair set; the matching weight label refers to the actual matching weight corresponding to the training data sample pair; the predicted matching weight refers to the training data sample pair input to the video The matching weight output by the matching model; the model loss value refers to the difference between the matching weight label and the predicted matching weight, which is used to measure the difference between the matching weight label and the predicted matching weight.

Specifically, the text data in the training data sample pair is obtained by the above-mentioned acquisition method of obtaining the retrieved text, and the video obtained based on the text data retrieval and the corresponding text data constitute the positive training data sample pair, and the other text-video data pairs in the same batch in the training data sample pair set constitute the negative training data sample pair. The training data sample pair is input into the video matching model, and the video matching model is used to predict the matching weight of the training data sample pair. At this time, the video matching model is a model that has not been trained yet. There will be a deviation between the predicted predicted matching weight and the actual matching weight label, and the model parameters of the video matching model need to be adjusted accordingly. Specifically, the model loss value of the video matching model is calculated according to the output predicted matching weight and matching weight label. The loss function for calculating the model loss value can be a 0-1 loss function, a square loss function, a cross entropy loss function, etc. in actual applications. In this specification, preferably, the cross entropy function is selected as the loss function for calculating the model loss value, and the model parameters of the video matching model are adjusted according to the model loss value. The adjusted model parameters are used for the next batch of training data sample pairs to continue training the video matching model until the stop condition of the model training is reached.

Specifically, the model training stopping conditions include that the model loss value is less than a preset threshold and/or the training rounds reach a preset round.

In a specific implementation provided in this specification, taking the training stop condition of the model loss value being less than a preset threshold as an example, the preset threshold is 0.3. When the model loss value is less than 0.3, it is considered that the video matching model training is completed.

In another specific implementation provided in the present specification, taking the preset training rounds as the training stop condition as an example, the preset training rounds are 30 rounds. When the training rounds of the training data sample pair reach 30 rounds, the video matching model training is considered to be completed.

In another specific implementation manner provided in the present specification, two training stop conditions, a preset threshold and a preset training round, are set, and the model loss value and the training rounds are monitored simultaneously. When either the model loss value or the training rounds meets the training stop condition, the video matching model training is considered completed.

The video retrieval method provided in this specification analyzes and processes the retrieval text and the target candidate video through a trained video matching model to obtain the matching weight corresponding to the target candidate video, thereby improving the accuracy of determining the matching weight of the target candidate video.

Step 206: Determine at least one target video from the at least one candidate video based on the matching weight corresponding to each candidate video.

The target video refers to a video that has a high degree of matching with the search text and can be used to be displayed on the client based on the search text.

Specifically, after determining the matching weights corresponding to the candidate videos, the target video corresponding to the search text can be determined in the candidate videos based on the matching weights corresponding to the candidate videos.

In a specific implementation provided in this specification, based on the matching weights corresponding to the candidate videos, determining at least one target video from the at least one candidate video includes:

Determine the candidate video whose matching weight is greater than or equal to a preset matching weight threshold as the target video; or

The candidate videos are sorted according to the matching weights corresponding to the candidate videos to obtain a candidate video list, and a target video is determined in the candidate video list based on a preset number of videos.

The preset matching weight threshold refers to the preset minimum value of the matching weight, which is used to measure the matching degree between each candidate video and the search text. The preset number of videos refers to the preset number of selectable target videos.

Specifically, after determining the matching weight corresponding to each candidate video, a preset matching weight threshold can be obtained, and candidate videos whose matching weights are greater than or equal to the preset matching weight threshold are determined among the candidate videos, and candidate videos whose matching weights are greater than or equal to the preset matching weight threshold are determined as target videos.

Furthermore, after determining the matching weight corresponding to each candidate video, a preset number of videos can also be obtained, and then, according to the matching weight corresponding to each candidate video, each candidate video is sorted to obtain a candidate video list, and a preset number of candidate videos are selected from the candidate video list as target videos.

The implementation method for determining the target video can be set according to actual application conditions, and this specification does not limit it here.

The video retrieval method provided in this specification, after obtaining the matching weights corresponding to each candidate video, determines the target video among the candidate videos based on the matching weights corresponding to each candidate video, thereby improving the accuracy of determining the target video.

Referring to Figure 4a, Figure 4a shows a schematic diagram of an interactive interface of a video retrieval method provided according to an embodiment of the present specification. As shown in Figure 4a, a user can enter a search text to be searched in the search bar of the client to generate a video retrieval instruction. After receiving the video retrieval instruction, the server uses the above-mentioned video retrieval method to determine at least one target video in a preset database, and feeds back and displays the target video to the client, and its display interface can be shown in Figure 4a.

Referring to FIG. 4b, FIG. 4b shows another embodiment of the present invention. Schematic diagram of the interactive interface of the video retrieval method. As shown in FIG4b, the user can enter the search text to be searched in the search bar of the client to generate a video retrieval instruction. After receiving the video retrieval instruction, the server uses the above-mentioned video retrieval method to determine at least one target video in the preset database, and feeds back and displays the target video to the client. The display interface can be shown in FIG4b.

It should be noted that the way in which the user inputs the search text is not limited to text input, and voice input may also be used. FIG. 4 a and FIG. 4 b are only exemplary illustrations.

The video retrieval method provided in this specification includes: obtaining a retrieval text and at least one candidate video; inputting the retrieval text and the target candidate video into a video matching model, and obtaining a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the retrieval text and the target candidate video, the first matching result being used to characterize the matching degree between the retrieval text and each target object in the target candidate video, and the second matching result being used to characterize the matching degree between the retrieval text and the video semantics of the target candidate video; based on the matching weight corresponding to each candidate video, at least one target video is determined in the at least one candidate video.

In one embodiment of the present specification, the first matching result between the search text and the target candidate video is calculated to determine the matching degree between the search text and each target object in the target candidate video, and the second matching result between the search text and the target candidate video is calculated to determine the matching degree between the search text and the video semantics of the target candidate video. Then, the matching weight of the target candidate video is determined by combining the first matching result and the second matching result, and the accuracy of determining the matching weight of the target candidate video is improved by analyzing the local content and the overall content of the target candidate video; then, the target video is determined in each candidate video according to the matching weight corresponding to each candidate video, thereby improving the accuracy of text-based video retrieval.

The following is combined with Figure 5, taking the application of the video retrieval method provided by this specification in text-based video retrieval as an example to further illustrate the video retrieval method. Among them, Figure 5 shows a processing flow chart of a video retrieval method provided by an embodiment of this specification, which specifically includes the following steps:

Step 502: Obtain the search text "cook noodles" and at least one candidate video.

Step 504: Input the search text "cook noodles" into the text encoder of the video matching model to obtain the text feature sequence of the search text "cook noodles", input the target candidate video into the video encoder of the video matching model to obtain the target video frame block feature sequence of the target candidate video.

Step 506: Input the text feature sequence and the target video frame block feature sequence into the spatial prototype generator of the video matching model to obtain a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence.

Step 508: Input the phrase feature sequence and the target object feature sequence into the target phrase matcher of the video matching model to obtain a first matching result.

Step 510: Input the target object feature sequence into the temporal prototype generator of the video matching model to obtain a semantic feature sequence corresponding to the target object feature sequence.

Step 512: Determine a global feature vector in the text feature sequence, input the semantic feature sequence and the global feature vector into the semantic matcher of the video matching model, and obtain a second matching result.

Step 514: Determine a matching weight corresponding to the target candidate video based on the first matching result and the second matching result.

Step 516: Determine at least one target video from the at least one candidate video based on the matching weight corresponding to each candidate video.

In one embodiment of the present specification, the first matching result between the search text "cook noodles" and the target candidate video is calculated to determine the matching degree between the search text "cook noodles" and each target object in the target candidate video, and the second matching result between the search text "cook noodles" and the target candidate video is calculated to determine the matching degree between the search text "cook noodles" and the video semantics of the target candidate video. Then, the matching weight of the target candidate video is determined by combining the first matching result and the second matching result, and the accuracy of determining the matching weight of the target candidate video is improved by analyzing the local content and the overall content of the target candidate video; then, the target video is determined in each candidate video according to the matching weight corresponding to each candidate video, thereby improving the accuracy of text-based video retrieval.

Corresponding to the above method embodiment, this specification also provides a video retrieval device embodiment. FIG6 shows a schematic diagram of the structure of a video retrieval device provided by an embodiment of this specification. As shown in FIG6, the device includes:

An acquisition module 602 is configured to acquire a search text and at least one candidate video;

The input module 604 is configured to input the search text and the target candidate video into the video matching model, and obtain the matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on the first matching result and the second matching result between the search text and the target candidate video, wherein the first matching result is used to characterize the matching degree between the search text and each target object in the target candidate video, and the second matching result is used to characterize the matching degree between the search text and each target object in the target candidate video. The result is used to represent the matching degree between the retrieved text and the video semantics of the target candidate video;

The determination module 606 is configured to determine at least one target video from the at least one candidate video based on the matching weight corresponding to each candidate video.

Optionally, the video matching model includes a coding layer, a first matching layer and a second matching layer;

The input module is further configured as follows:

Inputting the search text and the target candidate video into the coding layer to obtain a text feature sequence of the search text and a target video frame block feature sequence of the target candidate video;

Inputting the text feature sequence and the target video frame block feature sequence into the first matching layer, obtaining a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence, and determining a first matching result according to the phrase feature sequence and the target object feature sequence;

The target object feature sequence, the text feature sequence and the first matching result are input into the second matching layer, a second matching result is determined according to the target object feature sequence and the text feature sequence, and a matching weight corresponding to the target candidate video is determined based on the first matching result and the second matching result.

Optionally, the encoding layer includes a text encoder and a video encoder;

The input module is further configured as follows:

Inputting the search text into the text encoder to obtain a text feature sequence of the search text;

The target candidate video is input into the video encoder to obtain a target video frame block feature sequence of the target candidate video.

Optionally, the first matching layer includes a spatial prototype generator and a target phrase matcher;

The input module is further configured as follows:

Inputting the text feature sequence and the target video frame block feature sequence into the spatial prototype generator to obtain a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence;

The phrase feature sequence and the target object feature sequence are input into the target phrase matcher to obtain a first matching result.

Optionally, the input module is further configured as:

Determine the predicted text weight of the target text feature vector, wherein the target text feature vector The feature vector is any one of the text feature vectors in the text feature sequence;

Generate a target phrase feature vector based on the target text feature vector and the predicted text weight;

Based on each target phrase feature vector, a phrase feature sequence is obtained.

Optionally, the input module is further configured as:

Determine a prediction frame weight of a first frame block feature vector, wherein the first frame block feature vector is any one of the target video frame block feature vectors in the target video frame block feature sequence;

generating a first object feature vector based on the first frame block feature vector and the predicted frame weight;

Based on each first object feature vector, a target object feature sequence is obtained.

Optionally, the input module is further configured as:

Selecting a feature vector of an object to be processed from the target object feature sequence;

Calculating a first similarity between the feature vector of the object to be processed and each phrase feature vector in the phrase feature sequence, and calculating a second similarity between the feature vector of the object to be processed and each target video frame block feature vector in the target video frame block feature sequence;

Determine a target first similarity among each first similarity, and determine a target second similarity among each second similarity;

Determining an initial matching result of the feature vector of the object to be processed based on the first target similarity and the second target similarity;

Based on the initial matching results of the feature vectors of the objects to be processed, a first matching result is generated.

Optionally, the second matching layer includes a temporal prototype generator and a semantic matcher;

The input module is further configured as follows:

Inputting the target object feature sequence into the temporal prototype generator to obtain a semantic feature sequence corresponding to the target object feature sequence;

A global feature vector is determined in the text feature sequence, and the semantic feature sequence and the global feature vector are input into the semantic matcher to obtain a second matching result.

Optionally, the input module is further configured as:

Decoding the target object feature sequence to obtain a key frame feature sequence;

Determining the correlation relationship between the key frame feature vectors in the key frame feature sequence;

Based on the association relationship, generating at least one semantic feature vector;

Based on each semantic feature vector, a semantic feature sequence is obtained.

Optionally, the input module is further configured as:

Determining the semantic similarity between the global feature vector and each semantic feature vector in the semantic feature sequence;

Based on the semantic similarity of each semantic feature vector, a second matching result between the search text and the target candidate video is determined.

Optionally, the determining module is further configured to:

Determine the candidate video whose matching weight is greater than or equal to a preset matching weight threshold as the target video; or

The candidate videos are sorted according to the matching weights corresponding to the candidate videos to obtain a candidate video list, and a target video is determined in the candidate video list based on a preset number of videos.

Optionally, the device further comprises a training module configured to:

Obtaining training data sample pairs and matching weight labels corresponding to the training data sample pairs;

Inputting the training data sample pairs into the video matching model to obtain predicted matching weights;

Calculating a model loss value of the video matching model according to the matching weight label and the predicted matching weight;

The model parameters of the video matching model are adjusted according to the model loss value, and the layout generation model is continuously trained until a training stop condition is reached.

The video retrieval device provided in this specification includes: an acquisition module, configured to acquire a search text and at least one candidate video; an input module, configured to input the search text and the target candidate video into a video matching model, and obtain a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the search text and the target candidate video, the first matching result being used to characterize the matching degree between the search text and each target object in the target candidate video, and the second matching result being used to characterize the matching degree between the search text and the video semantics of the target candidate video; a determination module, configured to determine at least one target video in the at least one candidate video based on the matching weight corresponding to each candidate video.

In one embodiment of the present specification, by calculating the first matching result between the search text and the target candidate video, the matching degree between the search text and each target object in the target candidate video is determined, and the second matching result between the search text and the target candidate video is calculated to determine the matching degree between the search text and the video semantics of the target candidate video. The second matching result determines the matching weight of the target candidate video. By analyzing the local content and overall content of the target candidate video, the accuracy of determining the matching weight of the target candidate video is improved; then, according to the matching weights corresponding to each candidate video, the target video is determined among the candidate videos to improve the accuracy of text-based video retrieval.

The above is a schematic scheme of a video retrieval device of this embodiment. It should be noted that the technical scheme of the video retrieval device and the technical scheme of the above video retrieval method belong to the same concept, and the details not described in detail in the technical scheme of the video retrieval device can be referred to the description of the technical scheme of the above video retrieval method.

Fig. 7 shows a block diagram of a computing device 700 according to an embodiment of the present specification. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface, wired or wireless (e.g., a network interface card (NIC)), such as an IEEE802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

In one embodiment of the present specification, the above components of the computing device 700 and other components not shown in FIG. 7 may also be connected to each other, for example, through a bus. It should be understood that the computing device structure block diagram shown in FIG. 7 is only for illustrative purposes and is not intended to limit the scope of the present specification. Those skilled in the art may add or replace other components as needed.

The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

The processor 720 is used to execute the following computer executable instructions, which implement the steps of the above-mentioned video retrieval method when executed by the processor.

The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above video retrieval method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above video retrieval method.

An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned video retrieval method when executed by a processor.

The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above video retrieval method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above video retrieval method.

An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned video retrieval method.

The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above video retrieval method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above video retrieval method.

The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

The computer instructions include computer program codes, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

It should be noted that, for the convenience of description, the above-mentioned method embodiments are all described as a series of action combinations, but those skilled in the art should know that the embodiments of this specification are not limited to the described action sequence, because according to the implementation of this specification For example, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims (14)

A video retrieval method, comprising: Obtaining a search text and at least one candidate video; Input the search text and the target candidate video into a video matching model, and obtain a matching weight corresponding to the target candidate video output by the video matching model, wherein the target candidate video is any one of the at least one candidate video, and the matching weight is determined based on a first matching result and a second matching result between the search text and the target candidate video, wherein the first matching result is used to characterize the matching degree between the search text and each target object in the target candidate video, and the second matching result is used to characterize the matching degree between the search text and the video semantics of the target candidate video; At least one target video is determined from the at least one candidate video based on the matching weights corresponding to the candidate videos. The method according to claim 1, wherein the video matching model comprises a coding layer, a first matching layer, and a second matching layer; Inputting the search text and the target candidate video into a video matching model, and obtaining a matching weight corresponding to the target candidate video output by the video matching model, comprises: Inputting the search text and the target candidate video into the coding layer to obtain a text feature sequence of the search text and a target video frame block feature sequence of the target candidate video; Inputting the text feature sequence and the target video frame block feature sequence into the first matching layer, obtaining a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence, and determining a first matching result according to the phrase feature sequence and the target object feature sequence; The target object feature sequence, the text feature sequence and the first matching result are input into the second matching layer, a second matching result is determined according to the target object feature sequence and the text feature sequence, and a matching weight corresponding to the target candidate video is determined based on the first matching result and the second matching result. The method of claim 2, wherein the encoding layer comprises a text encoder and a video encoder; Inputting the search text and the target candidate video into a video matching model to obtain a text feature sequence of the search text and a target video frame block feature sequence of the target candidate video, including: Inputting the search text into the text encoder to obtain a text feature sequence of the search text; The target candidate video is input into the video encoder to obtain a target video frame block feature sequence of the target candidate video. The method of claim 2, wherein the first matching layer comprises a spatial prototype generator and a target phrase matcher; The text feature sequence and the target video frame block feature sequence are input into the first matching layer to obtain the phrase feature sequence corresponding to the text feature sequence and the target video frame block feature sequence. The target object feature sequence corresponding to the phrase feature sequence is determined, and a first matching result is determined according to the phrase feature sequence and the target object feature sequence, including: Inputting the text feature sequence and the target video frame block feature sequence into the spatial prototype generator to obtain a phrase feature sequence corresponding to the text feature sequence and a target object feature sequence corresponding to the target video frame block feature sequence; The phrase feature sequence and the target object feature sequence are input into the target phrase matcher to obtain a first matching result. The method according to claim 4, obtaining a phrase feature sequence corresponding to the text feature sequence, comprises: Determining a predicted text weight of a target text feature vector, wherein the target text feature vector is any one of the text feature vectors in the text feature sequence; Generate a target phrase feature vector based on the target text feature vector and the predicted text weight; Based on each target phrase feature vector, a phrase feature sequence is obtained. The method according to claim 4, obtaining the target object feature sequence corresponding to the target video frame block feature sequence, comprises: Determine a prediction frame weight of a first frame block feature vector, wherein the first frame block feature vector is any one of the target video frame block feature vectors in the target video frame block feature sequence; generating a first object feature vector based on the first frame block feature vector and the predicted frame weight; Based on each first object feature vector, a target object feature sequence is obtained. The method according to claim 4, obtaining a first matching result, comprising: Selecting a feature vector of an object to be processed from the target object feature sequence; Calculating a first similarity between the feature vector of the object to be processed and each phrase feature vector in the phrase feature sequence, and calculating a second similarity between the feature vector of the object to be processed and each target video frame block feature vector in the target video frame block feature sequence; Determine a target first similarity among each first similarity, and determine a target second similarity among each second similarity; Determining an initial matching result of the feature vector of the object to be processed based on the first target similarity and the second target similarity; Based on the initial matching results of the feature vectors of the objects to be processed, a first matching result is generated. The method according to claim 2, wherein the second matching layer comprises a temporal prototype generator and a semantic matcher; Inputting the target object feature sequence, the text feature sequence, and the first matching result into the second matching layer, and determining a second matching result according to the target object feature sequence and the text feature sequence, comprising: Inputting the target object feature sequence into the temporal prototype generator to obtain a semantic feature sequence corresponding to the target object feature sequence; Determine a global feature vector in the text feature sequence, and combine the semantic feature sequence and the The global feature vector is input into the semantic matcher to obtain a second matching result. The method according to claim 8, obtaining the semantic feature sequence corresponding to the target object feature sequence, comprises: Decoding the target object feature sequence to obtain a key frame feature sequence; Determining the correlation relationship between the key frame feature vectors in the key frame feature sequence; Based on the association relationship, generating at least one semantic feature vector; Based on each semantic feature vector, a semantic feature sequence is obtained. The method according to claim 8, obtaining the second matching result comprises: Determining the semantic similarity between the global feature vector and each semantic feature vector in the semantic feature sequence; Based on the semantic similarity of each semantic feature vector, a second matching result between the search text and the target candidate video is determined. The method according to claim 1, determining at least one target video from the at least one candidate video based on the matching weights corresponding to the candidate videos, comprises: Determine the candidate video whose matching weight is greater than or equal to a preset matching weight threshold as the target video; or The candidate videos are sorted according to the matching weights corresponding to the candidate videos to obtain a candidate video list, and a target video is determined in the candidate video list based on a preset number of videos. The method according to claim 1, wherein the video matching model is trained by the following method: Obtaining training data sample pairs and matching weight labels corresponding to the training data sample pairs; Inputting the training data sample pairs into the video matching model to obtain predicted matching weights; Calculating a model loss value of the video matching model according to the matching weight label and the predicted matching weight; The model parameters of the video matching model are adjusted according to the model loss value, and the layout generation model is continuously trained until a training stop condition is reached. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the video retrieval method according to any one of claims 1 to 12 are implemented. A computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the steps of the video retrieval method described in any one of claims 1 to 12.