CN118093938B - Video query retrieval method and system based on semantic depth model - Google Patents

Abstract

The invention discloses a video query retrieval method and a system based on a semantic depth model, and belongs to the technical field of information retrieval. In order to solve the problem of poor accuracy and correlation of a retrieval result of retrieving videos, rich and comprehensive feature information can be extracted from the videos through a deep learning technology, video contents can be described more accurately, the extracted video features are mapped into a semantic space through constructing a deep neural network model, semantic vector representations are generated, conversion from image features to semantic understanding is effectively achieved, accuracy and efficiency of video retrieval are remarkably improved, semantically most similar videos are rapidly retrieved in a video library through calculating similarity between two semantic vectors according to query conditions input by a user, and more friendly and practical query experience is provided for the user according to similarity sequencing, so that the method can be flexibly applied to various types of video libraries and different query conditions, and has good expandability and adaptability.

Description

Video query retrieval method and system based on semantic deep model

Technical Field

The present invention relates to the field of information retrieval technology, and in particular to a video query retrieval method and system based on a semantic depth model.

Background Art

With the explosive growth of video data, video retrieval has become an important technical requirement. Traditional video retrieval methods usually perform similarity matching based on visual features such as color, texture, and shape. However, these methods have difficulty understanding video content at the semantic level, resulting in inaccurate and irrelevant retrieval results.

The prior art often still has the following defects:

Traditional video retrieval methods usually only use some features in the video, such as color, texture, and shape, while ignoring other important feature information, such as scenes, characters, and actions. This incomplete feature extraction method may lead to inaccurate and irrelevant retrieval results. At the same time, traditional video retrieval methods usually only focus on the visual features of the video, while ignoring the semantic information of the video. Due to the lack of in-depth understanding of the video content, these methods are difficult to perform accurate similarity matching at the semantic level.

Summary of the invention

The purpose of the present invention is to provide a video query retrieval method and system based on a semantic depth model to solve the problems raised in the above background technology.

To achieve the above object, the present invention provides the following technical solution: a video query retrieval method based on a semantic depth model, comprising the following steps:

Step 1: Use deep learning technology to extract rich feature information from the video, including color, texture, shape, and motion;

Step 2: Build a deep neural network model to map the extracted video features into the semantic space and generate a semantic vector;

Step 3: Measure the similarity between the two videos at the semantic level by calculating the similarity between the two semantic vectors;

Step 4: According to the query condition input by the user, the most semantically similar video is retrieved from the video library. First, the query condition input by the user is represented by a semantic vector, and then the video most similar to the semantic vector is searched in the video library;

Step 5: Sort the retrieved videos based on the similarity calculation results, and put the videos that best meet the user's needs at the front.

Furthermore, in step four, the similarity between the semantic vector of the query condition input by the user and the semantic vector of each video in the video library is calculated, and the similarity of each video in the video library is compared according to the similarity threshold to obtain the video that is most similar to the semantic vector of the query condition input by the user.

Furthermore, the present invention provides a video query retrieval system based on a semantic depth model, which is applied in the above-mentioned video query retrieval method based on a semantic depth model, and comprises:

Feature extraction module for:

Use deep learning technology to deeply analyze the input video and extract color, texture, shape, and motion feature information from the input video;

Semantic modeling module for:

Build a deep neural network model to map video features into semantic space, capture the temporal information and contextual relationship of video content, and generate semantic vector representation;

Similarity calculation module, used for:

By calculating the similarity between two semantic vectors, the similarity between two videos at the semantic level is measured;

Video retrieval module for:

Retrieve the most semantically similar videos from the video library according to the query conditions input by the user, represent the query conditions input by the user with a semantic vector, and then search for the video library with the most similar video to the semantic vector;

Result sorting module, used to:

According to the similarity calculation results, the retrieved videos are sorted and the video that best meets the user's needs is adjusted to the front.

Furthermore, the feature extraction module includes:

Feature extraction unit, used to:

Use specific layers in CNN to detect and extract color distribution and color histogram information in the video, analyze texture patterns and structures in the video through CNN, perform contour detection and shape recognition on objects in the video, and combine optical flow technology to detect motion patterns and dynamic changes in the video;

Feature fusion unit, used for:

Integrate the extracted color, texture, shape and motion features, and effectively combine each feature to form a unified feature vector;

Feature normalization unit, used to:

The extracted features are normalized.

Furthermore, the semantic modeling module includes:

Timing information capture unit, used to:

Use RNN to capture the timing information in the video, analyze the temporal and spatial relationship between consecutive frames in the video, and capture dynamic changes and continuous actions;

Contextual relationship modeling unit, used to:

Use the self-attention mechanism in the Transformer architecture to model the contextual relationship of video content;

Feature fusion and semantic conversion unit, used for:

The information obtained from the temporal information capture unit and the contextual relationship modeling unit is fused, and the fused features are converted into a high-level semantic vector representation;

Semantic vector normalization unit, used to:

The generated semantic vectors are normalized.

Furthermore, the similarity calculation module includes:

Semantic vector input unit, used for:

Receive the semantic vector from the semantic modeling module, and format and preprocess the input semantic vector;

Similarity measurement unit, used to:

Select a suitable similarity measurement method according to specific needs, such as cosine similarity or Euclidean distance, calculate the similarity of the two input semantic vectors, and obtain a value as a measure of similarity;

A threshold setting unit, for:

Set the similarity threshold and dynamically adjust the similarity threshold based on actual retrieval results and user feedback.

Furthermore, the threshold setting unit includes:

A threshold initial value extraction module is used to extract the initial value of the similarity threshold;

A cycle duration setting module, used to set the observation cycle duration of the similarity threshold; wherein the observation cycle duration of the similarity threshold includes 30-90 unit durations, and the unit duration is 24 hours;

A comparison result extraction module is used to extract, for each observation period, the comparison result between the similarity values corresponding to all two semantic vectors in the observation period and the initial similarity threshold;

A real similarity value extraction module, used for retrieving the real similarity values of the two semantic vectors contained in the actual retrieval effect and user feedback when the comparison result between the similarity values corresponding to the two semantic vectors and the similarity initial threshold value indicates that the similarity values of the two semantic vectors are lower than the corresponding similarity threshold value;

A threshold adjustment judgment module, used to judge whether the initial similarity threshold needs to be adjusted by using the difference between the real similarity value and the similarity values corresponding to the two semantic vectors;

The adjustment execution module is used to adjust the initial similarity threshold using a threshold adjustment model when it is determined that the initial similarity threshold needs to be adjusted; wherein the structure of the threshold adjustment model is as follows:

Among them, _Syt represents the similarity threshold corresponding to the adjustment of the initial similarity threshold; _Swy represents the preset judgment threshold; _Sq represents the similarity judgment value; k represents the total number of observation cycles; _Swj represents the similarity judgment stability parameter of the j-th observation cycle; _S01j and _S02j represent the first similarity stability parameter and the second similarity stability parameter of the j-th observation cycle respectively; _Sy represents the similarity threshold; λ represents the ratio parameter.

Furthermore, the threshold adjustment judgment module includes:

A group number extraction module is used to extract the number of groups consisting of two semantic vectors below the corresponding similarity threshold in each observation period when the comparison result between the similarity values corresponding to the two semantic vectors and the similarity initial threshold indicates that the similarity values of the two semantic vectors are below the corresponding similarity threshold;

A value retrieving module, used for retrieving the similarity value corresponding to each group of two semantic vectors below the corresponding similarity threshold in each observation period;

The stability parameter acquisition module is used to obtain the similarity judgment stability parameter contained in each observation period by using the similarity values corresponding to the two semantic vectors below the corresponding similarity threshold in each observation period; wherein the similarity judgment stability parameter is obtained by the following formula:

Wherein, _Sw represents the similarity judgment stability parameter; n represents the number of groups corresponding to two semantic vectors below the corresponding similarity threshold; _S01 and _S02 represent the first similarity stability parameter and the second similarity stability parameter respectively; _Si represents the similarity value corresponding to the i-th group of two semantic vectors below the corresponding similarity threshold; _Sy represents the similarity threshold; m represents the number of groups corresponding to two semantic vectors not lower than the corresponding similarity threshold; _Szi represents the true similarity value corresponding to the two semantic vectors below the corresponding similarity threshold; _Sj represents the similarity value corresponding to the j-th two semantic vectors not lower than the corresponding similarity threshold;

The comprehensive parameter acquisition module is used to integrate the similarity judgment stability parameters contained in each observation period to generate a comprehensive similarity judgment value; wherein the similarity judgment value is obtained by the following formula:

Wherein, _Sq represents the similarity judgment value; k represents the total number of observation periods; _Swj represents the similarity judgment stability parameter of the j-th observation period; _S01j and _S02j represent the first similarity stability parameter and the second similarity stability parameter of the j-th observation period, respectively;

The threshold adjustment execution module is used to determine that the initial similarity threshold needs to be adjusted when the similarity determination value is lower than a preset determination threshold.

Furthermore, the video retrieval module includes:

The query condition processing unit is used to:

Receive the query conditions entered by the user, extract keywords, scenes, characters and other information, and convert them into semantic vector representation;

Video library index unit, used to:

Establish an index structure for the video library, pre-process the videos in the video library by interacting with the feature extraction module and the semantic modeling module, and generate a corresponding semantic vector representation for each video;

Retrieve matching unit, used to:

By interacting with the similarity calculation module, the similarity between the semantic vector of the query condition and the semantic vector of each video in the video library is calculated, and the videos most relevant to the user's query condition are screened out according to the similarity threshold.

Furthermore, the result sorting module includes:

The result receiving unit is used for:

Receiving a similarity value output by a similarity calculation module;

Sequencing application unit for:

Sort the videos according to the similarity calculation results;

Output display unit for:

The sorted video retrieval results are returned to the user.

Compared with the prior art, the present invention has the following beneficial effects:

1. The present invention can extract rich and comprehensive feature information from videos through deep learning technology, including color, texture, shape and motion, etc. These features can more accurately describe the video content and provide a solid foundation for subsequent semantic modeling. By building a deep neural network model, the extracted video features are mapped into the semantic space and a semantic vector representation is generated, which effectively realizes the transformation from image features to semantic understanding and significantly improves the accuracy and efficiency of video retrieval.

2. The present invention can accurately measure the similarity between two videos at the semantic level by calculating the similarity between two semantic vectors, thereby providing an accurate matching basis for subsequent video retrieval. According to the query conditions input by the user, the most semantically similar videos are quickly retrieved from the video library and sorted according to the similarity. This personalized sorting puts the videos that best meet the user's needs at the front, providing users with a more friendly and practical query experience. It can be flexibly applied to various types of video libraries and different query conditions, and has good scalability and adaptability.

3. The feature extraction module and semantic modeling module of the present invention can comprehensively extract the color, texture, shape and motion features in the video, so as to more accurately describe the video content. The similarity calculation module can accurately measure the semantic similarity between videos by calculating the similarity between two semantic vectors, thereby improving the accuracy and efficiency of retrieval. The video retrieval module is based on the user's query conditions. The system can quickly retrieve the most semantically similar videos in the video library and sort them according to the similarity. The personalized sorting of the result sorting module provides users with a more friendly and practical query experience.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a flow chart of a video query retrieval method based on a semantic depth model of the present invention;

FIG. 2 is a schematic diagram of the modules of the video query and retrieval system of the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

In order to solve the technical problem of poor accuracy and relevance of retrieval results for video retrieval, please refer to FIG. 1 , the present invention provides the following technical solutions:

The video query retrieval method based on the semantic depth model includes the following steps:

Step 1: Use deep learning technology to extract rich feature information from the video, including color, texture, shape, and motion. These feature information can fully describe the video content and provide basic data for subsequent semantic modeling;

Step 2: Build a deep neural network model to map the extracted video features into the semantic space and generate a semantic vector representation. This step is the bridge from image features to semantic understanding and is crucial to improving the accuracy and efficiency of video retrieval.

Step 3: By calculating the similarity between the two semantic vectors, the similarity between the two videos at the semantic level is measured, providing an accurate matching basis for subsequent video retrieval;

Step 4: According to the query condition input by the user, the most semantically similar video is retrieved from the video library. First, the query condition input by the user is represented by a semantic vector, and then the video most similar to the semantic vector is searched in the video library;

Step 5: Sort the retrieved videos according to the similarity calculation results, and put the videos that best meet the user's needs at the front. This step can provide users with a more friendly query experience and improve the practicality of video retrieval.

In the above embodiment, the similarity between the semantic vector of the query condition input by the user and the semantic vector of each video in the video library is calculated, and the similarity of each video in the video library is compared according to the similarity threshold to obtain the video that is most similar to the semantic vector of the query condition input by the user.

Specifically, deep learning technology can extract rich and comprehensive feature information from videos, including color, texture, shape, and motion. These features can more accurately describe the video content and provide a solid foundation for subsequent semantic modeling. By building a deep neural network model, the extracted video features are mapped into the semantic space and a semantic vector representation is generated, which effectively realizes the transformation from image features to semantic understanding and significantly improves the accuracy and efficiency of video retrieval.

In the above embodiment, by calculating the similarity between two semantic vectors, the similarity between two videos at the semantic level can be accurately measured, providing an accurate matching basis for subsequent video retrieval.

In the above embodiment, the method can quickly retrieve the most semantically similar videos in the video library according to the query conditions input by the user, and sort them according to the similarity. This personalized sorting puts the videos that best meet the user's needs at the front, providing the user with a more friendly and practical query experience. The method can be flexibly applied to various types of video libraries and different query conditions, and has good scalability and adaptability.

Please refer to FIG. 2 , a video query retrieval system based on a semantic deep model includes:

Feature extraction module for:

Use deep learning technology to deeply analyze the input video and extract color, texture, shape, and motion feature information from the input video;

Semantic modeling module for:

Build a deep neural network model to map video features into semantic space, capture the temporal information and contextual relationship of video content, generate semantic vector representation, and provide high-quality semantic data for subsequent video retrieval;

Similarity calculation module, used for:

By calculating the similarity between two semantic vectors, the similarity between two videos at the semantic level is measured. The cosine similarity and Euclidean distance measurement methods are used to provide accurate matching basis for video retrieval, ensuring that the retrieval results are highly relevant to the user's query conditions.

Video retrieval module for:

Retrieve the most semantically similar videos from the video library according to the query conditions input by the user, represent the query conditions input by the user with a semantic vector, and then search for the video library with the most similar video to the semantic vector;

Result sorting module, used to:

According to the similarity calculation results, the retrieved videos are sorted, and the videos that best meet the user's needs are adjusted to the front, providing a clear and orderly result display to facilitate users to quickly locate and select the required videos.

Specifically, the feature extraction module and the semantic modeling module can comprehensively extract the color, texture, shape and motion features in the video, thereby more accurately describing the video content. The similarity calculation module can accurately measure the semantic similarity between videos by calculating the similarity between two semantic vectors, thereby improving the accuracy and efficiency of retrieval. The video retrieval module is based on the user's query conditions. The system can quickly retrieve the most semantically similar videos in the video library and sort them according to similarity. The personalized sorting of the result sorting module provides users with a more friendly and practical query experience.

The feature extraction module includes:

Feature extraction unit, used to:

Use specific layers in CNN to detect and extract color distribution and color histogram information in the video, identify the main colors and color changes in the video to reflect the theme or emotion of the video, analyze the texture patterns and structures in the video through CNN to identify different materials and surface textures. The detailed analysis of texture helps to understand the details of the scene and the texture of the object. Perform contour detection and shape recognition on the objects in the video, analyze the basic shape, size, direction, etc. of the object, and provide a basis for subsequent object recognition. Combined with optical flow technology to detect motion patterns and dynamic changes in the video, analyze the motion trajectory, speed, direction, etc. of the object to understand the dynamic events in the video;

Feature fusion unit, used for:

Integrate the extracted color, texture, shape and motion features, and effectively combine each feature to form a unified feature vector to provide a more comprehensive description of the video content;

Feature normalization unit, used to:

The extracted features are standardized to ensure the comparability and consistency between different features.

In the above embodiment, by using CNN to extract multiple features such as color, texture, shape and motion, the system is able to provide a richer and more comprehensive description of the video content, which helps to more accurately understand the subject, emotion, objects and dynamic events of the video.

In the above embodiment, the feature fusion unit effectively combines different features to form a unified feature vector, providing a more comprehensive description of the video content, and the feature standardization unit ensures the comparability and consistency between different features, further improving the accuracy and efficiency of retrieval.

In the above embodiment, by combining the extraction and analysis of multiple features, the system can more accurately measure the similarity between videos, thereby improving the accuracy of video retrieval and providing users with more accurate and relevant retrieval results.

The semantic modeling module includes:

Timing information capture unit, used to:

Use RNN to capture the timing information in the video, analyze the temporal and spatial relationship between consecutive frames in the video, and capture dynamic changes and continuous actions;

Contextual relationship modeling unit, used to:

The self-attention mechanism in the Transformer architecture is used to model the contextual relationship of the video content. By analyzing the associations and dependencies between different video clips, the overall content and theme of the video can be understood.

Feature fusion and semantic conversion unit, used for:

The information obtained from the temporal information capture unit and the contextual relationship modeling unit is fused, and the fused features are converted into high-level semantic vector representations to ensure that the semantic vector can fully and accurately reflect the semantic content of the video;

Semantic vector normalization unit, used to:

The generated semantic vectors are standardized to ensure the comparability and consistency between different vectors, which helps to improve the accuracy of similarity calculation and the efficiency of retrieval.

In the above embodiments, by using the RNN and Transformer architecture, the timing information and contextual relationships in the video can be effectively captured, which helps to understand the dynamic changes, continuous actions and overall content themes in the video, and improves the in-depth understanding of the video content.

In the above embodiment, the feature fusion and semantic conversion unit integrates the information from different units and converts it into a high-level semantic vector representation, ensuring that the semantic vector can fully and accurately reflect the semantic content of the video, providing strong support for subsequent similarity calculation and retrieval.

The similarity calculation module includes:

Semantic vector input unit, used for:

Receive semantic vectors from the semantic modeling module, format and preprocess the input semantic vectors to ensure that they are suitable for similarity calculation;

Similarity measurement unit, used to:

Select a suitable similarity measurement method according to specific needs, such as cosine similarity or Euclidean distance, calculate the similarity of the two input semantic vectors, and obtain a value as a measure of similarity;

A threshold setting unit, for:

Set a similarity threshold and dynamically adjust the similarity threshold based on actual retrieval results and user feedback to ensure the accuracy and practicality of the retrieval results.

In the above embodiment, the similarity measurement unit accurately measures the semantic vector from the semantic modeling module, which helps to ensure the accuracy and reliability of similarity calculation and improve the accuracy of video retrieval.

In the above embodiment, the threshold setting unit allows the similarity threshold to be dynamically adjusted according to the actual retrieval effect and user feedback, which helps to balance the accuracy and recall rate of video retrieval and meet the needs of different users and practical applications. Accurate similarity calculation and flexible threshold setting help improve the efficiency and accuracy of video retrieval and provide users with more accurate and relevant retrieval results.

Specifically, the threshold setting unit includes:

A threshold initial value extraction module is used to extract the initial value of the similarity threshold;

A cycle duration setting module, used to set the observation cycle duration of the similarity threshold; wherein the observation cycle duration of the similarity threshold includes 30-90 unit durations, and the unit duration is 24 hours;

A comparison result extraction module is used to extract, for each observation period, the comparison result between the similarity values corresponding to all two semantic vectors in the observation period and the initial similarity threshold;

A real similarity value extraction module, used for retrieving the real similarity values of the two semantic vectors contained in the actual retrieval effect and user feedback when the comparison result between the similarity values corresponding to the two semantic vectors and the similarity initial threshold value indicates that the similarity values of the two semantic vectors are lower than the corresponding similarity threshold value;

A threshold adjustment judgment module, used to judge whether the initial similarity threshold needs to be adjusted by using the difference between the real similarity value and the similarity values corresponding to the two semantic vectors;

The adjustment execution module is used to adjust the initial similarity threshold using a threshold adjustment model when it is determined that the initial similarity threshold needs to be adjusted; wherein the structure of the threshold adjustment model is as follows:

Among them, _Syt represents the similarity threshold corresponding to the adjustment of the initial similarity threshold; _Swy represents the preset judgment threshold; _Sq represents the similarity judgment value; k represents the total number of observation cycles; _Swj represents the similarity judgment stability parameter of the j-th observation cycle; _S01j and _S02j represent the first similarity stability parameter and the second similarity stability parameter of the j-th observation cycle respectively; _Sy represents the similarity threshold; λ represents the ratio parameter.

The technical effect of the above technical solution is: the threshold initial value extraction module can extract the initial value of the similarity threshold, providing a basis for subsequent threshold adjustment. The cycle duration setting module can set the observation cycle duration of the similarity threshold to ensure the timeliness and accuracy of the threshold adjustment. By setting a suitable observation cycle, the similarity changes of the semantic vectors can be better monitored. The comparison result extraction module can extract the comparison results between the similarity values corresponding to all two semantic vectors in the observation cycle and the similarity initial threshold for each observation cycle. The extraction of such comparison results helps to better understand the similarity state of the semantic vectors. When the comparison result shows that the similarity values of the two semantic vectors are lower than the initial threshold, the real similarity value extraction module can retrieve the real similarity values of the two semantic vectors contained in the actual retrieval effect and user feedback. This retrieval mechanism can better understand the actual needs and feedback of users and improve the accuracy of semantic similarity judgment. The threshold adjustment judgment module uses the difference between the real similarity value and the similarity values corresponding to the two semantic vectors to determine whether the initial similarity threshold needs to be adjusted. This judgment mechanism can be dynamically adjusted according to actual needs and feedback to improve the accuracy and adaptability of similarity threshold setting. When it is determined that the initial similarity threshold needs to be adjusted, the adjustment execution module uses the threshold adjustment model to adjust the initial similarity threshold. This adjustment mechanism can be dynamically adjusted according to the actual situation to ensure that the setting of the similarity threshold is more accurate and reasonable. The above technical solution of this embodiment can automatically adjust and optimize the threshold through the preset threshold adjustment model and algorithm to achieve intelligent management of semantic similarity judgment. This automated and intelligent approach can greatly improve the efficiency and accuracy of semantic similarity judgment and reduce the possibility of manual intervention and misjudgment.

In summary, the threshold setting unit in the above technical solution provides a dynamic, accurate and automated method for determining semantic similarity, which helps to better understand the actual needs and feedback of users and improve the accuracy and adaptability of semantic similarity determination.

Specifically, the threshold adjustment judgment module includes:

A group number extraction module is used to extract the number of groups consisting of two semantic vectors below the corresponding similarity threshold in each observation period when the comparison result between the similarity values corresponding to the two semantic vectors and the similarity initial threshold indicates that the similarity values of the two semantic vectors are below the corresponding similarity threshold;

A value retrieving module, used for retrieving the similarity value corresponding to each group of two semantic vectors below the corresponding similarity threshold in each observation period;

The stability parameter acquisition module is used to obtain the similarity judgment stability parameter contained in each observation period by using the similarity values corresponding to the two semantic vectors below the corresponding similarity threshold in each observation period; wherein the similarity judgment stability parameter is obtained by the following formula:

Wherein, _Sw represents the similarity judgment stability parameter; n represents the number of groups corresponding to two semantic vectors below the corresponding similarity threshold; _S01 and _S02 represent the first similarity stability parameter and the second similarity stability parameter respectively; _Si represents the similarity value corresponding to the i-th group of two semantic vectors below the corresponding similarity threshold; _Sy represents the similarity threshold; m represents the number of groups corresponding to two semantic vectors not lower than the corresponding similarity threshold; _Szi represents the true similarity value corresponding to the two semantic vectors below the corresponding similarity threshold; _Sj represents the similarity value corresponding to the j-th two semantic vectors not lower than the corresponding similarity threshold;

The comprehensive parameter acquisition module is used to integrate the similarity judgment stability parameters contained in each observation period to generate a comprehensive similarity judgment value; wherein the similarity judgment value is obtained by the following formula:

Wherein, _Sq represents the similarity judgment value; k represents the total number of observation periods; _Swj represents the similarity judgment stability parameter of the j-th observation period; _S01j and _S02j represent the first similarity stability parameter and the second similarity stability parameter of the j-th observation period, respectively;

The threshold adjustment execution module is used to determine that the initial similarity threshold needs to be adjusted when the similarity determination value is lower than a preset determination threshold.

The technical effect of the above technical solution is: when the similarity value of two semantic vectors is lower than the similarity threshold, the group number extraction module can extract the number of semantic vector groups below the threshold in each observation period, and the value retrieval module can retrieve the similarity values corresponding to these semantic vectors below the threshold. The two modules work together to provide the necessary data basis for the subsequent stability parameter calculation. Using the similarity values corresponding to the two semantic vectors below the similarity threshold in each observation period, the stability parameter acquisition module can calculate the similarity judgment stability parameter of each observation period. This parameter reflects the stability of the similarity of the semantic vectors below the threshold in each observation period, which helps to determine whether the similarity threshold needs to be adjusted. The comprehensive parameter acquisition module integrates the similarity judgment stability parameters of each observation period to generate a comprehensive similarity judgment value. This judgment value integrates the information of all observation periods and provides a comprehensive indicator for determining whether the similarity threshold needs to be adjusted. The threshold adjustment execution module automatically determines whether the initial similarity threshold needs to be adjusted based on the comparison result between the comprehensive similarity judgment value and the preset judgment threshold. This automated judgment and execution method improves the intelligence and response speed of the system. By carefully analyzing and processing semantic vectors below the similarity threshold and making comprehensive judgments based on data from multiple observation periods, the module can more accurately identify situations where the similarity threshold needs to be adjusted, thereby improving the accuracy and stability of similarity judgments. Since the similarity of semantic vectors may change over time and context, the module can better adapt to this changing semantic environment by dynamically monitoring and adjusting the similarity threshold, thereby improving the adaptability and robustness of the system.

In summary, the threshold adjustment judgment module in the above technical solution realizes accurate, stable and dynamic management of the similarity threshold through meticulous data processing, comprehensive parameter acquisition and automated threshold adjustment judgment and execution mechanism, which helps to improve the accuracy of semantic similarity judgment and the overall performance of the system.

The video retrieval module includes:

The query condition processing unit is used to:

Receive the query conditions entered by the user, extract keywords, scenes, characters and other information, and convert them into semantic vector representation;

Video library index unit, used to:

Establish an index structure for the video library to quickly retrieve videos similar to the query conditions. Preprocess the videos in the video library by interacting with the feature extraction module and the semantic modeling module to generate a corresponding semantic vector representation for each video.

Retrieve matching unit, used to:

By interacting with the similarity calculation module, the similarity between the semantic vector of the query condition and the semantic vector of each video in the video library is calculated, and the videos most relevant to the user's query condition are screened out according to the similarity threshold.

In the above embodiment, by receiving the query conditions input by the user, the system can automatically extract keywords, scenes, characters and other information, and convert them into semantic vector representations, providing users with a more concise and intuitive query method and reducing the steps that users need to manually select and extract features.

In the above embodiment, the establishment of the video library index unit and the preprocessing of the videos in the video library enable the system to quickly retrieve videos similar to the query conditions. Through the interaction with the feature extraction module and the semantic modeling module, each video is converted into a corresponding semantic vector representation, thereby improving the efficiency and accuracy of the retrieval.

In the above embodiment, the retrieval matching unit calculates the similarity between the semantic vector of the query condition and the semantic vector of each video in the video library through interaction with the similarity calculation module, and selects the videos most relevant to the user's query condition according to the similarity threshold, thereby ensuring the accuracy and relevance of the retrieval results.

The result sorting module includes:

The result receiving unit is used for:

Receiving a similarity value output by a similarity calculation module;

Sorting application unit for:

According to the similarity calculation results, apply an appropriate sorting algorithm to sort the videos;

Output display unit for:

The sorted video retrieval results are returned to the user, providing a clear and orderly result display to facilitate users to browse and select the required videos.

In the above embodiment, the result sorting module sorts the videos according to the similarity calculation results, ensuring the accuracy and reliability of the retrieval results, and can put the most relevant and similar videos in the front, improving user satisfaction, providing users with a clear and orderly result display, facilitating users to browse and select the required videos, enhancing practicality, helping to enhance user satisfaction and loyalty, and promoting the widespread application and acceptance of video retrieval systems.

The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims (6)

1. The video query retrieval method based on the semantic depth model is characterized by comprising the following steps of:

Step one: extracting rich characteristic information from the video by using a deep learning technology, wherein the characteristic information comprises colors, textures, shapes and motions;

step two: constructing a deep neural network model, mapping the extracted video features into a semantic space, and generating semantic vectors;

step three: measuring the similarity degree of two videos at the semantic level by calculating the similarity between two semantic vectors;

step four: searching a semantically most similar video from a video library according to a query condition input by a user, firstly carrying out semantic vector representation on the query condition input by the user, and then searching the video most similar to the semantic vector from the video library;

step five: according to the similarity calculation result, ordering the searched videos, and arranging the video which meets the requirements of the user at the front;

In the fourth step, calculating the similarity between the semantic vector of the query condition input by the user and the semantic vector of each video in the video library, and comparing the similarity between each video in the video library according to a similarity threshold value to obtain a video which is most similar to the semantic vector of the query condition input by the user;

the video query retrieval system based on the semantic depth model is applied to the video query retrieval method based on the semantic depth model, and comprises the following steps:

the feature extraction module is used for:

Performing depth analysis on the input video by using a depth learning technology, and extracting color, texture, shape and motion characteristic information from the input video;

The semantic modeling module is used for:

Constructing a deep neural network model, mapping video features into a semantic space, capturing time sequence information and context relation of video content, and generating semantic vector representation;

The similarity calculation module is used for:

Measuring the similarity degree of two videos at the semantic level by calculating the similarity between two semantic vectors;

a video retrieval module for:

searching the semantically most similar video in the video library according to the query condition input by the user, carrying out semantic vector representation on query conditions input by a user, and then searching a video most similar to the semantic vector in a video library;

the result ordering module is used for:

According to the similarity calculation result, sequencing the searched videos, and adjusting the video sequencing most in line with the user requirement to the forefront;

wherein, the similarity calculation module includes:

a semantic vector input unit for:

Receiving a semantic vector from a semantic modeling module, and formatting and preprocessing the input semantic vector;

a similarity measurement unit for:

Selecting a proper similarity measurement method according to specific requirements, and performing similarity calculation on two input semantic vectors to obtain a numerical value as a similarity measurement;

a threshold setting unit configured to:

setting a similarity threshold value, and dynamically adjusting the similarity threshold value according to the actual retrieval effect and user feedback;

Wherein the threshold setting unit includes:

the threshold initial value extraction module is used for extracting a similarity threshold initial value;

the period duration setting module is used for setting the observation period duration of the similarity threshold; the observation period time of the similarity threshold comprises 30-90 unit time lengths, and the unit time length is 24 hours;

the comparison result extraction module is used for extracting comparison results between similarity values corresponding to all the two semantic vectors in the observation period and a similarity initial threshold value for each observation period;

the true similarity value extraction module is used for calling the true similarity values of the two semantic vectors contained in the actual retrieval effect and the user feedback when the comparison result between the similarity values corresponding to the two semantic vectors and the similarity initial threshold value shows that the similarity values of the two semantic vectors are lower than the corresponding similarity threshold value;

The threshold adjustment judging module is used for judging whether the initial threshold of the similarity is required to be adjusted or not by utilizing the difference between the true similarity value and the similarity value corresponding to the two semantic vectors;

the adjustment execution module is used for adjusting the initial threshold value of the similarity by using a threshold adjustment model when the initial threshold value of the similarity is determined to be required to be adjusted; the threshold adjustment model structure is as follows:

s _yt represents a similarity threshold value corresponding to the initial similarity threshold value after adjustment; s _wy represents a preset judgment threshold value; s _q represents a similarity determination value; k represents the total number of observation periods; s _wj represents a similarity judgment stability parameter of the j-th observation period; s _01j and S _02j respectively represent a first similarity stability parameter and a second similarity stability parameter of a j-th observation period; s _y represents a similarity threshold; λ represents a scale parameter.

2. The semantic depth model based video query retrieval system according to claim 1, wherein: the feature extraction module includes:

a feature extraction unit for:

Detecting and extracting color distribution and color histogram information in the video by using a specific layer in the CNN, analyzing a texture mode and a structure in the video by the CNN, performing contour detection and shape recognition on an object in the video, and detecting a motion mode and dynamic change in the video by combining an optical flow method technology;

The feature fusion unit is used for:

integrating the extracted color, texture, shape and motion characteristics, and effectively combining the characteristics to form a unified characteristic vector;

a feature normalization unit for:

And (5) carrying out standardization treatment on the extracted features.

3. The semantic depth model based video query retrieval system according to claim 1, wherein: the semantic modeling module includes:

A timing information capturing unit configured to:

Capturing time sequence information in the video by using the RNN, analyzing time and space relation between continuous frames in the video, and capturing dynamic changes and continuous actions;

A context modeling unit for:

modeling a context of video content using a self-attention mechanism in a Transformer architecture;

The feature fusion and semantic conversion unit is used for:

The information acquired from the time sequence information capturing unit and the context modeling unit is fused, and the fused features are converted into high-level semantic vector representation;

a semantic vector normalization unit configured to:

And carrying out standardization processing on the generated semantic vector.

4. The semantic depth model based video query retrieval system according to claim 1, wherein: the threshold adjustment judging module comprises:

The group number extraction module is used for extracting the group number formed by the two semantic vectors which are lower than the corresponding similarity threshold value in each observation period when the comparison result between the similarity value corresponding to the two semantic vectors and the similarity initial threshold value shows that the similarity value of the two semantic vectors is lower than the corresponding similarity threshold value;

The numerical value calling module is used for calling the similarity numerical value corresponding to the two semantic vectors, of which each group is lower than the corresponding similarity threshold value, in each observation period;

The stability parameter obtaining module is used for obtaining similarity judgment stability parameters contained in each observation period by using similarity values corresponding to two semantic vectors lower than the corresponding similarity threshold value in each observation period; the similarity judging stability parameter is obtained through the following formula:

Wherein S _w represents a similarity determination stability parameter; n represents the number of groups corresponding to two semantic vectors below the corresponding similarity threshold; s ₀₁ and S ₀₂ represent a first similarity stability parameter and a second similarity stability parameter, respectively; s _i represents similarity values corresponding to two semantic vectors of the ith group below the corresponding similarity threshold; s _y represents a similarity threshold; m represents the number of groups corresponding to two semantic vectors not lower than the corresponding similarity threshold; s _zi represents the true similarity value corresponding to two semantic vectors below the corresponding similarity threshold; s _j represents the similarity value corresponding to the j-th two semantic vectors which are not lower than the corresponding similarity threshold;

The comprehensive parameter acquisition module is used for integrating the similarity judgment stability parameters contained in each observation period to generate a comprehensive similarity judgment value; the similarity determination value is obtained through the following formula:

Wherein S _q represents a similarity determination value; k represents the total number of observation periods; s _wj represents a similarity judgment stability parameter of the j-th observation period; s _01j and S _02j respectively represent a first similarity stability parameter and a second similarity stability parameter of a j-th observation period;

And the threshold adjustment execution module is used for determining that the initial threshold of the similarity needs to be adjusted when the similarity judging value is lower than a preset judging threshold.

5. The semantic depth model based video query retrieval system according to claim 1, wherein: the video retrieval module comprises:

A query condition processing unit, configured to:

receiving query conditions input by a user, extracting keywords, scenes and characters, and converting the keywords, the scenes and the characters into semantic vector representations;

A video library index unit for:

establishing an index structure of a video library, preprocessing videos in the video library through interaction with a feature extraction module and a semantic modeling module, and generating corresponding semantic vector representations for each video;

a search matching unit for:

And calculating the similarity of the semantic vector of the query condition and the semantic vector of each video in the video library through interaction with the similarity calculation module, and screening the video most relevant to the query condition of the user according to the similarity threshold.

6. The semantic depth model based video query retrieval system according to claim 1, wherein: the result ordering module comprises:

A result receiving unit configured to:

receiving a similarity value output by a similarity calculation module;

a ranking application unit configured to:

ordering the videos according to the similarity calculation result;

an output display unit for:

And returning the ordered video retrieval results to the user.