CN115357754A - Deep learning-based large-scale short video retrieval method, system and equipment - Google Patents

Abstract

The invention discloses a large-scale short video retrieval method, a large-scale short video retrieval system and large-scale short video retrieval equipment based on deep learning, wherein the method firstly carries out key frame extraction and video standardization processing aiming at short videos to be retrieved; then inputting the processed short videos into a short video semantic feature extraction network, and calculating by utilizing similarity to obtain similar T videos; wherein T is a preset value; the invention designs a new short video semantic feature extraction network to extract the data representation of three-dimensional video data in a two-dimensional space. And introducing a hash representation layer under the structure of the inherent convolutional neural network to obtain a corresponding hash code of the short video in the Hamming space. The invention not only captures the relative semantic correlation of the hash code in different modes, but also obviously reduces the space-time complexity of short video retrieval. The invention fully utilizes the video space-time semantic information and further improves the retrieval performance.

Description

Large-scale short video retrieval method, system and equipment based on deep learning

technical field

The invention belongs to the technical field of video retrieval, and relates to a short video retrieval method, system and equipment, in particular to a large-scale short video retrieval method, system and equipment based on deep learning.

Background technique

With the development and development of Internet technology and smart devices, social networks and other information platforms have emerged a variety of multimedia data, such as text, voice, audio, images and video. In particular, short videos are favored by users. Short videos have risen due to the explosion of social media on the Internet. Short videos are an important carrier of information dissemination. They can convey a lot of information in a short period of time. According to "2021 China Network The Audiovisual Development Research Report shows that the market size of the pan-network audiovisual field exceeds 600 billion, and the short video field accounts for 30% of the market share, with a market size of 205.13 billion. Due to its fragmentation and social characteristics, short videos can better meet the individual needs of users. At the same time, the emergence of various video media has created a variety of video needs. How to find similar duplicates or users interested in videos uploaded by users based on large-scale video databases and limited equipment resources? video has become a meaningful topic in the era of big data.

In recent years, many scholars have used deep learning methods to solve short video data retrieval problems. A common practice is to manually annotate, label and other information on video data, relying on text search to indirectly complete the search task of video data. Although the existing short video retrieval methods have made some progress, there are still two deficiencies: 1) The short video information and the manually marked text information are not completely consistent or one-to-one correspondence, and it is difficult to extract the video information through manual marking. fully elaborated. 2) The video features extracted by machine learning are still very complex, and their storage requires a large amount of storage space, as well as a large amount of computing resources in terms of similarity calculation.

Contents of the invention

In order to solve the above technical problems, the present invention proposes a large-scale short video retrieval method, system and equipment based on deep learning, by learning the semantic information of short video data, using an improved convolutional neural network model to generate hash codes, and finally Use similarity calculation to retrieve a given number of short video items.

The technical solution adopted in the method of the present invention is: a large-scale short video retrieval method based on deep learning, comprising the following steps:

Step 1: Concentrate query data and retrieve short videos, perform key frame extraction and video standardization;

Step 2: Input the processed data into the short video semantic feature extraction network, and use the similarity calculation to obtain T similar videos; where T is a preset value;

The short video semantic feature extraction network consists of a feature extraction module and a feature hash code mapping module, including a first 3×3×3 convolutional layer, a first 1×2×2 pooling layer, a second 3×3 ×3 convolutional layer, second 2×2×2 pooling layer, third 3×3×3 convolutional layer, third 2×2×2 pooling layer, fourth 3×3×3 convolutional layer, The fourth 2×2×2 pooling layer, the fifth 3×3×3 convolutional layer, the fifth 2×2×2 pooling layer, the first 1×4096 fully connected layer, the second 1×4096 fully connected layer , Hash layer composition;

The first 3×3×3 convolutional layer, the first 1×2×2 pooling layer, the second 3×3×3 convolutional layer, the second 2×2×2 pooling layer, the third 3× 3×3 convolutional layer, third 2×2×2 pooling layer, fourth 3×3×3 convolutional layer, fourth 2×2×2 pooling layer, fifth 3×3×3 convolutional layer , the fifth 2×2×2 pooling layers are sequentially connected to form a feature extraction module together, input a short video, and obtain the original feature vector graph of the short video through the feature extraction module;

The first 1×4096 fully connected layer, the second 1×4096 fully connected layer, and the hash layer are sequentially connected to form a feature hash code mapping module, which inputs the original feature vector map of the short video, and passes through the short video feature hash code. The mapping module obtains the hash code after the feature mapping of the short video;

The hash layer is a 1×K fully connected layer activated by a sigmoid function, and K is a preset hash code length.

The technical solution adopted by the system of the present invention is: a large-scale short video retrieval system based on deep learning, including a preprocessing module and a retrieval module;

Described preprocessing module is used for querying data and retrieving centralized short video, carries out key frame extraction and video standardization processing;

Described retrieval module, for the data input after processing short video semantic feature extraction network, utilize similarity calculation to obtain similar T videos; Wherein, T is preset value;

The short video semantic feature extraction network is composed of a feature extraction module and a feature hash code mapping module as a whole, including a first 3×3×3 convolutional layer, a first 1×2×2 pooling layer, a second 3× 3×3 convolutional layer, second 2×2×2 pooling layer, third 3×3×3 convolutional layer, third 2×2×2 pooling layer, fourth 3×3×3 convolutional layer , the fourth 2×2×2 pooling layer, the fifth 3×3×3 convolutional layer, the fifth 2×2×2 pooling layer, the first 1×4096 fully connected layer, the second 1×4096 fully connected layer, hash layer;

The first 3×3×3 convolutional layer, the first 1×2×2 pooling layer, the second 3×3×3 convolutional layer, the second 2×2×2 pooling layer, the third 3× 3×3 convolutional layer, third 2×2×2 pooling layer, fourth 3×3×3 convolutional layer, fourth 2×2×2 pooling layer, fifth 3×3×3 convolutional layer , the fifth 2×2×2 pooling layers are sequentially connected to form a feature extraction module together, input a short video, and obtain the original feature vector graph of the short video through the feature extraction module;

The first 1×4096 fully connected layer, the second 1×4096 fully connected layer, and the hash layer are sequentially connected to form a feature hash code mapping module, which inputs the original feature vector map of the short video, and passes through the short video feature hash code. The mapping module obtains the hash code after the feature mapping of the short video;

The hash layer is a 1×K fully connected layer activated by a sigmoid function, and K is a preset hash code length.

The technical solution adopted by the device of the present invention is: a large-scale short video retrieval device based on deep learning, including:

one or more processors;

The storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors realize the described deep learning-based large-scale Scale short video retrieval method.

Compared with the prior art, the method proposed by the present invention not only captures the semantic correlation of videos in the time dimension, learns the relative semantic correlation of deep features, but also greatly optimizes the time-space complexity of video retrieval.

Description of drawings

Fig. 1 is a schematic frame diagram of an embodiment of the present invention.

Fig. 2 is a schematic diagram of the principle of an embodiment of the present invention.

FIG. 3 is a network structure diagram of short video semantic feature extraction according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of a short video semantic feature extraction network training process according to an embodiment of the present invention.

Detailed ways

In order to facilitate those of ordinary skill in the art to understand and implement the present invention, the present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described here are only used to illustrate and explain the present invention, and are not intended to limit this invention.

Please see Fig. 1 and Fig. 2, a kind of large-scale short video retrieval method based on deep learning provided by the present invention, comprises the following steps:

Step 1: Concentrate query data and retrieve short videos, perform key frame extraction and video standardization;

In this embodiment, the key frame extraction is based on the characteristics of the short video shot, that is, shooting in a one shot to the bottom mode, and using the technology of extracting key frames at equal intervals. Video standardization is based on the size of the video. If the video image is too large, downsampling is required. If the video image is too small, interpolation is required. Finally, the processed images of each frame are spliced to obtain a complete video.

In this embodiment, the video standardization process is based on the size of the video. If the video image is larger than the preset value, the downsampling operation is performed, and the M×N image is downsampled by s times to obtain (M/s)× (N/s) image; the specific formula is as follows:

DS(f)＝P _avg ({b ₁ ,b ₁ ,...,b _n })

Among them, f is the video frame, b is the different s×s small blocks that the picture is divided into, and P _avg () is the mean value pooling operation;

If the size of the video image is smaller than the preset value, the interpolation operation is performed. Here, in order to speed up the calculation, the missing pixel value is the average value of the horizontal nearest pixel and the vertical nearest pixel; finally, each frame of the processed image is processed. Stitching to get a complete video.

其中F_i为第i个视频帧，N为视频V 的总帧数；则提取的关键帧

其中，t代表以秒为单位的视频长度，FPS代表帧率， n是视频挑选的帧数，t＝1,2,...,n。If input video

Where F _i is the i-th video frame, N is the total number of frames of the video V; then the extracted key frame

Wherein, t represents the length of the video in seconds, FPS represents the frame rate, n is the number of frames selected from the video, and t=1,2,...,n.

Step 2: Input the processed data into the short video semantic feature extraction network, and use the similarity calculation to obtain T similar videos; where T is a preset value;

In this embodiment, use the short video semantic feature extraction network to calculate the hash code of the processed short video, sort the Hamming distances between the query data and the hash codes of each sample in the retrieval set from large to small, and calculate the ranking The first T precisions of the list are used as the search results.

Please see Figure 3. The short video semantic feature extraction network in this embodiment is composed of a feature extraction module and a feature hash code mapping module as a whole, including the first 3×3×3 convolutional layer and the first 1×2×2 pooling layer. layer, the second 3×3×3 convolutional layer, the second 2×2×2 pooling layer, the third 3×3×3 convolutional layer, the third 2×2×2 pooling layer, the fourth 3 ×3×3 convolutional layer, fourth 2×2×2 pooling layer, fifth 3×3×3 convolutional layer, fifth 2×2×2 pooling layer, first 1×4096 fully connected layer, The second 1×4096 fully connected layer and hash layer; the first 3×3×3 convolutional layer, the first 1×2×2 pooling layer, the second 3×3×3 convolutional layer, the second 2 ×2×2 pooling layer, third 3×3×3 convolutional layer, third 2×2×2 pooling layer, fourth 3×3 ×3 convolutional layer, fourth 2×2×2 pooling layer Layer, the fifth 3×3×3 convolutional layer, and the fifth 2×2×2 pooling layer are sequentially connected to form a feature extraction module, input a short video, and obtain the original feature vector map of the short video through the feature extraction module; the first The 1×4096 fully connected layer, the second 1×4096 fully connected layer, and the hash layer are sequentially connected to form a feature hash code mapping module. The original feature vector map of the short video is input, and the short video is obtained through the short video feature hash code mapping module. The hash code after video feature mapping; the hash layer is a 1×K fully connected layer activated by the sigmoid function, and K is the preset hash code length.

In this embodiment, the deep feature of the video represents f(V), and the hash represents layer H(V). The size of the feature vector obtained from the fully connected layer is [N, 4096, 1, 1, 1], that is, the number of nodes in the fully connected layer num is 4096. Assuming that the feature vector output from the fully connected layer is FC={FC ₁ , FC ₂ ,...,FC _num }, the feature vector obtained after the hash layer is H={h ₁ , h ₂ ,..., h _d }, where d is the number of hash layer nodes, that is, the vector dimension of the hash layer output. In order to ensure that each type of label in the data set has a unique hash code corresponding to it, at least d _min nodes are required, and the calculation formula for the number of nodes is as follows:

Assuming that the maximum value of d is d _max , then d _max = num, that is, the number of nodes in the hash layer needs to be lower than the number of nodes in the upper fully connected layer. Since the purpose of the hash layer is to map the fully connected features into a compact binary-like code, it is necessary to reduce the dimensionality of the features output by the previous layer. Then the value range of d is:

The hash function is defined as follows:

[ ₁ , ₂ ,..., _d ] ^T ＝ [Sign(W*FC+B)] ^T ;

Among them, Sign is the sign function, W is the weight of the hash layer, FC is the fully connected layer, and B is the bias.

In order to map hash features into binary codes, a threshold function is used to process the output value. The threshold function is designed as follows:

Please see Fig. 4, the short video semantic feature extraction network of the present embodiment is a trained short video semantic feature extraction network; its training process includes the following steps:

Step 2.1: Obtain the data set and divide it into training data set and test data set;

In this embodiment, use the HMDB51 and UCF-101 data sets, select 70% of the data sets as the training data set I _train , and the remaining 30% as the test data set I _test ;

Step 2.2: For the training data set and the test data set, perform key frame extraction and video standardization;

In this embodiment, the key frame extraction is based on the characteristics of the short video shot, that is, shooting in a one shot to the bottom mode, and using the technology of extracting key frames at equal intervals. Video standardization is based on the size of the video. If the video image is too large, downsampling is required. If the video image is too small, interpolation is required. Finally, the processed images of each frame are spliced to obtain a complete video.

其中F_i为第i个视频帧，N为视频V 的总帧数；则提取的关键帧

其中，t代表以秒为单位的视频长度，FPS代表帧率， n是视频挑选的帧数，t＝1,2,...,n。If input video

Where F _i is the i-th video frame, N is the total number of frames of the video V; then the extracted key frame

Wherein, t represents the length of the video in seconds, FPS represents the frame rate, n is the number of frames selected from the video, and t=1,2,...,n.

Step 2.3: Determine the training objective function, optimization algorithm, learning rate, momentum, weight decay, batch size and network training iteration number epoch;

In this embodiment, the objective function Loss is composed of a triplet loss function L _Triplet , a cross-entropy loss function L _CE and a smoothed average precision loss function L _AP ;

Loss = αL _Triplet + βL _CE + γL _AP ;

Among them, α, β, and γ are all hyperparameters, and the weight parameter W and bias parameter B of the network are obtained by training the model backpropagation technology;

其中，

为锚样本，是选定的某个类型的视频数据，

为与锚样本同类型的正例样本，

为与锚样本不同类型的负例样本；f()为将样本数据映射到同一个向量空间函数；l为预设 的负例样本与正例样本的最小距离；K为数据总样本数；In this example, the triplet loss function

in,

An anchor sample is a selected type of video data,

is a positive sample of the same type as the anchor sample,

is a different type of negative sample from the anchor sample; f() is a function that maps sample data to the same vector space; l is the minimum distance between a preset negative sample and a positive sample; K is the total number of data samples;

其中， K为数据总样本数，M为类别的数量；y_ic为符号函数，取值0或1，如果样本i的真是类别等于c则取1；p_ic为观测样本i属于类别c的预测概率；In this embodiment, the cross-entropy loss function of multi-classification

Among them, K is the total number of samples of the data, M is the number of categories; y _ic is a sign function, the value is 0 or 1, if the real category of sample i is equal to c, then it is 1; p _ic is the prediction that the observed sample i belongs to category c probability;

其中，m为样本个数；AP_k为 平滑平均估计值，

S_P为数据集中和查询视频属于 同一类型的视频集合，

表示将指示函数替换为了可导的Sigmoid函数；D_ij为排名矩阵，第 i行表示第i个样本和其余样本的相关性的排名分数；τ为平滑系数，不同的平滑系数可以提 供不同的梯度信息；In this example, the smoothing average precision loss function

Among them, m is the number of samples; AP _k is the smoothed average estimated value,

S _P is the video collection of the same type as the query video in the dataset,

Indicates that the indicator function is replaced by a derivable Sigmoid function; D _ij is a ranking matrix, and the i-th row represents the ranking score of the correlation between the i-th sample and the rest of the samples; τ is a smoothing coefficient, and different smoothing coefficients can provide different gradients information;

Step 2.4: Input the training data set into the short video semantic feature extraction network, and train the short video semantic feature extraction network;

In this example, the Adam algorithm is used for optimization, the learning rate is set to 10 ^-5 , the momentum is set to 0.9, the weight decay is set to 10 ^-5 , the batch size is set to 32, and the learning rate decay strategy is based on the Loss of the verification set. No more dips to adjust. The size of the input frame image is 3×112×112 and each convolution is 16 frames as a clip. The initial weights of the convolutional neural network ResNet34_3D are initialized with pre-trained weights. The weight parameter W and bias parameter B of the network are obtained by training the model.

Step 2.5: Use the trained short video semantic feature extraction network to calculate the hash codes of the samples in the test data set, sort the Hamming distances between the query samples and the hash codes of the samples in the training data set from large to small, and calculate The top n accuracy of the ranking list is used to obtain the average precision index MAP and the top n retrieval results; the category of the input video is the same as the category of the retrieved video, which means the retrieval is correct, and the value of the above loss function tends to be stable and no longer declines to obtain the effect good internet.

Platinum 8260 CPU＠2.4GHZ、NVIDIA A100TENSOR CORE GPU、Ubuntu 20.04系统，运用Python和开 源库Pytorch进行开发。In order to evaluate the effectiveness of the method of the present invention, the retrieval performance of the method of the present invention was compared with several state-of-the-art methods, including FIHTV, VHSL, SVH, DH, DVH, DBNVH, BRVH, DSH, DLBHC and SRH. Bits (16bits, 32bits, 64bits) hash codes, using UCF-101 data set and HMDB-51 data set, DVH method uses convolutional neural network to learn the feature expression and hash function of the image at the same time, and then they are corresponding The features projected into a common representation space, the FIHTV, VHSL, SVH, DH, DBNVH, BRVH, DSH, DLBHC and SRH methods perform as in the original text. The environment used in this experiment is

Platinum 8260 CPU@2.4GHZ, NVIDIA A100TENSOR CORE GPU, Ubuntu 20.04 system, using Python and open source library Pytorch for development.

Table 1

Table 1 is the experimental results of the comparison between the present invention and other methods on the video retrieval task on the UCF-101 dataset, where mAP is the average precision index.

Table 2

Table 2 is the comparative experimental results of the present invention and other methods on the video retrieval task on the HMDB-51 dataset, where mAP is the average precision index.

table 3

Table 3 is the comparison in speed of different video retrieval methods of the present invention. Multi-stage hash retrieval refers to using binary hash feature coding of 16 bits, 32bits, 64bits in order to compare with video feature coding representations in the database, step by step Narrow down the search scope, and finally use the real-valued feature encoding to perform an accurate comparison in the database. The invention makes full use of the temporal and spatial semantic information of the video, further improving the retrieval performance.

It should be understood that the above-mentioned descriptions for the preferred embodiments are relatively detailed, and should not therefore be considered as limiting the scope of the patent protection of the present invention. Within the scope of protection, replacements or modifications can also be made, all of which fall within the protection scope of the present invention, and the scope of protection of the present invention should be based on the appended claims.

Claims (7)

1. A large-scale short video retrieval method based on deep learning, is characterized in that, comprises the following steps: Step 1: Concentrate query data and retrieve short videos, perform key frame extraction and video standardization; Step 2: Input the processed data into the short video semantic feature extraction network, and use the similarity calculation to obtain T similar videos; where T is a preset value; The short video semantic feature extraction network consists of a feature extraction module and a feature hash code mapping module, including a first 3×3×3 convolutional layer, a first 1×2×2 pooling layer, a second 3×3 ×3 convolutional layer, second 2×2×2 pooling layer, third 3×3×3 convolutional layer, third 2×2×2 pooling layer, fourth 3×3×3 convolutional layer, The fourth 2×2×2 pooling layer, the fifth 3×3×3 convolutional layer, the fifth 2×2×2 pooling layer, the first 1×4096 fully connected layer, the second 1×4096 fully connected layer , Hash layer composition; The first 3×3×3 convolutional layer, the first 1×2×2 pooling layer, the second 3×3×3 convolutional layer, the second 2×2×2 pooling layer, the third 3× 3×3 convolutional layer, third 2×2×2 pooling layer, fourth 3×3×3 convolutional layer, fourth 2×2×2 pooling layer, fifth 3×3×3 convolutional layer , the fifth 2×2×2 pooling layers are sequentially connected to form a feature extraction module together, input a short video, and obtain the original feature vector graph of the short video through the feature extraction module; The first 1×4096 fully connected layer, the second 1×4096 fully connected layer, and the hash layer are sequentially connected to form a feature hash code mapping module, which inputs the original feature vector map of the short video, and passes through the short video feature hash code. The mapping module obtains the hash code after the feature mapping of the short video; The hash layer is a 1×K fully connected layer activated by a sigmoid function, and K is a preset hash code length. 2. The large-scale short video retrieval method based on deep learning according to claim 1, characterized in that: in step 1, the key frame extraction is based on the characteristics of the short video lens, that is, taking a shot to the end to shoot , the key frame extraction is carried out by using the key frame extraction technology at equal intervals; If input video

Among them, F _i is the i-th video frame, and N is the total number of frames of the video V; then the extracted key frame

Wherein, t represents the length of the video in seconds, FPS represents the frame rate, n is the number of frames selected from the video, and t=1,2,...,n. 3. The large-scale short video retrieval method based on deep learning according to claim 1, characterized in that: in step 1, the video standardization process is based on the size of the video size, if the video image is greater than a preset value, then Perform down-sampling operation processing, and perform s-fold down-sampling on the M×N image to obtain an (M/s)×(N/s) image; the specific formula is as follows: DS(f)＝P _avg ({b ₁ ,b ₁ ,...,b _n }) Among them, f is the video frame, b is the different s×s small blocks that the picture is divided into, and P _avg () is the mean value pooling operation; If the size of the video image is smaller than the preset value, the interpolation operation is performed, and the missing pixel value is the average value of the horizontal nearest pixel and the vertical nearest pixel; finally, the processed images of each frame are spliced to obtain a complete video . 4. The large-scale short video retrieval method based on deep learning according to claim 1, characterized in that: in step 2, use the hash code of the short video processed by the short video semantic feature extraction network calculation, query data and The Hamming distance between the hash codes of each sample in the retrieval set is sorted from large to small, and the top T accuracy of the ranking list is calculated as the retrieval result. 5. The large-scale short video retrieval method based on deep learning according to any one of claims 1-4, characterized in that: the short video semantic feature extraction network described in step 2 is a trained short video semantic feature extraction network; its training process includes the following steps: Step 2.1: Obtain the data set and divide it into training data set and test data set; Step 2.2: For the training data set and the test data set, perform key frame extraction and video standardization; Step 2.3: Determine the training objective function, optimization algorithm, learning rate, momentum, weight decay, batch size and network training iteration number epoch; The objective function Loss is composed of triplet loss function L _Triplet , cross-entropy loss function L _CE and smooth average precision loss function L _AP ; Loss = αL _Triplet + βL _CE + γL _AP ; Among them, α, β, and γ are all hyperparameters, and the weight parameter W and bias parameter B of the network are obtained by training the model backpropagation technology; The triplet loss function

in,

An anchor sample is a selected type of video data,

is a positive sample of the same type as the anchor sample,

is a different type of negative sample from the anchor sample; f() is a function that maps sample data to the same vector space; l is the minimum distance between a preset negative sample and a positive sample; K is the total number of data samples; The multi-class cross-entropy loss function

Among them, K is the total number of data samples, M is the number of categories; y _ic is a sign function, with a value of 0 or 1, if the real category of sample i is equal to c, then it is 1; p _ic is the prediction that observed sample i belongs to category c probability; The smoothed average precision loss function

Among them, m is the number of samples; AP _k is the smoothed average estimated value,

S _P is the video collection of the same type as the query video in the dataset,

Indicates that the indicator function is replaced by a derivable Sigmoid function; D _ij is a ranking matrix, and the i-th row represents the ranking score of the correlation between the i-th sample and the rest of the samples; τ is a smoothing coefficient; Step 2.4: input the training data set into the short video semantic feature extraction network, and train the short video semantic feature extraction network; Step 2.5: Use the trained short video semantic feature extraction network to calculate the hash codes of the samples in the test data set, sort the Hamming distances between the query samples and the hash codes of the samples in the training data set from large to small, and calculate The top n accuracy of the ranking list is used to obtain the average precision index MAP and the top n retrieval results; the category of the input video is the same as the category of the retrieved video, which means the retrieval is correct, and the value of the above loss function tends to be stable and no longer declines to obtain the effect good internet. 6. A large-scale short video retrieval system based on deep learning, characterized in that: it includes a preprocessing module and a retrieval module; The preprocessing module is used to collect query data and retrieve short videos, and perform key frame extraction and video standardization processing; The retrieval module is used to input the processed data into the short video semantic feature extraction network, and use similarity calculations to obtain similar T videos; wherein, T is a preset value; The short video semantic feature extraction network consists of a feature extraction module and a feature hash code mapping module, including a first 3×3×3 convolutional layer, a first 1×2×2 pooling layer, a second 3×3 ×3 convolutional layer, second 2×2×2 pooling layer, third 3×3×3 convolutional layer, third 2×2×2 pooling layer, fourth 3×3×3 convolutional layer, The fourth 2×2×2 pooling layer, the fifth 3×3×3 convolutional layer, the fifth 2×2×2 pooling layer, the first 1×4096 fully connected layer, the second 1×4096 fully connected layer , Hash layer composition; The first 3×3×3 convolutional layer, the first 1×2×2 pooling layer, the second 3×3×3 convolutional layer, the second 2×2×2 pooling layer, the third 3× 3×3 convolutional layer, third 2×2×2 pooling layer, fourth 3×3×3 convolutional layer, fourth 2×2×2 pooling layer, fifth 3×3×3 convolutional layer , the fifth 2×2×2 pooling layers are sequentially connected to form a feature extraction module together, input a short video, and obtain the original feature vector graph of the short video through the feature extraction module; The first 1×4096 fully connected layer, the second 1×4096 fully connected layer, and the hash layer are sequentially connected to form a feature hash code mapping module, which inputs the original feature vector map of the short video, and passes through the short video feature hash code. The mapping module obtains the hash code after the feature mapping of the short video; The hash layer is a 1×K fully connected layer activated by a sigmoid function, and K is a preset hash code length. 7. A large-scale short video retrieval device based on deep learning, characterized in that it comprises: one or more processors; A storage device, used to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors can realize any one of claims 1 to 5 A large-scale short video retrieval method based on deep learning.

Images