CN117540047A - Method, system, equipment and storage medium for retrieving video based on picture - Google Patents

Abstract

The present disclosure provides a method, system, device and storage medium for video retrieval based on pictures. The method includes: acquiring a plurality of video data; processing the plurality of video data, generating a first category and a first feature vector of the encoding of each video data and its key frames, and converting the generated plurality of first categories The items and multiple first feature vectors are stored in the video database; image data is obtained; the image data is processed to generate a second category and a second feature vector of the image data; among the multiple first categories in the video database Find the first category that is the same as the second category and confirm it as the query category; calculate the similarity between the second feature vector and multiple first feature vectors corresponding to the query category; sort the similarities and generate The video list corresponding to the similarity can accurately and quickly retrieve videos similar to the image based on the input image.

Description

Method, system, equipment and storage medium for video retrieval based on images

Technical field

The present disclosure relates to the field of image technology, and in particular, to a method, system, device and storage medium for video retrieval based on pictures.

Background technique

In the digital age, pictures and videos are widely used in various fields, and a large amount of media data is continuously generated and accumulated. How to easily retrieve and use this data is a challenging task, so the technology of using pictures to retrieve videos has become more and more important.

Contents of the invention

In view of this, the purpose of this disclosure is to propose a method, system, device and storage medium for video retrieval based on pictures, which can accurately and quickly retrieve videos similar to pictures based on input pictures.

In order to achieve one of the above disclosed purposes, the present disclosure provides a method for retrieving videos based on pictures. The method includes: acquiring multiple video data; processing multiple video data to generate encoding and decoding of each video data. the first category and the first feature vector of its key frame, and store the generated multiple first categories and multiple first feature vectors in the video database; obtain image data; process the image data to generate The second category and the second feature vector of the picture data; search for the first category that is the same as the second category among multiple first categories in the video database, and confirm it as the query category; Calculate the similarity between the second feature vector and multiple first feature vectors corresponding to the query category; sort the similarities to generate a video list corresponding to the similarities.

In some embodiments of the present disclosure, storing the generated plurality of first categories and the plurality of first feature vectors in a video database, and before obtaining the picture data, further includes: obtaining a plurality of the generated first categories and the plurality of first feature vectors in the video database. The single access amount of the video data within the set time; in response to the single access amount being less than the threshold, delete the first category and first feature vector of the video data and its key frames corresponding to the single access amount.

In some embodiments of the present disclosure, the plurality of video data are processed, a first category and a first feature vector of the encoding of each video data and its key frame are generated, and the generated plurality of first feature vectors are generated. Storing a category and a plurality of first feature vectors in a video database includes: storing each video data in a distributed file database and encoding each video data; and sequentially acquiring each of the plurality of video data. Key frames of video data; determine the category to which each group of key frames belongs, and store them in the video database as the first category; calculate pictures of each group of key frames based on the pre-trained VGG16 model feature vector and stored in the video database as the first feature vector.

In some embodiments of the present disclosure, sequentially acquiring key frames of each video data in a plurality of the video data further includes: acquiring an MPEG-4 standard table of the video data, and determining the MPEG-4 standard Whether the stss part is included in the table; in response to the stss part being included in the standard table, it is determined that the frame identified by the stss part is a key frame; in response to the stss part being not included in the standard table, the video frame is calculated using the edge detection calculation method similarity between them, generate multiple similar frames; cluster multiple similar frames to generate a set of similar frames; use a picture quality assessment algorithm to process the set of similar frames, and select each set of similar frames The optimal frame is the key frame.

In some embodiments of the present disclosure, determining the category to which each group of key frames belongs and storing it in the video database as the first category includes: based on a picture classification algorithm, determining the category to which each group of key frames belongs. The category to which the key frame belongs is stored in the video database as the first category.

In some embodiments of the present disclosure, before obtaining the MPEG-4 standard of the video data, the method further includes: dividing the video data into frames to obtain individual video frames; preprocessing the video frames, Optimized video frames are obtained; the preprocessing includes noise reduction and/or size unification.

In some embodiments of the present disclosure, processing the picture data to generate a second category and a second feature vector of the picture data includes: determining the category of the picture data based on a picture classification algorithm. , and stored as the second category; based on the pre-trained VGG16 model, calculate the picture feature vector of the picture data, and store it as the second feature vector.

In order to achieve another embodiment of the above disclosed purpose, the present disclosure provides a system for retrieving videos based on pictures, including: a first acquisition module for acquiring multiple video data; a first processing module for processing multiple video data Process the video data, generate the encoding of each video data and the first category and the first feature vector of its key frame, and store the generated multiple first categories and multiple first feature vectors in the video database; The second acquisition module is used to obtain picture data; the second processing module is used to process the picture data and generate a second category of the picture data and a second feature vector search module for searching in the video Search the database for the first category that is the same as the second category and confirm it as the query category; a calculation module for calculating a plurality of first features corresponding to the second feature vector and the query category Similarity between vectors; a generation module, used to sort the similarity and generate a video list corresponding to the similarity.

In order to achieve another embodiment of the above disclosed object, the present disclosure provides a computer device, including a processor and a memory; the memory stores a computer program, and when the computer program is executed by the processor, the processor Carry out the steps of any of the methods described above.

In order to achieve another embodiment of the above disclosed object, the present disclosure provides a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium stores computer instructions, the computer instructions are used to cause the computer to perform any of the above. A method for retrieving videos based on images.

Compared with the existing technology, the technical effect of the present invention is that: by processing the video data, the encoding of each video is retained, and the first category of the key frame and the first feature vector are generated in the video data stored; in the user After inputting the image for retrieval, the image is processed to obtain the second category and the second feature vector of the image, and the retrieval is performed within the same range of the first category as the second category, reducing the search scope and shortening the retrieval process. Reduce the calculation amount and time cost of model application to meet the requirements of scenarios that require high real-time retrieval.

Description of drawings

In order to explain the technical solutions in the present disclosure or related technologies more clearly, the drawings needed to be used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings in the following description are only for illustration of the present disclosure. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a flow chart of a method for retrieving videos based on images provided by an embodiment of the present disclosure;

Figure 2 is a flow chart of optimizing a video database provided by another embodiment of the present disclosure;

Figure 3 is a flow chart for processing video data provided by another embodiment of the present disclosure;

Figure 4 is a flow chart for obtaining key frames provided by another embodiment of the present disclosure;

Figure 5 is a flow chart for processing image data provided by another embodiment of the present disclosure;

Figure 6 is a schematic diagram of video retrieval based on volume pictures provided by another embodiment of the present disclosure;

FIG. 7 is a schematic diagram of the hardware structure of a computer device provided by another embodiment of the present disclosure.

Detailed ways

The present invention will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention. Structural, method, or functional changes made by those of ordinary skill in the art based on these embodiments are all included in the protection scope of the present invention.

Embodiments It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by those with ordinary skills in the field to which the disclosure belongs. "First", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprising" mean that the elements or things appearing before the word include the elements or things listed after the word and their equivalents, without excluding other elements or things.

Retrieval of videos by images is essentially a retrieval of images, and image retrieval has long been an important aspect in computer vision research. Image retrieval refers to the process of searching and retrieving relevant images from large image datasets. Over the years, various methods and techniques have been developed to achieve accurate and efficient image retrieval.

Among the existing image retrieval methods, one of the methods is based on the VGG16 network model, which has proven to be very effective in image recognition tasks.

VGG16 is a deep convolutional neural network. It consists of 16 layers and has more than 138 million parameters. The VGG16 network model is able to achieve state-of-the-art results in a variety of image recognition tasks, including object classification, object localization and object detection. The VGG16 model is trained on the ImageNet dataset, a large-scale dataset containing over a million images in 1000 different categories. This dataset is used to train the VGG16 model using supervised learning techniques. During training, the VGG16 model learns to recognize various features in pictures such as edges, corners, shapes, colors, and textures, etc.

Once the VGG16 model is trained, it can be used for various image processing tasks, including image retrieval. Image retrieval using the VGG16 model involves extracting features from an image and then comparing them to features of other images in the database. Feature similarity between two images can be measured using various metrics, including Euclidean distance, cosine similarity, and correlation coefficient, among others.

But the VGG16 model also has some flaws. First, it is highly dependent on the similarity between the image to be retrieved and the training data set. Therefore, in practical applications, the model may be biased due to over-adaptation to certain aspects of the input data. For example, when testing, it encounters an image that is different from the training data set. The performance of the model will be greatly affected by scenes, changing environments, etc.; secondly, due to its excessive depth and parameter volume, it is computationally expensive and time-consuming. All current image search methods often require the input of accurate Keywords or tags, users need to have certain professional knowledge and diverse thinking capabilities; finally, when the amount of input data is too large, the processing speed will further slow down, resulting in scenarios that require high real-time performance being unable to meet the needs. In practical applications, it is still necessary to consider the time and space complexity of the algorithm, as well as the operating efficiency of the applications developed based on the algorithm.

In view of this, the embodiment of the present disclosure provides a method for retrieving videos based on pictures, which can reduce the calculation consumption during the retrieval process. As shown in Figure 1, the method for retrieving videos based on pictures includes the following steps:

Step S100: Obtain multiple video data.

Specifically, the scope of obtaining video data is the scope of retrieval. For example, on a certain video platform, all videos on the video platform are used as the retrieval scope, or when the user searches for videos locally on the device, the local videos on the device are used as the retrieval scope. .

Step S200: Process the plurality of video data, generate the encoding of each video data and the first category and the first feature vector of the key frame, and use the generated multiple first categories and multiple first feature vectors stored in the video database.

Specifically, multiple video data are processed in sequence. Each video data has a separate encoding to facilitate video query and extraction. Each video data has its key frame. According to the length and content of the video data, Each video data may correspond to one key frame or multiple key frames. Each key frame may correspond to a first category and a first feature vector. Each video data may correspond to multiple first categories and multiple first features. Feature vector. The first category is a classification classification, so there may be one or more video data, key frames and first feature vectors sent to the first category.

After processing the video data, the features contained in the video data are extracted into the features of the image format data, which facilitates retrieval of the input image data and direct comparison with it.

In an implementation manner of the present disclosure, for step S200, see Figure 3, which specifically also includes:

Step S210, store each video data in the distributed file database, and encode each video data;

Step S220: Obtain the key frames of each video data in the plurality of video data in sequence;

Step S230, determine the category to which each group of key frames belongs, and store it in the video database as the first category;

Step S240: Calculate the picture feature vector of each group of key frames based on the pre-trained VGG16 model, and store it in the video database as the first feature vector.

Specifically, the encoding of each video data is a unique traceable ID, and the video data stored in different locations can be quickly queried based on the encoding. In addition, a video is a continuous image composed of many frames. For any end of the video, the number of image frames extracted is very large. Processing all the frames together may cause performance problems, and it is difficult to find a specific frame or time period. Therefore, when processing video, it is necessary to appropriately control the number of selected image frames to ensure processing efficiency.

Determine in advance the category to which the selected keyframes belong, and limit the search scope to specific categories before performing calculations, which can effectively reduce the amount of calculation and improve response speed. Specific categories include scene categories, classification based on feature rules such as color and shape, or classification based on statistical models.

Feature extraction using pre-trained VGG16 model and retrograde keyframes.

The specific process is: select the output of the convolution layer and fully connected layer of the VGG16 model, with a total of 4096 feature values. Since most of the key frame pictures are color pictures, a three-channel convolution kernel of 3×3 and a pooling kernel are used. is 2×2. Taking a 224×224 video image as an example, the specific calculation process is as follows:

1) The input image size is 224×224×3, and the convolution kernel of 64 channels is 3×3, with a step size of 1, and a total of 2 convolutions, and the output size is a feature vector of 224×224×64. For pooling, use a pooling kernel of 2×2, a stride of 2, and an output feature vector of size 112×112×64;

2) After 128 3×3 convolution kernels, the step size is 1, convolution is performed twice, the size becomes 112×112×128, and pooling is performed, the step size is 2, and the output size is 56×56×128;

3) After 256 3×3 convolution kernels, the step size is 1, convolution is performed 3 times, the size becomes 56×56×256, pooling is performed, the step size is 2, and the output size is 28×28×256;

4) After 512 3×3 convolution kernels, the step size is 1, convolution is performed 3 times, the size becomes 28×28×512, and pooling is performed, the step size is 2, and the output size is 14×14×256;

5) After 512 3×3 convolution kernels, the step size is 1, convolution is performed 3 times, the size becomes 14×14×512, and pooling is performed, the step size is 2, and the output size is 7×7×256;

6) Flatten the data into a one-dimensional array, 7×7×256=25088;

7) After two layers of 1×1×4096 and one layer of 1×1×1000 fully connected layer, a feature vector of 1×1000 is finally output.

In an implementation manner of the present disclosure, for sequentially obtaining key frames of each video data in multiple video data, see Figure 4. The method also includes:

Step S221, obtain the MPEG-4 standard table of video data;

Step S222, determine whether the MPEG-4 standard table includes the stss part;

Step S223, in response to the standard table including the stss part, determine that the frame identified by the stss part is a key frame;

Step S224, in response to the standard table not including the stss part, use the edge detection calculation method to calculate the similarity between the video frames and generate multiple similar frames;

Step S225, cluster multiple similar frames to generate a set of similar frames;

Step S226: Use the picture quality evaluation algorithm to process the similar frame sets, and select the optimal frame in each similar frame set as the key frame.

Specifically, in the MPEG-4 standard, the stss part helps the decoder quickly locate and parse key frames by identifying the location and number of key frames, thereby improving the efficiency and quality of video decoding. For example, the MPEG-4 standard includes stss part, the keyframes are obtained directly.

In addition, if there are no direct key frames, the edge detection algorithm is used to calculate the similarity between frames. The edge detection algorithm is an algorithm used to detect image edges. Edges are areas with large differences in gray value between adjacent pixels in an image, and usually represent the boundaries of objects in the image. The basic idea is to calculate the grayscale difference between each pixel in the image and its adjacent pixels, and determine whether the pixel is an edge pixel based on the size and direction of the difference. By comparing the grayscale differences on image frames, the similarity between frames is calculated, and frames whose similarity is greater than a certain threshold are considered similar frames. A similar frame set composed of multiple similar frames can be represented by a key frame.

It should be noted that the basic idea of key frame extraction is to select frames that represent the entire video to retain the content of the video as much as possible. These representative frames are called key frames. Key frames can be selected based on the following aspects:

1. Object movement: If the video contains only one moving object, the frame containing the object should be selected as the key frame.

2. Scene changes: If there are scene changes in the video, such as buildings, roads, sky, etc., frames containing such scenes should be selected as key frames. Therefore, when the scene changes, select an appropriate frame as a key frame.

3. Semantic information: If the video contains a lot of text or logos, we can identify and select frames containing such elements as key frames.

In an implementable embodiment of the present disclosure, step S221, before obtaining the MPEG-4 standard of the video data, also includes: dividing the video data into frames to obtain individual video frames; preprocessing the video frames to obtain optimized video frames; The preprocessing includes noise reduction and/or size unification.

The video data can also be optimized before extracting key frames. After noise reduction or size unification, clearer and more regular data frames can be obtained.

In an implementable embodiment of the present disclosure, determining the category to which each group of key frames belongs is determined and stored as the first category in the video database, including determining the category to which each group of key frames belongs based on a picture classification algorithm and storing it as the third category. A category is stored in the video database.

Specifically, the image classification algorithm training set contains a large amount of image data and labels, covering multiple categories of images. The image data is what the algorithm needs to learn and extract information, and the labels indicate the category of each image. The algorithm will automatically learn from the picture set and continuously optimize its own parameters so that it can classify the input pictures. Common image classification algorithms include rule-based algorithms and statistics-based algorithms. Choose a way to classify pictures based on deep learning algorithms and clustering-based algorithms.

Step S300: Obtain picture data.

Specifically, when searching, the user inputs pictures into the program, and the computer proceeds to the next step after obtaining the picture data.

Step S400: Process the image data to generate a second category and a second feature vector of the image data.

Specifically, after processing the image, the image data is generated with the same characteristics as the key frame, so as to facilitate comparison with the key frame. Under normal circumstances, the user inputs an image for retrieval, so after processing the image data, the generated second category and second feature vector are both one data.

In an implementation manner of the present disclosure, for step S400, see Figure 5, which specifically also includes:

Step S410, the image classification algorithm determines the category of the image data and stores it as the second category;

Step S420: Calculate the image feature vector of the image data based on the pre-trained VGG16 model, and store it as a second feature vector.

Specifically, use the same processing method as the key frame to process the image and obtain similar features. As in the previous step, the key frame is processed by the image classification algorithm. In this step, the image data is subjected to the same image classification algorithm, and the image classification algorithm is used. The same rule processing in , such as key frames, uses a statistical-based image classification algorithm, and the same statistical-based algorithm is applied to the image data. The use of the VGG16 model is also the same as the model used when processing key frames, so I will not go into details here.

Step S500: Search for a first category that is the same as the second category among multiple first categories in the video database and confirm it as the query category.

Specifically, the category to which the selected picture data belongs is determined in advance, and the video data has been classified according to the category in advance. Therefore, the video is only queried in the same category as the picture data, which reduces the calculation amount during the retrieval process. , improve the response time of retrieval.

Step S600: Calculate the similarity between the second feature vector and multiple first feature vectors corresponding to the query category.

Specifically, by calculating the distance between the first eigenvector and the second eigenvector to measure the similarity between the two, it is actually to find the K vectors closest to the target vector in the high-dimensional space and use the Euclidean distance calculation. , the formula is as follows:

According to the second category and the second feature vector to which the image data to be queried have been obtained in the previous step, among all the second feature vectors matching the image data under the query category, poll the second feature vectors that match the image data. vector similarity.

Step S700: Sort the similarities and generate a video list corresponding to the similarities.

Specifically, the similarity result corresponds to a specific first feature vector, thereby corresponding to the key frame and its video data. After sorting the similarities in a certain order, the video data corresponding to the similarities will be in the same order, and finally these ordered video data will be returned and displayed in the form of a video list.

In an implementation manner of the present disclosure, after step S200 and before step S300; see Figure 2, a video database optimization method is also included, specifically:

Step S800: In the video database, obtain the single access volume of multiple video data within a set time.

Step S900: In response to the single number of visits being less than a threshold, delete the first category and first feature vector of the video data and its key frames corresponding to the single number of visits.

It should be noted that the video database in this disclosure is a database generated for the retrieval process, which stores the encoding of video data and the characteristics of its key frames. The video data corresponding to the encoding in the video database is the scope of retrieval during the retrieval process. ,The video database is not a repository of video data on ,the platform.

Specifically, the video database is regularly inspected, and the set time is the time interval of the scheduled inspection, and the individual visits of each video data are queried. The visits include clicks, page views, retrieved times, etc., when accessing When the amount is less than the threshold, the video is excluded from the retrieval range, and the encoding of the video data is deleted in the video database, so that the subsequent retrieval process does not have to calculate this video, which can reduce the amount of calculation in the retrieval process and reduce Consume computing resources and improve retrieval speed.

Taking the e-commerce platform as an example, by recording the number of product visits, for products that have no traffic for a long time or whose traffic is below the threshold, the corresponding video data and key frames are deleted from the video database, and the first feature corresponding to the key frame is also deleted. Vector and first category.

To sum up, the present disclosure provides a method for retrieving videos based on pictures. By processing the video data, the encoding of each video is retained, and the first category of key frames and the video stored in the first feature vector are generated. In the data; after the user inputs the image for retrieval, the image is processed to obtain the second category and the second feature vector of the image, and the search is performed within the same first category range as the second category to reduce the search scope. The calculation amount during the retrieval process is reduced, and the calculation consumption and time cost of model application are reduced to meet the requirements of scenarios that require high real-time retrieval.

Specific embodiments of the present disclosure provide another method for retrieving videos based on images. The specific execution steps are:

Step S101, obtain multiple video data.

Step S102, store each video data in the distributed file database, and encode each video data.

Step S103: Obtain key frames of each video data in multiple video data in sequence.

Step S104: Determine the category to which each group of key frames belongs, and store it in the video database as the first category.

Step S105: Based on the pre-trained VGG16 model, the picture feature vector of each group of key frames is calculated and stored in the video database as the first feature vector.

Step S106: Obtain picture data.

Step S107: Based on the image classification algorithm, determine the category of the image data and store it as the second category.

Step S108: Calculate the image feature vector of the image data based on the pre-trained VGG16 model, and store it as a second feature vector.

Step S109: Search for a first category that is the same as the second category among multiple first categories in the video database and confirm it as the query category.

Step S1010: Calculate the similarity between the second feature vector and multiple first feature vectors corresponding to the query category.

Step S1011, sort the similarities and generate a video list corresponding to the similarities.

It should be noted that the method of the embodiment of the present disclosure can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario, and is completed by multiple devices cooperating with each other. In this distributed scenario, one of the multiple devices can only perform one or more steps in the method of the disclosed embodiment, and the multiple devices will interact with each other to complete all the steps. method described.

It should be noted that some embodiments of the present disclosure have been described above. Other implementations are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the above-described embodiments and still achieve desired results. Additionally, the processes depicted in the figures do not necessarily require the specific order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain implementations.

Based on the same inventive concept, corresponding to any of the above implementation methods, the present disclosure also provides a system for retrieving videos based on pictures.

Referring to Figure 6, a system for retrieving videos based on images includes:

The first acquisition module 100 is used to acquire multiple video data;

The first processing module 200 is used to process a plurality of the video data, generate a first category and a first feature vector of the encoding of each video data and its key frames, and combine the generated multiple first categories and a plurality of first feature vectors stored in the video database;

The second acquisition module 300 is used to acquire image data;

The second processing module 400 is used to process the image data and generate a second category and a second feature vector of the image data;

The search module 500 is used to search the first category that is the same as the second category in the video database and confirm it as a query category;

The calculation module 600 is used to calculate the similarity between the second feature vector and a plurality of first feature vectors corresponding to the query category;

The generation module 700 is used to sort the similarities and generate a video list corresponding to the similarities.

In a feasible implementation, the system for retrieving videos based on images also includes:

The third acquisition module 800 is to acquire, in the video database, a single number of visits to a plurality of the video data within a set time.

The response module 900, in response to the single visit being less than a threshold, deletes the first category and first feature vector of the video data and its key frames corresponding to the single visit.

For the convenience of description, when describing the above device, the functions are divided into various modules and described separately. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and/or hardware.

The devices in the above embodiments are used to implement the corresponding image-based video retrieval method in any of the foregoing embodiments, and have the beneficial effects of the corresponding method implementations, which will not be described again here.

Based on the same inventive concept, corresponding to any of the above implementation methods, the present disclosure also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor When the program is executed, the method for retrieving videos based on pictures described in any of the above embodiments is implemented.

Figure 7 shows a more specific hardware structure diagram of a computer device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input/output interface 1030, a communication interface 1040 and a bus 1050. The processor 1010, the memory 1020, the input/output interface 1030 and the communication interface 1040 implement communication connections between each other within the device through the bus 1050.

The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit, central processing unit), a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute related tasks. program to implement the technical solution provided by the implementation of this specification.

The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store operating systems and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 1020 and called and executed by the processor 1010.

The input/output interface 1030 is used to connect the input/output module to realize information input and output. The input/output/module can be configured in the device as a component (not shown in the figure), or can be externally connected to the device to provide corresponding functions. Input devices can include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices can include monitors, speakers, vibrators, indicator lights, etc.

The communication interface 1040 is used to connect a communication module (not shown in the figure) to realize communication interaction between this device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

Bus 1050 includes a path that carries information between various components of the device (eg, processor 1010, memory 1020, input/output interface 1030, and communication interface 1040).

It should be noted that although the above device only shows the processor 1010, the memory 1020, the input/output interface 1030, the communication interface 1040 and the bus 1050, during specific implementation, the device may also include necessary components for normal operation. Other components. In addition, those skilled in the art can understand that the above-mentioned device may only include components necessary to implement the embodiments of this specification, and does not necessarily include all components shown in the drawings.

The computer equipment in the above embodiments is used to implement the corresponding image-based video retrieval method in any of the above embodiments, and has the beneficial effects of the corresponding method implementations, which will not be described again here.

Based on the same inventive concept, corresponding to any of the above implementation methods, the present disclosure also provides a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions use To enable the computer to execute the method for retrieving videos based on pictures as described in any of the above embodiments.

The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), and read-only memory. (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, Magnetic tape cassettes, tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium can be used to store information that can be accessed by a computing device.

The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the method for retrieving videos based on pictures as described in any of the above embodiments, and have the beneficial effects of the corresponding method implementations, which will not be described again here.

Those of ordinary skill in the art will understand that the above discussion of any embodiments is illustrative only and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples; this description is provided for clarity only. , those skilled in the art should take the description as a whole. Under the idea of the present disclosure, the technical features in the above embodiments or different embodiments can also be appropriately combined, and the steps can be implemented in any order, and there are as mentioned above There are many other variations in various aspects of the disclosed embodiments, which have not been provided in detail for the sake of clarity.

Additionally, to simplify illustration and discussion, and so as not to obscure embodiments of the present disclosure, well-known power supplies/power supplies with integrated circuit (IC) chips and other components may or may not be shown in the provided figures. Ground connection. Furthermore, devices may be shown in block diagram form in order to avoid obscuring the embodiments of the present disclosure, and this is also in view of the fact that the details regarding the implementation of these block diagram devices are highly dependent on the manner in which the disclosed embodiments are to be implemented. platform (i.e., these details should be well within the understanding of those skilled in the art). Where specific details (eg, circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that systems may be constructed without these specific details or with changes in these specific details. The embodiments of the present disclosure are implemented below. Accordingly, these descriptions should be considered illustrative rather than restrictive.

Although the present disclosure has been described in connection with specific embodiments thereof, many substitutions, modifications and variations of these embodiments will be apparent to those of ordinary skill in the art from the foregoing description. For example, other memory architectures such as dynamic RAM (DRAM) may use the discussed implementations.

The disclosed embodiments are intended to embrace all such alternatives, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and principles of the embodiments of the present disclosure shall be included in the protection scope of the present disclosure.

Claims (10)

1. A method of retrieving video based on pictures, the method comprising:

acquiring a plurality of video data;

processing a plurality of video data, generating a first category and a first feature vector of each video data code and key frames thereof, and storing the generated plurality of first categories and plurality of first feature vectors in a video database;

acquiring picture data;

processing the picture data to generate a second category and a second feature vector of the picture data;

searching a plurality of first categories in the video database, wherein the first categories are the same as the second categories, and confirming the first categories as query categories;

calculating the similarity between the second feature vector and a plurality of first feature vectors corresponding to the query category;

and sequencing the similarity to generate a video list corresponding to the similarity.

2. The method of claim 1, wherein after storing the generated first plurality of categories and first plurality of feature vectors in the video database, the method further comprises, prior to obtaining the picture data:

acquiring single access amounts of a plurality of video data in a set time in the video database;

and deleting the first category and the first feature vector of the video data and the key frames thereof corresponding to the single access amount in response to the single access amount being smaller than a threshold value.

3. The method of picture-based video retrieval according to claim 1, wherein said processing the plurality of video data to generate a first category and a first feature vector for each video data encoding and key frames thereof, and storing the generated plurality of first categories and plurality of first feature vectors in a video database comprises:

storing each video data in a distributed file database and encoding each video data;

sequentially acquiring key frames of each video data in a plurality of video data;

judging the category of each group of the key frames, and storing the category as the first category in the video database;

based on a pretrained VGG16 model, picture feature vectors of each group of key frames are calculated and stored in the video database as the first feature vectors.

4. A method of retrieving video based on pictures as defined in claim 3 wherein said sequentially capturing key frames of each of a plurality of said video data further comprises:

acquiring an MPEG-4 standard table of the video data, and judging whether the MPEG-4 standard table comprises a stss part or not;

in response to the standard table including the stss portion, determining that the frame identified by the stss portion is a key frame;

in response to the standard table not including the stss part, calculating the similarity between the video frames by using an edge detection calculation method, and generating a plurality of similar frames;

clustering a plurality of similar frames to generate a similar frame set;

and processing the similar frame sets by using a picture quality evaluation algorithm, and selecting the optimal frame in each similar frame set as a key frame.

5. A method of retrieving video based on pictures according to claim 3, wherein said determining the category to which each of said key frames belongs and storing as said first category in said video database comprises:

and judging the category of each group of key frames based on a picture classification algorithm, and storing the category as the first category in the video database.

6. The method for retrieving video based on pictures as recited in claim 4 wherein prior to said obtaining said MPEG-4 standard of video data, further comprising:

framing the video data to obtain an independent video frame;

preprocessing the video frame to obtain an optimized video frame; the preprocessing includes noise reduction and/or size unification.

7. The method of retrieving video based on pictures of claim 1, wherein said processing the picture data to generate a second category and a second feature vector of the picture data comprises:

judging the category of the picture data based on a picture classification algorithm, and storing the category as the second category;

and calculating a picture feature vector of the picture data based on a pretrained VGG16 model, and storing the picture feature vector as the second feature vector.

8. A system for retrieving video based on pictures, characterized in that,

the first acquisition module is used for acquiring a plurality of video data;

the first processing module is used for processing a plurality of video data, generating a first category and a first feature vector of each video data code and key frames thereof, and storing the generated plurality of first categories and the generated plurality of first feature vectors in a video database;

the second acquisition module is used for acquiring the picture data;

the second processing module is used for processing the picture data and generating a second category and a second feature vector of the picture data;

the searching module is used for searching the first category which is the same as the second category in the video database and confirming the first category as a query category;

the computing module is used for computing the similarity between the second feature vector and a plurality of first feature vectors corresponding to the query category;

and the generation module is used for sequencing the similarity and generating a video list corresponding to the similarity.

9. A computer device, comprising: a processor and a memory;

the memory stores a computer program which, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1 to 7.

10. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the picture-based video retrieval method of any one of claims 1 to 7.