JP2021006956A - Retrieval device, retrieval method and retrieval program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To search a scene with high accuracy. A feature amount extraction unit 70 extracts a feature amount related to an image or an acoustic signal from a query video for each of a plurality of query videos. The scene feature amount acquisition unit 72 acquires the scene feature amount of the target scene using the extracted feature amount for each of the plurality of query videos. The search unit 74 performs a search using each feature amount and scene feature amount of the plurality of query images, and the feature amount and scene feature amount of the accumulated video. [Selection diagram] Fig. 4

Description

Disclosure techniques relate to search devices, search methods, and search programs.

Finding a stored video that shows a query (specific place or object) given as a query video from a large amount of video (called a stored video) is called instance search. In the instance search so far, after extracting the frame image from each of the query video and the stored video, the feature of the image is extracted from the frame image, and the image features are compared with each other to check whether the query is reflected in the stored video. The search was mainly based on image information, that is, determination and search (see, for example, Non-Patent Document 1).

Y. Peng, J. Zhang, X. Huang, M. Sun, X. He, P. Tang, Y. Zhao, J. Zhao, J. Qi, J. Zhang, “PKU-ICST at TRECVID 2015: Instance Search Task ”, in Proceedings of TRECVID 2015, NIST, USA, 2015.

In the conventional method based on conventional image information, for example, when you want to search for a stored image showing a specific place, there is an obstacle (such as another person) in the foreground of the image pointing to the query (here, that place). At that time, there is a problem that the query is shielded as an image, sufficient image information cannot be extracted, collation fails, and the target stored image cannot be searched. Further, even if the same place is shown in the query video and the stored video, there is a problem that the search cannot be performed if the video shot by the query video and the stored video is significantly different due to the difference in the shooting position and shooting direction of the camera.

The disclosed technique has been made in view of the above points, and an object of the present invention is to provide a search device, a search method, and a search program capable of searching a scene with high accuracy.

The first aspect of the present disclosure is a search device, which searches for a target scene included in a plurality of query images from at least a stored image or a database in which a feature amount obtained from the stored image is stored. Therefore, for each of the plurality of query videos, a feature quantity extraction unit that extracts a feature quantity related to an image or an acoustic signal from the query video, and for each of the plurality of query videos, the extracted feature quantity is used. Using the scene feature amount acquisition unit for acquiring the scene feature amount of the target scene, the feature amount and the scene feature amount of each of the plurality of query images, and the feature amount and the scene feature amount of the accumulated image. The scene feature amount is a process of acquiring the feature amount of the acoustic signal corresponding to the image from the feature amount of the image, or the feature amount of the image is obtained from the feature amount of the acoustic signal. This is the intermediate output value of the process to be acquired.

The second aspect of the present disclosure is a search method, which is a search method for searching a target scene included in a plurality of query images from at least a stored image or a database in which a feature amount obtained from the stored image is stored. Therefore, the feature amount extraction unit extracts the feature amount related to the image or the acoustic signal from the query video for each of the plurality of query images, and the scene feature amount acquisition unit extracts the feature amount for each of the plurality of query images. The scene feature amount of the target scene is acquired using the extracted feature amount, and the search unit uses the feature amount and the scene feature amount of each of the plurality of query images, and the feature amount and the feature amount of the accumulated image. The scene feature amount includes a search using the scene feature amount, and the scene feature amount is a process of acquiring the feature amount of the acoustic signal corresponding to the image from the feature amount of the image, or an image feature from the feature amount of the acoustic signal. This is the intermediate output value of the process for acquiring the amount.

The third aspect of the present disclosure is a search program for searching a target scene included in a plurality of query images from at least a stored image or a database in which a feature amount obtained from the stored image is stored. In the program, for each of the plurality of query videos, a feature amount related to an image or an acoustic signal is extracted from the query video, and for each of the plurality of query videos, the target scene is used using the extracted feature amount. The scene feature amount is acquired, and the computer executes a search using the feature amount and the scene feature amount of each of the plurality of query images and the feature amount and the scene feature amount of the accumulated image. The scene feature amount is a search program for obtaining the feature amount of the sound signal corresponding to the image from the feature amount of the image, or a process of acquiring the feature amount of the image from the feature amount of the acoustic signal. It is an intermediate output value of.

According to the disclosed technology, the scene can be searched with high accuracy.

It is a schematic block diagram of an example of a computer functioning as a learning device and a search device of the first embodiment and the second embodiment. It is a block diagram which shows the structure of the learning apparatus of 1st Embodiment and 2nd Embodiment. It is a figure which shows an example of the structure of a neural network. It is a block diagram which shows the structure of the search apparatus of 1st Embodiment and 2nd Embodiment. It is a figure which shows an example of the structure of the feature amount extraction model. It is a block diagram which shows the structure of the search part of the search apparatus of 1st Embodiment. It is a flowchart which shows the learning processing routine of the learning apparatus of 1st Embodiment and 2nd Embodiment. It is a flowchart which shows the search processing routine of the search apparatus of 1st Embodiment. It is a block diagram which shows the structure of the search part of the search apparatus of 2nd Embodiment. It is a flowchart which shows the search processing routine of the search apparatus of 2nd Embodiment. It is a figure which shows the experimental result.

Hereinafter, an example of the embodiment of the disclosed technique will be described with reference to the drawings. The same reference numerals are given to the same or equivalent components and parts in each drawing. In addition, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may differ from the actual ratios.

<Outline of this embodiment>
In the present embodiment, the sound information of the video is also used in the instance search, the image information and the sound information are embedded in one space (called an embedding space), and by using this information, the sound information and the video information can be obtained. Analyze and search the scene based on the information of the scene itself behind it. Specifically, first, before the search, image information (VGG features, etc.) and sound information (mfcc, etc.) of the video are extracted using a large number of videos, and sound information is output from this image information. Learn neural networks by supervised learning. At the time of search, this neural network is used, and image information of query video and stored video is input to this neural network. Then, the output of the hidden layer of the neural network at this time is collated as a scene feature amount to perform a search. It can be said that this hidden layer corresponds to the above-mentioned embedding space, and this neural network maps the input video information to the embedding space, and maps the mapping result to the embedding space to the sound information.

In the video search experiment using this method, a collation method using only the image feature amount of the video (a method of collating using only the VGG feature) and a sound feature amount of the video (a method of collating using only the mfcc feature of the video). ) Is also searching for scenes that could not be searched.

Then, in the method of the present embodiment, a search using this scene feature amount and a search using the image feature amount are performed, and the result of integrating the search scores for each feature amount is used as the final search score. The information of each modal is complemented and searched.

Further, in the instance search, a plurality of query videos for searching a certain instance may be given. In this case, one modal may be effective for the search in one query video, and another modal may be effective for the search in another query video. For example, one query video contains sounds that are valid for search, while another query video is silent and only image information is valid. Therefore, in the method of the present embodiment, a search is performed using a plurality of feature quantities including a scene feature quantity obtained from each of the plurality of query videos, the search scores are integrated for each feature quantity, and a score based on each feature quantity is further obtained. By integrating the above, a search using multiple modals of multiple query videos is performed.

[First Embodiment]
<Structure of the learning device according to the first embodiment>
FIG. 1 is a block diagram showing a hardware configuration of the learning device 10 of the present embodiment.

As shown in FIG. 1, the learning device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (a communication interface (Read) Memory) 12. It has an I / F) 17. Each configuration is communicably connected to each other via a bus 19.

The CPU 11 is a central arithmetic processing unit that executes various programs and controls each part. That is, the CPU 11 reads the program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a work area. The CPU 11 controls each of the above configurations and performs various arithmetic processes according to the program stored in the ROM 12 or the storage 14. In the present embodiment, the ROM 12 or the storage 14 stores a learning program for learning the neural network. The learning program may be one program, or may be a group of programs composed of a plurality of programs or modules.

The ROM 12 stores various programs and various data. The RAM 13 temporarily stores a program or data as a work area. The storage 14 is composed of an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used for performing various inputs.

The display unit 16 is, for example, a liquid crystal display and displays various types of information. The display unit 16 may adopt a touch panel method and function as an input unit 15.

The communication interface 17 is an interface for communicating with other devices, and for example, standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark) are used.

Next, the functional configuration of the learning device 10 will be described. FIG. 2 is a block diagram showing an example of the functional configuration of the learning device 10.

Functionally, the learning device 10 includes an input unit 15, a calculation unit 30, and an output unit 32, as shown in FIG.

The input unit 15 receives the input of the learning video.

The calculation unit 30 can be represented by a configuration including a learning feature amount extraction unit 20 and a model learning unit 22.

The learning feature amount extraction unit 20 extracts a pair of the image feature amount of the frame image and the feature amount of the acoustic signal from the learning video received by the input unit 15 for each frame.

For example, VGG features are extracted as image features, and mfcc features are extracted as acoustic signal features.

The model learning unit 22 learns a model that receives an image feature amount as an input and acquires a feature amount of a corresponding acoustic signal as shown in FIG.

For example, a neural network that takes a VGG feature as an input and outputs a corresponding mfcc feature is learned. This neural network is a neural network composed of three or more layers including an input layer, an intermediate layer, and an output layer.

In the present embodiment, with respect to the pair of the image feature amount extracted by the learning feature amount extraction unit 20 and the feature amount of the acoustic signal, when the image feature amount is input to the neural network, the features of the paired acoustic signal Learn the neural network parameters so that the quantity is output.

The model learning unit 22 outputs a model for obtaining the output of the intermediate layer obtained by processing from the input layer to the intermediate layer of the learned neural network as a feature amount extraction model by the output unit 32.

<Structure of the search device according to the first embodiment>
FIG. 1 is a block diagram showing a hardware configuration of the search device 50 of the present embodiment.

As shown in FIG. 1, the search device 50 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, and an input unit 15, similarly to the learning device 10. , Display unit 16 and communication interface (I / F) 17. In the present embodiment, the ROM 12 or the storage 14 stores a search program for searching a video.

Next, the functional configuration of the search device 50 will be described. FIG. 4 is a block diagram showing an example of the functional configuration of the search device 50.

Functionally, the search device 50 includes an input unit 15, a calculation unit 54, and an output unit 56, as shown in FIG.

The input unit 15 accepts inputs of a plurality of query videos. The query video includes a target scene (for example, a specific object or place) represented by an image component.

The calculation unit 54 includes a stored video database 60, a feature amount extraction unit 62, a feature amount extraction model storage unit 64, a scene feature amount acquisition unit 66, a scene feature amount database 68, a feature amount extraction unit 70, and a scene. It can be represented by a configuration including a feature amount acquisition unit 72 and a search unit 74.

The stored video database 60 stores a plurality of stored videos.

The feature amount extraction unit 62 extracts an image feature amount for each frame of each of the plurality of stored images stored in the stored image database 60.

The feature amount extraction model storage unit 64 stores the feature amount extraction model output by the learning device 10.

The scene feature amount acquisition unit 66 performs a process of acquiring the feature amount of the corresponding acoustic signal by inputting the average of the image feature amounts extracted by the feature amount extraction unit 62 for each frame for each of the plurality of stored images. The intermediate output value in the model is acquired as a scene feature. Specifically, the image feature amount is input to the input layer of the feature amount extraction model, and the output of the intermediate layer is acquired as the scene feature amount (see FIG. 5).

In the scene feature amount database 68, the acquired image feature amount and the scene feature amount are stored for each of the plurality of stored images.

The feature amount extraction unit 70 extracts an image feature amount for each frame of each of the plurality of query images.

The scene feature amount acquisition unit 72 acquires the scene feature amount by inputting the average of the image feature amounts extracted by the feature amount extraction unit 70 for each frame into the feature amount extraction model for each of the plurality of query images. Specifically, the image feature amount is input to the input layer of the feature amount extraction model, and the output of the intermediate layer is acquired as the scene feature amount.

The search unit 74 refers to the scene feature amount database 68, and for each of the plurality of stored images, the image feature amount and the scene feature amount of each of the plurality of query images, and the feature amount and the scene feature amount of the stored video. The search score is calculated using the above, and the accumulated video having the highest search score of N is output by the output unit 56 as a search result.

Specifically, as shown in FIG. 6, the search unit 74 includes a collation unit 80, an intra-modal score integration unit 82, and an inter-modal score integration unit 84.

The collation unit 80 calculates a feature amount search score in which the image feature amount of the query video and the image feature amount of the stored video are collated for each of the plurality of query videos for each stored video. For example, the cosine similarity between image features or the Z score obtained from the cosine similarity is calculated as the feature search score. Further, the collation unit 80 calculates a scene feature amount search score in which the scene feature amount of the query video and the scene feature amount of the stored video are collated for each of the plurality of query videos for each stored video. For example, the cosine similarity between scene features or the Z score obtained from the cosine similarity is calculated as the scene feature search score.

The in-modal score integration unit 82 integrates the feature amount search scores calculated for each of the plurality of query videos for each stored video, and also integrates the scene feature amount search scores calculated for each of the plurality of query videos. .. For example, the average value, the maximum value, or the weighted sum of the feature quantity search scores calculated for each of the plurality of query videos is used as the integration result of the feature quantity search scores. Further, the average value, the maximum value, or the weighted sum of the scene feature amount search scores calculated for each of the plurality of query videos is used as the integrated result of the feature amount search scores.

The inter-modal score integration unit 84 integrates the feature amount search score and the scene feature amount search score for each accumulated video to obtain a search score, and outputs the accumulated video having the highest N search scores as a search result. Is output by. For example, the average value, the maximum value, or the weighted sum of the feature amount search score and the scene feature amount search score is used as the search score.

In the above description, the case where the average of the image features extracted for each frame is input to the feature extraction model to acquire the scene features has been described as an example, but the present invention is not limited to this, and the frame is not limited to this. For each case, the extracted image features may be input to the feature extraction model to acquire the scene features. In this case, the scene feature amount acquisition unit 72 acquires the scene feature amount by inputting the image feature amount extracted by the feature amount extraction unit 70 into the feature amount extraction model for each frame of the query video. Further, the search unit 74 refers to the scene feature amount database 68, and for each of the plurality of stored images, the image feature amount and the scene feature amount for each frame of the stored video, and the image feature amount for each frame of the query video. The search score is calculated by collating with the scene feature amount in chronological order, and the accumulated video having the highest search score is output by the output unit 56 as the search result.

<Operation of the learning device according to the first embodiment>
Next, the operation of the learning device 10 will be described. FIG. 7 is a flowchart showing the flow of the learning process by the learning device 10. The learning process is performed by the CPU 11 reading the learning program from the ROM 12 or the storage 14, expanding it into the RAM 13 and executing it. Further, a plurality of learning images are input to the learning device 10.

In step S100, the CPU 11, as the learning feature amount extracting unit 20, extracts a pair of the image feature amount of the frame image and the feature amount of the acoustic signal from each learning video for each frame.

In step S102, the CPU 11 serves as the model learning unit 22, and when the image feature amount is input to the neural network for the pair of the image feature amount extracted in step S100 and the feature amount of the acoustic signal, the paired sound A model that learns the parameters of the neural network so that the feature amount of the signal is output and obtains the output of the intermediate layer by processing from the input layer to the intermediate layer of the learned neural network is used as a feature amount extraction model. It is output by the output unit 32.

<Operation of the search device according to the first embodiment>
Next, the operation of the search device 50 will be described.

First, when the feature amount extraction model is input to the search device 50, it is stored in the feature amount extraction model storage unit 64.

Further, when a plurality of stored videos are input to the search device 50, they are stored in the stored video database 60. Then, the feature amount extraction unit 62 extracts an image feature amount for each frame for each of the plurality of stored images stored in the stored image database 60, and the scene feature amount acquisition unit 66 extracts the image feature amount of the plurality of stored images. For each frame, the average of the image features extracted by the feature extraction unit 62 is input to the feature extraction model to acquire the scene features. Then, the acquired scene feature amount is stored in the scene feature amount database 68 for each of the plurality of stored images.

FIG. 8 is a flowchart showing the flow of the search process by the search device 50. The search process is performed by the CPU 11 reading the search program from the ROM 12 or the storage 14, expanding it into the RAM 13 and executing the search program. Further, a plurality of query videos including the target scene are input to the search device 50.

In step S110, the CPU 11 extracts the image feature amount for each frame of each of the plurality of query videos as the feature amount extraction unit 70.

In step S112, the CPU 11 inputs, as the scene feature amount acquisition unit 72, the average of the image feature amounts extracted by the feature amount extraction unit 70 for each frame into the feature amount extraction model for each of the plurality of query images. Acquire the scene feature amount.

In step S114, the CPU 11, as the collation unit 80, calculates a feature amount search score for each of the plurality of query videos by collating the image feature amount of the query video with the image feature amount of the stored video for each stored video. To do.

Further, the CPU 11 calculates, as the collation unit 80, a scene feature amount search score in which the scene feature amount of the query video and the scene feature amount of the stored video are collated for each of the plurality of query videos for each stored video. ..

In step S116, the CPU 11 integrates the feature amount search scores calculated for each of the plurality of query videos for each stored video as the in-modal score integration unit 82, and the scene calculated for each of the plurality of query videos. Integrate feature search scores.

In step S118, the CPU 11 integrates the feature amount search score and the scene feature amount search score for each stored video to obtain the search score as the inter-modal score integration unit 84.

In step S120, the CPU 11 creates a search result representing the accumulated video having the highest N search scores, outputs the search result by the output unit 56, and ends the search processing routine.

[Second Embodiment]
Next, the learning device and the search device according to the second embodiment will be described. Since the learning device according to the second embodiment has the same configuration as that of the first embodiment, the same reference numerals are given and the description thereof will be omitted. Further, in the search device according to the second embodiment, the parts having the same configuration as that of the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

<Structure of the search device of the second embodiment>
As shown in FIG. 1, the hardware configuration of the search device 250 of the present embodiment is the same as that of the search device 50 of the first embodiment.

Next, the functional configuration of the search device 250 will be described.

As shown in FIG. 4, the calculation unit 54 of the search device 250 includes a storage video database 60, a feature amount extraction unit 62, a feature amount extraction model storage unit 64, a scene feature amount acquisition unit 66, and a scene feature amount. It can be represented by a configuration including a database 68, a feature amount extraction unit 70, a scene feature amount acquisition unit 72, and a search unit 274.

As shown in FIG. 9, the search unit 274 includes an average feature amount calculation unit 280, a collation unit 282, and an inter-modal score integration unit 284.

The average feature amount calculation unit 280 calculates the feature amount obtained by averaging the image feature amounts as an integrated feature amount in which the image feature amounts of each of the plurality of query videos are integrated. For example, the average vector of each image feature of a plurality of query videos is used as the integrated feature.

The average feature amount calculation unit 280 calculates the feature amount obtained by averaging the scene feature amount as an integrated scene feature amount in which the scene feature amounts of each of the plurality of query images are integrated. For example, the average vector of the scene features of each of the plurality of query videos is used as the integrated scene features.

The collation unit 282 calculates a feature amount search score that collates the integrated feature amount with the image feature amount of the stored image for each stored image. Further, the collation unit 282 calculates a scene feature amount search score that collates the integrated scene feature amount with the scene feature amount of the stored image for each stored image.

Similar to the inter-modal score integration unit 84, the inter-modal score integration unit 284 integrates the feature amount search score and the scene feature amount search score for each stored video to obtain a search score, and the search score is the top N N. The stored video is output by the output unit 56 as a search result.

<Operation of the learning device according to the second embodiment>
Since the operation of the learning device 10 according to the second embodiment is the same as that of the first embodiment, the description thereof will be omitted.

<Operation of the search device according to the second embodiment>
Next, the operation of the search device 250 will be described.

FIG. 10 is a flowchart showing the flow of the search process by the search device 250. The search process is performed by the CPU 11 reading the search program from the ROM 12 or the storage 14, expanding it into the RAM 13 and executing the search program. In addition, a plurality of query videos including the target scene are input to the search device 250.

In step S110, the CPU 11 extracts the image feature amount for each frame of each of the plurality of query videos as the feature amount extraction unit 70.

In step S112, the CPU 11 inputs, as the scene feature amount acquisition unit 72, the average of the image feature amounts extracted by the feature amount extraction unit 70 for each frame into the feature amount extraction model for each of the plurality of query videos. Acquire the scene feature amount.

In step S200, the CPU 11 calculates the integrated feature amount by averaging the image feature amounts of each of the plurality of query videos as the average feature amount calculation unit 280. The CPU 11 calculates the integrated scene feature amount by averaging the scene feature amounts of each of the plurality of query images as the average feature amount calculation unit 280.

In step S202, the CPU 11 calculates, as the collation unit 282, a feature amount search score in which the integrated feature amount and the image feature amount of the stored image are collated for each stored image. Further, the CPU 11 uses the collation unit 282 to calculate a scene feature amount search score for collating the integrated scene feature amount with the scene feature amount of the stored image for each stored image.

In step S204, as the inter-modal score integration unit 284, the CPU 11 integrates the feature amount search score and the scene feature amount search score for each stored video to obtain a search score, similarly to the inter-modal score integration unit 84.

In step S120, the CPU 11 creates a search result representing the accumulated video having the highest N search scores, outputs the search result by the output unit 32, and ends the search processing routine.

<Experimental example>
The results of an experiment conducted to confirm the effectiveness of the method described in the above embodiment will be described. Here, the VGG feature is used as the image feature amount, and the sound feature of the 8-layer model of soundnet [Non-Patent Document 1], the sound feature of the 5th layer model [Reference 1], and the mfcc feature are used as the feature amount of the acoustic signal. We learned a neural network that converts VGG features into the sound features of the sound feature of the 8-layer model of sound, the sound feature of the 5th layer model, and the mfcc feature.

FIG. 11 shows the results of an experiment in which an image including a specific place is searched from 100,000 stored images. In this experiment, multiple video queries are given to find a place. The horizontal axis of the graph is the search accuracy (MAP value). vgg represents a comparative example of performing a search using only VGG features. In addition, im2s_soundnet8, im2s_soundnet5, im2s_vgish, and im2s_mfcc represent scene features that convert VGG features into sound features, and im2s_soundnet8, im2s_soundnet5, im2s_soundnet5, im2s_soundnet5, im2s_soundnet5, im2s_sondnet5, im2s_sondnet5 The scene feature amount when converted to the sound feature and mfcc feature of the fifth layer model is shown. Then, im2s_soundnet8 represents a comparative example in which a search is performed using only the scene features of im2s_soundnet8. In addition, vgg and im2s_mfcc is a combination of search results by VGG features and search results by scene features by im2s_mfcc, and vgg and im2s_vgish is a combination of search results by VGG features and search results by scene features by im2s_vggish. im2s_soundnet5 is a combination of search results by VGG features and search results by scene features by im2s_soundnet5, and vgg and im2s_soundnet8 is a combination of search results by VGG features and search results by scene features by im2s_soundnet8. The integration of the search score by the VGG feature and the search score by the scene feature amount was performed by calculating the weighted sum of each score. In addition, mean zscore uses the search score for each feature as the average value of the search scores (Z score calculated from the cosine similarity) of a plurality of query videos, and mean feature uses each feature as a plurality of query videos. The average of the features from the above, max cosine similarity is the search score for each feature as the maximum value of the search score (cosine similarity) by multiple query videos, and max zscore is for each feature. The search score is the maximum value of the search score (Z score calculated from the cosine similarity) of a plurality of query videos. Compared with the case of searching only by VGG features (vgg in FIG. 11) and the case of searching only by scene features (im2s_soundnet8 in FIG. 11) from the experimental results, by integrating the search results by VGG features and scene features. It can be seen that the search accuracy is improved.

[Reference 1] Y. Aytar, et al., “SoundNet: Learning Sound Representations from Unlabeled Video”, NIPS 2016.
[Reference 2] S. Hershey, et al., “CNN ARCHITECTURES FOR LARGE-SCALE AUDIO CLASSIFICATION”, ICASSP 2017.

As described above, according to the search device according to the present embodiment, the feature amount of the image is extracted for each of the plurality of query images, and the feature amount of the corresponding acoustic signal is acquired from the feature amount of the extracted image. By acquiring the intermediate output value of the processing to be performed as a scene feature amount and performing a search using the image feature amount and the scene feature amount of each of the plurality of query images and the image feature amount and the scene feature amount of the accumulated video. Even if there is a shield or the shooting conditions such as the shooting position and shooting direction are different, it is possible to search the stored video including the scene of the query video with high accuracy.

<Modification example>
The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

For example, the feature amount of the acoustic signal may be extracted from the video, and the scene feature amount may be acquired from the feature amount of the acoustic signal by using the feature amount extraction model. In this case, the model learning unit 22 learns a neural network that takes the feature amount of the acoustic signal as an input and converts it into the corresponding image feature amount, and processes the learned neural network from the input layer to the intermediate layer. A model for obtaining the output of the intermediate layer is used as a feature extraction model. In addition, the feature amount extraction unit 62 extracts the feature amount of the acoustic signal for each frame of each of the plurality of stored images stored in the stored video database 60. The scene feature amount acquisition unit 66 acquires the scene feature amount by inputting the feature amount of the acoustic signal extracted by the feature amount extraction unit 62 into the feature amount extraction model for each of the plurality of stored images. The feature amount extraction unit 70 extracts the feature amount of the acoustic signal for each frame of the query video including the specific object represented by the acoustic component. The scene feature amount acquisition unit 72 inputs the feature amount of the acoustic signal extracted by the feature amount extraction unit 70 to the input layer of the feature amount extraction model for the query video, and uses the output of the intermediate layer as the scene feature amount. get. The search unit 74 searches for each of the plurality of stored images by using the feature amount and the scene feature amount of each acoustic signal of the plurality of query images and the feature amount and the scene feature amount of the acoustic signal of the stored image. The score is calculated, and the accumulated video having the highest search score is output by the output unit 56 as a search result.

As a result, the intermediate output value of the process of extracting the feature amount of the acoustic signal of a plurality of query images and acquiring the feature amount of the corresponding image from the feature amount of the extracted acoustic signal is acquired as the scene feature amount, and a plurality of By performing a search using the feature amount and the scene feature amount of each acoustic signal of the query image and the feature amount and the scene feature amount of the acoustic signal of the accumulated image, the accumulation including the scenes of a plurality of query images can be performed with high accuracy. You can search the video.

Further, for each of the plurality of stored images, the search unit 74 includes the image feature amount, the acoustic signal feature amount, and the scene feature amount of each of the plurality of query images, and the image feature amount and the acoustic signal feature of the stored video. The search score may be calculated using the amount and the scene feature amount, and the accumulated video having the top N search scores may be output by the output unit 56 as the search result. In this case, as in the first embodiment, for each of the plurality of query videos, an image feature search score was calculated by collating the image feature of the query video with the image feature of the stored video. Integrate image feature search scores. In addition, for each of the plurality of query videos, a sound feature search score that collates the feature amount of the acoustic signal of the query video and the feature amount of the acoustic signal of the stored video is calculated, and the calculated sound feature search score is integrated. .. Further, for each of the plurality of query videos, a scene feature search score that collates the scene feature amount of the query video and the scene feature amount of the accumulated video is calculated, and the calculated scene feature search score is integrated. Then, the search may be performed using the integrated image feature amount search score, sound feature amount search score, and scene feature amount search score.

Alternatively, as in the second embodiment, the image integrated feature amount that integrates the image feature amount of each of the plurality of query images, the sound integrated feature amount that integrates the feature amount of each acoustic signal of the plurality of query images, and the sound integrated feature amount. The search may be performed using the integrated scene feature amount that integrates the scene feature amount of each of the plurality of query images, the image feature amount of the accumulated video, the acoustic signal feature amount, and the scene feature amount.

Further, the learning device and the search device may be configured as one device. Further, although described as an embodiment in which the program is pre-installed in the specification of the present application, it is also possible to provide the program by storing it in a computer-readable recording medium.

Further, various processors other than the CPU may execute various processes executed by the CPU reading software (program) in each of the above embodiments. In this case, the processor includes a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), and the like. An example is a dedicated electric circuit or the like, which is a processor having a circuit configuration designed exclusively for it. Further, the word embedding vector integration process may be executed by one of these various processors, or a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs, and a CPU and an FPGA). It may be executed by a combination of). Further, the hardware structure of these various processors is, more specifically, an electric circuit in which circuit elements such as semiconductor elements are combined.

Further, in each of the above embodiments, the mode in which the learning program and the search program are stored (installed) in the storage 14 in advance has been described, but the present invention is not limited to this. The program is stored in a non-temporary medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versaille Disk Online Memory), and a USB (Universal Serial Bus) memory. It may be provided in the form. Further, the program may be downloaded from an external device via a network.

The following additional notes will be further disclosed with respect to the above embodiments.

(Appendix 1)
A search device that searches for target scenes included in a plurality of query videos from a database in which at least the stored video or the feature amount obtained from the stored video is stored.
With memory
With at least one processor connected to the memory
Including
The processor
For each of the plurality of query videos, a feature amount related to an image or an acoustic signal is extracted from the query video.
For each of the plurality of query videos, the scene feature amount of the target scene is acquired by using the extracted feature amount.
A search is performed using the feature amount and the scene feature amount of each of the plurality of query images, and the feature amount and the scene feature amount of the accumulated video.
Is configured as
The scene feature amount is
A search device that is an intermediate output value of a process of acquiring a feature amount of an acoustic signal corresponding to an image from a feature amount of an image, or a process of acquiring a feature amount of an image from a feature amount of an acoustic signal.

(Appendix 2)
A program that stores a program that can be executed by a computer to execute a search process for searching a target scene included in a plurality of query videos from at least a stored video or a database in which a feature amount obtained from the stored video is stored. It is a temporary storage medium
The search process is
For each of the plurality of query videos, a feature amount related to an image or an acoustic signal is extracted from the query video.
For each of the plurality of query videos, the scene feature amount of the target scene is acquired by using the extracted feature amount.
A search is performed using the feature amount and the scene feature amount of each of the plurality of query images, and the feature amount and the scene feature amount of the accumulated video.
The scene feature amount is
A non-temporary storage medium that is an intermediate output value of a process of acquiring a feature amount of an acoustic signal corresponding to an image from a feature amount of an image or a process of acquiring a feature amount of an image from a feature amount of an acoustic signal.

10 Learning device 15 Input unit 20 Learning feature amount extraction unit 22 Model learning unit 30, 54 Calculation unit 32, 56 Output unit 50, 250 Search device 60 Accumulated video database 62, 70 Feature amount extraction unit 64 Feature amount extraction model storage unit 66, 72 Scene feature acquisition unit 68 Scene feature database 74, 274 Search unit 80, 282 Matching unit 82 In-modal score integration unit 84, 284 Inter-modal score integration unit 280 Average feature calculation unit

Claims (6)

A search device that searches for target scenes included in a plurality of query videos from a database in which at least the stored video or the feature amount obtained from the stored video is stored.
For each of the plurality of query videos, a feature quantity extraction unit that extracts a feature quantity related to an image or an acoustic signal from the query video, and a feature quantity extraction unit.
For each of the plurality of query videos, a scene feature amount acquisition unit for acquiring the scene feature amount of the target scene using the extracted feature amount, and a scene feature amount acquisition unit.
It has a search unit that searches using the feature amount and the scene feature amount of each of the plurality of query images, and the feature amount and the scene feature amount of the accumulated video.
The scene feature amount is
A search device that is an intermediate output value of a process of acquiring a feature amount of an acoustic signal corresponding to an image from an image feature amount or a process of acquiring an image feature amount from an acoustic signal feature amount. The search unit calculates, for each of the plurality of query videos, a feature amount search score in which the feature amount of the query video and the feature amount of the accumulated video are collated, and integrates the calculated feature amount search score. And
For each of the plurality of query videos, a scene feature search score that collates the scene feature amount of the query video with the scene feature amount of the stored video is calculated, and the calculated scene feature search score is integrated. ,
The search device according to claim 1, wherein a search is performed using the integrated feature amount search score and the scene feature amount search score. The search unit includes an integrated feature amount that integrates the feature amounts of each of the plurality of query images, an integrated scene feature amount that integrates the scene feature amounts of each of the plurality of query images, and the accumulated video. The search device according to claim 1, wherein a search is performed using the feature amount and the scene feature amount. The scene feature amount is the output value of the hidden layer of the neural network that acquires the feature amount of the acoustic signal corresponding to the image from the feature amount of the image, or the feature amount of the neural network that acquires the feature amount of the image from the feature amount of the acoustic signal. The search device according to any one of claims 1 to 3, which is an output value of the hidden layer. It is a search method for searching a target scene included in a plurality of query videos from at least a stored video or a database in which a feature amount obtained from the stored video is stored.
The feature quantity extraction unit extracts the feature quantity related to the image or the acoustic signal from the query video for each of the plurality of query videos.
The scene feature amount acquisition unit acquires the scene feature amount of the target scene by using the extracted feature amount for each of the plurality of query images.
The search unit includes performing a search using the feature amount and the scene feature amount of each of the plurality of query images and the feature amount and the scene feature amount of the accumulated video.
The scene feature amount is
A search method that is an intermediate output value of the process of acquiring the feature amount of the acoustic signal corresponding to the image from the feature amount of the image or the process of acquiring the feature amount of the image from the feature amount of the acoustic signal. It is a search program for searching a target scene included in a plurality of query videos from at least a stored video or a database in which a feature amount obtained from the stored video is stored.
For each of the plurality of query videos, a feature amount related to an image or an acoustic signal is extracted from the query video.
For each of the plurality of query videos, the scene feature amount of the target scene is acquired by using the extracted feature amount.
It is a search program for causing a computer to perform a search using the feature amount and the scene feature amount of each of the plurality of query images and the feature amount and the scene feature amount of the accumulated video. ,
The scene feature amount is
A search program that is an intermediate output value of the process of acquiring the feature amount of the acoustic signal corresponding to the image from the feature amount of the image or the process of acquiring the feature amount of the image from the feature amount of the acoustic signal.