JP2016009430A - Video classifier learning device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To highly accurately learn a discriminator for discriminating whether or not an image corresponds to a detection object such as a body of matter and a scene.SOLUTION: An explanation voice extraction unit 21 of an image discriminator learning device 1 compares a first voice signal as a voice signal of a program voice, with a second voice signal as a voice signal of the program voice to which the explanation voice is added, and extracts an explanation voice signal as the voice signal of the explanation voice. A voice recognition unit 24 recognizes the voice of the explanation voice signal extracted by the explanation voice extraction unit 21. A discrimination learning processing unit 3 extracts a word that serves as a label from a result of the voice recognition by the voice recognition unit 24, and learns a discriminator 34 for detecting whether or not image data relate to the extracted word, from a feature amount of the image data, using the extracted word and the feature amount extracted from the program image data in an image section corresponding to an utterance time of the word.

Description

  The present invention relates to a video classifier learning device and a program.

  In the object recognition technology for recognizing what kind of object appears in the video or what kind of scene the video is, a classifier for recognition is learned in advance. When performing object recognition, the discriminator receives the feature extracted from the video and determines whether or not the object to be detected is reflected in the video or whether the video is a scene to be detected. Then, the discrimination result is output as a binary value. For learning of the discriminator, a large amount of video data for learning to which a label representing the object or scene being shown is attached is used. As a method of assigning a label to video data for learning, there are a method of assigning manually (for example, refer to Non-Patent Document 1) and a method of using closed captions for broadcasting (for example, refer to Non-Patent Document 2).

P. Duygulu, K. Barnard, JFG de Freitas, DA Forsyth1, "Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary", ECCV '02 Proceedings of the 7th European Conference on Computer Vision-Part IV, 2002 , P. 97-112 Yoshihiko Kawai, Masato Fujii, “Video Retrieval System Based on Iterative Learning Considering Closed Captions and Image Features”, Video Information Media Society, Video Information Media Society Annual Conference Proceedings, 2012, 23-7 -1- "23-7-2"

  As described above, learning of the discriminator for recognizing a detection target such as an object reflected in a video or a scene of video requires video data for learning with a label corresponding to the detection target. .

  By the way, commentary broadcasting in a broadcast program is a service that explains the video of a program with commentary audio mainly for the visually impaired, apart from narration and dialogue of the broadcast program. In commentary broadcasting, the movement of a person is depicted, the location and time are conveyed, and the characters in the video are read. On the other hand, subtitle broadcasting is a service that converts TV audio into subtitles for people with hearing impairments and elderly people who are difficult to hear TV audio. The closed caption broadcasted by this service is “closed caption”.

  FIG. 5 is a diagram illustrating an example in which closed captions of a certain drama program and transcripts of commentary audio are arranged in time series. The start time and end time indicate the start time and end time when the content of the transcript is displayed on the closed caption caption, or the start time and end time when the content of the transcript is spoken in the commentary voice. The caption / explanation indicates the type of whether the content of the transcription is obtained from closed caption subtitles or commentary audio. When comparing the transcripts of <Subtitles> and <Description>, information that can be heard and recognized by the ear, such as the characters of the characters, is included in <Subtitles> but not in <Description>. . On the other hand, information that can be visually recognized such as “staff room” and “smile” is included in <Explanation> but not in <Subtitle>.

  As described above, closed captions often do not include an object appearing in a video or a word representing a video scene. Therefore, in order to reduce the burden of manual labeling, if video data is labeled using closed captions for broadcasting, sufficient video data for learning cannot be prepared, and high accuracy is achieved. There was a possibility that the classifier could not be learned.

  The present invention has been made in view of such circumstances, and is capable of accurately learning a discriminator for identifying whether or not the video corresponds to a detection target such as an object or a scene. A learning device and program are provided.

One aspect of the present invention compares the first audio signal, which is the audio signal of the program audio, with the second audio signal, which is the audio signal of the program audio, to which the explanation audio is added. A commentary speech extraction unit that extracts a commentary speech signal, a speech recognition unit that recognizes the commentary speech signal extracted by the commentary speech extraction unit, and a word that is a label is extracted from the result of speech recognition by the speech recognition unit Then, using the extracted word and the feature amount extracted from the program video data of the video section corresponding to the utterance time of the word, whether or not the video data is related to the extracted word is determined. An image discriminator learning device comprising: a discriminator learning processing unit that learns a discriminator for detection from a feature amount.
According to the present invention, the video discriminator learning apparatus recognizes the comment audio signal extracted by comparing the audio signal of the program audio and the audio signal of the program audio to which the comment audio is added. The video discriminator learning device extracts a word as a label from the result of speech recognition, and uses the extracted word and a feature amount extracted from program video data of a video section corresponding to the utterance time of the word. To learn.
As a result, the video discriminator learning device can label the video data with high accuracy and use it for learning of the discriminator without manpower, so that it can learn a discriminator with higher accuracy than before. it can.

One aspect of the present invention is the video discriminator learning device described above, wherein the discriminator learning processing unit includes a scene division unit that outputs divided video data obtained by dividing the program video data for each scene, and the voice recognition A label providing unit that extracts a word to be a label from a result of speech recognition by the unit, and assigns the word as a label to the divided video data at a time corresponding to the utterance time of the extracted word; And a discriminator learning unit that learns the discriminator using the divided video data to which is assigned.
According to the present invention, the video discriminator learning device assigns a label to each scene of the program video data, and learns the discriminator using the program video data to which the label is assigned for each scene.
As a result, the video discriminator learning device assigns a label based on the commentary audio for each scene of the video, so even if the timing of the commentary speech is different from the timing of the video corresponding to the utterance content, it is accurate. A label can be often given to video data.

One aspect of the present invention is the video classifier learning device described above, wherein the commentary speech signal extracted by the commentary speech extraction unit and the speech recognition unit extract an acoustic model used by the speech recognition unit for speech recognition. A process of adapting using a result of voice recognition, a language model used by the voice recognition unit for voice recognition, a result of the voice recognition by the voice recognition unit, a closed caption of the program, and one or more of program information. It further comprises an adapting unit that performs at least one of processing to be used and adapted.
According to the present invention, the video discriminator learning device recognizes a commentary speech signal using an acoustic model adapted to a commentator or a language model adapted to a program, and uses a word obtained as a result of the speech recognition as a label. Append to video data.
Thereby, since the video discriminator learning apparatus can recognize the commentary speech with high accuracy, it is possible to give an appropriate label to the video data.

One aspect of the present invention is the video discriminator learning device described above, further comprising a recognition unit that recognizes video data using the discriminator learned by the discriminator learning processing unit. .
According to the present invention, the video classifier learning device recognizes video data using the learned classifier.
Thereby, the video discriminator learning device can accurately identify an object shown in the video and a scene included in the video.

One aspect of the present invention is the video discriminator learning device described above, wherein the first audio signal and the second audio signal are a main audio signal and a sub audio signal of a broadcast program.
According to the present invention, the video discriminator learning device extracts a comment audio signal from the main audio signal and sub audio signal of the comment broadcast program, and broadcasts the word obtained as a result of the voice recognition of the extracted comment audio signal as a label. It is added to the video data of the program.
Thereby, the video discriminator learning device can learn the discriminator using the broadcast program.

  According to one embodiment of the present invention, a computer compares a first audio signal that is an audio signal of a program audio with a second audio signal that is an audio signal of the program audio to which the explanation audio is added. The commentary voice extracting means for extracting the commentary voice signal, which is a voice signal, the voice recognition means for recognizing the commentary voice signal extracted by the commentary voice extraction means, and the result of the voice recognition by the voice recognition means becomes a label. Extracting a word, and using the extracted word and a feature amount extracted from program video data of a video section corresponding to the utterance time of the word, whether or not video data is related to the extracted word It is a program for functioning as a video discriminator learning device comprising discriminator learning processing means for learning a discriminator for detection from a feature amount of video data.

  ADVANTAGE OF THE INVENTION According to this invention, the discriminator for identifying whether it is an image | video corresponding to detection targets, such as an object and a scene, can be learned accurately.

It is a functional block diagram which shows the structure of the image | video discriminator learning device by the embodiment. It is a figure which shows the processing flow of the discriminator learning process of the image | video discriminator learning device by the embodiment. It is a figure which shows the processing flow of the comment audio | voice signal extraction process of the comment audio | voice extraction part by the embodiment. It is a figure which shows the speech recognition result by the same embodiment. It is the figure which arranged closed caption and transcript of commentary voice in time series.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
The video discriminator learning apparatus according to the present embodiment extracts the commentary audio by comparing the main audio channel and the sub audio channel of the comment broadcast program. The video discriminator learning device of this embodiment gives a word obtained as a result of voice recognition of the extracted commentary voice as a label to program video data in a video section corresponding to the utterance time of the word. The video discriminator learning device according to the present embodiment learns a discriminator for determining whether video corresponds to a detection target such as an object or a scene, using program video data to which a label is attached. When the detection target is an object, the discriminator determines whether or not the detection target object is displayed in the video. If the detection target is a scene, a series of operations displayed in the video is detected. It is determined whether or not the target scene.

  As described above, the video discriminator learning device according to the present embodiment generates learning video data by assigning a label to video data based on a recognition result of commentary audio that explains an object or a scene in the video. Therefore, the video discriminator learning device of the present embodiment generates a large amount of video data for learning by assigning labels to video data more accurately than using closed captioning while reducing human costs and time costs. be able to. The video discriminator learning device according to the present embodiment can learn a discriminator with high accuracy by using a large amount of video data for learning with labels attached with high accuracy in this way.

  FIG. 1 is a functional block diagram showing the configuration of the video discriminator learning device 1 according to the present embodiment, in which only functional blocks related to the present embodiment are extracted and shown. The video discriminator learning device 1 is realized by a computer device, for example, and includes a commentary speech recognition result extraction unit 2, a discriminator learning processing unit 3, and a recognition processing unit 4.

The commentary speech recognition result extraction unit 2 includes a commentary speech extraction unit 21, an acoustic model storage unit 22, a language model storage unit 23, a speech recognition unit 24, and a model adaptation unit 25.
The commentary sound extraction unit 21 compares the main sound signal and the sub sound signal of the commentary broadcast program, and extracts the commentary sound signal. The main audio signal is an audio signal of the program audio without explanation (first audio signal), and the sub audio signal is an audio signal of the audio with explanation (second audio signal) obtained by adding the explanation audio to the program audio. is there. In the audio with explanation, the program audio and the explanation audio are usually uttered without overlapping. The commentary audio signal is a commentary sound signal.

  The acoustic model storage unit 22 stores an acoustic model representing the frequency characteristics of each phoneme. The language model storage unit 23 stores a language model that represents the ease with which words are connected. The voice recognizing unit 24 uses the acoustic model stored in the acoustic model storage unit 22 and the language model stored in the language model storage unit 23 as a voice for the commentary voice signal extracted by the commentary voice extraction unit 21. recognize. The voice recognition unit 24 outputs voice recognition result data in which the voice recognition result of the commentary voice signal is set. The speech recognition result includes a transcription of the utterance content in units of morphemes, and a start time and an end time at which each morpheme is uttered.

  The model adaptation unit 25 adapts the acoustic model stored in the acoustic model storage unit 22 according to the speaker of the commentary speech, and adapts the language model stored in the language model storage unit 23 according to the program. To do. For the adaptation of the acoustic model and the language model, the commentary speech signal and the commentary content indicated by the speech recognition result of the commentary speech signal are used.

The classifier learning processing unit 3 includes a scene dividing unit 31, a label adding unit 32, and a classifier learning unit 33.
The scene division unit 31 divides the video data (program video data) of the explanation broadcast program for each similar scene, and outputs divided video data that is video data divided for each scene. The label assigning unit 32 assigns a label to the divided video data output from the scene dividing unit 31 based on the voice recognition result of the commentary voice indicated by the voice recognition result data output from the voice recognition unit 24. The discriminator learning unit 33 learns the discriminator 34 using the divided video data provided with the label by the label adding unit 32 as learning video data. The discriminator 34 detects whether or not the video data is related to the word to be detected based on the video feature amount obtained from the video data.

The recognition processing unit 4 includes a scene dividing unit 41 and a recognition unit 42.
The scene division unit 41 divides the video data of the content to be recognized for each scene by the same processing as the scene division unit 31 and outputs the divided video data. The content to be recognized may be a commentary broadcast program, a broadcast program other than the commentary broadcast program, or a video other than the broadcast program. In the present embodiment, a case where the content to be recognized is a broadcast program will be described as an example. The recognizing unit 42 recognizes the divided video data output from the scene dividing unit 41 by the discriminator 34 learned by the discriminator learning unit 33, and outputs the recognition result data in which the recognition result is set.

Next, the operation of the video classifier learning device 1 will be described.
FIG. 2 is a diagram illustrating a processing flow of the discriminator learning process of the video discriminator learning device 1.
First, the main audio signal, sub audio signal, and video data of the commentary broadcast program are input to the video discriminator learning device 1 (step S110: YES). These are obtained from broadcast signals, for example. In the case of stereo duplex broadcasting, an L channel audio signal and an R channel audio signal of the main audio and sub audio are input. The commentary speech recognition result extraction unit 2 receives the input main audio signal and sub audio signal, and the discriminator learning processing unit 3 receives the input video data.

The commentary speech extraction unit 21 of the commentary speech recognition result extraction unit 2 compares the main sound signal and the sub sound signal of the comment broadcast, and extracts the comment sound signal (step S120). In order to accurately extract the explanation audio signal, if the main audio signal and the sub audio signal are not synchronized with each other, it is necessary to compare the synchronization of both signals after matching them accurately. Therefore, first, the commentary voice extraction unit 21 performs a process of synchronizing the main voice signal and the sub voice signal. The main audio signal and the sub audio signal having the start time t and the voice interval length T are respectively _xt ^{t + T} = [x (t)... X (t + T−1)], y _t ^{t + T} = [y (t)... Y (t + T -1)]. The commentary voice extraction unit 21 calculates the synchronization shift time a so that the correlation coefficient r expressed by the following formula (1) is maximized. Note that the speech interval length T is a time sufficiently longer than the expected synchronization shift time a.

  Here, S (x, y) represents the covariance of variables x and y, S (x) represents the standard deviation of variable x, and S (y) represents the standard deviation of variable y.

  The commentary voice extraction unit 21 synchronizes the main voice signal and the sub voice signal for each voice section length T using the calculated synchronization time a. The commentary voice extraction unit 21 extracts a voice signal of the commentary voice superimposed on the sub voice signal from the synchronized main voice signal and sub voice signal. For example, there are the following extraction methods A to C as methods for extracting the speech signal of the commentary speech.

(Extraction method A)
The commentary voice extraction unit 21 performs windowing for a short time on each of the main voice signal and the sub voice signal. The windowed portion is sequentially shifted in the time direction according to the size of the window. The commentary sound extraction unit 21 calculates the difference between the power of the main sound signal and the power of the sub sound signal for each window. When the calculated power difference is smaller than a predetermined threshold, the commentary voice extraction unit 21 determines that the voice segment of the window is a voice segment other than the commentary speech. On the other hand, when the calculated power difference is equal to or greater than a predetermined threshold, the commentary speech extraction unit 21 specifies the speech segment of the window as the commentary speech segment. The commentary voice extraction unit 21 extracts a sub-voice signal in the specified commentary voice section as a commentary voice signal. The comment audio signal is given a start time and an end time. Note that when the extraction method A is used, a process for synchronizing the main audio signal and the sub audio signal is not necessarily required.

(Extraction method B)
The explanation voice extraction unit 21 considers the sub voice signal as a voice signal in which noise other than the explanation voice is added to the explanation voice and the main voice signal as a voice signal of noise other than the explanation voice, and extracts the sub voice signal from the sub voice signal by the spectral subtraction method. Extract only the audio signal. At this time, the commentary voice extraction unit 21 acquires the characteristics of the voice signal to be removed as noise from the sub voice signal from the main voice signal.

(Extraction method C)
In the extraction method C, the commentary voice extraction unit 21 subtracts the voice indicated by the main voice signal from the voice indicated by the sub voice signal, and extracts the voice signal of the difference voice as the commentary voice signal. However, if there is a difference in the sound level between the main sound and the sub sound in the section where the commentary sound is not superimposed, the sound other than the commentary can be accurately removed by simply subtracting. Instead, it remains as noise for commentary speech. In order to remove such noise with high accuracy, the commentary speech extraction unit 21 calculates a correlation coefficient between the main speech signal and the sub-speech signal for each speech section length T, and for a section where the correlation coefficient is equal to or greater than a certain threshold. The values of the commentary audio signal may all be 0.

FIG. 3 is a diagram showing a processing flow of the commentary voice signal extraction process of the commentary voice extraction unit 21 by the extraction method C.
First, the commentary voice extraction unit 21 sets an initial value 0 to t (step S210). Comment sound extractor 21, the correlation coefficient of the main audio signal _x ^{t t + T} and the sub audio signal ^{_{y t + a t + a +}} T r t (x t t + T, y t + a t + a + T) to determine whether a threshold _{r thre} more (step S220). The correlation coefficient is calculated by the above formula (1).

When the commentary speech extraction unit 21 determines that the correlation coefficient is equal to or greater than the threshold value (step S220: YES), all the values of the commentary speech signal z _t ^{t + T} of the start time t and the speech segment length ^T are set to 0 (steps) S230).
On the other hand, when the commentary voice extraction unit 21 determines that the correlation coefficient is less than the threshold value (step S220: NO), the commentary voice signal z _t ^{t + T} is obtained as the main voice signal x _t ^{t + T} from the sub voice signal y _{t + at} ^{+ a + T.} The subtracted value is set (step S240).

  After the process of step S230 or step S240, the commentary voice extraction unit 21 determines whether or not the processes of step S220 to step S240 have been completed for all voice signals (step S250). If the commentary voice extraction unit 21 determines that it has not been completed yet (step S250: NO), it adds T to the value of t and updates it (step S260), and then repeats the processing from step S220. If the commentary voice extraction unit 21 determines that the process of steps S220 to S240 has been completed for all voice signals (step S250: YES), the commentary voice extraction unit 21 ends the process.

  The commentary speech extraction unit 21 outputs the commentary speech signal extracted by any of the above extraction methods to the speech recognition unit 24.

  In FIG. 2, the speech recognition unit 24 uses the commentary speech signal extracted by the commentary speech extraction unit 21, the acoustic model stored in the acoustic model storage unit 22, and the language model stored in the language model storage unit 23. Is recognized in the same manner as in the prior art (step S130).

FIG. 4 is a diagram illustrating an example of a speech recognition result. As shown in the figure, the speech recognition result of the commentary speech signal includes the words included in the utterance content of the commentary speech and the start time and end time of each word. A word included in the utterance content of the explanation voice corresponds to a morpheme. For example, “smile” is spoken from the start time “03: 46.52” to the end time “03: 47.01”, and from the start time “03: 47.02” to the end time “03: 47.04”. Indicates that “de” was spoken.
The voice recognition unit 24 outputs voice recognition result data in which a voice recognition result is set.

  The model adaptation unit 25 adapts the acoustic model and the language model (unsupervised adaptation) using the commentary speech signal and the speech recognition result of the commentary speech signal indicated by the speech recognition result data according to the prior art (step unsupervised adaptation). S140). For example, acoustic model adaptation methods include MLLR and MAP estimation. Further, there is a linear interpolation method as a language model adaptation method.

The MLLR is, for example, “CJ Leggetter and PC Woodland,“ Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models ”, Computer Speech and Language, 1995, Volume 9, p. 171-185 (Reference 1). It is described in.
MAP estimation is described in, for example, “J. Gauvain and Chin-Hui Lee,“ Maximum a Posteriori Estimation for Multivariate Gaussian Mixture Observations of Markov Chains ”, Speech and Audio Processing, IEEE Transactionson, 1994, Volume 2, Issue 2, p. 291-298 "(Reference 2).
The linear interpolation method is described in, for example, “Kita Kenji,“ Probabilistic Language Model ”, University of Tokyo Press, 1999, p. 63-66” (Reference 3).

  In the case of the same commentary broadcast program, the commentary sound of the same person is often used. In each drama, words that are easy to use vary depending on the broadcast program. Therefore, the model adaptation unit 25 adapts the acoustic model and the language model in accordance with the explanation broadcast program, and associates the adapted acoustic model and language model with the program ID of the explanation broadcast program, respectively. The language model storage unit 23 is registered. In addition to the speech recognition result of the commentary speech or instead of the speech recognition result of the commentary speech, the model adaptation unit 25 includes closed captions of the commentary broadcast program or program information of the commentary broadcast program acquired from the electronic program guide. Can also be used to adapt language models. Closed captions and electronic program guides can be obtained from broadcast signals, for example. By using this information for adaptation, it is possible to efficiently learn proper nouns such as characters in drama and names of places uttered by commentary speech. When the speech recognition unit 24 performs speech recognition again in the repeated processing after adaptation in step S130, the acoustic model and the language model associated with the program ID of the explanation broadcast program from which the main audio signal and the sub audio signal are obtained. Is used. An acoustic model and a language model associated with a program ID of a similar or related explanation broadcast program such as an explanation broadcast program of the same genre as the explanation broadcast program from which the main audio signal and the sub audio signal are obtained may be used. Note that the model adaptation unit 25 may perform only one of the adaptation of the acoustic model and the adaptation of the language model.

  The scene dividing unit 31 of the discriminator learning processing unit 3 divides the video data of the commentary broadcast program into similar scenes using existing technology (step S140). For this scene division, it is possible to use an existing technique in which shots (also referred to as cuts) of the same scene are collectively determined as a scene using image feature amounts.

  Specifically, image segment words generated from a plurality of videos are stored in advance in a storage unit (not shown) included in the scene division unit 31 (or an external storage device connected to the video classifier learning device 1). Keep it. In order to generate an image fragment word, a frame image sampled from a plurality of videos is divided into a predetermined image size to form a block image, and a block image set is clustered based on a feature vector representing a feature amount of each block image. The image fragment word is obtained as a vector that is an element of the center vector of each cluster. The scene division unit 31 divides the input video data of the commentary broadcast program for each shot using an existing arbitrary shot detection technique, and generates a shot sequence. When the scene dividing unit 31 samples a frame image at a predetermined interval from each shot of the video data of the commentary broadcast program, a feature vector representing a feature amount of the image is obtained from a block image obtained by dividing the sampled frame image into a predetermined image size. get. The scene dividing unit 31 calculates a histogram of the image fragment word based on the similarity between the feature vector and the image fragment word for each shot. With this histogram, an appearance ratio indicating how many types of block images exist in each shot can be obtained. The scene division unit 31 integrates shots and divides them into scenes based on the amount of change in the histogram of each shot. The scene dividing unit 31 divides the video data for each scene to generate divided video data, and outputs the divided video data to the label adding unit 32.

  The label assigning unit 32 assigns a label to the divided video data of each scene output from the scene dividing unit 31 based on the voice recognition result of the commentary voice indicated by the voice recognition result data output from the voice recognition unit 24 ( Step S150). For example, the label assigning unit 32 selects one piece of divided video data that has not yet been labeled, and the start time and the end time of the selected divided video data are selected from the recognition result words indicated by the voice recognition result data. Identify words that have a start or end time within the range. The label giving unit 32 excludes words having a predetermined part of speech such as particles and auxiliary verbs from the specified words, and acquires a word that becomes a label. The label assigning unit 32 adds label data indicating the acquired word to the selected divided video data. The label assigning unit 32 selects the divided video data to which no label has been added yet and repeats the same processing. As a result, for example, the label assigning unit 32 adds the word of the recognition speech recognition result shown in FIG. 4 to the divided video data of the scene at the start time “03: 45.00” and the end time “03: 55.00”. Among them, label data indicating “smile”, “corridor”, “go”, “seri”, “staff room”, “front” is assigned.

  If there is still an explanation broadcast program input (step S110: YES), the video discriminator learning device 1 repeats the processing from step S120 to step S160. When there is no input of the explanation broadcast program (step S110: NO), the discriminator learning unit 33 uses the discriminator 34 corresponding to each detection target using the divided video data to which the word to be detected is assigned as a label. Learning is performed (step S170). Accordingly, the discriminator learning unit 33 refers to the label data added to each divided video data, and extracts a target label for learning the discriminator 34. Alternatively, the classifier learning unit 33 may receive an input of a target label for learning the classifier 34 from the user. The discriminator learning unit 33 selects the divided video data to which the target label for learning the discriminator 34 is assigned. For example, the classifier learning unit 33 selects all the divided video data to which the label data including the label “smile” to be learned by the classifier 34 is assigned. The discriminator learning unit 33 learns the video discriminator 34 corresponding to “smile” by machine learning using the video feature amount acquired from the selected divided video data. For example, a support vector machine or a random forest can be used for machine learning, but other supervised learning methods may be used. Further, any one or more kinds of feature amounts can be used as the feature amount of the video. For example, SIFT (Scale-Invariant Feature Transform) feature quantity, PCA (Principal Component Analysis) -SIFT feature quantity, Haar-like feature quantity, HOG (Histograms of Oriented Gradients) feature quantity, LBP (Local Binary Pattern) feature quantity, etc. Although it can be used, other feature quantities may be used.

  In the classifier learning process described above, the video classifier learning apparatus 1 may perform the processes in steps S120 to S140 and the process in step S150 in parallel, and the processes in steps S120 to S120 after the process in step S150. You may perform the process of step S140. Further, the video discriminator learning device 1 may execute the process of step S140 after the process of step S150 or step S160 or in parallel with the process of step S150 or step S160. Not necessary.

  The recognition processing unit 4 of the video discriminator learning device 1 recognizes the video of the broadcast program using the discriminator 34 learned by the discriminator learning process, and outputs a recognition result for each scene included in the video.

When recognizing what scene is included in a video and what object is displayed, the video discriminator learning device 1 operates as follows.
First, video data of a broadcast program to be recognized is input to the recognition processing unit 4 of the video classifier learning device 1. The scene dividing unit 41 divides the video data of the broadcast program to be recognized for each scene by the same processing as the scene dividing unit 31, and outputs the divided video data.
The recognition unit 42 recognizes each divided video data using all the classifiers 34. For example, by recognizing the divided video data using the discriminator 34 corresponding to “smile”, it is possible to obtain a determination result as to whether or not the divided video data corresponds to “smile”. Whether or not “smile” is supported means whether or not the scene of “smile” to be recognized is included in the divided video data. Further, by recognizing the divided video data using the discriminator 34 corresponding to “corridor”, it is possible to obtain a determination result as to whether or not the divided video data corresponds to “corridor”. Whether or not it corresponds to “corridor” means whether or not “corridor” to be recognized is displayed in the video. The recognition unit 42 outputs recognition result data in which the recognition result of each divided video data is set. For example, the recognition result data includes the start time and end time of each divided video data (scene) and the recognition target that is determined to correspond to each divided video data.

When searching for a broadcast program video corresponding to the keyword, the video discriminator learning device 1 operates as follows.
The recognition processing unit 4 performs the recognition process using all the discriminators 34 on the video data of each broadcast program as described above, and internally associates the broadcast program identification information with the recognition result (not shown). The information is stored in a storage unit (an external storage device connected to the video classifier learning device 1 may be used). When a keyword is input to the recognition processing unit 4, the recognition unit 42 searches the recognition result of each broadcast program stored in the storage unit using the input keyword. When information about a broadcast program to be searched is input in addition to the keyword, the recognition unit 42 searches the recognition result of the broadcast program to be searched with the input keyword. The recognition unit 42 acquires the start time and end time of the divided video data identified as corresponding to the keyword, and broadcast program identification information in which the identified divided video data is included in the recognition result. The recognizing unit 42 reads program information corresponding to broadcast program identification information from a database or the like provided inside or outside the video discriminator learning device 1, and outputs the read program information and information of the divided video data as recognition result data. . The program information can be arbitrary, for example, the program ID of the broadcast program, the title of the broadcast program, or the description of the broadcast program. The information of the divided video data may be the start time and end time of the divided video data, or may be video data of a broadcast program at the start time and end time or a still image extracted from the video data.

When searching for whether or not a video corresponds to a keyword, the video discriminator learning device 1 operates as follows.
First, video data and keywords of a broadcast program to be recognized are input to the recognition processing unit 4 of the video classifier learning device 1. The scene dividing unit 41 divides the video data of the broadcast program to be recognized for each scene by the same processing as the scene dividing unit 31, and outputs the divided video data.
The recognition unit 42 recognizes each divided video data using the classifier 34 corresponding to the keyword. The recognition unit 42 outputs the start time and end time of the divided video data when there is divided video data determined to correspond to the keyword by recognition using the discriminator 34 corresponding to the keyword. Further, when there is no divided video data determined to correspond to the keyword, the recognizing unit 42 outputs that it does not correspond to the keyword.

In the above embodiment, the scene dividing unit 31 and the scene dividing unit 41 divide the video data for each scene, but divide the video data for each shot or for each predetermined time to obtain divided video data. It may be output.
Further, the video classifier learning device 1 can be configured without the scene dividing unit 31. In this case, the label assigning unit 32 obtains a word to be a label by excluding a word of a predetermined part of speech such as a particle or an auxiliary verb from the words of the recognition result indicated by the speech recognition result data. The label assigning unit 32 selects the acquired words one by one, and assigns the label data indicating the selected words to the video data at a predetermined time before and after the start time or end time of the selected word. For example, when the start time of the selected word w is t, label data representing the word w is given from time t-a to time t + b of the video data (a and b are 0 or more). Since the explained contents often appear in the video after the explanation, a <b may be satisfied.

  In the above-described embodiment, the case where the content to be recognized is a moving image has been described, but it may be a still image. When the content to be recognized is a still image, the image data of the still image is input directly to the recognition unit 42 without being input to the scene division unit 41. The classifier learning unit 33 learns a classifier using image feature amounts that can be extracted from image data of a still image.

  According to the above-described embodiment, the video discriminator learning device 1 can assign a discriminator learning label without manipulating the explanation broadcast program. In addition, the video discriminator learning device 1 can give a label with higher accuracy than the conventional method using closed captioning. Therefore, the image discriminator learning device 1 can learn a discriminator with higher accuracy than before.

  Note that the video classifier learning device 1 described above has a computer system therein. The operation process of the image discriminator learning device 1 is stored in a computer-readable recording medium in the form of a program, and the above processing is performed by the computer system reading and executing this program. The computer system here includes a CPU, various memories, an OS, and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

DESCRIPTION OF SYMBOLS 1 Image | video discriminator learning apparatus 2 Explanation voice recognition result extraction part 3 Classifier learning process part 4 Recognition process part 21 Explanation voice extraction part 22 Acoustic model memory | storage part 23 Language model memory | storage part 24 Voice recognition part 25 Model adaptation part 31 Scene division | segmentation Unit 32 Label giving unit 33 Classifier learning unit 34 Classifier 41 Scene division unit 42 Recognition unit

Claims (6)

The first audio signal, which is the audio signal of the program audio, is compared with the second audio signal, which is the audio signal of the program audio, to which the explanation audio is added, and the explanation audio signal which is the audio signal of the explanation audio is extracted. Commentary voice extractor,
A voice recognition unit that recognizes the commentary voice signal extracted by the commentary voice extraction unit;
A word as a label is extracted from a result of speech recognition by the speech recognition unit, and extracted using the extracted word and a feature amount extracted from program video data of a video section corresponding to the utterance time of the word. A discriminator learning processing unit for learning a discriminator for detecting whether or not video data is related to the word from the feature amount of the video data;
A video discriminator learning device comprising: The classifier learning processing unit
A scene division unit for outputting divided video data obtained by dividing the program video data for each scene;
A label providing unit that extracts a word as a label from a result of speech recognition by the speech recognition unit, and that gives the word as a label to the divided video data at a time corresponding to the utterance time of the extracted word;
A discriminator learning unit that learns the discriminator using the divided video data to which the label imparting unit has a label;
The video discriminator learning device according to claim 1. A process of adapting an acoustic model used by the speech recognition unit for speech recognition using the commentary speech signal extracted by the commentary speech extraction unit and a result of the speech recognition by the speech recognition unit; and the speech recognition unit An adaptation unit that performs at least one of adaptation of a language model used for speech recognition by using one or more of program closed caption and program information as a result of the speech recognition by the speech recognition unit. In addition,
The video discriminator learning device according to claim 1 or 2. A recognition unit that recognizes video data using the classifier learned by the classifier learning processing unit;
The video discriminator learning device according to any one of claims 1 to 3, wherein The first audio signal and the second audio signal are a main audio signal and a sub audio signal of a broadcast program,
The video discriminator learning device according to any one of claims 1 to 4, characterized in that: Computer
The first audio signal, which is the audio signal of the program audio, is compared with the second audio signal, which is the audio signal of the program audio, to which the explanation audio is added, and the explanation audio signal which is the audio signal of the explanation audio is extracted. Commentary voice extraction means,
Voice recognition means for recognizing the commentary voice signal extracted by the commentary voice extraction means;
A word to be a label is extracted from a result of speech recognition by the speech recognition means, and extracted using the extracted word and a feature amount extracted from program video data of a video section corresponding to the utterance time of the word. Discriminator learning processing means for learning a discriminator for detecting whether or not video data is related to the word from the feature amount of the video data;
A program for causing a video discriminator learning device to function.

Images