JP2009048334A - Video identification processing device, image identification processing device, and computer program - Google Patents

Abstract

Even when text data associated with video data or image data does not include a word representing the subject itself, video data or image data showing the desired subject is obtained as a search result. .
A video identification processing device classifies and stores video data and text data associated with the video data for each class, and a probability parameter value representing a relationship between the word and the class. And a probability parameter value database to be held in association with the class, and based on the appearance frequency of the word included in the text data read from the content database, to calculate a probability parameter value representing the relationship between the word and the class, A probability calculation processing unit that writes the calculated probability parameter value in the probability parameter value database in association with the word and the class.
[Selection] Figure 1

Description

  The present invention relates to a video identification processing device, an image identification processing device, and a computer program.

While the search technology for text content has advanced and the range of practical use has expanded, a technology for efficiently searching non-text content such as video and images has been demanded.
Japanese Patent Application Laid-Open No. 2004-228561 describes a technique of a keyword video search method in which a keyword text is given to a video in advance and a video having a keyword that matches a search word is output.
Patent Document 2 describes a technology of a video indexing device that indexes video based on words of specific parts of speech extracted based on the syntax from closed captions included in the video information.
Japanese Patent Application Laid-Open No. 2004-228561 describes a technique of a video subject extraction device that extracts a video including a desired subject based on text data including a script or closed caption used simultaneously with the video.
JP 2002-007415 A JP 2007-006116 A JP 2007-188169 A

In the technique disclosed in Patent Document 1 described in the background art described above, there is a problem that it is necessary to manually assign keywords in order to index videos, and it takes a lot of time and labor.
In addition, in the techniques disclosed in Patent Document 2 and Patent Document 3, it is not necessary to manually assign a keyword, but when searching for an image in which a desired subject is copied, a word representing the subject itself is converted into text. If it was not included, the video could not be obtained as a search result. For example, in order to search and obtain a video of an animal lion as the subject, “This is the king of beasts, the lion” in the text such as the closed caption associated with the video. , The word “lion” had to be included.

  In practice, however, words that directly represent the subject itself may not be included in text such as closed captions. For example, the closed caption corresponding to a video scene showing a baseball batter box contains words such as “batter”, “curve”, and “strike”, but the word “batter box” itself is May not be included. As another example, although closed captions corresponding to video scenes where the gas range for cooking has moved include words such as “fry pan”, “high heat”, “1 tablespoon”, etc., “ The word “gas range” may not be included.

  As in the above example, even if a word that directly represents the subject itself is not included in text such as closed captions, if such a video can be included in the search results, the search for the video or image is possible. This leads to improved accuracy.

  The present invention has been made in view of such circumstances and problems, and is desired even when a word representing the subject itself is not included in text data associated with video data or image data. An object of the present invention is to provide a video identification processing device, an image identification processing device, and a computer program for obtaining video data and image data showing a subject as a search result.

In order to solve the above problems, the present invention predicts video content to some extent even if a word representing the subject itself does not appear in a document from peripheral word information obtained from video data and text around image data. Provide a means.
Then, the present invention classifies video data and image data with similar ones according to image characteristics, video characteristics, etc., and lexical characteristics in the text of each class (the lexical characteristics of the class are: A probability value is calculated from a word appearance frequency vector) and used as an index of correlation between language and image. That is, the class occurrence probability when a lexical feature is given or the class occurrence probability when a word is given is an index of the correlation between the language and the image. For example, the words “frying pan”, “high heat”, and “one tablespoon” predict a video (class) of cooking in front of a gas range in a cooking program (that is, a high probability of occurrence), “batter”, “ From the words “strike” and “curve”, a video of a batter box of a baseball program is predicted. In terms of language, neither the word “gas range” nor the word “batter box” appears, but the surrounding words (in the above example, “fry pan”, “high heat”, “one tablespoon”, “ Video is estimated using clues such as “batter”, “strike”, and “curve”. We provide a system that models video and text pairs and predicts video when text-only input is received.

[1] A video identification processing device according to an aspect of the present invention represents a relationship between a content database that stores video data and text data associated with the video data by class, and a word and the class. Based on the probability parameter value database that holds the probability parameter value in association with the word and the class, and the appearance frequency of the word included in the text data read from the content database, the relationship between the word and the class is determined. A probability calculation processing unit that performs a process of calculating a probability parameter value to be expressed and writes the calculated probability parameter value in the probability parameter value database in association with the word and the class.
According to this, video data and text data are originally associated with each other. In the content database, a pair of video data and text data belongs to a certain class. The probability calculation processing unit calculates a probability parameter value representing the relationship between the word and the class based on the appearance frequency of the word included in the text data read from the classified content database. As one of the best forms of the probability parameter value, for example, the class occurrence probability for a given word is used. As a result, data representing the relationship between the word, the class, and the calculated probability parameter value is stored in the probability parameter value database.

[2] Further, in the video identification processing device according to one aspect of the present invention, the input text data is read, the appearance frequency of words included in the text data, and the probability parameter value read from the probability parameter value database. And an identification processing unit that calculates a class occurrence probability, which is a probability that the text data belongs to the class, for each class.
According to this, the identification processing unit generates a class from the input text data by a predetermined calculation process from the probability parameter value read from the probability parameter value database and the appearance frequency of the input text data. The probability can be calculated, that is, the class to which the input text belongs can be estimated. The class with the highest class occurrence probability calculated can be determined as the maximum likelihood class.

[3] In the video identification processing device according to one aspect of the present invention, the class to which the video data belongs is determined based on the video characteristics of the input video data, and the video data is associated with the video data. A video identification processing device, further comprising: a video classification processing unit that writes the text data to the content database.
According to this, the video classification processing unit classifies the pairs of video data and text data into classes based on the characteristics of the video regardless of the text data and the meaning represented by the video data, and the content database. Write to.

  [4] Further, the video identification processing device according to one aspect of the present invention uses the probability parameter value representing the relationship between the word and the class calculated based on the appearance frequency of the word included in the text data as the word and the class. A probability parameter value database stored in association with the text data, the input text data, the appearance frequency of words included in the text data, and the probability parameter value read from the probability parameter value database, the text An identification processing unit that calculates a class occurrence probability, which is a probability that data belongs to the class, for each class.

[5] Also, in the video identification processing device according to one aspect of the present invention, a video database that stores video data classified by class and a search result that matches the search word based on the input search word A search processing unit that obtains a class and outputs at least one of video data belonging to the search result class or reference information to video data belonging to the search result class by referring to a video database; and Video data is associated with the input text data, and the identification processing unit further determines a maximum likelihood class of the input text data based on the calculated class occurrence probability, and the video data Writing the video data to the video database so that the video data belongs to the maximum likelihood class determined. And features.
Thereby, a class that matches the search term can be obtained. Video data whose class is the maximum likelihood class (or reference information to the video data) can be read from the video database and output as a video search result.

  [6] In the video identification processing device according to one aspect of the present invention, the word is obtained by excluding particles, auxiliary verbs, and symbols from the result of morphological analysis processing of the text data. To do.

[7] Further, in the video identification processing device according to one aspect of the present invention, the word is obtained by extracting a noun, a verb, an adjective, and an unknown word from a result of morphological analysis processing on the text data. It is characterized by.
Here, as will be described later, the unknown word is a word that is treated as an unknown part of speech as a result of morphological analysis processing, and the part of speech treated as such is likely to be a noun actually.

  [8] In the video identification processing device according to the aspect of the present invention, the video data is video data of a broadcast program, and the text data is associated with text data of closed caption of the broadcast program and time information. Or the text data associated with time information obtained as a result of voice recognition processing based on the video data of the broadcast program.

[9] In addition, the image identification processing device according to an aspect of the present invention represents a content database that classifies and holds web content data including image data and text data for each class, and a relationship between words and the classes. Based on the probability parameter value database that holds the probability parameter value in association with the word and the class, and the appearance frequency of the word included in the text data read from the content database, the relationship between the word and the class is determined. A probability calculation processing unit that performs a process of calculating a probability parameter value to be expressed and writes the calculated probability parameter value in the probability parameter value database in association with the word and the class.
It is characterized by.
Thereby, the web content data classified based on the image characteristics of the image data included in the web content data is held in the content database for each class. Then, based on the lexical characteristics of the text data included in the web content data included in the class, the probability calculation processing unit calculates a probability parameter value representing the relationship between the word and the class (as one of the best modes, for example, The class occurrence probability for a given word) can be written and stored in the probability parameter value database.

  [10] A computer program according to an aspect of the present invention represents a relation between a content database that stores video data and text data associated with the video data by class, and a word and the class. A computer comprising a probability parameter value database that stores the probability parameter value in association with the word and the class, based on the appearance frequency of the word included in the text data read from the content database, the word and the A computer that executes a process of calculating a probability parameter value representing a relationship with a class, and writing the calculated probability parameter value in the probability parameter value database in association with the word and the class. It is a program.

  [11] Further, the computer program according to one aspect of the present invention associates a probability parameter value representing a relationship between the word and the class, calculated based on the appearance frequency of the word included in the text data, with the word and the class. A computer having a probability parameter value database to be stored, read the input text data, based on the appearance frequency of words included in the text data, and the probability parameter value read from the probability parameter value database, A computer program for executing a process of an identification processing step for calculating a class occurrence probability, which is a probability that the text data belongs to the class, for each class.

  [12] A computer program according to an aspect of the present invention includes a content database that classifies and holds web content data including image data and text data for each class, and a probability parameter that represents a relationship between a word and the class. A probability parameter value database that stores values in association with the word and the class, and based on the appearance frequency of the word included in the text data read from the content database, the word and the class A probability calculation processing step of performing a process of calculating a probability parameter value representing the relationship between the probability parameter value and writing the calculated probability parameter value to the probability parameter value database in association with the word and the class. It is.

According to the present invention, even if the word is not included in the text data related to the video data, the video can be searched using the word representing the subject or the like shown in the video. Alternatively, it is possible to create a probability parameter value database (identification model database) that is the basis of such video search.
In addition, according to the present invention, based on web content including image data (or video data) and text data, even a word not included in the text data represents a subject or the like reflected in the image or video. You can search for images and videos using words.
Note that text data used is, for example, closed caption data, script data, or voice recognition result data, so that it is not necessary to create new text data by taking time and effort. That is, the data to be searched can be accumulated efficiently.

[First Embodiment]
The first embodiment of the present invention will be described below with reference to the drawings.
<Identification model generation process>
FIG. 1 is a block diagram showing a functional configuration of an identification model generation device (video identification processing device) according to the present embodiment. In this figure, reference numeral 100 denotes an identification model generation device (video identification processing device). The identification model generation apparatus 100 includes a storage unit that stores program video data 1 (video data), a storage unit that stores closed caption data 2 (text data), a video classification processing unit 3, a content database 4, and the like. The probability calculation processing unit 5 and the identification model database 6 (probability parameter value database) are included.

  The program video data 1 is data representing a video of a program such as a television broadcast, for example. A video having a length of several minutes to several hours and audio of one or a plurality of channels recorded in synchronization with the video are recorded. Is included. The closed caption data 2 is text data corresponding to the program video data, and one or a plurality of sentences are recorded in association with the time. In other words, the text data of the closed caption data 2 is data synchronized with time. As the time here, standard time (local time such as Japan standard time or world standard time) corresponding to the broadcast date and time may be used, or relative time from the video start time may be used. The closed caption data 2 is also originally created for the purpose of being used for television broadcasting, and is typically text data for displaying a person's utterance in a program as characters on a television screen. The program video data 1 and the closed caption data 2 are stored as archive data of a broadcasting station for later use as materials after broadcasting. These data are recorded on a medium such as a magnetic tape, a magnetic hard disk, or an optical disk, and can be read as appropriate using a device that reads each data.

  The video classification processing unit 3 determines a class to which the program video data 1 belongs based on the video characteristics of the input program video data 1, and the program video data 1 and closed caption data associated with the video data. 2 is written in the content database 4.

  The content database 4 classifies and holds program video data 1 and closed caption data 2 associated with the program video data 1 for each class. As shown in the figure, the content database 4 stores video data and text data in association with each other, class 1, class 2, class 3,. Each class may include a plurality of pairs of video data and text data. The content database 4 is realized using storage means such as a magnetic disk device or an optical disk device. The data structure of the content database 4 will be described later with reference to another drawing.

The probability calculation processing unit 5 performs a process of calculating a probability parameter value representing the relationship between the word and the class based on the appearance frequency of the word included in the text data read from the content database 4. In addition, the probability calculation processing unit 5 writes the calculated probability parameter value in the probability parameter value database in association with the word and the class.
Here, the probability parameter value is, for example, a value of occurrence probability of a certain word in a certain class. Such an occurrence probability calculation method will be described in detail later.

  The identification model database 6 is a database that holds the probability parameter value representing the relationship between a word and a class calculated above in association with the word and the class. The identification model database 6 is realized using storage means such as a magnetic disk device or an optical disk device. The data structure of the identification model database 6 will be described later with reference to another drawing.

  FIG. 2 is a schematic diagram showing the data structure of the content database 4. As shown in the figure, the content database 4 includes data in a table (two-dimensional table) format, and the table includes data items such as class, video data, and text data. In each row of the table, the class, the video data, and the text data are associated with each other according to the value of each data item. For example, in the first line of the illustrated data example, “video data 1” corresponds to “text data 1”, and the class to which “video data 1” belongs is “class 1”. Yes. Here, “class 1”, “class 2”, “class 3”, and the like are data for identifying classes. “Video data 1”, “Video data 2”, and the like are binary data itself such as MPEG (Moving Picture Experts Group) -4 format or reference data to a data file storing such binary data. “Text data 1”, “Text data 2”, and the like are closed caption text data corresponding to video, or reference data to a data file storing such text data.

  FIG. 3 is a schematic diagram illustrating a data configuration and a data example of the identification model database 6. As shown in the figure, the content database 4 is configured by data in a table format, and the table includes data items such as class, word, and occurrence probability. In each row of the table, the class, the word, and the occurrence probability are associated with each other according to the value of the data item. For example, in the first row of the illustrated data example, the value of the probability that the word “curve” occurs in the class “class 1” is stored in the column of the occurrence probability (the occurrence probability value itself is not shown). )

Next, processing procedures by each unit of the identification model generation device 100 will be described in order.
First, the video classification processing unit 3 analyzes the program video data 1 to divide the program video data 1 and the closed caption data 2 corresponding thereto into an appropriate length in the time direction. The timing of division is, for example, when the scene of the program video is switched, or when the audio level is switched from less than a predetermined value to more than a predetermined value (or vice versa). The video scene switching timing is detected by differentiating the moving image data in the time direction.
In addition, the video classification processing unit 3 classifies the program video data into a plurality of classes based on the video features of each segmented section. The video features for classification here are features that are obtained purely from the image data itself, such as the luminance of each RGB (red, green, blue) channel and their distribution in the position on the screen. Classification by the semantic features of is not necessary. Then, the video classification processing unit 3 writes the program video data 1 and the closed caption data 2 associated therewith in the content database in a form classified for each class of the classification result.

Next, the probability calculation processing unit 5 analyzes the text corresponding to the class of the classified image (video data). At this time, a probability parameter value obtained by quantifying the relationship between the linguistic information (words included in the text) and the classification class is calculated.
FIG. 4 is a flowchart showing the procedure of the first half of the processing by the probability calculation processing unit 5. Hereinafter, it demonstrates along this flowchart.

First, in step S <b> 11, the probability calculation processing unit 5 reads a document from the content database 4. This document is text data stored in the content database 4. A document is a collection of all text data belonging to one class.
Next, in step S12, the probability calculation processing unit 5 takes out one sentence from the read document.
Next, in step S <b> 13, the probability calculation processing unit 5 performs a morphological analysis process on the extracted sentence. As a result of this morphological analysis, words (morphemes) included in the document are extracted.

Next, in step S14, the probability calculation processing unit 5 determines whether or not each extracted word is a word other than a particle, an auxiliary verb, and a symbol. If the determination result is affirmative, that is, if the word is not a particle, auxiliary verb, or symbol, the process proceeds directly to step S15. If the determination result is negative, that is, if the word is any of a particle, an auxiliary verb, or a symbol, step S15 is skipped and the process proceeds to step S16.
Next, in step S15, the probability calculation processing unit 5 calculates and updates the word frequency. The process of step S15 is performed for each word obtained as a result of the above morphological analysis and not a particle, auxiliary verb, or symbol. Details of the frequency calculation will be described later.

Next, in step S16, the probability calculation processing unit 5 determines whether or not the one sentence extracted in step S12 above is the last sentence of the document. If it is the last sentence, the process proceeds to step S17. If it is not the last sentence, the process returns to step S12 to process the next sentence.
Next, in step S <b> 17, the probability calculation processing unit 5 totals the frequencies of the words for all sentences to obtain a frequency vector. The frequency vector will be described later.

  Next, the process from step S11 to S17 mentioned above is demonstrated in detail using numerical formula. That is, the probability calculation processing unit 5 performs morphological analysis on the sentence in the text data read from the content database 4 and divides it into words. Then, particles, auxiliary verbs, and symbols are excluded from those words, and a document vector x and a frequency vector N (x) are obtained based on the appearance frequency of the remaining words. These document vector x and frequency vector N (x) are expressed by the following equations (1) and (2), respectively.

Here, the document is all text data belonging to one class. W _i (i = 1, 2, 3,..., N) is a word belonging to the document. Note that there are no overlapping words in the set of w _i . That is, n types of words appear in the document, excluding particles, auxiliary verbs, and symbols. In other words, for any i, j (where i ≠ j), w _i ≠ w _j . N _i (i = 1, 2, 3,..., N) is the appearance frequency (that is, the number of appearances) of the word w _{i in} the document.

Since the document vector x and the frequency vector N (x) are obtained in the processing from step S11 to S17 described above, the probability calculation processing unit 5 next uses the following expression (3) using these. The occurrence probability P (w _i | c) of the word w _i in each class is calculated.

However, N (i, x) is the frequency of the word _{w i} in the document x (number of occurrences). Also, in this equation, smoothing by Laplace estimation is performed in consideration of the zero frequency problem, and the smoothing coefficient δ is used as the parameter. V represents the number of types of words in class c.

The probability calculation processing unit 5 calculates the occurrence probability P (w _i | c) by the above method for all combinations of words and all classes. Then, the calculated occurrence probability P (w _i | c) is written in the table of the identification model database 6 in association with the word and the class.

Through the above-described series of processing, the identification model generation device 100 represents the relationship between the words included in the closed caption data 2 and the class based on the video features based on the pair of the program video data 1 and the closed caption data 2. The occurrence probability P (w _i | c), which is a parameter value, is calculated, and the identification model database 6 based on this is constructed.

<Identification process>
Next, processing for identifying an image class that matches newly input text data using the identification model constructed by the above-described identification model generation processing will be described.
FIG. 5 is a block diagram showing a functional configuration of the identification processing device (video identification processing device) according to the present embodiment. As shown in the figure, reference numeral 200 is an identification processing device. The identification processing device 200 includes an identification model database 6, a storage unit that stores the document 7, an identification processing unit 8, and a storage unit that stores the image class 9.

In the above configuration, the identification model database 6 holds the probability parameter value (ie, the occurrence probability P (w _i | c)) calculated in the identification model generation process for each word and for each class. Document 7 is input text data. The user of the identification processing device 200 can input the document 7 for the purpose of extracting a video suitable for the document 7. The identification processing unit 8 reads the document 7 (text data), calculates the appearance frequency of words included in the document 7, and calculates the appearance frequency and the probability parameter value read from the identification model database 6. Based on this, the class occurrence probability, which is the probability that the text of the document 7 belongs to a certain class, is calculated for each class. Further, the identification processing unit 8 determines the maximum likelihood class corresponding to the document 7 by using the calculated class occurrence probability. The image class 9 is obtained by outputting the maximum likelihood class determined by the identification processing unit 8.

  The storage means for storing the document 7 and the image class 9 is realized using, for example, a semiconductor memory, a magnetic disk device, an optical disk device, or the like.

FIG. 6 is a flowchart showing a procedure of identification processing by the identification processing unit 8. Hereinafter, the identification process performed by the identification processing unit 8 will be described with reference to FIG.
First, in step S31, the identification processing unit 8 reads the input document 7.
Next, in step S32, the identification processing unit 8 extracts words included in the read document 7 and calculates the frequency of those words. The method of word extraction and frequency calculation of this part is the same as the method in the identification model generation processing (the processing method from steps S11 to S17 described above). As a result of step S32, a document vector and a frequency vector for the input document 7 are obtained.

Next, in step S33, the identification processing unit 8 calculates the occurrence probability (class occurrence probability) of each class for the input document 7 using the following method.
First, the identification processing unit 8 reads the necessary occurrence probability P (w _i | c) with reference to the identification model database 6 as appropriate, and uses the following equation (4) to The attached probability P (c | x) is obtained.

Here, x is a document vector of the input document 7. N (i, x) is the frequency of the word w _i corresponding to the document vector x. “!” Is a factorial operator.

In the equation (4), P (w _i | c) is a probability parameter value read from the identification model database 6 by the identification processing unit 8 using the word w _i and the class c as keys. If the value of P (w _i | c) corresponding to the word w _i is not stored in the identification model database 6, P (w _i | c) = δ / (δ · | V |). That is, P (w _i | c) = 1 / | V | is set according to the expression (3).

In Expression (4), | x | is a document frequency. That is, the number of appearances of a document having the same document vector as the document vector x. P (| x |) is a probability that the document frequency is | x |.
That is, if there are two documents having a document vector of x1 in a certain class, | x | is 2. In addition to the vector x1, the ratio of document vectors in which | x | = 2 is p (| x |).
For example, in a certain class, there are two documents having the document vector x1 (that is, | x1 | = 2), one document having the document vector x2 (that is, | x2 | = 1), and a document having the document vector x3. If there are two (ie, | x3 | = 2) and there are no other documents, the frequency of | x | = 2 is 2 (x1 and x3), and the frequency of | x | = 1 Is 1 (x2). Therefore, p (| x1 |) = 2/3, p (| x2 |) = 1/3, and p (| x3 |) = 2/3.

  Next, the identification processing unit 8 obtains P (c | x) by using Bayes's theorem and using Equation (5) based on Equation (4).

Here, P (c) is simply the existence probability of class c based on the distribution of documents between classes.
Note that the rightmost denominator in the equation (5) is constant regardless of the class. Therefore, it is as Formula (6).

  In the above, Expression (4) and Expression (5) are shown as the calculation process. However, as a matter of course, the identification processing unit 8 directly calculates only the value on the right side of Expression (6) for each class, The class occurrence probability P (c | x) may be calculated.

  Returning to the flowchart, next, in step S34, the identification processing unit 8 determines the maximum likelihood class for the given document based on the calculated class occurrence probability P (c | x) of each class. That is, the maximum likelihood class is given by the following equation (7).

That is, through this process, the identification processing unit 8 can determine the video class most suitable for the given document 7.
As a matter of course, the identification processing unit 8 may directly calculate the maximum likelihood class of the document using the equation (7) for each class.
In addition, the process mentioned above is based on a Naive Bayes separator. In addition, the occurrence of each word is assumed to be independent.

  Note that video data may be associated with the document 7 that is input to the identification processing unit 8. In that case, the class occurrence probability of the video data can be calculated based on the document 7 and the class occurrence probability stored in the identification model database 6, and therefore the maximum likelihood class (image class) of the video data can be calculated. 9) can be output.

Next, an identification model generation and identification processing apparatus (video identification processing apparatus) that includes both a processing unit that generates an identification model and a processing unit that performs identification processing will be described.
FIG. 7 is a block diagram illustrating a configuration of the identification model generation and identification processing device 300. As illustrated, the identification model generation and identification processing device 300 includes an identification model generation device 100 and an identification processing device 200. Details of the identification model generation device 100 and the identification processing device 200 have already been described with reference to the drawings.

  As already described, the identification model generation device 100 and the identification processing device 200 have the identification model database 6. As a result of processing in the identification model generation device 100, a probability parameter value representing the relationship between a word and a class (that is, a class occurrence probability value based on a certain word) is written in the identification model database 6 of the identification model generation device 100. It is. The written value is reflected in the identification model database 6 of the identification processing device 200. On the identification processing device 200 side, the above-described identification processing is performed using the probability parameter value written in the identification model database 6. As a method of reflecting the database update between the two, for example, using mirroring of a hard disk device, using the mirroring mechanism of the database management system, or using database update on the identification model generation device 100 side as a trigger This can be realized by writing the same value to the database on the identification processing device 200 side using a mechanism for detecting and starting a predetermined procedure. Alternatively, a single database may be shared in a form that can be accessed from both the identification model generation device 100 and the identification processing device 200.

  That is, the identification model generation and identification processing device 300 performs classification of program video data into classes and identification model construction processing based on program video data and closed caption data in a content database (not shown). (The above is the processing of the identification model generation apparatus 100). Using the constructed identification model, the class occurrence probability for the document (text) is calculated, and based on this class occurrence probability, the class of the video that matches the document. Can be performed (the above is the processing of the identification processing device 200).

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the following, description of parts that are the same as those in the first embodiment described above will be omitted, and only parts that are unique to the present embodiment will be described.
In the first embodiment, words other than particles, non-auxiliary verbs, and non-symbols are used as words constituting the document vector. In this embodiment, instead of nouns, verbs, Use adjectives and unknown words.

  FIG. 8 is a flowchart illustrating a procedure of word extraction and frequency vector creation processing in the probability calculation processing unit 5 and the identification processing unit 8 of the present embodiment. Describing with reference to this figure, the processing up to reading a document in step S11, taking out one sentence from the document in step S12, and performing morphological analysis in step S13 is the same as in the first embodiment.

  The processing in the next step S14A is specific to this embodiment. In step S14A, the probability calculation processing unit 5 or the identification processing unit 8 determines whether each extracted word is a noun, a verb, an adjective, or an unknown word. It is determined whether it is a word. If the determination result is affirmative, that is, if the word is one of a noun, a verb, an adjective, or an unknown word, the process proceeds directly to step S15. If the determination result is negative, that is, if the word is not a noun, verb, adjective, or unknown word, step S15 is skipped and the process proceeds to step S16.

  In step S14A, the unknown word is a morpheme (word) determined to be an unknown part of speech as a result of morphological analysis. Depending on the method of morphological analysis processing, such unknown words whose part of speech cannot be specified are output as analysis results. Note that words that are determined to be unknown words are often nouns in practice, and it is effective to include such unknown words as part of the document characteristics by including them in the document vector and frequency vector. It is.

  The process of calculating and updating the word frequency in step S15, the process of determining whether or not it is the last sentence in step S16, and the process of calculating the frequency vector in step S17 are the first embodiment. It is the same.

  In the identification model generation device (video identification processing device) or the identification processing device (video identification processing device) according to the present embodiment by the processing in step S14A as described above, nouns, verbs, adjectives, and unknown words are targeted. Probability calculation processing and identification processing are performed.

[Third embodiment]
Next, a third embodiment of the present invention will be described. In the following, description of parts that are the same as those of the above-described embodiment will be omitted, and only parts that are unique to this embodiment will be described.

  FIG. 9 is a block diagram illustrating a functional configuration of the identification model generation device 100B (video identification processing device) according to the present embodiment. The feature of this embodiment is that, instead of using closed caption data 2 in the first embodiment, program script data 2B (text data) to which time information is added is used.

  The program script data 2B is script text data prepared when the program of the program video data 1 is produced. The content of the program script and the utterances of the performers and narrators in the actual program are not exactly the same, but they are very close. Therefore, even if the probability calculation processing by the probability calculation processing unit 5 is performed based on the program script, good results can be obtained. In recent years, program script data may be stored as text data. In such a case, even if there is no closed caption data, the identification model database 6 is created without enormous manpower. can do.

Since time information is given to the program script data 2B, a synchronization point with the program video data 1 can be specified using this time information. Thereby, even if the program video data 1 is divided into a plurality of time intervals, text data suitable for each time interval can be provided.
Since the program script data 2B may originally have time information added at the program production stage, the time information may be used as it is, or only the time information is manually added to the program script text. You may do it.

[Fourth Embodiment]
Next, a fourth embodiment of the present invention will be described. In the following, description of parts that are the same as those of the above-described embodiment will be omitted, and only parts that are unique to this embodiment will be described.

  FIG. 10 is a block diagram illustrating a functional configuration of the identification model generation device 100C (video identification processing device) according to the present embodiment. A feature of the present embodiment is that, instead of using the closed caption data 2 in the first embodiment, voice recognition result data 2C (text data) to which time information is added is used.

  The voice recognition result data 2C is created in advance by voice recognition processing based on the corresponding program video data 1. Since the video and audio included in the program video data 1 are originally synchronized with the time information, the time information is also given to the voice recognition result data 2C using the time information.

  By using the speech recognition result data 2C as described above, even if there is no closed caption data, the identification model database can be created without enormous manpower.

[Fifth Embodiment]
Next, a fifth embodiment of the present invention will be described. In the following, description of parts that are the same as those of the above-described embodiment will be omitted, and only parts that are unique to this embodiment will be described.

  FIG. 11 is a block diagram illustrating a functional configuration of the identification model generation device 100D (image identification processing device) according to the present embodiment. The characteristic configuration of the identification model generation device 100D according to the present embodiment is as follows. That is, instead of the program video data 1 in the first embodiment, image data 1D is used in this embodiment. Further, instead of the closed caption data 2 in the first embodiment, web text data 2D (text data) is used in the present embodiment. The image data 1D and the web text data 2D are collectively referred to as web content data. The content database 4 in the first embodiment stores a pair of video data and text data for each class, whereas the content database 4D in the present embodiment stores image data and text data for each class. Multiple pairs are stored.

  Here, the image data 1D and the web text data 2D are extracted from the same web content. For example, image data 1D is data of a single image or a plurality of images displayed on a page in a single web page. Here, the image may be a still image or a moving image. The web text data 2D is text data displayed on the web page. In many cases, image data 1D and web text data 2D displayed on the same web page are semantically connected.

  As an example, an image file referred to by the “<img>” tag is used as image data 1D in one HTML (hypertext markup language) file, and the HTML file other than the tag in the HTML file is used. Let the text part be the web text data 2D.

  In the present embodiment, the video classification processing unit 3 performs the feature on the image of the image data 1D (the feature of the pixel value, the feature of the distribution of the pixel value on the screen, and the distribution of the pixel value in the time direction in the case of a moving image. Classification) based on the characteristics of

  The identification model generation apparatus 100D of the present embodiment classifies images posted on a web page or the like into classes based only on their purely image features, and performs probability calculation processing based on the web text for each class. By calculating the probability parameter value indicating the relationship between the class and the word, the identification model database 6 in which the probability parameter value is written can be created. Then, by performing the above-described identification processing using the identification model database 6, the class occurrence probability corresponding to the match can be obtained for the input document, and the class of the image matching the document is determined. be able to.

[Sixth Embodiment]
Next, a sixth embodiment of the present invention will be described. In the following, description of parts that are the same as those of the above-described embodiment will be omitted, and only parts that are unique to this embodiment will be described.

FIG. 12 is a block diagram illustrating a functional configuration of the video search apparatus according to the present embodiment. As shown in the figure, the video search device 400 (video identification processing device) includes an identification model database 6 (probability parameter value database), means for storing a document 7 (text data), means for storing video data 1E, identification Processing unit 8E, means for storing image class 9, video database 12, search device 13 (search processing unit), video search interface unit 14, and means for at least temporarily storing input search word 15 And means for storing the search result 16 at least temporarily.
Note that the means for storing the document 7, the video data 1E, the image class 9, the search word 15, and the search result 16 are realized using a semiconductor memory, a magnetic hard disk device, an optical disk device, or the like, respectively. .
The video database 12 classifies and holds video data by class.

In this video search apparatus 400, the process of the identification processing unit 8E determining the image class 9 suitable for the input document 7 while referring to the identification model database 6 has already been described with reference to FIG. Since it is the same as that described above, the description is omitted here.
The processing unique to the present embodiment by the identification processing unit 8E reads the video data 1E associated with the document 7, and associates the video data 1E with the image class 9 (maximum likelihood class) determined based on the document 7. And a document 7 pair is written in the video database 12. That is, in addition to the processing of the identification processing unit 8 in the first embodiment, the identification processing unit 8E further determines the maximum likelihood class of the input document 7 based on the calculated class occurrence probability and supports the document 7 The video data 1E is written into the video database 12 so that the attached video data 1E belongs to the determined maximum likelihood class.

  When the identification processing unit 8E performs the above-described processing, the video data 1E is classified into classes according to the characteristics of the related document 7 regardless of the characteristics of the video or image, and for each classified class. The video database 12 can be written. By repeating such processing, video data classified by class is stored in the video database 12.

  In the video database 12, text information corresponding to each class (referred to as searched text) is also recorded. As the text to be searched, for example, a word appropriately given by a person (for example, the word may be a class name of the class) is used. For example, words such as “gas range” and “batter box” are used. In addition to these, texts closely related to the class may be automatically generated and used as searched text corresponding to each class. For this purpose, for example, by referring to a synonym dictionary or concept dictionary, the word “baseball” that is associated as a superordinate concept of the words “batter”, “strike”, “curve”, etc. is automatically Extracting or automatically extracting the word “batter box” associated with the same or similar concept of those words. In addition, when extracting a word from a synonym dictionary or a concept dictionary in this way, for example, a word that is automatically generated, such as a limitation that it is a concrete noun, a limitation that it is a noun that represents a place, etc. Rules regarding text may be provided in advance.

  The video search interface unit 14 has a function of accepting input or selection of a search word (15) by the user and displaying the search result (16) to the user. The video search interface unit 14 realizes the above-described user interface by controlling, for example, a display device of a personal computer, a keyboard, a mouse, and the like.

The search device 13 receives the input search word from the video search interface unit 14, obtains a class that matches the search word, sets the obtained class as a search result class, and sets video data belonging to the search result class as a video database. 13 and passes to the video search interface unit 14 as the inspection result 16.
In addition, the process of the part which calculates | requires the class which matches based on the search term in which the search device 13 was input uses the technique of the web search engine (search engine) for text. Using this technique, the search device 13 searches the search target text corresponding to each class described above, and sets the hit class as the search result class. Note that the number of search result classes may be one or plural.

  Instead of passing the video data itself as the search result 16 to the video search interface unit 14, the search device 13 may pass the reference information for the video data hit in the search as the search result 16 to the video search interface unit. Good. Here, as reference information to the video data, for example, a video (such as a URL (Uniform Resource Locator)) indicating the location of the video data file or a database index value of the target video data in the video database 12 is used. Use information that can identify the data.

By using the video search device 400, even if the document 7 (text data) originally corresponding to the video data 1E does not contain words such as “gas range” and “batter box”, the search term “gas range” is used. When the user inputs a word such as “Battery box” or the like, the corresponding video data (that is, a gas range, a batter box, or the like is displayed) can be taken out and shown to the user.
In the above, a form in which a search target text corresponding to each class is given by a person is described, but even if this search text is given manually, each video data is individually provided. Compared to the method of assigning metadata to the, it is possible to significantly reduce labor and improve efficiency.

  In addition, all or part of the identification model generation device, the identification processing device, the identification model generation and identification processing device, and the video search device in the above-described embodiment, for example, video classification processing, probability calculation processing, identification processing, and search These functions may be realized by a computer. In that case, a program for realizing these functions may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” dynamically holds a program for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It is also possible to include those that hold a program for a certain time, such as a volatile memory inside a computer system serving as a server or client in that case. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

Although a plurality of embodiments have been described above, the present invention can also be implemented in the following modifications.
For example, the same processing as described in the above embodiment may be performed using not only program video data but also general video data, image data, or the like associated with some text data.
The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

It is a block diagram which shows the function structure of the identification model production | generation apparatus (video identification processing apparatus) by 1st Embodiment of this invention. It is the schematic which shows the data structure of the content database by the embodiment. It is the schematic which shows the data structure and example of a data of the identification model database by the embodiment. It is a flowchart which shows the procedure of the first half part (process which calculates a document vector and a frequency vector) of the process of the probability calculation process part by the embodiment. It is a block diagram which shows the function structure of the identification processing apparatus (video identification processing apparatus) by the embodiment. It is a flowchart which shows the procedure of the identification process by the identification process part of the embodiment. It is a block diagram which shows the structure of the identification model production | generation and identification processing apparatus (video identification processing apparatus) by the embodiment. It is a flowchart which shows the procedure of the process of word extraction and frequency vector preparation in the probability calculation process part and the identification process part by 2nd Embodiment of this invention. It is a block diagram which shows the function structure of the identification model production | generation apparatus (video identification processing apparatus) by 3rd Embodiment of this invention. It is a block diagram which shows the function structure of the identification model production | generation apparatus (image | video identification processing apparatus) by 4th Embodiment of this invention. It is a block diagram which shows the function structure of the identification model production | generation apparatus (image identification processing apparatus) by 5th Embodiment of this invention. It is a block diagram which shows the function structure of the video search apparatus (video identification processing apparatus) by 6th Embodiment of this invention.

Explanation of symbols

1 Program video data (video data)
1D image data 1E video data 2 closed caption data (text data)
2B Script script data (text data)
2C Voice recognition result data (text data)
2D web text data (text data)
3 Video classification processing unit 4, 4D content database 5 probability calculation processing unit 6 identification model database (probability parameter value database)
7 Document (text data)
8, 8E Identification processing unit 9 Image class 12 Video database 13 Search device (search processing unit)
100, 100B, 100C Identification model generation device (video identification processing device)
100D identification model generation device (image identification processing device)
200 Identification processing device (video identification processing device)
300 Identification Model Generation and Identification Processing Device (Video Identification Processing Device)

Claims (12)

A content database that classifies and holds video data and text data associated with the video data for each class;
A probability parameter value database that stores a probability parameter value representing a relationship between a word and the class in association with the word and the class;
Based on the appearance frequency of the word included in the text data read from the content database, the probability parameter value representing the relationship between the word and the class is calculated, the calculated probability parameter value is the word and A probability calculation processor that writes to the probability parameter value database in association with the class;
A video identification processing apparatus comprising: The video identification processing device according to claim 1,
A class occurrence that is a probability that the text data belongs to the class based on the appearance frequency of the words included in the text data and the probability parameter value read from the probability parameter value database. An identification processing unit that calculates the probability for each class,
A video identification processing apparatus, further comprising: The video identification processing device according to claim 1,
A video classification processing unit that determines a class to which the video data belongs based on the video characteristics of the input video data and writes the video data and text data associated with the video data to the content database,
A video identification processing apparatus, further comprising: A probability parameter value database that stores the probability parameter value representing the relationship between the word and the class, calculated based on the appearance frequency of the word included in the text data, in association with the word and the class;
A class occurrence that is a probability that the text data belongs to the class based on the appearance frequency of the words included in the text data and the probability parameter value read from the probability parameter value database. An identification processing unit for calculating the probability for each class;
A video identification processing apparatus comprising: The video identification processing device according to claim 4,
A video database that stores video data classified by class, and
Based on an input search term, a search result class that matches the search term is obtained, and reference information to video data belonging to the search result class or video data belonging to the search result class is obtained by referring to a video database. A search processing unit that outputs at least one of
With
Video data is associated with the input text data,
The identification processing unit further determines a maximum likelihood class of the input text data based on the calculated class occurrence probability, and the video data so that the video data belongs to the determined maximum likelihood class. Write to the video database;
A video identification processing apparatus characterized by the above.   The video identification according to any one of claims 1 to 5, wherein the word is obtained by excluding a particle, an auxiliary verb, and a symbol from a result of morphological analysis processing of the text data. Processing equipment.   6. The word according to claim 1, wherein a noun, a verb, an adjective, and an unknown word are extracted from the result of the morphological analysis process on the text data. Video identification processing device. The video data is video data of a broadcast program,
The text data is obtained as a result of voice recognition processing based on the closed caption text data of the broadcast program, the text data of the broadcast program script associated with the time information, and the video data of the broadcast program. The video identification processing device according to claim 1, wherein the video identification processing device is associated text data. A content database that classifies and stores web content data including image data and text data,
A probability parameter value database that stores a probability parameter value representing a relationship between a word and the class in association with the word and the class;
Based on the appearance frequency of the word included in the text data read from the content database, the probability parameter value representing the relationship between the word and the class is calculated, the calculated probability parameter value is the word and A probability calculation processor that writes to the probability parameter value database in association with the class;
An image identification processing device comprising: A content database that classifies and holds video data and text data associated with the video data for each class;
A probability parameter value database that stores a probability parameter value representing a relationship between a word and the class in association with the word and the class;
On a computer with
Based on the appearance frequency of the word included in the text data read from the content database, the probability parameter value representing the relationship between the word and the class is calculated, the calculated probability parameter value is the word and Probability calculation processing step to write to the probability parameter value database in association with the class,
A computer program that executes the process. A computer comprising a probability parameter value database that stores the probability parameter value representing the relationship between the word and the class, which is calculated based on the appearance frequency of the word included in the text data, in association with the word and the class,
A class occurrence that is a probability that the text data belongs to the class based on the appearance frequency of the words included in the text data and the probability parameter value read from the probability parameter value database. An identification processing step for calculating the probability for each class;
A computer program that executes the process. A content database that classifies and stores web content data including image data and text data,
A probability parameter value database that stores a probability parameter value representing a relationship between a word and the class in association with the word and the class;
On a computer with
Based on the appearance frequency of the word included in the text data read from the content database, the probability parameter value representing the relationship between the word and the class is calculated, the calculated probability parameter value is the word and Probability calculation processing step to write to the probability parameter value database in association with the class,
A computer program that executes the process.

Images