JP4579638B2 - Data search apparatus and data search method - Google Patents

Description

  The present invention relates to an apparatus and method for retrieving data based on speech recognition result annotation data obtained by speech recognition of speech annotation data attached to data.

  In recent years, the spread of digital cameras and the like has been remarkable. In general, a user manages a digital image captured by a portable imaging device such as a digital camera using a PC or a server. For example, photographed images can be organized in folders on a PC or server, or specific images can be printed and incorporated into New Year's cards. In addition, when managed by the server, some images can be disclosed to other users.

  When performing such work, it is necessary to find a specific image intended by the user. When the number of images to be found is small, the images can be displayed as thumbnails, and can be found visually from the list. However, when the number of target images is hundreds, or when the target image group is divided and stored in a plurality of folders, it is difficult to find it visually.

  In view of this, an audio annotation (an audio annotation) is attached to an image on the imaging apparatus, and the information used at the time of search is used. For example, an image of a mountain is picked up by a portable imaging device, and “Hakone no Yama” is uttered to the image. This audio data is stored in the imaging device as a pair with the previous image data, and then recognized in the imaging device or in the PC to which the image is uploaded, and converted to text information “Hakoneyama”. To do. If the voice annotation data is converted into text information, it can be processed by a general text search technique, and the image can be searched for using words such as “Yama” and “Hakone”.

Prior art using such voice annotation includes Patent Literature 1, Patent Literature 2, and Patent Literature 3. In these prior arts, a user inputs a voice to be annotated when an image is taken or after the image is taken, and the voice data is used for image retrieval using an existing voice recognition technology.
JP 2003-219327 A JP 2002-325225 A JP-A-9-135417

  Misrecognition is unavoidable when searching by converting voice annotation by voice recognition. If the percentage of errors due to misrecognition is large, even if the search key is correctly inserted, the matching correlation is poor and the search is not performed correctly. However, even if only a small part of the annotation is wrong due to misrecognition, it can often be searched correctly if the majority is correct.

  Here, when the search images are ranked and arranged using the correlation degree of matching with the search key as a score, annotations with high correlation with the search key are often correctly searched for higher in the ranking. Images that have a low correlation with the search key are less likely to be different from other annotations, and the ranking drops rapidly. For images with low correlation, it is difficult to search for images in the order of rank. Rather than arranging images in order of name, time, etc. like conventional folders, do not display search results. It is preferable to notify the search failure.

In order to solve the above-described problems, the data search method of the present invention searches for desired data from a plurality of data in which each data is stored in association with predetermined audio data, and the search result is displayed on the display means. A data search method for a data search device to be displayed, the storage step for storing the plurality of data and predetermined voice data associated with each of the plurality of data, and voice recognition of each of the voice data An acquisition step of acquiring the first phoneme string obtained by performing the input, an input step of inputting a search key corresponding to a search condition in response to an operation by the user, and a word string obtained by performing morphological analysis on the search key divided into further reading is given to the word sequence, a conversion step to obtain the second phoneme sequence, for said obtained from each of the audio data first phoneme sequence, the phoneme Ma By performing quenching, a determination step of determining a correlation between the second phoneme sequence, respectively, the audio data to which the degree of correlation corresponding to said first phoneme string is above a predetermined threshold are correlated Rutotomoni is displayed side by side the data to rank order of ranking in the correlation, said data voice data is associated by the correlation degree corresponding to said first phoneme string is less than the threshold value, the name of the data A display control step of controlling the display means to display the data according to any one of the name order, the time information order of the data, the data size order of the data, and the display size order of the data . Further, the data search method of the present invention is a data search apparatus for searching for desired data from a plurality of data in which each data is stored in association with predetermined audio data and displaying the search result on a display means. A data search method, which is obtained by accumulating the plurality of data and predetermined audio data associated with each of the plurality of data, and recognizing each of the audio data. An obtaining step for obtaining a first word string; an input step for inputting a search key corresponding to a search condition in response to an operation by a user; and a conversion for obtaining a second word string by performing morphological analysis on the search key. Steps and word matching are performed on the first word string obtained from each of the audio data, thereby determining the degree of correlation with the second word string. And determining and displaying the data associated with the voice data corresponding to the first word string having the correlation degree equal to or greater than a predetermined threshold in order of ranking ranked by the correlation degree, and the correlation degree is The data associated with the speech data corresponding to the first word string that is less than the threshold is in the order of the name of the data, the order of the time information included in the data, the order of the data size of the data, the data A display control step of controlling the display means so that the display means are arranged and displayed according to any of the display size orders.

  As described above, in the data search in the present invention, the search result display method is switched depending on the score representing the correlation between the speech recognition result annotation data, which is the result of speech recognition of the data in the speech corresponding to the data, and the search condition. As a result, it is possible to distinguish and search for data that appears in a higher order and data that does not appear in a higher order, leading to improved user convenience.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram illustrating functions of a data search apparatus according to an embodiment of the present invention. In the figure, reference numeral 101 denotes a database. Reference numeral 102 denotes data such as images and documents stored in the database 101. Reference numeral 103 denotes voice annotation data that is annotated with voice corresponding to the data 102. Reference numeral 104 denotes speech recognition result annotation data obtained by speech recognition of the speech annotation data 103 and converted into a phoneme string or a word string. A search key input unit 105 inputs a search key as a search condition in order to search for desired data 102. A search key conversion unit 106 converts the search key into a phoneme string or a word string in the same format as the speech recognition result annotation data 104 in order to match with the check key.

  Reference numeral 107 denotes a match between the plurality of speech recognition result annotation data 104 in the database 101 and the search key, obtains a correlation score for each speech recognition result annotation data 104, and data 102 corresponding to the speech recognition result annotation data 104. It is a search part which ranks. A display switching unit 108 switches the data display method based on the correlation score of each data. Reference numeral 109 denotes a display unit that displays the data 102 based on the method designated by the display switching unit 108.

  The processing flow of the present embodiment will be described in detail with reference to FIG. The data 102 such as an image or a document includes corresponding voice annotation data 103 and voice recognition result annotation data 104 that is a result of voice recognition of the voice annotation data. Here, the voice recognition result annotation data may be created by having the voice recognition unit in the apparatus, or may be created by having the voice recognition unit on another apparatus such as a camera for taking an image. It doesn't matter. In addition, since the annotation data used in the data search in the present invention is the speech recognition result annotation data 104, the speech annotation data 103 may not exist.

  FIG. 2 shows an example of the voice recognition result annotation data 104. Reference numeral 201 in the figure denotes a recognition result phoneme string obtained by voice recognition of the voice annotation data 103 and converted into a phoneme string, and the top five most likely lines are arranged in order. Reference numeral 202 denotes a grammar name used for speech recognition. In this embodiment, the grammar for converting to a phoneme string will be described. However, the grammar for converting to a word string may be used. Reference numeral 203 denotes a recognition likelihood representing the likelihood of a phoneme string for speech. Of these pieces of information, only the recognition result phoneme string 201 is used in the data search in this embodiment, so that the grammar name 202 and the recognition likelihood 203 are not necessary.

  First, the user inputs a search key as a search condition in the search key input unit 105. FIG. 3 shows an example of a search key input dialog presented to the user in the search key input unit. In such a dialog, the user inputs a text or a word or sentence corresponding to the voice or a part of the data to be searched. When a search key is input and a search button is pressed, the search key is transmitted to the search key conversion unit 106, and the search key is converted into a phoneme string having the same format as the recognition result phoneme string 201. FIG. 4 is a diagram showing how the search key is converted into a phoneme string. The search key “Hakone no Yama” is morphologically analyzed and divided into word strings. Further, reading is given to the word string to obtain a phoneme string. A general natural language processing technique is applied to the morphological analysis and reading method.

Next, in the search unit 107, phoneme matching is performed between the phoneme string of the search key and the speech recognition result annotation data 104 corresponding to all data 102 to be searched, and the phoneme correct answer accuracy representing the degree of correlation with the search key is obtained. Ask. Matching can be performed by a general DP matching method or the like. FIG. 5 shows how the phoneme correct answer accuracy is obtained. The correct phoneme accuracy is obtained when the number of correct answers, the number of insertion errors, the number of deletion errors, the number of replacement errors is obtained by DP matching, etc.
{(Number of correct answers)-(Number of insertion errors)-(Number of deletion errors)-(Number of replacement errors)} × 100 / (Number of correct answers)
Ask for. In FIG. 5, there are two insertion errors, “o” and “a”, and one erroneous replacement error with “h” being “f”, and the correct phoneme accuracy is 75%. The data 102 is ranked as a score used for the search with the phoneme correct accuracy obtained in this way. Here, the speech recognition result annotation data in FIG. 2 includes the top five recognition result phoneme strings, and matching is performed with each of them to obtain the correct phoneme accuracy, and the best phoneme correct answer accuracy and the recognition result phoneme string are adopted. . However, the present invention is not limited to this, and the maximum value may be taken or the sum may be taken after applying the weighting coefficient to the correct phoneme accuracy according to the order. In addition, the speech recognition result annotation data is not limited to the form of holding the top N recognition results as shown in FIG. 2, but a lattice (word graph) composed of each phoneme (or word) is output, and the beginning of the lattice The correct phoneme accuracy may be obtained in each path from to the end.

  Next, in the display switching unit 108, the phoneme correct answer accuracy corresponding to each data 102 is compared with a threshold value, and data whose phoneme correct answer accuracy is equal to or higher than the threshold value are arranged in the order of ranking ranked by the phoneme correct answer accuracy in the display unit 109. Displayed data that is less than the threshold is different from the score order such as the order of the name of the data, the order of the name of the data, the order of the time information of the data, the order of the data size of the data, and the order of the data display size Display by reference.

  FIG. 6 shows how the search data is displayed. In the figure, reference numeral 601 denotes a search result display window in which data (here, images) whose phoneme accuracy with the search key is equal to or higher than a threshold value are displayed in order of rank. Reference numeral 602 denotes a data display window in which data whose phoneme accuracy is less than a threshold is displayed by a method other than the order of order such as name order or time order. In FIG. 6, images are arranged and displayed in the order of names in the data display window. You can also hear the corresponding voice annotation by pressing the note button. The user first looks at the data with high phoneme accuracy displayed in the search result display window 601. If there is no desired data, the user selects the desired data from the data arranged in the order of name, time, etc. in the data display window. Search for. In this embodiment, the windows are divided and displayed by the data arranged in the order of the order and the other data. However, the present invention is not limited to this. For example, the areas may be divided and displayed in the same window. In this way, users can first view a limited number of rank-ordered data by arranging data with high subword accuracy in rank order and displaying other data in parallel in name order and time order. If it is not there, it can be used in combination with the usual search by name and time as usual, which increases convenience.

  Here, the phoneme correct answer accuracy threshold setting method based on the relationship between the phoneme correct answer accuracy and the data search performance and the effectiveness of display method switching by threshold processing will be described. FIG. 7 is a scatter diagram in which the phoneme correct answer accuracy with the search key for the desired correct answer data and the search rank in which the correct answer data are ranked by the phoneme correct answer accuracy are plotted when 1000 data is searched. As shown in the figure, for data with a correct phoneme accuracy of over 60%, the search order is concentrated in a good rank, and there is no data that greatly drops the rank. On the other hand, data with a phoneme accuracy of less than 60% falls greatly in rank, and the rank range varies greatly depending on the data. Therefore, while correct data with a correct phoneme accuracy of over 60% can be robustly searched at the top, data with a phoneme accuracy of under 60% is greatly reduced in rank, and the range of the ranking is greatly influenced by the data. Therefore, it is not preferable to present it as a search result.

  Therefore, taking advantage of the characteristics confirmed in this preliminary experiment, the threshold value of phoneme accuracy is set to 60%. The correct answer data exceeding the phoneme correct accuracy of 60% (actually, all data exceeding the phoneme correct answer accuracy of 60% is displayed in the order of rank in the search result display window 601 because the system does not know the correct answer). A correct answer in which the phoneme correct answer accuracy with the search key exceeds a threshold value of 60% can be searched for in a robustly high order in the search result display window 601. For data that does not exceed the accuracy of phoneme accuracy of 60%, even if it is arranged in the order of rank, it does not know in which rank range the search efficiency is rather bad. The data is displayed according to a standard different from the score order such as the data size order and the data display size order.

  That is, if a system designer prepares a search set in which correct data corresponding to a search key is known in advance and creates a scatter diagram as shown in FIG. 7, the graph shape of FIG. An appropriate threshold value can be set according to the features of the above, and a hybrid user presentation of a limited number of rank order displays by search and normal displays such as name order and time order can be performed.

  FIG. 8 shows a hardware configuration diagram for realizing the data search apparatus of the present invention. In the figure, reference numeral 801 denotes a screen display unit such as a display for displaying data, a GUI panel, and the like. Reference numeral 802 denotes a data input unit such as a keyboard and a mouse for inputting a search key or pressing a GUI button. Reference numeral 803 denotes a sound output unit such as a speaker that outputs sound such as sound annotation data and warning sound. Reference numeral 804 denotes an external storage unit such as a ROM or a hard disk that holds the database 101 and a program for this data search method. Reference numeral 805 denotes a RAM for holding temporary information such as programs and data when the data search method is executed. Reference numeral 806 denotes a CPU that executes a program of this data search method.

(Other examples)
In the above embodiment, the phoneme correct answer accuracy is used as a search score by phoneme string matching. However, the present invention is not limited to this, for example, the correct answer accuracy by syllable matching instead of phoneme, or word unit matching. But it ’s okay. In addition, the recognition likelihood 203 obtained by speech recognition may be added to this, or the score may be weighted using the similarity between phonemes (eg, “p” and “t” are high in similarity). . Further, in the above embodiment, as shown in FIG. 5, the correct phoneme accuracy based on the entire phoneme string matching is used as a search score. However, the search key is improved by devising the score to suppress the deterioration of the score due to an insertion error. You may search by partial matching. In this embodiment, for example, when the annotation “Hakone no Yama” is added to the speech recognition result annotation data, it is possible to search by using “Hakone” and “Mountain” as search keys by partial matching.

(Other examples)
In the above-described embodiment, the phoneme correct answer accuracy threshold value processing is used to switch between the sorting in the order of rank and the other sorts and display them in another region. However, the present invention is not limited to this, and the phoneme correct answer accuracy threshold value process The present invention is applicable to all embodiments that switch the data display method. For example, not all data with a correct phoneme accuracy of less than a threshold is displayed, only data with a correct phoneme accuracy of a threshold is displayed, or only data with a correct accuracy of the phoneme of a threshold is displayed with a large image An embodiment in which only a small icon or link text is displayed is also conceivable.

(Other examples)
Note that the present invention may be applied to an apparatus composed of one device or a system composed of a plurality of devices. Further, a recording medium in which a program code of software for realizing the functions of the above-described embodiments is recorded is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus is stored in the recording medium. Needless to say, this can also be achieved by reading and executing the code. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  In the above embodiment, the case where the program is stored in the ROM has been described. However, the present invention is not limited to this and may be realized using any storage medium. Further, it may be realized by a circuit that performs the same operation.

  As a recording medium for supplying the program code, for example, a floppy (registered trademark) disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like is used. be able to.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS running on the computer performs actual processing based on an instruction of the program code. Needless to say, a case where the function of the above-described embodiment is realized by performing part or all of the processing, is also included.

  Further, after the program code read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

It is a functional block diagram of the data search device of the present invention. It is an example of the speech recognition result annotation data in the present invention. It is an example of the search key input dialog in the search key input part of this invention. It is a figure showing the mode of the process in the search key conversion part of this invention. It is a figure showing the phoneme matching in the search part of this invention. It is an example of the search result display window and data display window in the display part of this invention. It is a scatter diagram showing the relationship between phoneme correct answer precision and data search order. It is a hardware block diagram which implement | achieves the data search device of this invention.

Claims (7)

A data search method for a data search apparatus for searching for desired data from a plurality of data stored in association with predetermined audio data and displaying a search result on a display means.
An accumulation step of accumulating the plurality of data and predetermined audio data associated with each of the plurality of data;
An acquisition step of acquiring a first phoneme string obtained by recognizing each of the audio data;
An input step for inputting a search key corresponding to the search condition in response to an operation by the user;
A conversion step of obtaining a second phoneme string by morphologically analyzing the search key and dividing it into word strings , further adding readings to the word strings ;
A determination step of determining a degree of correlation with the second phoneme sequence by performing phoneme matching on the first phoneme sequence obtained from each of the speech data ;
The Rutotomoni is displayed side by side the data voice data is associated to the degree of correlation corresponding to said first phoneme string is above a predetermined threshold ranking order of ranking in the correlation, the correlation is below the threshold value The data associated with the speech data corresponding to the first phoneme sequence is the name order of the data name, the time information order of the data, the data size order of the data, the display size of the data A data search method comprising a display control step for controlling the display means so that the display means are arranged and displayed according to any one of the order . A data search method for a data search apparatus for searching for desired data from a plurality of data stored in association with predetermined audio data and displaying a search result on a display means.
An accumulation step of accumulating the plurality of data and predetermined audio data associated with each of the plurality of data;
An obtaining step of obtaining a first word string obtained by recognizing each of the audio data;
An input step for inputting a search key corresponding to the search condition in response to an operation by the user;
A conversion step of obtaining a second word string by morphological analysis of the search key;
A determination step of determining a degree of correlation with the second word string by performing word matching on the first word string obtained from each of the voice data;
The data associated with the voice data corresponding to the first word string having the correlation degree equal to or greater than a predetermined threshold is displayed side by side in the order ranked by the correlation degree, and the correlation degree is less than the threshold value. The data associated with the audio data corresponding to the first word string is arranged in the order of the name of the data, the order of time information included in the data, the order of the data size of the data, the order of the display size of the data A data search method comprising a display control step of controlling the display means so that the display means are arranged and displayed according to any of the above.   In the display control step, the data associated with the speech data corresponding to the first phoneme sequence having the correlation degree equal to or greater than a predetermined threshold is arranged in the order ranked by the correlation degree and displayed in the first window. And the data associated with the speech data corresponding to the first phoneme string having the correlation degree less than the threshold value, in the order of the name of the data, the order of the time information included in the data, the data 2. The data according to claim 1, wherein the display unit is controlled to display the second data in a second window different from the first window in accordance with either the data size order or the data display size order. retrieval method. The program which makes a computer perform the data search method of any one of Claim 1 thru | or 3 . A computer-readable storage medium storing the program according to claim 4 . A data search device for searching for desired data from a plurality of data stored in association with predetermined audio data, and displaying a search result on a display means,
Storage means for storing the plurality of data and predetermined audio data associated with each of the plurality of data;
Obtaining means for obtaining a first phoneme string obtained by voice recognition of each of the voice data;
An input means for inputting a search key corresponding to the search condition in response to an operation by the user;
Morphological analysis of the search key to divide it into word strings , further adding a reading to the word strings, and converting means for obtaining a second phoneme string ;
Determining means for determining the degree of correlation with the second phoneme string by performing phoneme matching on the first phoneme string obtained from each of the speech data ;
The Rutotomoni is displayed side by side the data voice data is associated to the degree of correlation corresponding to said first phoneme string is above a predetermined threshold ranking order of ranking in the correlation, the correlation is below the threshold value The data associated with the speech data corresponding to the first phoneme sequence is the name order of the data name, the time information order of the data, the data size order of the data, the display size of the data A data search apparatus comprising display control means for controlling the display means so that the display means are arranged and displayed according to any one of the order . A data search device for searching for desired data from a plurality of data stored in association with predetermined audio data, and displaying a search result on a display means,
Storage means for storing the plurality of data and predetermined audio data associated with each of the plurality of data;
Obtaining means for obtaining a first word string obtained by voice recognition of each of the voice data;
An input means for inputting a search key corresponding to the search condition in response to an operation by the user;
Conversion means for obtaining a second word string by morphological analysis of the search key;
Determining means for determining a degree of correlation with the second word string by performing word matching on the first word string obtained from each of the voice data;
The data associated with the voice data corresponding to the first word string having the correlation degree equal to or greater than a predetermined threshold is displayed side by side in the order ranked by the correlation degree, and the correlation degree is less than the threshold value. The data associated with the audio data corresponding to the first word string is arranged in the order of the name of the data, the order of time information included in the data, the order of the data size of the data, the order of the display size of the data A data search apparatus comprising display control means for controlling the display means so that the display means are arranged and displayed according to any of the above.

Images