JP2008305291A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

An object of the present invention is to reduce the memory capacity while maintaining the accuracy of giving a part of speech to a word.
Based on a co-occurrence probability co-occurrence probability stored in a co-occurrence probability table 63, which is a probability that two parts of speech co-occur, a part-of-speech candidate determination unit 101 selects among words constituting a word string. Determine a part-of-speech candidate that is a candidate for a part-of-speech candidate for a word of interest that may co-occur with a part-of-speech candidate for a part-of-speech candidate for a part-of-speech that is a word adjacent to the target word before or after the target word. The word appearance probability determination unit 102 determines a word appearance probability that is a probability that the attention word of the attention part-of-speech candidate appears as a part of speech based on the co-occurrence probability of the adjacent part-of-speech candidate and the attention part-of-speech candidate, and generates a word lattice. The unit 103 generates a word lattice based on the co-occurrence probability of adjacent words in the word string and the word appearance probability of each word constituting the word string. The present invention can be applied to, for example, a morphological analysis engine.
[Selection] Figure 9

Description

  The present invention relates to an information processing device, an information processing method, and a program, and more particularly, for example, an information processing device, an information processing method, and an information processing method capable of suppressing the memory capacity while maintaining the accuracy of giving parts of speech to words. And program.

  For example, in a morphological analysis engine that assigns the part of speech of each word to each word constituting a sentence (sentence), the part of speech of each word is assigned to each word constituting the sentence by performing a morphological analysis process on the sentence. .

  That is, for example, when attention is paid to a predetermined word constituting a sentence as an attention word, when the attention word is a known word stored in the dictionary database for information necessary to give the part of speech of the word, morphological analysis is performed. The engine assigns the part of speech of the attention word to the attention word based on the dictionary database of the morphological analysis engine.

  On the other hand, when the attention word constituting the sentence is an unknown word that is not stored in the dictionary database, for example, the morphological analysis engine has n parts of speech that are obtained in advance from a learning corpus that is a sample of a sentence or the like. Based on the co-occurrence probability of n-gram, which is the probability of co-occurrence, the part of speech is given to the attention word that is an unknown word.

  That is, for example, when giving parts of speech to the words t1, t2, and t3 constituting the sentence `` t1 (known word), t2 (unknown word), t3 (known word) '', the words t1 and t3 Part of speech stored in the database is given.

  Then, the maximum 3-gram co-occurrence probability of the 3-gram co-occurrence probabilities in which the part of speech of the word t1, the part of speech candidate of the word t2, and the part of speech of the word t3 co-occur in this order is obtained. The part of speech candidate of the word t2 when the 3-gram co-occurrence probability is maximized is given to the word t2 as the part of speech of the word t2.

  In Non-Patent Document 1, by using various models such as HMM-based model, Maximum entropy model, Conditional Markov model, Conditional random fields, Cyclic dependency networks disclosed in Non-Patent Documents 2 and 3, A morpheme analysis engine that divides a sentence into morphemes and gives parts of speech and stems to each morpheme is disclosed.

Masayuki Asahara "Natural Language Processing Technology for Constructing Morphological Analyzers" International Symposium Comparative Vocabulary Research X, 2006 Taku Kudo, Atsushi Yamamoto, Yuji Matsushita “Japanese Morphological Analysis Using Conditional Random Fields” Information Processing Society of Japan Natural Language Processing Study Group SIGNL-161,2004 Sheila M. Reynolds and Jeff A. Bilmes, “Part-of-Speech Tagging using Virtual Evidence and Negative Training”, Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT / EMNLP), pages 459-466 , Vancouver, October 2005.

  By the way, for example, when giving part of speech to each word t1, t2, t3, and t4 constituting the sentence `` t1 (known word), t2 (unknown word), t3 (unknown word), t4 (known word) '', For the words t1 and t4 which are known words, the part of speech stored in the dictionary database is given.

  After that, the maximum 3-gram co-occurrence probability is calculated from the 3-gram co-occurrence probabilities in which the part of speech of the word t1, the part of speech of the word t2, and the part of speech of the word t3 co-occur in this order. And the candidate for the part of speech of the word t2 when the 3-gram co-occurrence probability is maximized is given to the word t2 as the part of speech of the word t2, and the word when the 3-gram co-occurrence probability is maximized A candidate for part of speech of t3 is given to word t3 as part of speech of word t3.

  However, when the 3-gram co-occurrence probability is used, the part of speech of the word t4, which is a known word, is not considered, and the part of speech is given to the words t2, t3, which are unknown words. May not be able to give a good part of speech.

  That is, when there are more than n-1 unknown words in a sentence, when adding parts of speech to an unknown word using n-gram co-occurrence probabilities, the accuracy of assigning the part of speech of the unknown word to the unknown word is low. Sometimes.

  Here, when giving parts of speech to the words that make up the sentence `` t1 (known word), t2 (unknown word), t3 (unknown word), t4 (known word) '', the accuracy of giving the part of speech to the unknown word To preserve the part of speech of the word t4, the part of speech of the word t1, the part of speech of the word t2, the part of speech of the word t3 is given to the words t2 and t3. Candidate, the part of speech of the word t4 is the maximum 4-gram co-occurrence probability among the 4-gram co-occurrence probabilities that co-occur in this order, and the 4-gram co-occurrence probability is maximized The part-of-speech candidate for the word t2 is assigned to the word t2 as the part-of-speech for the word t2, and the part-of-speech candidate for the word t3 when the 4-gram co-occurrence probability is maximized is assigned to the word t3 as the part-of-speech for the word t3 It is necessary to.

  Therefore, in the method of assigning the part of speech of an unknown word to an unknown word using the co-occurrence probability of n-gram, considering the case where a plurality of unknown words continue, Co-occurrence probabilities of gram…, co-occurrence probabilities of multiple n-grams such as co-occurrence probabilities of N-grams need to be stored, and the memory capacity to store the co-occurrence probabilities of multiple n-grams is large It becomes.

  The present invention has been made in view of such a situation, and it is possible to suppress the memory capacity while maintaining the accuracy of giving a part of speech to a word.

  An information processing apparatus or program according to one aspect of the present invention is a program that causes a computer to function as an information processing apparatus that generates a word lattice from a word string and an information processing apparatus that generates a word lattice from a word string. Based on the co-occurrence probability stored in the storage means in which the co-occurrence probability, which is the probability of co-occurring two parts of speech, is obtained, attention is paid to the words constituting the word string. Part-of-speech candidate determination that determines a part-of-speech candidate that is a candidate for part-of-speech of the target word that may co-occur with a part-of-speech candidate that is a candidate for part-of-speech of the target word that is adjacent to the front or back of the target word A word appearance probability which is a probability that the attention word of the attention part-of-speech candidate appears, based on a means and a co-occurrence probability of the adjacent part-of-speech candidate and the attention part-of-speech candidate. Appearance probability determining means, word lattice generation means for generating the word lattice based on the co-occurrence probability of adjacent words in the word string and the word appearance probability of each word constituting the word string; As an information processing apparatus including the information processing apparatus, a program causing a computer to function.

  The word appearance probability determining means can determine the co-occurrence probability of the adjacent part-of-speech candidate and the attention part-of-speech candidate as the word appearance probability of the attention word.

  In the case where there are a plurality of adjacent part-of-speech candidates, the word appearance probability determining means sets the maximum value of the co-occurrence probabilities of each of the plurality of adjacent part-of-speech candidates and the attention part-of-speech candidate as the word appearance of the attention word. It can be determined as a probability.

  In the case where there are a plurality of adjacent part-of-speech candidates, the word appearance probability determining means calculates a total value of the co-occurrence probabilities of each of the plurality of adjacent part-of-speech candidates and the attention part-of-speech candidate as the word appearance of the attention word. It can be determined as a probability.

  The storage means can further store a part-of-speech appearance probability which is a probability of appearance of a part-of-speech obtained in advance, and the word appearance probability determination means co-occurs the adjacent part-of-speech candidate and the attention part-of-speech candidate. The product of the probability and the part of speech appearance probability of the part of speech candidate of interest can be determined as the word appearance probability of the word of interest.

  In the case where there are a plurality of adjacent part-of-speech candidates, the word appearance probability determination means determines the maximum value of the co-occurrence probabilities of each of the plurality of adjacent part-of-speech candidates and the part-of-speech candidate, and the part-of-speech appearance of the part-of-speech candidate The product with the probability can be determined as the word appearance probability of the word of interest.

  The co-occurrence probability or the part-of-speech appearance probability stored in the storage means may be learned in advance by a learning corpus that is a sentence sample.

  An information processing apparatus according to an aspect of the present invention includes: a word part-of-speech providing unit that assigns a part of speech of the attention word to the attention word based on the word lattice generated by the word lattice generation unit; Output means for outputting the attention word given the part of speech can be further provided.

  The storage means can further store a word table in which a word and a stem of the word are associated with each other, and the attention word is added to the attention word based on the word table stored in the storage means. A stem adding unit for adding a stem of the word is further provided, and the output unit can output the attention word to which the part of speech and the word stem of the attention word are added.

  The storage means can further store a compound word table in which a compound word composed of a plurality of words and a part of speech of the compound word are associated with each other, and the compound word table stored in the storage means can store the compound word table. On the basis of the above, the compound word included in the word string is further provided with compound word part-of-speech giving means for adding the part of speech of the compound word, and the output means is further provided with the part of speech of the compound word included in the word string. The compound word can be output.

  The storage means can further store a word appearance probability, which is a probability of appearance of a word having a predetermined part of speech that is obtained in advance, and the word appearance probability determination means determines the word appearance probability of the attention word. When stored in the storage means, the word appearance probability of the attention word is determined based on the word appearance probability stored in the storage means, and the word appearance probability of the attention word is stored in the storage means. If not, the co-occurrence probability of the attention word can be determined based on the co-occurrence probability of the adjacent part-of-speech candidate and the attention part-of-speech candidate.

  The word appearance probability stored in the storage unit may be learned in advance by a learning corpus that is a sample of a sentence.

  The part-of-speech candidate determination means can determine the part-of-speech with the maximum co-occurrence probability with the adjacent part-of-speech candidate as the target part-of-speech candidate.

  An information processing method according to an aspect of the present invention is an information processing method of an information processing apparatus that generates a word lattice from a word string, and stores a co-occurrence probability that is a probability of co-occurring two parts of speech that are obtained in advance. Based on the co-occurrence probabilities stored in the stored storage means, it is a candidate for a part of speech of an adjacent word that is a word adjacent before or after the attention word of interest among the words constituting the word string. Determining a part of speech candidate that is a candidate for the part of speech of the attention word that may co-occur with a certain part of speech candidate, and based on the co-occurrence probability between the part of speech candidate and the part of speech candidate of interest, Determining a word appearance probability, which is a probability that the attention word of the attention part-of-speech candidate appears, and determining the co-occurrence probability between adjacent words in the word string and the word appearance probability of each word constituting the word string Based on the above Including the step of generating a lattice.

  In one aspect of the present invention, the word string is configured based on the co-occurrence probability stored in storage means in which a co-occurrence probability that is a probability of co-occurring two parts of speech is obtained in advance. A part of speech candidate of the attention word that may co-occur with an adjacent part of speech candidate that is a candidate for the part of speech of the adjacent word that is adjacent to the attention word before or after the attention word of interest Participant part-of-speech candidates are determined, and based on the co-occurrence probability of the adjacent part-of-speech candidate and the part-of-speech part-of-interest candidate, a word appearance probability is determined, which is the probability that the part-of-speech part of the part-of-speech part candidate will appear. The word lattice is generated based on the co-occurrence probability between adjacent words in the word string and the word appearance probability of each word constituting the word string.

  ADVANTAGE OF THE INVENTION According to this invention, memory capacity can be restrained, maintaining the precision which gives a part of speech to a word.

  Embodiments of the present invention will be described below. Correspondences between the constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

An information processing apparatus or program according to one aspect of the present invention includes:
A program that causes a computer to function as an information processing device that generates a word lattice from a word string (for example, the morphological analysis engine of FIG. 1) or an information processing device that generates a word lattice from a word string,
Storage means (for example, the dictionary database of FIG. 1) in which the co-occurrence probability (for example, the co-occurrence probability held in the co-occurrence probability table 63 of FIG. 1), which is the probability of co-occurring two parts of speech, obtained in advance is stored. 12) Adjacent that is a candidate for part of speech of an adjacent word that is adjacent to the attention word in front of or behind the attention word of the words constituting the word string based on the co-occurrence probability stored in 12) Part-of-speech candidate determination means (for example, part-of-speech candidate determination unit 101 in FIG. 9) for determining a target part-of-speech candidate that is a candidate for part-of-speech of the target word that may co-occur with a part-of-speech candidate;
Based on the co-occurrence probability of the adjacent part-of-speech candidate and the target part-of-speech candidate, word appearance probability determining means for determining the word appearance probability that the part-of-speech is the probability that the target word of the target part-of-speech candidate appears (for example, FIG. 9 Word appearance probability determination unit 102),
Word lattice generation means for generating the word lattice based on the co-occurrence probability of adjacent words in the word string and the word appearance probability of each word constituting the word string (for example, the word in FIG. 9) A program that causes a computer to function as an information processing apparatus including the lattice generation unit 103) or an information processing apparatus.

In the information processing apparatus according to one aspect of the present invention,
Based on the word lattice generated by the word lattice generation means, word part-of-speech giving means (for example, the word part-of-speech giving unit 104 in FIG. 9) for giving the part-of-speech of the word of interest to the word of interest;
Output means (for example, the compound word part-of-speech giving unit 35 in FIG. 9) that outputs the attention word to which the part-of-speech of the word of interest is attached can be further provided.

In the information processing apparatus according to one aspect of the present invention,
The storage means further stores a word table (for example, the word table 61 in FIG. 1) in which a word is associated with a stem of the word,
Based on the word table stored in the storage means, further comprising a stem grant means (for example, the stem grant unit 33 in FIG. 1) that gives the attention word a stem of the attention word;
The output means can output the attention word to which the part of speech and the word stem of the attention word are assigned.

In the information processing apparatus according to one aspect of the present invention,
The storage means further stores a compound word table (for example, the compound word table 65 in FIG. 1) in which a compound word composed of a plurality of words and the part of speech of the compound word are associated with each other.
Based on the compound word table stored in the storage means, compound word part-of-speech giving means (for example, compound word part-of-speech giving unit 35 in FIG. 1) that gives the compound word included in the word string the part of speech of the compound word. )
The output means can further output the compound word to which the part of speech of the compound word included in the word string is given.

An information processing method according to one aspect of the present invention includes:
An information processing method of an information processing apparatus (for example, the morphological analysis engine of FIG. 1) that generates a word lattice from a word string,
Storage means (for example, the dictionary database of FIG. 1) in which the co-occurrence probability (for example, the co-occurrence probability held in the co-occurrence probability table 63 of FIG. 1), which is the probability of co-occurring two parts of speech, obtained in advance is stored. 12) Adjacent that is a candidate for part of speech of an adjacent word that is adjacent to the attention word in front of or behind the attention word of the words constituting the word string based on the co-occurrence probability stored in 12) A part of speech candidate that is a candidate for part of speech of the attention word that may co-occur with the part of speech candidate (for example, step S34 in FIG. 18);
Based on the co-occurrence probability of the adjacent part-of-speech candidate and the target part-of-speech candidate, a word appearance probability is determined, which is the probability that the part-of-speech word appears as the target part-of-speech candidate (for example, step S35 in FIG. 18).
The word lattice is generated based on the co-occurrence probability of adjacent words in the word string and the word appearance probability of each word constituting the word string (for example, step S36 in FIG. 18).
Includes steps.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a first configuration example of an embodiment of a morphological analysis engine as an information processing apparatus to which the present invention is applied.

  The morphological analysis engine of FIG. 1 includes a word analysis unit 11 and a dictionary database 12.

  In the morphological analysis engine of FIG. 1, for example, when a user operates an operation unit (not shown), a document as text data of a word string such as a sentence composed of a plurality of words is input. The document is supplied to the word analysis unit 11.

  The word analysis unit 11 includes a sentence delimiter unit 31, a word delimiter unit 32, a stem adding unit 33, a part of speech adding unit 34, and a compound word part of speech adding unit 35. A process of assigning the stem and part of speech of each word is performed.

  The sentence separator 31 is supplied with the input document.

  The sentence delimiter 31 divides the document supplied thereto into sentence units based on a predetermined rule for dividing the document into sentence units.

  That is, for example, the sentence delimiter 31 regards “.” And “?” In the document supplied thereto as a delimiter between adjacent sentences, and divides the document into units of sentences. Specifically, for example, when a document “I wanted to go to United States. So, I bought a ticket” is supplied to the sentence delimiter 31, the sentence delimiter 31 converts the document supplied thereto into It is divided into two sentences, “I wanted to go to United States.” And “So, I bought a ticket.”

  The sentence delimiter 31 appropriately converts sentences such as “I wanted to go to United States.” And “So, I bought a ticket” obtained by dividing the document supplied thereto into word delimiters. 32.

  The word delimiter 32 divides the sentence supplied from the sentence delimiter 31 into words constituting the sentence based on a predetermined rule for dividing the sentence into words.

  That is, for example, the word delimiter 32 regards “,”, “” (blank character) or the like in one sentence supplied from the sentence delimiter 31 as a delimiter between adjacent words, and converts the sentence into words. To divide. Specifically, for example, when the sentence “I wanted to go to United States.” Is supplied from the sentence delimiter 31 to the word delimiter 32, the word delimiter 32 converts the sentence supplied thereto into “ It is divided into eight words: “I”, “wanted”, “to”, “go”, “to”, “United”, “States”, and “.”.

  The word delimiter 32 supplies the sentence divided into words to the stem adding unit 33.

  The word stem assigning unit 33 refers to the word table 61 stored in the dictionary database 12 in which a word, a word ID (identification) for uniquely identifying the word, and a word stem of the word are associated with each other. A stem of each word is given to each word constituting the sentence supplied from the word delimiter 32.

  Further, the word stem assigning unit 33 reads the word ID of each word constituting the sentence supplied from the word delimiter 32 from the word table 61 and associates it with the word constituting the sentence corresponding to the word ID.

  The stem granting unit 33 supplies the part-of-speech giving unit 34 with a sentence in which a stem is given to a word and a word ID is associated.

  The part-of-speech providing unit 34 stores a part-of-speech ID that uniquely identifies a predetermined part-of-speech stored in the dictionary database 12, a word ID of a word whose part-of-speech is a predetermined part-of-speech, and a probability that a word whose part-of-speech is a predetermined part-of-speech appears The word appearance probability table 62 in which the word appearance probabilities are associated with each other, the part-of-speech IDs of the two parts of speech, and the co-occurrence probability that is the probability that the two parts of speech co-occur. , A word lattice is generated from a word string as a sentence supplied from the stem adding unit 33.

  Then, the part-of-speech giving unit 34, based on the word lattice generated from the word string as the sentence supplied from the stem giving unit 33, the part of speech ID of the part of speech of each word constituting the sentence supplied from the stem giving unit 33 , And by referring to the part-of-speech table 64 in which the part-of-speech and the part-of-speech ID of the part-of-speech associated with each other are stored in the dictionary database 12. , Give part of speech for each word.

  The part-of-speech giving unit 34 supplies the compound word part-of-speech giving unit 35 with a sentence in which the part of speech is given to the word.

  When the sentence supplied from the part-of-speech giving unit 34 includes a compound word composed of a plurality of words, the compound word part-of-speech giving unit 35 stores the compound word stored in the dictionary database 12 and the compound word. By referring to the compound word table 65 in which the part of speech and the compound word ID that uniquely identifies the compound word are associated, a plurality of words that are included in the sentence supplied from the part of speech adding unit 34 and that are compound words Is given as a compound word, and the part of speech of the compound word is given to the compound word.

  In addition, the compound word part-of-speech giving unit 35 outputs the compound word included in the sentence from the part-of-speech giving unit 34 in the form of “compound word / part of speech / stem” to a monitor (not shown) or the like for display.

  Further, the compound word part-of-speech giving unit 35 displays the words that do not constitute the compound word among the words constituting the sentence supplied from the part-of-speech giving unit 34 in the form of “word / part of speech / stem”. Output and display on a monitor not shown.

  The dictionary database 12 stores a word table 61, a word appearance probability table 62, a co-occurrence probability table 63, a part-of-speech table 64, and a compound word table 65. The word analysis unit 11 stores the word table 61 in the document supplied thereto. It is referred to when a word stem and part of speech of each word are given to each word.

  In the word table 61, a word, a word ID of the word, and a stem of the word are registered in association with each other.

  In the word appearance probability table 62, a part-of-speech ID of a predetermined part of speech, a word ID of a word whose part of speech is a predetermined part of speech, and a word appearance probability of a word whose part of speech is a predetermined part of speech are registered in association with each other.

  In the co-occurrence probability table 63, two part-of-speech IDs that uniquely identify each of the two parts of speech and the co-occurrence probabilities between the two parts of speech are registered in association with each other.

  In the part of speech table 64, a part of speech and a part of speech ID of the part of speech are registered in association with each other.

  In the compound word table 65, a compound word, a part of speech of the compound word, and a compound word ID of the compound word are registered in association with each other.

  FIG. 2 is a diagram showing the word table 61 stored in the dictionary database 12 of FIG.

  In the word table 61 of FIG. 2, a word (notation), a word ID of the word, and a stem of the word are associated with each other.

  As described above, the stem assigning unit 33 assigns a stem to each word constituting the sentence supplied from the word delimiter 32 by referring to the word table 61 of FIG. .

  Specifically, in the stem giving unit 33, for example, for the sentence “I wanted to go to United States.”, Each word “I”, “wanted” constituting the sentence “I wanted to go to United States.” ”,“ To ”,“ go ”,“ to ”,“ United ”,“ States ”,“. ”, The stems“ I ”,“ want ”,“ to ”,“ go ”,“ to ”,“ “Unit”, “state”, and “.” Are assigned respectively.

  Here, if each word to which a stem is assigned is expressed in the form of “word / word stem” in the order of the word and the stem of the word, the stem grant unit 33 assigns the stem to the word. 'I / I', 'wanted / want', 'to / to', 'go / go', 'to / to', 'United / unit', 'States / state', './.' Is supplied to the part-of-speech giving unit 34.

  Next, FIG. 3 is a diagram showing the part of speech table 64 stored in the dictionary database 12.

  The part of speech table 64 in FIG. 3 associates the part of speech with the part of speech ID of the part of speech.

  For example, in the part-of-speech giving unit 34, for the sentence “I wanted to go to United States.”, Each word “I / I” to which the stem is given is included in the sentence “I wanted to go to United States.” , "Wanted / want", "to / to", "go / go", "to / to", "United / unit", "States / state", "./." “NN”, “VBD”, “TO”, “VB”, “TO”, “VBN”, “NNS”, and “ST” are respectively assigned.

  Here, if each word with a stem and part of speech is given in the form of “word / part of speech / word stem” in the order of the word, the part of speech of the word, and the stem of the word, the part of speech is given. The part 34 is a sentence “I / NN / I”, “wanted / VBD / want”, “to / TO / to”, “go / VB / go”, “to / “TO / to”, “United / VBN / unit”, “States / NNS / state”, and “./ST/.” ”Are supplied to the compound word part-of-speech adding unit 35.

  The part of speech “NN” represents a singular noun, and the part of speech “VBD” represents a past tense verb. The part of speech “TO” represents to, and the part of speech “VB” represents the present verb. The part of speech “VBN” represents a past participle verb, and the part of speech “NNS” represents a plural noun. The part of speech “ST” represents the punctuation at the end of the sentence.

  FIG. 4 is a diagram showing the compound word table 65 stored in the dictionary database 12 of FIG.

  In the compound word table 65 of FIG. 4, the compound word, the part of speech of the compound word, and the compound word ID of the compound word are associated with each other.

  For example, in the compound word part-of-speech adding unit 35, for the sentence “I wanted to go to United States.”, Each word “I wanted to go to United States. "I / NN / I", "wanted / VBD / want", "to / TO / to", "go / VB / go", "to / TO / to", "United / VBN / unit", "States / The words "United / VBN / unit" and "States / NNS / state" in "NNS / state" and "./ST/." Are combined with the word "United States" consisting of the words "United" and "States". And the part of speech “Place” of the compound word is given to the compound word “United States”.

  After that, the compound word part-of-speech giving unit 35 reads the words “I / NN / I”, “wanted / VBD / want”, “to / TO / to”, “go / VB / go”, “to / TO / to”. , "./ST/." And the compound word "united states / Place / united states" are output.

  Note that the part of speech “Place” represents a place.

  FIG. 5 is a diagram illustrating the word lattice generated by the part of speech adding unit 34.

  FIG. 5 shows a word lattice generated from the sentence “Time fly like an arrow.” By the part-of-speech giving unit 34, for example.

  The word lattice is composed of nodes and links.

  The circles (◯) shown in FIG. 5 represent nodes, and the nodes are words constituting the sentence “Time fly like an arrow.”, Part-of-speech candidates that are candidates for part-of-speech of the word, and part-of-speech is the part-of-speech. It has the word appearance probability of the candidate word. The leftmost node is a sentence head node φ representing the beginning of a sentence, and has a value 1 as a word appearance probability.

  An arrow connecting two nodes adjacent to each other in the front and back shown in FIG. 5 represents a link, and the link has a co-occurrence probability between two part-of-speech candidates that each of the two adjacent nodes has.

  The part-of-speech giving unit 34 generates the word lattice shown in FIG. 5 from the word string as the sentence supplied from the stem giving unit 33, obtains the maximum likelihood path of the word lattice, and the node on the maximum likelihood path has Each part-of-speech candidate is given to each word as part-of-speech of each word that the node on the maximum likelihood path has.

  The maximum likelihood path refers to a path of a word lattice when the following formula (1) takes the maximum value.

…（１）

... (1)

Here, 表 し represents that t is changed from 1 to T and multiplication of P (w _t | p _t ) P (p _t | p _t−1 ) is performed. Represents the total number of words that make up the supplied sentence.

Furthermore, w _t represents the t-th word from the beginning of the sentence, and p _t represents the part of speech candidate (part of speech candidate) of the word w _t .

P (w _t | p _t ) represents the word appearance probability of the word w _t whose part of speech is the part of speech candidate p _t , and P (p _t | p _t-1 ) is the part of speech candidate p of the word w _{t-1 It} represents the co-occurrence probability between _t-1 and the part-of-speech candidate p _t of the word w _t .

The co-occurrence probability P (p ₁ | p ₀ ) represents the probability that the part-of-speech candidate p ₁ of the word w _{1 is} the part-of-speech that appears first from the beginning of the sentence.

  Next, with reference to FIGS. 6 to 8, a method in which the part-of-speech giving unit 34 generates the word lattice shown in FIG. 5 will be described.

  FIG. 6 is a diagram showing a word appearance probability table 62 stored in the dictionary database 12 of FIG.

  The word appearance probability table 62 in FIG. 6 includes a part of speech ID of a predetermined part of speech, a word ID of a word of a part of speech having a predetermined part of speech, and a word appearance probability of a word having a part of speech of a predetermined part of speech. The probability of appearing as a part-of-speech word of ID) is associated.

  FIG. 7 is a diagram showing a co-occurrence probability table 63 stored in the dictionary database 12 of FIG.

  In the co-occurrence probability table 63 in FIG. 7, the previous part-of-speech ID representing a predetermined part of speech, the subsequent part-of-speech ID representing the part of speech following the predetermined part of speech, the predetermined part of speech represented by the previous part of speech ID, and the predetermined part of speech The co-occurrence probability with the part of speech represented by the subsequent part of speech ID is associated.

  FIG. 8 shows an HMM (Hidden Markov Model) equivalent to the word appearance probability table 62 of FIG. 6 and the co-occurrence probability table 63 of FIG.

  FIG. 8 is a diagram showing HMMs for the word appearance probability table 62 of FIG. 6 and the co-occurrence probability table 63 of FIG.

  The circles shown in FIG. 8 represent the state of the HMM, and the state has a predetermined part of speech, a word whose part of speech is a predetermined part of speech, and a word appearance probability of a word whose part of speech is a predetermined part of speech. Note that the predetermined part-of-speech that each state has, the word whose part-of-speech is the predetermined part-of-speech, and the word appearance probability of the word whose part-of-speech is the predetermined part-of-speech are the predetermined part-of-speech stored in the word appearance probability table 62 of FIG. The part-of-speech ID corresponds to the word ID of a word whose part of speech is a predetermined part of speech, and the word appearance probability of a word whose part of speech is a predetermined part of speech.

  The arrows connecting the states shown in FIG. 8 represent the state transitions of the HMM, and the state transitions include the previous part-of-speech ID of the part-of-speech candidate that the state before the state transition has and the subsequent part-of-speech ID of the part-of-speech candidate that the state after the state transition has. And the co-occurrence probability of the co-occurrence probability table 63 of FIG.

  The part-of-speech providing unit 34 refers to the word appearance probability table 62 in FIG. 6 and the co-occurrence probability table 63 in FIG. 7, so that the nodes and links constituting the word lattice in FIG. 5, that is, the word lattice in FIG. Is generated.

  That is, of the words constituting the sentence “Time fly like an arrow.”, For example, when the word “like” is the attention word, the part-of-speech giving unit 34 reads the stem giving unit from the word appearance probability table 62 in FIG. The part-of-speech ID and the word appearance probability associated with the word ID of the attention word “like” constituting the sentence supplied from 33 are read, and a node having the part-of-speech indicated by the part-of-speech ID and the word appearance probability is generated. .

  In FIG. 5, the part of speech ID indicating the part of speech “IN”, the word appearance probability “0.01”, and the part of speech “VB” are set as a set of the part of speech ID and the word appearance probability associated with the word ID of the attention word “like”. There are three sets of part-of-speech ID and other word appearance probability “0.006”, another part-of-speech ID indicating part-of-speech “NN”, and another word appearance probability “0.002”, part-of-speech “IN” and word appearance probability “ A node having 0.01 ”(lower center in FIG. 5), a node having part of speech“ VB ”and a word appearance probability“ 0.006 ”(middle center in FIG. 5), a part of speech“ NN ”and a word appearance probability“ 0.002 ”. The node having the upper side (upper center in FIG. 5) is generated as a node for the word “like”.

  The part-of-speech providing unit 34 generates nodes (and sentence head nodes φ) for other words constituting the sentence “Time fly like an arrow.” In the same manner as when generating a node for the word “like”.

  The part-of-speech assigning unit 34 reads out the co-occurrence probability between two part-of-speech candidates that each adjacent node has from the co-occurrence probability table 63 in FIG. 7, and generates a link (FIG. 5) having the co-occurrence probability. .

  In FIG. 5, for example, for the adjacent words “time” and “fly” constituting the sentence “Time fly like an arrow.”, The part of speech candidate NN of the word “time” and the part of speech candidate VBZ of the word “fly” There are two co-occurrence probabilities of 0.3 and the co-occurrence probability NN between the part-of-speech candidate NN of the word “time” and the part-of-speech candidate NN of the word “fly”, and the node “time / NN” A link with a co-occurrence probability of 0.3 is generated between the node "fly / VBZ" and the node "time / NN" and the node "time / NN" A link with a co-occurrence probability of 0.4 is generated between "fly / NN". The part-of-speech giving unit 34 generates the other links shown in FIG. 5 in the same manner.

  After that, the part of speech giving unit 34 gives the part of speech of each word constituting the sentence supplied from the stem giving unit 33 to each word based on the word lattice of FIG. 5 obtained by generating the node and the link. To do.

  By the way, for example, the part-of-speech giving unit 34 reads out the part-of-speech ID and the word appearance probability for the words constituting the sentence supplied from the word stem giving unit 33 from the word appearance probability table 62 in FIG. The node about the word which comprises the sentence supplied from the stem provision part 33 is produced | generated.

  However, when the attention word constituting the sentence supplied from the word stem assigning unit 33 is an unknown word whose part-of-speech ID and word appearance probability for the attention word are not held in the word appearance probability table 62 of FIG. The part-of-speech ID and the word appearance probability for the word of interest cannot be obtained from the word appearance probability table 62 of 6, and therefore a node for the word of interest cannot be generated.

  Therefore, in the part-of-speech providing unit 34 in FIG. 1, when the word of interest is a known word whose part-of-speech ID and word appearance probability for the word of interest are stored in the word appearance probability table 62 in FIG. As described above, a node for the attention word is generated from the word appearance probability table 62 in FIG. 6. When the attention word is an unknown word, the attention that is an unknown word is determined from the co-occurrence probability table 63 in FIG. 7. Generate a node for the word.

  FIG. 9 is a block diagram showing a detailed configuration example of the part-of-speech giving unit 34 in FIG.

  The part-of-speech providing unit 34 includes a part-of-speech candidate determining unit 101, a word appearance probability determining unit 102, a word lattice generating unit 103, and a word part-of-speech providing unit 104.

  Note that the part-of-speech candidate determination unit 101 of the part-of-speech giving unit 34 is supplied with a sentence composed of each word to which a stem is given from the stem giving unit 33 in FIG.

  The part-of-speech candidate determination unit 101 refers to the word appearance probability table 62 in FIG. 6 and the co-occurrence probability table 63 in FIG. 7, and sequentially selects each word constituting the sentence supplied from the stem adding unit 33 as the attention word. Then, the part of speech candidate that is a candidate for the part of speech of the attention word is determined.

  That is, the part-of-speech ID for the attention word is held in the word appearance probability table 62 in FIG. 6. Therefore, when the attention word is a known word, the part-of-speech candidate determination unit 101 stores the word appearance probability table 62 in FIG. By referencing, the part of speech indicated by the part of speech ID for the attention word is determined as the attention part of speech candidate.

  On the other hand, if the part-of-speech ID for the attention word is not held in the word appearance probability table 62 in FIG. 6, and therefore the attention word is an unknown word, the part-of-speech candidate determination unit 101 performs the co-occurrence probability table 63 in FIG. , The part of speech that may co-occur with the adjacent part of speech candidate that is a candidate for the part of speech of the adjacent word that is the word adjacent before or after the target word is determined as the target part of speech candidate.

  The part of speech candidate determination unit 101 supplies the word appearance probability determination unit 102 with the attention word for which the attention part of speech candidate has been determined as the attention word whose part of speech is the attention part of speech candidate.

  The word appearance probability determining unit 102 refers to the word appearance probability table 62 in FIG. 6 and the co-occurrence probability table 63 in FIG. 7, so that the part of speech supplied from the part of speech candidate determining unit 101 is the attention word of the target part of speech candidate. Determine the word appearance probability.

  That is, the word appearance probability of the attention word is held in the word appearance probability table 62 in FIG. 6. Therefore, when the attention word is a known word, the word appearance probability determination unit 102 selects the word appearance probability table 62 in FIG. 6. The word appearance probability of the attention word is determined by referring to.

  On the other hand, when the word appearance probability of the attention word is not held in the word appearance probability table 62 in FIG. 6 and, therefore, the attention word is an unknown word, the word appearance probability determination unit 102 performs the co-occurrence probability table in FIG. The word appearance probability of the attention word is determined by referring to the co-occurrence probability between the adjacent part-of-speech candidate and the attention part-of-speech candidate held by 63.

  After the word appearance probability of each word constituting the sentence is determined, the word appearance probability determination unit 102 supplies the word appearance probability of each word constituting the sentence to the word lattice generation unit 103.

  The word lattice generation unit 103 forms a sentence supplied from the co-occurrence probability (co-occurrence probability between adjacent words of the sentence) held in the co-occurrence probability table 63 of FIG. 7 and the word appearance probability determination unit 102. Based on the word appearance probability of each word to be generated, a word lattice (FIG. 5) is generated and supplied to the word part-of-speech adding unit 104.

  The word part-of-speech providing unit 104 obtains the maximum likelihood path of the word lattice supplied from the word lattice generation unit 103 and determines the part-of-speech ID of the part of speech of each word based on the maximum likelihood path.

  Further, the word part-of-speech giving unit 104 is supplied from the word stem giving unit 33 through the part-of-speech candidate determination unit 101, the word appearance probability determination unit 102, and the word lattice generation unit 103 with reference to the part-of-speech table 64 of FIG. Part-of-speech is assigned to each word constituting the sentence and supplied to the compound-word part-of-speech giving unit 35 in FIG. The word part-of-speech giving unit 104 is configured by each word to which a word stem is given from the word stem giving unit 33 via the part-of-speech candidate decision unit 101, the word appearance probability decision unit 102, and the word lattice generation unit 103. A sentence is supplied.

  Next, referring to FIG. 10 to FIG. 17, when the attention word is an unknown word, the word appearance probability determination unit 102 in FIG. 9 is based on the co-occurrence probability between the adjacent part of speech candidate and the attention part of speech candidate. A word appearance probability determination process for determining the word appearance probability of a target word whose part of speech is a target part of speech candidate will be described.

  FIG. 10 shows a part-of-speech candidate (hereinafter referred to as the previous adjacent word) as an adjacent part-of-speech candidate, for example, a word adjacent to the target word (hereinafter referred to as the previous adjacent word as appropriate). FIG. 10 is a diagram for explaining word appearance probability determination processing for determining the co-occurrence probability between a previous adjacent part-of-speech candidate and a target part-of-speech candidate as a word appearance probability of a target word of the target part-of-speech candidate when the adjacent part-of-speech candidate exists); It is.

  Note that the part of speech Pos of the word L is hereinafter appropriately expressed as part of speech L / Pos.

  In FIG. 10, the part of speech L1 / Pos1 is the previous adjacent part of speech candidate Pos1 of the previous adjacent word L1, and the part of speech L2 / Pos1 is the attention of the attention word L2, which may co-occur with the previous adjacent part of speech candidate L1 / Pos1. It is a part of speech candidate.

  In FIG. 10, the number 0.02 shown below the target part-of-speech candidate L2 / Pos1 indicates the word appearance probability of the target word L2 whose part-of-speech is the target part-of-speech candidate L2 / Pos1.

  In FIG. 10, there is a part of speech Pos1 as a part of speech of the word L2 that may co-occur with the previous adjacent part of speech candidate L1 / Pos1, and the co-occurrence of the previous adjacent part of speech candidate L1 / Pos1 and the target part of speech candidate L2 / Pos1. The probability P (Pos1 | Pos1) is 0.02.

  For example, when there is a previous adjacent part-of-speech candidate L1 / Pos1 as one adjacent part-of-speech candidate for one attention-part-of-speech candidate L2 / Pos1, the word appearance probability determination unit 102 pays attention to the previous adjacent part-of-speech candidate L1 / Pos1. The co-occurrence probability P (Pos1 | Pos1) = 0.02 with the part-of-speech candidate L2 / Pos1 is determined as the word appearance probability of the attention word L2 with the part-of-speech candidate L2 / Pos1.

  Next, FIG. 11 illustrates a candidate part of speech (hereinafter, referred to as a “subsequent adjoining word”, for example) adjacent to the target word as one adjacent part of speech candidate, for example, as a neighboring part of speech candidate. A word appearance probability determination process that determines the co-occurrence probability of the back adjacent part-of-speech candidate and the target part-of-speech candidate as the word appearance probability of the target word of the target part-of-speech candidate when It is a figure explaining.

  In FIG. 11, part-of-speech L3 / Pos1 is post-neighbor part-of-speech candidate Pos1 of post-neighbor word L3, and part-of-speech L2 / Pos1 is the focus of attention word L2 that may co-occur with post-neighbor part-of-speech candidate L3 / Pos1. It is a part of speech candidate.

  In FIG. 11, the number 0.01 shown below the target part-of-speech candidate L2 / Pos1 indicates the word appearance probability of the target word L2 whose part-of-speech is the target part-of-speech candidate L2 / Pos1.

  In FIG. 11, there is a part of speech Pos1 as a part of speech of the word L2 that may co-occur with the rear neighboring part of speech candidate L3 / Pos1, and the co-occurrence of the rear neighboring part of speech candidate L3 / Pos1 and the target part of speech candidate L2 / Pos1. The probability P (Pos1 | Pos1) is 0.01.

  The word appearance probability determining unit 102 selects the next adjacent part-of-speech candidate L3 / Pos1 as one adjacent part-of-speech candidate L3 / Pos1 as one adjacent part-of-speech candidate L3 / Pos1, for example. The co-occurrence probability P (Pos1 | Pos1) = 0.02 with the part-of-speech candidate L2 / Pos1 is determined as the word appearance probability of the attention word L2 with the part-of-speech candidate L2 / Pos1.

  Next, FIG. 12 shows a case where there are two adjacent part-of-speech candidates, for example, one front part-of-speech candidate and one rear part-of-speech candidate, with respect to one target part-of-speech candidate. The word appearance that determines the co-occurrence probability of the attention part-of-speech candidate and the co-occurrence probability of the next adjacent part-of-speech candidate and attention part-of-speech candidate as the word appearance probability of the attention word of the attention part-of-speech candidate It is a figure explaining a probability determination process.

  In FIG. 12, the part of speech L1 / Pos1 is the previous adjacent part of speech candidate Pos1 of the previous adjacent word L1, and the part of speech L3 / Pos1 is the subsequent adjacent part of speech candidate Pos1 of the rear adjacent word L3. The part-of-speech L2 / Pos1 is a candidate part-of-speech candidate for the target word L2, which may co-occur with the previous neighboring part-of-speech candidate L1 / Pos1 and the rear neighboring part-of-speech candidate L3 / Pos1.

  Further, in FIG. 12, the number 0.02 shown below the target part-of-speech candidate L2 / Pos1 indicates the word appearance probability of the target word L2 having the part-of-speech candidate L2 / Pos1.

  In FIG. 12, there is a part of speech Pos1 as a part of speech of the word L2 that may co-occur with the previous adjacent part of speech candidate L1 / Pos1, and the co-occurrence of the previous adjacent part of speech candidate L1 / Pos1 and the target part of speech candidate L2 / Pos1. The probability P (Pos1 | Pos1) is 0.02.

  In FIG. 12, the part of speech Pos1 exists as the part of speech of the word L2 that may co-occur with the rear adjacent part of speech candidate L3 / Pos1, and the rear adjacent part of speech candidate L3 / Pos1 and the target part of speech candidate L2 / Pos1 The co-occurrence probability P (Pos1 | Pos1) is 0.01.

  The word appearance probability determining unit 102 has, for example, one previous adjacent part-of-speech candidate L1 / Pos1 and one subsequent adjacent part-of-speech candidate L3 / Pos1 as one adjacent part-of-speech candidate for one target part-of-speech candidate L2 / Pos1. The co-occurrence probability P (Pos1 | Pos1) = 0.02 between the previous neighboring part-of-speech candidate L1 / Pos1 and the focused part-of-speech candidate L2 / Pos1, and the co-occurrence of the following neighboring part-of-speech candidate L3 / Pos1 and the focused part of speech candidate L2 / Pos1 Of the probability P (Pos1 | Pos1) = 0.01, for example, the maximum co-occurrence probability P (Pos1 | Pos1) = 0.02 is determined as the word appearance probability of the attention word L2 whose part of speech is the attention part of speech candidate L2 / Pos1.

  FIG. 13 shows the maximum co-occurrence probability among the co-occurrence probabilities of each of the plurality of previous adjacent part-of-speech candidates and the target part-of-speech candidate when there are a plurality of previous adjacent part-of-speech candidates for one target part-of-speech candidate. FIG. 11 is a diagram for explaining word appearance probability determination processing in which the part of speech is determined as the word appearance probability of the attention word of the attention part of speech candidate.

  In FIG. 13, previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 exist as a plurality of previous adjacent part-of-speech candidates that may co-occur with one notable part-of-speech candidate L2 / Pos1.

  In FIG. 13, the co-occurrence probabilities P (Pos1 | Pos1), P (Pos1 | Pos2), and P (Pos1 |) of the three previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 and the target part-of-speech candidate L2 / Pos1. Pos3) is 0.02, 0.01 and 0.5, respectively.

  When there are a plurality of previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 for one target part-of-speech candidate L2 / Pos1, the word appearance probability determination unit 102 determines three previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Co-occurrence probability P (Pos1 | Pos1) = 0.02, P (Pos1 | Pos2) = 0.01, P (Pos1 | Pos3) = 0.5 for each Pos3 and the part of speech candidate L2 / Pos1 (Pos1 | Pos3) = 0.5 is determined as the word appearance probability of the attention word L2 whose part of speech is the part of speech candidate L2 / Pos1.

  FIG. 14 shows the maximum co-occurrence probability among the co-occurrence probabilities of the target part-of-speech candidate and each of the plurality of rear-adjacent part-of-speech candidates for a single target part-of-speech candidate. FIG. 11 is a diagram for explaining word appearance probability determination processing in which the part of speech is determined as the word appearance probability of the attention word of the attention part of speech candidate.

  In FIG. 14, there are rear adjacent part of speech candidates L3 / Pos1 to L3 / Pos3 as a plurality of rear adjacent part of speech candidates that may co-occur with one notable part of speech candidate L2 / Pos1.

  In FIG. 14, the co-occurrence probabilities P (Pos1 | Pos1), P (Pos2 | Pos1), P (Pos3 | Pos1) of the target part-of-speech candidate L2 / Pos1 and the three subsequent neighboring part-of-speech candidates L3 / Pos1 to L3 / Pos3 ) Are 0.01, 0.04, and 0.4, respectively.

  The word appearance probability determining unit 102, when there are a plurality of post-adjacent part-of-speech candidates L3 / Pos1 to L3 / Pos3 for one part-of-speech candidate L2 / Pos1, the part-of-speech candidate L2 / Pos1 and three parts Co-occurrence probabilities P (Pos1 | Pos1) = 0.01, P (Pos2 | Pos1) = 0.04, P (Pos3 | Pos1) = 0.4 with the candidates L3 / Pos1 to L3 / Pos3, respectively (Pos3 | Pos1) = 0.4 is determined as the word appearance probability of the attention word L2 whose part of speech is the part of speech candidate L2 / Pos1.

  FIG. 15 shows the co-occurrence probability of each of the plurality of previous adjacent part-of-speech candidates and the target part-of-speech candidate when there are a plurality of front-adjacent part-of-speech candidates and rear-adjacent part-of-speech candidates, and the target part-of-speech candidate FIG. 6 is a diagram for explaining word appearance probability determination processing in which the maximum value co-occurrence probability among the co-occurrence probabilities with each of a plurality of subsequent adjacent part-of-speech candidates is determined as the word appearance probability of the part-of-speech candidate of interest.

  In FIG. 15, there are previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 as a plurality of previous adjacent part-of-speech candidates that may co-occur with one notable part-of-speech candidate L2 / Pos1.

  In FIG. 15, the co-occurrence probabilities P (Pos1 | Pos1), P (Pos1 | Pos2), P (Pos1 |) of the three previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 and the target part-of-speech candidate L2 / Pos1. Pos3) is 0.02, 0.01 and 0.5, respectively.

  Further, in FIG. 15, there are rear adjacent part-of-speech candidates L3 / Pos1 to L3 / Pos3 as a plurality of rear adjacent part-of-speech candidates that may co-occur with one notable part-of-speech candidate L2 / Pos1.

  In FIG. 15, the co-occurrence probabilities P (Pos1 | Pos1), P (Pos2 | Pos1), and P (Pos3 |) of the target part-of-speech candidate L2 / Pos1 and the three subsequent neighboring part-of-speech candidates L3 / Pos1 to L3 / Pos3, respectively. Pos1) is 0.01, 0.04, and 0.4, respectively.

  The word appearance probability determining unit 102 includes a plurality of previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 and a plurality of subsequent adjacent part-of-speech candidates L3 / Pos1 to L3 / Pos3 for one target part-of-speech candidate L2 / Pos1. In this case, the co-occurrence probabilities P (Pos1 | Pos1) = 0.02, P (Pos1 | Pos2) = 0.01, P (Pos1) between each of the three previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 and the target part-of-speech candidate L2 / Pos1 | Pos3) = 0.5, and the co-occurrence probabilities P (Pos1 | Pos1) = 0.01 and P (Pos2 | Pos1) = 0.04 for the candidate part-of-speech L2 / Pos1 and the three subsequent neighboring part-of-speech candidates L3 / Pos1 to L3 / Pos3 , P (Pos3 | Pos1) = 0.4, the maximum co-occurrence probability P (Pos1 | Pos3) = 0.5 is determined as the word appearance probability of the attention word L2 having the part of speech candidate L2 / Pos1.

  FIG. 16 shows the co-occurrence probability between the previous adjacent part-of-speech candidate and the target part-of-speech candidate when there are a plurality of target part-of-speech candidates that may co-occur with one previous adjacent part-of-speech candidate. It is a figure explaining the word appearance probability determination process determined as a word appearance probability of a word.

  In FIG. 16, the part of speech L1 / Pos1 is the previous adjacent part of speech candidate Pos1 of the previous adjacent word L1, and the part of speech L2 / Pos (n) is a word of interest that may co-occur with the previous adjacent part of speech candidate L1 / Pos1. L2's feature part-of-speech candidates (n = 1, 2, ..., N).

  Also, in FIG. 16, the numbers shown below the target part-of-speech candidate L2 / Pos (n) indicate the word appearance probability of the target word L2 whose part-of-speech is the target part-of-speech candidate L2 / Pos (n).

  In FIG. 16, there are a plurality of N part-of-speech Pos1, Pos2,..., PosN as part-of-speech of the word L2 that may co-occur with the previous adjacent part-of-speech candidate L1 / Pos1.

  When there are a plurality of part-of-speech candidates L2 / Pos1, L2 / Pos2,..., L2 / PosN that may co-occur with one previous neighboring part-of-speech candidate L1 / Pos1, the word appearance probability determination unit 102 In the same manner as described in the above, the co-occurrence probability P (Pos (n) | Pos1) of the previous adjacent part-of-speech candidate L1 / Pos1 and the n-th attention part-of-speech candidate L2 / Pos (n) is the nth part-of-speech. It is determined as the word appearance probability of the attention word L2 of the part of speech candidate L2 / Pos (n).

  In FIG. 16, for example, the co-occurrence probability P (Pos1 | Pos1) of the previous adjacent part-of-speech candidate L1 / Pos1 and the target part-of-speech candidate L2 / Pos1 is 0.02, and the co-occurrence probability P (Pos1 | Pos1) = 0.02. However, the part of speech is determined as the word appearance probability of the attention word L2 of the attention part of speech candidate L2 / Pos1 as it is. Note that the word appearance probability determining unit 102 similarly determines the word appearance probability of the attention word L2 of other attention part-of-speech candidates L2 / Pos2, L2 / Pos3,..., L2 / PosN.

  FIG. 17 shows the maximum value of the co-occurrence probabilities of each of the plurality of previous adjacent part-of-speech candidates and the target part-of-speech candidate when there are a plurality of target part-of-speech candidates that may co-occur with a plurality of previous adjacent part-of-speech candidates. FIG. 11 is a diagram for explaining word appearance probability determination processing in which the part of speech is determined as the word appearance probability of the attention word of the attention part of speech candidate.

  In FIG. 17, there are previous adjacent part-of-speech candidates Pos1 to Pos3 as a plurality of previous adjacent part-of-speech candidates that may co-occur with the target part-of-speech candidate L2 / Pos (n).

  Similarly to the case described with reference to FIG. 13, the word appearance probability determination unit 102 includes the co-occurrence probabilities of the three previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 and the target part-of-speech candidate L2 / Pos (n). The co-occurrence probability of the maximum value is determined as the word appearance probability of the attention word L2 whose part of speech is the part of speech candidate L2 / Pos (n).

  Next, a word analysis process in which the morphological analysis engine in FIG. 1 gives a word stem and part of speech to each word constituting a word string as a sentence will be described with reference to the flowchart in FIG.

  In step S31, the sentence delimiter 31 divides the document supplied thereto into sentence units based on a predetermined rule for dividing the document into sentence units, and the sentence obtained by the division is a word delimiter 32. The process proceeds to step S32.

  In step S32, the word delimiter 32 divides the sentence supplied from the sentence delimiter 31 into words constituting the sentence based on a predetermined rule for dividing the sentence into words, and the sentence divided into words Is supplied to the stem giving unit 33.

  After completion of the process of step S32, the process proceeds to step S33, and the stem granting unit 33 refers to the word table 61 of FIG. 2 to each word constituting the sentence supplied from the word delimiter 32. Assign a stem for each word.

  Further, in step S33, the stem adding unit 33 reads the word ID of each word constituting the sentence supplied from the word delimiter 32 from the word table 61 of FIG. 2, and constructs a sentence corresponding to the word ID. Associate with a word.

  Further, in step S33, the stem adding unit 33 supplies the sentence with the stem added to the word and the associated word ID to the part-of-speech adding unit 34, and the process proceeds to step S34.

  In steps S34 to S36, the part-of-speech providing unit 34 supplies the word appearance providing unit 33 by referring to the word appearance probability table 62 in FIG. 6 and the co-occurrence probability table 63 in FIG. A word lattice is generated from the word string as the sentence.

  That is, in step S <b> 34, the part-of-speech candidate determination unit 101 of the part-of-speech providing unit 34 performs a part-of-speech candidate determination process for determining a part-of-speech candidate for each word constituting the sentence supplied from the word giving unit 33.

  That is, in step S34, the part-of-speech candidate determination unit 101 of the part-of-speech giving unit 34 refers to the word appearance probability table 62 in FIG. 6 or the co-occurrence probability table 63 in FIG. , The target part-of-speech candidate that is a candidate for the part of speech of the target word is determined, the target word for which the target part-of-speech candidate is determined, the part of speech as the target word of the target part-of-speech candidate, This is supplied to the word appearance probability determination unit 102.

  Thereafter, the process proceeds from step S34 to step S35, and the word appearance probability determination unit 102 performs word appearance probability determination processing for determining the word appearance probability of each word constituting the sentence.

  That is, in step S35, the word appearance probability determining unit 102 refers to the word appearance probability table 62 in FIG. 6 and the co-occurrence probability table 63 in FIG. The word appearance probability of the attention word of the part of speech candidate is determined. In step S35, after the word appearance probability of each word constituting the sentence is determined, the word appearance probability determination unit 102 supplies the word appearance probability of each word constituting the sentence to the word lattice generation unit 103. Then, the process proceeds to step S36.

  In step S36, the word lattice generation unit 103 converts the co-occurrence probability held in the co-occurrence probability table 63 of FIG. 7 and the word appearance probability of each word constituting the sentence supplied from the word appearance probability determination unit 102. Based on this, a word lattice (FIG. 5) is generated and supplied to the word part-of-speech adding unit 104, and the process proceeds to step S37.

  In step S37, the word part-of-speech giving unit 104 obtains the maximum likelihood path of the word lattice supplied from the word lattice generation unit 103, and determines the part-of-speech ID of the part of speech of each word based on the maximum likelihood path. Further, in step S37, the word part-of-speech giving unit 104 refers to the part-of-speech table 64 of FIG. 3 and from the word stem giving unit 33 via the part-of-speech candidate determination unit 101, the word appearance probability determination unit 102, and the word lattice generation unit 103. Part-of-speech is assigned to each word constituting the sentence to be supplied to the compound word part-of-speech giving unit 35 in FIG.

  The word part-of-speech giving unit 104 is configured by each word to which a word stem is given from the word stem giving unit 33 via the part-of-speech candidate decision unit 101, the word appearance probability decision unit 102, and the word lattice generation unit 103. A sentence is supplied.

  Thereafter, the process proceeds from step S37 to step S38, and the compound word part-of-speech giving unit 35 stores the compound word in the dictionary database 12 when the sentence supplied from the part-of-speech giving unit 34 (word part-of-speech giving unit 104) is included. By referring to the compound word table 65, a plurality of words that are compound words included in the sentence supplied from the part-of-speech adding unit 34 are compound words, and the part of speech of the compound word is added to the compound word. The processing proceeds to step S39.

  In step S39, the compound word part-of-speech giving unit 35 outputs and displays the compound word included in the sentence from the part-of-speech giving unit 34 in the form of “compound word / part of speech / word stem” on a monitor (not shown). Of the words constituting the sentence supplied from the part-of-speech giving unit 34, the words that do not constitute the compound word are output and displayed on a monitor (not shown) in the form of “word / part of speech / word stem”. Then, the word analysis process is terminated.

  Next, the part of speech candidate determination process performed in step S34 in FIG. 18 will be described in detail with reference to the flowchart in FIG.

  In step S61, the part-of-speech candidate determination unit 101 sequentially sets each word constituting the sentence supplied from the stem adding unit 33 as the attention word, and determines whether or not the attention word is an unknown word. If it is determined in step S61 that the word of interest is an unknown word, the process proceeds to step S63, and the part-of-speech candidate determination unit 101 refers to the co-occurrence probability table 63 in FIG. The part of speech that may co-occur is determined as a target part of speech candidate, the target word whose part of speech is the target part of speech candidate is supplied to the word appearance probability determination unit 102, and the process proceeds to step S64.

  On the other hand, if it is determined in step S61 that the word of interest is not an unknown word, that is, a known word, the process proceeds to step S62, and the part of speech candidate determination unit 101 refers to the word appearance probability table 62 in FIG. By doing so, the part of speech indicated by the part of speech ID for the attention word is determined as the attention part of speech candidate, the attention word whose part of speech is the attention part of speech candidate is supplied to the word appearance probability determination unit 102, and the process proceeds to step S64.

  In step S64, the part-of-speech candidate determination unit 101 determines whether or not all the words constituting the sentence supplied from the stem adding unit 33 are the attention words. If it is determined that the words are not the attention words, the process is performed. Returning to step S61, the same processing is repeated below, using a word that has not yet been set as the attention word among the words constituting the sentence as a new attention word.

  On the other hand, when it is determined in step S64 that all the words constituting the sentence supplied from the stem adding unit 33 are the attention words, the process returns to step S34 in FIG. 18, and then proceeds to step S35. move on.

  Next, the word appearance probability determination process performed in step S35 of FIG. 18 will be described in detail with reference to the flowchart of FIG.

  In step S91, the word appearance probability determination unit 102 determines whether or not the attention word whose part of speech is the attention part of speech candidate supplied from the part of speech candidate determination unit 101 is an unknown word.

  If it is determined in step S91 that the attention word is an unknown word, the process proceeds to step S93, and the word appearance probability determination unit 102 stores the adjacent part-of-speech candidates and attention as the co-occurrence probability table 63 in FIG. By referring to the co-occurrence probability with the part of speech candidate, the word appearance probability of the attention word supplied from the part of speech candidate determination unit 101 is determined, and the process proceeds to step S94.

  On the other hand, if it is determined in step S91 that the word of interest is a known word, the process proceeds to step S92, and the word appearance probability determination unit 102 refers to the word appearance probability table 62 in FIG. The word appearance probability of the attention word supplied from the candidate determination unit 101 is determined, and the process proceeds to step S94.

  In step S94, the word appearance probability determining unit 102 determines whether or not the word appearance probability of all the words constituting the sentence has been determined, and has not determined the word appearance probability of all the words constituting the sentence. If determined, the process returns to step S91, and the same process is repeated thereafter.

  On the other hand, when it is determined in step S94 that the word appearance probabilities of all the words constituting the sentence have been determined, the word appearance probability determining unit 102 determines the word appearance probability of each word constituting the sentence as the word lattice generating unit. 103, the process returns to step S35 of FIG. 18, and then proceeds to step S36.

  As described above, in the word analysis process performed by the morphological analysis engine of FIG. 1 described with reference to the flowchart of FIG. 18, when the attention word is a known word, by referring to the word appearance probability table 62 of FIG. , The part-of-speech candidate, and the word appearance probability of the target word of the target part-of-speech candidate are determined, and if the target word is an unknown word, by referring to the co-occurrence probability table 63 of FIG. And the word appearance probability of the attention word whose part of speech is the attention part of speech candidate.

  Therefore, even when an unknown word is included in a word string as a sentence, a word lattice is generated based on the co-occurrence probability between adjacent words in the word string and the word appearance probability of each word constituting the word string. Therefore, the part of speech of the unknown word can be given to the unknown word.

  Further, even when an unknown word continues in a word string, for example, when an adjacent word adjacent to an attention word that is an unknown word is a known word, the attention word is expressed based on the adjacent part-of-speech candidate of the adjacent word. Candidate part-of-speech candidates and the word appearance probability of the target word part-of-speech candidate part-of-speech candidate are determined, so that a word lattice can be generated from a word string followed by an unknown word, and a part-of-speech can be assigned to the unknown word .

  That is, based on the adjacent part-of-speech candidate of the adjacent word adjacent to the target word that is an unknown word, the target part-of-speech candidate of the target word and the word appearance probability of the target word whose target part-of-speech is the target part-of-speech candidate are determined. Based on the part-of-speech candidate of the word, the word-of-interest candidate of the unknown word to be the next attention word and the word appearance probability of the attention word of the part-of-speech candidate of interest are determined, and the same processing is repeated thereafter. The unknown word part-of-speech candidate and the unknown word appearance probability are determined for all unknown words, so even if the unknown word continues, a word lattice can be generated, and the unknown word is given a part of speech can do.

  Furthermore, even if n-1 or more unknown words continue in the word string, the candidate part-of-speech candidate of the word of interest, which is an unknown word, and the part of speech by the co-occurrence probability (bigram) of adjacent words in the word string Since the word appearance probability of the attention word of the attention part-of-speech candidate is determined, it is selected according to the number of times the unknown word continues, and the co-occurrence probabilities of multiple n-grams used to give part-of-speech to the unknown word are stored Since it is only necessary to store the co-occurrence probabilities for adjacent words in the word string in the dictionary database 12, compared with the case of storing the co-occurrence probabilities of a plurality of n-grams. , Can save memory capacity.

  In the word analysis process performed by the morphological analysis engine of FIG. 1, for example, among the words constituting the sentence “t1 (known word), t2 (unknown word), t3 (unknown word), t4 (known word)” To give parts of speech to the unknown words t2 and t3, by generating a word lattice that takes into account the word t4 (part of speech candidate) of the known word, that is, a word lattice having a node for the word t4, Since parts of speech can be given to words t2 and t3, the part of speech of word t1, the part of speech of word t2, the part of speech of word t3 co-occur in this order without considering the part of speech of word t4 Based on the co-occurrence probability of gram, the part of speech can be accurately assigned to the unknown word as compared with the case where the part of speech is assigned to the words t2 and t3.

  Further, if the probability distributions such as the word appearance probability held in the word appearance probability table 62 in FIG. 6 and the co-occurrence probability held in the co-occurrence probability table 63 in FIG. 7 represent the probability distribution of the actual language. Since part-of-speech can be assigned based on the probability distribution of the actual language, it is possible to obtain a result as if a person gave part-of-speech to each word constituting the sentence.

  In the word appearance probability determination process described with reference to FIGS. 12 to 15, for example, the maximum co-occurrence probability among the co-occurrence probabilities of each of the plurality of adjacent part-of-speech candidates and the target part-of-speech candidate is represented by the part of speech. Although it was decided to determine the word appearance probability of the attention word of the target part-of-speech candidate, the total value of the co-occurrence probabilities of each of the plurality of adjacent part-of-speech candidates and the target part-of-speech candidate is used, and the word of the attention word whose part of speech is the target part-of-speech candidate You may make it determine as an appearance probability.

  FIG. 21 shows the total value of the co-occurrence probabilities of each of the plurality of adjacent part-of-speech candidates and the target part-of-speech candidate when there are a plurality of previous part-of-speech candidates as adjacent part-of-speech candidates for one part-of-speech candidate. FIG. 10 is a diagram for explaining word appearance probability determination processing for determining the word appearance probability of the attention word of the attention part-of-speech candidate.

  In FIG. 21, as in the case of FIG. 13, there are previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 as a plurality of previous adjacent part-of-speech candidates that may co-occur with one notable part-of-speech candidate L2 / Pos1. Yes.

  In FIG. 21, the co-occurrence probabilities P (Pos1 | Pos1), P (Pos1 | Pos2), and P (Pos1 |) of each of the three previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 and the target part-of-speech candidate L2 / Pos1. Pos3) is 0.02, 0.01 and 0.5, respectively.

  When there are a plurality of previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 for one target part-of-speech candidate L2 / Pos1, the word appearance probability determination unit 102 determines three previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Co-occurrence probabilities P (Pos1 | Pos1) = 0.02, P (Pos1 | Pos2) = 0.01, P (Pos1 | Pos3) = 0.5, 0.53 (= 0.02 + 0.01 + 0.5) is determined as the word appearance probability of the attention word L2 whose part of speech is the part of speech candidate L2 / Pos1.

  FIG. 22 illustrates the total value of the co-occurrence probabilities of each of the plurality of previous adjacent part-of-speech candidates and the target part-of-speech candidate when there are a plurality of target part-of-speech candidates that may co-occur with a plurality of previous adjacent part-of-speech candidates. FIG. 10 is a diagram for explaining word appearance probability determination processing for determining the word appearance probability of the attention word of the attention part-of-speech candidate.

  In FIG. 22, there are previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 as a plurality of previous adjacent part-of-speech candidates that may co-occur with the target part-of-speech candidate L2 / Pos (n).

  Similar to the case described with reference to FIG. 21, the word appearance probability determination unit 102 calculates the total value of the co-occurrence probabilities of the three previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 and the target part-of-speech candidate L2 / Pos (n). Is determined as the word appearance probability of the attention word L2 whose part of speech is the part of speech candidate L2 / Pos (n).

  FIG. 23 is a block diagram showing a second configuration example of an embodiment of a morphological analysis engine to which the present invention is applied.

  In the figure, portions corresponding to those in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  That is, the morphological analysis engine of FIG. 23 is provided with a part-of-speech giving unit 44 instead of the part-of-speech giving unit 34 of the word analysis unit 11, and a new part-of-speech appearance probability table 201 is stored in the dictionary database 12. The rest of the configuration is the same as in FIG.

  The part-of-speech providing unit 44 is supplied with a sentence in which a word stem is assigned to a word and a word ID is associated from the word stem assigning unit 33.

  The part of speech giving unit 44 associates the word appearance probability table 62 in FIG. 6, the co-occurrence probability table 63 in FIG. 7, the part of speech ID that uniquely identifies the part of speech with the part of speech appearance probability that is the probability that the part of speech appears. By referring to the part-of-speech appearance probability table 201, a word lattice (FIG. 5) is generated from a word string as a sentence supplied from the stem adding unit 33.

  The part-of-speech giving unit 44 then, based on the word lattice generated from the word string as the sentence supplied from the stem giving unit 33, the part of speech ID of the part of speech of each word constituting the sentence supplied from the stem giving unit 33 And part of speech of each word is assigned to each word constituting the sentence supplied from the stem adding unit 33.

  The part-of-speech giving unit 44 supplies the compound word part-of-speech giving unit 35 with a sentence in which the part of speech is given to the word.

  FIG. 24 shows a part-of-speech appearance probability table 201 stored in the dictionary database 12 of FIG.

  The part-of-speech appearance probability table 201 in FIG. 19 associates the part-of-speech ID with the part-of-speech appearance probability of the part of speech represented by the part-of-speech ID.

  Next, FIG. 25 is a block diagram illustrating a detailed configuration example of the part-of-speech providing unit 44 of FIG.

  In the figure, portions corresponding to the part-of-speech providing unit 34 in FIG. 9 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  That is, the part-of-speech providing unit 44 in FIG. 25 is configured in the same manner as the part-of-speech providing unit 34 in FIG. 9 except that a word appearance probability determining unit 302 is provided instead of the word appearance probability determining unit 102 in FIG. ing.

  Note that the word appearance probability determination unit 302 is supplied from the part of speech candidate determination unit 101 with the attention word whose part of speech is the attention part of speech candidate.

  The word appearance probability determination unit 302 is supplied from the part of speech candidate determination unit 101 by referring to the word appearance probability table 62 in FIG. 6, the co-occurrence probability table 63 in FIG. 7, and the part of speech appearance probability table 201 in FIG. The word appearance probability of the attention word whose part of speech is the attention part of speech candidate is determined.

  That is, the word appearance probability of the attention word is held in the word appearance probability table 62 in FIG. 6, and therefore, when the attention word is a known word, as in the case of the word appearance probability determination unit 102 in FIG. The appearance probability determining unit 302 determines the word appearance probability of the attention word by referring to the word appearance probability table 62 of FIG.

  On the other hand, when the word appearance probability of the attention word is not held in the word appearance probability table 62 of FIG. 6 and, therefore, the attention word is an unknown word, the word appearance probability determination unit 302 displays the co-occurrence probability table of FIG. By referring to the co-occurrence probability between the adjacent part-of-speech candidate and the target part-of-speech candidate held by 63 and the part-of-speech appearance probability of the target part-of-speech candidate held by the part-of-speech appearance probability table 201 of FIG. To decide.

  After the word appearance probability of each word constituting the sentence is determined, the word appearance probability determining unit 302 supplies the word appearance probability of each word constituting the sentence to the word lattice generation unit 103.

  Next, referring to FIG. 26 and FIG. 27, when the attention word is an unknown word, based on the co-occurrence probability of the adjacent part-of-speech candidate and the attention part-of-speech candidate, and the part-of-speech appearance probability of the attention part-of-speech candidate, FIG. A word appearance probability determination process in which the word appearance probability determination unit 302 determines the word appearance probability of an attention word whose part of speech is an attention part of speech candidate will be described.

  FIG. 26 shows the co-occurrence probability between the previous adjacent part-of-speech candidate and the target part-of-speech candidate and the part-of-speech appearance probability of the target part-of-speech candidate when there is a plurality of target part-of-speech candidates that may co-occur with one previous adjacent part-of-speech candidate FIG. 6 is a diagram for explaining a word appearance probability determination process for determining the product of the above as the word appearance probability of a target word whose part of speech is a target part of speech candidate.

  In FIG. 26, the part of speech L1 / Pos1 is the previous adjacent part of speech candidate Pos1 of the previous adjacent word L1, and the part of speech L2 / Pos (n) is a word of interest that may co-occur with the previous adjacent part of speech candidate L1 / Pos1. L2's feature part-of-speech candidates (n = 1, 2, ..., N).

  Further, in FIG. 26, the numbers shown below the target part-of-speech candidate L2 / Pos (n) indicate the word appearance probability of the target word L2 having the part-of-speech candidate L2 / Pos (n).

  In FIG. 26, there are a plurality of N parts of speech Pos1, Pos2,..., PosN as parts of speech of the word L2 that may co-occur with the parts of speech L1 / Pos1.

  Here, in the word appearance probability determination process described with reference to FIG. 16, the co-occurrence probability P (Pos (n) | of the previous adjacent part-of-speech candidate L1 / Pos1 and the n-th target part-of-speech candidate L2 / Pos (n) Pos1) is determined as the word appearance probability of the attention word L2 of the nth part-of-speech candidate L2 / Pos (n).

  In the word appearance probability determination process described with reference to FIG. 26, the co-occurrence probability P (Pos (n) | Pos1) of the previous adjacent part-of-speech candidate L1 / Pos1 and the nth part-of-speech candidate L2 / Pos (n) is calculated. This is common to the word appearance probability determination process described with reference to FIG.

  However, in the word appearance probability determination process described with reference to FIG. 26, the co-occurrence probability P (Pos (n) | Pos1 between the previous adjacent part-of-speech candidate L1 / Pos1 and the n-th part-of-speech candidate L2 / Pos (n). ) And the part-of-speech appearance probability of the n-th attention part-of-speech candidate L2 / Pos (n) is determined as the word appearance probability of the attention word L2 of the part-of-speech candidate L2 / Pos (n) Is different.

  In FIG. 26, for example, the co-occurrence probability P (Pos1 | Pos1) between the previous adjacent part-of-speech candidate L1 / Pos1 and the target part-of-speech candidate L2 / Pos1 is 0.02, and the part-of-speech appearance probability of the target part-of-speech candidate L2 / Pos1 is The product 0.02 * 0.04 of the co-occurrence probability P (Pos1 | Pos1) = 0.02 and the part of speech appearance probability 0.04 is determined as the word appearance probability of the attention word L2 of the part of speech candidate L2 / Pos1 Yes. Note that the word appearance probability determination unit 302 similarly determines the word appearance probability of the attention word L2 of the other attention part-of-speech candidates L2 / Pos1, L2 / Pos2, L2 / Pos3,..., L2 / PosN.

  FIG. 27 shows the product of the maximum value of the co-occurrence probabilities of each of the plurality of previous adjacent part-of-speech candidates and the target part-of-speech candidate and the part-of-speech appearance probability of the target part-of-speech candidate when there are a plurality of previous adjacent part-of-speech candidates. It is a figure explaining the word appearance probability determination process in which a part of speech is determined as the word appearance probability of the attention word of the attention part of speech candidate.

  In FIG. 27, there are previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 as a plurality of previous adjacent part-of-speech candidates that may co-occur with one notable part-of-speech candidate L2 / Pos1.

  In FIG. 27, the co-occurrence probabilities P (Pos1 | Pos1), P (Pos1 | Pos2), P (Pos1 |) of each of the three previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 and the target part-of-speech candidate L2 / Pos1. Pos3) is 0.02, 0.01, 0.5, respectively, and the part-of-speech appearance probability of the target part-of-speech candidate L2 / Pos1 is 0.03.

  Here, in the word appearance probability determination process described with reference to FIG. 13, if there are a plurality of previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 for one target part-of-speech candidate L2 / Pos1, Co-occurrence probabilities P (Pos1 | Pos1) = 0.02, P (Pos1 | Pos2) = 0.01, P (Pos1 | Pos3) = 0.5 for each of the previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 and the target part-of-speech candidate L2 / Pos1 The maximum value co-occurrence probability P (Pos1 | Pos3) = 0.5 is determined as the word appearance probability of the attention word L2 whose part of speech is L2 / Pos1.

  In the word appearance probability determination process described with reference to FIG. 27, when there are a plurality of previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 for one target part-of-speech candidate L2 / Pos1, three previous adjacent parts-of-speech. Co-occurrence probabilities P (Pos1 | Pos1) = 0.02, P (Pos1 | Pos2) = 0.01, P (Pos1 | Pos3) = 0.5 for each of candidates L1 / Pos1 to L1 / Pos3 and the target part-of-speech candidate L2 / Pos1 This is the same as the word appearance probability determination process described with reference to FIG. 13 in that the maximum co-occurrence probability P (Pos1 | Pos3) = 0.5 is obtained.

  However, in the word appearance probability determination process described with reference to FIG. 27, the maximum value of the co-occurrence probabilities of each of the plurality of previous adjacent part-of-speech candidates L1 / Pos1 to L1 / Pos3 and the target part-of-speech candidate L2 / Pos1 is shared. The difference is that the product 0.5 * 0.03 of the occurrence probability P (Pos1 | Pos1) = 0.5 and the part-of-speech appearance probability 0.03 of the target part-of-speech candidate L2 / Pos1 is determined as the word appearance probability of the target word L2.

  As described above, in the word analysis process performed by the morphological analysis engine of FIG. 23 described with reference to FIGS. 23 to 27 as described above, when the attention word is a known word, the word appearance probability table of FIG. 62 is used to determine the candidate part-of-speech candidate and the word appearance probability of the target word whose target part-of-speech candidate is the target part-of-speech candidate. When the target word is an unknown word, the co-occurrence probability table 63 of FIG. 7 is referred to. Thus, the part of speech candidate of interest is determined, and the word appearance probability of the attention word whose part of speech is the candidate for part of speech candidate is determined by referring to the co-occurrence probability table 63 of FIG. 7 and the part of speech appearance probability table 201 of FIG.

  Therefore, even when an unknown word is included in a word string as a sentence, a word lattice is generated based on the co-occurrence probability between adjacent words in the word string and the word appearance probability of each word constituting the word string. Therefore, the part of speech of the unknown word can be given to the unknown word.

  Further, even when an unknown word continues in a word string, for example, when an adjacent word adjacent to an attention word that is an unknown word is a known word, the attention word is expressed based on the adjacent part-of-speech candidate of the adjacent word. Candidate part-of-speech candidates and the word appearance probability of the target word part-of-speech candidate part-of-speech candidate are determined, so that a word lattice can be generated from a word string followed by an unknown word, and a part-of-speech can be assigned to the unknown word .

  That is, based on the adjacent part-of-speech candidate of the adjacent word adjacent to the target word that is an unknown word, the target part-of-speech candidate of the target word and the word appearance probability of the target word whose target part-of-speech is the target part-of-speech candidate are determined. Based on the part-of-speech candidate of the word, the word-of-interest candidate of the unknown word to be the next attention word and the word appearance probability of the attention word of the part-of-speech candidate of interest are determined, and the same processing is repeated thereafter. The unknown word part-of-speech candidate and the unknown word appearance probability are determined for all unknown words, so even if the unknown word continues, a word lattice can be generated, and the unknown word is given a part of speech can do.

  Furthermore, even if n-1 or more unknown words continue in the word string, the candidate part-of-speech candidate of the word of interest, which is an unknown word, and the part of speech by the co-occurrence probability (bigram) of adjacent words in the word string Since the word appearance probability of the attention word of the attention part-of-speech candidate is determined, it is selected according to the number of times the unknown word continues, and the co-occurrence probabilities of multiple n-grams used to give part-of-speech to the unknown word are stored The co-occurrence probability between adjacent words in the word string and the part-of-speech appearance probability of the part of speech only need to be stored in the dictionary database 12, so that the co-occurrence probabilities of a plurality of n-grams can be obtained. Compared with storing, memory capacity can be saved.

  In the word analysis process performed by the morphological analysis engine of FIG. 23, for example, among the words constituting the sentence “t1 (known word), t2 (unknown word), t3 (unknown word), t4 (known word)” To give parts of speech to the unknown words t2 and t3, by generating a word lattice that takes into account the word t4 (part of speech candidate) of the known word, that is, a word lattice having a node for the word t4, Since parts of speech can be given to words t2 and t3, the part of speech of word t1, the part of speech of word t2, the part of speech of word t3 co-occur in this order without considering the part of speech of word t4 Based on the co-occurrence probability of gram, the part of speech can be accurately assigned to the unknown word as compared with the case where the part of speech is assigned to the words t2 and t3.

  Further, probabilities such as the word appearance probability held in the word appearance probability table 62 in FIG. 6, the co-occurrence probability held in the co-occurrence probability table 63 in FIG. 7, the part of speech appearance probability held in the part of speech appearance probability table 201 in FIG. If the distribution represents the probability distribution of the actual language, the part of speech can be assigned based on the probability distribution of the actual language, so it is as if a person has given a part of speech to each word constituting the document The following results can be obtained.

  By the way, in the word appearance probability determination process described with reference to FIG. 26, the word appearance probability determination unit 302 has a co-occurrence probability P (Pos1 | Pos1) between the target part-of-speech candidate L2 / Pos1 and the previous adjacent part-of-speech candidate L1 / Pos1. The product 0.02 * 0.04 of the part-of-speech candidate L2 / Pos1 of 0.02 * 0.04 is determined as the word appearance probability of the target word L2 of the target part-of-speech candidate L2 / Pos1, for example, The part of speech appearance probability 0.04 of the part of speech candidate L2 / Pos1 may be determined as the word appearance probability of the attention word L1 of the part of speech candidate L1 / Pos1.

  FIG. 28 is a diagram for explaining word appearance probability determination processing for determining the part-of-speech appearance probability of the target part-of-speech candidate as the word appearance probability of the target word whose part-of-speech is the target part-of-speech candidate.

  In FIG. 28, the part of speech L1 / Pos1 is the previous adjacent part of speech candidate Pos1 of the previous adjacent word L1, and the part of speech L2 / Pos (n) may co-occur with the previous adjacent part of speech candidate L1 / Pos1. L2's feature part-of-speech candidates (n = 1, 2, ..., N).

  In FIG. 28, the numbers shown below the target part-of-speech candidate L2 / Pos (n) indicate the word appearance probability of the target word L2 whose part-of-speech is the target part-of-speech candidate Pos (n).

  The word appearance probability determination unit 302 determines the part-of-speech appearance probability of the target part-of-speech candidate L2 / Pos (n) as the word appearance probability of the target word whose part-of-speech is the target part-of-speech candidate L2 / Pos (n).

  In FIG. 28, for example, the part-of-speech appearance probability of the target part-of-speech candidate L2 / Pos1 is 0.04, and the part-of-speech appearance probability 0.04 is directly determined as the word appearance probability of the target word L2 of the target part-of-speech candidate L2 / Pos1. Has been. Note that the word appearance probability determination unit 302 similarly determines the word appearance probability of the attention word L2 of other attention part-of-speech candidates L2 / Pos2, L2 / Pos3,..., L2 / PosN.

  In this case, the product of the co-occurrence probability between the previous adjacent part-of-speech candidate and the target part-of-speech candidate and the part-of-speech appearance probability of the target part-of-speech candidate described with reference to FIG. Compared with the case of determining as a probability, the process of reading the co-occurrence probability between the previous adjacent part-of-speech candidate and the target part-of-speech candidate from the co-occurrence probability table 63 of FIG. 7, the co-occurrence probability, and the part-of-speech appearance probability of the target part-of-speech candidate Therefore, the word appearance probability of the attention word can be determined more quickly.

  Further, when there are a plurality of target part-of-speech candidates, all the word appearance probabilities of the target word of the plurality of target part-of-speech candidates may be determined as the same word appearance probability.

  FIG. 29 is a diagram for explaining word appearance probability determination processing for determining all word appearance probabilities of attention words of a plurality of attention part-of-speech candidates as the same word appearance probability when there are a plurality of attention part-of-speech candidates.

  In FIG. 29, the part of speech L1 / Pos1 is the previous adjacent part of speech candidate Pos1 of the previous adjacent word L1, and the part of speech L2 / Pos (n) is a word of interest that may co-occur with the previous adjacent part of speech candidate L1 / Pos1. L2's feature part-of-speech candidates (n = 1, 2, ..., N).

  Also, in FIG. 29, the numbers shown below the target part-of-speech candidate L2 / Pos (n) indicate the word appearance probability of the target word L2 whose part-of-speech is the target part-of-speech candidate L2 / Pos (n).

  The word appearance probability determining unit 102 and the word appearance probability determining unit 302 represent the word appearance probabilities of the attention word L2 whose part of speech is the attention part of speech candidate L2 / Pos (n), for example, 1 / N representing the total number of attention part of speech candidates. Are determined to be the same word appearance probability.

  In FIG. 29, the total number of candidate part-of-speech candidates is 1000. Therefore, the same word in which the word appearance probabilities of the attention word L2 having the part-of-speech candidate L2 / Pos (n) take the value 1/1000 = 0.001. It is determined as the appearance probability.

  In this case, the part-of-speech appearance probability table 201 of FIG. 24 is compared with the case where the part-of-speech appearance probability of the target part-of-speech candidate described with reference to FIG. 28 is determined as the word appearance probability of the target word of the target part-of-speech candidate. Therefore, since it is not necessary to read the type appearance probability of the target part-of-speech candidate, the word appearance probability of the target word can be determined more quickly.

  Note that in the part of speech candidate determination unit 101 in FIG. 9 (FIG. 25), when the attention word is an unknown word, there is a possibility of co-occurring with the adjacent part of speech candidates by referring to the co-occurrence probability table 63 in FIG. 7. The part of speech is determined as a candidate part-of-speech candidate. For example, when a co-occurrence probability between an adjacent part-of-speech candidate and each of a plurality of part-of-speech takes the maximum value, one part-of-speech of a plurality of parts of speech You may make it determine as a candidate (part of speech of an attention word).

  FIG. 30 shows a part of speech in which the part of speech candidate determination unit 101 determines one part of speech as a target part of speech candidate when the co-occurrence probability between the adjacent part of speech candidate and each of the plurality of parts of speech has the maximum value. It is a figure explaining candidate determination processing.

  In FIG. 30, the part of speech L1 / Pos1 is the previous adjacent part of speech candidate Pos1 of the previous adjacent word L1, and the part of speech L2 / Pos10 has the maximum co-occurrence probability between the previous adjacent part of speech candidate L1 / Pos1 and the part of speech Pos (n). The part of speech when the co-occurrence probability P (Pos10 | Pos1) = 0.2 is shown (n = 1, 2,..., 10,..., N).

  In FIG. 30, there are part of speech Pos1, Pos2, ..., Pos10, ..., PosN as part of speech of word L2, which may co-occur with previous adjacent part of speech candidate L1 / Pos1, and the previous adjacent part of speech candidate L1 / Pos1 The co-occurrence probability P (Pos10 | Pos1) with the target part-of-speech candidate L2 / Pos10 is the maximum of the co-occurrence probabilities between the previous adjacent part-of-speech candidate L1 / Pos1 and each of the plurality of target part-of-speech candidates L2 / Pos (n). The co-occurrence probability is 0.2.

  The part-of-speech candidate determination unit 101 determines one part-of-speech out of part-of-speech Pos (n) that may co-occur with the previous adjacent part-of-speech candidate L1 / Pos1 as an attention part-of-speech candidate for the attention word L1.

  In FIG. 30, for example, when the co-occurrence probability of the adjacent part-of-speech candidate L1 / Pos1 and the plurality of part-of-speech Pos (n) takes the maximum co-occurrence probability P (Pos10 | Pos1) = 0.2, a plurality of part-of-speech Pos ( The part of speech Pos10 of n) is determined as the candidate part of speech candidate.

  In the part-of-speech candidate determination process described with reference to FIG. 30, one part-of-speech is set as the target part-of-speech candidate, and therefore, the target part-of-speech candidate is always set as the part of speech of the target word. As the word appearance probability of the word, for example, the value 1 may be taken.

  Further, when the word of interest is an unknown word, the part of speech candidate determination unit 101 can determine all the parts of speech held in the part of speech table 64 of FIG.

  Next, FIG. 31 is a block diagram showing a third configuration example of an embodiment of a morphological analysis engine to which the present invention is applied.

  In the figure, portions corresponding to those in FIG. 23 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  That is, the morphological analysis engine of FIG. 31 is configured in the same manner as in FIG. 23 except that a learning unit 13 is newly provided.

  In the morphological analysis engine of FIG. 31, for example, when a user operates an operation unit (not shown), a learning corpus that is a sample of a sentence such as a document or a newspaper as text data constitutes the learning corpus. Each word is input in a form in which the part of speech of each word is given. At this time, a learning corpus is supplied to the learning unit 13.

  For example, the learning unit 13 performs the word appearance probability table 62 in FIG. 6 or the co-occurrence probability table in FIG. 7 based on the input learning corpus before the word analysis processing performed by the morphological analysis engine in FIG. 31 is performed. 63, the part-of-speech appearance probability table 201 of FIG. 24 is generated and supplied to the dictionary database 12 for storage.

  Further, the learning unit 13 generates the word appearance probability table 62 in FIG. 6, the co-occurrence probability table 63 in FIG. 7, and the part-of-speech appearance probability table 201 in FIG. 19 as necessary based on the learning corpus. May be.

  Further, for example, the word table 61 in FIG. 2, the part of speech table 64 in FIG. 3, and the compound word table 65 in FIG. 4 are generated by the learning unit 13 by using an English dictionary or the like as a learning corpus.

  The above-described word analysis processing in FIG. 18, part-of-speech candidate determination processing in FIG. 19, and word appearance probability determination processing in FIG. 20 can be executed by dedicated hardware or can be executed by software. When the word analysis process of FIG. 18, the part-of-speech candidate determination process of FIG. 19, and the word appearance probability determination process of FIG. 20 are executed by software, a program constituting the software is incorporated in dedicated hardware. Installed from a program recording medium, for example, a general-purpose personal computer capable of executing various functions by installing various computers or various programs.

  FIG. 32 is a block diagram illustrating a configuration example of a computer that performs the above-described word analysis processing of FIG. 18, part of speech candidate determination processing of FIG. 19, and word appearance probability determination processing of FIG. 20 by executing a program.

  A CPU (Central Processing Unit) 901 executes various processes according to a program stored in a ROM (Read Only Memory) 902 or a storage unit 908. A RAM (Random Access Memory) 903 appropriately stores programs executed by the CPU 901 and data. The CPU 901, ROM 902, and RAM 903 are connected to each other by a bus 904.

  An input / output interface 905 is also connected to the CPU 901 via the bus 904. Connected to the input / output interface 905 are an input unit 906 made up of a keyboard, mouse, microphone, and the like, and an output unit 907 made up of a monitor, a speaker, and the like. The CPU 901 executes various processes in response to a command input from the input unit 906. Then, the CPU 901 outputs the processing result to the output unit 907.

  The storage unit 908 connected to the input / output interface 905 includes, for example, a hard disk, and stores programs executed by the CPU 901 and various data. A communication unit 909 communicates with an external device via a network such as the Internet or a local area network.

  A drive 910 connected to the input / output interface 905 drives a removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the program or data recorded therein. Get etc. The acquired program and data are transferred to and stored in the storage unit 908 as necessary.

  As shown in FIG. 32, a program recording medium that stores a program that is installed in a computer and is ready to be executed by the computer includes a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only). Memory), DVD (Digital Versatile Disc), a removable medium 911 which is a package medium composed of a magneto-optical disk, a semiconductor memory, or the like, or a ROM 902 in which a program is temporarily or permanently stored, or a storage unit 908 It is comprised by the hard disk etc. which comprise The program is stored in the program recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 909 that is an interface such as a router or a modem as necessary. Done.

  In the present specification, the step of describing the program stored in the program recording medium is not limited to the processing performed in time series in the described order, but is not necessarily performed in time series. Or the process performed separately is also included.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

It is a block diagram which shows the 1st structural example of one Embodiment of the morphological analysis engine to which this invention is applied. It is a figure which shows the word table 61 memorize | stored in the dictionary database 12 of FIG. It is a figure which shows the part of speech table 64 memorize | stored in the dictionary database 12 of FIG. It is a figure which shows the compound word table 65 memorize | stored in the dictionary database 12 of FIG. It is a figure which shows the word lattice which the part of speech provision part 34 of FIG. 1 produces | generates. It is a figure which shows the word appearance probability table 62 memorize | stored in the dictionary database 12 of FIG. It is a figure which shows the co-occurrence probability table 63 memorize | stored in the dictionary database 12 of FIG. It is a figure which shows HMM about the word appearance probability table 62 of FIG. 6, and the co-occurrence probability table 63 of FIG. It is a block diagram which shows the detailed structural example of the part of speech provision part 34 of FIG. It is a figure explaining the word appearance probability determination process which determines the co-occurrence probability of a previous adjacent part of speech candidate and an attention part of speech candidate as a word appearance probability of an attention word. It is a figure explaining the word appearance probability determination process which determines the co-occurrence probability of an attention part of speech candidate and a back adjacent part of speech candidate as a word appearance probability of an attention word. Word appearance probability that determines the co-occurrence probability between the previous adjacent part-of-speech candidate and the target part-of-speech candidate, and one of the co-occurrence probabilities between the target part-of-speech candidate and the subsequent adjacent part-of-speech candidate as the word appearance probability of the target word It is a figure explaining a determination process. It is a figure explaining the word appearance probability determination process which determines the co-occurrence probability of the maximum value among the co-occurrence probabilities of each of a plurality of previous adjacent part-of-speech candidates and the attention part-of-speech candidate as the word appearance probability of the attention word. It is a figure explaining the word appearance probability determination process which determines the co-occurrence probability of the maximum value among the co-occurrence probabilities of the attention part-of-speech candidate and each of a plurality of subsequent adjacent part-of-speech candidates as the word appearance probability of the attention word. The figure explaining the word appearance probability determination process which determines the co-occurrence probability of the maximum value among the co-occurrence probabilities of each of the plurality of previous adjacent part-of-speech candidates and rear adjacent part-of-speech candidates and the attention part-of-speech candidate as the word appearance probability of the attention word. It is. It is a figure explaining the word appearance probability determination process which determines the co-occurrence probability of a previous adjacent part of speech candidate and an attention part of speech candidate as a word appearance probability of an attention word. It is a figure explaining the word appearance probability determination process which determines the maximum value among the co-occurrence probabilities of each of a plurality of previous adjacent part-of-speech candidates and the attention part-of-speech candidate as the word appearance probability of the attention word. It is a flowchart explaining a word analysis process. It is a flowchart explaining the part of speech candidate determination process of step S34 of FIG. It is a flowchart explaining the word appearance probability determination process of step S35 of FIG. It is a 1st figure explaining the word appearance probability determination process which determines the total value of the co-occurrence probability of each of several adjacent part-of-speech candidates and an attention part-of-speech candidate as a word appearance probability of an attention word. It is a 2nd figure explaining the word appearance probability determination process which determines the total value of the co-occurrence probability of each of several previous adjacent part-of-speech candidates and an attention part-of-speech candidate as a word appearance probability of an attention word. It is a block diagram which shows the 2nd structural example of one Embodiment of the morphological analysis engine to which this invention is applied. It is a figure which shows the part of speech appearance probability table 201 memorize | stored in the dictionary database 12 of FIG. It is a block diagram which shows the detailed structural example of the part of speech provision part 44 of FIG. It is a figure explaining the word appearance probability determination process which determines the product of the co-occurrence probability of an adjacent part of speech candidate and an attention part of speech candidate and the part of speech appearance probability of an attention part of speech candidate as a word appearance probability of an attention word. Explains the word appearance probability determination process that determines the product of the maximum value of the co-occurrence probabilities of each of a plurality of adjacent part-of-speech candidates and the part-of-speech candidate and the part-of-speech appearance probability of the part-of-speech candidate as the word appearance probability of the target word It is a figure to do. It is a figure explaining the word appearance probability determination process which determines the part of speech appearance probability of an attention part of speech candidate as a word appearance probability of an attention word. It is a figure explaining the word appearance probability determination process which determines each word appearance probability of an attention word as the same word appearance probability. It is a figure explaining the word appearance probability determination process which determines one part of speech of a some part of speech as an attention part of speech candidate when the co-occurrence probability of an adjacent part of speech candidate and each of a plurality of part of speech takes the maximum value. It is a block diagram which shows the 3rd structural example of one Embodiment of the morphological analysis engine to which this invention is applied. It is a block diagram which shows the structural example of a computer.

Explanation of symbols

  11 word analysis section, 12 dictionary database, 31 sentence delimiter section, 32 word delimiter section, 33 word stem assigning section, 34 part of speech assignment section, 35 compound word part of speech assignment section, 44 part of speech assignment section, 61 word table, 62 word appearance probability table , 63 Co-occurrence probability table, 64 part-of-speech table, 65 compound word table, 101 part-of-speech candidate determination unit, 102 word appearance probability determination unit, 103 word lattice generation unit, 104 word part-of-speech assignment unit, 201 part-of-speech appearance probability table, 302 word appearance Probability determiner

Claims (15)

In an information processing apparatus that generates a word lattice from a word string,
Based on the co-occurrence probabilities stored in the storage means in which the co-occurrence probabilities, which are the probabilities of two parts of speech co-occurring, are stored in advance, pay attention to the words constituting the word string. Part-of-speech candidates for determining a part-of-speech candidate that is a candidate for part-of-speech for the target word that may co-occur with a part-of-speech candidate for a part-of-speech that is a candidate for part-of-speech that is a word adjacent to the front or back of the target word A determination means;
Word appearance probability determining means for determining a word appearance probability that is a probability that the attention word of the attention part of speech candidate appears, based on the co-occurrence probability of the adjacent part of speech candidate and the attention part of speech candidate;
An information processing apparatus comprising: a word lattice generation unit configured to generate the word lattice based on the co-occurrence probability of adjacent words in the word string and the word appearance probability of each word constituting the word string. The information processing apparatus according to claim 1, wherein the word appearance probability determining unit determines a co-occurrence probability between the adjacent part-of-speech candidate and the attention part-of-speech candidate as the word appearance probability of the attention word. In the case where there are a plurality of adjacent part of speech candidates,
The information according to claim 1, wherein the word appearance probability determining unit determines a maximum value among co-occurrence probabilities of each of the plurality of adjacent part-of-speech candidates and the attention part-of-speech candidate as the word appearance probability of the attention word. Processing equipment. In the case where there are a plurality of adjacent part of speech candidates,
2. The information according to claim 1, wherein the word appearance probability determining unit determines a total value obtained by summing up the co-occurrence probabilities of each of the plurality of adjacent part-of-speech candidates and the attention part-of-speech candidate as the word appearance probability of the attention word. Processing equipment. The storage means further stores a part-of-speech appearance probability which is a probability of appearance of a part-of-speech obtained in advance,
The word appearance probability determining means determines a product of a co-occurrence probability of the adjacent part-of-speech candidate and the target part-of-speech candidate and a part-of-speech appearance probability of the target part-of-speech candidate as the word appearance probability of the target word. The information processing apparatus according to 1. In the case where there are a plurality of adjacent part of speech candidates,
The word appearance probability determining means calculates the product of the maximum value of the co-occurrence probabilities of each of the plurality of adjacent part-of-speech candidates and the target part-of-speech candidate and the part-of-speech appearance probability of the target part-of-speech candidate. The information processing apparatus according to claim 5, wherein the information processing apparatus determines the word appearance probability. The information processing apparatus according to claim 5, wherein the co-occurrence probability or the part-of-speech appearance probability stored in the storage unit is learned in advance by a learning corpus that is a sentence sample. Based on the word lattice generated by the word lattice generating means, word part-of-speech giving means for giving the part-of-speech of the attention word to the attention word;
The information processing apparatus according to claim 1, further comprising: an output unit that outputs the attention word to which the part of speech of the attention word is assigned. The storage means further stores a word table in which a word is associated with a stem of the word,
Based on the word table stored in the storage means, further comprising a stem grant means for giving a stem of the attention word to the attention word;
The information processing apparatus according to claim 8, wherein the output unit outputs the attention word to which a part of speech and a stem of the attention word are assigned. The storage means further stores a compound word table in which a compound word composed of a plurality of words and a part of speech of the compound word are associated with each other,
Based on the compound word table stored in the storage means, further comprising compound word part-of-speech giving means for giving a part of speech of the compound word to the compound word included in the word string,
The information processing apparatus according to claim 8, wherein the output unit further outputs the compound word to which the part of speech of the compound word included in the word string is assigned. The storage means further stores a word appearance probability obtained in advance, which is a probability that a word having a predetermined part of speech appears.
The word appearance probability determining means is
When the word appearance probability of the attention word is stored in the storage means, the word appearance probability of the attention word is determined based on the word appearance probability stored in the storage means,
The co-occurrence probability of the attention word is determined based on the co-occurrence probability of the adjacent part-of-speech candidate and the attention part-of-speech candidate when the word appearance probability of the attention word is not stored in the storage unit. The information processing apparatus described. The information processing apparatus according to claim 11, wherein the word appearance probability stored in the storage unit is learned in advance by a learning corpus that is a sample of a sentence. The information processing apparatus according to claim 1, wherein the part-of-speech candidate determination unit determines a part-of-speech having the maximum co-occurrence probability with the adjacent part-of-speech candidate as the target part-of-speech candidate. In an information processing method of an information processing apparatus that generates a word lattice from a word string,
Based on the co-occurrence probabilities stored in the storage means in which the co-occurrence probabilities, which are the probabilities of two parts of speech co-occurring, are stored in advance, pay attention to the words constituting the word string. Determining a part of speech candidate that is a candidate for part of speech of the attention word that may co-occur with a part of speech candidate that is a candidate for part of speech of the adjacent word that is a word adjacent to the front or back of the attention word;
Based on the co-occurrence probability of the adjacent part-of-speech candidate and the attention part-of-speech candidate, determine a word appearance probability that is the probability that the part-of-speech word appears in the attention part-of-speech candidate;
An information processing method comprising: generating the word lattice based on the co-occurrence probability of adjacent words in the word string and the word appearance probability of each word constituting the word string. In a program that causes a computer to function as an information processing device that generates a word lattice from a word string,
Based on the co-occurrence probabilities stored in the storage means in which the co-occurrence probabilities, which are the probabilities of two parts of speech co-occurring, are stored in advance, pay attention to the words constituting the word string. Part-of-speech candidates for determining a part-of-speech candidate that is a candidate for part-of-speech for the target word that may co-occur with a part-of-speech candidate for a part-of-speech that is a candidate for part-of-speech that is a word adjacent to the front or back of the target word A determination means;
Word appearance probability determining means for determining a word appearance probability that is a probability that the attention word of the attention part of speech candidate appears, based on the co-occurrence probability of the adjacent part of speech candidate and the attention part of speech candidate;
A computer functioning as word lattice generation means for generating the word lattice based on the co-occurrence probability of adjacent words in the word string and the word appearance probability of each word constituting the word string Program to make.

Images