TWI452475B - A dictionary generating device, a dictionary generating method, a dictionary generating program product, and a computer readable memory medium storing the program - Google Patents

Description

Dictionary generating device, dictionary generating method, dictionary generating program product, and computer readable recording medium for memorizing the program

One aspect of the present invention relates to an apparatus, method, program, and computer readable recording medium for generating a word dictionary.

Since the prior art, a technique of dividing a plurality of words using a word dictionary to obtain a plurality of words (word division) has been known. The following Patent Document 1 associated therewith describes a technique in which a word that is compared with a partial character string of an input text is searched from a word dictionary, and is generated as a word candidate, and a partial character string of the input text that has not been compared with the word dictionary is generated. Selecting the possibility of unknown language as the candidate of the unknown language, using the unknown language model to infer the candidate word class of the unknown word to distinguish the word appearance probability, and using the dynamic programming method to obtain the word column with the highest probability at the same time.

Prior technical literature Patent literature

Patent Document 1: Japanese Patent Laid-Open Publication No. 2001-051996

In order to correctly segment the text, it is desirable to prepare a large number of words in the dictionary in order to enrich the vocabulary knowledge. However, it is not easy to build a large-scale dictionary by human hand. Therefore, it is required to easily construct a large-scale word dictionary.

A dictionary generating apparatus according to an aspect of the present invention includes: a model generating unit that generates a word segmentation model using a corpus and a word group prepared in advance, and Each text contained in the corpus is given information indicating the boundary of the word boundary; the analysis unit performs word segmentation incorporating the word segmentation model on the collected set of texts, and assigns boundary information to each text; the selection unit itself The word to be registered in the dictionary is selected from the text of the limit information given by the analysis unit, and the registration unit registers the word selected by the selection unit in the dictionary.

A dictionary generating method according to an aspect of the present invention is a dictionary generating device executor comprising: a model generating step of generating a word segmentation model using a corpus and a word group prepared in advance, and assigning a representation to each text included in the corpus a boundary information boundary; a parsing step that performs a word segmentation of a word segmentation model on a collection of collected texts, and assigns boundary information to each text; and a selection step that is given boundary information from the parsing step The word to be registered in the dictionary is selected in the text; and the login step is to register the word selected in the selection step in the dictionary.

The dictionary generating program of one aspect of the present invention causes the computer to execute: a model generating unit that generates a word segmentation model using a corpus and a word group prepared in advance, and assigns information indicating a boundary of a word to each text included in the corpus; And performing a word segmentation with a word segmentation model on the collected set of texts, and assigning boundary information to each text; the selection unit selects a word to be registered in the dictionary from the text of the boundary information given by the analysis unit And a registration unit that registers the word selected by the selection unit in the dictionary.

A computer-readable recording medium according to an embodiment of the present invention is a memory dictionary generating program that causes a computer to execute: a model generating unit that generates a word segmentation model using a corpus and a word group prepared in advance, and Each of the texts included in the corpus is given a limit information indicating a limit of a word; the analysis unit performs a word segmentation incorporating a word segmentation model on the collected set of texts, and assigns boundary information to each of the texts; The word to be registered in the dictionary is selected from the text of the limit information given by the analysis unit; and the registration unit registers the word selected by the selection unit in the dictionary.

According to such a form, a word segmentation model is generated using a corpus and a word group to which the boundary information is given, and a word segmentation in which the model is incorporated is applied to the text collection. Further, a word is selected from a set of texts to which the application limits the information, and is registered in the dictionary. In this way, by using the corpus of the bound information, the boundary information is also added to the text collection, and the words extracted from the text collection are registered, so that a large-scale word dictionary can be easily constructed.

In the dictionary generating device of the other embodiment, the selection unit may select a word to be registered in the dictionary based on the frequency of occurrence of each word calculated from the limit information given by the analyzing unit. The accuracy of the dictionary can be improved by considering the frequency of occurrence thus calculated.

Further, in the other form of the dictionary generating device, the selection unit may select a word whose frequency is equal to or greater than a specific threshold. Only words that have appeared more than a certain number of times are registered in the dictionary, so that the accuracy of the dictionary can be improved.

Further, in the dictionary generating apparatus of another aspect, the selection unit extracts a word whose frequency is equal to or greater than the threshold as a registration candidate, and selects a specific number of words from the registration candidate in order from the word having a high frequency of occurrence, and the registration unit may select the word. The word selected by the department is added to the dictionary in which the word group is recorded. Only words with a relatively high frequency are registered in the dictionary, thereby improving the accuracy of the dictionary. In addition, a word is added to the dictionary of the word group prepared in advance, so that Make the composition of the dictionary simple.

Further, in the dictionary generating apparatus of another aspect, the selection unit extracts a word whose frequency is equal to or greater than the threshold as a registration candidate, and selects a specific number of words from the registration candidate in order from the word having a high frequency of occurrence, and the registration unit may select the word. The words selected by the department are recorded in other dictionaries different from the dictionary in which the word group is recorded. Only the words with relatively high frequency are registered in the dictionary, so that the accuracy of the dictionary can be improved. Further, a word is added to a dictionary different from a dictionary (available dictionary) of a word group prepared in advance, and a dictionary having characteristics different from the existing dictionary can be generated.

Further, in the dictionary generation device of another aspect, the registration unit may register the word selected by the selection unit in a dictionary different from the dictionary in which the word group is recorded. A dictionary is added to a dictionary different from the dictionary of the word group prepared in advance (the existing dictionary), so that a dictionary having characteristics different from the existing dictionary can be generated.

Further, in the dictionary generating apparatus of another aspect, the selection unit extracts a word whose appearance frequency is equal to or greater than the threshold value as a registration candidate, and groups the words of the registration candidate according to the frequency of occurrence, and the registration unit may generate the plural number generated by the selection unit. Each group is individually registered in a plurality of dictionaries different from the dictionary in which the word group is recorded. The words are grouped according to the frequency of occurrence, and each of the generated groups is registered in each dictionary, so that a plurality of dictionaries different in characteristics due to the frequency of occurrence can be generated.

Further, in the dictionary generating device of another form, the information indicating the field of the text is associated with each of the collected texts, and the registration unit may individually register the word selected by the selecting unit based on the field containing the text of the word. A dictionary of field preparations. By generating a dictionary for each field, A plurality of dictionaries whose characteristics are different from each other can be generated.

Further, in the dictionary generating apparatus of another aspect, the boundary information includes the first information indicating that there is no boundary between the characters, the second information indicating that there is a limit between the positions of the characters, and the third meaning indicating that there is a limit between the positions of the characters. Information; the frequency of occurrence of each word can also be calculated based on the first, second and third information. Since it is not simply a matter of whether or not there is a limit, but a third information indicating the intermediate concept is introduced, the text can be more appropriately divided into a plurality of words.

Further, in the dictionary generating apparatus of another aspect, the analyzing unit includes a first binary classifier and a second binary classifier, and the first binary classifier determines whether to assign the first information or assign the first information to the position between the characters. In the information, the second binary classifier may determine whether to assign the second information or assign the third information to the inter-text position determination by which the first binary scoring device determines that the information other than the first information is assigned. The boundary information is determined step by step using a plurality of binary distributors, so that the boundary information can be given to the text at high speed and effectively.

Further, in another form of the dictionary generating device, the collected text set is divided into a plurality of groups; the analysis unit, the selection unit, and the registration unit perform processing based on one of the plurality of groups, and the model generation unit uses the corpus, The word group and the word registered by the registration unit generate a word segmentation model, and then the analysis unit, the selection unit, and the registration unit may perform processing based on the other of the plurality of groups.

According to one aspect of the present invention, a large-scale word dictionary can be easily constructed.

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are designated by the same reference numerals, and the repeated description is omitted.

First, the functional configuration of the dictionary generating apparatus 10 of the embodiment will be described with reference to Figs. 1 to 3 . The dictionary generating device 10 is a computer that analyzes a collection of a large amount of collected text (hereinafter also referred to as "large-scale text"), extracts words from the text collection, and adds the extracted words to the dictionary. .

As shown in FIG. 1, the dictionary generating device 10 includes a CPU 101 that executes a work system or an application program, a main memory unit 102 composed of a ROM and a RAM, an auxiliary memory unit 103 composed of a hard disk or the like, and a network card or the like. The communication control unit 104, the input device 105 such as a keyboard or a mouse, and the output device 106 such as a display.

Each functional component of the dictionary generating device 10 described later is realized by reading a specific software on the CPU 101 or the main memory unit 102, and causing the communication control unit 104, the input device 105, and the output device under the control of the CPU 101. The operation of 106 performs reading and writing of data in the main storage unit 102 or the auxiliary storage unit 103. The data or database required for processing is stored in the main memory unit 102 or the auxiliary memory unit 103. Further, although the dictionary generating device 10 is constituted by one computer in Fig. 1, the function of the dictionary generating device 10 may be distributed to a plurality of computers.

As shown in FIG. 2, the dictionary generation device 10 includes a model generation unit 11 as a functional component, an analysis unit 12, a selection unit 13, and a registration unit 14. When the dictionary generating device 10 executes the word extraction process, it refers to the learning corpus prepared in advance. The library 20, the existing dictionary 31, and the large-scale text 40 store the extracted words in the word dictionary 30. Furthermore, the word dictionary 30 includes at least the existing dictionary 31, and may further include one or more additional dictionaries 32. Before the detailed description of the dictionary generating device 10, the data will be described.

The learning corpus 20 is a set of texts that are provided (associated) with the boundary information (comment) indicating the boundary of the word (the divided position when the word is divided into words), and is prepared in advance as a database. A text consists of a plurality of words or strings. In the present embodiment, a text of a specific number randomly extracted from the title and description of the product stored in the website of the virtual shopping mall is used as the material of the learning corpus 20.

The extracted text is bounded by the evaluator's hand. The setting of the boundary information is based on two technical implementations of word segmentation using point speculation and a three-stage segmentation corpus.

[Using point speculation word segmentation]

The word boundary label b = b ₁ b ₂ ... b _n is assigned to the text (string) x = x ₁ x ₂ ... x _n (x ₁ , x ₂ ..., x _n is a character). Here, b _i is a label indicating whether or not a word boundary exists between the characters x _i and x _i+1 (the position between characters), b _i =1 means division, and b _i =0 means non-segmentation. Here, the value indicated by the label b _i can also be said to be the strength of the division.

Figure 3 shows " buy . ("bo-rupen wo katta" in Japanese ("I) bought a ballpoint pen." (n)" and " An example of a label between (wo). The value of the word boundary tag is determined by reference to the feature obtained from the surrounding text. For example, the value of the word boundary tag is set using three characteristics of a character characteristic, a character type characteristic, and a dictionary characteristic.

The character characteristic is a characteristic shown by a combination of all the characters (n-gram) of the length b below the length b _i or the inner packet limit b _i and the relative position with respect to b _i . For example, when n=3 is set in FIG. 3, it is obtained for " (n)" and " (wo) the boundary between the two, "-1/ (n)""1/ (wo)"-2/ (pen)"-1/ (n wo)""1/ Buy (wo ka)"-3/ (rupen)"-2/ (pen wo)"-1/ Buy (n wo ka)"1/ buy 9 characteristics of (wo kat).

The character type attribute is the same as the above character except that it replaces the word processing type. As the type of text, consider Hiragana, Katakana, Chinese characters, uppercase letters, lowercase letters, Arabic numerals, Chinese characters, and Chinese black (.). Furthermore, there is no limit to the type of text used and the number of types thereof.

The dictionary characteristic indicates whether the word of the length j (1≦j≦k) located around the boundary exists in the dictionary. The dictionary characteristic is represented by a combination of a graph indicating that the limit b _i exists at the end point (L) of the word, or at the start point (R) thereof, or within the word (M), and the length j of the word. If there is a login in the dictionary (pen)"" The word (wo)" produces the dictionary characteristics of L2 and R1 for the boundary b _i of Fig. 3. Further, when a plurality of dictionaries are used as will be described later, the dictionary characteristics are assigned to the dictionary characteristics. For example, if the identifier is DIC1, the dictionary A is registered. (pen)", in the dictionary B whose identification is DIC2, " (wo)", the dictionary characteristics are expressed as DIC1-L2, DIC2-R1, and the like.

Furthermore, in the present embodiment, the character characteristics and the character type characteristics are n- The maximum length n of the gram is set to 3, and the maximum length k of the dictionary feature is set to 8, but the value can be determined at will.

[3-stage word segmentation corpus]

In Japanese, there is a single word that is difficult to determine the word boundary, and there is a problem that the appropriate word segmentation pattern is different depending on the situation. As an example, imagine the inclusion of " (bo-rupen)" (English for "ballpoint pen") The text collection of words for keyword search. If not split (bo-rupen)", even if " Key search for (pen) (English "pen") also fails to extract text (reduced reproduction rate). On the other hand, will (bo-rupen)" is divided into " (bo-ru)"("ball" in English) and " In the case of (pen), by using sporting goods (bo-ru)" as a keyword search, resulting in extraction containing " (bo-rupen) text (decreased precision).

Therefore, as described above, a three-stage word segmentation corpus that introduces the concept of "half-segmentation" of non-"split" and "non-segmented" values is used. The 3-stage word segmentation corpus is a method for probabilistic word segmentation that uses a probabilistic value to represent a segmentation pattern. The intensity of the word segmentation that is actually recognizable by human beings is only a few stages, and the three-stage word segmentation corpus is used based on the reason that the necessity of representing the segmentation pattern is low with continuous probability values. For a word containing a half-divided word, both the word and the constituent elements of the word are extracted. Therefore, it is possible to record a word that is difficult for a person to judge whether to divide or not to be divided as a half-segment, and the boundary information is added. easily. "Semi-segmentation" means the probability of a positional boundary between words (more than 0 or less than 1) There is one aspect of the probability range.

The 3-stage word segmentation corpus is a three-stage segmentation discrete probability word segmentation by adding "half-segmentation" (b _i =0.5) to "segmentation" (b _i =1) and "non-segmentation" (b _i =0). And the generated corpus. For example, it also contains such as " / (bo-ru/pen)" or a compound / (ori/tatamu)" (English "fold") compound verb, " / The segmentation of the word (o/susume)" (English for "recommendation") and the lexicalization of the word (in this case, "/") is naturally defined as a semi-segmentation. Also, "juudenchi"("rechargeablebattery" in English) can be said to be "AB"("rechargeable" in English (recharge) and "denchi"("battery" in English). BC→ABC” compound, but such words are semi-divided in the form of “juu/den/chi”.

" buy . The text of (bo-rupen wo katta) is divided by a word segmentation and a three-stage word segmentation corpus estimated by the above-mentioned points, for example, as shown in FIG. In the example of Figure 3, the word boundary label of "split" (b _i =1) is given to the front of the text or " (n)" and " Wait between (wo). The word boundary label of "half-split" (b _i =0.5) is assigned to " (ru)" and " Between (pe). In Figure 3, the word boundary label of "non-segmentation" (b _i =0) is omitted, but the boundary between the characters is not shown (for example, " (pe)" and " The label is given between (n)".

A word boundary tag as boundary information is given to each text, and is stored in the database as the learning corpus 20. The method of assigning boundary information to text is arbitrary. As an example, "segmentation" is represented by space, and is represented by a hyphen "Semi-segmentation" can also embed the boundary information in each text by omitting the representation of "non-segmentation". At this time, the text to which the bounded information is given can be recorded while maintaining the character string.

The existing dictionary 31 is a collection of words of a specific number and is prepared in advance as a database. The existing dictionary 31 may be an electronic dictionary generally used, and may be, for example, a morphological analysis dictionary of UniDic.

The collection of texts collected by the large-scale text 40 is prepared in advance as a database. Any word or character string corresponding to the word to be extracted or the field of the word may be included in the large-scale text 40. For example, the title and description of the product may be collected in large quantities from the website of the virtual shopping mall, and the large-scale text 40 may be constructed from such insufficient information. The number of texts prepared as large-scale text 40 is overwhelming more than the amount of text contained in learning corpus 20.

The functional components of the dictionary generating device 10 will be described above as a premise.

The model generation unit 11 is a mechanism for generating a word segmentation model using the learning corpus 20 and the word dictionary 30. The model generation unit 11 includes a vector support device (SVM: Support Vector Machine), and inputs the learning corpus 20 and the word dictionary 30 to the device, and executes learning processing to generate a word segmentation model. The word segmentation model indicates a rule for how text should be divided, and is output as a parameter group used for word segmentation. Furthermore, the calculations used in mechanical learning are not limited to SVMs, but also to decision trees or logistic regression.

To analyze the large-scale text 40, the model generation unit 11 causes the SVM to perform learning based on the learning corpus 20 and the existing dictionary 31, thereby generating an initial word segmentation model (reference model). Then, the model generation unit 11 outputs the word division model to the analysis unit 12.

Then, when a word is added to the word dictionary 30 by the processing of the analysis unit 12, the selection unit 13, and the registration unit 14, which will be described later, the model generation unit 11 causes the SVM to perform learning based on the entire learning corpus 20 and the word dictionary 30 (again Learning) processing to produce a modified word segmentation model. Here, the entire word dictionary 30 means the words originally memorized in the existing dictionary 31 and all the words obtained from the large-scale text 40.

The analysis unit 12 executes a mechanism for analyzing the large-scale text 40 (word division) into a word segmentation model, and assigning (attached) the boundary information to each text. As a result, a large amount of text as shown in FIG. 3 is obtained. The analysis unit 12 performs such word segmentation on each of the texts forming the large-scale text 40, thereby indicating the above-mentioned "division" (second information), "half-segmentation" (third information), and "non-segmentation" (first information). The boundary information is given to each text, and all the processed text is output to the selection unit 13.

The analysis unit 12 includes two binary classifiers, and sequentially uses the classifiers to assign three pieces of limit information to each text. The first classifier judges that the position between characters is "non-segmentation" or the other mechanism, and the second classifier determines a mechanism that is determined to be "non-segmented" as "divided" or "semi-segmented". Since the position between characters is actually "non-segmented", it is first determined whether the position between characters is "non-segmentation", and then the division is determined for the portion other than the "non-segmentation", so that the boundary can be efficiently Information is given to a large amount of text. Further, the configuration of the analyzing unit 12 can be simplified by combining the binary classifiers.

The selection unit 13 is a mechanism for selecting a word registered in the word dictionary 30 from the text to which the analysis unit 12 gives the limit information.

First, the selection unit 13 obtains the total occurrence frequency f _r (w) of each word w included in the input text group based on the following formula (1). This calculation means that the frequency of occurrence is obtained from the boundary information b _i given to the position between the characters.

Here, O ₁ represents the occurrence of the writing of the word w as defined below.

Figure 3 buy . (bo-rupen wo katta) the word "a word" The frequency of occurrence of (bo-rupen)" is 1.0*1.0*1.0*0.5*1.0*1.0=0.5, the word of the phrase " The appearance frequency of (pen) is 0.5*1.0*1.0=0.5. This means to be treated as "in the phrase" (bo-rupen)" and " The word "(pen)" appears every 0.5 times. The selection unit 13 obtains the frequency of occurrence of each word included in each text, and sums the frequency of occurrence of each word, thereby obtaining the total appearance frequency of each word.

Then, the selection unit 13 selects only the words whose total occurrence frequency is equal to or greater than the first threshold value THa from the word group in the large-scale text 40 as the registration candidate V (the truncation of the word according to the frequency). Then, the selection unit 13 selects a word that is finally registered in the word dictionary 30 from the registration candidate V, and determines a dictionary (database) in which the word is stored, if necessary. The method of determining the final word and the dictionary of the storage terminal is not limited to one, and various methods as described below can be used.

The selection unit 13 may also determine that only words whose total occurrence frequency of the registration candidate V is equal to or greater than a certain threshold value are added to the existing dictionary 31. At this time, the selection unit 13 can select only the words whose total occurrence frequency is the second threshold THb (where THb>THa), or can select only the words whose total appearance frequency is up to the upper n bits. The following process is also referred to as "APPEND".

Alternatively, the selection unit 13 may determine that only the words whose total occurrence frequency of the registration candidate V is equal to or greater than a certain threshold value are registered in the additional dictionary 32. At this time, the selection unit 13 can select only the words whose total occurrence frequency is the second threshold THb (where THb>THa), or can select only the words whose total occurrence frequency is up to the upper n bits. The following process is also referred to as "TOP".

Alternatively, the selection unit 13 may determine that all the registration candidates V are registered in the additional dictionary 32. Hereinafter, such processing is referred to as "ALL".

Alternatively, the selection unit 13 may determine that the registration candidate V is divided into a plurality of partial sets based on the total appearance frequency, and each partial collection is registered in the individual additional dictionary 32. A set of parts in which the total occurrence frequency of the registration candidate V reaches the upper n bits is denoted as V _n . In this case, selection unit 13, for example generating section 1000 of the upper set of words containing the V _1000, including the upper portion 2000 of the words of the set V _2000, including the upper portion 3000 of the words of the set V _3000. Then, the selection unit 13 determines to register the partial sets V ₁₀₀₀ , V _{2000 ,} and V ₃₀₀₀ in the first additional dictionary 32 , the second additional dictionary 32 , and the third additional dictionary 32 . Furthermore, the number of partial collections generated or the size of each partial collection can also be arbitrarily determined. Hereinafter, such processing is referred to as "MULTI".

When the word finally registered is selected and the dictionary of the storage end is determined, the selection unit 13 outputs the selection result to the registration unit 14.

The registration unit 14 is a mechanism for registering the word selected by the selection unit 13 in the word dictionary 30. The dictionary entry word in the word dictionary 30 depends on the processing under the selection unit 13, and therefore the registration unit 14 may register a word only in the existing dictionary 31, or may register a word only in one additional dictionary 32. In the case of the above-mentioned "MULTI" processing, the registration unit 14 divides the selected word into a plurality of additional dictionaries 32 and registers them.

As described above, the word added to the word dictionary 30 is used for the correction of the word segmentation model, but the word dictionary 30 may be used for purposes other than word segmentation. For example, the word dictionary 30 may be used for morphological analysis, display of an input candidate sentence of an input box having an automatic input function, or a knowledge database for extracting a proper noun.

Next, the operation of the dictionary generating device 10 will be described with reference to Fig. 4, and the dictionary generating method of the present embodiment will be described.

First, the model generation unit 11 performs learning based on the learning corpus 20 and the existing dictionary 31 in the SVM, thereby generating an initial word segmentation model (reference model) (step S11, model generation step). Next, the analysis unit 12 performs analysis (word division) in which the reference model is incorporated in the large-scale text 40, and assigns (associates) the boundary information indicating "segmentation", "half-segmentation", or "non-segmentation" to each text. (Step S12, analysis step).

Then, the selection unit 13 selects a word registered in the dictionary (selection step). Specifically, the selection unit 13 calculates the total appearance frequency of each word based on the text of the boundary information (step S13), and selects a word whose frequency is equal to or greater than the specific threshold as the registration candidate (step S14). Then, the selection unit 13 selects a word that is finally registered in the dictionary from the registration candidates, and determines the word of the login word. Code (step S15). The selection unit 13 can select a word and specify a dictionary using the above-described methods such as APPEND, TOP, ALL, and MULTI.

Then, the registration unit 14 registers the selected word in the designated dictionary based on the processing in the selection unit 13 (step S16, registration step).

By the above processing, the word addition to the word dictionary 30 is completed. In the present embodiment, the word segmentation model is corrected using the expanded word dictionary 30. In other words, the model generation unit 11 generates a corrected word segmentation model based on the re-learning based on the entire learning corpus 20 and the word dictionary 30 (step S17).

Next, a dictionary generation program P1 for causing a computer to function as the dictionary generating device 10 will be described with reference to FIG.

The dictionary generation program P1 includes a main module P10, a model generation module P11, an analysis module P12, a selection module P13, and a registration module P14.

The main module P10 is part of the overall control dictionary generation function. The functions realized by the execution model generation module P11, the analysis module P12, the selection module P13, and the registration module P14 are the same as those of the model generation unit 11, the analysis unit 12, the selection unit 13, and the registration unit 14, respectively.

The dictionary program P1 is provided, for example, on a tangible recording medium such as a CD-ROM or a DVD-ROM or a semiconductor memory. Further, the dictionary generation program P1 can also be provided as a data signal overlapping with the transport wave via the network.

As described above, according to the present embodiment, the word segmentation model is generated using the learning corpus 20 that provides the limit information and the existing dictionary 31, and the word segmentation in which the model is incorporated is applied to the large-scale text 40. Then, by using the application, selecting a word from the set of texts that give the boundary information, and logging in the word dictionary 30. In this way, by using the analysis of the learning corpus 20 to provide the boundary information to the text collection, the words extracted from the text collection are registered, and the large-scale word dictionary 30 can be easily constructed.

E.g" (sumahoke-su) (English for "smartphone case") is divided into " (sumaho) and " (ke-su), the unknown language so far (sumaho) can be registered in the dictionary. Furthermore, " (sumaho) is Japanese" (suma-tofon) ellipsis. also," (uttororin) statement (Japanese) (uttori) (equivalent to the English "fascinated" unknown) can also be registered in the dictionary. Then, using the constructed dictionary for text analysis, the words containing the registered words can be executed with higher precision (for example, including " (sumaho) or " The word segmentation of (uttororin).

Next, an example of evaluation of word segmentation performance by the generating device 10 of the present embodiment will be described. The evaluation index formed by the word segmentation is set to N _{REF if the} number of extended words included in the positive corpus of the precision (Prec), the reproduction rate (Rec), and the F value is set to N _SYS , and the number of extended words included in the analysis result is set to N _SYS . When the number of extended words included in both the analysis result and the positive corpus is N _COR , the above three indexes are defined as follows.

Prec=N _COR /N _SYS

Rec=N _COR =N _REF

F=2Prec. Rec/(Prec+Rec)

Use UniDic's main entry list (different 304, 267 words) as the existing dictionary, and use LIBLINEAR as the vector support machine under the waiver parameter. learn The corpus and the half-width characters in the large-scale text are all unified into a full-width, and no further standardization is performed.

First, the validity of the learning corpus and large-scale text for the same field (learning in the same field) is explained. Here, the domain is based on the concept of stylistic and word grouping based on style, content (type), and the like. Under the same field of study, the title and description of the 590 product randomly selected from the website type of the virtual shopping mall A are randomly selected, and the description of the 50-products randomly extracted from the website of the virtual product B is used to create a 3-stage word segmentation learning. Corpus. The number of words in the learning corpus is about 110,000, and the number of words is about 340,000. Use this learning corpus to evaluate performance.

As a large-scale text, the title and explanatory text of all the product materials in the above-mentioned virtual shopping mall A are used. The number of goods is about 27 million, and the number of words is about 16 billion.

When the large-scale text is parsed by the reference model and the two-stage word segmentation is performed, 576,954 words are extracted, and when the three-stage word segmentation is performed after the analysis, 603, 187 words are extracted. Here, the threshold for the frequency of use of the truncation of words is 20. When the above-mentioned "MULTI" is used, a total of 100,000 words in the upper frequency, a top-ranked 200,000 words, a higher-ranking 300,000 words, a higher-ranking 400,000 words, and the whole are added as separate dictionaries. Only the top 100,000 words are used when using the above "TOP".

The learning result using the reference model, the relearning result using the word dictionary obtained by dividing the two-stage word, and the relearning result using the word dictionary obtained by the three-stage segmentation are shown in Table 1. The values in Table 1 are all percentages (%).

In the case of using the two-stage split re-learning, even if the method is used (APPEND/TOP/ALL/MULTI) to add a word, the F value rises, which means that the learning of the large-scale text using the proposal is effective. The increase in F value increases in the order of APPEND<TOP<ALL<MULTI. From this result, it is understood that when a word is added, it is more effective to add to another dictionary than to add to an existing dictionary. Further, it is more effective to add a word added to another dictionary based on the frequency of occurrence than to register the added word in an additional dictionary.

According to Table 1, it is considered that the classifier automatically learns the contribution and weight according to the frequency of occurrence of the word. Furthermore, in the case of using the three-stage word segmentation and re-learning, the benchmark model and the 2-stage word segmentation performance are improved in all cases. Specifically, by considering the half-segmentation, an improvement in obtaining a word or the like accompanying the affix is obtained.

Secondly, the effectiveness of the situation in the field of learning corpora and large-scale text is explained. The learning corpus used is the same as that of the same field as described above. On the other hand, the large-scale text uses the user reviews in the travel reservation website C, the name of the accommodation facility, the name of the accommodation plan, and the answer from the accommodation facility. The number of texts is 348,564, and the number of characters is about 126 million. The In the large-scale text, 150 and 50 comments were randomly extracted, and the word segmentation was performed by the human hand, and used as a text corpus and an active learning corpus (relative to the additional part of the learning corpus).

First, a large-scale text in the field of travel is parsed using a benchmark model of learning corpus learning from the above-mentioned commodity field. This analysis performance is the "reference" in Table 2 below.

Secondly, the learning corpus of the commodity field is added to the corpus of domain adaptation, and after learning the word segmentation model, it is used to parse large-scale text. This analysis performance is "field adaptation" in Table 2 below. After parsing the large-scale text, 41,671 words are extracted differently from the two-stage word segmentation, and 44,247 words different from the three-stage segmentation are extracted. In either case, only words with a total occurrence frequency of 5 or more are used.

The obtained words are added to the dictionary, and the results of the learning model using the learning corpus and the domain application corpus are shown in Table 2. The values in Table 2 are all percentages (%).

It can be seen from the table that when the learning corpus is different from the large-scale text field, the performance of the three-stage word segmentation is found to be improved.

The present invention has been described in detail based on the present embodiment. However, the present invention is not limited to the above embodiment. The present invention can be variously modified without departing from the spirit and scope of the invention.

In the above embodiment, the selection unit 13 selects a word based on the frequency of occurrence, but the selection unit 13 may register all the words in the existing dictionary 31 or the additional dictionary 32 without referring to the frequency of occurrence. Also, the truncation of words is not necessary.

In the above-described embodiment, the analysis unit 12 analyzes the entire large-scale text 40 and performs the processing by the selection unit 13 and the registration unit 14. However, the analysis unit 12 may divide the collected large amount of text into a plurality of times. At this time, the plurality of repetitions include a series of processing steps including a model generation step, an analysis step, a selection step, and a login step. For example, when the large-scale text 40 is divided into groups 1 to 3, the group 1 is analyzed by the processing of the first loop, the words are registered, the group 2 is analyzed by the processing of the second loop, and the words are registered, and the third ring is used. The analysis group 3 is processed, and the words are registered. In the process after the second ring, the model generation unit 11 refers to the entire word dictionary 30 to generate a corrected word segmentation model.

Since the three-stage division method is used in the above embodiment, the limit information is three types, but the aspect of the boundary information is not limited to this example. For example, it is also possible to perform two-stage word segmentation using only two types of boundary information, "segmentation" and "non-segmentation". In addition, "segmentation", "non-segmentation", and plural kinds of probabilistic divisions can be used to perform word segmentation of four or more stages. For example, a 4-stage word segmentation using a probabilistic division (third information) of b _i = 0.33 and b _i = 0.67 may be performed. Regardless of the use of either, the intensity of the segmentation equivalent to the third information is greater than the strength of the boundary information as "non-segmentation" (eg, b _i =0), and less than the strength of the boundary information as "segmentation" (eg, b _i =1) ).

Industrial availability

According to this embodiment, it is possible to easily construct a large-scale word dictionary.

10‧‧‧ dictionary production device

11‧‧‧Model Generation Department

12‧‧‧ Analysis Department

13‧‧‧Selection Department

14‧‧‧Login Department

20‧‧‧Learning corpus

30‧‧‧Word dictionary

31‧‧‧ Existing Dictionary (word group)

32‧‧‧Additional Dictionary

40‧‧‧ Large-scale text (collection of collected texts)

P1‧‧‧ dictionary production program

P10‧‧‧ main module

P11‧‧‧Model Generation Module

P12‧‧‧analysis module

P13‧‧‧Selection module

P14‧‧‧ Login Module

Fig. 1 is a view showing a hardware configuration of a dictionary generating device of an embodiment.

Fig. 2 is a block diagram showing the functional configuration of the dictionary generating device shown in Fig. 1.

Fig. 3 is a diagram for explaining setting of boundary information (word boundary label).

Fig. 4 is a flow chart showing the operation of the dictionary generating device shown in Fig. 1.

Fig. 5 is a view showing the configuration of a dictionary generating program of the embodiment.

10‧‧‧ dictionary production device

11‧‧‧Model Generation Department

12‧‧‧ Analysis Department

13‧‧‧Selection Department

14‧‧‧Login Department

20‧‧‧Learning corpus

30‧‧‧Word dictionary

31‧‧‧ Existing dictionary

32‧‧‧Additional Dictionary

40‧‧‧Collection of collected texts (large-scale text)

Claims (16)

A dictionary generating device includes: a model generating unit that generates a word segmentation model using a corpus and a word group prepared in advance, and assigns boundary information indicating a word limit to each of the texts included in the corpus, and the boundary information includes the representation The first information having the above-mentioned limit does not exist in the position between the characters, the second information indicating that the position exists between the characters, and the third information indicating that the position is probabilistic between the positions of the characters; and the analyzing unit is configured to collect the third information The set of texts performs a word segmentation in which the word segmentation model is incorporated, and the boundary information is given to each of the texts; and the selection unit selects a word to be registered in the dictionary from the text to which the analysis information is given by the analysis unit; The registration unit registers the word selected by the selection unit in the dictionary. The dictionary generating device according to claim 1, wherein the selection unit selects a word to be registered in the dictionary based on an appearance frequency of each word calculated from the limit information given by the analysis unit. The dictionary generating device of claim 2, wherein the selection unit selects the word whose appearance frequency is equal to or greater than a certain threshold. The dictionary generating device according to claim 3, wherein the selection unit extracts a word whose appearance frequency is equal to or higher than the threshold value as a registration candidate, and selects a specific number of words from the registration candidates in order from the word having a higher frequency of occurrence; The registration unit adds a word selected by the selection unit to a dictionary in which the word group is recorded. The dictionary generating device according to claim 3, wherein the selection unit extracts a word whose appearance frequency is equal to or greater than the threshold value as a registration candidate, and selects a specific number of words from the registration candidates in order from the word having a higher frequency of occurrence; The registration unit registers the word selected by the selection unit in a dictionary different from the dictionary in which the word group is recorded. The dictionary generating device of claim 3, wherein the registration unit registers a word selected by the selection unit in a dictionary different from a dictionary in which the word group is recorded. The dictionary generating device according to claim 3, wherein the selection unit extracts a word whose appearance frequency is equal to or higher than the threshold value as a registration candidate, and groups the words of the registration candidate according to the height of the appearance frequency; the registration unit is configured by the above The plurality of groups generated by the selection unit are individually registered in a plurality of dictionaries different from the dictionary in which the word group is recorded. The dictionary generating device of claim 3, wherein the information indicating the domain of the text is associated with each of the collected texts; and the registration unit individually registers the word selected by the selection unit based on the field containing the text of the word A dictionary prepared for each of these areas. The dictionary generating apparatus according to any one of claims 2 to 8, wherein the frequency of occurrence of each word is calculated based on the first, second, and third information. The dictionary generating device of claim 9, wherein the analyzing unit has the first two a value classifier and a second binary classifier; wherein the first binary classifier determines whether the first information is allocated or the information other than the first information is allocated to each of the inter-text positions, and the second binary classifier is The first binary scoring device determines that the inter-text position determination of the information other than the first information is the assignment of the second information or the allocation of the third information. The dictionary generating apparatus according to any one of claims 1 to 8, wherein the collected text set is divided into a plurality of groups; the analysis unit, the selection unit, and the registration unit are based on the plurality of groups After performing the processing, the model generating unit generates the word segmentation model using the corpus, the word group, and a word registered by the registration unit, and then the analysis unit, the selection unit, and the registration unit are based on the plurality of groups. The other one in the group performs processing. The dictionary generating device of claim 9, wherein the collected text set is divided into a plurality of groups; wherein the analyzing unit, the selecting unit, and the registration unit perform processing based on one of the plurality of groups, The model generating unit generates the word segmentation model using the corpus, the word group, and a word registered by the registration unit, and then the analysis unit, the selection unit, and the registration unit are based on the other one of the plurality of groups. Perform processing. The dictionary generating device of claim 10, wherein the collected text set is divided into a plurality of groups; After the analysis unit, the selection unit, and the registration unit perform processing based on one of the plurality of groups, the model generation unit generates the word segmentation model using the corpus, the word group, and a word registered by the registration unit. Then, the analysis unit, the selection unit, and the registration unit perform processing based on the other one of the plurality of groups. A dictionary generating method is performed by a dictionary generating device performer, comprising: a model generating step of generating a word segmentation model using a pre-prepared corpus and a word group, and assigning a word limit to each text included in the corpus The boundary information includes the first information indicating that the boundary does not exist between the characters, the second information indicating that the position exists between the characters, and the third information indicating that the position is probabilistic between the characters. And a parsing step of performing a word segmentation incorporating the word segmentation model on the collected set of texts, and assigning the boundary information to each of the texts; and a selecting step from the text to which the boundary information is given in the parsing step Selecting a word to be registered in the dictionary; and a registration step of registering the word selected in the selection step in the dictionary. A dictionary generating program product for causing a computer to execute: a model generating unit that generates a word segmentation model using a pre-prepared corpus and a word group, and assigns a table to each text included in the corpus The boundary information of the word boundary is displayed, and the boundary information includes a first information indicating that the boundary does not exist between the characters, a second information indicating that the position exists between the characters, and a probability that the position between the characters is probabilistic. a third information; an analysis unit that performs word segmentation incorporating the word segmentation model on the collected set of texts, and provides the boundary information to each of the texts; and the selection unit that gives the boundary by the analysis unit A word to be registered in the dictionary is selected from the text of the information; and a registration unit that registers the word selected by the selection unit in the dictionary. A computer-readable recording medium for a memory dictionary generating program, the dictionary generating program causing a computer to execute: a model generating unit that generates a word segmentation model using a pre-prepared corpus and a word group, and assigns each text included in the corpus The boundary information indicating the boundary of the word, and the boundary information includes the first information indicating that the boundary does not exist between the characters, the second information indicating that the position exists between the characters, and the probability that the position between the characters is probabilistic a third information; an analysis unit that performs word segmentation incorporating the word segmentation model on the collected set of texts, and provides the boundary information to each of the texts; and the selection unit that gives the boundary by the analysis unit Select the words to be registered in the dictionary in the text of the information; and The registration unit registers the word selected by the selection unit in the dictionary.