JP2020008951A - Terminology fluctuation extraction device and method - Google Patents

Abstract

【Task】
It is possible to determine the degree of similarity between a notational deviation candidate (compound word) and an OK word (compound word) with high accuracy, and to register them.
[Solution]
The notation fluctuation extracting device stores a document before correction and a document after correction in one domain, extracts a compound word from the document before correction and the document after correction, and appears only in the document before correction. A processing unit that makes a compound word a candidate for the fluctuation of the notation of the domain term. The processing unit registers the combination of the spelling fluctuation candidate and the domain term in the storage unit as a spelling fluctuation candidate / domain term pair, and stores the spelling fluctuation candidate and the domain of the spelling fluctuation candidate / domain term pair registered in the storage unit. The term is divided into general terms, and the maximum similarity between the general term of the divided notational candidate and the general term of the divided domain term is calculated. Based on the calculated maximum similarity of the general term, the notation is calculated. The average similarity of the swing candidate / domain term pair is calculated.
[Selection diagram] FIG.

Description

  The present invention relates to a spelling fluctuation extracting device and method.

  2. Description of the Related Art In recent years, a device that analyzes a document written in a natural language and extracts “notation fluctuation” of the document has been developed. Here, “oscillation” is a plurality of (different) notations indicating the same concept in one document or a series of documents.

  Patent Document 1 discloses an example of a technique for extracting a similar expression as a technique related to the transcription fluctuation. Patent Literature 1 discloses a compound word extracting unit that extracts a compound word that is a predetermined sequence of part of speech from a morphological sequence, and whether a compound word is a sway of another compound word in a specific set of compound words. There is disclosed a technology which includes a determination input unit for inputting a determination result by a human regarding whether or not to perform the determination, and registers the result in a spelling dictionary based on the determination result.

JP-A-2006-139164

  In addition, according to Patent Literature 1, the spelling fluctuation can be appropriately extracted only when a determination result by a human can be provided. Otherwise, there is a possibility that the spelling fluctuation cannot be properly extracted. Therefore, a sufficient recall may not be guaranteed.

  SUMMARY OF THE INVENTION It is an object of the present invention to provide a spelling variation extraction apparatus and method capable of determining and registering the similarity between a spelling variation candidate (compound word) and an OK word (compound word) with high accuracy.

  One aspect of the present invention that solves the above problems is a storage unit that stores a document before correction and a document after correction in one domain, and extracts a compound word from the document before correction and the document after correction, and corrects the compound word. A processing unit that sets a compound word appearing only in the previous document as a candidate for the fluctuation of the notation of the domain term. The processing unit registers the combination of the spelling fluctuation candidate and the domain term in the storage unit as a spelling fluctuation candidate / domain term pair, and stores the spelling fluctuation candidate and the domain of the spelling fluctuation candidate / domain term pair registered in the storage unit. The term is divided into general terms, and the maximum similarity between the general term of the divided notational candidate and the general term of the divided domain term is calculated. Based on the calculated maximum similarity of the general term, the notation is calculated. The average similarity of the swing candidate / domain term pair is calculated.

  ADVANTAGE OF THE INVENTION According to this invention, a correct term and the spelling fluctuation which is the erroneous description can be extracted from the document before correction and the document after correction accurately and with good recall.

FIG. 2 is a diagram illustrating a configuration example of a writing fluctuation extracting device according to a first embodiment. FIG. 3 is a diagram illustrating a hardware configuration of a notation fluctuation extraction device. FIG. 14 is a diagram illustrating an example of a general word vector 1221. FIG. 9 is a flowchart illustrating a processing procedure in an average similarity calculation unit. It is the figure which showed an example of the output of a notation fluctuation candidate / OK word pair. FIG. 9 is a diagram conceptually illustrating calculation of “average similarity”. It is the figure which showed an example of the output of a notation fluctuation / OK word pair. FIG. 13 is a diagram illustrating a configuration example of a spelling variation extraction device according to a second embodiment of the second embodiment. FIG. 5 is a diagram showing an example of a compound word vector. FIG. 14 is a flowchart illustrating a processing procedure in an average similarity calculation unit according to the second embodiment. FIG. 9 is a diagram showing pairs of spelling candidates and OK words registered in a vocabulary information storage unit. FIG. 18 is a flowchart illustrating a processing procedure in an average similarity calculation unit according to the third embodiment.

  Hereinafter, embodiments will be described with reference to the drawings. The embodiments described below do not limit the invention according to the claims. In addition, not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.

  A document targeted by the present technology is a document belonging to a specific “domain”. For example, a design document of a specific information system is a document belonging to a “domain” of the information system. As described above, the “domain” is a field to which the subject of the text belongs or a group of creators of the text. In a document such as a specification of a specific electronic device or a bank system written in a natural language, a compound word which is a term specific to a domain is often used.

  For this reason, there is a high possibility that a writing fluctuation peculiar to the domain occurs. For example, in a particular domain, the compound term "deposit item" is a term specific to that domain, but the word "type" may be used. At this time, the notation “deposit type” may fluctuate.

  In particular, when there is a similar expression with a low appearance frequency for a word with a high appearance frequency, it is difficult to extract the similar expression with a low appearance frequency because the co-occurrence expression is often biased. For example, it is assumed that there is a similar expression “deposit type” having an appearance frequency of 1 for a compound word “deposit type” having an appearance frequency of about several hundreds. Since the context of "deposit type" is "change of deposit type ...", its co-occurrence expression is "change". On the other hand, the co-occurrence expression of “deposit item” is often “registration”. In this situation, there is a high possibility that “deposit type” cannot be extracted as a similar expression of “deposit type”.

  Hereinafter, as an example, “deposit type” is a term specific to a domain, and “deposit type” is notationally changed.

FIG. 1 is a diagram functionally showing an outline of a writing fluctuation extracting device 10. The transcription fluctuation extracting device 10 includes a processing unit 11 and a storage unit 12.
The processing unit 11 includes an input / output unit 111, a compound word extraction unit 112, a vocabulary vector acquisition unit 113, an average similarity calculation unit 114, and a selection unit 115. The storage unit 12 includes a document storage unit 121 and a vocabulary information storage unit 122.

  In the first embodiment, the user registers a document before correction and a document after correction in the document storage unit 121 via the input / output unit 111 before executing the processing. The document before modification may be a single document or a plurality of documents. The same applies to the revised document.

  The spelling variation extraction device 10 is provided with a mechanism capable of mechanically distinguishing a document before correction and a document after correction. For example, the document storage unit 121 includes a folder for storing a document before correction and a folder for storing a document after correction. Further, for example, a function of adding “_before correction” to the file name of the document before correction (the same applies to the document after correction) may be provided.

  The function of registering a document in the input / output unit 111 also has a function of registering whether the document is before or after correction. For example, the user may be presented with a “check box for selecting whether the document is before or after modification”. Further, when receiving the user's selection result, the document is sorted into the pre-correction folder or the post-correction folder, or the document is registered in the document storage unit 121 so as to add “_before correction” or “_after correction”.

  When the user presses the “start” button on the input / output unit 111, the compound word extraction unit 112, the vocabulary vector acquisition unit 113, the average similarity calculation unit 114, and the selection unit 115 are sequentially activated. When all the processes are completed, all the extracted spelling variations are displayed on the input / output unit 111.

  Usually, in the corrected document, all the spelling has been corrected. Therefore, the following situation is taken as an example. The document before amendment includes both “deposit item” (domain term) and “deposit type” (notation). On the other hand, the document after the correction includes only the “deposit item” (domain term) and does not include the “deposit type” (notation fluctuation).

  The compound word extraction unit 112 performs a morphological analysis on the document before correction stored in the document storage unit 121 using a general word dictionary stored in the vocabulary information storage unit 122. Here, the “morpheme” is a minimum unit having a meaning in a sentence. The morpheme includes information of a character string indicating a notation of the morpheme and information of a part of speech to which the morpheme belongs. The compound word extraction unit 112 can be realized by using Mecab (MeCab is an open source morphological analysis engine) and TermExtract.

  The morpheme sequence of the document before correction, which is the result of the morphological analysis, is registered in the document storage unit 121. For example, the document before correction stored in the document storage unit 121 includes a fluctuation in the notation “deposit type”. The “deposit type” is two morphemes (general words) of “deposit” and “type”.

  The compound word extraction unit 112 extracts a predetermined part-of-speech sequence as a compound word from the morpheme sequence of the document before correction stored in the document storage unit 121. The result of the extraction is registered in the vocabulary information storage unit 122 as a pre-correction compound word dictionary. At this time, since “deposit” and “type” are a sequence of nouns, “deposit type” is extracted as a compound word and registered in the uncorrected compound word dictionary of the vocabulary information storage unit 122.

  The compound word extraction unit 112 performs a morphological analysis on the corrected document stored in the document storage unit 121 using a general word dictionary stored in the vocabulary information storage unit 122. The morpheme sequence of the corrected document, which is the result of the morphological analysis, is registered in the document storage unit 121. For example, the corrected document stored in the document storage unit 121 includes a term unique to the domain “deposit item”. The “deposit item” is two morphemes (general words) of “deposit” and “item”.

  The compound word extraction unit 112 extracts a predetermined part-of-speech sequence as a compound word from the morphological sequence of the document after correction stored in the document storage unit 121. The result of the extraction is registered in the vocabulary information storage unit 122 as a corrected compound word dictionary. At this time, since “deposit” and “item” are a sequence of nouns, “deposit item” is extracted as a compound word and registered in the compound word dictionary after correction, but the document after correction is “deposit type” Since the (description fluctuation) is not included, the “deposit type” is not registered in the compound word dictionary after correction.

  As described above, all the spelling changes are usually corrected in the corrected document, and the corrected document includes only correct compound words. Therefore, all the compound words extracted from the corrected document are correct, that is, all the compound words in the corrected compound word dictionary are correct. Therefore, the corrected compound word dictionary is called an “OK word dictionary”. Further, a compound word belonging to the OK word dictionary, that is, a compound word extracted from the document after correction is referred to as an “OK word”. For example, “deposit item” is an OK word.

  The compound word extraction unit 112 calculates a difference between the uncorrected compound word dictionary stored in the vocabulary information storage unit 122 and the OK word dictionary. That is, a compound word registered only in the uncorrected compound word dictionary and not registered in the OK word dictionary is acquired and registered in the vocabulary information storage unit 122 as a spelling candidate dictionary.

  The “deposit type” is registered in the pre-correction compound word dictionary, but is not registered in the OK word dictionary, and thus is registered in the spelling candidate dictionary.

  The compound word extraction unit 112 compares the correspondence between all the spelling candidates and all the OK words stored in the vocabulary information storage unit 122 with a “spelling candidate / OK word pair” (in this specification, “spelling candidate”). And a pair of OK words are described as a “spelling fluctuation candidate / OK word pair”) in the vocabulary information storage unit 122. Here, assuming that there are n spelling fluctuation candidates and m OK words, the number of spelling fluctuation / OK word pairs is (n × m).

  For example, there is a compound word pair “deposit type / deposit item”. Here, in the registration in the transcription fluctuation candidate dictionary, since the compound words registered in the OK word dictionary are excluded, the number of n can be reduced by the number of OK words, and the transcription fluctuation / OK word pair (in this specification, , "A pair of a spelling variation and an OK word" is referred to as a "spelling variation / OK word pair") (n × m) can be effectively reduced.

  The vocabulary vector acquisition unit 113 uses a technique such as machine learning to input the morpheme sequence of the document before correction and the morpheme sequence of the document after correction stored in the document storage unit 121 and to co-occur in common words. From the expression, calculate a vector expression of the general word. A method of calculating a vocabulary vector expression from a vocabulary co-occurrence expression is, for example, Word2Vec. The vector expression as a calculation result is registered as a general word vector 1221 in the vocabulary information storage unit 122 (see FIG. 3). For example, the vectors of “deposit”, “event” and “type” are calculated from the respective co-occurrence expressions. In this embodiment, vectors of the general words “deposit”, “item” and “type” are used, and vectors of compound words such as “deposit type” and “deposit item” are not used.

  FIG. 2 is a diagram illustrating a hardware configuration of the transcription fluctuation extracting device. A CPU 21 having an arithmetic function, a memory 22 for temporarily storing data, an interface 23 connected to a display device 25 such as a display and an input device 26 such as a mouse and a keyboard, and a storage unit 12 are connected via a bus 24. ing. The storage unit 12 uses a storage device such as a hard disk and an SSD. The storage unit 12 stores a program 13 for implementing various functions of the input / output unit 111, compound word extraction unit 112, vocabulary vector acquisition unit 113, average similarity calculation unit 114, and selection unit 115 of FIG. These programs are read into the memory 22, and various functions are realized by the CPU 21 executing the various programs. Further, a document storage unit 121 for storing a document before correction and a document after correction in the storage unit 12, a general term dictionary, a compound word dictionary before correction, a compound word dictionary after correction, a spelling candidate dictionary, a spelling candidate / OK word A vocabulary information storage unit 122 for storing pairs, general word vectors, average similarity U, spelling / OK word pairs, and compound word vectors.

  FIG. 3 shows an example of the general word vector 1221. A vector consisting of 50 to 200 numbers is assigned to one general word. This general vector is a value calculated from each co-occurrence expression for “deposit”, “item” and “type”.

  FIG. 4 is a flowchart illustrating a processing procedure in average similarity calculation section 114. The flow illustrated in FIG. 4 illustrates a process of calculating the average similarity for all the spelling candidate / OK word pairs stored in the vocabulary information storage unit 122 and registering the result in the vocabulary information storage unit 122. ing. However, since the spelling candidates are only compound words that are not registered in the OK word dictionary, the number of spelling candidate / OK word pairs has already been effectively reduced.

  In step S301, one spelling candidate / OK word pair is acquired from the vocabulary information storage unit 122. For example, “deposit type / deposit item” was acquired. The “deposit type” is a candidate for a sway, and the “deposit type” is an OK word.

  In step S302, a spelling candidate (compound) is divided into general words (elements). OK words (compound words) are also divided into general words (elements). Division refers to performing morphological analysis using a general word dictionary stored in the vocabulary information storage unit 122, similarly to the general word compound word extraction unit 112. For example, the “deposit type” (candidate fluctuation candidate compound word) is divided into two morphemes = general word (element) of “deposit” and “type”. The “deposit item” (OK compound word) is divided into two morphemes = general words (elements) of “deposit” and “item”. For convenience, a morpheme = general word that forms a compound is referred to as an element of the compound here.

  In step S303, the average similarity U is initialized to 0.

  In step S304, one element of the spelling fluctuation candidate is acquired. For convenience, this element is called an “NG element”. For example, “type” (NG element) is acquired.

  In step S305, the maximum similarity V is initialized to 0.

  In step S306, one element of the OK word is obtained. For convenience, this element is called an "OK element". For example, "deposit" (OK element) was acquired.

  In step S307, the NG element vector and the OK element vector are acquired from the general word vector 1221 (see FIG. 3) stored in the vocabulary information storage unit 122. Further, the similarity S between the NG element and the OK element is calculated. The similarity S indicates the cosine similarity between the NG element vector and the OK element vector. For example, the similarity S between the “type” (NG element) and the “deposit” (OK element) is set to 0.2.

  In step S308, the similarity S calculated in step S307 is compared with the maximum similarity V. If the similarity S> the maximum similarity V, the process proceeds to step S309; otherwise, the process proceeds to step S310. Since the similarity S = 0.2> the maximum similarity V = 0, the process proceeds to step S309.

  In step S309, the maximum similarity V is updated to the similarity S calculated in step S307. For example, the maximum similarity V is updated to the similarity S = 0.2.

  In step S310, it is confirmed whether or not all the elements of the OK word have been acquired. If all the elements of the OK word have been acquired, the process proceeds to step S311. Otherwise, the process returns to step S306. For example, the elements of “deposit item” (OK word) include “deposit” and “item”, but since only “deposit” has been acquired, the process returns to step S306.

  For the above example, the operation from step S306 to step S310 will be described again. In step S306, “event” (OK element) is obtained. Next, in step S307, the “type” (NG element) vector and the “event” (OK element) vector are acquired from the general word vector 1221 stored in the vocabulary information storage unit 122, and the “type” The cosine similarity between the (NG element) vector and the “event” (OK element) vector is calculated. As a result, the cosine similarity is 0.8, that is, the similarity S = 0.8. In step S308, the similarity S = 0.8 is compared with the maximum similarity V = 0.2. As a result, since the similarity S = 0.8> the maximum similarity V = 0.2, the process proceeds to step S309, and the maximum similarity V is updated to the similarity S = 0.8. Next, in step S310, since all the elements of the OK word have been acquired, the process proceeds to step S311.

  In step S311, the maximum similarity V is added to the average similarity U. For example, the maximum similarity V = 0.8 is added to the average similarity U = 0 to obtain the average similarity U = 0.8.

  In step S312, it is checked whether all elements of the spelling fluctuation candidate have been acquired. If all of the spelling fluctuation candidate elements have been acquired, the process proceeds to step S313; otherwise, the process returns to step S304. For example, the elements of the spelling candidate “deposit type” include “deposit” and “type”, but since only “type” has been acquired, the process returns to step S304.

  With respect to the above example, the operation from step S304 to step S312 will be briefly described again. In step S304, the other element “deposit” (NG element) of the spelling candidate “deposit type” is acquired. Subsequently, in steps S305 to S310, the similarity S = 1 between the “deposit” (NG element) and the element “deposit” (OK element) of the OK word “deposit item”, and the maximum similarity V = 1 Becomes Finally, in step S311, the maximum similarity V = 1 is added to the average similarity U = 0.8 to obtain an average similarity U = 1.8. Note that when the OK word “item” is selected in step S305, steps S305 to S310 are repeated, but detailed description is omitted.

  In step S313, the average similarity U is divided by the number of elements of the writing fluctuation candidate. Since the average similarity U = 1.8 and the number of elements of the writing fluctuation candidate is 2, the division results in the average similarity U = 0.9.

  In step S314, the average similarity U of the spelling candidate / OK word pair is registered in the vocabulary information storage unit 122 as the similarity of the spelling candidate / OK word pair. FIG. 5 shows a state in which the spelling candidate / OK word pair and the average similarity U are registered in the vocabulary information storage unit 122.

  In step S315, it is checked whether or not all the spelling candidate / OK word pairs in the vocabulary information storage unit 122 have been acquired. If all the spelling candidate / OK word pairs have been acquired from the vocabulary information storage unit 122, the process ends. If not, the process returns to step S301.

  In the above, the case where the average similarity is calculated as a normal average value of each maximum similarity has been illustrated. Other than the above, there is a method of calculating the average similarity as a multiplication of each maximum similarity. At this time, in step S303, the average similarity is initialized to 1. In step S311, the average similarity is multiplied by the maximum similarity. Step S313 is unnecessary.

  Further, there can be other methods than multiplication. For example, there is a harmonic mean. It will not be described in detail here. The point is that the average similarity of the pair of the notational shake candidate / OK word is determined based on the combination having the highest similarity of each element of the pair of the notational shake candidate / OK word.

  The calculation of the “average similarity” described above will be conceptually described with reference to FIG. From the vocabulary information storage unit 122, a pair of compound words of “deposit type / deposit item”, which is one set of spelling candidate / OK word pair, is acquired. The “deposit type” is a spelling candidate (NG word candidate), and the “deposit type” is an OK word.

  The average similarity calculation unit 114 divides the OK word (compound word) “deposit item” into “deposit” and “item” (element) based on a general word dictionary registered in the vocabulary information storage unit 122. Similarly, the NG word candidate (compound) “deposit type” is divided into “deposit” and “type” (element) based on the general word dictionary registered in the vocabulary information storage unit 122.

  The NG elements “deposit” and “type” of the notation fluctuation candidate are respectively acquired for similarity calculation, and each NG element is calculated for the similarity with “OK” elements “deposit” and “item”, respectively. Are used to calculate the average similarity of the compound words of “deposit type / deposit item”. In FIG. 6, the maximum similarity V between the NG element “type” and the OK element “deposit” “item” is “0.8” in the case of the NG element “type” and the OK element “event”.

  Similarly, the maximum similarity V between the NG element “deposit” and the OK element “deposit” “item” is “1.0” in the case of the NG element “deposit” and the OK element “deposit”. The result obtained by adding the maximum similarity between the NG element and the OK element divided into these general terms and dividing by the number of elements “2” is called “average similarity”. Unlike the similarity between the compound word vector of the compound word candidate and the compound word vector of the compound word OK, the maximum similarity of each element of the compound word is used. The accuracy of the degree determination can be improved.

  The selection unit 115 selects a spelling candidate / OK word pair that satisfies the condition from the spelling candidate / OK word pair in the vocabulary information storage unit 122, and finally selects the spelling candidate / OK word pair. OK word pair. The condition may be, for example, “the similarity is larger than a threshold”. That is, when the threshold value is set to 0.8, a spelling candidate / OK word pair that satisfies the similarity> 0.8 is set to a spelling / OK word pair. At this time, there is not always one or more writing fluctuation / OK word pairs for all the writing fluctuation candidates. If it is desirable that there be at least one spelling / OK word pair for all the spelling candidates, for example, a spelling candidate / OK word pair having the highest similarity for all the spelling candidates Finally, it may be sorted as a spelling / OK word pair. The spelling / OK word pair is registered in the vocabulary information storage unit 122.

  The input / output unit 111 presents the user with the spelling / OK word pair stored in the vocabulary information storage unit 122. It may be presented in a simple list format. When a plurality of spelling variations exist for one OK word, grouping may be performed using the OK word.

  FIG. 7 is an example of the output of the spelling / OK word pair. FIG. 7A shows the user of the spelling / OK word pair stored in the vocabulary information storage unit 122 with the spelling being “notation spelling (wrong)” and the OK word being “domain term”. 4 shows an example of a screen. FIG. 7B shows a case where grouping is performed using an OK word in a situation where a plurality of writing fluctuations exist for one OK word. Here, an example of a screen that presents the user with the spelling / OK word pair in the order of the OK word / spelling as a “domain term” and “spelling spelling (erroneous writing)” is shown.

  As described above, in the first embodiment, the compound word registered in the spelling candidate dictionary is excluded from the compound words registered in the OK word dictionary. Therefore, the number of n can be reduced by the number of OK words. The number of (n × m) word pairs can be effectively narrowed down, and the processing for obtaining the average similarity can be speeded up.

  In addition, unlike the similarity between a compound word candidate for a compound word of an OK word (compound word) and a compound word of an OK word (compound word), the candidate for a notation shake (compound word) and an OK word (compound word) are divided into general terms. Since the maximum similarity between the NG element and the OK element is obtained, and the maximum similarity of each element is divided by the number of elements “2” to calculate “average similarity”, the similarity determination of compound words can be performed with high accuracy. It can be carried out.

  In the second embodiment, the processing procedure of the average similarity calculation unit 114 in the first embodiment is changed to further reduce the processing time.

  FIG. 8 illustrates a configuration example of the spelling variation extraction device 10 according to the second embodiment. The components using the same reference numerals have the same functions and configurations as the first embodiment.

In the second embodiment, a vocabulary appearance frequency acquisition unit 116 is added to the processing unit 11.
The compound word extraction unit 112 performs a morphological analysis on the document before correction stored in the document storage unit 121 using the compound word dictionary before correction and the OK word dictionary stored in the vocabulary information storage unit 122. The resulting morpheme sequence containing the compound word of the uncorrected document is registered in the document storage unit 121. For example, the sentence “Confirmation dialog output by pressing the screen search button on the customer symbol number search screen” is “Customer symbol number search screen / on / screen search button / press / on / output / output / confirmation dialog” Is decomposed as follows. That is, compound words such as “customer symbol number search screen” are not broken down into general words. Next, the corrected document is similarly analyzed. The resulting morpheme sequence of the compound-containing document after modification is registered in the document storage unit 121.

  The vocabulary vector acquisition unit 113 receives the morpheme sequence containing the compound word of the document before correction and the morpheme sequence containing the compound word of the document after correction stored in the document storage unit 121 by using a technique such as machine learning. , Calculate the vector representation of the compound from the co-occurrence of the compound. The result is registered in the vocabulary information storage unit 122 as a compound word vector 1222. For example, vectors of “deposit item” and “deposit type” are calculated and registered in the vocabulary information storage unit 122.

  FIG. 9 shows an example of the compound word vector 1222 registered in the vocabulary information storage unit 122. A vector composed of 50 to 200 numbers is assigned to one compound word.

  The vocabulary appearance frequency acquisition unit 116 acquires the OK word dictionary and the spelling candidate dictionary stored in the vocabulary information storage unit 122. From the morpheme sequence containing the compound word of the document before correction and the morpheme sequence containing the compound word of the document after correction stored in the document storage unit 121, the appearance frequency of the OK word and the spelling variation candidate is measured, and the vocabulary information is stored. Registered in the section 122.

  FIG. 10 is a flowchart illustrating a processing procedure in the average similarity calculation unit 114 according to the second embodiment. In the flow illustrated in FIG. 10, all the spelling candidate / OK word pairs stored in the vocabulary information storage unit 122 are classified according to the appearance frequency difference, and the maximum similarity of each element of the compound word is considered. The processing of calculating the average similarity or the “cosine similarity” of a compound word and registering the result in the vocabulary information storage unit 122 as the similarity of the spelling candidate / OK word pair is shown. That is, when the appearance frequency is low, the average similarity is used, and when the appearance frequency is high and the reliability of the cosine similarity of the compound word by the co-occurrence expression is high, the cosine similarity of the compound word is used. The processing can be speeded up.

  In step S321, one spelling candidate / OK word pair is acquired from the vocabulary information storage unit 122.

  In step S322, the appearance frequency of the spelling variation candidate and the appearance frequency of the OK word are acquired from the vocabulary information storage unit 122. The appearance frequency of the spelling variation candidate can be obtained, for example, by counting the number of appearances of the spelling variation candidate from the document before correction. Similarly, the appearance frequency of the OK word can be obtained from the document after correction.

  In step S323, the appearance frequency difference between the spelling candidate / OK word pair is calculated from the appearance frequency of the spelling candidate and the OK word. There are various specific calculations. For example, the difference in the appearance frequency between the OK word and the spelling variation candidate may be used. Alternatively, a quotient obtained by dividing the appearance frequency of the spelling fluctuation candidate by the appearance frequency of the OK word may be used.

  In step S324, it is checked whether or not the appearance frequency difference of the spelling fluctuation candidate / OK word pair> the threshold value. The threshold value is appropriately set in advance in accordance with the formula for calculating the difference in the appearance frequency of the spelling candidate / OK word pair in step S323. If the difference in the appearance frequency of the OK word pair> the threshold, the process proceeds to step S325. Otherwise, the process proceeds to step S327. Assuming that the appearance frequency of the spelling fluctuation candidate is a quotient obtained by dividing the appearance frequency of the OK word, for example, the threshold value of the divided quotient is “0.01”. This is because, when the difference in appearance frequency is equal to or larger than the threshold value, it is determined that the accuracy of the cosine similarity of the compound word cannot be expected, and the determination based on the average similarity is performed. For example, when the difference between the appearance frequencies is large, the process moves to step S325. If the appearance frequency difference is small, the process moves to step S327.

  In step S325, the average similarity of the spelling candidate / OK word pair is calculated, and the result is registered in the vocabulary information storage unit 122 as the similarity of the spelling candidate / OK word pair. The calculation method is steps S302 to S314 shown in FIG.

  In step S326, it is confirmed whether or not all the spelling candidate / OK word pairs in the vocabulary information storage unit 122 have been acquired. If all the spelling candidate / OK word pairs have been acquired from the vocabulary information storage unit 122, the process ends. If not, the process returns to step S321.

  In step S327, a spelling candidate vector and an OK word compound word vector are acquired from the compound word vector 1222 stored in the vocabulary information storage unit 122.

  In step S328, the cosine similarity between the compound word vector of the spelling fluctuation candidate and the compound word vector of the OK word is calculated.

  In step S329, the cosine similarity between the spelling variation candidate vector and the OK word compound word vector is registered in the vocabulary information storage unit 122 as the spelling variation candidate / OK word pair similarity, and the process proceeds to step S326.

  FIG. 11 shows a pair of a spelling variation candidate and an OK word registered in the vocabulary information storage unit 122 in step S329, an appearance frequency difference, and a cosine similarity between the variation candidate vector and the OK word vector.

  According to the second embodiment, when the frequency of appearance is low, a pair of a notational shake candidate and an OK word can be registered with the similarity with high accuracy by using the average similarity. When the reliability of the word cosine similarity is high, by using the cosine similarity of the compound word, it is possible to register the spelling fluctuation candidate / OK word pair with high accuracy and high speed processing.

In the third embodiment, the processing procedure of the average similarity calculation unit 114 in the second embodiment is changed to improve the accuracy and recall of the spelling variation extraction.
A configuration example of the device 10 according to the third embodiment is the same as that of the second embodiment (FIG. 8).

  FIG. 12 is a flowchart illustrating a processing procedure in average similarity calculation section 114. The flow shown in FIG. 12 calculates the average similarity, the cosine similarity, and the appearance frequency difference for all the spelling candidate / OK word pairs stored in the vocabulary information storage unit 122, and calculates the average based on the appearance frequency difference. A process of calculating a weighted average of the similarity and the cosine similarity and registering the result in the vocabulary information storage unit 122 as the similarity of the spelling candidate / OK word pair is shown.

  By giving weight to the average similarity and the cosine similarity in consideration of the appearance frequency, it is possible to improve the accuracy of the similarity between the pair of the spelling variation candidate and the OK word.

  In step S331, one spelling candidate / OK word pair is acquired from the vocabulary information storage unit 122.

  In step S332, the average similarity of the notation fluctuation candidate / OK word pair is calculated. The calculation method is steps S302 to S313 shown in FIG.

  In step S333, a spelling candidate vector and an OK word vector are acquired from the compound word vector 1222 stored in the vocabulary information storage unit 122.

  In step S334, the cosine similarity between the notation fluctuation candidate vector and the OK word vector is calculated.

  In step S335, the appearance frequency of the spelling variation candidate and the appearance frequency of the OK word are acquired from the vocabulary information storage unit 122.

  In step S336, a difference in the appearance frequency between the spelling candidate / OK word pair is calculated from the appearance frequency of the spelling candidate and the OK word. There are various candidates for specific formulas. For example, the difference between the appearance frequency of the OK word and the appearance frequency of the spelling variation candidate may be used. Alternatively, a quotient obtained by dividing the appearance frequency of the spelling fluctuation candidate by the appearance frequency of the OK word may be used.

  In step S337, a weighted average of the average similarity and the cosine similarity is calculated based on the appearance frequency difference of the spelling candidate / OK word pair. In general, when there is a spelling with a low appearance frequency for an OK word with a high appearance frequency, the co-occurrence expression of the spelling sway is often biased. It is better to use the average similarity than the cosine similarity of the word with the compound word vector. The calculation method of the weighted average of the average similarity and the cosine similarity based on the difference in the appearance frequency of the spelling variation candidate / OK word pair is appropriately determined in advance according to the calculation formula of the difference in appearance frequency of the spelling variation candidate / OK word pair in step S336. Set. When the difference in appearance frequency of the notation fluctuation candidate / OK word pair is set as a difference, for example, the weight of the average similarity may be set to (1-1 / difference), and the weight of the cosine similarity may be set to (1 / difference). When the difference in the appearance frequency of the notation fluctuation candidate / OK word pair is used as the quotient, the weight of the average similarity may be set to (1-quotient), and the weight of the cosine similarity may be set to the quotient.

  In step S338, the weighted average is registered in the vocabulary information storage unit 122 as the similarity of the spelling candidate / OK word pair.

  In step S339, it is confirmed whether or not all the spelling candidate / OK word pairs in the vocabulary information storage unit 122 have been acquired. If all the spelling candidate / OK word pairs have been acquired from the vocabulary information storage unit 122, the process ends. If not, the process returns to step S331.

  According to the third embodiment, by giving the average similarity and the cosine similarity weights in consideration of the appearance frequency, it is possible to improve the accuracy of the similarity between the pair of the spelling variation candidate and the OK word.

10: This device, 11: processing unit, 12: storage unit, 111: input / output unit, 112: compound word extraction unit, 113: vocabulary vector acquisition unit, 114: average similarity calculation unit, 115: selection unit, 116: Vocabulary appearance frequency acquisition unit, 121: document storage unit, 122: vocabulary information storage unit, 1221: general word vector, 1222: compound word vector

Claims (15)

A storage unit for storing a document before correction and a document after correction in one domain,
A processing unit that extracts a compound word from the document before the correction and the document after the correction, and sets a compound word that appears only in the document before the correction as a spelling fluctuation candidate of the domain term,
The processing unit includes:
Register the combination of the spelling fluctuation candidate and the domain term as a spelling fluctuation candidate / domain term pair in the storage unit,
The spelling candidate and the domain term of the spelling candidate / domain term pair registered in the storage unit are divided into general terms, respectively, and the general term of the divided spelling candidate and the divided Calculating a maximum similarity between the domain term and the general term, and calculating an average similarity between the notation fluctuation candidate / domain term pair based on the calculated maximum similarity between the general terms. Extraction device. The storage unit has a vocabulary information storage unit,
The processing unit extracts a compound word by morphological analysis from the document before correction stored in the storage unit, registers the compound word in the vocabulary information storage unit as a compound word dictionary before correction, and stores the document in the storage unit. 2. The spelling variation extraction device according to claim 1, further comprising a compound word extraction unit that extracts a compound word from the corrected document by morphological analysis and registers the compound word in the storage unit as a corrected compound word dictionary.   The apparatus according to claim 2, wherein the processing unit includes a vocabulary vector acquisition unit that calculates a vector value from the co-occurrence expression of the general term.   The processing unit has an average similarity calculation unit that calculates the maximum similarity based on the vector value calculated by the vocabulary vector acquisition unit and calculates an average similarity of the spelling fluctuation candidate / domain term pair. The notation fluctuation extracting device according to claim 3, wherein:   A selecting unit that selects the spelling fluctuation candidate / domain term pair that satisfies a condition from the average similarity calculated by the average similarity calculation unit, and stores the spelling fluctuation / domain term pair in the vocabulary information storage unit. 5. The notation fluctuation extracting device according to claim 4, wherein: For the compound of the spelling candidate / domain term pair, the processing unit includes an appearance frequency at which the spelling candidate appears in the document before the correction and an appearance frequency at which the domain term appears in the corrected document. Has a vocabulary appearance frequency acquisition unit that acquires
The average similarity calculator calculates the average similarity when the difference in the frequency of appearance of the domain term in the corrected document is larger than a threshold, and the spelling candidate when the difference is smaller than the threshold. 5. The spelling variation extraction apparatus according to claim 4, wherein a similarity between the spelling variation candidate and the domain term pair is calculated based on a cosine similarity between the spelling variation candidate forming the domain term pair and the domain term. For the compound of the spelling candidate / domain term pair, the processing unit includes an appearance frequency at which the spelling candidate appears in the document before the correction and an appearance frequency at which the domain term appears in the corrected document. Has a vocabulary appearance frequency acquisition unit that acquires
The average similarity calculator calculates the average similarity and the notation of the notation fluctuation candidate / domain term pair according to a difference between an appearance frequency at which the notation fluctuation candidate appears and an appearance frequency at which the domain term appears. The apparatus according to claim 4, wherein a weighted average is calculated for the shake candidate and the cosine similarity of the domain term. From a document before correction and a document after correction in one domain, a domain term that is a correct compound word after correction and a correspondence of the sway of the notation of the document before correction corresponding to the domain term are stored as a pair. In the notation fluctuation extraction device,
A storage unit that stores the document before the correction and the document after the correction in the one domain,
A processing unit that extracts a compound word from the document before the correction and the document after the correction, and sets a compound word that appears only in the document before the correction as a spelling candidate of the domain term. Extraction device. The document before correction and the document after correction in one domain are stored in the storage unit,
By the processing unit, a compound word is extracted from the document before the correction and the document after the correction, and a compound word that appears only in the document before the correction is a notation fluctuation candidate of the domain term,
Register the combination of the spelling fluctuation candidate and the domain term as a spelling fluctuation candidate / domain term pair in the storage unit,
The spelling fluctuation candidate and the domain term of the spelling fluctuation candidate / domain term pair registered in the storage unit are each divided into general terms, and the general term of the divided spelling fluctuation candidate and the divided domain are divided. Calculating the maximum similarity of a term with a general term and calculating the average similarity of the notation fluctuation candidate / domain term pair based on the calculated maximum similarity of the general term. Method.   The processing unit extracts a compound word from the document before correction stored in the storage unit by morphological analysis, registers the compound word as a compound word dictionary before correction in the vocabulary information storage unit of the storage unit, and stores the compound word in the storage unit. The method according to claim 9, wherein the corrected document that has been corrected is subjected to morphological analysis to extract a compound word and to register the compound word in the storage unit as a corrected compound word dictionary.   The method according to claim 10, wherein the processing unit calculates a vector value from a co-occurrence expression of the general term.   The said processing part calculates the said maximum similarity based on the vector value calculated by the said vocabulary vector acquisition part, Computes the average similarity of the said notation fluctuation candidate / domain term pair, The Claims characterized by the above-mentioned. 12. A method for extracting notational deviation according to item 11.   13. The notation according to claim 12, wherein the spelling fluctuation candidate / domain term pair satisfying a condition is selected from the calculated average similarity, and stored as the spelling fluctuation / domain term pair in the vocabulary information storage unit. Shake extraction method. The processing unit includes:
For the compound word of the spelling fluctuation candidate / domain term pair, an appearance frequency in which the spelling fluctuation candidate appears in the document before the correction and an appearance frequency in which the domain term appears in the corrected document,
The difference in the frequency of appearance of the domain term in the document after the modification is based on the average similarity when the difference is larger than a threshold, and when the difference is smaller than the threshold, the spelling candidate / domain term pair is configured. 13. The method according to claim 12, wherein the similarity between the spelling candidate / domain term pair is calculated based on the cosine similarity between the spelling candidate and the domain term. The processing unit includes:
For the compound word of the spelling fluctuation candidate / domain term pair, an appearance frequency in which the spelling fluctuation candidate appears in the document before the correction and an appearance frequency in which the domain term appears in the corrected document,
The average similarity and the cosine of the spelling candidate and the domain term of the spelling candidate / domain term pair according to the difference between the appearance frequency of the spelling variation candidate and the appearance frequency of the domain term. The method according to claim 12, wherein a weighted average is calculated for the similarity.

Images