JP2018032324A - Parallel translation extraction apparatus, parallel translation extraction method, and program - Google Patents

Abstract

A bilingual corpus having higher quality is created as compared with a case where sentence matching is performed based only on the number of matching words. A bilingual sentence acquisition unit 105 of a bilingual sentence extracting apparatus 1 matches a first language sentence and a second language sentence constituting a bilingual document by using a bilingual dictionary to match a first language and a second language. Get one or more translations of a language. The translation model generation unit 107 generates a translation model based on the one or more parallel translation sentences. For each of the one or more parallel translation sentences, the translation unit 108 translates the sentence in the first language constituting the parallel translation sentence into the second language using the generated translation model. The edit distance calculation unit 109 calculates, for each of the one or more parallel translations, an edit distance between the first language sentence translated into the second language and the second language sentence corresponding to the sentence. . The bilingual sentence selection unit 110 selects a bilingual sentence whose calculated edit distance is greater than a threshold value from the one or more bilingual sentences. [Selection] Figure 1

Description

  The present invention relates to a technique for creating a bilingual corpus.

  In recent years, the importance of creating a large quantity of high-quality parallel corpora for use in statistical machine translation and text mining has been recognized. In general, it takes a great deal of cost to create a bilingual corpus, and its efficient creation method is a technical problem. As a method of creating a bilingual corpus, for example, a document in one language constituting a bilingual document is translated into the other language, and the number of matching words between the created translated sentence and the sentence in the other language is measured. A method for associating sentences is known (see Non-Patent Document 1).

Tatsuya Ishizaka, Masao Uchiyama, Eiichiro Sumida, Kazuhide Yamamoto, "Construction of a large-scale open source Japanese-English bilingual corpus", Information Processing Society of Japan, 2009-NL-191, p.1-6, May 2009

  However, in the conventional corpus creation method, as a result of associating sentences by focusing only on the number of matching words, there is a case where sentences that are not translated are associated with each other when viewed as a whole sentence.

  The present invention has been made in view of such circumstances, and an object of the present invention is to create a bilingual corpus having higher quality compared to a case where sentence matching is performed based only on the number of matching words. To do.

  In order to solve the above-described problems, the present invention provides a bilingual document acquisition unit that acquires bilingual documents in a first language and a second language, a sentence in the first language that constitutes the acquired bilingual document, and the second language. A parallel sentence acquisition unit that matches a sentence in a language using the bilingual dictionary of the first language and the second language and acquires one or more parallel sentences of the first language and the second language; and the acquisition A translation model generation unit that generates a translation model based on the one or more parallel translations that have been generated, and for each of the one or more acquired parallel translations, the translation model is configured using the generated translation model A translation unit that translates the sentence in the first language into the second language, and the sentence in the first language translated into the second language for each of the one or more acquired parallel translation sentences; Editing that calculates the editing distance between the corresponding sentence in the second language And a release calculator, among the obtained one or more sentence pairs, providing a bilingual sentence extraction device and a translated sentence selection unit editing the calculated distance is selected greater translated sentence than the threshold value.

  The present invention is also a bilingual sentence extraction method executed by one or more computers, the step of obtaining a bilingual document in a first language and a second language, and the first bilingual document constituting the obtained bilingual document. Matching a sentence in a language and a sentence in the second language using a bilingual dictionary for the first language and the second language to obtain one or more parallel sentences in the first language and the second language Generating a translation model based on the acquired one or more parallel translations, and constructing the parallel translation using the generated translation model for each of the acquired one or more parallel translations Translating the sentence in the first language into the second language, and for each of the one or more acquired parallel translation sentences, the sentence in the first language translated into the second language, and the sentence Between the corresponding sentence in the second language Calculating an edit distance, among the obtained one or more sentence pairs, providing a translated sentence extraction method comprising the steps of: editing the calculated distance is selected greater translated sentence than the threshold value.

  According to another aspect of the present invention, a computer acquires a bilingual document in a first language and a second language, a sentence in the first language and a sentence in the second language that constitute the acquired bilingual document, Matching using a bilingual dictionary of the first language and the second language to obtain one or more bilingual sentences of the first language and the second language, and based on the obtained one or more bilingual sentences Generating a translation model and, for each of the one or more acquired parallel translation sentences, using the generated translation model, the sentence of the first language constituting the parallel translation sentence is converted into the second language. Editing between the step of translating and the sentence of the first language translated into the second language and the sentence of the second language corresponding to the sentence for each of the one or more acquired parallel translation sentences Calculating a distance; and Among the one or more sentence pairs provides a program for executing a step of editing the calculated distance is selected greater translated sentence than the threshold value.

  According to the present invention, it is possible to create a bilingual corpus with higher quality compared to the case where sentence matching is performed based only on the number of matching words.

It is a block diagram which shows an example of a structure of the bilingual sentence extraction apparatus. It is a flowchart which shows an example of a bilingual sentence extraction process. It is a figure which shows an example of a bilingual document. It is a figure which shows an example of the data of the parallel translation memory | storage part. It is a figure which shows an example of the data of the parallel translation memory | storage part. It is a figure which shows an example of the data of the parallel translation memory | storage part.

1. Embodiment 1-1. Configuration FIG. 1 is a block diagram illustrating an example of a configuration of a parallel translation extraction apparatus 1 according to the present embodiment. The bilingual sentence extraction device 1 is a computer including an arithmetic processing device such as a CPU and a storage device such as an HDD. The bilingual sentence extraction apparatus 1 includes a bilingual document storage unit 101, a bilingual document acquisition unit 102, a word division unit 103, a bilingual dictionary storage unit 104, a bilingual sentence acquisition unit 105, a bilingual sentence storage unit 106, a translation The model generation unit 107, the translation unit 108, the edit distance calculation unit 109, the bilingual sentence selection unit 110, and the bilingual sentence editing unit 111 are provided. Among these functions, the functions of the bilingual document storage unit 101, the bilingual dictionary storage unit 104, and the bilingual sentence storage unit 106 are realized by a storage device. Other functions are realized by the arithmetic processing device executing a program stored in the storage device.

  The bilingual document storage unit 101 stores bilingual documents in the first language and the second language. Here, the first language is Japanese and the second language is English. A bilingual document is a pair of a Japanese document and an English document created by translating the document into English. The bilingual document is, for example, a pair of a Japanese patent application patent publication and a US patent application patent publication belonging to the same patent family. Or it is a pair of a Japanese newspaper article and an English version of the newspaper article. Alternatively, it is a pair of an English manual of open source software and a Japanese translation of the manual.

  The parallel translation document acquisition unit 102 acquires a parallel translation document from the parallel translation document storage unit 101.

  The word division unit 103 divides the bilingual document acquired by the bilingual document acquisition unit 102 into sentences and divides each sentence into words. For Japanese documents, morphological analysis is performed to divide the sentence into sentences with clues as clues, and each sentence into words. At that time, the utilization word may be converted into a basic form. For an English document, it is divided into sentences using a period as a clue, and each sentence is divided into words using a space as a clue. At that time, the ending word may be analyzed to convert the utilization word into a basic form. Further, upper case letters may be converted to lower case letters, and plural forms may be converted to singular forms.

  The bilingual dictionary storage unit 104 stores a bilingual dictionary. Here, the bilingual dictionary is a set of pairs of Japanese words and English words having the same meaning as the words.

  The bilingual sentence acquisition unit 105 matches the Japanese sentence and the English sentence cut out by the word dividing unit 103 using a bilingual dictionary stored in the bilingual dictionary storage unit 104, and thereby obtains one or more of Japanese and English Get the translation of Specifically, the bilingual sentence acquisition unit 105 translates the Japanese sentence cut out by the word dividing unit 103 into English, calculates the similarity between the created translated sentence and each English sentence, and calculates the similarity The pair of the English sentence with the maximum degree and the above Japanese sentence is acquired as a parallel translation. Here, the similarity is a value calculated based on the number of words that match between the created translated sentence and the English sentence. More specifically, it is a value expressed by the ratio of the number of independent words that match between the numbers of all independent words included in the translated sentence and the English sentence. For example, the bilingual sentence acquisition unit 105 acquires bilingual sentences using DP (Dynamic Programming) matching as in the bilingual corpus creation method described in the above non-patent document. As another example, the bilingual sentence acquisition unit 105 uses the DP matching as described in Takehito Utsuro, et al. “Bilingual Text Matching using Bilingual Dictionary and Statistics,” COLING, p.1076-1082, 1994. May be obtained. Here, the bilingual sentence is a pair of a Japanese sentence and an English sentence created by translating the sentence into English. In other words, it is a pair of a Japanese sentence and an English sentence having the same meaning as the sentence.

  The parallel translation storage unit 106 stores one or more parallel translations (in other words, parallel corpus) acquired by the parallel translation acquisition unit 105. At that time, the parallel translation storage unit 106 stores each parallel translation in association with a parallel translation ID that identifies the parallel translation.

  The translation model generation unit 107 generates a translation model based on one or more parallel translations stored in the parallel translation storage unit 106. At that time, the translation model generation unit 107 generates a translation model using, for example, a Moses decoder (http://www.statmt.org/moses/). See, for example, Philipp Koehn, et al. “Moses: Open Source Toolkit for Statistical Machine Translation,” Annual Meeting of the Association for Computational Linguistics, demonstration session, Prague, Czech Republic, June 2007.

  For each of one or more parallel translations stored in the parallel translation storage 106, the translation unit 108 uses the translation model generated by the translation model generation unit 107 to convert the Japanese sentences constituting the parallel translation into English. Translate to The translation unit 108 stores the created translated sentence in the parallel sentence storage unit 106 in association with the original Japanese sentence.

  The edit distance calculation unit 109 calculates, for each of one or more parallel translations stored in the parallel translation storage 106, a Japanese sentence translated into English by the translation part 108 and an English sentence corresponding to the sentence. The edit distance between them is calculated. Here, the editing distance is a value calculated based on the number of editing operations required to change a Japanese sentence translated into English into an English sentence corresponding to the sentence. Specifically, the edit distance calculation unit 109 calculates TER (Translation Error Rate) as the edit distance. Here, the editing operation is specifically four operations of insertion, deletion, replacement, and rearrangement. See, for example, Matthew Snover, et al. “A study of translation edit rate with targeted human annotation,” Proceedings of Association for Machine Translation in the Americas, p. 223-231, 2006. After calculating the TER, the edit distance calculation unit 109 stores the calculated TER in the bilingual sentence storage unit 106 in association with the corresponding bilingual sentence.

  The bilingual sentence selection unit 110 selects a bilingual sentence whose TER calculated by the edit distance calculation unit 109 is larger than a threshold value from among one or more bilingual sentences stored in the bilingual sentence storage unit 106. Here, the threshold value is set so that, for example, a predetermined percentage of parallel translation sentences are selected from one or more parallel translation sentences stored in the parallel translation storage section 106. When the parallel translation is selected, the edit distance calculation unit 109 notifies the parallel translation editing unit 111 of the parallel translation ID of the parallel translation as a deletion target.

  The parallel translation editing unit 111 deletes the parallel translation identified by the parallel translation ID notified from the parallel translation selection unit 110 from the parallel translation storage unit 106.

1-2. Operation The operation of the bilingual sentence extraction device 1 will be described. FIG. 2 is a flowchart showing an example of the bilingual sentence extraction process executed by the bilingual sentence extracting apparatus 1.

  In step S <b> 1 of the parallel translation extraction process, the parallel translation document acquisition unit 102 of the parallel translation extraction apparatus 1 acquires a parallel translation document from the parallel translation document storage unit 101. FIG. 3 is a diagram illustrating an example of a bilingual document.

  When the bilingual document is acquired by the bilingual document acquisition unit 102, the word dividing unit 103 divides the acquired bilingual document into sentences and divides each sentence into words (step S2). For Japanese documents, morphological analysis is performed to divide the sentence into sentences with clues as clues, and each sentence into words. For an English document, it is divided into sentences using a period as a clue, and each sentence is divided into words using a space as a clue.

  When the bilingual document is divided into sentences by the word dividing unit 103 and each sentence is divided into words, the bilingual sentence obtaining unit 105 divides the Japanese sentence and the English sentence cut out by the word dividing unit 103, Matching is performed using the bilingual dictionary stored in the bilingual dictionary storage unit 104 to obtain one or more bilingual sentences in Japanese and English (step S3). When acquiring one or more parallel translations, the parallel translation acquisition unit 105 stores each parallel translation in association with the parallel translation ID in the parallel translation storage unit 106 (step S4). FIG. 4 is a diagram illustrating an example of data in the parallel translation storage unit 106 in which the parallel translation is stored by the parallel translation acquisition unit 105.

  When one or more parallel translations are stored in the parallel translation storage 106 by the parallel translation acquisition unit 105, the translation model generation unit 107 generates a translation model based on the one or more stored parallel translations (step S5). .

  When the translation model is generated by the translation model generation unit 107, the translation unit 108 uses the generated translation model for each of the one or more parallel translations stored in the parallel translation storage unit 106. The constituent Japanese sentence is translated into English (step S6). The translation unit 108 stores the created translated sentence in the parallel sentence storage unit 106 in association with the original Japanese sentence. FIG. 5 is a diagram illustrating an example of data in the parallel translation storage unit 106 in which the translation is stored by the translation unit 108.

  When the Japanese sentence constituting the parallel translation is translated into English by the translation unit 108, the edit distance calculation unit 109 translates each of the one or more parallel translations stored in the parallel translation storage unit 106 into English. The TER between the Japanese sentence and the English sentence corresponding to the sentence is calculated (step S7). After calculating the TER, the edit distance calculation unit 109 stores the calculated TER in the bilingual sentence storage unit 106 in association with the corresponding bilingual sentence. FIG. 6 is a diagram illustrating an example of data in the parallel translation storage unit 106 in which the TER is stored by the edit distance calculation unit 109. The “total” of TER shown in the figure is a value obtained by summing the number of operations of insertion, deletion, replacement, and rearrangement.

  When the TER is calculated by the edit distance calculation unit 109, the bilingual sentence selection unit 110 selects a bilingual sentence having a TER larger than a threshold value from one or more parallel translations stored in the bilingual sentence storage unit 106 (Step S1). S8). When the parallel translation is selected, the edit distance calculation unit 109 notifies the parallel translation editing unit 111 of the parallel translation ID of the parallel translation as a deletion target. For example, if the threshold is set to “5”, the parallel translation selection unit 110 notifies the parallel translation editing unit 111 of the parallel translation ID “032” among the parallel translations shown in FIG.

When the parallel translation ID is notified from the parallel translation selection unit 110, the parallel translation editing unit 111 deletes the parallel translation identified by the parallel translation ID from the parallel translation storage 106 (step S9).
The above is the description of the bilingual sentence extraction process.

  According to the bilingual sentence extraction apparatus 1 described above, after the bilingual sentence is acquired from the bilingual document by the bilingual sentence acquiring unit 105 using DP matching, the bilingual sentence selecting unit 110 selects the bilingual sentence from the acquired bilingual sentences. , Sorting is performed based on TER. Therefore, according to this bilingual document storage unit 101, it is possible to create a bilingual corpus having higher quality compared to a case where sentence matching is performed based only on the number of matching words.

2. Modifications The above embodiments may be modified as described below. One or more modifications described below may be combined with each other.

2-1. Modification 1
The parallel translation extraction apparatus 1 according to the above embodiment may be a computer system including a plurality of computers. The storage device included in the bilingual sentence extraction device 1 according to the above embodiment may be connected to the bilingual sentence extraction device 1 via a communication line such as the Internet.

2-2. Modification 2
In the above embodiment, the first language may be English and the second language may be Japanese. The combination of the first language and the second language may be arbitrarily selected from natural languages such as German, French, Chinese, Korean, etc. in addition to Japanese and English.

2-3. Modification 3
The translation model generation unit 107 may generate a translation model using a decoder other than the Moses decoder. For example, a translation model may be generated using a Pharaoh decoder. For Pharaoh decoders, see, for example, Philipp Koehn, "Pharaoh: A Beam Search Decoder for Phrase-Based Statistical Machine Translation Models," Proceedings of the 6th Conference of the Association for Machine Translation in the Americas, p. 115-124, 2004. See

2-4. Modification 4
In the above embodiment, the translation model generation unit 107 is omitted, and the translation unit 108 uses the predetermined translation model for each of the one or more parallel translations stored in the parallel translation storage unit 106. Japanese sentences constituting the sentence may be translated into English.

2-5. Modification 5
The edit distance calculation unit 109 according to the above embodiment may calculate a value other than TER as the edit distance. For example, a Levenshtein distance, a Damerau-Levenshtein distance, or a Jaro-Winkler distance may be calculated.

As another example, the edit distance calculation unit 109 corresponds to each of one or more parallel translations stored in the parallel translation storage unit 106 and a Japanese sentence translated into English by the translation unit 108. BLUE or RIBES between English sentences may be calculated. For BLUE, for example,
Kishore Papineni, et al. "BLUE: a method for automatic evaluation of machine translation," Proc. Of the 40th Annual Meeting of the Association for Computational Linguistics, p.311-318, July 2002. For RIBES, see, for example, Tsutomu Hirao, et al. “RIBES: Automatic Evaluation of Translation Based on Rank Correlation”, Proc. Of the 17th Annual Conference of the Language Processing Society, p.1115-1118, March 2011 about.

2-6. Modification 6
The bilingual sentence selection unit 110 according to the above embodiment selects bilingual sentences whose TER is larger than the threshold, and then selects bilingual sentences other than the selected bilingual sentences and stored in the bilingual sentence storage unit 106. The bilingual sentence ID may be notified to the bilingual sentence editing unit 111 as a bilingual sentence that remains without being deleted. In this case, the bilingual sentence editing unit 111 deletes the bilingual sentence other than the bilingual sentence identified by the bilingual sentence ID notified from the bilingual sentence selecting unit 110 from the bilingual sentence storing unit 106.

2-7. Modification 7
A program for realizing each function of the bilingual sentence extraction apparatus 1 may be provided via a recording medium readable by a computer apparatus. Here, the recording medium is, for example, a magnetic recording medium such as a magnetic tape or a magnetic disk, an optical recording medium such as an optical disk, a magneto-optical recording medium, or a semiconductor memory. In addition, this program may be provided via a network such as the Internet.

DESCRIPTION OF SYMBOLS 1 ... Bilingual sentence extraction apparatus, 101 ... Bilingual document memory | storage part, 102 ... Bilingual document acquisition part, 103 ... Word division part, 104 ... Bilingual dictionary memory | storage part, 105 ... Bilingual sentence acquisition part, 106 ... Bilingual sentence memory | storage part, 107 ... Translation model generation unit, 108 ... Translation unit, 109 ... Editing distance calculation unit, 110 ... Parallel translation selection unit, 111 ... Parallel translation editing unit

Claims (3)

A bilingual document acquisition unit for acquiring bilingual documents in the first language and the second language;
The first language sentence and the second language sentence constituting the acquired bilingual document are matched using the first language and the second language bilingual dictionary, and the first language and the second language sentence are matched. A bilingual sentence acquisition unit for acquiring one or more bilingual sentences in two languages;
A translation model generation unit that generates a translation model based on the one or more acquired parallel translations;
For each of the obtained one or more parallel translation sentences, using the generated translation model, a translation unit that translates the sentence of the first language constituting the parallel translation sentence into the second language;
Editing that calculates an edit distance between the sentence in the first language translated into the second language and the sentence in the second language corresponding to the sentence for each of the one or more acquired parallel translations A distance calculator;
A bilingual sentence extraction device comprising: a bilingual sentence selecting unit that selects a bilingual sentence whose calculated edit distance is larger than a threshold value among the one or more acquired bilingual sentences. A bilingual sentence extraction method executed by one or more computers,
Obtaining a bilingual document in a first language and a second language;
The first language sentence and the second language sentence constituting the acquired bilingual document are matched using the first language and the second language bilingual dictionary, and the first language and the second language sentence are matched. Obtaining one or more parallel translations in two languages;
Generating a translation model based on the acquired one or more parallel translations;
For each of the obtained one or more parallel translation sentences, using the generated translation model, translating the sentence of the first language constituting the parallel translation sentence into the second language;
Calculating an edit distance between the sentence in the first language translated into the second language and the sentence in the second language corresponding to the sentence for each of the obtained one or more parallel translation sentences; When,
A bilingual sentence extracting method comprising: selecting a bilingual sentence whose calculated edit distance is greater than a threshold value from the one or more acquired bilingual sentences. On the computer,
Obtaining a bilingual document in a first language and a second language;
The first language sentence and the second language sentence constituting the acquired bilingual document are matched using the first language and the second language bilingual dictionary, and the first language and the second language sentence are matched. Obtaining one or more parallel translations in two languages;
Generating a translation model based on the acquired one or more parallel translations;
For each of the obtained one or more parallel translation sentences, using the generated translation model, translating the sentence of the first language constituting the parallel translation sentence into the second language;
Calculating an edit distance between the sentence in the first language translated into the second language and the sentence in the second language corresponding to the sentence for each of the obtained one or more parallel translation sentences; When,
A program for executing the step of selecting, from among the one or more acquired parallel translations, a parallel translation in which the calculated editing distance is greater than a threshold value.

Images