JP2007058311A - Corpus addition apparatus and corpus addition method - Google Patents

Abstract

A corpus for better natural language processing is created.
A first corpus storage unit 11 in which a first corpus is stored, a second corpus storage unit 12 in which a second corpus is stored, and similarity factors of subsets in the first corpus are stored. A first similarity coefficient storage unit 16, a second similarity coefficient storage unit 17 that stores similarity coefficients of subsets in the second corpus, and a set of subsets of the second corpus, the first corpus A subset extractor 18 for extracting a set of subsets having the same representative value as the representative value of the similarity coefficient in the second corpus, a set of subsets of the second corpus extracted by the subset extractor 18, An adder 19 for adding the corpus and an output unit 20 for outputting the corpus added by the adder 19 are provided.
[Selection] Figure 1

Description

  The present invention relates to a corpus adder that adds at least a part of another corpus to one corpus.

  Research has been conducted on what corpora are used to perform better processing in corpus-based natural language processing (for example, machine translation and summarization). For example, as a corpus used in natural language processing, it has been proposed to use a homogeneous one instead of a heterogeneous one (for example, Non-Patent Document 1).

As a related technique, it has been proposed to use a coefficient based on cross entropy as an index indicating the homogeneity of a corpus (for example, Non-Patent Document 2).
Gabriela Cavaglia, “Measuring corpus homogeneity using a range of measures for inter-document distance”, Processings of LREC, 2002, pp. 426-431 Etienne Annual, A method to qualitative corporate similarity and its application to qualitatively qualifying in the human beings, Proceeding and Quantitative

  As described above, there is a need to provide a corpus for better processing in natural language processing.

  The present invention has been made in the above situation, and provides a corpus adder and the like that creates a corpus for better natural language processing by adding at least a part of another corpus to one corpus The purpose is to do.

  In order to achieve the above object, a corpus adder according to the present invention includes a first corpus storage unit that stores a first corpus, a second corpus storage unit that stores a second corpus, and the first corpus. A first similarity coefficient storage unit storing a subset similarity coefficient in the second corpus, a second similarity coefficient storage unit storing a subset similarity coefficient in the second corpus, and a subset of the second corpus A subset extractor that extracts from the second corpus a subset set that has the same representative value as the representative value of the similarity coefficient of the subset in the first corpus, and the subset extractor extracts An adder that adds the set of subsets of the second corpus and the first corpus; and an output unit that outputs the corpus added by the adder; It is those with a.

  With such a configuration, a corpus for better natural language processing can be created, and higher performance can be obtained by executing natural language processing using the corpus.

In the corpus adder according to the present invention, a first reference corpus storage unit that stores a first reference corpus, a second reference corpus storage unit that stores a second reference corpus, and the first reference corpus storage The first reference corpus stored in the first corpus and the second reference corpus stored in the second reference corpus storage as a reference corpus is used as the first reference corpus stored in the first corpus. A similarity coefficient calculating unit that calculates a similarity coefficient of each subset in the second corpus stored in the corpus and the second corpus storage unit, the first similarity coefficient storage unit storing The similarity coefficient of the subset in the first corpus and the similarity coefficient of the subset in the second corpus stored in the second similarity coefficient storage unit are: The similarity coefficient calculating unit may be obtained by calculation.
With this configuration, the similarity coefficient can be calculated using the first reference corpus and the second reference corpus.

  In the corpus adder according to the present invention, the subset extracting unit may add the similarity coefficient distribution shape in the corpus as a result of addition to the first corpus to the similarity coefficient distribution shape in the first corpus. A subset to be added to the first corpus may be extracted from the second corpus so as to be the same.

  With such a configuration, as a result, it is possible to create a corpus in which a heterogeneous component is added to the first corpus while maintaining the representative value of the first corpus. Language processing can be performed.

  In the corpus adder according to the present invention, the subset extraction unit may obtain a similarity coefficient distribution obtained by multiplying a similarity coefficient distribution in the first corpus by a predetermined value to obtain a similarity coefficient in the second corpus. The second corpus subset set having the same distribution as the similarity coefficient distribution obtained by multiplying the distribution of similarity coefficients in the first corpus by the predetermined value. You may extract from the corpus.

  With such a configuration, at least a part of the second corpus can be added to the first corpus so as not to change the shape of the distribution of similarity coefficients in the first corpus.

  In the corpus adder according to the present invention, the predetermined value may be a similarity coefficient distribution obtained by multiplying the distribution of the similarity coefficient in the first corpus by the predetermined value, and the similarity in the second corpus. It may be the largest value when included in the coefficient distribution.

  With such a configuration, a set of as many subsets of the second corpus as possible can be added to the first corpus, and as a result, a corpus having a large number of subsets can be created, and better natural language processing can be performed. You can create a corpus to run.

  According to the corpus adder and the like according to the present invention, a corpus capable of executing better natural language processing can be created.

  Hereinafter, a corpus adder and the like according to the present invention will be described using embodiments. In the following embodiments, components and steps denoted by the same reference numerals are the same or equivalent, and repetitive description may be omitted.

(Embodiment 1)
A corpus adder according to Embodiment 1 of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing the configuration of a corpus adder 1 according to this embodiment. In FIG. 1, the corpus adder 1 according to the present embodiment is similar to a first corpus storage unit 11, a second corpus storage unit 12, a first reference corpus storage unit 13, and a second reference corpus storage unit 14. A degree coefficient calculation unit 15, a first similarity coefficient storage unit 16, a second similarity coefficient storage unit 17, a subset extraction unit 18, an addition unit 19, and an output unit 20.

  The first corpus storage unit 11 stores the first corpus. Here, the first corpus is a corpus to which all of them are added in the corpus adder 1. A corpus is an electronic language (document) database. In addition, the process in which a 1st corpus is memorize | stored in the 1st corpus memory | storage part 11 is not ask | required. For example, the first corpus may be stored in the first corpus storage unit 11 via a recording medium, and the first corpus transmitted via a communication line or the like is stored in the first corpus storage unit 11. The first corpus input via an input device or the like may be stored in the first corpus storage unit 11. The first corpus storage unit 11 can be realized by a predetermined recording medium (for example, a semiconductor memory, a magnetic disk, an optical disk, etc.). The storage in the first corpus storage unit 11 may be temporary storage in the first corpus RAM read from an external storage device or the like, or may be long-term storage in a hard disk or the like.

  The second corpus storage unit 12 stores the second corpus. Here, the second corpus is a corpus to which at least a part of the corpus is added in the corpus adder 1. In addition, the process in which a 2nd corpus is memorize | stored in the 2nd corpus memory | storage part 12 is not ask | required. For example, the second corpus may be stored in the second corpus storage unit 12 via a recording medium, and the second corpus transmitted via a communication line or the like is stored in the second corpus storage unit 12. Alternatively, the second corpus input via an input device or the like may be stored in the second corpus storage unit 12. The second corpus storage unit 12 can be realized by a predetermined recording medium (for example, a semiconductor memory, a magnetic disk, an optical disk, etc.). The storage in the second corpus storage unit 12 may be temporary storage in the RAM of the second corpus read from an external storage device or the like, or may be long-term storage in a hard disk or the like.

  The first reference corpus storage unit 13 stores a first reference corpus. Here, the first reference corpus is used as one reference corpus when the similarity coefficient calculation unit 15 described later calculates a similarity coefficient such as a first corpus. The reference corpus is a corpus that serves as a reference when calculating the similarity coefficient. The process of storing the first reference corpus in the first reference corpus storage unit 13 does not matter. For example, the first reference corpus may be stored in the first reference corpus storage unit 13 via the recording medium, and the first reference corpus transmitted via the communication line or the like is the first reference corpus. The first reference corpus may be stored in the storage unit 13, or the first reference corpus input via an input device or the like may be stored in the first reference corpus storage unit 13. The first reference corpus storage unit 13 can be realized by a predetermined recording medium (for example, a semiconductor memory, a magnetic disk, an optical disk, etc.). The storage in the first reference corpus storage unit 13 may be temporary storage in the RAM of the first reference corpus read from an external storage device or the like, or may be long-term storage in a hard disk or the like.

  The second reference corpus storage unit 14 stores a second reference corpus. Here, the second reference corpus is used as one reference corpus when the similarity coefficient calculation unit 15 described later calculates a similarity coefficient such as the first corpus. The first reference corpus and the second reference corpus are preferably separated as much as possible. A corpus is distant means that the corpora are not similar. For example, a colloquial corpus may be used as the first reference corpus, and a sentence corpus may be used as the second reference corpus. Further, for example, a corpus of words that can be used as the first reference corpus may be used, and a corpus of polite words may be used as the second reference corpus. Further, for example, a new-style word corpus may be used as the first reference corpus, and an old-style word corpus may be used as the second reference corpus. Although the first reference corpus and the second reference corpus are illustrated, the first reference corpus and the second reference corpus may be reversed. Note that the process of storing the second reference corpus in the second reference corpus storage unit 14 does not matter. For example, the second reference corpus may be stored in the second reference corpus storage unit 14 via a recording medium, and the second reference corpus transmitted via a communication line or the like may be stored in the second reference corpus. The second reference corpus may be stored in the storage unit 14, or the second reference corpus input via an input device or the like may be stored in the second reference corpus storage unit 14. The second reference corpus storage unit 14 can be realized by a predetermined recording medium (for example, a semiconductor memory, a magnetic disk, an optical disk, etc.). The storage in the second reference corpus storage unit 14 may be temporary storage in the RAM or the like of the second reference corpus read from an external storage device or the like, or may be long-term storage in a hard disk or the like.

  Note that any two or more storage units of the first corpus storage unit 11, the second corpus storage unit 12, the first reference corpus storage unit 13, and the second reference corpus storage unit 14 may be realized by the same recording medium. Alternatively, it may be realized by a separate recording medium. In the former case, for example, an area that stores the first corpus is the first corpus storage unit 11, and an area that stores the second corpus is the second corpus storage unit 12.

The similarity coefficient calculation unit 15 uses the first reference corpus stored in the first reference corpus storage unit 13 and the second reference corpus stored in the second reference corpus storage unit 14 as a reference corpus. The similarity coefficient of the subset in the first corpus stored in the first corpus storage unit 11 is calculated. Further, the similarity coefficient calculation unit 15 uses the first reference corpus stored in the first reference corpus storage unit 13 and the second reference corpus stored in the second reference corpus storage unit 14 as a reference corpus. As described above, the similarity coefficient of the subset in the second corpus stored in the second corpus storage unit 12 is calculated. Here, the similarity coefficient is an index indicating which one of two reference corpora is close to a certain corpus (including a corpus subset). For example, when the similarity coefficient I (A) is calculated using cross-entropy, it is as follows.

From the above equation, I (T1) = 0 and I (T2) = 1. Therefore, the value of the similarity coefficient I (A) from 0 to 1 indicates whether the test corpus A is close to the corpus T1 or the corpus T2. Further, the cross entropy H _T (A) of the N-gram model p constructed in the training corpus T and the test corpus A is as follows. However, the test corpus A has A = {s ₁ ,. . , S _Q }, and so on.

Each sentence s _i is composed of | s _i | characters as follows.

The similarity coefficient calculation unit 15 uses the first reference corpus as the training corpus T1, the second reference corpus as the training corpus T2, the first corpus subset as the test corpus A, and the similarity of the first corpus subset. Calculate the degree factor. The same applies when calculating the similarity coefficient of the subset of the second corpus. Here, the corpus subset is a subset of the corpus having an arbitrary size. The subset may be, for example, one sentence, a set of two or more sentences, or a set of phrases. In this embodiment, a case where the subset is one sentence (one sentence) will be described. In the present embodiment, the case where cross entropy is used in the calculation of the similarity coefficient will be described. However, the similarity coefficient may be calculated by other methods. For example, the similarity coefficient may be calculated using a χ ² test or the like instead of the cross entropy. Further, the similarity coefficient calculation unit 15 may calculate the similarity coefficient using the first reference corpus as the training corpus T2 and the second reference corpus as the training corpus T1.

  The first similarity coefficient storage unit 16 stores the similarity coefficient of the subset in the first corpus. This similarity coefficient is calculated by the similarity coefficient calculation unit 15. The first similarity coefficient storage unit 16 can be realized by a predetermined recording medium (for example, a semiconductor memory, a magnetic disk, an optical disk, etc.).

  The second similarity coefficient storage unit 17 stores the similarity coefficient of the subset in the second corpus. This similarity coefficient is calculated by the similarity coefficient calculation unit 15. The second similarity coefficient storage unit 17 can be realized by a predetermined recording medium (for example, a semiconductor memory, a magnetic disk, an optical disk, etc.).

  The first similarity coefficient storage unit 16 and the second similarity coefficient storage unit 17 may be realized by the same recording medium as the first corpus storage unit 11, the second corpus storage unit 12, and the like, or separately. It may be realized by the recording medium. In the former case, for example, a similarity coefficient may be stored in association with each corpus subset. In the latter case, for example, in the first corpus storage unit 11 or the like, a subset of the corpus is stored in association with a subset identifier for identifying the subset, and in the first similarity coefficient storage unit 16 or the like. The similarity coefficient of the subset identified by the subset identifier may be stored in association with the subset identifier.

  The subset extracting unit 18 extracts a set of subsets to be added to the first corpus in the adding unit 19 from the second corpus stored in the second corpus storage unit 12. Here, the subset set of the second corpus extracted by the subset extraction unit 18 has the same representative value as the representative value of the similarity coefficient in the first corpus. Here, the representative value is a value representing the distribution (group) of similarity coefficient, and the representative value may be an average value or a median value, for example. As a result, the subset extractor 18 only needs to have the same representative value as the representative value of the similarity coefficient in the first corpus in the subset of the second corpus extracted by the subset extractor 18. In the first corpus and the second corpus, the representative value may or may not be calculated. When the subset extraction unit 18 calculates a representative value such as the first corpus, the representative is based on the similarity coefficient stored in the first similarity coefficient storage unit 16 and the second similarity coefficient storage unit 17. A value may be calculated. In this embodiment, a case where an average value is used as a representative value will be described. A method in which the subset extracting unit 18 extracts a set of subsets of the second corpus having the same average value as the average value of the similarity coefficient in the first corpus will be described later.

The adder 19 adds the subset of the second corpus extracted by the subset extractor 18 and the first corpus stored in the first corpus storage 11.
The output unit 20 outputs the corpus added by the adding unit 19. Here, the output may be, for example, display on a display device (for example, a CRT or a liquid crystal display), transmission via a communication line to a predetermined device, or accumulation in a recording medium. In the present embodiment, it is assumed that the output unit 20 stores the corpus added by the adding unit 19 in the recording medium 21. The recording medium 21 may or may not be the same recording medium as the first corpus storage unit 11 or the like. Note that the output unit 20 may or may not include a device (for example, a display device) that performs output. The output unit 20 may be realized by hardware, or may be realized by software such as a driver that drives these devices.

  The first corpus is a corpus used in natural language processing, and may or may not be a corpus that has been adjusted in advance with respect to the processing target of natural language processing. Further, the second corpus may be anything as long as the similarity coefficient distribution is not a Dirac delta function distribution but has a certain extent.

  Next, the operation of the corpus adder 1 according to this embodiment will be described using the flowchart of FIG. FIG. 2 is a flowchart showing a process after the process of adding the first corpus and at least a part of the second corpus is started.

  (Step S101) The similarity coefficient calculation unit 15 reads the first corpus from the first corpus storage unit 11, and calculates the similarity coefficient of a subset of the first corpus. At the time of the calculation, the first reference corpus stored in the first reference corpus storage unit 13 and the second reference corpus stored in the second reference corpus storage unit 14 are used as the reference corpus. Further, the calculated similarity coefficient is stored in the first similarity coefficient storage unit 16.

  (Step S102) The similarity coefficient calculation unit 15 reads the second corpus from the second corpus storage unit 12, and calculates the similarity coefficient of the subset of the second corpus. At the time of the calculation, the first reference corpus stored in the first reference corpus storage unit 13 and the second reference corpus stored in the second reference corpus storage unit 14 are used as the reference corpus. Further, the calculated similarity coefficient is stored in the second similarity coefficient storage unit 17.

  (Step S <b> 103) The subset extraction unit 18 stores the similarity coefficient of each subset in the first corpus stored in the first similarity coefficient storage unit 16 and the second similarity coefficient storage unit 17. Using the similarity coefficient of each subset in the two corpora, an adder 19 determines a set of subsets of the second corpus to be added to the first corpus. Details of this processing will be described later.

  (Step S104) The subset extraction unit 18 extracts a set of subsets of the second corpus determined in step S103 from the second corpus storage unit 12.

  (Step S <b> 105) The adding unit 19 adds the subset of the second corpus extracted by the subset extracting unit 18 in step S <b> 104 and the first corpus stored in the first corpus storage unit 11.

  (Step S <b> 106) The output unit 20 stores the corpus added by the adding unit 19 in the recording medium 21. Then, the process ends.

  Next, the operation of the corpus adder 1 according to this embodiment will be described using a specific example. In this specific example, when a set of subsets of the second corpus is extracted using the distribution shape of the subset similarity coefficient in the first corpus (specific example 1), and the similarity of the subsets in the first corpus A case will be described where a set of subsets of the second corpus having the same average value as the average value of the similarity coefficients of the subsets in the first corpus is extracted regardless of the shape of the coefficient distribution (specific example 2). In addition, about the description common to the specific example 1 and the specific example 2, the description in the specific example 2 may be abbreviate | omitted.

[Specific Example 1]
In this specific example, a Japanese corpus is used. An SLDB (Spontaneous Speech Database) corpus is used as the first reference corpus. Further, as the second reference corpus, a corpus composed of Nikkei newspaper articles is used.

  In this specific example, as described above, a subset set is extracted using the shape of the distribution of similarity coefficients. Here, the distribution of the similarity coefficient is a distribution in which the horizontal axis represents the similarity coefficient and the vertical axis represents the number of subsets.

  The subset extraction unit 18 makes the distribution shape of the similarity coefficient in the corpus as a result of addition to the first corpus the same as the distribution shape of the similarity coefficient in the first corpus calculated by the similarity coefficient calculation unit 15. It is assumed that a subset to be added to the first corpus is extracted from the second corpus. Here, the shape of the distribution of one similarity coefficient and the shape of the distribution of another similarity coefficient are the same when the number of subsets in the distribution of one similarity coefficient is multiplied by a predetermined number. This is the case where the similarity coefficient distribution is the same. That is, all the similarity coefficient distributions multiplied by a predetermined value in the vertical axis direction have the same distribution shape.

  More specifically, the subset extraction unit 18 extracts a set of subsets as follows. The subset extracting unit 18 expands or reduces the similarity coefficient distribution obtained by multiplying the distribution of the similarity coefficient in the first corpus by a predetermined value (that is, expands or reduces in the vertical axis direction of the similarity coefficient distribution of the first corpus). 2) having the same distribution as the distribution of similarity coefficients obtained by multiplying the distribution of similarity coefficients in the first corpus by a predetermined value when the distribution of similarity coefficients is included in the distribution of similarity coefficients in the second corpus. A set of corpora subsets is extracted from the second corpus. In this specific example, the predetermined value includes the similarity coefficient distribution obtained by multiplying the distribution of the similarity coefficient in the first corpus by the predetermined value in the distribution of the similarity coefficient in the second corpus. It is assumed that it is the largest value in the case where

  When the corpus addition process is started, first, the similarity coefficient calculation unit 15 sends the first corpus, the first corpus storage unit 11, the first reference corpus storage unit 13, and the second reference corpus storage unit 14 to each other. The first reference corpus and the second reference corpus are read out, and the subset similarity coefficient in the first corpus is calculated using the above-described formula (step S101). A subset of the first corpus is used as the test corpus of the above equation. The similarity coefficient of the first corpus calculated by the similarity coefficient calculation unit 15 is stored in the first similarity coefficient storage unit 16.

FIG. 3 is a diagram illustrating an example of a subset similarity coefficient in the first corpus stored in the first similarity coefficient storage unit 16. As shown in FIG. 3, the subset identifier is associated with the similarity coefficient. Of course, as described above, the similarity coefficient may be stored in direct association with the subset of the first corpus in the first corpus storage unit 11. FIG. 4 is a diagram showing a distribution of similarity coefficients of the first corpus stored in the first similarity coefficient storage unit 16. In the distribution shown in FIG. 4, the average value I _{0 of the} distribution is 0.45. In FIG. 4, the similarity coefficient has discrete values. In FIG. 4, the values obtained by interpolating the values are shown as distributions. The same applies to the distribution of other similarity coefficients.

  Next, the similarity coefficient calculation unit 15 includes the second corpus, the first reference corpus, and the second reference from the second corpus storage unit 12, the first reference corpus storage unit 13, and the second reference corpus storage unit 14. Each corpus is read out and the similarity coefficient of the subset in the second corpus is calculated using the above-described equation (step S102). A subset of the second corpus is used as the test corpus of the above equation. The similarity coefficient of the second corpus calculated by the similarity coefficient calculation unit 15 is stored in the second similarity coefficient storage unit 17. In the second similarity coefficient storage unit 17 as well, as in FIG. 3, it is assumed that the similarity coefficient is stored in association with the subset identifier. FIG. 5 is a diagram illustrating a distribution of similarity coefficients of the second corpus stored in the second similarity coefficient storage unit 17.

  The subset extraction unit 18 refers to the first similarity coefficient storage unit 16 and the second similarity coefficient storage unit 17 to obtain the distribution of the similarity coefficient shown in FIGS. The subset extraction unit 18 then multiplies the first corpus similarity coefficient distribution by a predetermined value in the vertical axis direction so as to be included in the second corpus similarity coefficient distribution. Find the value. In this specific example, it is assumed that the value is 0.34. That is, as shown in FIG. 5, the distribution of the similarity coefficient of the first corpus multiplied by 0.34 is included in the distribution of the similarity coefficient of the second corpus. Therefore, the subset extracting unit 18 extracts a set of subsets of the second corpus having a distribution obtained by multiplying the distribution of the similarity coefficient of the first corpus shown in FIG. 5 by 0.34 from the second corpus. A subset is determined (step S103). Since the subset set extracted from the second corpus is determined in this way, as a result, the distribution of similarity coefficients in the corpus as a result of adding the extracted subset set and the first corpus Is the same as the shape of the similarity coefficient distribution of the first corpus.

  The subset extraction unit 18 extracts a set of subsets determined to be extracted from the second corpus storage unit 12. For example, in the distribution obtained by multiplying the similarity coefficient distribution of the first corpus shown in FIG. 5 by 0.34, when the number of subsets corresponding to the similarity coefficient 0.5 is 190, the subset extraction unit 18 randomly selects 190 subset identifiers of the subset having the similarity coefficient of 0.5 from the second similarity coefficient storage unit 17 and stores the subset identified by the selected subset identifier in the second corpus. Extract from part 12. Such subset extraction is performed for all subset ranges determined to be extracted (step S104).

  The adding unit 19 adds the subset of the second corpus extracted by the subset extracting unit 18 and the first corpus read from the first corpus storage unit 11 (step S105). As shown in FIG. 6, the corpus after the addition has a distribution obtained by multiplying the distribution of the similarity coefficient of the first corpus in FIG. 4 by 1.34. The output unit 20 accumulates the corpus added by the adding unit 19 in the recording medium 21 (step S106). The user can obtain a better processing result by using the corpus stored in the recording medium 21 than when the first corpus is used. This will be described later.

  In this specific example, the distribution of similarity coefficients obtained by multiplying the similarity coefficient distribution in the first corpus by a predetermined value is included in the distribution of similarity coefficients in the second corpus. In the case described above, the second corpus subset set having the same distribution as the similarity coefficient distribution obtained by multiplying the similarity coefficient distribution in the first corpus by the maximum value is extracted from the second corpus. However, the subset extraction unit 18 includes the first similarity coefficient distribution obtained by multiplying the similarity coefficient distribution in the first corpus by a predetermined value in the similarity coefficient distribution in the second corpus. From the second corpus, a set of subsets of the second corpus having the same distribution as the similarity coefficient distribution obtained by multiplying the distribution of the similarity coefficient in the other corpus by a non-maximum value It may put.

[Specific Example 2]
This specific example is the same as specific example 1 except that the method for determining the subset set to be extracted from the second corpus is different. The description of the same process as that of the specific example 1 is omitted.

  Assume that the similarity coefficient calculation unit 15 calculates the similarity coefficients of the subsets in the first corpus and the second corpus and stores them in the first similarity coefficient storage unit 16 and the second similarity coefficient storage unit 17. Then, the subset extraction unit 18 obtains the distribution of the similarity coefficient of the subset in the second corpus in the same manner as in the first specific example. Here, as described above, it is assumed that the average value of the similarity coefficients of the subsets in the first corpus is 0.45. As shown in FIG. 7, the subset extractor 18 makes the similarity coefficient symmetric with respect to the average value of the first corpus in the distribution of the similarity coefficients of the subset in the second corpus (that is, in FIG. 7). The distribution of the subset set to be extracted is determined so that the distribution is maximized. Thereafter, the subset extracting unit 18 extracts a subset set having the determined distribution from the second corpus, and the process of adding the extracted subset set and the first corpus is a specific example. Same as 1.

  Since the subset set extracted from the second corpus has the same average value as the average value of the similarity coefficients of the subsets in the first corpus, the average of the similarity coefficients of the subsets in the resultant corpus is added. The value is also the same as the average value of the similarity coefficients of the subsets in the first corpus.

  It goes without saying that the method of extracting a subset set having the same average value as the average value of the similarity coefficient of the first corpus from the second corpus is not limited to the description of this specific example. For example, in the distribution of similarity coefficients of the subset in the second corpus, by determining the distribution of the subset set so that the similarity coefficient is symmetric with respect to the average value of the similarity coefficients of the first corpus The subset set having the same average value as the average value of the similarity coefficient of the first corpus can be easily extracted from the second corpus. Specifically, as shown in FIG. 8, the distribution of the subset set extracted from the second corpus may be determined. Further, if a subset set having the same average value as the average value of the similarity coefficient of the first corpus can be extracted from the second corpus, the distribution of the subset set is shown in FIGS. As shown in FIG.

  In the above two specific examples, the case where the subset extracting unit 18 extracts a set of subsets of the second corpus having the same representative value as the representative value of the similarity coefficient in the first corpus has been described. If the set of subsets of the second corpus extracted by the extraction unit 18 has the same representative value as the representative value of the similarity coefficient in the first corpus as a result, the extraction method is the above two specific examples. Needless to say, it is not limited to.

  Further, in each of the above specific examples, the similarity coefficient of the subset in the first corpus is only used to obtain the distribution of the similarity coefficient or to obtain the representative value, so the first similarity coefficient The storage unit 16 simply stores the similarity coefficient of the subset and does not have to be associated with the subset identifier as shown in FIG.

[Description of effects]
Hereinafter, it will be described that the corpus obtained as a result of addition as described in the present embodiment is a corpus having better performance in natural language processing than the first corpus.

  Here, the effect of the corpus adder 1 according to the present embodiment will be described by evaluating the performance of the probabilistic language model and the performance of machine translation based on an example from Japanese to English in the EBMT system.

  Note that SLDB is used as the first reference corpus. This SLDB has both an English corpus and a Japanese corpus. As the second reference corpus, Calcorp Corpus is used in English, and a corpus composed of Nikkei newspaper articles is used in Japanese. In addition, BTEC (Basic Traveller's Expression Corpus), which is a corpus of basic travel conversation in Japanese and English, is used as a corpus in which the similarity coefficient is calculated using these reference corpora. As a task (data actually processed for evaluation), a set of 510 Japanese sentences of BTEC not included in the corpus is used. Assume that the similarity coefficient of this task (here, the similarity coefficient of the entire task) is 0.331. The similarity coefficient of each sentence included in the task is also a value close to the similarity coefficient of the task. Further, a subset of the corpus is a sentence.

  FIG. 9 is a diagram showing the distribution of similarity coefficients of subsets (here, sentences) in Japanese BTEC. The average value in the distribution shown in FIG. 9 is “0.315”, and the standard deviation is “0.118”. FIG. 10 is a diagram showing a distribution of similarity coefficients of subsets (here, sentences) in English BTEC. The average value in the distribution shown in FIG. 10 is “0.313”, and the standard deviation is “0.156”.

  First, consider perplexity. Perplexity was calculated for the aforementioned task using data randomly selected from Japanese BTEC. In addition, selection of random data from Japanese BTEC was performed from 0.5% to 100% of BTEC. The result is shown by the solid line graph in FIG. In addition, data from Japanese BTECs was selected in order from the vicinity of the similarity coefficient “0.331” of the above-mentioned task, and perplexity was calculated for the above-mentioned task. In this case, since the similarity coefficient of the task and the similarity coefficient of BTEC are almost the same, the perplexity is calculated based on homogeneous data. The selection of homogeneous data from Japanese BTEC was also performed from 0.5% to 100% of BTEC. The result is shown by the dashed-dotted line graph in FIG. When 100% data is selected from Japanese BTEC, the same data is selected for randomly selected data or homogeneous data, so the perplexity is the same.

  In general, the lower the perplexity, the higher the performance in the stochastic model. As can be seen from FIG. 11, when the percentage of the corpus (Japanese BTEC) used in the calculation of perplexity is small, that is, up to about 15%, homogeneous data has a lower perplexity. It has become. On the other hand, if the ratio is more than that, homogeneous data has higher perplexity (that is, lower performance) than randomly selected data. From the above, with regard to perplexity, it can be seen that data selected at random, excluding areas with a small amount of data, is more useful than homogeneous data.

  When this result is applied to the description of the effect of the corpus adder 1 according to the present embodiment, a corpus including a larger number of heterogeneous subsets has lower perplexity for tasks, and higher performance can be obtained. become. Thus, the corpus resulting from adding a subset to the first corpus will contain more heterogeneous subsets than the first corpus, resulting in a lower perplexity and in natural language processing. It can be seen that high performance can be obtained.

  Next, we will examine machine translation from Japanese to English in the EBMT system. As in the case of the above-described study on perplexity, machine translation is executed for a task consisting of 510 sentences using Japanese and English BTECs. Also in this case, as in the case of perplexity, the calculation of the BLEU score shown in FIG. 12 and the case where random data is selected and the case where homogeneous data are selected are shown in FIG. The NIST score shown by 13 and the mWER score shown in FIG. 14 were calculated. In each graph of FIG. 12 to FIG. 14, the result of randomly selected data is indicated by a solid line, and the result of homogeneous data is indicated by a one-dot chain line. Note that the BLEU score, NIST score, and mWER score are described in the following (Reference 1), (Reference 2), and (Reference 3), respectively.

(Reference 1) Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation.

(Reference 2) George Dodington. 2002. Automatic evaluation of machine translation quality using n-gram co-ocurrence statistics. Proceedings of Human Lang. Technol. Conf. (HLT-02), pp. 138-145.

(Reference 3) Franz Josef Och. 2003. Minimum Error Rate Training in Statistical Machine Translation. Proceedings of ACL 2003, pp. 160-167.

  In FIG. 12, the BLEU score takes a value in the range of 0 to 1, and a higher score indicates better translation. In FIG. 13, the NIST score has a value of 0 or more, and a higher score indicates better translation. In FIG. 14, the mWER score takes a value in the range from 0 to 1, and a lower score indicates better translation. From these results, it can be seen that the quality of machine translation improves as the size of the BTEC used increases. Randomly selected data are more homogeneous in the three evaluation methods except for 3% of BTEC in BLEU score, 18% of BTEC in NIST score, and 2% of BTEC in mWER score. It can be seen that the quality of translation is equal to or higher than that of simple data.

  When this result is applied to the explanation of the effect of the corpus adder 1 according to the present embodiment, a corpus including a larger number of different subsets of tasks has higher machine translation quality and higher performance. Will be obtained. Therefore, the corpus resulting from adding the subset to the first corpus will contain more heterogeneous subsets than the first corpus, resulting in higher translation quality, It can be seen that high performance in natural language processing can be obtained.

  Thus, unlike the above-mentioned non-patent document 1, it can be seen that a corpus including a different material can execute better natural language processing. Although only perplexity and machine translation have been confirmed this time, it can be predicted that similar results can be obtained in other natural language processing such as summarization.

  In each of the above specific examples, when the similarity coefficient of the task and the representative value of the distribution of the similarity coefficient of the first corpus are close, that is, the first corpus has already been adjusted for the task. As described above, the first corpus used in the corpus adder 1 according to the present embodiment may or may not be a corpus adjusted for a task as described above.

  As described above, according to the corpus adder 1 according to the present embodiment, a corpus with higher performance can be obtained, and by performing natural language processing using the corpus as a result of the addition, better processing is achieved. Will be done.

  If the distribution of similarity coefficients of subsets in the corpus is as shown in FIGS. 9 and 10, the distribution of similarity coefficients may be as smooth as in FIG. Alternatively, the distribution may be used. In the former case, an optimal smooth curve may be determined using a least square method or the like.

  In addition, the shape of the subset similarity coefficient distribution in the corpus is generally a Gaussian distribution as shown in FIGS. However, it goes without saying that the distribution of the similarity coefficient of the subset of the corpus used in the present embodiment may not be a Gaussian distribution.

  Further, in the above specific example, the case where BTEC is used as the corpus and the case where SLDB, Calgary, Nikkei Shimbun, etc. are used as the reference corpus has been described, but it goes without saying that the corpus is not limited thereto.

  In the present embodiment, the case where the similarity coefficient calculation unit 15 calculates the similarity coefficient of the subset in the first corpus and the similarity coefficient of the subset in the second corpus has been described. When the similarity coefficient is stored in advance in the coefficient storage unit 16 and the second similarity coefficient storage unit 17, the corpus adding device 1 may not include the similarity coefficient calculation unit 15. In this case, the corpus adding device 1 may not include the first reference corpus storage unit 13 and the second reference corpus storage unit 14. In this case, the process of storing the similarity coefficient in the first similarity coefficient storage unit 16 and the second similarity coefficient storage unit 17 does not matter. For example, the similarity coefficient may be stored in the first similarity coefficient storage unit 16 or the like via the recording medium, and the similarity coefficient transmitted via the communication line or the like is stored in the first similarity coefficient storage. It may be stored in the unit 16 or the like, or the similarity coefficient input via the input device may be stored in the first similarity coefficient storage unit 16 or the like.

  In the present embodiment, the case where the adder 19 actually adds the first corpus and at least a part of the second corpus has been described. However, the addition by the adder 19 is logical. There may be. That is, the adding unit 19 may only logically determine that the first corpus and at least a part of the second corpus are used as one corpus. In this case, the output unit 20 may output information that can specify the corpus after the addition. For example, in the first corpus stored in the first corpus storage unit 11 and the second corpus stored in the second corpus storage unit 12, the output unit 20 applies the subset included in the added corpus. Then, a process for setting a predetermined flag may be performed.

  In the present embodiment, the case where the average value is used as the representative value of the similarity coefficient distribution has been described. However, as described above, the median value may be used as the representative value of the similarity coefficient distribution. Needless to say.

  In each of the above embodiments, each processing or each function may be realized by centralized processing by a single device or a single system, or distributed processing by a plurality of devices or a plurality of systems. May be realized.

  In each of the above embodiments, each component may be configured by dedicated hardware, or a component that can be realized by software may be realized by executing a program. For example, each component can be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. The software that realizes the information processing apparatus in each of the above embodiments is a program as described below. That is, this program is stored in the computer in the first similarity coefficient storage unit, the subset similarity coefficient in the first corpus, and the second similarity coefficient storage unit. Using a subset similarity coefficient in the corpus, a second set of subsets of the corpus that has the same representative value as the representative value of the subset similarity coefficient in the first corpus An extraction step for extracting from the corpus, an addition step for adding the set of subsets of the second corpus extracted in the extraction step, and the first corpus, and an output for outputting the corpus added in the addition step Step.

  Further, the program stores, in the computer, the first reference corpus stored in the first reference corpus storage unit and the second reference corpus stored in the second reference corpus storage unit as a reference corpus. A first similarity coefficient calculation step of calculating a similarity coefficient of each subset in the first corpus and the second corpus is further executed, and the first similarity coefficient storage unit stores the first corpus The similarity coefficient of the subset in the corpus and the similarity coefficient of the subset in the second corpus stored in the second similarity coefficient storage unit may be calculated in the similarity coefficient calculation step. Good.

  In the above program, the output step for outputting information does not include at least processing performed only by hardware, for example, processing performed by a modem or an interface card in the output step.

  Further, this program may be executed by being downloaded from a server or the like, and a program recorded on a predetermined recording medium (for example, an optical disk such as a CD-ROM, a magnetic disk, a semiconductor memory, or the like) is read out. May be executed by

  Further, the computer that executes this program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

  FIG. 15 is a schematic diagram showing an example of the external appearance of a computer that executes the program and realizes the corpus adder 1 according to the embodiment. The above-described embodiment can be realized by computer hardware and a computer program executed on the computer hardware.

  In FIG. 15, a computer system 100 includes a computer 101 including a CD-ROM (Compact Disk Read Only Memory) drive 105, an FD (Flexible Disk) drive 106, a keyboard 102, a mouse 103, and a monitor 104.

  FIG. 16 is a diagram illustrating a computer system. In FIG. 16, in addition to the CD-ROM drive 105 and the FD drive 106, a computer 101 includes a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112 for storing a program such as a bootup program, A CPU (Random Access Memory) 113 that is connected to the CPU 111 and temporarily stores application program instructions and provides a temporary storage space, a hard disk 114 that stores application programs, system programs, and data, a CPU 111 and a ROM 112. Etc. to each other. The computer 101 may include a network card (not shown) that provides connection to the LAN.

  A program that causes the computer system 100 to execute the functions of the corpus adder 1 according to the above-described embodiment is stored in the CD-ROM 121 or FD 122, inserted into the CD-ROM drive 105 or FD drive 106, and stored in the hard disk 114. May be forwarded. Instead, the program may be transmitted to the computer 101 via a network (not shown) and stored in the hard disk 114. The program is loaded into the RAM 113 at the time of execution. The program may be loaded directly from the CD-ROM 121, the FD 122, or the network.

The program does not necessarily include an operating system (OS) or a third-party program that causes the computer 101 to execute the functions of the corpus adder 1 according to the above-described embodiment. The program may include only a part of an instruction that calls an appropriate function (module) in a controlled manner and obtains a desired result. How the computer system 100 operates is well known and will not be described in detail.
Further, the present invention is not limited to the above-described embodiment, and various modifications are possible, and it goes without saying that these are also included in the scope of the present invention.

  As described above, the corpus adder according to the present invention can create a corpus capable of executing better processing, and is useful as an apparatus for processing a corpus.

The block diagram which shows the structure of the corpus adding apparatus by Embodiment 1 of this invention. The flowchart which shows operation | movement of the corpus adder by the same embodiment The figure which shows an example of a response | compatibility with the similarity coefficient and subset identifier in the embodiment The figure which shows an example of distribution of the similarity coefficient of corpus in the embodiment The figure which shows an example of distribution of the similarity coefficient of corpus in the embodiment The figure which shows an example of distribution of the similarity coefficient of corpus in the embodiment The figure which shows an example of distribution of the similarity coefficient of corpus in the embodiment The figure which shows an example of distribution of the similarity coefficient of corpus in the embodiment The figure which shows an example of distribution of the similarity coefficient of corpus in the embodiment The figure which shows an example of distribution of the similarity coefficient of corpus in the embodiment The figure which shows an example of the perplexity in the same embodiment The figure which shows an example of the BLEU score in the embodiment The figure which shows an example of the NIST score in the embodiment The figure which shows an example of the mWER score in the same embodiment Schematic diagram showing an example of the appearance of the computer system in the embodiment The figure which shows an example of a structure of the computer system in the embodiment

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Corpus addition apparatus 11 1st corpus storage part 12 2nd corpus storage part 13 1st reference corpus storage part 14 2nd reference corpus storage part 15 Similarity coefficient calculation part 16 1st similarity coefficient memory | storage part 17 2nd similarity coefficient Storage unit 18 Subset extraction unit 19 Addition unit 20 Output unit 21 Recording medium

Claims (5)

A first corpus storage unit storing a first corpus;
A second corpus storage unit for storing a second corpus;
A first similarity coefficient storage unit that stores similarity coefficients of subsets in the first corpus;
A second similarity coefficient storage unit that stores similarity coefficients of subsets in the second corpus;
A subset extractor for extracting from the second corpus a subset set of the second corpus having a representative value that is the same as a representative value of a similarity coefficient of the subset in the first corpus;
An adder that adds a set of subsets of the second corpus extracted by the subset extractor and the first corpus;
An output unit that outputs the corpus added by the addition unit. A first reference corpus storage unit in which a first reference corpus is stored;
A second reference corpus storage unit for storing a second reference corpus;
The first corpus storage unit stores the first reference corpus stored in the first reference corpus storage unit and the second reference corpus stored in the second reference corpus storage unit as a reference corpus. A similarity coefficient calculation unit that calculates a similarity coefficient of each subset in the first corpus and the second corpus stored in the second corpus storage unit,
The subset similarity coefficient in the first corpus stored in the first similarity coefficient storage unit, and the subset similarity coefficient in the second corpus stored in the second similarity coefficient storage unit The corpus adder according to claim 1, which is calculated by the similarity coefficient calculation unit. The subset extractor is configured so that the shape of the similarity coefficient distribution in the corpus as a result of addition to the first corpus is the same as the shape of the similarity coefficient distribution in the first corpus. The corpus addition apparatus according to claim 1, wherein a subset to be added to the first corpus is extracted from the corpus of the corpus. A first corpus storage unit storing a first corpus, a second corpus storage unit storing a second corpus, and a first similarity coefficient storing a subset similarity coefficient in the first corpus In a corpus addition method processed using a storage unit, a second similarity coefficient storage unit that stores subset similarity coefficients in the second corpus, a subset extraction unit, an addition unit, and an output unit There,
The subset extractor extracts a set of subsets of the second corpus, the subset having a representative value that is the same as the representative value of the similarity coefficient of the subset in the first corpus from the second corpus. An extraction step to
An adding step in which the adding unit adds the set of subsets of the second corpus extracted in the extracting step and the first corpus;
A corpus addition method, comprising: an output step in which the output unit outputs the corpus added in the addition step. On the computer,
The subset similarity coefficient stored in the first corpus and stored in the first corpus, and the subset similarity coefficient stored in the second corpus and stored in the second corpus. Using an extraction step of extracting from the second corpus a set of subsets of the second corpus that has the same representative value as the representative value of the similarity coefficient of the subset in the first corpus;
An addition step of adding a set of subsets of the second corpus extracted in the extraction step and the first corpus;
An output step of outputting the corpus added in the adding step.

Images