JP2016161968A - Word vector learning device, natural language processing device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow efficient learning of word vectors.SOLUTION: A document data update part 26 updates a word list and co-occurrence information of contexts of words, and adds word vectors and texture vectors for added words. A number-of-dimensions increasing part 28 updates the number of dimensions. A repetition optimization part 30 repeats the successive steps of fixing rfor all the words i, optimizing a variable cfor all the words j, fixing cfor all the words j, and optimizing a variable rfor the words i, so that a solution to an optimizing problem can be obtained in the end.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a word vector learning device, a natural language processing device, a method, and a program, and more particularly, to a word vector learning device, a natural language processing device, a method, and a program for learning a word vector related to a word.

  Since each word is a discrete symbol and not based on a physical phenomenon or the like, it is not so simple to quantitatively express the similarity between words. For comparison, for example, voice is generally regarded as time-series data of frequency on a computer. Therefore, the similarity between arbitrary speech sections can be measured to some extent by calculating the distance between the vectorization of various feature quantities (continuous values) that can be calculated from the frequency. Similarly, similarity between images can be easily calculated to some extent by calculating a distance between pixel information vectorized as feature amounts. In this way, the similarity between waveforms, colors, and those based on physical phenomena can be handled relatively naturally on a computer, but discrete symbols such as languages can be used. The similarity between things that do not conform to the physical phenomenon described in (1) cannot simply be handled on a computer.

  From this background, various methodologies have been devised so far to calculate the similarity between discrete symbols such as words. One of them is a distributed semantic expression method. This is a method of assigning one vector to each word and trying to express the semantic similarity between words by the distance between the vectors, as in the case of speech and images. It can be said that this method has a very high affinity for computers because it expresses the semantic proximity between words by calculating the distance in the vector space.

  FIG. 9 shows an outline of the similarity between words by the distributed semantic expression.

Here, the i-th word is represented as w _i . Further, r ′ _i represents a vector assigned to the i-th word w _i . A symbol representing a vector is represented by adding “′” after the symbol.
Hereinafter, a vector assigned to a word is specifically referred to as a “word vector”. That is, the word vector of the word w _i is r ′ _i . At this time, as a calculation on the computer, as shown in the following formula (1), the similarity between the two words w _i and w _j is the inner product between the word vectors of w _i and w _j , or Generally, it is defined by the cosine distance.

In this case, the larger the inner product or cosine distance value, the more similar the words w _i and w _j are. This has the advantage that semantically similar words can be handled during processing in various language processing applications such as translation, dialogue, document summarization, or document proofing. As a result, it is shown that better results can be obtained than a processing method that does not use semantic proximity between words. Here, many methods have been proposed for acquiring word vectors for each word. As a basic methodology, first, for each word in a sentence, the context information of that word is defined. There are no specific rules for the context information and various information can be used, but most simply, the words appearing around each word are mostly handled as context information. Changing the context definition does not significantly affect the word vector estimation algorithm itself. Therefore, in the following discussion, the word context information is a word that appears in the vicinity.

  In recent years, methods that can be processed at such a high speed as to be able to deal with large-scale data obtained from web have become mainstream (see Non-Patent Document 1 and Non-Patent Document 2). The reason why the method that can handle large-scale data is the main reason is that the more data there is, the more accurately the language event can be captured, so it can be theoretically expected that the estimation accuracy of similarity will be improved. is there. Here, as a representative of the conventional method, a method for acquiring a word vector according to Non-Patent Document 2 will be described.

In the method of Non-Patent Document 2, in order to express a case where a word appears as the context of another word, another vector different from the word vector is assigned to each word. For convenience, this is called a “context vector” in contrast to the word vector. That is, the i-th word w _i has two vectors, a word vector r ′ _i and a context vector c ′ _j .

Assume that there are N words. At this time, (r ′ _i ) _{i = 1} ^N is a list of vectors in which all word vectors are arranged in order from i = 1 to N. Similarly, let (c ′ _j ) _{j = 1} ^{N be} a list of vectors arranged in order from 1 to N for j of all context vectors. Further, the number of times that the j-th word becomes a context with respect to the i-th word is X _{i, j} . At this time, the acquisition of the distributed semantic expression according to Non-Patent Document 2 can be formulated by the following equation (2) as a problem of minimizing the objective function of the following (3).

However, ^ _r'i and ^ _c'j is an estimation result of the word vectors and the context vector.

The finally obtained ^ r ′ _i is the word vector of the i-th word. This is the word vector used in the similarity calculation of equation (1). It is also used in application applications for natural language processing such as translation or document proofreading.

Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean, Efficient Estimation of Word Representations in Vector Space.Proceedings of Workshop at ICLR, 2013. Jeffrey Pennington, Richard Socher, and Christopher Manning, Glove: Global Vectors for Word Representation.Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.

  The problem of the method used in the current mainstream as in Non-Patent Document 2 described above is that the “number of words” to be handled before learning and the “number of dimensions” of the vector space to be output need not be explicitly determined. It is a point that should not be done. In order to clarify the discussion, the number of words is hereinafter referred to as N and the number of dimensions of the output vector space as D. That is, it means that the learning is started by determining the number of words N and the number of dimensions D before starting the learning of the distributed semantic expression.

  In general, considering the situation of constructing a distributed semantic expression, the data that can be used for learning is almost always increasing and updating every day. Ideally, learning is performed sequentially according to the update of the learning data. A situation occurs where you want to go. In addition, since the appropriate number of dimensions of the word vector is considered to vary depending on the amount of learning data to be handled, for example, it is desirable that the framework be able to handle the process of increasing the number of dimensions of the vector when the amount of learning data increases without uncomfortable feeling. Furthermore, even without these reasons, learning of distributed semantic representation is basically a non-convex optimization problem and there are multiple local optimal solutions, so even if the same data is used and the same objective function is used, learning is possible. Depending on the initial value and procedure of the time, the resulting word vectors may vary greatly. For this reason, if there is no relationship between the word vectors before and after relearning in a situation where the learning data has increased slightly, the system parameters of the system that uses it will be reestimated each time relearning the distributed semantic representation. There is a demerit that it must be done.

  The present invention has been made to solve the above problems, and an object of the present invention is to provide a word vector learning device, method, and program capable of efficiently learning a word vector.

  It is another object of the present invention to provide a natural language processing apparatus and program for performing natural language processing based on the semantic similarity of words using learned word vectors.

  In order to achieve the above object, the word vector learning device according to the first invention is based on document data, for each word, the word vector related to the word, and the word appears as the context of another word. A word vector learning device that learns a context vector representing a word vector and a context vector that is one of the dimensions of the word vector and the context vector in order, each different from the target dimension of the word vector and the context vector for each of the words A word vector and a context vector for each of the words so as to optimize the objective function for the target dimension based on the value of the dimension and the number of times one word has appeared as the context of the other word for the word pair It includes an iterative optimization unit that repeatedly estimates the value of the target dimension.

  In the word vector learning device according to the first invention, the iterative optimization unit fixes a value of a target dimension of the word vector for each of the words, and determines a word vector and a context vector for each of the words. Based on the value of each dimension different from the target dimension and the number of times one word has appeared as the context of the other word for the word pair, the objective function for the target dimension is optimized for each of the words A context vector optimization unit for estimating a target dimension value of a context vector, and a target dimension value of the context vector for each of the words is fixed, and is different from the target dimension of the word vector and the context vector for each of the words Based on the value of each dimension and the number of times one word appears as the context of the other word for a word pair, By the word vector optimization unit that estimates the value of the target dimension of the word vector for each of the words so as to optimize the objective function, and by the context vector optimization unit until a predetermined iteration termination condition is satisfied An iterative determination unit that alternately repeats estimation and estimation by the word vector optimization unit, and any one of the word vector and the context vector is set as a target dimension in order, the context vector optimization unit, The processing by the word vector optimization unit and the iterative determination unit may be repeated.

  Further, in the word vector learning device according to the first invention, when the iterative optimization unit learns the word vector and context vector by increasing the number of dimensions with respect to the learned word vector and context vector, Any dimension that is not learned of the word vector and the context vector is set as the target dimension in order, and the value of each dimension different from the target dimension of the word vector and the context vector for each of the words, and one word for the word pair Based on the number of appearances as the context of the other word, the estimation of the value of the target dimension of the word vector and the context vector for each of the words is repeated so as to optimize the objective function related to the target dimension. It may be.

In the word vector learning device according to the first invention, the iterative optimization unit optimizes the objective function related to the target dimension d represented by the following equation (5) according to the following equation (4): to, may be repeated to estimate a value of the target dimension d of the context vectors _c'j for each word vectors _r'i and word j for each word i.

_However, r _{i, d} is the elements of the dimension d of the word vectors _r'i words _i, c _{j, d} represents the elements of the dimension d of the context vectors _c'j words _{j, ^} r _{i, d} is _{r i} , _D , ^ c _j , _d represents the estimation result of c _j , _d , and b _{i, j, d} = −r ′ _i ^(d) · c ′ _j ^(d) + log (X _{i, j} ) , R ′ _i ^(d) represents a word vector obtained by replacing the element of the dimension d of the word vector r ′ _i of the word _i with 0, and c ′ _j ^(d) represents the dimension of the word vector c ′ _j of the word _j . A context vector in which the element of d is replaced with 0 is represented, and X _{i, j} represents the number of times that the word j appears as a context with respect to the word i.

  The natural language processing device according to the second invention uses the word vector of each word learned by the word vector learning device for an input document, and the meaning between words based on the word vector A natural language processing unit that performs natural language processing based on a similar degree of similarity.

  A word vector learning method according to a third invention is based on document data, and for each word, learns a word vector related to the word and a context vector indicating that the word appears as a context of another word. A word vector learning method in a vector learning device, wherein the iterative optimization unit sequentially sets any one dimension of the word vector and the context vector as a target dimension, and the target dimension of the word vector and the context vector for each of the words A word vector for each of the words so as to optimize the objective function for the target dimension based on the value of each dimension different from the number of times that one word has appeared as the context of the other word for the word pair And repeating the step of estimating the value of the target dimension of the context vector. .

  In the word vector learning method according to the third invention, the step of repeating the estimation by the iterative optimization unit is such that the context vector optimization unit fixes the value of the target dimension of the word vector for each of the words, Based on the value of each dimension different from the target dimension of the word vector and context vector for each of the words, and the number of times one word has appeared as the context of the other word for the word pair, the objective function for the target dimension is A step of estimating a value of a target dimension of a context vector for each of the words so as to optimize, and a word vector optimization unit fixes a value of a target dimension of the context vector for each of the words, The value of each dimension different from the target dimension of the word vector and context vector for each word, and one word for the other Estimating the value of the target dimension of the word vector for each of the words so as to optimize the objective function related to the target dimension based on the number of appearances as the context of the word; Alternately performing the estimation by the context vector optimization unit and the estimation by the word vector optimization unit until a predetermined iteration end condition is satisfied, and including any dimension of the word vector and the context vector Are sequentially set as the target dimension, and the process is repeated by the step of estimation by the context vector optimization unit, the step of estimation by the word vector optimization unit, and the iterative determination unit.

  A program according to a fourth aspect of the invention is a program for causing a computer to function as each unit constituting the word vector learning device or the natural language processing device.

  According to the word vector learning apparatus, method, and program of the present invention, any one dimension of the word vector and the context vector is set as the target dimension in order, and one word for each dimension value and word pair is different from the target dimension. By repeatedly estimating the value of the target dimension of the word vector and the context vector for each of the words so as to optimize the objective function for the target dimension based on the number of appearances as the context of the other word, There is an effect that each word vector can be learned efficiently.

  In addition, according to the natural language processing apparatus and the program of the present invention, it is possible to perform natural language processing based on the semantic similarity of words using a learned word vector.

It is a block diagram which shows the structure of the word vector learning apparatus which concerns on embodiment of this invention. It is a figure which shows the example of the word list and context co-occurrence information. It is a figure which shows the example of a one-dimensional word vector. It is a figure which shows the example of the two-dimensional word vector which increased the number of dimensions. It is a figure which shows the example of a 10-dimensional word vector. It is a block diagram which shows the structure of the natural language processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows the word vector learning process routine in the word vector learning apparatus which concerns on embodiment of this invention. It is a flowchart which shows the natural language processing routine in the natural language processing apparatus which concerns on embodiment of this invention. It is a figure which shows the outline | summary of the similarity between words by a distributed semantic expression.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Outline according to Embodiment of the Present Invention>

  First, an outline of the embodiment of the present invention will be described. In the embodiment of the present invention, in accordance with the situation that can occur when using the distributed semantic representation in the real environment, it is possible to add continuously and learn continuously from the viewpoint of updating the number of words N and the number of dimensions D. We propose a framework for obtaining a distributed semantic representation of words that can be performed and the results obtained behave as if the previous results were kept as much as possible. Also in the present embodiment, as in the conventional method, a word vector is acquired basically following the optimization problem shown in the above equation (2). However, the framework is changed so that learning can be continuously performed even in a situation where the number of words N and the number of dimensions D are sequentially increased.

First, all word vectors r ′ _i and context vectors c ′ _j are assumed to be D-dimensional vectors. However, D ≧ 1. At this time, with respect to an arbitrary target dimension d, only the elements of the dimension d of all the word vectors are set as optimization variables, and the values of the other elements (except the dimension d) are regarded as constants. However, 1 ≦ d ≦ D. Considering solving the optimization problem of the above equation (2) with this setting, it can be regarded as a problem of solving the optimization problem of one variable for each word. As a matter of course, since only the element of dimension d of each vector is determined as the optimization variable, the elements of the word vector that are changed after optimization are only elements of the selected dimension d. After the learning in the target dimension d is completed, the process is changed to the process of appropriately changing the selection of the target dimension d and repeating the same learning. Even if it changes in this way, it can be interpreted that the optimization algorithm is used to obtain the solution of the original optimization problem by simply repeating the optimization for each dimension. It can be interpreted as solving in a certain way.

  In the present embodiment, first, in a situation where learning is basically performed permanently, the learning environment is set. However, the actual processing occurs only when two patterns of events occur: (1) when document data is updated, or (2) when it is desired to increase the number of dimensions of a word vector. Learning requires a certain amount of time, but if the learning time is shorter than the occurrence of the two patterns of events, the process is in a state of waiting in a stopped state.

  Also, at the start of learning, the number of word vectors is set to N = 0, and the vector dimension is set to D = 1. As described above, in the present embodiment, it is always possible to start from the state where the number of words and the number of dimensions are minimum.

  The main processing flow is as shown in the following processing 1 to processing 4.

(Process 1): Standby When a document data update signal is received, the process proceeds to Process 2A.
When a signal for increasing the number of dimensions of the word vector is received, the process proceeds to process 2B.

(Process 2A): Update of document data The document data is read, and the co-occurrence information of the word and the context word is updated.
The value of the number of words N is updated according to the number of words that appear in the document data.
The target dimension d is set as d = 1.

(Process 2B): Addition of dimension to vector The word vector and context vector elements are incremented by one at the end and initialized.
The dimension number D is set to D = D + 1.
Set the initial value of d as d = D.

(Process 3)
The optimization problem related to the target dimension d is solved according to the equation (6) described later, and the value of the target dimension d of the word vector is updated.

(Process 4)
If d is d = D, return to process 1.
If d is other than d = D, d = d + 1 and the process returns to step 3.

  In the embodiment of the present invention, the above processing 1 to processing 4 are applied using the problem of obtaining a distributed semantic expression of a word using an electronic document described in a natural language existing on a web as a subject. A vector learning apparatus will be described as an example.

  <Configuration of Word Vector Learning Device According to Embodiment of the Present Invention>

  Next, the configuration of the word vector learning device according to the embodiment of the present invention will be described. As shown in FIG. 1, a word vector learning device 100 according to an embodiment of the present invention includes a CPU, a RAM, a ROM for storing a program and various data for executing a word vector learning processing routine described later, Can be configured with a computer including Functionally, the word vector learning device 100 includes an input unit 10, a calculation unit 20, and an output unit 50 as shown in FIG. The word vector learning device 100 assumes that the initial state is a state in which nothing is learned. Therefore, the number of words N is set to N = 0, and the number of dimensions D is set to D = 1.

  The input unit 10 receives a document data and a document data update signal or a signal for increasing the number of dimensions. As the document data, in this embodiment, document data generated by a general user who is posted to an SNS site on a certain day is used.

  The calculation unit 20 includes a document data update unit 26, a dimension number increase unit 28, an iterative optimization unit 30, and a vector storage unit 40.

  As described below, the document data update unit 26 updates the word list and the word context co-occurrence information when the document data update signal is received from the input unit 10 as the document data is updated. At the same time, a word vector and a context vector of the newly added word are prepared.

  The document data updating unit 26 first updates the word list and the word context co-occurrence information based on the received document data. N is set based on the number of words in the word list obtained here. In the present embodiment, the number of words used in the word list is all the words that appear in the accepted document data. For example, if the number of words is 10,000 words, N = 10,000. Or it is good also as a word which appeared more than a certain frequency.

  Next, the document data update unit 26 collects a pair of a word and a word extracted from the context of the word to obtain co-occurrence information. For example, two words before and after the target word are extracted as context information of each word. However, the number of words used before and after may be arbitrarily determined, and any number may be used. At that time, the number of appearances (co-occurrence frequency) is also recorded. This pair and frequency are acquired in advance as statistical information.

  FIG. 2 shows an example of word list and context co-occurrence information. For the sake of simplicity in the example shown here, it is assumed that word breaks and the like can be easily obtained using a commonly used tool or the like. In the case of Japanese, there are tools that can be used for free, and in the case of English, a blank separator can be used as a word separator. In this embodiment, an example of Japanese is described, but it may be applied to English. In this embodiment, in order to simplify and speed up the processing, word and context co-occurrence information is acquired from document data in advance.

  Next, the document data update unit 26 adds the word vector and context vector of the word to the word vector and context vector stored in the vector storage unit 40 for a word newly added by updating the document data. To do. In the initial state, since the dimension number D is D = 1, the document data update unit 26 prepares a one-dimensional word vector and context vector for each word and stores them in the vector storage unit 40.

  The dimension number increasing unit 28 executes processing when a signal indicating an increase in dimension number is received from the input unit 10. In order to cope with the increase in the number of dimensions, one dimension is added for each of all word vectors and context vectors, and the elements of the added dimension are initialized with arbitrary values. Next, the dimension number D is updated to D = D + 1.

  The iterative optimization unit 30 includes a context vector optimization unit 32, a word vector optimization unit 34, and an iterative determination unit 36.

  First, the principle of the iterative optimization unit 30 will be described.

  The iterative optimization unit 30 sets any one dimension of the word vector and the context vector as the target dimension in order, and the value of each dimension different from the target dimension of the word vector and the context vector for each word and one of the word pairs Based on the number of times a word appears as the context of the other word, the estimation of the value of the target dimension of the word vector and the context vector for each of the words is repeated so as to optimize the objective function for the target dimension.

Specifically, the iterative optimization unit 30 performs the following processing. First, as input, a word list and a context co-occurrence frequency are received from the document data update unit 26. Then, the initial value of the variable d representing the number of dimensions to be processed is set to 1. To simplify the description, for the i th word of the word vectors _r'i, the elements of the dimension d r _i, as _d. Next, let r ′ _i ^(d) be a vector in which the element of dimension d of word vector r ′ _i is fixed to 0. Similarly, let c ′ _j ^(d) be a vector in which the element of the dimension d of the context vector c ′ _j is fixed to 0. At this time, variables used for optimization are only r _{i, d} for all words i and c _{j, d} for all words j, and all r ′ _i ^(d) and c ′ _j ^(d) are constants. To do. To further simplify the description, terms that become constants during optimization are collectively represented as b _{i, j, d} = −r ′ _i ^(d) · c ′ _j ^(d) + log (X _{i, j} ).

  At this time, the optimization problem related to the target dimension d shown in the following equation (6) is solved.

  Compared with the above equation (6), the above equation (6) is simply a state in which only the elements of the dimension d of all the word vectors and context vectors are optimization variables, and the elements of the other dimensions are all fixed. This means that optimization is performed. Therefore, if D = 1, it means that the expressions (2) and (6) are completely equivalent.

When optimizing the above equation (6), r _{i, d} is fixed for all words i, and variables c _{j, d} are optimized for all words j. Conversely, the process of fixing c _{j, d} for all the words j and optimizing the variables r _{i, d} for all the words i is repeated alternately to finally optimize the expression (6). Get the solution to the problem. In this way, each problem can be obtained accurately because an analytical solution is obtained for each variable. Hereinafter, processing of each unit of the context vector optimization unit 32, the word vector optimization unit 34, and the iteration determination unit 36 in the iterative optimization unit 30 will be described.

Context vectors optimization unit 32, the value of the target dimension d of the word vectors _r'i for all words i, r _i, is fixed to _d stored in the vector storage unit 40, the words for each word i context vector _c'j for vector _r'i and each word j, and the value of each dimension different from the target dimension, the number one word i for the word pair has emerged as the context of the other words j X _i, and _j based on, to optimize the objective function shown in equation (7) relating to the target dimension d, the estimated value of the target dimension d of the context vectors _c'j for each word j, the vector storage section 40 Remember. Here, the context vector optimization unit 32, the value r _i of the target dimension d of the word vectors _r'i for all word _i, to fix the _d, target dimension d of the context vectors _c'j for each word j when optimizing the element c _j, and _d, as follows (8), each element c _j for all word _j, may be determined a value partial differential becomes zero _d.

Word vector optimization section 34, c _j where the value of the target dimension d is stored in the vector storage unit 40 of the context vectors _c'j for all word _j, fixed to _d, each word i, for j Based on the value of each dimension different from the target dimension of the word vector r ′ _i and the context vector c ′ _j and the number of times X _{i, j} that one word i appears as the context of the other word j for the word pair so as to optimize the objective function shown in equation (7) relates to the dimension d, and estimate the value of the target dimension d of the word vectors _r'i for each word i, and stores in the vector storage section 40. Here, the word vector optimization unit 34, the value c _j of the target dimension d of the word vector _c'j for all word _j, to fix the _d, target dimension d of the word vectors _r'i for each word i when optimizing the elements r _i, and _d, as follows (9), each element r _i for all words _i, may be determined a value partial differential becomes zero _d.

  The iterative determination unit 36 repeatedly performs the estimation by the context vector optimization unit 32 and the estimation by the word vector optimization unit 34 alternately until a predetermined iteration end condition is satisfied. As an iterative end condition, for example, when the above formulas (8) and (9) are calculated alternately, the value of the objective function in the above formula (7) is always the same or smaller, so the value of the objective function changes. Repeat the process until there is no change or almost no change. If the target dimension d is d = D, the process waits until a next learning data update signal or a signal indicating an increase in the number of dimensions arrives. On the other hand, if the target dimension d is d ≠ D, the target dimension is changed as d = d + 1, and the processing of the context vector optimization unit 32 and the word vector optimization unit 34 is repeated in the next target dimension. For example, if it is assumed that there is a request to expand the word vector to about 10 dimensions for use in reputation analysis or the like, a signal for increasing the number of dimensions is accepted. At this time, since the word vector learning device according to the present embodiment always increases only one dimension at a time, a series of processes of the dimension number increasing unit 28, the context vector optimizing unit 32, and the word vector optimizing unit 34 are performed 10 times. repeat.

  FIG. 3 shows an example of a one-dimensional word vector obtained by estimation by the iterative optimization unit 30, and FIGS. 4 and 5 show words obtained by repeating the estimation by the iterative optimization unit 30 by increasing the number of dimensions. An example of a vector is shown.

  <Configuration of Natural Language Processing Device According to Embodiment of the Present Invention>

  Next, the configuration of the natural language processing apparatus according to the embodiment of the present invention will be described. As shown in FIG. 6, a natural language processing apparatus 200 according to an embodiment of the present invention includes a CPU, a RAM, and a ROM that stores a program and various data for executing a natural language processing routine described later. It can be configured with a computer including. Functionally, the natural language processing apparatus 200 includes an input unit 210, a calculation unit 220, and an output unit 250 as shown in FIG. In the present embodiment, the natural language processing device 200 will be described by taking as an example a case where an unknown word is replaced with a word having a high similarity based on the word vector learned by the word vector learning device 100. However, the present invention is not limited to this, and summarization, document proofreading, and the like may be performed using the replaced word.

  The input unit 210 receives text to be translated.

  The calculation unit 220 includes a natural language processing unit 230 and a vector storage unit 240.

  The vector storage unit 240 stores the same as the vector storage unit 40.

  The natural language processing unit 230 includes a replacement unit 232 and a translation unit 234.

  The replacement unit 232 extracts an unknown word that is not in an existing dictionary (not shown) that stores the word from the words of the text received by the input unit 210, and a context vector for the word stored in the vector storage unit 40 Based on the above, the word in the dictionary having the highest similarity to the unknown word is estimated. And the text which replaced the unknown word with the word in the estimated dictionary is produced | generated.

  The translation unit 234 translates the text in which the word is replaced by the replacement unit 232 using an existing method, outputs the translated text to the output unit 250, and ends the processing.

  In the natural language processing apparatus 200, when performing other natural language processing, a word similar to a word appearing in a specific document is extracted from the dictionary and included in the processing target, thereby increasing information and accuracy. It is possible to improve. At this time, for each word that appears, the above formula (1) is calculated and a word having a high similarity is included in the process.

<Operation of the word vector learning device according to the embodiment of the present invention>

  Next, the operation of the word vector learning device 100 according to the embodiment of the present invention will be described. When the input unit 10 receives a document data and a document data update signal or a dimensionality increase signal, the word vector learning device 100 executes a word vector learning processing routine shown in FIG.

  First, in step S100, it is determined whether the signal received at the input unit 10 is a document data update signal or a dimensionality increase signal. If the signal is a document data update signal, the process proceeds to step S102. If the signal is a signal for increasing the number of dimensions, the process proceeds to step S108.

  In step S102, the word list and word context co-occurrence information are updated based on the document data received by the input unit 10.

  In step S104, it is determined whether a word is newly added based on the document data received by the input unit 10, and if it is added, the process proceeds to step S106, and if not, the process proceeds to step S110.

  In step S <b> 106, for the word added by the document data received by the input unit 10, the word vector and context vector are added to the word vector and context vector stored in the vector storage unit 40. In step S107, the target dimension d is set to 1, and the process proceeds to step S114.

  In step S108, one dimension is added and added for each of the word vectors and context vectors of all words stored in the vector storage unit 40 based on the signal of the increase in the number of dimensions received by the input unit 10. Initialize dimension elements with arbitrary values.

  In step S110, the number of dimensions D is updated to D + 1.

  In step S112, the target dimension d is set to D in order to set an unlearned dimension as a learning target.

In step S114, based on the word vectors and context vectors of all the words stored in the vector storage unit 40, the elements c _{j, d} of the target dimension d of the context vector of each word are optimized according to the above equation (8). And the context vector of each word stored in the vector storage unit 40 is updated.

In step S116, based on the word vectors and context vectors of all the words stored in the vector storage unit 40, the elements r _{i, d} of the target dimension d of the word vector of each word are optimized according to the above equation (9). And the word vector of each word stored in the vector storage unit 40 is updated.

  In step S118, the optimization of step S114 and step S116 determines whether the value of the objective function in equation (7) satisfies the iterative termination condition defined according to equation (6). The process proceeds to S120, and if not satisfied, the process returns to Step S114, and the optimization process in Steps S114 and S116 is repeated.

  In step S120, it is determined whether d = D. If d = D, the process proceeds to step S122, and if d = D, the process proceeds to step S118.

  In step S122, the target dimension is changed as d = d + 1, the process proceeds to step S114, and the optimization processing in steps S114 to S116 is repeated.

  In step S124, the word vector and the context vector obtained in steps S114 to S118 are output, the word vector learning processing routine is terminated, and the process waits until the next signal is received.

  As described above, according to the word vector learning device according to the embodiment of the present invention, any dimension of the word vector and the context vector is set as the target dimension in order, the value of each dimension different from the target dimension, and the word The word vector and context vector targets for each word so as to optimize the objective function of equation (7) above for the target dimension based on the number of times one word has appeared as the context of the other word for the pair By repeatedly estimating the dimension value, the word vector can be learned efficiently.

<Operation of Natural Language Processing Device According to Embodiment of the Present Invention>

  Next, the operation of the natural language processing apparatus 200 according to the embodiment of the present invention will be described. When the input unit 210 accepts a text to be translated, the natural language processing apparatus 200 executes a natural language processing routine shown in FIG.

  In step S200, an unknown word is extracted from the text to be translated received by the input unit 210.

  In step S202, based on the word vector of each word stored in the vector storage unit 240, the word in the dictionary having the highest similarity to the unknown word extracted in step S200 is estimated, and the translation target Is generated by replacing an unknown word with a word in the estimated dictionary.

  In step S204, translation is performed based on the text generated in step S202, and the text is output to the output unit 250 and the process is terminated.

  As described above, according to the natural language processing apparatus of the embodiment of the present invention, it is possible to perform translation processing based on the semantic similarity of words using learned word vectors.

  Further, according to the word vector learning device according to the embodiment of the present invention, optimization is performed in order from d = 1 to d = D in order to give meaning to each dimension of the vector. In this way, the first element of each vector expresses the strongest similarity by processing in order from the first dimension in this way. This is because it is possible to obtain a mechanism of learning a more detailed similarity that could not be grasped so far, using the similarity as a precondition.

  In addition, when the document data increases or is updated with respect to the learned word vector, the learning is continued with the learned word vector as an initial value, so that the difference information is obtained for each dimension. You can learn in the form of adding to the. Therefore, the change in the word vector before and after the relearning is a change in the value obtained from the difference information. This means that the value does not change if the data difference is not affected. Due to this property, the word vector before and after the re-learning is updated with the difference information of the document data and keeps the previous information as compared with the conventional case where the re-learning from the beginning when the document data increases. Can be obtained.

  Further, even if it is desired to increase the number of dimensions of the word vector, since the processing is decomposed into optimization for each dimension, the number of dimensions of the vector can be changed from D to D + 1 in a retrofit form that can be easily handled. Increased, d = D is set, and learning can be easily performed while increasing the dimensionality sequentially. Furthermore, when learning is started from a state in which there is no word vector, it is only necessary to set D = 1 and start processing from a one-dimensional vector, especially without determining the number of dimensions of the word vector in advance. It is possible to start.

  As described above, by using the present invention, unlike the conventional method, it is possible to learn distributed semantic expressions even in a situation where the number of words and the number of dimensions continuously increase. In addition, it is possible to maintain a dependency relationship with the difference from the previous word vector obtained. As a result, when the system is reconstructed using the distributed semantic representation, it is not necessary to re-estimate all the system parameters from the beginning, and there is an advantage that the cost of system maintenance can be greatly reduced.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

  For example, in the above-described embodiment, a case has been described in which the signal of increasing the number of dimensions is received, the number of dimensions D is updated to D + 1, and the word vector and the context vector are learned using the increased dimension as the target dimension. However, the present invention is not limited to this, and a signal for increasing the number of dimensions to an arbitrary two or more is accepted, and word vectors and context vectors are learned using the increased dimensions as target dimensions in order. May be.

10, 210 Input unit 20, 220 Calculation unit 26 Document data update unit 28 Number of dimension increase unit 30 Iterative optimization unit 32 Context vector optimization unit 34 Word vector optimization unit 36 Iteration determination unit 40 Vector storage unit 50, 250 Output unit 100 word vector learning device 200 natural language processing device 230 natural language processing unit 232 replacement unit 234 translation unit 240 vector storage unit

Claims (8)

A word vector learning device that learns, for each word, a word vector related to the word and a context vector representing that the word appears as a context of another word based on document data,
Any dimension of the word vector and the context vector is set as the target dimension in order, the value of each dimension different from the target dimension of the word vector and the context vector for each of the words, and one word for the word pair is the other word An iterative optimization unit that repeatedly estimates a word vector for each of the words and a value of the target dimension of the context vector so as to optimize an objective function related to the target dimension based on the number of appearances of Word vector learning device including The iterative optimization unit includes:
The value of the target dimension of the word vector for each of the words is fixed, the value of each dimension different from the target dimension of the word vector and the context vector for each of the words, and one word for the word pair of the other word A context vector optimization unit for estimating a value of a target dimension of a context vector for each of the words so as to optimize an objective function related to the target dimension based on the number of appearances as a context;
The value of the target dimension of the context vector for each of the words is fixed, the value of each dimension different from the target dimension of the word vector and the context vector for each of the words, and one word for the word pair of the other word A word vector optimization unit that estimates a value of a target dimension of a word vector for each of the words so as to optimize an objective function related to the target dimension based on the number of times it appears as context;
An iterative determination unit that alternately repeats the estimation by the context vector optimization unit and the estimation by the word vector optimization unit until a predetermined iteration end condition is satisfied,
The word vector learning device according to claim 1, wherein any one of the word vector and the context vector is set as a target dimension in order, and the processing by the context vector optimization unit, the word vector optimization unit, and the iterative determination unit is repeated. .   When the iterative optimization unit learns the word vector and context vector by increasing the number of dimensions with respect to the learned word vector and context vector, any dimension of the word vector and context vector that has not been learned , In turn, based on the value of each dimension different from the target dimension of the word vector and context vector for each of the words, and the number of times one word has appeared as the context of the other word for the word pair, The word vector learning device according to claim 1, wherein the estimation of the value of the target dimension of the word vector and the context vector for each of the words is repeated so as to optimize the objective function related to the target dimension. The iterative optimization unit optimizes the objective function related to the target dimension d represented by the following equation (2) according to the following equation (1), and the word vector r ′ i for each word _i and word j word vector learning device context vector _c'j repeatedly to estimate the value of the target dimension d of claim 1, wherein for each.

_However, r _{i, d} is the elements of the dimension d of the word vectors _r'i words _i, c _{j, d} represents the elements of the dimension d of the context vectors _c'j words _{j, ^} r _{i, d} is _{r i} , _D , ^ c _j , _d represents the estimation result of c _j , _d , and b _{i, j, d} = −r ′ _i ^(d) · c ′ _j ^(d) + log (X _{i, j} ) , R ′ _i ^(d) represents a word vector obtained by replacing the element of the dimension d of the word vector r ′ _i of the word _i with 0, and c ′ _j ^(d) represents the dimension of the word vector c ′ _j of the word _j . A context vector in which the element of d is replaced with 0 is represented, and X _{i, j} represents the number of times that the word j appears as a context with respect to the word i. Meaning between words based on the word vector using the word vector of each word learned by the word vector learning device according to any one of claims 1 to 4 with respect to the inputted input document A natural language processing apparatus including a natural language processing unit that performs natural language processing based on a similar degree of similarity. A word vector learning method in a word vector learning device that learns, for each word, a word vector related to the word and a context vector representing that the word appears as a context of another word based on document data,
The iterative optimization unit sequentially sets any dimension of the word vector and the context vector as a target dimension, and each of the word vector and context vector has a dimension value different from the target dimension and one of the word pairs. Estimating the value of the word vector and the target dimension of the context vector for each of the words so as to optimize the objective function for the target dimension based on the number of times the word of the word appears as the context of the other word The word vector learning method including the step of repeating. The step of repeating the estimation by the iterative optimization unit includes:
The context vector optimization unit fixes the value of the target dimension of the word vector for each of the words, and the word vector for each of the words, the value of each dimension different from the target dimension of the context vector, and one of the word pairs Estimating the value of the target dimension of the context vector for each of the words so as to optimize the objective function for the target dimension based on the number of times the word of
The word vector optimization unit fixes the value of the target dimension of the context vector for each of the words, the word vector for each of the words, the value of each dimension different from the target dimension of the context vector, and one of the word pairs Estimating the value of the target dimension of the word vector for each of the words so as to optimize the objective function for the target dimension based on the number of times that the word has appeared as the context of the other word;
The iterative determination unit alternately repeats the estimation by the context vector optimization unit and the estimation by the word vector optimization unit until a predetermined iteration end condition is satisfied, and
A process in which any one of the word vector and the context vector is sequentially set as a target dimension, the context vector optimization unit estimates, the word vector optimization unit estimates, and the iteration determination unit repeats the process. Item 7. The word vector learning method according to Item 6.   The program for functioning a computer as each part which comprises the word vector learning apparatus of any one of Claims 1-4, or the natural language processing apparatus of Claim 5.