KR102260646B1 - Natural language processing system and method for word representations in natural language processing - Google Patents

Abstract

The present invention relates to a natural language processing system and a word expression method in natural language processing, in a word expression method in natural language processing performed by a natural language processing system, a) a vocabulary including at least one word and each word providing a vocabulary dictionary dataset including previously learned word embedding information; b) When a vocabulary based on the vocabulary dictionary dataset is provided as input data, subword information about words existing in the input data is extracted using a word expression model, and the subword information is embedded in words calculating a vector; and c) matching the calculated word embedding vector with the pre-learned word embedding information of the corresponding word, replacing the pre-learned word embedding information with the calculated word embedding vector to learn a word expression for the word. wherein the word expression model includes a convolutional neural network-based convolutional neural network that calculates lower-order word feature vectors using the lower-order word information, and the lower-order word feature vectors calculated by the convolution module. It includes a highway network-based highway module that adaptively combines to calculate a word embedding vector of a corresponding word.

Description

NATURAL LANGUAGE PROCESSING SYSTEM AND METHOD FOR WORD REPRESENTATIONS IN NATURAL LANGUAGE PROCESSING

The present invention relates to a natural language processing system for generating word expressions for all words including Out Of Vocabulary (OOV) based on supervised learning of previously learned word embeddings, and to a word expression method in natural language processing.

As deep learning technology brings great development of computer vision systems, research on natural language processing systems using deep learning is also rapidly progressing. A natural language processing system using deep learning technology should embed words as low-dimensional vectors to quantify them.

At this time, natural language processing is a study of syntactic/semantic representations that a computer can understand human or natural language, and word embedding technology is a technology derived from a neural network-based language model. It is a technology that can express lexical meaning by arranging words closely on a vector space.

The language model is a model that calculates the probability that the next word will appear based on the preceding words in a given sentence. Because language models calculate the probability that a sentence actually exists, it can determine how well a given sentence is grammatically or semantically appropriate.

The language model can be applied to fields such as speech recognition, QA (Question Answering), natural language processing, predictive text, translation and interpretation. Out Of Vocabulary (OOV), that is, processing of unregistered words is required. In order to process OOV, a method of generating a temporary expression for an unregistered word was used by using an unregistered word as a pseudo word or using the context in which the word was used.

The existing method of processing unregistered words can be a good solution when there are a small amount of unregistered words, but when the number of unregistered words increases, there is a problem in that the natural language processing system cannot properly analyze the text.

These problems are more evident in the social media environment. Words consumed by users in social media are used differently from common words such as abbreviations, compound words, neologisms, and typos, and there is a very high probability that these words do not exist in the word embeddings of the natural language processing system.

Therefore, in an open vocabulary environment where the number of words to be processed by the natural language processing system is large or neologisms occur frequently, an expression for each unregistered word is generated, and the unique meaning of the unregistered word is inferred. It is required to develop a language model that can do this.

Republic of Korea Patent No. 10-1799681 (Title of the invention: apparatus and method for homogeneous word discrimination using lexical semantic network and word embedding)

The present invention according to an embodiment of the present invention in order to solve the above problems

Although the meaning of the existing unregistered words was grasped through the surrounding words without considering the unique information at all, in order to infer the unique meanings of all words as well as the unregistered words, the subword information of the corresponding word was used to infer the meaning. The purpose is to generate a unique word expression based on supervised learning of learned word embeddings.

However, the technical task to be achieved by the present embodiment is not limited to the technical task as described above, and other technical tasks may exist.

As a technical means for achieving the above technical problem, a word expression method in natural language processing according to an embodiment of the present invention is a word expression method in natural language processing performed by a natural language processing system, a) at least one or more providing a vocabulary dictionary dataset including vocabulary including words and pre-learned word embedding information for each word; b) When a vocabulary based on the vocabulary dictionary dataset is provided as input data, subword information about words existing in the input data is extracted using a word expression model, and the subword information is embedded in words calculating a vector; and c) matching the calculated word embedding vector with the pre-learned word embedding information of the corresponding word, replacing the pre-learned word embedding information with the calculated word embedding vector to learn a word expression for the word. wherein the word expression model includes a convolutional neural network-based convolutional neural network that calculates lower-order word feature vectors using the lower-order word information, and the lower-order word feature vectors calculated by the convolution module. It includes a highway network-based highway module that adaptively combines to calculate a word embedding vector of a corresponding word.

A natural language processing system according to another embodiment of the present invention is a natural language processing system for distributed expression of words included in a vocabulary, comprising: a memory in which a program for performing a word expression method in natural language processing is recorded; and a processor for executing the program, wherein, by the execution of the program, a vocabulary including at least one or more words and a vocabulary dictionary dataset including pre-learned word embedding information for each word. A vocabulary based on a vocabulary is provided as input data, subword information about words existing in the input data is extracted using a word expression model, a word embedding vector is calculated from the subword information, and the calculated By matching the word embedding vector with the previously learned word embedding information of the corresponding word, the previously learned word embedding information is replaced with the calculated word embedding vector to learn a word expression for the word, wherein the word expression model includes: A convolutional neural network-based convolution module that calculates lower-order word feature vectors using lower-order word information, and adaptively combining the lower-order word feature vectors calculated by the convolution module to embed the word in the corresponding word It includes a highway network-based highway module that calculates a vector.

According to the above-described problem solving means of the present invention, the unique meaning of a corresponding word is accurately extracted using sub-word information of not only unregistered words but also all words, and effectively in an open vocabulary environment with many unregistered words. can work

In addition, the present invention does not learn for a long time in a corpus, that is, a large corpus in order to generate new word embeddings, and can generate unregistered words using the word embeddings of the existing natural language processing system. efficiency and effectiveness can be improved.

1 is a diagram showing the configuration of a natural language processing system for generating a word expression according to an embodiment of the present invention.
2 is a diagram illustrating a word expression model according to an embodiment of the present invention.
FIG. 3 is a view for explaining a convolution module that is a part of FIG. 2 .
FIG. 4 is a view for explaining a highway module that is a part of FIG. 2 .
5 is a flowchart illustrating a learning process of a word expression model in a word expression method in natural language processing according to an embodiment of the present invention.
6 is a flowchart illustrating an inference process of a word expression model among a word expression method in natural language processing according to an embodiment of the present invention.
Fig. 7 is a diagram for explaining the nearest neighbor of a word expression for a vocabulary and an unregistered word.
8 is a diagram for explaining the word similarity results for the entire data set for comparative evaluation of a word expression method in natural language processing and other learning models according to an embodiment of the present invention.
9 is a diagram showing experimental results of applying the word expression method in natural language processing according to an embodiment of the present invention to an actual natural language processing system.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In the present specification, a 'terminal' may be a wireless communication device with guaranteed portability and mobility, for example, any type of handheld-based wireless communication device such as a smartphone, tablet PC, or notebook computer. In addition, the 'terminal' may be a wired communication device such as a PC that can connect to another terminal or server through a network. In addition, the network refers to a connection structure capable of exchanging information between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet (WWW). : World Wide Web), wired and wireless data networks, telephone networks, and wired and wireless television networks.

Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound Communication, Visible Light Communication (VLC), LiFi, etc. are included, but are not limited thereto.

The following examples are detailed descriptions to help the understanding of the present invention, and do not limit the scope of the present invention. Accordingly, an invention of the same scope performing the same function as the present invention will also fall within the scope of the present invention.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram showing the configuration of a natural language processing system for generating a word expression according to an embodiment of the present invention, FIG. 2 is a diagram for explaining a word expression model according to an embodiment of the present invention, FIG. 3 is FIG. 2 is a diagram for explaining a convolution module, which is a part of FIG. 2 , and FIG. 4 is a diagram for explaining a highway module, which is a part of FIG. 2 .

Referring to FIG. 1 , the natural language processing system 100 includes a communication module 110 , a memory 120 , a processor 130 , and a database 140 .

In detail, the communication module 110 provides a communication interface necessary to provide a transmission/reception signal between the natural language processing system 100 and the user terminal in the form of packet data in connection with a communication network. Furthermore, the communication module 110 may perform a role of receiving a data request from the user terminal and transmitting data as a response thereto.

Here, the communication module 110 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals through wired/wireless connection with other network devices.

In the memory 120, a program for performing a word expression method in natural language processing is recorded. In addition, the processor 130 performs a function of temporarily or permanently storing the processed data. Here, the memory 120 may include a volatile storage medium or a non-volatile storage medium, but the scope of the present invention is not limited thereto.

The processor 130 controls the entire process of providing a word expression method in natural language processing. That is, the processor 130 stores various vocabularies and pre-learned word embedding vectors of each word in the lexicon dataset, and creates a word expression model based on supervised learning of the word embeddings stored in the lexicon dataset. After generating word expressions for not only registered words but also unregistered words based on the learned word expression model, it is possible to infer the unique meaning of unregistered words through nearest neighbor search. Each operation performed by the processor 130 will be described in more detail later.

As shown in FIG. 2 , the word expression model 200 learns to generate a word expression based on supervised learning of previously-learned word embeddings. The word expression model 200 includes a convolution module 210 , a highway module 220 , and an optimization module 230 .

The convolution module 210 extracts character-based lower-order word features through a convolutional neural network. The highway module 220 adaptively combines the subword features extracted from the convolution module 210 using a highway network to calculate a word embedding vector. Also, the optimization module 230 optimizes the word embedding vector calculated by the highway module 220 to be similar to the previously learned word embedding.

Since the convolution module 210 can extract local features from natural language processing, the convolutional neural network is applied to the character sequence to extract lower-order word information. As shown in FIG. 3 , the convolution module 210 includes filters for extracting different features from a character sequence, and extracts feature maps, which are matrices calculated through each filter, through the filters. When the feature map is extracted, a non-linear function (tanh, hyperbolic tangent) is applied to change it to a non-linear value of the presence or absence of the corresponding feature.

In general, the performance improves as the depth of the learned word representation model increases. However, as the depth increases, optimization becomes difficult and training becomes difficult. Highway neural networks allow you to control the flow of information and maximize learning possibilities while deepening your word representation models.

As shown in FIG. 4 , the highway module 220 has a value of input(y) for the subword feature vectors received from the convolution module 210, and performs a nonlinear transformation (T) and a movement (C) is additionally applied. At this time, T is called a transform gate and C is called a carry gate because it expresses how much the output(z) is transformed and shifted with respect to the input(y).

Meanwhile, the processor 130 may include all kinds of devices capable of processing data, such as a processor. Here, the 'processor' may refer to a data processing device embedded in hardware, for example, having a physically structured circuit to perform a function expressed as a code or an instruction included in a program. As an example of the data processing device embedded in the hardware as described above, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit) and a processing device such as a field programmable gate array (FPGA), but the scope of the present invention is not limited thereto.

The database 140 stores data accumulated while performing a word expression method in natural language processing. For example, the database 140 may store a lexicon dataset, a word expression model, an embedding vector of a registered word and an unregistered word, and the like.

5 is a flowchart illustrating a learning process of a word expression model in a word expression method in natural language processing according to an embodiment of the present invention.

Referring to FIG. 5 , the word expression method in natural language processing provides a vocabulary dictionary dataset including vocabulary including at least one or more words and pre-learned word embedding information in the natural language processing system 100 ( S110 ) .

In the word expression model, when a vocabulary based on the vocabulary dictionary dataset is provided as input data (S120), subword information about words existing in the input data is extracted (S130), and the extracted subword information is used. to generate the lower-order word feature vectors, and then combine the lower-order word feature vectors to calculate a word embedding vector (S140).

The word expression model matches the calculated word embedding vector with the pre-learned word embedding information of the corresponding word (S150), and replaces the pre-learned word embedding with the calculated word embedding vector to learn the word expression for the word (S150). S160).

The range of a sub-word is various, such as a root, a character, an N-gram, and the like, but in the word expression model 200, the unit of the sub-word is used as a character. Therefore, the word expression model classifies all words existing in the vocabulary in character units, and uses one-hot encoding as an expression to represent each character. Here, the one-hot encoding has a value of 1 in the index of the corresponding character, otherwise it has a value of 0, so a word is formed by concatenating these character representations.

By concatenating the character representations constituting each word, a character sequence representation as shown in Equation 1 is generated.

[Equation 1]

이며, Vc는 문자들의 어휘이고, n은 단어의 길이이며,

은 연결 연산자이고, c_i은 해당 단어에서 i번째 문자를 표현하기 위한 문자 표현(즉, 원-핫 인코딩)을 각각 나타낸다.In Equation 1, r is a sequence representation for a word,

, where Vc is the vocabulary of characters, n is the length of the word,

is a concatenation operator, and c _i denotes a character representation (ie, one-hot encoding) for expressing the i-th character in the corresponding word, respectively.

At this time, zero-padding is inserted in r to perform the same number of convolutions on all characters, and the number of zero-padding becomes h-1/2.

The convolution module 210 of the word expression model 200 extracts lower-order word information by performing convolution on the padded character sequence expression r as in Equation 1 using Equation 2 below.

[Equation 2]

이고, h는 필터(F)의 폭이며, r_i:i+h-1는 문자 c_i에서 c_i+h-1 사이의 시퀀스 표현이고, b는 바이어스이며, tanh은 합성곱 결과 값들에 대한 비선형 함수를 각각 나타낸다.In Equation 2, F is a filter used in a convolutional neural network,

where h is the width of the filter F, r _i:i+h-1 is the sequence representation between the letters c _i to c _i+h-1 , b is the bias, and tanh is the Each nonlinear function is represented.

The convolution module 210 extracts unique lower-order word information existing in a word, and then applies a Max pooling operation to the extracted lower-order word information, thereby extracting only significant lower-order word features as shown in Equation 3 below. do.

[Equation 3]

In Equation 3, S is a low-order word feature vector, k is the length of the stride, and s _ki:k(i+1)-1 represents a sequence representation with one stride in s, respectively. At this time, since max pooling with stride applied generates features with different lengths depending on the length of the word, the elements of vector e are simply summed to create a fixed-size feature.

The convolution module 210 calculates a convolution while traversing input data at a specified interval for several filters, and the interval at which the filters are traversed at a specified interval is called a stride. When the input data has multiple channels, the filter traverses each channel, calculates the convolution, generates a feature map for each channel, concatenates all the feature maps of each channel, and returns the final feature map.

The resulting vector s has the same length as the input word due to zero padding, and max polling is used with stride to include suffixes, roots and prefixes in the subword information. Consequently, each feature derived from strided max pooling has a summary of subword information in the character sequence.

As such, the word expression model uses a vocabulary dictionary dataset including previously learned word embedding information, unlike the existing method of learning word expressions in an unsupervised learning method in a large corpus. learn word expressions.

The highway module 220 of the word expression model 200 routes the subword feature vector using the gate mechanism according to Equation 4 below, and is useful for convolutional neural networks by adaptively combining the local features of individual filters. . That is, the highway module 220 adaptively combines the lower-order word feature vectors derived from the convolution module 210, and allows the lower-order word feature vectors to be associated with the pre-learned word embedding information.

[Equation 4]

In Equation 4, H is the non-linear transformation at the input e, T is the transformation gate, and C denotes the moving gate, respectively.

In Equation 4, the moving gate C can be simplified to (1-T) and expressed as Equation 5 below.

[Equation 5]

In Equation 5, W _t and W _h are square matrices, b _t and b _h are biases, and tanh is a nonlinear function of the resulting values.

The highway module 220 performs linear transformation on the calculated word embedding vector to generate a word embedding vector having the same size as the previously learned word embedding information using Equation 6 below.

[Equation 6]

In Equation 6, w is the final word representation derived from the word representation model, W and b are parameters of the linear transformation, and y is the result vector. For optimization, the size of the pre-learned word embedding result vector y is set to be the same by using the linear transformation through Equation (6).

As such, the word expression model 200 uses the convolution module 210 and the highway module 220 to generate a word expression while considering lower word information. In addition, the optimization module 230 of the word expression model 200 uses the squared Euclidean distance as in Equation 7 as an objective function to learn the calculated word embedding vector to be similar to the previously learned word embedding vector. In this case, the Euclidean distance is also referred to as the L2 distance.

[Equation 7]

는 기학습된 단어 임베딩 벡터이다. In Equation 7, V _w is a vocabulary in the vocabulary dictionary dataset, w _v is the calculated word embedding vector,

is a pre-learned word embedding vector.

The optimization module 230 calculates the similarity between the word embedding vector calculated using the squared Euclidean distance or the L2 loss function according to Equation 7 and the pre-learned word embedding information, cosine similarity, L 1 distance (L1 Distance or Manhattan Distance) can be used to calculate the similarity between two vectors.

The optimization module 230 replaces the previously learned word embedding with the calculated word embedding vector by matching the word embedding vector calculated for the word with the previously learned word embedding information to learn a word expression for the word.

Referring back to FIG. 1 , when the word 'uncovered' is provided as input data in the lexicon dataset, the convolution module 210 has a different width than the three filters with h=5 and two zero padding added. It is assumed that a filter is used, the stride width is set to 3 for pooling, and a single highway network is used in the highway module 220 .

The word expression model 200 learns a word expression similarly to the pre-learned word embedding information of 'uncovered', and is lexically related to the input data but is not included in the pre-learned word embedding information (for example, , uncovering) can be shown. That is, since 'uncovered' and 'uncovering' share a similar character sequence expression, the word expression model 200 can reconstruct the previously learned word embeddings into characters and present the nearest neighboring words of the learned word expression. .

In this way, the pre-learned word embedding information for all words in the vocabulary dictionary dataset is generalized to the word expression newly created by the word expression model. In this case, since the word expression model classifies sub-words of each word in character units, it is possible to generate word expressions for all words including unregistered words.

6 is a flowchart illustrating an inference process of a word expression model in a word expression method in natural language processing according to an embodiment of the present invention, and FIG. 7 is a diagram illustrating the nearest neighbor of a word expression for a vocabulary and an unregistered word to be.

Referring to FIG. 6 , the word expression model 200, which has learned word expressions for registered words and unregistered words in the vocabulary dictionary dataset, separates the unregistered words into character units when the unregistered words are provided as input data (S210). , after generating a character sequence representation, an unregistered word embedding vector is generated through the convolution module 210 and the highway module 220 ( S220 ).

The word expression model 200 may infer the intrinsic meaning of the unregistered word through a nearest-neighbor search for the unregistered word embedding vector (S230).

Table 1 shows the neighboring words in the vector space for the word expression of the unregistered word through the reasoning process of the word expression model.

[Table 1]

As shown in Table 1, it can be confirmed that the inference process by the word expression model expresses the intrinsic meaning of the unregistered word well. That is, the word expression generated through the word expression model 200 well represents word transformations and compound words (computerization, bluejacket), captures words related to other word styles (global-globally-globalization), and misspelled words (vehicals). ), and can capture semantically related words in most unregistered words (bluejacket-pants, vehicals-bicycles).

As shown in FIG. 7 , when qualitative analysis is performed by searching for nearest neighbor words to evaluate the performance of the word expression model (GWR), the nearest neighbor words of word expressions learned in English by word2vec and GWR are extracted. , and the nearest neighbor words are sorted in descending order of similarity calculated by cosine similarity.

In addition, it can be seen that the nearest neighbor word of the vocabulary word is semantically or syntactically related word located in the nearest neighbor word (taxicab-minibus, teachteaching). Moreover, GWR makes lexically related word representations more similar. For example, it indicates that the variants (connects-connected-connecting) of the neighboring word of “connect” are more closely located than the pre-learned word embeddings in the vector space. This rendering shows similar trends in other words (teach-teaching, computes-compute-computable), and satisfies the characteristics of words regarding lexical relevance. This is because lexically related words share a large portion of a character sequence.

8 is a diagram for explaining the word similarity results for the entire data set for comparative evaluation of a word expression method in natural language processing and other learning models according to an embodiment of the present invention.

Word similarity can be measured by calculating a correlation coefficient between cosine similarities between words, and the ability of each model can be evaluated through word similarity.

First, we collected word similarity datasets for 6 languages: Arabic (Ar), German (De), English (En), Spanish (Es), French (Fr), and Russian (Ru), and We use pre-trained word2vec.

Here, word2vec is a model that considers a word as an individual atomic unit and adopts a window-based learning technique. Since word2vec is a word-based approach, it cannot represent OOV words, uses a null vector for word-like operations as the default value of OOV words, and initializes randomly for language modeling tasks and fine-tunes them.

On the other hand, the FastText method is an extension of word2vec, considers a letter n-gram as the minimum unit, and learns similarly to word2vec's technique. The Mimick method is a character-based model that learns functions from letter to word embeddings to represent words, and uses a skip gram version of word2vec for pre-learned word embeddings.

The word representation model is represented by a generated word representation (GWR) and uses word2vec for pre-learned word embeddings.

As shown in FIG. 8 , it can be seen that the lower-order word-based learning methods (FastText, Mimick, GWR) perform better than the word-based method (word2vec) in the word similarity data set for most languages. This indicates that considering the sub-word information, it is effective when expressing words and is useful in many languages. Among the subword-based methods, GWR outperforms Mimick in all languages except English.

Through the word similarity result shown in FIG. 8, the convolution model of the word expression model is useful for extracting local functions, that is, sub-word information, and supervised learning of word embedding information that has been learned more than FastText, which learns word expressions in a large corpus. It can be seen that GWR, which generates word expressions based on , shows better learning performance.

To confirm that the word representation of GWR-derived unregistered words (OOV) actually improves the performance, it shows significantly better performance than the ^{model that does not generate unregistered words (GWR - ).} This indicates that GWR effectively expands the lexical range and generates expressions of unregistered words very well. Also, GWR ^- shows slightly better performance than word2vec, which shows the effect of considering sub-word information in word representation.

9 is a diagram showing experimental results of applying the word expression method in natural language processing according to an embodiment of the present invention to an actual natural language processing system.

Referring to FIG. 9 , the natural language processing system performs language modeling using an external model to confirm the usefulness of GWR. The performance evaluation scale of the natural language processing system is perplexity, and the lower the perplexity, the stronger the model. Here, perplexity is an internal evaluation index for evaluating a language model and is usually expressed as PPL. The PPL measurement method is one of performance evaluation methods for an extrinsic task.

We perform language modeling tasks using data sets for six languages: Czech (Cs), German (De), English (En), Spanish (Es), French (Fr), and Russian (Ru). Based on the performance of word2vec used to learn GWR, it can be seen that the performance is significantly improved in the morphologically rich language.

For example, the performance of morphologically rich Slavic languages (Cs, Ru) exhibits more complex chaotic reduction (15 and 22% for Czech and Russian, respectively) than other languages in terms of morphology. This indicates that GWR is more useful and effective in morphologically rich languages.

As such, the word expression model shows that it is more effective in terms of processing unregistered words compared to the unregistered word processing method that uses the existing unregistered words to be expressed as pseudo-words.

On the other hand, the word expression model uses the highway module 220 based on the highway network rather than the multi-layer perceptron (MLP) widely used in CNN. Highway networks basically show faster convergence than MLP with respect to learning loss, so learning can be done faster, and 2-layer Highways show faster convergence than 1-layer Highways, so highways according to the layer stack A deeper word expression model can be learned while improving the semantic similarity of words by using the network characteristics.

As described above, the present invention provides a character-based word expression method marked with GWR. A word expression model using CNN and highway networks can generate high-quality expressions of unregistered words in consideration of sub-word information. This word representation model can also be applied to other areas such as text classification and named entity recognition.

The word expression method in natural language processing according to the embodiment of the present invention described above may be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Such recording media includes computer-readable media, and computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Computer readable media also includes computer storage media, which include volatile and nonvolatile embodied in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. , both removable and non-removable media.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: natural language processing system
110: communication module 120: memory
130: processor 140: database

Claims (19)

A method for expressing words in natural language processing performed by a natural language processing system, the method comprising:
a) providing a vocabulary dictionary dataset including a vocabulary including at least one word and pre-learned word embedding information for each word;
b) When a vocabulary based on the vocabulary dictionary dataset is provided as input data, subword information about words existing in the input data is extracted using a word expression model, and the subword information is embedded in words calculating a vector;
c) matching the calculated word embedding vector with pre-learned word embedding information of the corresponding word, thereby replacing the pre-learned word embedding information with the calculated word embedding vector to learn a word expression for the corresponding word;
d) When an unregistered word (Out of Vocabulary) is provided as input data to the learned word expression model, sub-word information is extracted for the unregistered word, and then the word embedding vector of the unregistered word is extracted using the extracted lower-order word information. calculating; and
e) calculating the similarity between word embedding vectors through a vector operation based on the calculated word embedding vector of the unregistered word, extracting neighboring words of the unregistered word, and inferring the unique meaning of the unregistered word,
The word expression model is
A convolutional neural network-based convolution module that calculates lower-order word feature vectors using the lower-order word information, and a word of the corresponding word by adaptively combining the lower-order word feature vectors calculated by the convolutional module Which includes a highway network-based highway module that calculates an embedding vector,
How to represent words in natural language processing. The method of claim 1,
Step b) is,
b-1) extracting lower-order words from all the words included in the input data, assigning individual codes to the extracted lower-order words, and concatenating codes constituting the corresponding words to generate a sequence expression;
b-2) extracting lower-order word information existing in the corresponding word through synthesis with the sequence expression in the convolution module;
b-3) extracting meaningful lower-order word features by applying a pooling operation to the extracted lower-order word information; and
b-4) concatenating all of the extracted sub-word features to calculate a sub-word feature vector through convolution. 3. The method of claim 2,
Step b) is,
When the sub-word is set as a character, a character-based sequence expression is generated for all words including an unregistered word (Out of Vocabulary),
A word expression method in natural language processing, wherein a word embedding vector of a registered word and a word embedding vector of an unregistered word are calculated by using the character-based sequence representation. 4. The method of claim 3,
Step c) is,
By matching the word embedding vector of the registered word with the previously learned word embedding information of the corresponding registered word, the previously learned word embedding is replaced with the word embedding vector of the registered word to learn the word expression for the registered word,
The word expression method in natural language processing, wherein the word expression of the non-registered word is learned by using the word embedding vector of the non-registered word. 3. The method of claim 2,
In b-1), one-hot encoding is applied to represent the sequence expression by Equation 1 below.
[Equation 1]

r: word expression,

Vc: vocabulary

: concatenation operator
c _i : One-hot encoding to represent the i-th character in the word 3. The method of claim 2,
b-2) is a word expression method in natural language processing, wherein the lower-order word information is extracted by synthesizing with the sequence expression through Equation 2 below.
[Equation 2]

F: convolution filter,

Vc: vocabulary
h: width of F
r _i:i+h-1 : represents the sequence between the letters c _i through c _i+h-1
b: bias
tanh: non-linear function of convolution result values 3. The method of claim 2,
b-3) extracts the lower-order word feature vectors by applying a stride pooling operation according to Equation 3 below, and the lower-order word feature vectors include a suffix, a root, and a prefix. How to express words in
[Equation 3]

S: subword feature vector
k: length of stride
s _ki:k(i+1)-1 : Represents a sequence with one stride in s The method of claim 1,
wherein the highway module routes the subword feature vector using a gate mechanism according to Equation 4 below.
[Equation 4]

H: non-linear transformation at input (e)
T: conversion gate
C: moving gate 9. The method of claim 8,
In Equation 4, the moving gate C is simplified to (1-T) and expressed as Equation 5 below, a word expression method in natural language processing.
[Equation 5]

W _t, W _h : square matrices
b _t , b _h : bias
tanh: non-linear function of the resulting values The method of claim 1,
The word expression model performs a linear transformation on the calculated word embedding vector through Equation 6 below to generate a word embedding vector having the same size as the previously learned word embedding information. Way.
[Equation 6]

w: final word expression
W, b: parameters of linear transformation
y: result vector The method of claim 1,
Step c) is,
Cosine similarity between the calculated word embedding vector and pre-learned word embedding information, L 1 distance (L1 Distance or Manhattan Distance), or L2 distance (L2 Distance or Euclidean Distance) Using a similarity calculation method of any one Matching Phosphorus, a word expression method in natural language processing. 12. The method of claim 11,
In step c), the similarity between the calculated word embedding vector and the pre-learned word embedding information is calculated using an L2 loss function according to Equation 7 below.
[Equation 7]

V _w : vocabulary in the lexical dictionary dataset
w _v : calculated word embedding vector

: Pre-Learned Word Embedding Vector delete The method of claim 1,
Step d) is,
When the sub-word is set as a character, a character-based sequence representation is generated for the Out of Vocabulary, and a word embedding vector of the non-registered word is calculated using the character-based sequence representation A method of representing words in natural language processing. 15. The method of claim 14,
The word expression model is
The convolution module extracts sub-word information existing in the non-registered word through synthesis with the character-based sequence expression, and applies a pooling operation to the extracted sub-word information to feature at least one sub-word extracts and connects the sub-word features to calculate a sub-word feature vector,
and associating the subword feature vector of the unregistered word calculated by the highway module with the pre-learned word embedding information using a gate mechanism. The method of claim 1,
The step e) extracts the neighboring words using a nearest-neighbor search algorithm, and sorts word expressions in the neighboring words in descending order of similarity between the word embedding vectors. How to express words in In the natural language processing system for distributed expression of words included in vocabulary,
a memory in which a program for performing a word expression method in natural language processing is recorded; and
a processor for executing the program;
The processor, by executing the program,
A vocabulary including at least one word and a vocabulary based on a vocabulary dictionary dataset including pre-learned word embedding information for each word are provided as input data, and words existing in the input data using a word expression model The pre-learned word embedding information by extracting subword information for , calculating a word embedding vector from the sub-word information, and matching the calculated word embedding vector with the pre-learned word embedding information of the corresponding word is replaced with the calculated word embedding vector to learn a word expression for the word, and when an unregistered word (Out of Vocabulary) is provided as input data to the learned word expression model, lower word information for the unregistered word is obtained After extraction, the word embedding vector of the non-registered word is calculated using the extracted sub-word information, and the similarity between the word embedding vectors is calculated through a vector operation based on the calculated word embedding vector of the non-registered word. Inferring the unique meaning of the unregistered word by extracting
The word expression model is
A convolutional neural network-based convolution module that calculates lower-order word feature vectors using the lower-order word information, and a word of the corresponding word by adaptively combining the lower-order word feature vectors calculated by the convolutional module A natural language processing system comprising a highway module based on a highway network for calculating an embedding vector. 18. The method of claim 17,
The word expression model is
Cosine similarity between the calculated word embedding vector and pre-learned word embedding information, L1 distance (L1 Distance or Manhattan Distance), or L2 distance (L2 Distance or Euclidean Distance) Any one of the similarity calculation method by matching using the method The natural language processing system of claim 1, further comprising an optimization module for reconstructing the previously learned word embedding information into the calculated word embedding vector. delete

Images