JPH11203286A - Natural language processing apparatus and method - Google Patents

Abstract

(57) [Summary] [Problem] To enable a part of an arbitrary size of a word dictionary to be loaded on an internal memory, thereby accelerating the processing and suppressing the size of a used memory according to the situation of natural language processing. Be able to do it. A word dictionary in which words are registered in a tree format.
05a, when processing a natural language sentence, the word dictionary 10
The notation use frequency information in the document processing is stored in 5a, and is loaded into the internal memory 107 by the partial dictionary load unit 102 of the word dictionary 105a to be usable, and the notation appearance frequency calculation unit 101 is displayed and displayed at the time of loading. Calculate the frequency,
The load condition setting unit 103 determines a load condition according to the calculation result and the contents of the predetermined load condition file 103a.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

TECHNICAL FIELD The present invention relates to a tree (TREE).
TECHNICAL FIELD The present invention relates to a natural language processing apparatus and method for analyzing a natural language sentence using a word dictionary of a format.

[0002]

2. Description of the Related Art As a word dictionary used for analyzing a natural language, a tree-type dictionary as shown in FIG. 4 has been frequently used (however, notation frequency is excluded). The word dictionary in the tree format is not registered in an internal storage device such as an IC memory because of its large overall capacity, but is stored as a file in a disk device as an external storage device.

[0003]

However, when performing natural language processing, the word dictionary is frequently accessed, and the access speed to the disk device greatly affects the processing speed of the natural language processing. Although the access speed of the disk device has been improved due to the progress of hardware, it has been a factor that hinders the improvement of the processing speed of the analysis process.

On the other hand, since the access speed to the internal memory is generally much faster than the access speed to the disk device, it is conceivable to load the contents of the dictionary into the internal memory. However, to load all dictionaries holding tens of thousands of words on the internal memory, a considerable amount of memory is required by itself, and large data such as dictionaries can be forcibly stored in insufficient memory. When loading, the processing speed is rather slowed down.

SUMMARY OF THE INVENTION The present invention solves the above-mentioned problems, and enables a part of an arbitrary size of a word dictionary to be loaded on an internal memory, thereby accelerating or using a processing according to the situation of natural language processing. An object of the present invention is to be able to suppress the memory size.

[0006]

As one means for achieving the above object, for example, the following arrangement is provided.

That is, a natural language processing apparatus for analyzing a natural language sentence using a word dictionary in which words are registered in a tree format, wherein an internal storage means capable of storing internal processing data and the internal storage means are provided separately. External storage means having a large storage capacity, and TREE stored in the external storage means.
A word dictionary in which words are registered in a format; and word loading means for loading at least a part of the word dictionary into the internal storage means.

[0008] For example, the invention is characterized by further comprising analysis means for analyzing a natural language using the word dictionary.

Also, for example, the word loading means obtains a probability that a natural language can be analyzed using only the word dictionary loaded in the internal storage means from the size of the word dictionary loaded in the internal storage means, And a load condition setting unit for determining a load portion of the word dictionary to be loaded into the internal storage unit from the obtained probability.

Further, for example, a document information holding means for holding document information, and a notation appearance frequency for calculating the occurrence frequency of each notation of a word registered in the word dictionary in the document information held in the document information holding means Calculation means. Alternatively, the load condition setting means determines a load portion of the word dictionary based on the probability obtained by the notation appearance frequency calculation means.

A natural language processing apparatus for analyzing a natural language sentence using a word dictionary in which words are registered in a tree format, wherein the word dictionary holds the frequency of use of notations in document processing, and stores internal processing data Means for loading a part of the content of the word dictionary into an internal memory capable of storing the word reference frequency with reference to the notation appearance frequency.

[0012] For example, in loading from the word dictionary to the internal storage means, the natural language is used only by using the word dictionary loaded into the internal storage means based on the size of the word dictionary loaded into the internal storage means. It is characterized in that a probability that can be analyzed is obtained, and a load portion of the word dictionary to be loaded into the internal storage means is determined from the obtained probability.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a natural language processing apparatus according to an embodiment of the present invention.

In FIG. 1, reference numeral 100 denotes a control unit for controlling the entire apparatus, 101 denotes a notation frequency calculation unit, 102 denotes a dictionary load unit, 103 denotes a load condition setting unit, 103a denotes a load condition setting file, 104 denotes a document database, 10
Reference numeral 5 denotes a disk device, and a word dictionary 105a is registered in the disk device 105.

A ROM 106 stores a control procedure of the control unit 100 or a control procedure of each part of the natural language processing apparatus, and functions as a control memory. 10
Reference numeral 7 denotes an internal memory having a high access speed, such as an IC, and an internal word dictionary load area 107a of a predetermined size is allocated to the internal memory 107. 110
Is a natural language analysis unit capable of analyzing a natural language stored in the document database 104.

In the above description, for example, the notation appearance frequency calculation unit 101 follows the control procedure shown in the flowchart of FIG. 3 stored in the ROM 106, and the word dictionary 105a in the document information held in the document database 104 described later. Calculate the appearance frequency of each notation of the word registered in. The ROM 106 stores the load condition setting unit 103
And the control procedure of the dictionary loading unit 102 shown in FIG.

In the above description, an example in which the natural language processing is performed by each individual configuration has been described. However, the present invention is not limited to the above example, and the central processing unit (computer device) collectively performs the processing. Each function may be realized. FIG. 2 shows an example in which the functions are collectively realized by the central processing unit (computer device).

FIG. 2 shows a central processing unit (computer unit).
FIG. 4 is a diagram showing an example of a case where each function is realized collectively by the.
2, reference numeral 21 denotes a control memory, which is a ROM of FIG.
106. A control program or the like according to a control procedure as shown in the flowcharts of FIGS. 3 and 6 may be stored and executed by the central processing unit 22.

Reference numeral 22 denotes a central processing unit for performing judgment and calculation in accordance with a control procedure stored in the control memory 21. 23 is an internal memory, 24 is a disk device,
It holds the document database, word dictionary, load condition setting file, and the like. 25 is a bus.

Next, the operation of the notation appearance reliability calculation unit 101 in the present embodiment as described above will be described below with reference to the flowchart shown in FIG.

The notation appearance reliability calculation unit 101 first retrieves the document Dn from the document database 104 in step S301. Subsequently, in step S302, "1" is set to a character position register i (not shown) indicating a character position to be analyzed, which is not shown. Subsequently, in step S303, the word dictionary tree (T
REE) Follow the structure.

For example, when the character position i of the document Dn is the position of the arrow in FIG. 5 in the word dictionary having the tree (TREE) structure shown in FIG. "" To "Arata". In other words, in the character string starting at position i, "ara", "arakata"
It can be seen that the notation has appeared.

Therefore, in the next step S304, the notation appearance frequency corresponding to the words "rough" and "rough" is increased by one.
Next, in step S305, the character position register i is incremented by one (the character position is shifted to the right by one). Then, in a step S306, it is checked whether or not this is the end of the document Dn. If not the end of document Dn, step S
Returning to step 303, the above processing is continued.

On the other hand, if the end of the document Dn is found in step S306, the process proceeds to step S307, where the document database 1
Check whether all 04 documents have been processed. When all the documents have been processed, the process ends. On the other hand, if there is a document that has not been processed, the process returns to step S301.
The next document Dn + 1 is retrieved from the document database 104, and the same processing is performed.

By performing the processing shown in FIG. 3 as described above, the notation appearance frequency for the word dictionary in the natural language document to be processed stored in the document database 104 is obtained, and as shown in FIG. It can be stored in the notation appearance frequency storage area.

The above-mentioned notation appearance frequency may be reset when the document processing is completed, or may be accumulated at a predetermined rate.

Next, the operations of the load condition setting unit 103 and the dictionary load unit 102 according to this embodiment will be described with reference to the flowchart shown in FIG. The following description is
As an example of the embodiment of the load condition setting unit 103,
The method of using the load condition setting file will be described.

FIG. 7 shows an example of the load condition setting file 103a in this embodiment. The load condition setting file 103a is a file for setting an upper limit of the load size of the word dictionary to the internal memory 107 and a lower limit of the non-disk access probability.

FIG. 7 shows a load condition setting file 103.
As an example of the setting condition of a, (a) is an example where the upper limit of the load size of the word dictionary to the internal memory 107 is 2 Mbytes, and indicates that the word dictionary can be loaded by 2 Mbytes. (B) does not specify the size of the internal memory, but sets the size based on the access ratio of the word dictionary stored in the internal memory 107 when processing the document. FIG.
Is an example in which it is set to 60%.

FIG. 7C shows that the upper limit of the load size is "0".
In this case, dictionary loading is not performed at all, and when word dictionaries are used, all word dictionaries stored on the disk device 105 are accessed.
(D) is an example in which the entire word dictionary is loaded into the internal memory 107, and (e) is a setting in which all documents are processed only by accessing the word dictionary in the internal memory 107 when processing a document. .

In this embodiment, the load ratio of the word dictionary to the internal memory 107 is controlled in accordance with the setting of the load condition setting file 103a.

With the above in mind, the control of FIG. 6 will be described. First, the load condition setting unit 103 first proceeds to step S60
In step 1, the current load condition setting file 103a is read. For example, FIG. 7A shows that the word dictionary is not loaded more than 2 Mbytes, and FIG. 7B accesses the dictionary part loaded on the internal memory 107 without accessing the word dictionary of the disk device. This means that the internal memory 107 is loaded with a size sufficient to achieve a probability of only 60%. The load size upper limit value is stored in a register 1 (not shown), and the non-disk access probability lower limit value is stored in a p register (not shown).

Then, in the following step S602, the load condition setting file 103a read in step S601
It is checked whether or not the load size upper limit value is “ALL”. If the upper limit of the load size is "ALL" as shown in FIG. 7D, the process proceeds to step 606, where the entire word dictionary 105a is loaded into the internal memory 107, and the process ends.

For example, a natural language processing device has a sufficiently large internal memory capacity, like a server connected to a large number of clients, and each client performs natural language processing using a word dictionary of the server. In such a case, it is desirable to set the load size upper limit value to “ALL” because the processing efficiency is reduced if the disk device is accessed one by one.

On the other hand, in step S602, step S60
If the load size upper limit value of the load condition setting file 103a read in step 1 is not "ALL", step S
Proceeding to step 603, the load size upper limit value of the load condition setting file 103a read in step S601 is “0”.
Check if it is. If the upper limit of the load size is “0”, the processing ends without loading the word dictionary into the internal memory 107.

On the other hand, in step S603, step S60
If the load size upper limit value of the load condition setting file 103a read in step 1 is not "0", step S60 is executed.
The process proceeds to step 4 to check whether the lower limit of the non-disk access probability is 100 or not. If the non-disk access probability lower limit value is 100, all word dictionary accesses are set to the word dictionary loaded in the internal memory, so that step S606 is performed.
Proceed to.

However, the present invention is not limited to the above example. For example, the notation frequency updated by the processing of FIG. 107 may be loaded. By performing such control, the occupied capacity of the internal memory 107 can be minimized.

On the other hand, if the lower limit of the non-disk access probability is not 100 in step S604, the process proceeds to step S60.
5, the load condition setting unit 101 extracts the notation appearance frequency shown in FIG. 4 in the word dictionary updated in the processing of FIG. 3, the extracted notation appearance frequency, the load size upper limit value of the register l, and the A word dictionary portion that can be loaded into the internal memory 107 is determined from the lower limit value of the non-disk access probability. Here, the parts to be sequentially loaded are determined in descending order of the notation frequency in the word dictionary 105a.

Then, the dictionary loading unit 102 loads the part of the word dictionary 105a set by the load condition setting unit 103 into the internal memory 107 into the internal word dictionary area 107a of the internal memory 107.

The details of the load condition setting of the load condition setting section 103 will be described below. The load condition setting unit 103
If the load size upper limit “l” is set, the notation appearance frequency of the word dictionary 105a calculated by the notation appearance frequency calculation unit 101 is checked, and the word dictionaries are checked in descending order of notation appearance degree.

For example, in the example of FIG. 8, those having a high notation appearance frequency are in the order of "thank you,""no,""no," and so on. If the load size when loading up to "Thank you so much" and "No" is smaller than "l", and the load size when loading up to "Thank you so much", "No" and "No way" exceeds "1", a word dictionary The part of the tree structure related to "Thank you so much" and "No"
That is, the setting is made to load the portion indicated by the thick frame in FIG.

Then, the dictionary loading unit 102 stores, for example, a portion related to “Thank you” and “No” according to this setting in the internal word dictionary area 107 of the internal memory 107.
Load to a.

When the lower limit p of the non-disk access probability is set, the word dictionary is also searched in the order of the frequency of appearance of the word in the word dictionary 105a. I do. For example, the probability of non-disk access when the word dictionary of FIG. 8 is loaded up to p = 65 “thank you” and “no” is (208 + 134) / (208 + 134 + 103 + 56 + 21 + 3) = 65.3%, which exceeds p, so the memory is loaded in the portion indicated by the thick frame in FIG.

As described above, according to the present embodiment, the upper limit of the load size and the lower limit of the non-disk access probability for determining the allocated capacity of the internal word dictionary that can be loaded into the internal memory 107 are set freely. The appropriate portion of the word dictionary 105a to be loaded into the internal memory 107 is determined in accordance with the condition, and the determined portion is loaded onto the internal memory 107. The effect that the memory size can be suppressed can be obtained.

That is, by loading a part of the words registered in the dictionary having a high frequency of appearance into the memory, the frequency of access to the disk device 105 can be reduced while the memory size required for the dictionary is reduced. Can be.

In this case as well, how much of the words registered in the dictionary should be loaded into the memory depends on the execution environment of the natural language processing, the purpose of the natural language processing, and the like. Accordingly, the processing speed can be increased and the size of the memory used can be suppressed, and the word dictionary in the internal memory 107 can be optimally allocated according to the situation.

[0047]

[Other Embodiments] In the embodiment described above,
To specify the conditions for loading the word dictionary into the internal memory,
Although the load condition setting file is used, even if the setting function is prepared in the program, if the central processing unit controls everything as shown in FIG. Various methods may be used.

In the above-described embodiment, the method of directly setting the values of the load size upper limit and the non-disk access probability lower limit, which are the load conditions, is adopted. However, the memory capacity of the computer on which the natural language processing software is installed is taken. , The upper limit may be automatically set for the load size.

The present invention may be applied to a system including a plurality of devices or to an apparatus including a single device. A recording medium storing the software program code for realizing the functions of the above-described embodiments is supplied to a system or an apparatus, and a computer (or CPU or MPU) of the system or the apparatus reads out the program code stored in the recording medium. Needless to say, it can also be achieved by executing.

In this case, the program code itself read from the recording medium realizes the function of the above-described embodiment, and the present invention is constituted by the recording medium on which the program code is recorded. As a recording medium for supplying the program code, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, and the like can be used.

When the computer executes the readout program code, not only the functions of the above-described embodiment are realized, but also the OS and the like running on the computer are actually executed based on the instructions of the program code. It goes without saying that a part or all of the above-described processing is performed, and the functions of the above-described embodiments are realized by the processing.

Further, after the program code read from the recording medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, based on the instruction of the program code, It goes without saying that the CPU included in the function expansion board or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

As described above, according to the present invention, in a natural language processing apparatus for analyzing a natural language sentence using a word dictionary in which words are registered in a tree format, a word dictionary stored in an external storage device Can be loaded into the internal storage means to perform natural language processing, and the processing efficiency can be optimized.

In this case as well, the upper limit of the load size and the lower limit of the non-disk access probability can be set freely according to the situation of the natural language processing, and the optimum load amount can be selected. Size reduction can be performed.

[0054]

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a natural language processing device according to an embodiment of the present invention.

FIG. 2 is a block diagram illustrating an example in which the functions illustrated in FIG. 1 are collectively realized by a central processing unit (computer device).

FIG. 3 is an operation flowchart illustrating a processing procedure of a notation appearance frequency calculation unit according to the embodiment;

FIG. 4 is a diagram showing an example of a tree structure of a word dictionary in the embodiment.

FIG. 5 is a diagram showing an example of a document in a document database according to the embodiment.

FIG. 6 is an operation flowchart illustrating a processing procedure of a load condition setting unit and a dictionary load unit according to the embodiment of the present embodiment;

FIG. 7 is a diagram illustrating an example of a load condition setting file according to the embodiment;

FIG. 8 is a diagram showing an example of a loaded portion of a word dictionary in the embodiment.

[Explanation of symbols]

 Reference Signs List 21 control memory 22 central processing unit 23 memory 24 disk device 25 bus 100 control unit 101 notation appearance frequency calculation unit 102 dictionary load unit 103 load condition setting unit 104 document database 105 disk device 105a word dictionary 106 ROM 107 internal memory 107a internal word dictionary Storage area 110 Natural language analysis unit

Continuation of the front page (72) Inventor Minoru Fujita 3-30-2 Shimomaruko, Ota-ku, Tokyo Inside Canon Inc.

Claims (9)

[Claims] 1. A natural language processing apparatus for analyzing a natural language sentence using a word dictionary in which words are registered in a tree format, comprising: an internal storage unit capable of storing internal processing data; External storage means having a large storage capacity, a word dictionary in which words are registered in a tree (TREE) format stored in the external storage means, and at least a part of the word dictionary is loaded into the internal storage means. A natural language processing device comprising: a word loading unit. 2. The natural language processing apparatus according to claim 1, further comprising an analysis unit that analyzes a natural language using a word dictionary. 3. The word loading means obtains a probability that a natural language can be analyzed using only the word dictionary loaded in the internal storage means from the size of the word dictionary loaded in the internal storage means. 2. A load condition setting means for determining a load portion of the word dictionary to be loaded into the internal storage means from the obtained probability.
Alternatively, the natural language processing device according to claim 2. 4. A document information holding means for holding document information, and a notation appearance for calculating an appearance frequency of each notation of a word registered in the word dictionary in the document information held in the document information holding means. 4. The natural language processing device according to claim 1, further comprising a frequency calculation unit. 5. The natural language processing apparatus according to claim 4, wherein said load condition setting means determines a load part of said word dictionary based on a probability obtained by said notation appearance frequency calculation means. 6. A natural language processing method in a natural language processing device for analyzing a natural language sentence using a word dictionary in which words are registered in a tree format, the method comprising: A natural language processing method comprising: loading a part of the contents of the word dictionary into an internal memory capable of storing processing data with reference to a notation frequency. 7. In loading from the word dictionary to the internal storage means, a probability that a natural language can be analyzed is obtained using only the word dictionary loaded in the internal storage means, and the internal storage is obtained from the obtained probability. 7. The method according to claim 6, wherein a load portion of the word dictionary to be loaded on the means is determined.
Natural language processing method described. 8. A computer storage medium storing a control procedure for realizing the function according to claim 1. Description: 9. A computer program sequence for realizing the function according to any one of claims 1 to 7.