JPH11238061A - Japanese text analysis method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a Japanese text analysis method without an adverse influence due to words displayed through the use of a visually similar character. SOLUTION: In a Japanese text analyzing method for dividing an arbitrary Japanese text into words from the head to the end of a sentence based on a longest matching method for retrieving the longest word which is matched with a title being the notation of a word dictionary by a dictionary retrieval processing using the notation matching of the words and dividing the Japanese text from the head by using the retrieved word; the following processings are executed by using a table where visually similar characters are stored. Namely, the table is referred to and the detected similar character is notation-matched as a similar character in a processing S28 at the time of notation-matching the words.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is incorporated in various devices and systems that require text analysis processing, such as a Japanese text-to-speech conversion device and a machine translation device, and divides an arbitrary Japanese text into words. It relates to a word text analysis method.

[0002]

2. Description of the Related Art In general, a Japanese text analysis method incorporated in a Japanese text-to-speech converter or the like includes, for example,
If the Japanese text is a horizontally written document, the processing is fast and the memory used is small due to the response speed and memory capacity limitation, etc.
The longest left match method is used, which has the advantage of simple processing. The left longest matching method is a method of dividing a text from the left (start of sentence) using the longest word that matches a heading of the word dictionary. FIG. 2 is a flowchart showing the flow of processing of a conventional general Japanese text analysis method based on the longest left matching method, and FIG. 3 is a flowchart showing conventional dictionary search processing by notation matching in FIG. Hereinafter, a general description of a conventional Japanese text analysis method based on the general left longest matching method shown in FIG. 2 (1), and a description of dictionary search processing by notation matching in FIG. 2 shown in FIG. I do. Note that Japanese text is usually composed of half-width characters (1 byte) and full-width characters (2 bytes). However, as preprocessing for text analysis, all half-width characters in the input text are converted to the corresponding full-width characters. It is assumed that the following text analysis processing is performed on a text composed of only full-width characters. This is for simplifying the dictionary search process by notation matching (character comparison).

(1) General description of Japanese text analysis method based on left longest match method In the Japanese text analysis method shown in FIG. 2, for example, a central processing unit (hereinafter, referred to as "CPU") having an arithmetic control function, By a computer having a memory and the like in which a text pointer, a character counter, a word dictionary, and the like are stored,
The following processes S1 to S7 are executed. First, in S1, a text pointer p is set at the head of the horizontally input Japanese text to be analyzed. In S2, the input text and the notation (that is, the heading) are matched with the pointer p at the head, and words that satisfy the connection conditions are searched for in the word dictionary, and all the obtained words are set as word candidates (this notation). The dictionary search by matching will be described later in detail). In S3, it is checked whether or not a word candidate has been obtained, and if it has been obtained, the longest written word is selected from the words (S4). However, if only one word candidate is obtained, it is selected as it is. If no word candidate is obtained in S2, backtracking is performed. Backtracking refers to returning the pointer p to the beginning of the immediately preceding word and selecting the next candidate for that word (the word with the second longest notation) (S
7). Then, the pointer p is advanced by the length of the selected word (S5). If the pointer p has reached the end of the input text, the analysis process is terminated. If not, the process returns to S2 to analyze the next word (S6). By performing the above procedure, the input text is divided into words from the beginning to the end of the sentence.

(2) Description of dictionary search processing S2 by notation matching in FIG. 2 In dictionary search processing S2 by notation matching in FIG. 2, for example, as shown in FIG. To S17 are executed. However, on the input Japanese text, the n-th character counted from the character indicated by the pointer p is p [n], and the n-th character on the notation (heading) of the word w obtained from the word dictionary is w [n]. Is defined. n
Is the zero origin, p [0] represents the character pointed to by the pointer p, and w [0] represents the first character of the notation (heading) of the word w. First, one word w whose notation (heading) starts with the character p [0] is extracted from the word dictionary (S10), and 0 is set in the character counter n (S11). Then, in S12, the character w [n] and the character p [n] are compared. If the result of the comparison is not the same, the notation did not match, so S
If it is the same, the value of the character counter n is incremented by one (S13), and the value of n is compared with the length (number of characters) of the notation (heading) of the word w (S14).

As a result of the comparison, if they are not equal, the process proceeds to S12, and the matching of the next character is attempted. If they are equal, the matching of the notation has been completed up to the end of the word w, so the connection condition of this word is checked (S15). The connection condition is a condition of whether the sentence can be present as the beginning of a sentence, or if the sentence can be grammatically connected to the immediately preceding word. If the connection condition is satisfied, the word w is set as one of the word candidates (S16), and the process proceeds to S17. If the connection condition is not satisfied, the process proceeds to S17. In S17, it is checked whether or not a word whose heading (heading) starts with the letter p [0] is registered in the word dictionary other than the current word w. If it has been registered, the process returns to S10 to retrieve the word from the dictionary, and attempts a similar notation matching as a new word w. If not registered, the process of S2 ends. In this way, input text and notation (heading)
Are found and zero or more word candidates satisfying the connection condition are obtained from the dictionary.

[0006]

However, the conventional Japanese text analysis method has the following problems (a) to (c), and it is difficult to solve these problems. (A) Words written in kana characters (for example, foreign words)
Is a long vowel character "-" (JIS Kanji code: 2
13C). For example, the third character of "department store" and the second character of "note" are represented by a long letter "-". However, instead of this long sound character, a minus sign character "-" (JIS Kanji code: 215D)
May be described as "depart" or "note". The reason why this occurs is that long sounds and minus signs cannot be distinguished on a screen of a word processor or a personal computer. Further, depending on personal preference in sight, notation using a long sign but not using a minus sign may be used.
As described above, if a certain word is described using a minus sign character instead of a long sound character, the notation does not match in the process of S2 in FIG. You will not be able to parse Japanese text correctly.

(B) As a method for solving the problem (a), for example, it is conceivable to additionally register words written using a minus sign character instead of a long sound character in a dictionary. This is because the problem of the notation matching described in (a) can be cleared. However, this method is not practical because the number of words including long letters in the notation is large, and the size of the word dictionary increases due to additional registration. (C) There are other problems caused by misuse or substitution of visually similar characters as in (a) above. For example,
As in "Kasumigaura" and "Kasumigaura", the lowercase "ka" (J
IS Kanji code: 2576) and capital letter "K" (JIS)
Which one of the kanji codes: 2531) is used is completely individual freedom and unpredictable. In addition, the hiragana characters "he", "be", and "@" (JIS kanji code: 2
458-245A) and katakana "he", "be", "pe"
(JIS kanji codes: 2558 to 255A) are also visually very similar, so that there is a good possibility that they will be misused or substituted. Although there is a solution of additionally registering words written using such similar characters in the dictionary as long as possible, as described above, another problem arises that the size of the word dictionary is dramatically increased by the additional registration. SUMMARY OF THE INVENTION It is an object of the present invention to provide a method for analyzing Japanese text that solves the problems of the prior art and does not cause any adverse effects due to words written using visually similar characters.

[0008]

In order to solve the above-mentioned problems, according to the first aspect of the present invention, the longest match with a heading which is a notation in a word dictionary is performed by a dictionary search process based on word notation matching. A Japanese text analysis method that searches for a word, and divides any Japanese text into words from the beginning to the end of the sentence based on the longest match method that uses the searched word to divide from the beginning of the Japanese text , A table in which characters that are visually similar are stored is provided, and at the time of the notation matching of the words, the notation matching is performed with the similar characters detected with reference to the table as the same characters. In the invention according to claim 2, in the method for analyzing Japanese text according to claim 1,
At the time of word notation matching, notation matching is performed using the long letter "-" and the minus sign character "-" as the same character. According to the third aspect of the present invention, in the Japanese text analysis method of the first aspect, when matching notation of words, lowercase letters "ka" and "ka" and uppercase letters "ke" and "ka" are the same. Notation matching is performed as characters.

According to a fourth aspect of the present invention, in the method of analyzing a Japanese text according to the first aspect, when matching notation of a word, hiragana "he", "be", "@" and katakana "he", Notation matching is performed by using "be" and "pe" as the same characters. By adopting such a configuration, when Japanese text is input,
A table in which visually similar characters are stored is referred to, and at the time of word notation matching, notation matching is performed as if the detected similar characters are the same character, and from the beginning of the input Japanese text. It is divided into words until the end of the sentence. In the invention according to claim 5, the longest word that matches the heading that is the notation of the word dictionary is searched by the dictionary search process based on the word notation matching, and the searched word is used to divide from the beginning of the Japanese text. A Japanese text analysis method for dividing an arbitrary Japanese text into words from the beginning to the end of a sentence based on the longest matching method, comprising a table in which corresponding horizontal and vertical characters are stored; At the time of matching, notation matching is performed with the corresponding characters detected with reference to the table as the same characters. In the invention according to claim 6, in the Japanese text analysis method according to claim 5, when performing notation matching of words, notation matching is performed by using the long letter "-" and the vertical line symbol character "|" as the same character. I have to. By adopting such a configuration, when Japanese text is input, a table storing the corresponding horizontal writing characters and vertical writing characters is referred to, and at the time of word notation matching, the detected corresponding character is Notation matching is performed as the same character, and the input Japanese text is divided into words from the beginning to the end of the sentence.

[0010]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a flowchart of a dictionary search process by notation matching performed by a Japanese text analysis method based on the longest left matching method according to an embodiment of the present invention.
FIG. 4 is a diagram showing a general example of a similar character table (visually similar characters) used in the processing of FIG. Hereinafter, with reference to FIGS. 1 and 4, an overall description (1) of the Japanese text analysis method based on the longest left match method of the present embodiment and a description (2) of a specific example of this analysis method will be described. Do.

(1) Overall description of Japanese text analysis method based on left longest match method In the Japanese text analysis method of this embodiment, for example, a CPU having an arithmetic control function, a character counter, a word dictionary, The processing is executed according to the flowchart of FIG. 2 in the same manner as in the related art using a computer having a memory or the like in which a similar character table or the like in which visually similar characters as described are described is stored. This embodiment is characterized in that FIG.
In the process S2, a dictionary search process by notation matching as shown in FIG. 1 is performed instead of the conventional process as shown in FIG. At this time, a similar character table as shown in FIG. 4 created in advance is referred to. A, A in FIG.
B, C, D, E, and F are table identifiers. As described above, the feature of the present embodiment is a dictionary search based on the notation matching in the process S2 in FIG. 2, and therefore, the description of the same processes as in the related art is omitted, and the description is based on the notation matching in FIG. The dictionary search processing will be described below.

In the dictionary search of FIG. 1, processes S20 to S28
Is executed. However, as in FIG. 3, the n-th character counted from the character pointed to by the pointer p on the horizontally input Japanese text is p [n], and n in the notation (heading) of the word w obtained from the word dictionary. The w th character is defined as w [n]. n
Is the zero origin, p [0] represents the character pointed to by the pointer p, and w [0] represents the first character of the notation (heading) of the word w. First, one word w whose notation (heading) starts with the character p [0] is extracted from the word dictionary (S20), and 0 is set in the character counter n (S21). Then, in S22, the character w [n] and the character p [n] are compared. As a result of the comparison, if they are not the same, the process proceeds to S28 to check whether they are similar characters, and if they are the same, the value of the character counter n is increased by one (S23), and the value of n and the word w
The length (number of characters) of the notation (heading) is compared (S2
4). If the result of the comparison is not equal, proceed to S22,
Attempts to match the next character. If they are equal, the matching of the notation has been completed up to the end of the word w, so the connection condition of this word is checked (S25). The connection condition is a condition of whether the sentence can be present as the beginning of a sentence, or if the sentence can be grammatically connected to the immediately preceding word.

If the connection condition is satisfied, the word w is set as one of the word candidates (S26), and the process proceeds to S27. If the connection condition is not satisfied, the process proceeds to S27. In S27, it is checked whether a word whose heading (heading) starts with the letter p [0] is not registered in the word dictionary other than the current word w. If it is registered, the process returns to S20, where the word is taken out of the dictionary, and the same notation matching is attempted as a new word w. If not registered, the dictionary search process ends. In S28, it is checked whether or not the character w [n] and the character p [n] exist in the similar character table shown in FIG. If it is in the table, it is determined that the character w [n] and the character p [n] are the same character, and the notation is matched, and the process proceeds to S23. If it is not in the table, the notation did not match, and the process proceeds to S27 as before. According to the above processing, even when the input Japanese text matches the notation (heading) and satisfies the connection condition, when extracting 0 or more word candidates from the dictionary, even if the word is written using similar characters, It will be correctly extracted as a word candidate.

(2) Description of Specific Examples of Analysis Method Some specific examples of the above (1) will be described. For example,
Consider a case where the input Japanese text is a "note" written in minus sign characters. In FIG. 2, first, the pointer p points to "No" (S1), and then a dictionary search by notation matching is performed (S2). In S2, a series of processing shown in FIG. 1 is performed. First, in S20, one word w whose notation (heading) starts with the letter p [0], that is, “no” is extracted from the dictionary. Here, it is assumed that “norma” has been extracted. In S21, the character counter n is set to 0, and in S22, the character p [n] and the character w are set.
When [n] is compared, they are both “No” and the same, so the value of the character counter n is increased by one (S23). Since the value of n is currently 1 and is not equal to the length (number of characters) 3 of the notation "norma" of the word w (S24), the process proceeds to S22.

When the character p [n] is compared with the character w [n] in S22, the former is "-" and the latter is "ru", which is not the same. In S28, it is checked whether or not the character p [n], that is, "-", and the character w [n], that is, "ru", are in the similar character table of FIG.
Proceed to. In S27, it is checked whether or not the word starting with the notation (heading) with the letter p [0] has not been registered in the dictionary other than "Norma". Since "Note" has been registered, the process returns to S20 and returns to S20. The note is taken from the dictionary as a new word w. In step S21, the character counter n is set to 0. When the character p [n] and the character w [n] are compared in step S22, they are both "No" and the same, so the value of the character counter n is increased by one (S23). ). Since the value of n is currently 1 and is not equal to the length (number of characters) 3 of the notation "note" of the word w (S24), the process proceeds to S22.

When the character p [n] is compared with the character w [n] in S22, the former is a minus sign character "-" and the latter is a long sound character "-", which is not the same. S2
At step 8, it is checked whether the character p [n], that is, the minus sign character "-", and the character w [n], that is, the long character "-", are in the similar character table of FIG. Then, the value of the character counter n is increased by one. Since the value of n is currently 2 and is not equal to the length (number of characters) 3 of the notation "note" of the word w (S24), S22
Proceed to. When the character p [n] and the character w [n] are compared in S22, both are "g" and are the same, so the value of the character counter n is increased by one (S23). Since the value of n is currently 3, which is equal to the length (number of characters) of the notation "note" of the word w, the process proceeds to S25. In S25, the connection condition is checked, but the word w is a noun “note” and can exist as the beginning of a sentence. Because of this, the connection conditions are satisfied.
In S26, the word "note" is adopted as one of the word candidates. Then, in S27, it is checked whether a word whose notation (heading) starts with the letter p [0] is not yet registered in the dictionary other than "norma" and "note",
Since "NO" has been registered, the process returns to S20, and "NO" is extracted from the dictionary as a new word w.

In step S21, the character counter n is set to 0. In step S22, the character p [n] and the character w [n] are compared.
Is increased by one (S23). The value of n is currently 1,
Since the length (number of characters) of the notation “No” of the word w is not equal to 2 (S24), the process proceeds to S22. The letter p in S22
When [n] is compared with the character w [n], the former is a minus sign character “−” and the latter is a long sound character “−”, which is not the same, so the process proceeds to S28. In S28, the character p [n], that is, the minus sign character "-", and the character w [n], that is, the long sound character "-"
Is checked in the similar character table of FIG. 4, and since it is in table A, the process proceeds to S23, where the value of the character counter n is incremented by one. Since the value of n is currently 2 and equal to the length (number of characters) of the notation “no” of the word w, 2
Proceed to S25. In S25, the connection condition is checked, but the word w is the interjection "no" and can exist as the beginning of a sentence. Since the connection condition is satisfied, the word “NO” is adopted as one of the word candidates in S26. Then, in S27, it is checked whether or not a word whose notation (heading) starts with the character p [0] is not registered in the dictionary other than "norma", "note", and "no", and is not registered anymore. Therefore, the dictionary search process of FIG.
End 2).

In this way, two word candidates "note" and "no" are obtained from the dictionary, and therefore, from S3 to S4 in FIG.
Proceed to. In S4, the longest word, ie, "note", is selected from the two word candidates "note" and "no". In S5, the pointer p is advanced by the length (= 3) of the selected word "note".
Has reached the end of the input text "note", the text analysis process of FIG. 2 ends (S6). In this way, the input text "note" written in minus sign character
Is correctly analyzed using the word "note" written in long letters. As described above, the present embodiment has the following effects. In the present embodiment, it is possible to take advantage of the longest matching method in which the processing is fast and uses less memory, and to prevent an error in analyzing Japanese text caused by misuse or substitution of visually similar characters. become. The present invention is not limited to the above embodiment,
Various modifications and usage forms are possible. For example, there are the following modifications (i) and (ii) as the modified examples and the use forms.

(I) FIG. 5 is a diagram showing a special example of the similar character table of FIG. 1 (when the input text is written vertically). In the above embodiment, the case where the input Japanese text is written horizontally is described. However, when the input Japanese text is created as a vertical writing document, for example, a long sound character "-" (JIS Kanji code: 213C) Instead of the vertical bar symbol character "|" (JIS Kanji code: 214
It may be described using 3). Also in this case, by using the analysis method of the above embodiment, it is possible to solve the problem that the analysis cannot be correctly performed because the notations do not match. That is, the long letter "-" (JIS
Kanji code: 213C) and vertical line symbol character "|" (JIS
Although the kanji code: 2143) is not a visually similar character, the problem in this case can be solved by adding a similar character table shown in FIG. 5 as a special example to the similar character table in FIG. Further, by adding new table data to the tables of FIGS. 4 and 5, it is possible to further improve the analysis accuracy for an arbitrary Japanese text. (Ii) The processing contents of FIGS. 1 and 2 can be changed to contents other than those shown. The device that performs these processes is not limited to a computer, and can be executed by a device having another configuration.

[0020]

As described in detail above, claims 1 to 5
According to the invention of Item 4, in the Japanese text analysis method based on the longest match method,
By referring to a table in which visually similar characters are stored, notation matching is performed using the detected similar characters as the same characters. This makes it possible to take advantage of the longest match method, which is fast in processing and uses less memory, and to prevent Japanese text analysis errors caused by misuse or substitution of visually similar characters. . According to the invention according to claims 5 and 6, in the Japanese text analysis method based on the longest match method, at the time of word notation matching, referring to a table in which corresponding horizontal writing characters and vertical writing characters are stored, Notation matching is performed using the detected corresponding character as the same character. Therefore, whether the input Japanese text is a horizontal writing document or a vertical writing document,
As in the invention according to the first aspect, it is possible to accurately prevent the analysis error of the Japanese text. Therefore, the Japanese text analysis method of the present invention can be used by incorporating it into various devices and systems that require text analysis processing, such as a Japanese text-to-speech conversion device and a machine translation device.

[Brief description of the drawings]

FIG. 1 is a flowchart illustrating a dictionary search process using notation matching according to an embodiment of the present invention.

FIG. 2 is a flowchart of a conventional general Japanese text analysis method based on the longest left matching method.

FIG. 3 is a flowchart of a conventional dictionary search process using notation matching in FIG. 2;

FIG. 4 is a diagram illustrating a general example (visually similar characters) of the similar character table in FIG. 1;

FIG. 5 is a diagram illustrating a special example of the similar character table in FIG. 1 (when the input text is written vertically).

[Explanation of symbols]

S1 Text pointer initialization processing S2 Dictionary search processing by notation matching S3 Check processing whether word candidates are obtained S4 Longest word selection processing S5 Text pointer updating processing S6 Judgment processing whether text pointer reaches the end of sentence S7 Backtrack processing S20 Processing to retrieve word w whose notation starts with character p [0] from dictionary S21 Initialization processing of character counter n S22 Comparison processing of whether character w [n] and character p [n] are the same S23 Character Update process of counter n S24 Comparison process of whether character counter n is equal to the length of word w S25 Check process of whether word w satisfies connection condition S26 Process of adding word w to word candidate S27 Notation Search processing to determine whether another word starting with p [0] is still in the dictionary S28 Referring to the similar character table The letter w
Processing for determining whether [n] and character p [n] are similar characters

Claims (6)

[Claims] 1. A longest word that matches a heading, which is a notation of a word dictionary, is searched by a dictionary search process based on word notation matching, and the longest match is divided from the beginning of the Japanese text using the searched word. In a Japanese text analysis method for dividing an arbitrary Japanese text into words from the beginning to the end of a sentence based on the method, a table in which visually similar characters are stored is provided. A method for analyzing Japanese text, comprising: performing notation matching on similar characters detected with reference to the table as the same character. 2. The Japanese text analysis method according to claim 1, wherein, when the notation matching of the words is performed, the notation matching is performed with the long character "-" and the minus sign character "-" as the same character. 3. The method according to claim 1, wherein when performing notation matching of words, the notation matching is performed by using lowercase “ga”, “ka” and uppercase “ke”, “ka” as the same characters. Japanese text analysis method described. 4. When matching notation of words, hiragana “he”, “be”, “ぺ” and katakana “he”, “be”,
2. The Japanese text analysis method according to claim 1, wherein the notation matching is performed using "pe" as the same character. 5. A longest match that is searched from the beginning of a Japanese text using the searched words by searching for the longest word that matches a heading that is a notation in a word dictionary by dictionary search processing based on word notation matching. A Japanese text analysis method for dividing an arbitrary Japanese text into words from the beginning to the end of a sentence based on the law, comprising a table in which corresponding horizontal and vertical characters are stored. And performing notation matching using the corresponding characters detected with reference to the table as the same characters. 6. The Japanese text analysis method according to claim 5, wherein, when the notation matching of the word is performed, the notation matching is performed by using the long character "-" and the vertical line symbol character "|" as the same character. .