WO2019100458A1 - Method and device for segmenting thai syllables - Google Patents

Abstract

The present invention relates to the technical field of information retrieval, and disclosed thereby are a method and device for segmenting Thai syllables. The method comprises: pre-processing a Thai text to be processed acquired from a Thai corpus, and determining non-Thai character strings and location syllable type information of each Thai character; labeling boundaries between each character in the Thai text to be processed, wherein each boundary consisting of at least one Thai syllable character is labeled as an identification to be segmented; extracting each syllable to be segmented in the Thai text to be processed, wherein each syllable to be segmented is composed of a Thai character which continuously appears n times and an identification to be segmented, and n is a positive integer; determining, according to the location syllable type information of the Thai characters in the syllables to be segmented, the segmentation probability of each syllable to be segmented by using a Markov chain probability voice model; and segmenting and configuring the syllables in a Thai sentence to be processed according to each syllable to be segmented and the segmentation probability corresponding thereto.

Description

Method and device for segmenting Thai syllables Technical field

The invention relates to the technical field of information retrieval, in particular to a method and a device for segmentation of a Thai syllable.

Background technique

也称傣语(Dai language)，是傣泰民族的语言，属于东亚语系/汉藏语系的一种语言。全球有约6800万人口使用泰语。泰语的文本中，词与词之间不用标点，不留空格，一句话从头到尾连续不断的拼写，一般，以空两个字母的间隔或句子当中的小停顿表示一个句子。然而，作为一个在语法中被明确定义的基本单元，文本中的泰语音节和音节之间却没有明显的空格存在。因此，泰语文本的处理操作首先得要对泰语文本进行音节的切分。这项切分处理工作为泰文的词法，句法，以及更复杂的自然语言处理算法任务提供了重要的基础。 Thai

Also known as Dai language, it is the language of the Shutai nationality and belongs to the East Asian language/Chinese-Tibetan language. About 68 million people worldwide use Thai. In the Thai text, there is no punctuation between words and words, no spaces are left, and a sentence is spelled continuously from beginning to end. Generally, a sentence is represented by a space between two letters or a small pause in a sentence. However, as a basic unit that is clearly defined in the grammar, there are no obvious spaces between the Thai syllables and syllables in the text. Therefore, the processing of Thai text must first be syllabic to the Thai text. This segmentation process provides an important foundation for Thai lexical, syntactic, and more complex natural language processing algorithm tasks.

At present, the syllable segmentation of Thai can be performed according to more than 200 rules that Thai scholars have summarized the grammatical patterns of syllables. However, because the grammar rules are complex and difficult to understand, there may be conflicts between a large number of rules, which makes the syllable segmentation speed of Thai is slower and the accuracy is not very high.

Summary of the invention

Embodiments of the present invention provide a method and apparatus for segmentation of Thai syllables. In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This generalization is not a general comment, nor is it intended to identify key/critical constituent elements or to describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the following detailed description.

According to a first aspect of the embodiments of the present invention, a method for segmentation of a Thai syllable is provided, including:

Preprocessing the pending Thai text obtained from the Thai corpus to determine the non-Thai character string and the location syllable type information of each Thai character;

Labeling a boundary between each character in the Thai text to be processed, wherein a boundary composed of at least one Thai syllable character is marked as a to-be-sliced identifier;

Extracting each to-be-segmented syllable in the Thai text to be processed, wherein the to-be-segmented syllable is composed of Thai characters that appear n times consecutively, and one of the to-be-segmented identifiers, and n is a positive integer;

Determining, according to the position syllable type information of the Thai character in the syllable to be segmented, using a Markov chain probability speech model to determine a singular probability of each syllable to be sliced;

The syllables in the pending Thai sentence are set according to each syllable to be sliced and its corresponding segmentation probability.

In an embodiment of the invention, the preprocessing of the Thai text to be processed, determining the non-Thai character string, and the position syllable type information of each Thai character include:

Identifying a non-Thai character string in the Thai text to be processed;

Determining position syllable type information of each Thai character in the to-be-processed Thai text according to a correspondence between the saved Thai character and the position syllable type information, wherein the position syllable type information is generated according to the basic grammatical rules of the Thai language , including syllable type information and location information.

In an embodiment of the invention, the labeling the boundary between each character in the Thai text to be processed includes:

Mark the boundary between two non-Thai syllable characters as the first segmentation identifier;

Marking the boundary between two Thai syllable characters as the to-be-divided identifier;

A boundary between a Thai syllable character and a non-Thai character string is marked as the to-be-sliced identifier.

In an embodiment of the invention, according to the position syllable type information of the Thai character in the syllable to be sliced, using the Markov chain model, determining the segmentation probability of each syllable to be sliced includes:

If the position syllable type information of the Thai character is a consonant character that is not at the end of the syllable, the cleavage probability corresponding to the syllable to be sliced by the Thai character arranged in the first order is zero;

If the position syllable type information of the Thai character is a vowel character that is not at the beginning of the syllable, the segmentation identifier corresponding to the to-be-segmented syllable of the Thai character is arranged by the second order The probability is zero.

In an embodiment of the invention, the syllable in the to-be-processed Thai sentence is divided according to each syllable to be sliced and its corresponding segmentation probability, including:

Determining a pre-processing segmentation probability of each pre-processed syllable in the set Thai sentence according to each to-be-segmented syllable and its corresponding segmentation probability;

The syllables in the set Thai sentence are divided according to the size of the pre-processed segmentation probability.

According to a second aspect of the embodiments of the present invention, there is provided an apparatus for segmentation of a Thai syllable, comprising:

a preprocessing unit for preprocessing the pending Thai text obtained from the Thai corpus, determining a non-Thai character string, and location syllable type information of each Thai character;

An identifier unit, configured to label a boundary between each character in the Thai text to be processed, wherein a boundary composed of at least one Thai syllable character is marked as a to-be-sliced identifier;

An extracting unit, configured to extract each to-be-segmented syllable in the Thai text to be processed, wherein the to-be-segmented syllable is composed of Thai characters that appear n times consecutively, and one of the to-be-segmented identifiers, n is Positive integer

a probability determining unit, configured to determine, according to the position syllable type information of the Thai character in the syllable to be sliced, a Markov chain probabilistic speech model to determine a puncturing probability of each syllable to be sliced;

A segmentation unit is configured to slice the syllables in the pending Thai sentence according to each syllable to be sliced and its corresponding segmentation probability.

In an embodiment of the present invention, the pre-processing unit is further configured to: identify a non-Thai character string in the Thai text to be processed; and, according to a correspondence between the saved Thai character and the location syllable type information, Determining position syllable type information of each Thai character in the to-be-processed Thai text, wherein the position syllable type information is generated according to basic grammatical rules of Thai, including syllable type information and position information.

In an embodiment of the invention, the identifier unit is further configured to mark a boundary between two non-Thai phonological characters as a first segmentation identifier; and, to mark a boundary between two Thai syllable characters as The to-be-sliced identifier marks a boundary between a Thai syllable character and a non-Thai character string as the to-be-sliced identifier.

In an embodiment of the present invention, the probability determining unit is further configured to: if the position syllable type information of the Thai character is a consonant character not at the end of the syllable, the Thai character arranged in the first order and the to-be-cut The segmentation probability corresponding to the sub-identified segmentation syllable is zero; if the position syllable type information of the Thai character is a vowel character not at the beginning of the syllable, the segmentation to be segmented by the second sequence is included The segmentation probability corresponding to the to-be-segmented syllable of the Thai character is zero.

In an embodiment of the present invention, the segmentation unit is further configured to determine, according to each syllable to be sliced and its corresponding segmentation probability, a pre-cut of each pre-processed syllable in the set Thai sentence a sub-probability; according to the size of the pre-processing segmentation probability, the syllables in the set Thai sentence to be processed are segmented.

The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

In the embodiment of the present invention, the Markov chain probability speech model can be used to determine the segmentation probability of each syllable by using the position syllable type information of the Thai character, and the syllables in the Thai sentence are segmented according to the segmentation probability. Another way to segment the Thai syllables. And the segmentation method is based on the basic grammar rules of Thai language. The n-gram grammar of Markov chain probability speech model is used to carry out probability statistics, which improves the accuracy and segmentation speed of Thai syllable segmentation.

The above general description and the following detailed description are merely illustrative and illustrative and are not restrictive.

DRAWINGS

The accompanying drawings, which are incorporated in the specification of FIG

FIG. 1 is a flowchart of a Thai syllable segmentation method according to an exemplary embodiment;

2 is a flow chart showing a Thai syllable segmentation method according to an exemplary embodiment;

FIG. 3 is a block diagram of a Thai syllabary segmentation device according to an exemplary embodiment; FIG.

FIG. 4 is a block diagram of a Thai syllable segmentation apparatus according to an exemplary embodiment.

Detailed ways

The detailed description of the embodiments of the invention are set forth in the description The examples represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operations may vary. Portions and features of some embodiments may be included or substituted for portions and features of other embodiments. The scope of the embodiments of the invention includes the full scope of the claims and all equivalents of the claims. In this context, various embodiments may be referred to individually or collectively by the term "invention," for convenience only, and if more than one invention is disclosed, it is not intended to automatically limit the scope of the application to any A single invention or inventive concept. Herein, relational terms such as first and second are used merely to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship between the entities or operations or order. Furthermore, the terms "comprises" or "comprising" or "comprising" or any other variations are intended to encompass a non-exclusive inclusion, such that a process, method, or device that includes a plurality of elements includes not only those elements but also other items not specifically listed. Elements. The various embodiments herein are described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same similar parts between the various embodiments may be referred to each other. For the structures, products, and the like disclosed in the embodiments, since they correspond to the parts disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method parts.

In Thai documents, there is no punctuation between words and words, no spaces are left, and a sentence is spelled continuously from beginning to end. It is difficult to identify Thai syllables from Thai documents. In the embodiment of the present invention, the Markov chain probability speech model can be used to determine the segmentation probability of each syllable by using the position syllable type information of the Thai character, and the syllables in the Thai sentence are segmented according to the segmentation probability. Another way to segment the Thai syllables. And the segmentation method is based on the basic grammar rules of Thai language. The n-gram grammar of Markov chain probability speech model is used to carry out probability statistics, which improves the accuracy and segmentation speed of Thai syllable segmentation.

FIG. 1 is a flow chart showing a Thai syllable segmentation method according to an exemplary embodiment. As shown in Figure 1, the process of segmentation of Thai syllables includes:

Step 101: Preprocess the pending Thai text obtained from the Thai corpus to determine a non-Thai character string and position syllable type information of each Thai character.

In the embodiment of the present invention, the Thai corpus includes various types of Thai texts, which may include news, encyclopedia, novels, essays, and the like, respectively. The corpus text is obtained from the Thai corpus, where the Thai text to be processed is obtained from the Thai corpus.

The Thai text to be processed contains not only Thai characters, but also non-Thai characters. Among them, non-Thai characters can include: foreign characters (such as English characters), Arabic numerals, punctuation and blank. Therefore, the non-Thai character string in the Thai text can be treated first, so that the Thai characters and non-Thai characters in the Thai text to be processed can be distinguished.

A syllable of Thai is mainly composed of vowel characters, consonant characters and tones. Among them, for 15 Thai vowel characters, they can be combined into at least 28 vowel forms such as diphthong or vowel, and appear in the syllable. For 44 Thai consonants, it can be used as the syllable head (open consonant) and tail (finish tail). There are five tones included in the syllable. Based on the basic grammar rules of the above Thai language, a correspondence between a Thai character and a position syllable type information may be generated and saved, wherein the position syllable type information is generated according to the basic grammatical rules of the Thai language, including syllable type information and position information.

For example, the correspondence between Thai characters and position syllable type information can be as shown in Table 1.

Table 1

则可根据表1，确定该泰文字符的位置音节类型信息为不在音节末尾的辅音。 Thereby, the position syllable type information of each Thai character in the Thai text to be processed can be determined according to the correspondence between the saved Thai characters and the position syllable type information. For example: Thai characters in Thai text to be processed include Thai characters

According to Table 1, the position syllable type information of the Thai character is determined to be a consonant not at the end of the syllable.

Step 102: Label the boundary between each character in the Thai text to be processed, wherein a boundary composed of at least one Thai syllable character is marked as a to-be-sliced identifier.

In the embodiment of the present invention, the boundary between each character may be marked by a method of manual identification, that is, after the artificial Thai syllable is segmented, the boundary between the characters is marked. Of course, the boundary can also be automatically tagged, and the smart match identifies the Thai syllable segment and marks the boundary between the characters. Either way, the label set {S, B} can be used to label the boundaries between Thai characters.

Wherein, the boundary between the two non-Thai phonological characters is marked as the first segmentation identifier; and the boundary of the two Thai syllable characters is marked as the to-be-segmented identifier, and a Thai syllable character is associated with a non-Thai language The boundary between the strings is marked as the identifier to be sliced.

Here, the boundary between the character and the character in the Thai text to be processed may be denoted by I _i , wherein the boundary between the two non-Thai syllable characters may be marked as the first segmentation identifier, that is, I _i =S, ie, S The first segmentation identifier. The boundary between the two Thai syllable characters can be marked as the identifier to be sliced; and the boundary between a Thai syllable character and a non-Thai character string is marked as the identifier to be sliced. At this time, I _i = B, that is, B is the identifier to be sliced. Through the above identification process, the Thai text to be processed can be indicated by C ₁ I ₁ C ₂ I ₂ ... C _i I _i ... CnIn, where C _i is the character in the Thai text to be processed, and I _i is the boundary between the characters. . Where I _i = S, or I _i = B, n is a positive integer.

Here, the labeling of the boundary between each character in the Thai text is a pre-processing process. The above-mentioned Thai syllable characters can be determined according to the method of manual identification or intelligent recognition, but the specific syllable division needs to be followed. step.

Step 103: Extract each to-be-segmented syllable in the Thai text to be processed, wherein the to-be-segmented syllable is composed of a Thai character that appears n times consecutively and a to-be-divided identifier.

Since the boundary between Thai characters is not easy to be divided, the syllable composed of the Thai character and the to-be-sliced identifier is extracted from the Thai text to be processed, and the extracted syllable is a syllable to be sliced, and the syllable to be sliced includes n consecutive occurrences. Thai characters and a to-be-divided logo. Here, n is a positive integer.

If the to-be-segmented syllable consists of a Thai character that appears once and a to-be-divided identifier, it can be represented by CB, or BC. If the to-be-segmented syllable consists of 2 Thai characters and a to-be-divided identifier, it is available. CCB, CBC, or BCC indicates that if the syllable to be sliced is composed of three Thai characters and one to be sliced, it can be represented by CCCB, CCBC, CBCC, or BCCC. If the to-be-segmented syllable consists of 4 Thai characters and a to-be-divided identifier, it can be represented by CCCCB, CCCBC, CCBCC, CBCCC or BCCC. And so on. Where C is the Thai character and B is the identity to be sliced.

The extracted syllables to be sliced may be the syllabic syllables to be successively appearing once in the Thai character, or the syllables to be sliced in which the Thai characters are consecutively displayed 2 times, or the syllables to be sliced in which the Thai characters are consecutively 3 times, or , including the to-be-segmented syllables in which the Thai characters appear one time in succession, the to-be-segmented syllables in which the Thai characters appear consecutively, and the to-be-segmented syllables in which the Thai characters appear consecutively three times. Of course, there are other options, depending on the application scenario, it will not be enumerated.

Step 104: Determine a segmentation probability of each syllable to be segmented by using a Markov chain probability speech model according to the position syllable type information of the Thai character in the syllable to be segmented.

Markov chain is a discrete event stochastic process with Markov property in exponentials. The Markov chain is a stochastic process that satisfies the Markov property. Markov Chain, describes a sequence of states, X ₁ , X ₂ , X ₃ ..., each of which depends on a finite number of states. A Markov chain is a sequence of random variables with Markov properties. The range of these variables, the set of all their possible values, is called the "state space", and the value of X _n is the state of time n. If the conditional probability distribution of X _n+1 for the past state is only a function of X _n , then

P(X _n+1 =x|X ₁ =x ₁ , X ₂ =x ₂ ,...,X _n =x _n )=P(X _n+1 =x|X _n =x _n )

Here x is a state in the process. The above identity can be seen as a Markov property.

The position syllable type information of the Thai character in the split syllable has been determined. Therefore, for the set position syllable type information, the corresponding splitting probability is fixed, including: if the position syllable type information of the Thai character is not in the syllable At the end of the consonant character, the segmentation probability corresponding to the to-be-segmented syllable to be sliced by the Thai character arranged in the first order is zero; if the position syllable type information of the Thai character is the element not at the beginning of the syllable In the case of a phonetic character, the segmentation probability corresponding to the segmentation syllable to be segmented by the second order and the Thai character to be sliced is zero.

For example, a syllable to be divided into two consecutive characters of Thai characters, a syllable to be cut for two consecutive Thai characters, and a syllable to be cut for three consecutive Thai characters, for example, if the Thai character C is not at the end of the syllable For consonant characters, the values of P _CB , P _CCB , P _CBC , P _CCCB , P _CCBC , and P _CBCC are all zero. That is, in the first order, C is in front of B. If the Thai character C is a vowel character that is not at the beginning of the syllable, the values of P _BC , P _BCC , P _CBC , P _CCBC , P _{BCCC ,} and P _CBCC are all zero. That is, in the second order, B is before C.

For the syllables of Thai characters containing information on the syllable type of other positions, the number of occurrences of each Thai character before the first segmentation mark in the pending Thai text is counted, and each Thai character appears in the pending Thai text. The total number of times, using the Markov chain probability speech model described above, can determine the segmentation probability of each syllable to be segmented.

Thus, for the statistics of the syllables to be segmented and the probability of segmentation in the Thai document in the Thai corpus, the segmentation model of the syllable and the corresponding segmentation probability can be determined.

Step 105: Splitting the syllables in the pending Thai sentence according to each syllable to be sliced and its corresponding segmentation probability.

The segmentation model of the syllable and the corresponding segmentation probability has been determined. Therefore, after the input of the pending Thai sentence is input, the setting of the pending Thai sentence can be determined according to each syllable to be sliced and its corresponding segmentation probability. The pre-processed segmentation probability of each pre-processed syllable is then segmented according to the size of the pre-processed segmentation probability.

Specifically, it may be matched in a segmentation model in which the syllables and the corresponding segmentation probabilities have been determined, and it is determined that each of the pre-processing syllables and their corresponding pre-processing segmentation probabilities in the pending Thai sentence are set, and then compared. The size of the pre-processed segmentation probability can be segmented according to the maximum set of segmentation probabilities for the syllables in the set Thai sentence. Alternatively, the syllables in the set Thai sentence are segmented according to the pre-processing splitting probability greater than the set value.

It can be seen that, in the embodiment of the present invention, the Markov chain probability speech model can be used to determine the segmentation probability of each syllable by using the position syllable type information of the Thai character, and the syllables in the Thai sentence are segmented according to the segmentation probability. , provides another way to segment the Thai syllables. And the segmentation method is based on the basic grammar rules of Thai language. The n-gram grammar of Markov chain probability speech model is used to carry out probability statistics, which improves the accuracy and segmentation speed of Thai syllable segmentation.

The operation flow is gathered into a specific embodiment to exemplify the method provided by the embodiment of the present disclosure.

In this embodiment, the Thai character of one consecutive syllable, the Thai character of two consecutive times, and the Thai character of three consecutive times are respectively corresponding to the one-gram grammar and the binary of the Markov chain probability speech model. Grammar, and ternary grammar.

FIG. 2 is a flowchart of a Thai syllable segmentation method according to an exemplary embodiment. As shown in Figure 2, the Thai syllable segmentation process includes:

Step 201: Identify a non-Thai character string in the Thai text to be processed obtained from the Thai corpus to determine a non-Thai language string.

Step 202: Determine position syllable type information of each Thai character in the Thai text to be processed according to the correspondence between the saved Thai characters and the position syllable type information.

Here, the position syllable type information is generated according to the basic grammar rules of the Thai language, including syllable type information and position information. The correspondence between the saved Thai characters and the position syllable type information can be as shown in Table 1.

Step 203: Label the boundary between two non-Thai syllabic characters in the Thai text to be processed as the first segmentation identifier, the boundary between the two Thai syllable characters, and a Thai syllable character with a non-Thai language The boundaries between strings are marked as the identifier to be sliced.

Thus, the Thai text to be processed can be identified by C ₁ I ₁ C ₂ I ₂ ... C _i I _i ... CnIn, where C _i is the character in the Thai text to be processed, and I _i is the boundary between the characters. Where I _i =S, or I _i =B.

Step 204: Extract each tonal syllable in the Thai text to be processed.

Here, the extracted to-be-segmented syllable includes: a syllable to be sliced for one consecutive Thai character, a to-be-segmented syllable in which two Thai characters appear consecutively, and a to-be-segmented syllable in which three consecutive Thai characters appear consecutively. Corresponding to the nine representation types CB, BC, CCB, CBC, BCC, CCCB, CCBC, CBCC, BCCC, respectively, where C is the Thai character and B is the identifier to be sliced.

Step 205: Determine a segmentation probability of each syllable to be segmented by using a Markov chain probability speech model according to position syllable type information of the Thai character in the syllable to be segmented.

Table 2

For the set position syllable type information, the corresponding segmentation probability is fixed, wherein if the Thai character C is a consonant character not at the end of the syllable, P _CB , P _CCB , P _CBC , P _CCCB , P _CCBC , and P The value of _CBCC is 0. If the Thai character C is a vowel character that is not at the beginning of the syllable, the values of P _BC , P _BCC , P _CBC , P _CCBC , P _{BCCC ,} and P _CBCC are all zero.

For the syllables of Thai characters containing information on the syllable type of other positions, the number of occurrences of each Thai character before the first segmentation mark in the pending Thai text is counted, and each Thai character appears in the pending Thai text. The total number of times, using the Markov chain probability speech model described above, calculates the segmentation probability of each of the nine syllabic syllables corresponding to the nine types in Table 2.

According to Table 2, the syllables to be segmented and the segmentation probability in the Thai document in the Thai corpus are counted, and the segmentation model of the syllable and the corresponding segmentation probability can be determined.

Step 206: Determine, according to each syllable to be sliced and its corresponding segmentation probability, a pre-processing segmentation probability of each pre-processed syllable in the pending Thai sentence.

Step 207: Split the syllables in the set Thai sentence according to the size of the pre-processed segmentation probability.

For example, if CCBC has the highest pre-cutting probability, I can determine I _i = B=1, which is the syllable boundary here.

It can be seen that, in this embodiment, the Markov chain probabilistic speech model can be used to determine the segmentation probability of each syllable by using a one-gram grammar, a binary grammar, and a ternary grammar, and the syllables in the Thai sentence are determined according to the segmentation probability. The segmentation improves the accuracy and segmentation speed of the Thai syllable segmentation.

The following is an apparatus embodiment of the present disclosure, which may be used to implement the method embodiments of the present disclosure.

According to the above-mentioned process of segmentation of the Thai syllables, a device for segmentation of Thai syllables can be constructed.

FIG. 3 is a block diagram of a Thai syllabary segmentation device according to an exemplary embodiment. As shown in FIG. 3, the apparatus includes: a pre-processing unit 100, an identification unit 200, an extraction unit 200, a probability determination unit 400, and a segmentation unit 500, where

The pre-processing unit 100 is configured to preprocess the Thai text to be processed obtained from the Thai corpus, determine a non-Thai character string, and position syllable type information of each Thai character.

The identifying unit 200 is configured to perform labeling on a boundary between each character in the Thai text to be processed, wherein a boundary composed of at least one Thai syllable character is marked as a to-be-segmented identifier.

The extracting unit 300 is configured to extract each to-be-segmented syllable in the Thai text to be processed, wherein the to-be-segmented syllable is composed of Thai characters that appear n times consecutively, and a to-be-divided identifier, where n is a positive integer.

The probability determining unit 400 is configured to determine a segmentation probability of each syllable to be segmented by using a Markov chain probability speech model according to the position syllable type information of the Thai character in the syllable to be segmented.

The segmentation unit 500 is configured to slice and set the syllables in the Thai sentence to be processed according to each syllable to be sliced and its corresponding segmentation probability.

In an embodiment of the present invention, the pre-processing unit 100 is further configured to identify a non-Thai character string to be processed in the Thai text; and determine the Thai language to be processed according to the correspondence between the saved Thai character and the location syllable type information. The position syllable type information of each Thai character in the text, wherein the position syllable type information is generated according to the basic grammatical rules of the Thai language, including syllable type information and position information.

In an embodiment of the present invention, the identifier unit 200 is further configured to mark a boundary between two non-Thai phonological characters as a first segmentation identifier; and, to mark a boundary between two Thai syllable characters as The segmentation identifier is used to mark the boundary between a Thai syllable character and a non-Thai character string as the identifier to be sliced.

In an embodiment of the present invention, the probability determining unit 400 is further configured to: if the position syllable type information of the Thai character is a consonant character not at the end of the syllable, include the Thai character arranged in the first order and the to-be-cut identifier to be cut. The segmentation probability corresponding to the partial syllable is zero; if the position syllable type information of the Thai character is a vowel character that is not at the beginning of the syllable, the to-be-segmented yoke and the Thai character to be sliced are arranged in the second order. The corresponding segmentation probability is zero.

In an embodiment of the present invention, the segmentation unit 500 is further configured to determine, according to each syllable to be sliced and its corresponding segmentation probability, a pre-processing segmentation probability of each pre-processed syllable in the pending Thai sentence; According to the size of the pre-processing splitting probability, the syllables in the set Thai sentence are divided.

The apparatus provided by the embodiment of the present disclosure is exemplified below.

FIG. 4 is a block diagram of a Thai syllable segmentation apparatus according to an exemplary embodiment. As shown in FIG. 4, the apparatus includes: a pre-processing unit 100, an identification unit 200, an extraction unit 200, a probability determination unit 400, and a segmentation unit 500. A storage unit 600 can also be included.

The storage unit 600 stores the correspondence between the Thai characters and the position syllable type information as shown in Table 1. In this way, the pre-processing unit 100 can identify the non-Thai character string in the Thai text to be processed, determine the non-Thai character string, and determine the pending relationship according to the correspondence between the Thai character and the position syllable type information saved by the storage unit 600. The position syllable type information of each Thai character in the Thai text.

The identification unit 200 can mark the boundary between two non-Thai syllabic characters in the Thai text to be processed as the first segmentation identifier, the boundary between the two Thai syllable characters, and one Thai syllable character and one The boundary between the non-Thai strings is marked as the identifier to be segmented, so that the Thai text to be processed can be marked by C ₁ I ₁ C ₂ I ₂ ... C _i I _i ... CnIn, where C _i is the Thai text to be processed The character in , I _i is the boundary between the characters. Where I _i =S, or I _i =B. S is the first segmentation identifier, and B is the identifier to be sliced.

Then, the extracting unit 300 extracts each syllable to be sliced in the Thai text to be processed. Thus, the extracted syllables to be sliced include: a syllable to be sliced with 1 Thai character in succession, a syllable to be sliced with 2 Thai characters in succession, and a syllable to be sliced with 3 consecutive Thai characters. Corresponding to the nine representation types CB, BC, CCB, CBC, BCC, CCCB, CCBC, CBCC, BCCC, respectively, where C is the Thai character and B is the identifier to be sliced.

If the Thai character C is a consonant character that is not at the end of the syllable, the probability determination unit 400 determines that the values of _CB , P _CCB , P _CBC , P _CCCB , P _CCBC , and P _CBCC are all zero. If the Thai character C is a vowel character that is not at the beginning of the syllable, the probability determining unit 400 determines that the values of P _BC , P _BCC , P _CBC , P _CCBC , P _{BCCC ,} and P _CBCC are all zero. For the to-be-segmented syllables of the Thai characters including the other position syllable type information, the probability determining unit 400 may perform probability statistics of the Markov chain probabilistic speech model, and determine the cut of each of the nine types of the syllables to be cut out in the above Table 2. Probability.

Therefore, the segmentation module 500 can determine, according to each syllable to be sliced and its corresponding segmentation probability, a pre-processing segmentation probability of each pre-processed syllable in the pending Thai sentence, and then, according to the pre-processing segmentation probability The size of the syllable in the set Thai sentence is divided.

It can be seen that, in this embodiment, the Markov chain probability speech model can be used to determine the segmentation probability of each syllable by using the position syllable type information of the Thai character, and the syllables in the Thai sentence are segmented according to the segmentation probability. Another way to segment the Thai syllables is provided. And the segmentation method is based on the basic grammar rules of Thai language. The n-gram grammar of Markov chain probability speech model is used to carry out probability statistics, which improves the accuracy and segmentation speed of Thai syllable segmentation.

Those skilled in the art will appreciate that embodiments of the present invention can be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or a combination of software and hardware. Moreover, the invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) including computer usable program code.

The present invention has been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (system), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions specified in one or more of the flow or in a block or blocks of the flow chart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.

It is to be understood that the invention is not to be construed as being limited to The scope of the invention is limited only by the appended claims.

Claims (10)

A method for segmentation of Thai syllables, characterized in that it comprises: Preprocessing the pending Thai text obtained from the Thai corpus to determine the non-Thai character string and the location syllable type information of each Thai character; Labeling a boundary between each character in the Thai text to be processed, wherein a boundary composed of at least one Thai syllable character is marked as a to-be-sliced identifier; Extracting each to-be-segmented syllable in the Thai text to be processed, wherein the to-be-segmented syllable is composed of Thai characters that appear n times consecutively, and one of the to-be-segmented identifiers, and n is a positive integer; Determining, according to the position syllable type information of the Thai character in the syllable to be segmented, using a Markov chain probability speech model to determine a singular probability of each syllable to be sliced; The syllables in the pending Thai sentence are set according to each syllable to be sliced and its corresponding segmentation probability. The method according to claim 1, wherein said preprocessing the Thai text to be processed, determining the non-Thai character string, and the position syllable type information of each Thai character include: Identifying a non-Thai character string in the Thai text to be processed; Determining position syllable type information of each Thai character in the to-be-processed Thai text according to a correspondence between the saved Thai character and the position syllable type information, wherein the position syllable type information is generated according to the basic grammatical rules of the Thai language , including syllable type information and location information. The method of claim 1 wherein said tagging a boundary between each character in said Thai text to be processed comprises: Mark the boundary between two non-Thai syllable characters as the first segmentation identifier; Marking the boundary between two Thai syllable characters as the to-be-divided identifier; A boundary between a Thai syllable character and a non-Thai character string is marked as the to-be-sliced identifier. The method according to claim 1 or 2, wherein said Markov chain model is used to determine the segmentation of each syllable to be sliced according to the position syllable type information of the Thai character in the syllable to be sliced Probabilities include: If the position syllable type information of the Thai character is a consonant character that is not at the end of the syllable, the cleavage probability corresponding to the syllable to be sliced by the Thai character arranged in the first order is zero; If the position syllable type information of the Thai character is a vowel character that is not at the beginning of the syllable, the segmentation identifier corresponding to the to-be-segmented syllable of the Thai character is arranged by the second order The probability is zero. The method according to claim 1, wherein the segmentation of the syllables in the pending Thai sentence according to each syllable to be sliced and its corresponding segmentation probability comprises: Determining a pre-processing segmentation probability of each pre-processed syllable in the set Thai sentence according to each to-be-segmented syllable and its corresponding segmentation probability; The syllables in the set Thai sentence are divided according to the size of the pre-processed segmentation probability. A device for segmentation of Thai syllables, characterized in that it comprises: a preprocessing unit for preprocessing the pending Thai text obtained from the Thai corpus, determining a non-Thai character string, and location syllable type information of each Thai character; An identifier unit, configured to label a boundary between each character in the Thai text to be processed, wherein a boundary composed of at least one Thai syllable character is marked as a to-be-sliced identifier; An extracting unit, configured to extract each to-be-segmented syllable in the Thai text to be processed, wherein the to-be-segmented syllable is composed of Thai characters that appear n times consecutively, and one of the to-be-segmented identifiers, n is Positive integer a probability determining unit, configured to determine, according to the position syllable type information of the Thai character in the syllable to be sliced, a Markov chain probabilistic speech model to determine a puncturing probability of each syllable to be sliced; A segmentation unit is configured to slice the syllables in the pending Thai sentence according to each syllable to be sliced and its corresponding segmentation probability. The device of claim 6 wherein: The preprocessing unit is further configured to: identify a non-Thai character string in the Thai text to be processed; and determine the to-be-processed Thai text according to a correspondence between the saved Thai character and the location syllable type information. The position syllable type information of each of the Thai characters, wherein the position syllable type information is generated according to the basic grammatical rules of the Thai language, including syllable type information and position information. The device of claim 6 wherein: The identifier unit is further configured to mark a boundary between two non-Thai phonological characters as a first segmentation identifier; and, to mark a boundary between two Thai syllable characters as the to-be-segmented identifier, A boundary between a Thai syllable character and a non-Thai character string is marked as the to-be-sliced identifier. A device according to claim 6 or claim 7 wherein: The probability determining unit is further configured to: if the position syllable type information of the Thai character is a consonant character not at the end of the syllable, including the Thai character arranged in the first order and the to-be-segmented syllable to be sliced Corresponding segmentation probability is zero; if the location syllable type information of the Thai character is a vowel character not at the beginning of the syllable, the to-be-segmented identifier and the Thai character arranged by the second order are included The segmentation probability corresponding to the segmented syllable is zero. The device of claim 6 wherein: The segmentation unit is further configured to determine, according to each to-be-segmented syllable and its corresponding segmentation probability, a pre-processing segmentation probability of each pre-processed syllable in the set Thai sentence; The size of the segmentation probability is processed, and the syllables in the set Thai sentence are divided.

Images