CN1570923A - Sentence boundary identification method in spoken language dialogue - Google Patents

Abstract

A sentence boundary segmentation method based on a bidirectional N-gram model and a Maximum Entrpy model includes two processes of training and segmentation. The training process includes the steps of: obtaining a spoken language corpus; performing pretreatments such as substitution on the spoken language corpus; counting n -N-element co-occurrence frequency of the gram model; estimate n-element forward dependency probability and n-element reverse dependency probability; obtain n-element positive and reverse dependency probability database; set the characteristic function of the Maximum Entropy model; cyclically calculate the characteristic function parameters; obtain A database of characteristic function parameters. The sentence boundary segmentation method based on the bidirectional n-gram model and the Maximum Entropy model is a pure statistical method, and its implementation only requires a background spoken language corpus, and the corpus does not need any deep segmentation or labeling. This method is not limited by language, and can be applied to sentence boundary segmentation in any language by changing the training corpus.

Description

Sentence Boundary Recognition Method in Spoken Conversation

technical field

The invention relates to speech recognition, in particular to a boundary recognition method for spoken sentences.

Background technique

With the rapid development of computer hardware conditions and the continuous improvement of speech recognition technology, language understanding and generation systems using speech as the interface (hereinafter referred to as speech-language combined systems) such as man-machine interface, man-machine dialogue system, simultaneous translation system, etc. Started to be practical. These systems have broad application prospects. For example, the man-machine voice interface, its perfection will make people no longer worry about learning cumbersome computer operations, because you only need to "speak" to the computer to listen to anything, and it will execute according to your requirements. Another example is simultaneous translation technology, which will eliminate communication barriers between users of different languages, which will greatly facilitate people's cross-border travel, and allow members from different countries to communicate in large international events (Olympic Games, Asian Games, etc.) Free communication becomes possible. In the military, the voice-language joint system also has important applications. The United States has begun to develop simultaneous translators for soldiers to facilitate their understanding of the situation from local residents when fighting in foreign lands. In addition, telephone eavesdropping has always been an effective means of obtaining military intelligence, and obtaining effective information from a large amount of voice information has always been completely dependent on manual work. If the machine can automatically obtain information, it will greatly improve efficiency and save manpower.

As can be seen from Figure 1, the speech-language joint system generally consists of three modules: a speech recognition module, a sentence boundary segmentation module, and a language analysis and generation module. Since the result of speech recognition is a continuous text without any punctuation, the next step of analysis, conversion and generation must be segmented, that is, the continuous text is divided into sentences, and the sentence boundary segmentation module is used in this way A function, it is in the middle of the speech recognition module and the language analysis and generation module, and it is a bridge connecting them. Speech recognition technology and language analysis and generation technology have always been research hotspots in the field of computer science, but sentence boundary segmentation technology has not received widespread attention before the initial practical application of speech-language joint systems. Nowadays, with the continuous expansion of the application of speech-language joint system, the sentence boundary segmentation technology, as one of the core technologies supporting this joint application, has been paid more and more attention.

Contents of the invention

The purpose of the present invention is to provide a sentence boundary recognition method in oral conversation, which solves the problem of converting continuous text after speech recognition into sentences that can be processed by subsequent analysis modules.

In order to achieve the above object, a sentence boundary recognition method in a spoken conversation includes two processes of training and segmentation, and the training process includes steps:

Obtain a spoken language corpus;

Perform preprocessing such as substitution on the spoken language corpus;

Statistical n-gram co-occurrence frequency of n-gram model;

Estimate the probability of n-ary forward dependence and the probability of n-ary reverse dependence;

Obtain the n-element positive and negative dependency probability database;

Set the characteristic function of the Maximum Entropy model;

Cyclic calculation of characteristic function parameters;

Obtain a database of characteristic function parameters.

The sentence boundary recognition method in oral conversation is a purely statistical method, and its implementation only requires a background spoken language corpus, and the corpus does not need any deep segmentation or labeling. This method is not limited by language, and can be applied to sentence boundary segmentation in any language by changing the training corpus.

Description of drawings

Figure 1 is a general schema of a speech speech application system.

Detailed ways

Various details involved in the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings.

Preprocessing of spoken corpus

The acquired oral corpus cannot be directly used for training, but must undergo some preprocessing. Sentence boundary segmentation is to find the end point of the sentence in the continuous text, that is, to predict the occurrence position of those sentence-end punctuation, so as long as it is the end-sentence punctuation, there is no difference for segmentation. The main task of preprocessing is to replace the end-of-sentence punctuation in the corpus with a unified symbol. For the convenience of description, the replacement symbol in this article is represented by "SB"; for other punctuation other than the end-of-sentence punctuation, it must be deleted, because the phonetic It is impossible for the recognized text to contain such punctuation marks. For Chinese, this work is very easy, just replace the end-of-sentence punctuation such as period, question mark, and exclamation mark with the same symbol, and then delete the non-end-of-sentence punctuation such as comma, colon, and quotation marks. However, the punctuation in some languages is ambiguous, such as the period "." in English, which is also used to indicate abbreviations, such as "Mr.", "Dr.", etc. At this time, we must first replace these abbreviations with " ".", and then replace "." with a unified symbol.

Statistics of N-gram Co-occurrence Frequency and Estimation of N-gram Dependency Probability

The statistics of N-gram co-occurrence probability are based on the preprocessed spoken language corpus. First of all, we need to count a primitive vocabulary. For Chinese, this vocabulary is all the words and "SB" that appear in the corpus. For English, this vocabulary includes all the words and abbreviations that appear in the corpus. and "SB". On the basis of the N-tuple frequency calculated from the corpus, we use the Modified Kneser-Ney Smoothing algorithm to estimate the dependency probability of the N-tuple combinations of all entries in the dictionary. Modified Kneser-Ney Smoothing gives different degrees of reduction to N-tuples with different occurrences to compensate those N-tuples with zero occurrences. This smoothing method has been evaluated by Stanley F. Chen et al., and its performance exceeds other smoothing methods.

Maximum Entropy model feature function setting and parameter training

The Maximum Entropy model is a statistical model used to estimate the joint probability. Its central idea is to maximize the entropy of joint events, that is, the uncertainty, under the condition of satisfying the constraints of the training corpus. In natural language processing, the joint probability is generally expressed as: P(b, c), b represents the possible situation, and c represents the context. In the sentence boundary segmentation method described in this paper, b is set as a Boolean variable, which is true to indicate that the judgment position is a sentence boundary, and false to indicate that the judgment position is not a sentence boundary. The corresponding eigenfunctions appear in groups, as follows:

It can be seen from the above formula that each set of feature functions corresponds to a S _j , and S _j represents a word group (Chinese) or phrase (English) of a certain length. In this method, S _j is set to appear in the training corpus All triples, binary and unary of . The prefix(c) and suffix(c) in the formula respectively represent the set of all prefixes and suffixes for judging the position, for example, such as the sentence "Please <1> tomorrow <2> day <3> and then <4> come to <5>", for position <3>, the prefix set is {day, tomorrow, please tomorrow}, and the suffix set is {again, come again}; include(prefix(c), S _j ) indicates that S _j belongs to prefix(c) . Each feature function has a weight corresponding to it, and the weight indicates the degree of influence of the feature to which the feature function belongs to the result. In this method, the weights also appear in groups, expressed as α _j10 , α _j11 , α _j20 , α _j21 , and these weights are calculated by the Generalized Iterative Scaling algorithm and stored in the maximum entropy parameter database. For a certain context, the probability of a certain situation is calculated as follows:

$\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&pi;\&Pi;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 10</span>}}^{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 10</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 11</span>}}^{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 11</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 20</span>}}^{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 20</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}^{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span>$

k is the group number of the set characteristic function, π is the normalized variable, in this example its value is:

        π = P(c, 0) + P(c, 1)

In particular, sometimes we only want to consider the joint probability of the left context or the right context and a certain situation. At this time, the calculation formulas are:

$\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; left</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&pi;\&Pi;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 10</span>}}^{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 10</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 11</span>}}^{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 11</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span>$

$\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; right</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&pi;\&Pi;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 20</span>}}^{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 20</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}^{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span>$

Sentence boundary segmentation method based on bidirectional n-gram model and Maximum Entropy model

For a given continuous text "W ₁ <1>W ₂ <2>...<n-1>W _n ", where W _i (1 i n-1) represents a primitive, sentence boundary segmentation is the judgment Whether the n-1 positions shown are sentence boundaries. Use P _is (i) to represent the probability that position i is a sentence boundary, and use P _no (i) to represent the probability that position i is not a sentence boundary, then position i is judged to be a sentence boundary if and only if P _is (i)>P _no (i).

In this method, P _is (i) and P _no (i) are composed of four parts: forward n-gram probability, reverse n-gram probability, maximum entropy forward correction probability and maximum entropy reverse correction probability. It is described by the formula as follows:

P _is (i)＝W _{n_is} (C _i )×P _is (i|NN)×W _{r_is} (C _i )×P _is (i|RN)

P _no (i)＝W _{n_no} (C _i )×P _no (i|NN)×W _{r_no} (C _i )×P _no (i|RN)

Among them, P _is (i/NN), P _no (i/NN) and P _is (i/RN), P _no (i/RN) represent positive and reverse n-gram probabilities respectively, W _{n_is} (C _i ), W _{n_no} (C _i ) and W _{r_is} (C _i ), W _{r_no} (C _i ) represent the weighted values of the forward and reverse n-gram probabilities respectively. Below we describe the calculation methods of the above items respectively.

Forward n-gram segmentation probability

Forward n-gram models treat text as a left-to-right Markov sequence. We use W ₁ W ₂ ...W _m (m is a natural number) to represent an input text sequence, W _i (1≤i≤m) represents a primitive, and according to the Markov property, the probability of a certain primitive appearing is only It is related to the n-1 primitives on its left, that is, P(W _m |W ₁ W ₂ ...W _m-1 )=P(W _m |W _m-n+1 ...W _m-1 ) . According to the conditional probability formula, the probability of word sequence occurrence can be written as: P(W ₁ W ₂ ...W _m )=P(W ₁ W ₂ ...W _m-1 )×P(W _m |W ₁ W ₂ ...W _m-1 ), combined we get:

P(W ₁ W ₂ ...W _m )＝P(W ₁ W ₂ ...W _m-1 )×P(W _m |W _m-n+1 ...W _m-1 )

Add the symbol "SB" representing the boundary of a sentence into the character sequence, and judge whether the position i is a boundary of a sentence, that is, calculate P(W ₁ W ₂ ...W _i SBW _i+1 ) (that is, P _is (i| NN)) and P(W ₁ W ₂ ... W _i W _i+1 ) (that is, the size of P _no (i|NN)). Taking the 3-gram model as an example, considering that there are two situations for position i-1, one is that it is a sentence boundary, and the other is that it is not a sentence boundary, calculate P(W ₁ W ₂ ...W _i SBW _i+1 ) and P(W ₁ W ₂ ...W _i W _i+1 ) the overlapping band formula is:

P(W ₁ W ₂ ...W _i SBW _i+1 )＝P(W ₁ W ₂ ...SBW _i ×P(SB|SBW _i ×P(W _i+1 |W _i SB)+P( W ₁ W ₂ ...W _i-1 W _i )

×P(SB|W _i-1 W _i )×P(W _i+1 |W _i SB)

P(W ₁ W ₂ ...W _i W _i+1 )＝P(W ₁ W ₂ ...W _i-1 SBW _i )×P(W _i+1 |SBW _i )+P(W ₁ W ₂ ... W _i-1 W _i )

×P(W _i+1 |W _i-1 W _i )

Assuming that the position number on the left of W ₁ is 0, then the initial value of the iterative band is:

P _is (0|NN)＝1

P _n0 (0|NN)＝0

Reverse n-gram segmentation probability

The reverse n-gram model is similar to the forward n-gram model, except that it regards the character sequence W ₁ W ₂ ... W _m as a Markov sequence from right to left, that is, the order in which they appear is regarded as are W _m W _m-1 . . . W ₁ . Similarly, through the conditional probability formula and the Markov property we get:

P(W _m W _m-1 ...W ₁ )＝P(W _m W _m-1 ...W ₂ )×P(W ₁ |W _n W _n-1 ...W ₂ )

W _i is the end point of a sentence if and only if P(W _m W _m-1 ...W _i+1 SBW _i )＞P(W _m W _m-1 ...W _i+1 W _i ) .

Similarly, we give iterative calculations P(W _m W _m-1 ...W _i+1 SBW _i ) (ie P _is (i+1|RN)) and P(W _m W _m-1 ... W _i+1 W _i ) is the formula (3-gram) of (P _no (i+1|RN)):

P(W _m W _m-1 ...W _i+1 SBW _i )＝P(W _m W _m-1 ...W _i+2 SBW _i+1 )×P(SB|SBW _i+1 )× P(W _i |W _i+1 SB)+P(W _m W _m-1 ...W _i+2 W _i+1 )

×P(SB|W _i+2 W _i+1 )×P(W _i |W _i+1 SB) with

P(W _m W _m-1 ...W _i+1 W _i )＝P(W _m W _m-1 ...W _i+2 SBW _i+1 )×P(W _i |SBW _i+1 ) +P(W _m W _m-1 ...W _i+2 W _i+1 )

×P(W _i |W _i+2 W _i+1 )

And initial conditions: P(SBW _m )=1, P(W _m )=0.

The reverse n-gram model stacks from right to left to calculate the probability that each position is a sentence boundary, which can avoid some errors in the forward model, such as the following sentence "Xiao Zhang was ill for a week", if the forward cut It is likely to output the following result "Xiao Zhang was sick with SB for a week", because searching from left to right, "Xiao Zhang is sick" is a complete sentence; and if reverse segmentation is used, searching from right to left, We obviously don't consider "a week" to be a complete sentence, so the search continues to the right until the true sentence boundary.

maximum entropy correction weight

Through the above description, based on the beneficial supplement of the reverse n-gram segmentation to the forward n-gram, we consider combining the positive and reverse n-gram probability weights, and the determination of the weight depends on the maximum entropy described in this method The parameters of the model.

As mentioned above, W _{n_is} (C _i ), W _{n_no} (C _i ), represents the weighting of the forward n-gram probability, and its calculation is equivalent to P(c_left, 1) and P(c_left, 0), as follows Show:

${\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; is</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&pi;\&Pi;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 11</span>}}^{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 11</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}$

${\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&pi;\&Pi;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 10</span>}}^{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 10</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}$

W _{r_is} (C _i ), W _{r_no} (C _i ) respectively represent the weighting of the forward n-gram probability, and their calculations are equivalent to P(c_right, 1) and P(c_right, 0), as follows:

${\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; is</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&pi;\&Pi;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}^{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}$

${\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&pi;\&Pi;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 20</span>}}^{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; 20</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}$

In order to verify the segmentation performance of this method, we used the collected Chinese and English spoken corpora to conduct a sentence boundary segmentation experiment, and compared it with the language model (forward n-gram model) in the reference. The training corpus, test corpus, and segmentation results are shown below. It should be noted that the accuracy rate in the test results is the ratio of the number of correct segmentations to the total number of segmentations, and the recall rate is the ratio of the number of correct segmentations to the number of original sentences. Ratio, while F-Score is a comprehensive measure of accuracy and recall, and its calculation formula is:

Table 1. Details of the training corpus language size number of sentences average sentence length Chinese 4.02MB 148967 8 words English 4.49MB 149311 6 words

Table 2. Details of the test corpus language size number of sentences average sentence length Chinese 412KB 12032 10 characters English 391KB 10518 7 words

Table 3. Results of Chinese segmentation test method Accuracy recall rate F-Score Literature[1] 79.4% 84.5% 81.9% This method 86.7% 86.0% 86.3%

Table 4. English Segmentation Test Results method Accuracy recall rate F-Score Literature[1] 73.4% 83.0% 77.9% This method 78.8% 84.9% 81.7%

It can be seen from the test results that the sentence boundary segmentation method based on the two-way n-gram model and the maximumentropy model we proposed is significantly better than the method based solely on the forward n-gram model used in the literature [1]. This is Because our method comprehensively considers the impact of forward and reverse searches on the segmentation results when judging whether a certain position is a sentence boundary, and reasonably adjusts the forward and reverse probabilities through the maximum entropy parameter.

Claims (3)

1. a sentence boundary recognition method in spoken conversation, comprise training and two processes of segmentation, described training process comprises steps: Obtain a spoken language corpus; Perform preprocessing such as substitution on the spoken language corpus; Statistical n-gram co-occurrence frequency of n-gram model; Estimate the probability of n-ary forward dependence and the probability of n-ary reverse dependence; Obtain the n-element positive and negative dependency probability database; Set the characteristic function of the Maximum Entropy model; Cyclic calculation of characteristic function parameters; Obtain a database of characteristic function parameters. 2. by the described method of claim 1, it is characterized in that the algorithm of described cyclic calculation feature function parameter is Generalized Iterative Scaling algorithm. 3. by the described method of claim 1, it is characterized in that described segmentation process comprises steps: Segment the text with a segmentation method based on the forward n-gram model; Segment the text with a segmentation method based on the reverse n-gram model; The context of the segmentation point is extracted, and the parameters of the characteristic function of the Maximum Entropy model are used to carry out weighted synthesis of the forward and reverse segmentation results.