CN1460948A - Method and device for amending or improving words application - Google Patents

Abstract

A database (3) is provided containing associations between words and possible values associated therewith, which provides a measure of the possible values of such correct or customary associations. Possible values are based on the frequency of associations obtained by analyzing large volumes of text, such as text authored by native speakers. In order to check a text segment for possible wrong or unnatural usages of one or more words of the text segment, the text is first analyzed (11) to determine the associations between its words. The likelihood of associations in the analyzed text is determined from a database (3). Calculate (14) the plausibility value of each word in the analyzed text, which is obtained by combining the possible values of the associations where the word appears. Another database (4) is used to index terms, which contains collections of terms that are easily confused by indexed terms. Select (13, 16) each confusing term in turn and replace the index term in its association. The possible values of these new associations are determined and the plausibility value of (14) the confusing word is calculated. In one error detection embodiment, confusable words are tried (23, 24) for those words whose plausibility falls below a threshold, and the confusable words with increasing plausibility are reported (25, 26) to the user. In a context-sensitive thesaurus embodiment, confusable words may be tried for all words and those with a plausibility value exceeding a second threshold may be reported.

Description

Method and apparatus for modifying or improving the use of words

technical field

The present invention relates to a method and apparatus for modifying or improving the selection and use of words in natural language texts. The invention also relates to a computer program for programming a computer to perform such a method, a storage medium containing such a program and a computer programmed with such a program.

Background technique

At the heart of writing or speaking in language is the choice of words to use. To help people make choices, native-speaking authors use dictionaries, and language learners generally use bilingual dictionaries. However, native-speaking authors find that dictionaries cannot give such detailed information about which synonyms are appropriate in the context, and beginners may choose the wrong translation from bilingual dictionaries or misspell one word for another due to lack of comprehensive ability or knowledge. a word.

In the Annotated Corpus of English Learners (Nichols, 1999 "The Cambridge Learner Corpus-Error Coding and Analysis for Writing Dictionaries and other books for English Learners", Summer Workshop on the Learner Corpus, Cambridge University Press), the verb or Misuse of prepositions is the second most common type of error after spelling and punctuation errors. For example, an author might have used "associate to" instead of "associate with," "loose one's temper" instead of "lose one's temper," or "wins me at tennis" instead of "beats me at tennis."

The present invention makes it possible to detect these and other types of errors and to propose corrections for them. It can handle real misspellings of words (like lose/loose), and it can also handle other different types of errors.

Look up a word like "make" in a dictionary and the author will find tons of synonyms. These synonyms can be classified into clusters that share a central meaning. A cluster may include synonyms such as "create", "construct", and "establish", but the author cannot find words such as "creates a diversion", "constructs a model", or "establishes a relationship".

The present invention makes it possible to provide these synonyms as suggestions in response to input such as "make a diversion", "make a model" or "make a relationship".

The present invention exploits the correlation or association constituted by the relationship between two words or phrases that occur simultaneously (not necessarily adjacently) in the language of a piece of writing or narration hereinafter referred to as text. An association may be related to a strength or similarity measured based on its frequency of occurrence in a large body of text. A word in a text may be associated with a plausibility value based on the possible values associated with that word. Words that make no sense in the text would be wrong or unnatural in the context.

US Patents 4,916,614, 4,942,526, 5,406,480 disclose the creation and use of co-occurrence information in syntactic analysis and translation.

The techniques disclosed in each of U.S. Pat. to" and "too". The presence of such words in the text indicates a potential error. The patents go on to describe different methods of correcting the errors.

US Patent 4,674,065 discloses a technique using a system of rules describing different contexts for distinguishing confusing word usage.

US Patents 4,868,750, 5,537,317 and 5,799,269 disclose systems for assigning probability values to part-of-speech sequences. The probability of a sequence containing a confusable word can be compared to the probability of a sequence containing the word it was confused into. If the latter is greater than the former, a possible error will be reported.

U.S. Patent No. 5,258,909 discloses a system that assigns a probability value to a sequence of words, assigns a probability value to the occurrence of one word being misspelled as another word, and combines these probabilities to determine whether a word is misspelled as another word. one word.

US Patents 5,659,771 and 5,907,839 disclose a system that associates words with features representing their context and uses a machine learning algorithm to compute a function from the feature values of a particular member of a confusing set. When a member of a confusing set appears in the text, use this function to classify it as correct or incorrect.

"An unsupervised method for detecting grammatical errors" by Jodorow and Rycock (pp. 140-147 in Proceedings of the first annual meeting of the North American Chapter of the Society for Computational Linguistics in 2002) discloses the use of consecutive words n A system for detecting errors in metasyntax models. The system can detect class changes and class preservation errors that have not been seen before, but because of the continuous model, it can only cover a very limited length. Modifications to bugs are not discussed.

US Patent No. 5,999,896 discloses a system that identifies possible errors in the use of words through the failure of a parser and resolves these problems by finding those confusing words that would allow subsequent parsing to succeed.

Contents of the invention

According to a first aspect of the present invention, a method of modifying or improving the selection of a first word or phrase in a written or spoken text passage containing a set of words in a first language, comprising the steps of:

(a) providing a first database of associations between words or phrases in the first language, wherein each association has at least one associated likelihood value based on the frequency of occurrence of the association in text in the first language;

(b) analyzing the text segment to determine a first association between said first word or phrase and a second word or phrase in the text segment, corresponding to at least a first possible value of said association and based on said at least a possible value corresponding to a first plausible value of said first word or phrase;

(c) prepare a second database in which each entry has at least one word or phrase associated with the set of words or phrases into which it can be confused;

(d) selecting or calculating a confusingly confusing word or phrase from the second database as a candidate alternative to the first word or phrase described in the text passage;

(e) deriving a second plausibility value of a confusing word or phrase based on a second possible value of the association in the first database between the confusing word or phrase and other words in the text segment or phrase composition; and

(f) selectively providing an indication of a confusing word or phrase based on the calculated plausibility value.

The possible value of each association in the first database may also be based on the frequency of occurrence of other associations each containing a word or phrase with the same correlation.

The possible value of each association in the first database may also be based on the frequency of occurrence of all other associations having the same correlation.

The possible value of each association in the first database consists of at least one of mutual information, T-value, Z-value, Yule's Q coefficient and log likelihood.

In step (e), the other word or phrase may be a second word or phrase, and the correlation relationship of the second association may be the same as that of the first association.

Step (b) may include establishing a set of first associations for a set of first words or phrases in the text segment and steps (d), (e) and (f) may be performed for each first association.

Step (b) may include establishing associations between non-adjacent words or phrases in the text.

Step (d) may include selecting each confusing word or phrase of a set of words or phrases and steps (e) and (f) may be performed for each confusing word or phrase.

Step (f) may include indicating the second plausible values in descending order of the second plausible values.

Steps (d), (e) and (f) may be performed if the first likelihood value is less than a first threshold.

Step (f) may comprise providing an indication when the or each second plausibility value exceeds a second threshold.

Step (f) may include providing an indication if the second plausible value is greater than the first plausible value.

Step (b) may include computing the first plausible value by means of a function learned by machine learning techniques from the annotated corpus of learner errors and their associated plausible values.

The method may include replacing the first word in the text segment with a confusing word.

The method may include generating the text segment by second language translation.

The method may include generating the text segment from a printed document by optical character recognition.

According to a second aspect of the invention there is provided a computer program for programming a computer to perform a method according to the first aspect of the invention.

According to a third aspect of the present invention, there is provided a storage medium containing a program according to the second aspect of the present invention.

The media may include computer readable media.

According to a fourth aspect of the present invention there is provided a computer comprising a program according to the second aspect of the present invention.

According to a fifth aspect of the present invention, there is provided an apparatus for modifying or improving the selection of a first word or phrase in a written or spoken text passage containing a set of words in a first language, comprising:

a first database of associations between words or phrases in the first language, wherein each association has at least one associated likelihood value based on the association's frequency of occurrence in the plurality of texts in the first language;

an analyzer for analyzing the text segment to establish a first association between said first word or phrase and a second word or phrase of the text segment, at least one first possible value corresponding to said association and a first plausible value based on said at least one possible value corresponds to said first word or phrase; and

a second database, each of which has at least one word or phrase associated with the set of words or phrases into which it can be confused;

means for selecting or calculating a confusingly confusing word or phrase from a second database as a candidate alternative to said first word or phrase in the text passage;

for deriving a second plausibility value of a confusing word or phrase based on the possible values in the first database of a second association between the confusing word or phrase and other words in the text segment or phrase composition; and

A tool for selectively providing an indication (25, 26) of a confusing word or phrase based on a calculated likelihood value.

By exploiting the possibility of association between words, it is possible to provide a technique that represents an improvement over known systems that use only the probabilities of part-of-speech sequences, since such known systems are unable to detect and modify very common class-preserving mistake.

Improvements will be achieved by using consecutive N-grams because dependency grammars can capture dependencies between words that are not adjacent but still directly affect the choice of other words. In principle the N-gram could be extended to cover such dependencies, but in practice this leads to several data sparsity problems. The use of associations will centralize the data used for the calculation of statistically possible values into units of linguistic meaning. Three-element correlation segments are almost always sufficient to obtain useful statistics, yet even four-element N-grams miss many possible or unlikely word combinations.

An important consequence of this restriction on linguistically meaningful entity statistics is that probability values are easier to interpret in a manner that requires error detection. To understand this, consider the implications of transition probabilities between adjacent words in a continuous bigram model. Within a composition, such as between "big" and "dog" in "a bigdog", transition probabilities can be directly compared to transition probabilities for similar sequences of adjectives and nouns. But the transition probabilities between "dog" and "a" in "give the dog a bone" are rather uninteresting (and unlikely) probabilities, which is a composition ending in "dog" followed by The probability of another component starting with "a". The probability of interest, i.e. the probability that a component beginning with "give" has a second object beginning with "bone", is not represented and cannot be compared with possible alternatives such as "give the dog a clone".

That is, in continuous N-gram models, low transition probabilities can indicate both linguistically interesting and linguistically uninterestingly unlikely. If a system based on successive N-grams used each low probability as a trigger for error handling, it would find a large number of possible "bugs", many of which are not real bugs. Dealing with these is expensive and there is a danger of classifying false errors as true errors.

This is why there is no known technique that uses low transition probabilities as a trigger for error handling, but rather uses the occurrence of a certain word in the text that is known to be confusing and then considers the relative possibility.

In contrast, "unlikely" is a more reliable error cue in this technique.

Any improbable association can cause error handling to start and only improbable associations can do so. Of course, unlikely correlations are not always the result of a mistake; however, in this technique, there will be far fewer of these false triggers.

Moreover, adding elements to a confusing set increases the chance of error handling being triggered when the presence of some elements in a confusing set is the only trigger for error handling (as in many known techniques). The number of times in turn increases the computational cost of evaluating each element.

When an associated likelihood and plausibility derived from a term are the triggers for error handling (as in the present invention), a wide range of error characteristics can be identified. Concepts that are easily confused are not limited to high frequency confusion of spelling and pronunciation.

In known techniques using learning algorithms and simultaneously using the presence of known confusing words as error handling triggers, there is no way to detect a word as a possible error other than applying the learning algorithm to distinguish it. Furthermore, like known N-gram-based techniques, learning systems do not fully benefit from aggregating data into units of linguistic meaning.

The present technique represents an improvement over known techniques based on parsing failures, since parsing failures are a very crude detection mechanism for word errors (especially those involving substitutions of words with the same part of speech). Conversely, the present technique provides very fine-grained quantitative measures of the likelihood of even very small sentence fragments, and includes grammatical analysis failures such as missing attachments hinted at as a particularly extreme case of improbability . Furthermore, the parsing success (coarse condition as an error has been corrected) can be replaced by a fine quantitative measure of gain improvement.

Description of drawings

The invention is further described by way of example and with reference to the accompanying drawings, which include:

Fig. 1 is the device block schematic diagram that constitutes an example of the present invention;

Figure 2 is a block diagram illustrating the correlation structure of the sentence "Love is the most important condition to marriage";

Figure 3 is a portion of the first database that associates possible values with associated ones; and

Figure 4 (comprising Figures 4a and 4b) shows an example of the invention as an error detector and modifier.

Detailed ways

Methods and apparatus are provided that detect errors and unnatural expressions in a user's writing and suggest ways in which usage of these languages can be improved. These techniques can also be used as context-sensitive dictionaries, which suggest expressions that have similar meanings to a given input expression in its context. Uses a statistical correlation model of word combinations as the basis for error detection and substitution checking. This solves several problems with known solutions, which either use continuous N-gram models, or use unanalyzed feature sets. Moreover, these techniques make it possible to provide a wider range of candidates for replacement. Detecting errors does not rely on the detection of specific words that are prone to misuse, so errors that have not been encountered before can be detected and corrected.

This method uses two types of relationships between terms. A relation type holds between two words at different positions in a single sentence. These are related relations, such as 'subject of', 'objectof' and 'modifier', and the example shown in Figure 2, which illustrates the results of the analysis of the sentence "Love is the most important condition for marriage". Words are represented by their prototypes and parts of speech, i.e., as terms, so "is" appears as "be_V". The subject of this verb is equivalent to "love_N" and its object is equivalent to "condition_N". The latter is defined by "the_DET" and decorated with "important_ADJ". "Most_ADV" is equivalent to an adverb that modifies "important_ADJ". "For_PREP" is equivalent to modifying the preposition of "condition_N", and "marriage_N" is equivalent to the object of the preposition "for_PREP". A triple is composed of two terms, and the relation linking them is called an association.

Another type of relationship includes a relationship that defines "possible alternatives", ie, relationships between alternatives for words at a given position in a sentence. Here are some examples of substitution relationships:

Dictionary relations, such as synonyms, antonyms, hyponyms, supernyms;

The misspellings that lead to other words of the language, such as "loose" to "lose", in which there is a special case of homonyms, speaking of words that sound the same but are spelled differently, such as "pane" and " pain";

Etymology, which refers to the different ways in which words are formed from the root of a word (such as "interested" and "interesting", or "safe" and "safety");

Interlingual confusion, speaking of alternative translations of a word in another language (e.g. French "marquer" into "mark" and "brand" are both acceptable);

False friends, where a word is not a possible translation of its cognate (e.g., "possible" and "actual", the correct and incorrect translations of the French "actual", respectively); and

Insertion or deletion errors, such as "he rang(at) the doorbell", "we paid(for) our meals", can also be considered as a substitute for or being substituted for an empty word;

When the use of the word w in a sentence is determined to be inappropriate, i.e. wrong, or otherwise non-idiomatic, each member of the set of words called w's confusing set C(w) can be considered is a possible replacement. The confusable set for w is drawn from those words associated with w, with the proviso that the actual membership may vary with the user's native language, the level of proficiency in the language in which it is written, and other factors.

Correlation is a widely used method to represent sentence structure. Many of the variations found are largely insignificant in the context of the present technique. A related relationship links two words called a relative word and a head word. In a typical pattern, no related word can be related to more than a single head word, but a head word can have any number of related words; other constraints, such as prohibiting loops, ensure that relations within a single sentence form a tree structure. In this specification, the association (also called association) between two words in a sentence is expressed in triplet form:

<first term, relationship, second term>

Here the entry is a term, such as 'chase_V' means all forms of the verb "to chase", namely chase, chased, chasing.

An association can be associated with its strength or number of possibilities. The frequency of an association, the number of times it is seen in a parsed corpus, is only a rough way of assessing its strength. A more accurate measure is to calculate the degree to which the frequency of the association deviates from what would be expected from the frequencies of its components. In some literature (for example, K. Kaqila, 1999, "Bigram Statistics Revisited: a Comparative Examination of some Statistical Measures in Morphological Analysis of Japanese KanjiSequences", Journal of Quantitative Linguistics, No. 6, No. 2, 1999, No. 149- 166, and "Methods for the Qualitative Evaluation of Lexical Association Measures" by Evert et al., Proceedings of the 30th Annual Meeting of the Society for Computational Linguistics in Toulouse, 2001, pp. 188-195, they gives a comparative evaluation of several measures in specific tasks) several such measures are disclosed and have applications in word segmentation, parsing, translation, information retrieval and lexicography. In these examples, generally only those associations that are significantly more likely than the expected frequency are of interest. However, the technique also focuses on associations that are significantly less likely than the expected frequency. Detection of such an association in text often indicates ungrammatical or non-idiomatic language usage.

Words that occur in one or more unlikely associations can then be substituted in turn by each member of its confusing set and the plausibility of the result of each such substitution can be calculated. If one or more members of the confusing set result in a sufficiently increased likelihood, those members may be suggested as replacements.

As a preliminary step, a large amount of spoken native language text is analyzed according to the relevant grammar to build a database of possible values for word combinations. Any suitable parsing program may be used, suitable examples are disclosed in M. Collins, "Three Generative Lexicalised Models for Statistical Parsing", Proceedings of the 35th Annual Meeting of the ACL/8th Conference of EACL, Madrid, 1997, and Slitter and Templey, "Parsing English with a Link Grammar", CMU-CS-91-196, Carnegie Mellon University, Department of Computer Science, 1991. The parser doesn't even have to be a parser as you might imagine, but can use finite state or similar techniques that add record dependency mechanisms.

According to one or more statistical measures, the frequency of each type of association (such as mutual information, T-value, and log-likelihood) is calculated, the likelihood value is calculated for each association and the results are stored in a table. Figure 3 shows some entries in such a database.

In Figure 3, the first column shows the association itself. The column titled 'freq' is the number of times this association occurs in the parsed corpus (in this case the UK National Corpus of about 80 million words). The remaining columns are mutual information, T-value, Yule's Q coefficient, and log-likelihood, respectively. Each of these is a different measure calculated from the frequency of the following four terms.

<first term, relationship, second term>

<first entry, relation, ^* >

< ^* , relation, first term>

< ^* , relation, ^* >

Here ' ^* ' stands for any entry. This parametric scheme is disclosed in "Automatic Retrieval and Clustering of Similar Words" by D. Lin, COLING-ACL 98, Montreal, Canada, August 1998. Different metrics have different ranges and are sensitive to the precise values of the four parameters in different ways. In each case, however, the value is related to the likelihood of the relationship. A positive value indicates that the combination is more likely than chance, and a negative value indicates that it is less likely.

For example, the formula for calculating the T value of <associate_V padv to_PREP> is: [P14-2] $\mathrm{associate}\_V.\mathrm{padv}.\mathrm{to}\_\mathrm{PREP}=$ $\frac{f/f\left(\mathrm{padv}\right)-\left(f(\mathrm{associate}\_V\&CenterDot;\mathrm{padv})f(\mathrm{padv}\&CenterDot;\mathrm{to}\_\mathrm{PREP})\right)/f{\left(\mathrm{padv}\right)}^2}{\sqrt{f(\mathrm{associate}\_V\&Center\; Dot;\mathrm{to}\_\mathrm{PREP})/f\left(\mathrm{padv}\right)}}$ $\mathrm{associate}\_V.\mathrm{padv}.\mathrm{to}\_\mathrm{PREP}=$ $\frac{25/10587833-(7680\&times;1020531)/{10587833}^2}{\sqrt{25/10587835}}=-143.050$

where f(associate_V·Padv·to_PREP)=F

In order to obtain high-quality estimates of the likelihood of word combinations, parsing native-language spoken corpora needs to be as accurate and as comprehensive as possible. However, accurate parsing requires the use of high-quality estimates of word combination probabilities, and this leads to a conflict. This conflict can be resolved by using an iterative or step-by-step approach. This is based on certain properties of the parsing algorithm.

Each individual association in a sentence is associated with a priority value. The priority value is a measure of the confidence that such an association exists between two words in a sentence. This priority value is a function of both sentence description coefficients, such as part-of-speech probability and word separation, and linguistic breadth coefficients, such as the strength of association between these words.

It returns a collection of associations that collectively satisfy the axioms of correlation structure (i.e. associations do not cross, each word is a related word of no more than one node, etc.); however, this collection is not required to form a single connection tree;

The relative effect of the sentence description coefficient and the language breadth coefficient on the priority value can be changed through appropriate parameter settings;

A threshold can be set so that only associations with priority values exceeding this threshold are returned;

The iterative nature of parsing will be exemplified in the analysis of a very simple phrase "world title fight".

According to the grammar, "title" must modify "fight", but it is not clear whether "world" modifies "title" or "fight". In English grammar, every noun in a noun sequence except the last can modify any noun to its right. In this example, knowledge of the strength of certain word combinations leads to the conclusion that "world" modifies "title". In other examples, like "plasticbaby pants," the first noun modifies not the noun that follows it but the last noun.

A full syntax analysis will give the association:

1. <title_N, mod_of, tight_N>

2. <world_N, mod_of, title_N>

In the first iteration of parsing the native spoken corpus, no possible value for the association between specific words was available, so the linguistic breadth factor had no effect on the priority value. The priority value threshold is set high so that, for example, words whose parts of speech are ambiguous or far apart are not associated, and the confidence in the correctness of the association is high. Following this example, only association 1 will be returned. The penultimate noun in a sequence definitely modifies the last noun, regardless of the language breadth coefficient. However, in the absence of language breadth information, neither association 2 nor erroneous <world_N, mod_of, fight_N> would have a high enough priority value to be returned in this example. However, associations with other instances of "world title" (and "world fight") that are not followed by other nouns in the corpus will be returned.

These high-certainty associations are then used to compute likely values. Later iterations can then use these language breadth coefficients to determine priority values, so the priority value threshold can be lowered. This increases the number of associations returned (coverage of the parse) and allows calculation of more accurate likelihood statistics. Following this example, the relative frequency and/or likelihood of <world, mod_of, title> and <world, mod_of, fight> will now give priority to the former over the latter. Further iterations will then continue to increase the effect of the language breadth factor on the priority value and decrease the priority value threshold. In this way, the coverage and reliability of probability data can be gradually strengthened.

After each iteration of parsing the native-language spoken corpus, the possible values associated with each type are determined and entered into the database.

When a sufficiently accurate database has been prepared or obtained by whatever means, it can be used in the present invention. The text to be checked is subjected to one iteration of such a parsing process. It is possible to reduce the contribution of the linguistic breadth coefficients to this parsing, and these coefficients, the possible values of the associations, will be considered in the next stage.

The possible value of each association in the text is then determined by examining the native language spoken database. Associations not seen in the original native-language spoken corpus are assigned a likelihood value by assuming that they have a low frequency. In a typical embodiment, all associations with a frequency of 1 found in the native language spoken corpus are discarded, greatly reducing the amount of data. It is then assumed that an association not found in the database has a frequency in the range 0 and 2, which is the optimal value determined experimentally, and its possible value is calculated accordingly.

Correlations that are likely to be low (eg, negative) are indicators of possible error. The possible values for which a word is associated are combined into the word's plausible value. Non-plausible words are replaced by members of their confusing sets to see if their plausibility results improve.

Figure 4 shows an embodiment of the invention as an error detector and modifier. As an example of syntax analysis, in step 10 an input text is provided and in step 11 it is analyzed. In step 12, the likelihood of associations in the input text is analyzed. In step 13, the first word in the text is selected and in step 14 the likelihood of this word is calculated. The input text is checked in step 15 to determine if all words have been used, if not, the next word is taken in step 16 and step 14 is repeated.

When all the words in the text have calculated plausibility values, in step 17 the words are sorted in ascending order of plausibility. The least plausible word is selected in step 18 and the method is terminated in step 20 if its plausibility is not lower than a first threshold in step 19 . Otherwise, get the confusing set of this word in step 21 and select the first confusing word in step 22 . In step 23 the word is replaced by the confusing word and in step 24 the plausibility of the confusing word in the context is calculated. If in step 25 an improvement in plausibility greater than a second threshold is detected, then in step 26 the confusing word is reported to the user.

Step 27 checks if all confusable words have been tried, if not, the next confusable word is selected in 28 and control returns to step 23. Otherwise, step 29 determines whether all words in the text have been processed, if not, step 30 gets the next word and returns control to step 19. Otherwise, the method ends in step 31 .

In this embodiment, we determine for each word w _i (1≤i≤n, the length of the sentence) the association set D(w _i ) it belongs to. We then apply to each D(w _i ) a function that maps the possible values of the association set to a single value, called the word's "likelihood" λ(w _i ). Rank the words by plausibility. If the likelihood of the minimum plausibility word w _λmin is below a threshold, then we try to find a correction. We replace w λmin with each word c _i (w _{λmin )} (1≤j≤n, the number of confusing words in C(w _λmin )) in turn, and calculate λ( _ci (w _λmin )). Those confusable words that improve the plausibility of the word after substitution may be provided to the user. Confusible words may be displayed in descending order of the improvements they produced.

Members of the confusing set may be associated with a confusing value indicating the likelihood of confusion. For example, from a learner-annotated corpus we can get an aggregate count of the frequency of each word that is mistakenly used as another word; errors in pronunciation and/or spelling of real words may be associated with values based on edit distance; Confusing words based on semantic relatedness may be associated with values based on path distance in a hierarchical network.

If such information is used, it is presented in a more _helpful order by _combining confounding _and improvements in plausibility into a single score, the surrogate score suggestion.

During the interaction with the user, an initial suggestion is provided to improve w _λmin by replacing the word with a member of the confusing set. If the user accepts one of these words, the substitution effect is passed on to the other words associated with it and the calculation of the new _wλmin value is repeated. The transfer process may include reattaching a substitute word to a different word than the original word.

Associations that are unlikely when used independently are possible when used as part of a larger structure, and vice versa. For example, "by accident" is a very strong collocation, while "by the accident" is unlikely and should be considered a possible error. There are however larger, possibly correct constructs that include the latter, such as "horrified by the accident".

Conversely, isolating "a knowledge" is a typical learner error, while "acknowledge of" is a reasonable expression. But when it came to "learn a knowledge of", it was a mistake.

These cases can be handled by computing the possible values of a correlation subgraph containing three or more elements linking two or more associations. Experimental observations indicate that no more than three elements are required in most cases. In the above case, the likelihood of a four-element phrase can be traced back to the likelihood of smaller elements. For example, "horrified by the accident" is possible because "horrified by" is such a strong collocation, while "learn a knowledge of" is unlikely because "knowledge" is such a strong collocation that "learn" does not Too probable object, irrespective of other elements.

The possible values for a triplet subgraph can be calculated in different ways. One method is to treat two of the elements and the association between them as a phrase unit, and then calculate the likelihood measure between this phrase unit and the third element, using the same calculation method as in the case of two elements The method is exactly the same.

Combining the possible values associated with two and three elements into a plausible value can also be achieved according to different schemes. We can weight the contribution of three-element phrases more than that of two-element phrases (a smoothing scheme) or if three-element phrases containing two-element phrases fail to satisfy some constraint on their frequency and/or likelihood, then Use only two-element phrases (a fallback scheme). The parameters of these schemes can be determined by experience or by a learning process, where the element of learning is not the presence or absence of a word in context, but the strength and frequency of combinations.

To increase the range of errors that can be detected and corrected, some extensions can be made to the basic method.

Computing the plausibility of a term may include a condition indicating the lack of affiliation of that term to any other term. Except in the case of finite verbs (or some other part of speech in lists and titles) which may be the root of a correlation tree, independent words always indicate an error (or an erroneous grammar). It is therefore appropriate to assign a very low probability value to null assignments, and this will trigger error handling.

In order to determine the corrections, the method will then need to be extended, as described below.

Provided that, as mentioned above, the grammatical analysis of the text to be corrected is not strongly influenced by the language breadth priority value coefficients, words will generally be attached if their parts of speech are appropriate. Conversely, if a word is not attached, the error will generally not be corrected by substituting a word with the same part of speech.

The error may not be a substitution, but an omission. For example, a noun will not be able to attach as an object to an intransitive verb. In many such cases, the error can be corrected by the insertion of a preposition. Insertion may be appropriate even when a noun is attached to a weakly associated verb. In either case, the insertion must be accompanied by the establishment of a new association, the likelihood of which will determine whether the error has been fixed.

Missing attachments can also be caused by substitution errors with class changes. This permutation may necessarily be accompanied by a partial reanalysis of the input if the confusing set of words of one class contains a word of another class. For example, if a beginner writes "get out of the building safety", the sequence "building safety" can be analyzed as an (unlikely) noun phrase. If the confusing set of the noun "safety" included the adverb "safely," reanalysis would be necessary to determine that the latter is a modifier of the verb "get out," with "building" rather than "safety" as its object.

This method can also be used as a context-sensitive dictionary, for example, without setting a threshold for the plausibility value of each word. In this case, all words are candidates for substitution regardless of plausibility. Likewise, substitution need not improve plausibility. Possible substitutions can be proposed, for example, if their plausibility values exceed a threshold.

The method may be carried out by any suitable means, but, in practice, it is most likely to be carried out by a computer which has been programmed to control it to carry out the method. Figure 1 illustrates a suitable computer system 100 based on a central processing unit (CPU) acting as a controller. The CPU 1 is equipped with a program memory 2, for example in the form of a disk drive containing storage media in the form of magnetic disks or optical disks, which in turn contain programs for controlling the CPU 1 . A first database 3, eg stored on a disk, contains associations and associated possible values. A second database, eg also stored on one or the aforementioned disks, contains confusing sets. A read/write or random access memory (RAM) 5 is provided in the usual way to hold temporary values of the parameters.

The CPU is equipped with an input interface 6 which allows the input of text to be detected for errors, unnatural expressions and the like. For example, text may have been entered manually via a keyboard or may already be in machine-readable form (eg, on a magnetic or optical disk). CPU 1 is also equipped with an output interface 7 which allows the user to monitor the output of the method. Also, in order to be able to interact with the method, the interfaces 6 and 7 provide the user with functions to enter data, commands, etc. and to monitor the operation of the method. For example, when the selection of easily confused words that increase the likelihood value is provided, these can be displayed through the display that constitutes part or all of the output interface 7, and the user can properly operate the keyboard and/or mouse that constitute part or all of the input interface 6 Choose a word that is easily confused.

A database containing associations between words and possible values associated with them is provided, which provides a measure of the possible values of such correct or customary associations. Possible values are based on the frequency of associations obtained by analyzing large volumes of text, such as text authored by native speakers. In order to check a text segment for possible wrong or unnatural usages of one or more words of the text segment, the text is first analyzed to determine the associations between its words. The likelihood of associations in the analyzed text is determined by the database. Computes the likelihood value for each word in the analyzed text, which is obtained by combining the possible values of the associations in which the word occurs. Use terms to index another database that contains collections of terms that are easily confused by index terms. Select each confusing term in turn and replace the index term in the index term's association. The possible values of these new associations are determined and the plausibility value of the confusing word is calculated. In one error detection embodiment, confusable words are tried for those words whose plausibility falls below a threshold, and the confusable words with increasing plausibility are reported to the user. In a context-sensitive thesaurus embodiment, confusable words may be tried for all words and those with a plausibility value exceeding a second threshold may be reported.

Although only one embodiment in which the present invention is applied to English is described above, the present invention is not limited to English but can be applied to other languages.

English text segments can be translated from languages other than English (such as Japanese).

A text segment can be generated by reading the words of a printed document using an optical character recognition system.

In accordance with the present invention, a method and an apparatus are provided for detecting errors and unnatural expressions in a user's writing and proposing ways in which these language usages can be improved.

According to the present invention, it is possible to detect errors and unnatural expressions in the user's writing and propose corrections to them. It handles misspellings of real words, as well as various other types of errors.

Claims (21)

1. One kind in the written or spoken text chunk that comprises one group of word of first language first word or the modification selected of phrase or improve one's methods, it is characterized in that, comprise the following steps:

   (a) provide first database (3) of the association between a first language word or the phrase, wherein each is associated to rare one and is associated in the relevant probable value of the frequency of occurrences in a large amount of first language texts based on this;

   (b) it is related with one first between described first word that is based upon text section or phrase and one second word or the phrase to analyze (14) text section, first probable value of corresponding at least described association and based on the first likelihood value of described at least one probable value described first word of correspondence or phrase;

   (c) provide one second database (4), wherein every has at least a word or phrase and its word that can be confused into or phrase set to link together;

   (d) from second database (4), select (22) or calculate easily be confused word or the phrase that a candidate as first word described in the text section or phrase substitutes;

   (e) derive (23,24) easily be confused word or phrase in first database (3) based on the second likelihood value of the probable value of one second association, this second association is made up of other word or phrase in easily be confused word or phrase and the text section; And

   (f) optionally provide the indication (25,26) of easily be confused a word or a phrase based on the truthlikeness value that calculates.
2. 2. the method for claim 1 is characterized in that, each the related probable value in first database (3) also is based on each and comprises one and have the word of identical correlationship or other related occurrence frequency of phrase.
3. 3. the method for claim 1 is characterized in that, each the related probable value in first database (3) also is based on all other related occurrence frequencies with identical correlationship.
4. 4. the method for claim 1 is characterized in that, each the related probable value in first database (3) is formed by in mutual information, T value, Yule ' s Q coefficient and the log likelihood at least one.
5. 5. the method for claim 1 is characterized in that, in step (e), described other word or phrase are second word or phrase, and the correlationship of second association is identical with the first related correlationship.
6. 6. the method for claim 1 is characterized in that, step (b) be included as text Duan Zhongyi organize first word or phrase set up one group first related and and (f) to each first related execution in step (d), (e).
7. 7. the method for claim 1 is characterized in that, step (b) comprises the association of setting up between non-conterminous word in the text or the phrase.
8. 8. the method for claim 1 is characterized in that, step (d) comprises each confusable word of the set of selecting a word or phrase or phrase and to each easily be confused word or phrase execution in step (e) and (f).
9. 9. method as claimed in claim 8 is characterized in that, step (f) comprises by the second likelihood value descending indicates the second likelihood value.
10. 10. the method for claim 1 is characterized in that, if the first likelihood value is less than a first threshold (19) then execution in step (d), (e) and (f).
11. 11. the method for claim 1 is characterized in that, step (f) comprises provides indication when described or each second likelihood value surpass one second threshold value (25).
12. 12. the method for claim 1 is characterized in that, step (f) comprise if the second likelihood value greater than the first likelihood value (25) then indication (26) is provided.
13. 13. the method for claim 1 is characterized in that, step (b) comprises function calculation (14) the first likelihood values that rely on to learn from beginner's mistake tagged corpus and relevant likelihood value thereof by machine learning techniques.
14. 14. the method for claim 1 is characterized in that, word replaces first word in (23) text section with easily being confused.
15. 15. the method for claim 1 is characterized in that, generates text section by the second language translation.
16. 16. the method for claim 1 is characterized in that, generates text section from document printing by optical character identification.
17. 17. be that computer compilation is to carry out the computer program of method according to claim 1.
18. 18. contain storage medium just like the described program of claim 17.
19. 19. medium as claimed in claim 18 comprises computer-readable medium.
20. 20. contain computing machine just like the described program of claim 17.
21. 21. first word or phrase a modification or the modifying device selected in the written or spoken text chunk that comprises one group of word of first language is characterized in that, comprising:

    First database (3) of the association between first language word or the phrase, wherein each is associated to rare one and is associated in the relevant probable value of the frequency of occurrences in a large amount of first language texts based on this;

    A controller that is used for analyzing (14) text section, with determine one first between described first word of text chunk or phrase and one second word or the phrase related, the corresponding described association of at least one first probable value and corresponding described first word of the first likelihood value based on described at least one probable value or phrase; And

    One second database (4), wherein every has at least a word or phrase and its word that can be confused into or phrase set to link together;

    Wherein:

    Controller (1) is selected (22) or is calculated easily be confused word or the phrase that a candidate as first word described in the text section or phrase substitutes from second database;

    Controller (1) derives (23,24) easily be confused a word or a phrase based on one the second second likelihood value that is associated in the probable value in first database (3), this second association is made up of other word or phrase in easily be confused word or phrase and the text section; And

    Controller optionally provides the indication (25,26) of easily be confused a word or a phrase based on the truthlikeness value that calculates.

Images