CN111971744A - Handling speech to text conversion - Google Patents

Abstract

The speech analysis is performed by: receiving an estimated glyph corresponding to an estimate made by speech-to-text software of what the user says aloud; comparing the estimated glyph to a reference glyph representing text from which the user is attempting to read; and providing feedback related to the user's speech based on the comparison of the estimated glyph to the reference glyph.

Description

Handling speech-to-text conversion

technical field

The present invention relates to processing speech-to-text (StT) transcription. The present invention includes methods, apparatus, computer programs, and non-transitory computer-readable storage media that may provide improvements and/or advantages related to processing speech-to-text conversion.

Background technique

The StT conversion software generates a text transcript of the text a person speaks aloud as estimated by the software. StT software usually does this when people speak. In some cases, the translations can be output, for example, to a word processor, so that the StT software provides a dictation-transcription service, allowing users to compose letters or other documents by reading aloud. In other cases, the text converter can be parsed and interpreted as commands, eg, to control devices such as cell phones, light bulbs, or climate control systems.

The claimed invention seeks to provide improved handling of speech-to-text conversion and additional functionality using speech-to-text conversion.

SUMMARY OF THE INVENTION

In one aspect, the present invention provides a method of analyzing speech, comprising: receiving an estimate made by speech-to-text software of a reader's oral reading of at least a portion of a reference text, the estimate comprising a plurality of estimated glyphs, each estimating the glyph representing at least one grapheme; comparing the estimated glyph to a plurality of reference glyphs representing at least part of the reference text, each reference glyph representing at least one grapheme; and providing a comparison with the reader speech based on the comparison of the estimated glyph to the reference glyph relevant feedback.

Embodiments of the present invention provide an efficient way of analyzing speech-to-text conversion from readings of reference texts so that feedback on the reader's fluency can be provided. In particular, embodiments of the present invention are language-independent by comparing glyphs representing graphemes or grapheme combinations.

In one embodiment, each reference glyph represents at least one word in the reference text, and each estimated glyph represents a set of one or more alternative words estimated by the speech-to-text software for the word spoken by the reader/user or phrase. This implementation improves the handling of homophones (such as "sail" and "sale"), near-homonyms ("are" and "our"), and errors or inaccuracies in translations due to stuttering or the reader's severe accent certainty.

In one embodiment, comparing the estimated glyphs to the reference glyphs includes: linking each estimated glyph to any matching reference glyph to generate multiple links; identifying conflicts between the links; and pruning the links by removing some conflicting links to resolve all identified conflicts. Link matching glyphs and trim conflicting links to eliminate slurs, mistakes, and repeated words such as stuttering in conversions. Linking all matching glyphs ensures the largest set of links for analysis. The removed proficiency can still be analyzed to provide feedback, directly or indirectly, based on the absence - for example, there will be time gaps in the remaining translations indicating where the proficiency occurs.

In one embodiment, identifying a conflict between links includes identifying a link that violates at least one of a set of rules, the set of rules comprising: (1) a reference glyph must not be linked with more than one estimated glyph; none of the reference glyphs There may be two links (2) an estimated glyph must not be linked with more than one reference glyph; and (3) no two links can cross each other. Applying these three rules removes links that represent typical errors in reading to get an accurate picture of the reader's fluency.

In one embodiment, pruning the links includes selecting a first link and identifying a set of links that conflict with the first link; determining a cost of maintaining each link in the identified set of conflicting links, the cost including the number of conflicting links for each link in the group; and removing said conflicting links except for the lowest cost conflicting link. Iteratively selects a link to determine the cost associated with maintaining that or conflicting links, and keeping the link with the lowest cost helps ensure that the end result provides the best possible correlation between the converted copy and the reference text.

In one embodiment, the method includes generating a plurality of link bundles from a plurality of links, each link bundle comprising one or more links, the links forming a continuous sequence of estimated glyphs that match a continuous sequence of reference glyphs; and wherein, Identifying conflicts between links includes identifying conflicts between linked bundles. Pruning links involves removing some conflicting link bundles to resolve the identified conflicts. Create Link Bundles Create linked sections that represent full fluent reading. This enables better and more consistent handling of transitions that contain read errors or similar "noise". Create the maximum number of link bundle sets to further improve your analysis by defining link bundles to include any single link or any contiguous group of links.

In one embodiment, pruning the links further includes maintaining larger link bundles in preference to smaller link bundles to resolve the identified conflicts. Large linked bundles represent fluent readings with little "noise", so prioritizing them over individual links or smaller linked bundles helps ensure that the best reading estimates are preserved.

In another aspect, the present invention provides a method of analyzing a user's speech, comprising: receiving one or more estimated glyphs, the estimated glyphs corresponding to estimates by speech-to-text software of what the user has spoken aloud; The one or more estimated glyphs are compared to one or more reference glyphs the user is attempting to read; and feedback related to the user's speech is provided based on the comparison of the one or more estimated glyphs to the one or more reference glyphs.

Optionally, providing feedback on the user's speech includes: generating a plurality of links, each link connecting a glyph of the one or more estimated glyphs and a glyph of the one or more reference glyphs; and applying a trimming process to the multiple glyphs. links to identify and resolve conflicts between links.

Optionally, the trimming process includes applying at least one rule to the plurality of links, the at least one rule specifying that: a glyph in the one or more reference glyphs must not be linked with more than one glyph in the one or more estimated glyphs; one or more A glyph in a plurality of estimated glyphs must not be linked to more than one glyph in one or more reference glyphs; or links linking glyphs in one or more estimated glyphs and one or more reference glyphs must not cross each other.

Optionally, the pruning process includes removing one or more links that do not meet the at least one rule.

Optionally, the pruning process includes determining the cost of retaining a link and comparing the cost of retaining the link to the cost of retaining a different link to determine which of the two links to remove.

Optionally, each link includes a pair of indices identifying a corresponding glyph in each of the one or more estimated glyphs and the one or more reference glyphs.

Optionally, the method further includes generating a plurality of link bundles, each link bundle including at least one link of the plurality of links.

Optionally, based on the output signal provided by the microphone, one or more estimated glyphs are generated corresponding to the estimation by the speech-to-text software of what the user has spoken aloud.

Optionally, the feedback includes one or more of: at least one parameter indicative of fluency of the user's speech; and at least one representation of the user's current reading position in one or more reference glyphs.

Aspects of the invention may be implemented, for example, as an apparatus, a computer program, or a non-transitory computer-readable storage medium.

Description of drawings

Specific embodiments of the present invention will now be described by way of non-limiting examples in which:

Fig. 1 shows the output example of StT conversion software;

Fig. 2 shows a kind of method that the word in the text to be read is correlated with the word in the StT conversion book;

Figure 3 shows an example of a first rule for trimming the correlation between the text to be read and the words in the StT translation;

Figure 4 shows an example of a second rule for trimming the correlation between the text to be read and the words in the StT translation;

Figure 5 shows an example of a third rule for trimming correlations between text to be read and words in the StT translation;

Figure 6 shows an example of a maximum link set that associates words in the text to be read with words in the StT translation;

Fig. 7 shows the example of Fig. 6, wherein a series of links from consecutive text words to consecutive translation words are represented by bold arrows and white boxes;

Figure 8 shows a variation of the example maximal link set of Figure 6;

Figure 9 shows the link set of Figure 8 after applying the first, second and third pruning rules of Figures 3 to 5;

Figure 10 shows a variation of the example maximal link set of Figure 6;

Figure 11 shows the link set of Figure 10 after applying the first, second and third pruning rules of Figures 3 to 5;

Figure ₁₂ shows _a set of links aggregated into link bundles b1 and b2;

Figure 13 shows the linked set of Figure 12 after applying the pruning rule of Figure 3 to linked bundles;

Figure ₁₄ shows _a set of links aggregated into link bundles b1 and b2;

Figure 15 shows a set of links aggregated into link bundles b ₁ and b ₃ ;

Figure 16 shows a set of links aggregated into link bundles b ₁ and b ₄ ;

Figure 17 shows a set of links aggregated into link bundles b ₁ and b ₆ ;

Figure 18 schematically illustrates a method for determining whether a pair of linked bundles overlap each other;

Figure 19 illustrates the method of Figure 2 applied to the example output of Figure 1;

Figure 20 shows the example of Figure 19 partially through the link pruning process;

Figure 21 shows the example of Figure 19 after the link pruning process is complete; and

Figures 22, 23 and 24 illustrate the linked bundle method applied to the output of Figure 1 .

Detailed ways

The embodiments of the present invention represent the applicant's present best ways to put the invention into practice, but they are not the only ways in which the invention can be practiced. They are only exemplified and will now be described.

In some cases, it may be necessary to compare the text translation output by the StT conversion software with another text, such as the text the user is reading. For example, this may allow the user's current reading position to be established in the text. An indication of the speech accuracy of the user who is reading the text may also be given. This enables users who wish to improve their spoken English to read stories, newspapers and other text aloud and receive feedback on data points such as reading accuracy, reading speed, fluency and pronunciation. The user may be provided with feedback on her or his intelligibility of the text he or he has spoken aloud.

By comparing the StT conversion to the text being read, the reader's current reading position can be determined. Under ideal conditions, this might be intuitive - the StT convertible might exactly match the text being read, so there is a one-to-one correspondence between the StT convertible and the text. In this case, the most recently added word in the StT translation can be used to identify the current reading position. However, there are a number of factors that can make tracking the current reading position difficult in practical situations. Readers, especially those who are learning to read, may not be able to read texts fluently. For example, he or she may slow down or stop in the middle of a sentence when encountering an unfamiliar word, and then resume the previous reading speed. The reader can use filler words (eg "um", "so" or "like"). The reader may recognize that he has made a mistake and then go back to the beginning of a word, line, sentence or paragraph to try another fluent reading. Readers may miss a word; inadvertently skip a line of text; reread a word, line, or sentence; garble a small piece of text; or get lost in the text. Additionally, StT software may sometimes incorrectly translate what the reader is saying, especially if, for example, the reader has an accent that the StT software cannot interpret correctly. Any of these problems can lead to conversion errors, which make it difficult and error-prone to compare the text the user is reading with the output of the StT software.

Figure 1 shows an example of the output of the StT software when the user is not fluent in reading. In the example shown, the user attempts to read aloud the text "Come along, Paddington. We'll take you home and you can have a nice hot bath" from Michael Bond's "Please Take Care of This Bear" . The StT software translates the user's attempts, and the various alternate words spoken by the StT software are shown in the table shown in Figure 1, and displayed in the alternate row in order from left to right, from high to low confidence. For example, StT software has interpreted user attempts at the beginning of a text excerpt for the word "come" as (in order of decreasing confidence) "come", "came", "Kim" or "cam". Likewise, StT software interprets "along" as "along", "long", "alone" or "on".

The user's reading begins fluently with "come along" and then tries a few stuttering "Paddington" before getting right. Next comes "we'll take you home", even though the reader actually says "we'll take you home" and then self-corrects to "take you home". In the end, we get "and you can have a", followed by a wrong transformation of "nice", "hot". "bath" appears to have been left out, but upon further inspection of the alternate word in the last line of StT's output, "hardball's" appears; after a bit of explanation, it can be thought of as "hot bath".

All of the above lack of fluency may make it difficult to determine the reader's current position in the reading text from the StT translation at any given time. Therefore, an attempt needs to be made to identify and eliminate the influency in the StT conversion book. To do this, first try to associate the words in the StT translation with the text you are reading.

For infrequently occurring words, the correlation may be trivial. For example, a word that occurs once in a read text and a word that occurs once in a translation can be correlated with confidence. However, this is usually not the case. In the short example above, the word "you" appears twice, and if the user further reads the book from which the excerpt comes, other words will appear multiple times, including less common words like "Paddington".

One way to establish a correlation between the text read and the StT convertible might be to look for hesitation. For example, one can look at hesitant words or phrases in the translation, such as "erm", "um", "I mean", etc., and remove them from relevance considerations. However, this approach poses several problems. The list of hesitant words and phrases will likely continue to grow as new hesitant phrases are discovered. Different readers are likely to use different "pet" hesitant phrases, so one reader may use "I mean" while another reader may use "er, sorry" which, when implemented as software, will result in Frequent software updates, which can be difficult to maintain. Hesitant words may also legitimately appear in the text being read and should not simply be removed.

Another approach might be to look for duplicates. As shown in the example in Figure 1, readers sometimes stutter and repeat a word or phrase, or even start rereading at the beginning of a sentence or paragraph. Duplicate words or word sequences can be looked for, identified as duplicates, and excluded from relevance considerations. However, there are also many problems with this approach. Duplicate words may legitimately appear in the text being read. For example, the structure "had had" in the sentence "I had had too many chocolates and so didn't feel very well." is a legitimately used repeated word. Since some readers will only repeat mispronounced words, while others may repeat from the beginning of a line, sentence, or paragraph, it is impossible to predict when to look for repeated word sequences. Also, repetition is often mixed with hesitant phrases, so it's not necessarily easy to spot. For example, it is difficult to codify how to distinguish between repeated and hesitant words from the words "Today is Wednes,er I mean,today isWednesday".

Figure 2 illustrates a general method of associating words in a text to be read with words in a StT transformation according to an embodiment of the present invention. Literal words are given in the following order: Literal words are intended to be read along the top of the graph. give an index to each word in the text to be read (shown as "i" in the example shown, where i is a number between 1 and 5), and similarly, give the value estimated by the StT software to have been read One index per word or group of words (shown as "j" in the example shown, where j is a number between 1 and 5).

Relevance is modeled with links, each link associating a text word with a translation word. In this case, "converted word" may actually include a number of alternate translations provided by the StT software. The transformbook word indexed j=5 in Figure 2 illustrates this concept - the StT software provides several options for it to estimate the content of the final pronunciation in the transformbook shown, from the transformbook in Figure 2 The concatenated square representation at index j=5 in the word row. If one of those alternate conversion words at index j=5 matches a text word, the text word and the set of alternate conversion words will be associated by a single link. For simplicity of presentation, alternate translation words are generally not shown in the figures, but any translation word in Figure 2 or other figures may also have multiple alternate translation words.

In the figures, the correlation between the text word and the translation word is represented pictographically as an arrow connecting the text word and the translation word. Links may sometimes be bolded for emphasis. However, dependencies may be represented and/or stored in other ways. For example, a given link l can be represented as a pair of values (i, j) indicating the start and end of the link, ie the word in the text to be read and the word or group of words in the StT translation are connected by the link. For example, the link shown in Figure 2 can be represented as (3, 2) because it connects the word with index i=3 in the text to be read and the word or group of words with index j=2 in the StT software output.

Ultimately, a set of links between text words and translation words should be modeled as close to fluent text reading as possible. Fluent reading can be classified as one of the following: reading words in the correct order, and words in the text are related to at most one word in the translation, and vice versa, words in the translation are related to at most one word in the text. These constraints are captured by three rules that must be satisfied by any link that associates a text word with a translation word if a set of links represents fluent reading.

Rule 1: No two links can link the same text word to the translation word. Since any word in the text should only be read once in a valid reading, there should be at most one link between the text word and the converted word. As a result of Rule 1, the two links shown in Figure 3 cannot be part of Fluent Reading at the same time because the two links shown associate one text word with two translation words. Text words may not be linked to any converted words - for example, if a word was skipped by the reader (i.e. not read), or if the word was incorrectly converted by the StT software.

Rule 2: No two links can link the same translation word to a text word. One word in the translation should correspond to at most one word in the text. In a perfect reading, each text word will be linked to a translation word. As a result of Rule 2, the links shown in Figure 4 cannot be part of Fluent Reading at the same time because the two links associate two text words with one translation word. Converting this word may not link to any text word for any of a number of reasons. Mistranslated words that were never spoken by the reader may not link to text; hesitant words will not link to text words; for example, reread words may not link.

Rule 3: No two links may cross. Crossed links indicate that the words were read out of order. As a result of Rule 3, the links shown in Figure 5 cannot all be part of Fluent Reading.

A link that violates Rule 1, Rule 2, or Rule 3 is called a conflict. Methods for resolving such conflicts are described in the following paragraphs.

To try to get a set of links that represent fluent reading, you must first create the largest set of links between text words and translation words. The largest set of links will contain a link wherever a text word matches a translation word (as mentioned above, a translation word may include several alternative translation words proposed by the StT software for a single utterance). Therefore, the largest link sets are unlikely to model fluent reading.

The steps to create the maximal link set are discussed in more detail in the following paragraphs, where each text word te in the text has an index in the text Te, as indicated by the letter i in Figure 2. Therefore, text can be thought of as an array of text words. Similarly, each translation word tr in the translation Tr has an index, as indicated by the letter J in FIG. 2 . These steps can be represented as follows.

1. Set L={}

This will be a set of candidate links. Initially L is empty, but is filled as candidate links are identified.

2. Set i=1

3. Set te _i to the text word at index i in Te.

4. Set Tri _{= match(te i ,} Tr)

_{Tri is a set of translation words that match te i .} match(te, Tr) searches each translation word tr in Tr and returns tr if te matches any of the alternative words in the translation word.

5. For each tr in Tr _i

5.1. Set L=L∪{Link(i,j)}—where j is the index of the transformed word tr.

The link creates a link between the text word te at index i and the translation word tr at index j. Note that this only records the index - not the word.

6. Set i=i+1

7. If i>te _last - where te _last is the index of the last word in Te.

stop

otherwise

Go to step 4.

As mentioned above, this creates the largest set of links that associate as many text words as possible with as many translation words as possible. The largest set of links is unlikely to represent fluent text reading, since in most cases a word will function in the text more than once, so rule 2 will be broken assuming such words are translated correctly at least once. Other rules may also be violated.

Once the maximal link set is established, a pruning process or method ("pruning") may be applied to the maximal link set to identify links that break one or more of rules 1, 2, and 3, and remove links until the rules are all complied with . The pruning process should maximize the number of remaining links, as this will result in the best relevance of the text and converted words.

The preferred pruning process removes links by evaluating the cost of keeping conflicting links. The cost of keeping links is the number of conflicting links that need to be removed in order to satisfy these three rules. Given a set of links, the pruning process will consider each link and its conflicting links in turn and keep the link with the lowest cost, as this is the link that resulted in the removal of the least conflicting link.

The steps of the linked preferred pruning process can be represented as follows.

1. Set L _remaining = {}, L _removed = {}.

These collections keep track of pruned (removed) links and remaining links. Initially they are empty, but are filled as the pruning process progresses.

2. Set L _unprocessed = L-{L _remaining ∪L _removed }

3. If L _unprocessed = {}

stop

4. Select a link in L _unprocessed l

5. Set L _conflicting =conflicts(l, L _unprocessed )

conflicts(l, L) returns a subset of L that contains links that conflict with l, i.e. violate at least one of rules R1, R2, or R3.

6. Set c _l = |L _conflicting |

This is the size of the set L _conflicting , the number of links that conflict with l.

7. Set c _min =min ({cl _' :cl _' =|conflicts(l', L unprocessed )|, l' _{∈L conflicting} })

In other words, find the cost c _l ' to keep each link l' in L _unprocess . Find the link with the lowest cost and save it in c _min .

8. If c _l <= c _min

8.1.L _removed ＝L _removed ∪L _conflicting

8.2.L _remaining =L _remaining ∪{l}

otherwise

8.3. L _removed =L _removed ∪{l}

Decide time. If c _l <= c _min , link l is reserved. Therefore, we add L to L _remaining and add all conflicting links to L _removed .

If c _l >c _min , l is removed, so L is added to L _removed .

9. Go to step 2

Note that in step 8, links from L to L _remaining or L _removed are added. Therefore, in the end, L _remaining ∪ L _removed = L and L _unprocessed will be {}, so the pruning process finally terminates in step 3. When terminated, all links _remaining in L will satisfy rules R1, R2, and R3. The pruning process works so that links, as previously described, can be represented as pairs of indices. While the index refers to words in the text and translations, the trimming process does not use words at all. The pruning process also does not assume any order of the links to be considered (eg, left-to-right or right-to-left). Thus, although the examples described and shown use English words for illustration purposes, the trimming process is language-independent and should work equally well for other languages. In addition to being applied to words, the trimming process can also be applied to any grapheme or combination of graphemes, such as letters, numbers, punctuation, syllables, logos, pictograms, ideograms, shorthand symbols or other representations of spoken or spoken words . In this document, the term "glyph" is intended to include any such representation of spoken language or discourse. In this document, one or more glyphs may constitute "text" (eg, "reference text" as referred to in the described and illustrated embodiments).

Figure 6 shows a set of links to prune. Suppose the pruning process first considers link 1 ₄ . l ₄ conflicts with link l ₃ (because l ₃ and l ₄ violate rule 1 causing the same text word to be associated with two translation words) and conflicts with link l ₅ (because l ₄ and l ₅ violate rule 2 causing two text words are associated with the same translation word). The rows in the table below give the cost of keeping l ₄ and its conflicting links, links that conflict with the corresponding links, and rules broken by the corresponding conflicts.

Link cost conflicting links l₃ 3 l₁(R3), l₂(R3), l₄(R1), l₄ 2 l₃(R1), l₅(R2) l₅ 1 l₄(R2)

In this example, l ₅ has the lowest retention cost (c _min =c ₅ =1 vs c ₄ =2) and we will remove l ₄ . Suppose the pruning process next considers link l ₁ . Only l ₃ conflicts with l ₁ . The cost of keeping l ₁ is c ₁ =1, and the cost of keeping l ₃ is c ₃ =2, because l ₃ conflicts with l ₁ and l ₂ after l ₄ is removed (both based on rule 3). Therefore, l ₃ is removed, the remaining links (l ₁ , l ₂ and l ₅ ) do not conflict with each other, so the pruning process ends.

As mentioned earlier, the pruning process assumes that the links to be pruned are not considered sequentially. In practice, reading in order does matter. Associating links in the same translation in a different order can lead to subtly different correlations, some of which have more links than others—the purpose of the pruning process is to maximize the number of related words in the translation. This variation in the number of related links tends to show up in many less-fluent readings, especially if a piece of text contains the same sequence of words. That said, fluent snippets often appear even in fluent reading. These segments can be identified as linked sequences from consecutive text words to consecutive translation words, as shown in Figure 7. Fluent reading is indicated by bold arrows and white boxes.

In the example of FIG. 7 , considering the order in which the links are pruned affects which links survive the pruning, which will now be described with reference to FIGS. 8 , 9 , 10 and 11 . In each instance, the links are numbered in the following order: they are considered for pruning (ie, link l ₁ is considered first, then link l ₂ , and so on).

In the example of FIG. 8, the pruning process first considers link l ₁ . The cost of keeping l1 is ₁ ( _l4 needs to be removed to keep _l1 ). l ₄ is the only link that conflicts with l ₁ . The cost of keeping l ₄ is 3 (requires removal of l ₁ , l ₂ and l ₃ ), so l ₄ is removed. There are no links that conflict with _l2 . There is a link l ₅ that conflicts with l ₃ . The cost of keeping l3 is ₁ ( _l4 has been removed, so the only other conflicting link is l5 ₎ . The cost of keeping l ₅ is also 1 (its only conflicting link is l ₃ ). _At this point, the pruning process will make an arbitrary decision to keep l3 and remove _l5 . While keeping the cost of each link the same, keep the considered links in favor of conflicting links. In this case, the pruning process can use the link's weight (i.e. how far down the list an alternative word must be searched before a match is found) to distinguish conflicting links, but even this would result in a "tie", so still Arbitrary decisions need to be made about which links to keep and which to remove.

The above pruning process results in the retention of links l ₁ , l ₂ and l ₃ , as shown in Figure 9. This is a good result in the sense that these links represent "fluent" readings represented by the white boxes and bold arrows in Figure 7.

Figure 10 shows the same example, but considering the links in a different order. In the case of Figure 10, the cost of keeping l ₁ is 2 (the conflicting links of l ₁ are l ₄ and l ₅ ). The cost of keeping l ₄ is 3 (l ₄ conflicts with l ₁ , l ₂ and l ₃ ). The cost of keeping l ₅ is 1 (l ₅ conflicts with l ₁ ), so l ₁ is removed! The cost of keeping l ₂ is 1 (the conflicting link of l ₂ is l ₄ ). The cost of keeping l ₄ is 2 (the conflicting links of l ₄ are l ₂ and l ₃ ), so l ₄ is removed. There are no conflicting links with l ₃ and l ₅ .

The pruning process results in links l ₂ , l ₃ and l ₅ remaining, which are sub-optimal - removing l ₁ , part of the fluent reading in the entire word set.

Therefore, there may be cases where the order in which the links are considered leads to different overall results, especially if the largest set of links is particularly "noisy", i.e. contains a large number of false links. Therefore, it is desirable to reduce the likelihood that noisy links remain after pruning at the expense of removing "good" links. Even non-fluent readings often have fluent reading segments that can be identified as a series of links that associate consecutive text words with consecutive translation words. One way to reduce the likelihood of retaining noisy links at the expense of good links is to group these good link sequences together into link bundles, with the aim of running the above pruning process on the link bundles. To do this, rules 1, 2, and 3 have to be redefined slightly to account for the behavior of linked bundles rather than individual links, as discussed in the following paragraphs.

The original definition of Rule 1 required that no two links could share the same text word. In the context of link bundles, this means that no link in one link bundle can share the same text word as a link in another text bundle. As a result, the link bundles b ₁ and b ₂ shown in FIG. 12 violate Rule 1. Specifically, the conflicting links shown in bold violate Rule 1. If the pruning process is applied to bundles b1 and b2, the entire linked bundle will be removed, so that non _- bold links (and conflicting links in removed bundles ₎ will be removed. If bundle b ₁ is considered first, the pruning process will decide to remove bundle b ₂ . In fact, the best result is to remove the two bold links in b2, and the surviving links form a new bundle, as shown by b _n in Figure 13. More generally, bundle b _n preferably contains all links in b ₂ that do not violate rule 1.

A bundle that violates rule 1 can look like this. If b is a link bundle, then b _from , an integer representing the index of the first word in the text contained in link bundle b, bto, an integer representing the index of the matching word in the translation, and b _size , representing the number of links in the bundle , then linking bundles b and b' violates R1 if:

maX(b _from +b _size ，b' _from +b' _size )-min(b _from ，b' _from )＜b _size +b' _size

Figure 18 schematically shows why this expression identifies a linked bundle that violates Rule 1. The left side of the above expression is shown in the upper part of FIG. 18 . The first link bundle b is shown as a rectangle containing a series of squares, each square representing a link in link bundle b (or in other words, each square representing a word in the text from which the link in link bundle b extends) . Link bundle b starts from the left side of the graph (representing index b _from ) and extends to the right by a distance (square) that represents the number of links in link bundle b (b _size ) (that is, represented by the number of links in bundle b by the number of consecutive words in the linked text). Therefore, the link bundle b ends up with b _from +b _size . The second link bundle b' (also shown as a rectangle) starts from b' _from and extends to the right by a distance that represents the number of links in link bundle b'(b' _size ). Therefore, the link bundle b' ends up with b' _from +b' _size . Therefore, the maximum span of text achieved by two link bundles is from the minimum of b _from and b' _from (far left in Figure 18) to the maximum of b _from +b _size and b' _from +b' _size ( far right in Figure 18).

The right side of this expression is shown in the lower part of Figure 18. If two link bundles b and b' do not overlap (that is, if no text word is linked by a link in link bundle b and a link in link bundle b'), the minimum span of text they get will be b _size + b' _size (represented by two rectangles representing two linked bundles end to end). Therefore, if there is an overlap between two linked bundles (i.e. rule 1 is violated by both linked bundles), the left side of the expression (represented by the upper half of Figure 18) will be smaller than the right side of the expression (represented by the top half of Figure 18) the lower part of ). If there is no overlap between the two linked bundles (i.e. rule 1 is not violated by the two linked bundles), then the left side of the expression is at least as large as the right side of the expression.

Rule 2 can be modified in a similar manner to identify bundles that violate Rule 2. Chaining bundles b and b' violates rule 2 when:

max(b _to +b _size , b' _to +b _size )min(b _to , b' _to )＜b _size +b' _size

Rule 3 also extends to the context of linked bundles. Chaining bundles b and b' violates rule 3 when:

sgn(b _from -b' _from )+sgn(b _to +b _size -(b' _to +b' _size ))=0

where sgn is the sign or sign function, which is defined as sgn(i)=+1 for i>0, 0 for i=0, and -1 for i<0.

The above case detects link bundles that violate rule 3, and some cases where link bundles violate rule 1 or 2. Since the pruning process works on linked sets, it has no effect on the working of the pruning process, since adding a link to a set more than once, for example if a link is detected to violate both rule 3 and rule 1, will not cause the same link to appear in that set multiple instances of .

An example of a cross-linked bundle is shown in Figure 14. _In this example, the bundles simply intersect, and the pruning process will remove either b1 or _b2 .

Like links, it is advantageous to create the largest set of candidate link bundles. On the surface, this seems intuitive: just collect the links in L (the largest set of candidate links), associating consecutive text words and consecutive transformed words into a link bundle. However, this will not provide the largest set of link bundles. Given a link bundle b containing say 3 consecutive links l ₁ , l ₂ and l ₃ , a linked bundle containing l ₁ and l ₂ , l ₂ and l ₃ , l ₁ , l ₂ and l ₃ will also be created. This gives the largest set of candidate link bundles, each containing only consecutive links, and addresses the issues highlighted when discussing Rule 1 in the context of link bundles above. The example in Figure 12 shows two bundles b ₁ and b ₂ that violate Rule 1. The pruning process will remove b2, so rule ₁ is no longer violated. What is not shown in the example in Figure 12 (for clarity) is that other candidate link bundles, such as those shown in Figures 15 and 16, are to be considered. _The largest set of linked bundles created will include bundles b3 and _b4 . These also conflict with b ₁ (according to rule 1), so will be removed by the pruning process. However, since the largest set of candidate linked bundles is created, there will be another candidate linked bundle b ₆ (as shown in FIG. 17 , corresponding to bundle _bn of FIG. 13 ), which does not conflict with b ₁ . Therefore, the problems highlighted in R1 can be solved without any special handling.

The steps of the preferred trimming process for linked bundles can be represented as follows.

Starting from a set of candidate links L, the steps of the pruning process are:

1. Set B=createLinkBundles(L)

createLinkBundles implements the steps outlined above to create the largest set of candidate link bundles

2. Set B _remaining = {}, B _removed = {}

These collections keep track of pruned (removed) link bundles and the remaining link bundles. Initially they are empty, but they are filled as the trimming process progresses.

3. Set i=max ({b _size : b∈BB _removed })

Find the size of the largest bundle so that the original pruning process can be run from the largest bundle, then the next largest bundle, and so on...

4. Set B _i = {b: b _size == i, b ∈ {BB _removed }}

5. Set B _{remaining_i} = trim(B _i )

This is the original pruning process described above, applied to linked bundles, using rules 1, 2, and 3 for linked bundle expansion as described above.

6. Set B _remaining =B _remaining ∪B _{remaining_i} and B _removed =B _removed ∪(B _i -B _{remaining_i} )

B _{remaining_i} is the set of linked bundles that survived pruning; these need to be added to B _remaining . B _{i-Bremaining_i} is the set of pruned linked bundles; these need to be added to B _removed .

7. For each linked bundle b in B _remaining

7.1. Set B _conflicting =conflicts(b,{b': _{b'∈B∧b'size} <i})

conflicts(b, B) returns the subset of B that contains link bundles that conflict with b - violating at least one of rules 1, 2, or 3, modified for link bundles.

7.2. Set B _removed =B _removed ∪B _conflicting

Any bundle b'∈B that collides with b and has size < i should also be removed. The main purpose of link bundles is to get rid of noisy links, so larger link bundles will remain in preference to any smaller link bundles that conflict with them.

8. If i==0

stop

otherwise

Go to step 3.

If i==0, the pruning process ends. There are no linked bundles of size 0. Otherwise, there may also be linked bundles that are not considered for pruning.

FIG. 19 illustrates the method of FIG. 2 applied to the non-fluent reading example illustrated and described in the context of FIG. 1 . Readers attempt to read aloud the text "Comealong, Paddington. We'll take you home and you can have a nice hot bath" from Michael Bond's "Please Take Care of This Bear." An index is provided in the left column of the figure, which can be compared with the text to be read (with the words in the middle column) and the output of the StT conversion software (with the words, including various suggested by the StT software for each utterance) Alternative words, in the right column of the figure) are used in combination.

An initial set of links has been created that connects the words in the text with the words in the translation. Each link connects a word in the text to a word or a set of alternate words in the translation, represented by arrows in the diagram. The link set is the largest, since a link is created whenever a word in the text matches one of the alternative words in the output of the StT software. For example, the word "Come" at index 1 in the text to be read is linked to the word groups "come", "came", "Kim" and "cam" at index 1 in the StT software output, not the StT software output any other word or group of words in , because the word "come" does not appear elsewhere in the output of the StT software. However, the word "take" at index 8 in the text to be read is connected by a link to the group of words at index 8 and index 11 in the StT output because the word "take" appears in two such positions in the StT software output . Likewise, the word "you" at index 12 in the text to be read is linked to the word groups at indices 9, 12, and 15 in the StT output, since each of these word groups contains the word "you".

As mentioned above, although shown as arrows in pictograms in the figures, links may actually be represented and/or stored in other ways. For example, a given link l can be represented as a pair of values (i, j) indicating the start and end of the link, i.e. the index of the word in the text to be read and the word in the output of the StT software connected by the link or The index of the word group. For example, a link connecting the word "hot" and the word groups "hot", "hope", "hop", "hawks", "hotdogs", etc. in the text to be read can be represented as (20, 20) because it The word at index 20 in the text to be read is concatenated with the word group at index 20 in the output of the StT software.

When the maximum link set has been created, a pruning process is applied to the maximum link set in an attempt to reduce the link set to as close as possible to fluent reading of the text to be read. The pruning process involves applying the largest set of links to Rules 1-3 discussed above in an attempt to identify and remove links that cause poor read through.

In the first case, the sets L _remaining and L _removed are empty, and the set L _unprocessed contains all 19 links shown in Figure 19 (links (1,1), (2,2), (6,6) Wait). Since L _unprocessed is not empty, the link (1,1) is chosen as the first link to be tested. No link conflicts with link (1,1), so the set L _conflicting = {} (ie the empty set). Therefore, set _{cl = 0 and cmin =} 0 (since there are no conflicting links, the cost of keeping these links is 0). Since c _l =c _min , the set L _removed ={} and L _remaining =L _remaining ∪{(1,1)}. Then remove (1,1) from L _unprocessed , the remaining 18 links are still pending.

Therefore, after applying the pruning process to the link (1,1), it is kept in L _remaining and removed from L _unprocessed . The same applies when pruning the links (2,2), (6,6) and (7,7).

Next, select the link (8,8) (still in L _unprocessed ). The set L _conflicting = {(8, 11)} because this link conflicts with link (8, 8) according to rule 1. Set c _l = 1 and c _min = 4 (since (8,11) conflicts with link (8,8) according to rule 1 and links (12,9), (13,10) and (15,9 according to rule 3) ) conflict - a total of four conflicts). In this case, c _l <c _min . So the set L _removed = L _removed ∪{(8,11)} and L _remaining =L _remaining ∪{(8,8)}={(1,1),(2,2),(6,6),( 7,7), (8,8)}. Remove (8,8) from L _unprocessed .

Take (12,9) as the next element to consider for L _unprocessed . The set L _conflicting = {(12,12), (12,15) and (15,9)}, since (12,9) conflicts with (12,12) and (12,15) according to rule 1, according to rule 2 Conflicts with (15,9). set c _l = 3 (since (12,9) has three conflicts in total), and c _min = 5 (keep linking (12,12) cost 5 because it is related to (12,9) and (12) according to rule 1 ,15) conflicts with (15,12) according to rule 2, and with (13,10) and (15,9) according to rule 3; keeping (12,15) costs 9 because it according to R1 Conflicts with (12,9) and (12,12), with (15,15) according to R2 and with (13,10), (13,13), (14,14), (15,9) according to R3 conflicts with (15,12); keeping (15,9) costs 7 because it conflicts with (12,9) according to R2, and with (13,10), (12,12), (13, 13), (14,14), (15,12) and (15,15) conflict). In this case, c _l <c _min . Hence the set L _removed = {(8,11)}∪{(12,12),(12,15),(15,9)} and L _remaining ={(1,1),(2,2),( 6,6), (7,7), (8,8)}∪{(12,9)}. Remove (12,9) from L _unprocessed . L _unprocessed still contains elements.

...

Figure 20 shows the links remaining in L _remaining or L _unprocessed at this point in the process.

Take (13,10) as the next element to consider for L _unprocessed . The set L _conflicting = {(13, 13)} because (13, 10) conflicts with (13, 13) according to rule 1. Setting cl = ₁ (because (13,10) only conflicts with (13,13)) and _cmin = 2 (retaining (13,13) costs 2 because it conflicts with (13,10) according to rule 1 and conflict with (15,12) according to the rules). In this case, c _l <c _min . Hence the set L _removed = {(8,11), (12,12), (12,15), (15,9)}∪{(13,13)} and L _remaining ={(1,1),( 2,2), (6,6), (7,7), (8,8), (12,9)}∪{(13,10)}. Remove (13,10) from L _unprocessed . L _unprocessed still contains elements.

Take (14,14) as the next element to consider for L _unprocessed . The set L _conflicting = {(15, 12)} because (14, 14) conflicts with (15, 12) according to rule 3. Setting cl = ₁ (because (14,14) only conflicts with (15,12)) and _cmin = 2 (retaining (15,12) costs 2 because it conflicts with (15,15) according to rule 1 and conflict with (14,14) according to rule 3). In this case, c _l <c _min . Hence the set L _removed = {(8,11), (12,12), (12,15), (15,9), (13,13)}∪{(15,12)} and L _remaining ={( 1,1), (2,2), (6,6), (7,7), (8,8), (12,9), (13,10)}∪{(14,14)}. Remove (14,14) from L _unprocessed . L _unprocessed still contains elements.

The remaining unprocessed links have no conflicting links, so when the remaining elements in L _unprocessed are processed, no other links are removed. After the last iteration of the pruning process using the link (20, 20), L _removed = {(8, 11), (12, 12), (12, 15), (15, 9), (13, 13), (15,12)} and L _remaining = {(1,1), (2,2), (6,6), (7,7), (8,8), (12,9), (13, 10), (14, 14), (15, 15), (16, 16), (17, 17), (18, 18), (20, 20)}. When L _unprocessed = L-{L _remaining ∪ L _removed }={}, the pruning process is terminated. Figure 21 shows the links in L _remaining when the pruning process is complete.

Figure 22 shows the same example of text and StT transforms output using linked bundles. Figure 22 shows the largest set of linked bundles. Link bundles that contain multiple links are represented by large dashed arrows. Links contained in link bundles appear as unfilled hollow arrows. Link bundles that contain only one link appear as simple black arrows. There are a total of 9 linked bundles in Figure 22 - 4 are shown as large arrows and 5 are shown as black line arrows. For clarity, link bundles are not explicitly labeled in the figure. Link bundles will be identified using the notation (j,k,m), representing a link bundle containing m links, where the first contained link goes from text word j to transition (StT output) word k. Therefore, the link bundle at the top of Figure 22 (the one containing the links (1,1) and (2,2)) is identified as (1,1,2) because the first link in the link bundle links index 1 The text word ("come") at index 1 and the translation word ("come", "came", "Kim", "cam") at index 1, and the link bundle contains a total of 2 links.

The process of applying the above-described link bundle trimming process to the example shown in FIG. 22 is as follows.

Set B = {b _1,1,2 ,b _6,6,3 ,b _8,11,1 ,b _12,9,2 ,b _12,15,1 ,b _1212,7 ,b _15,9,1 ,b _15,12,1 ,b _20,20,1 ,…}. B will include other linked bundles that make up the sub-bundles listed here. For example, in addition to b1,1,2, there are also b1,1,1 and b2,2,1. However, for simplicity, these beamlets are not listed in the steps above or below.

First, the sets B _remaining and B _removed are empty. Set i=7 (the size of the largest bundle, in this example b _12,12,7 ). B _i is the set of linked bundles of size i. In the example shown, b _12,12,7 is the only instance of a linked bundle that satisfies i=7. Therefore, for i=7, the set B _i ={b _12,12,7 }. B _{remaining_i} is the result of running the link pruning process on B _i . Since B _i contains only one bundle, nothing will be trimmed due to the link trimming process. Therefore, the set B _remaining = B _remaining ∪ B _{remaining_i} = {b _{12, 12, 7} } and B _removed = {}.

Next, for each b in B _remaining (in this case only b _12,12,7 ), determine B _conflicting : B _conflicting = {b _12,9,2 ,b _12,15,1 ,b _{15 ,9,1} ,b _15,12,1 } (because b _12,12,7 conflicts with b _12,9,2 and b _15,9,1 according to rule 2 and with b _12,15,1 and according to rule 1 b _15,12,1 conflict). Set B _removed =B _removed ∪B _conflicting ={b _12,9,2 ,b _12,15,1 ,b _15,9,1 ,b _15,12,1 }. Figure 23 shows the link bundles remaining at this point in the link bundle pruning process.

Next, set i=3 (the second largest link bundle size remaining in B - B _removed ). The set B _i = {b _6,6,3 }, since there is only one bundle of size 3 left. The set B _{remaining_i} = {b _6,6,3 }. For b _6,6,3 , B _conflicting is {b _8,11,1 } (because b _6,6,3 only conflicts with b _8,11,1 according to rule 2). Set B _removed ={b _12,9,2 ,b _12,15,1 ,b _15,9,1 ,b _15,12,1 }∪{b _8,11,1 }={b _8,11,1 ,b _12,9,2 ,b _12,15,1 ,b _15,9,1 ,b _15,12,1 }. FIG. 24 shows the remaining link bundles at this time.

Since there are no remaining conflicting link bundles to remove, further iterations of the link bundle trimming process do not result in any further changes to the link bundles. Finally, after testing the linked bundles of size 2 and 1 for collision (i=0), the linked bundle pruning process will stop.

In the linked case shown in Figure 21 and the linked bundle case shown in Figure 24, the result is both the word in the reference text (eg the text the user is reading) and the StT software translation (eg, the output of the software, which has been interpreted A set of correlations between words in an input signal from a microphone or other device that detects sounds produced by the reader).

The trimming process and method described above makes it possible to provide various feedbacks about the user's speech. For example, the pruning process and method makes it possible to track the reader's current reading position based on the StT transition, and in such a way that it can tolerate large amounts of non-fluent reading. This enables features such as highlighting the reader's current location on the screen. For example, if a user is reading text displayed on a display (such as a computer screen, phone screen, tablet screen, or other display whose output can be controlled), the display may be highlighted, underlined, or otherwise displayed as the user is reading Visually indicate the user's current location in the text. For example, the display can highlight the current word, line, sentence or paragraph depending on user preferences and specific circumstances. This helps readers to be consistent in their reading process so they don't get lost in the text. This can also help ensure that the speaker says all the words accurately. For example, if it is determined by the StT software that the speaker did not say a particular word accurately (e.g., the speaker pronounced it incorrectly or the user missed the word), highlighting may will not exceed the word. This can help improve the speaker's ability to read text aloud with confidence and accuracy.

The pruning process and method also makes it possible to track how many words were read correctly by the reader in a given reference text, and to identify words that were not read correctly. For example, pruning processes and methods can be used to maintain an ordered list of unlinked or intermittently linked words. The list can be sorted by the proportion of the number of times the word is not linked to the number of times the word is encountered in reading, giving an error rate (either reader-generated or software-generated). In some implementations, the list may only contain words with an error rate above a threshold.

The trimming process and method can differentiate reader errors from errors in the StT software listed above. For example, if a word first appears in the list, and the reader is known to have read the word correctly many times before (e.g., the reader has read it many times and read it correctly because the word already appears in the text No error), it may be an error in the StT software. This is especially true for common short words like "the," "and," and "you" that are sometimes ignored by StT software. New words that appear in the list of unlinked words and that the reader has never encountered before, or unlinked words that are already in the list, may be misreadings, and will be put into the reader's "practice" list. When the reader has practiced and pronounced a word in the list correctly, the word will move down the list.

The pruning process and method may result in metrics such as "words read/words read" (where the number of correct words is based on the number of links remaining after pruning and/or the number of links pruned), and "words read per minute", eg Based on the timing information in the conversion book provided by the StT software. Such timing information can be used to spot hesitations of silence, and/or to identify that punctuation was correctly interpreted by the reader - thus in speech when encountering things such as ",", ".", ":" Introduce appropriate pauses.

The trimming process and method can be performed on a mobile phone, tablet, laptop, desktop, other personal computing device, or any device that can receive signals or data representing user-generated sounds and processors that can process said signals or data. implemented in the software that executes on it. For example, the processor may receive data output by speech-to-text conversion software. The data may have been generated by the processor itself based on signals generated by a microphone or other voice input device. The device preferably includes a display with which the user can obtain substantially instantaneous feedback on his or her reading of the reference text.

The pruning process and method may alternatively or additionally use information about the frequency with which words in the reference text are linked to the first of "alternative words" for a given utterance in the translation to generate metrics. The pruning process may utilize weighted values, such as indices in an array of alternative words that identify where a particular match was found. In this case, the weight value can be set so that the smaller the weight, the greater the confidence of the StT software on the match. A weight of 0 may, for example, indicate a reference word linked to the first of the alternative words. This could potentially give the speaker an indication of the accuracy of his or her pronunciation.

Embodiments of the invention may be performed by a device including a processor and a memory. In this context, a processor may be any device or structure capable of executing instructions that cause the above-described processes and methods to be performed. The term "processor" is intended to include any suitable type of processor architecture. Similarly, in this case, the memory may be capable of storing data, whether temporarily or permanently, to enable the above-described processes and methods to be implemented. The term "memory" is intended to include any suitable type of memory, including both volatile and nonvolatile types of memory.

Claims (16)

1. A method of analyzing speech, comprising: receiving an estimate made by speech-to-text software of a reader's oral reading of at least a portion of the reference text, the estimate comprising a plurality of estimated glyphs, each estimated glyph representing at least one grapheme; comparing the estimated glyph to a plurality of reference glyphs representing at least a portion of the reference text, each reference glyph representing at least one grapheme; and Feedback related to the reader's speech is provided based on the comparison of the estimated glyph to the reference glyph. 2. The method of claim 1, wherein each reference glyph represents at least one word in the reference text, and each estimated glyph represents a fraction of a word estimated by the speech-to-text software as a word spoken by a reader. Group one or more alternate words or phrases. 3. The method of claim 1 or 2, wherein comparing the estimated glyph to the reference glyph comprises: link each estimated glyph to any matching reference glyph to produce multiple links; identify conflicts between said links; and The links are pruned by removing some conflicting links to resolve the identified conflicts. 4. The method of claim 3, wherein identifying a conflict between the links comprises identifying a link that violates at least one of a set of rules, the set of rules comprising: (1) A reference glyph may not be linked to more than one estimated glyph; no reference glyph can have two links (2) An estimated glyph may not be linked to more than one reference glyph; and (3) No two links can cross each other. 5. The method of claim 3 or 4, wherein pruning the links comprises: selecting a first link and identifying a set of links that conflict with the first link; determining the cost of maintaining each link in the identified set of conflicting links, the cost including the number of links that conflict with each link in the set; and The conflicting links are removed except for the lowest cost conflicting link. 6. The method of claim 3, further comprising generating a plurality of link bundles from the plurality of links, each link bundle comprising one or more links, the one or more links forming a continuation with a reference glyph a contiguous sequence of estimated glyphs whose sequence matches; and where: Identifying conflicts between links includes identifying conflicts between linked bundles; and Pruning the links includes removing some conflicting link bundles to resolve the identified conflicts. 7. The method of claim 6, wherein pruning the links further comprises maintaining larger link bundles in preference to smaller link bundles to resolve the identified conflicts. 8. The method of any of claims 3 to 7, wherein each link includes a pair of indices identifying the estimated glyph and a matching reference glyph. 9. The method of any preceding claim, wherein the feedback comprises one or more of: at least one parameter indicative of the reader's speech fluency; and At least one representation of the reader's current reading position in the reference text. 10. An apparatus comprising a processor and a memory, wherein the memory is configured to: receiving an estimate made by speech-to-text software of a reader's oral reading of at least a portion of the reference text, the estimate comprising a plurality of estimated glyphs, each estimated glyph representing at least one grapheme; comparing the estimated glyph to a plurality of reference glyphs representing at least a portion of the reference text, each reference glyph representing at least one grapheme; and Feedback related to the reader's speech is provided based on the comparison of the estimated glyph to the reference glyph. 11. The apparatus of claim 10, further comprising a microphone for detecting the voice of the reader's spoken reading and for generating an output signal to the speech-to-text software. 12. An apparatus according to claim 10 or 11, configured to perform the method according to any of claims 1 to 9. 13. A computer program comprising instructions which, when executed by a processor, cause the processor to: comparing a plurality of estimated glyphs corresponding to estimates made by the speech-to-text software of a reader's spoken reading of at least a portion of a reference text to a plurality of reference glyphs representing at least a portion of the reference text, each estimated glyph and a reference glyph representing at least one grapheme; and Feedback related to the reader's speech is provided based on the comparison of the estimated glyph to the reference glyph. 14. The computer program of claim 13, comprising instructions which, when the program is executed by a processor, cause the processor to perform the method of any one of claims 1 to 9. 15. A non-transitory computer-readable storage medium encoded with instructions that, when executed by a processor, cause the processor to: comparing a plurality of estimated glyphs corresponding to estimates made by the speech-to-text software of a reader's spoken reading of at least a portion of a reference text to a plurality of reference glyphs representing at least a portion of the reference text, each estimated glyph and a reference glyph representing at least one grapheme; and Feedback related to the reader's speech is provided based on the comparison of the estimated glyph to the reference glyph. 16. The non-transitory computer-readable storage medium of claim 15 encoded with instructions that, when executed by a processor, cause the processor to perform the performance of any one of claims 1 to 9 Methods.

Images