KR102405547B1 - Pronunciation evaluation system based on deep learning - Google Patents

Abstract

The present invention relates to a pronunciation evaluation system.
Pronunciation evaluation system according to the present invention includes a text providing unit that provides a sentence necessary for learning, an uttered voice collection unit that collects the voice uttered by the learner in response to the provided sentence, and converts the uttered voice into text form and converts it A text conversion unit that acquires the acquired text as an utterance, an evaluation unit that compares and analyzes the acquired utterance and the provided sentence to calculate a matching rate for the sentence, and the word and recognition that the learner continuously pronounces incorrectly using the evaluated result A misrecognized word collection unit that collects information on words that cause errors in the word and stores the collected words in a table format. and a control unit for providing a color-coded according to the utterance, and transmitting a re-speech request signal for the provided sentence when the utterance sentence and the provided sentence do not match.

Description

Pronunciation evaluation system based on deep learning

The present invention relates to a pronunciation evaluation system, and more particularly, a pronunciation evaluation that evaluates the user's pronunciation by matching the provided sentence with the voice information obtained as the user speaks by using deep learning and NLP (Natural Language Processing) technology. It's about the system.

Recently, the number of users learning foreign languages such as English using computers is increasing. In particular, programs for learning English pronunciation are increasing. When a user utters a specific word or sentence through a microphone, the utterance is analyzed and the user's pronunciation is evaluated and provided. In this case, a speech recognition technology is applied to find out the content of the user's utterance, and as an evaluation result, a feedback corresponding to a score or an evaluation level is provided to the user.

As the contents of the pronunciation learning result provided to the user, in the case of sentence pronunciation learning, only the overall score of the entire utterance is displayed in many cases. In many cases, it only indicates that it is not. In this case, when several words are uttered, such as a sentence, there is a problem in that an accurate pronunciation learning result is not provided because there is no point on the words having some problematic pronunciations.

On the other hand, there are cases where the voice recognition result is further utilized to point out and inform the user of the wrong pronunciation, but in this case, the part that is easy to make mistakes in English pronunciation of Koreans is established and knowledgeable in advance, and the wrong pronunciation is recognized through voice recognition. If the user is notified of this, there is a problem in that the pronunciation can be corrected only when information on the correct pronunciation is additionally established.

In addition to the accuracy of pronunciation itself, in some cases, a suprasegmental evaluation factor is included to measure the naturalness of speech, especially in sentence speech. It is difficult to point out the parts and provide details on how they went wrong. Here, the hypersegmental evaluation factors are items that are not segmented such as intonation, stress, and speed of speech, and the segmental factors are separable items such as sentences, phrases, syllables, words, and phonemes.

Accordingly, there is a need for a foreign language pronunciation evaluation technology that can evaluate and provide the result of pronunciation learning for a sentence even to a hypersegmental element for each predetermined word unit.

The technology that is the background of the present invention is disclosed in Korean Patent Application Laid-Open No. 10-2019-0068841 (published on June 19, 2019).

The technical problem to be achieved by the present invention is to provide a pronunciation evaluation system that evaluates the user's pronunciation by matching the provided sentences with the voice information obtained as the user speaks by utilizing deep learning and NLP (Natural Language Processing) technology. .

In the pronunciation evaluation system according to an embodiment of the present invention for achieving this technical task, a text providing unit for providing a sentence necessary for learning, a spoken voice collecting unit for collecting a voice uttered by a learner in response to the provided sentence, the above A text conversion unit that converts the spoken voice into a text form and obtains the converted text as an utterance, an evaluation unit that compares and analyzes the obtained utterance and the provided sentence to calculate a matching rate for the sentence, using the evaluated result Thus, it collects information on words that the learner continuously pronounces incorrectly and words that cause errors in recognition, and a misrecognized word collection unit that stores the collected words in a table format, and evaluates when the utterances and the provided sentences match and a control unit that digitizes the obtained results, classifies them by color according to pronunciation accuracy and stress matching, and provides them, and transmits a re-speech request signal for the provided sentences when the utterances and the provided sentences do not match.

The text conversion unit classifies the spoken voice by word, generates an utterance obtained using the classified word, and a plurality of candidate sentences similar to the utterance, and provides time information corresponding thereto for each of the classified words can do.

The evaluation unit may compare the provided sentence and the spoken sentence word by word, and calculate a matching rate according to the number of matching words.

The evaluation unit may calculate a matching rate by classifying the provided sentences and utterances in units of words, comparing each classified word, and assigning weights to the provided sentences or utterances.

The evaluation unit may give weight to an utterance when a word is correctly pronounced or a word is pronounced once more, and a weight can be given to a provided sentence when a word is incorrectly pronounced or a word is omitted.

The evaluation unit may classify the provided sentences and utterances in units of words, set the classified word as the number of a plurality of cases, and then calculate a matching rate using the set number of cases.

The misrecognized word collecting unit may extract numbers, infrequently used words, and words that the learner repeatedly incorrectly pronounces, classify them as misrecognized words, and store the classified misrecognized words in the form of a table.

The control unit acquires a waveform by using the learner's spoken voice, matches the acquired waveform with time information of the word included in the utterance to acquire the stress of the word, and calculates the acquired stress and the original word's stress. By comparison, the correct answer can be expressed in color and printed out.

As described above, according to the present invention, based on deep learning and natural language processing (NLP) technology, the user's speech level is evaluated, and the evaluated result is provided as an index, so that the user can feel a sense of achievement, and If you input the pronunciation of , it notifies the user by pointing out the location of inaccurate pronunciation in phoneme or syllable units, and at the same time, detailed analysis of pronunciation accuracy, stress, intonation, speed, etc. is possible, improving learning efficiency.

In addition, according to the present invention, it is possible to increase the recognition rate of the user's pronunciation by continuously updating the conversion table by monitoring the words that the user repeatedly pronounces incorrectly, and classifies the words with and without stress. Thus, it is possible to increase the matching rate between the spoken sentence and the provided sentence.

1 is a configuration diagram for explaining a pronunciation evaluation system according to an embodiment of the present invention.
2 is a flowchart illustrating a method of evaluating pronunciation using a pronunciation evaluation system according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating step S210 shown in FIG. 2 .
FIG. 4 is a view for explaining step S220 shown in FIG. 2 .
FIG. 5 is an exemplary diagram illustrating a state in which an utterance is converted and outputted in step S240 shown in FIG. 2 .
FIG. 6 is a view for explaining a method of calculating a matching rate by the third method in step S250 shown in FIG. 2 .
7 is an exemplary view showing a state in which the result derived in step S270 shown in FIG. 2 is output.
FIG. 8 is a view for explaining a method of determining the stress of a word in step S270 shown in FIG. 2 .

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In this process, the thickness of the lines or the size of the components shown in the drawings may be exaggerated for clarity and convenience of explanation.

In addition, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to intentions or customs of users and operators. Therefore, definitions of these terms should be made based on the content throughout this specification.

Hereinafter, a pronunciation evaluation system according to an embodiment of the present invention will be described in more detail.

1 is a configuration diagram for explaining a deep learning-based pronunciation evaluation system according to an embodiment of the present invention.

As shown in FIG. 1 , the pronunciation evaluation system 100 according to an embodiment of the present invention includes a text providing unit 110 , an spoken voice collecting unit 120 , a text converting unit 130 , an evaluation unit 140 , It includes a misrecognized word collection unit 150 and a control unit 160 .

First, the text providing unit 110 provides a sentence necessary for learning. The sentences provided at this time may be lines from movies or dramas, dialogues or music lyrics necessary for daily life.

The spoken voice collecting unit 120 collects the voices uttered by the learner in response to the provided sentences. The learner turns on the microphone function included in the terminal. And, by reading along the text provided to the microphone, the learner terminal collects the voice uttered by the learner.

The text converter 130 converts the collected spoken voice into a text form and obtains it.

The evaluation unit 140 compares the spoken sentence with the sentence provided for learning to obtain a matching rate. The evaluation unit 140 builds a deep learning model, and inputs an utterance sentence and a sentence provided for learning into the built deep learning model. Then, the deep learning model classifies the spoken sentence and the sentence provided for learning into word units, and then matches the classified words with each other to obtain a matching rate.

The misrecognized word collecting unit 150 collects information on words that the learner continuously pronounces incorrectly and words that cause errors in recognition by using the evaluated result, and stores the collected words in a table form.

Finally, when the utterance sentence and the provided sentence match, the controller 160 quantifies the evaluation result, classifies the result by color according to pronunciation accuracy and stress match, and provides it. When the utterance sentence and the provided sentence do not match, the to transmit a re-ignition request signal.

Hereinafter, a method of evaluating pronunciation using the pronunciation evaluation system according to an embodiment of the present invention will be described in more detail with reference to FIGS. 2 to 8 .

FIG. 2 is a flowchart illustrating a method for evaluating pronunciation using a pronunciation evaluation system according to an embodiment of the present invention, FIG. 3 is a diagram illustrating step S210 illustrated in FIG. 2, and FIG. 4 is S220 illustrated in FIG. It is a diagram for explaining the steps, and FIG. 5 is an exemplary view showing a state in which the output is converted into an utterance in step S240 shown in FIG. 2 , and FIG. 6 is a matching rate by the third method in step S250 shown in FIG. 2 . It is a view for explaining a method of calculating It is a diagram for explaining a method of judging.

First, the learner downloads an application that can evaluate pronunciation and activates it. Then, as shown in FIG. 3 , the text providing unit 110 provides a learnable sentence ( S210 ).

The learner may select any one sentence from among the provided sentences, or may select sequentially from the first sentence.

When the sentence selection is completed, the control unit 160 receives information on the gender of the learner selected by the learner and the intonation of a specific country to be evaluated (S220).

As shown in FIG. 4 , the present invention enables the evaluation to be made in response to the learner's selected language method, that is, according to a method selected from among British and American styles.

Then, when the learner records the provided sentence, the spoken voice collecting unit 120 receives the learner's spoken voice generated during the recording (S230).

The text conversion unit 130 converts the received spoken voice into a text form (S240).

And, as shown in FIG. 5 , the converted utterance is immediately output through the application.

In other words, the text conversion unit 130 classifies the spoken voice for each word, and obtains the utterance by using the classified word. In this case, the text conversion unit 130 additionally generates a plurality of candidate sentences in addition to the spoken sentence.

For example, when the provided sentence is "I am hungry", the text conversion unit 130 converts a voice uttered in response to the provided character to "I am hungry", and other "I was hungry", "I Create a candidate sentence like "am angry".

When step S250 is completed, the evaluation unit 140 calculates a matching rate by comparing the spoken sentence and the provided sentence with each other (S250).

The evaluation unit 140 calculates the matching rate by any one of three methods.

First, the first method will be described. The evaluation unit 140 compares the provided sentence and the utterance word by word, and calculates a matching rate according to the number of matching words.

For example, assuming that the provided sentence is “I want to go home” and the utterance is “I want to go home”, the evaluation unit 140 calculates a matching rate of 100% for the corresponding utterance. .

On the other hand, when the utterance is “I go home”, the evaluation unit 140 calculates a matching rate of 20% for the utterance.

Next, referring to the second invention, the evaluation unit 140 classifies the provided sentences and utterances in units of words, compares each classified word, and assigns weights to the provided sentences or utterances to calculate a matching rate.

For example, the provided sentence is classified into words as "I", "want", "to", "go", and "home" as "I want to go home", and the utterance corresponding to the provided sentence is It is assumed that "I go home" is classified in word units as "I", "go", and "home".

Then, the evaluation unit 140 compares the words one by one and gives weight to the spoken sentence when the word is correctly pronounced or the word is pronounced once more, and gives weight to the provided sentence when the word is incorrectly pronounced or omitted to calculate the matching rate.

Finally, when the third method is described, the evaluation unit 140 classifies the provided sentences and utterances in units of words. And, as shown in FIG. 6 , the evaluation unit 140 sets the number of classified words as a plurality of cases, and then calculates a matching rate using the set number of cases.

Next, the misrecognized word collecting unit 150 extracts unrecognized words or words for which the learner fails to pronounce from the evaluated utterances, and stores the extracted words in the form of a table ( S260 ).

Since the misrecognized word collecting unit 150 is not properly recognized and has a high probability of being converted incorrectly for numbers that are difficult to recognize, words that are not frequently used, or words that express human emotions, the misrecognized word collecting unit 150 extracts misrecognized words and stores the extracted words in the form of a table.

The reason for storing the misrecognized words in a table form is to achieve an accurate recognition rate when analyzing them as incorrect or correct answers compared to the spoken text.

Finally, the controller 160 outputs the analysis result for the utterance having a high matching rate (S270).

As shown in FIG. 7 , the control unit 160 analyzes and outputs pronunciation accuracy, stress, and a degree similar to intonation according to country.

At this time, as shown in FIG. 8 , the controller 160 acquires a waveform using the learner's uttered voice, and acquires the stress of the word by matching the acquired waveform with time information of the word included in the utterance.

Meanwhile, the controller 160 may determine the stress of a word according to the strength of the learner's voice.

As described above, according to the present invention, based on deep learning and natural language processing (NLP) technology, the user's speech level is evaluated, and the evaluated result is provided as an index, so that the user can feel a sense of achievement, and If you input the pronunciation of , it notifies the user by pointing out the location of inaccurate pronunciation in phoneme or syllable units, and at the same time, detailed analysis of pronunciation accuracy, stress, intonation, speed, etc. is possible, improving learning efficiency.

In addition, according to the present invention, it is possible to increase the recognition rate of the user's pronunciation by continuously updating the conversion table by monitoring the words that the user repeatedly pronounces incorrectly, and classifies the words with and without stress. Thus, it is possible to increase the matching rate between the spoken sentence and the provided sentence.

Although the present invention has been described with reference to the embodiment shown in the drawings, this is merely exemplary, and it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. will be. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the following claims.

100: pronunciation evaluation system
110: text providing unit
120: spoken voice collecting unit
130: text conversion unit
140: evaluation unit
150: misrecognized word collection unit
160: control unit

Claims (8)

In the pronunciation evaluation system based on deep learning and NLP (Natural Language Processing) technology,
A text providing unit that provides the sentences necessary for learning,
Speech voice collection unit for collecting the voice uttered by the learner in response to the provided sentence;
A text conversion unit that converts the spoken voice into a text form and obtains the converted text as an utterance;
An evaluation unit that compares and analyzes the acquired utterance and the provided sentence using a pre-established deep learning model to calculate a matching rate for the sentence;
A misrecognized word collecting unit that collects information on words that the learner continuously pronounces incorrectly using the evaluation results and words that cause errors in recognition, and stores the collected words in a table format; and
If the utterance and the provided sentence match, the evaluated result is digitized, and the result is classified by color according to pronunciation accuracy and stress match. It includes a control unit that
The evaluation unit,
A pronunciation evaluation system that classifies the provided sentences and utterances by word unit, compares each classified word, and assigns weights to the provided sentences or utterances to calculate a matching rate. According to claim 1,
The text conversion unit,
A pronunciation evaluation system for classifying the uttered speech for each word, generating an utterance obtained using the classified word, a plurality of candidate sentences similar to the utterance, and providing time information corresponding to each of the classified words. According to claim 1,
The evaluation unit,
A pronunciation evaluation system that compares the provided sentences and utterances word by word, and calculates a matching rate according to the number of matching words. delete According to claim 1,
The evaluation unit,
When a word is pronounced correctly or a word is pronounced once more, weight is given to the speech,
A pronunciation evaluation system that gives weight to a given sentence in case a word is pronounced incorrectly or a word is omitted. According to claim 1,
The evaluation unit,
A pronunciation evaluation system that classifies the provided sentences and utterances in units of words, sets the classified word as the number of multiple cases, and calculates a matching rate using the set number of cases. According to claim 1,
The misrecognized word collection unit,
A pronunciation evaluation system that extracts numbers, infrequently used words, and words that the learner repeatedly incorrectly pronounces, classifies them as misrecognized words, and stores the classified misrecognized words in a table format. According to claim 1,
The control unit is
Acquire a waveform using the learner's spoken voice,
A pronunciation evaluation system that matches the acquired waveform with time information of the words included in the utterance to obtain the stress of the word, compares the acquired stress with the stress of the original word, and expresses whether the correct answer is correct in color.

Images