KR20170081350A - Text Interpretation Apparatus and Method for Performing Text Recognition and Translation Per Frame Length Unit of Image - Google Patents

Abstract

In the present invention, the input of the image character is processed in units of a predetermined frame, and the character interval is normalized to a predetermined size, and then the feature vector is extracted. The DNN (Deep Neural Network) / HMM (Hidden Markov Model) Model, and language model). By recognizing texts and sentences and linking them with automatic translation and speech synthesis, we can extend the range of automatic interpreter - To a character interpretation apparatus and method.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a text interpreting apparatus and method for performing character recognition and translation by extracting a feature vector in units of frames for an image text,

More particularly, the present invention relates to a character interpretation apparatus and method, and more particularly, to a character interpretation apparatus and method, in which a character interval is set after acquiring an image, normalization is performed to a predetermined size, a feature vector is extracted in units of a predetermined frame, a DNN (Deep Neural Network) (Hidden Markov Model) learning and LM (Language Model) learning to recognize characters and sentences and link them with automatic translation and voice synthesis. To a character interpretation apparatus and method capable of expanding to a character interpretation on an image.

Generally, a character recognition technology receives an image through a camera, extracts the character from the background, extracts the feature, and compares the character with the feature pattern of the previously stored character to display the text character of the most similar pattern as the recognition result. According to this general character recognition technology, character recognition rate is high and it is put into practical use in case of recognizing a license plate of a certain standard using a limited range of letters or numbers or zip code recognition.

However, in most cases, in expressing characters, it is difficult to extract coherent features even if they are the same character due to the variety of contrast, size, thickness, spacing, pasting, etc., depending on the purpose and environment of use. Which makes it difficult to put the character recognition to practical use. Especially, unlike the English alphabet, in the case of Hangul, the syllable is composed by combining the initial, neutrality, and longitudinal in the two-dimensional space.

However, in case of domestic and international travelers, it is very inconvenient not only to communicate with the local people but also to understand the characters of the country. In order to solve this inconvenience, it is required to use a camera function of a smart phone, which most people carry, to perform a character interpretation in a more convenient manner.

In some cases of conventional character recognition, character recognition and translation are carried out at the level of recognition and combination of consonant division, word recognition, and translation. Therefore, there is an urgent need to provide a character interpretation service by securing a character recognition method and a technique capable of recognizing sentences even beyond word-based character recognition.

SUMMARY OF THE INVENTION The present invention has been made in order to solve the above-mentioned problems, and an object of the present invention is to provide a character recognition method and a character recognition method, In this paper, we propose a new method for extracting feature vectors from various types of text recognition and pattern recognition and classification. In order to overcome the above problem, the size of the character interval is normalized with respect to the image input such that the processing is performed in a predetermined frame unit (for example, 20 msec) in the speech recognition processing, Which is constructed using DNN (Deep Neural Network) / HMM (Hidden Markov Model) learning and LM (Language Model) learning. The present invention provides a character interpretation apparatus and method capable of searching for a recognized network, recognizing a character or a sentence, linking it with automatic translation and voice synthesis, and expanding an existing automatic voice interpretation range to a character interpretation on an image .

The technical problems of the present invention are not limited to the above-mentioned technical problems, and other technical problems which are not mentioned can be understood by those skilled in the art from the following description.

According to another aspect of the present invention, there is provided an apparatus for recognizing a character from an image according to an embodiment of the present invention to perform a character interpretation, the method comprising: Binarizing; Setting a character interval for the characters included in the binarized interpretation target image; A normalization step of changing the character interval to a normalized size; Extracting a character feature vector for characters pasted with information of an interval between characters in the character interval of a normalized size; Recognizing and outputting a word or a sentence in a character notation system selected in advance for the character characteristic vector by using a character model and a language model generated and constructed through a learning result according to a predetermined algorithm, And restoring and outputting the spacing; Performing a post-process including repetition of spacing and spacing between the characters through reconfirmation of the spacing and spacing; Outputting the word or sentence in which the interval and the spacing are restored to the screen; Translating the spacing and the spacing into a previously set correspondence language for the restored word or sentence; And outputting the speech or the sentence or the translated word or sentence in which the interval and the spacing are restored according to the selection.

The characters included in the binarized interpretation target image include a black character in a light background color or a bright character in a black background color.

In the step of setting the character interval, the character interval of the box type determined by the maximum value and the minimum value of the row value and the maximum value and the minimum value of the column value of the area where the characters exist in the binarized interpretation target image can be set .

In the normalization step, the character interval may be normalized to a predetermined size according to an adjustment ratio determined by a ratio of a predetermined value to a difference between a maximum height value and a minimum height value of the character interval.

In the step of extracting the character feature vector, the character feature vector having a value of 1 or 0 may be extracted for each pixel of the character interval having the normalized size according to the brightness.

Recognizing and outputting the word or sentence, the character model generated and constructed through an iterative HMM (Hidden Markov Model) learning result on the character included in the image, and a word or sentence included in the image The language model created and constructed through the repeated LM (Language Model) learning results can be used.

In the step of recognizing and outputting the word or sentence, HMM learning is performed by referring to the image information stored in the image character DB and the image state recognition result information of each character interval included therein, The LM learning is performed with reference to the character model managed for the reference pattern of the character string and the word or sentence information of each character section included in the text language DB to refer to recognition of the word or sentence according to the determined grammar The language model being managed can be used.

According to another aspect of the present invention, there is provided a character interpretation apparatus for recognizing a character and performing character interpretation, the apparatus comprising: an image binarization unit for acquiring and binarizing an interpretation target image through a camera; A character segment setting unit for setting a character segment for characters included in the binarized interpretation target image; A character interval normalization unit for changing the character interval to a normalized size; A character feature vector extraction unit for extracting a character feature vector for characters pasted with information of an interval between characters in the character interval of a normalized size in frame units of a predetermined size; Recognizing and outputting a word or a sentence in a character notation system selected in advance for the character characteristic vector by using a character model and a language model generated and constructed through a learning result according to a predetermined algorithm, And a character recognition network for restoring and outputting the spacing; A post-processing unit for performing post-processing including repetition of the spacing and spacing between the characters through reconfirmation of the spacing and spacing; A character recognition unit for outputting the word or sentence in which the interval and the spacing are restored to the screen; An automatic translation unit for translating the word or sentence in which the interval and the spacing are restored into a preset language of the partner country; And a speech synthesizer for outputting the speech or words of the word or sentence or the translated word or sentence in which the interval and the spacing are restored according to the selection.

The characters included in the binarized interpretation target image may include a black character in a light background color or a bright character in a black background color.

The character interval setting unit may set the maximum value and the minimum value of the row value of the area where the characters exist in the binarized interpretation target image and the character interval of the box type determined by the maximum value and the minimum value of the column value.

The character interval normalization unit may normalize the character interval to a predetermined size according to an adjustment ratio determined by a ratio of a predetermined value to a difference between a maximum height value and a minimum height value of the character interval.

The character characteristic vector extraction unit may extract the character characteristic vector having a value of 1 or 0 according to the brightness for each pixel of the character interval of the normalized size.

The character recognition network includes a character model generated and constructed through a repetitive HMM learning result on characters included in the image, and a repetitive LM (Language Model) for a word or a sentence included in the image ) ≪ / RTI > learning result.

The character recognition network performs HMM learning with reference to image information stored in the image character DB and image state recognition result information of each character interval included in the image character DB, Manages the character model, manages the language model to be referred to in order to recognize the word or sentence according to a predetermined grammar by performing LM learning with reference to information of a word or a sentence of each character section included in the text language DB .

According to the character interpretation apparatus and method according to the present invention, the automatic interpretation range of conventional voice is expanded, and the effect is expanded to the character interpretation on the image. In particular, the procedure of various types of character characteristics and pattern recognition and classification according to character recognition is performed by extracting feature vectors in units of frames (characters) on an image, HMM learning for image character DB, LM learning, and character recognition network (constituted by parameters generated by learning), thereby recognizing the characters in a manner that does not separate them separately as in the conventional method. Can be reduced. In addition, it has an effect of easily realizing a character interpretation by linking with automatic translation and voice synthesis.

Further, according to the character interpretation apparatus and method according to the present invention, it is possible to eliminate the inconvenience that the user has to individually set the character recognition target section by automating the character segment setting. In addition, the user may manually set or change the character recognition section according to the user's needs, if an error occurs in the character section setting by miscellaneous. On the other hand, since there is a contrast relationship between the background color and the character color (for example, a black character in a light background color and a bright character in a black background color), it is considered in the character interval setting process, And the background can be identified.

In addition, according to the apparatus and method for translating a character according to the present invention, the size of the character feature vector can be made constant through character segment normalization, thereby eliminating the complexity due to extraction of various types of character feature vectors such as character size and spacing .

According to the character interpretation apparatus and method according to the present invention, when extracting character characteristic vectors, it is possible to distinguish syllables, words, and phrases in a character recognition network search and post- Thereby preventing an excessive feature parameter generation and an increase in the amount of calculation in the character recognition process.

Further, according to the apparatus and method for translating a character according to the present invention, HMM learning using an image character DB and LM learning using a text language DB to generate parameters necessary for character recognition and building a character recognition network, It is possible to effectively replace various feature parameter generation, pattern recognition, and classification according to the character recognition.

In addition, according to the apparatus and method for translating a character according to the present invention, a character interpretation result can be realized through automatic translation and speech synthesis in the language of the correspondent station including the sentence, Interpreter resources can be utilized effectively.

In addition, according to the apparatus and method for translating a character according to the present invention, a synthesized voice of a character recognition word and a sentence can be output as needed, so that a voice of a partner station can be heard and learned.

1 is a block diagram of a character interpretation apparatus included in a user terminal for character interpretation based on a character recognition according to an embodiment of the present invention.
FIG. 2 is a flow chart for explaining an automatic procedure and procedure for character interval setting in the character interval setting unit of FIG. 1. FIG.
FIG. 3 is a flowchart for explaining a procedure for adjusting the size of a character segment based on a predetermined character height for standardization of character feature vector extraction in the character segment normalization unit in FIG. 1. FIG.
FIG. 4 is a flowchart for explaining a procedure of extracting a character feature vector by extracting a pixel value of height while proceeding from left to right in the character feature vector extraction unit of FIG.
5 is an exemplary view of a camera image and an information file of each character interval stored in the image character DB of FIG.
FIG. 6 is an exemplary view showing name, image number, and character information included in the file, in which information on characters included in the image stored in the text language DB of FIG. 1 is recorded in each file.
FIG. 7 is a conceptual diagram of a method for searching for a character combined with an HMM and an LM parameter for a feature vector extracted in units of frames from an image character in the character recognition network of FIG. 1. FIG.
8 is a diagram for explaining an example of a method of implementing a character interpretation apparatus according to an embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. It should be noted that, in adding reference numerals to the constituent elements of the drawings, the same constituent elements are denoted by the same reference symbols as possible even if they are shown in different drawings. In the following description of the embodiments of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the difference that the embodiments of the present invention are not conclusive.

In describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are intended to distinguish the constituent elements from other constituent elements, and the terms do not limit the nature, order or order of the constituent elements. Also, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and are to be interpreted in an ideal or overly formal sense unless explicitly defined in the present application Do not.

1 is a block diagram of a character interpretation apparatus 100 included in a user terminal 20 for character interpretation based on a character recognition according to an embodiment of the present invention. Referring to FIG. 1, the overall structure and operation sequence of the character recognition-based character interpretation by extracting the feature vector of each frame according to the present invention will be described.

The character interpretation apparatus 100 of the present invention may be embodied as a dedicated apparatus to be driven together with the camera 21 or may be embedded in a user terminal 20 equipped with a camera 21 such as a smart phone. The user terminal 20 may be a wireless terminal such as a wearable device, a tablet PC, a notebook PC, or the like capable of voice / video phone communication in addition to a smart phone, or may be a wired terminal such as a desktop PC or other communication terminal.

First, when the user can not communicate with the other party in the native language (e.g., Korean), the user executes a predetermined application program loaded on the user terminal 20 to display a character interpretation (button) 26 or voice interpretation Button) 27 so as to communicate with the other party. The voice interpreter 27 outputs the result of translation between languages of the voice input of the native language and the language of the partner station (for example, English, Japanese, English, etc.) through the screen of the user terminal 20 or the speaker 28, In this case, a detailed description of the voice interpretation 27 will be omitted.

When the user selects the character interpretation 26 on the screen of the user terminal 20 for the character interpretation based on the character recognition based on the present invention, the camera 21 is operated and the image 22 ) Appears on the screen. The user can direct the camera 21 to the predetermined image 10 and capture and acquire an image of the portion of the image 10 including the character to be interpreted, that is, the interpretation target image 29. [ For example, when a predetermined mark (e.g., +) displayed on the screen is set so as to be located at the center of the character-containing portion of the image 10 and photographed to photograph the interpretation target image 29 of a certain area Can be obtained. In the following, necessary settings and resources in the character interpreter 26 are already present in the voice interpreter 27 or are utilized if they are already set. That is, the settings and resources necessary for the character interpretation 26 or the voice interpretation 27 can be shared and utilized.

The interpretation target image 29 acquired through the camera 21 is composed of a predetermined frame unit (for example, 15 frames or the like) and processed in units of frames by the character interpretation apparatus 100 included in the user terminal 20 Character recognition 24 is performed and automatically translated 25 into a language of the other station (e.g., preset English, Japanese, etc.), and the character (or sentence) (Or sentence) translated into a voice may be synthesized and output.

1, a character interpretation apparatus 100 included in a user terminal 20 according to an embodiment of the present invention includes an image binarization unit 110, a character region setting unit 120, a character region normalization unit A character recognition vector extracting unit 140, a character recognition network 150, a post-processing unit 160, a character recognizing unit 170, an automatic translating unit 180, and a voice synthesizing unit 190.

The image binarization unit 110 binarizes the interpretation target image 29 obtained through the camera 21. [ For example, if the brightness value of each pixel exceeds a predetermined threshold value, it can be binarized by replacing it with 1, otherwise, by 0. At this time, if the interpretation target image 29 is a color image, it is converted into grayscale, and a locally threshold comparison can be performed.

The character interval setting unit 120 recognizes the character portion in the binarized interpretation target image 29 and sets a character interval (or area) so that all the characters are included (see FIG. 2). The character interval setting unit 120 can check the pixel values of the characters in the up, down, left, and right directions around the direction of the camera 21 and automatically set the corresponding character interval. By automating the setting of the character interval in this manner, it is possible to eliminate the inconvenience that the user must individually set the character recognition target section.

In addition, when there is an error in the setting of the character interval by a miscellaneous or the like, or when it is necessary for the user to directly set, it is possible that the user directly sets and changes the character recognition interval according to the selection on the predetermined menu on the screen. This can be supported by the character interval setting unit 120. [

In addition, when there is a contrast between the background color and the character color (for example, black characters in a light background color and bright characters in a black background color), when the character region setting unit 120 automatically sets a character region, You can set the character interval by identifying the character interval and background. For example, it is possible to calculate the number of pixels of the background color and the character color together, analyze the frequency thereof, and set the character interval (or area) so that all the characters are included by recognizing the character portion. That is, the smaller frequency may be the character portion.

The character interval normalization unit 130 normalizes the character interval including all the characters to a predetermined size (see FIG. 3). That is, since the size of characters on the interpretation target image 29 can be variously obtained, the size is changed to a predetermined normalized size. By thus normalizing the character segment size in response to the diversity of the character size, the character segment is adjusted to a constant size based on the height of the character, so that the size of the character feature vector can be uniformly standardized, The complexity due to various types of character feature vector extraction can be solved.

The character characteristic vector extracting unit 140 extracts a character characteristic vector from the left (column) to the right (column) with respect to the image (normalized character interval) changed to the normalized size , And generates a character feature vector so that recognized characters are extracted without being pasted in a space (see FIG. 4). For example, a feature vector is assigned to 1 or 0 according to brightness for each pixel based on the vertical (vertical axis) of the character portion while proceeding from the left side (column) to the right side (column) with respect to the image changed to the normalized size , It is possible to extract a character feature vector for characters pasted so that recognized characters are spaced apart (or at minimum pixel intervals) by removing character spacing or spacing beyond a certain size. If we combine these feature vectors, we can check the shape of the characters like 23. The character spacing and spacing excluded by the character characteristic vector extracting unit 140 are restored through a method of recognizing only words that match the grammar according to the language model in the search process of the character recognition network 150, It is possible to further perform necessary processing such as character spacing and spacing reprocessing through reaffirmation of whether or not the character spacing and spacing are correct.

By extracting variously spaced characters based on the character feature vector in a state without space, it is possible to improve the complexity according to the diversity of character spacing and to prevent excessive feature parameter generation and increase in the amount of calculation in the character recognition process. The restoration of the spacing according to the distinction of syllables, words and phrases can be performed through a character recognition network searching and post-processing process as described below.

The character recognition network 150 includes a character model 153 generated and constructed through a repetitive HMM learning result on the characters included in the image based on the information of the image character DB 151 By using the language model 154 generated and constructed through the repetitive LM learning results for the languages such as the words and sentences included in the image based on the information of the text language DB 152 (see Fig. 6) (Words or sentences) are recognized and output to the character characteristic vector generated so that the extracted characters are extracted in a state in which they are pasted without spaces. The character notation 155 mode can be appropriately selected by the user. For example, it is assumed that the recognition unit syllable in the Hangul character notation conforms to the romanization notation for convenience of character processing.

In this manner, HMM learning is performed using the image character DB 151 and LM learning is performed using the text language DB 152 to generate parameters necessary for character recognition and building the character recognition network 150, It is possible to effectively substitute various feature parameter generation and pattern recognition and classification according to the pattern.

In order to recognize from the character-based character recognition to the sentence, the model parameter is generated through the extraction of the feature vector of each frame and the HMM learning and the LM learning, and is reflected in the configuration of the character recognition network 150. The character modeling by the HMM is constructed by obtaining a character model 153 (or a reference pattern) that is a reference by using the character feature vector extracted in the learning step. In the actual character recognition step, a Viterbi algorithm The pattern matching unit 140 compares the extracted pattern with the corresponding characters of the feature vector from the character feature vector extracting unit 140 to find the most similar pattern and outputs the same as the character recognition result. At this time, the LM language model 154 allows only words and sentences matching the predetermined grammar to be recognized. By using the LM language model 154 in the search process of character recognition, it is possible to reduce the search space and increase the probability of a word or a sentence matching the grammar, thereby contributing to the improvement of the character recognition rate. In addition, in order to extend beyond word-based character recognition to sentence recognition, syllables, words, and phrases must be distinguished efficiently, and a large-scale language model 154 is required for this. In addition, DNN (Deep Neural Network) learning results can be used to construct the character model 153, but the HMM model will be used as a reference here.

On the other hand, the character characteristic vector extracting unit 140 generates a character characteristic vector so that the recognized characters (words or sentences) are extracted without being spaced apart. However, the character characteristic vector extracting unit 140 searches the character recognition network 150 and the post- Character spacing and spacing can be restored.

Accordingly, the character recognition unit 170 restores the syllable, word, and phrase from the recognized characters (word or sentence) through the character recognition network 150, and displays the text-refined word or sentence 24 on the screen .

The automatic translation unit 180 automatically translates the translated text or sentence 25 into a language (eg, preset English, Japanese, English, etc.) of the text-refined word or sentence 24 You can print on the screen.

The speech synthesis unit 190 can output a speech to the text-refined word or sentence 24 output from the character recognition unit 170 on the speaker of the source language of the local station by selecting a predetermined menu on the screen, Similarly, by selecting a predetermined menu on the screen, the translated character or sentence 25 can also be output to the speaker.

In this way, the character recognition result including the sentence can be realized by automatic translation and voice synthesis in the correspondent language, and it can be utilized as an effective voice-based automatic interpretation resource by maximizing the environment setting for this. In addition, by making it possible to output a synthesized voice of a character recognition word and a sentence as necessary, it is possible to listen to and learn a voice of a partner station language.

FIG. 2 is a flowchart for explaining an automatic procedure and procedure for character interval setting in the character interval setting unit 120 of FIG.

2, the character interval setting unit 120 first sets the character interval (or area) so that all characters are included in the binarized interpretation target image 29, 21) It is confirmed that the direction point b (point where a predetermined mark (e.g., +) displayed on the screen exists) is on the area of the characters (S10) (The largest row value) is calculated and set (S11). At this time, if a character having a larger row value is not detected up to a predetermined distance while confirming whether the pixel value at the lower end of the character is background or character while confirming a predetermined width to the right and left of the center axis of the character area, The row value of the point detected as the row value can be set as the maximum value (c) of the height.

Next, the character segment setting unit 120 identifies an area in which the characters are present while proceeding left and right at the upper end of the characters at the bending point (b) of the camera 21. If no character having a smaller row value is detected up to a predetermined distance, The row value of the point detected as the minimum row value is set as the minimum value d of height (S12).

Then, in the same manner, the character segment setting unit 120 sets the maximum value (e) (S13) and the minimum value (f) (S14) of the width (column value) of the region where the characters exist.

Thus, the character region setting unit 120 recognizes the character portion in the binarized interpretation target image 29 and determines the maximum value (c) and the minimum value (d) of the row value of the region in which the characters exist, And a box-shaped character interval (or area) defined by the maximum value (e) and the minimum value (f) of the column value.

In the character interval setting process, the maximum value and the minimum value of the background color portion are set so that the contrast of the background color and the character color is set differently from that of the normal case (the bright character in the black background color) At the same time. At this time, if the background color portion is determined as an area in which characters exist, the character interval may be set to the maximum value and the minimum value thereof.

FIG. 3 is a flowchart for explaining a procedure for adjusting the size of a character segment based on a predetermined character height for standardization of a character feature vector extraction in the character segment normalization unit 130 in FIG.

As described above, when a character interval (a in FIG. 3) including all the characters in the area in which characters exist is set, the character interval normalization unit 130 normalizes the corresponding character interval (a in FIG. 3) (Width min (Wmin), minimum height (Hmin)) (for example, 0,0) as the start point in the character section (a) (S20) by applying the adjustment ratio to the pixel values of all the coordinates in the character section (a), from the lower right end (the maximum width Wmax and the maximum height Hmax) The value of the coordinates of each pixel position in the normalized size adjusted by the ratio can be determined to generate the image b of the normalized size (S21).

Here, the adjustment ratio is determined by the ratio of the predetermined value T to the difference between the height maximum value Hmax and the minimum height value Hmin of the character section (a) with respect to the width and the height .

[Equation 1]

Width adjustment ratio = T * (Hmax - Hmin)

Adjustment ratio of height = T * (Hmax - Hmin)

By thus normalizing the character segment size in response to the diversity of the character size, the character segment is adjusted to a constant size based on the height of the character, so that the size of the character feature vector can be uniformly standardized, The complexity due to various types of character feature vector extraction can be solved.

FIG. 4 is a flowchart for explaining a procedure for extracting a character feature vector by extracting a pixel value of height while proceeding from left to right in the character feature vector extractor 140 of FIG.

As described above, when an image of a normalized size is generated, pixel values (1, 0) on the upper and lower (vertical axes) of each character proceed from the left (column) to the right In order to generate a character feature vector (a in FIG. 4) such that the recognized character is extracted in a pasted state, first, the character feature vector extraction unit 140 extracts a character image having a normalized size (B) From (0,0) to (c) (width maximum) to the right (column) end, to the end of each column row (d) 1 for each pixel above and below the character interval 0 " (S30).

In this way, the character characteristic vector extracting unit 140 extracts a character characteristic vector in units of upper and lower (vertical axes) of a character section, skips characters spaced apart or spaced apart by a predetermined size or more, ). By extracting the feature vector up to the end, the character feature vector can be extracted so that recognized characters are extracted without being spaced (or minimum pixel spacing) (S31).

By extracting variously spaced characters based on the character feature vector in a state without space, it is possible to improve the complexity according to the diversity of character spacing and to prevent excessive feature parameter generation and increase in the amount of calculation in the character recognition process. The restoration of the spacing according to the distinction of syllables, words and phrases can be performed through a character recognition network searching and post-processing process as described below.

5 is an exemplary view of a camera image and an information file of each character interval stored in the image character DB (database) 151 of FIG.

As shown in FIG. 5, the image character DB 151 serving as a basis for HMM learning for constructing the character model 153 includes image information (a) photographed by the camera 21 and each character interval 52, and 53 of the image state recognition result information of the region (region) in which the image is recognized. There are three character intervals in the illustrated image information (a), and the file of the image status recognition result information for each character interval includes an image name (Image_Name), a resolution for the width and height, Information such as a start point (width: X, length: Y), a width (Width), a height (Height), and a word name (Word) Here, the image information (a) may correspond to the information of the interpretation target image 29 as described above.

FIG. 6 is an exemplary view showing the name of the file, the image number, and the character information included therein, in which information on characters included in the image stored in the text language DB 152 of FIG. 1 is recorded in each file.

As shown in FIG. 6, in the text language DB 152 serving as a basis of the LM learning for constructing the language model 154, each character period included in the image information (a) Area) can be previously stored in a separate file and managed. At this time, information about the image index (00001, 00002, ..) and the word or sentence (skin care, body care, ..) is managed for each file name (1,2, Thereby making it easy to refer to the operation of the character recognition network 150.

7 is a conceptual diagram illustrating a method of searching for a character combined with an HMM and an LM parameter for a feature vector extracted in units of frames from an image character in the character recognition network 150 of FIG.

The character characteristic vector extracting unit 140 extracts a character characteristic (character string) that has been appended to the recognized characters from the left (column) to the right (column) with respect to the image (normalized character interval) 75 changed to the normalized size As the vector 74 is generated, the character recognition network 150 recognizes the characters (word or sentence) for the character feature vector 74 using the character model 153 and the language model 154, can do.

The character recognition network 150 searches the statistically optimal HMM state 73 for the character feature vector 74 using the internally combined character model 153 and the language model 154 to determine The character recognition according to the character model of the most similar pattern can be performed and the recognized characters can be corrected by referring to the language model of the word and sentence matching the grammar so as to be composed of words and sentences matching the grammar.

For example, with respect to the feature vector "goodbye " in FIG. 7, the character recognition network 150 starts character recognition in a predetermined initial state S0 of the HMM state 73, For example, in accordance with the character notation 155 mode of the romanization notation 71 for the sake of convenience, among the words starting with "an ","Quot; nyeong ", and can output a character recognition result indicating that the image contains "goodbye" (72) unless the grammar is correct.

FIG. 8 is a diagram for explaining an example of a method of implementing a character interpretation apparatus 100 according to an embodiment of the present invention. The text interpreting apparatus 100 according to an embodiment of the present invention may be implemented by hardware, software, or a combination thereof. For example, the character interpretation apparatus 100 may be implemented in the computing system 1000 as shown in FIG.

The computing system 1000 includes at least one processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, a storage 1600, And an interface 1700. The processor 1100 may be a central processing unit (CPU) or a memory device 1300 and / or a semiconductor device that performs processing for instructions stored in the storage 1600. Memory 1300 and storage 1600 may include various types of volatile or non-volatile storage media. For example, the memory 1300 may include a ROM (Read Only Memory) 1310 and a RAM (Random Access Memory)

Thus, the steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by processor 1100, or in a combination of the two. The software module may reside in a storage medium (i.e., memory 1300 and / or storage 1600) such as a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, You may. An exemplary storage medium is coupled to the processor 1100, which can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor 1100. [ The processor and the storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and the storage medium may reside as discrete components in a user terminal.

As described above, the conventional character recognition method is a method of recognizing a pattern by segmenting each one of the characters. However, according to the character interpretation device 100 according to the present invention, , 20 msec), the image input is processed in a predetermined frame unit (for example, 15 frames, etc.), the character interval is normalized to a predetermined size, the feature vector is extracted, Using DNN (Deep Neural Network) / HMM (Hidden Markov Model) learning and LM (Language Model) learning, it can recognize characters and sentences, link with automatic translation and voice synthesis, It is possible to extend the interpretation of characters on images.

In addition, by recognizing characters in such a way that they do not separate themselves, it is possible to reduce errors caused by separating the individual characters separately, and to extend them to sentence recognition by applying LM. In particular, We can overcome the limitations of existing methods of vector extraction and pattern recognition and classification.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention.

Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

In the image binarization unit 110,
The character interval setting unit 120
The character interval normalization unit 130
The character characteristic vector extracting unit 140 extracts
The character recognition network 150,
The post-
The character recognition unit 170,
The automatic translation unit (180)
The voice synthesizer 190,

Claims (1)

A method of translating a character in an apparatus for recognizing characters from an image and performing character interpretation,
Binarizing an interpretation target image obtained through a camera; Setting a character interval for the characters included in the binarized interpretation target image;
A normalization step of changing the character interval to a normalized size; Extracting a character feature vector for characters pasted with information of an interval between characters in the character interval of a normalized size;
Recognizing and outputting a word or a sentence in a character notation system selected in advance for the character characteristic vector by using a character model and a language model generated and constructed through a learning result according to a predetermined algorithm, And restoring and outputting the spacing;
Performing a post-process including repetition of spacing and spacing between the characters through reconfirmation of the spacing and spacing; Outputting the word or sentence in which the interval and the spacing are restored to the screen;
Translating the spacing and the spacing into a previously set correspondence language for the restored word or sentence; And outputting the voice or the sentence or the translated word or sentence in which the interval and the spacing are restored according to the selection.

Images