WO2016117564A1 - Program, information storage medium, and recognition device - Google Patents

Abstract

Provided is a recognition device or the like which is capable of recognizing mathematical formulae including character strings. This recognition device includes: a registration unit which determines whether inputted strokes or stroke strings correspond to a specific gesture for demarcating a character string and a mathematical formula, and, in cases when the inputted strokes or stroke strings are determined to correspond to the specific gesture, registers information related to the range specified by the specific gesture; a demarcation unit which sets, as strokes corresponding to the character string, the strokes included in the range specified by the specific gesture among the inputted strokes, and sets, as strokes corresponding to the mathematical formula, the strokes not included in the range specified by the specific gesture; and a recognition unit which uses a character-string-recognition engine to recognize the character string from the strokes set as the strokes corresponding to the character string, and uses a mathematical-formula-recognition engine to recognize the mathematical formula from the stokes set as the strokes corresponding to the mathematical formula.

Description

Program, information storage medium and recognition device

The present invention relates to a program, an information storage medium, and a recognition device.

Conventionally, as a method of inputting a mathematical formula, a method using a specialized language such as LaTeX or a method using a mathematical formula input editor has been used. However, the former needs to learn technical terms, and the latter requires menu selection and symbols. It was necessary to repeat the selection, both of which were unusable for ordinary people. On the other hand, a method for recognizing a mathematical formula handwritten on a tablet or the like by a computer has been studied for a long time. The recognition rate has been improved by many years of research, and the possibility of penetrating the general public is increasing due to the rapid spread of tablet PCs in recent years.

JP 2009-80615 A

Mathematical expressions consist of the relationship between symbols and the position and size of symbols representing the four arithmetic operators, parentheses, fractional symbols and roots, integral symbols, and powers. In a handwritten mathematical expression recognition system, it is normal to assume an alphanumeric symbol as a symbol. However, in practice, a character string other than an alphanumeric symbol is often written as a symbol. There was a problem that the character string contained in was not recognized. Formulas containing these strings often appear in textbooks and are often written on a blackboard.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a program, an information storage medium, and a recognition device that can recognize mathematical expressions including character strings. is there.

(1) The present invention is a program for recognizing a mathematical expression including a character string from a handwritten input stroke string, and the input stroke or stroke string is a specific gesture for distinguishing a character string from a mathematical expression. A registration unit for registering information regarding a range specified by the specific gesture and a specified by the specific gesture among the input strokes when it is determined that the specific gesture is satisfied. A category in which a stroke included in the range to be set is set as a stroke corresponding to one of the character string and the mathematical expression, and a stroke not included in the range specified by the specific gesture is set as a stroke corresponding to the other of the character string and the mathematical expression Text and the stroke set as the stroke corresponding to the character string. It recognizes the sequence, a program for causing a computer to function the formula as recognition unit for recognizing a mathematical expression recognition engine from the set stroke as a stroke which corresponds to the formula. The present invention also relates to an information storage medium that can be read by a computer and stores a program for causing the computer to function as each of the above-described units. The present invention also relates to a recognition device including the above-described units.

According to the present invention, it is determined whether or not the input stroke corresponds to the specific gesture, the stroke included in the range specified by the specific gesture is set as a stroke corresponding to one of the character string and the mathematical expression, A stroke that is not included in the range specified by the specific gesture is set as a stroke corresponding to the other of the character string and the mathematical expression, and the character string is recognized by the character string recognition engine from the stroke set as the stroke corresponding to the character string. By recognizing the mathematical expression from the stroke set as the stroke corresponding to the mathematical expression by the mathematical expression recognition engine, the character string included in the mathematical expression and the mathematical expression can be clearly divided and recognized. A method of automatically separating (classifying) mathematical expressions and character strings without using specific gestures is also conceivable. However, since it is difficult to automatically distinguish between mathematical expressions and character strings, and erroneous division is inevitable, the present invention introduces a specific gesture for distinguishing between character strings and mathematical expressions.

(2) In the program, the information storage medium, and the recognition device according to the present invention, the classification unit is designated by the specific gesture when it is determined that the input stroke or stroke sequence corresponds to the specific gesture. When the stroke included in the range is set as a stroke corresponding to one of the character string and the mathematical expression, and it is determined that the input stroke does not correspond to the specific gesture, the input stroke is specified by the specific gesture It is possible to determine whether the input stroke is included in the range specified by the specific gesture, and the input stroke may be set as a stroke corresponding to one of the character string and the mathematical expression.

According to the present invention, even when writing a specific gesture after writing a character string (or mathematical expression), even when writing a character string (or mathematical expression) after writing a specific gesture, Character strings and mathematical formulas can be distinguished, and user convenience can be improved.

(3) In the program, the information storage medium, and the recognition device according to the present invention, the sorting unit sets a stroke included in a range specified by the specific gesture as a stroke corresponding to a character string. And set a stroke that is not included in the range specified by the specific gesture as a stroke corresponding to the mathematical expression, and the recognition unit uses a stroke included in the range specified by the one specific gesture as one symbol. It may be handled and a mathematical expression is recognized by a mathematical expression recognition engine.

According to the present invention, a stroke included in a range specified by one specific gesture (a stroke set as a stroke corresponding to a character string) is treated as one symbol, and a mathematical expression is recognized by a mathematical expression recognition engine. A mathematical expression including a character string can be recognized using a mathematical expression recognition engine that cannot recognize the character string.

(4) In the program and the information storage medium according to the present invention, the computer further functions as a display control unit that performs control to display the input stroke sequence on the display unit, and the display control unit corresponds to the specific gesture. Then, control may be performed to display the determined stroke or stroke sequence as an image representing a shape corresponding to the specific gesture.

The recognition apparatus according to the present invention further includes a display control unit that performs control to display the input stroke sequence on the display unit, and the display control unit displays the stroke or the stroke sequence determined to correspond to the specific gesture. The image may be displayed as an image representing the shape corresponding to the specific gesture.

According to the present invention, it is possible to cause the user to recognize that the specific gesture has been correctly determined by displaying the stroke determined to correspond to the specific gesture as an image representing the shape corresponding to the specific gesture. Convenience can be improved.

(5) In the program, the information storage medium, and the recognition device according to the present invention, the display control unit does not include a stroke included in a range specified by the specific gesture within a range specified by the specific gesture. You may control to display with a color different from a stroke.

According to the present invention, the stroke included in the range specified by the specific gesture is displayed in a color different from the stroke not included in the range specified by the specific gesture, whereby the character string and the mathematical expression are displayed by the specific gesture. It is possible to make the user recognize that it is correctly classified, and to improve the convenience for the user.

(6) In the program, the information storage medium, and the recognition device according to the present invention, when the input stroke or stroke sequence forms a substantially rectangular shape, the registration unit corresponds to the specific gesture. Then, it may be determined.

FIG. 1 is an example of a functional block diagram of the recognition apparatus of the present embodiment. FIG. 2A is a diagram illustrating an example of writing in which mathematical expressions, character strings, and specific gestures are mixed. FIG. 2B is a diagram illustrating an example of the specific gesture table. FIG. 3A is a diagram for explaining the order in which specific gestures are written. FIG. 3B is a diagram for explaining the order in which specific gestures are written. FIG. 3C is a diagram for explaining the order in which the specific gesture is written. FIG. 3D is a diagram for explaining the order in which specific gestures are written. FIG. 4A is a diagram illustrating an example of writing of a rectangular specific gesture. FIG. 4B is a diagram illustrating an example of writing of a rectangular specific gesture. FIG. 5 is a flowchart illustrating an example of processing in the sequential method. FIG. 6A is a diagram illustrating a specific display example in the sequential method. FIG. 6B is a diagram illustrating a specific display example in the sequential method. FIG. 6C is a diagram illustrating a specific display example in the sequential method. FIG. 6D is a diagram illustrating a specific display example in the sequential method. FIG. 6E is a diagram illustrating a specific display example in the sequential method. FIG. 6F is a diagram illustrating a specific display example in the sequential method. FIG. 7 is a flowchart showing an example of processing in the batch method. FIG. 8A is a diagram illustrating a specific display example in the batch method. FIG. 8B is a diagram illustrating a specific display example in the batch method. FIG. 8C is a diagram illustrating a specific display example in the batch method.

Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Configuration FIG. 1 shows an example of a functional block diagram of the recognition apparatus of the present embodiment. In addition, the recognition apparatus of this embodiment is good also as a structure which abbreviate | omitted a part of component (each part) of FIG.

The character input unit 160 is for the user to input handwritten characters with a writing medium (pen, fingertip, etc.), and the function can be realized by a writing surface such as a tablet or a touch panel. The character input unit 160 detects coordinate data representing the position of the writing medium from when the writing medium touches the writing surface until it leaves, and strokes (strokes) the detected coordinate data string (coordinate point series). Is output to the processing unit 100. A vector from the end point of the stroke to the start point of the next stroke is called an off stroke (handwriting vector), and a series of strokes and off strokes is called a stroke sequence.

The storage unit 170 stores a program and various data for causing the computer to function as each unit of the processing unit 100, and also functions as a work area of the processing unit 100. The function can be realized by a hard disk, a RAM, or the like.

The display unit 190 outputs the image generated by the processing unit 100, and its function can be realized by a display such as a touch panel, LCD, or CRT that also functions as the character input unit 160.

The processing unit 100 (processor) performs processing such as recognition processing and display control based on the coordinate data and program from the character input unit 160. The processing unit 100 performs various processes using the main storage unit in the storage unit 170 as a work area. The functions of the processing unit 100 can be realized by hardware such as various processors (CPU, DSP, etc.), ASIC (gate array, etc.), and programs. The processing unit 100 includes a registration unit 110, a sorting unit 112, a recognition unit 114, and a display control unit 120.

The registration unit 110 determines whether or not the input stroke or stroke sequence corresponds to a specific gesture for distinguishing a character string from a mathematical expression, and determines that the input gesture corresponds to the specific gesture. A process for registering information related to the range (area) specified by. The registered information is stored in the storage unit 170. Here, when the input stroke or stroke sequence forms a substantially rectangular shape, the registration unit 110 may determine that the stroke or stroke sequence corresponds to the specific gesture.

The sorting unit 112 sets a stroke included in the range specified by the specific gesture among the input strokes as a stroke corresponding to one of the character string and the mathematical expression, and is included in the range specified by the specific gesture. A stroke that does not exist is set as a stroke corresponding to the other of the character string and the mathematical expression. That is, the sorting unit 112 sets a stroke included in the range specified by the specific gesture among the input strokes as a stroke corresponding to the character string, and is not included in the range specified by the specific gesture. May be set as a stroke corresponding to the mathematical formula, or vice versa.

The recognition unit 114 recognizes a character string by the character string recognition engine from the stroke set as a stroke corresponding to the character string by the classification unit 112, and uses the mathematical expression recognition engine from the stroke set by the classification unit 112 as a stroke corresponding to the mathematical expression. The process of recognizing the mathematical formula is performed. Here, when the classification unit 112 sets a stroke included in the range specified by the specific gesture as a stroke corresponding to a character string, the recognition unit 114 is specified by one specific gesture and the specific gesture. The stroke included in the range may be handled as one symbol, and the mathematical expression may be recognized by the mathematical expression recognition engine. For example, in the example shown in FIG. 2A, the character strings “area”, “height”, and “width” are treated as one symbol such as “x”, “y”, and “z”, respectively, and applied to the mathematical expression recognition engine. It may be recognized.

The display control unit 120 performs control to display the input stroke sequence on the display unit 190. Here, the display control unit 120 may perform control to display the stroke or the stroke sequence determined to correspond to the specific gesture as an image representing a shape corresponding to the specific gesture (for example, an image representing a rectangle). Good. Further, the display control unit 120 may perform control to display a stroke included in a range specified by the specific gesture in a color different from a stroke not included in the range specified by the specific gesture. In addition, the display control unit 120 performs control to display a mathematical expression (recognition result) including a character string recognized by the recognition unit 114 on the display unit 190.

2. Next, the method of this embodiment will be described with reference to the drawings.

When letting a computer recognize mathematical formulas containing handwritten character strings, ideally, the mathematical formulas and character strings are automatically separated from the input stroke sequence (handwritten pattern) and recognized. Is preferred. However, in the first place, it is difficult to automatically classify mathematical formulas and character strings, and if an attempt is made to recognize mathematical formulas and character strings on incomplete classification, the recognition rate may be significantly reduced.

Therefore, in the method of this embodiment, a gesture for distinguishing a character string and a mathematical expression is introduced. This is called a specific gesture. Here, the specific gesture means an operation of specifying a stroke included in a range specified by the gesture and performing a process corresponding to the stroke other than a character to be recognized or a symbol other than a mathematical expression.

In the method of this embodiment, a rectangle surrounding a character string is adopted as the specific gesture. That is, when a stroke (or stroke sequence) input by handwriting forms a substantially rectangular shape, it is determined that the stroke corresponds to the specific gesture, and the character string is recognized from the stroke included in the range specified by the specific gesture. The mathematical expression is recognized from the stroke not included in the range specified by the specific gesture. The reason for adopting the rectangle as the shape of the specific gesture is that the rectangle does not appear in an ordinary mathematical formula, and it is natural for humans to write the rectangle to specify the range. Note that squares and rectangles may be used as symbols in certain areas of mathematics. In this case, symbols having a different shape may be adopted as specific gestures instead of rectangles. In other words, the specific gesture may be any shape as long as the shape is unique (easily distinguishable from other symbols and symbols) and the range indicated by the specific gesture is clear. For example, the shape of the specific gesture may be one that writes one or both of the diagonal lines of the rectangle from right to left (because writing from left to right makes it indistinguishable from “X”), or both of the diagonal lines of the rectangle What is continuously written (a shape in which “α” is inverted) may be used. Here, a specific gesture is used to distinguish a character string included in a mathematical expression, but a specific gesture (for example, a rectangle surrounding the mathematical expression) may be used to distinguish a mathematical expression included in the character string.

FIG. 2A is a diagram illustrating an example of writing in which mathematical expressions, character strings, and specific gestures are mixed. Each time a stroke is input, the registration unit 110 monitors whether or not the stroke is a specific gesture. When the registration unit 110 determines that the stroke is a specific gesture, the registration unit 110 determines the stroke order of the specific gesture (the stroke of the stroke corresponding to the specific gesture). (Order) and coordinates (rectangular rectangular coordinates or rectangular opposing rectangular coordinates) are registered in the specific gesture table. FIG. 2B is a diagram showing a specific gesture table corresponding to the writing example of FIG. 2A. For example, in the specific gesture table shown in FIG. 2B, the specific gesture BG ₁ is written in the fifth stroke after the 4-stroke character string “area”, and the coordinates of the upper left corner and the lower right corner (with the specific gesture). An example of information relating to the designated range) is registered, and for the specific gesture BG ₂ is written in the 13th screen after the first “x”, and the coordinates of its upper left corner and lower right corner are registered, The specific gesture BG ₃ is written in the 18th screen after the second “x”, and the coordinates of its upper left corner and lower right corner are registered. The classification unit 112 refers to the specific gesture table as illustrated in FIG. 2B, classifies the stroke surrounded by any specific gesture as a part of the character string (stroke corresponding to the character string), and other than that The stroke is divided as a part of the formula (stroke corresponding to the formula).

In the example shown in FIG. 2A, the case where the specific gesture is written in one stroke is shown, but it may be allowed to write the specific gesture in multiple strokes. In this case, for the convenience of processing, the maximum number of strokes (for example, 4 strokes) in which a specific gesture is written is specified, and the stroke corresponding to a mathematical expression or character is written from the start of writing the specific gesture to the end of writing. Subject to not writing. Under such a condition, there is no big restriction for the user. When it is allowed to write a specific gesture in a plurality of strokes, the stroke order of the first stroke and the stroke order of the last stroke constituting the specific gesture are registered in the specific gesture table. The “adjusted stroke order” in the specific gesture table of FIG. 2B will be described later.

In the method according to the present embodiment, a process for distinguishing a character string from a mathematical expression is executed regardless of the timing at which a specific gesture is written. This is because if the order in which the specific gesture is written is specified, it becomes a great restriction for the user, and the usability becomes poor. As shown in FIG. 3, there are four cases in which the specific gesture is written. FIG. 3A shows a case where a specific gesture BG is first written, and then a character string is written in the specific gesture BG. FIG. 3B first writes a character string, and then, FIG. 3C shows a case where a specific gesture BG surrounding the character string is written, and FIG. 3C shows a case where the character string is first written and then another stroke is written, and then the specific gesture BG surrounding the character string is written. FIG. 3D shows a case where a character string is added to a specific gesture BG surrounding a character string already written.

Fig. 4 shows an example of writing a rectangular specific gesture. When writing the specific gesture BG, the user starts writing from the upper left corner of the rectangle, proceeds clockwise (see FIG. 4A) or counterclockwise (see FIG. 4B), and returns to the upper left corner. When the specific gesture includes at least a part (for example, 1/3 or more) of brushstrokes constituting the input stroke, the sorting unit 112 is surrounded by the specific gesture (specified by the specific gesture). To be included in the range to be included. This is because the specific gesture written by the user may cross another stroke (see FIG. 3).

In the method of this embodiment, in order to recognize a specific gesture with high accuracy, an online recognition method that uses time-series information of a stroke and an offline recognition method that uses a feature as an image of a stroke are used together to recognize the specific gesture. . The recognition target when recognizing a specific gesture is a mathematical expression (a symbol such as an English character, an arithmetic operator, an operator such as a fractional symbol, a root or an integral symbol, a parenthesis) and a specific gesture (a surrounding rectangle).

In the online recognition method, MRF (Markov Random Field) model is used. First, the input pattern is normalized to a standard size, and feature points such as end points and corners are extracted by the Ramer-Douglas-Peucker method. Then, expansion / contraction matching is performed with the feature points of the MRF model. For the score evaluation, a threshold is used, and if it is smaller than the threshold, the score is rejected.

In the offline recognition method, MQDF (Modified Quadrantic Discriminant Function) is used. The stroke can be easily converted into an image by connecting the strokes in chronological order. Although the time-series information is lost when the stroke is converted into an image, the influence of the stroke order difference, overwriting, or the like can be eliminated by converting it into an image. Gradient features are extracted from this image by the Sobel filter and projected in eight directions. The original image is 64 × 64 pixels and is divided into 8 × 8 sections of the same size. On the other hand, a Gaussian filter of 18 × 18 pixels is applied to blur. In this way, the direction feature close to the center greatly contributes, the contribution of the periphery becomes small, and some contribution is also received from the adjacent section, which makes it strong against misalignment. As a result, features in 8 directions are extracted for each 8 × 8 section, and 512-dimensional features are obtained as a whole. Furthermore, dimensions are compressed by Fisher's discriminant analysis to obtain 256-dimensional features. MQDF is used for identification. Furthermore, a threshold value is set, and if the distance to the rectangle of the input pattern is the shortest, it is rejected if it is larger than the threshold value.

Finally, we integrate the results of the online and offline recognition methods. Here, the online recognition method gives a logarithmic value of a probability value (a negative value, but a larger value is better), and the offline recognition method gives a distance (a smaller value is better). Therefore, it is necessary to match these scales. Therefore, the integrated score is calculated by multiplying the online recognition score by −1 to make it a positive value (thus, the smaller the better) and taking the weighted sum. The integrated score score _combination is calculated by the following equation.

Here, score _online and score _offline are the score of the online recognition method and the score of the offline recognition method, respectively. Further, w ₁ and w ₂ are integration weights, and these are optimized using a training pattern. More specifically, usually, score _online takes a value from about −10 to 0, and score _offline takes a value of 0 or more and less than 1024. That is, -score _online is about 1 / 100th of the score _offline , and in reality, an optimal value is obtained when the conditions of 0 <w ₁ <0.1 and w ₂ = 1-w ₁ are satisfied. The registration unit 110 recognizes the smallest gesture _combination as the specific gesture among the recognition targets.

3. Processing Next, a processing flow of the recognition apparatus of the present embodiment will be described. The recognition device is equipped with an interactive device that displays time-series writing points in real time and provides recognition feedback for each stroke (stroke), as in a tablet PC, and a memory for time-series writing points. There is a non-interactive device in which recording is performed with a tablet or an electronic pen, and reading and processing of the data is performed collectively by a PC or the like.

3-1. Sequential processing In the sequential method adopted in interactive devices, the user designates the language (for example, English or Japanese) for recognizing the character string, starts up the system (program), and enters the input section such as a touch panel. Start writing mathematical formulas and strings. The system recognizes a specific gesture and classifies a character string and a mathematical expression each time a stroke is written, and recognizes a character string and a numerical string and displays a recognition result after writing. FIG. 5 is a flowchart illustrating an example of processing in the sequential method.

First, the processing unit 100 acquires a stroke input to the character input unit 160 (step S10). At this time, the display control unit 120 performs control to display the acquired stroke on the display unit 190. Next, the registration unit 110 determines whether or not the latest stroke up to the maximum number of strokes (for example, 4 strokes) of the specific gesture corresponds to the specific gesture (configures the specific gesture) from the acquired stroke ( Step S12). For example, the registration unit 110 calculates an integrated score score _combination for each of a recent stroke of one stroke, a stroke sequence of two strokes, a stroke sequence of three strokes, a stroke sequence of four strokes (maximum number of strokes), When the specific _combination gives the smallest score _combination among the recognition targets, it is recognized as a specific gesture. However, for a stroke rejected by either the online recognition method or the offline recognition method, the score _combination is not calculated, and if any stroke is rejected, it is determined that the stroke does not correspond to the specific gesture.

In the case of corresponding to the specific gesture (Y in step S12), the registration unit 110 specifies the information of the specific gesture (stroke order and opposing two-dimensional coordinates) from the stroke information determined to be applicable to the specific gesture. Register in the gesture table (step S14). Here, the registration unit 110 cancels the setting when the stroke determined to correspond to the specific gesture has been previously set as a stroke corresponding to the mathematical expression. In addition, the display control unit 120 controls to display an image of a stroke determined to be a specific gesture as an image representing a rectangle (a shape corresponding to the specific gesture) of a color (for example, red) different from other strokes. I do. Thereby, it can be fed back to the user that the specific gesture has been correctly recognized.

Next, the sorting unit 112 refers to the specific gesture table, and sets a stroke included in a range specified by the registered specific gesture as a stroke corresponding to the character string (step S16). At this time, the display control unit 120 performs control to change the color of the stroke included in the range specified by the specific gesture (for example, change from black to blue). Thereby, it can be fed back to the user that the character string has been correctly classified by the specific gesture. If there is no stroke included in the range specified by the specific gesture, the process of step S16 is skipped.

When it does not correspond to the specific gesture (N in Step S12), the sorting unit 112 refers to the specific gesture table, and determines whether or not the acquired stroke is included in the range specified by any specific gesture. Is determined (step S18). When the acquired stroke is included in the range specified by any specific gesture (Y in Step S18), the sorting unit 112 sets the acquired stroke as a stroke corresponding to the character string (Step S18). S20), the display control unit 120 performs control to change the color of the acquired stroke (for example, change from black to blue). On the other hand, when the acquired stroke is not included in the range specified by the specific gesture (N in Step S18), the sorting unit 112 sets the acquired stroke as a stroke corresponding to the mathematical expression (Step S18). S22).

Next, the processing unit 100 determines whether or not there is an input of a stroke (step S24). If there is an input (Y in step S24), the processing unit 100 proceeds to the process of step S10, and writing by the user is completed. Until this is done, the processing from step S10 is repeated.

If it is determined that no stroke is input (N in step S24), the recognition unit 114 recognizes the character string by the character string recognition engine from the stroke set as the stroke corresponding to the character string in steps S16 and S20. The mathematical expression recognition engine recognizes the mathematical expression from the stroke set as the stroke corresponding to the mathematical expression in step S22 (step S26). Then, the display control unit 120 performs control for causing the display unit 190 to display the recognition result (the mathematical expression including the character string).

Here, the mathematical expression recognition engine does not understand the stroke corresponding to the specific gesture or the stroke corresponding to the character string, but there is no information on the character string surrounded by the specific gesture (that is, from the stroke information corresponding to the mathematical expression). Alone), the structure of the formula cannot be analyzed. Therefore, in the method of the present embodiment, one specific gesture and a character string surrounded by the specific gesture are treated as one symbol (one symbol in the mathematical expression), and the coordinate information and mathematical expression of the specific gesture handled as the symbol are supported. Recognize mathematical formulas from stroke information. Therefore, the stroke order of the specific gesture is adjusted. Specifically, of the first stroke constituting the specific gesture and the stroke order of the first stroke corresponding to the character string surrounded by the specific gesture, the earlier stroke order of the specific gesture is set as the stroke order after adjustment of the specific gesture. Register with. In the example illustrated in FIG. 2, “1”, “9”, and “12” are registered as the adjusted stroke order of the specific gestures BG ₁ , BG ₂ , and BG ₃ , respectively. In this case, the recognition unit 114 (the mathematical expression recognition engine) treats the specific gesture BG ₁ and the character string “area” included in the specific gesture BG ₁ as one symbol written in the first stroke by referring to the specific gesture table, The specific gesture BG ₂ and the character string “height” included therein are treated as one symbol written in the ninth stroke, and the specific gesture BG ₃ and the character string “width” included therein are written in the 12th stroke. Treated as one symbol, the structure of the mathematical formula can be analyzed from the relationship between the position and size of these symbols and the stroke corresponding to the mathematical formula.

In the sequential method, the user may make a mistake in writing, so it is desirable to have an Undo function. In this case, when the user inputs an Undo command and the latest undo stroke is a specific gesture, the specific gesture and its registration information are deleted and included in the range specified by the specific gesture. The stroke (the stroke set as the stroke corresponding to the character string) is reset as the stroke corresponding to the mathematical expression. On the other hand, if the latest undo stroke is a stroke corresponding to a character string or a mathematical expression, the stroke is simply deleted.

FIG. 6 is a diagram illustrating a specific display example in the sequential method. In the example shown in FIG. 6, the user first writes a character string “value” and a mathematical expression, and then writes a specific gesture BG ₁ surrounding the character string “value” (FIG. 6A). Then, the registration unit 110 recognizes the specific gesture BG ₁ , and the display control unit 120 displays the specific gesture BG ₁ in red as an image BI ₁ (specific gesture image) that represents a rectangle having the same size as the specific gesture BG _1. It was replaced to change the color of a character string enclosed by a particular gesture BG ₁ "value" in blue (Fig. 6B). Next, the user writes a new specific gesture BG ₂ (FIG. 6C). Then, the registration unit 110 recognizes the specific gesture BG ₂ and the display control unit 120 replaces the specific gesture BG ₂ with the specific gesture image BI ₂ (FIG. 6D). Next, when the user writes the character string “result” in the specific gesture image BI ₂ , the display control unit 120 displays the character string “result” in blue every time a stroke is acquired (FIG. 6E). ). Finally, when the user performs an operation to instruct execution of recognition, the recognition unit 114 recognizes the character string and the mathematical expression, and the display control unit 120 displays the recognition result (FIG. 6F). Here, the specific gesture images BI ₁ and BI ₂ are removed and the recognition result is displayed.

3-2. Batch Processing In the batch method employed in non-interactive devices, the user starts up the system and writes mathematical formulas and character strings on an interface such as a tablet. Then, after writing, the data of the stroke string is read by a PC or the like (recognition device), and the language for recognizing the character string is designated at that time. In the collective method, the three processes of recognition of specific gestures, classification of character strings and mathematical expressions, recognition of character strings and numerical sequences, and display of recognition results are executed after all stroke data is sent to a PC or the like. . At this time, when the input stroke is sequentially processed, the same processing as the sequential method can be performed. In the batch method, since it is not necessary to perform recognition of specific gestures and classification of character strings and mathematical expressions in real time, after recognizing all the specific gestures, character strings and mathematical expressions for other strokes are used. A method of dividing and finally recognizing them may be adopted. FIG. 7 is a flowchart showing an example of processing of the latter method in the batch method.

First, the processing unit 100 acquires the stroke sequence input to the character input unit 160 (step S30). Next, the processing unit 100 sets 1 to the variable i (step S32). Next, the registration unit 110 determines whether or not the most recent stroke from the i-th (i-th stroke) stroke to the maximum number of strokes (for example, 4 strokes) of the specific gesture corresponds to the specific gesture (step) S34).

In the case of corresponding to the specific gesture (Y in step S34), the registration unit 110 specifies the information of the specific gesture (stroke order and opposing two-dimensional coordinates) from the stroke information determined to be applicable to the specific gesture. Register in the gesture table (step S36). If it does not correspond to the specific gesture (N in step S34), the i-th stroke is set as a non-gesture stroke (stroke not corresponding to the specific gesture) (step S38).

Next, the processing unit 100 increases the value of the variable i by 1 (step S40), and whether or not the value of the variable i is n (n is the total number of strokes included in the acquired stroke sequence) or not. Is determined (step S42). When the value of the variable i is n or less (Y in step S42), the process proceeds to step S34, and the processing from step S34 is repeated until the value of the variable i exceeds n.

When the value of the variable i exceeds n (N in step S42), the processing unit 100 sets 1 to the variable i (step S44). Next, the sorting unit 112 refers to the specific gesture table to determine whether or not the i-th non-gesture stroke set in step S38 is included in a range specified by any specific gesture ( Step S46). When the i-th non-gesture stroke is included in the range specified by any specific gesture (Y in step S46), the sorting unit 112 sets the i-th non-gesture stroke as a stroke corresponding to the character string. Set (step S48). On the other hand, when the i-th non-gesture stroke is not included in the range specified by the specific gesture (N in step S46), the sorting unit 112 sets the i-th non-gesture stroke as a stroke corresponding to the mathematical expression. (Step S50).

Next, the processing unit 100 increases the value of the variable i by 1 (step S52), and determines whether or not the value of the variable i is equal to or less than m (m is the total number of non-gesture strokes) (step S54). ). When the value of the variable i is less than or equal to m (Y in step S54), the process proceeds to step S46, and the processes in and after step S46 are repeated until the value of the variable i exceeds m.

If the value of the variable i exceeds m (N in step S54), the recognition unit 114 recognizes the character string by the character string recognition engine from the stroke set as the stroke corresponding to the character string in step S48, The mathematical expression is recognized by the mathematical expression recognition engine from the specific gesture table (coordinate information of the specific gesture treated as a symbol) and the stroke set as the stroke corresponding to the mathematical expression in step S50 (step S56). Then, the display control unit 120 performs control for causing the display unit 190 to display the recognition result (the mathematical expression including the character string).

FIG. 8 is a diagram illustrating a specific display example in the batch method. In the collective method, all strokes written by the user and taken into the recognition device are displayed (FIG. 8A), and when the user performs an operation to instruct execution of recognition, recognition of a specific gesture, classification of character strings and mathematical expressions, The recognition of the character string and the numerical sequence is executed, and the recognition result is displayed (FIG. 8C). In addition, after performing recognition of a specific gesture and classification of a character string and a mathematical expression, and before performing recognition of a character string and a numerical sequence, as shown in FIG. 8B, the recognition result of the specific gesture and the classification of the mathematical expression and the character string The result may be displayed. In the example illustrated in FIG. 8B, the specific gestures BG ₁ , BG ₂ , and BG ₃ are replaced with the specific gesture images BI ₁ , BI ₂ , and BI ₃ , respectively, and the characters surrounded by the specific gestures BG ₁ , BG ₂ , and BG ₃ are used. The colors of the columns “area”, “height”, and “width” are changed to blue.

According to this embodiment, a specific gesture for dividing a character string and a mathematical expression is introduced, a stroke surrounded by the specific gesture is divided into a character string, and a stroke not surrounded by the specific gesture is divided into a mathematical expression, By recognizing each by the character string recognition engine and the mathematical expression recognition engine, it is possible to clearly distinguish the character string and the mathematical expression included in the mathematical expression. Further, when recognizing a mathematical expression, by treating the specific gesture and the stroke surrounded by the specific gesture as one symbol, the structural analysis of the mathematical expression including the character string can be performed using the mathematical expression recognition engine.

Further, according to the present embodiment, in the sequential method, when a specific gesture is written, the stroke surrounded by the specific gesture is processed into a character string (step S16 in FIG. 5), and the stroke is also written. When the stroke is surrounded by a specific gesture, the process of dividing the stroke into a character string is performed (step S20 in FIG. 5), so that the character string and the mathematical expression are divided regardless of the timing at which the specific gesture is written. It is possible to improve user convenience. That is, according to the present embodiment, when writing a character string after writing a specific gesture (FIG. 3A), writing a specific gesture after writing a character string (FIG. 3B, FIG. 3C), Even when a character string is added to the specific gesture (FIG. 3D), the process of dividing the character string and the mathematical expression can be performed.

In addition, this invention is not limited to the above-mentioned embodiment, A various change is possible. The present invention includes configurations that are substantially the same as the configurations described in the embodiments (for example, configurations that have the same functions, methods, and results, or configurations that have the same objects and effects). In addition, the invention includes a configuration in which a non-essential part of the configuration described in the embodiment is replaced. In addition, the present invention includes a configuration that exhibits the same operational effects as the configuration described in the embodiment or a configuration that can achieve the same object. Further, the invention includes a configuration in which a known technique is added to the configuration described in the embodiment.

100 processing unit, 110 registration unit, 112 sorting unit, 114 recognition unit, 120 display control unit, 160 character input unit, 170 storage unit, 190 display unit

Claims (8)

A program for recognizing a mathematical expression including a character string from a stroke sequence input by handwriting,
It is determined whether or not the input stroke or stroke sequence corresponds to a specific gesture for distinguishing a character string from a mathematical expression, and when it is determined to correspond to the specific gesture, a range specified by the specific gesture A registration unit that registers information about
Among the input strokes, a stroke included in the range specified by the specific gesture is set as a stroke corresponding to one of the character string and the mathematical expression, and a stroke not included in the range specified by the specific gesture is set as a character string. And a section to be set as a stroke corresponding to the other of the formulas;
Recognizing a character string from a stroke set as a stroke corresponding to a character string by a character string recognition engine, and causing the computer to function as a recognition unit for recognizing a mathematical formula from a stroke set as a stroke corresponding to a mathematical expression A program characterized by In claim 1,
The section is
When it is determined that the input stroke or stroke sequence corresponds to the specific gesture, the stroke included in the range specified by the specific gesture is set as a stroke corresponding to one of the character string and the mathematical expression, and the input When it is determined that the stroke does not correspond to the specific gesture, it is determined whether or not the input stroke is included in the range specified by the specific gesture, and is included in the range specified by the specific gesture. In this case, the input stroke is set as a stroke corresponding to one of the character string and the mathematical expression. In claim 1 or 2,
The section is
Among the input strokes, strokes included in the range specified by the specific gesture are set as strokes corresponding to the character string, and strokes not included in the range specified by the specific gesture are set as strokes corresponding to the mathematical expression. Set,
The recognition unit
A program which recognizes a mathematical expression by a mathematical expression recognition engine, treating a stroke included in a range specified by one specific gesture as one symbol. In any one of Claims 1 thru | or 3,
Further causing the computer to function as a display control unit that performs control to display the input stroke sequence on the display unit,
The display control unit
A program for performing control to display a stroke or a stroke sequence determined to correspond to the specific gesture as an image representing a shape corresponding to the specific gesture. In claim 4,
The display control unit
A program for performing control to display a stroke included in a range specified by the specific gesture in a color different from a stroke not included in the range specified by the specific gesture. In any one of Claims 1 thru | or 5,
The registration unit
When the input stroke or stroke sequence forms a substantially rectangular shape, it is determined that the stroke or stroke sequence corresponds to the specific gesture. A computer-readable information storage medium, wherein the program according to any one of claims 1 to 6 is stored. A recognition device for recognizing a mathematical expression including a character string from a stroke sequence input by handwriting,
It is determined whether or not the input stroke or stroke sequence corresponds to a specific gesture for distinguishing a character string from a mathematical expression, and when it is determined to correspond to the specific gesture, a range specified by the specific gesture A registration unit that registers information about
Among the input strokes, a stroke included in the range specified by the specific gesture is set as a stroke corresponding to one of the character string and the mathematical expression, and a stroke not included in the range specified by the specific gesture is set as a character string. And a section to be set as a stroke corresponding to the other of the formulas;
A recognition unit that recognizes a character string by a character string recognition engine from a stroke set as a stroke corresponding to the character string, and recognizes a mathematical expression by a mathematical expression recognition engine from a stroke set as a stroke corresponding to the mathematical expression. A recognition device.

Images