[go: up one dir, main page]

WO2016117564A1 - Programme, support de stockage d'informations et dispositif de reconnaissance - Google Patents

Programme, support de stockage d'informations et dispositif de reconnaissance Download PDF

Info

Publication number
WO2016117564A1
WO2016117564A1 PCT/JP2016/051457 JP2016051457W WO2016117564A1 WO 2016117564 A1 WO2016117564 A1 WO 2016117564A1 JP 2016051457 W JP2016051457 W JP 2016051457W WO 2016117564 A1 WO2016117564 A1 WO 2016117564A1
Authority
WO
WIPO (PCT)
Prior art keywords
stroke
specific gesture
character string
mathematical expression
range specified
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2016/051457
Other languages
English (en)
Japanese (ja)
Inventor
正樹 中川
ミン カン ファン
ドゥック アィン レ
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tokyo University of Agriculture and Technology NUC
University of Tokyo NUC
Original Assignee
Tokyo University of Agriculture and Technology NUC
University of Tokyo NUC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tokyo University of Agriculture and Technology NUC, University of Tokyo NUC filed Critical Tokyo University of Agriculture and Technology NUC
Priority to KR1020177018253A priority Critical patent/KR101909485B1/ko
Priority to JP2016570658A priority patent/JP6694638B2/ja
Priority to CN201680006635.XA priority patent/CN107209862B/zh
Publication of WO2016117564A1 publication Critical patent/WO2016117564A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/18—Extraction of features or characteristics of the image
    • G06V30/1801—Detecting partial patterns, e.g. edges or contours, or configurations, e.g. loops, corners, strokes or intersections
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition

Definitions

  • the present invention relates to a program, an information storage medium, and a recognition device.
  • Mathematical expressions consist of the relationship between symbols and the position and size of symbols representing the four arithmetic operators, parentheses, fractional symbols and roots, integral symbols, and powers.
  • a handwritten mathematical expression recognition system it is normal to assume an alphanumeric symbol as a symbol.
  • a character string other than an alphanumeric symbol is often written as a symbol.
  • Formulas containing these strings often appear in textbooks and are often written on a blackboard.
  • the present invention has been made in view of the above problems, and an object of the present invention is to provide a program, an information storage medium, and a recognition device that can recognize mathematical expressions including character strings. is there.
  • the present invention is a program for recognizing a mathematical expression including a character string from a handwritten input stroke string, and the input stroke or stroke string is a specific gesture for distinguishing a character string from a mathematical expression.
  • a registration unit for registering information regarding a range specified by the specific gesture and a specified by the specific gesture among the input strokes when it is determined that the specific gesture is satisfied.
  • a category in which a stroke included in the range to be set is set as a stroke corresponding to one of the character string and the mathematical expression, and a stroke not included in the range specified by the specific gesture is set as a stroke corresponding to the other of the character string and the mathematical expression Text and the stroke set as the stroke corresponding to the character string.
  • the present invention also relates to an information storage medium that can be read by a computer and stores a program for causing the computer to function as each of the above-described units.
  • the present invention also relates to a recognition device including the above-described units.
  • the stroke included in the range specified by the specific gesture is set as a stroke corresponding to one of the character string and the mathematical expression
  • a stroke that is not included in the range specified by the specific gesture is set as a stroke corresponding to the other of the character string and the mathematical expression
  • the character string is recognized by the character string recognition engine from the stroke set as the stroke corresponding to the character string.
  • the classification unit is designated by the specific gesture when it is determined that the input stroke or stroke sequence corresponds to the specific gesture.
  • the stroke included in the range is set as a stroke corresponding to one of the character string and the mathematical expression, and it is determined that the input stroke does not correspond to the specific gesture, the input stroke is specified by the specific gesture It is possible to determine whether the input stroke is included in the range specified by the specific gesture, and the input stroke may be set as a stroke corresponding to one of the character string and the mathematical expression.
  • the sorting unit sets a stroke included in a range specified by the specific gesture as a stroke corresponding to a character string. And set a stroke that is not included in the range specified by the specific gesture as a stroke corresponding to the mathematical expression, and the recognition unit uses a stroke included in the range specified by the one specific gesture as one symbol. It may be handled and a mathematical expression is recognized by a mathematical expression recognition engine.
  • a stroke included in a range specified by one specific gesture (a stroke set as a stroke corresponding to a character string) is treated as one symbol, and a mathematical expression is recognized by a mathematical expression recognition engine.
  • a mathematical expression including a character string can be recognized using a mathematical expression recognition engine that cannot recognize the character string.
  • the computer further functions as a display control unit that performs control to display the input stroke sequence on the display unit, and the display control unit corresponds to the specific gesture. Then, control may be performed to display the determined stroke or stroke sequence as an image representing a shape corresponding to the specific gesture.
  • the recognition apparatus further includes a display control unit that performs control to display the input stroke sequence on the display unit, and the display control unit displays the stroke or the stroke sequence determined to correspond to the specific gesture.
  • the image may be displayed as an image representing the shape corresponding to the specific gesture.
  • the present invention it is possible to cause the user to recognize that the specific gesture has been correctly determined by displaying the stroke determined to correspond to the specific gesture as an image representing the shape corresponding to the specific gesture. Convenience can be improved.
  • the display control unit does not include a stroke included in a range specified by the specific gesture within a range specified by the specific gesture. You may control to display with a color different from a stroke.
  • the stroke included in the range specified by the specific gesture is displayed in a color different from the stroke not included in the range specified by the specific gesture, whereby the character string and the mathematical expression are displayed by the specific gesture. It is possible to make the user recognize that it is correctly classified, and to improve the convenience for the user.
  • the registration unit corresponds to the specific gesture. Then, it may be determined.
  • FIG. 1 is an example of a functional block diagram of the recognition apparatus of the present embodiment.
  • FIG. 2A is a diagram illustrating an example of writing in which mathematical expressions, character strings, and specific gestures are mixed.
  • FIG. 2B is a diagram illustrating an example of the specific gesture table.
  • FIG. 3A is a diagram for explaining the order in which specific gestures are written.
  • FIG. 3B is a diagram for explaining the order in which specific gestures are written.
  • FIG. 3C is a diagram for explaining the order in which the specific gesture is written.
  • FIG. 3D is a diagram for explaining the order in which specific gestures are written.
  • FIG. 4A is a diagram illustrating an example of writing of a rectangular specific gesture.
  • FIG. 4B is a diagram illustrating an example of writing of a rectangular specific gesture.
  • FIG. 5 is a flowchart illustrating an example of processing in the sequential method.
  • FIG. 6A is a diagram illustrating a specific display example in the sequential method.
  • FIG. 6B is a diagram illustrating a specific display example in the sequential method.
  • FIG. 6C is a diagram illustrating a specific display example in the sequential method.
  • FIG. 6D is a diagram illustrating a specific display example in the sequential method.
  • FIG. 6E is a diagram illustrating a specific display example in the sequential method.
  • FIG. 6F is a diagram illustrating a specific display example in the sequential method.
  • FIG. 7 is a flowchart showing an example of processing in the batch method.
  • FIG. 8A is a diagram illustrating a specific display example in the batch method.
  • FIG. 8B is a diagram illustrating a specific display example in the batch method.
  • FIG. 8C is a diagram illustrating a specific display example in the batch method.
  • FIG. 1 shows an example of a functional block diagram of the recognition apparatus of the present embodiment.
  • the recognition apparatus of this embodiment is good also as a structure which abbreviate
  • the character input unit 160 is for the user to input handwritten characters with a writing medium (pen, fingertip, etc.), and the function can be realized by a writing surface such as a tablet or a touch panel.
  • the character input unit 160 detects coordinate data representing the position of the writing medium from when the writing medium touches the writing surface until it leaves, and strokes (strokes) the detected coordinate data string (coordinate point series). Is output to the processing unit 100.
  • a vector from the end point of the stroke to the start point of the next stroke is called an off stroke (handwriting vector), and a series of strokes and off strokes is called a stroke sequence.
  • the storage unit 170 stores a program and various data for causing the computer to function as each unit of the processing unit 100, and also functions as a work area of the processing unit 100.
  • the function can be realized by a hard disk, a RAM, or the like.
  • the display unit 190 outputs the image generated by the processing unit 100, and its function can be realized by a display such as a touch panel, LCD, or CRT that also functions as the character input unit 160.
  • the processing unit 100 performs processing such as recognition processing and display control based on the coordinate data and program from the character input unit 160.
  • the processing unit 100 performs various processes using the main storage unit in the storage unit 170 as a work area.
  • the functions of the processing unit 100 can be realized by hardware such as various processors (CPU, DSP, etc.), ASIC (gate array, etc.), and programs.
  • the processing unit 100 includes a registration unit 110, a sorting unit 112, a recognition unit 114, and a display control unit 120.
  • the registration unit 110 determines whether or not the input stroke or stroke sequence corresponds to a specific gesture for distinguishing a character string from a mathematical expression, and determines that the input gesture corresponds to the specific gesture. A process for registering information related to the range (area) specified by. The registered information is stored in the storage unit 170.
  • the registration unit 110 may determine that the stroke or stroke sequence corresponds to the specific gesture.
  • the sorting unit 112 sets a stroke included in the range specified by the specific gesture among the input strokes as a stroke corresponding to one of the character string and the mathematical expression, and is included in the range specified by the specific gesture.
  • a stroke that does not exist is set as a stroke corresponding to the other of the character string and the mathematical expression. That is, the sorting unit 112 sets a stroke included in the range specified by the specific gesture among the input strokes as a stroke corresponding to the character string, and is not included in the range specified by the specific gesture. May be set as a stroke corresponding to the mathematical formula, or vice versa.
  • the recognition unit 114 recognizes a character string by the character string recognition engine from the stroke set as a stroke corresponding to the character string by the classification unit 112, and uses the mathematical expression recognition engine from the stroke set by the classification unit 112 as a stroke corresponding to the mathematical expression.
  • the process of recognizing the mathematical formula is performed.
  • the classification unit 112 sets a stroke included in the range specified by the specific gesture as a stroke corresponding to a character string
  • the recognition unit 114 is specified by one specific gesture and the specific gesture.
  • the stroke included in the range may be handled as one symbol, and the mathematical expression may be recognized by the mathematical expression recognition engine.
  • the character strings “area”, “height”, and “width” are treated as one symbol such as “x”, “y”, and “z”, respectively, and applied to the mathematical expression recognition engine. It may be recognized.
  • the display control unit 120 performs control to display the input stroke sequence on the display unit 190.
  • the display control unit 120 may perform control to display the stroke or the stroke sequence determined to correspond to the specific gesture as an image representing a shape corresponding to the specific gesture (for example, an image representing a rectangle). Good.
  • the display control unit 120 may perform control to display a stroke included in a range specified by the specific gesture in a color different from a stroke not included in the range specified by the specific gesture.
  • the display control unit 120 performs control to display a mathematical expression (recognition result) including a character string recognized by the recognition unit 114 on the display unit 190.
  • a gesture for distinguishing a character string and a mathematical expression is introduced.
  • This is called a specific gesture.
  • the specific gesture means an operation of specifying a stroke included in a range specified by the gesture and performing a process corresponding to the stroke other than a character to be recognized or a symbol other than a mathematical expression.
  • a rectangle surrounding a character string is adopted as the specific gesture. That is, when a stroke (or stroke sequence) input by handwriting forms a substantially rectangular shape, it is determined that the stroke corresponds to the specific gesture, and the character string is recognized from the stroke included in the range specified by the specific gesture. The mathematical expression is recognized from the stroke not included in the range specified by the specific gesture.
  • the reason for adopting the rectangle as the shape of the specific gesture is that the rectangle does not appear in an ordinary mathematical formula, and it is natural for humans to write the rectangle to specify the range. Note that squares and rectangles may be used as symbols in certain areas of mathematics. In this case, symbols having a different shape may be adopted as specific gestures instead of rectangles.
  • the specific gesture may be any shape as long as the shape is unique (easily distinguishable from other symbols and symbols) and the range indicated by the specific gesture is clear.
  • the shape of the specific gesture may be one that writes one or both of the diagonal lines of the rectangle from right to left (because writing from left to right makes it indistinguishable from “X”), or both of the diagonal lines of the rectangle What is continuously written (a shape in which “ ⁇ ” is inverted) may be used.
  • a specific gesture is used to distinguish a character string included in a mathematical expression, but a specific gesture (for example, a rectangle surrounding the mathematical expression) may be used to distinguish a mathematical expression included in the character string.
  • FIG. 2A is a diagram illustrating an example of writing in which mathematical expressions, character strings, and specific gestures are mixed.
  • the registration unit 110 monitors whether or not the stroke is a specific gesture.
  • the registration unit 110 determines the stroke order of the specific gesture (the stroke of the stroke corresponding to the specific gesture). (Order) and coordinates (rectangular rectangular coordinates or rectangular opposing rectangular coordinates) are registered in the specific gesture table.
  • FIG. 2B is a diagram showing a specific gesture table corresponding to the writing example of FIG. 2A. For example, in the specific gesture table shown in FIG.
  • the specific gesture BG 1 is written in the fifth stroke after the 4-stroke character string “area”, and the coordinates of the upper left corner and the lower right corner (with the specific gesture).
  • An example of information relating to the designated range) is registered, and for the specific gesture BG 2 is written in the 13th screen after the first “x”, and the coordinates of its upper left corner and lower right corner are registered,
  • the specific gesture BG 3 is written in the 18th screen after the second “x”, and the coordinates of its upper left corner and lower right corner are registered.
  • the classification unit 112 refers to the specific gesture table as illustrated in FIG. 2B, classifies the stroke surrounded by any specific gesture as a part of the character string (stroke corresponding to the character string), and other than that The stroke is divided as a part of the formula (stroke corresponding to the formula).
  • the case where the specific gesture is written in one stroke is shown, but it may be allowed to write the specific gesture in multiple strokes.
  • the maximum number of strokes for example, 4 strokes
  • the stroke corresponding to a mathematical expression or character is written from the start of writing the specific gesture to the end of writing.
  • the stroke order of the first stroke and the stroke order of the last stroke constituting the specific gesture are registered in the specific gesture table.
  • the “adjusted stroke order” in the specific gesture table of FIG. 2B will be described later.
  • FIG. 3A shows a case where a specific gesture BG is first written, and then a character string is written in the specific gesture BG.
  • FIG. 3B first writes a character string, and then,
  • FIG. 3C shows a case where a specific gesture BG surrounding the character string is written, and
  • FIG. 3A shows a case where a specific gesture BG is first written, and then a character string is written in the specific gesture BG.
  • FIG. 3C shows a case where the character string is first written and then another stroke is written, and then the specific gesture BG surrounding the character string is written.
  • FIG. 3D shows a case where a character string is added to a specific gesture BG surrounding a character string already written.
  • Fig. 4 shows an example of writing a rectangular specific gesture.
  • the user starts writing from the upper left corner of the rectangle, proceeds clockwise (see FIG. 4A) or counterclockwise (see FIG. 4B), and returns to the upper left corner.
  • the specific gesture includes at least a part (for example, 1/3 or more) of brushstrokes constituting the input stroke
  • the sorting unit 112 is surrounded by the specific gesture (specified by the specific gesture). To be included in the range to be included. This is because the specific gesture written by the user may cross another stroke (see FIG. 3).
  • an online recognition method that uses time-series information of a stroke and an offline recognition method that uses a feature as an image of a stroke are used together to recognize the specific gesture.
  • the recognition target when recognizing a specific gesture is a mathematical expression (a symbol such as an English character, an arithmetic operator, an operator such as a fractional symbol, a root or an integral symbol, a parenthesis) and a specific gesture (a surrounding rectangle).
  • MRF Markov Random Field
  • MQDF Modified Quadrantic Discriminant Function
  • the stroke can be easily converted into an image by connecting the strokes in chronological order. Although the time-series information is lost when the stroke is converted into an image, the influence of the stroke order difference, overwriting, or the like can be eliminated by converting it into an image.
  • Gradient features are extracted from this image by the Sobel filter and projected in eight directions.
  • the original image is 64 ⁇ 64 pixels and is divided into 8 ⁇ 8 sections of the same size.
  • a Gaussian filter of 18 ⁇ 18 pixels is applied to blur.
  • the direction feature close to the center greatly contributes, the contribution of the periphery becomes small, and some contribution is also received from the adjacent section, which makes it strong against misalignment.
  • features in 8 directions are extracted for each 8 ⁇ 8 section, and 512-dimensional features are obtained as a whole.
  • dimensions are compressed by Fisher's discriminant analysis to obtain 256-dimensional features.
  • MQDF is used for identification.
  • a threshold value is set, and if the distance to the rectangle of the input pattern is the shortest, it is rejected if it is larger than the threshold value.
  • the online recognition method gives a logarithmic value of a probability value (a negative value, but a larger value is better), and the offline recognition method gives a distance (a smaller value is better). Therefore, it is necessary to match these scales. Therefore, the integrated score is calculated by multiplying the online recognition score by ⁇ 1 to make it a positive value (thus, the smaller the better) and taking the weighted sum.
  • the integrated score score combination is calculated by the following equation.
  • score online and score offline are the score of the online recognition method and the score of the offline recognition method, respectively.
  • the registration unit 110 recognizes the smallest gesture combination as the specific gesture among the recognition targets.
  • the recognition device is equipped with an interactive device that displays time-series writing points in real time and provides recognition feedback for each stroke (stroke), as in a tablet PC, and a memory for time-series writing points.
  • an interactive device that displays time-series writing points in real time and provides recognition feedback for each stroke (stroke), as in a tablet PC, and a memory for time-series writing points.
  • a non-interactive device in which recording is performed with a tablet or an electronic pen, and reading and processing of the data is performed collectively by a PC or the like.
  • Sequential processing In the sequential method adopted in interactive devices, the user designates the language (for example, English or Japanese) for recognizing the character string, starts up the system (program), and enters the input section such as a touch panel. Start writing mathematical formulas and strings. The system recognizes a specific gesture and classifies a character string and a mathematical expression each time a stroke is written, and recognizes a character string and a numerical string and displays a recognition result after writing.
  • FIG. 5 is a flowchart illustrating an example of processing in the sequential method.
  • the processing unit 100 acquires a stroke input to the character input unit 160 (step S10).
  • the display control unit 120 performs control to display the acquired stroke on the display unit 190.
  • the registration unit 110 determines whether or not the latest stroke up to the maximum number of strokes (for example, 4 strokes) of the specific gesture corresponds to the specific gesture (configures the specific gesture) from the acquired stroke (Ste S12). For example, the registration unit 110 calculates an integrated score score combination for each of a recent stroke of one stroke, a stroke sequence of two strokes, a stroke sequence of three strokes, a stroke sequence of four strokes (maximum number of strokes), When the specific combination gives the smallest score combination among the recognition targets, it is recognized as a specific gesture. However, for a stroke rejected by either the online recognition method or the offline recognition method, the score combination is not calculated, and if any stroke is rejected, it is determined that the stroke does not correspond to the specific gesture.
  • the registration unit 110 specifies the information of the specific gesture (stroke order and opposing two-dimensional coordinates) from the stroke information determined to be applicable to the specific gesture. Register in the gesture table (step S14).
  • the registration unit 110 cancels the setting when the stroke determined to correspond to the specific gesture has been previously set as a stroke corresponding to the mathematical expression.
  • the display control unit 120 controls to display an image of a stroke determined to be a specific gesture as an image representing a rectangle (a shape corresponding to the specific gesture) of a color (for example, red) different from other strokes. I do. Thereby, it can be fed back to the user that the specific gesture has been correctly recognized.
  • the sorting unit 112 refers to the specific gesture table, and sets a stroke included in a range specified by the registered specific gesture as a stroke corresponding to the character string (step S16).
  • the display control unit 120 performs control to change the color of the stroke included in the range specified by the specific gesture (for example, change from black to blue). Thereby, it can be fed back to the user that the character string has been correctly classified by the specific gesture. If there is no stroke included in the range specified by the specific gesture, the process of step S16 is skipped.
  • the sorting unit 112 refers to the specific gesture table, and determines whether or not the acquired stroke is included in the range specified by any specific gesture. Is determined (step S18).
  • the sorting unit 112 sets the acquired stroke as a stroke corresponding to the character string (Step S18).
  • the display control unit 120 performs control to change the color of the acquired stroke (for example, change from black to blue).
  • the sorting unit 112 sets the acquired stroke as a stroke corresponding to the mathematical expression (Step S18). S22).
  • step S24 the processing unit 100 determines whether or not there is an input of a stroke (step S24). If there is an input (Y in step S24), the processing unit 100 proceeds to the process of step S10, and writing by the user is completed. Until this is done, the processing from step S10 is repeated.
  • the recognition unit 114 recognizes the character string by the character string recognition engine from the stroke set as the stroke corresponding to the character string in steps S16 and S20.
  • the mathematical expression recognition engine recognizes the mathematical expression from the stroke set as the stroke corresponding to the mathematical expression in step S22 (step S26). Then, the display control unit 120 performs control for causing the display unit 190 to display the recognition result (the mathematical expression including the character string).
  • the mathematical expression recognition engine does not understand the stroke corresponding to the specific gesture or the stroke corresponding to the character string, but there is no information on the character string surrounded by the specific gesture (that is, from the stroke information corresponding to the mathematical expression). Alone), the structure of the formula cannot be analyzed. Therefore, in the method of the present embodiment, one specific gesture and a character string surrounded by the specific gesture are treated as one symbol (one symbol in the mathematical expression), and the coordinate information and mathematical expression of the specific gesture handled as the symbol are supported. Recognize mathematical formulas from stroke information. Therefore, the stroke order of the specific gesture is adjusted.
  • the earlier stroke order of the specific gesture is set as the stroke order after adjustment of the specific gesture.
  • “1”, “9”, and “12” are registered as the adjusted stroke order of the specific gestures BG 1 , BG 2 , and BG 3 , respectively.
  • the recognition unit 114 treats the specific gesture BG 1 and the character string “area” included in the specific gesture BG 1 as one symbol written in the first stroke by referring to the specific gesture table,
  • the specific gesture BG 2 and the character string “height” included therein are treated as one symbol written in the ninth stroke, and the specific gesture BG 3 and the character string “width” included therein are written in the 12th stroke.
  • the structure of the mathematical formula can be analyzed from the relationship between the position and size of these symbols and the stroke corresponding to the mathematical formula.
  • the user may make a mistake in writing, so it is desirable to have an Undo function.
  • the specific gesture and its registration information are deleted and included in the range specified by the specific gesture.
  • the stroke (the stroke set as the stroke corresponding to the character string) is reset as the stroke corresponding to the mathematical expression.
  • the latest undo stroke is a stroke corresponding to a character string or a mathematical expression, the stroke is simply deleted.
  • FIG. 6 is a diagram illustrating a specific display example in the sequential method.
  • the user first writes a character string “value” and a mathematical expression, and then writes a specific gesture BG 1 surrounding the character string “value” (FIG. 6A).
  • the registration unit 110 recognizes the specific gesture BG 1 , and the display control unit 120 displays the specific gesture BG 1 in red as an image BI 1 (specific gesture image) that represents a rectangle having the same size as the specific gesture BG 1. It was replaced to change the color of a character string enclosed by a particular gesture BG 1 "value" in blue (Fig. 6B).
  • the user writes a new specific gesture BG 2 (FIG. 6C).
  • the registration unit 110 recognizes the specific gesture BG 2 and the display control unit 120 replaces the specific gesture BG 2 with the specific gesture image BI 2 (FIG. 6D).
  • the display control unit 120 displays the character string “result” in blue every time a stroke is acquired (FIG. 6E).
  • the recognition unit 114 recognizes the character string and the mathematical expression, and the display control unit 120 displays the recognition result (FIG. 6F).
  • the specific gesture images BI 1 and BI 2 are removed and the recognition result is displayed.
  • FIG. 7 is a flowchart showing an example of processing of the latter method in the batch method.
  • the processing unit 100 acquires the stroke sequence input to the character input unit 160 (step S30). Next, the processing unit 100 sets 1 to the variable i (step S32). Next, the registration unit 110 determines whether or not the most recent stroke from the i-th (i-th stroke) stroke to the maximum number of strokes (for example, 4 strokes) of the specific gesture corresponds to the specific gesture (step) S34).
  • the registration unit 110 specifies the information of the specific gesture (stroke order and opposing two-dimensional coordinates) from the stroke information determined to be applicable to the specific gesture. Register in the gesture table (step S36). If it does not correspond to the specific gesture (N in step S34), the i-th stroke is set as a non-gesture stroke (stroke not corresponding to the specific gesture) (step S38).
  • the processing unit 100 increases the value of the variable i by 1 (step S40), and whether or not the value of the variable i is n (n is the total number of strokes included in the acquired stroke sequence) or not. Is determined (step S42). When the value of the variable i is n or less (Y in step S42), the process proceeds to step S34, and the processing from step S34 is repeated until the value of the variable i exceeds n.
  • the processing unit 100 sets 1 to the variable i (step S44).
  • the sorting unit 112 refers to the specific gesture table to determine whether or not the i-th non-gesture stroke set in step S38 is included in a range specified by any specific gesture (Ste S46).
  • the sorting unit 112 sets the i-th non-gesture stroke as a stroke corresponding to the character string. Set (step S48).
  • the sorting unit 112 sets the i-th non-gesture stroke as a stroke corresponding to the mathematical expression. (Step S50).
  • the processing unit 100 increases the value of the variable i by 1 (step S52), and determines whether or not the value of the variable i is equal to or less than m (m is the total number of non-gesture strokes) (step S54). ). When the value of the variable i is less than or equal to m (Y in step S54), the process proceeds to step S46, and the processes in and after step S46 are repeated until the value of the variable i exceeds m.
  • the recognition unit 114 recognizes the character string by the character string recognition engine from the stroke set as the stroke corresponding to the character string in step S48,
  • the mathematical expression is recognized by the mathematical expression recognition engine from the specific gesture table (coordinate information of the specific gesture treated as a symbol) and the stroke set as the stroke corresponding to the mathematical expression in step S50 (step S56).
  • the display control unit 120 performs control for causing the display unit 190 to display the recognition result (the mathematical expression including the character string).
  • FIG. 8 is a diagram illustrating a specific display example in the batch method.
  • all strokes written by the user and taken into the recognition device are displayed (FIG. 8A), and when the user performs an operation to instruct execution of recognition, recognition of a specific gesture, classification of character strings and mathematical expressions, The recognition of the character string and the numerical sequence is executed, and the recognition result is displayed (FIG. 8C).
  • FIG. 8B After performing recognition of a specific gesture and classification of a character string and a mathematical expression, and before performing recognition of a character string and a numerical sequence, as shown in FIG. 8B, the recognition result of the specific gesture and the classification of the mathematical expression and the character string The result may be displayed.
  • FIG. 8A all strokes written by the user and taken into the recognition device
  • FIG. 8C the recognition result is displayed.
  • FIG. 8B after performing recognition of a specific gesture and classification of a character string and a mathematical expression, and before performing recognition of a character string and a numerical sequence, as shown in FIG. 8B,
  • the specific gestures BG 1 , BG 2 , and BG 3 are replaced with the specific gesture images BI 1 , BI 2 , and BI 3 , respectively, and the characters surrounded by the specific gestures BG 1 , BG 2 , and BG 3 are used.
  • the colors of the columns “area”, “height”, and “width” are changed to blue.
  • a specific gesture for dividing a character string and a mathematical expression is introduced, a stroke surrounded by the specific gesture is divided into a character string, and a stroke not surrounded by the specific gesture is divided into a mathematical expression,
  • the character string recognition engine and the mathematical expression recognition engine By recognizing each by the character string recognition engine and the mathematical expression recognition engine, it is possible to clearly distinguish the character string and the mathematical expression included in the mathematical expression. Further, when recognizing a mathematical expression, by treating the specific gesture and the stroke surrounded by the specific gesture as one symbol, the structural analysis of the mathematical expression including the character string can be performed using the mathematical expression recognition engine.
  • the stroke surrounded by the specific gesture is processed into a character string (step S16 in FIG. 5), and the stroke is also written.
  • the process of dividing the stroke into a character string is performed (step S20 in FIG. 5), so that the character string and the mathematical expression are divided regardless of the timing at which the specific gesture is written. It is possible to improve user convenience. That is, according to the present embodiment, when writing a character string after writing a specific gesture (FIG. 3A), writing a specific gesture after writing a character string (FIG. 3B, FIG. 3C), Even when a character string is added to the specific gesture (FIG. 3D), the process of dividing the character string and the mathematical expression can be performed.
  • the present invention includes configurations that are substantially the same as the configurations described in the embodiments (for example, configurations that have the same functions, methods, and results, or configurations that have the same objects and effects).
  • the invention includes a configuration in which a non-essential part of the configuration described in the embodiment is replaced.
  • the present invention includes a configuration that exhibits the same operational effects as the configuration described in the embodiment or a configuration that can achieve the same object.
  • the invention includes a configuration in which a known technique is added to the configuration described in the embodiment.
  • processing unit 110 registration unit, 112 sorting unit, 114 recognition unit, 120 display control unit, 160 character input unit, 170 storage unit, 190 display unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Character Discrimination (AREA)

Abstract

L'invention concerne un dispositif de reconnaissance ou analogue qui est susceptible de reconnaître des formules mathématiques comprenant des chaînes de caractères. Le dispositif de reconnaissance de la présente invention comprend : une unité d'enregistrement qui détermine si des segments ou chaînes de segments entrés correspondent ou non à un geste spécifique pour délimiter une chaîne de caractères et une formule mathématique, et, dans des cas dans lesquels les segments ou chaînes de segments entrés sont déterminés comme correspondant au geste spécifique, enregistre des informations associées à la plage spécifiée par le geste spécifique ; une unité de délimitation qui règle, comme segments correspondant à la chaîne de caractères, les segments inclus dans la plage spécifiée par le geste spécifique parmi les segments entrés, et règle, comme segments correspondant à la formule mathématique, les segments non compris dans la plage spécifiée par le geste spécifique ; et une unité de reconnaissance qui utilise un moteur de reconnaissance de chaîne de caractères pour reconnaître la chaîne de caractères à partir des segments réglés comme segments correspondant à la chaîne de caractères, et utilise un moteur de reconnaissance de formule mathématique pour reconnaître la formule mathématique à partir des segments réglés comme segments correspondant à la formule mathématique.
PCT/JP2016/051457 2015-01-21 2016-01-19 Programme, support de stockage d'informations et dispositif de reconnaissance Ceased WO2016117564A1 (fr)

Priority Applications (3)

Application Number Priority Date Filing Date Title
KR1020177018253A KR101909485B1 (ko) 2015-01-21 2016-01-19 프로그램, 정보기억매체 및 인식장치
JP2016570658A JP6694638B2 (ja) 2015-01-21 2016-01-19 プログラム、情報記憶媒体及び認識装置
CN201680006635.XA CN107209862B (zh) 2015-01-21 2016-01-19 识别装置和信息存储介质

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2015-009207 2015-01-21
JP2015009207 2015-01-21

Publications (1)

Publication Number Publication Date
WO2016117564A1 true WO2016117564A1 (fr) 2016-07-28

Family

ID=56417105

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2016/051457 Ceased WO2016117564A1 (fr) 2015-01-21 2016-01-19 Programme, support de stockage d'informations et dispositif de reconnaissance

Country Status (4)

Country Link
JP (1) JP6694638B2 (fr)
KR (1) KR101909485B1 (fr)
CN (1) CN107209862B (fr)
WO (1) WO2016117564A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2019191815A (ja) * 2018-04-23 2019-10-31 国立大学法人東京農工大学 プログラム、情報記憶媒体及びクラスタリング装置
CN111539383A (zh) * 2020-05-22 2020-08-14 浙江蓝鸽科技有限公司 公式知识点识别方法及装置
JP2023510904A (ja) * 2020-01-28 2023-03-15 マイスクリプト 手書きにおける数学検出

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111079622A (zh) * 2019-12-10 2020-04-28 黄淮学院 一种在统一识别框架下小型化手写体文本识别器的方法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11203412A (ja) * 1998-01-16 1999-07-30 Toshiba Corp 文書画像処理装置、文書画像処理方法及び文書画像処理プログラムを記録したコンピュータ読み取り可能な記録媒体
JP2009080615A (ja) * 2007-09-26 2009-04-16 Univ Of Tokyo オンライン手書き数式認識方法及び装置
JP2014127188A (ja) * 2012-12-27 2014-07-07 Toshiba Corp 整形装置及び方法

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3150468B2 (ja) * 1993-01-07 2001-03-26 株式会社東芝 オンライン文字認識装置
JPH09319828A (ja) * 1996-05-30 1997-12-12 Nec Corp オンライン文字認識装置
JP3827453B2 (ja) * 1998-09-07 2006-09-27 積水樹脂株式会社 吸音構造体
US20060001667A1 (en) * 2004-07-02 2006-01-05 Brown University Mathematical sketching
JP5098504B2 (ja) * 2007-08-09 2012-12-12 富士通株式会社 文字認識プログラム、文字認識装置および文字認識方法
CN101388068B (zh) * 2007-09-12 2011-06-22 汉王科技股份有限公司 数学公式识别编码方法
CN101685497B (zh) * 2008-09-28 2011-10-12 汉王科技股份有限公司 手写信息处理方法和装置
JP2011053986A (ja) * 2009-09-02 2011-03-17 Sharp Corp 計算装置、計算プログラムおよびこれが記録された記録媒体
CN103679164A (zh) * 2012-09-21 2014-03-26 阿里巴巴集团控股有限公司 一种基于移动终端的标志识别处理方法和系统
CN103295008B (zh) * 2013-05-22 2017-04-05 华为终端有限公司 一种文字识别方法及用户终端
CN104063063B (zh) * 2014-06-05 2018-10-09 北京搜狗科技发展有限公司 输入法的数学计算输入方法和输入法系统
CN104268118A (zh) * 2014-09-23 2015-01-07 赵方 一种包括触摸屏的数学式计算器及数学公式识别的方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11203412A (ja) * 1998-01-16 1999-07-30 Toshiba Corp 文書画像処理装置、文書画像処理方法及び文書画像処理プログラムを記録したコンピュータ読み取り可能な記録媒体
JP2009080615A (ja) * 2007-09-26 2009-04-16 Univ Of Tokyo オンライン手書き数式認識方法及び装置
JP2014127188A (ja) * 2012-12-27 2014-07-07 Toshiba Corp 整形装置及び方法

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
ATSUSHI MURASE ET AL.: "A System for Recognizing Handwritten Mathematical Expressions", IPSJ SIG NOTES, vol. 91, no. 34, 9 May 1991 (1991-05-09), pages 1 - 8 *
KEISUKE MOCHIDA ET AL.: "A Method for Recognizing On-line Handwritten Patterns mixed of Text and Formulas", IEICE TECHNICAL REPORT, vol. 104, no. 741, 10 March 2005 (2005-03-10), pages 43 - 48 *
MASAKI NAKAGAWA ET AL.: "Fast Algorithms to Determine Object Inclusion within a Pen Movement Enclosure on a Display Integrated Tablet", IEICE TECHNICAL REPORT, vol. J77-D-II, 25 August 1994 (1994-08-25), pages 1630 - 1639 *
SO I ET AL.: "On Line Recognition of Handwritten Mathematical Formulae Using Stroke Relations", IEICE TECHNICAL REPORT, vol. 98, no. 489, 17 December 1998 (1998-12-17), pages 9 - 16 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2019191815A (ja) * 2018-04-23 2019-10-31 国立大学法人東京農工大学 プログラム、情報記憶媒体及びクラスタリング装置
JP7104965B2 (ja) 2018-04-23 2022-07-22 国立大学法人東京農工大学 プログラム及びクラスタリング装置
JP2023510904A (ja) * 2020-01-28 2023-03-15 マイスクリプト 手書きにおける数学検出
JP7573246B2 (ja) 2020-01-28 2024-10-25 マイスクリプト 手書きにおける数学検出
CN111539383A (zh) * 2020-05-22 2020-08-14 浙江蓝鸽科技有限公司 公式知识点识别方法及装置
CN111539383B (zh) * 2020-05-22 2023-05-05 浙江蓝鸽科技有限公司 公式知识点识别方法及装置

Also Published As

Publication number Publication date
KR101909485B1 (ko) 2018-10-18
KR20170100541A (ko) 2017-09-04
JP6694638B2 (ja) 2020-05-20
CN107209862A (zh) 2017-09-26
JPWO2016117564A1 (ja) 2017-10-26
CN107209862B (zh) 2021-03-09

Similar Documents

Publication Publication Date Title
CN110837403B (zh) 机器人过程自动化
CN110232311B (zh) 手部图像的分割方法、装置及计算机设备
US10360473B2 (en) User interface creation from screenshots
US8542928B2 (en) Information processing apparatus and control method therefor
CN113223025B (zh) 图像处理方法及装置、神经网络的训练方法及装置
US20140143721A1 (en) Information processing device, information processing method, and computer program product
US10769427B1 (en) Detection and definition of virtual objects in remote screens
US9099007B1 (en) Computerized processing of pictorial responses in evaluations
KR20180064371A (ko) 다수의 객체 입력을 인식하기 위한 시스템 및 그 방법 및 제품
US20130155026A1 (en) New kind of multi-touch input device
CN114365075A (zh) 用于选择图形对象的方法和对应装置
JP4712613B2 (ja) 情報処理装置、情報処理方法およびプログラム
JP6694638B2 (ja) プログラム、情報記憶媒体及び認識装置
JP2023153238A (ja) デジタルインク生成装置、方法及びプログラム
CN104094283B (zh) 字符切取方法、使用该方法的字符识别装置
CN111062377A (zh) 一种题号检测方法、系统、存储介质及电子设备
CN114241486A (zh) 一种提高识别试卷学生信息准确率的方法
Suganya et al. Detection and recognition of hand gestures to control the system Applications by neural networks
CN112183253B (zh) 数据处理方法、装置、电子设备及计算机可读存储介质
JP5674535B2 (ja) 画像処理装置、方法、及びプログラム
Dey et al. Algorithm For Multi-Hand Finger Counting: An Easy Approach
KR100983779B1 (ko) 도서 정보 서비스 장치 및 방법
US20240184494A1 (en) Method, electronic device, and storage medium for adjusting document style
US20150142784A1 (en) Retrieval device and method and computer program product
Chen et al. Design of Interactive Teaching System Based on Computer Vision Technology

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16740171

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 20177018253

Country of ref document: KR

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2016570658

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16740171

Country of ref document: EP

Kind code of ref document: A1