WO2008096192A1 - Methods and apparatuses for processing information in written form by means of a mobile ocr appliance - Google Patents

Abstract

The options for use of conventional mobile OCR appliances are extended in that, in a manual first action, a large image of an information item and, in a manual second action using the same appliance, an image element of the large image are recorded, and the image element is identified in the large image in order to initiate subsequent processing with specific functions. The image element can be recorded by a contact-inherent snap-shot by optics which are fitted in the interior of the housing and have autofocus, while the large image is recorded without making contact, using the same optics, but now at a reasonable distance from the original. By way of example, the processing of written numbers, simple columns of numbers, multistage columns of numbers and columns of numbers in a multiple-column arrangement is improved by means of this OCR computation machine. In this case, simple numbers and columns of numbers are preferably detected contact-inherently, and are automatically added or are added subsequently. Numbers in relatively complicated multiple-column arrangements, in contrast, are recorded without making contact by recording a large image at a reasonable distance from the written original. Instead of having to carry out the necessary processing of the multiple columns in sequence on a display which is, of course, small of a mobile OCR computation machine such as this, this processing can be initiated on the original document itself: The additional contact-inherent reading associated with this of numbers that are present there or of other information - once the still image has been stored - initiates a function, for example subsequent addition, in this OCR computation machine, by means of which the stored multiple columns are processed as desired and automatically. If it is in fact necessary at all, the display on the small screen can be restricted to the subsequently calculated sums. An OCR appliance such as this can also advantageously be used for applications in mobile OCR text recording and processing.

Description

 Method and devices for processing written information by means of a mobile OCR device

description

The invention relates to methods and devices for processing written information by means of a mobile OCR device.

Mobile OCR devices are used for a wide variety of purposes. Their application limits are described in a first example on the basis of an OCR calculating machine, as explained in EP 0 416 072.

By means of such an OCR calculating machine, a number column in writing is recorded in an uninterrupted motion along the number column, from top to bottom, or from the first to the last number of the column, and the numbers are automatically added or added.

However, such an OCR calculating machine is normally limited to the post-addition or addition of a single column of numbers arithmetically linked to a single result, the sum.

In particular from the work of accountants are present documents in writing, which not only a number column but a plurality of columns of numbers - hereinafter referred to as Multikolonnen Anordnunq - have, in which often not only a final total but also intermediate totals are included. Such columns are referred to below as multi-stage. Documents with such more complicated columns can not be processed in a simple manner by means of a known OCR calculating machine.

According to the state of the art, by contrast, such a document can be completely recorded with a single snap-shot (still picture) from a distance, then digitized, buffered and recognized by a character recognition software (OCR) and displayed on a screen for further processing.

For the post-addition (or addition) of a single column of numbers or a multi-column arrangement so that different methods applied to different devices, each having advantages in one arrangement and disadvantages in the other arrangement:

For the simple post-addition of a single column of numbers, the described OCR calculating machine is advantageously used for the post-addition of a multi-column arrangement, advantageously a digital camera connected to a computer.

Similar problems arise in other mobile applications. For example, an entire page of text is advantageously recorded with a digital camera - in contrast to a small text excerpt, for example in the form of individual lines, which is sensibly detected with an OCR reading pen.

(The detection of a multi-column arrangement or a text page or an otherwise more extensive text, among others, is hereinafter referred to as recording a large image, that of a smaller section as a recording of a partial image (of the large image).

If a partial image is now captured with a reading pen, the full image is not available. If a large image is captured, only a small display is available in a mobile application, so that the display of the large image and the search and editing of a partial image is tedious and exhausting.

Here the invention seeks to remedy this situation: It presents methods and devices with which both applications, both the recording, display and processing of large and partial images, can be carried out in a simple manner with a single device. From the combination of previously separately used devices then further inventive methods are presented.

It will first be shown below how the combination of the recording of a large and partial image can be realized according to the invention on the basis of an OCR calculating machine. For this purpose, the OCR calculating machine is equipped with optics with autofocus. As with previous OCR calculators, the focus is not fixed on the reading of the written template or its contact-inherent reading and a partial image, but can be set up at different distances. This allows the OCR calculator to record a larger still image at some distance from the original. It can also be used to record larger multi-column arrangements and then to add or add them.

In order for the user of the OCR calculating machine to still be able to read single numbers (snap-shots) or a single number column by movement of the device along the number column in a simple manner on the original (contact-inherent), the autofocus optical system must also be able to read the light source for illuminating numbers be secured inside the housing. In contrast to this, with a mobile camera, any flash lighting as well as the camera module itself are regularly found on the exterior of the device in order to ensure the widest possible recording angle. Since the optics with auto focus according to the invention is now mounted inside, although the image pickup angle is reduced, but this is still sufficient for the capture of a large image - for example in the form of a multi-column arrangement.

In Fig. 1 the inventive simple construction of such an OCR calculating machine is shown. The optics with autofocus (1) is mounted in the middle of the housing (3), the illumination (2) when taking a contact-inherent snap-shot (Figure 1.2) also inside the housing, here on a housing edge. A multicolumn arrangement (5) is received in the necessary distance (4) for presentation without contact according to FIG. 1.1. The contact-inherent detection of a single number or a column of numbers is shown in Fig. 1.2, the image to be recorded in (6). Selected for further processing is from Figure (6) that number (7), which is located closest to the front housing edge. A single column is contact-inherently detected by moving the OCR calculator along the number column on the pad (8).

It is thus possible to capture a large and partial image with one and the same OCR calculating machine by means of autofocus, without losing the main advantages of a conventional OCR computing machine with contact-inherent detection: With the contact-inherent detection of the image to be captured is greatly limited and on the relevant there Partial information of the document reduced. At most, still necessary delimitations of numbers, etc. can be made simple and reliable. The readable information can be increased by contact-inherent reading by not only a contact-inherent snap-shot is made but a contact-inherent movement is carried out on the document, as was already the case in the OCR calculator for reading number columns. This extends the limited nature of contact-inherent reading - precisely to the size required in an additional dimension.

All this is not so comfortable with a non-contact still image as it can be captured by a conventional digital camera. Additional and sometimes complex procedures have to be introduced for the necessary demarcation of image and recognition. All these additional methods have in common that a simple edition of the device on the document to be detected - not only for easy determination of the relevant part information to be read but also to relieve the hand holding the digital camera - when using the principles of a digital camera, i. non-contact, not possible.

Incidentally, such an OCR calculating machine can also be used for text recognition, unless the OCR is limited to numerical recognition.

Since the functions of such an OCR computing machine should be housed in a single handy device, is in such a calculator regularly but only a small screen available, comparable in size as a cell phone or a PDA. For the representation of a Muitikolonnen arrangement is thus a very small, if not too small area available.

The fact that with the same OCR calculating machine according to the invention now both recordings of large images as well as images of partial images are possible and considered, now opens at the same time an inventive solution of the disadvantage, a large image on a small display of a mobile device for editing to have to represent. This and how further rationalization potential due to the inventive integration of large image and sub-image capture in an OCR computing machine can be developed, is now shown in detail:

1. After the non-contact recording of a still picture, the numbers to be counted are not marked on a very small or too small screen as totals, but determined so that they are read directly from the written documents. This can be done with contact-inherent detection using a simple snap-shot on the template itself. If, for example, three totals are to be recalculated in a multi-column arrangement, these three totals are read in separately by means of one contact-inherent snap-shot (cf., for example, a simple example in FIG. 1.1, sub-image 9, in which the total 10 is recalculated in this way). , After that, the automatic post-addition or addition of these sums takes place with automated methods on the basis of the other stored, non-contact information of the large image.

Multi-stage additions in multicolumn arrangement can thus be carried out or checked in a simple manner very pleasant. All intermediate totals and totals are processed in this manner, advantageously from top to bottom, because thus an automated assignment of the summands to the totals is easily possible (see also patent application CH 0133/06.) Only if such a post-addition reveals an error , it is necessary to list the relevant numbers on a display for control and correction purposes. In all other cases, an acoustic signal is sufficient. Thus, the calculation by means of the original document is possible without a small or at most too small screen must be taken to help.

Any correct post-addition can be easily documented on the template by the examiner. Of course it is also possible to digitally save all numbers and the correctness of the postaddition.

It is obvious that editing and entering computational commands on the template itself is very clear and simple. On the Display must and can be displayed only relevant information - and thus in an optimal, easy to read size.

If a number which is to be recalculated as a sum can not be unambiguously identifiable - for example as a result of two identical numbers - these two numbers can still be displayed with their 'environment numbers' on the screen in order to enable a correct identification by the user.

2. In many cases text is also stored in a multi-column arrangement. In most cases, the text refers to a line or a column. A column is often identified by a calendar date, while a text marks a line.

For example, in a statement of financial position, the details of cash and cash equivalents (cash, postal check, various banks) and their total amount are to be arranged in three columns with each other as of 31.12.2005, 2004 and 2003. These totals are to be added. For this purpose, in a method according to the invention, the text "liquid agent" is simply read in by means of a snap-shot, as a result of which all totals of the liquid agent located on this row are added after every three years. The same can then be done for 'customer claims' and inventories, which should also be on this balance sheet. Advantageously, in the present example, these sums are stored as sums of the first stage, after which the same method for the current assets can be made as the sum of the second stage. (see, for example, patent application CH 0133/06 or PCT application AlpnaCom of 26.01.2007).

For example, only the sums of the first and second stages, ie the liquid assets, the customer receivables and the inventories and the sum of the second stage of the current assets, can be displayed on the screen, while all the other details do not appear. This makes it possible to obtain a good overview of the recalculated sums on a small screen of the OCR calculating machine. A similar process can be carried out not only for post-additions but also for additions. The procedure should be adapted to the fact that there are no totals yet.

Instead of lines, the method can also be carried out for columns which, as in the present example, represent individual years.

In particular, after the user of such an OCR calculating machine has acquired the methods of the present invention, there is nothing to prevent him from using these methods even when all the non-contact information is completely displayed on a larger screen of a personal computer. If, for example, only recalculated sums are displayed, the information is reduced to the essentials in this case as well.

3. For example, in a multi-column arrangement, the OCR calculating machine may be similar to a mouse or, in the case of a touch screen, similar to a pen. A number is read in by means of a snap-shot, whereupon from the wealth of information exactly this number with its immediate surroundings is displayed in clearly readable size. This eliminates a tedious search from a wealth of numbers and the selected number or information can be easily edited on the small screen.

If the number is not uniquely identifiable, then the selection can be further restricted by collecting additional numbers from the environment of the relevant number, either by multiple contact-inherent snap-shots or a short contact-inherent motion of the OCR calculating machine.

Contact-inherent snap-shots and read-in movements directly on the document along the column of numbers - possibly also in the row direction - can also be used to carry out arithmetic operations in the sense of a spreadsheet: The snap-shot can be used to determine the input field, with additional snap-shots the fields be defined individually or by means of a read-in the fields of subsequent numbers. 4. In addition, individual numbers can be deleted or moved by clicking on them using the OCR calculator, etc.

Common to all these first additional methods according to the invention is the advantage that the information for editing on the original document itself is available in full size and therefore readability, individual steps or processing or overviews on the small screen are immediately readily available, but at the same time stored full, captured information of the full screen on the device until their deletion and thus more information is always available digitally ready for further processing.

In the case of a pure post-addition of a multi-column arrangement, the screen is not needed at all, if the post-additions do not result in differences and no corrections are necessary.

The functions and commands to be selected can be made available on a screen, for example a touch screen. At best, individual commands can also be attached to a 'keyboard template' - for example, made of paper or plastic - from where they are scanned using snap-shots.

The advantages of the inventive combination between receiving a large and partial image written information and their processing by triggering a specific function, as described above for an OCR computing machine, are also feasible if the partial image is not kontaktinhärent but contact detected. Only this additional, not very simple method to use and solve problems, as shown for example in the literature in the following document and were essentially solved:

Bertucci Eve; Pius Maurizio; Mirmehdi Majid "Text Selection by Structured Light Marketing for Handheld Cameras". In: Proceedings of the Seventh International Conference on Document Analysis and Recognition (ICDAR'03), ISBN 0-7695-1960-1 / 03, 2003 IEEE.

In this article, a non-contact detection of a partial image is made possible by the fact that the text to be detected is marked during the detection with a laser, which is installed in a mobile camera, which is guided from left to right without contact over a text. The mark by the laser beam corresponds to the center of the images taken. Several pictures are taken every second. Then the words corresponding to the laser marking and the selection through the center of the image are selected.

However, this is merely a method in which, on the occasion of non-contact image acquisition by means of a mobile camera, a partial image can be obtained by selecting information from an image of the image acquisition.

For this purpose, not only a camera module with autofocus but also a device for generating a laser beam is necessary. Since the user can not place the device on a support for guiding, the quality of the image capture (blurring, distance to the base, etc.) is also worse than with contact-inherent detection or a sufficient quality must be ensured with additional technical aids.

Example Mobile Text Editing

Another field of application of the inventive mobile OCR devices is the text capture.

A rail traveler, for example, is interested in a multi-page specialist article that he or she wants to edit and at least partially archive in the office. He takes as the first of each page of the subject with a mobile device according to the invention a full screen. Each page is then edited by an OCR and stored in his handheld device.

Now he reads the article more or less and edited it simultaneously, but not on a - small - screen, but according to the invention very conveniently directly in his journal. In doing so, the traveler exercises the following functions, for example, by reading in a partial image a second time:

- The user reads with the inventive OCR device along a side edge, preferably left, or in the middle of a page a vertical section of text that captures only part of the text on each line. Due to the stored text, each detected line is supplemented with the missing text, so that the horizontally recorded text in the horizontal direction is completely available for further processing.

For clarification:

One line of text has the length of 12 cm, its OCR device has a reading width of 3.5 cm. Thus, the user reads in a vertical text section 3.5 cm wide, i. from each line 3.5 cm. Thus, each line is missing the text of the remaining 8.5 cm. These missing 8.5 cm are now completed and completed after identification of the partial image in the stored large image from the large screen.

- By activating a function 1, vertical reading causes the selection and storage of the completed text block as part of a text summary that the user wants to create.

- Activation of function 2 translates the text block from a foreign language.

- The selection of a function 3 marks the vertically recorded text block in the sense of a marker / felt-tip pen.

- Function 4 deletes text by vertical capture.

- With a function 5, only the vertically captured text block is finally saved.

A variety of other triggerable functions are conceivable.

The user does not initiate the basis of these methods on the - small - screen of his handy device, but comfortably while reading through his specialist article on the pages of the magazine itself. The reading field of his handheld device is wide enough that it can be used during the second recording process. vertical detection, recognize a part of the text from the selected vertical section and reliably associate it with the text that has already been stored completely from the first detection and can then complete the lines of text. The method may be allowed a margin of error such that within a particular Marge an existing non-match from unequal recognition of identical text is allowed.

Other similar methods in the same context, which are not described here in detail, are also possible for a horizontal 2nd detection and other applications.

The user no longer has to read in the entire text horizontally with a mobile OCR reading pen that he wants to edit. It is sufficient if he very quickly in a small vertical section from top to bottom takes over text to perform various functions.

It is conceivable to carry out the identification of information in the context of the second detection from the information stored in the first acquisition also from a comparison of digital partial information, without this having previously been converted by OCR into characters, for example an ASCII code. From this, methods can be derived which are based on the same inventive principles, but will not be discussed further here. It is to be expected that these methods require a higher memory requirement and at least for the identification by comparison more processing time, as compared to the OCR detection is performed.

These are only first applications of the inventive methods and devices. Others are easy to find and expect.

The use of the inventive method in connection with the mobile recording of documents for the efficient determination and completion of text of expected input fields of semistuctured entry masks while at the same time original storage of the documents processed according to the invention is briefly indicated.

From the comments on the inventive method can be readily derived the hiezu necessary inventive devices, as they are included and described in the claims.

In addition, the reading device additionally has various memories, for example for storing the digital still picture recorded without contact or its information (information memory) recognized in the form of a character a device for identifying the partial image in the large image (with and without storage of the partial image in the information memory), via an operative connection, via which functions are triggered and possibly via a function memory.

Parts of the inventive means of the device may also be outsourced. For example, the screen or certain functions and memory could be swapped out.

In addition, the full screen can also be transmitted to a high-performance server in order to relieve the handheld in terms of computing and software. On the handset, only the less complex identification of the partial image in the large image or even only the triggering of functions is performed.

Claims

claims 1) A method for processing written information by means of a mobile OCR device, characterized in that in a manual recording with the OCR device a large image is recorded in writing information available, this large image is stored at least temporarily, a partial image of the large image of existing written information is recorded in another manual acquisition with the same OCR device, the partial image is identified in the saved large image, a function is triggered in order to carry out automated processing from the partial image / large image of the written information in accordance with the rules of this function. 2) Method according to claim 1, characterized in that the partial image is detected by contact-inherent detection by means of a snap-shot. 3) Method according to claim 1, characterized in that the partial image is captured on the occasion of an image acquisition by means for selecting information from this image acquisition. 4) Method according to claim 1 to 3, characterized in that the post-addition of numbers and their expected structure as Number columns are stored on a written template in a function memory, Numbers and their structure are recognized as number columns on the written template in a manual capture of a large image by means of OCR and structure recognition methods and stored in an information store, in a second acquisition, a partial image is detected by means of a contact-inherent snap-shot and a number, recognized by means of an OCR and determined as an actual sum to be recalculated and stored in a number memory, the second manual detection triggers the stored function of the postaddition, the summands of the total to be recalculated are determined on the basis of the structures stored in the information store of the full screen of the written template and temporarily stored and added to the desired sum, the target sum is compared with the actual sum and a difference is displayed. 5) Method for processing written information, numbers, simple number columns, multi-level number columns and number columns in multi-column arrangement by means of an OCR calculating machine, characterized in that Numbers, columns of numbers and other information relevant for the OCR calculating machine can be recognized and processed with the same OCR calculating machine, depending on the application, however, optionally by contact-inherent detection by means of the OCR calculating machine on the written template or by contactless detection by means of the same OCR calculating machine. 6) Method according to claim 5, characterized in that Numbers in a multi-column arrangement are recorded and OCR-recognized by non-contact capturing of a still image with the OCR calculating machine. a single number is detected and OCR-recognized by contact-inherent detection of the number by means of a contact-inherent snap-shot of the OCR calculating machine on a written template and a simple number column is detected and OCR-recognized by contact-inherently moving the OCR calculating machine along the number column on a written template. 7) Method according to claim 1 to 6, characterized in that the large image and partial image are identified in advance by the identification of the partial image in the large image by means of an OCR and the identification is based on the OCR recognized characters. 8) Method according to claim 1 to 3, characterized in that the triggered function contains an OCR recognition and the partial image is saved as recognized text in the large image. 9) Method according to claim 1 to 3, 7 and 8, characterized in that the triggered function includes a text editing function. 10) Method according to claim 1 to 3, 7 and 9, characterized in that a structured large picture contains various structured text elements, a part of a text element is captured when capturing a partial image, the subelement is identified in the text, identified on the basis of the identified subelement the text element in the text or the subelement is completed with the missing text to the text element and a function for editing the identified text element is triggered. 11) A method according to claim 9 and 10, characterized in that at least one text editing function is stored in a function memory, a text and its structure are recorded on the written template in a first manual capture of the large image and at least the structure is recognized, Text and recognized structure are stored in an information store, as part of a second acquisition along a text column, a column section of the text column is recorded in the width of a read window, this column section is identified in the stored large image, due to the identified column section, the text column in Large image is identified or the identified column section is supplemented with the missing text to the text column, and the identified text column is edited with a stored function. 12) Method according to claim 1 to 8, characterized in that identified in a multi-column arrangement a text columns or rows, the multi-column arrangement is recorded in a first acquisition of the large image and recognized by means of OCR and structure recognition methods and stored in an information memory, in a second acquisition, a text is captured and recognized as a partial image, the recognized text identifies a column arrangement of the stored multi-column arrangement, the identified column arrangement is completed by its stored numbers and a function for the post-addition of the numbers of the column is triggered. 13) Method according to claim 1 to 12, characterized in that On a screen of the mobile OCR device only a relevant part of the stored or processed information is displayed. 14) Mobile OCR device according to claim 1 to 13, characterized in that the device includes an optical device for capturing a large image of a written original and a partial image with a smaller image detail from this written template, an information store for storing at least the captured large image, a device for identifying the partial image in a large image, an operative connection for triggering a function such that automated processing is carried out with the partial image / large image in accordance with the rules of the function. 15) Device according to claim 14, characterized in that the optical device has an adjustable focus for adaptation to a larger recording distance in the large image and a smaller recording distance in the partial image. 16) Device according to claim 15, characterized in that the optical device has a manually adjustable focus. 17) Device according to claim 15, characterized in that the optical device has an autofocus. 18) Apparatus according to claim 14, characterized in that comprising the device an optical device with a fixed focus for taking the Grossund part image and a device for detecting the partial image by selecting information from an image of an image recording. 20) OCR calculating machine, characterized in that the OCR calculator has an optical reading device for contact-inherent reading and recognition of at least one number by means of a snap-shot or a movement on the original and an optical reading device for a non-contact reading in and recognizing at least one number column by means of a still picture freely above the original. 21) Device according to claim 14 to 17 and 20, characterized in that having the device inside the housing an optical reading device with autofocus whose focus adapts sufficiently to a smaller distance at a contact-inherent and a greater distance at a non-contact detection and at least one device inside the OCR calculating machine for the illumination of the reading area on the written template for a contact-inherent reading on the template itself. 22) Device according to one of claims 12 to 21, characterized that individual individual facilities are outsourced for the storage, processing or presentation of information.