DE102023127343A1 - Method for creating a database for document classes and methods for scanning and processing a document and a device for carrying out these methods - Google Patents

Abstract

The invention relates to methods and a device that allow the partially or fully automatic processing of documents, even if they were bent or folded and are therefore no longer smooth. Scanned images generated by scanning the documents can be automatically assigned to document classes and processed and, in particular, read out accordingly in a class-specific manner. This allows, in particular, the processing of machine-printed documents into which people have inserted handwritten text.

Description

The invention relates to a method for creating a database for document classes and a method for scanning and processing a document as well as a device for carrying out these methods.

There are methods for recognizing similar images based on features. So-called "Content Based Image Retrieval (CBIR)" systems are known that find similar images in large image databases. Such CBIR methods, for example, are based on X. Wangming et. al. (December 2008), “Application of Image SIFT Features to the Context of CBIR,” in “2008 International Conference on Computer Science and Software Engineering” (Issue 4, pages 552-555 ), IEEE, and from KR Reddy et. al. (2016), "A Comparative Study of SIFT and PCA for Content-Based Image Retrieval, Inter. Refereed J. Ing. Sci. (IRJES) 5 (11), 12-19 Such CBIR systems often result in mismatches, but these are not problematic, as these systems are designed to return multiple matching images in case of doubt.

Methods for correcting image distortion sometimes use feature matching. In feature matching, features are extracted from two images that are intended to show at least some of the same content and then matched to one another. If one of the two images is a reference image, then based on this match of features, the pixels of the image that is not the reference image can be shifted so that they are arranged in the same way or in a similar order to that in the reference image. If the reference image is not distorted, then after this shift of the pixels the non-reference image is corrected. The matching of features is described, for example, in R. C. Gonzalez "Digital Image Processing", 4th edition Pearson, pages 915, 916.

Such feature mapping can be used for a wide variety of applications in image processing. For example, it can be used for image stitching or for stabilizing video signals. It can also be used to track specific objects in a video signal or to generate 3D models from multiple images. When rectifying an input image with respect to a reference image, local displacements are determined from the paired features, which can include both linear displacements and rotations. These displacements are interpolated across the entire image domain, creating a "dense displacement field" for each pixel, as described, for example, in Lee Seungyong, Georg Wolberg, and Sung Yong Shin, "Scattered Data Interpolation with Multilevel B-Splines," IEEE Transactions on Visualization and Computer Graphics 3.3 (1997), pages 228 to 244.

From the US 6, 711, 293 B1 A method is known with which scale-invariant features (SIFT features: Scale Invariant Feature Transform) can be extracted. Such features have the advantage that they are equally present in the scanned images from different viewing directions of a camera, for example. These features can thus be correctly assigned to one another in images from different viewing directions.

There are other types of features, such as ORB features or SURF features ( Rublee Ethan et al. “ORB: An Efficient Alternative to SIFT or SURF” 2011 International Conference on Computer Vision, IEEE 2011 ).

From the EP 1 594 078 B1 A system and method are disclosed for identifying points of interest (POIs) in each image at different resolutions. A point of interest is a point whose position within an image is defined by at least one property attributable to the pixels in a neighborhood of pixels of a predefined size around the point. Furthermore, each point of interest is a point that can be assigned a unique orientation based on at least one property attributable to the pixels. This can be a property attributable to the point that defines the point position or another property in a neighborhood around the point. Once the points of interest have been identified, a descriptor is created for each point of interest. This descriptor characterizes the point in a manner that is essentially invariant to changes in image position, orientation, and scale, as well as to changes in image size. Furthermore, the descriptor is essentially invariant to scaling of the image and to changes in the intensity of the pixels in the region around the point of interest. These descriptors are also called feature vectors, where a feature is defined by the feature vector, the corresponding coordinates and a principal direction.

The present invention is based on the object of creating a method for creating a database for document classes, a method for scanning and processing a document and a device for carrying out these methods, with which a reliable assignment It is possible to assign optically scanned documents to corresponding document classes. In particular, the assignment should be as clear as possible. The device should also be easy to use.

According to a first aspect, the present invention relates to a method for creating a database for document classes, wherein each document class is defined by a plurality of features. In this method, a plurality of copies of a document type to be classified are scanned with a scanning device, wherein a scan image is generated for each copy of the document type (= document). Each scan image is aligned with a digital reference image of the document type to be classified. The scan images aligned in this way are superimposed on one another, wherein the individual pixels are averaged, thus generating a prototype image. From the prototype image, features are again determined, which are entered in the database as the database features defining the document class.

The method involves scanning multiple copies of a document type, i.e., multiple documents, and aligning, superimposing, and average the resulting scanned images. This produces a prototype image that is prototypical for the document type. Since the prototype image was generated by optical scanning with a scanning device, the prototype image reproduces the document type as perceived by the scanning device. In other words, this means that the prototype image also exhibits properties determined by the scanning method and the scanning device and thus does not have to exactly match the digital reference image. Such a prototype image is therefore more similar to an image of such a document scanned with the scanning device than to the digital reference image. The features extracted from the prototype image are therefore more specific to the scanned images generated from these documents with the scanning device than the corresponding features generated from the digital reference image alone. This creates a database that very reliably defines the individual document classes using database features.

If you want to process documents automatically, then the documents should be clearly assigned to a document class, as this is the only way to ensure that the document content can be reliably and correctly extracted automatically. The purpose of a database for document classes is to identify documents and assign them to the document class. Additional information may be available for the individual document classes, which makes it easier to extract the information contained in the documents. This additional information describes, for example, fields with checkboxes, fields with machine-readable text or numbers. However, this only works reliably if a specific document is correctly assigned to a document class. In the event of an incorrect assignment, instructions for extracting from an incorrect document class would be used, which would then be unsuitable for the respective document. The features derived from the prototype image allow such a clear assignment of a document to a specific document class.

Preferably, one type of scanning device or a single specific scanning device is used to create a specific database. This means that the database characteristics are specific to the type of scanning device or to the single specific scanning device. Scanning devices for optically scanning documents can differ considerably, so that correspondingly different scanned images are produced, even if the same document is scanned with different scanning devices. The differences can lie in the color or brightness sensitivity and in the sharpness with which the individual pixels are captured. By using a single type of scanning device or a single specific scanning device, such deviations are reduced or eliminated and the documents can be captured reliably.

The alignment of each scanned image can be performed using feature mapping or other methods, such as an optical flow method. Because the individual scanned images are aligned to the digital reference image, they can essentially be arranged in any position within the scanning device. Automatic alignment brings the scanned images into register with the digital reference image. Alignment using feature mapping allows for very precise alignment of the scanned images to the digital reference image.

The database features can each comprise a feature vector describing the respective feature and coordinates defining the location of the respective feature, with the feature vector being obtained from the prototype image and the corresponding coordinates from the reference image. By obtaining the features from the prototype image, they describe the individual features as they are recognized or seen by the respective scanning device. By assigning the However, the exact coordinates of the reference image are used to determine the coordinates of the respective features, since the corresponding coordinates of the features in the prototype image may be somewhat blurred or imprecise due to the averaging of the multiple images. This results in an optimal combination of information in the features, resulting in a very high quality of the features.

When averaging the pixels of the multiple scans aligned to the reference image, the pixels of the superimposed scans with the same coordinates are averaged. The term "pixel averaging" means that the color values and/or brightness values of the respective pixels are averaged. Averaging can be carried out by equally weighting all pixels that have the same coordinates. The individual pixels can also be weighted differently. For example, it can be useful to analyze the distribution of the color values and/or brightness values of the pixels and exclude values that deviate significantly from the mean. Averaging can also be carried out by determining a median value for the pixels with the same coordinates in the superimposed scans.

Preferably, certain predefined areas of the aligned scan images are excluded. These are primarily areas with different content in the respective copies of the document type. Typical examples of such areas are fields in which document users are asked to enter certain individual information, such as their name or address. The characters that represent this information differ from document to document. It makes little sense to use these characters as a feature classifying the document class. This exclusion means that, at least in these areas, no features are extracted.

Meanings can be assigned to certain predefined areas. These meanings can be stored in a database and used when evaluating the content of the document by assigning the respective meaning to the information contained in these areas or by evaluating this information with knowledge of its meaning.

The "specific areas" that are not considered and to which specific meanings are assigned may be the same areas. But they may also be different or only partially the same areas.

These areas can be excluded before the overlay, for example, by cutting out the corresponding areas in the not-yet-aligned scan images. It is also possible to cut out these areas only once the individual scan images are superimposed. These areas can be "cut out," for example, by assigning a predetermined color value to these pixels, such as a color value representing the color white or a color value representing the color black.

• - Entzerren und Ausrichten des jeweiligen Abtastbildes mittels einer Merkmalszuordnung (= erste Merkmalszuordnung), wobei Merkmale des Abtastbildes, welche jeweils einen Merkmalsvektor umfassen, zu korrespondierenden Merkmalen des Referenzbildes des Dokuments zugeordnet werden und entsprechend dieser Merkmalszuordnung eine Homographie-Matrix ermittelt wird, mit der alle Bildpunkte des jeweiligen Abtastbildes zur Ausbildung eines Homographie-Bildes abgebildet werden,
• - Hinzufügen der Koordinaten der Merkmale im Homographie-Bild zu den jeweiligen Merkmalsvektoren und
• - Hinzufügen der Koordinaten der Merkmale im Referenzbild zu den jeweiligen Merkmalsvektoren und,
• - erneutes Zuordnen der Merkmale des Homographie-Bildes zu korrespondierenden Merkmalen des Referenzbildes des Dokuments (= zweite Merkmalszuordnung), wobei bei der Zuordnung die den Merkmalsvektoren zugeordneten Koordinaten berücksichtigt werden,
• - Entzerren des Homographie-Bildes nach Maßgabe der Zuordnung der Merkmale des Homographie-Bildes zu den Merkmalen des Referenzbildes.

To align the scanned images to the digital reference image, the following steps can be performed:

• - Rectifying and aligning the respective scanned image by means of a feature assignment (= first feature assignment), whereby features of the scanned image, each comprising a feature vector, are assigned to corresponding features of the reference image of the document and, in accordance with this feature assignment, a homography matrix is determined with which all pixels of the respective scanned image are mapped to form a homography image,
• - Adding the coordinates of the features in the homography image to the respective feature vectors and
• - Adding the coordinates of the features in the reference image to the respective feature vectors and,
• - re-assigning the features of the homography image to corresponding features of the reference image of the document (= second feature assignment), whereby the coordinates assigned to the feature vectors are taken into account during the assignment,
• - Rectification of the homography image according to the assignment of the features of the homography image to the features of the reference image.

The homography matrix generated from the first feature assignment allows for global perspective distortion correction. The scanned image is corrected using the homography matrix through rotation, translation, and global perspective distortion correction. This allows the document to be positioned arbitrarily with respect to the optical scanning device used for optical scanning. Homography mapping rotates and shifts the scanned image until its orientation essentially matches that of the digital reference image. Distortions that vary locally are included in the homography image. pronounced distortions, such as those caused by bending and folding of the document, are not corrected. Such homography mapping does not allow for the correction of locally varying distortions.

In the second feature assignment, by assigning the features of the homography image to the features of the reference image, both the coordinates of the features of the homography image and the coordinates of the features of the reference image are taken into account. During such an assignment, the distance between the corresponding feature vectors is determined. The smaller the distance, the better the assignment. The feature pairs with the smallest distance are selected as feature pairs of the features of the homography image and the features of the reference image. By taking the coordinates of the features into account, the distance between the features of the homography image and the corresponding features in the reference image is small if the corresponding features are arranged in the same or a similar position in both images. This allows features to be clearly assigned, even if identical or similar features occur repeatedly in the corresponding images. By spatially assigning these features, identical or similar features can be clearly distinguished from one another. However, such a spatial assignment only makes sense if the scanned image is aligned in essentially the same way as the reference image. For example, if the scanned image is rotated by an angle of more than 45° relative to the reference image, the features to be assigned to each other would have completely different coordinates, and the coordinates would have no meaningful significance for the similarity or assignment of the features. Only the alignment of the scanned image by mapping it to the homography image using the homography matrix allows the meaningful use of the feature coordinates as components of the feature vectors, since the mapping using the homography matrix aligns the scanned image to the reference image.

After the second feature assignment, the features of the homography image and the reference image are assigned to each other so precisely that the homography image can be rectified very precisely according to the second feature assignment, whereby locally different rectifications, such as those caused by bending and folding of a document, are also possible.

The rectification of the homography image is preferably performed using a freeform rectification method, for example, by interpolating a displacement field as explained above (see discussion by Lee, Seungyong, et al.).

The features are preferably scaling and/or rotation invariant features.

Preferably, the scanned images are rectified before superimposition, whereby the rectification can be performed by means of feature assignment and/or by means of image section assignment. Both rectification by means of feature assignment and rectification by means of image section assignment can be performed in combination, so that certain areas of the scanned images are rectified by means of feature assignment and other areas by means of image section assignment. These rectification methods each enable freeform rectification.

• - Abtasten eines Dokuments mit einer optischen Abtastvorrichtung, wobei ein Abtastbild des Dokuments erzeugt wird,
• - Klassifizieren des Abtastbildes zu einer Dokumentenklasse, wobei aus dem Abtastbild Merkmale extrahiert werden, und diese Merkmale mit Datenbankenmerkmalen einer Merkmalsdatenbank, die mehrere Dokumentenklassen definieren, verglichen werden, wobei das Abtastbild der Dokumentenklasse zugeordnet wird, zu der die beste Übereinstimmung mit den Datenbankmerkmalen der jeweiligen Dokumentenklasse erzielt wird.

According to a further aspect, the present invention relates to a method for scanning and processing a document comprising the steps

• - scanning a document with an optical scanning device, whereby a scanned image of the document is generated,
• - Classifying the scanned image into a document class, wherein features are extracted from the scanned image and these features are compared with database features of a feature database that defines a plurality of document classes, wherein the scanned image is assigned to the document class for which the best match with the database features of the respective document class is achieved.

Classifying documents using feature mapping allows for precise and accurate classification. This is especially true when using a feature database, as described above.

Documents classified in this way can be registered with the digital reference. The digital reference can contain instructions according to which specific information from the document is to be read and processed accordingly. These instructions can, for example, include instructions specifying what type of information (e.g., name, address, telephone number, etc.) should be included in specific areas of the document. This information can then be automatically read and processed according to its meaning.

Preferably, a type of scanning device or a A single specific scanning device is used. In particular, the same type of scanning device or the same specific scanning device that was also used to create the feature database is used. This maintains the quality of uniform scanning, which in turn ensures a correspondingly high quality of the assignment.

• 1a ein Ausführungsbeispiel einer Abtastvorrichtung zum optischen Abtasten von Dokumenten in einer grob schematisch vereinfachten Darstellung,
• 1b eine Abtasteinheit der Abtastvorrichtung aus 1A und ein Abtastelement und eine Beleuchtungseinrichtung in ihrer geometrischen Zuordnung zum Abtastbereich,
• 2 ein Verfahren zum Erstellen einer Datenbank für Dokumentenklassen in einem Flussdiagramm,
• 3 ein Abtastbild und ein digitales Referenzbild eines Dokuments, jeweils nach bestimmten Bearbeitungsschritten, und
• 4 ein Verfahren zum Abtasten und Verarbeiten eines Dokuments mit einer Datenbank, die mit dem Verfahren nach 2 erzeugt worden ist, in einem Flussdiagramm.

The invention is explained in more detail below by way of example with reference to the drawings. The drawings show:

• 1a an embodiment of a scanning device for optically scanning documents in a roughly schematic simplified representation,
• 1b a scanning unit of the scanning device 1A and a scanning element and an illumination device in their geometric assignment to the scanning area,
• 2 a method for creating a database for document classes in a flowchart,
• 3 a scanned image and a digital reference image of a document, each after certain processing steps, and
• 4 a method for scanning and processing a document with a database which is compatible with the method according to 2 has been generated, in a flowchart.

The invention is explained in more detail below using a device for optically scanning documents ( 1a , 1b) . Such a scanning device 1 comprises a scanning unit 2 and an evaluation unit 3.

In the present embodiment, the evaluation unit 3 is formed by a conventional workstation or personal computer 4 with an input unit 5 and a display unit 6. The input unit 5 is a computer keyboard, and the display unit 6 is a computer screen.

However, the evaluation unit 3 can also be formed by any other multi-purpose computer, such as a computer tablet or a smartphone, or can be fully integrated into the scanning unit 2.

In the present embodiment, the evaluation unit 3 is designed in the form of a cuboid body with a base plate 7, two side walls 8, 9, and a top wall 10. The base plate 7 and the walls 8-10 define a cavity, with the base plate 7 forming a scanning area 11 on which a document 12 to be scanned can be placed. A scanning element 13 and an illumination device 14 are arranged in the top wall 10 ( 1b) .

The scanning element 13 is a digital camera with a two-dimensional camera chip and a lens. The camera's viewing direction is directed toward the scanning area 11, allowing the entire scanning area to be optically scanned by the camera.

The illumination device 14 comprises light-emitting diodes as illuminants and is designed to emit white light. For this purpose, light-emitting diodes that emit white light can be provided, or light-emitting diodes of different colors can be provided that together generate white light. If several light-emitting diodes of different colors are provided, it is possible to control them so that they can also emit light with different color ranges or spectral ranges by adjusting the light intensity of the differently colored light-emitting diodes accordingly. The illumination device can have reflectors (not shown), which are preferably designed to direct the light onto the scanning area 11 as diffusely as possible. Such illumination is also referred to as dark-field illumination. If primarily or exclusively non-reflective documents are to be scanned, it can also be useful to provide bright-field illumination, since with bright-field illumination, a higher light intensity is achieved at the scanning area 11 than with dark-field illumination, with the same light source intensity.

The surface of the scanning area 11 is preferably matt with a uniform color (especially white).

The surfaces of the side walls 8, 9 facing into the interior are preferably also matt and have a light color, in particular white, so that the light emitted by the lighting device and reflected by the side walls 8, 9 is directed into the scanning area 11 as diffusely as possible.

The evaluation unit 3 is connected to the scanning unit 2 via a data line 15 for bidirectional data transmission, so that the evaluation unit 3 can control both the scanning element 13 and the illumination device 14 and read out the data acquired by the scanning element 13. The evaluation unit 3 thus serves not only to evaluate images captured by the camera 13, but also to control the entire process.

Other scanning devices can also be used within the scope of the invention, such as scanning devices that use a line-scan camera that moves parallel to a document to be scanned during the scanning process, as is known, for example, from copiers, or even a digital camera that is arranged, for example, on a tripod in a specific spatial relationship to an evaluation area. It is also not necessary for the scanning unit to have an illumination device, since, in principle, a document could also be scanned using ambient light.

The scanning unit 2 according to the present embodiment has a specific, fixed geometric relationship between the scanning element 13, the illumination device 14, and the scanning area 11. Furthermore, the tubular design of the scanning device 1 with the two side walls, the top wall, and the base plate 7 minimizes interference from ambient light, yet the scanning area 11 is easily accessible from two sides, so that a document 12 to be scanned can be easily placed in the scanning area 11 by a user.

Such scanning devices 1 are particularly suitable for scanning identification documents, such as identity cards, driving licenses or the like, or for scanning certain forms, such as lottery tickets, registration forms or the like.

The scanning device described above represents one possible embodiment. However, it can be adapted in a variety of ways, provided a camera and a scanning area are provided. For example, it is possible to provide a single, vertically extending, rod-shaped support between the scanning area and the ceiling wall instead of the side walls.

A method for creating a database for document classes is explained below, which is carried out using documents scanned with the scanning device 1 explained above.

The process begins in step S1 ( 2 ).

In step (S2), several documents 12 are scanned by the scanning element 13. For this purpose, a document 12 is first placed on the scanning area 11 of the scanning unit 2. The document 12 is illuminated by the illumination device 14 in order to accurately capture all details of the document 12 and to obtain the most uniform scanning conditions possible when scanning different documents. Once the document 12 is in place and correctly illuminated, a user initiates the scanning process, for example, by pressing a button on the scanning unit 2 or on the input unit 5.

This process is repeated for several documents, i.e. n documents 12, so that n scan images are generated from the respective documents.

In the scanning device 1, the individual documents are manually placed onto the scanning area 11. It is of course also possible to provide an automatic feeder for feeding the documents into the scanning area 11, so that the scanning of multiple documents can be carried out fully automatically.

A scanning process can also be initiated automatically, without having to press a button. For example, a proximity sensor can be provided to detect that an object is located in the scanning area 11. Following this, an initial preliminary optical scan is performed to check whether the object is stationary in the scanning area 11, i.e., that it is not moving. If this is the case, it can be illuminated with the illumination device 14, and the actual optical scan can be performed.

In the present embodiment, the scanning element 13 is designed as a camera having a two-dimensional camera chip and a lens. The camera's viewing direction is directed toward the scanning area 11, so that the entire scanning area, and thus the entire document 12, can be captured with a single image. The scanned image 16 thus generated contains a representation of the document 12, which can initially be rotated and/or distorted as desired ( 3 , top row).

After the scanning image 16 has been captured, the data representing the scanning image 16 are transmitted to the evaluation unit 3 via the data line 15. The evaluation unit 3 evaluates the scanning images 16.

During this evaluation, a first feature matching of a scanned image of a first document is performed (step S3). Here, features, also referred to as features or keypoints, are identified and extracted in the scanned image 16 using a so-called SIFT algorithm (Scale-Invariant Feature Transform algorithm).

This involves identifying features that are unchanging, i.e. invariant, with respect to a change in scale. This is carried out, for example, by generating a scale space that essentially comprises a series of images with different scales (= different scales) or resolutions, whereby these images are calculated from the sample image 16. For each scale, the sample image 16 is convolved with a Gaussian filter with increasing sigma value to generate a series of blurred images. The Difference Of Gaussian (DOG) is then calculated by subtracting two consecutive blurred images. The result is a series of DOG images. Possible feature points are identified as local minima or maxima in the DOG images by comparing each pixel with its neighbors in the current sample image 16 and the neighbors in the scale images.

A feature point specifies the coordinates of a central point of a feature.

Potential feature points are verified by fitting a quadratic function to the local image patterns around the potential feature points. Features with low contrast or poorly localized to an edge are discarded. Low-contrast features are sensitive to noise. Feature points along edges are spatially unstable.

A final position of the feature points is determined by a maximum or minimum of the fitted quadratic function. Each identified feature point is assigned one or more orientations that pass 16 on the local gradient direction of the scan image. This ensures rotation invariance. For this purpose, the gradient magnitude and orientation are calculated for the neighborhood of the feature point. An orientation histogram with 36 columns covering 360° is created from the gradient orientation of key points within a region and the key point. The maxima in this histogram determine the orientation of the feature point. The highest peak in the histogram and any subsequent local maximums that lie within 80% of the highest peak are used to assign the orientation. After the feature points have been identified and the orientation assigned, a descriptor, also called a feature vector, is created for each feature to capture the local appearance of the feature.

The descriptor is a unique fingerprint for each feature point, enabling matching between two images, as the corresponding feature points in the respective images can be mapped to each other. The descriptor can be viewed as a histogram of the gradient orientation in a region around the feature point, providing a top-level representation of the local image texture.

To create a descriptor, the region around the keypoint is first divided into a 4 × 4 grid of subregions, for example. For each subregion, the gradient strength is calculated in the orientation of the pixels. An orientation histogram with 8 columns or bins is then created for each subregion, corresponding to a total of 128 columns or dimensions. The histogram values capture the dominant gradient direction in the local neighborhood of the feature point. The histogram values form the elements of the descriptor or vector. This descriptor is normalized to increase its robustness to lighting and contrast variations. The normalized vector is the descriptor for the feature points and provides an invariant representation of the local image structure.

This feature recognition is also performed on a digital reference image 17. The digital reference image is a perfect digital representation of essential elements of the document 12. The individual documents 12 may contain additional information, which has been added manually or mechanically, in particular, by a user of the document. Otherwise, the content of the document 12 largely corresponds to the digital reference image 17.

The actual assignment of the feature points of the scanned image 16 and the reference image 17 then takes place by comparing them with each other. When comparing the feature points between the scanned image 16 and the reference image 17, a distance (e.g., Euclidean distance) between the corresponding vectors is determined based on the descriptors, and the feature points of the two images with the smallest distance are assigned to each other. This can be carried out, for example, by initially evaluating distances between feature points that are below a certain threshold value as potential matches. A ratio test can then be performed, in which the distance of a feature point of the scanned image 16 to the second-closest feature point of the reference image 17 is calculated. The ratio of the distances between the closest and the second-closest feature point is then calculated. If the ratio is below a certain threshold value, for example, 0.8, the match is considered valid. After the correspondence between features of the scanned image 16 and the reference image 17 were found, the process is repeated by comparing feature points of the reference image 17 with the scan image 16. Only the feature points that match in both directions can be retained.

The retained feature points can then be filtered to remove outliers caused, for example, by noise. For this purpose, the so-called RANSAC (Random Sample Consensus) algorithm can be used to detect outliers and errors. However, other algorithms for outlier detection are also possible.

Matching features between two images is well known. Other known matching methods can also be used, provided they reliably match similar features.

The feature assignment is used to calculate a homography matrix. The homography matrix allows for perspective rectification of the scanned image and alignment with the reference image. By applying the homography matrix to the scanned image 16, the scanned image is aligned (rotated or shifted) with respect to the reference image 17 and also rectified, resulting in a homography image 18 ( 3 This homography mapping or homography rectification can correct perspective distortions, but not inhomogeneous distortions within the image. Distortions such as those caused by folding or creasing a document cannot be eliminated with homography rectification.

A direct linear transform (DLT) is used to calculate the homography matrix. It requires at least four matching, non-collinear features from the sample image 16 and the reference image 17. However, more matches can be used, in which case a least-squares solution is calculated. Due to noise, false matches, and other factors, not all matches will match perfectly, so the homography matrix must be determined using methods such as the RANSAC algorithm mentioned above. The RANSAC algorithm repeatedly selects a random subset of matching feature points and calculates the homography matrix. It then determines how many matches match this calculated homography matrix, and the homography matrix with the highest number of matches is selected as the final result.

During feature determination in step S3, certain predefined areas can be hidden. These are usually areas that contain individual information in the individual documents and are therefore not assignable to one another. Hiding can be achieved, for example, by setting the color and/or brightness values of these areas to a specific value, which corresponds, for example, to the color white. These hidden areas can thus be defined based on the features.

Once the homography matrix is determined, it is used to transform the scanned image 16 into the homographic image 18. For this purpose, each pixel in the scanned image 16 is multiplied by the homography matrix to obtain the transformed point in the homographic image 18. This mapping step represents the perspective distortion correction mentioned above (step S4). During this perspective distortion correction, a scanned image 16 can be adjusted in size and orientation to the reference image 17, whereby individual parts of the acquired scanned image 16 can be cropped. However, this is designed such that the information content of the scanned image 16 relating to the document 12 is not cropped. For example, edge detection is performed to detect the edges of the document 12. The scanned image 16 is now cropped such that all edges of the document 12 still remain in the scanned image 16.

After the transformation, there may be 18 regions in the homography image that do not contain corresponding pixels from the distorted scan image. These regions are filled with neighboring pixels or other inpainting techniques.

According to this embodiment, the homography image 18 is sharpened using corresponding known algorithms. Algorithms for contrast adjustment or for smoothing artifacts created during the transformation are known and can be applied here. The (location) coordinates of the features in both the homography image 18 and the reference image 17 are then added to the respective descriptors. This addition of the coordinates is referred to as stamping (step S5).

In the subsequent step S6, the features of the homography image 18 are again assigned to corresponding features of the reference image 17, whereby the assignment takes into account the coordinates assigned to the descriptors (second feature assignment). The descriptors comprise, for example, 130 dimensions, of which 128 of these dimensions are derived from a 4 x 4 grid of subregions. each with 8 columns or bins (see above). The additional two dimensions are the coordinates. These two additional dimensions can be weighted in the same way as all the other dimensions combined. However, it also makes sense to weight the two additional dimensions (= coordinates) more heavily, for example with a weighting factor that results in at least five times the weighting or at least ten times the weighting of the coordinates compared to the other dimensions. The coordinates can also be weighted so heavily that they have the same weight as the remaining 128 dimensions. The coordinates can therefore be included in the comparison of the characteristics for assignment with the same weight as the other dimensions.

By taking the coordinates into account, the location of the features is considered much more closely than in the first feature assignment, which means that only the features that are close to each other in the reference image and the scanned image are actually assigned to each other. This is reliably possible here because a perspective correction was previously performed, in which the reference image and the scanned image were aligned with each other so that the corresponding features are located in similar locations in the respective images. Without this alignment, the two images can be arranged, for example, rotated by 90° to each other, which would result in corresponding features being located in completely different locations. Taking the location coordinates into account in the feature vector would cause significant misassignments.

Based on the newly assigned features, displacement vectors are calculated between assigned features of the homography image 18 and the reference image 17, which is also referred to as vectoring.

It may be that contiguous areas of the documents have no or very few features, so that no displacement vectors exist for these areas due to the feature assignment. In this case, it makes sense to assign image sections of the homography image 18 and the reference image 17 to each other and determine a corresponding displacement vector for the assigned image sections. The assignment of image sections is also referred to as template matching.

A displacement vector field is generated from the displacement vectors of the feature assignment and the assignment of the image sections. The displacement vectors are included in the displacement vector field either directly or in an averaged or interpolated form. For regions where no displacement vectors are present, corresponding displacement vectors are interpolated.

Using this displacement vector field, the homography image 18 is subjected to freeform distortion correction. First, control points are selected using the displacement vector field, which are distributed as evenly as possible across the homography image 18 and the reference image 17. The connection between the control points in these images is given by the displacement vectors. A system of linear equations is established based on the control points. This system of equations is solved to determine weights for each control point. These weights determine the strength and direction of the transformation of each control point.

For each pixel in the homography image 18, a new position in a transformation image 19 is calculated based on the weights and a radial basis function of the control points. The pixel values are transferred directly if they perfectly match the grid of the transformation image 19. Since this is not the case in most cases, the pixel values for the transformation image 19 are interpolated. This can be done, for example, using bilinear or bicubic interpolation.

To avoid overfitting, especially when many control points are used, a regular realization term can be added to the transformation. This smooths the transformation and avoids high-frequency deformations. Further post-processing steps can be performed on the transformation image 19, for example, by sharpening, cropping, or performing other image adjustments to improve quality. The transformation image 19 is thus a freeform-distorted scan image. Using the methods explained above with reference to steps S3 to S7, the scan image 16 is aligned and rectified with respect to the reference image 17, whereby a pixel-precise match can be achieved.

In step S8, a check is made to determine whether a scanned image of another document is present that is to be aligned and rectified with respect to the reference image 17. If this is the case, the method flow proceeds to step S3, and the alignment and rectification of the additional scanned image 16 is performed.

If it is determined in step S8 that the scanned images of all documents are aligned and rectified, the process flow proceeds to step S9, in which the scanned images aligned and rectified in this way are superimposed on one another. Here, the pixels at the same locations of the aligned scan images 19 are averaged, and the average value thus generated is entered as a point value at the corresponding location in a prototype image. The average value can be a mean value, a median value, or even a mean value in which the pixel values of the different aligned scan images 19 are weighted differently. The prototype image thus represents an image of the document 12 that is rectified and aligned with respect to the reference image 17 and shows the document 12 as it is captured by the scanning device 1. Unlike the reference image, the prototype image therefore contains the effects of the optical scanning by the scanning device 1. Compared to the reference image, the prototype image may contain certain blurring and/or brightness and/or colors may differ. The deviations may be small, but even small deviations can cause errors during further processing.

Features are extracted from the resulting prototype image. The features are preferably SIFT features. However, other types of features, such as ORB features or SURF features, can also be used. Preferably, the features are scale- and/or rotation-invariant features.

The features comprise at least one descriptor or feature vector and coordinates that define the location of the features in the image. Preferably, the descriptor or feature vector is determined based on the prototype image and the associated coordinate is determined based on the reference image. This provides a description of the feature as seen by the scanning device 1, with the exact location of the reference image being used as the location. The features thus generated are stored in a database, with the features of such a prototype image representing a document class. The matching representations in the documents of a document class are contained in the respective prototype image. The features derived from this therefore apply to all documents of a document class. They are thus representative of a specific document class.

The method is terminated with step S11.

This process can be repeated for different document classes, with multiple copies of the documents of a document class being scanned and the scanned images being evaluated according to the above feature extraction procedure. Such a database can be used to classify documents, as explained below using an exemplary method ( 4 ).

In this method, a document of any document class can be scanned with the scanning device 1 (step S13).

Features are extracted from the resulting scanned image, and these features are compared with the feature groups of the different document classes. The comparison can be performed, for example, by calculating a distance, particularly the Euclidean distance. The document is then assigned to the document class with the smallest deviation from the corresponding features.

In the next step, the scanned image is registered with the digital reference image of the corresponding document class, i.e., aligned. This can be carried out in a similar manner to steps S3 to S7 of the method explained above. The database preferably contains processing instructions for the individual document classes, specifying how the individual documents are to be processed. These processing instructions typically include instructions as to which areas of the document are to be read out and what information they contain. For example, the areas in which a surname, first name, address, telephone number, email address, or the like are to be contained can be defined, with the type of respective information also being stored accordingly. This allows for the reliable reading of predetermined information content and its further processing. Due to the registration of the scanned image with the digital reference image, the corresponding instructions defined based on the reference image can be directly applied to the respective area of the scanned image.

In step S16, the corresponding information is read from the scanned image based on these instructions and passed on for further processing.

The method is terminated with step S17.

The methods and device described above allow for the partially or fully automatic processing of documents, even if they were bent or folded and are therefore no longer smooth. The documents can be assigned to document classes and processed and, in particular, read out accordingly. This allows, in particular, the processing of machine-generated printed documents in which people have inserted handwritten text.

List of reference symbols

1

scanning device

2

scanning unit

3

Evaluation unit

4

Workstation computer

5

Input unit

6

Display unit

7

Base plate

8

side wall

9

side wall

10

Ceiling wall

11

Scanning range

12

document

13

scanning element

14

Lighting equipment

15

data line

16

Scanning image

17

Reference image

18

Homography image

19

Transformation image

QUOTES CONTAINED IN THE DESCRIPTION

This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 6,711,293 B1 [0005]
• EP 1 594 078 B1 [0007]

Cited non-patent literature

• X. Wangming et. al. (December 2008), “Application of Image SIFT Features to the Context of CBIR,” in “2008 International Conference on Computer Science and Software Engineering” (Issue 4, pages 552-555 [0002]
• K.R. Reddy et. al. (2016), "A Comparative Study of SIFT and PCA for Content-Based Image Retrieval, Inter. Refereed J. Ing. Sci. (IRJES) 5 (11), 12-19 [0002]
• Rublee Ethan et al. “ORB: An Efficient Alternative to SIFT or SURF” 2011 International Conference on Computer Vision, IEEE 2011 [0006]

Claims (14)

A method for creating a database for document classes, each document class being defined by multiple features, wherein: multiple instances of a document type to be classified are scanned with a scanning device, each generating a scanned image, and each scanned image is aligned with a digital reference image of the document type to be classified. The scanned images thus aligned are superimposed on one another, with the individual pixels being averaged to generate a prototype image. whereby features are again determined from the prototype image, which are entered in the database as the database features defining the document class. Procedure according to Claim 1 , characterized in that one type of scanning device or a single specific scanning device is used to create a specific database. Procedure according to Claim 1 or 2 , characterized in that the alignment of each scanned image is carried out by means of a feature assignment or by means of an image section assignment. Method according to one of the Claims 1 until 3 , characterized in that the database features each comprise a feature vector describing the respective feature and coordinates defining the location of the respective feature, the feature vector being obtained from the prototype image and the corresponding coordinates being obtained from the reference image. Method according to one of the Claims 1 until 4 , characterized in that certain predefined areas of the aligned scan images are not taken into account. Method according to one of the Claims 1 until 5 , characterized in that a meaning is assigned to certain predefined areas. Method according to one of the Claims 1 until 6 , characterized in that the following steps are carried out in each case to align the scanned images to the digital reference image: - rectifying and aligning the respective scanned image by means of a feature assignment, wherein features of the scanned image, which each comprise a feature vector, are assigned to corresponding features of the reference image of the document and a homography matrix is determined in accordance with this feature assignment, with which matrix all pixels of the respective scanned image are mapped to form a homography image, - adding the coordinates of the features in the homography image to the respective feature vectors and adding the coordinates of the features in the reference image to the respective feature vectors and, - re-assigning the features of the homography image to corresponding features of the reference image of the document, wherein the coordinates assigned to the feature vectors are taken into account in the assignment, - rectifying the homography image in accordance with the assignment of the features of the homography image to the features of the reference image. Procedure according to Claim 7 , characterized in that the rectification of the homography image is carried out using a free-form rectification method. Method according to one of the Claims 1 until 8 , characterized in that the features are scaling and/or rotation invariant features. Method according to one of the Claims 1 until 6 , characterized in that before the scanning images are superimposed, they are rectified, wherein the rectification is carried out by means of a feature assignment and/or by means of an assignment of image sections. A method for scanning and processing a document, comprising the steps of: - scanning a document with an optical scanning device, generating a scanned image of the document, - classifying the scanned image into a document class, extracting features from the scanned image, and comparing these features with database features of a feature database that defines multiple document classes, and assigning the scanned image to the document class that best matches the database features of the respective document class. Procedure according to Claim 11 , characterized in that the scanned image is registered to a reference image on the basis of a feature assignment to the corresponding database features. Procedure according to Claim 11 or 12 , characterized in that the feature database contains a feature database created with the method according to one of the Claims 1 until 10 generated database. Method according to one of the Claims 11 until 13 , characterized in that a type of scanning device or a single specific scanning device is used as the scanning device, in particular a type of scanning device or a single specific scanning device which has also been used to create the feature database.

Images