CN111832403B - Document structure recognition method, document structure recognition model training method and device - Google Patents

Abstract

The present application discloses a document structure recognition method, a model training method and a device for document structure recognition, which relate to the fields of natural language processing and deep learning technology and are used for document layout analysis. The specific implementation scheme is: obtaining a document image, selecting a candidate area from the document image, performing image feature extraction on the candidate area to obtain image features, performing semantic recognition on the characters contained in the candidate area to obtain semantic features, and classifying according to the image features and semantic features to determine the document structure type to which the candidate area belongs. In the present application, when performing document structure type recognition, semantic features are added on the basis of image features, fully considering the importance of semantic information in document structure recognition, and improving the accuracy of document structure type judgment.

Description

Document structure recognition method, document structure recognition model training method and device

Technical Field

The present application relates to the field of computer technology, and in particular, to the field of natural language processing and deep learning technology.

Background technique

In real life, in order to facilitate the preservation of paper documents, paper documents are usually converted into electronic documents in the form of pictures. Electronic documents have obvious advantages in transmission and preservation.

After converting the document into an image, analyzing the document layout, that is, identifying the document structure, is an indispensable step in understanding the document content. Many subsequent tasks, such as information extraction and text classification, need to be based on accurate document structure type recognition.

Summary of the invention

The present application provides a document structure recognition method, a model training method and a device for document structure recognition. In the present application, when identifying the document structure type, semantic features are added on the basis of image features, the importance of semantic information in document structure recognition is fully considered, and the accuracy of document structure type judgment is improved.

In a first aspect of the present application, a document structure recognition method is provided, the method comprising:

Acquire document images;

Selecting a candidate region from the document image;

Extracting image features from the candidate area to obtain image features;

Performing semantic recognition on the characters contained in the candidate area to obtain semantic features;

Classification is performed according to the image features and the semantic features to determine the document structure type to which the candidate region belongs.

In a second aspect of the present application, a model training method for document structure recognition is provided, the method comprising:

Obtain a training sample set;

The target detection model is trained using the training sample set, wherein the target detection model is used to select a candidate area from a document image, perform image feature extraction on the candidate area to obtain image features, perform semantic recognition on characters contained in the candidate area to obtain semantic features, and perform target detection based on the image features and the semantic features to determine the document structure type to which the candidate area belongs.

In a third aspect of the present application, a document structure recognition device is provided, the device comprising:

An image acquisition module, used for acquiring a document image;

A selection module, used for selecting a candidate area from the document image;

An extraction module, used for extracting image features from the candidate area to obtain image features;

A recognition module, used for performing semantic recognition on the characters contained in the candidate area to obtain semantic features;

The detection module is used to perform target detection according to the image features and the semantic features to determine the document structure type to which the candidate area belongs.

In a fourth aspect of the present application, a model training device for document structure recognition is provided, the device comprising:

A sample acquisition module is used to obtain a training sample set;

A training module is used to train a target detection model using the training sample set, wherein the target detection model is used to select a candidate area from a document image, perform image feature extraction on the candidate area to obtain image features, perform semantic recognition on characters contained in the candidate area to obtain semantic features, and perform target detection based on the image features and the semantic features to determine the document structure type to which the candidate area belongs.

According to a fifth aspect of the present application, an electronic device is provided, comprising:

at least one processor; and

a memory communicatively connected to the at least one processor; wherein,

The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the document structure recognition method described in the first aspect, or the model training method for document structure recognition described in the second aspect.

The sixth aspect of the present application provides a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable the computer to execute the document structure recognition method described in the first aspect, or the model training method for document structure recognition described in the second aspect.

The technical solution provided by the embodiment of the present application has the following beneficial effects:

Acquire a document image, select a candidate area from the document image, extract image features from the candidate area to obtain image features, perform semantic recognition on the characters contained in the candidate area to obtain semantic features, and classify according to the image features and semantic features to determine the document structure type to which the candidate area belongs. In the present application, when performing document structure type recognition, semantic features are added on the basis of image features, fully considering the importance of semantic information in document structure recognition, and improving the accuracy of document structure type judgment.

It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

FIG1 is a schematic diagram of a flow chart of a document structure recognition method provided in an embodiment of the present application;

FIG2 is a flow chart of another document structure recognition method provided in an embodiment of the present application;

FIG3 is a flow chart of another document structure recognition method provided in an embodiment of the present application;

FIG4 is a schematic diagram of a flow chart of another document structure recognition method provided in an embodiment of the present application;

FIG5 is a schematic diagram of a candidate region closing operation according to the present embodiment;

FIG6 is a second schematic diagram of the candidate region closing operation of this embodiment;

FIG7 is a flow chart of another document structure recognition method provided in an embodiment of the present application;

FIG8 is a flow chart of a model training method for document structure recognition provided in an embodiment of the present application;

FIG9 is a schematic diagram of text content annotation in a training sample of the present application;

FIG10 is a schematic diagram of the structure of a document structure recognition device provided in an embodiment of the present application;

FIG11 is a schematic diagram of the structure of a model training device for document structure recognition provided in an embodiment of the present application;

FIG. 12 is a block diagram of an electronic device according to an embodiment of the present application.

Detailed ways

The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

The document structure recognition method, document structure recognition model training method and device of the embodiments of the present application are described below with reference to the accompanying drawings.

FIG1 is a flow chart of a document structure recognition method provided in an embodiment of the present application.

As shown in FIG1 , the method comprises the following steps:

Step 101, obtaining a document image.

The executor of this embodiment is a processor, in which a trained model for document structure recognition is running. The trained model for document structure recognition can be used to implement the document structure recognition method of this application.

The document image refers to an image of a document that needs to be used for document structure recognition, for example, a document in an image format obtained by scanning, such as a resume in an image format, an instruction manual in an image format, and the like.

Step 102: Select a candidate region from the document image.

The candidate region refers to a region that may contain various document structure types, including tables, texts, images, paragraphs, titles, and footnotes (for supplementing the text), which are not listed one by one in this embodiment.

As a first possible implementation method, a fast object detection model Faster R-CNN is used to identify the region of each object in the image, and the region of each object is used as a candidate region.

As a second possible implementation, a candidate region algorithm, such as selective search, is used to select a candidate region from the document image.

As a third possible implementation method, a sliding detection frame is used to select multiple candidate areas from the document image. Specifically, according to the preset sliding detection frame, it moves sequentially from left to right and from top to bottom in the document image according to a preset step size, and the area framed by the sliding detection frame each time it moves is used as a candidate area, thereby obtaining multiple candidate areas.

Step 103: extract image features from the candidate area to obtain image features.

Among them, the image features include color features, texture features, shape features and spatial relationship features of the image, granularity features of the text in the candidate area, etc., wherein the spatial relationship features also include the foreground area or background area to which different areas in the candidate area belong, and the text granularity includes at least one of a word, a sentence or a paragraph.

As a possible implementation method, a neural network model is used to extract image features from the candidate region to obtain image features.

Step 104: perform semantic recognition on the characters contained in the candidate area to obtain semantic features.

In one embodiment, each character contained in the candidate area is converted into a corresponding character vector, each character vector is concatenated to obtain a concatenated vector, and semantic recognition is performed on the concatenated vector to obtain a corresponding semantic feature.

For example, if a candidate region is an area about education experience in a resume, semantic features of the corresponding education experience can be obtained after semantic recognition of the characters contained in the candidate region. For another example, if a candidate region is an area about work experience in a resume, semantic features of the corresponding work experience can be obtained after semantic recognition of the characters contained in the candidate region.

Step 105 , classifying according to the image features and semantic features to determine the document structure type to which the candidate region belongs.

Among them, the document structure type includes text, table, title, paragraph, footnote and image, etc.

In the present application, classification is performed based on image features and semantic features, wherein candidate regions can be identified based on shape features, texture features, content density, etc. contained in image features to determine whether the candidate region is a table, text, or image, etc., and the semantic information contained in candidate regions corresponding to different document structure types is different, and different semantic information can indicate the corresponding document structure type. For example, the semantic information recognized by the characters contained in the table is different from the semantic information recognized by the characters contained in the text, and the semantic information contained in different text paragraphs is also different. For example, a resume contains more text regions, and different text regions have different semantic information. There are text regions corresponding to educational experience, text regions corresponding to work experience, and text regions corresponding to personal information. Therefore, the document structure type can be identified based on the extracted semantic features of the candidate regions in combination with the image features, which can improve the accuracy of document structure identification in the candidate regions.

In the document structure recognition method of the embodiment of the present application, a document image is obtained, a candidate area is selected from the document image, image features are extracted for the candidate area to obtain image features, semantic recognition is performed on the characters contained in the candidate area to obtain semantic features, and classification is performed based on the image features and semantic features to determine the document structure type to which the candidate area belongs. In the present application, when performing document structure type recognition, semantic features are added on the basis of image features, the importance of semantic information in document structure recognition is fully considered, and the accuracy of document structure type judgment is improved.

Based on the above embodiment, the embodiment of the present application provides another document structure recognition method. FIG2 is a flow chart of another document structure recognition method provided by the embodiment of the present application. As shown in FIG2, the method includes the following steps:

Step 201, obtaining a document image.

Step 202: Select a candidate region from the document image.

Specifically, reference may be made to steps 101-102 in the previous embodiment, the principles are the same and will not be repeated here.

Step 203: Identify content attribute features for each pixel unit in the candidate area.

Among them, the content attribute features indicate the attribute-related features contained in each pixel unit, such as whether the pixel unit belongs to the foreground or the background, and the target text granularity corresponding to the pixel unit, wherein the target text granularity includes one or more of words, sentences and paragraphs.

Step 204: Generate an input image based on the pixel value and content attribute characteristics of each pixel in the candidate area.

In one embodiment, each pixel point in the regenerated input image has multiple channels, each channel is used to indicate the pixel value or content attribute feature of the corresponding pixel in the candidate area, and the feature information contained in each pixel point in the input image is increased.

The feature information contained in each pixel can be expressed as [pixel value, content attribute feature]

Step 205: extract image features from the input image to obtain image features.

Specifically, reference may be made to step 103 in the previous embodiment, the principle is the same and will not be repeated here.

The regenerated input image in this embodiment contains pixel values and content attribute features. Compared with the prior art which only contains pixel value information, the amount of information in the extracted image features is increased. Therefore, when the document structure type is subsequently identified based on the image features, a more fine-grained document structure type can be identified. For example, it can be identified whether the text in the candidate area is a sentence or a paragraph.

Step 206: Obtain the document content corresponding to the document image.

As a possible implementation method, the document content corresponding to the document image may be recognized by using optical character recognition (OCR) recognition technology.

As another possible implementation manner, a source document corresponding to the document image is obtained to obtain corresponding document content from the source document.

Step 207: query the relative position in the document content according to the relative position of the candidate area in the document image to obtain the characters contained in the candidate area.

As a possible implementation method, the coordinates of each pixel in the document image are identified, the pixel corresponding to the border of the candidate area is determined, and the relative position relationship of the candidate area in the document image is determined according to the coordinates of the pixel corresponding to the border. Since the document image and the corresponding document content have a one-to-one correspondence in position and content, the relative position in the document content is queried according to the relative position, and the characters contained in the relative position corresponding to the candidate area in the document content are taken as the characters contained in the candidate area.

Step 208: Perform semantic recognition on the characters contained in the candidate area to obtain semantic features.

Furthermore, semantic recognition is performed on the characters contained in the candidate area to obtain semantic features. Since each character is obtained from the document content corresponding to the document image, the accuracy of the characters contained in the candidate area is improved, thereby improving the accuracy of the recognized semantic features.

Step 209 , concatenating the image features of the candidate region with the semantic features of the candidate region to obtain synthetic features of the candidate region.

Step 210 , classifying the candidate region according to its synthetic features to determine the document structure type to which the candidate region belongs.

In one embodiment of the present application, the image features of the candidate area are spliced with the semantic features of the candidate area to obtain the synthetic features of the candidate area, and the information contained in the synthetic features of the candidate area is increased so that under the supervision of the semantic information, the document structure type to which the candidate area belongs is obtained through a classification model, which can improve the accuracy of document structure identification in the candidate area. Compared with the prior art, in which only image features are used for classification to determine the document structure type to which the candidate area belongs, the accuracy of determining the document structure type is improved.

In the document structure recognition method of the embodiment of the present application, a document image is obtained, a candidate area is selected from the document image, image features are extracted for the candidate area to obtain image features, semantic recognition is performed on the characters contained in the candidate area to obtain semantic features, and classification is performed based on the image features and semantic features to determine the document structure type to which the candidate area belongs. In the present application, when performing document structure type recognition, semantic features are added on the basis of image features, the importance of semantic information in document structure recognition is fully considered, and the accuracy of document structure type judgment is improved.

The above embodiments illustrate that the content attribute feature may include whether the corresponding pixel unit belongs to the foreground or the background, or may also include the target text granularity corresponding to the pixel unit, which will be described below through embodiments.

Based on the above embodiment, this embodiment illustrates that the content attribute feature contained in the pixel unit includes whether the corresponding pixel unit belongs to the foreground or the background. FIG3 is a flow chart of another document structure recognition method provided by the embodiment of the present application. As shown in FIG3, as a possible implementation method, the above step 203 may include the following steps:

Step 301, binarize the candidate region to determine the foreground part and the background part from the candidate region.

In one embodiment of the present application, the candidate area is first grayscale processed to obtain a grayscale image, and then the grayscale image is binarized. The grayscale image is obtained by making the values of the three color components of red R, green G, and blue B the same. Since the range of color values is [0, 255], there are 256 grayscale levels, that is, the grayscale image can express 256 grayscale colors. In the present application, a grayscale image can be generated in the following three ways.

As a first possible implementation method, the maximum value method, that is, R=B=G=Max(R,G,B), the brightness of the grayscale image after processing by this method is relatively high.

As a second possible implementation method, the average value method: R=B=G=(R+G+B)/3. After being processed by this method, the grayscale image has a softer color.

As a third possible implementation, the weighted average method is: R = G = B = R * weight 1 + G * weight 2 + B * weight 3, where weight 1, weight 2 and weight 3 are the weights of R, G and B. When the weight values are different, grayscale images with different grayscales can be formed.

After converting the candidate area into a grayscale image as described above, the candidate area is binarized. For example, an adaptive threshold binarization algorithm is used to set the grayscale value of the pixel unit of the foreground part of the image corresponding to the candidate area to 255, and the grayscale value of the pixel unit of the background part is set to 0. That is to say, the candidate area after binarization presents a visual effect that the foreground part is white and the background part is black, so as to determine the foreground part and the background part from the candidate area.

Step 302: Generate content attribute features of each pixel unit according to whether each pixel unit belongs to the foreground part or the background part.

The content attribute features correspond to multiple dimensions, each dimension corresponds to the pixel value of each pixel unit, and the features belonging to the foreground or the background.

In this embodiment, each pixel unit belonging to the foreground and each pixel unit belonging to the background are identified, thereby generating content attribute features of each pixel unit. That is to say, by adding the feature of whether the pixel unit belongs to the foreground or the background to the content attribute feature, the feature information contained in the content attribute feature of each pixel unit is improved.

In the document structure recognition method of the present application, the candidate area is binarized to determine the foreground part and the background part from the candidate area, and the content attribute features of each pixel unit are generated according to whether each pixel unit belongs to the foreground part or the background part, thereby increasing the information contained in the content attribute features of each pixel unit.

In the previous embodiment, it is explained that the content attribute feature includes the foreground or background. In practical applications, the content attribute feature can be further expanded, that is, the content attribute feature in this embodiment also includes the granularity of the target text to which the pixel unit belongs, so as to increase the information contained in the content attribute feature. To this end, this embodiment also provides a document structure recognition method. FIG4 is a flow chart of another document structure recognition method provided by the embodiment of the present application. As shown in FIG4, as another possible implementation method, the above step 203 can include the following steps:

Step 401, binarize the candidate region to determine the foreground part and the background part from the candidate region.

Specifically, reference may be made to step 301 in the previous embodiment, the principle is the same and will not be repeated here.

Step 402 , using the structural element size corresponding to the target text granularity, performs a closing operation on the binary candidate region to expand the foreground portion.

The target text granularity includes at least one of a word, a sentence and a paragraph. The structural element size corresponds to the target text granularity. The structural element size can be represented by a matrix and is a size that can cover the corresponding text granularity.

In one embodiment, due to different target text granularities, different target text granularities correspond to different spatial scales, i.e., structural element sizes. If the structural element size is too large or too small, the target text granularity cannot be accurately determined. Therefore, in this embodiment, different structural element sizes are corresponding to different target text granularities, i.e., the character granularity has the structural element size corresponding to the character, the sentence granularity has the structural element size corresponding to the sentence, and the paragraph has the structural element size corresponding to the paragraph. The structural element size corresponding to the target text granularity is used to perform a closing operation on the binarized candidate area, wherein the closing operation includes an expansion operation and an erosion operation. Specifically, the corresponding structural element size is first used to process the target text granularity of the foreground part of the binarized candidate area using an expansion operation to expand the corresponding target text granularity in the foreground, and then an erosion operation is used to eliminate the noise in the expanded foreground, thereby achieving the expansion of the foreground part while reducing the noise points in the foreground part.

For example, the target text granularity is described as a sentence.

FIG5 shows the result of the first closing operation on a candidate region of binarization, using a structural element size corresponding to the sentence granularity, such as a 5*5 matrix. The sentences in the candidate region are expanded, so that the foreground area corresponding to the sentences is expanded. Then, the erosion operation is used to expand the black background part to eliminate some noise. In order to enhance the connectivity between sentences, the closing operation can be performed multiple times to enhance the expansion effect of the foreground part. FIG6 shows the expanded foreground part obtained after the second closing operation, which achieves full connectivity between sentences to improve the effect of subsequent contour detection.

It should be noted that the size of the structural element can be adjusted according to the specific text granularity and precision, and is not limited in this embodiment.

Step 403: perform contour detection on the foreground portion to obtain a bounding box of the target text granularity.

In the application embodiment, a contour detection algorithm is used to perform contour detection on the foreground part to obtain a bounding box of the target text granularity. Taking sentence granularity as an example, contour detection is performed on the connected sentence parts in Figure 6 to determine the bounding box of each sentence, such as bounding box 1 in Figure 6, where each sentence in Figure 6 can determine a bounding box, which is not marked one by one in Figure 6, nor is it listed one by one in this embodiment.

Step 404: determine the pixel units in the bounding box as belonging to the target text granularity, and determine the pixel units not in the bounding box as not belonging to the target text granularity.

As shown in Figure 6, the bounding box corresponding to each white area in the figure is the bounding box of the target text granularity, so that the pixel units in the bounding box are determined to belong to the target text granularity, and the pixel units not in the bounding box are determined to not belong to the target text granularity. For example, the sentence in the bounding box 1 is sentence 1, and the pixel units in the white bounding box 1 belong to sentence 1, while the black area around the bounding box 1 is not in the bounding box 1, so it is determined not to belong to sentence 1.

Step 405: Generate content attribute features of each pixel unit according to the target text granularity to which each pixel unit belongs.

Among them, the content attribute features correspond to multiple dimensions, and each dimension corresponds to whether each pixel unit corresponds to the foreground or background, and whether it is one of the attribute features of a word, sentence or paragraph. In other words, the feature information contained in each pixel point can be expressed as [pixel value, foreground/background, target text granularity]

In this embodiment, content attribute features of each pixel unit are generated based on the target text granularity to which each pixel unit belongs, that is, the content attribute features of each pixel unit are identified, thereby increasing the amount of information contained in the content attribute features, thereby increasing the amount of information contained in the image features extracted subsequently, thereby improving the accuracy of determining the document structure type.

In the document structure recognition method of the present embodiment, after the candidate area is binarized, a closing operation is performed on the binarized candidate area using the structural element size corresponding to the target text granularity to expand the foreground part, and a bounding box of the target text granularity is obtained by performing contour detection on the foreground part, and the pixel units within the bounding box are determined to belong to the target text granularity, and the pixel units not within the bounding box are determined to not belong to the target text granularity, and the content attribute features of each pixel unit are generated according to the target text granularity to which each pixel unit belongs, thereby increasing the amount of information contained in the content attribute features of each pixel unit.

Based on the above embodiment, this embodiment further provides a document structure recognition method. When selecting a candidate area from a document image, a sliding detection frame method can also be used. FIG. 7 is a flow chart of another document structure recognition method provided by an embodiment of the present application. As shown in FIG. 7, the method includes the following steps:

Step 701, obtaining a document image.

Step 702: Select multiple candidate regions from the document image using a sliding detection frame.

In one embodiment, according to a preset sliding detection frame, in the document image, the sliding detection frame is sequentially moved from left to right and from top to bottom according to a preset step length, and the area framed by the sliding detection frame each time is used as a candidate area to obtain multiple candidate areas. In this embodiment, in order to improve the accuracy of obtaining the candidate area, the step length of the sliding detection frame movement is set to be small, so that adjacent candidate areas have overlapping parts.

Step 703: Identify content attribute features for each pixel unit in the candidate area.

Step 704: Generate an input image based on the pixel value and content attribute characteristics of each pixel in the candidate area.

Each pixel in the input image has multiple channels, and each channel is used to indicate the pixel value or content feature of the corresponding pixel in the candidate area.

Step 705: extract image features from the input image to obtain image features.

Step 706, obtaining the document content corresponding to the document image.

Step 707: query the relative position in the document content according to the relative position of the candidate area in the document image to obtain the characters contained in the candidate area.

Step 708: Perform semantic recognition on the characters contained in the candidate area to obtain semantic features.

Step 709: concatenate the image features of the candidate region with the semantic features of the candidate region to obtain synthetic features of the candidate region.

Step 710: Classify the candidate region according to its synthetic features to determine the document structure type to which the candidate region belongs.

Among them, steps 703 to 710 may be specifically referred to the relevant explanations in the above embodiments, and will not be described in detail in this embodiment.

Step 711: If at least two candidate regions continuously distributed in the document image belong to the same document structure type, the at least two candidate regions are merged to obtain a merged region and the document structure type of the merged region.

Specifically, if there are continuously distributed overlapping candidate areas among the candidate areas determined by the sliding detection box, that is, multiple continuously distributed candidate areas are classified and identified as belonging to the same document structure type because they contain the same content, in order to reduce the number of candidate areas belonging to the same document structure type, at least two candidate areas can be merged, and the document structure type corresponding to the at least two candidate areas can be used as the document structure type of the merged area to reduce the redundancy of the model output information.

In the document structure recognition method of the embodiment of the present application, a sliding detection frame is used to select multiple candidate areas from a document image, and after classifying the candidate areas according to their synthetic features to determine the document structure type to which the candidate areas belong, if at least two candidate areas continuously distributed in the document image belong to the same document structure type, the at least two candidate areas are merged to obtain a merged area and the document structure type of the merged area to reduce the redundancy of the model output information.

In the above embodiment, a trained document structure recognition model is used to recognize the text structure. In order to implement the above embodiment, the present application also provides a model training method for document structure recognition. FIG8 is a flow chart of a model training method for document structure recognition provided in an embodiment of the present application. As shown in FIG8, the method comprises the following steps:

Step 801, obtaining a training sample set.

In a possible implementation method of an embodiment of the present application, a page is obtained, for example, by using crawler technology to crawl a large number of web documents, extract text content in the page, and parse the document structure tree of the page to obtain text areas corresponding to each document structure type, and the text content is annotated according to the text areas corresponding to each document structure type, wherein the annotation includes annotating the table area with the document structure type as a table, annotating the image area with the document structure type as an image, annotating the title area with the document structure type as a title, annotating the paragraph area with the document structure type as a paragraph, and annotating the footnote area with the document structure type as a footnote, so as to obtain training samples in the training sample set, thereby achieving the generation of a large-scale training sample set based on the existing page data, thereby improving the generation efficiency.

For example, as shown in Figure 9, taking a resume text as an example, the text content of the resume text is extracted, and the document structure tree of the resume page is parsed to obtain a text area 91 corresponding to the document structure type of the table, a text area 93 corresponding to the document structure type of the picture, a text area 92 corresponding to the document structure type of the title, and a text area 94 corresponding to the document structure type of the text. Then, the text content corresponding to each text area in the resume text is annotated, that is, area 92 is annotated with the title, area 93 is annotated with the image, area 91 is annotated with the table, and area 94 is annotated with the text.

In another possible implementation of the embodiment of the present application, layout information is randomly generated, and a training document is generated based on the layout information. According to the layout information, text areas corresponding to each document structure type are annotated in the training document to obtain training samples in a training sample set. This enables corresponding layout information to be preset based on demand, thereby meeting sample generation requirements in different scenarios.

For example, layout information is randomly generated according to preset layout information and filling content. For example, the layout information includes pictures, texts, and titles, and then a training document is generated after filling in the corresponding content according to the layout information. Since the document structure type and content of each text area in the training document are known, the content contained in each text area can be annotated to generate training samples in the training sample set, thereby realizing the generation of a large-scale training sample set.

In another possible implementation of the embodiment of the present application, the existing large-scale English version training sample set can be generated by not only retaining the corresponding English version training samples, but also obtaining the corresponding Chinese version through text translation to generate a large-scale training sample set.

Step 802, using a training sample set to train a target detection model, wherein the target detection model is used to select a candidate area from the document image, perform image feature extraction on the candidate area to obtain image features, perform semantic recognition on characters contained in the candidate area to obtain semantic features, and perform target detection based on the image features and semantic features to determine the document structure type to which the candidate area belongs.

In one embodiment of the present application, a training sample set is used to train the target detection model. Specifically, the target detection model can be trained by deep learning. Compared with other deep learning methods, deep learning performs better on large data sets and can improve the training effect of the target detection model. The goal of optimizing the target detection model training is to minimize the objective function, that is, the error between the document structure type of each text area in the classified training sample and the document structure type of each labeled text area is minimized, and then the target detection model training is completed, so that the trained target detection model is used to select candidate areas from document images, extract image features of the candidate areas to obtain image features, perform semantic recognition on the characters contained in the candidate areas to obtain semantic features, and perform target detection based on image features and semantic features to determine the document structure type to which the candidate areas belong.

It should be noted that the method of using the trained target detection model to identify the document structure type can be referred to the description in the corresponding embodiments of Figures 1 to 7, and will not be repeated in this embodiment.

In the model training method for document structure recognition in the embodiment of the present application, a target detection model is trained by constructing a large-scale training corpus, so that the trained target detection model can be used to obtain a document image, select a candidate area from the document image, extract image features from the candidate area to obtain image features, perform semantic recognition on the characters contained in the candidate area to obtain semantic features, and classify according to the image features and semantic features to determine the document structure type to which the candidate area belongs. In the present application, when performing document structure type recognition, semantic features are added on the basis of image features, the importance of semantic information in document structure recognition is fully considered, and the accuracy of the document structure type judgment made by the document structure recognition model is improved.

In order to implement the above embodiment, the present application also proposes a document structure recognition device.

FIG. 10 is a schematic diagram of the structure of a document structure recognition device provided in an embodiment of the present application.

As shown in FIG. 10 , the device includes: an image acquisition module 101 , a selection module 102 , an extraction module 103 , a recognition module 104 and a detection module 105 .

The image acquisition module 101 is used to acquire a document image.

The selection module 102 is used to select a candidate region from the document image.

The extraction module 103 is used to extract image features from the candidate area to obtain image features.

The recognition module 104 is used to perform semantic recognition on the characters contained in the candidate area to obtain semantic features.

The detection module 105 is used to perform object detection based on image features and semantic features to determine the document structure type to which the candidate region belongs.

In a possible implementation of the embodiment of the present application, the device further includes

The acquisition module is used to acquire the document content corresponding to the document image.

The query module is used to query the relative position in the document content according to the relative position of the candidate area in the document image, so as to obtain the characters contained in the candidate area.

As a possible implementation, the detection module 105 is specifically used to:

The image features of the candidate region are concatenated with the semantic features of the candidate region to obtain synthetic features of the candidate region, and classification is performed according to the synthetic features of the candidate region to determine the document structure type to which the candidate region belongs.

In a possible implementation of the embodiment of the present application, the extraction module 103 includes:

An identification unit, used for identifying content attribute features of each pixel unit in the candidate area;

A generating unit, configured to generate an input image according to a pixel value of each pixel in the candidate area and the content attribute feature, wherein each pixel in the input image has a plurality of channels, and each channel is used to indicate the pixel value or the content feature of a corresponding pixel in the candidate area;

The extraction unit is used to extract image features from the input image to obtain the image features.

As a possible implementation manner, the above identification unit is specifically used to:

The candidate area is binarized to determine a foreground part and a background part from the candidate area; and the content attribute feature of each pixel unit is generated according to whether each pixel unit belongs to the foreground part or the background part.

As another possible implementation, the identification unit is further configured to:

Binarize the candidate area to determine the foreground part and the background part from the candidate area; use the structural element size corresponding to the target text granularity to perform a closing operation on the binarized candidate area to expand the foreground part; wherein the target text granularity includes at least one of a word, a sentence and a paragraph; perform contour detection on the foreground part to obtain a bounding box of the target text granularity; determine the pixel units in the bounding box as belonging to the target text granularity, and determine the pixel units not in the bounding box as not belonging to the target text granularity; generate the content attribute feature of each pixel unit according to the target text granularity to which each pixel unit belongs.

In a possible implementation of the embodiment of the present application, the selection module 102 is specifically used to:

Selecting a plurality of candidate regions from the document image using a sliding detection frame;

Correspondingly, the device further includes:

The merging module is used for merging at least two candidate areas distributed continuously in the document image to obtain a merged area and the document structure type of the merged area if the at least two candidate areas belong to the same document structure type.

It should be noted that the above explanation of the document structure recognition method embodiment is also applicable to the document structure recognition device of this embodiment, and the principles are the same, which will not be repeated here.

In the document structure recognition method of the embodiment of the present application, a document image is obtained, a candidate area is selected from the document image, image features are extracted for the candidate area to obtain image features, semantic recognition is performed on the characters contained in the candidate area to obtain semantic features, and classification is performed based on the image features and semantic features to determine the document structure type to which the candidate area belongs. In the present application, when performing document structure type recognition, semantic features are added on the basis of image features, the importance of semantic information in document structure recognition is fully considered, and the accuracy of document structure type judgment is improved.

In order to implement the above embodiment, the present application also proposes a model training device for document structure recognition.

FIG11 is a schematic diagram of the structure of a model training device for document structure recognition provided in an embodiment of the present application.

As shown in FIG. 11 , the device includes: a sample acquisition module 111 and a training module 112 .

The sample acquisition module 111 is used to acquire a training sample set.

The training module 112 is used to train the target detection model using a training sample set, wherein the target detection model is used to select a candidate area from a document image, perform image feature extraction on the candidate area to obtain image features, perform semantic recognition on characters contained in the candidate area to obtain semantic features, and perform target detection based on the image features and semantic features to determine the document structure type to which the candidate area belongs.

In a possible implementation of the embodiment of the present application, the sample acquisition module 111 includes:

The acquisition unit is used to acquire the page.

The extraction module is used to extract the text content in the page.

The parsing unit is used to parse the document structure tree of the page to obtain the text area corresponding to each document structure type.

The marking unit is used to mark the text content according to the text area corresponding to each of the document structure types to obtain training samples in the training sample set.

In another possible implementation of the embodiment of the present application, the sample acquisition module 111 further includes:

The generating unit is used to randomly generate layout information, and generate a training document according to the layout information.

The above-mentioned marking unit is used to mark the text area corresponding to each document structure type in the training document according to the layout information, so as to obtain the training samples in the training sample set.

It should be noted that the aforementioned explanation of the embodiment of the model training method for document structure recognition is also applicable to the model training device for document structure recognition in this embodiment. The principles are the same and will not be repeated here.

In the model training device for document structure recognition of the embodiment of the present application, a target detection model is trained by constructing a large-scale training corpus, so that the trained target detection model can be used to obtain a document image, select a candidate area from the document image, extract image features from the candidate area to obtain image features, perform semantic recognition on the characters contained in the candidate area to obtain semantic features, and classify according to the image features and semantic features to determine the document structure type to which the candidate area belongs. In the present application, when performing document structure type recognition, semantic features are added on the basis of image features, the importance of semantic information in document structure recognition is fully considered, and the accuracy of the document structure type judgment made by the document structure recognition model is improved.

In order to implement the above embodiment, the embodiment of the present application further provides an electronic device, including:

at least one processor; and

a memory communicatively connected to the at least one processor; wherein,

The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the document structure recognition method described in the aforementioned method embodiment or the model training method for document structure recognition described in the aforementioned method embodiment.

In order to implement the above-mentioned embodiments, the embodiments of the present application propose a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the document structure recognition method described in the above-mentioned method embodiments or the model training method for document structure recognition described in the above-mentioned method embodiments.

According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

As shown in Figure 12, it is a block diagram of an electronic device according to the document structure recognition method or the model training method of document structure recognition of an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and/or required herein.

As shown in Figure 12, the electronic device includes: one or more processors 1201, memory 1202, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input/output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and/or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). In Figure 12, a processor 1201 is taken as an example.

The memory 1202 is the non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to execute the document structure recognition method provided in the present application, or to execute the model training method for document structure recognition. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to execute the document structure recognition method provided in the present application, or to execute the model training method for document structure recognition.

The memory 1202, as a non-transient computer-readable storage medium, can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions/modules corresponding to the document structure recognition method in the embodiment of the present application (for example, the image acquisition module 101, the selection module 102, the extraction module 103, the recognition module 104 and the detection module 105 shown in FIG. 10). The processor 1201 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1202, that is, implements the document structure recognition method in the above method embodiment. Similarly, the model training method for document structure recognition in the above method embodiment can be implemented, and the principle is the same, which will not be repeated here.

The memory 1202 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the use of an electronic device according to a document structure recognition method or a model training method for document structure recognition, etc. In addition, the memory 1202 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1202 may optionally include a memory remotely arranged relative to the processor 1201, and these remote memories may be connected to the electronic device of the document structure recognition method or the model training method for document structure recognition via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The electronic device of the document structure recognition method or the document structure recognition model training method may further include: an input device 1203 and an output device 1204. The processor 1201, the memory 1202, the input device 1203 and the output device 1204 may be connected via a bus or other means, and FIG12 takes the bus connection as an example.

The input device 1203 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device of the document structure recognition method or the model training method of document structure recognition, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device Y04 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and/or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and/or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and/or data to a programmable processor.

To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

According to the technical solution of the embodiment of the present application, which involves the field of natural language processing and deep learning technology, a document image is obtained, a candidate area is selected from the document image, image features are extracted from the candidate area to obtain image features, semantic recognition is performed on the characters contained in the candidate area to obtain semantic features, and classification is performed based on the image features and semantic features to determine the document structure type to which the candidate area belongs. In the present application, when identifying the document structure type, semantic features are added on the basis of image features, the importance of semantic information in document structure identification is fully considered, and the accuracy of document structure type judgment is improved.

It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims (21)

1. A document structure identification method, the method comprising:

Acquiring a document image;

Selecting a candidate region from the document image;

Extracting image features of the candidate areas to obtain image features;

carrying out semantic recognition on characters contained in the candidate region to obtain semantic features;

classifying according to the image features and the semantic features to determine document structure types of the candidate regions, wherein the document structure types comprise texts, tables, titles, paragraphs, footnotes and images;

The extracting the image features of the candidate region to obtain the image features includes:

Identifying a content attribute feature for each pixel cell in the candidate region;

generating an input image according to the pixel value of each pixel in the candidate region and the content attribute characteristics, wherein each pixel point in the input image is provided with a plurality of channels, and each channel is used for indicating the pixel value and the content attribute characteristics of the corresponding pixel in the candidate region;

and extracting image features of the input image to obtain the image features.

2. The document structure recognition method of claim 1, wherein the classifying according to the image feature and the semantic feature to determine a document structure type to which the candidate region belongs comprises:

Splicing the image features of the candidate region with the semantic features of the candidate region to obtain synthetic features of the candidate region;

classifying according to the synthesized characteristics of the candidate region to determine the document structure type of the candidate region.

3. The document structure recognition method of claim 1, wherein the identifying content attribute features for each pixel unit in the candidate region comprises:

Binarizing the candidate region to determine a foreground portion and a background portion from the candidate region;

And generating the content attribute characteristics of each pixel unit according to whether each pixel unit belongs to the foreground part or the background part.

4. The document structure recognition method of claim 1, wherein the identifying content attribute features for each pixel unit in the candidate region comprises:

Binarizing the candidate region to determine a foreground portion and a background portion from the candidate region;

Performing a closed operation on the binarized candidate region by adopting the size of the structural element corresponding to the granularity of the target text so as to expand the foreground part; wherein the target text granularity comprises at least one of words, sentences, and paragraphs;

performing contour detection on the foreground part to obtain a bounding box of the target text granularity;

Determining that the pixel units in the bounding box belong to the target text granularity, and determining that the pixel units not in the bounding box do not belong to the target text granularity;

And generating the content attribute characteristics of each pixel unit according to the target text granularity to which each pixel unit belongs.

5. The document structure recognition method according to any one of claims 1 to 4, wherein the selecting a candidate region from the document image includes:

selecting a plurality of candidate areas from the document image by adopting a sliding detection frame;

Correspondingly, after classifying according to the image features and the semantic features of the candidate region to determine the document structure type to which the candidate region belongs, the method further comprises:

and if at least two candidate areas continuously distributed in the document image belong to the same document structure type, merging the at least two candidate areas to obtain a merging area and the document structure type of the merging area.

6. The document structure recognition method according to any one of claims 1 to 4, wherein, before said semantically recognizing the characters contained in each of the candidate regions, further comprising:

acquiring document content corresponding to the document image;

And according to the relative position of the candidate region in the document image, inquiring the relative position in the document content to obtain the characters contained in the candidate region.

7. A model training method for document structure recognition, the method comprising:

acquiring a training sample set;

training a target detection model by adopting the training sample set, wherein the target detection model is used for selecting a candidate region from a document image, extracting image features of the candidate region to obtain image features, carrying out semantic recognition on characters contained in the candidate region to obtain semantic features, and carrying out target detection according to the image features and the semantic features to determine the document structure type of the candidate region, wherein the document structure type comprises texts, tables, titles, paragraphs, footnotes and images;

When training the target detection model by adopting the training sample set, selecting a candidate region from a document image by the target detection model, and extracting image features of the candidate region to obtain image features, wherein the method comprises the following steps:

Identifying a content attribute feature for each pixel cell in the candidate region;

generating an input image according to the pixel value of each pixel in the candidate region and the content attribute characteristics, wherein each pixel point in the input image is provided with a plurality of channels, and each channel is used for indicating the pixel value and the content attribute characteristics of the corresponding pixel in the candidate region;

and extracting image features of the input image to obtain the image features.

8. The model training method of claim 7, wherein the acquiring a training sample set comprises:

Acquiring a page;

Extracting text content in the page;

Analyzing the document structure tree of the page to obtain text areas corresponding to the document structure types;

And labeling the text content according to the text region corresponding to each document structure type to obtain training samples in the training sample set.

9. The model training method of claim 7, wherein the acquiring a training sample set comprises:

randomly generating layout information;

generating a training document according to the layout information;

And marking text areas corresponding to the document structure types in the training documents according to the layout information to obtain training samples in the training sample set.

10. A document structure identification apparatus, the apparatus comprising:

the image acquisition module is used for acquiring a document image;

the selecting module is used for selecting candidate areas from the document image;

the extraction module is used for extracting the image characteristics of the candidate areas to obtain the image characteristics;

the identification module is used for carrying out semantic identification on the characters contained in the candidate region to obtain semantic features;

the detection module is used for carrying out target detection according to the image characteristics and the semantic characteristics so as to determine the document structure type of the candidate region, wherein the document structure type comprises a text, a table, a title, a paragraph, a footnote and an image;

Wherein, the extraction module includes:

an identifying unit configured to identify a content attribute feature for each pixel unit in the candidate region;

A generating unit, configured to generate an input image according to a pixel value of each pixel in the candidate region and the content attribute feature, where each pixel point in the input image has a plurality of channels, and each channel is configured to indicate the pixel value and the content attribute feature of a corresponding pixel in the candidate region;

and the extraction unit is used for extracting the image characteristics of the input image so as to obtain the image characteristics.

11. The document structure identification device of claim 10, wherein the detection module is specifically configured to:

Splicing the image features of the candidate region with the semantic features of the candidate region to obtain synthetic features of the candidate region;

classifying according to the synthesized characteristics of the candidate region to determine the document structure type of the candidate region.

12. The document structure identification device according to claim 10, wherein the identification unit is specifically configured to:

Binarizing the candidate region to determine a foreground portion and a background portion from the candidate region;

And generating the content attribute characteristics of each pixel unit according to whether each pixel unit belongs to the foreground part or the background part.

13. The document structure identification device according to claim 10, wherein the identification unit is further specifically configured to:

Binarizing the candidate region to determine a foreground portion and a background portion from the candidate region;

Performing a closed operation on the binarized candidate region by adopting the size of the structural element corresponding to the granularity of the target text so as to expand the foreground part; wherein the target text granularity comprises at least one of words, sentences, and paragraphs;

performing contour detection on the foreground part to obtain a bounding box of the target text granularity;

Determining that the pixel units in the bounding box belong to the target text granularity, and determining that the pixel units not in the bounding box do not belong to the target text granularity;

And generating the content attribute characteristics of each pixel unit according to the target text granularity to which each pixel unit belongs.

14. The document structure identification apparatus according to any one of claims 10-13, wherein the selecting module is specifically configured to:

selecting a plurality of candidate areas from the document image by adopting a sliding detection frame;

correspondingly, the device further comprises:

And the merging module is used for merging the at least two candidate areas if at least two candidate areas continuously distributed in the document image belong to the same document structure type, so as to obtain a merging area and the document structure type of the merging area.

15. The document structure identification apparatus according to any one of claims 10 to 13, wherein the apparatus further comprises:

the acquisition module is used for acquiring the document content corresponding to the document image;

And the query module is used for querying the relative position in the document content according to the relative position of the candidate region in the document image so as to obtain the characters contained in the candidate region.

16. A model training apparatus for document structure recognition, the apparatus comprising:

the sample acquisition module is used for acquiring a training sample set;

The training module is used for training a target detection model by adopting the training sample set, wherein the target detection model is used for selecting a candidate region from a document image, extracting image features of the candidate region to obtain image features, carrying out semantic recognition on characters contained in the candidate region to obtain semantic features, and carrying out target detection according to the image features and the semantic features to determine the document structure type of the candidate region, wherein the document structure type comprises a text, a table, a title, a paragraph, a footnote and an image;

When the training sample set is adopted to train the target detection model, the training module is specifically configured to:

Identifying a content attribute feature for each pixel cell in the candidate region;

generating an input image according to the pixel value of each pixel in the candidate region and the content attribute characteristics, wherein each pixel point in the input image is provided with a plurality of channels, and each channel is used for indicating the pixel value and the content attribute characteristics of the corresponding pixel in the candidate region;

and extracting image features of the input image to obtain the image features.

17. The model training apparatus of claim 16 wherein said sample acquisition module comprises:

the acquisition unit is used for acquiring the page;

The extraction module is used for extracting text content in the page;

The analysis unit is used for analyzing the document structure tree of the page to obtain text areas corresponding to the document structure types;

and the labeling unit is used for labeling the text content according to the text region corresponding to each document structure type so as to obtain training samples in the training sample set.

18. The model training apparatus of claim 16 wherein said acquisition module comprises:

the generating unit is used for randomly generating layout information and generating a training document according to the layout information;

And the labeling unit is used for labeling text areas corresponding to the structure types of each document in the training document according to the layout information so as to obtain training samples in the training sample set.

19. An electronic device, comprising:

At least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the document structure identification method of any one of claims 1-6 or the model training method of document structure identification of any one of claims 7-9.

20. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the document structure recognition method of any one of claims 1-6, or the model training method of document structure recognition of any one of claims 7-9.

21. A computer program product comprising a computer program which, when executed by a processor, implements the document structure identification method of any one of claims 1 to 6 or the model training method of document structure identification of any one of claims 7 to 9.