RU2774665C1 - Method for recognising chemical information from images of document and system for implementation thereof - Google Patents

Abstract

FIELD: computing technology.

SUBSTANCE: invention relates to the field of computing technology for data recognition. The computer-implemented method includes the following stages: inputting an image of a document page into a detector; the detector identifying fragments on the page; obtaining coordinates of the fragment on the page for each identified fragment; and classifying the fragments; the structure recognition unit recognises the chemical structure for each fragment; inputting the identified reaction arrow fragments into the arrow recognition unit; obtaining coordinates on the page for each arrow and attributes of the reaction; supplying the coordinates on the page to the input of the reaction recognition unit for each fragment of the recognised chemical structures; and, based on the received data, the reaction recognition unit determines the relation between the arrows and the recognised chemical structures; recognised chemical structures are obtained as a result based on the recognised data for the image of a document page.

EFFECT: ensured automatic recognition of chemical information from images of documents, reduction in the duration and increase in the accuracy of recognition of chemical information from images of documents.

17 cl, 7 dwg

Description

FIELD OF TECHNOLOGY

The invention relates to the field of data recognition, in particular to a method for recognizing chemical information from document images and a system for its implementation.

The presented solution can be used at least in pharmaceutical companies to collect data on a variety of chemical information, for example, chemical compounds presented in various formats, chemical reactions and additional chemical information, in other areas of technology in which it is necessary to collect data on such chemical information. Also, this solution can be used by chemical information providers to compile databases.

BACKGROUND OF THE INVENTION

In the patent application CN111860507A, published on 10/30/2020, a method for extracting the molecular structural formula of a composite image based on adversarial learning related to the field of deep learning, image recognition and extracting a composite molecular formula is described. A method for extracting a molecular structural formula of a composite image based on adversarial learning includes the following steps: building a set of data pairs consisting of composite images and molecular structures in the form of SMILES notation; creation of adversarial networks: SMILES generator and structure recognizer in SMILES notation, and adversarial training of the proposed neural network model.

In the international application for invention WO2019148852A1, publication date 08/08/2019, a method for recognizing chemical information from images of hand-drawn drawings by identifying structures, identifying handwriting, identifying atoms corresponding to structures, identifying bonds using deep learning methods is disclosed.

Patent document EP2567338B1, publication date 04/08/2020, discloses a device for electronic identification and compilation of chemical structures found in a vault in which electronic files are stored. The optical structure recognition module identifies a plurality of possible chemical structures in the electronic storage files, wherein at least one of the electronic files contains non-embedded images of the chemical structures identified by the optical structure. The recognition module outputs (for each potential chemical structure identified) a chemical structure object with an associated set of properties, including the number of carbon atoms (eg, number of heteroatoms, number of bonds, number of bonds of the chosen bond order, number of rings, and formula weight). The optical structure recognition module also applies (for each derived chemical structure object) one or more filters, including a filter to exclude objects identified as having less than a selected number of carbon atoms, with the selected number of carbon atoms being configured and set by the user based on the expected content of electronic files. , and stores objects not removed by one or more filters in a searchable electronic database of identified objects.

In the application for invention CN112818645A, publication date 05/18/2021, a method and a device for extracting chemical information are described, in which a document containing chemical information is obtained, an image and text are extracted from a document containing chemical information, the chemical structure and its corresponding label are extracted, and connections between the chemical structure and the label, the extraction of the chemical object and the relationship between chemical objects from the text.

In the article [1], the authors presented a new model DECIMER (Deep lEarning for Chemical Image Recognition). A network based on the neural architecture of the Transformer is disclosed that can recognize molecular structures in the form of SMILES notation with an accuracy of more than 96% for images of chemical structures without stereochemical information and with an accuracy of more than 89% for images with stereochemical information.

In the article [2], the authors address the problem of image-to-text translation specifically for molecular structures, where the result is a predicted chemical designation in the InChI format for a given molecular structure. Current approaches are mostly rule based or CNN + RNN based methodology. However, according to the authors of [2], they show worse results on noisy images and images with a small amount of distinguishable details. To overcome these limitations, the authors proposed an end-to-end transformer model. Compared to attention-based methods, the proposed model demonstrates better performance.

In [3], the authors present a fast and accurate model that combines a deep convolutional neural network that learns from molecular images and a pretrained decoder that translates the latent representation into a representation of SMILES molecules. The method, named Img2Mol by the authors, is capable of correctly recognizing up to 88% of molecular images and converting them to the SMILES representation.

In chemical journals, scientific articles, patents, technical reports, dissertations and other chemical documents, key information is presented in the form of a representation not only of molecular structures in a standardized format, but also of structures written in a non-standardized format, indicating "pseudochemical groups" R1, R2 and etc. In general, such structures are called Markush structures. Also, chemical formulas often include the designations of chemical groups in the form of abbreviations - for example, "Ph-", "MeO-". In addition, key chemical information is also presented in the form of chemical reactions. However, in the solutions of the prior art there is no possibility of optical recognition from the original optical scans of various chemical documents of chemical information in the form of structures recorded in a non-standardized or abbreviated format, as well as in the form of chemical reactions.

Optical recognition of this kind of information is a very difficult task. The technical problem to be solved by the claimed invention is the recognition of chemical information from chemical documents, containing both chemical structures written in a standardized and non-standardized or abbreviated format, as well as chemical reactions.

SUMMARY OF THE INVENTION

The technical result of the claimed invention is the provision of automatic recognition of chemical information from document images, containing both chemical structures recorded in a standardized and non-standardized or abbreviated format, as well as chemical reactions, reducing the time and increasing the accuracy of recognition of chemical information from document images. An additional technical result is an increase in the performance of the computing system when solving the task, i.e. The present solution makes it possible to process documents to obtain a recognition result in less time, thereby reducing the load on the central processor of the computing device.

This technical result is achieved due to the fact that a computer-implemented method for recognizing chemical information from document images, in which a computing device containing a processor and memory stores in memory instructions executed by the processor, and executes instructions, including the steps at which:

- the image of the document page is fed to the input of the detector; the detector using the first neural network identifies on the page one or more fragments containing chemical information; for each identified fragment get the coordinates of the fragment on the page; and classifying the fragments into at least the following categories: chemical structure, reaction arrow;

- one or more identified fragments of chemical structures are fed to the input of the structure recognition block, each fragment being an image; the structure recognition unit for each fragment recognizes the chemical structure using the second neural network;

- one or more identified fragments of reaction arrows are fed to the input of the arrow recognition block; the arrow recognition unit, using the third neural network, determines the type of the arrow, and using the fourth neural network, the coordinates on the page for each arrow, and the response attributes are obtained;

- the coordinates on the page of each fragment of the recognized chemical structures, the corresponding recognized chemical structures, the coordinates on the page of each reaction arrow, the arrow type, the reaction attributes are transmitted to the input of the reaction recognition block; and based on the received data, the reaction recognition block determines how the arrows connect the recognized chemical structures;

- as a result, based on the recognized data for the document page image, one or more recognized chemical structures are obtained, coordinates on the page for each recognized chemical structure, recognized relationships between substances participating in a chemical reaction, represented as chemical structures, coordinates on the page for each recognized relations.

In the method, the chemical structure may be at least a chemical compound, a Markush structure, a chemical structure with substituents.

The method may additionally identify fragments containing additional information to help recognize reactions.

In the method, additional information may include at least the following: title, legend.

In the method, the detector additionally for each identified fragment can determine confidence - a number from 0 to 1, which evaluates the reliability of the identified fragment, where 0 is absolutely not sure, 1 is completely sure.

In the method, the identified fragments may be filtered by a predetermined confidence threshold.

The method may set a confidence threshold for each category of fragments.

In the method, the first neural network may be a Faster R-CNN neural network or other convolutional network of equal or greater power.

In the way the second neural network may be a neural network based on a transformer architecture, and the pattern recognition block comprises a convolutional block and a transformer decoder.

The method can use ResNet-50 without the last two layers as a convolutional block, or another convolutional network that works with images.

In the method, the recognized chemical structure may be a text sequence that uniquely describes the chemical structure.

The method may describe the chemical structure as a text sequence using a SMILES modification capable of describing Markush structures and chemical structures with substituents.

The method can implement a mechanism for converting the SMILES modification, which is capable of describing Markush structures and chemical structures with substituents, to SMILES and vice versa.

In the method, the third and fourth neural networks may be ResNet-based convolutional neural networks.

In the method, the arrows can be classified into the following types: a straight arrow, an arrow that is not a straight arrow.

In the method, the substances participating in the chemical reaction may be the initial substances of the chemical reaction, the products of the chemical reaction.

The system for recognition of chemical information from document images contains:

- detector;

- structure recognition block;

- arrow recognition unit;

- reaction recognition block;

and wherein the computing device, comprising the processor and a memory storing instructions executable by the processor, performs the above-described method.

DESCRIPTION OF THE DRAWINGS

The implementation of the invention will be described hereinafter in accordance with the accompanying drawings, which are presented to explain the essence of the invention and in no way limit the scope of the invention.

The claimed invention is illustrated by figures 1-4, which depict:

Fig. 1 illustrates a block diagram of a system for recognizing chemical information from document images.

Fig. 2 illustrates a block diagram of a modified transformer.

Fig. 3a, 3b, 3c, 3d illustrate an example of the operation of a system for recognizing chemical information from document images.

Fig. 4 illustrates a general diagram of a computing device for implementing the present invention.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the implementation of the invention, numerous implementation details are provided to provide a clear understanding of the present invention. However, it will be apparent to one skilled in the art how the present invention can be used, both with and without these implementation details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to unnecessarily obscure the features of the present invention.

Furthermore, it will be clear from the foregoing that the invention is not limited to the present implementation. Numerous possible modifications, changes, variations and substitutions that retain the spirit and form of the present invention will be apparent to those skilled in the subject area.

The present invention is an automated mechanism for recognizing chemical information from real chemical documents.

On FIG. Figure 1 shows the architecture of a system for recognizing chemical information from document images. System (100) contains the following main blocks:

- A detector (101) that localizes individual elements of chemical information on a scan of a chemical document using a neural network based on the Faster R-CNN architecture or any other more powerful convolutional network.

- A structure recognition block (102), which, using a neural network based on a modified transformer architecture (Transformer), solves the problem of translating an image into text (image captioning). The architecture of the Transformer is modified in such a way that not a sequence, but an image is fed to the network input. The modified Transformer architecture lacks the encoder block.

- An arrow recognition unit (103), which classifies the arrows representing the direction of the reaction with the aid of a neural network. The neural network recognizes the coordinates of the beginning and end of the arrows, as well as the attributes of the reactions.

- A Reaction Recognition Block (104), which determines how the arrows link the recognized chemical structures.

The system input is a scan of the document page, which is transferred to the detector (101). The detector (101) using a neural network finds rectangular areas on the page containing chemical information - molecules and arrows of reactions, as well as additional information that helps in recognizing reactions, for example, titles and legends of schemes.

The detector (101) basically has a Faster R-CNN network, but can equally be implemented on the basis of any neural network architecture that solves the detection problem (YOLO, SDD, EfficientDet, etc.).

For each fragment found, the following are returned:

1. Coordinates on the original page;

2. Category (molecule, arrow, title, legend);

3. Confidence - a number from 0 (absolutely not sure) to 1 (completely sure).

The received objects are filtered by a preset confidence threshold (for example, 0.8), i.e. a fragment with a confidence value less than 0.8 will be rejected as unreliable. The threshold can be set for each category of fragments.

To train the detector (101), we used manually labeled data using a labeling interface specially written for this purpose. In total, 2500 pages of articles were marked up. Further filling of the training data set is possible in a semi-automatic mode, when the detector prediction is corrected manually.

The structure recognition block (102) converts images of chemical structures in chemical documents into structures in the form of a text sequence using a neural network. The images of chemical structures that are fed to the structure recognition unit (102) describe, for example, the spatial structures of molecules, the starting materials involved in chemical reactions, the products of chemical reactions, and so on. In addition, the structure recognition unit (102) recognizes chemical structures that are displayed in both standardized and non-standardized or abbreviated format. The neural network is implemented on the basis of the Transformer architecture. An ordinary transformer solves the sequence2sequence problem, i.e. translation from one sequence to another, for example, machine translation. In the present invention, the input is not a 1D sequence, but an image (2D sequence), so a modified version of the transformer is used (Fig. 2). The modified transformer (200) contains a convolutional block (201) and a transformer decoder (202). The encoder block, which is used in a conventional transformer, has been completely replaced by a convolutional block (201). ResNet-50 is used as the convolution block (201) without the last two layers. Thus, when a 384x384 image is supplied, the output from the convolution block (201) is a 512x48x48 matrix, which is equivalent to a transformer encoder with a depth of 512 and an input sequence length of 48. At the same time, the convolution block (201) can be not only ResNet, but any other convolutional network that works with an image, from a variety of options, for example, EfficientNet, DenseNet, etc.

The transformer (200) converts the image into a text sequence that uniquely describes the spatial structure of the molecule. The description language of a molecule can be any representation: SMILES and its variations (DeepSMILES, SELFIES), as well as InChI or IUPAC name. SMILES-based options are preferred because they are the most concise and reflect the structure directly.

For textual representation of Markush structures, as well as structures with substituents, the FG-SMILES (Functional Group SMILES) language has been developed. FG-SMILES is an extension of the conventional SMILES chemical text description language. FG-SMILES allows you to write both structures with functional groups in abbreviated form, and Markush structures. The CXSMILES language, a well-known analogue for writing substituent groups and Markush structures, is too verbose in comparison with FG-SMILES, and also does not allow writing R-groups in an indefinite position. In addition, for FG-SMILES a mechanism for converting to SMILES and vice versa is implemented.

This notation allows you to write functional groups as substituents of atoms, for example:

[Et]N([Et])CCCNc1nc([X])nc([R3])c1[R2]

In classic SMILES, inorganic atoms, ions, isotopes, and also stereo atoms are written in square brackets. In this modification, the abbreviated names of functional groups, as well as R-groups, are written in a similar way. A way is also presented of linking an R-group to a cycle, rather than to a specific position in the cycle (if you want to show that the R-group is in an indefinite position in the cycle).

There is also a CXSMILES extension that solves a similar problem (except for the indeterminate position), but the corresponding representation is unintuitive and many times longer, which negatively affects the ability to use CXSMILES in machine learning. CXSMILES string corresponding to the above example:

*c1nc(*)c(*)c(NCCCN(*)*)n1 |atomProp:0.dummyLabel.X:4.dummyLabel.R3:6.dummyLabel.R2:13.dummyLabel.Et:14.dummyLabel.Et |

The implemented SMILES ↔ FG-SMILES conversion mechanism makes it possible to find known functional groups in SMILES and replace them. The list of functional groups includes more than 100 groups. These are not all possible groups, but this list covers the vast majority of real examples from articles, and can also be expanded.

The data for training the structure recognition model is artificially generated using a data generator based on the method of creating an artificial dataset that simulates data from real scientific articles. This is due to the fact that the authors of real chemical articles take considerable liberties when depicting structures. In addition to differences in fonts, line weights, and padding, there are often elements of artistic design. In order for the model to be resistant to such features of real data, the generator applies random non-linear geometric distortions to images, and also adds random chemically meaningful garbage - fragments of other molecules, arrows and inscriptions to the free space of the image.

The data generator generates a random modification of the base molecule, produces an image of the modified molecule, and the corresponding FG-SMILES.

The generator takes a SMILES string as input. It then looks for functional groups in the corresponding molecule, and replaces a randomly selected part of them with a short representation, thus forming FG-SMILES. Part of the methyl substituents are changed to a random R-group. Then the molecule is drawn using the RDKit library in accordance with what substitutions of functional groups were made. The resulting image-FG-SMILES pairs are used to train the model.

The trained model returns the forecast as a string FG-SMILES, as well as the confidence in the forecast - a number from 0 to 1. A clear pattern was found between the returned confidence value and the correctness of the answer. Without taking into account the confidence values, the base model has an accuracy of about 90% on the test data. Accuracy is understood as the proportion of complete matches between the real value and the forecast, up to the difference between indices and the number of strokes near R-groups. If we discard examples with a confidence value less than 0.98, then 10% is discarded, but on the remaining examples, the accuracy becomes 97%. At the threshold of 0.99, 15% is cut off, the accuracy on the remaining ones is 98.6%, at the threshold of 0.995, 22% is cut off, the accuracy on the remaining 99.8%.

In fact, this allows us to talk about the achievement of absolute accuracy, if the task is not to recognize everything. In the problem of mass recognition of structures and reactions for automatic filling of databases, it is not of fundamental importance if some of the structures are discarded, but it is of fundamental importance to prevent false data from entering the database. Threshold cutting thus achieves almost absolute precision.

Fragments of the original page, recognized by the detector (101) as "reaction arrows", are transferred to the arrow recognition unit (103). The first network in the arrow recognition block determines whether the fragment is a straight arrow, indicating a reaction. By architecture, the network is the simplest convolutional network based on ResNet. The network was trained on 10,000 fragments obtained from the detector (101) and labeled manually. The network returns "Yes" if the fragment is a single straight arrow, indicating an irreversible reaction, and "No" otherwise. Thus, options are excluded when the arrow represents a reversible reaction, an equilibrium state, a reaction mechanism, or a fragment was erroneously returned by the detector (101).

The arrow start and end coordinates are recognized by another ResNet-based convolutional network with four outputs representing the X and Y position of the arrow start and end coordinates, respectively, in fractions of the length/width of the fragment. To train the network, 7000 real fragments obtained using the detector (101) were manually labeled, for which the first network returned "Yes".

The tasks facing both networks are not difficult by the standards of modern technologies, in contrast to the detector (101) or the pattern recognition unit (102), both networks were trained with an accuracy close to 100%.

The reaction arrow recognition block (103) determines the coordinates of the beginning and end of the arrows on the initial page. Together with the coordinates of the rectangles of the recognized structures, this information is passed to the reaction recognition block (104), which is a logical algorithm that determines for each arrow whether there is a recognized structure for it near the beginning and end of the arrow. If a plausible option is found, then the structure at the beginning of the arrow is considered a reactant, and the structure behind the end of the arrow is considered a reaction product. The reaction recognition block (104) generates the result in the form of ready-made reactions. At the output of block (104), a list of recognized structures with coordinates and confidence values is formed, as well as a list of reactions.

Thus, after processing the page scan, the output of the system (100) is the recognized chemical structures, the coordinates on the page for each recognized chemical structure, the recognized relationships between reactants, agents and chemical reaction products presented as chemical structures, the coordinates on the page for each recognized relations.

An example of recognizing chemical information from document images, which is presented to explain the essence of the invention and in no way limits the scope of the invention.

The input of the detector (101) is fed with an image of a document page (Fig. 3a) containing chemical information. The detector (101) using the first neural network identifies fragments on the page containing chemical information. On FIG. 3b, the fragments identified by the detector (101) are marked with rectangles. For each identified fragment, the coordinates of the fragment on the page are obtained; and classify the fragments into the following categories: chemical structure - rectangles with the structure of molecules, reaction arrow - rectangle with an arrow, additional information - under the rectangles of molecules, two small rectangles "AZD9496", "A1", and one long rectangle containing additional information about the chemical reaction , as well as the title rectangle "Scheme 1" (Fig. 3b). Molecule rectangles have the image class, the arrow rectangle has the condition class, two small rectangles and one long rectangle has the description class, and the title rectangle has the legend class.

One or more identified fragments of chemical structures are fed to the input of the structure recognition block (102), each fragment being an image (Fig. 3c, 301, 302); the structure recognition block (102) for each fragment recognizes the chemical structure using the second neural network.

For fragment (301), the block recognizes FG-SMILES:

FC=1C=C(C=C(C1[C@H]1N([C@@H](CC2C1NC1=CC=CC=C21)C)CC(C)(C)F)F)/C=C /C(=O)O

For fragment (302), the block recognizes FG-SMILES:

C[C@@H]1CC2c3ccccc3N(C)C2[C@@H](c2c(F)cc(/C=C/C(=O)O)cc2F)N1CC(C)(C)F

The identified fragment of the reaction arrow is fed to the input of the arrow recognition block (103) (Fig. 3d). The arrow recognition block (103) using the third neural network determines the type of arrow - a straight arrow, and using the fourth neural network receives the coordinates of the beginning and end of the arrow on the page, and the response attributes.

At the input of the reaction recognition block (104), the coordinates on the page of each fragment of the recognized chemical structures, the corresponding recognized chemical structures, the coordinates on the page of each reaction arrow, the arrow type, the reaction attributes are transmitted; and based on the received data, the reaction recognition block (104) determines how the arrows connect the recognized chemical structures.

As a result, based on the recognized data for the document page image, one or more recognized chemical structures are obtained, coordinates on the page for each recognized chemical structure, recognized relationships between substances participating in a chemical reaction, represented as chemical structures, coordinates on the page for each recognized relationship. .

On FIG. 4 shows a general diagram of a computing device (400) that provides the data processing necessary to implement the claimed solution.

In general, the device (400) contains such components as: one or more processors (401), at least one memory (402), data storage medium (403), input/output interfaces (404), I/O means ( 405), networking tools (406).

The processor (401) of the device performs the basic computing operations necessary for the operation of the device (400) or the functionality of one or more of its components. The processor (401) executes the necessary machine-readable instructions contained in the main memory (402).

The memory (402) is typically in the form of RAM and contains the necessary software logic to provide the required functionality.

The data storage means (403) can be in the form of HDD, SSD disks, raid array, network storage, flash memory, optical information storage devices (CD, DVD, MD, Blue-Ray disks), etc. Means (403) allows you to perform long-term storage of various types of information.

Interfaces (404) are standard means for connecting and working with the server part, for example, USB, RS232, RJ45, LPT, COM, HDMI, PS/2, Lightning, FireWire, etc.

The choice of interfaces (404) depends on the specific implementation of the device (400), which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, and the like.

The keyboard should be used as the data I/O (405) in any embodiment of the system. The keyboard hardware can be any known: it can be either a built-in keyboard used on a laptop or netbook, or a separate device connected to a desktop computer, server, or other computer device. In this case, the connection can be either wired, in which the keyboard connection cable is connected to the PS / 2 or USB port located on the system unit of the desktop computer, or wireless, in which the keyboard exchanges data via a wireless communication channel, for example, a radio channel, with base station, which, in turn, is directly connected to the system unit, for example, to one of the USB ports. In addition to the keyboard, the following I/O devices can also be used: joystick, display (touchscreen), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc.

Means of networking (406) are selected from devices that provide network reception and transmission of data, for example, an Ethernet card, WLAN/Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. With the help of tools (405) the organization of data exchange over a wired or wireless data transmission channel is provided, for example, WAN, PAN, LAN (LAN), Intranet, Internet, WLAN, WMAN or GSM, 3G, 4G, 5G.

The components of the device (400) are coupled via a common data bus (407).

The present application materials provide a preferred disclosure of the implementation of the claimed technical solution, which should not be used as limiting other, private embodiments of its implementation, which do not go beyond the scope of the requested legal protection and are obvious to specialists in the relevant field of technology.

It should be clear to a person skilled in the art that various variations of the proposed method and system do not change the essence of the invention, but only determine its specific embodiments and applications.

Sources

[1] Rajan et al., DECIMER 1.0: Deep Learning for Chemical Image Recognition using Transformers, 2021 https://chemrxiv.org/ndownloader/files/27775521

[2] Sundaramoorthy et al., End-to-End Attention-based Image Captioning, 04/30/2021 https://arxiv.org/pdf/2104.14721.pdf

[3] Clevert et al., Img2Mol-Accurate SMILES Recognition from Molecular Graphical Depictions, 2021 https://chemrxiv.org/ndownloader/files/27273986.

Claims (27)

1. A computer-implemented method for recognizing chemical information from document images, in which a computing device containing a processor and memory stores in memory instructions executed by the processor and executes instructions, including the steps of: - the image of the document page is fed to the input of the detector; the detector using the first neural network identifies on the page one or more fragments containing chemical information; for each identified fragment get the coordinates of the fragment on the page; and classify the fragments into at least the following categories: chemical structure, reaction arrow; - one or more identified fragments of chemical structures are fed to the input of the structure recognition block, each fragment being an image; the structure recognition unit for each fragment recognizes the chemical structure using the second neural network; - one or more identified fragments of reaction arrows are fed to the input of the arrow recognition block; the arrow recognition unit, using the third neural network, determines the type of the arrow, and using the fourth neural network, the coordinates on the page for each arrow, and the response attributes are obtained; - the coordinates on the page of each fragment of the recognized chemical structures, the corresponding recognized chemical structures, the coordinates on the page of each reaction arrow, the arrow type, the reaction attributes are transmitted to the input of the reaction recognition block; and based on the received data, the reaction recognition block determines how the arrows connect the recognized chemical structures; - as a result, based on the recognized data for the document page image, one or more recognized chemical structures are obtained, coordinates on the page for each recognized chemical structure, recognized relationships between substances participating in a chemical reaction, represented as chemical structures, coordinates on the page for each recognized relations. 2. The method according to claim 1, characterized in that the chemical structure is at least a chemical compound, a Markush structure, a chemical structure with substituents. 3. The method according to p. 1, characterized in that additionally identify fragments containing additional information that contributes to the recognition of reactions. 4. The method according to p. 3, characterized in that additional information includes at least the following: title, legend. 5. The method according to p. 1, characterized in that the detector additionally determines confidence for each identified fragment - a number from 0 to 1, which evaluates the reliability of the identified fragment, where 0 is absolutely not sure, 1 is completely sure. 6. The method according to claim 5, characterized in that the identified fragments are filtered by a predetermined confidence threshold. 7. The method according to p. 6, characterized in that which set a confidence threshold for each category of fragments. 8. The method according to p. 1, characterized in that the first neural network is a Faster R-CNN neural network or other convolutional network of equal or greater power. 9. The method according to p. 1, characterized in that the second neural network is a neural network based on the transformer architecture, and the structure recognition block contains a convolutional block and a transformer decoder. 10. The method according to claim 9, characterized in that ResNet-50 without the last two layers or another convolutional network that works with images is used as a convolutional block. 11. The method according to claim 1, characterized in that the recognized chemical structure is a text sequence that uniquely describes the chemical structure. 12. The method according to p. 11, characterized in that describe the chemical structure as a text sequence using the SMILES modification capable of describing Markush structures and chemical structures with substituents. 13. The method according to p. 12, characterized in that implemented a mechanism for converting the SMILES modification, which is capable of describing Markush structures and chemical structures with substituents, to SMILES and vice versa. 14. The method according to p. 1, characterized in that the third and fourth neural networks are ResNet-based convolutional neural networks. 15. The method according to p. 1, characterized in that arrows are classified into the following types: straight arrow, arrow that is not a straight arrow. 16. The method according to p. 1, characterized in that the substances involved in a chemical reaction are the initial substances of a chemical reaction, the products of a chemical reaction. 17. System for recognition of chemical information from images of documents, containing: - detector; - structure recognition block; - arrow recognition unit; - reaction recognition block; and in which the computing device, containing the processor and memory storing instructions executable by the processor, implements the method according to paragraphs. 1-16.

Images