JP7732704B2 - Deep learning-based optical character recognition method and system - Google Patents

Description

Applicable to Article 30, Paragraph 2 of the Patent Act [2203.05122] DEER: Detection-agnostic End-to-End Recognizer for Scene Text Spotting (arxiv.org) Publication date March 10, 2022

Article 30, Paragraph 2 of the Patent Act applies. Oral presentation at a research conference held at NAVER DEVIEW 2023 (TRACK D) at COEX Grand Ballroom, Seoul, Korea. https://deview.kr/2023/sessions/560. Published on February 27, 2023.

Article 30, paragraph 2 of the Patent Act applies. Oral presentation at the research meeting at ECCV 2022, TiE workshop (David Intercontinental, Gallery Hall) https://sites.google.com/view/tie-eccv2022/schedule https://sites.google.com/view/tie-eccv2022/schedule com/view/tie-eccv2022/home Published on October 24, 2022 Kil Tae-ho, Kim Sung-hyun, Seo Sook-min The above-mentioned researchers affiliated with Naver Corporation published "Out of Vocabulary Scene Text Understanding" at the ECCV 2022, TiE workshop at the request of Naver Corporation.

This disclosure relates to optical character recognition methods and systems, and more particularly to deep learning-based optical character recognition methods and systems that extract text from an image based on the image and reference points by generating reference points for referencing text regions contained in the image.

Optical character recognition (OCR) is a technology that detects text from an image and recognizes the type of text that is detected. Conventional optical character recognition technology has been used to recognize characters in documents. However, recognizing text that is commonly seen in everyday life, such as on roadside signs, is technically difficult due to the wide variety of possible cases. For example, in the case of text contained in an image that is diagonal or curved, it can be difficult to recognize the text because the text areas may be non-rectangular and have various shapes. Text displayed in this way in the surrounding environment or on objects is called scene text.

To recognize such scene text, conventional technologies consist of a first stage in which a detector is used to detect text regions from an image, and then the detected regions are edited (for example, by adjusting the size of the regions) and transmitted to a recognizer, followed by a second stage in which the recognizer recognizes the text. However, with such conventional technologies, if the text regions cannot be detected, both text recognition and text recognition may fail, and the content of the text may be lost when adjusting the size of the detected regions. Furthermore, the artificial neural networks executed at each stage must be trained repeatedly, which can reduce the efficiency of computational resources. Furthermore, when updating the detector, the recognizer must also be retrained, which can be disadvantageous in terms of management and maintenance of related services.

Korean Patent Publication No. 10-2015-0125376

This disclosure provides a deep learning-based optical character recognition method and system (apparatus) to solve the problems described above.

The present disclosure may be implemented in various ways, including as a method, an apparatus (system), or a computer-readable computer program.

According to one embodiment of the present disclosure, a deep learning-based optical character recognition method may include detecting at least one text region from an image, generating at least one reference point associated with the at least one text region, and extracting at least one piece of text from the image based on the image and the at least one reference point.

A computer-readable computer program may be provided for executing a method according to one embodiment of the present disclosure on a computer.

An optical character recognition system according to one embodiment of the present disclosure may include a detector that detects at least one text region from an image and generates at least one reference point associated with the at least one text region, and a recognizer that extracts at least one piece of text from the image based on the image and the at least one reference point.

According to various embodiments of the present disclosure, even if an error occurs in the process of detecting a text region in an image, text can be correctly extracted from the image by utilizing the entire image and reference points. In other words, even if an error occurs in detecting a text region, loss of text content can be prevented. Furthermore, since the final text recognition result does not solely depend on the detected text region, this method can also have advantages when extracting rotated text within an image.

According to various embodiments of the present disclosure, the detector and recognizer are trained in one step instead of two, which can improve training efficiency. Furthermore, by using images and reference points without adjusting the size of detected text regions, it is possible to prevent recognition errors that occur when long characters are distorted or truncated. Furthermore, by sharing multi-scale features extracted from the backbone with the detector and recognizer, it is possible to improve model performance and inference speed.

According to various embodiments of the present disclosure, text in an image can be recognized and decoded by utilizing reference points and overall image features without pooling or masking regions of interest within the image. This allows for smooth text extraction even if regions of interest are not detected correctly.

Various embodiments of the present disclosure can more accurately extract complex scene text, such as crossed text within an image, text within text, and text of various fonts and sizes. Furthermore, since the final text recognition result does not heavily depend on the detection of text regions within an image, it can be resilient to errors in detecting text regions. Furthermore, text can be correctly extracted even if the text within an image is rotated.

According to various embodiments of the present disclosure, by matching which text regions contain specific word units of text, the detected text can be linked more organically. This can result in a higher quality translation service being provided in the process of extracting text from a document using optical character recognition and then providing a translation service.

The effects of the present disclosure are not limited to those mentioned above, and other effects not mentioned will be clearly understood by a person with ordinary skill in the art to which the present disclosure pertains (referred to as an "ordinary artisan") from the claims.

Embodiments of the present disclosure will be described with reference to the accompanying drawings, as described below, in which like reference numerals indicate like elements, but are not limited to the drawings.

FIG. 1 illustrates an example of an optical character recognition method according to one embodiment of the present disclosure. FIG. 2 is a schematic diagram illustrating an example configuration in which an information processing system is communicatively coupled to multiple user terminals for optical character recognition according to one embodiment of the present disclosure. FIG. 3 is a block diagram illustrating the internal configuration of a user terminal and an information processing system according to an embodiment of the present disclosure. FIG. 4 is a schematic diagram illustrating an optical character recognition system according to one embodiment of the present disclosure. FIG. 5 is a diagram illustrating an example of extracting text from an image according to one embodiment of the present disclosure. FIG. 6 is a diagram illustrating an example process of an optical character recognition system according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating an example of an optical character recognition result according to an embodiment of the present disclosure. FIG. 8 is a diagram showing an example in which text is detected character by character according to an embodiment of the present disclosure. FIG. 9 is a diagram illustrating an example of detecting line-based and paragraph-based text regions in a document according to an embodiment of the present disclosure. FIG. 10 is a flowchart illustrating a method according to one embodiment of the present disclosure.

Specific details for implementing this disclosure will be described in detail below with reference to the accompanying drawings. However, in the following description, specific descriptions of well-known functions and configurations will be omitted if they may unnecessarily obscure the gist of this disclosure.

In the accompanying drawings, identical or corresponding components are given the same reference numerals. Furthermore, in the following description of the embodiments, duplicate descriptions of identical or corresponding components may be omitted. However, the omission of a description of a component does not imply that such a component is not included in a certain embodiment.

Advantages and features of the disclosed embodiments, as well as methods for achieving them, will become apparent from the following detailed description of the embodiments taken in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, and may be embodied in a variety of different forms. These embodiments are provided solely so that this disclosure will be thorough and complete, and will adequately convey the scope of the invention to those skilled in the art.

The terms used in this specification are briefly explained, followed by a detailed description of the disclosed embodiments. The terms used in this specification are generally selected based on current widespread usage, taking into account the functionality of this disclosure. However, this may change depending on the intentions or legal precedents of engineers in the relevant field, the emergence of new technology, and other factors. In addition, in certain cases, the applicant may arbitrarily select terms, and in such cases, their meanings will be described in detail in the description of the relevant invention. Therefore, the terms used in this disclosure should be defined based on the meanings of the terms and the overall content of this disclosure, rather than simply by their names.

In this specification, the singular includes the plural unless the context clearly dictates otherwise. Furthermore, the plural includes the singular unless the context clearly dictates otherwise. When a part of the entire specification is said to include certain elements, this does not mean that other elements are excluded, but that other elements may also be included, unless otherwise specified.

Furthermore, as used herein, the terms "module" or "unit" refer to a software or hardware component, and the "module" or "unit" performs some function. However, the term "module" or "unit" is not limited to software or hardware. A "module" or "unit" may be configured to reside on an addressable storage medium or to execute one or more processors. Thus, by way of example, a "module" or "unit" may include components such as software components, object-oriented software components, class components, and task components, as well as at least one of a process, function, attribute, procedure, subroutine, program code segment, driver, firmware, microcode, circuit, data, database, data structure, table, array, or variable. The components and "modules" or "units" may indicate that the functionality provided therein may be combined into fewer components and "modules" or "units," or further separated into additional components and "modules" or "units."

According to one embodiment of the present disclosure, a "module" or "unit" may be implemented with a processor and memory. "Processor" should be broadly interpreted to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, "processor" may also refer to an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. "Processor" may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such combination. Additionally, "memory" should be broadly interpreted to include any electronic component capable of storing electronic information. "Memory" may refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage devices, registers, etc. Memory is said to be in electronic communication with a processor if the processor can read information from and/or write information to the memory. Memory that is integrated into a processor is in electronic communication with the processor.

In this disclosure, a "system" may include at least one of a server device and a cloud device, but is not limited to these. For example, a system may be composed of one or more server devices. As another example, a system may be composed of one or more cloud devices. As yet another example, a system may be configured and operated by a server device and a cloud device together.

In this disclosure, "display" may refer to any display device associated with a computing device, for example, any display device capable of displaying any information/data controlled by or provided by a computing device.

In this disclosure, "each of a plurality of A's" or "each of a plurality of A's" may refer to each of all components included in the plurality of A's, or may refer to each of some components included in the plurality of A's.

End-to-end scene text spotting techniques that recognize arbitrary forms of text instances have made significant improvements. Common methods for scene text detection are region-of-interest pooling or segmentation masking to limit the features to a single text instance. However, if the detection is inaccurate (e.g., if one or more characters are cut off), the recognizer in such methods may have difficulty decoding or generating the correct character image sequence.

This disclosure provides an optical character recognition method that includes a detection-agnostic end-to-end recognizer, taking into account the difficulty of accurately determining word boundaries using a detector alone, for example, in scene text spotting problems. The disclosed method can reduce the tight dependency between the detector and recognizer by linking the detector and recognizer using a single reference point for each piece of text instead of using detected text regions. Furthermore, the disclosed method can enable the recognizer to recognize text displayed as a reference point along with features of the entire image. In other words, because only a single point is required to recognize text, text can be extracted from an image without any type of detector or bounding polygon. Additionally, this disclosure provides an optical character recognition method and system that is competitive in regular and arbitrary text spotting benchmarks and robust to text detection errors.

End-to-end scene text spotting techniques are used in a variety of fields, including information extraction, image retrieval, and visual query response. In general, an end-to-end scene text spotting pipeline consists of a detector and a recognizer. The detector localizes text instances in an image in the form of boxes or polygons, and the recognizer receives each localized text region as input and decodes the characters in each patch of the image.

Traditional scene text spotting pipelines use a somewhat tightly coupled framework between the detector and recognizer. In particular, if the detector encounters a recognition error for a text region, the recognizer may be supplied with an image containing only the text itself, with the target text cut off. In this case, the recognition performance of the pipeline can be heavily influenced by the performance of the image patches output by the detector and recognizer. Recently, end-to-end scene text spotting methods have used a more loosely coupled framework by extracting features using region of interest (ROI) pooling or masking and limiting the recognizer's input region to a single word. Using localized features in the recognizer can reduce the recognizer's reliance on the detector's cut-off regions, but detector errors can still accumulate and recognition failures can occur. Furthermore, feature pooling and masking require data with bounding boxes or polygons to train an end-to-end scene text spotting model, even if the final application does not require accurate boundary information.

This disclosure provides a new end-to-end recognizer that significantly reduces dependency on the accuracy of detection results. Instead of relying on the detector to extract precise text regions, the detector can generate reference points for each text instance. The recognizer can then comprehensively recognize the text surrounding the reference points. Specifically, given the reference points, the recognizer can be trained to attend to specific text instance regions while decoding text sequences. Furthermore, by requiring the detector to only have a single reference point, a greater variety of detection algorithms and annotations can be applied. Furthermore, rotated or curved text instances can be handled naturally without pooling operations and polygon-type annotations.

In this disclosure, a "text area" may refer to an area in an image that contains text. Here, a text area may be displayed in the form of a rectangular bounding box, but is not limited to this, and may also be displayed in the form of a polygonal bounding polygon. A text area may also be configured with vertex coordinates. For example, if a text area is in the form of a bounding box, the text area may be configured with top-left coordinate values, top-right coordinate values, bottom-right coordinate values, and bottom-left coordinate values.

In this disclosure, a "recognizer" may refer to a module in an optical character recognition system that recognizes text regions within an image. The recognizer may also include a text decoder. That is, the recognizer may recognize text regions to generate a character image sequence and convert the character image sequence into text using the text decoder.

FIG. 1 illustrates an optical character recognition method according to one embodiment of the present disclosure. A first example 110 is an example of a conventional optical character recognition method. The conventional optical character recognition method includes a first stage in which a detector detects a text region 112 from an image, adjusts the size of the detected text region 112, and transmits the size to a recognizer. A second stage in which the recognizer recognizes text in the detected text region 112. However, in this conventional method, if the detector erroneously detects the text region 112, both stages of text recognition may fail. Furthermore, the content of the text may be lost when the size of the detected text region 112 is adjusted. Furthermore, the need to redundantly train the artificial neural network executed at each stage may reduce the efficiency of computing resources. Furthermore, when the detector is updated, the recognizer must also be newly trained, which may increase the time and cost required for management or maintenance of related services.

The second example 120 is an example of the optical character recognition method of the present disclosure. In one embodiment, at least one text region may be detected from an image. Specifically, features related to text in units of words may be extracted from the image. Furthermore, position information for the at least one text region may be generated based on the features. Note that the position information for the text region may be the vertex coordinates of a bounding box or bounding polygon of the text region.

In one embodiment, a reference point 122 associated with a text region may be generated. The reference point 122 may include the center point of the text region. In this case, the center point of the text region may be determined based on the position information of the text region. As shown in the figure, if the bounding box of the text region includes the character "sesame", the reference point 122 may be generated at the center of the character "sesame". An example of how the reference point is displayed will be described in detail below with reference to Figure 7.

In one embodiment, text may be recognized or extracted from the image based on the image and the reference point 122. Specifically, multiple offset points 124_1 to 124_4 may be generated adjacent to the reference point 122. The multiple offset points 124_1 to 124_4 may guide the recognizer to focus on areas adjacent to the multiple offset points 124_1 to 124_4 and extract text. Furthermore, at least one piece of text may be extracted from the image based on the image and the multiple offset points 124_1 to 124_4. For example, the recognizer may extract "sesame" from the image by focusing on the multiple offset points 124_1 to 124_4 in the image. While FIG. 1 shows four offset points, this is not limiting.

In one embodiment, multiple text extractions may be performed at least partially in parallel. For example, a region containing "sesame" may be detected as a first text region, and a region containing "stick" may be detected as a second text region. In this case, a first reference point may be generated in the first text region, and a second reference point may be generated in the second text region. Furthermore, "sesame" may be extracted from the first text region based on the image and the first reference point. Furthermore, "stick" may be extracted from the second text region based on the image and the second reference point, at least partially in parallel with the extraction of "sesame."

With this configuration, even if there is an error in the process of detecting the text region of the image, the text can be correctly extracted from the image by using the entire image and reference points. In other words, even if an error occurs in detecting the text region, the loss of text content can be prevented. Furthermore, since the final text recognition result does not solely depend on the detected text region, it can also have the advantage of extracting rotated text within an image.

FIG. 2 is a schematic diagram illustrating a configuration in which an information processing system 230 is communicatively coupled to multiple user terminals 210_1, 210_2, and 210_3 for optical character recognition according to one embodiment of the present disclosure. As shown in the figure, multiple user terminals 210_1, 210_2, and 210_3 may be coupled to information processing system 230, which may provide optical character recognition services, via network 220. Note that the multiple user terminals 210_1, 210_2, and 210_3 may include terminals of users who receive the optical character recognition service.

In one embodiment, information processing system 230 may include one or more server devices and/or databases, or one or more cloud computing service-based distributed computing devices and/or distributed databases, that may store, provide, and execute computer-executable programs (e.g., downloadable applications) and data related to providing optical character recognition services, etc.

The optical character recognition service provided by the information processing system 230 may be provided to users through a web browser or web browser extension program of an optical character recognition service application installed in each of the user terminals 210_1, 210_2, and 210_3. For example, the information processing system 230 may provide information or respond to a request for text extraction from an image received from the user terminals 210_1, 210_2, and 210_3 through an optical character recognition service application.

Multiple user terminals 210_1, 210_2, and 210_3 may communicate with information processing system 230 via network 220. Network 220 may be configured to enable communication between multiple user terminals 210_1, 210_2, and 210_3 and information processing system 230. Depending on the installation environment, network 220 may be configured as a wired network such as Ethernet (registered trademark), a wired home network (Power Line Communication), a telephone line communication device, or RS serial communication, a mobile communication network, a wireless network such as WLAN (Wireless LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), and ZigBee (registered trademark), or a combination thereof. The communication method is not limited, and may include not only communication methods that utilize communication networks that may be included in network 220 (for example, mobile communication networks, wired internet, wireless internet, broadcast communication networks, satellite communication networks, etc.), but also short-range wireless communication between user terminals 210_1, 210_2, and 210_3.

2, mobile phone terminal 210_1, tablet terminal 210_2, and PC terminal 210_3 are shown as examples of user terminals, but are not limited thereto. User terminals 210_1, 210_2, and 210_3 may be any computing device capable of wired and/or wireless communication and on which an optical character recognition service application or a web browser, etc., can be installed and executed. For example, user terminals may include AI speakers, smartphones, mobile phones, navigation systems, computers, notebooks, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, game consoles, wearable devices, IoT (Internet of Things) devices, VR (Virtual Reality) devices, AR (Augmented Reality) devices, set-top boxes, etc. Also, while FIG. 2 shows three user terminals 210_1, 210_2, and 210_3 communicating with the information processing system 230 via the network 220, this is not limited to this, and a different number of user terminals may be configured to communicate with the information processing system 230 via the network 220.

While FIG. 2 shows user terminals 210_1, 210_2, and 210_3 receiving optical character recognition services from information processing system 230, this is not intended to be limiting. For example, optical character recognition services may be provided by optical character recognition programs/applications installed on user terminals 210_1, 210_2, and 210_3 without communication with information processing system 230. Also, while information processing system 230 is shown as a single device, this is not intended to be limiting and information processing system 230 may be comprised of multiple devices.

3 is a block diagram showing the internal configuration of a user terminal 210 and an information processing system 230 according to one embodiment of the present disclosure. The user terminal 210 refers to any computing device capable of executing applications, a web browser, etc., and capable of wired/wireless communication, and may include, for example, the mobile phone terminal 210_1, tablet terminal 210_2, and PC terminal 210_3 of FIG. 2. As shown in the figure, the user terminal 210 may include memory 312, a processor 314, a communication module 316, and an input/output interface 318. Similarly, the information processing system 230 may include memory 332, a processor 334, a communication module 336, and an input/output interface 338. As shown in FIG. 3, the user terminal 210 and the information processing system 230 may be configured to communicate information and/or data over the network 220 using their respective communication modules 316, 336. Additionally, the input/output device 320 may be configured to input information and/or data to the user terminal 210 or output information and/or data generated from the user terminal 210 via the input/output interface 318.

Memory 312, 332 may include any non-transitory computer-readable recording medium. According to one embodiment, memory 312, 332 may include a permanent mass storage device such as a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. As another example, a permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be included in user terminal 210 or information processing system 230 as a separate permanent storage device distinct from memory. In addition, memory 312, 332 may store an operating system and at least one program code.

Such software components may be loaded from a computer-readable recording medium separate from memory 312, 332. Such separate computer-readable recording medium may include a recording medium directly connectable to the user terminal 210 and information processing system 230, but may also include computer-readable recording media such as a floppy drive, disk, tape, DVD/CD-ROM drive, or memory card. As another example, the software components may be loaded into memory 312, 332 via communication modules 316, 336 rather than a computer-readable recording medium. For example, at least one program may be loaded into memory 312, 332 based on a computer program installed by a file provided over network 220 by a developer or a file distribution system that distributes application installation files.

Processors 314, 334 may be configured to process computer program instructions by performing basic arithmetic, logic, and input/output operations. The instructions may be provided to processors 314, 334 by memory 312, 332 or communication modules 316, 336. For example, processors 314, 334 may be configured to execute instructions received from program code stored in a storage device such as memory 312, 332.

The communication modules 316, 336 may provide configurations or functions for the user terminal 210 and the information processing system 230 to communicate with each other via the network 220, and may provide configurations or functions for the user terminal 210 and/or the information processing system 230 to communicate with other user terminals or other systems (e.g., a separate cloud system). For example, a request or data (e.g., a request to extract text from an image) generated by the processor 314 of the user terminal 210 using program code stored in a storage device such as the memory 312 may be transmitted to the information processing system 230 via the network 220 under the control of the communication module 316. Conversely, a control signal or command provided under the control of the processor 334 of the information processing system 230 may be received by the user terminal 210 via the communication module 316 of the user terminal 210 via the communication module 336 and the network 220.

The input/output interface 318 may be a means for interfacing with the input/output device 320. For example, input devices may include devices such as a camera including an audio sensor and/or an image sensor, a keyboard, a microphone, and a mouse, while output devices may include devices such as a display, a speaker, and a haptic feedback device. As another example, the input/output interface 318 may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen. For example, when the processor 314 of the user terminal 210 processes instructions from a computer program loaded into the memory 312, a service screen constructed using information and/or data provided by the information processing system 230 or another user terminal may be displayed on the display via the input/output interface 318. While FIG. 3 illustrates the input/output device 320 as not being included in the user terminal 210, this is not limiting and the input/output device 320 may be configured as a single device together with the user terminal 210. Furthermore, the input/output interface 338 of the information processing system 230 may be a means for interfacing with an input or output device (not shown) that is coupled to the information processing system 230 or that may be included in the information processing system 230. In FIG. 3, the input/output interfaces 318, 338 are shown as elements configured separately from the processors 314, 334, but are not limited to this, and the input/output interfaces 318, 338 may also be configured to be included in the processors 314, 334.

The user terminal 210 and the information processing system 230 may include more components than those shown in FIG. 3. However, it is not necessary to explicitly show most of the conventional components. In one embodiment, the user terminal 210 may be implemented to include at least some of the input/output devices 320 described above. The user terminal 210 may also include other components such as a transceiver, a GPS (Global Positioning System) module, a camera, various sensors, a database, etc. For example, if the user terminal 210 is a smartphone, it may include components typically included in smartphones, such as an acceleration sensor, a gyro sensor, a microphone module, a camera module, various physical buttons, buttons using a touch panel, input/output ports, a vibrator for vibration, etc.

While a program for an optical character recognition service application or the like is running, the processor 314 can receive text, images, videos, sounds, and/or actions, etc., entered or selected via input devices such as a touch screen, keyboard, camera including an audio sensor and/or image sensor, microphone, etc. connected to the input/output interface 318, and can store the received text, images, videos, sounds, and/or actions, etc. in memory 312 or provide them to the information processing system 230 via the communication module 316 and the network 220.

The processor 314 of the user terminal 210 may be configured to manage, process, and/or store information and/or data received from the input/output device 320, other user terminals, the information processing system 230, and/or multiple external systems. The information and/or data processed by the processor 314 may be provided to the information processing system 230 via the communication module 316 and the network 220. The processor 314 of the user terminal 210 may transfer and output information and/or data to the input/output device 320 via the input/output interface 318. For example, the processor 314 may display the received information and/or data on the screen of the user terminal 210.

The processor 334 of the information processing system 230 may be configured to manage, process, and/or store information and/or data received from multiple user terminals 210 and/or multiple external systems. The information and/or data processed by the processor 334 may be provided to the user terminal 210 via the communication module 336 and the network 220.

Figure 4 is a schematic diagram illustrating an example of an optical character recognition system according to one embodiment of the present disclosure. In one embodiment, the optical character recognition system may include a backbone (not shown), a transformer encoder 420, a detector (or location head) 430, and a recognizer 440. Note that the recognizer 440 may include a text decoder.

In one embodiment, the transformer encoder 420 may combine multi-scale feature maps generated by the backbone. The detector 430 may also set reference points 432 for text instances and bounding boxes. The recognizer 440 may then generate a character image sequence by recognizing text regions based on the multi-scale features and reference points 432 of the input image 410, and convert the generated character image sequence into text using a text decoder.

In one embodiment, the input image 410 is provided to a backbone, which extracts feature maps (e.g., _C2 , _C3 , _C4 , and _C5 ) from the input image 410. The resolution of the extracted feature maps may be ¼, ⅛, ⅛-sixteenth, or ⅓-thousandth of the resolution of the input image 410, respectively. The feature maps may be projected onto multiple channels (e.g., 256 channels) by applying a fully-connected (FC) layer and group normalization. The projected feature maps may then be merged, flattened, and concatenated into feature tokens of size ( _L2 + _L3 + _L4 + _L5 ) × 256, where L _i may represent the flattened length of C _i (or H/2 ⁱ × W/2 ⁱ ). In this case, the transformer encoder 420 may use the feature maps as input and output refined features. The refined features, via detector 430, are used in conjunction with reference points 432 to enable recognizer 440 to autoregressively generate text sequences within text instances.

In one embodiment, the Transformer Encoder 420 may use deformable attention, which scales linearly with input length. Because the cost of learning or inference for conventional self-attention increases quadratically with input length, using a Transformer to concatenate multi-scale features may be inefficient. In contrast, deformable attention may provide the encoder and decoder with more efficient and accurate localization results. Deformable attention may be calculated as shown in Equation 1 below.

Here, x(v, p) represents bilinear interpolation to extract a feature from the feature value v at position p. K is the number of sampled keypoints, and k represents the key index. h is the index of the attention head, and W _h ^o ∈R ^C×Cm and W _h ^k ∈R ^Cm×C can be linear projections. P _ref , ΔP _hqk , and A _hqk can be reference points, sampling offsets, and attention weights, respectively. P _ref , ΔP _hqk , and A _hqk can be calculated by applying linear projections to the query features. In this case, softmax can be applied to A _hqk . A fixed reference point with normalized coordinates to [0,1]×[0,1] can be used in the encoder 420.

In one embodiment, to recognize a text object using position information, a reference point (or center position of a text instance) 432 can be predicted using a detector 430 that detects text from multi-scale features. A bounding polygon of the text instance can be extracted using a segmentation map. For example, feature tokens of size _L2 can be extracted from features corresponding to feature map _C2 and reconstructed to a size of (H/4, W/4). A segmentation head consisting of transposed convolution, group normalization, and ReLU (Rectified Linear Unit) can be used to obtain binary and threshold maps. Additionally, the center coordinates of the text instance can be determined as the reference point 432 in the detected bounding polygon. While the text detection method described here is based on a segmentation-based detection method, various detection methods can be applied.

In one embodiment, a recognizer 440 including a text decoder, e.g., a Transformer decoder, may autoregressively predict text sequences from text instances associated with text regions, referencing the image 410 and reference points 432 with deformable attention 444. In this case, a query Q for the text decoder may include text embeddings, positional embeddings, and reference points q _ref . Also, the key K and value V of the text decoder may be feature tokens of the Transformer encoder 420. The query Q may be propagated through self-attention 442, deformable attention 444, and a feed-forward layer 446.

In one embodiment, during the training phase of the recognizer 440, the bounding box of the text region is sampled from the image, and the center coordinate of the bounding box can be used as the reference point 432 for the text decoder. This allows the detector 430 and the recognizer 440 to be trained independently. Furthermore, to reduce the coordinate difference between the model's prediction and the actual value, the center coordinate can be randomly varied during the training phase using Equation 2 below.

Here, p _c represents the center point of the ground truth polygon, and p _tl , p _tr , and p _bl represent the top-left, top-right, and bottom-left coordinates, respectively. In addition, the center point of the text region extracted in the detection phase can be used as a reference point in the inference phase.

In one embodiment, the loss function L used for training can be expressed as Equation 3 below:

where _Lr represents the autoregressive text recognition loss, and _Ls , _Lb , and _Lt represent differentiable binarization losses for the probability map, binary map, and threshold map, respectively. Furthermore, the softmax cross entropy between the predicted probability of a string sequence and the ground truth text label corresponding to the corresponding text bounding box can be calculated as _Lr . After differentiable binarization, binary cross entropy with hard negative mining can be applied to _Ls , dice loss can be applied to _Lb , and _L1 distance loss can be applied to _Lt.

In one embodiment, the inference stage may use the probability map of the detector 430. In this case, the probability map is binarized to a specified threshold, and connected components may be extracted as regions in the binary map. Note that the size of the extracted region may be smaller than the actual text region. Therefore, the extracted region may be expanded using an offset D in the Vatti clipping algorithm, as expressed in Equation 4 below.

Here, A is the area of the polygon region, L is the perimeter of the polygon, and r may represent a predetermined dilation factor. After the polygon is extracted in the dilated region, the calculated center coordinates may be provided to the recognizer 440 as a reference point 432. As a result, the recognizer 440 may predict a character sequence in the corresponding text region by referring to the reference point 432.

This configuration allows the detector and recognizer to be trained in one step instead of two, improving training efficiency. Furthermore, by using images and reference points without adjusting the size of the detected text region, it is possible to prevent recognition errors caused by distorted or truncated long characters. Furthermore, by sharing multi-scale features extracted from the backbone with the detector and recognizer, model performance and inference speed can be improved.

Figure 5 illustrates an example of extracting text from an image 510 according to an embodiment of the present disclosure. In one embodiment, text can be recognized from the image 510 using a reference point 512 and offset points 514_1 to 514_4. Specifically, each character can be sequentially and autoregressively decoded by a recognizer 520 including deformable attention by referring to the features of the image 510 and the reference point 512. Note that deformable attention may not perform attention on the entire features of the image 510, but may instead sample only a few points. That is, attention may be performed using the reference point 512 and offset points 514_1 to 514_4.

In one embodiment, offset points 514_1 to 514_4 may be generated at positions adjacent to reference point 512. Specifically, the position of each of the multiple offset points 514_1 to 514_4 may be calculated by adding the offset predicted by the model from the coordinates of reference point 512. In this case, recognizer 520 may determine the position to perform attention based on the keys and values of offset points 514_1 to 514_4. As a result, recognizer 520 may decode text by generating characters autoregressively, with reference to reference point 512 as the center and attentioning the surrounding offset points 514_1 to 514_4. For example, recognizer 520 may sequentially and autoregressively decode each character of "sesame" by referring to offset points 514_1 to 514_4. While FIG. 5 shows four multiple offset points 514_1 to 514_4, this is not limiting.

This configuration allows text in an image to be recognized and decoded by utilizing reference points and overall image features without pooling or masking regions of interest within the image. This allows for smooth text extraction even if regions of interest are not detected correctly.

Figure 6 illustrates an example process for an optical character recognition system according to one embodiment of the present disclosure. In one embodiment, a backbone 620 may generate image features (or multi-scale feature maps) from an image 610. A transformer encoder 630 may also tune and encode the image features.

In one embodiment, the word detector 640 may detect the location of text in units of words from the features encoded in the image 610. Specifically, the word detector 640 may extract features related to the text in units of words. The word detector 640 may also generate location information for at least one text region based on the features. Furthermore, the word detector 640 may generate a reference point based on the location information of the text region. The reference point may be the center point of the detected text region.

In one embodiment, based on the reference points generated by the word detector 640 and the encoded features of the image 610, recognizers (recognizers 1 to N) 650_1 to 650_N may recognize and extract text from the image 610. Specifically, recognizers 650_1 to 650_N may generate multiple offset points adjacent to the reference points. Furthermore, recognizers 650_1 to 650_N may recognize the text from the image 610 by paying attention to the reference points and the multiple offset points, and generate a character image sequence. Furthermore, recognizers 650_1 to 650_N may convert the generated character image sequence into text by decoding it.

In one embodiment, the word detector 640 may detect multiple text regions from the feature encoded image 610 at least partially in parallel. In this case, a reference point may be generated for each text region. This allows recognizers 650_1 through 650_N to extract text in parallel for each of the N text regions.

In one embodiment, the line detector 660 may detect line-by-line text positions in the feature-encoded image 610. Specifically, the line detector 660 may extract features related to line-by-line text. The line detector 660 may also generate position information for at least one text region based on the features. An example of detecting line-by-line text positions will be described in detail below with reference to FIG. 9.

In one embodiment, the paragraph detector 670 may detect paragraph-based text positions in the feature-encoded image 610. Specifically, the paragraph detector 670 may extract features related to the text in paragraph units. The paragraph detector 670 may also generate position information for at least one text region based on the features. An example of detecting paragraph-based text positions will be described in detail below with reference to FIG. 9.

In one embodiment, the word detector 640, line detector 660, and paragraph detector 670 may be coupled to the transformer encoder 630 in parallel. That is, the word detector 640, line detector 660, and paragraph detector 670 may detect word, line, and paragraph-based text regions at each level at least partially in parallel.

In one embodiment, the detected text regions at each level can be correlated with each other through a post-processing step 680. Specifically, the positions of words detected by the word detector 640, the positions of lines detected by the line detector 660, and the positions of paragraphs detected by the paragraph detector 670 can be matched with each other. This allows position information of line-based text regions and/or paragraph-based text regions containing text extracted by the recognition units 650_1 to 650_N to be detected.

Figure 7 illustrates an example of an optical character recognition result according to an embodiment of the present disclosure. A first example 710 is an example of a conventional optical character recognition result. Conventional optical character recognition methods extract text within an image by pooling or masking regions of interest. In this case, extraction of complex, dense text or text with geometric shapes may fail. For example, in the ambiguous word "HOME" containing a geometric shape, a conventional optical character recognition method may determine only "ME" as the region of interest due to the geometric shape of the "O." Such a failure to detect text may also result in failure to recognize text within the image.

A second example 720 is an example of an optical character recognition result of the present disclosure. The optical character recognition method of the present disclosure can detect text regions and generate reference points (denoted by the "+" shape in FIG. 7) without pooling or masking regions of interest. In this case, the recognizer does not need to rely heavily on the detected text regions. In other words, even if the text region detection result is inaccurate, the recognizer can still correctly extract the text in the image by considering the entire extracted image and concentrating on the reference points.

This configuration allows for more accurate extraction of complex scene text, such as crossed text within an image, text within text, and text of various fonts and sizes. Furthermore, since the final text recognition result does not depend heavily on the detection of text regions within the image, it can be resilient to errors in detecting text regions. Furthermore, text can be correctly extracted even if the text in the image is rotated.

Figure 8 is a diagram showing an example of character-level text detection according to an embodiment of the present disclosure. In one embodiment, after the recognizer 810 recognizes word-level text, character-level text can be detected. Specifically, to prevent the optical character recognition model from becoming heavy, an auto-regressive decoder 820 can be added to the recognizer 810. Note that the auto-regressive decoder 820 can auto-regressively recognize each character included in the extracted text using a classification score for the extracted text.

In one embodiment, the autoregressive decoder 820 can simultaneously recognize and detect each character by predicting the location and angle of each recognized character. Furthermore, the autoregressive decoder 820 can be trained to correctly recognize and detect characters for foreign languages and all character orientations. As shown in Figure 8, the autoregressive decoder 820 can detect each of "c, h, o, c, o...".

In one embodiment, pseudo-labeling may be applied to train the autoregressive decoder 820. Specifically, a teacher model may train labeled data. In this case, the trained teacher model may generate pseudo-labeled data for weakly labeled data. Here, the weakly labeled data may include data in which word-based text regions are detected in an image. The pseudo-labeled data may also include data in which the teacher model detects characters in the weakly labeled data. The autoregressive decoder 820 may then train both the pseudo-labeled data and the labeled data as a student model.

Figure 9 shows an example of detecting line-based and paragraph-based text regions in a document according to an embodiment of the present disclosure. A first example 910 is an example of word-based text detection in a document using the optical character recognition method of the present disclosure. A second example 920 is an example of line-based text detection in a document using the optical character recognition method of the present disclosure. And a third example 930 is an example of paragraph-based text detection in a document using the optical character recognition method of the present disclosure.

In one embodiment, features extracted from the Transformer encoder can be used to extract text-related features from a document. Furthermore, based on the features, a word detector, a line detector, and a paragraph detector can perform optical character recognition at the word, line, and paragraph levels, respectively. In this case, the word detector, line detector, and paragraph detector can detect text at each level at least partially in parallel. Furthermore, in a post-processing stage, matching can be performed to determine which line-level text region and which paragraph-level text region a specific word unit is contained in.

This configuration allows for more organic linking of detected text by matching which text regions contain specific word units. This allows for better quality translation services to be provided after extracting text from a document using optical character recognition.

FIG. 10 is a flowchart illustrating an example of a method 1000 according to an embodiment of the present disclosure. In one embodiment, the method 1000 may be performed by at least one processor. The method 1000 may begin with the processor detecting at least one text region from an image (S1010). Specifically, the processor may extract features related to text on a word-by-word basis from the image. The processor may also generate position information for the at least one text region based on the features.

The processor may then generate at least one reference point associated with the at least one text region (S1020). Note that each of the at least one reference point may include a center point of each of the at least one text region.

Then, the processor may extract at least one piece of text from the image based on the image and the at least one reference point (S1030). Specifically, the processor may generate multiple offset points adjacent to the at least one reference point. The multiple offset points may guide the recognizer to focus on regions adjacent to the multiple offset points and extract text. The processor may also extract at least one piece of text from the image based on the image and the multiple offset points.

In one embodiment, the at least one text region may include a first text region and a second text region. The at least one reference point may include a first reference point associated with the first text region and a second reference point associated with the second text region. In this case, the processor may extract the first text from the first text region based on the image and the first reference point. Furthermore, the processor may extract the second text from the second text region based on the image and the second reference point, in parallel with at least a portion of the step of extracting the first text.

In one embodiment, the extracted text may be word-by-word text. In this case, the processor may autoregressively detect each of the at least one character contained in the extracted text using the classification score for the extracted text. The processor may also predict the position and angle of each of the at least one character within the image.

In one embodiment, the processor may extract features associated with text line by line from the image. Based on the features, the processor may generate position information for at least one text region.

In one embodiment, the processor may extract features associated with paragraph-by-paragraph text from the image. Based on the features, the processor may generate position information for at least one text region.

In one embodiment, the processor may detect at least one word-based text region from the image. The processor may also detect at least one line-based text region from the image. And the processor may detect at least one paragraph-based text region from the image. In this case, detecting the word-based text region, detecting the line-based text region, and detecting the paragraph-based text region may be performed at least partially in parallel. Furthermore, the processor may detect position information for at least one of the line-based text region or paragraph-based text region containing the extracted text.

The above method may be provided as a computer program stored on a computer-readable recording medium for execution by a computer. The medium may permanently store the computer-executable program or may temporarily store it for execution or download. The medium may also be various recording or storage means in the form of a single or multiple pieces of hardware, and is not limited to media directly connected to any computer system, but may also be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and ROM, RAM, flash memory, and other media configured to store program instructions. Other examples of media include recording or storage media managed by app stores that distribute applications, or sites or servers that supply or distribute various other software.

The methods, operations, or techniques of the present disclosure may be implemented by various means. For example, such techniques may be implemented by hardware, firmware, software, or a combination thereof. Those of ordinary skill in the art will understand that the various illustrative logical blocks, modules, circuits, and algorithmic steps described in connection with the present disclosure may also be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design requirements imposed on the overall system. Those of ordinary skill in the art may implement the described functionality in various ways for each particular application, but such implementations should not be interpreted as a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may be implemented within one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in this disclosure, computers, or combinations thereof.

Accordingly, the various example logic blocks, modules, and circuits described in connection with this disclosure may be implemented or performed by a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but the processor may also be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

In a firmware and/or software implementation, the techniques may be implemented as instructions stored on a computer-readable medium such as random access memory (RAM), read-only memory (ROM), nonvolatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage device, etc. The instructions may also be executable by one or more processors to cause the processors to perform certain aspects of the functionality described in this disclosure.

If implemented in software, the techniques may be stored on or transmitted via a computer-readable medium as one or more instructions or code. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. Storage media may also be any available medium that can be accessed by a computer. By way of non-limiting example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to transport or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection is properly termed a computer-readable medium.

For example, if software is transferred from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted wire, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted wire, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. As used herein, disk and disc include CDs, laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Alternatively, the processor and the storage medium may reside as discrete components in a user terminal.

Although the embodiments described above are described as utilizing aspects of the presently disclosed subject matter in one or more stand-alone computer systems, the present disclosure is not limited thereto and may be implemented in connection with any computing environment, such as a network or distributed computing environment. Furthermore, aspects of the subject matter in the present disclosure may be implemented in multiple processing chips or devices, and storage may be affected similarly across multiple devices. Such devices may include PCs, network servers, and even portable devices.

Although the present disclosure has been described herein with reference to some embodiments, various modifications and variations that would be understood by those of ordinary skill in the art to which the presently disclosed invention pertains may be made without departing from the scope of the present disclosure. Furthermore, such modifications and variations should be considered to fall within the scope of the claims appended hereto.

110: First example 112: Text area 120: Second example 122: Reference point 124_1 to 124_4: Multiple offset points

Claims (18)

1. A deep learning-based Optical Character Recognition (OCR) method executed by at least one processor, comprising:
Detecting at least one text region from the image based on model data obtained by deep learning ;
inferring and generating at least one reference point associated with the at least one detected text region so as to reduce a difference between the model data and the at least one detected text region;
predicting and extracting at least one text from the image based on the image and the at least one reference point so as to minimize a loss function ;
Including ,
Detecting at least one text region from the image comprises:
extracting text-related features from the image on a word-by-word basis;
and generating position information for the at least one text region based on the features . The deep learning-based optical character recognition method of claim 1, wherein each of the at least one reference point includes a center point of each of the at least one text region. 1. A deep learning-based Optical Character Recognition (OCR) method executed by at least one processor, comprising:
Detecting at least one text region from an image based on model data obtained by deep learning;
inferring and generating at least one reference point associated with the at least one detected text region so as to reduce a difference between the model data and the at least one detected text region;
predicting and extracting at least one text from the image based on the image and the at least one reference point so as to minimize a loss function;
Including,
the at least one text area includes a first text area and a second text area;
the at least one reference point includes a first reference point associated with the first text region and a second reference point associated with the second text region;
The step of extracting at least one text from the image comprises:
extracting a first text from the first text region based on the image and the first reference points;
extracting second text from the second text region based on the image and the second reference points, at least in part in parallel with extracting the first text;
Deep learning-based optical character recognition methods, including: 1. A deep learning-based Optical Character Recognition (OCR) method executed by at least one processor, comprising:
Detecting at least one text region from the image based on model data obtained by deep learning;
inferring and generating at least one reference point associated with the at least one detected text region so as to reduce a difference between the model data and the at least one detected text region;
predicting and extracting at least one text from the image based on the image and the at least one reference point so as to minimize a loss function;
Including,
The step of extracting at least one text from the image comprises:
generating a plurality of offset points adjacent to the at least one reference point;
extracting at least one text from the image based on the image and the plurality of offset points;
Deep learning-based optical character recognition methods, including: The deep learning-based optical character recognition method of claim 4 , wherein the offset points guide text extraction by attention to regions adjacent to the offset points. 1. A deep learning-based Optical Character Recognition (OCR) method executed by at least one processor, comprising:
Detecting at least one text region from an image based on model data obtained by deep learning;
inferring and generating at least one reference point associated with the at least one detected text region so as to reduce a difference between the model data and the at least one detected text region;
predicting and extracting at least one text from the image based on the image and the at least one reference point so as to minimize a loss function;
Including,
the extracted text is word-by-word text,
The method comprises:
The deep learning-based optical character recognition method further includes a step of autoregressively detecting at least one character included in the extracted text using a classification score for the extracted text. The deep learning based optical character recognition method of claim 6 , further comprising predicting a position and an angle of each of the at least one character within the image. 1. A deep learning-based Optical Character Recognition (OCR) method executed by at least one processor, comprising:
Detecting at least one text region from an image based on model data obtained by deep learning;
inferring and generating at least one reference point associated with the at least one detected text region so as to reduce a difference between the model data and the at least one detected text region;
predicting and extracting at least one text from the image based on the image and the at least one reference point so as to minimize a loss function;
Including,
Detecting at least one text region from the image comprises:
extracting line-by-line text-related features from the image;
generating position information for at least one text region based on the features;
Deep learning-based optical character recognition methods, including: 1. A deep learning-based Optical Character Recognition (OCR) method executed by at least one processor, comprising:
Detecting at least one text region from an image based on model data obtained by deep learning;
inferring and generating at least one reference point associated with the at least one detected text region so as to reduce a difference between the model data and the at least one detected text region;
predicting and extracting at least one text from the image based on the image and the at least one reference point so as to minimize a loss function;
Including,
Detecting at least one text region from the image comprises:
extracting text-related features from the image in paragraph units;
generating position information for at least one text region based on the features;
Deep learning-based optical character recognition methods, including: 1. A deep learning-based Optical Character Recognition (OCR) method executed by at least one processor, comprising:
Detecting at least one text region from an image based on model data obtained by deep learning;
inferring and generating at least one reference point associated with the at least one detected text region so as to reduce a difference between the model data and the at least one detected text region;
predicting and extracting at least one text from the image based on the image and the at least one reference point so as to minimize a loss function;
Including,
Detecting at least one text region from the image comprises:
detecting at least one word-based text region from the image;
detecting at least one line-by-line text region from the image;
detecting at least one paragraph-based text region from the image;
The deep learning-based optical character recognition method, wherein the steps of detecting word-based text regions, detecting line-based text regions, and detecting paragraph-based text regions are performed at least partially in parallel. The deep learning-based optical character recognition method of claim 10 , further comprising detecting position information of at least one of a line-based text region or a paragraph-based text region including the extracted text. A computer readable computer program for executing the method according to any one of claims 1 to 11 on a computer. 1. A deep learning-based Optical Character Recognition (OCR) system executed by at least one processor , comprising:
a detector that detects at least one text region from an image based on model data obtained by deep learning, and infers and generates at least one reference point associated with the at least one text region so as to reduce a difference between the model data and the detected at least one text region;
a recognizer that predicts and extracts at least one text from the image based on the image and the at least one reference point so as to minimize a loss function ;
Including ,
a backbone for extracting at least one text-related feature from the image;
a transformer encoder for encoding the features;
an optical character recognition system further comprising : 1. A deep learning-based Optical Character Recognition (OCR) system executed by at least one processor, comprising:
a detector that detects at least one text region from an image based on model data obtained by deep learning, and infers and generates at least one reference point associated with the at least one text region so as to reduce a difference between the model data and the detected at least one text region;
a recognizer that predicts and extracts at least one text from the image based on the image and the at least one reference point so as to minimize a loss function;
Including,
The detector comprises:
extracting bounding polygons of text regions from the image using a segmentation map;
extracting the center coordinates of the text region from the bounding polygon;
An optical character recognition system that determines the center coordinate as the reference point. 1. A deep learning-based Optical Character Recognition (OCR) system executed by at least one processor, comprising:
a detector that detects at least one text region from an image based on model data obtained by deep learning, and infers and generates at least one reference point associated with the at least one text region so as to reduce a difference between the model data and the detected at least one text region;
a recognizer that predicts and extracts at least one text from the image based on the image and the at least one reference point so as to minimize a loss function;
Including,
The recognizer autoregressively predicts at least one text from a text instance associated with the at least one text region using deformable attention while referring to the image and the at least one reference point. 1. A deep learning-based Optical Character Recognition (OCR) system executed by at least one processor, comprising:
a detector that detects at least one text region from an image based on model data obtained by deep learning, and infers and generates at least one reference point associated with the at least one text region so as to reduce a difference between the model data and the detected at least one text region;
a recognizer that predicts and extracts at least one text from the image based on the image and the at least one reference point so as to minimize a loss function;
Including,
The recognizer
a first recognizer for extracting first text from the image based on first reference points associated with a first text region and the image;
a second recognizer for extracting second text from the image based on second reference points associated with second text regions and the image;
An optical character recognition system, wherein the first recognizer and the second recognizer perform text extraction at least partially in parallel. 1. A deep learning-based Optical Character Recognition (OCR) system executed by at least one processor, comprising:
a detector that detects at least one text region from an image based on model data obtained by deep learning, and infers and generates at least one reference point associated with the at least one text region so as to reduce a difference between the model data and the detected at least one text region;
a recognizer that predicts and extracts at least one text from the image based on the image and the at least one reference point so as to minimize a loss function;
Including,
The recognizer
an optical character recognition system that uses a classification score for the extracted text to autoregressively detect at least one character contained in the extracted text and predict a position and angle of each of the at least one character within the image. 1. A deep learning-based Optical Character Recognition (OCR) system executed by at least one processor, comprising:
a detector that detects at least one text region from an image based on model data obtained by deep learning, and infers and generates at least one reference point associated with the at least one text region so as to reduce a difference between the model data and the detected at least one text region;
a recognizer that predicts and extracts at least one text from the image based on the image and the at least one reference point so as to minimize a loss function;
Including,
The detector comprises:
a first detector for detecting word-based text regions;
a second detector for detecting line-by-line text regions;
a third detector for detecting text regions in paragraph units;
The optical character recognition system, wherein the first detector, the second detector, and the third detector perform text region detection at least partially in parallel.