TWI854101B - Instant-messaging bot for robotic process automation and robotic textual-content extraction from images - Google Patents

Abstract

An enterprise instant-messaging (IM) platform containing an enterprise chatbot is connected to one or more public IM platforms over the Internet. A robotic-process-automation (RPA) manager contains multiple modules of enterprise workflows and receives instructions from the enterprise chatbot for executing individual workflows. The system allows enterprise users connected to the enterprise IM platform, and external users connected to the public IM platforms, to use instant messaging to initiate enterprise internal or customer-facing workflows that are automated with the help of the enterprise chatbot. Furthermore, textual-content extraction from digital images can be incorporated in the RPA manager as an enterprise workflow, and improved convolutional neural network (CNN) methods for textual-content extraction are provided.

Description

Using instant messaging robots to automate robotic processes and extract text from images Robotic processes for content

The invention relates to robotic process automation, text extraction, text detection, text recognition, computer vision, convolutional neural networks, conversational robots, natural language processing, conversational user interfaces, instant messaging (SMS applications), and human-computer dialogue. This disclosure describes systems and methods for instant messaging robots for robotic process automation, and robotic process methods for extracting text content from images.

The present invention aims to solve three problems related to robotic process automation (RPA): (1) RPA that combines artificial intelligence and a conversational user interface, (2) robotic workflows delivered via instant messaging and mobile Internet, and (3) methods for robotic extraction of text content from digital images. These problems will be described one by one below.

Robotic process automation refers to the automation of business processes using software solutions; similar to physical robots that improve production and logistics efficiency, software robots can improve the efficiency of business processes, so RPA is crucial for enterprises to achieve digital transformation. For enterprises, business workflows automated using RPA can include their internal processes as well as customer-facing processes. Traditional RPA is performed by computer software based on rule-based algorithms, but in order to make RPA more intelligent and easy to use, modern RPA needs to combine machine learning and deep learning algorithms, and even use conversational robots as its conversational user interface.

More and more business processes are being conducted over the Internet, especially mobile Internet. Mobile applications developed for specific business processes of individual companies are expensive, and users outside the company are not interested in such dedicated applications and are unwilling to download and use them. Therefore, there is a considerable demand in the market for RPA that can serve customers through mobile Internet without downloading specific applications. In addition, instant messaging has replaced telephones and emails and has become the most common way of communication in our daily lives; public instant messaging programs are readily available, and several popular messaging platforms, such as WhatsApp, Facebook Messenger and WeChat, each have more than one billion active users. Taking advantage of the popularity of public instant messaging programs as the primary user interface for external customers of an enterprise, the present invention provides robotic processes via mobile Internet using instant messaging robots without requiring external customers to download any specific mobile phone applications.

In recent years, the demand for real-time extraction of text messages from digital images transmitted online has increased significantly. For example, the process of a user depositing a check into a bank account online: taking a photo of the check through the camera in the mobile app, uploading the check image to extract the text content in real time, and confirming the result; the entire process can be completed with just a few simple touch actions on the mobile phone panel. Similarly, users can upload photos of receipts or other proof of loss documents through the app to make insurance claims. In order to automate the above process, real-time extraction of text content from digital images is an indispensable step.

In the past few decades, text content extraction has been achieved through traditional optical character recognition (OCR) technology; however, although traditional OCR has good text extraction effects on images with clear document formats on clean backgrounds, it is not as effective for images with unformatted or complex backgrounds. In recent years, Convolutional Neural Network (CNN) technology has been applied to text content extraction, and has better versatility for images with multiple formats and backgrounds; however, practical methods for text content extraction based on CNN still need further development and improvement.

The purpose of the present invention is to solve the above problems through an instant messaging robot combined with robotic process automation, one implementation of which includes a new CNN method for extracting text content from digital images.

The present invention discloses a system and method for implementing robotic process automation (RPA) with an instant messaging robot, and a new method for extracting text content from digital images based on a convolutional neural network (CNN). The system includes a conversational robot application built for an enterprise (or organization), a software RPA manager, and an instant messaging platform. The RPA manager includes a plurality of enterprise workflow modules and can receive instructions from the enterprise conversational robot to execute each workflow. The enterprise instant messaging platform is also connected to one or more public instant messaging platforms via the Internet.

Through instant messaging, the system enables enterprise users connected to the enterprise instant messaging platform and external users connected to the public instant messaging platform to communicate with the enterprise conversation robot in one-on-one or group mode. Furthermore, enterprise users and external users can use the system to activate internal or customer-facing workflows and automate workflows with the assistance of the enterprise conversation robot.

In addition, the implementation of the robot process in the RPA manager of the present invention also includes extracting text content from digital images, and provides an improved convolutional neural network (CNN) method for such processes.

100:Enterprise instant messaging robot

102: Enterprise dialogue robot

104: Enterprise instant messaging platform

106:RPA Manager

108: Text extraction process

108a~108n: Enterprise workflow

108tp: Third-party RPA workflow

110: Public instant messaging platform

112: Enterprise users

114: External user

116: Enterprise database

200: CT-SSD architecture

204:ResNet-18 Convolutional Core

206: Conv4 convolutional layer

208: Text slice detection layer

210: Non-maximum suppression algorithm

212: Output: Text line border

300: Watershed U-net segmentation architecture

304~322: U network segmentation model

324: Text distribution map

326: Distribution map of eroded text

328: Watershed Algorithm

330: Output: Optimized text distribution map

400:CTC-CNN architecture

402: Input text line image

404:ResNet-18 Convolutional Core

406: Conv5 feature map

408, 410: One-dimensional convolution operation

412: Softmax operation

414:CTC loss function

416: Output: Text line content

FIG1 is a block diagram of an instant messaging robot for providing robot process automation according to an embodiment.

Figure 2 is a diagram of the CT-SSD architecture used in an embodiment for text detection in digital images.

Figure 3 is a diagram of the watershed U-net segmentation architecture used in an embodiment for text detection in digital images.

Figure 4 is a diagram of the CTC-CNN architecture used for text recognition of digital text line images in an embodiment; the resolution of the input text line image and the resolution of its convolutional layer feature map are marked in brackets respectively.

Figure 5 is a diagram of the VGG-16 architecture used for image recognition.

Figure 6 is a diagram of the ResNet-18 architecture for image recognition; the convolution backbone contains shortcut connections that bypass specific convolution operations (solid arrows represent constant connections; dashed arrows represent projection connections).

Figure 7 is a diagram of the SSD architecture for general object detection in digital images.

Figure 8 is a schematic diagram of the SSD object detection mechanism (from W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, A. C. Berg, <SSD: Single Shot Multi-Frame Detector>, arXiv: 1512.02325 (2016)). (a) Input image, and display of the outer frames of two objects detected by SSD: cat and dog. (b) 8x8 convolution feature map; each cell is configured with detection frames of different aspect ratios, and the two detection frames of cats are represented by thick dashed lines. (c) 4x4 convolution feature map; each cell is configured with detection boxes of different aspect ratios, and one of the detection boxes that detected a dog is indicated by a thick dashed line.

Figure 9 is a diagram of the CTPN architecture for text detection in digital images.

Figure 10 is a schematic diagram of the CTPN text detection mechanism. The 3x3 detection box that slides horizontally in the Conv5 feature map first generates a sequence of text slice proposals for subsequent processing. The spatial resolution (WxH) of the Conv5 feature map of VGG-16 depends on the resolution of the input image; if the input image is 512x512 pixels, then W=32, that is, the width of each feature unit cell is equivalent to 16 pixels in the original image.

Figure 11 is a schematic diagram of the CT-SSD text slice detector for sampling the Conv4 feature map of ResNet-18 in an embodiment. If the input image is 512x512 pixels, the width of the detector is fixed to 16 pixels (i.e., the width of the feature unit cell).

Figure 12 is an example of using the CT-SSD text detection of the present invention, showing (a) the detected candidate text slices, and (b) the text lines generated after the text slices are combined and calculated.

Figure 13 is a U-grid structure used for digital image semantic segmentation.

FIG14 is an example of using the watershed U-net segmentation of the present invention to perform text detection, showing (a) input image, (b) text distribution map after U-net segmentation, and (c) eroded text distribution map after U-net segmentation. The two and a half frames of completed text distribution map of FIG14 (b) and (c) are further processed by the watershed algorithm to generate the optimized text distribution map at the output.

FIG15 is an example of an embodiment using overlapping block method for image preprocessing before text detection. In this example, the resolution of the input image (shaded rectangle) is 1920x1080 pixels, the resolution of each block (dashed rectangle) is 512x512 pixels, the overlap between adjacent blocks is 32 pixels, and the blocks at the edge are padded as needed.

FIG1 illustrates an embodiment of an enterprise instant messaging robot 100 for robotic process automation (RPA). The system includes an enterprise (or organization)-specific conversation robot software 102, an instant messaging platform 104, and an RPA manager software 106. The RPA manager 106 includes a plurality of workflow modules 108-108n, each of which can provide an automated workflow for the enterprise. The enterprise instant messaging platform 104 is also connected to a public instant messaging platform 110 via the Internet. The system allows enterprise users 112 connected to the enterprise instant messaging platform 104, external users 114 connected to the public instant messaging platform 110, and the enterprise conversation robot 102 to communicate using instant messaging in one-to-one or group mode.

The enterprise conversation robot 102 includes software for receiving, processing, analyzing and responding to human messages in a simulated real-person manner. It consists of three main parts: (1) a natural language processing and understanding (NLP/NLU) module for analyzing the intent of incoming messages from enterprise users 112 or external users 114; (2) a conversation management (DM) module for interpreting the output (intent) of the NLP/NLU module and analyzing the ongoing human-computer conversation context, including previous messages or other information related to the conversation, and providing response instructions as output; and (3) a natural language generation (NLG) module for receiving response instructions from the DM module and generating response messages to enterprise users 112 or external users 114.

The enterprise instant messaging platform 104 includes software for managing instant messaging exchanges between enterprise users 112, external users 114, and enterprise conversation robot 102. In FIG1, solid arrows, dotted arrows, and dashed arrows represent message exchanges between enterprise users 112 and conversation robot 102, between external users 114 and conversation robot 102, and between enterprise users 112 and external users 114, respectively. The exchanged messages may include text, images, videos, hyperlinks, or other digital content. For more details about the enterprise conversation robot 102 and enterprise instant messaging platform 104 of the present invention, please refer to U.S. Patent Application No. 16/677,645, dated November 7, 2019.

The RPA manager 106 includes software for configuring and executing built-in workflows 108-108n. One or more RPA workflows 108tp provided by third-party developers can also be optionally connected to and controlled by the RPA manager 106 through an application programming interface (API). To activate a robotic workflow within the enterprise or facing customers, an enterprise user 112 or an external user 114 can send a message to the enterprise dialogue robot 102 to express such intention, and the enterprise dialogue robot 102 will respond and instruct the RPA manager 106 to execute the specified workflow.

Some workflows are end-to-end, i.e., they can directly generate outputs by simply receiving inputs; other workflows are interactive, i.e., users 112 or 114, dialogue robots 102, and RPA managers 106 need to interact back and forth in the process to generate outputs. In some cases, the enterprise dialogue robot 102 and RPA manager 106 also need to connect to the enterprise database 116 to obtain necessary information related to the ongoing instant messaging conversation or robot workflow.

It is not uncommon for a conversational robot to respond to a user's query with an incorrect answer or no answer at all, due to its inability to understand the nuances of human language or due to a lack of sufficient information; this may be considered a poor user experience, leading to a negative impression of the enterprise. The present invention provides a mechanism for an enterprise user 112 (e.g., a customer service person) to immediately intervene in an ongoing text message conversation between an external user 114 (e.g., a customer) and the enterprise conversational robot 102; in this way, friction or frustration that an external user may encounter in a conversation with the conversational robot can be immediately remedied, which can help enterprises improve the user experience of robot processes.

The present invention provides a robot workflow including daily office processes, such as automated meeting arrangements within an enterprise. The meeting initiator, that is, the enterprise user 112, only needs to send a meeting request to the enterprise dialogue robot 102, including the meeting theme, participants, expected time and venue; with the assistance of the RPA manager 106, the enterprise dialogue robot 102 will first compare the schedule of each participant and venue behind the scenes, and then propose the best time and venue to the meeting initiator and obtain confirmation; once confirmed, the enterprise dialogue robot 102 will send a meeting invitation SMS to each participant (also an enterprise user), and will also send a reminder SMS before the meeting.

Another example of an office workflow is the automated leave application and leave approval process. In this example, an enterprise user 112 submits a leave application to his supervisor (another enterprise user) through the enterprise dialogue robot 102; with the assistance of the RPA manager 106, the enterprise dialogue robot 102 will guide the supervisor to sign and approve to ensure that the enterprise's leave regulations are strictly followed, and will also send reminder text messages to the supervisor in a timely manner to ensure that the approval can be completed in time.

The present invention also provides a robotic workflow for extracting text content from digital images. The most basic of such workflows is end-to-end, in which a user 112 or 114 transmits a digital image to an enterprise dialogue robot 102, which forwards the image to an RPA manager 106 for text content extraction; the result is then transmitted back to the user 112 or 114 via the dialogue robot 102.

The interactive robotic workflow involving text content extraction can be illustrated by the following implementation of a robotic return merchandise authorization (RMA). RMA is crucial for a company's after-sales service; although most RMA claims are routine and repetitive, they take up the valuable time of customer service personnel, so there is a strong demand for robotic RMA in the market.

The following robotic RMA process represents an embodiment of the present invention: (1) Due to a damaged product, an external user 114 (customer) sends a message to the enterprise dialogue robot 102 through the public instant messaging platform 110 to request a return authorization RMA, and attaches a photo of the product label; (2) Using its NLP/NLU function, the enterprise dialogue robot 102 is able to understand the external user's intention; it forwards the product label image to the RPA manager 106, which then performs text content extraction 108 and returns the result to the enterprise dialogue robot 102; (3) The enterprise dialogue The robot 102 compares the text content (such as model number, serial number) extracted from the product label with the sales record in the enterprise database 116 to confirm whether the product is within the warranty period; (4) Based on the warranty status of the product, the enterprise dialogue robot 102 sends an RMA authorization code or a rejection message to the external user 114; and (5) If the response message of the external user 114 contains negative emotions during the process, the dialogue robot 102 immediately escalates the problem to the enterprise user 112 (customer service personnel), and then the enterprise user 112 takes over the subsequent dialogue with the external user 114.

Textual-Content Extraction Methods

The text content extraction methods provided by the present invention include two main steps: (1) performing text detection on the input image to extract the position and size of the text line in the image, and (2) performing text recognition on the individual text line images detected in step (1) to extract their text content. Figures 2, 3, and 4 respectively show two independent text detection methods and a text recognition method; these methods are all derived from a deep learning model based on a convolutional neural network (CNN). To facilitate the explanation of the present invention, the basic concept and methodology of CNN, as well as the related deep learning model, are first explained below.

Basic Convolutional Neural Network

In recent years, CNNs have been widely used to automatically detect, classify, and identify objects in digital images. Figures 5 and 6 show two commonly used CNN model architectures: VGG-16 (K. Simonyan, A. Zissermen, Very deep convolutional networks for large-scale image recognition, arXiv: 1409.1556 (2014)) and ResNet-18 (K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, arXiv: 1512.03385 (2015)). A complete CNN pipeline consists of a convolutional skeleton for extracting spatial features of an image, followed by a decoding path for detecting, classifying, or identifying objects in the image.

Typically, the convolutional backbone contains a number of convolutional operations, Rectified-Linear-Unit (ReLU) operations, and pooling operations arranged in a specific order. As shown in Figures 5 and 6, the input image first passes through the convolutional backbone to generate a series of feature maps, each of which contains WxH cells; the spatial resolution of these feature maps gradually decreases (as shown in the decreasing number of cells in the figure), indicating that they extract larger and larger local features in the original image. The resolution of the feature map in each convolutional layer (represented as Conv1 to Conv5) depends on the resolution of the input image; Figures 5 and 6 assume that the input resolution is 512×512 pixels. In the convolutional skeleton, each convolution operation involves scanning the input image or individual feature maps with multiple spatial filters (called convolution kernels), and the number of convolution kernels (called the number of channels) provides the depth of the feature map. Each pooling operation is also performed by scanning through a spatial filter (pooling filter), and the output value of the operation can be the maximum value (maximum pooling) or the average value (mean pooling) of the feature-cell values covered by the pooling filter. The ReLU operation usually follows each convolution operation, which sets all negative feature-cell values to zero and keeps the positive feature-cell values unchanged. In Figures 5 and 6, a convolution or pooling operation is labeled with its filter size, number of channels, and stride (size of each scan step), for example: "3x3 conv,256,/2" means "3x3 convolution kernel, 256 channels, stride 2 cells". For simplicity, all feature map edges in the following description are assumed to be padded, so that convolution and pooling operations with stride 1 do not change the size of the feature map (WxH), and the same operation with stride 2 will halve the size (W/2xH/2). In addition, ReLU operations are omitted in Figures 5 and 6 and are not shown. The decoding path of VGG-16 or ResNet-18 consists of one or more fully-connected (fc) neural layers and a normalized exponential function (Softmax) operation; Softmax is a standard activation function that normalizes the output value.

In CNN, each convolution operation or fully connected layer contains trainable weighted parameters; the actual applied CNN model may contain tens of millions of trainable parameters (e.g., VGG-16 contains 138.4 million parameters and ResNet-18 contains 11.5 million parameters). Therefore, in order to train a complete CNN model, a large amount (e.g., hundreds of thousands to millions of frames) of annotated training images is required. This is impractical for most applications that can only provide a limited number (e.g., hundreds to tens of thousands of frames) of training images. Fortunately, there are already some pre-trained CNN models for certain domains on the open source platform, and application developers can use all or part of these models for effective use; under appropriate conditions, it is possible to use a small set of training images in similar domains to successfully train CNN or its derivative models for new applications. Some embodiments of the present invention use the pre-trained convolutional backbone of VGG-16, ResNet-18 or their respective variants and derive them into new CNN methods. Generally speaking, compared with VGG variants, ResNet variants require fewer trainable parameters and are easier to train.

Single Shot Multi-Box Detector (SSD)

SSD is a CNN method for detecting general objects from digital images (W.Liu, D.Anguelov, D.Erhan, C.Szegedy, S.Reed, C.Y.Fu, A.C.Berg, <SSD: Single Shot Multi-Box Detector>, arXiv: 1510.22325 (2016)). It combines accuracy and speed and is a more efficient object detection model. The SSD architecture is shown in Figure 7. In addition to using the Conv1 to Conv5 convolutional backbone of VGG-16, it also adds several convolutional layers to produce a gradually decreasing feature map size (i.e., smaller WxH).

The detection layer of SSD takes all the feature unit values of the Conv4 feature map and the additional convolution feature map after Conv6 as its input (as shown by the thick arrows in Figure 7), and configures multiple detection frames of different aspect ratios on each feature cell (as shown in Figure 8 (b) and (c)). Generally, in a smaller feature map (with a larger cell), it is more likely to detect a larger object, and vice versa. This mechanism can detect the category and location of objects of different sizes in one shot. Figure 8 shows the simultaneous detection of cats and dogs in the input image: in Figure 8(b), the smaller cat is detected by two detection frames in the 8×8 feature map; in Figure 8(c), the larger dog is detected by a single detection frame in the 4x4 feature map.

The output of the detection layer provides the category score of each detected object, the coordinates and size of the object bounding box. The object bounding box does not have to coincide with the detection box of the detected object; usually, there will be deviations in coordinates, height, and width between the two. In addition, the same object may be detected by several detection boxes, thus generating several candidate object bounding boxes. The last step of SSD is the Non-Maximum Suppression algorithm, which is a rule-based algorithm that aims to select the best one from several candidate object bounding boxes.

Connectionist Text Proposal Network (CTPN)

There are two main differences between text detection and general object detection: first, general objects usually have clear closed boundaries, while text is composed of discrete elements (e.g., English letters, Chinese characters, punctuation marks, spaces), so the boundaries of text are less clear; second, text detection usually requires higher accuracy than general object detection, because if a line of text is only partially detected, it will cause serious errors in subsequent text recognition. Therefore, methods used to detect general objects (such as SSD) are not very effective for detecting text.

The Connectionist Text Proposal Network (CTPN) is designed specifically for text detection (Z. Tian, W. Huang, T. He, P. He, Y. Qiao, Detecting text in natural image with Connectionist Text Proposal Network, arXiv: 1609.03605 (2016)). The CTPN architecture is shown in Figure 9; similar to SSD, it uses the Conv1 to Conv5 layers of VGG-16 as the convolutional backbone, but the similarities end there. As shown in Figure 10, immediately after the convolution, CTPN uses a small detection box (3x3 window) to slide horizontally and scan line by line in the Conv5 feature map; from each line scanning process, CTPN generates a sequence of fine "text-slice proposals"; the width of each text slice proposal is fixed and only represents a small part of a text line (for example, if the input image is 512x512 pixels, the text slice width is only 16 pixels). In addition, CTPN provides a vertical anchor mechanism that can simultaneously predict the vertical position, height, and text/non-text score of each text slice proposal. The following is the method by which CTPN further detects text lines from these text slice proposal sequences.

As shown in Figure 9, each text slice proposal sequence is fed into a recurrent neural network (RNN) architecture with bidirectional long short-term memory (LSTM), and the LSTM encoder is used to extract the features of the text sequence; in other words, the recurrent neural network is used to analyze whether each text slice proposal is part of a text. Finally, the CTPN combines the text slice proposals whose text/non-text scores exceed the threshold (for example, 0.7) into a single or multiple text lines; if the horizontal distance between adjacent text slice proposals is less than 50 pixels and the vertical overlap exceeds 0.7, they are combined into the same text line.

Although CTPN is accurate in detecting text, its speed (e.g., 7 frames per second) is far slower than that of SSD (e.g., 59 frames per second). Therefore, a text detection method that is both accurate and fast is needed in practice.

The text detection method 1 of the present invention: Connectionist Text Single Shot Multi-Slice Detector (CT-SSD)

The present invention provides a text detection method that mixes CTPN and SSD, called a concatenated text single shot multi-slice detector. The purpose of CT-SSD is to achieve the text detection accuracy of CTPN and the speed of SSD at the same time. Figure 2 shows the architecture 200 of CT-SSD; similar to SSD and CTPN, CT-SSD uses a convolutional backbone 204 of a pre-trained CNN model (e.g., the convolutional backbone of ResNet-18 or VGG-16), but CT-SSD does not include the additional convolutional layers of SSD nor the large LSTM encoder network of CTPN.

CT-SSD adopts the multi-frame detection mechanism of SSD, but like CTPN, it only samples a single convolutional layer 206 (e.g., the Conv4 layer of ResNet-18) to detect fine text slices. Although each CNN convolutional layer in Figures 5 and 6 contains several feature maps with the same resolution (WxH) and depth (number of channels), CT-SSD's text detection only needs to sample the last feature map of the convolutional layer 206. In addition, the detection box configured by CT-SSD on each feature cell has a fixed width and several different heights (i.e., multiple aspect ratios), as shown in Figure 11, and the preferred embodiment of CT-SSD adopts an aspect ratio range of [1,8]; these fine detection boxes are called "text slice detectors". The spatial resolution of text slice detection is determined by the width of the text slice detector, which in turn is determined by the width of the sampled feature cell. As shown in the embodiment of FIG. 11 , if the input image 202 is 512×512 pixels, the resolution of the Conv4 layer feature map of ResNet-18 used for sampling is 32×32, and the text slice detection resolution is 16 pixels; if other detection resolutions are to be used, it can be achieved by sampling feature maps of different resolutions.

The output of the text slice detection layer 208 provides the coordinates, height, and text/non-text score of preliminary candidate text slices; they are then filtered by the non-maximum suppression algorithm 210 to select the most likely candidate text slices. Figure 12(a) shows an example of text slice detection using CT-SSD, in which the detected candidate text slices are marked with slice boxes of different heights and widths. Unlike the fixed-width text slice proposals in CTPN, the width of the candidate text slices in CT-SSD is variable. With the candidate text slices, the output layer 212 can use text combination rules similar to CTPN to connect them into text lines: if the horizontal distance between adjacent candidate text slices is less than a preset value (e.g., 50 pixels) and their vertical overlap exceeds a preset value (e.g., 0.7), they are combined into the same text line, as shown in Figure 12(b). The text detection efficiency of CT-SSD and CTPN is experimentally compared, and the results show that the former can not only achieve better accuracy, but also is 10 times faster than the latter.

U-Net

Semantic image segmentation is another method for detecting objects from digital images; in this method, each pixel of the image is classified according to the type of object to be detected. U-Net is a CNN method designed specifically for semantic image segmentation (O. Ronneberger, P. Fischer, T. Brox, <U-Net: Convolutional networks for biomedical image segmentation> (U-Net: Convolutional networks for biomedical image segmentation), arXiv: 1505.04597 (2015)). The present invention provides a text detection method based on U-Net semantic segmentation.

As shown in Figure 13, the U-Net architecture includes a typical convolutional path (Conv1 to Conv5) for extracting features of different sizes from the input image, and a transposed-convolutional path (T-Conv1 to T-Conv4) symmetrical to the convolutional path for accurate pixel-level positioning. The transposed convolution operation (labeled as "up conv" in Figure 13) projects the feature value of a cell in the feature map to the weighted feature values of multiple cells in the higher resolution feature map; therefore, the feature map size (WxH) in the transposed convolutional path is gradually increased. In addition, the up-sampled feature map of each transposed convolution layer is concatenated with the corresponding feature map of the same size in the convolution path (as shown by the thick arrows in Figure 13) to enhance the simultaneous detection of the content and position of the object. The output layer (denoted as "1x1 conv,C", where C is the number of categories of the object to be detected) provides the category score of each pixel in the input image.

Text detection method 2 of the present invention: Watershed U-Net Segmentation

As mentioned above, since text has no clear boundaries, the detection of the text line frame is sometimes ambiguous, especially when the distance between adjacent text lines is small; using pixel segmentation alone as text detection may not be able to solve this type of problem. Therefore, the present invention provides a text detection method that combines U-net segmentation with the watershed algorithm (the latter is often used to segment objects that touch each other). This new method is called watershed U-net segmentation.

FIG3 is a diagram of the watershed U-net segmentation architecture 300, which includes a complete U-net model 304-322. In addition to the normal text distribution map 324 (i.e., pixel-level text distribution) output channel, an additional auxiliary output channel is added: an eroded text distribution map 326, as shown in FIG14(b) and FIG14(c). In order to train the U-net model to output the text distribution map and the eroded text distribution map at the same time, the present invention expands the original training image set: the edge of each annotated text line in the original image is cut off by 15% to generate a new training image. After U-net segmentation, the watershed algorithm 328 is used to process the two and a half frames of the completed text distribution map to generate an optimized text distribution map of the output layer 330.

In practical applications, the resolution of the input image may have a wide range; in order to achieve fully automatic text detection, the embodiment of the present invention uses overlapping-tiles to pre-process the image before the aforementioned text detection step. As shown in the example of Figure 15, the resolution of the input image (shaded rectangle) is 1920x1080 pixels, the resolution of each tile (dashed rectangle) is fixed at 512x512 pixels, the overlapping part between adjacent tiles is 32 pixels, and the tiles at the edge are filled with padding as needed. For the image in each tile, CT-SSD or watershed U-net segmentation is first used to detect the text line, and then the results are merged to obtain the position and size of each text line relative to the original input image.

The text recognition method of the present invention: Connectionist Temporal Classification CNN (CTC-CNN)

The two text detection methods of the present invention provide the position and size (wxh) of the text line frame in the input image; followed by a text recognition process for identifying the text content of each text line, which is a sequence-to-sequence process that can be performed using a CNN method based on a connection-time classification (CTC) loss function. (F.Borisyuk, A.Gordo, V.Sivakumar, <Rosetta: Large scale system for text detection and recognition in images>, arXiv: 1910.05085 (2019)).

FIG4 shows the architecture diagram 400 of CTC-CNN, which includes a convolutional backbone 404 of a pre-trained CNN model (e.g., the convolutional backbone of ResNet-18 Conv1-Conv5). During the training and testing of the model, the size of all text-line images 402 is proportionally adjusted to wx32 (i.e., the image height is fixed to 32 pixels, and the aspect ratio is the same as the original size). Therefore, the resolution of the feature map 406 at the end of the convolutional backbone (Conv5) is w/32x1; this is a one-dimensional feature sequence containing w/32 cells, each cell representing a position in the input text-line image 402. Following the Conv5 layer, there are two consecutive one-dimensional convolution operations 408 and 410 using 3x1 convolution kernels, and the feature sequence 410 at its output contains C channels, corresponding to the total number of text elements to be recognized. In one embodiment of the present invention, C=4,593, corresponding to 52 English letters, 10 numbers, 40 punctuation marks, and 4,491 Chinese characters. The Softmax operation 412 combined with the CTC loss function 414 provides the probability distribution of all text elements for each cell in the sequence; after removing the blank and repeated text elements, the text content 416 of the input text line image 402 can be obtained.

The present invention discloses a system and method for implementing Robotic Process Automation (RPA) with an instant messaging robot, and a new method for extracting text content from digital images based on a convolutional neural network (CNN). The system includes a conversational robot built for an enterprise, a software RPA manager, and an instant messaging platform. The RPA manager includes a plurality of enterprise workflow modules and can receive instructions from the enterprise conversational robot to execute each workflow. The enterprise instant messaging platform is also connected to one or more public instant messaging platforms via the Internet.

Through instant messaging, the system enables enterprise users connected to the enterprise instant messaging platform and external users connected to the public instant messaging platform to communicate with the enterprise conversation robot in one-on-one or group mode. Furthermore, enterprise users and external users can use the system to activate internal or customer-facing workflows and automate workflows with the assistance of the enterprise conversation robot.

In addition, the implementation example of the robot process in the RPA manager of the present invention also includes extracting text content from digital images, and provides an improved convolutional neural network (CNN) method for such processes. The above is only a preferred embodiment of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope of the present invention.

100:Enterprise instant messaging robot

102: Enterprise dialogue robot

104: Enterprise instant messaging platform

106:RPA Manager

108: Text extraction process

108a~108n: Enterprise workflow

108tp: Third-party RPA workflow

110: Public instant messaging platform

112: Enterprise users

114: External user

116: Enterprise database

Claims (2)

A method for automatically detecting text lines in a digital image comprises: a convolution neural network backbone extracting the spatial features of the digital image; a text slice detection layer detecting a plurality of candidate text slices in the digital image; a non-maximum value based A filter layer of a Suppression algorithm selects a group of best candidate text slices from the candidate text slices; and a text construction layer combines the group of best candidate text slices into one or more text lines based on the following rule: if the horizontal distance between two adjacent candidate text slices is less than a preset value and the vertical overlap is greater than another preset value, then the two belong to the same text line; wherein the convolution neural network backbone is composed of a plurality of convolution layers in an orderly manner, including a plurality of convolution operations, linear rectification function operations, and pooling operations, and is used to generate a series of multi-channel features from the digital image. A convolutional layer comprises a plurality of feature maps, wherein each convolutional layer comprises a plurality of feature maps, and the feature maps have the same spatial resolution and the same number of channels, and the feature map contained in the next convolutional layer of the convolutional layer has the same or lower spatial resolution and a larger number of channels as the feature map of the previous layer; and the text slice detection layer samples a single feature map of a single convolutional layer in the convolutional bone trunk, and a set of a plurality of text slice detectors is arranged in each cell of the feature map, and the width of the set of text slice detectors is equal to the width of the cell, and the heights are different, and the height-to-width ratio is between 1 and 8. A method for automatically detecting Chinese characters in pixelated digital images, comprising: a convolutional neural network generating two and a half frames of pixel-level text distribution maps from the digital image, the convolutional neural network comprising a convolutional path followed by a transposed convolutional path; and an adjustment layer generating an optimal pixel-level text distribution map; wherein the convolutional path The method comprises a plurality of convolution layers, which include a plurality of convolution operations, linear rectifier operations, and pooling operations, and is used to generate a series of multi-channel feature maps from the digital image; wherein each convolution layer includes a plurality of feature maps, and the feature maps all have the same spatial resolution and the same number of channels, and the feature map contained in the next convolution layer of the convolution layer has the same or lower spatial resolution as the feature map of the previous layer. resolution and a larger number of channels; wherein the transposed convolution path is composed of a plurality of transposed convolution layers in order, including a plurality of transposed convolution operations, concatenation operations, convolution operations, and linear rectifier function operations, for generating a series of multi-channel feature maps with gradually increasing resolutions; wherein each transposed convolution layer contains a plurality of feature maps with the same spatial resolution , wherein the first feature map is generated by the following two steps: 1. performing a transposed convolution operation on the last feature map of the previous layer to improve the resolution; 2. performing a sequential operation on the feature map generated by the operation and the feature map with the same resolution in the convolution path; wherein the two and a half frames of text distribution maps generated by the convolution neural network are one original text and the other eroded text; and wherein the adjustment layer applies the watershed algorithm to generate an optimized text distribution map with non-overlapping text lines from the two and a half frames of text distribution maps.