CN114596571B - An intelligent lens-free text recognition system - Google Patents

Abstract

The invention discloses an intelligent lens-free character recognition system. The system comprises an optical module and a calculation imaging and intelligent text positioning recognition module, wherein the optical module consists of a mask plate with a modulatable amplitude and a sensor, wherein the amplitude distribution of transmitted light of the mask plate is modeled as a two-dimensional convolution layer and can be optimized as a parameter; the computational imaging and text positioning recognition module comprises a computational imaging model, a text positioning model and a text recognition model, wherein input data are raw data obtained on a sensor after passing through the optical module, the input data are output into a text form of predicted text, and meanwhile, the light transmission amplitude distribution of a mask plate in the optical module and the computational imaging network parameters are optimized through result feedback. The invention realizes the optimization of the lens-free imaging and word recognition deep learning model of the software and hardware integration, improves the accuracy of word positioning and word recognition under the lens-free condition, and has the advantages of universality and universality of each module of the system and strong practical applicability.

Description

An intelligent lens-free text recognition system

Technical Field

The invention belongs to the field of lens-free imaging, and in particular relates to an intelligent lens-free text recognition system.

Background Art

With the rapid development and application of visual tasks, cameras are integrated into various hardware devices. Some application scenarios have strict requirements on the size of the camera. Lensless cameras are imaging systems that use thin masks instead of lenses, so the size of the camera can be greatly reduced.

Compared with cameras with lenses, lensless cameras need to perform computational imaging on the data collected on the sensor to restore the image. However, the images reconstructed based on lensless technology have the disadvantages of blur and resolution, which makes them incapable of many visual tasks. Currently, there is no research on non-single-character text detection and recognition based on lensless technology.

Therefore, a lens-free text recognition system is needed.

Summary of the invention

In view of the fact that the current lensless imaging technology has not been applied to the positioning and recognition of texts other than single letters due to its poor imaging quality, the present invention provides a lensless text positioning and recognition system with high recognition accuracy and universal applicability.

The technical solution adopted by the present invention is as follows:

The intelligent lens-free text recognition system of the present invention comprises an optical module and a computational imaging and text positioning recognition module. The optical module is mainly composed of a modulated amplitude mask plate and an optical sensor placed in parallel. The target to be recognized is placed in front of the optical module. The light emitted by the target to be recognized is scattered by the modulated amplitude mask plate and then projected on the plane of the optical sensor to form a projection image (raw data). The optical sensor transmits the projection image to the computational imaging and text positioning recognition module.

The computational imaging and text recognition module includes a computational imaging model, a text positioning model and a text recognition model, and the three models are connected in series; the input of the computational imaging and text recognition module is the projection image obtained on the sensor after passing through the optical module, and the output is the text form of the text on the projection image.

The modulated amplitude mask plate is a binary mask plate composed of k*k cells, and the value of each cell is 1 or 0, 1 indicates that light can pass through, and 0 indicates that light cannot pass through.

The projected image is outputted as a predicted reconstructed image by the computational imaging model; the text positioning model processes the input reconstructed image and outputs the position of the text in the image; after the output result of the text positioning model is inputted into the text recognition model, the text recognition result of the image is outputted;

During the training of computational imaging and text recognition modules, only the computational imaging model participates in the training and needs to update parameters. The text positioning model and text recognition model do not participate in the training.

The computational imaging model is a neural network of an encoder-decoder system, specifically using U-NET; the text localization model adopts an arbitrary text localization model structure, specifically using CTPN; the text recognition model adopts an arbitrary text recognition model structure, specifically using CRNN.

The pattern on the modulated amplitude mask is displayed through a liquid crystal display, and the pattern on the mask is randomly generated or determined after training optimization; the method for determining the mask pattern after training optimization includes the following steps:

1) The imaging process of the target to be identified and the optical module is modeled as a two-dimensional convolutional layer, specifically:

m＝w*o

Wherein, w represents the amplitude distribution on the mask, that is, the value distribution of the cell on the mask; a coordinate system is constructed with the center point of the mask as the origin, (i, j) is the coordinate of the center point of the cell on the mask, and w _{i, j} represents the value of the cell with coordinates (i, j) on the mask;

o represents the image after scaling on the sensor plane when the target to be identified does not pass through the mask (i.e., o represents the image after scaling on the sensor plane when the target to be identified passes through the aperture); a coordinate system is constructed with the center point of the sensor plane as the origin, (x, y) represents the coordinate value of the pixel point of the projected image on the sensor plane, o _{x, y} represents the pixel value at (x, y) on the sensor plane when the target to be identified does not pass through the mask; o _{x+i, y+j} represents the pixel value at (x+i, y+j) on the sensor plane;

m represents the image of the target to be identified after passing through the mask and projected onto the sensor plane; m _{x, y} represents the pixel value of the target to be identified at (x, y) on the sensor plane after passing through the mask;

k represents the number of rows or columns of cells on the mask, i∈[1, k];

2) Binarize the two-dimensional convolutional layer to obtain a two-dimensional convolutional layer of a binary neural network. The results are as follows:

in,

Among them, w ^b represents the result after binarization of w;

Since the mask has only 0 and 1 values, we use a binary neural network for training. The binary neural network uses the sign function to map two continuous values to -1 or +1, then adds 1 and divides by 2;

3) The parameters w ^b of the two-dimensional convolutional layer of the binary neural network are used as model parameters to train and optimize together with the computational imaging and text positioning and recognition modules;

3.1) During the training process, the pattern of the mask is randomly initialized through circuit adjustment, and the result of random initialization is used as the initial parameter of the convolution layer of the binary neural network;

3.2) Training of the system forward propagation process: Fix the target to be identified, measure the projection image of the target to be identified on the plane of the optical sensor after passing through the mask in the real physical scene, and use it as the input of the computational imaging and text positioning recognition module;

Training of the back-propagation process: Calculate the loss function Loss between the predicted image and the real image label output by the imaging and text positioning recognition module, back-propagate the loss function Loss to the binary neural network convolution layer, update the binary neural network convolution layer parameter w ^b , and modulate the adjustable mask according to the updated parameter w ^b . The modulation result is used as the mask pattern in the forward propagation process of the model in the next round of training;

3.3) The mask pattern obtained after training is the optimized result.

The cell size of the modulatable mask is the same as the pixel size on the sensor plane; the distance d1 between the target to be identified and the modulatable amplitude mask is much larger than the distance d2 between the modulatable amplitude mask and the optical sensor, d1>100d2; therefore, the amplitude distribution on the mask is approximately equal to the projection of the amplitude distribution on the mask on the sensor plane.

The loss function Loss of the computational imaging and text positioning recognition module during the training process is:

Loss = a × Loss1 + b × Loss2;

Among them, Loss1 is the error between the predicted image output by the computational imaging model and the real image label (the target image to be identified); Loss2 is the error between the predicted text finally output by the computational imaging and text positioning and recognition module and the real text label of the target to be identified (the text information on the target image to be identified); a and b are weights.

Effective benefits of the present invention:

The lens-free text recognition system of the present invention can reduce the size limitation caused by the lens, making it more convenient to integrate the camera into other devices.

The present invention realizes the optimization of deep learning models of lens-free imaging and text recognition with integrated hardware and software, improves the accuracy of text positioning and text recognition in the lens-free environment, and each module of the system is universal and applicable and has strong practical applicability.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an overall data flow of the present invention.

FIG. 2 is a schematic diagram of an optical module in the present invention.

FIG. 3 is a schematic diagram of the computational imaging and text positioning and recognition module of the present invention.

DETAILED DESCRIPTION

The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

As shown in Figure 1, the lensless text recognition system of the present invention includes an optical module, a computational imaging and text positioning recognition module. The target to be recognized obtains raw data through the optical module, and the raw data obtains the text form of the text through the computational imaging and text positioning recognition module.

The optical module is composed of a modulated amplitude mask and a sensor, wherein the modulated amplitude mask is a binary mask, which is divided into k*k cells, the value of the cell is 1 or 0, 1 represents that the light can pass through, and 0 represents that the light cannot pass through. It is placed in front of the optical sensor, and the two are placed in parallel. The object is placed in front of the mask, and the light emitted by the object is scattered by the modulated amplitude mask, and a specific projection image (raw data) is projected on the plane where the sensor is located, which is recorded by the sensor and transmitted to the computational imaging and text positioning recognition module. The pattern of the modulated amplitude mask can be controlled in real time by the circuit, and can be displayed by a liquid crystal display. The pattern on the mask can be randomly generated or updated by training optimization; the mask obtained by training optimization can make the recognition effect of the computational imaging and text positioning recognition module better. When the mask pattern in the optical module is fixed, without optimizing the mask, using a fixed mask pattern, the positioning and recognition results of the text can still be obtained by the raw data obtained on the sensor through the computational imaging and text positioning recognition module.

As shown in FIG2 , the method for training and optimizing the mask pattern is as follows:

1) Model the interaction between the imaging target and the mask as a two-dimensional convolutional layer

m＝w*o

Wherein, w represents the amplitude distribution on the mask, that is, the value distribution of the cell on the mask; a coordinate system is constructed with the center point of the mask as the origin, (i, j) is the coordinate of the center point of the cell on the mask, and w _{i, j} represents the value of the cell with coordinates (i, j) on the mask;

o represents the scaled image of the target to be identified on the sensor plane when the modulated amplitude mask is fully transparent and the target does not pass through the mask; a coordinate system is constructed with the center point of the sensor plane as the origin, (x, y) represents the coordinate value of the pixel point on the sensor plane, and o _{x, y} represents the pixel value at (x, y) on the sensor plane when the target to be identified does not pass through the mask;

m represents the image of the target to be identified after passing through the mask and projected onto the sensor plane; m _{x, y} represents the pixel value of the target to be identified at (x, y) on the sensor plane after passing through the mask;

k represents the number of rows or columns of cells on the mask, i∈[1,k].

Since the pixel size of the mask unit cell is the same as the sensor pixel size, and d1>>d2 (d1 is greater than 100 times d2), w can be approximately equal to the projection of the amplitude distribution on the mask on the sensor plane.

2) The parameters w of the two-dimensional convolutional layer are used as model parameters to train and optimize together with the subsequent computational imaging and text positioning and recognition modules.

2.1) Binarize the two-dimensional convolutional layer to obtain a binary neural network:

Among them, w ^b represents the result of binarization of w.

Since the mask has only 0 and 1 values, we use a binary neural network for training. The binary neural network uses the sign function to map w to -1 or +1, then adds 1 and divides by 2.

2.2) During the training process, the mask pattern is randomly initialized through circuit adjustment, and this value is used as the initial parameter of the convolutional layer of the binary neural network.

In the forward propagation process of the system: fix the target object to be measured, measure the raw data obtained on the sensor after passing through the mask in the real physical scene, and use it as the input of the computational imaging and text positioning recognition module.

After the computational imaging and text positioning and recognition modules, the error with the actual label is obtained.

During the gradient back propagation process: calculate the error between the output of the imaging and text positioning recognition module and the true label, calculate the gradient and back propagate it, back propagate the error to the binary neural network convolution layer, update the binary neural network convolution layer, and modulate the adjustable mask according to the updated weight result as the mask pattern in the forward propagation process of the model in the next round of training.

After training is completed, the mask pattern is fixed.

As shown in Figure 3, the computational imaging and text recognition module includes a computational imaging model, a text positioning model, and a text recognition model. During use, the module is composed of three models in series, the input data is the original data obtained on the sensor after passing through the optical module, and the output is the text form of the predicted text. During the training process, only the computational imaging model needs to update the parameters, and the text positioning model and the text recognition model do not participate in the training.

The computational imaging model is an imaging method based on deep learning, which can calculate the predicted reconstructed image (model output) from the raw data (model input) obtained on the sensor. The computational imaging model is a neural network with an encoder-decoder structure, and a U-NET structure network can be used specifically. During the training process, images containing letters and numbers are used as targets to be identified, and the raw data obtained on the sensor after passing through the optical module is used as the model input. The training loss function consists of two parts: 1) the error Loss1 between the target image to be identified (real image label) and the predicted image output by the model; 2) the predicted reconstructed image is input into the subsequent text positioning model and text recognition model to obtain the predicted text, and the error Loss2 between the predicted text and the text label of the target to be identified is calculated. The loss function Loss = a × Loss1 + b × Loss2.

Gradient back propagation: This loss function calculates the gradient backpropagation to update the computational imaging model and the two-dimensional convolutional layer mentioned above, and finally obtains the corresponding trained computational imaging model.

The text localization model is a neural network model. The input data is an image, and the output data is the location of the text. The model can use any text localization model structure, and the specific structure can be CTPN: the image passes through the VGG16 network, and each row of the last convolution layer of the network calculates a 3*3 sliding window, and the results are connected through the BLSTM structure, and finally input into a fully connected layer, and the output is the predicted coordinates and confidence score. And the model is trained, and does not participate in the back propagation update parameter of the loss function in this system.

The text recognition model is a neural network model. The input data is an image with letters and numbers, and the output is the result of text recognition. The model can use any text recognition model structure, such as CRNN: the image first passes through several convolutional layers to extract the feature sequence, enters the BLSTM, and finally outputs the score of the predicted letter. And the model is trained, and does not participate in the back propagation of the loss function to update the parameters in this system.

Claims (6)

1. An intelligent lens-free text recognition system, characterized in that it comprises an optical module and a computational imaging and text positioning recognition module, wherein the optical module is composed of a modulated amplitude mask plate and an optical sensor placed in parallel, and a target to be recognized is placed in front of the optical module, and the light emitted by the target to be recognized is scattered by the modulated amplitude mask plate and then projected on the plane of the optical sensor to form a projection image, and the optical sensor transmits the projection image to the computational imaging and text positioning recognition module; The pattern on the modulated amplitude mask is displayed on a liquid crystal display, and the pattern on the mask is determined after training optimization; The computational imaging and text recognition module includes a computational imaging model, a text positioning model and a text recognition model, and the three models are connected in series; the input of the computational imaging and text recognition module is the projection image obtained on the sensor after passing through the optical module, and the output is the text form of the text on the projection image; The method for determining the mask pattern after training optimization comprises the following steps: 1) The imaging process of the target to be identified and the optical module is modeled as a two-dimensional convolutional layer, specifically: m＝w*o Wherein, w represents the amplitude distribution on the mask, that is, the value distribution of the cell on the mask; a coordinate system is constructed with the center point of the mask as the origin, (i, j) is the coordinate of the center point of the cell on the mask, and w _{i, j} represents the value of the cell with coordinates (i, j) on the mask; o represents the scaled image of the target to be identified on the sensor plane when it does not pass through the mask; a coordinate system is constructed with the center point of the sensor plane as the origin, (x, y) represents the coordinate value of the pixel point of the projected image on the sensor plane, o _{x, y} represents the pixel value at (x, y) on the sensor plane when the target to be identified does not pass through the mask; o _{x+i, y+j} represents the pixel value at (x+i, y+j) on the sensor plane; m represents the image of the target to be identified after passing through the mask and projected onto the sensor plane; m _x,y represents the pixel value of the target to be identified at (x,y) on the sensor plane after passing through the mask; k represents the number of rows or columns of cells on the mask, i∈[1,k]; 2) Binarize the two-dimensional convolutional layer to obtain a two-dimensional convolutional layer of a binary neural network. The results are as follows: in, Among them, w ^b represents the result after binarization of w; 3) The parameters w ^b of the two-dimensional convolutional layer of the binary neural network are used as model parameters to train and optimize together with the computational imaging and text positioning and recognition modules; 3.1) During the training process, the pattern of the mask is randomly initialized through circuit adjustment, and the result of random initialization is used as the initial parameter of the convolution layer of the binary neural network; 3.2) Training of the system forward propagation process: Fix the target to be identified, measure the projection image of the target to be identified on the plane of the optical sensor after passing through the mask in the real physical scene, and use it as the input of the computational imaging and text positioning recognition module; Training of the back-propagation process: Calculate the loss function Loss between the predicted image and the real image label output by the imaging and text positioning recognition module, back-propagate the loss function Loss to the binary neural network convolution layer, update the binary neural network convolution layer parameter w ^b , and modulate the adjustable mask according to the updated parameter w ^b . The modulation result is used as the mask pattern in the forward propagation process of the model in the next round of training; 3.3) The mask pattern obtained after training is the optimized result. 2. An intelligent lens-free text recognition system according to claim 1, characterized in that the modulated amplitude mask is a binary mask consisting of k*k cells, and the value of each cell is 1 or 0, 1 indicates that light can pass through, and 0 indicates that light cannot pass through. 3. The intelligent lens-free text recognition system according to claim 1 is characterized in that the projection image is outputted as a predicted reconstructed image by the computational imaging model; the text positioning model processes the input reconstructed image and outputs the position of the text in the image; after the output result of the text positioning model is inputted into the text recognition model, the text recognition result of the image is outputted; During the training process of computational imaging and text recognition modules, only the computational imaging model participates in the training, while the text positioning model and text recognition model do not participate in the training. 4. An intelligent lens-free text recognition system according to claim 3, characterized in that the computational imaging model is a neural network of an encoder-decoder system, specifically U-NET; the text positioning model adopts an arbitrary text positioning model structure, specifically CTPN; the text recognition model adopts an arbitrary text recognition model structure, specifically CRNN. 5. According to claim 1, an intelligent lens-free text recognition system is characterized in that the cell size of the modulated mask is the same as the pixel size on the sensor plane; the distance d1 between the target to be identified and the modulated amplitude mask is much larger than the distance d2 between the modulated amplitude mask and the optical sensor, d1>100d2; therefore, the amplitude distribution on the mask is approximately equal to the projection of the amplitude distribution on the mask on the sensor plane. 6. The intelligent lens-free text recognition system according to claim 3, characterized in that the loss function Loss of the computational imaging and text positioning recognition module during the training process is: Loss = a × Loss1 + b × Loss2; Among them, Loss1 is the error between the predicted image output by the computational imaging model and the real image label; Loss2 is the error between the predicted text finally output by the computational imaging and text positioning and recognition module and the real text label of the target to be recognized; a and b are weights.