License plate recognition method and system
Technical Field
The invention relates to the field of image recognition, in particular to a method and a system for improving the success rate of remote small-target license plate detection and recognition by combining a semantic segmentation technology.
Background
With the development of scientific technology, the use of artificial intelligence is more and more extensive, such as license plate recognition, face recognition and the like, and especially with the maturity of various deep learning technologies, the exploration of various deep network models of image semantic segmentation, image detection, image classification, image recognition and image retrieval is more and more deep and fine. The automatic license plate recognition technology is widely applied to scenes such as communities, hotels, road checkpoints and the like, and the recognition success rate is over 98%.
The identification of license plates is technically divided into two types: 1. and the license plate recognition is based on traditional image processing 2 and deep learning. The traditional license plate recognition is divided into three steps: license plate positioning, character segmentation and character recognition, and the recognition precision is limited; the vehicle license plate recognition based on deep learning has stronger generalization capability, and the vehicle license plate recognition success rate is higher due to an end-to-end recognition mode based on a neural network architecture. The license plate recognition based on deep learning is divided into two main steps, the license plate is detected firstly, namely the position of the license plate is found in a picture, and then OCR recognition of license plate characters is carried out. The license plate detection scheme also comprises two schemes of target detection and semantic segmentation. However, in any scheme, the image needs to be subjected to resize operation before being subjected to the neural network, because the neurons of the neural network are limited, the network which is too deep and wide will require more training sets, and the difficulty of training is greatly increased, so that input resolution is necessary for performing the resize operation. However, resize operation causes a problem, if the position of the vehicle is far away, or if the whole license plate occupies a small area in the image, the detected and segmented license plate is very fuzzy, resulting in a low recognition success rate.
The Chinese patent application entitled "dynamic identification method of multiple license plates in real-time traffic scene" (CN201810368034.4) provides a dynamic identification method of multiple license plates in real-time traffic scene, which is mainly divided into several main steps of image preprocessing, license plate positioning network training, image post-processing, identification network training and the like. The license plate positioning network and the character recognition network are utilized, a plurality of license plate information in a single picture can be recognized efficiently in real time under the condition that character information is not split, and compared with the traditional character splitting and recognizing method, the method has the innovative advantage and can be applied to monitoring of violation behaviors such as parking lot charging, expressways and the like in a large quantity. However, this approach directly resizes the picture to 224 x 224, with a low success rate for far-end scene detection and recognition.
A chinese patent application entitled "a license plate detection and integral recognition method based on deep learning" (CN201710187289.6) proposes a license plate detection and integral recognition method based on deep learning, which is characterized by comprising the following steps: step a, vehicle detection is carried out to obtain a target vehicle, and the target vehicle determines the whole license plate detection area; b, dividing the license plate detection area into n small blocks, wherein the small blocks are partially overlapped; c, fitting by using a first deep neural network model to obtain a rough license plate area, and meanwhile, obtaining the credibility that the area is a real license plate; d, fusing according to the relation between the position of the license plate area and the reliability to obtain a final license plate area; and e, integrally identifying the license plate number of the license plate area through a second deep neural network model. The scheme has the disadvantages that the vehicle detection part uses the traditional image processing scheme, the accuracy rate is low, in addition, the license plate detection uses a simple block dividing mode, the target can be divided into different blocks for a larger target, and the missing rate is high; and the noise of a smaller target background is larger, and the false detection rate is higher.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
In order to improve the success rate of license plate detection and recognition, particularly the success rate of the detection and recognition of a remote small-target license plate, the invention firstly uses a semantic segmentation technology to carry out semantic segmentation on a vehicle and a background, then extracts the vehicle from the background, and then improves the screen occupation ratio of the license plate in a picture through processing, thereby improving the success rate of the detection and recognition of the remote small-target license plate.
According to one aspect of the invention, a license plate recognition method is provided, and the method comprises the following steps:
performing semantic segmentation processing on the picture to be recognized to obtain a two-classification segmentation picture, wherein the semantic segmentation distinguishes a vehicle from a background region;
using the two classification segmentation maps to perform vehicle matting processing;
carrying out logic processing on the image subjected to the cutout processing so as to improve the screen ratio of the vehicle to the license plate; and performing license plate recognition on the logically processed picture.
According to a further embodiment of the present invention, the semantic segmentation processing on the picture to be recognized further comprises:
using a pre-trained vehicle semantic segmentation model to distinguish a vehicle from a background, wherein the vehicle semantic segmentation model is trained by:
performing two-classification marking on a vehicle picture data set for training, wherein the vehicle is marked to be in one color, and the background is in another color; and
and inputting the marked training data set into a neural network for training to obtain the vehicle semantic segmentation model.
According to a further embodiment of the present invention, the vehicle matting process using the breakdown map further comprises:
carrying out binarization operation on the two classification segmentation graphs;
adjusting the binarized two-class segmentation graph to the size of an original graph; and
and performing a masking operation on the original image by using the adjusted binary image to replace the background part in the original image with white.
According to a further embodiment of the present invention, the logic processing the matting processed picture further comprises:
detecting whether a vehicle exists in the image subjected to the cutout processing; and
when one or more vehicles are detected to exist, each vehicle is cut out of the original image to obtain a picture containing the vehicle, so that each vehicle has a high screen ratio in the cut picture.
According to a further embodiment of the present invention, the license plate recognition of the logically processed picture further comprises:
preprocessing the image after logic processing to adapt to a license plate recognition model; and
and inputting the preprocessed picture into a license plate recognition model to obtain a license plate recognition result.
According to another aspect of the present invention, there is provided a license plate recognition system, including:
a vehicle semantic segmentation inference module configured to perform semantic segmentation processing on a picture to be recognized to obtain a two-class segmentation picture, wherein the semantic segmentation distinguishes a vehicle from a background region;
a vehicle automatic cutout module configured to perform a vehicle cutout process using the binary segmentation map;
a business logic processing module configured to logically process the matting processed picture to improve a screen fraction of the vehicle and the license plate; and
a license plate recognition module configured to perform license plate recognition on the logically processed picture.
According to a further embodiment of the present invention, the semantic segmentation processing on the picture to be recognized further comprises:
using a pre-trained vehicle semantic segmentation model to distinguish a vehicle from a background, wherein the vehicle semantic segmentation model is trained by:
performing two-classification marking on a vehicle picture data set for training, wherein the vehicle is marked to be in one color, and the background is in another color; and
and inputting the marked training data set into a neural network for training to obtain the vehicle semantic segmentation model.
According to a further embodiment of the present invention, the vehicle matting process using the breakdown map further comprises:
carrying out binarization operation on the two classification segmentation graphs;
adjusting the binarized two-class segmentation graph to the size of an original graph; and
and performing a masking operation on the original image by using the adjusted binary image to replace the background part in the original image with white.
According to a further embodiment of the present invention, the logic processing the matting processed picture further comprises:
detecting whether a vehicle exists in the image subjected to the cutout processing; and
when one or more vehicles are detected to exist, each vehicle is cut out of the original image to obtain a picture containing the vehicle, so that each vehicle has a high screen ratio in the cut picture.
According to a further embodiment of the present invention, the license plate recognition of the logically processed picture further comprises:
preprocessing the image after logic processing to adapt to a license plate recognition model; and
and inputting the preprocessed picture into a license plate recognition model to obtain a license plate recognition result.
Compared with the scheme in the prior art, the license plate recognition method and the license plate recognition system provided by the invention at least have the following advantages:
(1) has high accuracy. According to the invention, semantic segmentation is carried out on the vehicle, then the vehicle picture is fed into a license plate detection neural network, the resolution reduction range after the picture resize is smaller, the license plate characters are clearer, and the subsequent license plate number identification is more accurate;
(2) the training efficiency is high. The invention filters the irrelevant noise by using the semantic segmentation technology, thereby accelerating the convergence speed of model training;
(3) has flexible adaptability. The semantic segmentation, the license plate detection and the license plate recognition of the vehicle can use a classical neural network and can also use a traditional image processing-based method; and
(4) has economical efficiency. The method greatly reduces the number of training samples and the use of GPU resources.
These and other features and advantages will become apparent upon reading the following detailed description and upon reference to the accompanying drawings. It is to be understood that both the foregoing general description and the following detailed description are explanatory only and are not restrictive of aspects as claimed.
Drawings
So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only some typical aspects of this invention and are therefore not to be considered limiting of its scope, for the description may admit to other equally effective aspects.
Fig. 1 is a schematic configuration diagram of a license plate recognition system according to an embodiment of the present invention.
FIG. 2 is a schematic flow diagram of a method of training a deep learning network-based license plate recognition model according to one embodiment of the present invention.
FIG. 3 illustrates a neural network architecture diagram that may be used with the vehicle semantic segmentation model of the present invention.
FIG. 4 illustrates an exemplary architecture of a character recognition model according to one embodiment of the invention.
Detailed Description
The present invention will be described in detail below with reference to the attached drawings, and the features of the present invention will be further apparent from the following detailed description.
FIG. 1 is a schematic block diagram of a license plate recognition system 100 according to one embodiment of the present invention. As shown in FIG. 1, the system 100 may include a vehicle semantic segmentation inference module 102, a vehicle automatic matting module 104, a business logic processing module 106, and a license plate recognition module 108.
The vehicle semantic segmentation inference module 102 is configured to perform semantic segmentation on a picture to be subjected to license plate recognition based on a pre-trained vehicle semantic segmentation model, and segment a background in the picture from a vehicle. The training process of the vehicle semantic segmentation model will be described in detail below in conjunction with fig. 2. As one example, the vehicle semantic segmentation inference module 102 may output a segmentation map of two classes, where the vehicle is one color and the background is another color (e.g., white).
The automatic vehicle matting module 104 is used for obtaining high-resolution vehicle matting according to the segmentation map and the original map output by the semantic segmentation inference module 102. As one example, the vehicle automatic cutout module 104 may resize the segmentation map to the resolution of the original image and then perform a masking operation with the original image to replace the portion marked as background with, for example, white, thereby obtaining the vehicle cutout.
The business logic processing module 106 is used for performing further logic processing on the vehicle cutout output by the vehicle automatic cutout module 104 so as to further improve the screen occupation ratio of the vehicle and the license plate. For example, if there is only one car in the car cutout output by the car auto cutout module 104, the car can be cut out and resized so that the car has a higher screen fraction in the cut picture. Similar operations can be performed for each vehicle if there are multiple vehicles in the vehicle cutout output by the vehicle automatic cutout module 104. If the vehicle is not available in the vehicle cutout, the processing is not needed, and the original image is directly used as an output result.
The license plate recognition module 108 is configured to perform license plate recognition on the result output by the service logic processing module 106. As one example, the license plate recognition module 108 may further include a license plate detection module 110 and a license plate recognition model 112. The license plate detection module can perform operations such as license plate positioning, affine transformation and alignment on the input picture, and the operations are used for preprocessing the picture to be subjected to license plate recognition so as to better adapt to the input requirement of the license plate recognition model 112, so that the recognition success rate and the accuracy rate are improved. The license plate recognition model 112 may be any of various existing license plate recognition models. As one example, the license plate recognition model 112 may be a neural network-based recognition model.
FIG. 2 is a schematic flow diagram of a method 200 of training a vehicle semantic segmentation model according to one embodiment of the invention. The method 200 begins at step 202 with a binary signature of a vehicle picture dataset. As an example, a picture dataset containing vehicles for training may be labeled with a semantic segmentation of two classes (i.e., ground route), such as labeling vehicles in a picture in one color and background in another color.
Subsequently, in step 204, the labeled training data set is input into a neural network for training of the semantic segmentation model to obtain the vehicle semantic segmentation model. As one example, the neural network may be constructed using a fully convolutional network, U-Net, SegNet, or any other suitable neural network structure. Fig. 3 shows a neural network structure diagram of a semantic segmentation model of a vehicle, which can be used in the present invention, and by means of this structure, the semantic segmentation model of the vehicle can be trained, and when a picture of the vehicle is inputted, the model can segment the vehicle from the background and output a two-class segmentation map.
FIG. 4 is a flow diagram of a license plate recognition method 400 according to one embodiment of the invention. The method begins at step 402 by performing semantic segmentation on a picture to be recognized to obtain a two-class segmentation map. The picture to be recognized may be an image of an environment containing a vehicle captured by an image capturing device such as a monitor probe, a camera of an access control system, or the like. As an example, a trained vehicle semantic segmentation model as mentioned above may be used, and after the picture to be recognized is input into the model, a two-classification segmentation map is output that distinguishes the vehicle from the background, where the vehicle is labeled in one color and the background is labeled in another color.
Subsequently, at step 404, a vehicle cutout process is performed using the binary segmentation map. As one example, the vehicle matting process may further include performing a binarization operation on the binary segmentation map (e.g., to 0 and 1), then adjusting the binary segmentation map to the original image size, and then performing a mask operation on the original image using the adjusted binarization map to replace a background portion in the original image with white, while a vehicle portion in the original image remains in the original color. The processing is similar to the process of extracting the vehicle from the original image, and the vehicle picture on the white background is obtained after the processing. It is understood that the background is replaced by white for example, and any other color that can facilitate distinguishing the background from the vehicle may be used.
In step 406, the matting processed picture is logically processed to improve the screen ratio of the vehicle and the license plate. In different shooting scenarios, the size of the vehicle in the picture (i.e., screen duty) may vary greatly. For example, in an access control system such as a garage, the captured picture usually has only one vehicle, and the proportion of the vehicle in the picture is very high. In contrast, in the image captured by the road surface monitoring camera, there are usually a plurality of vehicles, the size of each vehicle varies with the distance between the vehicle and the camera, and the screen occupation ratio of the vehicle at a far distance may be very small. Therefore, the screen occupation ratio of the vehicle and the license plate can be improved through processing, for example, whether the vehicle exists in the image subjected to the cutout processing is firstly detected, and when one or more vehicles are detected, each vehicle is cut out from the original image (for example, a rectangular frame) to obtain the image containing the vehicle, so that each vehicle has a high screen occupation ratio in the cut image. For example, the entire body of the detected vehicle is framed with a rectangular frame, leaving only as little space as possible around the body, so that the screen occupation ratio is as large as possible. If no vehicle is detected in the image subjected to the matting processing, for example, the image is completely blank, which indicates that no vehicle is in the image or the vehicle is incomplete, the original image can be directly used as an output result.
In step 408, license plate recognition is performed on the logically processed picture. The license plate recognition may be performed using any existing license plate recognition model or algorithm. As one example, license plate recognition may further include pre-processing the picture, including license plate positioning, affine transformation, alignment, etc., to better fit a license plate recognition model or algorithm to be used subsequently.
What has been described above includes examples of aspects of the claimed subject matter. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the claimed subject matter, but one of ordinary skill in the art may recognize that many further combinations and permutations of the claimed subject matter are possible. Accordingly, the disclosed subject matter is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims.