TWI869931B - License plate recognition system and method thereof - Google Patents

Abstract

The present invention relates to a license plate recognition system and method thereof. The license plate recognition system comprises at least a server device storing at least one picture file of a license plate to be identified and a plurality of picture files of sample license plates. Each of the plurality of picture files of sample license plates has a double row of characters. Through deep learning training, the system generates a neural network model, inputs the at least one picture file of a license plate to be identified into the neural network model to output analysis result information, and decodes the analysis result information by a decoding algorithm to obtain a character content of an identified license plate.

Description

License plate recognition system and method

The present invention relates to a license plate recognition system and method, in particular to a recognition system and method capable of recognizing license plates with double rows of characters.

Generally, license plates have license plate numbers. Due to the limitation of license plate specifications, double-row license plates often occur. For the traditional method of recognizing double-row character license plates, it is generally necessary to locate the position of the characters first, and then use the OCR recognition module to recognize the characters at different positions. After that, the recognition results at different positions are concatenated to obtain the final recognition result.

However, traditionally, for this type of license plate with double rows of characters, it is often necessary to mark the upper and lower rows of characters separately and then identify them separately. This method is very time-consuming. However, it is difficult to identify the double rows of characters traditionally, and there is often a problem of reduced recognition. Therefore, if this problem can be overcome, it will be able to effectively reduce the time and cost of identification.

Therefore, this case uses a deeper neural network as the backbone. Through the training process, the network not only extracts character features from the image, but also rearranges the double-row image character features (single-row image character features are also acceptable) to a certain extent, so that its feature graph arranges the character features in the image in sequence. When entering the final output layer, it can be processed using a recognition method to obtain the recognized license plate character content. Therefore, this invention should be an optimal solution.

The license plate recognition system of the present invention comprises: at least one server device, which at least comprises: a license plate data storage module, which stores at least one license plate image file to be recognized and a plurality of license plate sample image files, wherein the license plate sample image file has a marking area, and the image content of the marking area is an image character feature, and the image character feature is a double-row character; a neural network learning module, which is connected to the license plate data storage module, and is used to store a plurality of license plate sample image files; The marked area of the image file is deep-learned to generate a neural network model; a license plate recognition module is connected to the license plate data storage module and the neural network learning module to input the license plate image file to be recognized into the neural network model to output an analysis result information; and a decoding output module is connected to the license plate recognition module to decode the analysis result information through a decoding algorithm to obtain a recognized license plate character content.

More specifically, the content of the recognized license plate characters corresponds to the license plate code on the license plate image file to be recognized, and the license plate code is a single-row character or a double-row character.

More specifically, the decoding algorithm is a greedy algorithm or a beam search algorithm.

More specifically, the neural network model has at least a plurality of convolutional layers and a deconvolutional layer, wherein the image content of the labeled area can be extracted into a feature map through a plurality of convolutional layers to rearrange the character features in the image content, and the deconvolutional layer is used to enlarge the feature map to increase the upper limit of the length of the recognized string, wherein the feature map corresponds to a plurality of time feature regions.

More specifically, the deconvolution layer is connected to a character feature extraction layer, which extracts a character feature matrix for each time feature region of the feature map according to multiple character types. The character feature matrix at least includes output channel quantity information, vertical feature information, and horizontal feature information, wherein the output channel quantity information is the number of character types, wherein the vertical feature information is the height of the time feature region, and wherein the horizontal feature information is the width of the time feature region.

More specifically, the character feature extraction layer is connected to an average dimension reduction layer, which is used to extract an average value of all vertical feature information of the character feature matrix to output a reduced-dimensional character feature matrix.

More specifically, the averaging dimension reduction layer is connected to an output layer, which processes the dimension reduction character feature matrix through a linked temporal classification method, and the decoding output module recognizes each time feature region as a character through the decoding algorithm, and removes continuous characters and blanks to obtain the recognized license plate character content.

A license plate recognition method, the steps of which are: (1) a server device stores at least one license plate image file to be recognized and multiple license plate sample image files, the license plate sample image file has a marked area, and the image content of the marked area is an image character feature, and the image character feature is a double-row character; (2) the server device is used to perform deep learning training on the marked areas of the multiple license plate sample image files to generate a neural network model; (3) the server device is used to input the license plate image file to be recognized into the neural network model to output an analysis result information; (4) the server device is used to decode the analysis result information through a decoding algorithm to obtain a recognized license plate character content.

More specifically, the neural network model has at least a plurality of convolutional layers, a deconvolutional layer, a character feature extraction layer, an average dimension reduction layer and an output layer. The plurality of convolutional layers extract the image content of the labeled area into a feature map having a plurality of temporal feature regions to rearrange the character features in the image. The deconvolutional layer is used to enlarge the feature map. The character feature extraction layer is used to average the feature map. A character feature matrix is extracted from each time feature region, and the average dimension reduction layer is used to reduce the dimension of the character feature matrix and output a reduced dimension character feature matrix. The output layer then processes the reduced dimension character feature matrix through a linked temporal classification method, and uses the decoding algorithm to identify each time feature region as a character, and removes continuous characters and blanks to obtain the recognized license plate character content.

1: Server equipment

11: Processor

12: Computer-readable recording media

121: Applications

1211: License plate data storage module

1212:Neural network learning module

1213: License plate recognition module

1214: Decoding output module

2: License plate body

21: Annotation area

[Figure 1] is a schematic diagram of the overall structure of the license plate recognition system and method of the present invention.

[Figure 2] is a schematic diagram of the application program architecture of the license plate recognition system and method of the present invention.

[Figure 3] is a schematic diagram of the image content of the license plate sample image file of the license plate recognition system and method of the present invention.

[Figure 4A] is a schematic diagram of the network architecture of the neural network model of the license plate recognition system and method of the present invention.

[Figure 4B] is a schematic diagram of the network architecture of the bottleneck layer of the neural network model of the license plate recognition system and method of the present invention.

[Figure 4C] is a schematic diagram of the network architecture of the character feature extraction layer of the neural network model of the license plate recognition system and method of the present invention.

[Figure 5] is a schematic diagram of the process of the license plate recognition system and method of the present invention.

Other technical contents, features and effects of the present invention will be clearly presented in the following detailed description of the preferred embodiment with reference to the drawings.

Please refer to Figure 1, which is a schematic diagram of the overall structure of the license plate recognition system and method of the present invention. As can be seen from the figure, the license plate recognition system includes a server device 1, which includes at least one processor 11 and at least one computer-readable recording medium 12. The computer-readable recording medium 12 stores at least one application 121, wherein the computer-readable recording medium 12 further stores computer-readable instructions. When the processors 11 execute the computer-readable instructions, the application 121 can be operated.

As shown in FIG. 2, the application 121 includes a license plate data storage module 1211, a neural network learning module 1212, a license plate recognition module 1213 and a decoding output module 1214.

The license plate data storage module 1211 stores at least one license plate image file to be recognized and multiple license plate sample image files (license plate sample image files are used as data sets for deep learning training). As shown in Figure 3, the license plate sample image file has a license plate body 2, and the license plate body 2 has a label area 21 on its surface. The image content of the label area 21 is an image character feature, and the image character feature is a double-row character or a single-row character. In this case, the license plate image with a single-row character and the license plate image with a double-row character are used for training.

The content of the recognized license plate characters corresponds to the license plate code on the image file of the license plate to be recognized. The license plate code is a single row of characters or a double row of characters, where each license plate contains at least 6 characters.

The neural network learning module 1212 is connected to the license plate data storage module 1211 to perform deep learning training on the labeled areas of multiple license plate sample image files to generate a neural network model. As shown in FIG. 4, in this embodiment, a neural network with resnet50 as the backbone is constructed to train the neural network.

When the license plate sample image file is input into the neural network learning module 1212, the neural network input size is 240 pixels wide and 96 pixels high, and the input dimension is 1x3x96x240. After being processed by the neural network, the network output layer matrix is obtained, and the output layer dimension is 1x36x30.

In this embodiment, the image character feature set is the numbers 0-9, English capital letters A-Z excluding I, O and dash (-), a total of 36 characters, but not limited to this, different types of characters can also be applied to the technical architecture of this case.

In this embodiment, resnet50 is used as the backbone (the backbone in this case is not limited to resnet50, and the lighter resnet34 can also be used), and the front N convolutional layers are retained as the image feature extractor (input->2D convolutional layer (conv2D)->batch normalization layer (BN)->rectified linear unit function (ReLU)->2D maximum pooling layer (MaxPooling2D)->bottleneck layer C=64->bottleneck layer C=128->bottleneck layer C=256), where *3, *4, *6 next to the dashed box represent that the block is repeated 3 times, 4 times, and 6 times.

The above-mentioned multiple convolutional layers can extract the image content of the marked area into a feature map, which is used to rearrange the character features in the image content.

As shown in Figure 4B, the bottleneck layer in the network architecture diagram (input->2D convolution layer (conv2D) convolution kernel 1x1 output channel C-> batch normalization layer (BN)-> rectified linear unit function (ReLU)->2D convolution layer (conv2D) convolution kernel 3x3 output channel C-> batch normalization layer (BN)-> rectified linear unit function (ReLU)->2D convolution layer (conv2D) convolution kernel 3x3 output channel C-> batch normalization layer (BN)-> rectified linear unit function (ReLU)->2D convolution layer (conv2D) Convolution layer (conv2D) convolution kernel 3x3 output channel Cx4-> batch normalization layer (BN)-> feature addition layer (Add)-> rectified linear unit function (ReLU)-> output layer (Output))(input (input)-> 2D convolution layer (conv2D) convolution kernel 1x1 output channel Cx4-> batch normalization layer (BN)-> feature addition layer (Add)).

Continuing from Figure 4A, we have the two-dimensional deconvolution layer (TransposConv2d), the character feature extraction layer, the average dimension reduction layer (ReduceMean), and the output layer (Output).

The deconvolution layer (two-dimensional deconvolution layer) is used to enlarge the feature map to increase the upper limit of the length of the recognized string, wherein the feature map corresponds to multiple time feature regions (timestep). In this embodiment, the timestep of the output layer is increased from 15 to 30. The output layer of the subsequent character feature extraction layer has more time feature regions (timestep), providing the connectionist temporal classification method (CTC) corresponding to the characters at each horizontal position in the feature map.

To explain further, the main purpose of the deconvolution layer is to increase the maximum string length that can be recognized in theory.

To explain further, in sequence processing models, timestep usually refers to the time feature region in the sequence (also called time step), and each time feature region corresponds to an element in the sequence. For example, if we have a sentence "I like to eat apples", each word can be regarded as a timestep, so this sentence has six timesteps.

Among them, Connectionist temporal classification (CTC) is a technology used to process sequence data, which is particularly suitable for processing situations where the length of input and output are inconsistent. Since the output of each timestep (each area of the feature map) corresponds to a possible character, CTC is used to determine which character each timestep should correspond to.

The deconvolution layer is connected to a character feature extraction layer, which extracts a character feature matrix for each time feature region of the feature map according to multiple character types. The character feature matrix at least includes output channel quantity information, vertical feature information, and horizontal feature information, wherein the output channel quantity information is the number of character types, the vertical feature information is the height of the time feature region, and the horizontal feature information is the width of the time feature region.

As shown in Figure 4C, it is the character feature extraction layer in the network architecture diagram (input->two-dimensional convolution layer (conv2D) convolution kernel 13x1 output channel 1024->rectified linear unit function (ReLU)->two-dimensional convolution layer (conv2D) convolution kernel 1x30 output channel 36->rectified linear unit function (ReLU)->output layer (Output)). As can be seen from the figure, this embodiment uses a 13x1 convolution kernel and a 1x30 convolution kernel to extract relevant information in the vertical and horizontal directions respectively. The number of output channels is 36, corresponding to 35 possible characters plus blank, and blank is CTC loss. Special characters required for function training, and the output matrix dimension is 1x36x13x30.

The 13x1 convolution kernel is mainly used to capture the vertical features of the image, while the 1x30 convolution kernel is used to capture the horizontal features.

The number of output channels is 36, which means that the output has 36 independent feature maps, each of which is a different representation of the original input information. In this case, these 36 channels correspond to 35 possible characters plus a special character representing blank. The blank character is required in the CTC loss function, and it is used to represent the separation between different characters.

The number of output channels is 36 because this embodiment uses 35 possible characters and a special blank symbol. These 35 possible characters usually include 26 English letters and 9 numbers (or may be different depending on the actual license plate system).

The output matrix dimension is 1x36x13x30. The first dimension (1) is the batch size, which represents the number of images processed simultaneously. In this case, we process one image at a time.

The second dimension (36) is the number of channels, which, as mentioned above, corresponds to 35 possible characters and one whitespace character.

The third and fourth dimensions (13 and 30) represent the height and width of the feature map respectively. This means that we have 13 different positions in the vertical direction and 30 different positions in the horizontal direction, so we have a total of 13x30=390 positions, and each position has a 36-dimensional vector to represent the information of that position.

The character feature extraction layer is connected to the average dimension reduction layer (ReduceMean), which is used to extract an average value of all vertical feature information (height) of the character feature matrix to output a reduced dimension character feature matrix.

To further explain, the average dimension reduction layer (ReduceMean) averages the output of the previous layer for the third dimension (height) to reduce the dimension. As the final output layer of the network, the output matrix dimension is 1x36x30, where 36 corresponds to the possible character categories and 30 corresponds to the time feature region (timestep) of the feature map.

The average dimension reduction layer is connected to an output layer, which processes the dimension reduction character feature matrix through a link temporal classification method. The output layer represents the results of the neural network operation, and then needs to be converted into corresponding strings through decoding processing.

The loss function in this embodiment uses the connectionist temporal classification method (CTC), and the network is trained through the Adam optimization method, where the initial learning rate is 0.0001, the learning rate uses exponential decay, the decay rate is 0.5 every 20 epochs, the batch size is 128 pictures each time, and the validation data set is used every 2 epochs. The total training is 100 epochs, and the weight with the highest validation accuracy during the training process is saved. This weight is the final model weight after training.

As shown in Table 1 below, a 4 x T matrix is used as an example of the analysis result information (each timestep feature value) output by the output layer, and further explains how the output matrix (36 x 30) of the neural network is decoded into the final output string. In this example, the vertical axis is each character, the horizontal axis is timesteps, the possible character types (A~C+blank) are 4, the number of timesteps is T, ε represents blank characters, and the numbers in the matrix represent probabilities, where each timestep corresponds to a row, and the sum of each row is 1 (the sum of all possible probabilities is 1); Table 1 is the matrix value (analysis result information) through the network output layer, but it has not yet been converted into the final string, so decoding processing is still required to convert these results into corresponding strings.

The license plate recognition module 1213 is connected to the license plate data storage module 1211 and the neural network model to input the license plate image file to be recognized into the neural network model to output analysis result information (output each timestep feature value).

The decoding output module 1214 is connected to the license plate recognition module 1213 to decode the analysis result information through a decoding algorithm to obtain the content of the recognized license plate characters. The decoding algorithm is a series of operations based on specific rules, such as a greedy algorithm or a beam search algorithm.

The decoding output module 1214 recognizes each time feature region output by the output layer as a character through the decoding algorithm, and removes consecutive characters and blanks to obtain the recognized license plate character content.

To explain further, the decoding algorithm processes each time feature region (timestep) in sequence, calculates the character corresponding to the maximum value, and represents the character recognized by the time feature region (timestep). After processing each time feature region (timestep), the respective characters are concatenated, and the continuous characters and blanks are removed to obtain the final output result.

The decoding technology of this case is explained with reference to the example in Table 1. First, in the first step, the character with the highest probability is taken out at each timestep. The highest probability of the first timestep is 0.7, and the corresponding character is A. The character with the highest probability of the second timestep is ε , and so on. The characters with the highest probability of all timesteps are recorded. In the second step, repeated characters in consecutive identical characters are removed, for example, AAA is removed and corrected to "A", and AABBBCC is corrected to "ABC". In the third step, blank characters are removed to obtain the final result.

In the example of Table 1, for the first three timesteps, the string obtained in the first step is A ε C, and the final result is "AC".

The first step of directly taking the highest probability is a feature of the Greedy algorithm, while the second and third steps are in accordance with the CTC rules, which is the same as Beam Search.

In addition, if the string obtained in the first step is AεCCCεBB, the second step removes the repeated characters in the consecutive identical characters (AεCCCεBB=>AεCεB), and the third step removes the blank characters to obtain the final result (ACB).

In this experimental example, 13,630 license plate images were used to train the neural network, and 8,800 license plate images were used for testing, including 2,300 double-row license plate images. The accuracy of single-row license plate images was 98.04%, and the accuracy of double-row license plate images was 94.69%.

The license plate recognition method of this case is shown in Figure 5, and its steps are: (1) A server device stores at least one license plate image file to be recognized and multiple license plate sample image files, and the license plate sample image file has a marking area, and the image content of the marking area is an image character feature, and the image character feature is a double-row character 501; (2) The server device is used to convert multiple (1) The server device is used to perform deep learning training on the labeled area of the license plate sample image file to generate a neural network model 502; (2) The server device is used to input the license plate image file to be recognized into the neural network model to output an analysis result information 503; (3) The server device is used to decode the analysis result information through a decoding algorithm to obtain a recognized license plate character content 504.

The license plate recognition system and method provided by the present invention have the following advantages when compared with other commonly used technologies:

(1) The present invention uses a relatively deep neural network as the backbone. Through the training process, the network not only extracts character features from the image, but also rearranges the single-row and double-row image character features to a certain extent, so that its feature map arranges the character features in the image in sequence. When entering the final output layer, it can be processed using the general method for single-line text recognition.

(2) The neural network used in the present invention can be applied not only to single-row license plates, but also has a high degree of recognition capability for double-row license plates.

(3) Compared with the traditional method of identifying double-row license plates, the present invention does not need to identify the upper and lower rows separately, but can identify both rows at the same time, which will effectively reduce the time and cost of identification.

The present invention has been disclosed through the above-mentioned embodiments, but they are not used to limit the present invention. Anyone familiar with this technical field and having common knowledge can make some changes and modifications without departing from the spirit and scope of the present invention after understanding the above-mentioned technical features and embodiments of the present invention. Therefore, the scope of patent protection of the present invention shall be subject to the definition of the claim attached to this specification.

121: Applications

1211: License plate data storage module

1212:Neural network learning module

1213: License plate recognition module

1214: Decoding output module

Claims (6)

A license plate recognition system includes: at least one server device, which includes at least: a license plate data storage module, which stores at least one license plate image file to be recognized and multiple license plate sample image files, the license plate sample image file has a label area, and the image content of the label area is an image character feature, and the image character feature is a double-row character or a single-row character; a neural network learning module, which is connected to the license plate data storage module, and is used to store the label area of the multiple license plate sample image files. A deep learning training is performed in the domain to generate a neural network model; a license plate recognition module is connected to the license plate data storage module and the neural network learning module, and is used to input the license plate image file to be recognized into the neural network model to output an analysis result information; and a decoding output module is connected to the license plate recognition module, and is used to decode the analysis result information through a decoding algorithm to obtain a recognized license plate character content, wherein the decoding algorithm is a greedy algorithm (Greedy algorithm) or a beam search algorithm (Beam Search); wherein the neural network model has at least a plurality of convolutional layers, a deconvolutional layer, a character feature extraction layer, an average dimension reduction layer and an output layer, wherein the plurality of convolutional layers extract the image content of the marked area into a feature map having a plurality of time feature regions to rearrange the character features in the image content, the deconvolutional layer is used to enlarge the feature map, and the character feature extraction layer is used to extract a character from each time feature region of the feature map. The average dimension reduction layer is used to reduce the dimension of the character feature matrix and output a reduced dimension character feature matrix. The output layer then processes the reduced dimension character feature matrix through a linked temporal classification method to obtain the analysis result information, which is the feature value of each time feature region. Each time feature region is then identified as a character through the decoding algorithm, and continuous characters and blanks are removed to obtain the recognized license plate character content. A license plate recognition system as described in claim 1, wherein the recognized license plate character content corresponds to the license plate code on the license plate image file to be recognized, and the license plate code is a single-row character or a double-row character. A license plate recognition system as described in claim 1, wherein the deconvolution layer enlarges the feature map to increase the upper limit of the length of the recognition string. The license plate recognition system as described in claim 1, wherein the deconvolution layer is connected to the character feature extraction layer, and the character feature extraction layer extracts the character feature matrix for each time feature region of the feature map according to multiple character types, and the character feature matrix at least includes an output channel quantity information, a vertical direction feature information, and a horizontal direction feature information, wherein the output channel quantity information is a character type quantity, wherein the vertical direction feature information is the height of the time feature region, and wherein the horizontal direction feature information is the width of the time feature region. The license plate recognition system as described in claim 1, wherein the character feature extraction layer is connected to the averaging dimensionality reduction layer, and the averaging dimensionality reduction layer is used to extract an average value of all vertical feature information of the character feature matrix to output the reduced dimensionality character feature matrix. A license plate recognition method, the steps of which are: a server device stores at least one license plate image file to be recognized and a plurality of license plate sample image files, the license plate sample image file has a marking area, and the image content of the marking area is an image character feature, and the image character feature is a double row of characters; the server device is used to store the marking area of the plurality of license plate sample image files Perform deep learning training to generate a neural network model; the server device is used to input the license plate image file to be recognized into the neural network model to output an analysis result information; the server device is used to decode the analysis result information through a decoding algorithm to obtain a recognized license plate character content, wherein the decoding algorithm is a greedy algorithm (Greedy algorithm) or a beam search algorithm (Beam Search); wherein the neural network model has at least a plurality of convolutional layers, a deconvolutional layer, a character feature extraction layer, an average dimension reduction layer and an output layer, wherein the plurality of convolutional layers extract the image content of the labeled area into a feature map having a plurality of time feature regions to rearrange the character features in the image content, the deconvolutional layer is used to enlarge the feature map, and the character feature extraction layer is used to extract a character feature from each time feature region of the feature map. The character feature matrix is obtained by the average dimension reduction layer, and the averaging dimension reduction layer is used to reduce the dimension of the character feature matrix and output a reduced dimension character feature matrix. The output layer then processes the reduced dimension character feature matrix through a linked temporal classification method to obtain the analysis result information, which is the feature value of each time feature region. The decoding algorithm then identifies each time feature region as a character, and removes continuous characters and blanks to obtain the recognized license plate character content.

Images