CN115565165A - Semantic segmentation and Transformer-based container number detection and identification method - Google Patents

Abstract

The invention provides a container number detection and identification method based on semantic segmentation and Transformer, which comprises the following steps: s1, detecting container numbers and constructing an identification data set; s2, constructing a box number detection network based on semantic segmentation and training; s3, performing text correction on the box number area obtained by detection; s4, constructing a container number recognition network and training; and S5, detecting and identifying the container number by using the trained detection and identification network. By the technical scheme, the container number area in the image can be efficiently and accurately segmented, three different types of container numbers are classified, and the corrected container number area image is obtained through TPS text correction.

Description

A Container Number Detection and Recognition Method Based on Semantic Segmentation and Transformer

technical field

The invention relates to the technical field of computer vision, in particular to a container number detection and recognition method based on semantic segmentation and Transformer.

Background technique

With the rapid development of international trade and social economy, my country's demand for logistics and transportation industry is increasing day by day. As an important container for cargo transportation, containers occupy a pivotal position in the entire transportation system. In order to realize the automation, informatization and intelligence of large-scale container transportation and management, it is very necessary to design an efficient and accurate container number identification system.

Traditionally, there are three commonly used container number detection methods based on edge detection, based on mathematical morphology and based on the maximum stable extremum region (MSER). These traditional methods are mainly based on artificial design features. After the image is acquired, the traditional method uses image grayscale to obtain the grayscale image of the image; then uses the directional gradient histogram and scale-invariant feature transformation to obtain image features; then searches through sliding windows or methods based on image region similarity To generate the box number position and box number character position information; in the recognition stage, use decision tree, support vector machine and template matching methods to identify the detected box number area. Due to the complex background of the picture, picture noise and other factors, the above-mentioned traditional image processing method inevitably has certain limitations in box number detection, and the detection speed is relatively low.

Contents of the invention

In view of this, the purpose of the present invention is to provide a container number detection and recognition method based on semantic segmentation and Transformer, which can efficiently and accurately segment the container number area in the image, and classify three different types of container numbers. TPS text correction obtains the corrected box number area image.

In order to achieve the above object, the present invention adopts the following technical solutions: a container number detection and recognition method based on semantic segmentation and Transformer, comprising the following steps:

Step S1: the construction of container number detection and identification data set;

Step S2: construct and train the box number detection network based on semantic segmentation;

Step S3: Carry out text correction in the box number area that detects;

Step S4: construct container number identification network and train;

Step S5: the process of using the trained detection and recognition network to detect and recognize the container number.

In a preferred embodiment, the step S1 is specifically:

Step S11: analyze the type of the container number to be detected and identified, and determine that the picture containing this type of information is a training picture;

Step S12: the container data set is collected at the port on the spot;

Step S13: Utilize LabeIme labeling software, label container number area, the square frame position information of number area, classification information and box number mark information are saved with json file form, obtain initial container number data set;

Step S14: Perform data enhancement processing on the initial container number data set to obtain the container number data set.

In a preferred embodiment, the types of container numbers to be detected and identified include horizontal, vertical, and double-line container numbers, wherein the horizontal and vertical container numbers are marked with 4 coordinates, and the double-line container numbers are marked with 6 coordinates.

In a preferred embodiment, the step S2 is specifically:

Step S21: using the Swin Transformer as the backbone network to obtain feature maps C1, C2, C3, and C4 of the size of the original image 1/4, 1/8, 1/16, and 1/32 pixels respectively;

Step S22: feature map C1, C2, C3, C4 obtain feature map F by FPN structure;

Step S23: feature map F predicts the head by semantic segmentation network, obtains the feature map of predicting three different box number types;

Step S24: utilize the prediction three maps that step S23 obtains, the loss function of design, calculate loss value;

Step S25: start training, save weight file after training finishes;

Step S26: Utilize step S25 to save the weight file to detect the picture to be recognized, and obtain the box number coordinate data based on the semantic segmentation detection network.

In a preferred embodiment, the calculation of the loss value is specifically as follows:

Loss=l _k , (k=1,2,3)

Among them, Loss represents the total loss, Loss is the sum of losses of 3 maps, l _k (k=1,2,3) represents the loss of predicting 3 maps of different box number types, k represents the kth map; y _k Indicates the true pixel segmentation map, f _k (x) represents the predicted box number segmentation map, y _ki represents the i-th pixel value of the true value heat map, f _ki (x) represents the i-th pixel value of the predicted heat map, n represents the total The pixel value of ; the predicted loss function of the three box number segmentation heatmaps is added to obtain the total loss function of the detection network.

In a preferred embodiment, the box number area correction method in step S3 is based on a thin plate spline interpolation method.

In a preferred embodiment, the container number recognition network training step in the step S4 is:

Step S41: Build a container number recognition network based on a multi-head attention mechanism;

Step S42: Carry out preprocessing operation to the input container box number picture and box number label;

Step S43: design the loss function of the network, calculate the loss and update the identification network parameters according to the loss;

Step S44: set the number of network training rounds, combine the box number label as supervision information, train the network to the end, save the network framework and network parameters of the trained container box number recognition part.

In a preferred embodiment, in the step S43, the container recognition network consists of 6 convolutional layers; 2 layers of multi-head attention mechanism network; 4 layers of maximum pooling layer; 3 fully connected layers.

In a preferred embodiment, in the step S43, the calculation of the loss value is specifically as follows:

Among them, l _rec represents the loss value of recognition, w is the character sequence data encoded by the box number label, |w| is the number of characters in the training target sequence, where |w|=11, w _j is the jth of the character prediction sequence data characters, y _j is the jth character of the character truth sequence data.

Compared with the prior art, the present invention has the following beneficial effects: the present invention designs a container number detection network based on semantic segmentation with SwinTransformer as the backbone network, and the detection network in this paper can efficiently and accurately segment the container number area in the image , and classify three different types of box numbers, and get the corrected box number area image through TPS text correction. In the identification stage of the container number, the network of this paper combines the convolutional neural network and the multi-head attention mechanism, and then designs a prediction network specifically for the container number sequence, which effectively solves the problems of blurred characters, broken characters, and hyphens in the container number area. question.

Description of drawings

Fig. 1 is a structural flow chart of a preferred embodiment of the present invention.

Fig. 2 is an effect diagram of part of the data set collected in step S1 in the preferred embodiment of the present invention.

Fig. 3 is an example of three different box number types in step S13 in the preferred embodiment of the present invention.

Fig. 4 is a network diagram of box number detection in step S2 in the preferred example of the present invention.

Fig. 5 is an example diagram of TPS box number correction in step S3 in the preferred embodiment of the present invention.

Fig. 6 is a network diagram of box number identification in step S4 in the preferred embodiment of the present invention.

Fig. 7 is a network structure for identifying CNN feature extraction in step S4 in a preferred embodiment of the present invention.

Fig. 8 is an example diagram of detection and identification of the overall box number in step S5 in the preferred embodiment of the present invention.

detailed description

The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

It should be pointed out that the following detailed description is exemplary and intended to provide further explanation to the present application. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terms used here are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include In addition, it should also be understood that when the terms "comprising" and/or "comprising" are used in this specification, it indicates the presence of features, steps, operations, means, components and/or combinations thereof.

Please refer to Figures 1-8, the present invention provides a container number detection and recognition method based on semantic segmentation and Transformer, including the following steps:

Step S1: the construction of container number detection and identification data set;

Step S2: construct and train the box number detection network based on semantic segmentation;

Step S3: Carry out text correction in the box number area that detects;

Step S4: construct container number identification network and train;

Step S5: the process of using the trained detection and recognition network to detect and recognize the container number.

In this embodiment, step S1 is specifically:

Step S11: analyze the type of the container number to be detected and identified, and determine that the picture containing this type of information is a training picture;

Step S12: as shown in Figure 2, the container data set is collected at the port on the spot;

Step S13: Utilize LabeIme labeling software, label container number area, the quadrangular frame position information of number area, classification information and case number mark information are saved with json file form, obtain initial container number data set; As shown in Figure 3, described container The types of numbers to be detected and identified include horizontal, vertical, and double-line container numbers, where the horizontal and vertical container numbers are marked with 4 coordinates, and the double-line container numbers are marked with 6 coordinates.

Step S14: Perform data enhancement processing on the image of the initial container number data set, and perform dictionary encoding processing on the annotation of the data set to obtain the container number data set.

In this embodiment, the step S2 is specifically:

Step S21: As shown in Figure 4, use the Swin Transformer as the backbone network to obtain feature maps C1, C2, C3, and C4 with pixel sizes of 1/4, 1/8, 1/16, and 1/32 of the original image, respectively.

Step S22: feature map C1, C2, C3, C4 obtain feature map F by FPN structure;

Step S23: feature map F predicts the head by semantic segmentation network, obtains the feature map of predicting three different box number types;

Step S24: utilize the prediction three maps that step S23 obtains, the loss function of design, calculate loss value;

Step S25: Calculation of loss value:

Loss=l _k, (k=1,2,3)

Among them, Loss represents the total loss, Loss is the sum of losses of 3 maps, l _k (k=1,2,3) represents the loss of predicting 3 maps of different box number types, k represents the kth map; y _k Indicates the true pixel segmentation map, f _k (x) represents the predicted box number segmentation map, y _ki represents the i-th pixel value of the true value heat map, f _ki (x) represents the i-th pixel value of the predicted heat map, n represents the total The pixel value of ; the predicted loss function of the three box number segmentation heatmaps is added to obtain the total loss function of the detection network.

Step S26: start training, save weight file after training finishes;

Step S27: Utilize step S25 to save the weight file to detect the picture to be recognized, and obtain the box number coordinate data based on the semantic segmentation detection network.

In this embodiment, step S3 is specifically:

Step S31: As shown in Figure 5, convert the quadrilateral frame obtained by precise positioning into an evenly spaced input control point C' in the input image, and now set a rectangular area with an appropriate length and width as the output image according to the coordinates of the detection frame, and set K an output control point;

Step S32: linearly project the control point C of the output image to obtain the control point C' in the input image, and the conversion relationship is shown in the following formula:

in:

φ(r)=r ² log(r)

T=[C'O ^2×3 ]ΔC ^-1

r=||c _i -c _k ||, represents the Euclidean distance between the output control point c _i and c _k , and the transformation matrix T is the target matrix to be obtained; the calculation of △C is shown in the following formula:

组成的矩阵。where C ⁺ is made by

composed matrix.

Step S33: For all pixel coordinates p' and p of the input image and the output image, the corresponding relationship also conforms to the TPS conversion relationship in formula 6, and the mapping relationship between p' and p is shown in the following formula:

Step S34: Calculate the corresponding two-dimensional coordinate p' of the two-dimensional coordinate p in the input image in the predicted image according to the above coordinate conversion formula, and the sampler calculates the predicted image by performing bilinear interpolation operations on adjacent pixels of p' The value of p, and finally get the correction result map of the scene text.

In this embodiment, step S4 is specifically:

Step S41: As shown in Figure 6, build a container number recognition network based on the multi-head attention mechanism;

Step S42: Carry out preprocessing operation to the input container box number picture and box number label;

Step S43: design the loss function of the network, calculate the loss and update the identification network parameters according to the loss;

Step S44: Calculate the recognition loss function:

Among them, l _rec represents the loss value of recognition, w is the character sequence data encoded by the box number label, |w| is the number of characters in the training target sequence, where |w|=11, w _j is the jth of the character prediction sequence data characters, y _j is the jth character of the character truth sequence data.

Step S45: set the number of network training rounds, combine the box number label as supervision information, train the network to the end, save the network framework and network parameters of the trained container box number recognition part.

In this embodiment, step S5 is specifically:

Step S51: the box number detection network and the box number identification network are spliced together;

Step S52: load the network weight data that training preserves;

Step S53: input the box number picture to be detected;

Step S54: as shown in Figure 8, obtain the recognition result of container number in the picture;

The above descriptions are only preferred embodiments of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope of the present invention.

Claims (9)

1. A container number detection and identification method based on semantic segmentation and Transformer is characterized by comprising the following steps:

s1, detecting container numbers and constructing an identification data set;

s2, constructing a box number detection network based on semantic segmentation and training;

s3, performing text correction on the box number area obtained by detection;

s4, constructing a container number recognition network and training;

and S5, detecting and identifying the container number by using the trained detection and identification network.

2. The semantic segmentation and Transformer-based container number detection and identification method according to claim 1, wherein the step S1 specifically comprises:

s11, analyzing the type of the container number to be detected and identified, and determining the picture containing the type information as a training picture;

s12, collecting a container data set at a field port;

step S13, labeling a container number area by using LabeIMe labeling software, and storing position information, classification information and container number labeling information of a quadrilateral frame in the number area in a json file form to obtain an initial container number data set;

and S14, performing data enhancement processing on the initial container number data set to obtain a container number data set.

3. The semantic segmentation and Transformer-based container number detection and identification method according to claim 2, wherein the types to be detected and identified by the container number include a horizontal container number, a vertical container number, and a double-row container number, wherein the horizontal container number and the vertical container number are labeled with 4 coordinates, and the double-row container number is 6 coordinates.

4. The semantic segmentation and Transformer-based container number detection and identification method according to claim 1, wherein the step S2 specifically comprises:

s21, using Swin transform as a backbone network to respectively obtain characteristic graphs C1, C2, C3 and C4 of original images with 1/4, 1/8, 1/16 and 1/32 pixel sizes;

s22, obtaining a characteristic diagram F through the FPN structure by the characteristic diagrams C1, C2, C3 and C4;

s23, predicting the head by the semantic segmentation network to obtain feature maps for predicting three different box number types by the feature map F;

s24, calculating a loss value by using the three maps obtained by predicting in the step S23 and a designed loss function;

s25, starting training, and saving a weight file after the training is finished;

and S26, detecting the picture to be identified by using the weight file stored in the step S25 to obtain the box number coordinate data of the detection network based on semantic segmentation.

5. The semantic segmentation and Transformer-based container number detection and identification method according to claim 4, wherein the loss value is calculated as follows:

Loss＝l _k, (k＝1,2,3)

where Loss denotes the total Loss, loss is the sum of the losses of 3 maps, l _k (k =1,2,3) denotes predicting the loss of 3 different bin number type maps, k denotes the kth map; y is _k Representing a true value pixel segmentation map, f _k (x) A division diagram representing the predicted case number, y _ki Indicating the ith pixel value, f, of a truth heat map _ki (x) Representing the ith pixel value of the predicted heat map and n representing the total pixel value; and adding the predicted 3 box number segmentation heat map loss functions to obtain the total loss function of the detection network.

6. The container number detection and identification method based on semantic segmentation and Transformer according to claim 1, wherein the box number region correction method in step S3 is a thin-plate spline interpolation-based method.

7. The semantic segmentation and Transformer-based container number detection and identification method according to claim 1, wherein the step of training the container number identification network in step S4 is as follows:

step S41: constructing a container number identification network based on a multi-head attention mechanism;

s42, preprocessing the input container number picture and the container number label;

s43, designing a loss function of the network, calculating loss and updating and identifying network parameters according to the loss;

and S44, setting the number of network training rounds, training the network to the end by taking the box number label as monitoring information, and storing the network frame and the network parameters of the trained container number identification part.

8. The semantic segmentation and Transformer-based container number detection and identification method according to claim 7, wherein in step S43, the container identification network consists of 6 convolutional layers; 2 layers of multi-head attention mechanism network; 4 maximum pooling layers; 3 full connection layers.

9. The method for detecting and identifying container number based on semantic segmentation and Transformer according to claim 7, wherein in the step S43, the loss value is calculated as follows:

wherein l _rec Representing the loss value of recognition, w is the character sequence data after box number marking and coding, | w | is the character number of the training target sequence, wherein | w | =11, w _j Predicting the jth character, y, of sequence data for a character _j Is the jth character of the truth-valued sequence data.

Images