CN111476843A - Chinese wolfberry branch recognition and positioning method based on attention mechanism and improved PV-RCNN network - Google Patents

Abstract

The invention relates to a wolfberry branch identification and positioning method based on an attention mechanism and an improved PV-RCNN network, and compared with the prior art, the method overcomes the defect that a two-dimensional image is difficult to accurately identify due to the fact that wolfberry branches are shielded, broken points, overlapped and the like. The invention comprises the following steps: collecting and preprocessing a training sample; carrying out voxelization processing on the three-dimensional point cloud; constructing a Chinese wolfberry branch and a key point detection network thereof; training medlar branches and key point detection networks thereof; collecting and preprocessing images of the medlar branches to be identified; and identifying and positioning the medlar branches. The method can supplement a large amount of positioning information lost by convolution operation in the voxelization and sparse 3D convolution network, and meanwhile, the accuracy of the detection of key points at the tail ends of the branches of the Chinese wolfberry is improved by obtaining the contribution degree of relevant points to target detection and characteristic enhancement in a refinement network according to the attention network, so that the accurate identification and positioning of the branches of the Chinese wolfberry are realized.

Description

Recognition and localization of wolfberry branches based on attention mechanism and improved PV-RCNN network method

technical field

The invention relates to the technical field of wolfberry harvesting, in particular to a wolfberry branch identification and positioning method based on an attention mechanism and an improved PV-RCNN network.

Background technique

With the continuous expansion of wolfberry planting area, the picking of wolfberry has become a bottleneck problem that restricts the sustainable development of wolfberry industry. Since there is no mature picking machinery in the domestic and foreign markets, the harvesting of wolfberry mainly relies on manual work, but the efficiency of manual harvesting of wolfberry is only 3-5kg/h, and the cost is more than 50% of the production cost. The development of wolfberry harvesting machinery adapted to my country's national conditions is of great significance for reducing the required costs, increasing farmers' income, and ensuring the steady and sustainable development of the wolfberry industry.

All kinds of wolfberry harvesting machines rely on the subjective judgment of the operator. Using the wolfberry harvesting and clamping device to clamp the wolfberry branches, swinging or brushing the wolfberry branches, the efficiency is relatively low. However, the number of wolfberry fruit is relatively large and the volume is small, and the fruit is blocked by leaves and branches during picking, which makes the accurate identification and positioning of wolfberry branches difficult to accurately identify under two-dimensional images.

If the computer recognition technology based on 3D point cloud data (3D point cloud data can obtain the specific spatial information such as the spatial dimension, distribution characteristics and 3D shape of the target) can be used to accurately identify and judge the position of wolfberry branches and the key points at the ends of the branches. Coordinates, according to the position and trend of the branches and the coordinates of the key points at the ends of the branches, the robotic arm is used to pick them up at a fixed point for high-efficiency picking of wolfberry, which can not only improve the picking efficiency and cleaning rate of wolfberry, but also minimize the damage to wolfberry and Protection of the tree from damage. However, the current detection of wolfberry branches is still mainly based on two-dimensional images, but the natural environment is complex and there are problems such as occlusion, breakpoints, and overlaps, so it is difficult to directly determine the position of branches by using two-dimensional images.

Therefore, how to improve the accuracy of wolfberry branch detection has become a key technical problem to be solved urgently.

SUMMARY OF THE INVENTION

The purpose of the present invention is to solve the defects in the prior art that the two-dimensional images are difficult to accurately identify due to the occlusion, breakpoints, overlapping and other reasons of the wolfberry branches, and to provide a wolfberry branch identification based on an attention mechanism and an improved PV-RCNN network. positioning method to solve the above problems.

In order to achieve the above object, technical scheme of the present invention is as follows:

The identification and positioning method of wolfberry branches based on attention mechanism and improved PV-RCNN network includes the following steps:

Collection and preprocessing of training samples: Obtain 20 images of wolfberry trees from different angles through a binocular camera, build a 3D model to obtain a 3D point cloud, label the 3D point cloud, and create a sphere with a radius of r for the key points at the end of the branches. ;

Voxelization of 3D point cloud: Voxelize the 3D point cloud through the VolexNet network to form multiple grids; among them, the size of the point cloud input area is (L, W, H), and the size of each grid is The size is (l,w,h), the total number of grids is (L/l,W/w,H/h), and the number of point clouds in each grid is set to 8;

Construction of wolfberry branches and their key points detection network: Based on the PV-RCNN network, a wolfberry branch and its key point detection network was constructed, and the wolfberry branches and key points were obtained by integrating the attention mechanism in the PV-RCNN of the wolfberry branch and its key point detection network. Refinement network for targeting;

Training wolfberry branch and its key point detection network: use training samples to train wolfberry branch and its key point detection network;

Collection and preprocessing of images of wolfberry branches to be identified: 20 images of the wolfberry tree to be identified from different angles captured by the binocular camera were obtained, and the three-dimensional point cloud to be identified was obtained by using the constructed three-dimensional model, and the three-dimensional point cloud to be identified was processed. voxelization;

Recognition and positioning of wolfberry branches: Input the processed three-dimensional point cloud data to be identified into the trained wolfberry branch and its key point detection network to obtain the positions of wolfberry branches and key points at the ends of the branches, so as to realize the identification and positioning of wolfberry branches.

The described construction of wolfberry branches and their key point detection network includes the following steps:

The detection network of wolfberry branches and their key points was constructed based on PV-RCNN network, and the input layer was set as: many grids after voxelization of the three-dimensional point cloud of wolfberry branches and a sphere with a radius of r at the key points at the end of the branches;

The feature extraction layer is set as: using a sparse 3D convolutional network to perform multi-scale layer-by-layer feature extraction on the input grid and a sphere whose key point radius is r; Perform point cloud feature extraction;

Constructing a refinement network integrating attention mechanism in the detection network PV-RCNN of wolfberry branches and their key points: based on the attention mechanism, a precise localization network of wolfberry branches and their key point target candidate boxes is constructed as a target regression refinement network;

The output layer of the wolfberry branch and its key point detection network based on the improved PV-RCNN network is set as the position of the wolfberry branch and the coordinates of the key point at the end of the branch.

The training of wolfberry branches and their key point detection network includes the following steps:

Input many grids and spheres with key point radius r into 3D sparse convolutional neural network for layer-by-layer feature extraction;

The sparse convolutional neural network consists of four layers of C1, C2, C3, C4, 3×3×3 3D sparse convolution, and feature extraction is performed layer by layer;

Convert the C4 feature map to a top-down feature map, and the size of the top-view feature map is

个anchorboxes，角度分别为0度、45度、135度，通过NMS非极大值抑制操作生成3Dproposal，最终获得3Dproposal对应的类别和坐标位置；Generated by the RPN network according to the feature map size

Anchorboxes with angles of 0 degrees, 45 degrees, and 135 degrees, respectively, generate 3D proposals through NMS non-maximum suppression operation, and finally obtain the corresponding category and coordinate position of 3D proposals;

Use the k relevant points selected by FPS to extract the feature of point cloud through the PointNet network based on the attention mechanism;

The training of the target regression refinement network: cascade the top-view feature corresponding to 3D proposals and k related point weight features F _i ′; then use the Fusion model to multiply the cascaded result and the attention feature generated by the 3D proposals convolution. Fusion; finally, the precise position of the refined bounding box 3Dbox is obtained through the multi-layer perceptron;

The training of the loss function is performed during the training process: the loss function includes the multi-task objective loss function L _RPN of the RPN and the regression box refinement loss function L _REFINE .

The feature extraction of the point cloud using the k relevant points selected by FPS and the PointNet network based on the attention mechanism includes the following steps:

Using the FPS algorithm to select k relevant points from the 3D point cloud, the formula is as follows:

κ={p ₁ , p ₂ ,...,p _k };

The feature representation of each relevant point _pi is as follows:

其中i＝1，2，3，...，k；

where i=1, 2, 3, ..., k;

为每一层3D稀疏卷积上产生的特征图，c＝1，2，3，4；in,

For the feature map generated on each layer of 3D sparse convolution, c=1, 2, 3, 4;

是三维点云通过SA模型计算的第i个相关点p_i特征；

is the i-th related point p _i feature calculated by the SA model of the 3D point cloud;

是对俯视图利用双线性插值获得的特征；

is the feature obtained by bilinear interpolation for the top view;

The weight of the feature F _i of the relevant point p _i is calculated as follows:

F′ _i =Λ(pi ) _{⊙Fi i =} 1, 2, 3,...,k;

Among them, Λ(·)∈[0,1] is the attention network, and its value represents the attention vector corresponding to the input related point, that is, the importance of the related point, and F _i is the feature of the related point _pi .

The training of the loss function in the training process includes the following steps:

Training multi-task target loss function L _RPN , the loss function includes classification task loss function L _cls ; target regression box loss function L _boxreg ; key point regression loss function L _keyreg :

When IoU>0.6, the anchor is considered as a positive sample; when IoU<0.45, the anchor is considered as a negative sample; its expression is as follows:

L _RPN = L _cls + L _boxreg + L _keyreg ;

The loss function L _cls of the classification task, its expression is as follows:

Among them, L _cls (x, y)=-(xlog(y)+(1-x)log(1-y)), N ₊ represents the number of positive samples, N _- represents the number of negative samples;

The target regression box loss function L _boxreg , whose expression is as follows:

Among them, let σ=2;

Train the keypoint regression loss function L _keyreg , whose expression is as follows:

是标记的关键点坐标，f(x_i)预测的关键点坐标；in,

is the marked keypoint coordinates, f( _xi ) predicted keypoint coordinates;

The _3Dbox regression box refinement loss function L REFINE including the branch target box and the keypoint target box is trained, and its expression is as follows:

是标记的目标框，3Dbox是预测目标框。in,

is the labeled target box, and 3Dbox is the predicted target box.

beneficial effect

Compared with the prior art, the method for identifying and locating wolfberry branches based on the attention mechanism and the improved PV-RCNN network of the present invention integrates the attention network of relevant points in the PV-RCNN network and the 3D proposals convolution feature in the refined network. The generated attention can supplement a large amount of localization information lost in the convolution operation in the voxelized and sparse 3D convolutional network. At the same time, according to the contribution of the relevant points to the target detection obtained by the attention network and the feature enhancement in the refined network. The detection accuracy of the key points of wolfberry branches and branch ends is improved, and the accurate identification and positioning of wolfberry branches is realized.

Description of drawings

FIG. 1 is a sequence diagram of the method of the present invention.

Detailed ways

In order to have a further understanding and understanding of the structural features of the present invention and the effects achieved, the preferred embodiments and accompanying drawings are used in conjunction with detailed descriptions, and the descriptions are as follows:

As shown in Figure 1, the method for identifying and locating wolfberry branches based on an attention mechanism and an improved PV-RCNN network according to the present invention includes the following steps:

The first step is the collection and preprocessing of training samples. The binocular camera was used to obtain 20 images of the wolfberry tree from different angles, and the existing technology was used to build a 3D model to obtain a 3D point cloud. The 3D point cloud will be preprocessed by voxelization to form a grid. We use a sphere with a radius of r as annotation just so that it can be input into the sparse 3D convolutional network as data in the same format as the grid for feature extraction.

The second step is the voxelization of the 3D point cloud. The 3D point cloud is voxelized through the VolexNet network to form multiple grids; the size of the point cloud input area is (L, W, H), and the size of each grid is (l, w, h) , the total number of grids is (L/l, W/w, H/h), and the number of point clouds in each grid is set to 8.

The third step is to construct the detection network of wolfberry branches and their key points. Based on the PV-RCNN network, the detection network of wolfberry branches and their key points was constructed, and the attention mechanism was integrated in the PV-RCNN of the detection network of wolfberry branches and their key points to obtain a refined network for the target location of wolfberry branches and key points. Here, the PV-RCNN network architecture is used to construct the detection network of wolfberry branches and their key points, and in order to realize the attention mechanism, a refinement network is established in the PV-RCNN network to integrate the attention mechanism. The specific steps are as follows:

(1) The detection network of wolfberry branches and their key points is constructed based on the PV-RCNN network, and the input layer is set as: many grids after voxelization of the three-dimensional point cloud of wolfberry branches and a sphere with a radius of r at the end of the branch.

(2) Set its feature extraction layer as: using a sparse 3D convolutional network to perform multi-scale layer-by-layer feature extraction on the input grid and a sphere whose key point radius is r; The PointNet network performs point cloud feature extraction.

(3) Constructing a refinement network integrating attention mechanism in the detection network PV-RCNN of wolfberry branches and their key points: Based on the attention mechanism, a precise localization network of wolfberry branches and their key point target candidate boxes is constructed as the target regression refinement network.

(4) Set the output layer of the wolfberry branch and its key point detection network based on the improved PV-RCNN network as the position of the wolfberry branch and the coordinates of the key point at the end of the branch.

The fourth step is to train the detection network of wolfberry branches and their key points: use the training samples to train the detection network of wolfberry branches and their key points. The specific steps are as follows:

(1) Input many grids and spheres with key point radius r into the 3D sparse convolutional neural network for layer-by-layer feature extraction;

The sparse convolutional neural network consists of four layers of C1, C2, C3, C4, 3×3×3 3D sparse convolution, and feature extraction is performed layer by layer;

Convert the C4 feature map to a top-down feature map, and the size of the top-view feature map is

个anchorboxes，角度分别为0度、45度、135度，通过NMS非极大值抑制操作生成3Dproposal，最终获得3Dproposal对应的类别和坐标位置。Generated by the RPN network according to the feature map size

Anchorboxes with angles of 0 degrees, 45 degrees, and 135 degrees, respectively, generate 3D proposals through the NMS non-maximum suppression operation, and finally obtain the corresponding categories and coordinate positions of 3D proposals.

(2) Use FPS to select k relevant points and perform feature extraction of point cloud through the P _o intNet network based on the attention mechanism. Since the three-dimensional point cloud of wolfberry branches will lose a lot of important information after being voxelized, k related points are selected from the point cloud for feature extraction to make up for the lost information as much as possible. Importance. In this way, the method of processing 3D point cloud for target detection can not only improve the training speed but also improve the accuracy of target detection. The specific steps are as follows:

A1) Use the FPS algorithm to select k relevant points from the 3D point cloud, and the formula is as follows:

κ={p ₁ , p ₂ , ..., p _k };

The feature representation of each relevant point _pi is as follows:

其中i＝1，2，3，...，k；

where i=1, 2, 3, ..., k;

勾每一层3D稀疏卷积上产生的特征图，c＝1，2，3，4；in,

Check the feature map generated on each layer of 3D sparse convolution, c=1, 2, 3, 4;

是三维点云通过SA模型计算的第i个相关点p_i特征；

is the i-th related point p _i feature calculated by the SA model of the 3D point cloud;

是对俯视图利用双线性插值获得的特征；

is the feature obtained by bilinear interpolation for the top view;

B2) Calculate the weight of the feature F _i of the relevant point p _i as follows:

F′ _i =Λ(pi ) _{⊙Fi i =} 1, 2, 3,...,k;

Among them, Λ(·)∈[0,1] is the attention network, and its value represents the attention vector corresponding to the input related point, that is, the importance of the related point, and F _i is the feature of the related point _pi .

(3) The training of the target regression refinement network and the voxelization of the point cloud will generally lose a certain amount of positioning information, thus affecting the accuracy of target detection, so the convolution feature and the relevant point weight feature are used as the input of the refinement network. , which can provide semantic supplementary information to each other; at the same time, adding attention mechanism can attach surrounding context information to each point, enhance features, and further optimize detection results. The steps are: cascade the top-view feature corresponding to 3D proposals and k related point weight features F′ _i ; at the same time, use the Fusion model to multiply the concatenated result and the attention feature generated by 3D proposals convolution to fuse; The refined bounding box 3Dbox precise location is obtained by a multilayer perceptron.

(4) Training the loss function in the training process: the loss function includes the multi-task target loss function L _RPN of the RPN and the regression box refinement loss function L _REFINE .

According to the importance of determining the grasping position of key points of wolfberry branches, we introduce key points as supervision data, so that after training a network for coarsely localizing wolfberry branches and their key points, we further refine the target frame of wolfberry branches and key point target boxes. The training of the network finally obtains the precise positioning of the target.

A1) Training multi-task target loss function L _RPN , the loss function includes classification task loss function L _cls ; target regression box loss function L _boxreg ; key point regression loss function L _keyreg :

When IoU>0.6, the anchor is considered as a positive sample; when IoU<0.45, the anchor is considered as a negative sample; when 0.45<IoU<0.6, it is difficult to judge the positive and negative samples of the anchor, so, when calculating the loss function will not be considered. Its expression is as follows:

L _RPN = L _cls + L _boxreg + L _keyreg ;

The loss function L _cls of the classification task, its expression is as follows:

Among them, L _cls (x, y)=-(xlog(y)+(1-x)log(1-y)), N ₊ represents the number of positive samples, N _- represents the number of negative samples;

The target regression box loss function L _boxreg , whose expression is as follows:

Among them, let σ=2;

Train the keypoint regression loss function L _keyreg , whose expression is as follows:

是标记的关键点坐标，f(x_i)预测的关键点坐标；in,

is the marked keypoint coordinates, f( _xi ) predicted keypoint coordinates;

A2) Train the _3Dbox regression box refinement loss function L REFINE including the branch target frame and the key point target frame, and its expression is as follows:

是标记的目标框，3Dbox是预测目标框。in,

is the labeled target box, and 3Dbox is the predicted target box.

The fifth step is the collection and preprocessing of the images of the wolfberry branches to be identified: 20 images of the wolfberry tree to be identified from different angles captured by the binocular camera are obtained, and a three-dimensional model is constructed to obtain the three-dimensional point cloud to be identified. The point cloud is voxelized.

The sixth step, identification and positioning of wolfberry branches: Input the processed three-dimensional point cloud data to be identified into the trained wolfberry branch and its key point detection network to obtain the positions of wolfberry branches and key points at the ends of the branches, and realize the identification and identification of wolfberry branches. position.

The foregoing has shown and described the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited by the above-mentioned embodiments. The above-mentioned embodiments and descriptions describe only the principles of the present invention. Without departing from the spirit and scope of the present invention, there are various Variations and improvements are intended to fall within the scope of the claimed invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims (5)

1. a method for identifying and positioning medlar branches based on attention mechanism and improving PV-RCNN network, is characterized in that, comprises the following steps: 11) Collection and preprocessing of training samples: 20 images of the wolfberry tree from different angles are obtained through a binocular camera, a 3D model is constructed to obtain a 3D point cloud, the 3D point cloud is annotated, and a sphere with a radius of r is established for the key points at the end of the branch. mark; 12) Voxelization of 3D point cloud: Voxelize the 3D point cloud through the VolexNet network to form multiple grids; among them, the size of the point cloud input area is (L, W, H), and each grid The size of the grid is (l, w, h), the total number of grids is (L/l, W/w, H/h), and the number of point clouds in each grid is set to 8; 13) Constructing the detection network of wolfberry branches and their key points: The detection network of wolfberry branches and their key points is constructed based on the PV-RCNN network, and the attention mechanism is fused in the PV-RCNN of the wolfberry branches and their key points detection network to obtain the wolfberry branches and their key points. Refinement network for key point targeting; 14) Training the detection network of wolfberry branches and their key points: using the training samples to train the detection network of wolfberry branches and their key points; 15) Collection and preprocessing of the images of the wolfberry branches to be identified: 20 images of the wolfberry tree to be identified from different angles captured by the binocular camera are obtained, the three-dimensional point cloud to be identified is obtained by using the constructed three-dimensional model, and the three-dimensional point cloud to be identified is obtained. The cloud is voxelized; 16) Identification and positioning of wolfberry branches: Input the processed three-dimensional point cloud data to be identified into the trained wolfberry branch and its key point detection network, and obtain the positions of wolfberry branches and key points at the ends of the branches, so as to realize the identification and positioning of wolfberry branches. 2. the method for identifying and locating wolfberry branches based on attention mechanism and improving PV-RCNN network according to claim 1, is characterized in that, described constructing wolfberry branches and key point detection network thereof comprises the following steps: 21) A detection network for wolfberry branches and their key points is constructed based on the PV-RCNN network, and the input layer is set as: many grids after voxelization of the three-dimensional point cloud of wolfberry branches and a sphere with a radius of r at the end of the branch; 22) Set its feature extraction layer as: using a sparse 3D convolutional network to perform multi-scale layer-by-layer feature extraction on the input grid and a sphere whose key point radius is r; PointNet network for feature extraction of point cloud; 23) Constructing a refinement network integrating attention mechanism in the detection network PV-RCNN of wolfberry branches and their key points: Based on the attention mechanism, an accurate localization network of wolfberry branches and their key point target candidate boxes is constructed as a target regression refinement network ; 24) Set the output layer of the wolfberry branch and its key point detection network based on the improved PV-RCNN network as the position of the wolfberry branch and the coordinates of the key point at the end of the branch. 3. the method for identifying and positioning wolfberry branches based on attention mechanism and improving PV-RCNN network according to claim 1, is characterized in that, described training wolfberry branches and key point detection network thereof comprises the following steps: 31) Input many grids and spheres with key point radius r into the 3D sparse convolutional neural network for layer-by-layer feature extraction; The sparse convolutional neural network consists of four layers of C1, C2, C3, C4, 3×3×3 3D sparse convolution, and feature extraction is performed layer by layer; Convert the C4 feature map to a top-down feature map, and the size of the top-view feature map is

Generated by the RPN network according to the feature map size

Anchorboxes, the angles are 0 degrees, 45 degrees, and 135 degrees, respectively, through the NMS non-maximum suppression operation to generate 3Dproposa], and finally obtain the corresponding category and coordinate position of 3Dproposal; 32) Use the k relevant points selected by FPS and perform feature extraction of the point cloud through the PointNet network based on the attention mechanism; 33) Training of the target regression refinement network: cascade the top-view feature corresponding to 3D proposals and k related point weight features F′ _i ; then use the Fusion model to match the cascaded results with the attention features generated by 3D proposals convolution. Multiply and fuse; finally obtain the precise position of the refined bounding box 3Dbox through the multi-layer perceptron; 34) The training of the loss function in the training process: the loss function includes the multi-task target loss function L _RPN of the RPN and the regression box refinement loss function L _REFINE . 4. according to claim 3 based on attention mechanism and improving the wolfberry branch identification and positioning method of PV-RCNN network, it is characterized in that, described utilizing FPS to select k relevant points and by the PointNet network based on attention mechanism Feature extraction from point cloud includes the following steps: 41) Use the FPS algorithm to select k relevant points from the 3D point cloud, and the formula is as follows: κ={p ₁ , p ₂ , ..., p _k }; The feature representation of each relevant point _pi is as follows:

where i=1, 2, 3, ..., k; in,

For the feature map generated on each layer of 3D sparse convolution, c=1, 2, 3, 4;

is the i-th related point p _i feature calculated by the SA model of the 3D point cloud;

is the feature obtained by bilinear interpolation for the top view; 42) Calculate the weight of the feature F _i of the relevant point p _i as follows: F′ _i =A(pi ) _{⊙Fi i =} 1, 2, 3,...,k; Among them, Λ(·)∈[0,1] is the attention network, and its value represents the attention vector corresponding to the input related point, that is, the importance of the related point, and F _i is the feature of the related point _pi . 5. the wolfberry branch identification and positioning method based on attention mechanism and improved PV-RCNN network according to claim 3, is characterized in that, the described training that carries out loss function in training process comprises the following steps: 51) Training multi-task target loss function L _RPN , the loss function includes classification task loss function L _cls ; target regression box loss function L _boxreg ; key point regression loss function L _keyreg : When IoU>0.6, the anchor is considered as a positive sample; when IoU<0.45, the anchor is considered as a negative sample; its expression is as follows: L _RPN = L _cls + L _boxreg + L _keyreg ; The loss function L _cls of the classification task, its expression is as follows:

Among them, L _cls (x, y)=-(xlog(y)+(1-x)log(1-y)), N ₊ represents the number of positive samples, and N represents the number of negative samples; The target regression box loss function L _boxreg , whose expression is as follows:

Among them, let σ=2; Train the keypoint regression loss function L _keyreg , whose expression is as follows:

in,

is the marked keypoint coordinates, f( _xi ) predicted keypoint coordinates; 52) Train the _3Dbox regression box refinement loss function L REFINE including the branch target frame and the key point target frame, and its expression is as follows:

in,

is the labeled target box, and 3Dbox is the predicted target box.

Images