CN106570903A - Visual identification and positioning method based on RGB-D camera - Google Patents

Abstract

The present invention provides a kind of visual recognition and positioning method based on RGB-D camera, comprising the following steps: 1), by using Microsoft Kinect camera sensor to carry out the acquisition of color image and depth image and convert them into three-dimensional point cloud, and in the scene 2), after extracting the plane in step 1), perform object extraction and segmentation on the remaining point cloud; 3), identify the point cloud collection of the object obtained in step 2) respectively and matching; 4), the object point cloud obtained in step 2) is calculated to realize the positioning of the object. This method is based on the 3D point cloud images collected by Microsoft's RGB-D sensor Kinect II for object recognition and positioning, and does not involve complex operations such as matching between multiple images when locating objects, which greatly improves computing efficiency and is real-time Strong, suitable for the advantages of complex daily life environment.

Description

A visual recognition and positioning method based on RGB-D camera

technical field

The invention relates to the field of machine vision recognition and positioning, in particular to a visual recognition and positioning method based on an RGB-D camera.

Background technique

At present, most of the existing object recognition and positioning systems based on multi-eye color image cameras obtain the position of each pixel in space by stereo matching the images collected by different sensors, which are costly and slow. , system complexity and other issues.

Object edge segmentation is mostly realized by the method of convex hull extraction based on the image of the color camera. This processing method needs to consider the appearance color of the object, and it is easy to cause misjudgment when the background color is similar to the object, and the method of convex hull extraction is also difficult. There are problems with the wrong convex hull outline of the object and the inclusion of the background part.

Compared with the existing methods of object recognition and positioning based on multi-eye color image cameras, the method of using RGB-D sensors for object recognition and positioning has many advantages:

First of all, the amount of calculation is small, the calculation speed is fast, the real-time performance is strong, and the cost of object positioning is low. The RGB-D sensor Kinect II launched by Microsoft reduces the cost of 3D scanning and directly provides users with high-resolution color images and depth images. And point cloud images, the position of each pixel point in the camera coordinate system can be obtained directly through only one RGB-D sensor, without the need to obtain the spatial position of each pixel point by stereo matching the images collected by different sensors in the multi-eye system position in

Secondly, the accuracy and robustness have been improved. Based on the depth image and point cloud image provided by the RGB-D camera, plane extraction and object segmentation and positioning can be directly performed, effectively avoiding the influence of the appearance of the object itself and the background color. The occurrence of misjudgment can be reduced, and the accuracy and stability of the system can be improved.

Contents of the invention

The purpose of the present invention is to provide a visual recognition and positioning method based on an RGB-D camera, which has a small amount of calculation, strong real-time performance, and can adapt to daily life scenes.

The purpose of the present invention can be achieved through the following technical solutions:

A visual recognition and positioning method based on an RGB-D camera, said method comprising the following steps:

1) The object is converted into a three-dimensional point cloud image after the color image and depth image are collected by the Microsoft Kinect camera sensor;

2) Carry out corresponding normal vector calculation to each point of the three-dimensional point cloud image that step 1) obtains;

3) For the set of normal vectors obtained in step 2), use the region growing algorithm to extract the background plane where the object is placed;

4) removing the point of the background plane extracted in step 3), and performing object point cloud set extraction and convex hull extraction processing on the remaining point cloud;

5) Combine the point cloud collection of each object extracted in step 4) with the corresponding convex hull, perform secondary region growth, and realize the segmentation of the complete contour of each object and the extraction of the complete point set;

6) According to the complete outline of each object obtained in step 5), extract the corresponding color image and perform feature extraction and matching recognition respectively;

7) The point cloud set in the complete outline of each object obtained in step 5) is subjected to an average calculation to obtain the position information of each object in the camera coordinate system;

8) Transform the position information of each object in the camera coordinate system obtained in step 7) into the world coordinate system to realize the positioning of each object.

Preferably, in step 2), the method for calculating the normal vector is:

Let P _k be the point where you want to obtain the surface normal vector, first find the four points P ₁ , P ₂ , P ₃ and P ₄ near the point P _k in the image, P ₁ and P ₃ form a vector P ₂ and P ₄ make up the vector Then the surface normal vector of point P _k able to pass cross product get, as follows:

The normal vector of each point of the 3D point cloud image is calculated by the above formula.

Preferably, in step 3), first sequentially scan the normal vector of each point of the three-dimensional point cloud image, and then continue to search for a point near the point where the normal vector is approximately vertical when encountering an approximately vertical normal vector, and add it to the potential Plane point set, if the number of points in the potential plane point set is greater than the set threshold, the potential plane point set is considered to be a plane point set and the potential plane point set is added to the plane set, otherwise continue to scan the remaining normal vectors , until the scanning is finished, the plane set can be obtained to realize the plane extraction.

Preferably, in step 5), the segmentation of the complete contour of the object and the extraction method of the complete point set include the following steps:

a) Input a 3D point cloud, a set of planar points and points within the range of the planar convex hull;

b) Sequentially scan the points within the range of the convex hull of the plane to find points that belong to the range of the convex hull but do not belong to the plane;

c) After the non-planar point within the convex hull range found in step b), continue to search for all non-planar points within the convex hull near this point, and add them to the potential object point set;

d) If the number of potential object point concentration points in step c) is less than the set threshold, then return to step b) and continue to scan the remaining plane convex hull points;

e) If the number of potential object point collection points in step c) is greater than or equal to the set threshold, then the potential object point collection is considered to be an object point collection;

f) Continue to search for points on the boundary of the convex hull near the potential object point set in step e) to add to the potential object point set, that is, to find object points outside the convex hull and add them to the potential object point set;

g) adding the potential object point set obtained in step f) to the object set;

h) If there are still points in the point cloud that have not been scanned, go back to step b) and continue to search for new object point sets;

i) If all the points in the point cloud have been scanned, the algorithm ends to obtain the object set.

Preferably, step 6) is specifically:

First, obtain the corresponding area of the object in the color image through the 3D point cloud collection of the object, intercept the image area where the object is located, and use the Feature Detector::detect() function of Open CV to obtain the surf feature point;

Secondly, further use the Feature2D::compute() function of Open CV to obtain the surf feature vector. The surf feature vector of the object is added to the recognition library as the recognition feature of the object, or directly used as the feature vector of the object recognition and matching. When matching the surf eigenvectors with the corresponding surf eigenvectors in the library, use the nearest neighbor open source library FLANN to match the eigenvectors;

Finally, use the nearest neighbor open source library FLANN to match the feature vectors, specifically using the Descriptor Matcher::match() function of Open CV.

Compared with the prior art, the present invention has the following advantages and beneficial effects:

1. The present invention uses an RGB-D sensor. Compared with the existing object recognition and positioning system based on multi-eye color image cameras, it does not involve complex operations such as matching between multiple images when locating objects, and has a small amount of calculation. High efficiency, fast computing speed, strong real-time performance, low object positioning cost, high accuracy, strong robustness and other advantages can realize accurate, fast and stable object recognition and positioning;

2. The present invention uses the region growing algorithm to extract the background plane in the three-dimensional point cloud. It has the characteristics of small calculation and accurate segmentation, and can realize the rapid separation and extraction of the background plane, and improves the precision of object extraction, positioning and recognition. possibility;

3. The present invention combines the point cloud collection of each object with the corresponding convex hull, uses the secondary region growing algorithm, eliminates the points belonging to the background part in the convex hull of the object, and increases the points belonging to the object part outside the convex hull, realizing the object The complete extraction of contours improves the accuracy of object positioning.

Description of drawings

Fig. 1 is the flowchart of the method of the present invention.

Fig. 2 is a schematic diagram of the cross product of the normal vector of the surface of the present invention.

Fig. 3 is a flow chart of the planar region growing algorithm of the present invention.

Fig. 4 is a flow chart of the segmentation of the complete contour of the object and the extraction of the complete point set in the present invention.

Figure 5(a) is a schematic diagram of the Kinect camera coordinate system, and Figure 5(b) is a schematic diagram of the actual world coordinate system.

detailed description

The present invention will be further described in detail below in conjunction with the embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.

Example:

This embodiment provides a visual recognition and positioning method based on an RGB-D camera. As shown in FIG. Composition, described method specifically comprises the following steps:

Step 1, convert the color image and depth image of the object into a three-dimensional point cloud image after being collected by the Microsoft Kinect camera sensor;

In this step, Microsoft's Kinect sensor can collect RGB-d images, through the built-in API function or third-party function libraries such as Open Natural Interaction (Open NI), Point Cloud Library (Point Cloud Library, PCL), The 3D point cloud image can be obtained.

Step 2, performing corresponding normal vector calculation for each point of the 3D point cloud image obtained in step 1;

As shown in Figure 2, it is a schematic diagram of the cross product of the normal vector of the surface of the present invention, P _k is the point where the surface normal vector is to be obtained, first find the four points P ₁ , P ₂ , P near the top, bottom, left, and right of the point P _k in the image ₃ and P ₄ , P ₁ and P ₃ form a vector P ₂ and P ₄ make up the vector Then the surface normal vector of point P _k able to pass cross product get, as follows:

The normal vector of each point of the 3D point cloud image can be calculated by the above formula.

Step 3. For the set of normal vectors obtained in step 2, use the region growing algorithm to extract the background plane where the object is placed;

In this step, as shown in Figure 3, first scan the normal vector of each point of the 3D point cloud image sequentially, and then continue to search for a point near the point where the normal vector is approximately vertical when encountering an approximately vertical normal vector, and add In the potential plane point set S, if the number N _s of points in the potential plane point set S is greater than the set threshold N, the potential plane point set S is considered to be a plane point set and the potential plane point set S is added to the plane Otherwise, continue to scan the remaining normal vectors until the end of the scan, then you can get the plane set C to realize the plane extraction.

Step 4, remove the points of the background plane extracted in step 3, and perform object point cloud set extraction and convex hull extraction processing on the remaining point cloud;

Step 5. Combining the point cloud collection of each object extracted in step 4 with the corresponding convex hull, performing secondary region growth, and realizing the segmentation of the complete outline of each object and the extraction of the complete point set;

As shown in Figure 4, it is the flow chart of the segmentation of the complete contour of the object and the extraction of the complete point set. After performing the convex hull operation on the plane set obtained in step 3, the obtained plane convex hull contains both the plane point set and the Contains a collection of object points. In order to realize the complete extraction of object contours and improve the accuracy of object positioning, the present invention combines the point cloud collection of each object with the corresponding convex hull, and uses the secondary region growing algorithm to eliminate the points belonging to the background part in the convex hull of the object. Adding the points belonging to the object part outside the convex hull, the segmentation of the complete outline of the object and the extraction method of the complete point set include the following steps:

a) Input a 3D point cloud, a set of planar points and points within the range of the planar convex hull;

b) Sequentially scan the points within the range of the convex hull of the plane to find points that belong to the range of the convex hull but do not belong to the plane;

c) After the non-planar point within the convex hull range found in step b), continue to search for all non-planar points within the convex hull near this point, and add them to the potential object point set S';

d) If the number N _s of points in the potential object point set S' in step c) is less than the set threshold N, then return to step b) and continue to scan the remaining planar convex hull points;

e) If the number N _s of points in the potential object point set S' in step c) is greater than or equal to the set threshold N, then the potential object point set S' is considered to be an object point set;

f) Continue to find points on the boundary of the convex hull near the potential object point set S' in step e) and add them to the potential object point set S', that is, find the object points outside the convex hull and add them to the potential object points set S';

g) adding the potential object point set S' obtained in step f) to the object set C';

h) If there are still points in the point cloud that have not been scanned, go back to step b) and continue to search for new object point sets;

i) If all the points in the point cloud have been scanned, the algorithm ends to obtain the object set C'.

Step 6. According to the complete outline of each object obtained in step 5, extract the corresponding color image and perform feature extraction and matching recognition respectively;

In this step, firstly, the corresponding area of the object in the color image can be obtained through the 3D point cloud collection of the object, and the image area where the object is located is intercepted and then the surf feature point is obtained by using the Feature Detector::detect() function of Open CV. Further use the Feature2D::compute() function of Open CV to obtain the surf feature vector. The surf feature vector of the object can be added to the recognition library as the recognition feature of the object, or directly used as the feature vector of the object recognition and matching. When matching the surf feature vector with the corresponding surf feature vector in the library, use the nearest neighbor open source library FLANN to match the feature vector, and finally, use the nearest neighbor open source library FLANN to match the feature vector, specifically using Open CV's Descriptor Matcher: :match() function implementation.

Step 7. Perform an averaging operation on the point cloud set in the complete outline of each object obtained in step 5 to obtain the position information of each object in the camera coordinate system;

Step 8: Transform the position information of each object in the camera coordinate system obtained in step 7 into the world coordinate system to realize the positioning of each object.

Figure 5(a) and Figure 5(b) are the schematic diagrams of the Kinect camera coordinate system and the actual world coordinate system respectively. Both coordinate systems are right-handed coordinate systems. To realize the positioning of objects in the world coordinate system, it is necessary to The camera is calibrated, and there is the following relationship between the camera coordinate system and the world coordinate system:

Among them, X _C , Y _C , Z _c represent the position components of the object in the camera coordinate system, X _W , Y _W , Z _W represent the position components of the object in the world coordinate system, with They are the camera's rotation matrix and offset matrix, both of which are external parameters of the camera.

In the process of coordinate system transformation, the rotation matrix needs to be passed and offset matrix To calculate the world coordinate system of the object, and these two matrices are obtained through camera calibration. When performing camera calibration, the Matlab Camera Calibration Toolbox (Camera Calibration Toolbox for Matlab) is used to calibrate Kinect through the chessboard method. Since the default camera coordinate system of Kinect is set at the infrared camera, as shown in Figure 5, and the pictures collected by Kinect are mirror images, the infrared camera of Kinect should be used to collect infrared images when performing camera calibration, and the collected The images were mirrored left and right one by one before they were input into the toolbox for calibration. When calibrating, let the camera collect more than 20 pictures around the chessboard at different angles and different distances to calculate the internal parameters of the camera, and finally fix the chessboard at the target position to collect an image to calculate the external parameters of the camera.

Perform the following operations on the external parameter matrix obtained after calibration and the calculated coordinates of the object in the camera coordinate system:

The spatial coordinate information of the object in the world coordinate system can be obtained.

The above is only a preferred embodiment of the patent of the present invention, but the scope of protection of the patent of the present invention is not limited thereto. The equivalent replacement or change of the technical solution and its invention patent concept all belong to the protection scope of the invention patent.

Claims (5)

1. a kind of visual identity and localization method based on RGB-D photographic head, it is characterised in that：Methods described includes following step Suddenly：

1) three are converted into after the collection for carrying out coloured image and depth image to object by Microsoft Kinect camera sensing devices Dimension point cloud chart picture；

2) to step 1) obtain three-dimensional point cloud image each point carry out corresponding normal vector calculating；

3) to step 2) the normal direction duration set that obtains, the background plane that deployment area growth algorithm is placed to object carries Take；

4) by step 3) in the point of background plane that extracts remove, and remaining cloud is carried out object point converge conjunction extract and Convex closure extraction process；

5) by step 4) in extract each object point converge conjunction in combination with corresponding convex closure, carry out second zone growth, realize The segmentation of each object integrity profile and the extraction of complete point set；

6) according to step 5) in the integrity profile of each object that obtains, extracting corresponding coloured image simultaneously carries out respectively feature extraction And match cognization；

7) by step 5) point in the integrity profile of each object that obtains converges conjunction and carries out computing of averaging, and obtains each object in phase Positional information in machine coordinate system；

8) by step 7) positional information of each object for obtaining in camera coordinates system, coordinate system transformation is carried out, it is transformed into the world In the middle of coordinate system, the positioning of each object is realized.

2. a kind of visual identity and localization method based on RGB-D photographic head according to claim 1, it is characterised in that： Step 2) in, the method for calculating normal vector is：

If P_kFor want to claim surface normal point, point P is found first_kNeighbouring four point P up and down in the picture₁、P₂、P₃ And P₄, P₁And P₃Composition of vectorP₂And P₄Composition of vectorThen point P_kSurface normalCan pass throughMultiplication cross Arrive, it is specific as follows：

$\stackrel{\→}{{\mathrm{\ν}}_p}=\stackrel{\→}{{\mathrm{\ν}}_2}\×\stackrel{\→}{{\mathrm{\ν}}_1}$

The normal vector of each point of three-dimensional point cloud image is calculated by as above formula.

3. a kind of visual identity and localization method based on RGB-D photographic head according to claim 1, it is characterised in that： Step 3) in, the normal vector of each point of first sequential scan three-dimensional point cloud image, the normal vector for running near vertical then continues Find the point nearby and normal vector is the point of near vertical, in being added to potential plane point set, if potential plane point set midpoint Threshold value of the number more than setting, then it is assumed that the potential plane point set is that a plane point set merges and potential plane point set is added To in plane set, remaining normal vector is otherwise continued to scan on, after the end of scan, you can obtain plane set, realized flat The extraction in face.

4. a kind of visual identity and localization method based on RGB-D photographic head according to claim 1, it is characterised in that： Step 5) in, the segmentation of the object integrity profile and the extraction of complete point set are comprised the following steps：

A) point being input in three-dimensional point cloud, planar point set peace face Convex range；

B) point in sequential scan plane Convex range, searching belongs in Convex range but is not belonging to the point of plane；

C) after the non-flat cake in the Convex range found in step b), the point is continually looked for all non-flat in convex closure nearby Cake, and it is added to potential object point concentration；

If the number of the potential object point centrostigma d) in step c) less than setting threshold value, return to step b) continue to scan on it is surplus Under plane convex closure point；

If the number of the potential object point centrostigma e) in step c) is more than or equal to the threshold value of setting, then it is assumed that the potential object point Collect for an object point set；

F) continue the borderline point of searching convex closure near the potential object point set in step e) and be added to potential object point concentration, The object point outside convex closure is also looked for out and add potential object point to concentrate；

G) the potential object point set obtained in step f) is added in collection of objects；

If the point h) still suffered from a cloud is not scanned, returns to step b) and continually look for new object point set；

If the whole points i) put in cloud are scanned, algorithm terminates to obtain collection of objects.

5. a kind of visual identity and localization method based on RGB-D photographic head according to claim 1, it is characterised in that： Step 6) it is specially：

First, object corresponding region in coloured image is obtained by the three-dimensional point cloud set of object, the figure that object is located Using the Feature Detector of Open CV after intercepting as region::Detect () function obtains surf characteristic points；

Secondly, further using the Feature2D of Open CV::Compute () function obtains surf characteristic vectors, object Surf characteristic vectors are added in identification storehouse as the identification feature of the object, or directly as the spy of the object identification matching Vector is levied, when corresponding surf characteristic vectors are matched in the surf characteristic vectors and storehouse for realizing object, using arest neighbors The storehouse FLANN that increases income carries out the matching of characteristic vector；

Finally, the matching of characteristic vector is carried out using the arest neighbors storehouse FLANN that increases income, particularly as being to use Open CV Descriptor Matcher::Match () function is realized.