WO2019100933A1 - Method, device and system for three-dimensional measurement - Google Patents

Abstract

A three-dimensional measurement method, and a device (20) and a system (10) for implementing the method. The three-dimensional measurement method comprises: extracting feature points (P₁, P₂, P₃) in images (IM₁, IM₂, IM₃) from three cameras (C₁, C₂, C₃); screening a matching feature point group on the basis of a first disparity d₁ generated by the object point (P) between the first image (IM₁) and the second image (IM₂) in a first direction (A) and a second disparity d₂ generated by the object point (P) between the second image (IM₂) and the third image (IM₃) in a second direction (B); and calculating three-dimensional coordinates of the object point (P) corresponding to the matching feature point group. According to the three-dimensional measurement method, it can help to reduce the calculation amount of the feature point matching process. The accuracy and matching efficiency of feature point matching can be improved.

Description

Method, device and system for three-dimensional measurement Technical field

The present invention generally relates to methods, apparatus, and systems for three-dimensional measurements, and more particularly to three-dimensional measurement methods, apparatus, and systems based on computer vision techniques.

Background technique

Three-dimensional measurement based on computer vision and three-dimensional reconstruction based on three-dimensional measurement are widely used in industry, security, transportation and entertainment. Industrial robots sense the real world and make decisions that require three-dimensional spatial information. Security monitoring adds 3D scenes to improve target recognition accuracy. Auto-driving, drones, etc. need to sense the position of surrounding objects in real time. Building restoration, cultural relics restoration, etc. require three-dimensional reconstruction of buildings and cultural relics, especially three-dimensional reconstruction based on high-density true color point clouds. In addition, the characters formed by three-dimensional reconstruction are widely used in the movie, animation, and game industries. Three-dimensional virtual characters formed based at least in part on three-dimensional reconstruction are also widely used in the VR and AR industries.

Three-dimensional reconstruction based on binocular cameras has been developed for decades, but restoring a true-color three-dimensional point cloud with high accuracy is not an easy task. Specifically, the three-dimensional reconstruction based on the binocular camera recovers the three-dimensional coordinate information of the object by calculating the positional deviation between the image points corresponding to the same object point in the binocular image, the core of which is to select the feature points in the image, and Find/screen feature point groups (ie, feature point matching) on different images that may correspond to the same object point. The methods currently used to filter matching feature point groups are as follows:

1) Calculate the sum of squared differences (SSD) of the pixel difference between the corresponding points of different images to be matched, and find the minimum value;

2) Calculate the sum of the squares of the zero-to-average sum of squared differences between the corresponding points of the different images to be matched, and find the minimum value;

3) calculating the absolute value of (SAD, sum of absolute difference) between the corresponding points of the different images to be matched, and finding the minimum value;

4) Calculate the minimum value of (ZSAD, zero mean sum of squared differences) between the corresponding points of the different images to be matched, and find the minimum value.

5) Calculate the normalization cross correlation (NCC) of the corresponding points of the corresponding images to be matched, and find the maximum value;

6) Calculate the zero-mean normalized cross correlation (ZNCC) of different image to be matched corresponding points, and find the maximum value.

A common feature of the above method for screening matching feature point groups is that which feature point on one image is most similar to the feature point to be matched on another image, and one problem exists in the processing of images of complex scenes. The medium match error rate is often very high. For example, in an image of a periodic structure, two image points that do not belong to the same object or on the same period of the same period, it is likely that the normalized cross-correlation of the neighborhood is the largest, or the pixel difference is the smallest, and the true corresponding point combination may be It is in the position where the normalized cross-correlation value is the second largest or the pixel difference is the second. When there are no other dimensions to verify that the match is valid, then that match error will persist and be passed. Another problem with the above-described existing method of screening matching feature point groups is that the amount of calculation is very large, resulting in problems such as excessive processing time and/or difficulty in miniaturization and high cost of the required computing device.

Summary of the invention

It is an object of the present invention to provide a three-dimensional measurement method and apparatus and system for implementing the same that at least partially solve the above-described problems in the prior art.

According to an aspect of the present invention, a three-dimensional measurement method based on a parallax ratio relationship is provided, the method comprising: receiving first, second, and third images from a first camera, a second camera, and a third camera, respectively The first camera, the second camera, and the third camera have the same focal length and optical axes parallel to each other, and the optical centers of the first camera, the second camera, and the third camera are disposed on the same plane perpendicular to the optical axis; Extracting feature points in the first image, the second image, and the third image; matching feature points in the first image, the second image, and the third image, the matching including filtering matching features based on the following parallax ratio relationship a group of points: a first parallax d ₁ in a first direction and a second direction generated between the first image and the third image, which are generated between the first image and the second image, in the second direction The second parallax d _{2 above} satisfies d ₁ :d ₂ =D ₁ :D ₂ , where D ₁ is the offset of the optical center of the first camera relative to the optical center of the second camera in the first direction, D ₂ is the optical center with respect to the second camera An offset of the optical center of the three cameras in the second direction, wherein the first direction is a direction parallel to the plane and not perpendicular to a line connecting the optical centers of the first camera and the second camera, and the second direction is a direction parallel to the plane and not perpendicular to a line connecting the optical centers of the second camera and the third camera; and calculating a three-dimensional coordinate of the object point corresponding to the matched feature point group.

According to another aspect of the present invention, there is provided a three-dimensional measuring apparatus comprising: a processor; and a memory storing program instructions, wherein when the program instructions are executed by the processor, causing the processor to perform the following operations Receiving a first image, a second image, and a third image; extracting feature points in the first image, the second image, and the third image, respectively; matching feature points in the first image, the second image, and the third image The matching includes: filtering the matched feature point group based on the following coordinate relationship: a difference between the abscissa of the feature point in the first image and the abscissa of the feature point in the second image and the feature point in the second image The abscissa has a predetermined proportional relationship with the difference between the abscissas of the feature points in the third image, and the feature points in the first image and the third image have the same ordinate; and the object corresponding to the matched feature point group is calculated The three-dimensional coordinates of the point.

According to still another aspect of the present invention, a three-dimensional measuring apparatus for use with a camera array for three-dimensional measurement is provided. The camera array includes at least a first camera, a second camera, and a third camera, the first camera, the second camera, and the third camera having the same focal length and optical axes parallel to each other, and the first camera and the second camera The optical centers of the third camera and the third camera are arranged on the same plane perpendicular to the optical axis. The three-dimensional measuring apparatus includes a processing unit that receives a first image, a second image, and a third image from the first camera, the second camera, and the third camera, respectively, and is configured to perform processing of: Extracting feature points in an image, a second image, and a third image; matching feature points in the first image, the second image, and the third image, the matching including filtering the matched feature point groups based on the following parallax ratio relationship: a same object point, a first parallax d ₁ generated in a first direction between the first image and the second image, and a second generated in the second direction between the second image and the third image The parallax d ₂ satisfies d ₁ :d ₂ =D ₁ :D ₂ , where D ₁ is the offset of the optical center of the first camera relative to the optical center of the second camera in the first direction, and D ₂ is the first An offset of the optical center of the two cameras relative to the optical center of the third camera in the second direction, wherein the first direction is an optical center connection parallel to the plane and non-perpendicular to the first camera and the second camera The direction of the line, the second direction being parallel to the plane and not perpendicular to Two cameras and the direction of the camera optical center of the third connection; and calculating a set of matching feature points corresponding to the three-dimensional coordinates of the object point.

According to still another aspect of the present invention, a three-dimensional measurement system based on a parallax ratio relationship is provided, the system comprising: a camera array and any one of the above three-dimensional measurement devices, the camera array including at least a first camera, a second camera, and a a three camera, the first camera, the second camera, and the third camera having the same focal length and an optical axis parallel to each other, and the optical centers of the first camera, the second camera, and the third camera are arranged in a same direction perpendicular to the optical axis on flat surface.

DRAWINGS

Other features, objects, and advantages of the present invention will become more apparent from the Detailed Description of Description

1A, 1B and 1C show the relationship between the binocular parallax and the relative position of the camera optical center;

2 shows a parallax ratio relationship of cameras disposed on the same plane perpendicular to the optical axis;

3 is a schematic structural block diagram showing an example of a three-dimensional measuring system of the present invention;

Figure 4 shows a schematic overall flow chart of the three-dimensional measuring method of the present invention;

FIG. 5 illustrates an example of a camera array that can be used in conjunction with the three-dimensional measurement method according to the first embodiment of the present invention;

6A, 6B, and 6C show the parallax ratio relationship between the cameras shown in Fig. 5.

FIG. 7 is a view schematically showing an example of an image obtained by the camera array shown in FIG. 5 and illustrating coordinate difference values of image points;

FIG. 8 is a schematic flow chart showing a three-dimensional measuring method according to a first embodiment of the present invention; FIG.

FIG. 9 shows an example of a process of screening a feature point group which can be used for the three-dimensional measurement method according to the first embodiment of the present invention;

FIG. 10 is a view schematically comparing feature point matching based on similarity calculation and feature point matching based on coordinate relationship in the three-dimensional measurement method according to the first embodiment of the present invention;

Figure 11 is a flow chart showing an example of a three-dimensional measuring method according to a first embodiment of the present invention;

12 shows an example of a camera array that can be used in combination with the three-dimensional measurement method according to the second embodiment of the present invention and shows its parallax ratio relationship;

Figure 13 is a view schematically showing an example of an image obtained by the camera array shown in Figure 12 and illustrating coordinate difference values of image points;

FIG. 14 shows an example of a process of screening a feature point group which can be used for the three-dimensional measurement method according to the second embodiment of the present invention;

15 shows an example of a camera array that can be used in combination with the three-dimensional measurement method according to the third embodiment of the present invention and shows its parallax ratio relationship;

Figure 16 is a view schematically showing an example of an image obtained by the camera array shown in Figure 12 and illustrating coordinate difference values of image points;

17 shows an example of a process of screening a feature point group that can be used in the three-dimensional measurement method according to the third embodiment of the present invention;

Figure 18 is a flowchart showing one example of a three-dimensional measuring method according to a third embodiment of the present invention;

19 illustrates an example of a camera array arrangement that can be used in a three-dimensional measurement system in accordance with an embodiment of the present invention.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the invention, rather than the invention. It should also be noted that, for the convenience of description, only parts related to the invention are shown in the drawings.

In the present application, "left" and "right", "upper" and "lower" respectively denote only relative positions unless otherwise stated, when "left" and "right", "upper" and "lower", The positions indicated to each other in the mutually perpendicular directions are not limited to those in which the person usually refers to the left and right. In addition, in the present application, "vertical" and "horizontal" indicate directions perpendicular to each other unless otherwise stated, the former is not limited to the "vertical" direction, and the latter is not limited to the "horizontal direction".

It should be noted that the embodiments in the present application and the features in the embodiments may be combined with each other without conflict. The invention will be described in detail below with reference to the drawings in conjunction with the embodiments.

First, the relationship between binocular parallax and camera optical distance will be described with reference to FIGS. 1A, 1B, and 1C. In the figure, O _l and O _r represent the optical centers of the left and right cameras respectively (the optical center of the camera lens), and I _l and I _r represent the image planes of the left and right cameras, respectively (hereinafter referred to as left image and right, respectively). Image side). The image plane of the camera is determined by the position of the photosensitive surface of the image sensor contained in the camera, such as a CCD or CMOS, typically located at a focal length f from the optical center. The camera is usually a real image that is inverted on the object. The image plane is located on the opposite side of the object with respect to the optical center. However, for the convenience of illustration and analysis, the image surface is shown at a symmetrical position on the same side as the object. Show. It should be understood that this does not change the parallax relationship that will be discussed in this application.

Both cameras for binocular vision have the same focal length. The optical centers O _l and O _{r of the} left and right cameras are separated by a distance D (also referred to as "baseline"), and the corresponding optical axes Z _l and Z _r are parallel to each other.

FIGS. 1A, 1B and 1C, the optical center of the camera to the left O _l camera coordinate system origin to the left and right of a camera optical axis is the Z-direction. The optical centers of the left and right cameras are located in the same plane (ie, the XY plane) perpendicular to the optical axis. 1A and 1B, a direction to connect with the optical center O _l O _r as the X direction of the camera coordinate system, a direction perpendicular to the optical axis of the optical center and the connection to the Y-direction. It should be appreciated that the camera coordinate system may also be set in other ways, for example, the optical center of the camera to the right of the origin O _r, is set in a manner different from the camera coordinate system, discussed below, does not affect the parallax than a given relationship.

In the present application, the image planes corresponding to the respective cameras have image plane coordinate systems set in the same manner. For example, in the illustrated example of the present application, the point at which the camera optical axis intersects the image plane (the image point imaged by the object point on the optical axis) is the origin of the image plane coordinate system, and is parallel to the X coordinate of the camera coordinate system, respectively. The direction of the Y-axis is the x-axis and the y-axis of the image plane coordinate system. It should be understood that the image coordinate system can also be set in other ways, for example, using an apex angle of the photosensitive surface of the image sensor as the origin, and the setting of different image coordinate systems does not affect the parallax discussed below. The ratio relationship.

Fig. 1A shows the relationship between the binocular parallax and the optical center distance in the direction in which the camera is connected in the optical center. Figure 1A shows the projection of the entire imaging system on the XZ plane. The same object point P[X, Y, Z] in space is imaged by the left and right cameras to the image points P _l and P _{r , respectively} . As more clearly shown in FIG. 1B, based on the triangle similarity principle, for the object point P, the left and right cameras generate a parallax d=x _l -x _r in the direction (x direction) of the image corresponding to the camera's optical center line. And d / D = f / Z, where x _l and x _r are the x-axis coordinates of the image points P _l and P _r in the left image plane I _l and the right image plane I _r , respectively.

Fig. 1B shows binocular parallax in a direction perpendicular to the optical line of the camera. Figure 1 B schematically illustrates the projection of the entire imaging system on the YZ plane. As shown in FIG. 1B, the object point P does not generate a parallax in the direction (y direction) of the image perpendicular to the optical line of the camera by the left and right cameras, that is, d=y _l -y _r =0.

Next, the relationship between the parallax in the arbitrary direction on the image plane and the distance D between the optical centers of the cameras is examined. For the convenience of discussion, the direction of the parallax to be investigated is set to the X direction in FIG. 1C, and the optical line connection between the left camera and the right camera does not coincide with the X direction, in other words, the direction of the parallax to be inspected relative to the camera optical center. The connection can be in any direction. Intermediate can be assumed that there is a camera, which is the optical center and the optical center O _m O _l O _r and disposed in the same plane perpendicular to the optical axis of the camera is aligned with the intermediate left camera in the X direction, the Y direction and the right camera Alignment, as shown in Figure 1C. According to the discussion above with reference to FIGS. 1A, 1B and 1C, the parallax d _lm = x _{l -} x _m = D _x × f / Z of the left camera and the intermediate camera in the X direction, where x _m is the object point passing through the intermediate camera The x-axis coordinate of the imaged point on the image plane of the intermediate camera, D _x is equal to the offset of the right camera relative to the left camera in the X direction, ie D _x =X _r -X _l . Since the middle camera and the right camera are aligned in the Y direction, there is no parallax in the X direction, that is, d _mr = x _{m -} x _r =0. In summary, the left camera and the right camera produce a parallax d _x = x _{l -} x _r = d _lm + d _mr = D _x × f / Z in the X direction. Similarly, the left camera and the right camera produce a parallax dy=y _l -y _r =D _y ×f/Z in the Y direction, where D _y is equal to the offset of the right camera relative to the left camera in the Y direction, ie D _y = Y _r -Y _l .

2 expands the situation shown in FIG. 1C into the case of three cameras arranged in the same plane perpendicular to the optical axis, wherein the first camera C ₁ , the second camera C _{2 ,} and the third camera C ₃ have the same The focal length f is an optical axis (not shown) parallel to each other, and the respective optical centers O ₁ , O ₂ and O _{3 are} located in the same plane perpendicular to the optical axis. According to the analysis above with reference to FIG. 1C, for the same object point, the first parallax d ₁ =D ₁ ×f/Z in the first direction generated between the images obtained by the first camera and the second camera, wherein D ₁ Is the offset of the optical center O ₂ of the second camera relative to the optical center O _{1 of the} first camera in the first direction A, and the image obtained between the second camera and the third camera, in the second direction The second parallax on B is D ₂ = D ₂ × f / Z, where D ₂ is the offset of the optical center O ₃ of the third camera relative to the optical center O _{2 of the} second camera in the second direction B, wherein The first direction A is parallel to the plane of the camera's optical center and is not perpendicular to the direction of the optical connection of the first camera and the second camera, and the second direction B is parallel to the plane of the camera's optical center and is non-perpendicular to the first The direction of the optical connection between the two cameras and the third camera. Although the first direction is different from the second direction in FIG. 2, the first direction may be the same as the second direction. The inventors of the present invention have found that the following parallax ratio relationship exists for three cameras arranged in the same plane perpendicular to the optical axis, that is, the first parallax d ₁ and the second parallax d ₂ satisfy d ₁ : d ₂ =D ₁ :D ₂ . It should be noted that the equal sign in the above equation does not mean that the present invention is limited to a strictly equal case, and those skilled in the art can understand that if the difference between the two sides of the equal sign is within a certain range, it can be considered as engineering significance. Equal on. For example, it can be expressed as d ₁ :d ₂ =(1+k)*D ₁ :D ₂ , where k represents an allowable relative deviation, and |k|<<1,|k| can be, for example, 0.05 or 0.01.

Based on the above findings, a three-dimensional measurement system and a three-dimensional measurement method based on a parallax ratio relationship are proposed according to the present invention. FIG. 3 shows a schematic structural block diagram of one example of a three-dimensional measurement system 10 in accordance with the present invention. FIG. 4 shows a schematic overall flow diagram of a three dimensional measurement method 100 of the present invention.

3, three-dimensional measurement system includes a camera array 10 in FIG. CA, the CA includes at least a first camera array camera C _1, C _2, and the second camera third camera ₃ C, they have the same focal length and the optical axes are parallel to one another And the respective optical centers are arranged on the same plane perpendicular to the optical axis. Preferably, each camera has the same aperture, ISO, shutter time, image sensor, and the like. More preferably, each camera is a camera of exactly the same type.

The three-dimensional measurement system 10 includes a processing unit 11 that receives images from a camera array, including first, second, and third images from a first camera C ₁ , a second camera C _{2 ,} and a third camera C ₃ , and Processing is performed based on these images to achieve three-dimensional measurement. Of course, the camera array CA can include additional cameras, and the processing unit 11 can receive images from these additional cameras and process them.

The three-dimensional measurement system 10 can include a control unit 12 operative to control the first camera, the second camera, and the third camera to acquire images simultaneously. For the three-dimensional reconstruction of dynamic scenes, such as the traffic on the road, the instantaneous expression of people, the result of asynchronous shooting is that the matching result error rate is very high. The control unit 12 can also control a multiple zoom such as a camera. For example, for a farther scene, the three cameras need to be adjusted to a larger focal length identically. The control unit 12 may be connected through a wired or wireless manner with the first camera C _1, C _2, and the second camera third camera C _3, to realize the above control. The control unit 12 can be in communication with the processing unit 11 to receive information from the processing unit 11 to generate the control signals for controlling the camera to acquire images simultaneously, or to operate independently to achieve the above control.

The three-dimensional measurement method 100 based on the parallax ratio relationship of the present invention is implemented based on, for example, the camera array CA of the three-dimensional measurement system 10. The three-dimensional measurement method 100 includes:

S110: receiving a first image, a second image, and a third image from the first camera, the second camera, and the third camera, respectively;

S120: extract feature points in the first image, the second image, and the third image;

S130: Matching feature points in the first image, the second image, and the third image, the matching includes filtering the matched feature point group based on the following parallax ratio relationship: for the same object point, in the first image and the second image The first disparity d ₁ generated in the first direction and the second disparity d ₂ in the second direction generated between the second image and the third image satisfy d ₁ :d ₂ =D ₁ : D ₂ , where D ₁ is the offset of the optical center of the first camera relative to the optical center of the second camera in the first direction, and D ₂ is the optical center of the second camera relative to the optical center of the third camera An offset in the second direction, wherein the first direction is a direction parallel to the plane and non-perpendicular to a line connecting the optical centers of the first camera and the second camera, the second direction being parallel to the plane And not perpendicular to the direction of the optical connection of the second camera and the third camera;

S140: Calculate the three-dimensional coordinates of the object points corresponding to the matching feature point group.

Similarly, the equal sign in the above equation d ₁ :d ₂ =D ₁ :D ₂ does not mean that the present invention is limited to strictly equal cases, and those skilled in the art can understand that the difference between the two sides of the equal sign is within a certain range. In this case, it can be considered as equal in engineering sense. For example, it can be expressed as d ₁ :d ₂ =(1+k)*D ₁ :D ₂ , where k represents an allowable relative deviation, and |k|<<1,|k| can be, for example, 0.05 or 0.01.

The processing unit 11 of the three-dimensional measurement system 10 is configured to perform the above-described processes s110 to s140 in the three-dimensional measurement method 100. Processing unit 11 may be implemented by a processor and a memory storing program instructions that, when executed by the processor, cause the processor to perform the operations of processes s110-s140 described above. The processing unit 11 can constitute a three-dimensional measuring device 20 according to the invention.

In process s110, the image may be received directly from the first camera, the second camera, and the third camera, or may be received via other units or devices.

In process s120, feature points in each image are found and described by a combination of attributes. For example, a grayscale gradient can be used to find a point with a large change in gray as a feature point, or a sift algorithm can be used to find a sift feature point, or a corner point detection algorithm such as the Harris algorithm can be used to find a corner point of an image as a feature point. It should be understood that the present invention is not limited to the specific method for extracting feature points.

The processing s130 is configured to match the feature points in each image, and includes processing for filtering the matching feature point groups based on the parallax ratio relationship described above. The process of screening matching feature point groups based on the parallax ratio relationship in process s130 will be described in more detail below in conjunction with various embodiments.

Processing s130 may further include the process of filtering the matched set of feature points in other manners. In other words, in some embodiments, the process s130 may implement feature point matching only by processing based on a parallax ratio relationship. In other embodiments, feature point matching may be implemented in processing s130 in conjunction with processing based on disparity ratio relationships and processing based on other matching/screening methods. Other ways of filtering matching feature point groups include, for example, applying a similarity calculation to a pixel or neighborhood pixel group to filter matching feature point groups. Here, the similarity calculation includes a sum of squares of pixel gradation differences, a sum of squares of zero-crossing average pixel gradation differences, an absolute value of pixel gradation differences, an absolute value of a zero-crossing average pixel gradation difference, and an adjacent The calculation of at least one of the normalized cross-correlation between the domains and the zero-crossing average normalized cross-correlation.

For example, in some embodiments, the process s130 further includes applying a similarity calculation to the pixel or neighborhood pixel group to further filter the matched feature point group for the matched feature point group selected based on the parallax ratio relationship. In some examples, the similarity calculation may be applied only to two or more sets of feature points that contain the same feature points, ie, only if the match result is not unique.

In other embodiments, in the process s130, the matching feature point group is first filtered by, for example, applying a similarity calculation to the feature point or its neighboring pixel group, and then the matching feature point group obtained by the similarity calculation screening is judged. Whether the group feature points satisfy the parallax ratio relationship, thereby further screening the matching feature point groups.

Each of the matched feature point groups obtained by processing s130 includes one feature point from the first image, the second image, and the third image, respectively. In the process s140, the depth (Z coordinate) of the corresponding object point may be calculated based on any two feature points in a matched feature point group, and then the X, Y coordinates of the object point are calculated according to the similar triangle principle. A method of calculating a depth value based on two matched feature points and calculating an X, Y coordinate based on the depth value is known, and will not be described herein.

In some preferred embodiments, a depth value is calculated in each of the two feature points in a matched feature point group in the processing s130, and the average value is taken as the depth value of the corresponding object point. In theory, the calculated depth values should be equal, but in reality, considering the effect of image noise, there will be a slight difference between the three values. Therefore, taking the average value can reduce the influence of noise and improve the accuracy of the depth value. In some other embodiments, two pairs of feature points in the feature point group may also be selected to calculate the depth value and take the average value as the depth value of the object point.

For three-dimensional measurement for three-dimensional reconstruction, the process s140 may further include calculating a color value of the object point corresponding to each matched feature point group, such as [R, G, B]. This color value can form voxel information together with the coordinates [X, Y, Z] of the object point, such as [X, Y, Z, R, G, B]. In 3D reconstruction, voxel information of all object points can be combined to form a true color 3D model of the object or scene.

The three-dimensional measurement method 100 and the three-dimensional measurement system 10 will be described in more detail below in connection with various embodiments, particularly in which a matching feature point group is screened based on a parallax ratio relationship.

5 to 11 illustrate a three-dimensional measurement system and a three-dimensional measurement method according to a first embodiment of the present invention.

5 shows a camera array CA in a three-dimensional measuring system according to a first embodiment of the present invention, wherein the first camera C ₁ , the second camera C _{2 ,} and the third camera C ₃ have the same focal length and optical axes parallel to each other And the optical centers of the three cameras are arranged on the same straight line (X direction) perpendicular to the optical axis. In the arrangement (a) shown in Fig. 5, D ₁ = D ₂ ; in the arrangement (b), D ₁ ≠ D ₂ .

6A, 6B, and 6C show the parallax ratio relationship between the three cameras shown in Fig. 5. As shown, the space of the same object point P [X, Y, Z] through a first camera C _1, C _2, and the second camera C _3, respectively, third camera image in the first image plane I ₁ P ₁ point, the first two plane image I P ₂ of _two points and the third image plane I P ₃ point _3. Each of the image planes has an image plane coordinate system set in the same manner as described above. In FIG. 6, the x-axis direction of each image plane corresponds to the direction of the line where the camera's optical center is located, and the y-axis is perpendicular to the x-axis. As shown more clearly in Figure 6C, in the x-axis direction, a first parallax d ₁ = x _{1 -} x ₂ = D ₁ × f / Z generated between the first camera and the second camera, the second camera and The second parallax d ₂ = x _{2 -} x ₃ = D ₂ × f / Z generated between the third cameras satisfies d ₁ : d ₂ = D ₁ : D ₂ . Similarly, in the case of d ₁ :d ₂ =(1+k)*D ₁ :D ₂ , it can be considered equal. Where k represents the relative deviation allowed, and |k|<<1, |k| may be, for example, 0.05 or 0.01.

FIG. 7 schematically shows an example of the first image IM ₁ , the second image IM _{2 ,} and the third image IM ₃ obtained by the camera array shown in FIG. 5, wherein the abscissa axis (u axis) of the image corresponds to the camera light. The direction of the line where the heart is located. Ideally, as shown in FIG. 7 by superimposing the first image _{IM. 1,} the second and the third image IM _{2. 3} obtained image IM artifact image IM 'as shown, the same object point P [X, Y, Z] in the image The corresponding image points P ₁ [u ₁ , v ₁ ], P ₂ [u ₂ , v ₂ ], P ₃ [u ₃ , v ₃ ] in IM ₁ , IM ₂ and IM ₃ have the same ordinate, ie v ₁ = v ₂ = v ₃ , and the abscissa difference s ₁ = u _{1 -} u ₂ = d ₁ / d _{p of the} image points P ₁ [u ₁ , v ₁ ], P ₂ [u ₂ , v ₂ ], The abscissa difference s ₂ = u _{2 -} u ₃ = d ₂ / d _{p of the} image point P ₂ [u ₂ , v ₂ ], P ₃ [u ₃ , v ₃ ] satisfies s ₁ : s ₂ = d ₁ :d ₂ =D ₁ :D ₂ , where d _p is the side length of the pixel unit of the image sensor.

The first image, the second image, and the third image obtained directly from the first camera, the second camera, and the third camera due to manufacturing errors, mounting errors, and changes in camera internal parameters and/or external parameters generated during use It usually deviates from the above ideal situation. This can bring the image closer to the ideal situation by physically calibrating or correcting the camera and/or by correcting the image by a calculation program. Therefore, the three-dimensional measurement method according to an embodiment of the present invention may include correction processing of the first image, the second image, and the third image, the correction processing such that the first image, the second image, and the third image correspond to the first camera The points of the optical axes of the second camera and the third camera have the same abscissa and ordinate, and the abscissa directions of the first image, the second image, and the third image all correspond to the direction of the straight line, wherein The directions of the abscissa and the ordinate are perpendicular to each other.

The correction process is performed before the process s130 shown in Fig. 4, that is, the process of matching the feature points, preferably after the image is received and before the feature points are extracted, that is, between the processes s110 and s120 shown in Fig. 4. However, it should be noted that the three-dimensional measurement method according to an embodiment of the present invention is not limited to the case including the above-described correction processing, for example, in some applications, correction may be performed by physical adjustment of the camera array without an image Correction.

Accordingly, as shown in FIG. 3, a correction unit 13 may be further included in the three-dimensional measurement system 10 according to the present invention, the correction unit 13 receiving an image from the camera column CA and generating a correction matrix based at least in part on the image, The correction matrix is provided to the processing unit 11. The correction matrix, when applied to the first image, the second image, and the third image by the processing unit 11, implements the above-described correction processing in the three-dimensional measurement method. As shown in Fig. 3, the three-dimensional measuring device 20 can include the correcting unit 13. Processing unit 11 and correction unit 13 may be implemented based on the same processor and memory or different processors and memories.

Based on the camera array having the arrangement shown in FIG. 5, as shown in FIG. 8, in the process s230 of the three-dimensional measurement method 200 according to the first embodiment of the present invention, the ratio based on the parallax ratio in the process s130 shown in FIG. The processing of matching the feature point groups is implemented by: filtering the matched feature point groups based on the following coordinate relationships: the abscissa of the feature points P ₁ [u ₁ , v ₁ ] in the first image and the feature points P ₂ in the second image [ The difference s ₁ = u ₁ -u ₂ of the abscissa of u ₂ , v ₂ ] and the abscissa of the feature point P ₂ [u ₂ , v ₂ ] in the second image and the feature points in the third image The difference s ₂ = u _{2 -} u ₃ of the abscissa of P ₃ [u ₃ , v ₃ ] satisfies s ₁ : s ₂ = D ₁ : D ₂ and is in the first image, the second image, and the third image The feature points have the same ordinate, ie v ₁ = v ₂ = v ₃ . The other processing of the three-dimensional measurement method 200 is the same as the corresponding processing in the three-dimensional measurement method 100 described above with reference to FIG. 4, and details are not described herein again.

FIG. 9 shows an example of a process of screening matching feature point groups based on the above-described coordinate relationship s ₁ : s ₂ = D ₁ : D ₂ and v ₁ = v ₂ = v ₃ , process 300. As shown, process 300 includes:

S310: The first feature point and the second feature point are respectively selected in the first image, the second image, and the third image, and the ordinates of the first feature point and the second feature point satisfy relative to one The target ordinate is within a predetermined range;

S320: Calculate a desired abscissa of a third feature point matching the first feature point and the second feature point among the third one of the first image, the second image, and the third image, such that the difference s ₁ =u ₁ -u ₂ and the difference s ₂ =u ₂ -u ₃ satisfy s ₁ :s ₂ =D ₁ :D ₂ ;

S330: Search for the third feature point in the third party based on the expected position formed by the expected abscissa and the target ordinate of the third feature point.

It should be noted that the equal sign in the above equation s ₁ : s ₂ = D ₁ : D ₂ does not mean that the present invention is limited to strictly equal cases, and those skilled in the art can understand that the difference between the two sides of the equal sign is within a certain range. In the case of the inside, it can be considered as equal in engineering sense. For example, it can be expressed as s ₁ :s ₂ =(1+k)*D ₁ :D ₂ , where k represents an allowable relative deviation, and |k|<<1, |k| can be, for example, 0.05 or 0.01.

Process 300 can be used to scan an image line by line, for example, in a certain order of ordinates (increment or decrement) to filter matching sets of feature points. The above-described process 300 will be described in more detail below with reference to FIG. 7 taking the case where the first camera, the second camera, and the third camera are arranged at equal intervals (i.e., D ₁ = D ₂ ). It is to be understood that the following description is only illustrative and not restrictive.

In the processing s310, among the J feature points P ₂ [u ₂ (j), v ₂ (j)] (1 ≤ j ≤ J) having the target ordinate v _t in the second image IM ₂ , among them, a feature point P ₂ [u ₂ , v ₂ ] (first feature point), wherein the ordinate v ₂ (j)=v _t ; searching for features having a target ordinate in the first image IM ₁ and the third image IM ₃ Point P ₁ [u ₁ (i), v ₁ (i)] and feature point P ₃ [u ₃ (j), v ₃ (j)], where v ₁ (i)=v ₃ (j)=v _t Obtaining ₁ feature points P ₁ [u ₁ (i), v ₁ (i)] satisfying the condition in the first image IM ₁ , ₁ ≤ i ≤ I, and K in the second image IM ₂ satisfying the condition Characteristic point P ₃ [u ₃ (k), v ₃ (k)], 1 ≤ j ≤ K (when I or K is zero, the search is ended, the first feature point is reselected); assuming I ≥ K, in order to reduce Searching times, starting from the third image IM ₃ , that is, selecting one of the feature points P ₃ [u ₃ , v ₃ ] (second feature point) in IM ₃ in the third image (if M < N, from The picture on the left starts searching).

The first feature point is selected from the second image in the above description, which is merely exemplary, and the present invention is not limited to which image is selected from the first feature point. For example, in other examples, the number of feature points I, J, and K that meet the ordinate requirement in each image may be first compared, and the first feature point is selected in the image with the smallest number of feature points, and the number of feature points is The second feature point is selected in the small image, and finally the third feature point is searched for in the image with the largest number of feature points.

Considering the image error/noise, in the processing s310, the feature points can be searched and selected within a predetermined range of the ordinate of v _t - ε - v _t + ε, and ε is an integer greater than or equal to 0, which can be installed according to, for example, a camera array. It is determined by the condition of use or the quality of the image, preferably 0 ≤ ε ≤ 2.

Then, the process in s320, the symmetry is calculated by using the first image IM in _an expected occurrence of the feature point _{P 2 [u 2, v 2} ] and the feature point _{P 3 [u 3, v 3} ] matching feature points P ₁ (first The abscissa position of the three feature points (expected abscissa u _e ) such that the difference s ₁ =u _e -u ₂ and the difference s ₂ =u ₂ -u ₃ satisfy s ₁ :s ₂ =D ₁ :D ₂ =1, ie u _e = 2u ₂ -u ₃ . It can be seen that the abscissa of the feature point has a symmetric coordinate relationship at this time.

Next, in process s330, based on the expected abscissa u _e of the feature point P ₁ expected to appear and the target ordinate v _t , the expected position [u _e , v _t ] of the feature point P ₁ can be obtained, and based on the The expected position is searched in the first image IM ₁ for the presence or absence of the feature point P ₁ . Also, considering image error/noise, in some examples, the abscissa tolerance ε _u and the ordinate tolerance ε _v can be set, and [u _e -ε _u ～u _e + ε _u , v _t - ε _v ~ _{v t} + ε v] range inherent in the first image IM ₁ searches the feature point P ₁ (e.g. see FIG. _1, shown in a broken line range image IM. 7). Preferably, 0 ≤ ε _u ≤ 3, 0 ≤ ε _v ≤ 3 In other examples, only one of the abscissa tolerance and the ordinate tolerance may be set, and details are not described herein again.

If the expected feature point P ₁ is searched for in s330, the feature points P ₁ , P ₂ , P _{3 are taken} as one matching feature point group. If the expected feature point P ₁ is not searched, the selected first feature point (P ₂ ) and the second feature point (P ₃ ) do not have the possibility of matching, and the screening of the round ends, and then the selection can be selected. The next second feature point is then repeated for the above processes s320 and s330. After traversing all of the second feature points, a new first feature point is selected and the above process is similarly repeated.

The process 300 has been described above by taking a linear arrangement of the camera and a symmetric distribution (D ₁ : D ₂ =1) as an example. For the case of D ₁ : D ₂ =1: R, R ≠ 1, the operation of the above process 300 is similar, except that when calculating the expected abscissa of the third feature point, taking the specific case discussed above as an example , u _e =(1+R)×u ₂ -R×u ₃ , in this case, considering that the image coordinates are integers, it is necessary to round up the calculated values to obtain the desired expected abscissa, ie u _e =round {(1+R)×u ₂ -R×u ₃ }, “round” here denotes a rounding operation.

FIG. 9 shows only one example of a process of filtering matching feature point groups based on the coordinate relationship. In the case where the candidate matching feature point group has been obtained by other means (for example, calculation of the similarity of the pixel or the neighborhood pixel group), it is also possible to check whether the abscissa of each feature point in the candidate feature point group is certain. Satisfying s ₁ : s ₂ = D ₁ : D ₂ within the tolerance range, and judging whether the ordinate of each feature point is within an allowable range, so that the screening satisfies the above coordinate relationship s ₁ : s ₂ = D ₁ : D ₂ and A feature point group of v ₁ = v ₂ = v ₃ .

According to the three-dimensional measurement method 200 of the first embodiment of the present invention, the processing based on the parallax ratio relationship generated between the three cameras is simplified to the processing based on the coordinate relationship of the corresponding feature points in the image, and the feature point matching/filter matching is performed. In the process of feature point group, compared with the traditional method of feature point matching based on the similarity calculation of pixel or neighborhood pixel group, the former mainly adds and subtracts coordinates and a small number of multiplication operations (in D ₁ : In the case of D ₂ =1, only addition and subtraction operations are required, and the latter usually requires dense multiplication operations, such as convolution of matrices, so in comparison, the former greatly reduces the matching process. Calculated amount. A significant reduction in the amount of computation is of great significance for three-dimensional measurement and reconstruction based on high definition images and for real-time three-dimensional measurement and reconstruction, which offers the possibility of implementation of the latter.

FIG. 10 schematically compares feature point matching based on similarity calculation and feature point matching based on the coordinate relationship s ₁ : s ₂ = D ₁ : D ₂ in the three-dimensional measurement method 200 according to the first embodiment of the present invention. FIG. 10(a) shows a certain feature point P _l1 in one image obtained by, for example, a binocular camera, and three candidate feature points P _r1 which are very similar to the self attribute and the neighborhood attribute in another image, P _r2 , P _r3 . According to the unique requirement of feature point matching in 3D reconstruction, the above non-unique matching result can only be abandoned. In addition, it is conceivable that when due to image noise or the like, the incorrectly matched feature point P _r3 may exhibit greater similarity to the feature point P _l1 than the correctly matched feature point P _r1 , thereby based on the pair of pixels or neighborhoods Feature point group screening for pixel group similarity calculation may obtain erroneous matching results. (b) of FIG. 10 shows that for images from the first camera, the second camera, and the third camera arranged as shown in FIG. 5, based on the coordinate relationship s ₁ : s ₂ = D ₁ : D ₂ , the unique one can be selected The matching feature point groups (P _11, P _21, P ₃₁ ) are excluded, while the other two candidate feature points P ₃₂ , P ₃₃ that may be matched according to the attribute similarity are excluded. It should be noted that the equal sign in the above equation s ₁ : s ₂ = D ₁ : D ₂ does not mean that the present invention is limited to strictly equal cases, which may result in the inability to find a matching feature point group. Those skilled in the art can understand that if the difference between the two sides of the equal sign is within a certain range, it can be considered that the engineering sense is equal. For example, it can be expressed as s ₁ :s ₂ =(1+k)*D ₁ :D ₂ , where k represents an allowable relative deviation, and |k|<<1, |k| can be, for example, 0.05 or 0.01.

It can be seen that the feature point matching based on the coordinate relationship can help to eliminate the error result obtained by the matching based on the similarity calculation, and improve the correct rate of the matching; at the same time, since the unique matching result can be obtained with a larger probability, the matching result is avoided. The matching failure caused by not unique, so the feature point matching based on the parallax ratio relationship/the above coordinate relationship also contributes to the improvement of the matching rate, thereby contributing to the realization of the high density feature point cloud.

FIG. 11 is a flow chart showing an example of a three-dimensional measurement method, a three-dimensional measurement method 400, according to a first embodiment of the present invention. The three-dimensional measurement method 400 includes:

S410: receiving a first image, a second image, and a third image from the first camera, the second camera, and the third camera, respectively;

S420: performing a correction process on the first image, the second image, and the third image, as discussed above, such that the first image, the second image, and the third image correspond to the first camera, the second camera, and The points of the optical axis of the third camera have the same abscissa and ordinate, and the abscissa directions of the first image, the second image, and the third image all correspond to the direction of the straight line;

S430: extract feature points in the first image, the second image, and the third image;

S440: Matching feature points in the first image, the second image, and the third image, the matching includes:

S441: Filter matching feature point groups based on the following coordinate relationship: the abscissa of the feature points P ₁ [u ₁ , v ₁ ] in the first image and the horizontal of the feature points P ₂ [u ₂ , v ₂ ] in the second image The difference s ₁ = u ₁ -u ₂ of the coordinates and the abscissa of the feature point P ₂ [u ₂ , v ₂ ] in the second image and the feature point P ₃ [u ₃ , v ₃ in the third image The difference s ₂ = u _{2 -} u ₃ of the abscissa satisfies s ₁ : s ₂ = D ₁ : D ₂ , and the feature points in the first image, the second image, and the third image have the same vertical Coordinates, ie v ₁ = v ₂ = v ₃ ;

S442: further performing screening based on the similarity calculation of the pixel group of the pixel or the neighborhood for the matching feature point group obtained by processing s441;

S450: Calculate the three-dimensional coordinates of the object points corresponding to the matching feature point group.

The process s441 can be implemented as, for example, the process 300 shown in FIG. 9, but is not limited thereto.

Process s420 may be a correction process implemented using any correction method, including a correction process implemented by applying a certain correction matrix to the image.

According to the present invention, the parallax ratio relationship can also be applied to the correction processing, and for example, the correction matrix for the correction processing can be generated based on the parallax ratio relationship. As an example, the process of generating a matrix for the correction process based on the parallax ratio relationship may include extracting feature points in the first image, the second image, and the third image, respectively, for example, extracting a sparse feature dot lattice by using a sift algorithm Matching feature points in the first image, the second image, and the third image to obtain a plurality of matching feature point groups, for example, by using a RANSAC algorithm; using coordinates of the feature points in each image in the matching feature point group, according to the correction a matrix is applied to each of the images to match the parallax ratio relationship satisfied between feature points in the feature point group, to establish an overdetermined system of equations; and to solve the overdetermined equations by, for example, a least squares method to obtain a correction matrix .

In the three-dimensional measurement method 400, the matching feature point group is first filtered by the processing s441, so that the number of feature points entering the subsequent feature point matching processing s442 based on the similarity calculation is greatly reduced, and the calculation amount in the feature point matching process can be significantly reduced. . In addition, the combination of processing s441 and s442 also helps to improve the correct rate and matching rate of feature point matching, so that a higher density feature point cloud can be obtained.

12 to 14 illustrate a three-dimensional measurement system and a three-dimensional measurement method according to a second embodiment of the present invention.

12 shows a camera array CA in a three-dimensional measuring system according to a second embodiment of the present invention, wherein the first camera C ₁ , the second camera C _{2 ,} and the third camera C ₃ have the same focal length and optical axes parallel to each other (Z direction), the optical centers O ₁ , O _{2 of} the first camera C ₁ and the second camera C ₂ are aligned in the X direction, and the optical centers O ₂ , O _{3 of} the second camera C ₂ and the third camera C ₃ are along the Y Align the directions. The shift of the optical center of the second camera with respect to the optical center of the first camera in the X direction is D ₁ , and the shift of the optical center of the third camera with respect to the optical center of the second camera in the Y direction is D ₂ .

Referring to the discussion above in connection with FIG. 1C, and as shown in FIG. 12, in the x-axis direction, a first parallax d ₁ = x _{1 -} x ₂ = D ₁ × f / generated between the first camera and the second camera Z; in the y-axis direction, a second parallax d ₂ = y _{2 -} y ₃ = D ₂ × f / Z generated between the second camera and the third camera satisfies d ₁ : d ₂ = D ₁ : D ₂ . Similarly, those skilled in the art can understand that if the difference between the two sides of the equal sign is within a certain range, it can be considered as equivalent in engineering sense. For example, it can be expressed as d ₁ :d ₂ =(1+k)*D ₁ :D ₂ , where k represents an allowable relative deviation, and |k|<<1,|k| can be, for example, 0.05 or 0.01.

FIG. 13 is a view schematically showing an example of the first image IM ₁ , the second image IM _{2 ,} and the third image IM ₃ obtained by the camera array shown in FIG. 12, wherein the abscissa axis (u axis) of the image corresponds to the first The direction of the line where the camera and the second camera are located (x-axis direction), and the ordinate of the image (v-axis) corresponds to the direction of the line where the second camera and the third camera's optical center are located (y direction). Ideally, as shown in FIG. 13 by superimposing the first image _{IM. 1,} the second and the third image IM _{2. 3} obtained image IM artifact image IM 'as shown, the same object point P [X, Y, Z] in the image The corresponding image points P ₁ [u ₁ , v ₁ ], P ₂ [u ₂ , v ₂ ] in IM ₁ , IM ₂ have the same ordinate, ie v ₁ = v ₂ , in the images IM ₂ , IM ₃ The corresponding image points P ₂ [u ₂ , v ₂ ], P ₃ [u ₃ , v ₃ ] have the same abscissa, that is, u ₂ = u ₃ , and the image points P ₁ [u ₁ , v ₁ ], P ₂ [u ₂ , v ₂ ] the abscissa difference s ₃ = u _{1 -} u ₂ = d ₁ / d _p , the image point P ₂ [u ₂ , v ₂ ], P ₃ [u ₃ , v ₃ ] The ordinate difference s ₄ = v ₂ - v ₃ = d ₂ / d _p , satisfies s ₃ : s ₄ = d ₁ : d ₂ = D ₁ : D ₂ , where d _p is the side length of the pixel unit of the image sensor . Similarly, if the difference between the two sides of the equal sign is within a certain range, it can be considered as equivalent in the engineering sense. For example, it can be expressed as s ₃ : s ₄ = (1 + k) * D ₁ : D ₂ , where k represents an allowable relative deviation, and |k|<<1, |k| can be, for example, 0.05 or 0.01.

The three-dimensional measurement method according to the present embodiment may include correction processing of the first image, the second image, and the third image, the correction processing such that the first image, the second image, and the third image correspond to the first camera, the second image The points of the optical axes of the camera and the third camera have the same abscissa and ordinate, and the abscissa directions of the first image, the second image, and the third image correspond to the optical centers of the first camera and the second camera The direction, the ordinate direction corresponds to the direction of the optical line connection of the second camera and the third camera.

According to the camera array having the arrangement shown in FIG. 12, in the three-dimensional measurement method according to the second embodiment of the present invention, the process of screening the matching feature point group based on the parallax ratio relationship in the process s130 shown in FIG. 4 is implemented to include: The matching feature point group is filtered based on the following coordinate relationship: the abscissa of the feature point P ₁ [u ₁ , v ₁ ] in the first image and the abscissa of the feature point P ₂ [u ₂ , v ₂ ] in the second image The difference s ₃ =u ₁ -u ₂ and the ordinate of the feature point P ₂ [u ₂ , v ₂ ] in the second image and the feature point P ₃ [u ₃ , v ₃ ] in the third image The difference s ₄ = v ₂ - v _{3 of the} ordinate satisfies s ₃ : s ₄ = D ₁ : D ₂ and v ₁ = v ₂ , u ₂ = u ₃ . The other processing of the three-dimensional measuring method according to the present embodiment may be the same as or similar to the corresponding processing in the three-dimensional measuring method 100 described above with reference to FIG. 4, and details are not described herein again.

FIG. 14 shows an example of a process of screening matching feature point groups based on the above-described coordinate relationship, process 500. As shown in Figure 14, process 500 includes:

S510: selecting a first feature point and a second feature point in the first image and the second image, the ordinates of the first feature point and the second feature point satisfying a predetermined range with respect to a target ordinate;

S520: calculating a difference s ₃ = u ₁ -u ₂ between the abscissa of the first feature point and the abscissa of the second feature point;

S530: Calculate the difference s ₄ such that s ₃ : s ₄ = D ₁ : D ₂ ;

S540: Calculate a expected ordinate of the third feature point in the third image that matches the first feature point and the second feature point, such that a difference between the ordinate of the second feature point and the expected ordinate of the third feature point The second difference s ₄ calculated for the above;

S550: Search for the third feature point in the third image based on the expected position formed by the expected ordinate of the third feature point and the abscissa of the second feature point.

The operation of selecting feature points in the predetermined range with respect to the target coordinates in the processing s510 may refer to the operations described above in the processing s310 in the combination processing 300. In addition, similar to the process 300, the process s550 of the process 500 may also set a tolerance range with respect to the expected position, and details are not described herein again.

It should be noted that since the abscissa and the ordinate in the image in the present application represent two directions that are relatively perpendicular, in this embodiment, the first camera and the third camera actually have a pair with respect to the second camera. a positional relationship, such that in process s510, the first feature point and the second feature point are selected in the first image and the second image, and the ordinates of the first feature point and the second feature point satisfy The first and second feature points are selected first in the horizontally aligned two images by being within a predetermined range with respect to a target ordinate.

In some preferred embodiments, after the feature points are selected in the second image, feature points in the first image and the third image that have the same ordinate and the same abscissa as the feature point may be searched for based on the first image. And the number of feature points searched in the third image to determine the next search order. For example, when the number of feature points having the same ordinate in the first image is greater than the number of feature points having the same abscissa in the third image, the feature points may be selected from the third image to be calculated. And determining the expected position of the feature points in the first image and searching. Processing s510 is intended to cover this situation.

The process 500 can be used to traverse feature points in the second image, for example, row by row or column by column, to filter feature points in the first image and the third image to match, thereby filtering the matched feature point groups.

From a technical effect point of view, similar to the three-dimensional measurement method according to the first embodiment of the present invention, the three-dimensional measurement method according to the second embodiment of the present invention is relative to a pixel-based or neighborhood-based pixel group in the feature point matching process. In the feature point matching method of similarity calculation, the calculation amount of the matching process is greatly reduced, which helps to improve the spatial precision and real-time performance of the three-dimensional measurement. Meanwhile, the three-dimensional measurement method according to the present embodiment may be combined with a feature point matching method based on the similarity calculation of a pixel or a neighborhood pixel group, in which case the feature point matching based on the parallax ratio relationship (screening matching feature point group) The method can effectively eliminate the mismatching result of the feature point matching calculated based on the similarity of the pixel or the neighborhood pixel group, improve the correct rate of the matching; and help to avoid the matching failure due to the non-unique matching result, thereby Helps improve the matching rate and obtain a high-density feature point cloud.

15 to 18 illustrate a three-dimensional measurement system and a three-dimensional measurement method according to a third embodiment of the present invention.

Figure 15 shows a camera array CA in a three-dimensional measuring system according to a third embodiment of the present invention, wherein the first camera C ₁ , the second camera C ₂ and the third camera C ₃ have the same focal length and optical axes parallel to each other (Z direction), the first camera C ₁ , the second camera C ₂ and the third camera C ₃ are arranged in a triangle, and their optical centers O ₁ , O ₂ , O ₃ are arranged on the same plane perpendicular to the optical axis Wherein the optical centers O ₁ , O _{2 of} the first camera C ₁ and the second camera C ₂ are aligned in the X direction. The shift of the optical center of the second camera with respect to the optical center of the first camera in the X direction is D ₁ , and the shift of the optical center of the third camera with respect to the optical center of the second camera in the X direction is D ₂ .

Referring to the discussion above in connection with FIG. 1C, and as shown in FIG. 15, in the x-axis direction, a first parallax d ₁ = x _{1 -} x ₂ = D ₁ × f / generated between the first camera and the second camera Z, a second parallax generated between the second camera and the third camera d ₂ = x _{2 -} x ₃ = D ₂ × f / Z, satisfying d ₁ : d ₂ = D ₁ : D ₂ .

FIG. 16 schematically shows an example of the first image IM ₁ , the second image IM _{2 ,} and the third image IM ₃ obtained by the camera array shown in FIG. 15 , wherein the abscissa axis (u axis) of the image corresponds to the first The direction of the line where the camera and the third camera are located (x-axis direction). Ideally, as shown in FIG. 16, the same object point P[X, Y, Z] obtains image points P ₁ [u ₁ , v ₁ ], P ₂ [u] in the images IM ₁ , IM ₂ , and IM ₃ , respectively. ₂ , v ₂ ], P ₃ [u ₃ , v ₃ ], as shown in the imaginary image IM′, wherein corresponding image points in the images IM ₁ , IM ₃ have the same ordinate, ie v ₁ = v ₃ , And the abscissa difference s ₅ = u _{1 -} u ₂ = d ₁ / d _{p of the} image points P ₁ [u ₁ , v ₁ ], P ₂ [u ₂ , v ₂ ], the image point P ₂ [u ₂ , v ₂ ], P ₃ [u ₃ , v ₃ ] The abscissa difference s ₆ =u ₂ -u ₃ =d ₂ /d _p , satisfying s ₅ :s ₆ =d ₁ :d ₂ =D ₁ :D ₂ , where d _p is the side length of the pixel unit of the image sensor. Similarly, if the difference between the two sides of the equal sign is within a certain range, it can be considered as equivalent in the engineering sense. For example, it can be expressed as s ₅ :s ₆ =(1+k)*D ₁ :D ₂ , where k represents an allowable relative deviation, and |k|<<1,|k| can be, for example, 0.05 or 0.01.

The three-dimensional measurement method according to the present embodiment may include correction processing of the first image, the second image, and the third image, the correction processing such that the first image, the second image, and the third image correspond to the first camera, the second image The points of the optical axes of the camera and the third camera have the same abscissa and ordinate, and the abscissa directions of the first image, the second image, and the third image correspond to the optical connections of the first camera and the third camera The direction.

According to the camera array having the arrangement shown in FIG. 15, in the three-dimensional measurement method according to the third embodiment of the present invention, the process of screening the matching feature point group based on the parallax ratio relationship in the process s130 shown in FIG. 4 is implemented to include: The matching feature point group is filtered based on the following coordinate relationship: the abscissa of the feature point P ₁ [u ₁ , v ₁ ] in the first image and the abscissa of the feature point P ₂ [u ₂ , v ₂ ] in the second image The difference s ₅ =u ₁ -u ₂ and the abscissa of the feature point P ₂ [u ₂ , v ₂ ] in the second image and the feature point P ₃ [u ₃ , v ₃ ] in the third image The difference s ₆ = u _{2 -} u _{3 of the} abscissa satisfies s ₅ : s ₆ = D ₁ : D ₂ , and the feature points in the first image and the third image have the same ordinate, that is, v ₁ = v ₃ . The other processing of the three-dimensional measuring method according to the present embodiment is the same as the corresponding processing in the three-dimensional measuring method 100 described above with reference to FIG. 4, and details are not described herein again.

FIG. 17 shows an example of a process of screening a matching feature point group based on the above coordinate relationship, process 600. As shown in Figure 17, process 600 includes:

S610: respectively selecting a first feature point and a second feature point in the first image and the third image, wherein the ordinate of the second feature point is within a predetermined range with respect to the ordinate of the first feature point;

S620: Calculate a desired abscissa of the third feature point in the second image that satisfies the following relationship with the first feature point and the second feature point, such that the difference value s ₅ and the difference value s ₆ satisfy s ₅ :s ₆ =D ₁ : D ₂ ; and

S630: Search for the third feature point in the second image based on the expected abscissa.

The operation of selecting the feature points in the predetermined range with respect to the target coordinates in the processing s610 may refer to the operations described above in the processing s310 in the combination processing 300. Further, similar to the process 300, the processing s630 600 may be expected with respect to the abscissa is set tolerance range (horizontal dashed line ₂ of the feature point P in FIG. 16 Referring range), are not repeated here.

It should be noted that the ordinate range of the third feature point in the second image is not constrained in the process 600 as compared to the process 300 shown in FIG. 9, so that the matching feature is searched based on the expected abscissa in the second image. Point, the resulting matching result is more likely to be non-unique than the first embodiment and the second embodiment. In view of this, it is preferable to incorporate a feature point matching method based on the similarity calculation for a pixel or a neighborhood pixel group in the three-dimensional measurement method according to the third embodiment of the present invention.

FIG. 18 shows an example of a three-dimensional measurement method, a three-dimensional measurement method 700, according to a third embodiment of the present invention. As shown, the three-dimensional measurement method 700 includes:

S710: receiving a first image, a second image, and a third image from the first camera, the second camera, and the third camera, respectively;

S720: performing a correction process on the first image, the second image, and the third image, and as the above discussion, the correction process is such that the first image, the second image, and the third image correspond to the first camera, the second camera, and The points of the optical axis of the third camera have the same abscissa and ordinate, and the abscissa directions of the first image, the second image, and the third image correspond to the directions of the optical centers of the first camera and the third camera. ;

S730: extract feature points in the first image, the second image, and the third image;

S740: Matching feature points in the first image, the second image, and the third image, the matching includes:

S741: Filter matching matching feature point groups based on similarity calculation of pixel groups of pixels or neighborhoods;

S742: Filter matching feature point groups based on the following coordinate relationship: the abscissa of the feature points P ₁ [u ₁ , v ₁ ] in the first image and the cross-section of the feature points P ₂ [u ₂ , v ₂ ] in the second image The difference s _{5 of the} coordinates θ ₅ -u ₁ -u ₂ and the abscissa of the feature point P ₂ [u ₂ , v ₂ ] in the second image and the feature point P ₃ [u ₃ , v ₃ in the third image The difference s ₆ = v ₂ - v ₃ of the abscissa satisfies s ₅ : s ₆ = D ₁ : D ₂ , and the feature points in the first image and the third image have the same ordinate, ie v ₁ = v ₃ ;

S750: Calculate the three-dimensional coordinates of the object points corresponding to the matching feature point group.

Wherein, the process s720 may be a correction process implemented by using any correction method, including a correction process implemented by applying a certain correction matrix to the image.

In addition, it should be understood that the three-dimensional measurement method 700 is not limited to the processing using the similarity calculation based on the pixel group or the neighborhood of the pixel group in the processing s741, and may be replaced with other existing or later emerging others. Filter the processing of matching feature point group/feature point matching.

The process s742 can be implemented as, for example, the process 600 shown in FIG. 17, but is not limited thereto.

The three-dimensional measurement method 700 according to the third embodiment of the present invention can help improve the correct rate and matching rate of feature point matching by employing the process s742, and contribute to obtaining a higher density feature point cloud.

Although in the example shown in FIG. 18 above, the process of screening the matched feature point group satisfying the coordinate relationship s ₅ :s ₆ =D ₁ :D ₂ is set to perform filter matching based on the similarity of the pixel group to the pixel or the neighborhood. After the processing of the feature point group, but as discussed above in connection with the process s130 of the three-dimensional measurement method 100 shown in FIG. 4, the three-dimensional measurement method according to the third embodiment of the present invention may be reversed, that is, first The matched feature point groups satisfying the coordinate relationship s ₅ :s ₆ =D ₁ :D ₂ are selected, the candidate matching feature point groups are obtained, and then the similarity calculation is applied to the matched matching feature point groups, thereby further screening the matching feature point groups. . The three-dimensional measurement method thus realized can greatly reduce the matching process because the number of feature points entering the screening process based on the similarity calculation is greatly reduced due to the screening based on the coordinate relationship s ₅ :s ₆ =D ₁ :D ₂ Calculating the amount, as described above, also helps to improve the correct rate and matching rate of feature point matching, and helps to obtain a higher density feature point cloud.

Figure 19 shows a more arrangement of a camera array that can be used in a three dimensional measurement system in accordance with the present invention. As shown, the camera array can include three cameras arranged in an equilateral triangle, and can be further expanded to arrange more than three cameras forming a plurality of equilateral triangles, such as the honeycomb arrangement shown in the lower right corner of FIG. Furthermore, a camera array comprising three cameras arranged in a right triangle may be further expanded into a variety of other forms, such as rectangular (including square), T-shaped, cross-shaped, diagonal, and expanded forms in units of these shapes.

The three-dimensional measurement method according to the present invention can be implemented based on images from three cameras or more than three cameras in a camera array. Although the three-dimensional measurement system and the three-dimensional measurement method according to the first, second and third embodiments of the present invention are respectively described above, those skilled in the art should understand that these embodiments or features therein may be combined to form different technologies. Program. For example, in some other embodiments, the first, second, third, and fourth cameras are included in the camera array, and the three-dimensional measurement method according to the present invention may be for a set of images from the first, second, and third cameras. And the set of images from the first, third and fourth cameras respectively perform the feature point matching processing based on the parallax ratio relationship proposed by the present invention, and combine the matching results of the two sets of images to determine the last in the first sum. The set of feature points in the third camera is matched to calculate the spatial position of the corresponding object point. Such a technical solution still adopts the general inventive concept of the present invention, and the scope of the invention of the present application is intended to cover such technical solutions.

Next, a three-dimensional measurement system 10 in accordance with the present invention will be further described with reference to FIG. As shown in FIG. 3, the three-dimensional measurement system 10 may further include a projection unit 14 in addition to a different camera array CA. The projection unit 14 is for projecting a projection pattern onto a shooting area of the camera array CA, which can be captured by a camera in the camera array CA. The projection pattern can add more feature points in the shooting area, and in some applications, the feature points can be distributed more evenly or compensate for the lack of feature points in some areas.

The projected pattern can include dots, lines, or a combination thereof. The dots may also be formed as larger or have a specific shape after the size is increased, and the lines may be formed into a strip pattern having a wider or other shape characteristic after the size is increased. Preferably, the projection pattern includes a line, and the extending direction of the line is not parallel to the optical fiber connection direction of at least two of the first camera, the second camera, and the third camera, which helps provide more available for use according to the present invention. A feature point of the matching process based on the parallax ratio relationship in the three-dimensional measurement method of the invention.

The projected pattern can also be encoded by features such as color, intensity, shape, distribution, and the like. For an encoded projection pattern, the feature points with the same encoding in the image obtained by the camera are necessarily matching points. Added matching dimensions to improve match rate and match accuracy.

The projection unit 14 may be configured to include a light source and an optical element for forming a projection pattern based on illumination light from the light source, such as a diffractive optical element or grating or the like. The illumination light emitted by the light source includes light having wavelengths in an operating wavelength range of the first camera, the second camera, and the third camera, and may be monochromatic light or multi-color light, and may include visible light and/or Or non-visible light, such as infrared light. In some applications, the projection unit 14 can be configured to be able to adjust the projection direction to selectively project a projection pattern to different regions depending on different shooting scenes. In addition, the projection unit 14 may be configured to enable sequential single pattern projection or sequential multi-pattern projection, or to project different patterns according to different shooting scenes.

In some embodiments, the three-dimensional measurement system 10 can also include a sensor 15 for detecting at least a portion of the pattern features projected by the projection unit 14 to obtain additional information that can be used for three-dimensional measurements. For example, in some applications, the camera array CA operates at visible wavelengths and infrared wavelengths, the projection unit 14 projects a projected pattern at infrared wavelengths, and the sensor 15 is an infrared sensor or an infrared camera, at which point the information acquired by the camera array CA includes both visible light formation. The image information in turn includes a projected pattern that can provide more feature points for three-dimensional measurement based on binocular vision, while the projected pattern obtained by sensor 15 can be used for measurements based on other three-dimensional measurement techniques, such as based on structured light. measuring. In the three-dimensional measuring system 10 according to the present invention, the information obtained by the sensor 15 can be transmitted, for example, to the correcting unit 13, which can, for example, use three-dimensional measurement results obtained based on information from the sensor 15 and other three-dimensional measuring techniques for the camera. Calibration and/or correction of the array or its obtained images.

For the three-dimensional measurement system 10 according to the invention, it should be understood that it can be implemented as an integrated system or as a distributed system. For example, the camera array CA in the three-dimensional measurement system 10 can be mounted on one device, and the processing unit 11 can be implemented based on an internet server to be physically separate from the camera array CA. The projection unit 14 and the sensor 15 may be mounted together with the camera array CA or may be provided independently. The same is true for the control unit 12. In addition, the correcting unit 13 may be implemented together with the processing unit 11 by the same processor and associated memory or the like (which may be formed as part of the three-dimensional measuring device 20 represented by the dashed box in FIG. 3), or may be implemented separately. For example, the correction unit 13 can be implemented by a processor integrated with the camera array CA and an associated memory or the like.

Several application examples of the three-dimensional measurement system and measurement method according to the present invention will be described below.

[Application Example 1]

In this application example, the three-dimensional measuring system according to the present invention is implemented as a three-dimensional measuring device based on a mobile phone and an external camera module.

The camera module consists of three cameras of the same model. The centers of the three cameras are in a straight line, and the distances between the centers of adjacent cameras are equal. The optical axes of the cameras are parallel to each other and face the same, all perpendicular to the line where the camera center is located.

The camera module is connected to the phone via WiFi and/or data cable. The phone can control the camera's 3 cameras for simultaneous shooting (photo and / or video) and equal magnification.

Photos and/or videos captured by the camera module are transmitted to the phone via WiFi and/or data lines. The mobile phone corrects the image frames in the photo and/or video by the correction application. For example, a checkerboard placed in front of the camera module can be utilized during the calibration process, and the checkerboard mesh size is known. Corrections based on an auxiliary correction tool such as a checkerboard are known in the art and will not be described again. The method for correction in this application example is also not limited to this particular method.

After the calibration is completed, the camera module is used to capture the scenes and objects to be modeled. The mobile phone can be further integrated or externally connected with a projection module for projecting stripes onto the object to be photographed, and the direction of the stripes is not parallel to the direction of the center of the three cameras. A processing unit (consisting of a processing chip and a storage unit of the mobile phone) integrated in the mobile phone extracts feature points and feature areas common to the three camera photos. In addition to the feature points that are attached to the object on the image, the feature points also include new feature points and feature regions formed on the object by the stripes projected by the projection module. The matching feature point group having the image coordinate symmetry relationship and the similar attribute (the similarity attribute is calculated by the similarity calculation of the pixel and the neighborhood pixel group) is selected. A plurality of depth values are calculated based on each of the matched feature point groups, and an average of the depth values is taken. Then, the three-dimensional space coordinates of the object points corresponding to the feature point group are calculated and fused with the color information to form voxel information. Finally, a true color point cloud of the entire object or scene is displayed on the mobile phone or a three-dimensional model reconstructed based on the point cloud.

In a variant, the handset can transmit the received photos and/or videos from the camera module to the cloud server. The server can quickly correct the photos, extract feature points, match the corresponding matching feature point groups with image coordinate symmetry relationship and similar attributes, calculate the three-dimensional coordinates of the object points corresponding to the matched feature point groups, and form voxel information by color information fusion. The true color point cloud result or the three-dimensional model reconstructed based on the point cloud is transmitted back to the mobile phone for display.

The system can also be integrated with TOF to solve image-free features (such as large white walls) or occlusion problems. The fusion of the two point clouds not only ensures that large object targets can be sampled, but also is compatible with the surface details of the objects, and can form a three-dimensional point cloud with more comprehensive spatial sampling and higher resolution. For example, on the mobile phone side, the "forward camera + TOF" fusion 3d sampling module can scan the pixel level precision of the face, and the current TOF can only scan the approximate surface contour. The 3D module of "backward camera + TOF" is a shortcoming that complements the TOF system's limited projection distance. In addition, the three-dimensional point cloud formed by the TOF is transformed and projected by the space coordinate system to calculate the initial parallax of the corresponding sampling point on the image, which can accelerate the image matching process.

The basic process of image 3D measurement system and laser radar fusion is as follows: 1. TOF generates 3D point cloud; 2. Converts the point cloud 3D coordinates of lidar to reference camera coordinates through conversion between TOF coordinate system and reference camera coordinate system The three-dimensional coordinates of the corresponding points are obtained; 3. According to the projection equation of the camera, the three-dimensional coordinates of the corresponding sampling points in the reference camera coordinate system are converted into the two-dimensional coordinates and the parallax of the image; 4. The two-dimensional coordinates corresponding to the sampling points by referring to the camera image Coordinates and disparity find the initial position of the two-dimensional coordinates on other camera images; 5. Set the search neighborhood centered on the initial position, and find the corresponding points in other camera images with precision to the pixel level or sub-pixel level; The exact parallax of the matching point is calculated by searching for the matching result, and then the three-dimensional coordinates of the matching point are converted; 7. The image is partitioned according to the initial matching result, and the area clamped by the adjacent corresponding matching point is corresponding to the area to be matched; 8. Corresponding Matching feature points to be matched on multiple camera images in the to-be-matched area; 9. Outputting a point cloud of the TOF in the reference camera coordinate system and matching the image feature points The results of the integration of 3D point cloud.

The system can also be integrated with structured light to solve the problem of images without features such as large white walls. Structured light can form high density features on the surface of the object. Structured light is more flexible to use. It can only increase the feature points on the surface of the object (allowing the pattern arrangement of different positions to be repeated), and the three-dimensional point cloud can be generated by multi-camera system matching, and the coding pattern projection can also be generated (guarantee that the pattern coding or neighborhood pattern coding of each point is unique) The three-dimensional point cloud is calculated from the triangle original understanding between the camera and the projection device.

In addition, a combination of "camera + TOF + structured light" can be formed, which can form a high-density and ultra-high-resolution 3D point cloud, which can be used for applications such as VR/AR games or movies that require very realistic 3D virtual objects. in.

[Application Example 2]

In this application example, the three-dimensional measuring system according to the present invention is realized as a device for automatic driving of a vehicle.

The camera module is mounted on the front of the vehicle and includes three cameras of the same model. The centers of the three cameras are in a straight line with the same distance from the center of the camera. The optical axes of the cameras are parallel to each other and face the same, all perpendicular to the line where the camera center is located. The camera module is connected to the onboard computer via a data cable. The on-board computer can control the camera's 3 cameras for simultaneous shooting (photo and/or video) and equal magnification.

The photos and/or videos captured by the camera module are transmitted to the onboard computer via the data cable. The on-board computer generates a correction matrix from multiple images (image frames in a photo or video) taken at the same time. After the correction, the on-board computer extracts the feature points and feature regions shared by the images taken by the three cameras, selects the corresponding matching group with symmetric structure and similar properties, and selects the matched feature point groups with the image coordinate symmetry relationship and the similar attributes, and calculates The three-dimensional coordinates of the object points corresponding to the feature point group are matched, and the true color three-dimensional point cloud is output after being merged with the color information, and is pushed to the decision system of the vehicle automatic driving.

The system can also be integrated with Lidar to solve image-free features (such as large white walls) or occlusion problems. Lidar is suitable for spatial sampling of large objects, even in the case of images without features. However, due to the spatial sampling rate limitation, such as a power pole of several tens of meters, the leakage angle may exist when the opening angle is smaller than the spatial sampling angle resolution of the radar. The image system is more sensitive to various features of the object, including edge features. The fusion of the two point clouds not only ensures that large targets can be sampled, but also is compatible with the surface details of objects and the sampling of small objects, which can form a three-dimensional point cloud with more comprehensive spatial sampling and higher resolution. In addition, the three-dimensional point cloud formed by the laser radar is converted into the initial parallax of the corresponding sampling point on the image, which can accelerate the image matching process.

The basic process of image 3D measurement system and laser radar fusion is as follows: 1. Lidar generates 3D point cloud; 2. Converts lidar 3D coordinates of lidar into reference by conversion between lidar coordinate system and reference camera coordinate system The three-dimensional coordinates of the corresponding points in the camera coordinate system; 3. According to the projection equation of the camera, the three-dimensional coordinates of the corresponding sampling points in the reference camera coordinate system are converted into two-dimensional coordinates and parallax of the image; 4. by referring to the camera image corresponding to the sampling points Two-dimensional coordinates and parallax find the initial position of the two-dimensional coordinates on other camera images; 5. Set the search neighborhood centered on the initial position, and find the corresponding points in other camera images with precision to the pixel level or sub-pixel level; 6. Calculate the exact parallax of the matching point by searching for the matching result, and then convert the three-dimensional coordinates of the matching point; 7. According to the initial matching result, the image is partitioned, and the area clamped by the adjacent matching point is the corresponding matching area; Matching feature points to be matched on multiple camera images in the corresponding to-match region; 9. Output point cloud and graph of lidar in reference camera coordinate system The result of the fusion of a three-dimensional point cloud formed by feature point matching.

[Application Example 3]

In this application example, the three-dimensional measurement system according to the present invention is realized as a device for performing aerial photography based on a drone and an external camera module.

The camera module is placed on the airborne head of the drone, including five cameras of the same model. The five cameras are distributed in a cross shape with the center of the camera on the same plane. The distance between the centers of adjacent cameras in the longitudinal and lateral directions is equal. The optical axes of the cameras are parallel to each other and face the same, all perpendicular to the plane of the center of the camera. The camera module is connected to the drone's onboard computer via a data cable. The onboard computer can control the camera's 5 cameras for simultaneous shooting (photo and/or video) and equal magnification.

Photos and/or videos captured by the camera module are transmitted to the onboard computer via the data cable. The onboard computer generates a correction matrix from multiple images (image frames in a photo or video) taken at the same time. The image is corrected by a correction matrix. The onboard computer can display the feature points and feature areas shared by the images of the five cameras on the camera module in real time, extract corresponding matching feature point groups with image coordinate symmetry relationship and similar attributes, and calculate the three-dimensional object points corresponding to the matched feature point groups. Coordinates, combined with color information, output true color 3D point clouds.

It is easy to understand that the equal signs in the formulas in the present invention are all equal in engineering sense, and a certain deviation can be tolerated. That is, if the difference between the two sides of the equal sign is within a certain range, it can be considered equal. The range of the deviation is, for example, plus or minus 5%, or plus or minus 1%.

The above description is only a preferred embodiment of the present application and a description of the principles of the applied technology. It should be understood by those skilled in the art that the scope of the invention referred to in the present application is not limited to the specific combination of the above technical features, and should also be covered by the above technical features without departing from the inventive concept. Other technical solutions formed by any combination of their equivalent features. For example, the above features are combined with the technical features disclosed in the present application, but are not limited to the technical features having similar functions.

Claims (51)

A three-dimensional measurement method comprising: Receiving a first image, a second image, and a third image from the first camera, the second camera, and the third camera, respectively; Extracting feature points in the first image, the second image, and the third image, respectively; Matching feature points in the first image, the second image, and the third image, the matching including a first parallax d ₁ in the first direction generated between the first image and the second image based on the same object point Matching a set of matching feature points with a second disparity d ₂ generated in a second direction generated between the second image and the third image, wherein the first direction is an optical center connection that is non-perpendicular to the first camera and the second camera The direction of the line, the second direction being a direction that is not perpendicular to the optical line of the second camera and the third camera; Calculate the three-dimensional coordinates of the object points corresponding to the matching feature point group. The three-dimensional measuring method according to claim 1, wherein the first camera, the second camera, and the third camera have the same focal length and an optical axis parallel to each other, and the first camera, the second camera, and the third camera The optical centers are arranged on the same plane perpendicular to the optical axis, Wherein the first direction is an optical fiber connection direction of the first camera and the second camera, parallel to the plane, and the second direction is a connection direction of the second camera and the third camera optical center, parallel to the plane, Wherein the matching comprises screening the matched feature point group based on the following parallax ratio relationship: d ₁ :d ₂ =D ₁ :D ₂ , wherein D ₁ is the optical center of the first camera relative to the optical center of the second camera The offset in the first direction, D ₂ is the amount of shift of the optical center of the second camera relative to the optical center of the third camera in the second direction. The three-dimensional measuring method according to claim 2, wherein the optical centers of the first camera, the second camera, and the third camera are sequentially arranged on a straight line perpendicular to the optical axis, the first direction and the second direction Is the direction of the line; and The processing of screening the matched feature point group based on the parallax ratio relationship includes: screening the matched feature point group based on the following coordinate relationship: the abscissa of the feature point in the first image and the abscissa of the feature point in the second image s ₁ of the abscissa and the difference between the abscissa and the third feature points in the image feature points in the second image difference satisfies s _{2 s 1: s 2 = D 1} : D 2, and the first image The feature points in the second image and the third image have the same ordinate, and the direction of the abscissa corresponds to the direction of the straight line. The three-dimensional measuring method according to claim 3, further comprising: performing correction processing on the first image, the second image, and the third image such that the first image, the second image, and the third image correspond to the first camera, The points of the optical axes of the two cameras and the third camera have the same abscissa and ordinate, and the abscissa directions of the first image, the second image, and the third image all correspond to the direction of the straight line. The three-dimensional measurement method according to claim 3, wherein the processing of filtering the matched feature point group based on the coordinate relationship comprises: Selecting a first feature point and a second feature point in each of the first image, the second image, and the third image, the ordinates of the first feature point and the second feature point satisfying a target longitudinal The coordinates are within a predetermined range; Calculating a desired abscissa of a third feature point of the third one of the first image, the second image, and the third image that matches the first feature point and the second feature point, such that the difference s ₁ The difference s ₂ satisfies s ₁ : s ₂ = D ₁ : D ₂ ; The third feature point is searched for in the third person based on the expected position composed of the expected abscissa and the target ordinate of the third feature point. The three-dimensional measuring method according to claim 5, wherein the first feature point and the second feature in which the ordinate is the target ordinate are respectively selected in the first image, the second image, and the third image point. The three-dimensional measurement method according to claim 5, wherein the processing of searching for the third feature point based on the expected position further comprises: Setting a tolerance range including at least one of an abscissa tolerance range and an ordinate tolerance range; In the third one of the first image, the second image, and the third image, the third feature point is searched for in an area within the tolerance range with respect to the expected position. The three-dimensional measurement method according to claim 5, wherein the processing of filtering the matched feature point groups based on the coordinate relationship further comprises: Calculating a number of feature points whose ordinates are within a predetermined range with respect to the target ordinate in the first image, the second image, and the third image; It is determined that the third one of the first image, the second image, and the third image is the one having the largest number of feature points having a given ordinate in the first image, the second image, and the third image. The three-dimensional measuring method according to claim 5, wherein D ₁ : D ₂ = 1:1, and the processing of calculating the expected abscissa of the third feature point comprises: calculating the expected abscissa such that the first image The sum of the feature points in the feature point and the abscissa of the feature points in the third image is twice the abscissa of the feature points in the second image. The three-dimensional measuring method according to claim 2, wherein the optical centers of the first camera and the second camera are arranged on a first straight line perpendicular to the optical axis, and the optical centers of the second camera and the third camera are arranged perpendicular to An optical axis and a second line perpendicular to the first line, the first direction being a direction of a first line, and the second direction being a direction of a second line; The processing of screening the matched feature point group based on the parallax ratio relationship includes: screening the matched feature point group based on the following coordinate relationship: the abscissa of the feature point in the first image and the abscissa of the feature point in the second image s ₃ and the difference of the feature points in the second image and the ordinate is the ordinate of the feature point of the third image difference satisfies s _{4 s 3: s 4 = D 1} : D 2, a first image, and The feature points of the second image have the same ordinate, the feature points of the second image and the third image have the same abscissa, and the direction of the abscissa corresponds to the first direction, the ordinate The direction corresponds to the second direction. The three-dimensional measuring method according to claim 10, further comprising: correction processing of the first image, the second image, and the third image, the correction processing such that the first image, the second image, and the third image correspond to the first The points of the optical axes of the camera, the second camera, and the third camera have the same abscissa and ordinate, and the abscissa directions of the first image, the second image, and the third image correspond to the first direction, and the ordinate direction corresponds to The second direction. The three-dimensional measurement method according to claim 10, wherein the processing of filtering the matched feature point groups based on the coordinate relationship comprises: Selecting, in the first image and the second image, a first feature point and a second feature point, respectively, wherein the ordinates of the first feature point and the second feature point satisfy a predetermined range with respect to a target ordinate; Calculating a difference s ₃ between the abscissa of the first feature point and the abscissa of the second feature point; Calculate the difference s ₄ such that s ₃ : s ₄ = D ₁ : D ₂ ; Calculating a desired ordinate of the third feature point matching the first feature point and the second feature point in the third image, such that a difference between the ordinate of the second feature point and the expected ordinate of the third feature point is Calculating the second difference s ₄ ; and The third feature point is searched for in the third image based on the expected position formed by the expected ordinate of the third feature point and the abscissa of the second feature point. The three-dimensional measurement method according to claim 12, wherein the processing of searching for the third feature point based on the expected position further comprises: Set the tolerance range; In the third image, the third feature point is searched for in an area within the tolerance range with respect to the expected position. The three-dimensional measuring method according to claim 2, wherein the first direction and the second direction are both optical fiber connection directions of the first camera and the third camera. The three-dimensional measurement method according to claim 14, wherein the processing of screening the matched feature point group based on the parallax ratio relationship comprises: The matching feature point group is filtered based on the following coordinate relationship: a difference s ₅ between the abscissa of the feature point in the first image and the abscissa of the feature point in the second image and the abscissa of the feature point in the second image The difference s ₆ of the abscissa of the feature points in the third image satisfies s ₅ : s ₆ = D ₁ : D ₂ , and the feature points in the first image and the third image have the same ordinate, the horizontal The direction of the coordinates corresponds to the optical fiber connection direction of the first camera and the third camera. The three-dimensional measuring method according to claim 15, further comprising: performing correction processing on the first image, the second image, and the third image such that the first image, the second image, and the third image correspond to the first camera, The points of the optical axes of the two cameras and the third camera have the same abscissa and ordinate, and the abscissa directions of the first image, the second image, and the third image correspond to the optical centers of the first camera and the third camera The direction of the line. The three-dimensional measurement method according to claim 15, wherein the processing of filtering the matched feature point groups based on the coordinate relationship comprises: Selecting a first feature point and a second feature point in the first image and the third image, respectively, wherein the ordinate of the second feature point is within a predetermined range with respect to the ordinate of the first feature point; Calculating a desired abscissa of the third feature point in the second image that satisfies s ₅ :s ₆ =D ₁ :D ₂ relationship with the first feature point and the second feature point; A third feature point is searched for in the second image based on the expected abscissa. The three-dimensional measuring method according to claim 1, wherein the processing of matching the feature points in the first image, the second image, and the third image further comprises: matching the matching based on the parallax ratio relationship A set of feature points, applying a similarity calculation to a pixel or neighborhood pixel group to further filter the matched feature point set. The three-dimensional measuring method according to claim 18, wherein the similarity calculation is applied only to two or more feature point groups including the same feature point. The three-dimensional measurement method according to claim 1, wherein the matching the feature points in the first image, the second image, and the third image further comprises: applying a similarity calculation to the feature point or a neighborhood pixel group thereof Filter matching feature point groups; and The process of screening the matched feature point groups based on the parallax ratio relationship is applied to the matched feature point groups selected by the similarity calculation. The three-dimensional measuring method according to any one of claims 18 to 20, wherein the similarity calculation includes a sum of squares of grayscale differences by pixels, a sum of squares of zero-crossing average pixel grayscale differences, and pixel grayscale differences The calculation of at least one of the absolute value sum, the absolute value sum of the zero-crossing average pixel gradation difference, the normalized cross-correlation between the neighborhoods, and the zero-crossing average normalized cross-correlation. The three-dimensional measuring method according to claim 2, further comprising: generating a correction matrix for applying the first camera, the second camera, and the like to be received later based on the first image, the second image, and the third image The image of the third camera, the process of generating the correction matrix includes: Extracting feature points in the first image, the second image, and the third image, respectively; Matching feature points in the first image, the second image, and the third image to obtain a plurality of matching feature point groups; Using the coordinates of the feature points in each image in the matched feature point group, the overdetermined equations are established according to the parallax ratio relationship satisfied between the feature points in the matching feature point group after the correction matrix is applied to each image; The overdetermined system of equations is solved to obtain a correction matrix. The three-dimensional measurement method according to claim 1, wherein the calculating the three-dimensional coordinates of the object points corresponding to the respective feature point groups based on the matched feature point groups comprises: calculating the depth based on at least two pairs of the feature points in one of the matched feature point groups And taking the average value of the depth value as the depth value of the object point corresponding to the matching feature point group. The three-dimensional measuring method according to claim 1, further comprising: projecting a pattern to a photographing area of the camera with light of a wavelength within an operating wavelength range of the first camera, the second camera, and the third camera. The three-dimensional measuring method according to claim 24, wherein the pattern includes a stripe whose direction is not parallel to an optical center line of at least two of the first camera, the second camera, and the third camera. A three-dimensional measuring apparatus comprising: a processor; and a memory storing program instructions, wherein when the program instructions are executed by the processor, the processor is caused to perform the following operations: Receiving the first image, the second image, and the third image; Extracting feature points in the first image, the second image, and the third image, respectively; Matching feature points in the first image, the second image, and the third image, the matching comprising: filtering the matched feature point group based on the following coordinate relationship: an abscissa of the feature point in the first image and a feature in the second image The difference between the abscissa of the point and the abscissa of the feature point in the second image and the abscissa of the feature point in the third image are in a predetermined proportional relationship, and the first image and the third image The feature points have the same ordinate; Calculate the three-dimensional coordinates of the object points corresponding to the matching feature point group. The three-dimensional measuring apparatus according to claim 26, wherein the coordinate relationship further comprises that the feature point in the second image has the same ordinate as the feature point in the first image and the third image. The three-dimensional measuring apparatus according to claim 27, wherein said program instructions, when executed, cause said processor to perform an operation of filtering matching feature point groups based on said coordinate relationship by: Selecting a first feature point and a second feature point in each of the first image, the second image, and the third image, the ordinates of the first feature point and the second feature point satisfying a target longitudinal The coordinates are within a predetermined range; Calculating a desired abscissa of a third feature point of the third one of the first image, the second image, and the third image that matches the first feature point and the second feature point, such that the feature point in the first image a difference between a horizontal coordinate and an abscissa of a feature point in the second image and a difference between an abscissa of the feature point in the second image and an abscissa of the feature point in the third image; and The third feature point is searched for in the third person based on the expected position composed of the expected abscissa and the target ordinate of the third feature point. The three-dimensional measuring apparatus according to claim 28, wherein the first feature point and the second feature whose ordinate is the target ordinate are respectively selected in the first image, the second image, and the third image point. The three-dimensional measuring apparatus according to claim 28, wherein said program instructions, when executed, cause said processor to perform an operation of searching for a third feature point based on said expected position by: Setting a tolerance range including at least one of an abscissa tolerance range and an ordinate tolerance range; In the third one of the first image, the second image, and the third image, the third feature point is searched for in an area within the tolerance range with respect to the expected position. A three-dimensional measuring device for use with a camera array for performing three-dimensional measurement, the camera array comprising at least a first camera, a second camera, and a third camera, the three-dimensional measuring device comprising: A processing unit that receives the first image, the second image, and the third image from the first camera, the second camera, and the third camera, respectively, and is configured to perform the following processing: Extracting feature points in the first image, the second image, and the third image, respectively; Matching feature points in the first image, the second image, and the third image, the matching including a first parallax d ₁ in the first direction generated between the first image and the second image based on the same object point Matching a set of matching feature points with a second disparity d ₂ generated in a second direction generated between the second image and the third image, wherein the first direction is an optical center connection that is non-perpendicular to the first camera and the second camera The direction of the line, the second direction being a direction that is not perpendicular to the optical line of the second camera and the third camera; Calculate the three-dimensional coordinates of the object points corresponding to the matching feature point group. The three-dimensional measuring apparatus according to claim 31, wherein said first camera, said second camera, and said third camera have the same focal length and an optical axis parallel to each other, and the first camera, the second camera, and the third camera The optical centers are arranged on a same plane perpendicular to the optical axis, the first direction and the second direction being parallel to the plane, Wherein the matching comprises screening the matched feature point group based on the following parallax ratio relationship: d ₁ :d ₂ =D ₁ :D ₂ , wherein D ₁ is the optical center of the first camera relative to the optical center of the second camera The offset in the first direction, D ₂ is the amount of shift of the optical center of the second camera relative to the optical center of the third camera in the second direction. The three-dimensional measuring apparatus according to claim 32, wherein optical centers of said first camera, said second camera, and said third camera are sequentially arranged on a straight line perpendicular to an optical axis, said first direction and said second direction Is the direction of the line; and The processing unit is configured to filter the matched feature point groups based on the parallax ratio relationship by: The matching feature point group is filtered based on the following coordinate relationship: a difference between the abscissa of the feature point in the first image and the abscissa of the feature point in the second image, and the abscissa and the third point of the feature point in the second image The difference in the abscissa of the feature points in the image satisfies s ₁ : s ₂ = D ₁ : D ₂ , and the feature points in the first image, the second image, and the third image have the same ordinate, the horizontal The direction of the coordinates corresponds to the direction of the line. The three-dimensional measuring apparatus according to claim 33, further comprising: a correction unit that generates a correction matrix based on images from the first camera, the second camera, and the third camera and provides the correction matrix to the processing unit, the correction matrix being applied to the first image, the second by the processing unit And the third image, the points corresponding to the optical axes of the first camera, the second camera, and the third camera in the first image, the second image, and the third image have the same abscissa and ordinate, and the first The abscissa directions of the image, the second image, and the third image all correspond to the direction of the straight line. The three-dimensional measuring apparatus according to claim 33, wherein the processing unit is configured to filter the matched feature point group based on the coordinate relationship by the following processing: Selecting a first feature point and a second feature point in each of the first image, the second image, and the third image, the ordinates of the first feature point and the second feature point satisfying a target longitudinal The coordinates are within a predetermined range; Calculating a desired abscissa of a third feature point of the third one of the first image, the second image, and the third image that matches the first feature point and the second feature point, such that the difference s ₁ The difference s ₂ satisfies s ₁ : s ₂ = D ₁ : D ₂ ; The third feature point is searched for in the third person based on the expected position composed of the expected abscissa and the target ordinate of the third feature point. The three-dimensional measuring apparatus according to claim 35, wherein the processing unit is configured to search for the third feature point based on the expected position by: Setting a tolerance range including at least one of an abscissa tolerance range and an ordinate tolerance range; In the third one of the first image, the second image, and the third image, the third feature point is searched for in an area within the tolerance range with respect to the expected position. The three-dimensional measuring apparatus according to claim 32, wherein the optical centers of the first camera and the second camera are arranged on a first straight line perpendicular to the optical axis, and the optical centers of the second camera and the third camera are arranged perpendicular to An optical axis and a second line perpendicular to the first line, the first direction being a direction of a first line, and the second direction being a direction of a second line; The processing unit is configured to filter the matched feature point groups based on the parallax ratio relationship by: Filtering the matched feature point groups based on the following coordinate relationship: a difference s ₃ between the abscissa of the feature points in the first image and the abscissa of the feature points in the second image and the ordinate of the feature points in the second image The difference s ₄ from the ordinate of the feature point in the third image satisfies s ₃ : s ₄ = D ₁ : D ₂ , the feature points of the first image and the second image have the same ordinate, the second image And the feature points of the third image have the same abscissa, the direction of the abscissa corresponds to the first direction, and the direction of the ordinate corresponds to the second direction. The three-dimensional measuring apparatus according to claim 37, further comprising: a correction unit that generates a correction matrix based on images from the first camera, the second camera, and the third camera and provides the correction matrix to the processing unit, the correction matrix being applied to the first image, the second by the processing unit And the third image, the points corresponding to the optical axes of the first camera, the second camera, and the third camera in the first image, the second image, and the third image have the same abscissa and ordinate, and the first The abscissa directions of the image, the second image, and the third image all correspond to the direction of the straight line. The three-dimensional measuring apparatus according to claim 37, wherein said processing unit is configured to filter the matched feature point group based on said coordinate relationship by: Selecting, in the first image and the second image, a first feature point and a second feature point, respectively, wherein the ordinates of the first feature point and the second feature point satisfy a predetermined range with respect to a target ordinate; Calculating a difference s ₃ between the abscissa of the first feature point and the abscissa of the second feature point; Calculate the difference s ₄ such that s ₃ : s ₄ = D ₁ : D ₂ ; Calculating a desired ordinate of the third feature point matching the first feature point and the second feature point in the third image, such that a difference between the ordinate of the second feature point and the expected ordinate of the third feature point is Calculating the second difference s ₄ ; and The third feature point is searched for in the third image based on the expected position formed by the expected ordinate of the third feature point and the abscissa of the second feature point. The three-dimensional measuring apparatus according to claim 39, wherein said processing unit is configured to search for a third feature point based on said expected position by: Setting a tolerance range including at least one of an abscissa tolerance range and an ordinate tolerance range; In the third image, the third feature point is searched for in an area within the tolerance range with respect to the expected position. The three-dimensional measuring apparatus according to claim 32, wherein said first direction and said second direction are both optical fiber connection directions of the first camera and the third camera; The processing unit is configured to filter the matched feature point groups based on the parallax ratio relationship by: Filtering the matched feature point groups based on the following coordinate relationship: a difference s ₅ between the abscissa of the feature points in the first image and the abscissa of the feature points in the second image and the abscissa of the feature points in the second image The difference s ₆ from the abscissa of the feature point in the third image satisfies s ₅ : s ₆ = D ₁ : D ₂ , the feature points in the first image and the third image have the same ordinate, The direction of the abscissa corresponds to the direction of the optical fiber connection of the first camera and the third camera. The three-dimensional measuring apparatus according to claim 41, further comprising: a correction unit that generates a correction matrix based on images from the first camera, the second camera, and the third camera and provides the correction matrix to the processing unit, the correction matrix being applied to the first image, the second by the processing unit And the third image, the points corresponding to the optical axes of the first camera, the second camera, and the third camera in the first image, the second image, and the third image have the same abscissa and ordinate, and the first The abscissa directions of the image, the second image, and the third image all correspond to the directions of the optical centers of the first camera and the third camera. The three-dimensional measuring apparatus according to claim 37, wherein the processing unit is configured to filter the matched feature point group based on the coordinate relationship by the following processing: Selecting a first feature point and a second feature point in the first image and the third image, respectively, wherein the ordinate of the second feature point is within a predetermined range with respect to the ordinate of the first feature point; Calculating a desired abscissa of the third feature point in the second image that satisfies s ₅ :s ₆ =D ₁ :D ₂ relationship with the first feature point and the second feature point; A third feature point is searched for in the second image based on the expected abscissa. The three-dimensional measuring apparatus according to claim 31, wherein the processing unit is further configured to: apply a similarity calculation to a pixel or a neighborhood pixel group for the matched feature point group selected based on the parallax ratio relationship To further filter the matched feature point groups. The three-dimensional measuring apparatus according to claim 31, wherein the processing unit is further configured to: apply a similarity calculation to the feature point or its neighboring pixel group to filter the matched feature point group; and filter out the similarity calculation The matching feature point group performs the process of filtering the matching feature point group based on the parallax ratio relationship. The three-dimensional measuring apparatus according to claim 34, 38 or 42, wherein the correcting unit is configured to generate a correction matrix by the following processing: Extracting feature points in the first image, the second image, and the third image, respectively; Matching feature points in the first image, the second image, and the third image to obtain a plurality of matching feature point groups; Using the coordinates of the feature points in the matched feature point groups in each image, the overdetermined equations are established according to the coordinate relationship that is satisfied between the feature points in the matched feature point groups after the correction matrix is applied to each image; The overdetermined system of equations is solved to obtain a correction matrix. A three-dimensional measurement system based on a parallax ratio relationship, comprising: a camera array including at least a first camera, a second camera, and a third camera; A three-dimensional measuring device according to any one of claims 26-46. The three-dimensional measurement system according to claim 47, wherein said first camera, said second camera, and said third camera have the same focal length and an optical axis parallel to each other, and the first camera, the second camera, and the third camera The optical center is disposed on a same plane perpendicular to the optical axis, the three-dimensional measurement system further comprising: a control unit coupled to the camera array and in communication, and configured to control the first camera, the second camera, and the third camera to synchronize Collect images. The three-dimensional measurement system of claim 48, wherein the control unit is further configured to control multiple zooms of the first camera, the second camera, and the third camera. A three-dimensional measurement system according to claim 49, further comprising: a projection unit including a light source and an optical element for forming a projection pattern based on illumination light from the light source, the illumination light comprising the first camera, Light of a wavelength within a working wavelength range of the two cameras and the third camera, and the projection unit projects the projected pattern toward a photographing area of the camera array. The three-dimensional measurement system of claim 50, wherein the projection pattern comprises a stripe, and a direction of the stripe is not connected to an optical center of at least two of the first camera, the second camera, and the third camera parallel.

Images