JP2008304269A - Information processing apparatus, information processing method, and computer program - Google Patents

Abstract

A configuration for calculating a three-dimensional position of a feature point included in a captured image based on an acquired image of a camera is realized.
Generating camera position and orientation information having an existence probability distribution according to a normal distribution based on an acquired image of the camera, and generating a feature point tracking error distribution according to the normal distribution based on the acquired image of the camera Then, the existence probability distribution of the feature points in the three-dimensional space is calculated by applying the camera position and orientation information having the existence probability distribution according to the normal distribution and the feature point tracking error distribution information according to the normal distribution. With this configuration, the three-dimensional position of the feature point included in the acquired image of the camera can be analyzed more accurately.
[Selection] Figure 1

Description

  The present invention relates to an information processing apparatus, an information processing method, and a computer program. More specifically, the present invention relates to an information processing apparatus, an information processing method, and a computer program for obtaining a three-dimensional position such as a feature point included in an image based on an image photographed by a camera.

  Processing for analyzing a photographed image of a camera and obtaining a three-dimensional position of an object included in the photographed image is used in various fields. For example, an agent (moving body) such as a robot equipped with a camera analyzes the captured image of the camera, observes the moving environment, and moves while grasping the environment around the agent according to the observation situation, It is used for environment map construction processing for creating a map of the surrounding environment (environment map) based on

  By tracking the same feature point from a plurality of frames of a moving image captured by the camera, the camera position and the feature point position can be calculated. Non-Patent Document 1 discloses a method of tracking feature point positions in all frames and calculating the positions of feature points and camera positions by batch processing after data of all frames is obtained.

  However, the method disclosed in Non-Patent Document 1 has a problem that the existence probability distribution of the feature point positions cannot be obtained, and a correct feature point position can be obtained only from the judgment of whether or not the feature points are converged, that is, the analysis result. It is difficult to determine whether or not In addition, there is a problem that correct analysis becomes difficult unless all the feature points to be tracked are shown in all frames.

  Non-Patent Document 2 proposes a method for estimating a three-dimensional position of a feature point using a particle filter. The three-dimensional position detection process to which the particle filter is applied is an identification process technique that approximates the state distribution using a large number of particles (sampling points).

In the three-dimensional position detection process using this particle filter, the existence probability distribution of a new feature point is calculated, but the information used to calculate the existence probability distribution is the frame information before the frame where the new feature point appears. This is camera position / orientation information and feature point tracking error, and there is a problem that information from the appearance frame of the new feature point is not applied and the correct three-dimensional position of the feature point is not obtained.
“Shape and motion from image stream under orthography: a factorization method”, C.I. Tomasi and T. Kanade, International Journal of Computer Vision, Volume 9, Number 2, pp. 137-154, (1992). "Real-time Simulaneous Localization and Mapping with a Single Camera", Andrew J. et al. Davison, Computer Vision Proceedings. 9th IEEE International Conference, (2003).

  The present invention has been made in view of the above-described problems, and based on the three-dimensional position and orientation existence probability distribution of the camera obtained based on the image photographed by the camera and the tracking result of the feature point, An object of the present invention is to provide an information processing apparatus, an information processing method, and a computer program capable of accurately obtaining a three-dimensional position existence probability distribution.

The first aspect of the present invention is:
An information processing device that calculates a three-dimensional existence probability distribution of feature points included in a captured image of a camera,
A camera for taking images,
A camera position and orientation detection unit for detecting the position and orientation of the camera based on an image acquired by the camera;
A feature point tracking unit that tracks feature points extracted from images acquired by the camera;
A feature point 3D position detection unit that calculates a 3D existence probability distribution of feature points by inputting the camera position and orientation information detected by the camera position and orientation detection unit and the feature point trajectory information acquired by the feature point tracking unit Have
The camera has a pinhole camera model configuration that is configured to sequentially shoot a plurality of image frames by changing at least one of a position and a posture,
The camera position and orientation detection unit is configured to generate camera position and orientation information having an existence probability distribution according to a normal distribution,
The feature point tracking unit is configured to generate a feature point tracking error distribution according to a normal distribution,
The three-dimensional position detection unit includes camera position and orientation information having an existence probability distribution according to a normal distribution generated by the camera position and orientation detection unit, and a feature point tracking error according to a normal distribution generated by the feature point tracking unit. An information processing apparatus is characterized in that the distribution probability is applied to calculate a presence probability distribution of feature points in a three-dimensional space.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the camera position / orientation detection unit generates information having an existence probability distribution according to a normal distribution for each of the camera position (c) and the camera attitude (R). The feature point tracking unit generates a feature point tracking error distribution according to a normal distribution corresponding to the feature point position (m) in the image, and the three-dimensional position detection unit The configuration is such that the presence probability distribution in the three-dimensional space of the feature points is calculated by inputting the camera position (c) having the presence probability distribution according to the distribution, the camera posture (R), and the feature point position (m). Features.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the three-dimensional position detection unit includes a camera position (c), a camera posture (R), and a feature point position (m) having an existence probability distribution according to a normal distribution. And the existence probability distribution in the three-dimensional space of the feature points using linear approximation by the first-order differentiation of the three-dimensional position calculation formula of the feature points set as input parameters.

  Furthermore, in one embodiment of the information processing apparatus according to the present invention, the feature point tracking unit generates a feature point tracking error distribution reflecting a quantization error determined based on a pixel setting of an image sensor of the camera. It is characterized by being.

  Furthermore, in one embodiment of the information processing apparatus of the present invention, the three-dimensional position detection unit is configured to detect a ray connecting the camera and the feature point in a plurality of images including the same feature point included in the photographed image of the camera. A sampling unit that selects data obtained based on an image having an inner angle larger than a predetermined threshold angle as an angle (inner angle) formed as data to be applied to feature point existence probability distribution calculation, and is selected by the sampling unit And a presence probability distribution calculating unit that calculates a presence probability distribution of feature points in a three-dimensional space by applying data.

  Furthermore, in one embodiment of the information processing apparatus according to the present invention, the camera position / orientation detection unit detects a position of the feature point in the image input from the camera and a position / orientation of the camera by using SLAM (simultaneous localization and). (mapping) is executed.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the feature point tracking unit performs window matching with the local region image of the previous frame captured by the camera, and performs frame matching between the frames of the local region (feature points). The present invention is characterized in that a process for obtaining a movement amount is executed.

Furthermore, the second aspect of the present invention provides
In the information processing apparatus, an information processing method for calculating a three-dimensional existence probability distribution of feature points included in a captured image of a camera,
An image capturing step in which the camera captures an image;
A camera position and orientation detection unit, wherein a camera position and orientation detection unit detects the position and orientation of the camera based on an image acquired by the camera;
A feature point tracking unit for tracking a feature point extracted from an image acquired by the camera;
The three-dimensional position detection unit inputs the camera position and orientation information detected by the camera position and orientation detection unit and the feature point trajectory information acquired by the feature point tracking unit, and calculates a three-dimensional existence probability distribution of the feature points. A feature point three-dimensional position detection step,
The image capturing step is a step of sequentially capturing a plurality of image frames by changing at least one of a position and a posture with a camera having a pinhole camera model configuration,
The camera position and orientation detection step is a step of generating camera position and orientation information having an existence probability distribution according to a normal distribution,
The feature point tracking step is a step of generating a feature point tracking error distribution according to a normal distribution,
The three-dimensional position detection step includes camera position and orientation information having an existence probability distribution according to a normal distribution generated by the camera position and orientation detection unit, and a feature point tracking error according to a normal distribution generated by the feature point tracking unit. An information processing method is characterized in that it is a step of calculating an existence probability distribution of a feature point in a three-dimensional space by applying distribution information.

  Furthermore, in an embodiment of the information processing method of the present invention, the camera position / orientation detection step generates information having an existence probability distribution according to a normal distribution for each of the camera position (c) and the camera attitude (R). The feature point tracking step is a step of generating a feature point tracking error distribution according to a normal distribution corresponding to the feature point position (m) in the image, and the three-dimensional position detecting step is a normal step. The step of calculating the existence probability distribution of the feature points in the three-dimensional space with the camera position (c) having the existence probability distribution according to the distribution, the camera posture (R), and the feature point position (m) as inputs. Features.

  Furthermore, in an embodiment of the information processing method of the present invention, the three-dimensional position detecting step includes a camera position (c) having a presence probability distribution according to a normal distribution, a camera posture (R), and a feature point position (m). Is a step of calculating the existence probability distribution of the feature points in the three-dimensional space using linear approximation by first-order differentiation of the feature point three-dimensional position calculation formula set as an input parameter.

  Furthermore, in an embodiment of the information processing method of the present invention, the feature point tracking step includes a step of generating a feature point tracking error distribution reflecting a quantization error determined based on a pixel setting of an image sensor of the camera. It is characterized by being.

  Furthermore, in an embodiment of the information processing method of the present invention, the three-dimensional position detection step includes: a light ray connecting the camera and the feature point in a plurality of images including the same feature point included in the photographed image of the camera. A sampling step for selecting data obtained based on an image having an inner angle that is larger than a predetermined threshold angle as the data to be applied to the calculation of the feature point existence probability distribution; An existence probability distribution calculating step of calculating an existence probability distribution of the feature points in the three-dimensional space by applying the data.

  Furthermore, in one embodiment of the information processing method of the present invention, the camera position / orientation detection step includes SLAM (simultaneous localization and and) that detects the position of the feature point in the image input from the camera and the position and orientation of the camera. (mapping) is performed.

  Furthermore, in one embodiment of the information processing method of the present invention, the feature point tracking step performs window matching with the local region image of the previous frame photographed by the camera to perform a local region (feature point) between frames. It is a step for executing a process for obtaining a movement amount.

Furthermore, the third aspect of the present invention provides
A computer program that causes an information processing device to calculate a three-dimensional existence probability distribution of feature points included in a captured image of a camera,
An image capturing step for causing the camera to capture an image;
A camera position and orientation detection step for causing the camera position and orientation detection unit to detect the position and orientation of the camera based on the image acquired by the camera;
A feature point tracking step for causing the feature point tracking unit to track a feature point extracted from the image acquired by the camera;
The 3D position detection unit inputs the camera position and orientation information detected by the camera position and orientation detection unit and the feature point trajectory information acquired by the feature point tracking unit, and calculates a 3D existence probability distribution of the feature points. A feature point three-dimensional position detection step,
The image capturing step is a step of sequentially capturing a plurality of image frames by changing at least one of a position and a posture with a camera having a pinhole camera model configuration,
The camera position and orientation detection step is a step of generating camera position and orientation information having an existence probability distribution according to a normal distribution,
The feature point tracking step is a step of generating a feature point tracking error distribution according to a normal distribution,
The three-dimensional position detection step includes camera position and orientation information having an existence probability distribution according to a normal distribution generated by the camera position and orientation detection unit, and a feature point tracking error according to a normal distribution generated by the feature point tracking unit. In the computer program, the distribution information is applied to calculate a presence probability distribution of the feature points in the three-dimensional space.

  The computer program of the present invention is, for example, a computer program that can be provided by a storage medium or a communication medium provided in a computer-readable format to a general-purpose computer system that can execute various program codes. . By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of the embodiment of the present invention, the camera position and orientation information having the existence probability distribution according to the normal distribution is generated based on the image acquired by the camera, and the normal distribution is generated based on the acquired image of the camera. A feature point tracking error distribution is generated, and the camera position and orientation information having the existence probability distribution according to these normal distributions and the feature point tracking error distribution information according to the normal distributions are applied to the three-dimensional space of the feature points The existence probability distribution at is calculated. According to this configuration, it is possible to more accurately analyze the three-dimensional position of the feature point included in the acquired image of the camera.

  Details of an information processing apparatus, an information processing method, and a computer program according to embodiments of the present invention will be described below with reference to the drawings.

  A configuration of an information processing apparatus according to an embodiment of the present invention will be described with reference to FIG. The information processing apparatus 100 according to the present invention generates and outputs a feature point existence probability distribution 107 that is three-dimensional position information of feature points such as objects included in a captured image based on image data captured by the camera 101.

  As shown in FIG. 1, an information processing apparatus according to an embodiment of the present invention includes a camera 101 that captures the surrounding environment, and a camera position and orientation detection unit 102 that inputs a captured image of the camera 101 and estimates the position and orientation of the camera. A feature point tracking unit 103 that inputs a captured image of the camera 101 and executes a tracking process of a feature point included in the captured image, and a camera that stores camera position and orientation information detected by the camera position and orientation detection unit 102 Position / orientation storage unit 104, feature point trajectory information storage unit 105 that stores trajectory information of feature points detected by the feature point tracking unit 103, camera position / orientation information stored in the camera position / orientation storage unit 104, and feature point trajectory The feature point trajectory information stored in the information storage unit 105 is input, and the feature point existence probability distribution 10 as the three-dimensional position information of the feature points based on the input information. And a feature point three-dimensional position detector 107 which generates a.

The camera 101 is mounted on a moving robot, for example, or is appropriately moved by a user (person). That is, the position and orientation of the camera 101 are not fixed, and a captured image from the changed position and orientation, that is, a moving image consisting of continuous frames is input. The camera 101 is suitable for the pinhole camera model. The pinhole camera model represents a projective transformation between the position of a point in a three-dimensional space and the pixel position on the camera image plane, and is represented by the following equation.

  The meaning of the above formula will be described with reference to FIGS. The above formula is obtained by calculating the pixel position 212 on the camera image plane of the point (m) of the object 211 included in the captured image 210 of the camera, that is, the position represented by the camera coordinate system, and the three-dimensional object 200 in the world coordinate system. It is an expression showing the correspondence with the position (M) 201.

  A pixel position 212 on the camera image plane is expressed by a camera coordinate system. The camera seat cost system is a coordinate system in which the focal point of the camera is the origin C, the image plane is a two-dimensional plane of Xc and Yc, and the depth is Zc, and the origin C is moved by the movement of the camera.

  On the other hand, the three-dimensional position (M) 201 of the object 200 is indicated by a world coordinate system composed of XYZ three axes having an origin O that does not move due to the movement of the camera. An expression indicating the correspondence between the positions of the objects in the different coordinate systems is defined as the above-described pinhole camera model.

Λ, A, Cw, and Rw included in this equation are as shown in FIG.
λ: normalization parameter A: camera internal parameter,
Cw: camera position,
Rw: camera rotation matrix,
Means.

The camera internal parameter A includes the following values.
f: Focal length θ: Image axis orthogonality (ideal value is 90 °)
ku: scale of the vertical axis (conversion from the 3D position scale to the 2D image scale)
kv: Scale on the horizontal axis (conversion from 3D position scale to 2D image scale)
(U0, v0): Image center position

  The camera position / orientation detection unit 102 shown in FIG. 1 inputs from the camera 101 using this pinhole camera model, that is, an expression representing the projective transformation between the position of the point in the three-dimensional space and the pixel position on the camera image plane. Using the feature point in the video, the position of the feature point and the camera position are corrected every frame, and the estimated position and orientation information of the camera represented by the world coordinate system determined by the camera position and orientation detection unit 102 A position (Cw) and a camera rotation matrix (Rw) are generated and stored in the camera position and orientation storage unit 104.

  This process is performed in the above-mentioned paper [Andrew J. Davison, “Real-time simulated localization and mapping with a single camera”, Proceedings of the 9th International Conference on Computer Vision, Ninth, 200.

  The camera position / orientation detection unit 102 generates camera position / orientation information having an existence probability distribution according to a normal distribution for each of the camera position (Cw) and the camera rotation matrix (Rw), and stores the camera position / orientation memory. Stored in the unit 104.

  It should be noted that the creation of a surrounding map (environment map) in conjunction with the self-pose (position and orientation) identification (localization) executed as confirmation of the pose (position and orientation) of the camera position and direction, that is, the environment map The process of creating is called SLAM (simultaneous localization and mapping). The camera position / orientation detection unit 102 may perform a mapping process based on the feature point information included in the photographed image along with the camera position / orientation by using this SLAM technique, and store these results in the camera position / orientation storage unit 104. Good.

  The captured image of the camera 101 is also input to the feature point tracking unit 103. The detailed configuration and processing of the feature point tracking unit 103 will be described with reference to FIG. As shown in FIG. 4, the feature point tracking unit 103 includes a feature point tracking unit 301, a feature point extraction unit 302, and a preceding frame tracking information storage unit 303 that stores tracking information of the preceding frame.

  The feature point tracking unit 301 obtains a movement amount between frames of a local region (feature point) selected from the frame image acquired by the camera 101. For example, the feature point tracking unit 301 performs window matching with the local region image of the previous frame to obtain the amount of movement of the local region (feature point) between frames. Note that the calculated tracking result includes an error in the amount of movement, and processing is performed in consideration of this error. That is, in this example, an image is used in the movement amount calculation, and processing is performed in consideration of a quantization error due to the pixel position.

The feature point tracking unit 301 generates a feature point tracking error distribution reflecting the quantization error determined based on the pixel setting of the image sensor of the camera 101. Using a constant error distribution for the feature point tracking result as an error distribution Σ, a feature point tracking result having an error distribution is generated. The feature point tracking unit 301 generates a feature point tracking error distribution according to the normal distribution.

  The feature point tracking result obtained by the feature point tracking unit 301 is output as feature point tracking information 321 and recorded in the feature point tracking information storage unit 105 shown in FIG. Further, the feature point tracking result is input to the feature point extracting unit 302 and also stored in the preceding frame tracking information storage unit 303 to be applied to the feature point tracking process for the subsequent frame.

  The feature point extraction unit 302 extracts an image area where the feature point has not been tracked from the feature point tracking result input from the feature point tracking unit 301, and creates a new feature point using a Harris Corner Detector (Harris Corner Detector). Execute the extraction process. That is, a new feature point is extracted from an image area having no feature point, and the extracted feature point information is stored in the preceding frame tracking information storage unit 303. It is preferable that the minimum image area section for detecting feature points is set in advance, and one or more feature points are always detected in the section area.

  A feature point extraction process using a Harris corner detector will be described with reference to FIG. The data processing unit of the information processing apparatus generates a plurality of Harris corner images 410 to 412 and Laplacian images 420 to 422 from the acquired image 400 photographed by the camera as shown in FIG.

  A Harris corner image is image data generated by applying a Harris Corner Detector to an acquired image. From these Harris corner images 410 to 412, for example, a pixel point (maxima point) 415 having a higher value than the surrounding eight pixels is extracted as a detection point. Furthermore, a LoG (Laplacian of Gaussian) filter is applied to the acquired image 400 to generate multiple levels (resolution) of Laplacian images 420 to 422. The LoG (Laplacian of Gaussian) filter is a kind of second-order differential filter used for image edge enhancement, and is performed until information from the retina is relayed by the outer knee in the human visual system. It is used as an approximate model of the processing.

  The feature point extraction processing corresponds to the position of the detection point obtained from the Harris corner images 410 to 412 as to whether there is a change in position due to a resolution change within a predetermined level range of the Laplacian images 420 to 422 that are LoG filter output images. A point is examined, and a point having no change is defined as a feature point. Thereby, it is possible to realize matching between feature points that is robust to an enlargement / reduction operation of an image. Details of these feature point extraction processes are described in, for example, Japanese Patent Application Laid-Open No. 2004-326693 (Japanese Patent Application No. 2003-124225), and this method is applied as a feature point extraction process in the feature point extraction unit 302. Is possible.

  The camera position / posture storage unit 104 and the feature point trajectory information storage unit 105 in FIG. 1 are all about the camera position / posture information detected by the camera position / posture detection unit 102 and the feature point trajectory information detected by the feature point tracking unit 103, respectively. Holds the output result of the frame. It is assumed that the frame times of the respective data are synchronized.

  A detailed configuration and processing of the feature point three-dimensional position detection unit 106 in FIG. 1 will be described with reference to FIG. The feature point three-dimensional position detection unit 106 includes a sampling unit 501 and an existence probability distribution calculation unit 502 as shown in FIG.

  In the sampling unit 501, camera position / orientation information corresponding to each frame image corresponding to all frame images recorded in the camera position / orientation storage unit 104 and feature points corresponding to each frame stored in the feature point locus information storage unit 105. Information to be applied for generating feature point existence probability distribution data is selected from the trajectory information.

  For example, in a continuous frame in a captured image of the camera 101, there is a high possibility that the captured image is a similar image with almost no change, and the camera position and orientation information and feature point trajectory information obtained from these similar images are not significantly changed. . Even if such information with poor change is used, the processing becomes inefficient, and it is inappropriate for the calculation of the three-dimensional position of the feature point. Therefore, the sampling unit 501 performs processing for selecting information that enables accurate calculation of the three-dimensional position of the feature point.

A sampling method executed by the sampling unit 501 will be described with reference to FIG. The position and posture of the camera 101 are changed with time. In FIG.
the camera 101-i at the position where the i-th frame is shot,
the camera 101-j at the position where the j-th frame is shot,
Is shown. These cameras are the same camera, meaning that an image is taken at a position moved with the passage of time.

  As described above, it is assumed that the camera 101 captures images from different positions and the same feature point M601 is captured in the different frame images. In this way, the camera observes feature points from various directions. The line connecting the camera and the feature point is called a ray, but if samples with different ray directions are sampled, data with different information will be sampled and the accuracy of calculating the 3D position information of the feature point will be increased. Can do. Therefore, it is preferable to sample data with different ray directions.

Light rays having greatly different directions have a large internal angle θ between the light rays. Conversely, the internal angle θ between rays in similar directions is small. In this example, sampling is performed with reference to the internal angle θ. The inner angle θ can be obtained by using a vector inner product formula, that is, the following mathematical formula.

However, in the above (Formula 2),
R ^T _i · x _i : unit vector of light rays from the camera 101-i toward the feature point M601,
R ^T _j · x _j : unit vector of light rays from the camera 101-j toward the feature point M601,
It is.
As described above, when the internal angle θ is calculated, a unit vector of light rays (line connecting the camera and the feature point) observed from cameras with different positions and orientations is necessary. Hereinafter, a method for deriving a unit vector of light will be described.

Processing for calculating a unit vector of light rays from the camera 101-i shown in FIG. 7 toward the feature point M601 will be described.
The unit vector [R ^T _i · x _i ] is calculated using observation information mi obtained from the captured image of the camera 101-i, where the camera position of the camera 101-i is ci and the posture (rotation matrix) is Ri.

A unit vector in the light ray direction connecting the focal point of the camera 101-i and the feature point M601 is represented by the following equation (Equation 3).

The unit vector of the light ray connecting the feature point M601 and the focus of the camera 101-i is shown in FIGS. 8A to 8C from the pinhole camera formula described above with reference to FIGS. , I.e., the following expressions 4-6.

First, (Formula 5) is obtained from the formula (Formula 4) of the pinhole camera. The right side of (Formula 5) indicates “the position of the feature point in the camera coordinate system in which the camera position is the origin and the posture represents each axis”, and di on the right side is “the distance between the feature point and the camera”, xi represents “a unit vector of rays in the camera coordinate system”. The unit vector xi of light in the camera coordinate system is obtained by the following equation (Equation 7).

  In the above formula, xi is expressed in the camera coordinate system, and thus the vector of the ray in the world coordinate system is obtained by transformation as in the above formula (Formula 6). However, the left side of the above formula (Formula 6) represents “a vector drawn from the camera to the feature point in the world coordinate system”, di is the “distance between the feature point and the camera”, and xi is “the camera coordinate system” Represents a unit vector of rays. RTx in the above formula (Formula 6) is “a unit vector of rays (line connecting the camera and the feature point) observed from the camera in the world coordinate system”.

  By substituting the obtained RTx into the above-described equation (Equation 2) for obtaining the internal angle θ, the internal angle θ can be obtained. The sampling unit 501 of the feature point three-dimensional position detection unit 106 illustrated in FIG. 6 samples frame data (camera position / posture information and feature point tracking information) whose internal angle θ calculated by the above formula (Formula 2) is not small. Like that. Specifically, for example, camera position / posture information and feature point tracking information obtained based on frame data having an internal angle θ equal to or greater than a preset threshold is acquired as sampling data.

  The sample data of the sampling unit 501 of the feature point three-dimensional position detection unit 106 shown in FIG. 6 is input to the existence probability distribution calculation unit 502, and the existence probability distribution calculation unit 502 corresponds to the feature points based on the input sample data. The feature point existence probability distribution as the three-dimensional position information to be calculated is calculated.

  As described above, the camera position / orientation detection unit 102 generates, for each of the camera position (c) and the camera attitude (R), information having an existence probability distribution according to a normal distribution corresponding to each captured image frame. The feature point tracking unit 103 generates a feature point tracking error distribution according to a normal distribution corresponding to the feature point position (m) in the image corresponding to each captured image frame. As described with reference to FIG. 7, the sampling unit 501 of the feature point trajectory information storage unit 105 and the sampling point 501 of the feature point three-dimensional position detection unit 106 has, for example, a frame having an internal angle θ equal to or greater than a preset threshold value. Camera position / posture information and feature point tracking information obtained based on the data are acquired as sampling data and input to the existence probability distribution calculation unit 502. The existence probability distribution calculation unit 502 calculates a feature point existence probability distribution as three-dimensional position information corresponding to the feature point based on the input sample data.

  The existence probability distribution calculation unit 502 receives the camera position (c), camera posture (R), and feature point position (m) having an existence probability distribution according to the normal distribution selected as the sample data, and inputs the feature points. The existence probability distribution in the three-dimensional space is calculated. The feature point existence probability distribution calculation process executed by the existence probability distribution calculation unit 502 will be described below.

The position M of the feature point is obtained using the formula of the pinhole camera model. However, the camera position is c, the camera posture (rotation matrix) is R, the camera internal parameter is A, and the position of the feature point in the image is m.
The following is the pinhole camera model formula (Formula 8).

From the above (Equation 8), the equation relating to the position M of the feature point is transformed into the following (Equation 9).

However, in the above (Equation 9), the vector x written in lower case is the one defined by Equation 10 below.

D represents the distance between the camera position c and the feature point M. Since the above (Formula 9) holds only for a positive value, the following formula (Formula 11) regarding the position M of the feature point holds for the measured value.

However, the symbol [^] means a measured value, and ε represents an error between the left side and the right side caused by a measurement error. In the above (Formula 11),
It is. A subscript i represents a frame number.

The input frame is not data for one frame but data for n frames, that is, data corresponding to a plurality of frames selected by the above-described sampling processing (camera position and orientation c and R and position m in the feature point image). Therefore, in order to support data corresponding to a plurality of frames, the above (Formula 11) is expanded to obtain the following Formula (Formula 13).

c ^ _i : Measurement value of camera position (camera position (c) having an existence probability distribution according to a normal distribution of i frames selected as sample data)
R ^ _i : Measured value of camera posture (rotation matrix) (camera posture (R) having an existence probability distribution according to a normal distribution of i frames selected as sample data),
Mx, My, Mz: three-dimensional positions of feature points (feature point positions having a feature point tracking error distribution according to a normal distribution of i frames selected as sample data),
d _i : distance between camera and feature point (i-frame shooting time) = unknown,
x ^ _i : unit vector of light ray from camera to feature point (i-frame shooting time) = unknown,
ε _i : Error caused by measurement error (error between left side and right side of Formula 11) (corresponding to i frame)

Further, each matrix / vector of the following formula (formula 14), that is,

Each matrix / vector is defined as shown in (Formula 15) below. However, the matrix _eq A is different from A of the pinhole camera.

  If the vector X is calculated such that the norm (vector length) of ε on the left side of (Equation 15) is minimum (error minimum), the desired vector X (three-dimensional position M of the feature point and each frame) A distance di) between the camera position of the data and the feature point is obtained.

The equation for the square of the norm of ε is given by the following (Equation 16).

When the vector X is assumed to be a variable and the others are all constants, the above (Equation 16) is a quadratic function relating to the vector X. Therefore, X having the smallest left side is the partial differential equation relating to the vector X in (Equation 16), That is,

X where the partial differential value of the above (Formula 17) becomes zero. FIG. 9 shows the relationship between ε ^t ε included in (Expression 17) and partial differentiation [(δ (ε ^t ε) / δX)]. The vector X is represented by the following formula (Formula 18).

When all the vector terms of the above formula (formula 18) are expanded, the following formula (formula 19) is obtained.

When the term “distance between camera and feature point [d _i ]” in the formula (formula 19) is deleted and reconstructed, the following formula (formula 20) is obtained.

However, in the above formula (Formula 20),

It is.

The above (Equation 20) means that if the camera position and orientation and the image position of the feature point are obtained, the three-dimensional position of the feature point is obtained. The two matrices included in (Equation 20) above are
[NewA] and [newB] are set and modified as follows.

Based on this formula modification, the following relational expression (Formula 24) is obtained.

  However, the function G in the above (Equation 24) is input data from the sampling unit 501 to the existence probability distribution calculation unit 502 shown in FIG. 6, that is, the position (m ^) of the feature point in the image, the camera position (c ^), It is composed of camera posture (R ^) information.

  Specifically, the position (m ^), camera position (c ^), and camera attitude (R ^) information of the feature point corresponding to the image frame selected as sample data by the sampling unit 501 are in accordance with the normal distribution. This is feature point tracking error distribution data according to a normal distribution corresponding to the camera position (c), camera posture (R), and feature point position (m) having a presence probability distribution.

The existence probability distribution calculation unit 502 obtains the existence probability distribution of the position of the feature point using the above (Equation 24). The position (m ^), the camera position (c ^), and the camera posture (R ^) of the feature point in the input image have an existence probability distribution according to the normal distribution, and the function G in the above (Equation 24) Is a non-linear function with respect to the input value, so the existence probability distribution becomes very complicated. However, if the function G is linearly approximated using the first-order differential ∇G as shown in the following (Equation 25), the feature point position M can be approximated by multiplication of normal distributions. The existence probability distribution calculation unit 502 can calculate the existence probability distribution of feature points by this linear approximation.

  For details on this development, refer to “Greg Welch and Gary Bishop,“ An Introduction to Kalman Filter ”, Technical Report 95-041 in Department of Computer 200 th. .Unc.edu / ˜welch / media / pdf / kalman_intro.pdf) ”.

The average value of the existence probability distribution of the position of the feature point can be obtained from the above formula (Formula 24) using the average value of the normal distribution as an input value. Further, the variance of the existence probability distribution of the feature point positions is obtained by the following formula (Formula 26).

However, in the above (Equation 26), Σg represents a linear approximation model error of the function G and is set as a constant. Further, the first-order differential ∇G of the function G is obtained by using the following expression (Expression 27) of the inverse matrix of the 3 × 3 diagonal matrix.

  Thus, the existence probability distribution in the three-dimensional space of the feature point positions is obtained.

  A usage example of the existence probability distribution of the feature points obtained in the above method in the three-dimensional space will be described with reference to FIGS. The processing example of FIG. 10 is a configuration example that waits until the accuracy of the existence probability distribution at the feature point position becomes high and outputs the converged existence probability distribution information.

  “The accuracy of the existence probability distribution of the feature point position is high” means that the standard deviation (norm of the covariance matrix) of the normal distribution is small in this example using the existence probability distribution being a normal distribution. Refers to the case. As shown in FIG. 10, the existence probability distribution information of the points that have been destroyed by the feature point three-dimensional position detection unit is input to the convergence determination unit 701.

The convergence determination unit 701 determines whether the existence probability distribution of the feature point positions in the three-dimensional space has converged. As the evaluation formula, for example, the following (Formula 28) is used.

  However, in the above (Formula 28), the function [diagonal] extracts the diagonal component of the covariance matrix ΣM of the existence probability distribution according to the normal distribution of feature points, and the function [sum] calculates the sum of all inputs.

  One advantage of the processing in the information processing apparatus of the present invention is that “the presence probability distribution of the output feature point position is calculated in consideration of the error distribution (existence probability distribution) of the input values”. Therefore, if the error distribution of the input value is correct, it is possible to robustly obtain the three-dimensional position distribution of the feature points even if the error distribution of the input value is large.

  FIG. 11 is a processing example in which presence probability distribution information of feature point positions in a three-dimensional space is fed back to the camera position and orientation detection unit 102 to enable more accurate camera position and orientation detection.

  The result of the example of “waiting until the accuracy of the feature point position becomes high” is fed back to the camera position and orientation detection unit. As described above, since the three-dimensional position distribution of feature points obtained robustly is handled as a feedback value of the camera position and orientation detection unit, the camera position and orientation detection unit is also robust.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be recorded in advance on a recording medium. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet and can be installed on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of the embodiment of the present invention, the camera position and orientation information having the existence probability distribution according to the normal distribution is generated based on the image acquired by the camera, and the camera acquisition is further performed. Generate feature point tracking error distribution according to normal distribution based on image, and apply camera position and orientation information with existence probability distribution according to these normal distribution and feature point tracking error distribution information according to normal distribution. The feature probability distribution in the three-dimensional space is calculated. According to this configuration, it is possible to more accurately analyze the three-dimensional position of the feature point included in the acquired image of the camera.

It is a figure explaining the structure and process of the information processing apparatus which concerns on one Example of this invention. It is a figure explaining the meaning of the formula which shows the correspondence of the pinhole camera model, ie, the position expressed by the camera coordinate system, and the three-dimensional position of the object in the world coordinate system. It is a figure explaining the meaning of the formula which shows the correspondence of the pinhole camera model, ie, the position expressed by the camera coordinate system, and the three-dimensional position of the object in the world coordinate system. It is a figure explaining the structure and process of the feature point tracking part of the information processing apparatus which concerns on one Example of this invention. It is a figure explaining the feature point extraction process using a Harris corner detector (Harris Corner Detector). It is a figure explaining the structure and process of a feature point three-dimensional position detection part of the information processing apparatus which concerns on one Example of this invention. It is a figure explaining the sampling process in the sampling part of the feature point three-dimensional position detection part of the information processing apparatus which concerns on one Example of this invention. It is a figure explaining the sampling process in the sampling part of the feature point three-dimensional position detection part of the information processing apparatus which concerns on one Example of this invention. And epsilon ^t epsilon included in Equation 17 is a diagram showing the relationship between the partial differential ^{[(δ (ε t ε)} / δX)]. It is a figure explaining the usage example of the feature point presence probability distribution information of the information processing apparatus which concerns on one Example of this invention. It is a figure explaining the usage example of the feature point presence probability distribution information of the information processing apparatus which concerns on one Example of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Information processing apparatus 101 Camera 102 Camera position / orientation detection unit 103 Feature point tracking unit 104 Camera position / orientation storage unit 105 Feature point trajectory information storage unit 106 Feature point three-dimensional position detection unit 107 Feature point existence probability distribution information 301 Feature point tracking unit 302 feature point extraction unit 303 preceding frame tracking information storage unit 321 feature point tracking information 400 acquired image 410 to 412 Harris corner image 420 to 422 Laplacian image 501 sampling unit 502 existence probability distribution calculation unit 601 feature point 701 convergence determination unit

Claims (15)

An information processing device that calculates a three-dimensional existence probability distribution of feature points included in a captured image of a camera,
A camera for taking images,
A camera position and orientation detection unit for detecting the position and orientation of the camera based on an image acquired by the camera;
A feature point tracking unit that tracks feature points extracted from images acquired by the camera;
A feature point 3D position detection unit that calculates a 3D existence probability distribution of feature points by inputting the camera position and orientation information detected by the camera position and orientation detection unit and the feature point trajectory information acquired by the feature point tracking unit Have
The camera has a pinhole camera model configuration that is configured to sequentially shoot a plurality of image frames by changing at least one of a position and a posture,
The camera position and orientation detection unit is configured to generate camera position and orientation information having an existence probability distribution according to a normal distribution,
The feature point tracking unit is configured to generate a feature point tracking error distribution according to a normal distribution,
The three-dimensional position detection unit includes camera position and orientation information having an existence probability distribution according to a normal distribution generated by the camera position and orientation detection unit, and a feature point tracking error according to a normal distribution generated by the feature point tracking unit. An information processing apparatus having a configuration for calculating an existence probability distribution in a three-dimensional space of feature points by applying distribution information. The camera position and orientation detection unit is configured to generate information having an existence probability distribution according to a normal distribution for each of the camera position (c) and the camera orientation (R).
The feature point tracking unit is configured to generate a feature point tracking error distribution according to a normal distribution corresponding to a feature point position (m) in an image;
The three-dimensional position detector
The configuration is such that the presence probability distribution in the three-dimensional space of the feature points is calculated by inputting the camera position (c) having the presence probability distribution according to the normal distribution, the camera posture (R), and the feature point position (m). The information processing apparatus according to claim 1. The three-dimensional position detector
Linear by first-order differentiation of the feature point three-dimensional position calculation formula set with the camera position (c), camera posture (R), and feature point position (m) having an existence probability distribution according to the normal distribution as input parameters The information processing apparatus according to claim 1, wherein the information processing apparatus is configured to calculate an existence probability distribution in a three-dimensional space of feature points using approximation. The feature point tracking unit includes:
The information processing apparatus according to claim 1, wherein the information processing apparatus is configured to generate a feature point tracking error distribution reflecting a quantization error determined based on a pixel setting of an image sensor of the camera. The three-dimensional position detector
Among a plurality of images including the same feature point included in the photographed image of the camera, an angle (inner angle) formed by a light beam connecting the camera and the feature point is obtained based on an image having an inner angle larger than a predetermined threshold angle. A sampling unit that selects the obtained data as data to be applied to the feature point existence probability distribution calculation;
An existence probability distribution calculating unit that calculates the existence probability distribution in the three-dimensional space of the feature points by applying the data selected by the sampling unit;
The information processing apparatus according to claim 1, wherein the information processing apparatus has a configuration. The camera position and orientation detection unit
The information processing apparatus according to claim 1, wherein SLAM (simultaneous localization and mapping) is performed to detect the position of the feature point in the image input from the camera and the position and orientation of the camera. . The feature point tracking unit includes:
2. The configuration according to claim 1, wherein the window matching with the local area image of the previous frame captured by the camera is executed to calculate a movement amount between frames of the local area (feature point). Information processing device. In the information processing apparatus, an information processing method for calculating a three-dimensional existence probability distribution of feature points included in a captured image of a camera,
An image capturing step in which the camera captures an image;
A camera position and orientation detection unit, wherein a camera position and orientation detection unit detects the position and orientation of the camera based on an image acquired by the camera;
A feature point tracking unit for tracking a feature point extracted from an image acquired by the camera;
The three-dimensional position detection unit inputs the camera position and orientation information detected by the camera position and orientation detection unit and the feature point trajectory information acquired by the feature point tracking unit, and calculates a three-dimensional existence probability distribution of the feature points. A feature point three-dimensional position detection step,
The image capturing step is a step of sequentially capturing a plurality of image frames by changing at least one of a position and a posture with a camera having a pinhole camera model configuration,
The camera position and orientation detection step is a step of generating camera position and orientation information having an existence probability distribution according to a normal distribution,
The feature point tracking step is a step of generating a feature point tracking error distribution according to a normal distribution,
The three-dimensional position detection step includes camera position and orientation information having an existence probability distribution according to a normal distribution generated by the camera position and orientation detection unit, and a feature point tracking error according to a normal distribution generated by the feature point tracking unit. An information processing method, which is a step of calculating an existence probability distribution in a three-dimensional space of feature points by applying distribution information. The camera position and orientation detection step is a step of generating information having an existence probability distribution according to a normal distribution for each of the camera position (c) and the camera orientation (R).
The feature point tracking step is a step of generating a feature point tracking error distribution according to a normal distribution corresponding to the feature point position (m) in the image;
The three-dimensional position detection step includes
A step of calculating a presence probability distribution in a three-dimensional space of feature points by inputting a camera position (c) having a presence probability distribution according to a normal distribution, a camera posture (R), and a feature point position (m). The information processing method according to claim 8. The three-dimensional position detection step includes
Linear by first-order differentiation of the feature point three-dimensional position calculation formula set with the camera position (c), camera posture (R), and feature point position (m) having an existence probability distribution according to the normal distribution as input parameters 9. The information processing method according to claim 8, wherein the information processing method is a step of calculating an existence probability distribution in a three-dimensional space of feature points using approximation. The feature point tracking step includes:
The information processing method according to claim 8, wherein the information processing method is a step of generating a feature point tracking error distribution reflecting a quantization error determined based on a pixel setting of an image sensor of the camera. The three-dimensional position detection step includes
Among a plurality of images including the same feature point included in the photographed image of the camera, an angle (inner angle) formed by a light beam connecting the camera and the feature point is obtained based on an image having an inner angle larger than a predetermined threshold angle. A sampling step for selecting the obtained data as data to be applied to the feature point existence probability distribution calculation;
An existence probability distribution calculating step of calculating an existence probability distribution in a three-dimensional space of feature points by applying the data selected in the sampling step;
The information processing method according to claim 8, further comprising: The camera position and orientation detection step includes
9. The information processing method according to claim 8, wherein the information processing method is a step of performing SLAM (simultaneous localization and mapping) for detecting the position of the feature point in the image input from the camera and the position and orientation of the camera. . The feature point tracking step includes:
9. The step of executing a window matching with a local area image of a previous frame captured by the camera to obtain a movement amount between frames of a local area (feature point). Information processing method. A computer program that causes an information processing device to calculate a three-dimensional existence probability distribution of feature points included in a captured image of a camera,
An image capturing step for causing the camera to capture an image;
A camera position and orientation detection step for causing the camera position and orientation detection unit to detect the position and orientation of the camera based on the image acquired by the camera;
A feature point tracking step for causing the feature point tracking unit to track a feature point extracted from the image acquired by the camera;
The camera position / orientation information detected by the camera position / orientation detection unit and the feature point trajectory information acquired by the feature point tracking unit are input to the three-dimensional position detection unit to calculate a three-dimensional existence probability distribution of the feature points. A feature point three-dimensional position detection step,
The image capturing step is a step of sequentially capturing a plurality of image frames by changing at least one of a position and a posture with a camera having a pinhole camera model configuration,
The camera position and orientation detection step is a step of generating camera position and orientation information having an existence probability distribution according to a normal distribution,
The feature point tracking step is a step of generating a feature point tracking error distribution according to a normal distribution,
The three-dimensional position detection step includes camera position and orientation information having an existence probability distribution according to a normal distribution generated by the camera position and orientation detection unit, and a feature point tracking error according to a normal distribution generated by the feature point tracking unit. A computer program which is a step of calculating distribution probability distribution of feature points in a three-dimensional space by applying distribution information.