JP2016114445A - Three-dimensional position calculation device, program for the same, and cg composition apparatus - Google Patents

Abstract

A three-dimensional position calculation apparatus for calculating a three-dimensional position of a feature point using a fisheye camera is provided. A three-dimensional position calculation apparatus 10 includes feature point extraction means 12 for extracting feature points from images photographed by two fisheye cameras of an equidistant projection method, and images at the extracted feature points. In the feature point pair whose feature quantity is most similar between both images, the radial distance, which is the difference between the radial diameters of the feature points when both images are represented in polar coordinates, is smaller than a predetermined threshold value, and A feature point filtering means 13 for filtering a feature point pair whose angle distance, which is a difference in angles, is smaller than a predetermined threshold value as a feature point pair indicating the same feature point, a position on both images of the filtered feature point pair, Feature point three-dimensional position calculation means 15 for calculating the three-dimensional position of the feature point according to the distance of the fisheye camera. [Selection] Figure 2

Description

  The present invention relates to a three-dimensional position calculation device that calculates the position of an object in a three-dimensional space using a fisheye camera, a program for the same, and a CG composition device.

In recent years, with the development of CG (Computer Graphics) technology, CG composite video in which a CG object and a live-action video are combined is used in movies, television programs, and the like.
Thus, when combining a CG object and a live-action video, it is necessary to match the CG object with the illumination conditions of the live-action video.

Conventionally, in order to capture light from a wide range as much as possible in the real world, a method of modeling an actual light source environment from images taken by two fisheye cameras and performing CG synthesis in that environment (hereinafter referred to as an actual light source measurement method) ) Is disclosed (Patent Document 1, Non-Patent Document 1).
In this actual light source measurement method, first, two fisheye cameras are pointed in the light source direction, and the distribution of the light sources is photographed. In this method, the three-dimensional position of the matching feature point is obtained from the two captured images, and a triangular mesh having the vertex as the feature point is generated. In this method, the luminance distribution is calculated from the captured image for each triangular mesh, so that information on the light source distribution is approximately obtained by a model in which the triangular meshes are connected.
In the actual light source measurement method, the feature point that minimizes the intersection error of two straight lines from the center of each image to the feature point in the three-dimensional space is selected as the feature point that matches between the images, and the intersection error is minimized. When there are a plurality of feature points, the feature point having the closest pixel coordinate on the image is selected.

In addition to Patent Document 1 and Non-Patent Document 1, various methods for matching feature points between images photographed by a plurality of fisheye cameras have been proposed (for example, Patent Documents 2 and 3; Non-patent documents 2 and 3).
Note that the method of Patent Document 2 uses RANSAC (RANdom SAmpleConsensus) for associating feature points. In the method of Patent Document 3, a fish-eye image (all-sky image) photographed by a fish-eye camera is developed on a plane, and feature points are matched by calculating an SSD (Sum of Squared Difference). In the method of Non-Patent Document 2, feature points are matched by calculating ZNCC (Zero-mean Normalized Cross-Correlation) for all feature points. Further, the method of Non-Patent Document 3 uses a bundle adjustment algorithm in order to minimize the error corresponding to the feature points.

JP-A-11-175762 International Publication No. 2012/063467 International Publication No. 2006/075528

Imari Sato, Yoichi Sato, Katsushi Ikeuchi, “Measurement of real light source environment by omnidirectional stereo and superimposition of virtual object based on it”, IEICE Transactions D-II, J81-DII, No. 5, pp.861-871, May 1998 M. Lhuillier, “Automatic Structure and Motion using a Catadioptric Camera”, Proc. IEEE Workshop OMNIVIS’05, 2005 E. Mouragnon, M. Lhuillier, M. Dhome, F. Dekeyser, P. Sayd, “Generic and Real-Time Structure from Motion”, Proc. BMVC, 2007

However, the distance measured in the real world is not necessarily reflected as it is on the image photographed by the fisheye camera as in the above-described real light source measurement method. In other words, the images taken by the fisheye camera (all-sky image) have different real-world distances even if the image center and the image periphery have the same distance on the image.
For this reason, there is a problem in that, when a feature point is matched using a distance on the whole sky image as an index as in the past, there is a high possibility of causing a false matching.

  In addition, when performing the feature point matching in the conventional method, it is necessary to flatten the whole sky image, or it is necessary to perform operations such as RASSNSAC, SSD, and ZNCC for many feature points. There is a problem that the calculation cost is increased due to the large number of preprocessing and calculation objects.

  The present invention has been made in view of such a problem. When a three-dimensional position of a feature point is calculated using a fisheye camera, the three-dimensional position is more accurate and less expensive than the conventional one. It is an object of the present invention to provide a calculation device, a program thereof, and a CG synthesis device using a fisheye camera.

  In order to solve the above-described problem, a three-dimensional position calculation apparatus according to the present invention is configured such that a three-dimensional position of a feature point in an image is obtained from images captured by two fisheye cameras having equidistant projection-type fisheye lenses. Is a configuration including a feature point extracting unit, a feature point filtering unit, and a feature point three-dimensional position calculating unit.

  In this configuration, the three-dimensional position calculation device extracts feature points whose image features change from one image and the other image captured by the two fisheye cameras by the feature point extraction unit. This feature point can be extracted by, for example, SIFT (Scale Invariant Feature Transform) or the like.

Then, the three-dimensional position calculation device uses the feature point filtering unit to represent both images in polar coordinates in the feature point pair having the most similar image feature amount between the two images. The feature point pairs in which the radial distance of each of the feature points is smaller than a predetermined threshold and the angular distance is smaller than the predetermined threshold are filtered as feature point pairs indicating the same feature point.
The radial distance is a radial difference in polar coordinates on the feature point pair image. The angular distance is a difference in declination in polar coordinates on the feature point pair image.
Thereby, the feature point filtering means can perform the matching determination of the feature points according to the characteristics of the fisheye camera such that the farther from the center of the image, the larger the parallax between the two images.

Then, the three-dimensional position calculation device uses the feature point three-dimensional position calculation means, after using the feature point filtering means, the positions of the feature point pairs determined to be correct feature point matching on both images, Based on the triangulation principle, the three-dimensional position of the feature point is calculated based on the distance from the center of the fisheye camera.
The three-dimensional position calculation device can be incorporated into a CG composition device that captures an image of the illumination environment, models the illumination environment based on the calculated three-dimensional position of the feature point, and generates a CG object.

The present invention has the following excellent effects.
According to the present invention, since the matching of feature points can be determined in accordance with the characteristics of a fish-eye camera in images taken by two fish-eye cameras, false matching can be reduced, and A highly accurate three-dimensional position can be obtained.
Further, according to the present invention, since it is a simple method of performing threshold determination in feature point matching determination, the calculation cost can be kept low.

1 is a configuration diagram showing an overall configuration of a video composition system according to an embodiment of the present invention. It is a block block diagram which shows the structure of CG synthetic | combination apparatus containing the three-dimensional position calculation apparatus (three-dimensional position calculation means) which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the coordinate system of the image (all-sky image) image | photographed with the fisheye camera. It is explanatory drawing for demonstrating the matching judgment of the feature point by a threshold value, Comprising: (a) is a figure which shows the example which makes the threshold value the difference of the radial and declination in this invention, (b) is the conventional simple distance. It is a figure which shows the example used as a threshold value. It is explanatory drawing for demonstrating the principle of the triangulation which calculates the three-dimensional position of the feature point detected with two fisheye cameras. It is explanatory drawing for demonstrating the feature point direction vector used by the principle of a triangulation, (a) is a figure for demonstrating the zenith angle of a feature point direction vector, (b) is a bias of a feature point direction vector. It is a figure for demonstrating a corner | angular. It is a schematic diagram for demonstrating the role of a light source distribution map. It is a flowchart which shows operation | movement of the CG synthetic | combination apparatus containing the three-dimensional position calculation apparatus (three-dimensional position calculation means) concerning embodiment of this invention.

Embodiments of the present invention will be described below with reference to the drawings.
≪Overall configuration of video composition system≫
First, the overall configuration of the video composition system S according to the embodiment of the present invention will be described with reference to FIG.

  The video composition system S synthesizes the live-action video in the studio ST photographed by the studio camera Cs and the CG object in accordance with the illumination conditions in the live-action video, and generates a composite video D. Here, the video composition system S includes a CG composition device 1 and a live-action CG composition device 3.

The CG synthesizer 1 is based on images (left camera image G _L , right camera image G _R ) captured by a sensor camera (hereinafter referred to as fish eye cameras C _L , C _R ) using a fish-eye lens installed in the studio ST in advance. The CG object (CG image) is generated by modeling the illumination environment such as the light source L and rendering the material CG data CG _D to be synthesized in the illumination environment. In this specification, in order to represent each fisheye camera and its captured image, it is expressed as “left” and “right” for convenience in accordance with the drawings, but this is simply “one”, “other”. It just represents the meaning of

Here, the fisheye cameras C _L and C _R are arranged at a predetermined interval so that the captured images are parallel with the zenith as the imaging target in order to capture the illumination environment. Further, fisheye camera C _L, C _R is a lens equidistant projection method, and it is the same resolution. Here, the equidistant projection method is a method in which the distance from the center of the captured image is proportional to the zenith angle.
The CG composition device 1 outputs the generated CG object to the live-action CG composition device 3. The configuration and operation of the CG synthesizer 1 will be described in detail later.

Stock CG synthesizing unit 3, the image photographed by the studio camera C _S, is to synthesize a CG object generated by the CG synthesizing apparatus 1. The live-action CG composition device 3 is a general composition device that composes a CG object for each frame of a video.
Moreover, the studio camera C _S for capturing the images input to the live-action CG synthesizing unit 3 captures the subject as an image in a studio ST, a general camera for use in broadcasting stations or the like.
In this way, the video composition system S can synthesize a CG that is visually uncomfortable with a live-action video by applying the actual lighting environment to the CG object by the CG composition device 1.
Here, the video composition system S is shown as an example configured as a system in the studio ST, but may be configured inside a general building or outdoors.
Hereinafter, the configuration and operation of the CG synthesizer 1 will be described in detail.

≪Configuration of CG synthesizer≫
First, referring to FIG. 2 (refer to FIG. 1 as appropriate), the configuration of the CG synthesizer 1 according to the embodiment of the present invention will be described.
As shown in FIG. 2, the CG synthesizing apparatus 1 includes a three-dimensional position calculation unit 10 and a rendering unit 20.

3D position calculating unit 10 inputs fisheye camera C _L, C _R in captured each image (left camera image G _L, the right camera image G _R) and, in association with the feature points in the image, The three-dimensional position of the feature point is calculated.
Here, the three-dimensional position calculation apparatus 10 includes an image input unit 11, a feature point extraction unit 12, a feature point filtering unit 13, a calibration unit 14, and a feature point three-dimensional position calculation unit 15.

Image input means 11 is used for inputting fisheye camera _C L, _{C R} are photographed image (left camera image _{G L,} the right camera image _{G R)} to. Here, the image input unit 11 is provided with fish-eye camera _C L to enter, the left image input unit 11L corresponding to _{C R,} and the right image input unit 11R, a. That is, the left image input unit 11L receives the left camera image G _L that fisheye camera C _L is taken, the right image input unit 11R inputs right camera image G _R which fisheye camera C _R is taken.
The left image input unit 11L and the right image input unit 11R output the input images (left camera image G _L and right camera image G _R ) to the feature point extraction unit 12.

The feature point extraction unit 12 extracts each feature point from the image (left camera image G _L , right camera image G _R ) input from the image input unit 11. Here, the feature point extracting unit 12 includes a left image feature point extracting unit 12L corresponding to an input image (left camera image G _L , right camera image G _R ), and a right image feature point extracting unit 12R. That is, the left image feature point extracting means 12L is a feature point (left image feature point) is extracted from the left camera image G _L, the right image feature point extracting unit 12R are feature points from the right camera image G _R (right image feature Point).

Here, the feature point is a point at which the image feature in the photographed image changes, for example, a point at which a pixel value or a luminance value with respect to an adjacent pixel changes. This feature point can be extracted using a general method. For example, the left image feature point extracting unit 12L and the right image feature point extracting unit 12R extract feature points by SIFT, SURF (Speeded Up Robust Features), or the like.
The left image feature point extracting unit 12L and the right image feature point extracting unit 12R output the extracted feature points (position and feature amount on the image) to the feature point filtering unit 13. The feature amount is, for example, a 128-dimensional SIFT feature amount when a feature point is detected by SIFT.

The feature point filtering unit 13 is a radial distance and an angle between the left image feature point and the right image feature point extracted by the left image feature point extracting unit 12L and the right image feature point extracting unit 12R of the feature point extracting unit 12. Based on the distance, corresponding feature points are filtered.
The feature point filtering unit 13 includes a left image feature point and a right image feature point that have the most similar feature amounts (image feature amounts) extracted by the left image feature point extraction unit 12L and the right image feature point extraction unit 12R, respectively. On the other hand, the feature points are filtered based on whether the radial distance and the angular distance are above or below the predetermined threshold values, respectively. Here, the feature point filtering unit 13 adopts a correspondence in which the radial distance is smaller than a predetermined threshold and the angular distance is smaller than the predetermined threshold as correct matching.

Note that the feature amount is most similar means, for example, that the Euclidean distance of the feature amount (for example, SIFT feature amount) between the left image feature point and the right image feature point is minimized.
The radial distance is a radial distance (difference) when images (left camera image G _L , right camera image G _R ) are represented in a polar coordinate system. The angular distance is a declination distance (difference) when an image is expressed in a polar coordinate system.
As a result, the feature point filtering unit 13 removes erroneous matching from the feature point group corresponding to the feature amount that is most similar by SIFT or the like in advance.
Then, the feature point filtering unit 13 outputs the filtered corresponding feature points (position and feature amount on the image) to the calibration unit 14.

Here, with reference to FIG. 3, the radial distance and the angular distance, which serve as a reference for the feature point filtering unit 13 to perform the matching determination, will be described. FIG. 3 is a diagram illustrating a coordinate system of the left camera image G _L (full sky image). The same applies to the right camera image G _R.
As shown in FIG. 3, the coordinates of the feature point (left image feature point) of the left camera image _GL are p = (p _x , p _y ) when expressed in an orthogonal coordinate system, and when expressed in a polar coordinate system, As the moving radius r and the deflection angle θ, p = (r cos θ, r sin θ).

That is, the feature point filtering means 13, in the left camera image G _L and the right camera image G _R, the distance of the radius vector r and declination θ used in a polar coordinate system, the filtering feature points.
Here, the target of the distance calculation, coordinates p = characteristic points of the left camera image _{G L (p x, p y} ), although not shown, the feature point of the right camera image G _R coordinates Is q = (q _x , q _y ).
At this time, the feature point filtering means 13 obtains a radial distance (radial distance r _d ) between the two points by the following equation (1).

Further, the feature point filtering means 13 obtains the declination distance (angular distance θ _d ) between the two points by the following equation (2).

Then, the feature point filtering means 13, as shown in the following equation (3), radial distance _{r d} is less than the threshold value _{r Thresshold,} and angular distance theta _d threshold theta 2 points p less than _Thresshold, Let q be a matching point.

The threshold _{r Thresshold} and threshold theta _Thresshold is a value that is calculated by the following equation (4) and (5).

Here, p _width represents the horizontal width (number of pixels) of the image (left camera image G _L , right camera image G _R ), and p _height represents the vertical width (number of pixels) of the image (see FIG. 3).
R _partition indicates the number of radial radius divisions (radial radius division number) when the images (left camera image G _L , right camera image G _R ) are expressed in polar coordinates, and θ _partition indicates the image in polar coordinates. The number of divisions of declination θ (the number of declination divisions) is shown.
The radial division number r _partition and the declination division number θ _partition are indices indicating the fineness of dividing an image (left camera image G _L , right camera image G _R ) by radial and declination, It is determined in advance based on the resolution or the like. Of course, the number of divisions may be set from the outside.
Also, radial division number _{r partition} the threshold _{r Thresshold} is at least a fish left camera image _{G L,} the value equal to or larger than the disparity of the right camera image _{G R} which corresponds to the distance between the eye cameras _C L, _{C R.}
Note that it is desirable that the feature point filtering means 13 further excludes outliers from the corresponding feature points by the minimum median method or the like.

Here, with reference to FIG. 4, the matching determination by the threshold value which the feature point filtering means 13 performs is demonstrated visually. Incidentally, FIG. 4 shows the left camera image G _L and the right camera images G _R on the same coordinates.
As shown in FIG. 4 (a), or the feature point filtering means 13, whether the feature point q _a is matched to the feature point p _a, the difference between the radius vector and polarization angle, a range of the area Ba Depending on whether or not.
Further, the feature point filtering means 13, for the feature point p _b at the position movement diameter is smaller than the feature point p _a, narrow range than the region Ba, the difference between the radius vector and declination, region Bb It is performed depending on whether or not it is within the range.
As a result, the feature point filtering means 13 narrows the area range to be determined as the same feature point as the radius is smaller, and widens the area range as the radius is larger, and performs the feature point matching judgment. .

For reference, FIG. 4B illustrates a threshold range in the case of performing matching determination based on a simple distance on a conventional image. As shown in FIG. 4B, conventionally, even if the feature points p _c and p _d have different radial distances, the feature points are within the same distance (the circular regions Bc and Bd having the same radius). Was matching.
As described above, the feature point filtering unit 13 sets more pixels as matching determination pixels as the distance from the image center increases (the radius is larger), so that the corresponding feature points are matched to the characteristics of the fisheye camera. Can be accurately filtered.
Returning to FIG. 2, the description of the configuration of the CG synthesizer 1 will be continued.

The calibration unit 14 performs camera calibration based on the feature points filtered by the feature point filtering unit 13. That is, calibration means 14, so that the left camera image G _L and the right camera image G _R, parallel optical axis of the fisheye camera C _L, C _R, the horizontal axis of the imaging device in the fisheye camera matches The position correction of the feature point of the other image (for example, the right camera image G _R ) is performed on the basis of the feature point of one image (for example, the left camera image G _L ).

The calibrating unit 14 obtains a basic matrix such that one image and the other image are parallel from the corresponding (matching) feature point group filtered by the feature point filtering unit 13, and feature points of both images Is converted (rotated) by a basic matrix.
This basic matrix can be obtained from, for example, at least eight corresponding feature points. Since the 8-point algorithm for obtaining the basic matrix is a general algorithm, description thereof is omitted here.
The calibrating unit 14 outputs the position of the corresponding feature point after calibration (position on the image) to the feature point three-dimensional position calculating unit 15.

Feature point three-dimensional position calculating means 15, on the image after calibration the position of the feature point of the corresponding two points, fisheye camera C _L, and a distance between the C _R, the feature points in the three-dimensional space The position (feature point three-dimensional position) is calculated.

Specifically, the feature point three-dimensional position calculating means 15, as shown in FIG. 5, fisheye camera C _L, C principal point feature point in a three-dimensional space from the position of _R (3-dimensional feature point position P) Vectors (unit direction vectors) u and v are obtained. Then, the feature point three-dimensional position calculating means 15, a fish-eye camera C _L position on advance three-dimensional space is _known, the distance w between C _R, the principle of triangulation by a vector u, v, wherein The position of the point in the three-dimensional space (feature point three-dimensional position P) is calculated.
Note that each of the vectors, fisheye camera C _L, C _R in the captured left camera image G _L, the right camera image G _{R (if} it is calibrated, the image after calibration) determined from the coordinates of the feature points of be able to.

Here, with reference to FIG. 6, the vector which the feature point three-dimensional position calculation means 15 calculates is demonstrated. Since fisheye camera C _L, vector calculation of C _R is the same technique will be described here vector u whose starting point is the fisheye camera C _L.
Since fisheye camera C _L is a camera equidistant projection method, when the focal length of the fisheye camera C _L is f, as shown in FIG. 6 (a), the zenith of the fisheye camera C _L The following equation (6) is established between the angle between the vector T and the vector u (the zenith angle L _θ1 ) and the radius r of the coordinate p of the feature point on the image (left camera image G _L ).

Further, as shown in FIG. 6 (b), declination L _.theta.2 in projecting the vector u on the image (left camera image G _L), as described in FIG. 3, the image (left camera image G _L ) Is the same as the declination when expressed in the polar coordinate system.
Therefore, the feature point three-dimensional position calculating unit 15, the vector u corresponding to fisheye camera C _L, identified in zenith angle L _.theta.1 and declination L _.theta.2. Similarly, the feature point three-dimensional position calculating unit 15, the vector v corresponding to the fisheye camera C _R, identified in zenith angle R _.theta.1 and declination R _.theta.2.

Then, as shown in FIG. 5, the feature point three-dimensional position calculation means 15 calculates the distance w between the fisheye cameras C _L and C _R and the vectors u (L _θ1 , L _θ2 ) and v (R _θ1 , R _θ2). The position of the feature point in the three-dimensional space (feature point three-dimensional position P) is calculated by the principle of triangulation.
In the three-dimensional space, the vectors u and v have a twisted relationship and may not have an intersection. Therefore, the feature point three-dimensional position calculation means 15 calculates the closest point of the vectors u and v as the feature point three-dimensional position P.
Returning to FIG. 2, the description of the configuration of the CG synthesis apparatus 1 will be continued.

The feature point three-dimensional position calculation unit 15 outputs the feature point three-dimensional position obtained by the calculation to the rendering unit 20. Here, the feature point three-dimensional position calculation means 15 outputs the position of the feature point of the fisheye camera image (for example, the left camera image G _L ) corresponding to the feature point three-dimensional position to the rendering means 20. .

  As a result, the three-dimensional position calculation means 10 does not require preprocessing for developing the whole sky image into a flat image when performing feature point matching, and by simple threshold determination of the radial distance and the angular distance, Since the corresponding feature points are filtered, the calculation cost can be reduced. In addition, since the three-dimensional position calculation means 10 uses a distance (radial distance, angular distance) in consideration of the characteristics of the fisheye camera, compared to the conventional method of filtering feature points by the actual distance in the image, Feature point matching can be performed with high accuracy.

The rendering unit 20 models an illumination environment specified by the feature point 3D position calculated by the 3D position calculation unit 10 and generates a CG image (CG object) corresponding to the illumination environment from the material CG data. It is.
Here, the rendering unit 20 includes a light source distribution map generation unit 21 and a shadow processing unit 22.

The light source distribution map generating means 21 associates the information on the light source distribution of the illumination environment from the three-dimensional position of the feature point calculated by the three-dimensional position calculation means 10 with a triangular mesh having the feature point three-dimensional position as a vertex, A light source distribution map is generated.
That is, the light source distribution map generation unit 21 sets the region in the triangular mesh for each triangular mesh having the three-dimensional position of the feature point as a vertex in the light source distribution image (here, the left camera image _GL ). Are associated with the region surrounded by the feature points that are the vertices of the triangular mesh.

As a result, the light source distribution map generating means 21 can generate a light source distribution map as an illumination environment by associating image regions with each triangular mesh.
The light source distribution map generation unit 21 outputs a light source distribution map, which is modeled illumination environment data, to the shadow processing unit 22.

The shadow processing unit 22 performs a shading process on the material CG data CG _D on the basis of the light source distribution map modeled by the light source distribution map generation unit 21 and generates a CG object to be combined with a live-action image. .
The shadow processing means 22 generates a CG object having a camera (studio camera C _S ) as a viewpoint position at a position on the real space (three-dimensional space) where the CG object is placed. At this time, the shading processing means 22 performs shading processing on the material CG data CG _D based on the light source luminance specified in each triangular mesh region of the light source distribution map.

The role of the light source distribution map when performing shadow processing in the shadow processing means 22 will be briefly described.
As shown in FIG. 7, the light source distribution map M is obtained by modeling the illumination environment with a triangular mesh. Each triangular mesh includes, for example, luminance information of the light source L itself and luminance information of indirect light of the light source L. Has been dealt with.

That is, the shadow processing means 22 determines the pixel value of the CG object based on the light source information of the triangular mesh corresponding to the CG shape and the direction of the light that irradiates the shape.
The shading processing in the shading processing means 22 may use a general method, for example, a method such as IBL (Image Based Lighting).
The shadow processing means 22 outputs the generated CG object to the live-action CG composition device 3.

  Thus, the rendering unit 20 can generate a light source distribution map from the three-dimensional position of the feature point calculated by the three-dimensional position calculation unit 10 and generate a CG object suitable for the actual lighting environment.

  As described above, by configuring the CG synthesizer 1, the CG synthesizer 1 increases the accuracy of feature point matching by filtering feature points according to the characteristics of the fisheye camera, thereby providing a more accurate lighting environment. Can be modeled. Thereby, the CG synthesizing apparatus 1 can generate a CG object that does not feel uncomfortable even if it is synthesized with a live-action video.

≪Operation of CG synthesizer≫
Next, referring to FIG. 8 (refer to FIGS. 1 and 2 as appropriate), the operation of the CG synthesis apparatus 1 according to the embodiment of the present invention will be described.
First, the CG synthesizing apparatus 1 is photographed from the fisheye cameras C _L and C _R that photographed the illumination environment by the image input unit 11 (the left image input unit 11L and the right image input unit 11R) of the three-dimensional position calculation unit 10. Images (left camera image G _L , right camera image G _R ) are input (step S1).

Then, the CG synthesizing apparatus 1 extracts feature points from the image input in step S1 by SIFT or the like by the feature point extraction unit 12 (left image feature point extraction unit 12L, right image feature point extraction unit 12R) (step S1). S2).
That is, the left image feature point extracting unit 12L extracts feature points (left image feature point) from the left camera image G _L, the right image feature point extracting unit 12R is, feature points from the right camera image G _R (right image feature Point).
Then, the CG synthesis apparatus 1 uses the feature point filtering unit 13 to extract the radial distance from the predetermined threshold with respect to the left image feature point and the right image feature point with the most similar feature amount extracted in step S2. That is smaller and the angular distance is smaller than a predetermined threshold is adopted (filtered) as correct matching (step S3).
Thereby, the feature point filtering means 13 can accurately filter the corresponding feature points according to the characteristics of the fisheye camera.

Then, the CG synthesizing apparatus 1 uses the calibration unit 14 to correct the position of the feature point by camera calibration (step S4).
That is, calibration means 14, the feature point group matching is filtered in step S3, determined the basic matrix for performing the position correction, the feature points of two images (left camera image G _L and the right camera image G _R) Perform position correction on the image.

Then, CG synthesizing apparatus 1, the feature point three-dimensional position calculating unit 15, the position of the corresponding two points characteristic points on the calibration image after the step S4, fisheye camera C _L, and the distance between C _R Then, the position of the feature point in the three-dimensional space (the feature point three-dimensional position) is calculated (step S5).
The operation up to this step S5, fisheye camera C _L, C _R are photographed image (left camera image G _L, the right camera image G _R) from, it is possible to calculate the three-dimensional position of the feature point.

Thereafter, the CG synthesis device 1 uses the light source distribution map generation unit 21 of the rendering unit 20 to obtain the light source distribution information of the illumination environment from the three-dimensional position of the feature points calculated by the operations up to step S5. A light source distribution map is generated in association with the triangular mesh whose position is the vertex (step S6).
As a result, the lighting environment in the real space is modeled.

Then, the CG composition device 1 performs a shadow process on the material CG data based on the light source distribution map generated in step S6 by the shadow processing means 22, and generates a CG object to be combined with the live-action video ( Step S7).
As a result, the CG composition device 1 can generate a CG object that is suitable for the lighting environment in the real space and does not feel uncomfortable even if it is synthesized with the live-action video.

As mentioned above, although the structure and operation | movement of CG synthesizing | combining apparatus 1 which concern on embodiment of this invention were demonstrated, this invention is not limited to this embodiment.
For example, here, the configuration of the CG synthesizing apparatus 1 includes the three-dimensional position calculation unit 10 and the rendering unit 20, but these may be configured independently.
That is, the three-dimensional position calculating unit 10 may be configured as a three-dimensional position calculating device and the rendering unit 20 as a rendering device.

Here, the three-dimensional position calculation means 10 of the CG synthesizer 1 is configured to include the calibration means 14, but as long as the fisheye cameras C _L and _CR are arranged in advance in a calibrated state. The calibration unit 14 may be omitted from the configuration.
Conversely, as long as the fisheye camera C _L, C _R are arranged in a non-calibration state, between the image input unit 11 and feature point extracting unit 12, a second calibration performing camera calibration Means may be provided.
In this case, for example, the second calibration unit uses the one image (for example, the left camera image G _L ) as a reference so that the other image (for example, the right camera image G _R ) is most similar. The image is rotated.

The CG synthesizer 1 described above and its modification can be operated by a program (CG synthesis program) that causes a computer to function as each of the above-described means.
When the CG composition device 1 is configured separately as a three-dimensional position calculation device and a rendering device, the three-dimensional position calculation device causes the computer to function as each unit of the three-dimensional position calculation unit 10 described above. It can be operated by a program (three-dimensional position calculation program). In addition, the rendering apparatus operates the computer with a program (rendering program) that functions as each unit of the rendering unit 20 described above. Also, these programs can be recorded on a recording medium and distributed, or distributed via a communication line.

S image composition system 1 CG composition device 10 3D position calculation means (3D position calculation device)
DESCRIPTION OF SYMBOLS 11 Image input means 11L Left image input means 11R Right image input means 12 Feature point extraction means 12L Left image feature point extraction means 12R Right image feature point extraction means 13 Feature point filtering means 14 Calibration means 15 Feature point three-dimensional position calculation means 20 Rendering means (rendering device)
21 light source distribution map generating means 22 shading means C _L fisheye camera C _R fisheye camera

Claims (5)

A three-dimensional position calculation device for calculating a three-dimensional position of a feature point in an image from images captured by two fisheye cameras having equidistant projection type fisheye lenses,
Feature point extracting means for extracting feature points with varying image characteristics from one image and the other image captured by the two fisheye cameras;
In the feature point pair in which the image feature quantity at the feature point extracted by the feature point extracting unit is the most similar between the two images, a motion that is a difference in radius of each feature point when the two images are expressed in polar coordinates. A feature point filtering means for filtering feature point pairs in which the radial distance is smaller than a predetermined threshold and the angular distance that is a difference in declination is smaller than the predetermined threshold as a feature point pair indicating the same feature point;
A feature point three-dimensional position calculation means for calculating a three-dimensional position of the feature point based on the position of both of the feature point pairs filtered by the feature point filtering means and the distance of the fisheye camera;
A three-dimensional position calculation apparatus comprising: A basic matrix for parallelizing the other image with the one image is obtained from positions on both images of the plurality of feature point pairs filtered by the feature point filtering means, and the feature points of the other image are obtained. Calibration means for converting according to the basic matrix further comprises
The three-dimensional position calculation apparatus according to claim 1, wherein the feature point three-dimensional position calculation unit calculates the three-dimensional position based on the converted feature point pair.   The feature point three-dimensional position calculation means is represented by a direction vector composed of a zenith angle and a declination angle specified by a position on each image of the feature point pair through the principal point position of the fisheye camera. The three-dimensional position calculation apparatus according to claim 1, wherein a three-dimensional position where two straight lines are closest is calculated as a three-dimensional position of the feature point.   The three-dimensional position calculation program for functioning a computer as a three-dimensional position calculation apparatus as described in any one of Claims 1-3. CG synthesis for generating a CG object to be synthesized with a live-action video photographed in the illumination environment from images photographed by two fisheye cameras equipped with equidistant projection fisheye lenses for photographing the illumination environment A device,
The three-dimensional position calculation device according to any one of claims 1 to 3, wherein a three-dimensional position of a feature point in the image is calculated from images captured by the two fisheye cameras;
For each triangular mesh whose vertex is the three-dimensional position calculated by the three-dimensional position calculation device, each vertex of the triangular mesh in the image photographed by any of the fisheye cameras A light source distribution map generating means for generating a light source distribution map that models the illumination environment in association with a region surrounded by feature points that are:
Based on the light source distribution map generated by the light source distribution map generating means, the shadow processing means for generating the CG object by shading the material CG data,
A CG synthesizer comprising:

Images