JP2001194114A - Image processing apparatus, image processing method, and program providing medium - Google Patents

Abstract

(57) [Problem] To provide an image processing apparatus and method capable of simultaneously capturing a pattern image for generating a distance image and a texture image from which a pattern has been removed. SOLUTION: A filter having two different characteristics is arranged in an image sensor for photographing a measurement object in correspondence with each pixel. Irradiate the pattern light with the infrared light to the measurement target, place an infrared light cut filter in about half of the pixels of the image sensor that captures the measurement target image, and place a visible light cut filter in the other half. did. The pixels on which the infrared light cut filter is arranged do not capture the pattern light and are used as a texture image, and the pixels on which the visible light cut filter is arranged capture the pattern light and generate a distance image using this image. . A texture image and an image with a pattern can be simultaneously captured, and a high-precision three-dimensional image is generated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image processing apparatus and an image processing method for obtaining a three-dimensional shape of an object and generating a distance image, and a program providing medium. In particular, an image processing apparatus suitable for a stereo image method of generating a distance image by measuring a three-dimensional shape of a subject surface using images captured from a plurality of different positions of an object present in a distance range on a three-dimensional space and The present invention relates to an image processing method and a program providing medium.

[0002]

2. Description of the Related Art Conventional methods for obtaining a three-dimensional shape of an object are roughly classified into an active method (active method) and a passive method (passive method). As an active method,
A method of calculating the distance to each measurement object by projecting light and measuring the time until the light is reflected back, or applying a slit-shaped pattern light to the measurement object and projecting it on the measurement object There is a method such as a light-section method for measuring a three-dimensional shape by examining the shape of a given pattern light.

[0003] A typical passive method is as follows.
This is a stereo method using the principle of triangulation.
This is a method in which a positional relationship (parallax) is obtained by finding a corresponding point of each pixel between the images using more than one camera, and a distance to a measurement target is measured.

[0004] The principle of the stereo method will be briefly described. The stereo method associates pixels in a plurality of images obtained by photographing the same object from two or more viewpoints (different line-of-sight directions) using a plurality of cameras, and thereby positions a measurement object in a three-dimensional space. It is what we seek. For example, the same object is photographed from different viewpoints by the reference camera and the detection camera, and the distance between the measurement objects in each image is measured based on the principle of triangulation.

FIG. 1 is a diagram for explaining the principle of the stereo method. Reference camera (Camera 1) and detection camera (C
amera 2) captures the same object from different viewpoints. Consider obtaining the depth of a point “mb” in an image captured by a reference camera.

[0006] Objects that appear at the point "mb" in the image captured by the reference camera are identified by "m1,""m2," and "m3" in the images captured by the detection cameras capturing the same object from different viewpoints. Will be developed on a straight line. This straight line is referred to as an epipolar line Lp.

[0007] The position of the point "mb" in the reference camera appears on a straight line called "epipolar line" in the image obtained by the detection camera. Point P to be imaged
(Points existing on a straight line including P1, P2, and P3) are the same observation point “mb” on the reference image regardless of the depth, that is, the distance from the reference camera, as long as they are on the line of sight of the reference camera. Appears in On the other hand, the point P on the image captured by the detection camera appears on the epipolar line at a position corresponding to the magnitude of the distance between the reference camera and the observation point P.

FIG. 1 illustrates a correspondence between an epipolar line and an observation point “mb” in a detected camera image. As shown in the figure, the position of the observation point P is P1, P2,
As the position changes to P3, the observation point in the detected camera image shifts to “m1,” “m2,” and “m3”.

By using the above geometrical optical properties to search for the observation point "mb" on the epipolar line, the distance of the point P can be identified. This is the basic principle of the “stereo method”. In this way, three-dimensional information on all pixels on the screen is obtained. The acquired three-dimensional information can be used as pixel attribute data corresponding to each pixel.

Although the above-described stereo image method uses one reference camera and one detection camera, a multi-baseline stereo (Multi) using a plurality of detection cameras is used.
An evaluation value may be obtained by an iBaseline Stereo) method, and three-dimensional information for each pixel may be obtained based on the evaluation value. The multi-baseline stereo image method uses an image captured by one reference camera and a plurality of detection cameras, obtains an evaluation value representing a correlation between the plurality of detection camera images and the reference camera image, and calculates each evaluation value. The sum is added and the sum is used as a final evaluation value. For details of the multi-baseline stereo image method, see, for example, “Stereo matching using a plurality of baseline lengths”, IEICE Transactions D
-11Vol. J75-D-II No. 8 pp. 1
317-1327, August 1992.

[0011] As described above, the stereo method associates pixels in a plurality of images obtained by photographing the same object from two or more viewpoints (different line-of-sight directions) using a plurality of cameras, that is, By performing "corresponding scoring (matching)", the position of the measurement target in the three-dimensional space is obtained.

Conventionally, the method of “corresponding scoring” that is often used is pixel-based matching, A
Area-based matching and Feature-ba
It is roughly classified into sed matching. Pixel-base
The d-matching is a method of searching for the correspondence of points in one image as it is in the other image (C. Lawr.
ence Znickick and Jon A. W
ebb: Multi-baseline Stereo
o Using Surface Extraction
n, Technical Report, CMU-
CS-96-196, (1996)).

Area-based matching is a method of searching for the correspondence of a point in one image by using a local image pattern around the point when searching for the other image (Okutomi, Kinide: Stereo matching using multiple baseline lengths, IEICE Transactions D-II, V
ol. J75-DII, No. 8, pp. 1317
-1327, (1992), Yokoyama, Miwa, Ashigahara, Koyanatsu, Hayashi, later: Stereo Camera Sys
tem and Its Application,
SRF '97, (1997), Kanade, Kimura: Video rate stereo machine, Journal of the Robotics Society of Japan, Vol.
13, No. 3, pp. 322 to 326, (19
95), Kinade, Mosquito Field, Kimura, Kawamura, Yoshida, Oda: Development of Video Rate Stereo Machine, Journal of the Robotics Society of Japan,
Vol. 15, No. 2, pp. 261-267,
(1997), Yamaguchi, Takachi, Iguchi: Stereo matching for stone image measurement using the adaptive window method, Humanities and Computers, Vol. 32, no. 10, p
p. 55-60, (1996), Yokoya: Recent Special Issue on Signal Processing Recent topics on computer vision,
System / Control / Information, Vol. 38, no. 8,
pp. 436-441, (1994)).

The feature-based matching is a method of extracting features such as dark and light edges from images and associating them using only features between images (HH.
Baker and T.M. O. Binford: De
pth from edgeand intensit
y based stereo, In Proc.
IJCAI'81, (1981), Ishiyama, Kakuho, Kawai, Ueshiba, Tomita: Search for Correspondence Candidates in Segment-Based Stereo, IEICE Technical Report, Vol. 96, No. 13
6, (1997); E. FIG. L. Grimson:
Computational experiments
with a feature based ste
reo algorithm, IEEE Tran
s. PAMI, Vol. 7, No. 1 pp.
17-34, (1985)).

The characteristics of each of the above methods are summarized as follows. (1) Pixel-based matching and Area-
In the based matching, a corresponding point is searched for each pixel, so that the obtained distance image is dense. on the other hand,
Since the feature-based matching associates only feature points, the obtained distance image is sparse.

(2) In the Area-based matching, a kind of correlation operation is performed.
Although computational cost is higher than d-matching and Feature-based matching, it is not necessarily a problem that cannot be solved by speeding up the algorithm.

(3) Pixel-based matching only performs the correspondence between pixels, so that the calculation speed is considerably high. However, due to the difference in characteristics between the right and left cameras, it is not easy to perform the correspondence using the gray value between pixels. Absent.

From the above-described characteristics, Area-based matching is generally effective and widely used as a method for obtaining a three-dimensional shape (or depth) of an object with high accuracy for each pixel.

A method of obtaining a corresponding point in stereo vision by general area-based matching will be described with reference to FIG. FIG. 2A is an observation image of the reference camera, and FIG. 2B is an observation image of the detection camera. Using the small area W around the point mb on the image observed by the reference camera as a template, image correlation values at several points on the epipolar line of the detected camera image are obtained. In the case of the example shown in this figure, the distance resolution is six points of m1 to m6, and the distance numbers 1 to 6 are, for example, 1 m, 2 m
m, 3 m, 4 m, 5 m, and 6 m.

As the image correlation value of each point, for example, an evaluation value obtained by using the following equation (1) can be used. In the following equations, I (x) indicates a luminance value in a reference image captured by the reference camera, and I ′ (x ′) indicates a luminance value of a detected camera image captured by the detection camera.

[0021]

(Equation 1)

M1 to m6 in FIG. 2 obtained by using the above equation
Among the evaluation values at the six points, the lowest point is defined as the corresponding point.
This is shown in the lower graph of FIG. In the case of the example of FIG. 2, the position of m3, that is, the position of 3 m from the camera is set as distance data. In addition, it is also possible to obtain the lowest point in a portion other than the sample data by executing an interpolation process between the sampling data. When this interpolation processing is performed, the point between m3 and m4 in the graph of FIG. 2 is the minimum evaluation value, and in this case, the measurement target is assumed to be a distance of about 3.3 m from the camera. Note that the relationship between the epipolar line and the position on the epipolar line and the distance to the object is determined in advance by calibration. For example, for all pixels on the reference camera image, the coordinates of corresponding points on the detected camera image corresponding to each distance are stored in a table.

As described above, by repeatedly executing the matching process between the reference camera image and the detected camera image for the pixels at each measurement point, three-dimensional shape data can be obtained for all the pixels.

[0024]

However, in practice, it is difficult to accurately associate all pixels from the reference camera image and the detected camera image. This is because it becomes difficult to perform matching in a matching process for an object having almost no features (shade, shape, color, and the like) such as a white wall and a human face.

Since the corresponding points between the images are searched using the image of the small area around each pixel, there is no pattern on the object surface,
If there is little change in the luminance value of each pixel of an image captured by the reference camera and the detection camera, it becomes difficult to find a corresponding point. As a method of solving this, there is a method of irradiating a non-periodic random pattern to the measurement target and capturing the measurement target together with the pattern. According to this pattern irradiation method,
It is possible to take an image with a random pattern and perform a process of associating the images based on the pattern. As described above, it is effective to use an image irradiated with a random pattern to generate a distance image.

On the other hand, in order to paste the color of the actual measurement target (called a texture) on the three-dimensional shape (for example, a polygon) obtained by the above-described method, the object to be measured is photographed with, for example, visible light. You need an image. However, the image obtained by irradiating the above random pattern has a random pattern projected on the surface of the object, and is provided with a pattern different from the original color or pattern of the object. It cannot be treated as the color of an object.

As a method for obtaining an original color of an object, that is, a texture image, there is a method of performing image processing on a pattern-added image obtained by capturing an object on which a random pattern is projected and deleting the pattern. But,
This image processing is complicated, and it is difficult to completely reproduce the color of the object.

The present invention has been made in view of the above-mentioned problems of the prior art, and has an image processing apparatus and an image processing method capable of obtaining a texture image without performing the above-described image processing. , And a program providing medium.

[0029]

SUMMARY OF THE INVENTION The present invention has been made in consideration of the above problems, and a first aspect of the present invention is based on an image obtained by irradiating a measurement target with pattern light and projecting the pattern light. In an image processing apparatus that generates distance data of a measurement target, a pattern projection unit that projects pattern light onto the measurement target, and an imaging that captures an image of the measurement target in pixel units and stores luminance data of the image for each pixel An image of the element and a plurality of pixels constituting the image sensor are captured with the pattern light projected on some of the pixels, and a texture image with the pattern light removed is captured on the other pixels constituting the image sensor. Optical filter configured as described above, the image data stored in the image sensor,
A data selecting unit that separates and outputs the pattern image capturing pixel data obtained by capturing the pattern-added image and a texture image capturing pixel data that captures the texture image; A distance image generating unit that calculates distance data of the measurement target based on the pattern-added image.

Further, in one embodiment of the image processing apparatus according to the present invention, the optical filter has different light transmission characteristics corresponding to each of a pixel for taking a pattern image and a pixel for taking a texture image in the image sensor. Characterized in that an optical filter having the following is arranged.

Further, in one embodiment of the image processing apparatus according to the present invention, the optical filter includes two different types of band cut filters for cutting light of two different wavelength bands for capturing an image with a pattern of the image sensor. It is characterized in that pixels and texture image capturing pixels are arranged in correspondence with each other.

Further, in one embodiment of the image processing apparatus of the present invention, the optical filter has different light transmission characteristics corresponding to each of the image capturing pixel with a pattern and the texture image capturing pixel of the image sensor. It is a configuration in which different types of optical filters having
At least one of the optical filters is constituted by a polarizing plate.

Further, in one embodiment of the image processing apparatus according to the present invention, the optical filter has different light transmission characteristics corresponding to each of the image capturing pixel with pattern and the texture image capturing pixel of the image sensor. It is a configuration in which different types of optical filters having
At least one of the optical filters is constituted by a light intensity reduction filter for reducing light intensity.

Further, in one embodiment of the image processing apparatus according to the present invention, the pattern light projecting means has a structure for projecting pattern light of infrared light onto the object to be measured, and the optical filter includes the image pickup device. Characterized in that it has a configuration in which an infrared light cut filter is arranged corresponding to a pixel for taking a texture image.

Further, in one embodiment of the image processing apparatus according to the present invention, the optical filter has a configuration in which a visible light cut filter is arranged corresponding to a pixel of the image pickup device for taking an image with a pattern. And

Further, in one embodiment of the image processing apparatus of the present invention, the optical filter has different light transmission characteristics corresponding to each of the image capturing pixel with pattern and the texture image capturing pixel of the image sensor. It is a configuration in which different types of optical filters having
It is characterized in that different optical filters are arranged in adjacent pixels of the image pickup device and arranged optical filters are arranged.

Further, in one embodiment of the image processing apparatus according to the present invention, the optical filter has a different light transmission characteristic corresponding to each of the image capturing pixel with a pattern and the texture image capturing pixel of the image sensor. It is a configuration in which different types of optical filters having
A configuration in which different optical filters are arranged so that about one-half of the number of pixels constituting the image pickup device is a pixel for capturing an image with a pattern and the remaining one-half of the pixels are pixels for capturing a texture image. Features.

Further, in one embodiment of the image processing apparatus of the present invention, the image processing apparatus measures a three-dimensional shape of the measurement object using images obtained by photographing the measurement object from different directions, and applies a stereo method. An image processing apparatus that generates a distance image by using a first camera that captures an image of the measurement target, a second camera that captures an image of the measurement target from a direction different from the first camera, and the first camera. Distance image that generates parallax data by an image-to-image association process between a first image captured by the second camera and a second image captured by the second camera, and generates a distance image based on the parallax data Generating means, the first camera;
And at least one of the second camera captures an image of the measurement target in pixel units, and an image sensor that stores luminance data of an image for each pixel, and a partial pixel of a plurality of pixels forming the image sensor. An optical filter configured to capture an image with a pattern by projecting the pattern light, and capture a texture image from which the pattern light has been removed at another pixel configuring the image sensor. And

Further, according to a second aspect of the present invention, there is provided an image processing method for irradiating pattern light on a measurement object and generating distance data of the measurement object based on an image obtained by projecting the pattern light. A texture image obtained by capturing an image with a pattern by projecting pattern light at a part of a plurality of pixels constituting the image sensor, and removing the pattern light at other pixels constituting the image sensor. Image capturing step for capturing the image data, the image data stored in the image sensor, the pattern image capturing pixel data capturing the pattern-added image, and the texture image capturing pixel data capturing the texture image. Image data separating step of separating and outputting the image data, and separating in the image data separating step. In the image processing method characterized by comprising: a distance image generation step of calculating the distance data of the measurement target on the basis of the force has been the patterned image.

Further, in one embodiment of the image processing method according to the present invention, the optical filter has different light transmission characteristics corresponding to each of a pixel for taking a pattern image and a pixel for taking a texture image in the image sensor. Wherein the image capturing step captures a patterned image and a texture image simultaneously via optical filters having different light transmission characteristics.

Further, in one embodiment of the image processing method of the present invention, a three-dimensional shape of the object to be measured is measured by using images of the object to be measured from different directions, and a distance image is generated by applying a stereo method. An image processing method, wherein an image to be measured is simultaneously captured by a first camera and a second camera arranged in a different direction from the first camera, and the image is captured by the first camera and the second camera. Each of the cameras captures a textured image obtained by projecting a pattern image at a part of a plurality of pixels constituting the configured imaging device and removing the pattern light at other pixels constituting the imaging device. An image capturing step for capturing at
The image data stored in the image sensor of the camera and the second camera is separated into pattern image capturing pixel data capturing the pattern-added image and texture image capturing pixel data capturing the texture image. An image data separating step to be output, a first pattern-attached image captured by the first camera, which is separated and output in the image data separating step, and a second pattern-attached image captured by the second camera. A distance image generating step of calculating the distance data of the measurement target.

Further, according to a third aspect of the present invention, a computer system executes image processing for irradiating pattern light on a measurement target and generating distance data of the measurement target based on an image obtained by projecting the pattern light. A program providing medium for providing a computer program, wherein the computer program is a step of capturing an object to be measured by an image sensor, and a pattern in which pattern light is projected on some of a plurality of pixels constituting the image sensor. Capturing an image with the image, capturing an image of the texture with the pattern light removed at the other pixels constituting the image sensor, and capturing the image with the pattern using the image data stored in the image sensor. The captured pixel data and the texture image Image data separating step of separating and outputting the texture image capturing pixel data, and a distance image generating step of calculating the distance data of the measurement target based on the patterned image separated and output in the image data separating step. And a program providing medium characterized by having:

The program providing medium according to the third aspect of the present invention is, for example, a medium for providing a computer program in a computer-readable format to a general-purpose computer system capable of executing various program codes. . The form of the medium is not particularly limited, such as a storage medium such as a CD, an FD, and an MO, and a transmission medium such as a network.

Such a program providing medium defines a structural or functional cooperative relationship between the computer program and the providing medium for realizing the functions of a predetermined computer program on a computer system. Things. In other words, by installing the computer program into the computer system via the providing medium, a cooperative operation is exerted on the computer system, and the same operation and effect as the other aspects of the present invention can be obtained. You can do it.

Still other objects, features and advantages of the present invention are:
It will become apparent from the following more detailed description based on the embodiments of the present invention and the accompanying drawings.

[0046]

FIG. 3 is a block diagram showing the configuration of an image processing apparatus according to the present invention. FIG. 3 shows an apparatus for measuring a three-dimensional shape or a distance image of a measurement target by projecting an image pattern. For example, an image pattern composed of random textures (for example, a binary point or a group of light and shade random points) is generated by a light projection image pattern generation device, and the image pattern is projected onto a measurement target by a light projector. By projecting a random texture on the measurement target as described above, even if the measurement target is a target having almost no features (shade, shape, color, etc.) such as a white wall or a human face, matching processing between images is performed. Can be performed more accurately.

The outline of the image processing apparatus shown in FIG. 3 will be described. The image processing apparatus measures a three-dimensional shape of a measurement target by applying the stereo method described above, and has a three-dimensional shape, for example, a position at a different angle with respect to a measurement target object such as a human face. An image is captured using two cameras arranged in the camera, a so-called reference camera and a detection camera, and based on the images captured by the two cameras, the surface shape of the object to be measured is determined based on the stereo method described in FIG. It is to measure.

The random pattern is irradiated by the light projecting pattern irradiation means 311. This light emitting pattern is an aperiodic light emitting pattern based on a uniform random number or a normal random number.
The pattern to be projected is, for example, an aperiodic light projection pattern, for example, a pattern having no periodicity in dot size, line length, thickness, position, density, and the like.

The non-periodic pattern is formed in advance on a slide or the like, for example. In this embodiment, the pattern is projected on the object to be measured by irradiating infrared light through the slide. Alternatively, a random pattern may be dynamically generated using a transmissive liquid crystal display element or the like, and the random pattern may be projected using infrared light. By imaging the non-periodic pattern formed on the surface of the measurement target 300 together with the measurement target, it is possible to easily associate the image data of the reference camera and the detection camera when generating the distance image.

Note that, as shown in FIG.
Irradiating the random texture projection pattern on the object has few features that distinguish each part of the surface, such as a wall or a human face, and it is difficult to associate the reference camera image with the detected camera image. In some cases, a random pattern is added to a captured image to make it easier to execute an image-to-image association process. The pattern to be illuminated may be any pattern that can execute the image-to-image association process,
If the measurement target itself has a characteristic configuration, a simple pattern configuration may be used.

The cameras 1 and 301 and the cameras 2 and 302
It is arranged in a different gaze direction with respect to the measurement target 300,
An image in each gaze direction, that is, a measurement target 300 on which a random pattern is projected is photographed. One corresponds to the reference camera, and the other corresponds to the detection camera. Images taken by each camera are taken at synchronized timing. The images captured by the cameras 1 and 301 are divided into a pattern-added image 1 and 303, which is an image to which a pattern is added, and a texture image 1 and 304, which reflect the original color of the measurement target 300 to which no pattern is added. It is taken separately. This separation method will be described later with reference to FIG. On the other hand, the images captured by the cameras 2 and 302 also include a pattern-added image 2 and a pattern-added image 2305 and a measurement target 3 to which no pattern is added.
The image is separated and taken into texture images 2 and 306 reflecting the original color of 00.

The image association means 307 includes the cameras 1
Patterned images 1, 30 captured by 301
3 and a pattern-added image 2305 captured by the camera 2302, and associating processing between the respective light-projection pattern images. That is, a process of associating each pixel of the image captured by the cameras 1 and 301 with the pixel of the detected image captured by the camera 2 and 302 is performed, and the parallax data of the detected image with respect to the reference image is stored in the distance image generation unit 30.
8 is output. The process of associating the images in the image associating unit 307 is performed using a calibration parameter in which a correction value or the like of each image is set.

The light projection pattern emitted by the light projection pattern irradiating means 311 is a non-periodic pattern as described above, and it is relatively easy to associate images taken by each camera. The distance image generation unit 308 generates a distance image based on the disparity data generated by the image association unit 307.

The processing in each block having the configuration shown in FIG. 3 and the control of data transfer between blocks are controlled by control means (not shown), and predetermined memories, for example, semiconductor memories such as RAM and ROM, magnetic disks, and optical disks And the like can be controlled by a control program recorded in a storage medium such as. Further, the user can input commands, parameters, and the like using input means (not shown) to change each control mode.

The image processing apparatus shown in FIG. 3 is an apparatus capable of simultaneously taking an image with a pattern to be measured to which pattern light is applied and a texture image to be measured with the pattern light removed. According to this configuration, a highly accurate distance image based on the measurement target to which the pattern light is added and a texture image that is the measurement target image from which the pattern light has been removed can be obtained at the same time. A three-dimensional image can be easily obtained. That is, the conventional image processing for removing a pattern from a pattern-added image is not required.

FIG. 4 shows the internal structure of the camera unit in the image processing apparatus of the present invention, that is, the cameras 1, 301 and 2, 302 in FIG. A captured image of a measurement target is captured by an image sensor 402 via a lens 401. The imaging element 402 is, for example, a CCD (Charged
coupled device) or CMOS
(Complementary metal oxidid
and each pixel is constituted by an element in which light is converted into electric charge by photoelectric conversion in each pixel.

The front surface of the image sensor 402, that is, the lens 4
On the 01 side, an optical filter 403 is formed corresponding to each pixel of the image sensor 402. That is, the image sensor 4
Reference numeral 02 denotes a configuration for capturing a captured image via the optical filter 403. The optical filter 403 has a configuration in which two types of filters, an optical filter A and an optical filter B, are associated with each pixel. For example, the optical filter A is a visible light cut filter, and the optical filter B is an infrared light cut filter.

FIG. 5 shows two examples of the optical filter 403. 5A and 5B, one square corresponds to one pixel of the image sensor 402. The example of FIG. 5A is an example in which adjacent pixels have different filters. FIG. 5B shows an example in which four pixels are set as one set and correspond to the same filter. Note that these drawings show a part of the arrangement of the optical filter 403. Actually, a pattern similar to the arrangement shown in FIG. 5 is arranged corresponding to all the pixels of the image sensor 402. is there.

FIG. 6 shows two different examples of the optical filter 403. 6 (3) and (4), one frame corresponds to one pixel of the image sensor 402. The example of FIG. 6 (3) is an example in which the same filter is arranged in the column direction and pixels in adjacent columns have different filters. FIG. 6D shows an example in which the same filter is arranged in the row direction and pixels in adjacent rows have different filters.

The image captured by the image sensor 402 via the optical filter 403 as shown in FIG. 5 or FIG.
It is read by the reading unit 404 shown in FIG. The reading unit 404 converts the electric charge of each pixel of the image sensor 402 into a voltage, reads the voltage, and outputs the voltage to the A / D conversion unit 405. The A / D converter converts the voltage signal of each pixel into digital data indicating the luminance of each pixel,
6 is stored.

The luminance data stored in the frame memory 406 is sequentially output to the luminance data selecting section 407 for each pixel, and the luminance data of the pixel to which the optical filter A is associated and the optical filter B are associated. And separates the luminance data of the pixels. The luminance data selection unit 407 has the configuration data of the optical filter 403 shown in FIG. 5, holds the position data of each pixel based on this configuration data, and separates and outputs the luminance data of each pixel.

FIG. 7 shows an example of luminance data separated and output. FIG. 7 shows separated output data obtained when the optical filter of FIG. 5A is used.

FIG. 7A shows data in which luminance data of pixels associated with an optical filter A (visible light cut filter) is integrated. In each pixel, the portion indicated by (A) is a pixel having luminance data, and the portion indicated by (-) is a pixel having no luminance data. The pixel indicated by (A) has luminance data based on the image filtered by the optical filter A (visible light cut filter). That is, an image with a pattern, which is an image obtained by capturing the pattern light irradiated by the infrared light, is captured in a portion (A) of FIG.

On the other hand, FIG. 7B shows data in which luminance data of pixels associated with an optical filter B (infrared light cut filter) is integrated. In each pixel, a portion indicated by (B) is a pixel having luminance data, and a portion indicated by (-) is a pixel having no luminance data.
The pixel indicated by (B) has luminance data based on the image filtered by the optical filter B (infrared light cut filter). That is, the pattern irradiated by the infrared light is cut by the filter B, and the texture image, which is an image obtained by capturing the texture image reflecting the original color data of the measurement target, is shown in FIG.
The part is taken.

The luminance data selector 407 shown in FIG. 4 outputs the two separated pixel data shown in FIG. 7, respectively. 4 has at least the camera 1 shown in FIG.
The luminance data shown in FIG. 7A is output as a pattern-added image 1,303, and the luminance data shown in FIG. 7B is output as a texture image 1,304.

On the other hand, the cameras 2 and 302 may be configured to obtain a pattern-added image for generating a distance image and a texture image as the same configuration as the camera 1, or may be configured to obtain only a pattern-added image for generating a distance image. Good.

If the cameras 2 and 302 are configured to obtain a pattern-added image for generating a distance image and a texture image in the same manner as the camera 1, the same luminance data as shown in FIG. FIG. 7
The luminance data similar to that shown in (2)
Output as 306. Patterned images 2 and 305 captured by cameras 2 and 302 are output to image-to-image associating means 307 as distance image generation data, and texture images 2 and 306 captured by cameras 2 and 302 are output by camera 1. , 301 can be used as interpolation data for the texture images 1 and 304 captured by the camera.

Note that the texture images are stored in the cameras 1 and 301.
Camera 2306 if it is not necessary to capture a texture image
May be configured to obtain only an image with a pattern for generating a distance image. In this case, the cameras 2 and 302 need not have a configuration in which the two types of optical filters shown in FIG. 4 are associated with each pixel, and have a configuration in which a visible light cut filter is associated with all pixels of the image sensor. be able to.

The image-to-image associating means 307 shown in FIG.
A pattern-added image 1,303 taken by the camera 1,301 and a pattern-added image 2,305 taken by the camera 2,302 are input. These pattern-added images are the image data shown in FIG. 7A and do not hold the data for all the pixels. It is possible to do so only on the basis of: If necessary, an interpolation process may be performed on a pixel having no luminance data based on luminance data of peripheral pixels. Alternatively, it is also possible to use luminance data shifted in the adjacent X-axis direction or Y-axis direction for a pixel having no luminance data. When this processing is performed, there is a possibility that a different point shifted by one pixel is set as a corresponding point in the inter-image association processing. However, when the entire image is observed macroscopically, this micro error is a problem. No.

Also, the texture images 1 and 304 obtained by the cameras 1 and 301 do not hold data on all pixels as shown in FIG. 7B, similarly to the distance image generation image. However, even if an image obtained based on these pixel data is used as a texture image, there is no macro problem. If necessary,
For pixels having no data, an interpolation process based on data of peripheral pixels may be performed. In addition, cameras 2, 30
Interpolation processing may be performed using the texture images 2 and 306 obtained from Step 2.

Although the configuration shown in FIG. 4 has been described as the configuration of the camera unit, it is not necessary that all the components of these configurations be configured in the camera. For example, the frame memory 406, the luminance data selection unit 407, etc. May be configured outside the camera to transfer data from the camera. Also, the luminance data is directly transmitted from the A / D converter 405 without passing through the frame memory 406 to the luminance data selector 407.
May be output. FIG. 8 shows an example of this configuration.

Next, a specific example of image processing using the image processing apparatus of the present invention will be described with reference to FIG. In FIG. 9, a measurement target 901 is a cup, and the measurement target 901 is irradiated with a random pattern from the pattern light emitting unit 900 using infrared light. This pattern is a pattern for enabling the image-to-image associating process by the image-to-image associating means to be executed with high accuracy.

Images of the object to be measured 901 illuminated with the pattern by the cameras 1 and 902 and the cameras 2 and 903 are taken from different directions. The positions of the cameras 1 and 902 and the cameras 2 and 903 are fixed, and parameters necessary for obtaining the distance are calculated in advance by calibration.

Camera 1 902 and camera 2 903
Has an image sensor in which two different optical filters A and B are associated with each pixel as described in FIG. 5, and an infrared light cut filter and a visible light cut filter correspond to each pixel. Capture an image through the attached filter. The combination of the optical filters A and B is not limited to the visible light cut filter and the infrared light cut filter.

For example, a pattern image is captured using a band cut filter that passes only infrared light as the optical filter A and cuts not only visible light but also light of another wavelength such as ultraviolet light. A configuration may be adopted in which a texture image is captured using an external light cut filter. Alternatively, a configuration may be adopted in which a texture image is captured using an infrared light cut filter as the optical filter B, and a pattern image is captured as a configuration in which no filter is attached to the optical filter A portion.

With such a configuration, the texture images 1 and 904 from which the irradiation patterns are removed and the images 1 and 906 with the irradiation patterns are removed from the cameras 1 and 902.
Are captured, and the texture images 2 and 905 from which the irradiation pattern is removed and the images 2 and 907 with the irradiation pattern are captured from the cameras 2 and 903.

The distance image generation unit 908 includes the cameras 1 and 90
2 and the images 2 and 90 with the irradiation pattern taken by the cameras 2 and 903, respectively.
7 is input, parallax data is acquired by using the parameters calculated by the previously executed calibration to obtain parallax data, and a distance image is generated. The distance image generation unit 908 can execute the image-to-image association process using the image in which the random pattern is projected on the measurement target 901, so that it is possible to generate the high-accuracy distance image 909.

As texture images on which no pattern is projected, texture images 1 and 904 photographed by cameras 1 and 902 and texture images 2 and 905 photographed by cameras 2 and 903 are photographed simultaneously with a pattern-added image. . High-precision three-dimensional image data can be generated based on the distance image data generated by the distance image generation unit 908 and the texture images 1 and 904. Note that interpolation processing may be performed on the texture images 1 and 904 captured by the cameras 1 and 902 using the texture images 2 and 905 captured by the cameras 2 and 903, respectively.

FIG. 10 shows a flow of the distance image generation processing in the image processing apparatus of the present invention. Each step of the flow in FIG. 10 will be described.

Step S1001 is a step of irradiating an object to be measured with an infrared light pattern and simultaneously capturing images of the measurement object by cameras 1 and 2 installed in different directions.

In step S1002, the texture image 1 and the pattern-added image 1 are acquired from the camera 1,
Is a step of acquiring a texture image 2 and a pattern-attached image 2 from. This is a step executed by the configuration shown in FIG. 4 or FIG. 8 described above.

In step S1003, in the distance image generation unit, the image 1 with the irradiation pattern
This is a step of inputting the image 2 with the irradiation pattern captured by the camera 2 and executing an image-to-image association process using calibration parameters to acquire parallax data and generate a distance image.

In the above-described embodiment, the pattern irradiating light is applied to the object to be measured using infrared light, and the texture image from which the pattern is removed is provided by the configuration in which the infrared light cut filter is provided as a filter of the camera. Although the configuration for capturing an image was described, for example, pattern irradiation light was applied to the measurement target using ultraviolet light, and an ultraviolet light cut filter was used as a camera filter, and a pixel corresponding to the ultraviolet light cut filter was captured. A configuration in which a texture image is obtained based on an image may be adopted.

Further, in addition to the method of separating the light from visible light by a wavelength band such as infrared light and ultraviolet light, for example, a structure is adopted in which pattern irradiation is performed as a fixed polarized light, and an optical filter provided in an image sensor is provided. It is also possible to adopt a configuration in which a polarizing plate is applied so that pattern light is not input to a pixel portion that has passed through the polarizing plate.

Alternatively, the pattern light is irradiated onto the object to be measured by irradiation light having a low light intensity (brightness), and a light intensity reduction filter for reducing the light intensity is provided in the image pickup device, so that the pixels passing through the filter can be obtained. The configuration may be such that the pattern light is not input to the portion or the pattern light is made inconspicuous.

As described above, the filter configuration of the image sensor is set to a combination of the pattern light input pixel and the pattern light non-input pixel according to the light characteristic of irradiating the pattern light. Capture, separate these two types of image data according to the pixels that have captured each image data, use the pixel data that captures the pattern light as an image for distance image generation, and use the pixel data that does not capture the pattern light Use as a texture image eliminates the need for complicated image processing such as removing a pattern from an image in which pattern light is projected as in the related art.

In the above-described embodiment, the configuration in which the distance image is obtained from the two light projection pattern images by using two cameras for capturing images has been described.
The present invention can be applied to a multi-baseline stereo system using more than one camera. Furthermore, the configuration of the present invention can be applied not only to the distance measurement based on the stereo method described above, but also to a light cutting method for performing distance measurement by projecting pattern light and a distance measurement by a space code (two-dimensional coded pattern) method. Applicable.

The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the spirit of the present invention. That is, the present invention has been disclosed by way of example, and should not be construed as limiting. Further, what is configured by appropriately combining the above-described embodiments is also included in the scope of the present invention, and in order to determine the gist of the present invention, refer to the claims section described at the beginning. Should.

[0089]

As described above in detail, according to the image processing apparatus and the image processing method of the present invention, a filter for capturing two different images on an image sensor for capturing a measurement object for generating a distance image. And a configuration in which some pixels of the image sensor are used as pattern light input pixels and other pixels are used as pattern light non-input pixels, so that a pattern image for generating a distance image and a texture image from which a pattern has been removed are provided. Two images can be captured at the same time, and a high-accuracy distance image using a pattern image can be generated. In addition, a high-accuracy three-dimensional image can be easily obtained based on distance data based on the generated distance image and a texture image. Image data can be generated.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a stereo method applicable as a three-dimensional information acquisition configuration used in the present invention.

FIG. 2 is a diagram illustrating an image-to-image association process performed in generation of a distance image to which a stereo method is applied.

FIG. 3 is a diagram illustrating a configuration of an image processing apparatus according to the present invention.

FIG. 4 is a diagram illustrating a configuration of a camera unit in the image processing apparatus according to the present invention.

FIG. 5 is a diagram showing a configuration example (part 1) of an optical filter applied in the image processing apparatus of the present invention.

FIG. 6 is a diagram showing a configuration example (part 2) of an optical filter applied in the image processing apparatus of the present invention.

FIG. 7 is a diagram for explaining a pattern-added image obtained by the image processing apparatus of the present invention and a data acquisition pixel configuration of a texture image.

FIG. 8 is a diagram illustrating a configuration (example 2) of a camera unit in the image processing apparatus according to the present invention.

FIG. 9 is a diagram illustrating a specific example of image processing in the image processing apparatus according to the present invention.

FIG. 10 is a diagram illustrating a flow of a process of capturing an image and generating a distance image in the image processing apparatus of the present invention.

[Explanation of symbols]

 300 Measurement object 301 Camera 1 302 Camera 2 303 Patterned image 1 304 Texture image 1 305 Patterned image 2 306 Texture image 2 307 Image-to-image association means 308 Distance image generation means 311 Light projection pattern irradiation means 401 Lens 402 Imaging element 403 Optical filter 404 Readout unit 405 A / D conversion unit 406 Luminance data selection unit 900 Light emission pattern irradiation unit 901 Measurement target 902 Camera 1 903 Camera 2 904 Texture image 1 905 Texture image 2 906 Patterned image 1 907 Patterned image 2 908 Distance image generator 909 Distance image

Continued on the front page F term (reference) 2F065 AA04 AA06 AA53 FF05 FF49 HH00 HH07 LL22 LL24 LL26 LL32 QQ03 QQ24 QQ25 QQ31 2H059 AA10 AA18 CA01 5B057 BA02 BA15 CA08 CA12 CA16 CB08 CB20 DC13 A07

Claims (14)

[Claims] An image processing apparatus for irradiating pattern light on a measurement target and generating distance data of the measurement target based on an image obtained by projecting the pattern light, comprising: a pattern light projecting means for projecting the pattern light on the measurement target; An image of a measurement target is captured in pixel units, and an image sensor that stores luminance data of the image for each pixel, and an image with a pattern obtained by projecting the pattern light at a part of a plurality of pixels constituting the image sensor. Including
An optical filter configured to capture a texture image from which the pattern light has been removed at another pixel configuring the image sensor, and image data stored in the image sensor, a pattern image capturing the image with the pattern. Data selection means for separating and outputting the captured pixel data and the texture image captured pixel data obtained by capturing the texture image; and the measurement object based on the pattern-added image separated and output by the data selection means. An image processing apparatus comprising: a distance image generating unit that calculates distance data of: 2. The optical filter according to claim 1, wherein the image sensor has a configuration in which pixels for capturing a pattern image and optical filters having different light transmission characteristics are arranged in correspondence with the pixels for capturing a texture image. The image processing apparatus according to claim 1, wherein: 3. The optical filter according to claim 1, wherein two different types of band cut filters for cutting light in two different wavelength bands correspond to each of the image capturing pixel with pattern and the texture image capturing pixel of the image sensor. The image processing apparatus according to claim 1, wherein the image processing apparatus is arranged so as to be disposed. 4. The optical filter according to claim 1, wherein different types of optical filters having different light transmission characteristics are arranged corresponding to each of the image capturing pixels with a pattern of the image sensor and the texture image capturing pixels. Yes,
2. The image processing apparatus according to claim 1, wherein at least one of the optical filters is formed of a polarizing plate. 5. The optical filter according to claim 1, wherein different types of optical filters having different light transmission characteristics are arranged corresponding to each of the image capturing pixels with a pattern of the image sensor and the texture image capturing pixels. Yes,
2. The image processing apparatus according to claim 1, wherein at least one of the optical filters includes a light intensity reduction filter that reduces light intensity. 6. The pattern light projecting means has a structure for projecting pattern light of infrared light onto the object to be measured, and the optical filter has a red color corresponding to a pixel of the image sensor for taking a texture image. The image processing apparatus according to claim 1, further comprising a configuration in which an external light cut filter is arranged. 7. The image processing apparatus according to claim 1, wherein the optical filter has a configuration in which a visible light cut filter is arranged corresponding to a pixel of the image pickup device that captures an image with a pattern. 8. The optical filter according to claim 1, wherein a plurality of types of optical filters having different light transmission characteristics are arranged corresponding to each of the image capturing pixels with a pattern of the image sensor and the texture image capturing pixels. Yes,
The image processing apparatus according to claim 1, wherein a different optical filter is arranged in a pixel adjacent to the image sensor. 9. The optical filter according to claim 1, wherein different types of optical filters having different light transmission characteristics are arranged in correspondence with each of the image capturing pixels with a pattern of the image sensor and the texture image capturing pixels. Yes,
A configuration in which different optical filters are arranged so that about one-half of the number of pixels constituting the image pickup device is a pixel for capturing an image with a pattern and the remaining one-half of the pixels are pixels for capturing a texture image. The image processing apparatus according to claim 1, wherein: 10. The image processing apparatus according to claim 1, wherein the image processing apparatus measures a three-dimensional shape of the measurement object using images captured from different directions and generates a distance image by applying a stereo method. A first camera that captures an image of the measurement target; a second camera that captures an image of the measurement target from a different direction from the first camera; a first image captured by the first camera; Distance image generation means for generating disparity data by an image-to-image association process with a second image captured by the two cameras, and generating a distance image based on the disparity data; And at least one of the second camera captures an image of the measurement target in pixel units, stores an image sensor that stores luminance data of the image for each pixel, and a plurality of pixels forming the image sensor. In Part pixels uptake a patterned image projected with the pattern light,
The image processing apparatus according to claim 1, further comprising: an optical filter configured to capture a texture image from which the pattern light has been removed in another pixel included in the imaging element. 11. An image processing method for generating distance data of a measurement target based on an image obtained by irradiating the measurement target with the pattern light and projecting the pattern light, comprising:
An image capturing step of capturing an image with a pattern by projecting pattern light at a part of a plurality of pixels constituting the image sensor, and capturing a texture image with the pattern light removed at another pixel constituting the image sensor; An image to be output by separating the image data stored in the image sensor into pattern image capturing pixel data capturing the pattern-added image and texture image capturing pixel data capturing the texture image. An image processing method, comprising: a data separation step; and a distance image generation step of calculating distance data of the measurement target based on the pattern-attached image separated and output in the image data separation step. 12. The optical filter according to claim 1, wherein the image sensor has optical filters having different light transmission characteristics corresponding to pixels for capturing a pattern image and pixels for capturing a texture image, respectively. The image processing method according to claim 11, wherein in the image capturing step, the image with the pattern and the texture image are captured simultaneously through optical filters having different light transmission characteristics. 13. The image processing method according to claim 1, wherein a three-dimensional shape of the measurement target is measured using images obtained by photographing the measurement target from different directions, and a distance image is generated by applying a stereo method. An image capturing step of simultaneously capturing an image of a measurement target by a first camera and a second camera arranged in a direction different from that of the first camera, wherein an image sensor configured by the first camera and the second camera is Image capturing in which a pattern image is captured by projecting pattern light at a part of a plurality of pixels constituting the image, and a texture image from which the pattern light is removed at other pixels configuring the image sensor is captured by each camera. And, for the image data stored in the image sensors of the first camera and the second camera, An image data separating step of separating and outputting the captured pattern image capturing pixel data and the texture image capturing pixel data capturing the texture image, and the first data separated and output in the image data separating step. A distance image generating step of calculating distance data of the measurement target based on an image with a first pattern captured by a camera and an image with a second pattern captured by the second camera. The image processing method according to claim 11. 14. A program providing medium for providing a computer program for causing a computer system to execute image processing for generating distance data of a measurement target based on an image obtained by projecting the pattern light onto the measurement target and projecting the pattern light. Wherein the computer program is a step of capturing an object to be measured by an image sensor,
An image capturing step of capturing an image with a pattern by projecting pattern light at a partial pixel of the plurality of pixels constituting the image sensor, and capturing a texture image from which the pattern light has been removed at another pixel constituting the image sensor; And an image for separating and outputting the image data stored in the imaging device into pattern image capturing pixel data capturing the pattern-added image and texture image capturing pixel data capturing the texture image. A program providing medium, comprising: a data separation step; and a distance image generation step of calculating distance data of the measurement target based on the pattern-attached image separated and output in the image data separation step.