JP2010276469A - Image processor and image processing method of ranging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor which obtains a captured image required for ranging even in the night and a shadow area where the quantity of photons is reduced, and also to provide an image processing method of a ranging apparatus. <P>SOLUTION: A corresponding point is optimally searched by comparing image signals (comprising W, Mg, Cy, Ye and Ri pixels) obtained by passing them through filters (such as W, Mg, Cy, Ye and Ri filters) higher in order than primary color filters (R, G, B) in the saturation order, based on a signal having a S/N ratio larger than an image signal comprising at least R pixels, G pixels and B pixels. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus capable of measuring a distance to an object with a pair of imaging elements and an image processing method of the distance measuring apparatus.

  For example, as shown in FIG. 1, in an autonomously moving robot RBT or the like, a front obstacle OB is detected to warn of a danger, or in order to avoid this automatically, the distance to an object is measured. Is done. Here, as a distance measuring device that measures the distance to an object, there is an apparatus that uses reflection of electromagnetic waves such as ultrasonic waves and laser light. However, a distance measuring device using ultrasonic waves, laser light, etc. can measure the distance to the object relatively accurately, but the distance measurement range is relatively narrow. There is a problem that it is difficult. On the other hand, as shown in Patent Document 1, there has been proposed an apparatus for measuring a distance by acquiring a captured image using a pair of imaging elements by a stereo camera (stereoscopic method) and performing image processing on the captured image. ing.

JP 2009-14445 A

  By the way, a solid-state imaging device (CCD / CMOS) used for a general digital camera or the like has a primary color filter that transmits visible blue light (B), visible green light (G), and visible red light (R), for each pixel. In response to this, as shown in FIG. 2A, they are physically arranged (so-called Bayer arrangement structure). Such an image sensor has no problem in daytime imaging as shown in FIG. 3A, but the amount of photons decreases at night or in the shade as shown in FIG. There is a problem that an image signal with a small ratio is output, and comparison of images necessary for distance measurement cannot be performed effectively. On the other hand, an infrared camera in which a uniform infrared filter that transmits infrared light is physically arranged in front of the pixels as shown in FIG. Although photons can be obtained, there is a problem that only a so-called black-and-white grayscale image can be taken because there is no color data in the output image signal. A similar problem occurs when a stereo camera is mounted on the surveillance camera MC (FIG. 1).

  An object of the present invention is to provide an image processing apparatus and an image processing method for the distance measuring apparatus that can obtain a captured image necessary for distance measurement even at night or in the shade where the amount of photons is small.

The image processing apparatus according to claim 1 is provided with a plurality of pixels which are provided apart from each other by a predetermined interval and convert light into a pixel signal by receiving light from an object through a plurality of different filters. Based on the parallax obtained by the parallax detection means, the parallax detection means for obtaining the parallax between the imaging elements in the separation direction by comparing the image signals output from the pair of imaging elements, and the pair of imaging elements In the image processing apparatus having a distance measuring means for sequentially obtaining the distance to the object,
Image signals or image data obtained through a filter higher than the primary color filter in a saturation order of filters in which the pixels of the image sensor are likely to be saturated in the different filters are compared. In this specification, “pixel is saturated” means “data acquired by the pixel is saturated”.

  In recent years, in order to enable color images even at night imaging, a solid-state imaging device employing a so-called complementary color filter having sensitivity in a plurality of bands of R, G, B, and Ir has been developed. FIG. 4A is a diagram illustrating an arrangement according to an example of a complementary color filter. The complementary color filter shown in FIG. 4A has one Ye filter, Ri filter, Ir filter, and W filter each having a sensitive wavelength band in the visible wavelength region and the infrared wavelength region corresponding to each pixel. A unit filter group is formed. In contrast, conventional R filters, G filters, and B filters are referred to as primary color filters.

  In the example of FIG. 4A, in the unit filter group, Ri filters are arranged in the first row and first column, W filters are arranged in the second row and first column, and Ye filters are arranged in the first row and second column. The Ir filters are arranged in the second row and the second column. Pixels corresponding to these filters are referred to as Ri pixel, W pixel, Ye pixel, and Ir pixel, respectively. However, the above is an example, and Ri patterns, Ir pixels, W pixels, and Ye pixels may be arranged in a zigzag pattern in other patterns.

  Since the Ye pixel includes a Ye filter, it captures an image component Ye (original image component) and an infrared image component, which are visible color image components of Ye. Since the Ri pixel includes a Ri filter, it captures an image component Ri (original image component) and an infrared image component, which are visible color image components of Ri. Since the Ir pixel includes an Ir filter, an image component Ir (original image component) that is an infrared image component is captured. Since the W pixel does not include a filter, an image component W (original image component) that is a luminance image component including the visible luminance image component and the image component Ir is captured. Thus, the four filters have different spectral transmission characteristics.

  FIG. 5 is a diagram showing the spectral transmission characteristics of the Ye, Ri, and Ir filters, where the vertical axis indicates the transmittance (sensitivity) and the horizontal axis indicates the wavelength (nm). A graph indicated by a dotted line indicates the spectral sensitivity characteristic of the pixel in a state where the filter is removed. It can be seen that this spectral sensitivity characteristic has a peak near 600 nm and changes in a convex curve. In FIG. 5, 400 nm to 700 nm is a visible wavelength region, 700 nm to 1100 nm is an infrared wavelength region, and 400 nm to 1100 nm is a sensitive wavelength band.

  As shown in FIG. 5, the Ye filter has a characteristic of transmitting light in the sensitive wavelength band excluding the blue region in the visible wavelength region. Therefore, the Ye filter mainly transmits yellow light and infrared light.

  The Ri filter has a characteristic of transmitting light in a sensitive wavelength band excluding a blue region and a green region in the visible wavelength region. Therefore, the Ri filter mainly transmits red light and infrared light.

  The Ir filter has a characteristic of transmitting light in a sensitive wavelength band excluding the visible wavelength region, that is, in the infrared wavelength band. W indicates a case where no filter is provided, and all light in the sensitive wavelength band of the pixel is transmitted.

  To achieve similar characteristics, Ye, M (magenta) + Ir, C (cyan) + Ir instead of Ye, Ri, Ir (M + Ir shields only green and C + Ir only red) Is also possible. However, the Ri pixel, the Ir pixel, and the Ye pixel can make the spectral transmission characteristics steep, and, for example, the spectral transmission characteristics are better than those of the M + Ir filter and the C + Ir filter. In other words, each of the M + Ir filter and the C + Ir filter has a characteristic of shielding only the green region and the red region, which are partial regions in the center, in the sensitive wavelength band. It is difficult to provide steep spectral transmission characteristics such as Ir filter and Ye filter. Therefore, each of the M + Ir filter and the C + Ir filter cannot extract the RGB image components with high accuracy even if the calculation is performed. Therefore, by configuring the imaging device with Ri pixels, Ir pixels, Ye pixels, and W pixels, the performance of the imaging device can be improved. Furthermore, complementary color filters as shown in FIGS. 4B to 4F can also be used. FIG. 4G is a diagram showing the correlation between the filters.

  FIG. 6 is a diagram illustrating a flowchart for performing interpolation processing on an image signal from an image sensor using a complementary color filter. First, one frame of original image data is captured by an image sensor using a complementary color filter. Thereby, image components Ye, Ri, Ir, and W are obtained (step S1).

  Here, the imaging element images the image component Ye by the Ye pixel, images the image component Ri by the Ri pixel, images the image component Ir by the Ir pixel, and images the image component W by the W pixel.

  Next, color interpolation processing is performed on the image components Ye, Ri, Ir, and W. Further, calculation of dR = Ri−Ir, dG = Ye−Ri, and dB = W−Ye is performed to calculate the color signals dR, dG, and dB (step S2).

  Next, calculation of Y = 0.3 dR + 0.59 dG + 0.11 dB, Cb = dB−Y, Cr = dR−Y is performed to calculate the luminance signal Y and the color difference signals Cr and Cb (step S3).

  Next, the color difference signals Cr and Cb are smoothed to calculate the color difference signals Crs and Cbs (step S4). Here, as the smoothing process, for example, a cascade filter process, which is a filter process that multi-resolutions the color difference signals Cb and Cr by repeatedly using a relatively small size low-pass filter such as 5 × 5, is adopted. Also good. Further, a filter process using a low-pass filter of a predetermined size having a relatively large size may be employed.

  Also, an edge-preserving filter that smoothes areas other than edges without blurring the subject that emits light (smoothing when the signal level difference between pixels is smaller than a certain reference value, and smoothing the part larger than the reference value Filter) processing that does not enable conversion may be employed. Note that it can be estimated that light emission is detected by comparing an infrared component and a visible light component.

  As described above, by performing the smoothing process on the color difference signals Cb and Cr, the noise components included in the color difference signals Cb and Cr are blurred, and the S / N ratio of the color difference signals Cb and Cr can be improved.

  Next, a calculation of Yadd (1/4) × (Ri + Ir + W + Ye) is performed to calculate a luminance signal Yadd (step S5).

  Next, color difference signals Crm and Cbm are calculated as Crm = Crs × Yadd / Y and Cbm = Cbs × Yadd / Y (step S6).

  Next, the color signals dR ′, dG ′, and dB ′ are calculated from the luminance signal Yadd and the color difference signals Crm and Cbm by performing inverse conversion in step S2 (step S7). As described above, color interpolation processing can be performed.

  By using such a complementary color filter, RGB signals necessary for forming a color image can be obtained by performing not only daytime imaging but also nighttime and shaded imaging and performing appropriate color interpolation processing. Can do. On the other hand, the distance to the object can also be measured using a signal from a pixel that has received light that has passed through the complementary color filter.

  First, a method for obtaining the distance to the object by the technique described in Japanese Patent Application Laid-Open No. 2009-14445 will be described. FIG. 7 is a diagram illustrating a state in which the distance to an object is measured with a stereo camera. In FIG. 7, the cameras 1 and 2 having a pair of imaging elements are arranged so as to be separated from each other by a predetermined baseline interval L and the optical axes are parallel to each other. The captured images of the objects captured by the cameras 1 and 2 are searched for corresponding points on a pixel-by-pixel basis using, for example, the SAD (Sum of Absolute Difference) method, which is a corresponding point search method. The parallax with respect to the object in the direction can be obtained, and the distance to the object can be obtained based on the following equation based on the obtained parallax.

In FIG. 7, at least the focal length (f), the number of pixels of the image sensor (CCD), the size of one pixel (μ) are equal to each other, and two cameras 1 and 2 are used, and only a predetermined baseline length (L) is obtained. The object OB is photographed by arranging the optical axes 1X and 2X in parallel so as to be separated from each other to the left and right. At this time, in the example of FIG. 3A, the pixel number of the tip of the left tree on the imaging surface 1b of the camera 1 (assuming counting from the left end or the right end) is the same as x1, on the imaging surface 2b of the camera 2. If the pixel number at the tip of the left tree is x2 (assuming y is equal), the parallax (number of shifted pixels) on the imaging surfaces 1b and 2b is d (= x1-x2), and the object OB The distance (Z) up to is similar to the triangle shown by hatching. Therefore,
Z: f = L: μ × d
From the relationship
Z = (L × f) / (μ × d) (1)
Can be obtained.

  One of the important image processes in the distance measurement described above is to search for corresponding points in pixel units in captured images obtained from two cameras and to accurately detect the same object (in the example of FIG. 3A). It is to find a pixel corresponding to the tip of the left tree. Corresponding point search in the prior art performs color interpolation processing from RGB signals, creates three image data for each of two captured images, compares at least one image data, and uses the same signal value If there is a pixel having, the same object is estimated. This process is called pattern matching.

  FIG. 8A is a diagram showing color interpolation processing for the primary color filter. In such color interpolation processing, by replacing the signal from the G pixel between two adjacent R pixels with the average value of the signal of the R pixel sandwiching the signal, the first image data (only the R pixel signal ( G pixel signal by replacing the signal from the R pixel or the B pixel between two adjacent G pixels with the average value of the G pixel signals sandwiching it The second image data consisting of only the B pixel is formed, and the signal from the G pixel between the two adjacent B pixels is replaced with the average value of the B pixel signals sandwiching the second image data, thereby consisting only of the B pixel signal Third image data is formed.

  FIG. 8B is a diagram showing color interpolation processing for an example of a complementary color filter. In such color interpolation processing, the signal from the Ye pixel or the W pixel between two adjacent Ri pixels is replaced with the average value of the signals of the Ri pixels sandwiching this, whereby the first pixel consisting of only the signals of the Ri pixels. A second image comprising only Ye pixel signals is formed by forming image data and replacing the signal from the Ri pixel or Ir pixel between two adjacent Ye pixels with the average value of the signals of the Ye pixels sandwiching the image data. By forming image data and replacing the signal from the Ri pixel or Ir pixel between two adjacent W pixels with the average value of the signals of the W pixels sandwiching the image data, the third data consisting of only the signals of the W pixels By forming the image data and replacing the signal from the W pixel or Ye pixel between two adjacent Ir pixels with the average value of the Ir pixel signal between them, only from the Ir pixel signal Forming a fourth image data that.

  Here, when pattern matching is performed for any one of the R pixel, G pixel, and B pixel, in a captured image at night or in the shade, the pixel value becomes too low to be easily hidden by noise. There is a fear that it cannot be done. On the other hand, for example, a W pixel as an example of the complementary color filter has sensitivity over the entire sensitive wavelength band, and thus has a relatively high pixel value not only in the daytime but also in a captured image at night or in the shade. In other words, the W pixel is easily saturated with respect to the R pixel, the G pixel, and the B pixel.

  According to the research result of the present inventor, it was found that the complementary color filter of FIG. 8B has a saturation order as shown in FIG. More specifically, the W pixel is most easily saturated, followed by the Ye pixel and the Ri pixel, and the Ir pixel is most hardly saturated with the primary color filter interposed therebetween. FIG. 10 shows the saturation order according to another complementary color filter. In the saturation order of FIG. 10, the W pixel is most likely to be saturated, then the Mg pixel having the Mg (magenta) filter, the Cy pixel having the Cy (cyan) filter, and the Ye pixel in the complementary color filter pixel group are likely to be saturated. In contrast, the B pixel, G pixel, R pixel, and Ir pixel in the primary color filter pixel group are relatively difficult to saturate.

  The present inventor has derived that the saturation order of a filter in which pixels are likely to be saturated becomes as shown in FIG. 9 or FIG. 10, and in accordance with this saturation order, the primary color filters (R, G, B). By comparing image data (for example, composed of W, Mg, Cy, Ye, Ri pixels) obtained through a higher-order filter (for example, W, Mg, Cy, Ye, Ri filter), at least R pixel, G The present inventors have found that an optimum corresponding point search can be performed based on a signal having an S / N ratio larger than that of image data including pixels and B pixels. This makes it possible to perform effective distance measurement even at night or in the shade. Note that “saturation order of filters in which pixels are likely to be saturated” is synonymous with “order in which sensor sensitivity is good” or “order in which spectral sensitivity band is wide”, and these can be used interchangeably.

  The image processing apparatus according to claim 2, wherein when a pixel is saturated when light from an object is incident through the higher-order filter, the parallax detection unit has a higher saturation order than a filter in which the pixel is saturated. It is characterized in that image signals obtained through a lower filter are compared with each other. Since light passing through a filter (for example, W, Mg, Cy, Ye, Ri filter) that is higher than the primary color filter has a relatively large amount of photons, pixels that receive it (for example, W, Mg, Cy, Ye, Ri) Pixel) has a high sensitivity, but is easily saturated. If the pixels are saturated, there will be many pixels with the same signal value, which may cause a so-called occlusion phenomenon that is a distance measurement error, and there is a possibility that accurate stereo matching cannot be performed. Therefore, when the pixel is saturated when light from the object is incident through a certain filter, the parallax detection means obtains the image signals obtained through the filters lower in the saturation order than the filter in which the pixel is saturated. By comparing, it is possible to perform accurate ranging.

  The image processing apparatus according to claim 3, wherein the parallax detection unit obtains a pixel average value in one image formed by an image signal obtained through a certain filter, and the pixel average value is equal to or greater than a threshold value. It is characterized by determining that is saturated. Thereby, it can be easily determined whether or not it is saturated.

  5. The image processing apparatus according to claim 4, wherein when the light from the object is incident through a certain filter and the pixels in a part of the region are saturated, the parallax detection unit Is characterized in that image signals obtained through a filter that is lower in the saturation order than a saturated filter are compared with each other. For example, when a bright subject such as the sun appears on the screen, an image other than the sun may be used effectively. Thus, when sunlight or the like is directly incident and pixels in a part of the region are saturated, the parallax detection unit is lower than the saturated filter in the saturation order for pixels in the part of the saturated region. Appropriate processing can be performed by comparing image signals obtained through filters.

  The image processing apparatus according to claim 5, wherein the parallax detection unit obtains a pixel average value of a partial region formed by an image signal obtained through a certain filter, and the pixel average value is equal to or greater than a threshold value. It is characterized in that it is determined that pixels in a part of the region are saturated. Thereby, it can be easily determined whether or not it is saturated.

  The image processing apparatus according to claim 6, when comparing the image signals output from the pair of image pickup devices for each filter, selects a filter in which the positions of pixels having the same pixel signal value coincide with each other. The image signals obtained through the filtered filters are compared with each other. Thereby, stereo matching with high accuracy can be performed.

The image processing method according to claim 7 is provided with a plurality of pixels which are provided apart from each other by a predetermined interval and which respectively receive light from the object through a plurality of different filters to convert them into pixel signals. Based on the parallax obtained by the parallax detection means, the parallax detection means for obtaining the parallax between the imaging elements in the separation direction by comparing the image signals output from the pair of imaging elements, and the pair of imaging elements In the image processing method of the distance measuring device having distance measuring means for sequentially obtaining the distance to the object,
If the pixel does not saturate when light from an object enters through the higher order filter among the saturation ranks of the filters in which the pixels of the image sensor are likely to be saturated in the different filters, the pixels are obtained through the higher order filter. Compare image signals
When the pixel is saturated when light from an object enters through the upper filter, the image signals or image data obtained through the lower filter in the saturation order are compared with the filter in which the pixel is saturated. It is characterized by that. The effect of the present invention is the same as that of the first aspect described above.

  According to the present invention, it is possible to provide an image processing apparatus and an image processing method for a distance measuring apparatus that can obtain a captured image necessary for distance measurement even at night or in the shade where the amount of photons is small.

It is a figure which shows the robot carrying the ranging device of this invention. It is a figure which shows a RGB primary color filter and an infrared filter. It is a figure which shows the frame of the daytime (a) and nighttime (b) each image | photographed using a pair of camera. It is a figure which shows the example of a complementary color filter. It is the figure which showed the spectral transmission characteristic of Ye, Ri, and Ir filter, The vertical axis | shaft has shown the transmittance | permeability (sensitivity), and the horizontal axis has shown the wavelength (nm). It is a figure which shows the flowchart which performs the interpolation process of the image signal from the image pick-up element using a complementary color filter. It is a figure which shows the state which measures the distance to a target object with a stereo camera. It is a figure which shows the difference of the color interpolation process of a primary color filter and a complementary color filter. It is a figure which shows the example of a saturation order. It is a figure which shows the example of a saturation order. 1 is a block diagram of a distance measuring device 10 according to the present embodiment. FIG. 6 is a schematic diagram for explaining the operation of the distance measuring device 10. FIG. 5 is a flowchart for explaining the operation of the distance measuring device 10. It is a schematic diagram for demonstrating operation | movement of the distance measuring device 10 concerning another embodiment. It is a figure which shows the flame | frame produced by replacing only some pixel values.

  Hereinafter, the distance measuring device 10 according to the embodiment of the present invention will be described. FIG. 11 is a block diagram of the distance measuring apparatus 10 that also serves as an imaging unit according to the present embodiment. In FIG. 11, a distance measuring device 10 is a system that is mounted on a robot or the like and measures the distance to an object by a stereo method from images captured by a pair of reference cameras 1 and reference cameras 2 provided on the left and right sides of the body. .

  As shown in FIG. 7, the cameras 1 and 2 as imaging means are mounted on the body so as to be separated by a predetermined baseline interval L and the optical axes are parallel to each other. In the cameras 1 and 2, the light of the object passes through the lenses 1a and 2a, enters the imaging surfaces of the imaging elements 1b and 2b having complementary filters, and the converted image signals are input to the image processing units 1c and 2c. Then, the image is processed and output as image data composed of dR ′, dG ′, and dB ′ as described above. The control units 1d and 2d receive control signals from the outside and control the image sensors 1b and 2b and the image processing units 1c and 2c. According to the present invention, a color image similar to that using the primary color filter can be obtained by performing image processing as shown in FIG.

  Next, image processing necessary for distance measurement will be described. The image processing units 1c and 2c store the saturation order of filters in which the pixels of the image pickup devices 1b and 2b are likely to be saturated in the complementary color filter, which is obtained in advance through experiments and simulations. Whether or not the pixel is saturated when light from an object enters through a filter (for example, a W filter) compared to a threshold value (for example, a value that is greater than or less than the inflection point for a Lin-Log sensor) Judging. When the image processing units 1c and 2c determine that the pixel is not saturated when light from an object enters through a higher-order filter, the image processing units 1c and 2c output an image signal obtained through the filter to the stereo processing unit 3. When it is determined that the pixel is saturated, an image signal obtained through a filter (for example, a Ye filter) that is lower in saturation order than the filter in which the pixel is saturated is output to the stereo processing unit 3. Based on these two image signals, the stereo processing unit 3 performs image processing including stereo matching, and based on the image-processed signals, the arithmetic unit 4 performs processing based on the equation described with reference to FIG. Measure the distance to the object. In the present embodiment, the stereo processing unit 3 and the calculation unit 4 constitute a parallax detection unit, and the calculation unit 4 constitutes a distance measurement unit.

  FIG. 12 is a schematic diagram for explaining the operation of the distance measuring device 10. First, as described with reference to FIG. 8B, the image processing units 1c and 2c correspond to the four pixels shown in FIG. 12B from the original image data shown in FIG. Create frame data. Next, the image processing units 1c and 2c calculate an average pixel value (average pixel value) in each frame data. The pixel value can be expressed in 256 levels. Here, as shown in FIG. 12 (c), the pixel average value is 255 in the frame of W pixel, the pixel average value is 200 in the frame of Ye pixel, the pixel average value is 180 in the frame of Ri pixel, and the frame of Ir pixel. Let us assume that the average pixel value is 150.

  The image processing units 1c and 2c set a threshold value of 240 as to whether or not the pixel is saturated. Therefore, as shown in FIG. 12D, it is determined that saturation has occurred in the W pixel frame whose pixel average value is equal to or greater than the threshold, and the next Ye pixel frame is selected in the saturation order. As shown in e), image signals of two Ye pixel frames obtained from the cameras 1 and 2 are output to the stereo processing unit 3. The stereo processing unit 3 performs stereo matching processing using a frame of two Ye pixels.

  FIG. 13 is a flowchart for explaining the operation of the distance measuring device 10. In step S101, the image processing unit 1c acquires a frame corresponding to each filter for the reference camera, and in step S102, the image processing unit 2c acquires a frame corresponding to each filter for the reference camera. In step S103, the image processing units 1c and 2c determine whether or not the frame of W pixels is saturated as described above, and if it is determined that the frame is not saturated, the W pixels are determined in step S104. The stereo processing unit 3 performs stereo matching processing using two W pixel frames, and outputs the matching result to the calculation unit 4 in step S110.

  On the other hand, if it is determined in step S103 that the frame of W pixels is saturated, the image processing units 1c and 2c further saturate the frame of Ye pixels in step S105 as compared with the threshold value as described above. In step S106, the image signal of the Ye pixel frame is output to the stereo processing unit 3, and the stereo processing unit 3 outputs the two Ye pixel frames. The stereo matching process is performed, and the matching result is output to the calculation unit 4 in step S110.

  On the other hand, if it is determined in step S105 that the frame of Ye pixels is saturated, the image processing units 1c and 2c further saturate the frame of Ri pixels in step S107 as compared with the threshold value as described above. In step S108, the image signal of the Ri pixel frame is output to the stereo processing unit 3, and the stereo processing unit 3 outputs the two Ri pixel frames. The stereo matching process is performed, and the matching result is output to the calculation unit 4 in step S110.

  On the other hand, if it is determined in step S107 that the frame of Ri pixel is saturated, the image processing units 1c and 2c output the image signal of the frame of Ir pixel to the stereo processing unit 3 in step S109, and the stereo processing unit 3 The processing unit 3 performs a stereo matching process using two Ir pixel frames, and outputs a matching result to the calculation unit 4 in step S110.

  As a modification of the present embodiment, when the complementary color filter shown in FIG. 4E is provided in the imaging devices 1b and 2b of the cameras 1 and 2, W is replaced with Ye and Ye is replaced with Mg in the flowchart of FIG. Replacement, Ri may be replaced with Cy, and Ir may be replaced with G. The same applies to other complementary color filters.

  However, with respect to the three-color complementary filters shown in FIGS. 4B, 4C, and 4D, only the determination as to whether or not the upper filter is saturated based on the saturation order is only twice. Different.

  FIG. 14 is a schematic diagram for explaining the operation of the distance measuring apparatus 10 according to another embodiment. Similar to the embodiment shown in FIG. 12, the image processing units 1c and 2c are based on the original image data shown in FIG. 14A as described with reference to FIG. 4 frames data corresponding to the four pixels shown in FIG. Next, the image processing units 1c and 2c calculate an average pixel value in each frame data. Here, as shown in FIG. 14C, the pixel average value is 255 in the frame of W pixel, the pixel average value is 200 in the frame of Ye pixel, the pixel average value is 180 in the frame of Ri pixel, and the frame of Ir pixel. Let us assume that the average pixel value is 150.

  As shown in FIG. 14D, the image processing units 1c and 2c set the threshold value of whether or not the pixel is saturated to 240, and saturation occurs in the W pixel frame whose pixel average value is equal to or larger than the threshold value, as shown in FIG. However, all the frames that are not saturated are selected. Specifically, the image processing units 1c and 2c select a frame of Ye pixels, a frame of Ri pixels, and a frame of Ir pixels, and 2 obtained from the cameras 1 and 2 as shown in FIG. The pair of Ye pixel frames, Ri pixel frames, and Ir pixel frame image signals are output to the stereo processing unit 3. The stereo processing unit 3 performs stereo matching processing by comparing two Ye pixel frames, Ri pixel frames, and Ir pixel frames.

  At this time, the pixel whose Ye pixel value of the reference camera 2 matches the Ye pixel value of the target pixel of the base camera 1 is the 100th, and the reference camera is compared with the Ri pixel value of the target pixel of the base camera 1. It is assumed that the pixel with the matching Ri pixel value of 2 is the 110th pixel, and the pixel with the matching Ir pixel value of the reference camera 2 is the 100th pixel with respect to the Ir pixel value of the target pixel of the reference camera 1. In such a case, the image processing units 1c and 2c determine that the 100th pixel of the reference camera 2 corresponds to the target pixel of the base camera 1 based on the principle of majority decision (FIG. 14 (f)). The parallax can be obtained by substituting into the equation (1) described with reference to FIG.

  Depending on the imaging scene, a high-luminance subject such as the sun may appear on the screen, but pixels that receive light other than sunlight are not saturated and may be used effectively. Therefore, the image processing units 1c and 2c, for example, determine whether or not the pixel value output from the W pixel has exceeded the threshold value, and the W pixel value of a part of the region exceeds the threshold value, but the remaining exceeds. If it is determined that the pixel is not saturated, the W pixel in a part of the saturated region is replaced with the Ye pixel value as shown in FIG. 15, and a frame is created. Can be processed.

  The present invention can be applied to a robot camera, a surveillance camera, or the like, but the application is not limited thereto.

DESCRIPTION OF SYMBOLS 1, 2 Camera 1a, 2a Lens 1b, 2b Image pick-up element 1c, 2c Image processing part 1d, 2d Control part 3 Stereo processing part 4 Calculation part 10 Distance measuring device

Claims (7)

A pair of image sensors provided with a plurality of pixels that are provided apart from each other by a predetermined interval and convert light into a pixel signal by receiving light from an object through a plurality of different filters, and the pair of image sensors The parallax detection means for obtaining the parallax between the image sensors in the separation direction by comparing the image signals output from the distance, and the distance for sequentially obtaining the distance to the object based on the parallax obtained by the parallax detection means In an image processing apparatus having a measuring means,
An image processing apparatus that compares image signals or image data obtained through a filter that is higher than a primary color filter in a saturation order of filters in which pixels of the image sensor are likely to be saturated in the different filters.   When a pixel is saturated when light from an object is incident through the upper filter, the parallax detection unit is configured to detect image signals obtained through the lower filter in the saturation order than the filter in which the pixel is saturated. The image processing apparatus according to claim 1, wherein the image processing apparatuses are compared.   The parallax detection means obtains a pixel average value in one image formed by an image signal obtained through a certain filter, and determines that the pixel is saturated when the pixel average value is equal to or greater than a threshold value. The image processing apparatus according to claim 1.   When pixels in a part of the region are saturated when light from an object enters through a certain filter, the parallax detection means performs saturation order of the pixels in the part of the region with a saturation order from the saturated filter. The image processing apparatus according to claim 1, wherein image signals obtained through a lower-order filter are compared with each other.   The parallax detection means obtains a pixel average value of a partial region formed by an image signal obtained through a certain filter, and when the pixel average value is equal to or greater than a threshold, the pixel of the partial region is saturated The image processing apparatus according to claim 4, wherein the determination is made.   When the image signals output from the pair of image sensors are compared for each filter, a filter in which the positions of the pixels having the same pixel signal value match is selected, and the image signals obtained through the selected filter are The image processing apparatus according to claim 1, wherein the image processing apparatuses are compared. A pair of image sensors provided with a plurality of pixels that are provided apart from each other by a predetermined interval and convert light into a pixel signal by receiving light from an object through a plurality of different filters, and the pair of image sensors The parallax detection means for obtaining the parallax between the image sensors in the separation direction by comparing the image signals output from the distance, and the distance for sequentially obtaining the distance to the object based on the parallax obtained by the parallax detection means In an image processing method of a distance measuring device having a measuring means,
If the pixel does not saturate when light from an object enters through the higher order filter among the saturation ranks of the filters in which the pixels of the image sensor are likely to be saturated in the different filters, the pixels are obtained through the higher order filter. Compare image signals
When the pixel is saturated when light from an object enters through the upper filter, the image signals or image data obtained through the lower filter in the saturation order are compared with the filter in which the pixel is saturated. An image processing method characterized by that.

Images