JP2013187573A - Vehicle periphery monitoring apparatus - Google Patents

Abstract

Provided is a vehicle periphery monitoring device that makes it possible to obtain appropriate images of objects even if objects with a large luminance difference are mixed in a subject to be captured by a camera that captures the periphery of the vehicle.
A vehicle periphery monitoring device, as a camera 2 for photographing the periphery of a vehicle, receives a plurality of colors of light of each of a plurality of predetermined colors as composition colors of a color image. Color light receiving pixels (R light receiving pixels, G light receiving pixels, and B light receiving pixels) and a plurality of sub light receiving pixels (Rx light receiving pixels) that receive light of a predetermined color with light receiving sensitivity lower than that of the color light receiving pixels are arranged. A camera having an image sensor 22 is provided.
[Selection] Figure 2

Description

  The present invention relates to a vehicle periphery monitoring device including a camera capable of capturing a color image around a vehicle.

  A camera capable of capturing a color image is mounted on a vehicle, and driving assistance such as alerting the driver is performed by monitoring the situation around the vehicle based on an image (color image) captured by the camera. Such a vehicle periphery monitoring device is conventionally known.

  For example, Patent Document 1 discloses that a preceding vehicle is detected by detecting the lighting (red lighting) of a brake lamp of a preceding vehicle existing ahead of the host vehicle based on a captured image (color image) in front of the host vehicle. Has been proposed that recognizes the braking state of the vehicle and appropriately notifies the driver of the vehicle according to the braking state of the preceding vehicle.

  In addition, a camera mounted on a vehicle and capable of capturing a color image usually has a color filter for each type of light of the three primary colors R (red), G (green), and B (blue). A plurality of color light receiving pixels (specifically, a set of a light receiving pixel capable of receiving R light, a light receiving pixel capable of receiving G light, and a light receiving pixel capable of receiving B light) arranged in a Bayer array It has an image sensor that is arranged vertically and horizontally in a predetermined arrangement pattern such as. Then, a color image is generated by interpolating the gradation values of the R, G, and B colors in the pixel at the position of each light receiving pixel of the image sensor by so-called demosaicing processing.

Japanese Patent Publication No. 6-10839

  By the way, the brightness (brightness) of an object projected on an image sensor of a camera for monitoring the periphery of a vehicle is generally various, and there are many cases where objects with greatly different brightness are mixed. For example, at night, high-luminance self-luminous objects such as headlights of oncoming vehicles, taillights when braking the preceding vehicle, or lighting lamps of traffic lights are markedly higher than the surrounding road structures. It tends to be bright.

  On the other hand, when monitoring the surroundings of a vehicle using a captured image of a camera for the purpose of avoiding contact between the vehicle and an external object or assisting driving of the vehicle, generally high brightness such as a headlight of an oncoming vehicle In addition to the self-luminous object image, it is also necessary to acquire an image of a relatively low-luminance object such as a road surface or a pedestrian.

  For this reason, the degree of exposure of the camera of the conventional vehicle periphery monitoring device is generally set so that an image of a relatively low-luminance object such as a road surface or a pedestrian can be obtained.

  However, in this case, in the captured image such as a color image obtained from the image data of the camera (the output signal of each light receiving pixel of the image sensor), the gradation values of the location where the self-luminous object is projected and the surrounding locations. Tends to be saturated (so-called whiteout state). As a result, in the captured image, the self-luminous object and its surrounding objects cannot be distinguished (no difference in brightness or color on the image), or the light emission color of the self-luminous object is unknown. easy.

  Furthermore, when two self-luminous light sources are close to each other, such as left and right headlights and left and right taillights of a vehicle, a spot in the whiteout state near the image of those self-luminous objects In some cases, it will lead to a lump.

  Therefore, in the conventional vehicle periphery monitoring device, the color of a high-luminance self-luminous object such as a headlight or taillight of a vehicle, a lighting lamp of a traffic light, or the location thereof is determined from the captured image of the camera by other objects in the vicinity. In many cases, it is difficult to recognize them separately. As a result, there is an inconvenience that it is difficult to correctly recognize other vehicles existing around the vehicle and objects such as traffic lights from the captured image of the camera.

  The present invention has been made in view of such a background, and even when objects having a large luminance difference are mixed in a photographing object of a camera that captures the periphery of a vehicle, it is possible to obtain an appropriate image of those objects. An object of the present invention is to provide a vehicle periphery monitoring device.

  In order to achieve such an object, the vehicle periphery monitoring device of the present invention is a camera mounted on the vehicle for photographing the periphery of the vehicle. A camera having an imaging device in which a plurality of color light receiving pixels that receive light of each type of color and a plurality of sub light receiving pixels that receive light of a predetermined color with a light receiving sensitivity lower than that of the color light receiving pixels (First invention).

  In the first aspect of the present invention, the “plurality of color light-receiving pixels” are defined as the “plurality of color light-receiving pixels” when the predetermined plural kinds of colors are the first to Nth N colors. Means a light receiving pixel that receives light of any one of the first to Nth colors and includes light receiving pixels corresponding to all the first to Nth colors. . Moreover, the light of a certain color means light of a specific wavelength band (or light having a specific wavelength band as a main component).

  According to the first aspect of the invention, the image pickup device of the camera includes a plurality of sub light receiving pixels that receive light of a predetermined color with a light receiving sensitivity lower than that of the color light receiving pixels, in addition to the plurality of color light receiving pixels. It is. In this case, each sub-light-receiving pixel has lower light-receiving sensitivity than the color light-receiving pixel, so that the intensity of the output signal of the color light-receiving pixel (the signal that the color light-receiving pixel outputs according to the light-receiving level) Even if a relatively high-brightness object that causes saturation is included, the sub-light-receiving pixel that receives the light of the high-brightness object outputs its output signal (the sub-light-receiving pixel outputs according to its light reception level). Signal saturation) is less likely to occur.

  For this reason, even if the object to be photographed includes a relatively high-luminance object that causes saturation of the intensity of the output signal of the color light receiving pixel, Based on at least the output signal of the sub-light-receiving pixel in the output signal data of each light-receiving pixel constituting the element, it is possible to generate an image indicating the brightness state of the high-brightness object at an appropriate resolution. .

  In addition, when the camera's shooting target does not include the high-brightness object as described above, the color and brightness of the shooting target are appropriately set based on at least the output signal of the color light receiving pixel in the camera's imaging data. It is possible to generate a color image having a high resolution.

  Therefore, according to the first aspect of the present invention, even when objects with a large luminance difference are mixed in the photographing target of the camera that captures the periphery of the vehicle, it is possible to obtain appropriate images of those objects.

  In the first invention, the predetermined plurality of colors received by the color light receiving pixels are, for example, three primary colors of R (red), G (green), and B (blue), cyan, magenta, and yellow. Complementary color three primary colors can be adopted.

  Various colors such as gray can be adopted as the color of light received by the sub light-receiving pixels (the predetermined color). In this case, the predetermined color may be the same as any one of the predetermined plurality of colors related to the color light receiving pixel.

  On the other hand, in the vehicle periphery monitoring device, it is possible to appropriately detect red light emitting objects such as the taillight of the preceding vehicle of the own vehicle and the red lamp of the traffic light, thereby preventing contact between the own vehicle and an external object. In many cases, it is important to provide appropriate driving support for the vehicle. Therefore, in the first invention, as one form thereof, it is preferable that the predetermined color of the sub light receiving pixel is red (second invention).

  According to the second aspect of the present invention, based on at least the output signal of the sub-light-receiving pixel in the imaging data of the camera, the color and luminance state of the red high-luminance object such as the taillight of the preceding vehicle and the red lamp of the traffic light Can be generated with an appropriate resolution.

  Alternatively, as another form of the first invention, the predetermined color of the sub light-receiving pixel may be a gray color (third invention).

  According to the third aspect of the present invention, it is possible to generate images of various high-intensity objects such as headlights of oncoming vehicles and road signs with appropriate resolution, not limited to the taillight of the preceding vehicle and the red lamp of the traffic light. It becomes possible.

  In the first to third aspects of the invention, the sub light receiving pixels may be arranged so as to be distributed over the entire area of the imaging element. However, if it is known that a high-luminance light-emitting object important for the host vehicle exists in a specific part of the imaging region of the camera, or if it is known that the possibility is high, The light receiving pixels may be arranged only in a specific partial area of the image sensor.

  For example, when the camera is mounted on the vehicle so as to image the front or rear of the vehicle (own vehicle), the headlights and taillights of other vehicles in front of or behind the vehicle are In many cases, it exists in an area near the infinity point on the road surface in front of or behind the vehicle in the entire imaging area. In many cases, the lighting lamp of the traffic light is present in the upper area of the entire imaging area of the camera.

  Therefore, when the camera is mounted on the vehicle so as to image the front or rear of the vehicle (own vehicle), the sub light-receiving pixel is located within the partial area of the image sensor. It is arranged together with the color light receiving pixels, and the partial area includes an area corresponding to an upper area of the entire imaging area of the camera and a front of the vehicle in the entire imaging area of the camera. It is also possible to include at least one of the regions corresponding to the partial region including the infinity point of the road surface (fourth invention).

  According to the fourth aspect of the present invention, in the partial area, an appropriate image of a high-luminance object can be generated using the output signal of the sub light-receiving pixel, while an area other than the partial area is generated. In the (region not including the sub light receiving pixels), a color image with high reliability can be generated using the output signals of the color light receiving pixels.

  In the first to fourth inventions, it is possible to generate various types of images. However, in order to appropriately monitor the surroundings of the vehicle, it is preferable to employ, for example, the following modes.

  That is, the image is an original image captured by the camera, and is composed of a plurality of pixels each assigned a gradation value indicating the intensity of an output signal output according to the light reception level of each light receiving pixel of the image sensor. Original image acquisition means for acquiring the original image, and the original image in the processing target area that is at least a part of the original image including the plurality of color light receiving pixels and the plurality of sub light receiving pixels. Based on the gradation value of each pixel, by determining the gradation value of each of the plurality of colors and the gradation value of the predetermined color in each pixel of the processing target region, the plurality of colors and A color-specific image generating unit that generates a plurality of color-specific images corresponding to each of the predetermined colors and a color-specific image corresponding to each of the plurality of types of colors among the plurality of color-specific images. First color image And at least one of the predetermined color image and a second color image obtained by synthesizing all the color images of the plurality of color images. It is preferable to include monitoring image acquisition means for acquiring the first image (fifth invention).

  According to the fifth aspect of the invention, based on the gradation value of each pixel of the original image in the processing target area, the color-specific image generation unit corresponds to each of the plurality of types of colors and the predetermined color. A plurality of color-specific images are generated.

  Each color image is an image in which the gradation value of each pixel is a gradation value of a color corresponding to the color image, that is, an image showing the luminance distribution of the color. Note that each color-specific image is obtained by, for example, determining the gradation of each pixel of the color-specific image based on the gradation value of the pixel of the color corresponding to the color-specific image among the pixels of the original image in the processing target region. A value (a gradation value of the corresponding color) can be generated by determining by a demosaicing process.

  The monitoring image acquisition means includes a first color image obtained by combining color images corresponding to the plurality of colors of the plurality of color images, and a color-specific image of the predetermined color, At least one of the second color image obtained by synthesizing all the color-specific images among the plurality of color-specific images is acquired as an image for monitoring around the vehicle.

  In this case, the first color image appropriately reflects the color information of the imaging target in the processing target area, particularly when the intensity of the output signal of the light receiving pixel in the processing target area is not saturated. Become. Therefore, the first color image can be used as a color image with good color reproducibility.

  Further, the color-specific image of the predetermined color appropriately reflects the luminance distribution of the shooting target, particularly when the shooting target in the processing target area includes a high-luminance object. . Therefore, the color-specific image of the predetermined color can be used as an image for easily detecting a high-luminance object.

  Further, the second color image is a color image reflecting the luminance distribution of the imaging target in the processing target area indicated by the color-specific image of the predetermined color. For this reason, in particular, even when a subject to be imaged in the processing target region includes a high-luminance object, the second image is a color image in which the dynamic range is pseudo-expanded while suppressing the overexposure state. Color images can be used.

  Therefore, according to the fifth aspect of the invention, various brightness and color objects are detected using any one of the first color image, the color-specific image of the predetermined color, and the second color image. be able to.

  In the fifth aspect of the invention, the monitoring image acquisition unit monitors the periphery of the vehicle among the first color image, the luminance distribution image, and the second color image according to the brightness level or contrast of the first color image. It is preferable to switch the image acquired as the image for use (sixth invention).

  According to the sixth aspect of the invention, an appropriate image (desired image) of the first color image, the luminance distribution image, and the second color image is selected according to the luminance level of the photographing target in the processing target region and the distribution state thereof. An image in which an object can be easily detected) can be used as an image for monitoring around the vehicle.

The figure which shows the structure of the vehicle periphery monitoring apparatus in embodiment of this invention. FIG. 2A is a diagram illustrating an arrangement example of the color filter of the camera in the embodiment, and FIG. 2B is a diagram for explaining a captured image (original image) obtained from the image sensor of the camera of the embodiment. The graph which shows the characteristic of the light reception sensitivity of the light reception pixel of the camera of embodiment. 3 is a flowchart showing processing of the image controller 3 shown in FIG. The figure for demonstrating the process of STEP2 of FIG. The flowchart which shows the detail of the process of STEP3 of FIG. The figure for demonstrating the G image produced | generated by STEP3-2 or 3-3 of FIG. The figure for demonstrating the Rx image produced | generated by STEP3-2 or 3-3 of FIG. The figure for demonstrating the R image produced | generated by STEP3-3 of FIG. The figure for demonstrating the B image produced | generated by STEP3-3 of FIG. The figure for demonstrating the 1st color image produced | generated by the process of STEP4 of FIG. The figure for demonstrating the 2nd color image produced | generated by the process of STEP4 of FIG. Explanatory drawing regarding the area | region of the image used by the process of STEP4 of FIG. FIGS. 14A and 14B are diagrams showing examples of other arrangement patterns of light receiving pixels of the camera. FIGS. 15A, 15 </ b> B, and 15 </ b> C are diagrams each illustrating an example of an array pattern of light receiving pixels of a camera including gray light receiving pixels. The graph which shows the light reception characteristic of each light reception pixel relevant to Fig.15 (a), (b), (c).

  An embodiment of a vehicle periphery monitoring device of the present invention will be described with reference to FIGS. With reference to FIG. 1, the vehicle periphery monitoring device of the present embodiment includes a camera 2 mounted on a vehicle 1 and an image controller 3 connected to the camera 2.

  The camera 2 is a camera for photographing the periphery of the vehicle 1. In the present embodiment, the camera 2 is mounted on the front portion of the vehicle 1 so as to photograph the front of the vehicle 1.

  The camera 2 is a camera that can shoot a color video (video including color information of a shooting target) by an imaging device 22 (CCD, CMOS, etc.) in which a filter 21 is incorporated. Output to the control circuit 30 of the controller 3. The imaging element 22 is configured by arranging a plurality of m × n light receiving pixels in an array pattern of m rows and n columns in a two-dimensional manner.

  Referring to FIG. 2A, the filter 21 of the camera 2 is a color filter of three primary colors R (red), G (green), and B (blue), that is, light in the R, G, and B wavelength ranges. It is composed of three types of color filters (hereinafter referred to as basic color filters) that can each transmit (visible light) and a color filter of a predetermined color (hereinafter referred to as sub color filters).

  In the present embodiment, the predetermined color, which is the color of light in the wavelength range that can be transmitted through the sub color filter, is R (red) among the three primary colors R, G, and B. However, the light transmittance of the sub color filter is lower than the light transmittance of the basic color filter.

  A color filter of one of the three primary color basic color filters is attached to each light receiving path side of a predetermined number of light receiving pixels among the m × n light receiving pixels of the image sensor 22. Yes. Accordingly, each light receiving pixel on which the basic color filter of any one of R, G, and B is mounted receives light through the corresponding color filter.

  Note that the basic color filters mounted on these light receiving pixels (the predetermined number of light receiving pixels) include all color filters of R, G, and B. Accordingly, the entire light receiving pixel to which the basic color filter is attached is composed of a light receiving pixel (hereinafter referred to as an R light receiving pixel) to which an R color filter (hereinafter referred to as an R filter) is attached, and a G color filter. A light receiving pixel (hereinafter referred to as a G light receiving pixel) to which a light receiving pixel (hereinafter referred to as a G light receiving pixel) is attached and a light receiving pixel (hereinafter referred to as a B light receiving pixel) to which a B color filter (hereinafter referred to as a B filter) is attached. It consists of. These R light receiving pixel, G light receiving pixel, and B light receiving pixel correspond to the color light receiving pixels in the present invention.

  The sub color filter is mounted on each light receiving path side of the remaining light receiving pixels (the light receiving pixels excluding the R light receiving pixel, the G light receiving pixel, and the B light receiving pixel) among the m × n light receiving pixels. ing. Thereby, each light receiving pixel to which the sub color filter is attached receives light through the sub color filter. Hereinafter, the sub color filter is referred to as an Rx filter, and a light receiving pixel on which the Rx filter is mounted is referred to as an Rx light receiving pixel. This Rx light receiving pixel corresponds to the sub light receiving pixel in the present invention.

  In the image sensor 22 having the arrangement pattern shown in FIG. 2A, light receiving pixels, for example, G light receiving pixels, to which one of the basic color filters is mounted are arranged in a checkered arrangement pattern. . The Rx light receiving pixels having the red Rx filter are arranged so that every other pixel is arranged in the vertical direction and the horizontal direction. Of the basic color filters, the R light receiving pixels to which the R filter is attached and the B light receiving pixels to which the B filter is attached are alternately arranged in the vertical and horizontal directions. . Therefore, in the image sensor 22 having the arrangement pattern shown in FIG. 2A, the number ratio of the R light receiving pixels, the G light receiving pixels, the B light receiving pixels, and the Rx light receiving pixels is 1: 4: 1: 2.

  The camera 2 includes R light receiving pixels (indicated by R11, R15,...), G light receiving pixels (indicated by G12, G14,...), And B light receiving pixels (B13, B31,. And output signals of Rx light receiving pixels (indicated by Rx22, Rx24,... In the figure) are output to the image controller 3 as imaging data. The output signal of each light receiving pixel is a signal having an intensity (magnitude) corresponding to the light receiving level per predetermined time at the light receiving pixel.

  The light receiving level of each light receiving pixel corresponds to the intensity of light received by the light receiving pixel (light after passing through the color filter corresponding to the light receiving pixel) out of the light incident on the image sensor 22 of the camera 2. To do.

  Here, the light receiving characteristics of each light receiving pixel will be described with reference to FIG. Relationship between the light receiving sensitivity of each of the R light receiving pixel, the G light receiving pixel, and the B light receiving pixel received through the basic color filter (ratio of the intensity of the output signal of the light receiving pixel to the intensity of the incident light) and the wavelength of the incident light ( Spectral characteristics) are characteristics shown by solid line graphs a r, a g, and a b in FIG. That is, the light receiving sensitivity of each of the R light receiving pixel, the G light receiving pixel, and the B light receiving pixel is high with respect to light in the wavelength region of the corresponding color.

  On the other hand, the relationship between the light-receiving sensitivity of the Rx light-receiving pixel that receives light through the red sub-color filter and the wavelength of the incident light has a characteristic as indicated by a solid line graph “arx” in FIG.

  In this case, although the light receiving sensitivity of the Rx light receiving pixel is highly sensitive to light in the same wavelength range as the R light receiving pixel (red light), the transmittance of the Rx filter (sub color filter) is Thus, it is lower than the transmittance of the basic color filter. For this reason, as shown in the figure, the light receiving sensitivity of the Rx light receiving pixels is lower than that of the R light receiving pixels, the G light receiving pixels, and the B light receiving pixels.

  Note that the intensity of the output signal of each light receiving pixel varies between a predetermined minimum value and a maximum value in accordance with the light reception level (the intensity of light received). Therefore, when the light receiving level of each light receiving pixel becomes a large level exceeding a certain level, the intensity of the output signal of the light receiving element is saturated and the intensity is held at or near the maximum value. Further, when the light receiving level of each light receiving pixel becomes a small level lower than a certain level, the intensity of the output signal of the light receiving pixel is held at a minimum value or a level close thereto.

  Supplementally, other types of color filters other than R, G, and B may be used as the basic color filter, such as filters of the three primary colors complementary colors Cy (cyan), Mg (magenta), and Ye (yellow). . In that case, the transmittance of the Rx filter is set so that the light receiving sensitivity of the Rx light receiving pixel to which the Rx filter is mounted is lower than the light receiving sensitivity of the light receiving pixel to which the complementary primary color filter is mounted. You just have to.

  The image controller 3 includes a control circuit 30 including a CPU, a memory, an input / output circuit, and the like (not shown), an image memory 40, and a CAN (Controller Area Network) driver 50.

  The control circuit 30 functions as an original image acquisition unit 31, a color-specific image generation unit 32, a monitoring image acquisition unit 33, and an object detection unit 34 by executing an image processing program stored in the memory by the CPU. . Note that some or all of the original image acquisition unit 31, the color-specific image generation unit 32, the monitoring image acquisition unit 33, and the object detection unit 34 may be configured by hardware.

  The original image acquisition unit 31 corresponds to the original image acquisition means in the present invention. The original image acquisition unit 31 outputs a control signal to the camera 2 to image the front of the vehicle 1, acquires an original image 41 based on imaging data (output signal of each light receiving pixel) output from the camera 2. Stored in the image memory 40.

  As shown in FIG. 2B, the original image 41 is output from each light receiving pixel (R light receiving pixel, G light receiving pixel, B light receiving pixel, Rx light receiving pixel) of the image sensor 22 shown in FIG. The gradation value indicating the signal intensity is individually assigned as the gradation value of the pixel at the arrangement position corresponding to the light-receiving pixel (pixels having the same arrangement position). In FIG. 2B, the gradation value of each pixel is set to S (upper case), one of lower case letters r, g, b, rx, and subscripts i, j (i = 1, 2,..., M, j = 1, 2,..., N).

  Here, Srij represents a gradation value of a pixel at an arrangement position corresponding to the R light receiving pixel in FIG. 2A (hereinafter referred to as R pixel), and Sgij represents an arrangement corresponding to the G light receiving pixel in FIG. 2 indicates the gradation value of the pixel at the position (hereinafter referred to as G pixel), Sbij indicates the gradation value of the pixel at the arrangement position (hereinafter referred to as B pixel) corresponding to the B light receiving pixel in FIG. 2A, and Srxij Indicates a gradation value of a pixel (hereinafter referred to as an Rx pixel) at an arrangement position corresponding to the Rx light receiving pixel in FIG.

  The color-specific image generation unit 32 corresponds to the color-specific image generation means in the present invention. The color-specific image generation unit 32 generates four types of color-specific images 42 corresponding to each of the R light-receiving pixels, the G light-receiving pixels, the B light-receiving pixels, and the Rx light-receiving pixels, and stores the color-specific images 42 in the image memory. 40.

  More specifically, the four types of color-specific images 42 correspond to an R image (hereinafter referred to as reference numeral 42r) that is a color-specific image 42 of a color (red) corresponding to the R light receiving pixel, and a G light receiving pixel. A G image (hereinafter referred to as reference numeral 42g) that is a color-specific image 42 of the color (green) to be performed, and a B image (hereinafter referred to as reference numeral 42) that is a color (blue) image 42 corresponding to the B light receiving pixel. 42b) and an Rx image (hereinafter referred to as reference numeral 42rx) which is a color-specific image 42 of the color (red) corresponding to the Rx light receiving pixel.

  The R image 42r has a red tone monotone image (a red luminance distribution is determined by a demosaicing process) in which the tone value of each pixel is determined based on at least the tone value of the R pixel in the original image 41. Image).

  The G image 42g has a green tone monotone image (green luminance) in which the tone value of each pixel is determined by demosaicing processing based on at least the tone value of the G pixel in the original image 41. Image showing the distribution).

  Further, the B image 42b has a blue tone monotone image (blue luminance) in which the tone value of each pixel is determined by demosaicing processing based on at least the tone value of the B pixel in the original image 41. Image showing the distribution).

  In addition, the Rx image 42rx has a red tone monotone image (red luminance) in which the tone value of each pixel is determined by demosaicing processing based on at least the tone value of the Rx pixel in the original image 41. Image showing the distribution).

  The Rx image 42rx is a monotone image with a red gradation similar to the R image 42r, but the R image 42r and the Rx image 42rx have different gradation values used for image generation. Therefore, the R image 42r and the Rx image 42rx are generated as different color-specific images 42.

  The monitoring image acquisition unit 33 corresponds to the monitoring image acquisition means in the present invention. The monitoring image acquisition unit 33 selects, from the color-specific images 42, a monitoring image used by the object detection unit 34 to detect a monitoring object (an object to be monitored) existing in front of the vehicle 1. Alternatively, it is acquired by generating, and the acquired monitoring image is output to the object detection unit 34.

  In this case, the monitoring image acquisition unit 33 has a function of generating any one of the first color image 43 and the second color image 44 as the monitoring image as needed. The first color image 43 is a color image obtained by synthesizing the R image 42r, the G image 42g, and the B image 42b among the color-specific images 42. The second color image 44 includes all types of the color-specific images 42. This is a color image obtained by combining the images (42r, 42g, 42b, 42rx).

  The object detection unit 34 detects the monitoring object using the monitoring image provided from the monitoring image acquisition unit 33 and transmits various control signals to the vehicle controller 6 according to the detection result. . In the present embodiment, the object detection unit 34 uses a lane (lane mark) on the road surface in front of the vehicle 1, another vehicle, a traffic light (or road sign), and a pedestrian as monitoring objects, and monitors each type of those objects. Objects are detected by a predetermined detection algorithm corresponding to each object.

  The vehicle controller 6 is an electronic circuit unit that includes a CPU, a memory, an input / output circuit, and the like (not shown). The vehicle controller 6 executes the control program for the vehicle 1 held in the memory by the CPU, thereby controlling the operation of the steering device 71, and the braking control unit for controlling the operation of the braking device 72. 62 and a display display control unit 63 that controls display on the display 73.

  The image controller 3 and the vehicle controller 6 communicate with each other via the CAN drivers 50 and 64.

  Next, the details of the processing of the image controller 3 will be described focusing on the processing of the color-specific image generation unit 32 and the monitoring image acquisition unit 33.

  The image controller 3 executes processing by the original image acquisition unit 31, the color-specific image generation unit 32, the monitoring image acquisition unit 33, and the object detection unit 34 as shown in the flowchart of FIG.

  STEP 1 is processing performed by the original image acquisition unit 31. As described above with reference to FIG. 2B, the original image acquisition unit 31 uses the image data (output signals of the light receiving pixels of the image sensor 22) output from the camera 2 to generate the original image 41 (each light reception). An image in which a gradation value indicating the intensity of the output signal of the pixel is assigned to the pixel at the arrangement position corresponding to the light receiving pixel is acquired and held in the image memory 40.

  Next, the processing of STEP 2 and 3 is executed by the color-specific image generation unit 32. In STEP 2, the color-specific image generation unit 32 sets one or more processing target areas in the original image 41. This processing target area is an area that is a target of the generation process of the color-specific image 42 by the color-specific image generation unit 32.

  FIG. 5 shows an example of setting the processing target area. In this example, the entire original image 41 is divided into a plurality of regions (5 × 6 regions in the illustrated example), and each region is set as a processing target region.

  However, the number of processing target areas to be set, their sizes, and arrangement positions depend on the shooting environment, the type of monitoring target to be detected from the image of the processing target area, the purpose of use of the image, etc. You may make it differ. For example, when trying to detect another vehicle as an object to be monitored using a captured image of the camera 2, an area including an infinite point on the road in the original image 41 (the other image may be shown in the original image 41. A region having high characteristics) may be set as a processing target region.

  Alternatively, when trying to detect a traffic signal or a road sign standing on the road side as an object to be monitored, an upper area (an area where there is a high possibility that a traffic signal or a road sign is present) in the original image 41 is processed. It may be set as a target area.

  Further, the entire original image 41 may be set as one processing target area. Further, when a plurality of processing target areas are set, two or more of the processing target areas may have portions that overlap each other.

  In STEP 3 subsequent to STEP 2, the color-specific image generation unit 32 generates four types of color-specific images 42 (R image, G image, B image, Rx image) for each processing target region.

  Specifically, this process is executed as shown in the flowchart of FIG. First, the color-specific image generation unit 32 selects one of the plurality of processing target areas in STEP 3-1.

  Next, the color-specific image generation unit 32 determines, in STEP 3-2, one of the R color image 42r, the G image 42g, and the B image 42b as one specific color image and the Rx image 42rx as a reference luminance distribution image. Generate as

  Here, in the present embodiment, the image of the specific color is a color corresponding to the largest number of G light receiving pixels among the R light receiving pixels, the B light receiving pixels, the G light receiving pixels, and the Rx light receiving pixels constituting the image sensor 22. (Green) image. Therefore, in this embodiment, the image of the specific color is the G image 42g among the R image 42r, the G image 42g, and the B image 42b.

  In STEP 3-2, one of the G image 42g and the Rx image 42rx is generated as a reference luminance distribution image. This reference luminance distribution image means an image having a luminance distribution state that has a relatively high degree of matching with the actual luminance distribution of the subject to be photographed in the selected processing target area.

  In this case, among the G image 42g and the Rx image 42rx, the image generated as the reference luminance distribution image is included in the gradation value of the G pixel included in the processing target area and the processing target area of the original image 41. Is selected based on the gradation value of the Rx pixel.

  For example, out of the total number of all the pixels included in the processing target area (or the total number of all G pixels), the ratio of the number of G pixels having a gradation value within a predetermined appropriate range, and all included in the processing target area Of the total number of pixels (or the total number of all Rx pixels) is compared with the ratio of the number of Rx pixels having gradation values within a predetermined appropriate range.

  Then, an image of a color corresponding to a pixel (G pixel or Rx pixel) having a larger ratio among the G image 42g and the Rx image 42rx is selected as an image to be generated as a reference luminance distribution image.

  Here, the appropriate range relating to the gradation value of the G pixel is a predetermined upper limit value slightly smaller than the maximum value of the G pixel gradation value and a predetermined lower limit value slightly larger than the minimum value of the G pixel gradation value. A range between values. This proper range includes a so-called whiteout state (a state where the light receiving level of the light receiving pixel is too high and the intensity of the output signal of the light receiving pixel is saturated) This range is set in advance so as not to be in a collapsed state (a state in which the light receiving level of the light receiving pixel is too small and the intensity of the output signal of the light receiving pixel becomes minute). Therefore, when the gradation value of the G pixel is within the above appropriate range, the gradation value of the G pixel (or the intensity of the output signal of the G light receiving pixel corresponding to the G pixel) It can be considered that the value appropriately reflects the brightness of the.

  The appropriate range related to the gradation value of the Rx pixel is also a preset range similar to the appropriate range related to the gradation value of the G pixel.

  Therefore, when the luminance of the subject to be photographed in the selected processing target area is relatively low overall (that is, when the Rx pixel tends to be blacked out), basically, the G image 42g is selected as an image to be generated as a reference luminance distribution image. In addition, when the luminance of the subject to be captured in the processing target area being selected is relatively high overall (that is, when the G pixel tends to be overexposed), basically, the Rx image 42rx is selected as an image to be generated as a reference luminance distribution image.

  Note that an image (G image or Rx image) to be generated as a reference luminance distribution image is selected based on the average value or the histogram of the gradation value of the G pixel and the gradation value of the Rx pixel in the processing target region. May be.

  The G image or Rx image as the reference luminance distribution image is generated as follows. First, a case where a G image as a reference luminance distribution image is generated will be described.

  As shown in FIG. 7, the G image 42g is an image to which a G value (G (green) luminance value) is assigned as a gradation value of each pixel. In FIG. 7, the gradation value (G value) of each pixel is represented by G′i, j (i = 1, 2,..., M, j = 1, 2,..., N).

  The G value G′i, j of each pixel of the G image 42g as the reference luminance distribution image is determined as follows.

  Among the pixels of the G image 42g, the pixel at the arrangement position (the pixel having the same arrangement position) corresponding to the G pixel of the original image 41 (the pixel whose gradation value is Sgi, j) is expressed by the following equation (1). As described above, the gradation value Sgi, j of the G pixel of the corresponding original image 41 is directly determined as the G value (G′i, j) of the pixel of the G image 42g. For example, the gradation value (Sg2,3) of the pixel at the position of (i, j) = (2,3) in the original image 41 shown in FIG. 2B is (i, j) = It is determined as the G value (G′2, 3) of the pixel at the arrangement position of (2, 3).

G'i, j = Sgi, j (1)

In addition, among the pixels of the G image 42g, regarding the pixels at the arrangement positions corresponding to the pixels (R pixels, B pixels, or Rx pixels) other than the G pixels of the original image 41, the G pixels at the arrangement positions around the pixels are changed. The G value (G′i, j) is complementarily determined using at least the gradation value.

  Specifically, among the pixels of the G image 42g, for the pixels at the arrangement positions corresponding to the R pixel and the B pixel of the original image 41, the gradation values Sgi of the four G pixels adjacent to the upper, lower, left, and right sides of the pixel. On the basis of + 1, j, Sgi-1, j, Sgi, j + 1, Sgi, j-1, depending on the magnitude relationship between Ig and Jg in the following formulas (2) and (3), formula (4) Alternatively, the G value (G′i, j) is determined by (5) or (6).

Ig = | Sgi + 1, j−Sgi-1, j | (2)
Jg = | Sgi, j + 1−Sgi, j-1 | (3)
When Ig <Jg G'i, j = (Sgi + 1, j + Sgi-1, j) / 2 (4)
When Ig> Jg G'i, j = (Sgi, j + 1 + Sgi, j-1) / 2 (5)
When Ig = Jg G'i, j = (Sgi, j + 1 + Sgi, j-1 + Sgi + 1, j + Sgi-1, j) / 4 (6)

Further, among the pixels of the G image 42g, for the pixels at the arrangement positions corresponding to the Rx pixels of the original image 41, the gradation values Sgi + 1, j, Sgi of the four G pixels adjacent to the upper, lower, left, and right sides of the pixel. −1, j, Sgi, j + 1, Sgi, j−1, the corresponding Rx pixel gradation value Srxi, j, and four Rx pixels that exist at intervals of 1 pixel above and below and on the left and right Based on the gradation values Srxi + 2, j, Srxi-2, j, Srxi, j + 2 and Srxi, j-2, depending on the magnitude relationship between Ig and Jg in the following equations (7) and (8) The G value (G′i, j) is determined by the equation (9) or (10) or (11).

Ig = | Sgi + 1, j-Sgi-1, j | + | 2Srxi, j-Srxi + 2, j-Srxi-2, j | (7)
Jg = | Sgi, j + 1−Sgi, j−1 | + | 2Srxi, j−Srxi, j + 2−Srxi, j−2 | (8)
When Ig <Jg G'i, j = (Sgi + 1, j + Sgi-1, j) / 2
+ (2Srxi, j-Srxi + 2, j-Srxi-2, j) / 4 (9)
When Ig> Jg G'i, j = (Sgi, j + 1 + Sgi, j-1) / 2
+ (2Srxi, j-Srxi, j + 2-Srxi, j-2) / 4 (10)
When Ig = Jg G'i, j = (Sgi, j + 1 + Sgi, j-1 + Sgi + 1, j + Sgi-1, j) / 4
+ (4Srxi, j-Srxi + 2, j-Srxi-2, j-Srxi, j + 2-Srxi, j-2) / 8
...... (11)

Through the above process (demosaicing process), using the G pixel gradation value (G value) of the original image 41, a green gradation value (luminance value) is assigned to each pixel of the G image 42g in the processing target region. The indicated G value is assigned. As a result, a G image 42g (green tone luminance distribution image) as a reference luminance distribution image is generated.

  Next, a case where the Rx image 42rx as the reference luminance distribution image is generated will be described.

  As shown in FIG. 8, the Rx image 42rx is an image in which an Rx value (red luminance value) is assigned as a gradation value of each pixel. In FIG. 8, the gradation value (Rx value) of each pixel is represented by Rx′i, j (i = 1, 2,..., M, j = 1, 2,..., N).

  The Rx value (Rx′i, j) of each pixel of the Rx image 42rx as the reference luminance distribution image is determined as follows.

  Among the pixels of the Rx image 42rx, the pixel at the arrangement position (the pixel having the same arrangement position) corresponding to the Rx pixel of the original image 41 (the pixel whose gradation value is Srxi, j) is expressed by the following equation (12). As described above, the gradation value Srxi, j of the Rx pixel of the corresponding original image 41 is determined as it is as the Rx value (Rx′i, j) of the pixel of the Rx image 42rx.

Rx'i, j = Srxi, j (12)

Further, among the pixels of the Rx image 42rx, for the pixels at the arrangement positions corresponding to the pixels (R pixel, G pixel, or B pixel) other than the Rx pixels of the original image 41, the gradation values of the Rx pixels around the pixel Rx value (Rx′i, j) is determined complementarily using at least.

  Specifically, among the pixels of the Rx image 42rx, the pixel at the arrangement position corresponding to the R pixel of the original image 41 is adjacent to the upper right, lower right, upper left, and lower left of the pixel. Four Rx pixel gradation values Srxi-1, j + 1, Srxi + 1, j + 1, Srxi-1, j-1, Srxi + 1, j-1 and corresponding R pixel gradation values Sri , j and the gradation values Sri-2, j + 2, Sri + 2, of the four R pixels existing at an interval of one pixel at the upper right, lower right, upper left, and lower left. Based on j + 2, Sri-2, j-2, Sri + 2, j-2, depending on the magnitude relationship between Irx and Jrx in the following formulas (13) and (14), formula (15) or ( The Rx value (Rx′i, j) is determined by 16) or (17).

Irx = | Srxi + 1, j + 1-Srxi-1, j-1 |
+ | 2Sri, j-Sri + 2, j + 2-Sri-2, j-2 | (13)
Jrx = | Srxi + 1, j-1-Srxi-1, j + 1 |
+ | 2Sri, j-Sri + 2, j-2-Sri-2, j + 2 | (14)
When Irx <Jrx, Rx'i, j = (Srxi + 1, j + 1 + Srxi-1, j-1) / 2
+ (2Sri, j-Sri + 2, j + 2-Sri-2, j-2) / 4 (15)
When Irx> Jrx Rx'i, j = (Srxi + 1, j-1 + Srxi-1, j + 1) / 2
+ (2Sri, j-Sri + 2, j-2-Sri-2, j + 2) / 4 (16)
When Irx = Jrx Rx'i, j = (Srxi + 1, j-1 + Srxi-1, j + 1 + Srxi + 1, j + 1 + Srxi-1, j-1) / 4
+ (4Sri, j-Sri + 2, j + 2-Sri-2, j-2-Sri + 2, j-2-Sri-2, j + 2) / 8
...... (17)

Further, among the pixels of the Rx image 42rx, the pixels at the arrangement position corresponding to the B pixel of the original image 41 are four pixels adjacent to the pixel in the oblique direction, similarly to the pixels at the arrangement position corresponding to the R pixel. The gradation values Srxi-1, j + 1, Srxi + 1, j + 1, Srxi-1, j-1, Srxi + 1, j-1 of the pixel and the gradation values Sbi, j of the corresponding B pixel The gradation values Sbi-2, j + 2, Sbi + 2, j + 2, Sbi-2, j-2, Sbi + 2, of the four B pixels existing at an interval of one pixel in the diagonal direction. Based on j-2, the Rx value (Rx′i, j) is obtained by the equation (20) or (21) or (22) according to the magnitude relationship between Irx and Jrx in the following equations (18) and (19). Is determined.

Irx = | Srxi + 1, j + 1-Srxi-1, j-1 |
+ | 2Sbi, j-Sbi + 2, j + 2-Sbi-2, j-2 | (18)
Jrx = | Srxi + 1, j-1-Srxi-1, j + 1 |
+ | 2Sbi, j-Sbi + 2, j-2-Sbi-2, j + 2 | (19)
When Irx <Jrx, Rx'i, j = (Srxi + 1, j + 1 + Srxi-1, j-1) / 2
+ (2Sbi, j-Sbi + 2, j + 2-Sbi-2, j-2) / 4 (20)
When Irx> Jrx Rx'i, j = (Srxi + 1, j-1 + Srxi-1, j + 1) / 2
+ (2Sbi, j-Sbi + 2, j-2-Sbi-2, j + 2) / 4 (21)
When Irx = Jrx Rx'i, j = (Srxi + 1, j-1 + Srxi-1, j + 1 + Srxi + 1, j + 1 + Srxi-1, j-1) / 4
+ (4Sbi, j-Sbi + 2, j + 2-Sbi-2, j-2-Sbi + 2, j-2-Sbi-2, j + 2) / 8
...... (22)

Further, among the pixels of the Rx image 42rx, for the pixel at the arrangement position corresponding to the G pixel of the original image 41, the gradation values (Srxi-1, j, Srxi + 1, j) or (Srxi, j−1, Srxi, j + 1), the corresponding G pixel gradation value Sgi, j, and 2 above and below or on the left and right sides of the pixel 2 Based on the gradation values (Sgi-2, j, Sgi + 2, j) or (Sgi, j-2, Sgi, j + 2) of two G pixels, Rx is obtained by the following equation (23) or (24). The value (Rx'i, j) is determined.

When Rx pixels are adjacent on the top and bottom Rx'i, j = (Srxi + 1, j + Srxi-1, j) / 2
+ (2Sgi, j-Sgi + 2, j-Sgi-2, j) / 4 (23)
When Rx pixels are adjacent to the left and right Rx'i, j = (Srxi, j + 1 + Srxi, j-1) / 2
+ (2Sgi, j-Sgi, j + 2-Sgi, j-2) / 4 (24)

Through the above process (demosaicing process), using the gradation value (Rx value) of the Rx pixel of the original image 41, a red gradation value (luminance value) is assigned to each pixel of the Rx image 42rx in the processing target region. The indicated Rx value is assigned. Thereby, an Rx image 42rx (a luminance distribution image of red gradation) as a reference luminance distribution image is generated.

  The above is the details of the processing in STEP 3-2.

  Next, the color-specific image generation unit 32 determines the remaining three color-specific images (42r, 42b, 42rx, or 42r, 42) other than the image (G image or Rx image) generated as the reference luminance distribution image in STEP3-3. 42b, 42g). In this case, the remaining three color-specific images in the selected processing target region are generated by reflecting the luminance distribution of the reference luminance distribution image. Details of STEP 3-3 will be described later.

  Next, the color-specific image generation unit 32 determines whether or not all processing target areas have been selected in STEP 3-4. If the determination result is negative, a new process target area is selected in STEP 3-1, and the processes of STEP 3-2 and 3-3 are similarly executed for the selected process target area. . If the determination result in STEP 3-4 is affirmative, the processing in STEP 3 is completed. As a result, color-specific images 42 are generated for all processing target areas.

  Details of the processing of STEP 3-3 will be described below. First, in STEP 3-2, a process of generating the R image 42r, the B image 42b, and the Rx image 42rx when the G image is generated as the reference luminance distribution image will be described.

  First, as shown in FIG. 9, the R image 42r is an image to which an R value (red luminance value) is assigned as the gradation value of each pixel. In FIG. 9, the gradation value (R value) of each pixel is represented by R′i, j (i = 1, 2,..., M, j = 1, 2,..., N).

  In the process of generating the R image 42r in the processing target region being selected by reflecting the luminance distribution of the G image 42g generated as the reference luminance distribution image, the R value (R′i) of each pixel of the R image 42r is generated. , j) is determined as follows.

  That is, among the pixels of the R image 42r, the pixel at the arrangement position corresponding to the R pixel of the original image 41 (pixels having the same arrangement position) is represented by the R pixel of the corresponding original image 41 as shown in Expression (26). Gradation value Sri, j is determined as the R value (R′i, j) of the pixel of the R image 42r.

R'i, j = Sri, j (26)

In addition, among the pixels of the R image 42r, the R value (R′i, j) of the pixel at the arrangement position corresponding to a pixel other than the R pixel of the original image 41 (G pixel, B pixel, or Rx pixel) is Using the gradation value of the R pixel around the pixel (gradation value in the original image 41) and the G value of the pixel and the surrounding pixels (gradation value (G value) in the G image 42g). It is determined.

  Specifically, among the pixels of the R image 42r, with respect to the pixel at the arrangement position corresponding to the B pixel of the original image 41, four R pixels (one pixel interval above and below and right and left of the pixel) ( The gradation value Sri + 2, j, Sri-2, j, Sri, j + 2, Sri, j-2 of the original image 41) and the corresponding pixel (pixel in the G image 42g) G Value (luminance value) G′i, j, and G values (luminance values) G′i + 2, of four pixels (pixels in the G image 42g) existing at intervals of one pixel above and below and on the left and right. Based on j, G'i-2, j, G'i, j + 2, G'i, j-2, depending on the magnitude relationship between Ir and Jr in the following equations (27) and (28), The R value (R′i, j) is determined by the equation (29), (30), or (31).

Ir = | Sri + 2, j−Sri−2, j | + | 2G′i, j−G′i + 2, j−G′i−2, j | (27)
Jr = | Sri, j + 2-Sri, j-2 | + | 2G'i, j-G'i, j + 2-G'i, j-2 | (28)
When Ir <Jr R'i, j = (Sri + 2, j + Sri-2, j) / 2
+ (2G'i, j-G'i + 2, j-G'i-2, j) / 2 (29)
When Ir> Jr R'i, j = (Sri, j + 2 + Sri, j-2) / 2
+ (2G'i, j-G'i, j + 2-G'i, j-2) / 2 (30)
When Ir = Jr R'i, j = (Sri, j + 2 + Sri, j-2 + Sri + 2, j + Sri-2, j) / 4
+ (4G'i, j-G'i + 2, j-G'i-2, j-G'i, j + 2-G'i, j-2) / 4
...... (31)

Further, among the pixels of the R image 42r, for the pixel at the arrangement position corresponding to the G pixel of the original image 41, one R pixel (a pixel in the original image 41) adjacent to any one of the upper, lower, left, and right of the pixel. The gradation value Sri + 1, j or Sri-1, j or Sri, j + 1 or Sri, j-1 and the G value (luminance value) G′i, of the corresponding pixel (the pixel in the G image 42g) j and the G value (luminance value) G′i + 1, j, G′i−1, j, G′i, of one pixel (a pixel in the G image 42g) adjacent to either the top, bottom, left, or right thereof Based on j + 1, G′i, j−1, the R value (Ci, j_r) is determined by the following equation (32) or (33) or (34) or (35).

When R pixel is adjacent to the lower side R′i, j = Sri + 1, j + (G′i, j−G′i + 1, j) (32)
When R pixel is adjacent on the upper side R′i, j = Sri−1, j + (G′i, j−G′i−1, j) (33)
When R pixel is adjacent to the right side R′i, j = Sri, j + 1 + (G′i, j−G′i, j + 1) (34)
When R pixel is adjacent to the left side R'i, j = Sri, j-1 + (G'i, j-G'i, j-1) (35)

Among the pixels of the R image 42r, the pixels at the arrangement positions corresponding to the Rx pixels of the original image 41 are adjacent to the upper right corner and the lower left corner, or the upper left corner and the lower right corner of the pixel. The gradation values (Sri-1, j + 1, Sri + 1, j-1) or (Sri-1, j-1, Sri + 1, j + 1) of two R pixels (pixels in the original image 41) And the G value (luminance value) G′i, j of the corresponding pixel (the pixel in the G image 42g) and two adjacent pixels on the upper right and lower left sides, or on the upper left and lower right sides. G value (luminance value) (G′i−1, j + 1, G′i + 1, j−1) or (G′i−1, j−1, G−1) of a pixel (pixel in the G image 42g) Based on 'i + 1, j + 1), the R value (R'i, j) is determined by the following equation (36) or (37).

R'i, j = (Sri + 1, j-1 + Sri-1, j + 1) / 2 when R pixels are adjacent to the upper right and lower left
+ (2G′i, j−G′i + 1, j−1−G′i−1, j + 1) / 2 (36)
R'i, j = (Sri + 1, j + 1 + Sri-1, j-1) / 2 when R pixel is adjacent to upper left and lower right
+ (2G'i, j-G'i + 1, j + 1-G'i-1, j-1) / 2 (37)

When the G image 42g is generated as the reference luminance distribution image in the processing target region being selected, the pixels other than the pixel at the arrangement position corresponding to the R pixel of the original image 41 among the respective pixels of the R image 42r as described above. The R value (R′i, j) of each pixel is determined based on the gradation value (gradation value in the original image 41) of the R pixel around the pixel (expressions (29) to (29)). 37) is the correction value (Equations (29) to (37)) determined according to the G value (G value in the G image 42g) of the pixel and the surrounding pixels. This is determined by correcting according to the second term on the right-hand side.

  The R value (R′i, j) of each pixel at the arrangement position corresponding to the R pixel of the original image 41 among the pixels of the R image 42r is the level of the R pixel of the original image 41 corresponding to the pixel. It is determined so as to coincide with the key value Sri, j.

  Thereby, the R image 42r in the processing target region being selected is generated by reflecting the luminance distribution of the G image 42g as the reference luminance distribution image.

  Next, as shown in FIG. 10, the B image 42b is an image to which a B value (blue luminance value) is assigned as the gradation value of each pixel. In FIG. 10, the gradation value (B value) of each pixel is represented by B′i, j (i = 1, 2,..., M, j = 1, 2,..., N).

  In the process of generating the B image 42b in the processing target region being selected by reflecting the luminance distribution of the G image 42g generated as the reference luminance distribution image, the B value (B′i) of each pixel of the B image 42b is generated. , j) is determined in the same manner as the R value (R′i, j) of each pixel of the R image 42r.

  That is, among the pixels of the B image 42b, the pixels at the arrangement position corresponding to the B pixels of the original image 41 (the pixels at the same arrangement position) are represented by the B pixels of the corresponding original image 41 as shown in Expression (38). Gradation value Sbi, j is determined as the B value (B′i, j) of the pixel of the B image 42b.

B'i, j = Sbi, j (38)

In addition, among the pixels of the B image 42b, the B value (B′i, j) of the pixel at the arrangement position corresponding to the pixels other than the B pixel of the original image 41 (R pixel, G pixel, or Rx pixel) is Using the gradation value of the B pixel around the pixel (gradation value in the original image 41) and the G value of the pixel and the surrounding pixels (gradation value (G value) in the G image 42g) It is determined.

  Specifically, among the pixels of the B image 42b, with respect to the pixel at the arrangement position corresponding to the R pixel of the original image 41, four B pixels (one pixel interval above and below and right and left of the pixel) ( The gradation value Sbi + 2, j, Sbi-2, j, Sbi, j + 2, Sbi, j-2 of the original image 41) and the corresponding pixel (pixel in the G image 42g) G Value (luminance value) G′i, j, and G values (luminance values) G′i + 2, of four pixels (pixels in the G image 42g) existing at intervals of one pixel above and below and on the left and right. Based on j, G'i-2, j, G'i, j + 2, and G'i, j-2, depending on the magnitude relationship between Ib and Jb in the following equations (39) and (40), The B value (B′i, j) is determined by the equation (41) or (42) or (43).

Ib = | Sbi + 2, j−Sbi−2, j | + | 2G′i, j−G′i + 2, j−G′i−2, j | (39)
Jb = | Sbi, j + 2-Sbi, j-2 | + | 2G'i, j-G'i, j + 2-G'i, j-2 | (40)
When Ib <Jb B'i, j = (Sbi + 2, j + Sbi-2, j) / 2
+ (2G′i, j−G′i + 2, j−G′i−2, j) / 2 (41)
When Ib> Jb B′i, j = (Sbi, j + 2 + Sbi, j−2) / 2
+ (2G'i, j-G'i, j + 2-G'i, j-2) / 2 (42)
When Ib = Jb B'i, j = (Sbi, j + 2 + Sbi, j-2 + Sbi + 2, j + Sbi-2, j) / 4
+ (4G'i, j-G'i + 2, j-G'i-2, j-G'i, j + 2-G'i, j-2) / 4
...... (43)

In addition, among the pixels of the B image 42 b, for the pixel at the arrangement position corresponding to the G pixel of the original image 41, one B pixel adjacent to either the upper, lower, left, or right of the pixel (B pixel in the original image 41) Gradation value Sbi + 1, j or Sbi-1, j or Sbi, j + 1 or Sbi, j-1 and the G value (luminance value) G′i of the corresponding pixel (the pixel in the G image 42g). , j and the G value (luminance value) G′i + 1, j, G′i−1, j, G′i of one pixel (pixel in the G image 42g) adjacent to any one of the top, bottom, left, and right , j + 1, G′i, j−1, the B value (B′i, j) is determined by the following equation (44) or (45) or (46) or (47).

When B pixel is adjacent to the lower side B′i, j = Sbi + 1, j + (G′i, j−G′i + 1, j) (44)
When B pixel is adjacent on the upper side B′i, j = Sbi−1, j + (G′i, j−G′i−1, j) (45)
When B pixel is adjacent to the right side B′i, j = Sbi, j + 1 + (G′i, j−G′i, j + 1) (46)
When B pixel is adjacent to the left side B′i, j = Sbi, j−1 + (G′i, j−G′i, j−1) (47)

Among the pixels of the B image 42b, the pixels at the arrangement positions corresponding to the Rx pixels of the original image 41 are adjacent to the pixel on the upper right side and the lower left side, or on the upper left side and the lower right side. The gradation values (Sbi-1, j + 1, Sbi + 1, j-1) or (Sbi-1, j-1, Sbi + 1, j + 1) of two B pixels (B pixels in the original image 41) ) And the G value (luminance value) G′i, j of the corresponding pixel (the pixel in the G image 42 g), 2 adjacent to the upper right side and the lower left side, or the upper left side and the lower right side. G values (luminance values) (G′i−1, j + 1, G′i + 1, j−1) or (G′i−1, j−1,) of two pixels (pixels in the G image 42g) Based on G′i + 1, j + 1), the B value (B′i, j_b) is determined by the following equation (48) or (49).

When B pixels are adjacent to the upper right and lower left, B'i, j = (Sbi + 1, j-1 + Sbi-1, j + 1) / 2
+ (2G′i, j−G′i + 1, j−1−G′i−1, j + 1) / 2 (48)
When B pixel is adjacent to upper left and lower right B'i, j = (Sbi + 1, j + 1 + Sbi-1, j-1) / 2
+ (2G′i, j−G′i + 1, j + 1−G′i−1, j−1) / 2 (49)

When the G image 42g is generated as the reference luminance distribution image in the processing target region being selected, the pixels other than the pixel at the arrangement position corresponding to the B pixel of the original image 41 among the pixels of the B image 42b are processed as described above. The B value (B′i, j) of each pixel is a basic value (expressions (41) to (49) determined according to the gradation value of the B pixel around the pixel (the gradation value in the original image 41)). ) Of the right side of each of the correction values (expressions (41) to (49)) determined in accordance with the G value (G value in the G image 42g) of the pixel and the surrounding pixels. It is determined by correcting by the second term on the right side).

  Then, among the pixels of the B image 42b, the B value (B′i, j) of each pixel at the arrangement position corresponding to the B pixel of the original image 41 is the level of the B pixel of the original image 41 corresponding to the pixel. It is determined so as to coincide with the key value Sbi, j.

  Thereby, the B image 42b in the processing target region being selected is generated by reflecting the luminance distribution of the G image 42g as the reference luminance distribution image.

  Next, in the process of generating the Rx image 42rx in the selected processing target area by reflecting the luminance distribution of the G image 42g generated as the reference luminance distribution image, the Rx image 42rx (the image shown in FIG. 8) The Rx value (Rx′i, j) of each pixel is determined as follows.

  That is, among the pixels of the Rx image 42rx, the pixel at the arrangement position corresponding to the Rx pixel of the original image 41 (the pixel having the same arrangement position) is represented by the Rx pixel of the corresponding original image 41 as shown in Expression (50). Gradation value Srxi, j is determined as the Rx value (Rx′i, j) of the pixel of the Rx image 42rx.

Rx'i, j = Swi, j (50)

Also, among the pixels of the Rx image 42rx, the Rx value (Rx′i, j) of the pixel at the arrangement position corresponding to the pixel (R pixel, G pixel, or B pixel) other than the Rx pixel of the original image 41 is Using the gradation value of the Rx pixel around the pixel (gradation value in the original image 41) and the G value of the pixel and the surrounding pixels (gradation value (G value) in the G image 42g). It is determined.

  Specifically, among the pixels of the Rx image 42rx, the pixels at the arrangement positions corresponding to the R pixel or the B pixel of the original image 41 are diagonally upper right, lower right, upper left, and lower left of the pixel. Gradation values Srxi-1, j + 1, Srxi + 1, j + 1, Srxi-1, j-1, Srxi + 1, j-1 of four Rx pixels adjacent to (Rx pixel in the original image) And the G value G′i, j of the corresponding pixel (pixel in the G image) and four pixels (in the G image 42g) that are adjacent to the upper right side, the lower right side, the upper left side, and the lower left side. Pixel) G values G′i−1, j + 1, G′i + 1, j + 1, G′i−1, j−1, G′i + 1, j−1 The Rx value (Rx′i, j) is determined by the equation (53) or (54) or (55) according to the magnitude relationship between Irx and Jrx in the equations (51) and (52).

Irx = | Srxi + 1, j + 1-Srxi-1, j-1 |
+ | 2G′i, j−G′i + 1, j + 1−G′i−1, j−1 | (51)
Jrx = | Srxi + 1, j-1-Srxi-1, j + 1 |
+ | 2G′i, j−G′i + 1, j−1−G′i−1, j + 1 | (52)
When Irx <Jrx, Rx'i, j = (Srxi + 1, j + 1 + Srxi-1, j-1) / 2
+ (2G'i, j-G'i + 1, j + 1-G'i-1, j-1) / 2 (53)
When Irx> Jrx Rx'i, j = (Srxi + 1, j-1 + Srxi-1, j + 1) / 2
+ (2G′i, j−G′i + 1, j−1−G′i−1, j + 1) / 2 (54)
When Irx = Jrx Rx'i, j = (Srxi + 1, j + 1 + Srxi + 1, j-1 + Srxi-1, j + 1 + Srxi-1, j-1) / 4
+ (4G′i, j−G′i + 1, j + 1−G′i + 1, j−1−G′i−1, j + 1−G′i−1, j−1) / 4
...... (55)

Further, among the pixels of the Rx image 42rx, for the pixel at the arrangement position corresponding to the G pixel of the original image 41, the gradation of two Rx pixels (pixels in the original image 41) adjacent to the pixel vertically and horizontally The value (Srxi-1, j, Srxi + 1, j) or (Srxi, j-1, Srxi, j + 1) and the G value (luminance value) G ′ of the corresponding pixel (the pixel in the G image 42g) i, j and G values (luminance values) (G′i−1, j, G′i + 1, j) or (G Based on 'i, j-1, G'i, j + 1), the Rx value (Rx'i, j) is determined by the following equation (56) or (57).

When Rx pixels are adjacent on the top and bottom Rx'i, j = (Srxi + 1, j + Srxi-1, j) / 2
+ (2G'i, j-G'i + 1, j-G'i-1, j) / 2 (56)
When Rx pixels are adjacent to the left and right Rx'i, j = (Srxi, j + 1 + Srxi, j-1) / 2
+ (2G'i, j-G'i, j + 1-G'i, j-1) / 2 (57)

When the G image 42g is generated as the reference luminance distribution image in the processing target region being selected, the pixels other than the pixels at the arrangement positions corresponding to the Rx pixels of the original image 41 among the pixels of the Rx image 42rx are as described above. The Rx value (Rx′i, j) of each pixel is a basic value (expressions (53) to (57) determined in accordance with the gradation value of the Rx pixel around the pixel (the gradation value in the original image 41). ) Is the correction value (Equations (53) to (57)) determined in accordance with the G value (G value in the G image 42g) of the pixel and its surrounding pixels. This is determined by correcting according to the second term on the right-hand side.

  Then, among the pixels of the Rx image 42rx, the Rx value (Rx′i, j) of each pixel at the arrangement position corresponding to the Rx pixel of the original image 41 is the level of the Rx pixel of the original image 41 corresponding to the pixel. It is determined so as to match the key value Srxi, j.

  Thereby, the Rx image 42rx in the processing target region being selected is generated by reflecting the luminance distribution of the G image 42g as the reference luminance distribution image.

  The above is the remaining three color-specific images (R image 42r, B image 42b, and Rx image 42rx) reflecting the luminance distribution of the G image 42g generated as the reference luminance distribution image in the processing target region being selected. Is a process for generating

  Next, in STEP 3-2, processing for generating the R image 42r, the B image 42b, and the G image 42g when the Rx image is generated as the reference luminance distribution image will be described.

  First, regarding the R image 42r, in the process of generating the R image 42r in the processing target region being selected by reflecting the luminance distribution of the Rx image 42rx generated as the reference luminance distribution image, R of each pixel of the R image 42r is generated. The value (R′i, j) is determined as follows.

  That is, among the pixels of the R image 42r, the pixel at the arrangement position corresponding to the R pixel of the original image 41 (pixels having the same arrangement position) is represented by the R pixel of the corresponding original image 41 as shown in Expression (58). Gradation value Sri, j is determined as the R value (R′i, j) of the pixel of the R image 42r.

R'i, j = Sri, j (58)

In addition, among the pixels of the R image 42r, the R value (R′i, j) of the pixel at the arrangement position corresponding to a pixel other than the R pixel of the original image 41 (G pixel, B pixel, or Rx pixel) is It is determined using the gradation value of the R pixel around the pixel (gradation value in the original image 41) and the Rx value of the pixel and the surrounding pixels (gradation value in the Rx image 42rx).

  Specifically, among the pixels of the R image 42r, with respect to the pixel at the arrangement position corresponding to the B pixel of the original image 41, four R pixels (one pixel interval above and below and right and left of the pixel) ( Rx value of the pixel (pixel in the Rx image 42rx) corresponding to the gradation value Sri + 2, j, Sri-2, j, Sri, j + 2, Sri, j-2 of the original image 41) (Luminance value) Rx'i, j, and Rx values (luminance values) Rx'i + 2, j of four pixels (pixels in the Rx image 42rx) existing at intervals of one pixel vertically and horizontally , Rx'i-2, j, Rx'i, j + 2, Rx'i, j-2 based on the relationship between Ir and Jr in the following equations (59) and (60) The R value (R′i, j) is determined by (61), (62), or (63).

Ir = | Sri + 2, j-Sri-2, j | + | 2Rx'i, j-Rx'i + 2, j-Rx'i-2, j | (59)
Jr = | Sri, j + 2-Sri, j-2 | + | 2Rx'i, j-Rx'i, j + 2-Rx'i, j-2 | (60)
When Ir <Jr R'i, j = (Sri + 2, j + Sri-2, j) / 2
+ (2Rx'i, j-Rx'i + 2, j-Rx'i-2, j) / 2 (61)
When Ir> Jr R'i, j = (Sri, j + 2 + Sri, j-2) / 2
+ (2Rx'i, j-Rx'i, j + 2-Rx'i, j-2) / 2 (62)
When Ir = Jr R'i, j = (Sri, j + 2 + Sri, j-2 + Sri + 2, j + Sri-2, j) / 4
+ (4Rx'i, j-Rx'i + 2, j-Rx'i-2, j-Rx'i, j + 2-Rx'i, j-2) / 4
...... (63)

Further, among the pixels of the R image 42r, for the pixel at the arrangement position corresponding to the G pixel of the original image 41, one R pixel (R pixel of the original image 41) adjacent to either the upper, lower, left, or right of the pixel. The gradation value Sri + 1, j or Sri-1, j or Sri, j + 1 or Sri, j-1 and the corresponding pixel (pixel in the Rx image 42rx) Rx value (luminance value) Rx'i, j and the Rx value (luminance value) Rx′i + 1, j, Rx′i−1, j, Rx′i, of one pixel (a pixel in the Rx image 42rx) adjacent to either the top, bottom, left, or right thereof Based on j + 1 and Rx′i, j−1, the R value (R′i, j) is determined by the following equation (64) or (65) or (66) or (67).

When R pixel is adjacent to the lower side R′i, j = Sri + 1, j + (Rx′i, j−Rx′i + 1, j) (64)
When R pixel is adjacent on the upper side R′i, j = Sri−1, j + (Rx′i, j−Rx′i−1, j) (65)
When R pixel is adjacent on the right side R′i, j = Sri, j + 1 + (Rx′i, j−Rx′i, j + 1) (66)
When R pixel is adjacent to the left side R'i, j = Sri, j-1 + (Rx'i, j-Rx'i, j-1) (67)

Among the pixels of the R image 42r, the pixels at the arrangement positions corresponding to the Rx pixels of the original image 41 are adjacent to the upper right corner and the lower left corner, or the upper left corner and the lower right corner of the pixel. The gradation values (Sri-1, j + 1, Sri + 1, j-1) or (Sri-1, j-1, Sri + 1, j + 1) of two R pixels (R pixels in the original image 41) ) And the Rx value (luminance value) Rx′i, j of the corresponding pixel (the pixel in the Rx image 42rx), 2 adjacent to the upper right and lower left, or the upper left and lower right. Rx values (luminance values) (Rx′i−1, j + 1, Rx′i + 1, j−1) or (Rx′i−1, j−1,) of two pixels (pixels in the Rx image 42rx) Based on Rx′i + 1, j + 1), the R value (R′i, j) is determined by the following equation (68) or (69).

R'i, j = (Sri + 1, j-1 + Sri-1, j + 1) / 2 when R pixels are adjacent to the upper right and lower left
+ (2Rx'i, j-Rx'i + 1, j-1-Rx'i-1, j + 1) / 2 (68)
R'i, j = (Sri + 1, j + 1 + Sri-1, j-1) / 2 when R pixel is adjacent to upper left and lower right
+ (2Rx'i, j-Rx'i + 1, j + 1-Rx'i-1, j-1) / 2 (69)

When the Rx image 42rx is generated as the reference luminance distribution image in the processing target region being selected, the pixels other than the pixel at the arrangement position corresponding to the R pixel of the original image 41 among the pixels of the R image 42r as described above. The R value (R′i, j) of each pixel is determined based on the gradation values (gradation values in the original image 41) of the R pixels around the pixel (expressions (61) to (61)). 69) is a correction value (expressions (61) to (69)) determined in accordance with the Rx values (Rx values in the Rx image 42rx) of the pixel and its surrounding pixels. This is determined by correcting according to the second term on the right-hand side.

  The R value (R′i, j) of each pixel at the arrangement position corresponding to the R pixel of the original image 41 among the pixels of the R image 42r is the level of the R pixel of the original image 41 corresponding to the pixel. It is determined so as to coincide with the key value Sri, j.

  Thereby, the R image 42r in the processing target region being selected is generated by reflecting the luminance distribution of the Rx image 42rx as the reference luminance distribution image.

  Next, regarding the B image 42b, in the process of generating the B image 42b in the processing target region being selected by reflecting the luminance distribution of the Rx image 42rx generated as the reference luminance distribution image, each pixel of the B image 42b is generated. The B value (B′i, j) is determined in the same manner as the R value (R′i, j) of each pixel of the R image 42r.

  That is, among the pixels of the B image 42b, the pixels at the arrangement position corresponding to the B pixels of the original image 41 (the pixels having the same arrangement position) are represented by the B pixels of the corresponding original image 41 as shown in Expression (70). Gradation value Sbi, j is determined as the B value (B′i, j) of the pixel of the B image 43b.

B'i, j = Sbi, j (70)

In addition, among the pixels of the B image 42b, the B value (B′i, j_b) of the pixel at the arrangement position corresponding to the pixel (R pixel, G pixel, or Rx pixel) other than the B pixel of the original image 41 is Using the gradation value of the B pixel around the pixel (gradation value in the original image 41) and the Rx value of the pixel and the surrounding pixels (gradation value (Rx value) in the Rx image 42rx) It is determined.

  Specifically, among the pixels of the B image 42b, with respect to the pixel at the arrangement position corresponding to the R pixel of the original image 41, four B pixels (one pixel interval above and below and right and left of the pixel) ( Rx value of the corresponding pixel (the pixel in the Rx image 42rx) and the gradation value Sbi + 2, j, Sbi-2, j, Sbi, j + 2, Sbi, j-2 of the pixel in the original image 41) (Luminance value) Rx'i, j, and Rx values (luminance values) Rx'i + 2, j of four pixels (pixels in the Rx image 42rx) existing at intervals of one pixel vertically and horizontally , Rx'i-2, j, Rx'i, j + 2, Rx'i, j-2 based on the magnitude relationship between Ib and Jb in the following formulas (71) and (72) The B value (B′i, j) is determined by (73), (74), or (75).

Ib = | Sbi + 2, j-Sbi-2, j | + | 2Rx'i, j-Rx'i + 2, j-Rx'i-2, j | (71)
Jb = | Sbi, j + 2-Sbi, j-2 | + | 2Rx'i, j-Rx'i, j + 2-Rx'i, j-2 | (72)
When Ib <Jb B'i, j = (Sbi + 2, j + Sbi-2, j) / 2
+ (2Rx'i, j-Rx'i + 2, j-Rx'i-2, j) / 2 (73)
When Ib> Jb B′i, j = (Sbi, j + 2 + Sbi, j−2) / 2
+ (2Rx'i, j-Rx'i, j + 2-Rx'i, j-2) / 2 (74)
When Ib = Jb B'i, j = (Sbi, j + 2 + Sbi, j-2 + Sbi + 2, j + Sbi-2, j) / 4
+ (4Rx'i, j-Rx'i + 2, j-Rx'i-2, j-Rx'i, j + 2-Rx'i, j-2) / 4
...... (75)

In addition, among the pixels of the B image 42 b, for the pixel at the arrangement position corresponding to the G pixel of the original image 41, one B pixel adjacent to either the upper, lower, left, or right of the pixel (B pixel in the original image 41) Gradation value Sbi + 1, j or Sbi-1, j or Sbi, j + 1 or Sbi, j-1 and the Rx value (luminance value) Rx'i of the corresponding pixel (the pixel in the Rx image 42rx). , j and one pixel (pixel in the Rx image 42rx) adjacent to either the top, bottom, left, or right thereof, the Rx value (luminance value) Rx'i + 1, j, Rx'i-1, j, Rx'i , j + 1, Rx′i, j−1, the B value (B′i, j) is determined by the following formula (76) or (77) or (78) or (79).

When B pixel is adjacent to the lower side B′i, j = Sbi + 1, j + (Rx′i, j−Rx′i + 1, j) (76)
When B pixel is adjacent on the upper side B′i, j = Sbi−1, j + (Rx′i, j−Rx′i−1, j) (77)
When B pixel is adjacent to the right side B′i, j = Sbi, j + 1 + (Rx′i, j−Rx′i, j + 1) (78)
When B pixel is adjacent to the left side B'i, j = Sbi, j-1 + (Rx'i, j-Rx'i, j-1) (79)

Among the pixels of the B image 42b, the pixels at the arrangement positions corresponding to the Rx pixels of the original image 41 are adjacent to the pixel on the upper right side and the lower left side, or on the upper left side and the lower right side. The gradation values (Sbi-1, j + 1, Sbi + 1, j-1) or (Sbi-1, j-1, Sbi + 1, j + 1) of two B pixels (B pixels in the original image 41) ) And the Rx value (luminance value) Rx′i, j of the corresponding pixel (the pixel in the Rx image 42rx), 2 adjacent to the upper right and lower left, or the upper left and lower right. Rx values (luminance values) (Rx′i−1, j + 1, Rx′i + 1, j−1) or (Rx′i−1, j−1,) of two pixels (pixels in the Rx image 42rx) Based on Rx′i + 1, j + 1), the B value (B′i, j) is determined by the following equation (80) or (81).

When B pixels are adjacent to the upper right and lower left, B'i, j = (Sbi + 1, j-1 + Sbi-1, j + 1) / 2
+ (2Rx'i, j-Rx'i + 1, j-1-Rx'i-1, j + 1) / 2 (80)
When B pixel is adjacent to upper left and lower right B'i, j = (Sbi + 1, j + 1 + Sbi-1, j-1) / 2
+ (2Rx'i, j-Rx'i + 1, j + 1-Rx'i-1, j-1) / 2 (81)

When the Rx image 42rx is generated as the reference luminance distribution image in the processing target region being selected, the pixels other than the pixels at the arrangement positions corresponding to the B pixels of the original image 41 among the respective pixels of the B image 42b as described above. The B value (B′i, j) of each pixel is a basic value (expressions (73) to (73)) determined according to the gradation value of the B pixel around the pixel (the gradation value in the original image 41). 81) is a correction value (expressions (73) to (81)) determined in accordance with the Rx values (Rx values in the Rx image 42rx) of the pixel and its surrounding pixels. This is determined by correcting according to the second term on the right-hand side.

  Then, among the pixels of the B image 42b, the B value (B′i, j) of each pixel at the arrangement position corresponding to the B pixel of the original image 41 is the level of the B pixel of the original image 41 corresponding to the pixel. It is determined so as to coincide with the key value Sbi, j.

  Thereby, the B image 42b in the processing target region being selected is generated by reflecting the luminance distribution of the Rx image 42rx as the reference luminance distribution image.

  Next, in the process of generating the G image 42g in the selected processing target region by reflecting the luminance distribution of the Rx image 42rx generated as the reference luminance distribution image, the G value (G ′ i, j) is determined as follows.

  That is, among the pixels of the G image 42g, the pixel at the arrangement position corresponding to the G pixel of the original image 41 (pixels having the same arrangement position) is represented by the G pixel of the corresponding original image 41 as shown in Expression (82). Gradation value Sgi, j is determined as the G value (G′i, j) of the pixel of the G image 42g.

G'i, j = Sgi, j (82)

In addition, among the pixels of the G image 42g, the G value (G′i, j) of the pixel at the arrangement position corresponding to a pixel (R pixel, B pixel, or Rx pixel) other than the G pixel of the original image 41 is Using the gradation value of the G pixel around the pixel (gradation value in the original image 41) and the Rx value of the pixel and the surrounding pixels (gradation value (Rx value) in the Rx image 42rx) It is determined.

  Specifically, among the pixels of the G image 42g, for the pixels at the arrangement positions corresponding to the pixels other than the G pixel of the original image 41, four G pixels adjacent to the upper and lower sides and the left and right of the pixel (the original image 41). The tone value Sgi + 1, j, Sgi-1, j, Sgi, j + 1, Sgi, j-1 of the G pixel in the middle) and the Rx value (luminance) of the corresponding pixel (the pixel in the Rx image 42rx) Value) Rx′i, j, and Rx values (luminance values) Rx′i + 1, j, Rx′i−1, j of four pixels (pixels in the Rx image 42rx) adjacent to the top, bottom, left and right thereof, Based on Rx′i, j + 1 and Rx′i, j−1, depending on the magnitude relationship between Ig and Jg in the following formulas (83) and (84), formula (85) or (86) or ( 87), the G value (G'i, j) is determined.

Ig = | Sgi + 1, j-Sgi-1, j | + | 2Rx'i, j-Rx'i + 1, j-Rx'i-1, j | (83)
Jg = | Sgi, j + 1-Sgi, j-1 | + | 2Rx'i, j-Rx'i, j + 1-Rx'i, j-1 | (84)
When Ig <Jg G'i, j = (Sgi + 1, j + Sgi-1, j) / 2
+ (2Rx'i, j-Rx'i + 1, j-Rx'i-1, j) / 2 (85)
When Ig> Jg G'i, j = (Sgi, j + 1 + Sgi, j-1) / 2
+ (2Rx'i, j-Rx'i, j + 1-Rx'i, j-1) / 2 (86)
When Ig = Jg G'i, j = (Sgi, j + 1 + Sgi, j-1 + Sgi + 1, j + Sgi-1, j) / 4
+ (4Rx'i, j-Rx'i + 1, j-Rx'i-1, j-Rx'i, j + 1-Rx'i, j-1) / 4
...... (87)

When the Rx image 42rx is generated as the reference luminance distribution image in the processing target region being selected, the pixels other than the pixels at the arrangement positions corresponding to the G pixels of the original image 41 among the pixels of the G image 42g as described above. The G value (G′i, j) of each pixel is determined based on the gradation values (gradation values in the original image 41) of G pixels around the pixel (expressions (85) to ( 87), the first term on the right side of each of the correction values (expressions (85) to (87)) determined in accordance with the Rx values of the pixel and the surrounding pixels (Rx values in the Rx image 42rx). This is determined by correcting according to the second term on the right-hand side.

  Then, among the pixels of the G image 42g, the G value (G′i, j) of each pixel at the arrangement position corresponding to the G pixel of the original image 41 is the level of the G pixel of the original image 41 corresponding to the pixel. It is determined so as to coincide with the key value Sgi, j.

  Thereby, the G image 42r in the processing target region being selected is generated by reflecting the luminance distribution of the Rx image 42rx as the reference luminance distribution image.

  The above is the remaining three color-specific images (R image 42r, B image 42b, and G image 42g) reflecting the luminance distribution of the Rx image 42rx generated as the reference luminance distribution image in the processing target region being selected. Is a process for generating

  Returning to FIG. 4, four types of color-specific images (R image 42 r, G image 42 g, and B image 42 b) for each processing target area are obtained by the processing of STEP 4 described above (processing by the color-specific image generation unit 32). And the Rx image 42rx), the processing of STEP 4 is executed by the monitoring image acquisition unit 33.

  In this processing, the monitoring image acquisition unit 33 is suitable for detection of the monitoring target by the target detection unit 34 according to the type of the monitoring target to be detected by the target detection unit 34, the shooting environment, and the like. A monitoring image (which is easy to detect) is acquired, and the acquired monitoring image is output to the object detection unit 34. The monitoring image is an image selected from four types of color-specific images 42 (R image 42r, G image 42g, B image 42b, and Rx image 42rx) of each processing target region, or a plurality of types of processing target regions. It is an image generated by combining the color-specific images 42.

  In the present embodiment, the monitoring image acquisition unit 33, for example, the Rx image 42rx (or the combined image of the Rx images 42rx of a plurality of processing target areas adjacent to each other) among the color-specific images 42 of each processing target area. And the first color image 43 (or a plurality of processing target areas adjacent to each other) generated by combining the R image 42r, the G image 42g, and the B image 42b of the Rx color-specific images 42 of each processing target area. Are combined with each color image 42 (R image 42r, G image 42g, B image 42b, and Rx image 42rx) of each processing target area. One or a plurality of images of the second color image 44 (or a combination image of the second color images 44 of a plurality of processing target areas adjacent to each other) are acquired and output.

  As shown in FIG. 11, the first color image 43 of each processing target region includes an R value (C1i, j_r) that is a gradation value of R (red) and a gradation of G (green). Gradation value C1i configured as a set (a set of gradation values of three colors) of a G value (C1i, j_g) as a value and a B value (C1i, j_b) as a gradation value of B (blue) , j is the assigned image.

  In this case, as indicated by the following equations (88a) to (88c), the R value (C1i, j_r) and G value (G values) of the gradation values C1i, j of each pixel of the first color image 43 in each processing target region ( C1i, j_g) and B value (C1i, j_b) are respectively the R value (R′i, j) of the corresponding pixel of the R image 42r in the processing target area and the corresponding pixel of the G image 42g of the processing target area. The G value (G′i, j) and the B value (B′i, j) of the corresponding pixel of the B image 42b in the processing target area are set as they are.

C1i, j_r = R'i, j (88a)
C1i, j_g = G'i, j (88b)
C1i, j_b = B'i, j (88c)

In addition, as shown in FIG. 12, the second color image 44 of each processing target region includes R value (C2i, j_r), which is a gradation value of R (red), and G (green). Gradation configured as a set of G values (C2i, j_g) as gradation values and B values (C2i, j_b) as gradation values of B (blue) (a set of gradation values of three colors) This is an image to which the value C2i, j is assigned.

  In this case, the R value (C2i, j_r) of the gradation values C2i, j of each pixel of the second color image 44 in each processing target area is the R value (C2i, j_r) of the corresponding pixel of the R image 42r in the processing target area ( R′i, j) is determined by correcting by the following equation (89a) according to the Rx value (Rx′i, j) of the corresponding pixel of the Rx image 42rx of the processing target region.

  Similarly, the G value (C2i, j_b) among the gradation values C2i, j of each pixel of the second color image 44 in each processing target area is the G value (C2i, j_b) of the corresponding pixel of the G image 42g in the processing target area ( G′i, j) is determined by correcting by the following equation (89b) according to the Rx value (Rx′i, j) of the corresponding pixel of the Rx image 42rx of the processing target region.

  Similarly, the B value (C2i, j_b) of the gradation values C2i, j of each pixel of the second color image 44 in each processing target area is the B value (C value of the corresponding pixel of the B image 42b in the processing target area ( B′i, j) is determined by correcting by the following equation (89c) according to the Rx value (Rx′i, j) of the corresponding pixel of the Rx image 42rx of the processing target region. In other words, the second color image 44 is a color image obtained by correcting the first color image 43 according to the luminance distribution of the Rx image 42rx.

C2i, j_r = (R'i, j + ([alpha] r * Rx'i, j + [beta] r)) / [gamma] r (89a)
C2i, j_g = (G'i, j + (αg × Rx'i, j + βg)) / γg (89b)
C2i, j_r = (B'i, j + ([alpha] b * Rx'i, j + [beta] b)) / [gamma] b (89c)

Here, αr in Expression (89a) is for adjusting the contribution ratio of the Rx value (Rx′i, j) of the Rx image 42rx to the R value (C2i, j_r) of each pixel of the second color image 44. Gain factor. Αg in equation (89b) and αb in equation (89c) are gain coefficients having the same purpose as αr.

  The gradation value Rx′i, j of the Rx image 42rx in the processing target region takes an effective value when the luminance (brightness) of the photographing target is a high luminance region, and thus the contrast of the high luminance region with respect to the photographing target. When it is required to use the second color image 44 having a relatively high value as the monitoring image, it is desirable to set the gain coefficients αr, αg, and αb to be larger.

  When the second color image 44 having a relatively high color resolution is required to be used as the monitoring image, it is desirable to set the gain coefficients αr, αg, αb to be small.

  Note that the gain coefficients αr, αg, and αb may be set to different values.

  In addition, βr in the equation (89a) is the case where the Rx value (Rx′i, j) of the Rx image 42rx has a gradation value equal to or higher than a certain value (for example, the luminance of the shooting target is the gradation of the R image 42r). Rx value (Rx'i, j) is reflected in the R value (C2i, j_r) of the second color image 44 only when the brightness is in a high brightness region such that the value is saturated or close to that value. This is a negative offset value for making (contribute). Βg in the equation (89b) and βb in the equation (89c) are also offset values (<0) having the same purpose as βr.

  In this case, the value of (αr × Rx′i, j + βr) in the equation (89a) is the same value when αr × Rx′i, j + βr ≧ 0, but αr × Rx′i, j + βr. When <0, the value of αr × Rx′i, j + βr is forcibly set to “0”. Therefore, only when Rx′i, j ≧ −βr / αr, the Rx value (Rx′i, j) is reflected in the R value (C2i, j_r) of the second color image 44, and Rx′i, j When <−βr / αr, the R value (C2i, j_r) of the second color image 44 can be made independent of the Rx value (Rx′i, j).

  The same applies to βg in the formula (89b) and βb in the formula (89c). Note that the offset values βr, βg, and βb may be set to different values.

  Further, γr in the equation (89a) is a correction coefficient for adjusting the data length of C2i, j_r (variable range of gradation values) to the data length of R′i, j and Rx′i, j. Γg in Expression (89b) and γb in Expression (89c) are correction coefficients having the same purpose as γr.

  Γr in the formula (89a) is determined according to the values of αr and βr. For example, when R′i, j and Rx′i, j are data represented by 8-bit gradation values (gradation values from 0 to 255), αr = 1, βr = 0, or αr = 1 and βr = −128, γr is set to γr = 3/2. As a result, C2i, j_r can be represented by an 8-bit gradation value.

  The same applies to γg in formula (89b) and γb in formula (89c).

  The gain coefficients αr, αg, αb, the offset values βr, βg, βb and the correction coefficients γr, γg, γb used for generating the second color image 44 by the equations (89a) to (89c) The light receiving characteristics of the R light receiving pixel, the G light receiving pixel, the B light receiving pixel, and the Rx light receiving pixel of the element 22 and the image characteristics required for the second color image 44 for detecting the monitoring object (color resolution, high luminance region, etc.) Is set so that the second color image 44 having the required image characteristics can be generated on the basis of the image characteristics such as contrast with respect to the imaging object having the brightness of (2).

  As these set values, predetermined default values or values variably determined according to a request from the object detection unit 34 or the like can be used.

  Note that the second color image 44 generated as described above can have the dynamic range pseudo-expanded more than the first color image 43. However, the first color image 43 is more advantageous than the second color image 44 in increasing the color resolution and the overall contrast of the image.

  The monitoring image acquisition unit 33 includes an Rx image 42rx (or a combination image) of each processing target region and each processing target region according to the type of the monitoring target detected by the target detection unit 34, the shooting environment, and the like. One or a plurality of images of the first color image 43 (or a combination image thereof) and the two color images 44 (or a combination image thereof) of each processing target region are acquired and output as a monitoring image. .

  For example, when the object detection unit 34 tries to detect another vehicle (a preceding vehicle or an oncoming vehicle) in front of the vehicle 1 at night, the monitoring image acquisition unit 33 corresponds to an area on the road surface. The Rx image 42rx of one or a plurality of processing target areas is output to the target object detection unit 34 as a monitoring image.

  Since the Rx image 42rx is an image generated based on the gradation value of the Rx pixel of the original image 41 corresponding to the Rx light receiving pixel having low light receiving sensitivity, the taillight of another vehicle as a high-luminance luminescent material is obtained. In addition, the headlight image is easily obtained as an image that can be distinguished from other images without being overexposed. For this reason, based on the Rx image 42rx, the taillight of the other vehicle and the image of the headlight can be detected with high reliability, and thus the other vehicle can be appropriately detected.

  The Rx image 42rx can be effectively used when detecting a traffic light having a high-luminance luminescent material.

  When trying to detect other vehicles and traffic lights at night, the second color image 44 may be output to the object detection unit 34 as a monitoring image instead of the Rx image 42rx. Since the second color image 44 has a wide dynamic range, an image of a high-luminance taillight of another vehicle, a headlight, or a light emitting part of a traffic light can be obtained with a good color resolution. Also, it is possible to obtain an image of a relatively dark object around the taillight, the headlight, and the light emitting part of the traffic light.

  In addition, when the object detection unit 34 is trying to detect a pedestrian in front of the vehicle 1 or a lane (lane mark) on the road surface, the monitoring image acquisition unit 33 may detect an area or road surface where a pedestrian may exist. The first color image 43 of one or a plurality of processing target areas corresponding to this area is output to the object detection unit 34 as a monitoring image.

  Since the first color image 43 has a higher color resolution than the second color image 44, it can detect pedestrians and road lanes (white or yellow lanes, etc.) with high reliability by distinguishing them from other object images. can do.

  At night, the second color image 44 having a wide dynamic range may be output as a monitoring image in order to detect pedestrians and road lanes (lane marks).

  In the present embodiment, the monitoring image acquisition unit 33 acquires any one of the Rx image 42rx, the first color image 43, and the second color image 44 as the monitoring image as described above, and uses it as the object detection unit 34. Output to. In this case, when the first color image 43 or the second color image 44 is used for monitoring, the color images 43 and 44 are generated by the monitoring image acquisition unit 33 and held in the image memory 40. .

  Supplementally, as the processing mode of the monitoring image acquisition unit 33, for example, the following modes may be adopted in addition to the above mode. That is, the monitoring image acquisition unit 33 generates and outputs only the first color image 43 as a monitoring image in a bright environment such as daytime.

  Then, from the brightness of each pixel of the first color image 43 in one or more processing target areas at night (linear combination value of R value, G value, and B value of each pixel) and the exposure degree of the camera 2 When the calculated average value of the brightness level of the first color image 43 reaches a predetermined threshold or more, or the contrast of the first color image 43 (more specifically, for example, the brightness of the first color image 43 When the distribution of the color histogram or the dispersion of the color histogram reaches a predetermined threshold value or more, the monitoring image acquisition unit 33 sets the pair of the Rx image 42rx and the first color image 43 or the second color image 44. Is output as a monitoring image.

  Here, at night, in a situation in which a subject to be photographed by the camera 2 includes a high-luminance luminescent object such as a taillight or headlight of another vehicle or a light emitting unit of a traffic light, the luminescent object is not included. Compared to the case, the average value of the brightness and the contrast tend to be relatively large.

  Therefore, when a high-luminance luminescent object is included in the shooting target of the camera 2, the Rx image 42rx or the second color image 44 that easily detects the luminescent object can be used as a monitoring image.

  The average value of the brightness level and the area for calculating the contrast may be the entire area of the captured image of the camera 2 (area that matches the entire original image 41). It may be a part of the area. For example, as shown in FIG. 13, out of the entire captured image of the camera 2, an area A <b> 1 in a certain range including an infinite point on the road surface in front of the vehicle 1 (host vehicle) (an area where another vehicle can exist), The average value of the brightness level and the contrast are calculated only in one or both of the upper area A2 (area where a traffic signal or a sign can be present) in the entire captured image of the camera 2. It may be.

  Returning to the description of FIG. 4, subsequent to the processing of the monitoring image acquisition unit 33 in STEP 4, processing of detecting the monitoring target ahead of the vehicle 1 is executed by the target detection unit 34 in STEP 5.

  In this processing, the object detection unit 34 detects the monitoring object using the monitoring image provided from the monitoring image acquisition unit 33. For example, when detecting another vehicle as a monitoring target at night, the target detection unit 34 uses the taillight of the other vehicle from the Rx image 42rx (or the second color image 44) as the monitoring image. Alternatively, the headlight image is detected based on the feature of the color and the feature of the arrangement form (feature such that two pairs of high-intensity images are arranged side by side). Detects another vehicle (front vehicle or oncoming vehicle) present in

  In this case, in particular, since the Rx image 42rx is a red gradation image, it is possible to suitably detect the taillight of another vehicle.

  In a bright environment in the daytime, the other vehicle can be detected by using the first color image 43 as the monitoring image.

  In addition, when detecting a traffic signal or a road sign as an object to be monitored, when detecting a lane (lane mark) on a road surface, or when detecting a pedestrian, the object detection unit 34 is, for example, By using the Rx image (or the second color image 44) as the monitoring image at night and using the first color image 43 as the monitoring image during the day, the color characteristics, the luminance distribution characteristics, Those monitoring objects are detected based on the shape features and the like.

  In this case, the object detection unit 34 is prone to show noticeable colors, luminance distributions, and shapes that are characteristic of the monitoring object depending on the shooting environment (brightness, etc.) and the type of the monitoring object to be detected. The monitoring object can be detected using the monitoring image (any one of the Rx image 42rx, the first color image 43, and the second color image 44). Therefore, various objects can be detected with high reliability under various environments.

  When the object detection unit 34 detects a pedestrian or another vehicle, the object detection unit 34 determines the possibility of contact between the pedestrian or other vehicle and the vehicle 1 (own vehicle). If it is determined, a control signal for instructing the vehicle controller 6 to perform the contact avoidance measure is transmitted. At this time, the display display control unit 63 of the vehicle controller 6 displays an alarm on the display 73. Moreover, the braking control part 62 performs the process which operates the braking device 72 and avoids a contact as needed.

  In addition, the object detection unit 34 performs lane keeping control in which the vehicle 1 (own vehicle) is maintained and travels within the traveling region of the own vehicle defined by the lane in response to detection of the lane (lane mark). The control signal for transmitting to the vehicle controller 6 is transmitted. At this time, the steering control unit 61 controls the operation of the steering device 71 to perform lane keep control.

  According to the embodiment described above, in addition to the R light receiving pixel, the G light receiving pixel, and the B light receiving pixel as the color light receiving pixels, the Rx light receiving pixel having lower light receiving sensitivity than these light receiving pixels. As a result, the object to be imaged by the camera 2 includes a high-luminance object (such as a taillight or headlight of another vehicle or a light emitting unit of a traffic light) that causes saturation of the output signal of the color light receiving pixel. Even in such a case, it is possible to acquire the Rx image 42rx or the second color image 44 showing the luminance distribution of the high-luminance object at an appropriate resolution.

  For this reason, by using these Rx images 42rx or the second color image 44, it is possible to detect a high-luminance object such as a taillight or headlight of another vehicle or a light emitting unit of a traffic light with high reliability. .

  In particular, since the Rx image 42rx is a red gradation image, a high-intensity red object that is highly important for the vehicle 1 traveling in a road environment (such as a taillight of another vehicle or a red light emitting unit of a traffic light). Can be appropriately detected using the Rx image 42rx.

  In addition, among the four types of color-specific images 42 of the R image 42r, the G image 42g, the B image 42b, and the Rx image 42rx generated from the original image 41 by the demosaicing process, The Rx image 42rx based on the intensity (the gradation value of the Rx pixel of the original image 41) and the R image 42r, the G image 42g, and the B image 42b for the three primary colors among the four types of color-specific images 42 are synthesized. And a second color image 44 (a color image obtained by correcting the first color image 43 to an Rx image 42rx according to the luminance distribution). ) Is used as a monitoring image.

  In this case, the Rx image 42rx is an image that can obtain an image of a high-luminance shooting target with an appropriate resolution. The first color image 43 is a color image with high color reproducibility. The third color image 44 is a color image having a dynamic range wider than that of the first color image 43, although color reproducibility and contrast are likely to be lower than those of the first color image 43.

  For this reason, by using any one of the Rx image 42rx, the first color image 43, and the second color image 44, various monitoring objects are detected with high reliability under various photographing environments. Can be.

[Modification]
Next, some modifications regarding the embodiment of the present invention will be described.

  In the embodiment, the Rx image 42rx, the first color image 43, and the second color image 44 are used as the monitoring image. However, only one of the Rx image 42rx and the second color image 44 and the first color image are used. 43 may be used as a monitoring image.

  Moreover, in the said embodiment, although the camera 2 image | photographs the front of the vehicle 1 (own vehicle), the camera which can image | photograph area | regions (for example, back) other than the front of the vehicle 1 may be sufficient.

  In the above-described embodiment, the Rx light receiving pixels as the sub light receiving pixels are arranged so as to be distributed over the entire image pickup device 22, but the Rx light receiving pixels (sub light receiving pixels) are provided only in a part of the image pickup device 22. May be arranged so as to be distributed. For example, in the entire image sensor 22, an area corresponding to A1 in FIG. 13 (partial area including an infinite point on the road surface in the imaging area of the camera 2) and an area corresponding to A2 in FIG. The arrangement of the Rx light receiving pixels may be included only in one or both of the upper imaging area).

  In the camera 2 that captures the front of the vehicle 1 (or the camera that captures the rear of the vehicle 1), a high-luminance light-emitting object (taillight or headlight of another vehicle or traffic light) is displayed in the imaging region corresponding to the above A1 and A2. In many cases, a light emitting part) is included. Therefore, even if the region where the Rx light receiving pixels (sub light receiving pixels) are arranged is limited to a part of the entire image sensor 22 as described above, the Rx image 42rx and the second color image in the region are also included. 44 can be used to detect a high-brightness monitoring object in front of (or behind) the vehicle 1.

  In this case, in the region other than the region where the Rx light receiving pixels are arranged in the entire image pickup device 22, the R light receiving pixels, the G light receiving pixels, and the B light receiving pixels are arranged in a known Bayer arrangement, for example. It may be arranged in a pattern. Then, in the processing by the color-specific image generation unit 32, four types of color-specific images 42 including the Rx image 42rx are generated only in the region where the Rx light receiving pixels are arranged, and in the other regions, a general known publicly known image is generated. An R image 42r, a G image 42g, and a B image 42b for generating the first color image 43 may be generated by the demosaicing process.

  In the above embodiment, the image sensor 22 in which the light receiving pixels are arranged in the arrangement pattern shown in FIG. 2A has been described as an example. The arrangement pattern is shown in FIG. 14A or FIG. An arrangement pattern as shown may be used.

  The arrangement pattern of FIG. 14A is a pattern in which the ratio of the number of R light receiving pixels, the number of G light receiving pixels, the number of B light receiving pixels, and the number of Rx light receiving pixels is 1: 1: 1: 1. 14B is a pattern in which the ratio of the number of R light receiving pixels, the number of G light receiving pixels, the number of B light receiving pixels, and the number of Rx light receiving pixels is 1: 4: 2: 1. is there.

  In the arrangement pattern of FIG. 14A, in the process of the color-specific image generation unit 32, in addition to the aspect of generating either the G image 42g or the Rx image 42rx as the reference luminance distribution image, the Rx image 42rx, Either the R image 42r or the B image 42b may be generated as the reference luminance distribution image.

  On the other hand, in the arrangement pattern of FIG. 14B, the number of the G light receiving pixels is larger than the number of each of the R light receiving pixels and the B light receiving pixels as in the case of FIG. Similarly to the above, either the G image 42g or the Rx image 42rx may be generated as the reference luminance distribution image.

  Supplementally, depending on the information required for detection of the object, depending on the type of the object to be detected, the detection environment, the characteristics of the light receiving pixel, etc., regardless of the arrangement pattern of the color filter, You may make it select the image according to color produced | generated as a reference | standard brightness distribution image. However, in the light receiving pixels constituting the image sensor 22, generally, when the ratio of the number of light receiving pixels for each color filter of each color is not a uniform ratio (the same number ratio), the largest number of colors It is desirable to generate a color-specific image as the reference luminance distribution image based on the gradation value corresponding to the light receiving pixel.

  In the above embodiment, the Rx light receiving pixel that receives light in the red wavelength range is provided in the image sensor 22 as the sub light receiving pixel. However, instead of the Rx light receiving pixel, a color other than red, for example, a gray color is provided. The image sensor 22 may include a light-receiving pixel that receives the light as a sub-light-receiving pixel. Specifically, the sub light receiving pixel receives light through a gray color filter and has a light receiving sensitivity lower than that of the color light receiving pixels (R light receiving pixel, G light receiving pixel, and B light receiving pixel) (hereinafter referred to as “light receiving pixels”). Gr light receiving pixel).

  For example, in an array pattern as shown in FIG. 15A, FIG. 15B, or FIG. 15C, an R light receiving pixel, a G light receiving pixel, and a B light receiving pixel, and a Gr light receiving pixel (sub light receiving pixel) You may employ | adopt the image pick-up element which arranged.

  In this case, the light receiving sensitivity of the Gr light receiving pixel has characteristics as shown in the graph of FIG. 16 with respect to the wavelength of the incident light, and is lower than the light receiving sensitivity of the R light receiving pixel, the G light receiving pixel, and the B light receiving pixel. Is done.

  When such an image sensor including a Gr light receiving pixel is used, a Gr image which is a gray-tone monotone image in addition to the R image 42r, the G image 42g, and the B image 42 from the original image by a demosaicing process. Can be generated.

  For example, the array pattern of FIG. 15C is obtained by replacing the Rx light receiving pixels in the array pattern of the embodiment shown in FIG. 2A with Gr light receiving pixels.

  For this reason, when the light receiving elements having the arrangement pattern of FIG. 15C are employed, the demosaicing process similar to that of the above embodiment (the demosaicing process using the Gr gradation value instead of the R gradation value). ) To generate a Gr image. Further, by synthesizing the Gr image, the R image 42r, the G image 42g, and the B image 42, a second color image can be generated as in the above embodiment.

  In this case, the Gr image is not limited to a red light emitting object such as a taillight of another vehicle, but is an image indicating the luminance distribution of the object at an appropriate resolution when the object to be photographed includes a high luminance object. For this reason, various high-intensity objects such as taillights and headlights of other vehicles and light emitting parts of traffic lights are detected with high reliability using the Gr image or the second color image generated using the Gr image. Can be.

  DESCRIPTION OF SYMBOLS 2 ... Camera, 22 ... Image pick-up element, 31 ... Original image acquisition part (original image acquisition means), 32 ... Color-specific image generation part (color-specific image generation means), 33 ... Monitoring image acquisition part (Monitoring image acquisition means) ).

Claims (6)

  As a camera mounted on the vehicle for photographing the periphery of the vehicle, a plurality of color light receiving pixels that respectively receive light of each type of a predetermined plurality of types of colors that are composition colors of a color image; A vehicle periphery monitoring device comprising: a camera having an image pickup device in which a plurality of sub light receiving pixels that receive light of a predetermined color with light receiving sensitivity lower than that of the color light receiving pixels are arranged. The vehicle periphery monitoring device according to claim 1,
The vehicle periphery monitoring device, wherein the predetermined color of the sub light receiving pixel is red. The vehicle periphery monitoring device according to claim 1,
The vehicle periphery monitoring device according to claim 1, wherein the predetermined color of the sub light-receiving pixel is a gray color. In the vehicle periphery monitoring device according to any one of claims 1 to 3,
The camera is mounted on the vehicle so as to image the front or the rear of the vehicle, and the sub light receiving pixels are arranged together with the color light receiving pixels in a partial area of the image sensor, The partial area includes an area corresponding to an upper area of the entire imaging area of the camera, and a portion including an infinite point on the road surface ahead of the vehicle in the entire imaging area of the camera A vehicle periphery monitoring device comprising at least one of the regions corresponding to the region. In the vehicle periphery monitoring device according to any one of claims 1 to 4,
An original image picked up by the camera, which is composed of a plurality of pixels each assigned with a gradation value indicating the intensity of an output signal output by each light receiving pixel of the image sensor in accordance with the light receiving level. An original image acquisition means for acquiring an image;
Based on the gradation value of each pixel of the original image in a processing target area that is at least a part of the original image including a plurality of color light receiving pixels and a plurality of sub light receiving pixels. A plurality of colors corresponding to each of the plurality of colors and the predetermined color by determining a gradation value of each of the plurality of colors and a gradation value of the predetermined color in each pixel of the target region Color-specific image generation means for generating another image;
Of the plurality of color-specific images, the first color image formed by combining the color-specific images corresponding to the plurality of types of colors, the color-specific image of the predetermined color, and the plurality of color-specific images A vehicle periphery comprising: a monitoring image acquisition means for acquiring at least one image of a second color image obtained by combining all the color-specific images as a monitoring image around the vehicle Monitoring device. In the vehicle periphery monitoring device according to claim 5,
The monitoring image acquisition means is for monitoring around the vehicle of the first color image, the color-specific image of the predetermined color, and the second color image according to the brightness level or contrast of the first color image. A vehicle periphery monitoring device, wherein an image acquired as an image is switched.