JP2008026998A - Obstacle position calculation system and obstacle position calculation method - Google Patents

Abstract

An obstacle position calculation system capable of calculating a position where an obstacle exists more accurately by calculating parallax including image data of a peripheral area of a target area to be subjected to parallax calculation, and an obstacle An object position calculation method is provided.
Image data captured by a plurality of far-infrared imaging devices that capture the periphery of a vehicle is acquired, the presence of an obstacle in the image is detected, and the position of the obstacle is calculated. Image data acquired from a plurality of far-infrared imaging devices is divided into target areas of a predetermined size, and an extended area composed of the divided target areas and peripheral areas existing around the target areas is set. . A correlation value is calculated between a plurality of images for each extension region, and a parallax with a region in other image data corresponding to the extension region in one image data is calculated based on the calculated correlation value. Adjacent areas having substantially the same parallax are combined and specified as an obstacle candidate area, and an actual position of a representative point of the specified obstacle candidate area is estimated.
[Selection] Figure 5

Description

  The present invention relates to an obstacle position calculation system and an obstacle position calculation method that can accurately calculate a position where an obstacle exists based on image data captured by a far-infrared imaging device that images the periphery of a vehicle. .

  A vehicle that is equipped with a far-infrared imaging device equipped with a bolometer or pyroelectric image sensor in a vehicle such as an automobile. Many monitoring systems have been developed.

  For example, when a person is imaged by a far-infrared imaging device, the person who is a heat source has a high luminance value with respect to other background objects. A conventional ambient monitoring system stores a reference pattern indicating a luminance distribution according to a human body temperature distribution in advance, and pattern matching is performed between the image data captured by the far-infrared imaging device and the stored reference pattern. It is possible to detect human presence.

For example, in Patent Document 1, when the presence of a human being is detected as the presence of an obstacle, a warning display is displayed on the head-up display by calculating the distance and direction to the obstacle and calculating the possibility of a collision. There is disclosed a periphery monitoring device provided with means for alerting a driver, such as outputting and sounding a horn.
JP 2001-018738 A

  In Patent Document 1, the parallax of a region indicating a person detected as an obstacle is obtained from image data captured by far-infrared imaging devices juxtaposed on the left and right, and the distance to the obstacle and the obstacle using the principle of triangulation The direction in which the object exists is estimated. Image data captured by a far-infrared imaging device has less color information or high-frequency components of the image spectrum than image data captured by other types of imaging devices. For this reason, the images captured by the far-infrared imaging device have a small difference between the image data to be compared when obtaining the parallax, and it is difficult to determine whether or not the regions match only with the magnitude of the correlation. There was a problem that there is a risk of becoming.

  For example, when calculating a correlation value based on image data captured by far-infrared imaging devices arranged on the left and right, it is limited to determining whether or not the same object appears in the left and right images. The correlation value could not be calculated in consideration of the positional relationship of a plurality of objects such as “a person in front of the utility pole”. Therefore, the parallax is obtained for one obstacle candidate region including a plurality of different objects, and the distance to the obstacle to be detected cannot be obtained accurately.

  In addition, when the far-infrared imaging device is mounted on a vehicle, the objects existing in both the near and near directions with a distance to the object of 30 m to 80 m are simultaneously imaged. For example, when an object with a distance of 30 m and an object with a distance of 80 m are captured in one image, it is difficult to accurately specify the object for which parallax is to be obtained, and between different objects existing at similar distances. However, there is a possibility that a high correlation value may be calculated, and as a result, an erroneous parallax may be calculated.

The present invention has been made in view of such circumstances, and by calculating the parallax including the image data of the peripheral area of the target area that is the target of the parallax calculation, the position where the obstacle is present can be determined more accurately. An object of the present invention is to provide an obstacle position calculation system and an obstacle position calculation method that can be calculated.

  In order to achieve the above object, an obstacle position calculation system according to a first invention acquires a plurality of far-infrared imaging devices that image the periphery of a vehicle and image data captured by the plurality of far-infrared imaging devices. An obstacle position calculating system that calculates the position of an obstacle based on images captured by a plurality of far-infrared imaging devices. An extended area composed of area dividing means for dividing an image acquired from the far-infrared imaging device into a target area of a predetermined size, a divided target area, and a peripheral area existing around the target area Extended area setting means for setting, correlation value calculating means for calculating a correlation value between a plurality of images for each set extended area, and other corresponding to an extended area in one image based on the calculated correlation value Painting A parallax calculating means for calculating the parallax with the inner area, an obstacle candidate area specifying means for specifying adjacent obstacle areas where the calculated parallaxes substantially coincide with each other as an obstacle candidate area, and the specified obstacle candidate And a position estimating means for estimating the actual position of the representative point of the region.

  The obstacle position calculation system according to a second invention is the obstacle position calculation system according to the first invention, wherein the region dividing means is binarized with means for binarizing image data acquired from the plurality of far-infrared imaging devices. Means for specifying a range for position calculation based on the data, and an area corresponding to the specified range in the image is divided into target areas of a predetermined size. .

  In the obstacle position calculation system according to a third aspect of the present invention, in the first or second aspect, the target area is set as a rectangular area, and the peripheral area is a direction orthogonal to each side of the rectangular area for each target area It is a rectangular area existing in.

  According to a fourth aspect of the present invention, there is provided an obstacle position calculation method that acquires image data captured by a plurality of far-infrared imaging devices that capture the periphery of a vehicle, detects the presence of an obstacle in the image, and detects the detected obstacle. In the obstacle position calculation method for calculating a position, an image acquired from the plurality of far-infrared imaging devices is divided into target areas of a predetermined size, and the divided target areas and surroundings around the target areas An extended area composed of areas is set, a correlation value is calculated between a plurality of images for each set extended area, and another corresponding to the extended area in one image is calculated based on the calculated correlation value Calculates the parallax with the area in the image, combines adjacent areas where the calculated parallax is approximately the same, identifies it as an obstacle candidate area, and estimates the actual position of the representative point of the identified obstacle candidate area It is characterized by doing.

In the first invention and the fourth invention, image data captured by a plurality of far-infrared imaging devices that capture the periphery of the vehicle is acquired to detect the presence of an obstacle in the image, and the position of the detected obstacle is determined. calculate. An image acquired from a plurality of far-infrared imaging devices is divided into target areas of a predetermined size, and an extended area composed of the divided target areas and peripheral areas existing around the target areas is set. A correlation value is calculated between a plurality of images for each set extension region, and a parallax with a region in another image corresponding to the extension region in one image is calculated based on the calculated correlation value. Adjacent areas where the calculated parallax substantially coincides are combined and specified as an obstacle candidate area, and an actual position of a representative point of the specified obstacle candidate area is estimated. The target area for which the correlation value is calculated is expanded, and the correlation value is calculated for the expanded extended area, so that information around the obstacle, for example, a part of the same object as the target object is imaged. The correlation value can be calculated in consideration of a state in which an obstacle existing before the object is imaged, an obstacle existing behind the object is imaged, and the like. Therefore, even if there is an area where the luminance change is relatively small, or an area similar to the area corresponding to the object is present in the object range, the corresponding point, block, area for the object to be detected as an obstacle Etc. can be specified with high accuracy, and the parallax between the left and right images can be calculated with higher accuracy.

  In the second aspect of the invention, the image data acquired from a plurality of far-infrared imaging devices is binarized to specify a range that is a position calculation target, and an area corresponding to the specified range in the image is set to a predetermined size. The target area is divided. As a result, for example, the actual distance can be estimated for the representative points of the high luminance area obtained by the binarization process, and there are areas where the luminance change is relatively small and areas similar to the area corresponding to the object. Even when the object is within the target range, the corresponding points, blocks, areas, etc. can be specified with high accuracy for the object to be detected as an obstacle, and the parallax between the left and right images can be calculated with higher accuracy. Is possible.

  In the third invention, the target area is set as a rectangular area, and the peripheral area is a rectangular area that exists in a direction orthogonal to each side of the rectangular area for each target area. Thus, when detecting an object that is highly likely to have a luminance change dependency in the vertical direction or the horizontal direction of the image, for example, a pedestrian, the target is considered in consideration of surrounding luminance values in the upper, lower, left, and right peripheral regions. The correlation value of the region can be calculated, and the parallax can be accurately calculated from the left and right images.

  According to the present invention, the target area for calculating the correlation value is expanded, and the correlation value is calculated for the expanded extended area, so that a part of the information around the obstacle, for example, the same object as the target object is imaged. The correlation value can be calculated in consideration of a state where an obstacle existing in front of the object is imaged, an obstacle existing behind the object is imaged, and the like. Therefore, even if there is an area where the luminance change is relatively small, or an area similar to the area corresponding to the object is present in the object range, the corresponding point, block, area for the object to be detected as an obstacle Etc. can be specified with high accuracy, and the parallax between the left and right images can be calculated with higher accuracy.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following embodiment, an example will be described in which the distance to an obstacle existing in front of the vehicle is calculated based on an image captured by the far-infrared imaging device during night driving.

  FIG. 1 is a schematic diagram showing a configuration of an obstacle position calculation system according to an embodiment of the present invention. In the present embodiment, far-infrared imaging devices 1 and 1 that capture surrounding images during night driving are juxtaposed in a front grill near the center in front of the vehicle. The far-infrared imaging devices 1 and 1 are imaging devices using infrared light having a wavelength of 7 to 14 micrometers. Image data captured by the far-infrared imaging devices 1 and 1 is transmitted to a detection device 3 connected via a video cable 7 corresponding to an analog video system such as NTSC or a digital video system.

  In addition to the far-infrared imaging devices 1 and 1, the detection device 3 is connected to a display device 4 having an operation unit via a cable 8 corresponding to a video system such as NTSC, VGA, DVI, etc. An output device such as an alarm device 5 that emits an audible warning by a sound effect or the like is connected via an in-vehicle LAN cable 6 compliant with CAN.

FIG. 2 is a block diagram showing a configuration of the far-infrared imaging device 1 of the obstacle position calculation system according to the embodiment of the present invention. The image pickup unit 11 includes image pickup elements that convert optical signals into electric signals in a matrix. As an image sensor for infrared light, an infrared sensor such as a vanadium oxide bolometer type using a micromachining technique or a pyroelectric type BST (Barium-Strontium-Titanium) is used. The image capturing unit 11 reads an infrared light image around the vehicle as a luminance signal, and transmits the read luminance signal to the signal processing unit 12.

  The signal processing unit 12 is an LSI, converts the luminance signal received from the image capturing unit 11 into a digital signal, performs processing for correcting variations in the image sensor, correction processing for defective elements, gain control processing, and the like, and performs image data processing. Is stored in the image memory 13. Needless to say, it is not essential to temporarily store the image data in the image memory 13, and the image data may be transmitted directly to the detection device 3 via the video output unit 14.

  The video output unit 14 is an LSI, and outputs video data to the detection device 3 via a video cable 7 compatible with an analog video system such as NTSC or a digital video system.

  FIG. 3 is a block diagram showing a configuration of the detection device 3 of the obstacle position calculation system according to the embodiment of the present invention. The video input unit 31 a inputs video signals from the far infrared imaging devices 1 and 1. The video input unit 31a stores the image data input from the far-infrared imaging devices 1 and 1 in the image memory 32 in synchronization with each frame. The video output unit 31 b outputs image data to the display device 4 such as a liquid crystal display via the video cable 8, and the communication interface unit 31 c outputs an alarm device 5 such as a buzzer and a speaker via the in-vehicle LAN cable 6. An output signal such as a synthesized sound is transmitted.

  The image memory 32 is an SRAM, flash memory, SDRAM or the like, and stores image data input from the far-infrared imaging devices 1 and 1 via the video input unit 31a.

  The LSI 33 that performs image processing reads the image data stored in the image memory 32 in units of frames, and the image data acquired from the far-infrared imaging devices 1 and 1 is applied to a target area of a predetermined size, for example, 3 × 3 pixels. To divide. The LSI 33 sets an extension area obtained by extending the divided target area to the peripheral area. The LSI 33 calculates the parallax of the left and right image data acquired from the far-infrared imaging devices 1 and 1 for the extended region, and calculates the distance to the object based on the principle of triangulation. The RAM 331 stores data generated during the arithmetic processing and the calculated time-series position data of the obstacle.

  Detailed processing in the LSI 33 will be described below. FIG. 4 is a flowchart showing the procedure of the obstacle position calculation process of the LSI 33 of the detection device 3 of the obstacle position calculation system according to the embodiment of the present invention.

  The LSI 33 reads the image data stored in the image memory 32 (step S401), and divides the read image data into target areas of a predetermined size (step S402). The method for setting the target area to be divided is not particularly limited. For example, you may set as a square area | region which consists of m * n (m and n are natural numbers) pixels. In the present embodiment, the target area is set as a square area composed of 3 × 3 pixels.

  The LSI 33 sets an extension area composed of the divided target area and a peripheral area existing around the target area (step S403). The direction and size for setting the extension region differ depending on the object to be detected. For example, when the object to be detected is a pedestrian, the high luminance area by the binarization process is an area centered on the pedestrian's head. Therefore, it is easier to detect the luminance difference from other objects such as the background and road surface, and the parallax can be calculated more accurately by extending the target area in the vertical and horizontal directions than simply expanding it in the surrounding direction. It becomes possible to do.

For example, when the target area is a square area composed of 3 × 3 pixels, it is preferable to extend the target area in each side direction of the square area. FIG. 5 is a diagram illustrating an example of setting an extension area when the target area is a square area.

  In the conventional parallax calculation, correlation values of images captured by the left and right far-infrared imaging devices 1 and 1 are calculated on the basis of a target area 51 composed of 3 × 3 pixels, and corresponding points, blocks, areas, and the like are calculated. I was exploring. FIG. 6 is a diagram illustrating a variation example of a region detected as an obstacle candidate region due to variation in the target region 51. As shown in FIG. 6A, when the target area 51 is relatively narrow, there are few areas in which high correlation is detected between the target areas, and it is difficult to specify the obstacle candidate area accurately. is there.

  Therefore, the target area 51 is expanded to an extended area including the peripheral area 52 as shown in FIG. 5B, for example, an area composed of 5 × 5 pixels, and the left and right far-infrared imaging devices 1 on the basis of the extended area, The correlation value of the image imaged in 1 is calculated, and the corresponding point, block, region, etc. are searched. In this case, as shown in FIG. 6B, there are a relatively large number of areas in which high correlation is detected between the extended areas, and it becomes easier to specify an obstacle candidate area. Therefore, the reliability of the correlation value calculated between the images picked up by the left and right far-infrared imaging devices 1 and 1 becomes high, and it becomes easier to search for corresponding points, blocks, regions, etc. between the two images. By obtaining the parallax reliably, it is possible to accurately estimate the whole image of the obstacle to be detected.

  In addition, it is more preferable to specify the extension direction according to the characteristics of the target object to be detected, instead of simply expanding the target area 51 to an area composed of 5 × 5 pixels. For example, when the object to be detected is a pedestrian, there is a strong dependence on the background image only in the vertical and horizontal directions of the image. In this case, as shown in FIG. 5C, the partial extension regions 53, 53, 53, in which the target region 51 is expanded only in the outer direction of each side of the rectangular region, that is, in the vertical direction and the horizontal direction of the target region 51, The correlation value of the images imaged by the left and right far-infrared imaging devices 1 and 1 is calculated on the basis of the extended area including 53, and the corresponding points, blocks, areas, etc. are searched.

  FIG. 6C shows the detection state of the obstacle candidate area when the extension area including the partial extension areas 53, 53, 53, 53 extended only in the vertical direction and the horizontal direction of the target area 51 is used as a reference. Yes. Considering the dependency relationship with the surrounding background image and the like, it is easier to specify the obstacle candidate region and the far-infrared imaging on the left and right sides as compared with FIG. 6B when the target region 51 is simply expanded. The reliability of the correlation value calculated between the images captured by the devices 1 and 1 becomes high, and it becomes easier to search for corresponding points, blocks, regions, and the like between both images, and parallax is reduced in many regions. By obtaining with certainty, it is possible to accurately estimate the entire image of the obstacle to be detected.

  The LSI 33 calculates a correlation value in units of images for each set extension area (step S404). For example, with the extended region set based on the image captured by the left far-infrared imaging device 1 as a reference, the same shape and the same size as the expanded region in the image captured by the right far-infrared imaging device 1 A correlation value with the region group is calculated. The region group having the same shape and the same size as the extended region means a region group in the case of parallax = 1, 2,..., N (n is a natural number), for example.

  When the LSI 33 calculates the correlation value R in image units for each extended region, the correlation value R is calculated based on (Equation 1).

  In (Expression 1), N is the total number of pixels in the region to be matched, k is an integer of 0 ≦ k ≦ (N−1), and Fk is based on an image captured by the left far-infrared imaging device 1. Gk is the pixel value of the kth pixel in the extension region set in the above, Gk is the k in the region group having the same shape and the same size as the extension region in the image captured by the right far-infrared imaging device 1 The pixel values of the second pixel are shown.

  The LSI 33 extracts the parallax having the maximum correlation value calculated for each parallax (step S405). That is, there is a high possibility that an obstacle exists with the parallax having the maximum calculated correlation value.

  The LSI 33 identifies adjacent candidate areas where the extracted parallaxes are approximately the same as an obstacle candidate area (step S406). The LSI 33 estimates the actual position of the representative point of the specified obstacle candidate area (step S407). For example, the center of gravity of the obstacle candidate area is set as the representative point.

  Specifically, the LSI 33 determines the distance between the vehicle front center point and the detected obstacle based on the parallax of the region corresponding to the image captured by the left and right far-infrared imaging devices 1, 1. And an angle from the vehicle front center point indicating the direction of the obstacle. The angle indicating the direction is calculated as a positive angle on the left side and a negative angle on the right side with respect to the traveling direction of the vehicle.

  The LSI 33 performs a collision determination process according to a known algorithm based on the calculated distance to the obstacle and the direction in which the obstacle exists, and detects the presence of the obstacle when there is a high possibility of a collision. A signal indicating the fact is displayed to the display device 4 or the alarm device 5. The display device 4 outputs a warning display such as outputting a display indicating that the presence of an obstacle such as a pedestrian has been detected or blinking the warning display. The alarm device 5 notifies the driver of the presence of an obstacle such as a pedestrian by sounding a buzzer.

  When the object to be detected as an obstacle is not a pedestrian, the extension area is not limited to extending in the upper limit direction and the left-right direction. For example, when the above-described system is applied to the detection of a white line on a road, the white line on the road surface extends in an oblique direction. It becomes possible to detect. As described above, since the dependency direction with the background image differs depending on the target object to be detected, the correlation value can be calculated with higher accuracy by setting the extended region according to the target object. The position of the object can be estimated.

As described above, according to the present embodiment, the target area for calculating the correlation value is expanded, and the correlation value is calculated for the expanded extended area. The correlation value is calculated taking into account the state where a part of the object is imaged, an obstacle existing in front of the object is imaged, and an obstacle existing behind the object is imaged. can do. Therefore, even if there is an area where the luminance change is relatively small, or an area similar to the area corresponding to the object is present in the object range, the corresponding point, block, area for the object to be detected as an obstacle Etc. can be specified with high accuracy, and the distance to the obstacle can be accurately estimated by calculating the parallax between the left and right images with high accuracy.

  In addition, although the case where the target area is a square area has been described as an example, the target area is not limited to the square area, and the shape is not particularly limited as long as there is a possibility that an obstacle exists. Further, the present invention is not limited to dividing all captured images into target regions, and for example, region division may be performed only on a high-luminance region after binarization processing is executed.

  In the above-described embodiment, the LSI 33 of the detection device 3 performs the above-described control. However, a separate control device may be provided, or a control device of another device may also be used.

It is a mimetic diagram showing composition of an obstacle position calculation system concerning an embodiment of the invention. It is a block diagram which shows the structure of the far-infrared imaging device of the obstacle position calculation system which concerns on embodiment of this invention. It is a block diagram which shows the structure of the detection apparatus of the obstacle position calculation system which concerns on embodiment of this invention. It is a flowchart which shows the procedure of the obstacle position calculation process of LSI of the detection apparatus of the obstacle position calculation system which concerns on embodiment of this invention. It is a figure which shows the example of a setting of an extended area | region in case an object area | region is a square area | region. It is a figure which shows the example of a fluctuation | variation of the area | region detected as an obstacle candidate area | region by the fluctuation | variation of an object area | region.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Far-infrared imaging device 3 Detection apparatus 4 Display apparatus 5 Alarm apparatus 31a Image | video input part 31b Image | video output part 31c Image | video output part 32 Image memory 33 LSI
331 RAM

Claims (4)

A plurality of far-infrared imaging devices for imaging the periphery of the vehicle;
A detection device that acquires image data captured by the plurality of far-infrared imaging devices and detects the presence of an obstacle in the image, and the position of the obstacle based on the images captured by the plurality of far-infrared imaging devices In the obstacle position calculation system for calculating
The detection device includes:
Area dividing means for dividing the images acquired from the plurality of far-infrared imaging devices into target areas of a predetermined size;
An extension area setting means for setting an extension area composed of a divided target area and a peripheral area existing around the target area;
Correlation value calculating means for calculating a correlation value between a plurality of images for each set extension region;
Disparity calculating means for calculating a disparity with a region in another image corresponding to the extended region in one image based on the calculated correlation value;
An obstacle candidate area specifying unit that combines adjacent areas where the calculated parallax substantially coincides and specifies the obstacle candidate area;
An obstacle position calculation system comprising: position estimation means for estimating an actual position of a representative point of the identified obstacle candidate area. The region dividing means includes
Means for binarizing image data acquired from the plurality of far-infrared imaging devices;
Means for specifying a range for position calculation based on the binarized data,
2. The obstacle position calculation system according to claim 1, wherein an area corresponding to the specified range in the image is divided into target areas of a predetermined size. The target area is set as a rectangular area,
The obstacle position calculation system according to claim 1, wherein the peripheral area is a rectangular area that exists in a direction orthogonal to each side of the rectangular area for each target area. In the obstacle position calculation method for acquiring image data captured by a plurality of far-infrared imaging devices that image the periphery of a vehicle, detecting the presence of an obstacle in the image, and calculating the position of the detected obstacle,
Dividing images acquired from the plurality of far-infrared imaging devices into target areas of a predetermined size,
Set an extended area consisting of the divided target area and the peripheral area around the target area,
Calculate the correlation value between multiple images for each set extension area,
Based on the calculated correlation value, a parallax with an area in another image corresponding to an extended area in one image is calculated,
Combining adjacent areas where the calculated parallax substantially coincides and specifying as an obstacle candidate area,
An obstacle position calculation method characterized by estimating an actual position of a representative point of an identified obstacle candidate area.

Images