JP2001273488A - Obstacle detection device - Google Patents

Abstract

(57) [Summary] [Problem] This is a technology for supporting safe driving of a car and realizing automatic driving, and a obstacle existing on a road surface such as a preceding car or a pedestrian using a vehicle-mounted stereo camera. Provide an obstacle detection device that detects an object at high speed and with high accuracy. SOLUTION: An image input unit 1 having a plurality of television cameras for inputting images, an image storage unit 2 for storing a plurality of images input by the plurality of cameras, and an image storage unit for storing images taken by an arbitrary camera. An image conversion unit 3 that converts the image into an image viewed from the viewpoint of the camera, and a second predetermined area corresponding to a first predetermined area in the converted image and a first predetermined area in an image captured by another camera. In performing the matching process with the area,
Making the first predetermined area smaller than the second predetermined area,
A matching processing unit 4 for extracting a region having the lowest matching degree with the first predetermined region among the predetermined regions, and an obstacle detecting unit 5 having an area having the minimum matching degree of each first predetermined region as an obstacle. Is done.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an obstacle detecting device, and more particularly to an obstacle detecting device for detecting an obstacle existing on a road such as a preceding vehicle, a parked vehicle, or a pedestrian while a host vehicle is running. About.

[0002]

2. Description of the Related Art It is important to detect an obstacle existing in the traveling direction of a vehicle in order to realize safe driving assistance and automatic traveling of a vehicle.

Conventionally, techniques for detecting obstacles include:
Some use lasers or ultrasonic waves, and others use television cameras. The apparatus using a laser requires an expensive apparatus, and the apparatus using an ultrasonic wave has a low resolution of the ultrasonic wave. Therefore, there is a possibility that a problem may occur in the detection accuracy of an obstacle, and the practicality is low. In addition, the active lane using the laser or the ultrasonic wave alone cannot recognize the traveling lane.

On the other hand, a television camera is relatively inexpensive and is suitable for obstacle detection in terms of resolution, measurement accuracy, and measurement range. It is also possible to recognize the traveling lane. When a television camera is used, there is a method using one camera and a method using a plurality of cameras (stereo cameras).

In a method using one camera, a road area and an obstacle area are separated from one image picked up by the camera by using information such as luminance, color, and texture.

For example, a medium luminance region having low saturation, that is, a gray region in a captured image, that is, a gray region is extracted and used as a road region. In some cases, the area is extracted and the other area is set as an obstacle area. However, since there are many obstacles having the same brightness, color, or texture as roads, it is difficult to separate the obstacle region from the road region by this method.

On the other hand, the method using a plurality of cameras detects an obstacle using three-dimensional information as a clue. This method is generally called stereo vision. For example, two cameras are arranged on the left and right, a point that is the same point in a three-dimensional space is associated between left and right images, and the three points of the point are determined in a triangulation manner. This is for obtaining a dimensional position. If the position, posture, and the like of each camera with respect to the road plane are obtained in advance, the height of an arbitrary point in the image from the road plane can be obtained by stereo vision.

With such a stereoscopic technique, it is possible to determine the presence or absence of a height and to separate an obstacle area from a road area.

However, ordinary stereo vision has a problem of corresponding point search. Stereo vision generally refers to a coordinate system fixed to a stereo camera at an arbitrary point on an image (hereinafter referred to as a stereo camera coordinate system).
This is a technique for determining the dimensional position. Corresponding point search means a search calculation required for associating the same point in space between left and right images, and has a problem that the calculation cost is extremely high. This search for corresponding points is a factor hindering the practical use of stereo vision.

However, if it is only necessary to separate the road area and the obstacle area on the image, the corresponding point search is not necessary, and the presence or absence of the height from the road plane can be determined as follows, for example. .

If the projection points of points on the road plane to the left and right images are (u, v) and (u ', v'),

(Equation 1) Holds.

[Outside 1] These parameters are dependent on the position and orientation with respect to the road plane, the focal length of each camera lens, and the image origin.

[Outside 2] If the point P exists on the road plane, the points P and P 'form a correct set of corresponding points, and the two points have the same luminance. Therefore, when the luminances of the points P and P ′ are different, it can be determined that the point P belongs to the obstacle area. According to this method, it is possible to directly determine the presence or absence of an arbitrary point on the image from the road surface from only the equation (1), and the coefficient of the equation (1) is four or more feature points on the road. Can be obtained only from the projection points on the left and right images, and a corresponding point search between the left and right images is unnecessary.

[Outside 3] However, when the vehicle travels outdoors, the relationship between the road plane and the relative position and attitude of each camera changes from moment to moment due to the vibration of the vehicle itself and changes in the inclination of the road.

[Outside 4] However, when an obstacle is detected based on a mere difference from another camera image, there is a problem that the accuracy of detecting the obstacle is significantly reduced.

[0012]

As described above, in the conventional obstacle detection device using a television camera, the use environment is limited, a corresponding search requiring a high calculation cost is required, or the vehicle is running. Since it cannot cope with the vibration of the road and the inclination of the road, there is a problem that the performance is significantly deteriorated in an outdoor environment.

In view of the above, the present invention has been made in view of the above-mentioned conventional problems, and suppresses the influence of vibration and the inclination of the road itself and stably suppresses the vehicle on any road. An object of the present invention is to provide an obstacle detection device that detects an obstacle.

[0014]

In order to achieve the above object, in the obstacle detecting device according to the present invention, a light receiving section of the vehicle is separated from a light receiving section separated from the own vehicle so that the optical axes thereof are parallel to each other. A plurality of image capturing means for simultaneously capturing an area in the traveling direction as an image; an image storage unit for storing an image captured by the image capturing means; and a first image storage unit stored in the image storage unit.
Converting an area in the first image captured by the imaging means to a predetermined area on an epipolar line corresponding to the area in the second image captured by the second imaging means;
An image conversion unit that obtains the converted image as a converted image, a first predetermined area in the converted image converted from one of the first image and the second image, and a second predetermined area in the other image. When performing the matching with the predetermined region, the second predetermined region is made larger than the first predetermined region, and the first predetermined region moves within the second predetermined region and the first predetermined region moves. The position where the degree of coincidence between the predetermined region and the second predetermined region is maximum is defined as a matching region, and the matching region is detected for each of the first regions to obtain a matching image including a plurality of matching regions. The image processing apparatus includes a processing unit, and an obstacle detection unit that sets an area lower than a reference value in the matching image obtained by the matching processing unit as an obstacle.

Here, the operation of the image conversion unit will be described with reference to FIGS.

The operation of the image conversion section is to convert the first image picked up by the first image pickup means into an image which can be seen from the viewpoint of the second image pickup means, and to obtain this converted image. is there.

The parameters used for this conversion are:
The camera serving as a plurality of imaging means and the road plane on which the own vehicle travels have a typical geometric relationship (for example, when a stationary vehicle is placed on an unsloped road surface) only once in advance. It is obtained, and is not changed while the vehicle is running, that is, when the operation of obstacle detection is being performed.

First, as shown in a perspective view for explaining the operation of FIG. 1, a vehicle on which two cameras 10a and 10b are mounted is arranged on a flat road surface without inclination. On the road surface, there are white lines drawn in parallel to the traveling direction of the vehicle, and these white lines are denoted by l and l '.

It is assumed that the relationship between the positions and postures of these two cameras 10a and 10b is unknown to the obstacle detection device, and only the epipolar constraint is known. , Posture, and epipolar constraint do not change.

The epipolar constraint is a constraint that holds for a general stereo image. As shown in the explanatory diagram of the epipolar constraint in FIG. 2, an epipolar constraint on an image (left image) captured by the camera 10a is used. The arbitrary point P refers to a state in which the point P is constrained to be on a predetermined straight line including the corresponding point P ′ on the image (right image) captured by the camera 10b. This straight line is called an epipolar line.

For example, when the optical axes of the cameras are arranged so as to be parallel to each other, the point corresponding to an arbitrary point P in the left image exists on the same scanning line on the right screen, so that the epipolar line and the scanning Will match the line. Since the epipolar constraint depends on the relationship between the relative position and orientation between the stereo cameras and the internal parameters of each camera, that is, the focal length of the camera lens and the image origin, it is stereo that the epipolar constraint does not change and remains unchanged. This means that the relative positional relationship of the cameras and the internal parameters do not change while the vehicle is running.

This epipolar constraint is formulated as shown in the following equation (2).

(Equation 2)

[Outside 5] F is a 3 × 3 matrix and is called a fundamental matrix. When formula (2) is developed and arranged, it can be expressed as (3) shown below.

(Equation 3)

[Outside 6] The matrix F consists of 9 elements, but each element is not independent,
Theoretically, each element can be obtained from a set of seven or more corresponding points. Since the three-dimensional position of each set of corresponding points is unnecessary, the calculation of the matrix F, that is, the epipolar high speed, is relatively easy.

Although l and l 'in each image are parallel to each other in the three-dimensional space, on the images picked up by the left and right cameras, as shown in the white line area picked up by each camera in FIG. , l 'intersect at a point at infinity called a vanishing point on the screen.

Next, a relational expression that is established between corresponding points on the road surface will be obtained. As shown in the diagram for explaining the corresponding points in FIG. 4, arbitrary two points on the straight line 1 in the left image are A and C.
And two arbitrary points on the straight line l ′ are B and D.

The corresponding points on the right image of these four points are A ',
B ′, C ′, and D ′ can be easily calculated by using the previously obtained epipolar constraint. That is,
The corresponding point A ′ of the point A coincides with the intersection of the straight line 1 and the epipolar line LA of the point A on the right image. Similarly, the points B ′, C ′, and D ′ can be obtained as intersections between the points B, C, and D and the epipolar lines LB, LC, and LD.

[Outside 7]

[Outside 8]

(Equation 4) The following relational expression holds.

[Outside 9]

(Equation 5) The conversion thus obtained is, for example, as shown in the converted image example of FIG. 5, in the case of a stereo image, when the left camera image (a) is converted to the right camera viewpoint, an image as shown in (c) is obtained. . In other words, pixels on the road surface, in the figure, at the point of contact between the tire of the vehicle and the road surface can be correctly converted to corresponding points, whereas objects with a spatial height are displayed in the image. It is transformed with falling distortion.

[Outside 10] Since a set of corresponding points is formed, the brightness of the points P and P 'is basically the same. Conversely, if the brightness is different, the points P and P
'Is a point not on the road surface.

If the relationship between the road surface and the camera is constant,

(Equation 6) (However, || indicates an absolute value.) In consideration of D ≠ 0 or an error such as a difference in characteristics between the right and left cameras,
By setting the threshold value Thr, it is possible to determine that the point P satisfying D> Thr belongs to the obstacle area.

However, in practice, various changes occur, such as camera vibration and road surface inclination accompanying the movement of the vehicle.
It is difficult to determine an obstacle by the above equation (6). For example, since the luminance difference between a landmark ("stop", speed limit display, white line, etc.) and the road surface is large, the assumed geometrical relationship between the road surface and the camera (the image conversion parameters described above were obtained). Due to the difference between the camera and the road surface at the time) and the geometric relationship between the actual road surface and the camera, Equation (6) gives a large value around the landmark (= around the edge) even though it is not an obstacle. Because we have.

[0028]

Embodiments of the present invention will be described below with reference to the drawings.

FIGS. 6 and 7 show an obstacle detecting device according to the present invention.

FIG. 6 is a block diagram of an obstacle detection device according to the present invention. The obstacle detection device is provided in a host vehicle on which a user rides, and their optical axes are substantially parallel to each other. Input unit 1 composed of two stereo cameras in which the light receiving units as lenses are spaced apart from each other as described above.
(Imaging means), an image accumulating section 2 for storing an image captured by the image input section 1, an image converting section 3 for performing processing (to be described in detail later) on the captured image, and a plurality of processed images. A matching processing unit 4 for performing a matching process between images;
An obstacle detection unit 5 detects an obstacle based on the result of the matching processing from the matching processing unit 4.

It should be noted that obstacles are detected under the condition that vibrations that may occur when the vehicle travels and changes in the slope of the road surface are present. It is assumed that an obstacle that can exist in the vehicle is detected. It is assumed that the geometric relationship between the two stereo cameras and the configuration when the stereo cameras are mounted on the own vehicle do not change or change from the above-described calculation of the image conversion parameters. Further, an image captured by the image input unit 1 disposed on the left is a first camera left camera image, and an image captured by the image input unit 1 disposed on the right is a second camera right camera image. And

The operation of the obstacle detecting device having such a configuration will be described.

First, an area in the traveling direction of the host vehicle is simultaneously imaged as two images using two television cameras.

Next, the image storage unit 2 stores the two images input by the image input unit 1 in an image memory.

Next, the image conversion unit 3 reads the left camera image as the first image stored in the image storage unit 2 and uses one of the image input parameters according to the conversion parameters obtained in advance by the method described above. The image captured by the unit 1 (for example, the left stereo camera) is converted to the viewpoint of the other image input unit 1 (for example, the right stereo camera), and the converted image is obtained as a converted image. Here, an arbitrary region (for example, one pixel) in the left camera image is obtained by converting into a predetermined region (one pixel) on the epipolar line corresponding to this region in the right camera image as the second image. It is. In the following, a case in which the left camera image is converted to the right camera viewpoint will be described. However, the same processing is essentially performed even when the right camera image is converted to the left camera viewpoint.

Next, in the matching processing section 4, as shown in the explanatory diagram of the matching processing shown in FIG.
And a second predetermined region corresponding to the first predetermined region of the right camera image. A position in the second area corresponding to the first predetermined area obtained by the matching processing is set as a matching area, and a matching image including a plurality of matching areas is obtained. Here, the first predetermined area and the second predetermined area are areas including at least one pixel, and the size of the first predetermined area is smaller than the size of the second predetermined area. . On the converted image of FIG. 3A, the road surface is almost correctly projected, while a tall object such as a vehicle or a pedestrian in front of the own vehicle becomes larger as the height from the road surface increases. Since the projection is performed with distortion, a high degree of coincidence is obtained at a position on the road surface (a position where there is almost no height), and an object with a height has low degree of coincidence.

The matching process is to calculate the similarity between the second predetermined region and the first predetermined region at each position where the first predetermined region is moved and moved in the second predetermined region. , The maximum degree of coincidence of the obtained similarities
Is determined as the first predetermined area in the predetermined area. When calculating the degree of coincidence, the normalized correlation or SA
D (Sum of Absolute Differe)
nce), SSD (Sumof Square Dif)
For example, a matching method used in image processing such as a reference is used. However, the normalized correlation has a larger value as the degree of coincidence is higher, whereas the SAD and SSD have a higher degree of coincidence as the value is smaller, and can be appropriately used.

The first predetermined area moving in the second predetermined area is the first area before the movement and the first area after the similarity calculation is completed and moved to a new place. That is, even if the region moves so that a part of the region overlaps, the region can move so as to be adjacent to the region without overlapping.

When the matching processing between the second predetermined area and the first predetermined area is completed, a first predetermined area different from the first predetermined area for which the matching processing has been completed, and this new first predetermined area Is performed with the second predetermined region corresponding to the first camera region, and the matching process is performed on all the first predetermined regions existing in the left camera image. The second predetermined regions corresponding to the first predetermined regions may be either overlapping or adjacent to each other. Further, it is possible to set the entirety of the converted image as the first predetermined area, or to set only the position where the first predetermined area is obtained as the first predetermined area.

When the matching processing for all the first predetermined areas is completed, a two-dimensional image (see FIG. 10C) made of the first predetermined area and the one having the highest degree of coincidence is generated.

Next, the obstacle detection unit 5 performs threshold processing on the two-dimensional image obtained and generated by the matching processing unit 4 so that an area having a degree of coincidence lower than the reference value on this image is determined as an obstacle. Output as object area. In addition, the reference value includes
The operator who operates the own vehicle can use the value obtained empirically or can divide the coincidence into a plurality of, for example, nine levels and set the coincidence from the one with the lowest coincidence to the fourth or lower region, for example. It is also possible to use an arbitrary degree of coincidence as a reference value from a two-dimensional image, and to use a road surface that occupies the largest area in a two-dimensional image, because the road surface is the part other than the area with the highest degree of coincidence. Can be set as the reference value.

The matching processing section 4 obtains the degree of coincidence for each first predetermined area as a two-dimensional image of the degree of coincidence.
A threshold (for example, the above-described reference value) may be provided in advance, and only those lower than this threshold may be output as a two-dimensional image.

Since the obstacle detection unit 5 detects an area lower than the reference value as an obstacle, after the degree of coincidence of the second predetermined area with the first predetermined area is calculated,
If the degree of coincidence is equal to or less than an arbitrary threshold value (for example, the above-described reference value), the degree of coincidence is set as the degree of coincidence with the first predetermined area, and the subsequent degree of coincidence calculation in the first predetermined area is not performed. The processing can be performed as follows.

The obstacle information detected by the obstacle detection unit 5 is transmitted to the user by auditory means such as voice, visual means such as light (including images), and sensible means such as vibration. Can be provided as appropriate.

In the above-described obstacle detecting device, no matter what road the host vehicle is traveling on, it is not affected by the fluctuation of brightness or the shadow of the preceding vehicle, etc. And the obstacle can be detected stably. Further, by alerting the user of the existence of the obstacle, it is possible to avoid an event that the user may want to avoid due to the presence of the obstacle.

Further, by performing a calculation with a threshold value when performing the matching process, the calculation time required for the matching process can be reduced, and high-speed obstacle detection can be performed. this is,
For example, if an error occurs between a parameter for obtaining a converted image and an optimal image conversion parameter derived from the relationship between the actual camera and the road surface, and even if an error occurs in the conversion of the left camera image, In an area without a texture (pattern) above, there is often no texture even in the second predetermined area of the corresponding right camera image, so even if such an area is searched for the entire second predetermined area, This is because there is no significant difference from the result obtained by searching all the second predetermined areas, and even if the search processing is terminated, the obstacle detection performance does not deteriorate.

The present invention is not limited to the above embodiment, and it goes without saying that various modifications can be made without departing from the spirit of the present invention. For example, although the description has been made assuming that the road surface is a plane, even when the road surface is assumed to be a curved surface, for example, the curved surface is approximated to a plurality of piecewise planes, and a plurality of image conversion parameters are prepared, By performing the above-described piecewise image conversion, an obstacle can be detected.

Further, it is possible to easily realize hardware and parallelization of each component such as a matching processing unit, and to prepare SIMD (Single) which is prepared for speeding up multimedia data processing by a recent processor.
Instruction stream Multip
It is very easy to increase the speed by using the “le Date stream” calculation function.

The size of the first or second predetermined area is not constant, but can be changed as long as the degree of coincidence can be calculated.

[0050]

As described above, according to the present invention, a stable obstacle can be detected without being affected by the vibration applied to the image pickup means or the inclination of the road surface on which the vehicle runs.

[Brief description of the drawings]

FIG. 1 is a view for explaining a positional relationship between an image pickup means of an obstacle detection device of the present invention and a road surface.

FIG. 2 is an explanatory view of an epipolar constraint of the obstacle detection device of the present invention.

FIG. 3 is an explanatory diagram of epipolar constraint of the obstacle detection device of the present invention.

FIG. 4 is an explanatory diagram of epipolar constraint of the obstacle detection device of the present invention.

FIG. 5 is an explanatory diagram of an operation of an image metamorphosis section of the obstacle detection device of the present invention.

FIG. 6 is a block diagram of an obstacle detection device according to the present invention.

FIG. 7 is an explanatory diagram of an operation of a matching processing unit of the obstacle detection device of the present invention.

[Explanation of symbols]

 Reference Signs List 1 image input unit 2 image storage unit 3 image conversion unit 4 matching processing unit 5 obstacle detection unit

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification code FI Theme coat ゛ (Reference) B60R 21/00 626 B60R 21/00 626G G01B 11/00 G01B 11/00 H G01C 3/06 G01C 3/06 V G06T 7/00 300 G06T 7/00 300C G08G 1/16 G08G 1/16 C (72) Inventor Kazunori Onoguchi 1-1-30 Oyodonaka, Kita-ku, Osaka City, Osaka Toshiba Kansai Branch Office F-term ( (Reference) 2F065 AA00 AA02 AA06 CC11 FF05 JJ05 QQ00 QQ08 QQ23 QQ36 QQ38 2F112 AC04 AC06 BA01 CA00 CA05 CA12 FA03 FA09 FA21 FA36 FA38 5B057 AA16 BA02 BA11 CA13 CA16 CB13 CB16 DA11 DB03 DC09 504 A04 CC04 CC04 JA26

Claims (2)

[Claims] A plurality of image pickup means for simultaneously picking up an image of a region in the traveling direction of the vehicle as an image from a light receiving section which is separated from the own vehicle so that the optical axes thereof are substantially parallel to each other; An image storage unit for storing the captured image; and a second image captured by the second imaging unit, the area in the first image captured by the first imaging unit stored in the image storage unit. An image conversion unit that converts the image into a predetermined region on an epipolar line corresponding to the region in the image, and obtains the converted image as a conversion image; and an image conversion unit that converts the image from one of the first image and the second image. When performing matching between the first predetermined area in the converted image and the second predetermined area in the other image, the second predetermined area is made larger than the first predetermined area, 2 in the first area. A position where the area moves and the degree of coincidence between the first predetermined area and the second predetermined area is maximized is determined as a matching area. An obstacle, comprising: a matching processing unit that obtains a matching image including a region; and an obstacle detection unit that uses a region lower than a reference value as an obstacle in the matching image obtained by the matching processing unit. Object detection device. 2. The matching processing unit according to claim 1, wherein a degree of coincidence between the first predetermined area and the second predetermined area that has been matched with the first predetermined area is equal to or greater than a predetermined threshold. 2. The obstacle detecting device according to claim 1, wherein in this case, a position where the degree of coincidence is a certain threshold value is determined to be a position where the degree of coincidence is the maximum, and subsequent matching is stopped.