JP2010002985A - Moving vector calculation device, obstacle detection device and moving vector calculation method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving vector calculation device, an obstacle detection device and a moving vector calculation method for accurately calculating the moving vector of a vehicle even when the vehicle is traveling at a low speed in the case of detecting an obstacle by using only one camera. <P>SOLUTION: This moving vector calculation device is provided with: a tracking target image extraction part 340 for extracting a partial image at a predetermined position as a tracking target image in a vehicle peripheral image picked up by an in-vehicle camera 140; and a moving vector calculation part 380 for calculating the moving vector of the vehicle based on the change of the position of the tracking target image detected by a position detection part 360 in each of the plurality of vehicle peripheral images. Thus, it is possible to calculate the moving vector of the vehicle by image processing without using the detection result of a vehicle speed sensor. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a motion vector calculation device, an obstacle detection device, and a motion vector calculation method, and more particularly to a technique for calculating a motion vector on a vehicle image at a predetermined time and a technique for detecting an obstacle from the calculated motion vector.

  Conventionally, as a driving support system in the case of parking, it has a function of converting an image captured by an in-vehicle camera that monitors the back of a vehicle into an overhead image, and detects an obstacle from overhead images obtained at two different points. A system for displaying the detected obstacle on the monitor has been developed. However, in this system, in order to detect an obstacle, it is necessary that the vehicle is not in a stationary state, and a vehicle motion vector (movement amount and movement direction) at a predetermined time using a vehicle speed sensor, a steering angle sensor, or the like. ) Must be accurately measured in real time.

A technique for detecting an obstacle using a stereo camera has also been developed. However, detecting obstacles using a stereo camera requires two cameras, which is expensive and requires matching processing between images taken by the two cameras. Becomes enormous. Therefore, a technique for detecting an obstacle using only one camera has been proposed (see, for example, Patent Document 1).
JP 2008-48094 A

  In the technique described in Patent Literature 1, an image obtained by capturing the rear of the vehicle with a camera at time (t−Δt) is two-dimensionally converted to generate a first overhead image, and an image captured with the camera at time t. Is converted into a two-dimensional image to generate a second overhead image. Then, the first overhead image is shifted based on the amount of movement of the vehicle, the difference between the shifted first overhead image and the second overhead image is extracted to generate a differential overhead image, and this difference overhead image More obstacles are detected.

  By the way, in the technique described in Patent Document 1, the amount of movement of the vehicle in a predetermined time (Δt) is obtained based on the vehicle speed signal. Therefore, when a detection error occurs in the vehicle speed sensor (that is, an error occurs in the movement amount of the vehicle), an error also occurs in the first overhead image that is the basis of the differential overhead image that is generated in order to detect an obstacle, In some cases, an object that is not an obstacle is erroneously detected as an obstacle. Especially when driving at low speeds, such as when parking, the rotation speed of the tire decreases and the detection accuracy of the vehicle speed sensor decreases, so errors are likely to occur in the detection value of the vehicle speed sensor. There is a problem that it is easy to detect by mistake.

  The present invention has been made to solve such a problem. When an obstacle is detected using only one camera, the vehicle at a predetermined time can be detected even when the vehicle is traveling at a low speed. It is an object to make it possible to accurately calculate a motion vector (movement amount and movement direction).

  In order to solve the above-described problems, in the present invention, a partial image at a predetermined position in a vehicle surrounding image captured by an imaging device installed in a vehicle is extracted as a tracking target image and sequentially captured by the imaging device. Further, the position of the extracted tracking target image is detected from each of the plurality of vehicle surrounding images, and the motion vector of the vehicle is calculated based on the change in the position of the detected tracking target image.

  According to the present invention configured as described above, since the motion vector of the vehicle at a predetermined time is calculated based on the vehicle surrounding image sequentially captured by the imaging device, it is not necessary to use the detection result of the vehicle speed sensor. A motion vector of the vehicle can be calculated. Thereby, even when the vehicle travels at a low speed where the detection accuracy of the vehicle speed sensor is lowered, the motion vector of the vehicle can be accurately calculated. Therefore, even when the vehicle is traveling at a low speed, an accurate difference image can be generated based on the accurately calculated motion vector, and it is ensured that an object that is not actually an obstacle is erroneously detected as an obstacle. Can be prevented.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration example of an obstacle detection apparatus 100 according to the present embodiment. Obstacle detection device 100 is mounted on a vehicle. The obstacle detection apparatus 100 is connected to an in-vehicle camera 140 (corresponding to the imaging apparatus of the present invention) and a monitor 160 via an in-vehicle network. As shown in FIG. 1, the obstacle detection device 100 includes a motion vector calculation device 200, a difference image generation unit 220, an obstacle detection unit 240, a display screen generation unit 260, an output buffer 270, and a display control unit 280. Has been. The in-vehicle camera 140 sequentially captures the rear periphery of the vehicle at a predetermined imaging time interval Δt (for example, 1/30 seconds). In addition, a timing earlier than Δt is represented by t, and a timing later is represented by (t + Δt).

  The motion vector calculation device 200 uses a captured image captured by the in-vehicle camera 140 (hereinafter referred to as “vehicle surrounding image”) to calculate a vehicle motion vector from the vehicle surrounding image at the imaging time interval Δt. As shown in FIG. 1, the motion vector calculation apparatus 200 includes a captured image acquisition unit 300, a distortion correction unit 320, an input buffer 330, a tracking target image extraction unit 340, a position detection unit 360, and a motion vector calculation unit 380. Has been.

  The captured image acquisition unit 300 sequentially acquires vehicle surrounding images captured by the in-vehicle camera 140 and sequentially outputs them to the distortion correction unit 320. The distortion correction unit 320 corrects the influence of lens distortion of the in-vehicle camera 140 by correcting the value of each pixel in the vehicle surrounding image acquired from the captured image acquisition unit 300. Specifically, the distortion correction unit 320 uses a mapping table that describes which pixel of the vehicle surrounding image captured by the in-vehicle camera 140 corresponds to which pixel of the vehicle surrounding image after correction. The value of each pixel in is corrected. The distortion correction unit 320 performs correction processing on the vehicle surrounding image, and then sequentially stores the corrected vehicle surrounding image in the input buffer 330.

  The tracking target image extraction unit 340 reads the vehicle surrounding image stored in the input buffer 330, and partial images at three positions determined in advance from the read vehicle surrounding image (for example, a partial image including 16 × 16 pixels). Are extracted as three tracking target images. When a vehicle surrounding image is first stored in the input buffer 330, the tracking target image extracting unit 340 extracts partial images at three predetermined positions from the stored vehicle surrounding image as three tracking target images. In addition, the tracking target image extraction unit 340, when the vanishing point calculation unit 400 (to be described later) cannot calculate the vanishing point, performs tracking of the partial images at three predetermined positions from the latest vehicle surrounding image. A new target image is extracted. Further, even when the vanishing point calculating unit 400 can calculate the vanishing point, the tracking target image extracting unit 340 passes through the vanishing point when there is at least one extension line that does not pass through the calculated vanishing point. Instead of the tracking target image that forms a non-extended line, a partial image at one predetermined position is extracted as the tracking target image from the latest vehicle surrounding image. The tracking target image extraction unit 340 outputs the extracted tracking target image to the position detection unit 360 and the motion vector calculation unit 380. Details of vanishing points and extension lines will be described later.

  The position detection unit 360 sequentially reads the vehicle surrounding images stored in the input buffer 330 and detects the positions of the three tracking target images output from the tracking target image extraction unit 340 on each of the read vehicle surrounding images. To do. Specifically, the position detection unit 360 searches for the same partial images as the three tracking target images output from the tracking target image extraction unit 340 from the vehicle surrounding image read from the input buffer 330, and Position coordinates on the vehicle surrounding image are detected as the position of the tracking target image. Here, the determination as to whether or not the image is the same as the tracking target image is performed by, for example, correlation calculation using image feature amounts such as color distribution characteristics and luminance values. Actually, the position detection unit 360 detects the position of the tracking target image by regarding the partial images having a similarity equal to or greater than a certain value as the same. Each time the position detection unit 360 detects the positions of the three tracking target images on the vehicle surrounding image read from the input buffer 330, the position detection unit 360 outputs the detected positions of the three tracking target images to the motion vector calculation unit 380.

  The motion vector calculation unit 380 changes the positions of the three tracking target images detected by the position detection unit 360 in the vehicle surrounding image captured by the in-vehicle camera 140 at the imaging time t and the imaging time (t + Δt) at which the imaging timing continues. Based on the above, the motion vector of the vehicle at the imaging time interval Δt is calculated. If the vehicle is moving between the imaging time t and the imaging time (t + Δt), there is a difference between the detection position of the tracking target image at the imaging time t and the detection position of the tracking target image at the imaging time (t + Δt). Occurs. The motion vector calculation unit 380 detects the difference as a motion vector. As a specific configuration therefor, the motion vector calculation unit 380 includes a vanishing point calculation unit 400, an overhead image generation unit 420, a second position detection unit 440, and a calculation unit 460.

  The vanishing point calculation unit 400 is the same tracking for the positions of the three tracking target images detected by the position detection unit 360 in each of the vehicle surrounding images captured by the in-vehicle camera 140 at the imaging time t and the imaging time (t + Δt). An extension line of a line connecting the positions of the target images is obtained, and an attempt is made to calculate a vanishing point where the obtained extension lines intersect on the vehicle surrounding image captured at the imaging time (t + Δt). When the vanishing point can be calculated (when three extension lines intersect at one point), the vanishing point calculation unit 400 notifies the overhead image generation unit 420 to that effect. In addition, when the vanishing point can be calculated (when two extension lines intersect at one point), the vanishing point calculation unit 400 generates a bird's-eye view image to that effect and which extension line of the tracking target image intersects. Notification to the unit 420.

  In addition, when the vanishing point cannot be calculated, the vanishing point calculation unit 400 notifies the tracking target image extraction unit 340 to that effect. In response to the notification that the vanishing point cannot be calculated from the vanishing point calculation unit 400, the tracking target image extraction unit 340 reads the vehicle surrounding image captured at the imaging time (t + Δt) from the input buffer 330. Then, the tracking target image extraction unit 340 newly extracts partial images at three predetermined positions from the read vehicle surrounding image as three tracking target images, and outputs them to the position detection unit 360. Thereafter, the position detection unit 360 detects the positions of the three tracking target images newly output from the tracking target image extraction unit 340 on the vehicle surrounding image stored in the input buffer 330, and sends it to the motion vector calculation unit 380. Output.

  Even if the vanishing point calculation unit 400 can calculate the vanishing point, when there is at least one extension line that does not pass through the calculated vanishing point (in the present embodiment, one of the three extension lines is extended). When only the line does not pass through the vanishing point), information for specifying the tracking target image forming the extension line that does not pass through the vanishing point (hereinafter referred to as “tracking target image specifying information”) is used as the tracking target image extracting unit 340. Output to. The tracking target image extraction unit 340 that has acquired the tracking target image specifying information reads the vehicle surrounding image captured at the imaging time (t + Δt) from the input buffer 330. Then, the tracking target image extraction unit 340 replaces the tracking target image indicated by the acquired tracking target image identification information with one partial image at a predetermined position from the vehicle surrounding image read from the input buffer 330 as one tracking. A new target image is extracted and output to the position detection unit 360. Thereafter, the position detection unit 360 detects the positions of three tracking target images including one newly extracted tracking target image on the vehicle surrounding image stored in the input buffer 330, and sends it to the motion vector calculation unit 380. Output.

  The bird's-eye view image generation unit 420 receives a notification that the vanishing point can be calculated from the vanishing point calculation unit 400, and reads the vehicle surrounding image at the imaging time t and the imaging time (t + Δt) from the input buffer 330. The bird's-eye view image generation unit 420 uses viewpoint conversion information that describes the correspondence between the pixels of the vehicle surrounding image captured by the in-vehicle camera 140 and the pixels of the vehicle surrounding bird's-eye view image of the rear periphery of the vehicle viewed from the upper virtual viewpoint. Then, by performing viewpoint conversion processing on the read vehicle surrounding image, a vehicle surrounding overhead image viewed from a virtual viewpoint above the vehicle is generated. The overhead image generation unit 420 outputs the generated vehicle surroundings overhead view image to the second position detection unit 440 and the difference image generation unit 220.

  The second position detection unit 440 includes the three tracking outputs output from the tracking target image extraction unit 340 at the imaging time t output from the overhead image generation unit 420 and the overhead view image around the vehicle at the imaging time (t + Δt). The position of the target image is detected. Specifically, the second position detection unit 440 performs viewpoint conversion processing on the three tracking target images output from the tracking target image extraction unit 340 from the vehicle surroundings overhead view image output from the overhead view image generation unit 420. The same partial image is searched for, and the position coordinates of the partial image on the vehicle surroundings overhead image are detected as the position of the tracking target image. The second position detection unit 440 outputs the detected positions of the three tracking target images to the calculation unit 460.

  The calculation unit 460 acquires the position of the tracking target image output from the second position detection unit 440. Then, the calculation unit 460 tracks the position of the tracking target image on the vehicle surroundings overhead image acquired this time (imaging time t + Δt) from the second position detection unit 440 and the tracking on the vehicle surroundings overhead image acquired previously (imaging time t). From the position of the target image, a change in the position of the tracking target image is calculated as a vehicle motion vector in the imaging time interval Δt. The calculation unit 460 outputs the motion vector of the vehicle at the calculated imaging time interval Δt to the difference image generation unit 220.

  Next, an example of an operation for calculating the motion vector of the vehicle at a predetermined time from the vehicle surrounding images continuously captured by the in-vehicle camera 140 will be described with reference to FIG. FIG. 2A is a diagram illustrating a vehicle surrounding image (here, it is assumed that correction processing by the distortion correction unit 320 has already been performed) captured by the in-vehicle camera 140 at a certain imaging timing (imaging time t). is there. In FIG. 2A, 502, 504 and 506 are three tracking target images extracted by the tracking target image extracting unit 340 from the vehicle surrounding image captured by the in-vehicle camera 140. The positions of the three tracking target images are positions where a road is highly likely to be captured and the vanishing point can be obtained. Therefore, the positions of the three tracking target images are positioned at equal intervals below the horizontal line and on the same horizontal line on the vehicle surrounding image. Although the image on the road does not have a large feature in appearance, it can be used as a feature amount because there is a difference in values between adjacent pixels.

  2B shows a vehicle surrounding image (here, correction processing by the distortion correction unit 320) captured by the in-vehicle camera 140 at an imaging timing (imaging time (t + Δt)) after the imaging timing of FIG. It has already been done). In FIG. 2B, 502 (Δt), 504 (Δt) and 506 (Δt) are obtained from the vehicle surrounding image captured at the imaging timing after Δt from the imaging timing in FIG. ) Is a partial image (hereinafter referred to as a tracking target image after movement) determined by the position detection unit 360 to be the same as the three tracking target images 502, 504, and 506 extracted by the tracking target image extraction unit 340. In FIG. 2B, the three tracking target images 502, 504, and 506 extracted by the tracking target image extraction unit 340 in FIG. That is, the three tracking target images 502 and 504 extracted by the tracking target image extraction unit 340 in FIG. 2A by the vehicle traveling backward during the time Δt from the imaging time t to the imaging time (t + Δt). , 506 have moved to the positions of the tracking target images 502 (Δt), 504 (Δt), and 506 (Δt) after the movement.

  FIG. 2C shows a vanishing point 530 that can be calculated by the vanishing point calculation unit 400. Specifically, the vanishing point calculation unit 400 is detected by the position detection unit 360 in each of the vehicle surrounding images in FIGS. 2A and 2B captured at the imaging time t and the imaging time (t + Δt) by the in-vehicle camera 140. The positions of the detected tracking target images 502, 504, and 506, and the positions of the same tracking target images 502, 504, and 506 (that is, the position of the tracking target image 502 and the tracking target image 502 (Δt) after the movement). Three extended lines 520 and 522 connecting the position, the position of the tracking target image 504 and the tracking target image 504 (Δt) after movement, and the position of the tracking target image 506 and the position of the tracking target image 506 (Δt) after movement. 524, and the points where the obtained extension lines 520, 522, 524 intersect Calculated as runs 530.

  As an operation of the motion vector calculation unit 380 thereafter, the overhead image generation unit 420 uses the vehicle surrounding images in FIGS. 2A and 2B captured by the in-vehicle camera 140 at the imaging time t and the imaging time (t + Δt). Each viewpoint conversion process generates a vehicle surroundings overhead image viewed from a virtual viewpoint above the vehicle. Thereafter, the second position detection unit 440 includes the tracking target images 502, 504, and 506 and the tracking target image 502 after movement (Δt) on each of the two vehicle surrounding overhead images generated by the overhead image generation unit 420. , 504 (Δt) and 506 (Δt) are detected. Finally, the calculation unit 460 includes the tracking target images 502, 504, and 506 detected by the second position detection unit 440 on the two vehicle surroundings overhead view images generated by the overhead view image generation unit 420 and the moved images. Changes in the positions of the tracking target images 502 (Δt), 504 (Δt), and 506 (Δt) are calculated as vehicle motion vectors in the imaging time interval Δt.

  FIG. 2D shows vehicle motion vectors 550, 552, and 554 (corresponding to the tracking target images 502, 504, and 506, respectively) at the imaging time interval Δt calculated by the calculation unit 460. As shown in FIG. 2 (c), when all the extension lines 520, 522, 524 pass through the vanishing point 530 calculated by the vanishing point calculation unit 400, the extension lines 520, 522, 524 passing through the vanishing point 530. The vehicle motion vectors 550, 552, and 554 calculated based on the change in the position of the tracking target images 502, 504, and 506 on the vehicle surroundings bird's-eye view image that coincide with each other have the same size and direction.

  Next, another operation example for calculating the motion vector of the vehicle at a predetermined time from the vehicle surrounding images continuously captured by the in-vehicle camera 140 will be described with reference to FIG. FIG. 3A is a diagram illustrating a vehicle surrounding image (here, it is assumed that correction processing by the distortion correction unit 320 has already been performed) captured at a certain imaging timing (imaging time t) by the in-vehicle camera 140. is there. In FIG. 3A, reference numerals 602, 604, and 606 denote three tracking target images extracted by the tracking target image extraction unit 340 from the vehicle surrounding image captured by the in-vehicle camera 140.

  FIG. 3B shows a vehicle surrounding image (here, correction processing by the distortion correction unit 320) captured by the in-vehicle camera 140 at an imaging timing (imaging time (t + Δt)) after the imaging timing of FIG. It has already been done). In FIG. 3B, 602 (Δt), 604 (Δt), and 606 (Δt) are obtained from the vehicle surrounding image captured at the imaging timing after Δt from the imaging timing in FIG. ) Is a partial image (tracking target image after movement) determined by the position detection unit 360 to be the same as the three tracking target images 602, 604, and 606 extracted by the tracking target image extraction unit 340. In FIG. 3B, the three tracking target images 602, 604, and 606 extracted by the tracking target image extraction unit 340 in FIG. That is, the three tracking target images 602 extracted by the tracking target image extraction unit 340 in FIG. 3A when the vehicle travels backward during the time Δt from the imaging time t to the imaging time (t + Δt), It can be seen that the positions 604 and 606 have moved to the positions of the tracking target images 602 (Δt), 604 (Δt), and 606 (Δt) after the movement.

  However, only the position of the tracking target image 606 is different from the example shown in FIG. 2B, and the shadow 610 is avoided due to the influence of the shadow 610 newly reflected in the vehicle surrounding image of FIG. Moved to position. This is because the position of the road corresponding to the tracking target image output from the tracking target image extraction unit 340 is included in the range of the shadow 610, and the pixel value of the road position greatly changes due to the influence of the shadow 610. This is because the same road position is not extracted by the correlation calculation. That is, a partial image that is the same as the tracking target image is found within a range excluding the shadow 610 from the vehicle surrounding image, and the position coordinates of the partial image on the vehicle surrounding image are detected as the position of the tracking target image.

  FIG. 3C shows the vanishing point 630 that can be calculated by the vanishing point calculation unit 400. Specifically, the vanishing point calculator 400 uses the position detector 360 in each of the vehicle surrounding images in FIGS. 3A and 3B captured at the imaging time t and the imaging time (t + Δt) by the in-vehicle camera 140. The positions of the detected tracking target images 602, 604, and 606, and the positions of the same tracking target images 602, 604, and 606 (that is, the position of the tracking target image 602 and the tracking target image 602 (Δt) after the movement). Three extension lines 620 and 622 that connect the position, the position of the tracking target image 604 and the tracking target image 604 (Δt) after movement, and the position of the tracking target image 606 and the position of the tracking target image 606 (Δt) after movement. , 624, and two of the obtained extension lines 620, 622, 624 Calculating a point at which the long line 620, 622 intersect the vanishing point 630. Here, only the extension line 624 formed from the positions of the tracking target images 606 and 606 (Δt) does not pass through the vanishing point 630 due to the influence of the shadow 610 reflected in the vehicle surrounding image of FIG. In this case, the tracking target image extraction unit 340 replaces the tracking target image 606 extracted from the vehicle surrounding image captured at the imaging time t with the same position on the vehicle surrounding image captured at the imaging time (t + Δt). A new tracking target image is extracted.

  As an operation of the motion vector calculation unit 380 thereafter, the overhead image generation unit 420 uses the vehicle surrounding images in FIGS. 3A and 3B captured by the in-vehicle camera 140 at the imaging time t and the imaging time (t + Δt). Each viewpoint conversion process generates two vehicle surrounding overhead images viewed from a virtual viewpoint above the vehicle. Thereafter, the second position detection unit 440 forms tracking target images 602 and 604 that form extension lines 602 and 622 that pass through the vanishing point 630 on the two vehicle surroundings overhead images generated by the overhead image generation unit 420, respectively. In addition, the positions of the tracking target images 602 (Δt) and 604 (Δt) after the movement are detected. Finally, the calculation unit 460 includes the tracking target images 602 and 604 detected by the second position detection unit 440 and the tracking target after movement on each of the two vehicle surrounding overhead images generated by the overhead image generation unit 420. Changes in the positions of the images 602 (Δt) and 604 (Δt) are calculated as vehicle motion vectors in the imaging time interval Δt.

  FIG. 3D shows vehicle motion vectors 650 and 652 (corresponding to tracking target images 602 and 604, respectively) in the imaging time interval Δt calculated by the calculation unit 460. As shown in FIG. 3C, when the extension lines 620 and 622 pass through the vanishing point 630 calculated by the vanishing point calculation unit 400, the tracking target that forms the extension lines 620 and 622 passing through the vanishing point 630. The vehicle motion vectors 650 and 652 calculated based on the change in the position of the images 602 and 604 on the vehicle surroundings bird's-eye view image match in both size and direction. Here, as shown in FIG. 3C, the tracking target images 606 and 606 (Δt) forming the extended line 624 that does not pass through the vanishing point 630 are not used for the calculation of the vehicle motion vector by the calculation unit 460. The motion vectors corresponding to the tracking target images 606 and 606 (Δt) are not shown in FIG.

  The difference image generation unit 220 acquires the vehicle motion vector in the imaging time interval Δt output from the calculation unit 460. Further, the difference image generation unit 220 acquires the vehicle surroundings overhead view image at the imaging time t and the imaging time (t + Δt) output from the overhead image generation unit 420. In addition, the difference image generation unit 220 moves the vehicle surroundings overhead image at the imaging time t by the motion vector based on the acquired vehicle motion vector at the imaging time interval Δt. And the difference for every pixel of this moved vehicle surroundings bird's-eye view image and the vehicle surroundings bird's-eye view image of imaging time (t + Δt) is extracted, and a difference image is generated. The difference image generation unit 220 outputs the generated difference image to the obstacle detection unit 240.

  The obstacle detection unit 240 analyzes the difference image output from the difference image generation unit 220 and attempts to detect an obstacle existing behind the vehicle. Specifically, the obstacle detection unit 240 extracts only an area having a pixel value equal to or greater than a preset threshold value from the difference image, and detects the extracted area as an obstacle. When the obstacle detection unit 240 detects an obstacle, the obstacle detection unit 240 notifies the display screen generation unit 260 to that effect. The obstacle detection unit 240 detects the position of the detected obstacle on the vehicle surrounding image. Specifically, the obstacle detection unit 240 refers to the viewpoint conversion information that the obstacle detection unit 240 has, based on the difference image of the detected obstacle (the vehicle surrounding image captured at the imaging time t and the imaging time (t + Δt)). The position of the obstacle on the vehicle surrounding image captured at the imaging time (t + Δt) is calculated from the position on the generated vehicle surrounding overhead view image) and output to the display screen generation unit 260. Further, when the obstacle detection unit 240 does not detect the obstacle, the obstacle detection unit 240 notifies the display screen generation unit 260 to that effect.

  When the display screen generation unit 260 receives a notification that an obstacle has been detected from the obstacle detection unit 240, the display screen generation unit 260 reads the vehicle surrounding image captured at the imaging time (t + Δt) from the input buffer 330. In addition, the display screen generation unit 260 acquires the position of the obstacle on the vehicle surrounding image captured at the imaging time (t + Δt) from the obstacle detection unit 240. Then, the display screen generation unit 260 notifies the driver that there is an obstacle behind the vehicle from the vehicle surrounding image read from the input buffer 330 and the position of the obstacle acquired from the obstacle detection unit 240. A warning display screen is generated. In the warning display screen generated by the display screen generation unit 260, in order to emphasize the presence of the obstacle detected by the obstacle detection unit 240, the position where the obstacle exists is surrounded by a frame of a predetermined size. A frame of a predetermined size is blinked at a predetermined time interval. Further, the display screen generation unit 260 stores the generated warning display screen in the output buffer 270.

  In addition, when the display screen generation unit 260 receives a notification from the obstacle detection unit 240 that the obstacle has not been detected, the display screen generation unit 260 normally displays the state behind the vehicle from the vehicle surrounding image read from the input buffer 330. Generate a screen. The display screen generation unit 260 stores the generated normal display screen in the output buffer 270.

  The display control unit 280 reads the warning display screen stored in the output buffer 270 and displays the read warning display screen on the monitor 160. In addition, the display control unit 280 reads the normal display screen stored in the output buffer 270 and displays the read normal display screen on the monitor 160. By checking the warning display screen displayed on the monitor 160, the driver can recognize an obstacle existing behind the vehicle and take appropriate measures thereafter.

  Next, the display operation by the display control unit 280 of the obstacle detection apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of a warning display screen displayed on the monitor 160 by the display control unit 280 of the obstacle detection apparatus 100 according to the present embodiment.

  4A shows a vehicle surrounding image captured at a certain imaging timing (imaging time (t + Δt)) by the in-vehicle camera 140. As shown in FIG. There are obstacles 700, 702, 704 such as cones, etc. As described above, the obstacles 700, 702, 704 on the vehicle surrounding image are detected by the operation of the obstacle detection apparatus 100.

  FIG. 4B is a diagram illustrating an example of a warning display screen generated based on the vehicle surrounding image in FIG. In this warning display screen, the entire vehicle surrounding image captured by the in-vehicle camera 140 is displayed in color video, and the presence of the obstacles 700, 702, and 704 detected by the obstacle detection unit 240 is emphasized. The positions where the obstacles 700, 702, and 704 exist are surrounded by predetermined size frames 708, 710, and 712, respectively, and the predetermined size frames 708, 710, and 712 are blinked at predetermined time intervals (for example, 1 second). It is additionally displayed in a manner.

  Note that the aspect of emphasizing the presence of the obstacles 700, 702, and 704 on the warning display screen is not limited to the above example. For example, only the portions corresponding to the obstacles 700, 702, and 704 are displayed as color images, and the other portions are displayed as monochrome images so that the obstacles and other portions can be clearly recognized. Also good.

  Next, the operation of the obstacle detection apparatus 100 in this embodiment will be described in detail. FIG. 5 is a flowchart showing an operation example of the obstacle detection apparatus 100 according to the present embodiment. The process of step S100 in FIG. 5 is started when, for example, an ignition key (not shown) of the vehicle is operated by the user to turn on the obstacle detection device 100.

  First, the obstacle detection apparatus 100 determines whether or not the shift lever has entered R (reverse) (step S100). If the obstacle detection device 100 determines that the shift lever has not entered R (reverse) (NO in step S100), the obstacle detection device 100 has detected that the previous shift lever has entered R (reverse). It is determined whether or not a predetermined time has passed since step S120 (step S120). If obstacle detection device 100 determines that the predetermined time has elapsed (YES in step S120), the process in FIG. 5 is terminated. On the other hand, when obstacle detection device 100 determines that the predetermined time has not elapsed (NO in step S120), the process transitions to step S140.

  On the other hand, when the obstacle detection device 100 determines that the shift lever has entered R (reverse) (YES in step S100), the motion vector calculation device 200 performs processing for calculating the vehicle motion vector in the imaging time interval Δt. This is performed (step S140). FIG. 6 is a flowchart showing an example of the flow of motion vector calculation processing of the motion vector calculation apparatus 200 according to the present embodiment. First, the captured image acquisition unit 300 acquires a vehicle surrounding image captured by the in-vehicle camera 140 (step S300). Then, the captured image acquisition unit 300 outputs the vehicle surrounding image acquired from the in-vehicle camera 140 to the distortion correction unit 320. The distortion correction unit 320 corrects the influence of the lens distortion of the in-vehicle camera 140 on the vehicle surrounding image output from the captured image acquisition unit 300 (step S320). Then, the distortion correction unit 320 stores the corrected vehicle surrounding image in the input buffer 330.

  Next, the tracking target image extraction unit 340 determines whether or not the tracking target image has already been extracted from the vehicle surrounding image (step S340). If it has not been extracted yet (NO in step S340), that is, the first processing timing (timing for processing the first vehicle surrounding image) of motion vector calculation apparatus 200 after the shift lever enters R (reverse). The tracking target image extraction unit 340 reads the vehicle surrounding image stored in the input buffer 330 and extracts partial images at three predetermined positions from the vehicle surrounding image as three tracking target images (step S360). Then, the tracking target image extraction unit 340 outputs the extracted three tracking target images to the position detection unit 360 and the motion vector calculation unit 380. The motion vector calculation unit 380 also outputs the positions of the extracted three tracking target images. Thereafter, the process proceeds to step S300.

  On the other hand, when the tracking target image has already been extracted from the vehicle surrounding image (YES in step S340), that is, after the shift lever enters R (reverse), the second and subsequent processing timings (2 At the timing of processing the vehicle surrounding image after the first sheet), the position detection unit 360 reads the latest vehicle surrounding image stored in the input buffer 330 after the processing of step S320, and on the read vehicle surrounding image, In step S360, the positions of the three tracking target images extracted by the tracking target image extraction unit 340 are detected (step S380). Then, the position detection unit 360 outputs the detected positions of the three tracking target images to the motion vector calculation unit 380.

  Next, the vanishing point calculation unit 400 is detected by the position detection unit 360 in each of the vehicle surrounding images imaged at successive imaging timings (for example, imaging time t and imaging time (t + Δt)) by the in-vehicle camera 140. An attempt is made to calculate the vanishing point from the positions of the two tracking target images (step S400).

  If the vanishing point cannot be calculated (NO in step S420), vanishing point calculation unit 400 notifies tracking target image extraction unit 340 to that effect. In response to the notification that the vanishing point cannot be calculated from the vanishing point calculating unit 400, the tracking target image extracting unit 340 reads the vehicle surrounding image captured at the imaging time (t + Δt) from the input buffer 330. Then, three tracking target images are newly extracted from different positions on the read vehicle surrounding image in place of the previously extracted three tracking target images (step S440). Then, the tracking target image extraction unit 340 outputs the three newly extracted tracking target images to the position detection unit 360.

  On the other hand, if the vanishing point can be calculated (YES in step S420), vanishing point calculation unit 400 determines whether or not there is even one extension line that does not pass through the vanishing point that can be calculated. Determination is made (step S460). If the vanishing point calculation unit 400 determines that there is an extension line that does not pass through the vanishing point (YES in step S460), the vanishing point calculation unit 400 forms the extension line that does not pass through the vanishing point. The tracking target image specifying information for specifying the image is output to the tracking target image extracting unit 340.

  The tracking target image extracting unit 340 that has acquired the tracking target image specifying information reads the vehicle surrounding image captured at the imaging time (t + Δt) from the input buffer 330, and specifies the tracking target image from the read vehicle surrounding image. Instead of the tracking target image indicated by the information, a new tracking target image is extracted (step S480). Then, the tracking target image extraction unit 340 outputs the newly extracted tracking target image to the position detection unit 360. Thereafter, the process proceeds to step S560. On the other hand, when the vanishing point calculation unit 400 determines that there is no extension line that does not pass through the vanishing point (that is, all extension lines pass through the vanishing point) (NO in step S460), the process transitions to step S500. To do.

  In step S500, the bird's-eye view image generation unit 420 receives a notification that the vanishing point can be calculated from the vanishing point calculation unit 400 and captures the imaging time t and imaging time (t + Δt) captured by the in-vehicle camera 140. By reading the vehicle surrounding image from the input buffer 330 and subjecting the read vehicle surrounding image to viewpoint conversion processing, a vehicle surroundings overhead image viewed from a virtual viewpoint above the vehicle is generated. Then, the bird's-eye view image generation unit 420 outputs the generated vehicle surroundings bird's-eye view image to the second position detection unit 440 and the difference image generation unit 220.

  Next, the second position detection unit 440 outputs the tracking target image extraction unit 340 at each of the imaging time t and the imaging time (t + Δt) on the vehicle surroundings overhead image output from the overhead image generation unit 420. The positions of the three tracking target images are detected (step S520). Then, the second position detection unit 440 outputs the detected positions of the three tracking target images to the calculation unit 460. Next, the calculation unit 460 acquires the position of the tracking target image output from the second position detection unit 440, and the position of the three tracking target images on the vehicle surroundings overhead image at the imaging time t and the imaging time ( From the position of the tracking target image on the vehicle surroundings overhead view image at (t + Δt), a change in the position of the tracking target image is calculated as a vehicle motion vector at the imaging time interval Δt (step S540). Then, the calculation unit 460 outputs the motion vector of the vehicle at the calculated imaging time interval Δt to the difference image generation unit 220. Thereafter, the motion vector calculation apparatus 200 ends the processing in FIG. 6 and proceeds to step S160 in FIG.

  In step S160 of FIG. 5, the difference image generation unit 220 is a bird's-eye view of the surroundings of the imaging time t output from the overhead image generation unit 220 based on the motion vector of the vehicle at the imaging time interval Δt output from the calculation unit 460. Move the image by the motion vector. And the difference image is produced | generated by extracting the difference of the vehicle surroundings bird's-eye view image of this moved imaging time t, and the vehicle surroundings bird's-eye view image of the imaging time (t + (DELTA) t) output from the bird's-eye view image generation part 220.

  Next, the obstacle detection unit 240 analyzes the difference image output from the difference image generation unit 220 and attempts to detect an obstacle existing behind the vehicle (step S180). If obstacle detection unit 240 has not detected an obstacle (NO in step S200), display screen generation unit 260 receives notification from obstacle detection unit 240 that no obstacle has been detected. Then, a normal display screen representing the state behind the vehicle is generated from the vehicle surrounding image read from the input buffer 330 (step S220). Then, the display screen generation unit 260 stores the generated normal display screen in the output buffer 270. Finally, the display control unit 280 reads the normal display screen stored in the output buffer 270 and displays the read normal display screen on the monitor 160 (step S240).

  On the other hand, when obstacle detection unit 240 detects an obstacle (YES in step S200), obstacle detection unit 240 detects the position of the detected obstacle on the vehicle surrounding image. Then, the obstacle detection unit 240 outputs the detected position of the obstacle on the vehicle surrounding image to the display screen generation unit 260.

  Next, the display screen generation unit 260 receives the notification that the obstacle is detected from the obstacle detection unit 240, and the vehicle surrounding image read from the input buffer 330 and the position of the obstacle acquired from the obstacle detection unit 240. Thus, a warning display screen for notifying the driver that there is an obstacle behind the vehicle is generated (step S260). Then, the display screen generation unit 260 stores the generated warning display screen in the output buffer 270. Finally, the display control unit 280 reads the warning display screen stored in the output buffer 270, and displays the read warning display screen on the monitor 160 (step S280). Note that when the display process in step S240 or step S280 ends, the process transitions to step S100.

  As described above in detail, in this embodiment, the tracking target image extraction unit 340 extracts partial images at three predetermined positions in the vehicle surrounding image captured by the in-vehicle camera 140 as the tracking target image, and the in-vehicle In each of the two vehicle surrounding images continuously captured by the camera 140, the position detection unit 360 detects the position of the tracking target image. Then, based on the change in the position of the tracking target image detected by the position detection unit 360, the motion vector calculation unit 380 calculates a vehicle motion vector at a predetermined time (imaging time interval Δt).

  Accordingly, since the vehicle motion vector is calculated based on the vehicle surrounding images continuously captured by the in-vehicle camera 140, the vehicle motion vector can be calculated without using the detection result of the vehicle speed sensor. That is, even when the vehicle is traveling at a low speed where the detection accuracy of the vehicle speed sensor is reduced, the vehicle motion vector is accurately calculated as long as the amount of movement (pixels) of the vehicle on the vehicle surrounding image continuously captured by the in-vehicle camera 140 is known. be able to. Therefore, even when the vehicle is traveling at a low speed, an accurate difference image can be generated based on the accurately calculated motion vector, and it is ensured that an object that is not actually an obstacle is erroneously detected as an obstacle. Can be prevented.

  Further, in the present embodiment, the motion vector calculation unit 380, only when the vanishing point can be calculated, based on the change in the position of the tracking target image that forms the extension line passing through the vanishing point, Is calculated. The motion vector may be obtained from the vehicle surroundings overhead image without obtaining the vanishing point on the vehicle surrounding image. However, by checking whether the vanishing point is obtained, an incorrect vehicle motion vector is erroneously determined. Can be avoided. For example, if the position of the tracking target image on the vehicle surrounding image cannot be correctly detected due to the influence of a shadow existing on the road surface, the vanishing point cannot be calculated. This is effective for determining whether or not it can be calculated.

  In the present embodiment, even if an extension line that does not pass through the vanishing point exists, a new tracking target image is extracted from the latest vehicle surrounding image instead of the tracking target image that forms the extension line. A mode in which a new tracking target image is not extracted may be used, but in that case, the extension line does not pass through the vanishing point. On the other hand, if a new extraction is performed, the possibility that an extension line formed by the position of the tracking target image from which the tracking target image is newly extracted passes through the vanishing point at the next and subsequent imaging timings can be restored. .

  In the present embodiment, the tracking target image extraction unit 340 extracts the tracking target image from a predetermined position in the vehicle surrounding image. Thereby, as a premise of the correlation calculation by the position detection unit 360, image processing for searching for and extracting feature points such as edge portions from the vehicle surrounding image becomes unnecessary. Accordingly, the load of the vehicle motion vector calculation process is reduced, and high-speed processing is possible.

  In this embodiment, the example in which the tracking target image extraction unit 340 extracts the tracking target image from a predetermined position in the vehicle surrounding image has been described. However, the present invention is not limited to this. For example, the tracking target image extraction unit 340 calculates the edge strength of each pixel for the vehicle surrounding image, and the calculated edge strength is equal to or higher than a first predetermined value and higher than the first predetermined value. A partial image that is less than a predetermined value may be extracted as a tracking target image. In this way, the point where the brightness in the image changes abruptly (for example, a person or a vehicle is likely to move independently of the host vehicle, so the motion vector is likely to be unrelated to the movement of the host vehicle. Point), or the point where the brightness in the image hardly changes (that is, the point where there is almost no feature as an image and the error in detecting the position of the tracking target image on the vehicle surrounding image) is selected as the tracking target image Therefore, an error in detecting the position of the tracking target image on the vehicle surrounding image can be prevented as much as possible. In this example, the vanishing point may not be obtained. Therefore, it is better not to perform the process of re-taking the tracking target image depending on whether the vanishing point is obtained.

  In the present embodiment, the motion vector calculation unit 380 calculates an angle between the direction indicated by the previously calculated motion vector and the direction indicated by the currently calculated motion vector, and the calculated angle is less than a predetermined value. Only in this case, the motion vector calculated this time may be adopted. As a result, when the angle between the direction indicated by the previously calculated motion vector and the direction indicated by the currently calculated motion vector becomes large (for example, tracking on the vehicle surrounding image due to the influence of a shadow existing on the road surface). When the position of the target image changes greatly), by not using the vehicle motion vector calculated this time, it is possible to avoid erroneously using an incorrect vehicle motion vector.

  Further, in the present embodiment, the position of the tracking target image is detected in each of the vehicle peripheral images continuously captured by the in-vehicle camera 140, and a predetermined time (imaging time interval Δt) is determined based on the detected change in the position of the tracking target image. ), The position of the tracking target image is detected in each of the two vehicle surrounding images whose imaging timings are not continuous, and based on the detected change in the position of the tracking target image. The motion vector of the vehicle at a predetermined time (between the time when each vehicle peripheral image is captured) may be calculated.

  In the present embodiment, the tracking target image extraction unit 340 extracts the three tracking target images from the vehicle surrounding image. However, the two tracking target images may be extracted from the vehicle surrounding image. . However, as in the present embodiment, it is preferable to extract three tracking target images from the vehicle surrounding image from the viewpoint of increasing the possibility of calculating the vanishing point. Note that four or more tracking target images may be extracted from the vehicle surrounding image from the viewpoint of increasing the possibility of calculating the vanishing point.

  Moreover, in this embodiment, although the 2nd position detection part 440 demonstrated the example which detects the position of a tracking object image in the vehicle surroundings overhead view image produced | generated by the overhead view image generation part 420, the 2nd position detection part 440 performs viewpoint conversion processing on the vehicle surrounding image from the position of the tracking target image on the vehicle surrounding image detected by the position detection unit 360 and the viewpoint conversion mapping table of the overhead image generation unit 420 itself. The position of the tracking target image in the vehicle surroundings overhead image that would have been obtained by the above may be detected.

  In addition, each of the above-described embodiments is merely an example of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. In other words, the present invention can be implemented in various forms without departing from the spirit or main features thereof.

It is a block diagram explaining the structural example of the obstruction detection apparatus by this embodiment. It is explanatory drawing explaining an example of operation | movement of the motion vector calculation apparatus by this embodiment. It is explanatory drawing explaining an example of operation | movement of the motion vector calculation apparatus by this embodiment. It is a figure which shows the example of the warning display screen displayed on the monitor by this embodiment. It is a flowchart which shows the operation example of the obstruction detection apparatus by this embodiment. It is a flowchart which shows the operation example of the motion vector calculation apparatus by this embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Obstacle detection apparatus 200 Motion vector calculation apparatus 220 Difference image generation part 240 Obstacle detection part 340 Tracking object image extraction part 360 Position detection part 380 Motion vector calculation part 400 Vanishing point calculation part 420 Overhead image generation part 440 2nd position Detection unit

Claims (9)

A tracking target image extraction unit that extracts a partial image at a predetermined position in a vehicle surrounding image captured by an imaging device installed in the vehicle as a tracking target image;
A position detection unit that detects the position of the tracking target image in each of a plurality of vehicle surrounding images sequentially captured by the imaging device;
A motion vector comprising: a motion vector calculation unit that calculates a motion vector of the vehicle based on a change in position of the tracking target image detected in the plurality of vehicle surrounding images by the position detection unit. Calculation device. The motion vector calculation device according to claim 1,
The tracking target image extraction unit extracts a plurality of tracking target images from a plurality of predetermined positions in the vehicle surrounding image,
The position detection unit detects positions of the plurality of tracking target images in each of a plurality of vehicle surrounding images sequentially captured by the imaging device;
The motion vector calculation unit is configured to obtain positions of the plurality of tracking target images detected by the position detection unit in each of a plurality of vehicle surrounding images sequentially captured by the imaging device and positions of the same tracking target images. Each of the extension lines of the connecting lines is obtained, it is determined whether or not a vanishing point where the obtained plural extension lines intersect can be calculated, and only when the vanishing points can be calculated, the positions of the plurality of tracking target images are determined. A motion vector calculation device that calculates a motion vector of the vehicle based on a change. The motion vector calculation apparatus according to claim 2,
The motion vector calculation unit is configured to obtain positions of the plurality of tracking target images detected by the position detection unit in each of a plurality of vehicle surrounding images sequentially captured by the imaging device and positions of the same tracking target images. A vanishing point calculation unit that calculates an extension line of each connecting line and calculates a vanishing point where the obtained plurality of extension lines intersect,
Only when the vanishing point can be calculated by the vanishing point calculation unit, the plurality of vehicle surrounding images sequentially captured by the imaging device are respectively subjected to viewpoint conversion processing, and viewed from a virtual viewpoint above the vehicle. An overhead image generation unit for generating an overhead image of the surroundings of the vehicle;
A second position detection unit that detects positions of the plurality of tracking target images in each of the plurality of vehicle surroundings overhead images generated from the plurality of vehicle surroundings images by the overhead view image generation unit;
A motion vector calculation apparatus that calculates a motion vector of the vehicle from a change in position of the tracking target image detected in the plurality of vehicle surrounding overhead images by the second position detection unit. The motion vector calculation apparatus according to claim 3,
When there is an extended line that does not pass through the vanishing point calculated by the vanishing point calculating unit, the second position detecting unit is configured to generate a plurality of the vehicle surrounding images generated by the overhead image generating unit. In each of the vehicle surrounding bird's-eye view images, among the plurality of tracking target images, the position of only the tracking target image that forms an extension line passing through the vanishing point is detected,
The motion vector calculation unit calculates a motion vector of the vehicle from a change in position of the tracking target image detected in the plurality of vehicle surrounding overhead images by the second position detection unit. Vector calculation device. The motion vector calculation apparatus according to claim 4,
The tracking target image extraction unit forms an extension line that does not pass through the vanishing point among the plurality of tracking target images when there is an extension line that does not pass through the vanishing point calculated by the vanishing point calculation unit. A motion vector calculation device characterized by newly extracting a tracking target image. The motion vector calculation device according to claim 1,
The tracking target image extraction unit calculates an edge strength of each pixel for the vehicle surrounding image instead of extracting the tracking target image from a predetermined position in the vehicle surrounding image, and calculates the calculated edge strength. A motion vector calculation apparatus that extracts a partial image that is greater than or equal to a first predetermined value and less than a second predetermined value that is greater than the first predetermined value as the tracking target image. In the motion vector calculation device according to any one of claims 1 to 6,
The motion vector calculation unit includes a direction indicated by a motion vector previously calculated from a plurality of vehicle surrounding images captured at a first timing by the imaging device, and a second later than the first timing by the imaging device. An angle formed by a direction indicated by the motion vector calculated this time from a plurality of vehicle surrounding images including a vehicle surrounding image captured at timing is calculated, and the current time only when the calculated angle is less than a predetermined value. A motion vector calculation apparatus that employs a calculated motion vector. A tracking target image extraction unit that extracts a partial image at a predetermined position in a vehicle surrounding image captured by an imaging device installed in the vehicle as a tracking target image;
A position detection unit that detects the position of the tracking target image in each of a plurality of vehicle surrounding images sequentially captured by the imaging device;
A motion vector of the vehicle is calculated based on a change in the position of the tracking target image detected by the position detection unit in the plurality of vehicle surrounding images captured by the imaging device from a certain timing to a later timing. A motion vector calculation unit;
A bird's-eye view image generation unit that generates a vehicle surroundings bird's-eye view image viewed from a virtual viewpoint above the vehicle by performing viewpoint conversion processing on each of the plurality of vehicle surrounding images sequentially captured by the imaging device;
Based on the vehicle motion vector calculated by the motion vector calculation unit, the vehicle surroundings overhead image generated by the overhead image generation unit is moved from the vehicle surroundings imaged at a certain timing by the imaging device, A difference image generation unit that extracts a difference between the moved vehicle surrounding bird's-eye view image and the vehicle surrounding bird's-eye view image generated from the vehicle surrounding image captured at the later timing by the imaging device and generates a difference image When,
An obstacle detection apparatus comprising: an obstacle detection unit that detects an obstacle from the difference image generated by the difference image generation unit. A first step of extracting, as a tracking target image, a partial image at a predetermined position in a vehicle surrounding image captured by an imaging device installed in the vehicle;
A second step of detecting a position of the tracking target image in each of a plurality of vehicle surrounding images sequentially captured by the imaging device;
And a third step of calculating a motion vector of the vehicle based on a change in position of the tracking target image detected in the plurality of vehicle surrounding images in the second step. Vector calculation method.