JP2010079582A - Apparatus, method and program for detecting object - Google Patents

Abstract

An object detection apparatus for detecting an object with a wide visual field range with high accuracy is provided.
An object detection apparatus includes: a parallax calculation unit that calculates parallax between a plurality of input images; a parameter calculation unit that calculates a surface parameter of a road surface on which the object exists; A region calculation unit 104a that calculates a region and a non-common region, and a rectangle on the road surface of the common region, and a rectangle whose disparity uniformity of each point included in the rectangle on the road surface of the common region is greater than a threshold value Is generated as a candidate rectangle for detecting an object, a second generation unit 104c is generated as a candidate rectangle for a rectangle on the road surface of the non-common area, and an object to be detected in the generated candidate rectangle And a determination unit 105 that determines whether or not is included.
[Selection] Figure 3

Description

  The present invention relates to an apparatus, a method, and a program for detecting a specific object (for example, a person) using a plurality of cameras mounted on a moving body (for example, an automobile).

  An object detection technique for detecting a specific object included in an input image has a wide application range and is a very useful technique. Using this technology, for example, it is possible to detect pedestrians around the vehicle with a camera mounted on a car, and to support safe driving by alerting the driver with an alarm or controlling the brake Can do. When a camera is mounted on a moving body such as an automobile to detect an object, stereo viewing with two cameras is effective.

An outline of a pedestrian detection process using an in-vehicle stereo camera is shown below (for example, Non-Patent Document 1).
1. Calculate the parallax of each point between stereo images.
2. Based on the parallax of each point, a solid object on the road is detected as a pedestrian candidate area.
3. Each candidate area is pattern-identified to detect a pedestrian area. Pattern identification is performed by a classifier generated by learning in advance.

  If stereo vision is used, not only the distance to the detection target can be measured, but also the region having a height relative to the road surface can be extracted using the obtained three-dimensional information. Can be limited. In general, since the pattern identification processing has a high calculation cost, it is possible to reduce the overall processing cost and improve the identification accuracy by stereo processing (for example, Non-Patent Document 1).

D. M. Gavrila and S. Munder, “Multi-Cue Pedestrian Detection and Tracking from a Moving Vehicle”, International Journal of Computer Vision, Springer Verlag, vol.73, no.1 (June), pp.41-59, 2007.

  However, the region where the stereo parallax can be calculated is limited to the common visual field region of both the left and right cameras. For this reason, for example, even when a pedestrian enters the imaging range of the right camera, it cannot be detected as a pedestrian candidate if it is outside this common visual field region. For this reason, a problem such as a delay in the alarm and a delay in the brake timing as a result may occur.

  The present invention has been made in view of the above, and an object of the present invention is to provide an apparatus, a method, and a program that can detect an object in a wide visual field range with high accuracy.

  In order to solve the above-described problems and achieve the object, the present invention is an object detection device that detects an object existing on a reference plane, and is configured to reduce parallax between images captured by each of a plurality of imaging devices. A parallax calculation unit to calculate, an area calculation unit to calculate a common area of the plurality of images and a non-common area other than the common area based on the parallax, and a rectangle in the common area, The non-common with the first generation unit that generates a rectangle whose parallax uniformity of each point included in the rectangle is larger than a predetermined threshold as a candidate rectangle that is a candidate for the rectangle for detecting the object A second generation unit that generates a rectangle in a region as the candidate rectangle, and a determination unit that determines whether or not an object to be detected is included in the generated candidate rectangle.

  The present invention is also a method and program capable of executing the above-described apparatus.

  According to the present invention, it is possible to detect an object with a wide visual field range with high accuracy.

  Exemplary embodiments of an apparatus, a method, and a program according to the present invention will be described below in detail with reference to the accompanying drawings.

  The object detection apparatus according to the present embodiment is an apparatus that detects an object existing on a reference plane such as a road surface or a floor surface by stereo viewing using a stereo camera or the like, and includes each camera ( By combining monocular vision that detects objects only from individual images captured by the imaging device with stereo vision, both disadvantages are compensated for, and objects with a wider field of view are more accurately compared to conventional methods. To detect.

  First, the visual field of the stereo camera will be described with reference to FIGS. FIG. 1 is a diagram illustrating an example of a common visual field of a stereo camera. FIG. 2 is a diagram illustrating an example of an image captured when a pedestrian enters the moving body from the right side.

  FIG. 1A shows an example in which two cameras 11a and 11b constituting a stereo camera are installed at a predetermined interval B. FIG. 1B shows imaging ranges of the individual cameras 11a and 11b.

  As shown in FIG. 1A, the shaded portion that is the overlapping portion of the imaging ranges of the two cameras 11a and 11b is a common field of view. The space of width B existing on the left and right of the common visual field is an area where stereo parallax cannot be calculated. The area that can be stereo-processed is narrower by the width B than the imaging range of one camera (FIG. 1B).

  Therefore, as shown in FIG. 2, when the pedestrian enters the moving body from the right side, the left camera is present even if the pedestrian is within the imaging range of the right camera (FIG. 2 (a)). If it exists outside the imaging range (FIG. 2B), it cannot be detected as a pedestrian candidate by stereo processing. In such a case, the pedestrian is detected after entering the common visual field. For this reason, as described above, problems such as delays in alarm and brake timing occur.

  Since the width B coincides with the camera interval, if the camera interval is reduced, an area that can be seen only by one camera can be reduced. However, since the principle of stereo vision is triangulation, there arises a problem that the distance measurement accuracy decreases as the camera interval becomes narrower.

  Therefore, the object detection apparatus according to the present embodiment makes it possible to detect an object in a wide visual field range with high accuracy by combining monocular vision and stereo vision.

  Next, the configuration of the object detection apparatus according to the present embodiment will be described with reference to FIG. FIG. 3 is a block diagram showing a configuration of object detection apparatus 100 according to the present embodiment. The object detection apparatus 100 includes an image input unit 101, a parallax calculation unit 102, a parameter calculation unit 103, a candidate rectangle generation unit 104, a determination unit 105, and an output unit 106. Note that the image input unit 101 may be disposed outside the object detection device 100 and connected to the object detection device 100.

  In this embodiment, as shown in FIG. 4, the road in the traveling direction of the automobile using two cameras 11 a and 11 b functioning as the image input unit 101 mounted in front of the moving body 401 (automobile). An example of detecting a pedestrian 411 present on the surface 410 is shown. The object detection device 100 determines whether or not a pedestrian that is a detection target is included in the input images input from the cameras 11a and 11b, and if included, the position of the pedestrian on the image and the position in the real space Is output.

  Returning to FIG. 3, the image input unit 101 inputs a plurality of images having different shooting viewpoints using at least two cameras. If there is an overlap in the field of view, the positions and directions of the cameras can be arbitrarily set. In the present embodiment, it is assumed that the same two cameras 11a and 11b are arranged in parallel on the left and right to shoot a stereo image.

  FIG. 5 is a diagram illustrating an example of a stereo camera coordinate system used in the present embodiment. In the present embodiment, the origin O of the stereo camera coordinate system is taken as the viewpoint (lens center) of the right camera (camera 11b). Further, a straight line connecting the viewpoints of the left and right cameras is set as the X axis, the Y axis is set vertically downward, and the Z axis is set in the optical axis direction of the camera.

If the distance between cameras (baseline length) is B, the position of the left camera can be expressed as (−B, 0, 0). For the sake of simplicity, if the road surface is modeled as a plane and is ignored because the slope in the horizontal direction is very small, the equation for the road surface (road plane) is expressed by the following equation (1).
Y = αZ + β (1)

  α and β represent the inclination of the road plane observed from the stereo camera and the height of the stereo camera from the road surface, respectively. Hereinafter, α and β are collectively referred to as surface parameters. In general, the slope of the road surface varies from place to place, and the camera vibrates when traveling on a moving body, so that the surface parameters change from moment to moment as the moving body moves.

  Returning to FIG. 3, the parallax calculation unit 102 calculates the parallax between the stereo images acquired by the image input unit 101. Specifically, the parallax calculation unit 102 obtains corresponding points (corresponding points) between a plurality of images constituting a stereo image, and each of the obtained corresponding points represents a difference in position on the plurality of images. Is calculated.

  Here, a coordinate system for each image of the stereo image will be described. An image coordinate system is set for each image constituting the stereo image. An x-axis and a y-axis are set in the horizontal direction and the vertical direction, respectively, in an image captured by the right camera (camera 11b) (hereinafter referred to as “right image”). Similarly, an x ′ axis and a y ′ axis are set in the horizontal direction and the vertical direction, respectively, in an image captured by the left camera (camera 11a) (hereinafter referred to as “left image”). It is assumed that the horizontal direction (x-axis and x′-axis) of each image matches the X-axis direction of the stereo camera coordinate system.

  In such a case, if the corresponding point on the left image of the point (x, y) on the right image is (x ′, y ′), y = y ′. Therefore, only the difference in position in the horizontal direction needs to be considered. Hereinafter, the difference in the horizontal position of the corresponding point between the right image and the left image is referred to as parallax. In the following, the parallax is expressed as d = x−x ′ with the right image as the reference image.

  Next, a parallax calculation method performed by the parallax calculation unit 102 will be described with reference to FIG. FIG. 6 is a schematic diagram for explaining an example of a parallax calculation method. The parallax calculation unit 102 searches the same scanning line on the left image for an arbitrary point (x, y) of the right image that is the reference image, and finds a corresponding point (x ′, y) = (x + d, y). Ask.

  Since d ≧ 0, the parallax calculation unit 102 only needs to examine a point on the right side of the same coordinate on the left image as illustrated in FIG. More specifically, the parallax calculation unit 102 sets a window around a point (x, y) on the right image as shown in FIG. 6B, and selects a luminance pattern most similar to the internal luminance pattern. The surrounding points are obtained from the same scanning line of the left image.

For example, normalized cross-correlation C can be used as an evaluation measure of similarity of luminance patterns. When the size of the search window is (2w + 1) × (2w + 1) pixels and the luminance in the window set for the left and right images is expressed as f (ξ, η) and g (ξ, η), the normalized cross-correlation C is (2).

  If a corresponding point is searched for any point on the reference image using the normalized cross-correlation C, a parallax map including the parallax for all the points can be obtained.

The parameter calculation unit 103 calculates the plane parameters α and β using the parallax map calculated by the parallax calculation unit 102. First, a method for obtaining the position (X, Y, Z) of the point in the three-dimensional space from the parallax d of the point (x, y) on the reference image will be described. The following relational expression (5) is established between the point (X, Y, Z) in the space and (x ′, y) and (x, y) which are the projected images on the left and right images.

When the equation (5) is solved for X, Y, and Z, the following equation (6) is obtained.

  The parameter calculation unit 103 obtains the position (three-dimensional position) in the three-dimensional space of the point on the reference image from which the parallax is obtained by the parallax calculation unit 102 using each of the above equations. And the parameter calculation part 103 selects the point with the near distance with a road surface among the calculated | required three-dimensional positions, and substitutes in said (1) Formula (Y = (alpha) Z + (beta)) which is an equation of a road surface. Then, α and β which are surface parameters are calculated.

The parameter calculation unit 103 extracts a point that is close to the road surface as a point that satisfies the following expression (7).

Here, ΔY is a threshold value, and an appropriate value is set in advance. Y _p represents the point (X, Y, Z) Y coordinate of the intersection between a straight line parallel to the reference road surface in the Y-axis through. The reference road surface refers to a road surface that is a predetermined reference such as a flat road. The parameters of the reference road surface are measured on a flat road when the moving body is stationary, for example. If the parameters of the reference road surface are α ′ and β ′, Y _p is given by the following equation (7-1).
Y _p = α′Z + β ′ (7-1)

  Returning to FIG. 3, the candidate rectangle generation unit 104 generates a candidate rectangle that is a rectangle candidate for detecting a pedestrian in each image of the stereo image. The candidate rectangle generation unit 104 includes an area calculation unit 104a, a first generation unit 104b, and a second generation unit 104c.

  The area calculation unit 104a divides each image into a monocular processing area and a stereo processing area. The stereo processing area means an area on each image corresponding to the common visual field range of the left and right cameras. That is, the stereo processing area is an area (common area) in which corresponding points are included in both the left and right images. The monocular processing area means an area (non-common area) on each image that is outside the common visual field range of the left and right cameras.

  First, details of these two areas will be described. In addition, although the shape of the candidate area | region which detects a pedestrian is arbitrary, below, a rectangle is demonstrated to an example. For convenience of explanation, an image coordinate system having the origin at the upper left of the image will be described below as an example. However, a region can be calculated by a similar method even in other coordinate systems.

If the point (x, y) on the reference image is a point on the road plane Y = αZ + β, the parallax d _p of the point (x, y) is given by the following equation (8) from the above equation (5). It is done.

Here, α and β use the surface parameters calculated by the parameter calculation unit 103. If it is assumed that all objects are present above the road surface (camera side), since the object is present in front of the road plane as observed from the camera, the parallax d _p represents the lower limit (minimum value) of the parallax d. To express. On the other hand, since d is non-negative according to the above equation (5), the following constraint equation (9) holds for the parallax d.

Therefore, in the range of y ≧ α, the following expression (10) holds for the right image.

Here, x ′ _max represents the maximum value of x ′, that is, the horizontal width of the image. Similarly, for the left image, since the minimum value of x is 0, an inequality such as the following expression (11) holds.

  The area calculation unit 104a calculates an area that does not satisfy these conditions on each of the left and right images as a monocular processing area that does not have a corresponding point and that is outside the common visual field range of the left and right cameras. The area calculation unit 104a calculates an area other than the monocular processing area as a stereo processing area.

  FIG. 7 is a diagram illustrating an example of a monocular processing region for each of the left and right images. In the left image, a monocular processing area 701a exists at the lower left of the image, and in the right image, a monocular processing area 701b exists at the lower right of the image.

  Returning to FIG. 3, the candidate rectangle generation unit 104 uses the first generation unit 104 b and the second generation unit 104 c to generate candidate rectangles using different methods for the stereo processing region and the monocular processing region. Specifically, the first generation unit 104b generates a candidate rectangle in the stereo processing area. The second generation unit 104c generates a candidate rectangle in the monocular processing area. First, a method for generating candidate rectangles in the stereo processing area by the first generation unit 104b will be described.

  FIG. 8 is a diagram illustrating an example of a candidate rectangle generated in the stereo processing area. The first generation unit 104b first sets a rectangle having an arbitrary point (x, y) in the stereo processing area with the lower side as a midpoint. The first generation unit 104b uses the surface parameters α and β to set a rectangle whose bottom side is in contact with the road surface and whose size is determined in consideration of the vertical and horizontal sizes of a person.

For example, when the representative values of the human height and the horizontal width are H and W, respectively, the height h and the width w on the rectangular image can be obtained as follows. That is, since the following equation (12) is established from the equation (1) (Y = αZ + β) and the equation (5) that are road surface equations, the following equation (13) is obtained.

  As described above, the size of the rectangular image generated by the first generation unit 104b varies depending on the vertical position on the image, that is, the y coordinate. The first generation unit 104b generates a plurality of types of rectangles for each point (x, y) on the image as illustrated in FIG. FIG. 8 shows an example in which three types of rectangles are set.

  Next, the first generation unit 104b evaluates the possibility that a person (pedestrian) is included in the rectangle from the parallax in the rectangle set to each point in this way, and selects a rectangle having a high possibility as a candidate rectangle. Generate as Specifically, the first generation unit 104b evaluates a rectangle having a degree of uniformity greater than a predetermined threshold representing a degree of parallax included in the rectangle as a rectangle having a high possibility of including a person. FIG. 9 is a schematic diagram for explaining a rectangular evaluation method.

As shown in FIG. 9A, when a person is included in a rectangle 901, since the depth in the rectangle is substantially uniform, the corresponding parallax is also uniform. Further, since the point (x, y) is in contact with the road surface, the parallax at this point is given by the above equation (8). Therefore, if the parallax at an arbitrary point in the rectangle is d _i , the first generation unit 104b can evaluate the uniformity of the parallax with the number N of points that satisfy the following expression (14). Here, Δd represents a predetermined parallax difference threshold (difference threshold).

In order to consider the influence of the size of the rectangle, the first generation unit 104b normalizes the number N with the area S = w × h of the rectangle, and the normalized number N satisfies the following formula (15) as candidates. Register as a rectangle.

Here, N _min is a threshold value, and an appropriate value is set in advance. Note that, as shown in FIG. 9B, even in a rectangle 902 smaller than an object to be detected (a person such as a pedestrian), the parallax in the rectangle becomes uniform, and the above equation (15) may be satisfied. In order to prevent such a rectangle from being a candidate, when the first generation unit 104b satisfies the above equation (15) for a plurality of rectangles for a certain point, only the rectangle having the largest area is set as a candidate rectangle. You may comprise so that it may select. For example, when a rectangle 901 and a rectangle 902 are candidate rectangles as shown in FIG. 9B, only the rectangle 901 may be selected as a candidate rectangle.

  Moreover, the evaluation method of the said uniformity is an example, and any evaluation method can be applied if it can evaluate the uniformity of the parallax within a rectangle. Moreover, you may comprise so that the uniformity which represents that a degree of uniformity is so large that a value is small may be used.

  Next, a method for generating candidate rectangles in the monocular processing area by the second generation unit 104c will be described. Similar to the processing in the stereo processing region, the second generation unit 104c uses the surface parameters α and β calculated by the parameter calculation unit 103 for any point (x, y), and uses the pedestrian size candidates. Generate a rectangle. Specifically, the second generation unit 104c uses the surface parameters α and β to determine, as candidate rectangles, rectangles whose bottom sides are in contact with the road surface and whose sizes are determined in consideration of human vertical and horizontal sizes. Generate.

  FIG. 10 is a diagram illustrating an example of a candidate rectangle generated in a monocular processing region on each of the left and right images. FIG. 11 is a diagram illustrating the position of the generated candidate rectangular stereo camera in real space. For example, FIGS. 10 and 11 show correspondences between the midpoints 1001 to 1004 of the lower side of the candidate rectangle generated in the left image and the midpoints 1011 to 1014 of the lower side of the candidate rectangle generated in the right image. .

  The width of each rectangle of the monocular processing areas 701a and 701b corresponds to the camera interval B when converted into a real space. Here, the case where the camera interval B is substantially equal to or smaller than the size in the horizontal direction of a human is illustrated. Moreover, although the case where one rectangle is generated for each point is illustrated, the second generation unit 104c generates a plurality of types of rectangles for each point in order to correspond to various sizes of a person. May be.

  Returning to FIG. 3, the determination unit 105 determines whether or not the generated candidate rectangle includes a pedestrian (person) to be detected, and classifies the candidate rectangle into a rectangle including a person and a rectangle not including a person. For example, the determination unit 105 uses a classifier that has been obtained in advance through learning so that a person and a person other than a person (non-person) can be identified. That is, a large number of luminance patterns including a person and luminance patterns not including a person are prepared as sample images, and an identification function that can correctly identify them is calculated. The learning method is arbitrary, but AdaBoost, Support Vector Machine, etc. are relatively effective.

  The output unit 106 outputs the rectangular position determined by the determination unit 105 as including a person.

  Next, an object detection process performed by the object detection apparatus 100 according to the present embodiment configured as described above will be described with reference to FIG. FIG. 12 is a flowchart showing the overall flow of the object detection process in the present embodiment.

  First, the image input unit 101 inputs left and right images captured by the cameras 11a and 11b constituting the stereo camera (step S1201). Next, the parallax calculation unit 102 obtains corresponding points between the input left image and right image, calculates the parallax for each point, and creates a parallax map (step S1202). Next, the parameter calculation unit 103 calculates the plane parameters α and β of the road plane using the parallax map (step S1203).

  Next, the region calculation unit 104a of the candidate rectangle generation unit 104 calculates a monocular processing region and a stereo processing region for each of the left and right images using the calculated parallax (step S1204).

  Next, the first generation unit 104b of the candidate rectangle generation unit 104 generates a candidate rectangle for the stereo processing area (step S1205). Further, the second generation unit 104c of the candidate rectangle generation unit 104 generates a candidate rectangle for the monocular processing region (step S1206).

  Next, the determination unit 105 determines whether or not a person is included in each candidate rectangle generated by the first generation unit 104b and the second generation unit 104c (step S1207). Then, the output unit 106 outputs information such as the position of a rectangle determined to include a person (step S1208), and ends the object detection process.

  As described above, the conventional object detection method using only stereo vision cannot detect an object existing outside the common visual field of the left and right cameras. That is, there is a problem that the visual field range for object detection is narrower than the range actually captured by each camera. On the other hand, when only monocular vision is used, the object can be detected within the actually captured range, but the candidate area cannot be narrowed down using surface parameters or the like as in the method using stereo vision. It could not be detected with accuracy.

  On the other hand, the object detection apparatus of the present embodiment makes use of the monocular vision and the stereo vision by the above-described processing, thereby compensating for the disadvantages when both are used alone, and adopting the monocular vision system. Compared with the stereo vision method, it is possible to detect an object in a wider visual field range with higher accuracy. That is, according to the present embodiment, it is possible to detect an object in a wide range not only for the stereo processing area but also for the monocular processing area. Also, candidate areas can be narrowed down using monochromatic processing areas using surface parameters, so that an object can be detected with high accuracy.

  In this embodiment, the case where a pedestrian existing in the traveling direction is detected using a camera installed in front of the moving body has been described. However, a camera may be installed on the side or rear of the moving body. . Further, the present invention can also be applied when a camera is mounted on a moving body other than a moving body such as a mobile robot. Further, the present invention is not limited to the case where a camera is installed on a moving body, and can also be applied to a case where an object is detected from an image captured by a monitoring camera installed at a fixed position.

  Moreover, although the case where a person (pedestrian) was detected was demonstrated, a detection target object is not limited to this, Even if it is another detection target, it is applicable. Further, the present invention can also be applied when detecting multi-class objects such as simultaneous detection of a person and a moving object.

  In addition, the stereo view when two cameras are arranged in parallel on the left and right has been described. However, the number of cameras is arbitrary as long as there are two or more cameras. You may arrange as follows.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined. In addition, modifications can be made without departing from the scope of the present invention.

  Below, the specific example of a deformation | transformation is demonstrated.

(Modification 1)
In the first modification, the number of candidate rectangles generated in the monocular processing region is increased or decreased according to the traveling direction of the moving object. FIG. 13 is a block diagram illustrating a configuration of an object detection device 1300 according to the first modification of the present embodiment. The object detection apparatus 1300 includes an image input unit 101, a parallax calculation unit 102, a parameter calculation unit 103, a candidate rectangle generation unit 1304, a determination unit 105, an output unit 106, and a steering angle detection unit 1307. ing. Note that the image input unit 101 may be disposed outside the object detection device 1300 and connected to the object detection device 1300.

  In the first modification, the addition of the steering angle detection unit 1307 and the function of the second generation unit 1304c of the candidate rectangle generation unit 1304 are different from those in the above embodiment. Other configurations and functions are the same as those in FIG. 3, which is a block diagram showing the configuration of the object detection apparatus 100, and thus are denoted by the same reference numerals and description thereof is omitted here.

  The steering angle detection unit 1307 is a sensor that detects the steering angle of the moving body on which the stereo camera is mounted. The second generation unit 1304c estimates the traveling direction of the moving body from the detected steering angle, and more candidate rectangles than the monocular processing region existing in the direction other than the traveling direction with respect to the monocular processing region existing in the traveling direction. Is different from the second generation unit 104c.

  For example, when the moving body turns to the right, the moving body goes to the monocular processing area (hereinafter referred to as “right end area”) of the right image. The risk of collision is higher than On the other hand, since the moving body moves away from the monocular processing area of the left image (hereinafter referred to as “left end area”), the possibility of colliding with a pedestrian existing in this area is lower than that when the moving body is going straight. Become. Therefore, the second generation unit 1304c increases the candidate rectangles in the right end region and decreases the candidate rectangles in the left end region when the mobile object turns right. Conversely, when making a left turn, the second generation unit 1304c increases the candidate rectangles in the left end region and decreases the candidate rectangles in the right end region. By increasing or decreasing the number of candidate rectangles using the steering angle in this way, more efficient object detection processing can be performed.

(Modification 2)
In the second modification, a candidate rectangle is generated using the correspondence between temporally continuous images (frames). FIG. 14 is a block diagram showing a configuration of an object detection apparatus 1400 according to the second modification of the present embodiment. The object detection device 1400 includes an image input unit 101, a parallax calculation unit 102, a parameter calculation unit 103, a candidate rectangle generation unit 1404, a determination unit 1405, and an output unit 106. The image input unit 101 may be disposed outside the object detection device 1400 and connected to the object detection device 1400.

  In the second modification, the function of the second generation unit 1404c of the candidate rectangle generation unit 1404 is different from that of the above embodiment. Other configurations and functions are the same as those in FIG. 3, which is a block diagram showing the configuration of the object detection apparatus 100, and thus are denoted by the same reference numerals and description thereof is omitted here.

  The second generation unit 1404c sets a rectangle corresponding to the candidate rectangle generated for the frame input at the previous time among the images (frames) input in time series by the image input unit 101 to the frame at the current time. The obtained rectangle is generated as a candidate rectangle.

  FIG. 15 is a diagram illustrating an example of a candidate rectangle generation method according to the second modification. FIG. 15A corresponds to the right image, and FIG. 15B corresponds to the left image. As shown in FIGS. 15A and 15B, at time t-1, the pedestrian is present in the stereo processing area and is reflected in both the left and right images. For this reason, a candidate area | region can be produced | generated using the parallax by the 1st production | generation part 104b. However, at the next time t, the pedestrian moves out of the angle of view on the left image and moves to the monocular processing region on the right image. For this reason, a candidate area cannot be generated using stereo parallax at time t. Therefore, in this case, the second generation unit 1404c generates a candidate rectangle in the monocular processing region.

  In the present modification, the second generation unit 1404c extracts a rectangle corresponding to the candidate rectangle generated at the previous time t-1 from the frame at the time t, and generates the extracted rectangle as a candidate rectangle. For example, the second generation unit 1404c extracts, as an associated rectangle, a rectangle having the highest degree of similarity with the candidate rectangle generated at time t-1 among arbitrary rectangles in the frame at time t.

  By such processing, a more accurate candidate rectangle can be generated for the monocular processing area of the current frame using the candidate rectangle of the previous frame. In addition, you may comprise so that the number of candidate rectangles may be decreased by not producing | generating a candidate rectangle in the area | region shielded by the extracted candidate rectangle. This is because if only the person closest to the moving body can be detected, it is not necessary to detect the person existing behind the person. In this case, it may be configured not to generate candidate rectangles in a completely shielded area, or may be configured to exclude candidate rectangles shielded more than a predetermined ratio.

(Modification 3)
In the third modification, the number of candidate rectangles in the monocular processing region is increased or decreased according to the determination result of person detection for the candidate rectangles generated in the stereo processing region. FIG. 16 is a block diagram showing a configuration of an object detection apparatus 1500 according to the third modification of the present embodiment. The object detection device 1500 includes an image input unit 101, a parallax calculation unit 102, a parameter calculation unit 103, a candidate rectangle generation unit 1504, a determination unit 1505, and an output unit 106. The image input unit 101 may be disposed outside the object detection device 1500 and connected to the object detection device 1500.

  In Modification 3, the function of the second generation unit 1504c of the candidate rectangle generation unit 1504 and the function of the determination unit 1505 are different from those of the above embodiment. Other configurations and functions are the same as those in FIG. 3, which is a block diagram showing the configuration of the object detection apparatus 100, and thus are denoted by the same reference numerals and description thereof is omitted here.

  The determination unit 1505 is different from the determination unit 105 in that the determination unit 1505 first determines the presence or absence of a person with respect to the candidate rectangle generated by the first generation unit 104b.

  When the determination unit 1505 determines that the candidate rectangle generated by the first generation unit includes a person, the second generation unit 1504c reduces the number of candidate rectangles generated in the monocular processing region. For example, if a pedestrian is detected at a position corresponding to the position immediately before the moving object in the stereo processing area when the moving object is moving straight, even if there are other pedestrians in the monocular processing area at both ends It can be considered that the sex is low compared to pedestrians in the stereo processing area. For this reason, the second generation unit 1504c is configured to reduce the number of candidate rectangles in the monocular processing region or not generate candidate rectangles.

  Next, the hardware configuration of the object detection apparatus according to the present embodiment will be described with reference to FIG. FIG. 17 is an explanatory diagram illustrating a hardware configuration of the object detection device according to the present embodiment.

  The object detection device according to the present embodiment is connected to a control device such as a CPU (Central Processing Unit) 51, a storage device such as a ROM (Read Only Memory) 52 and a RAM (Random Access Memory) 53, and a network. A communication I / F 54 that performs communication and a bus 61 that connects each unit are provided.

  An object detection program executed by the object detection apparatus according to the present embodiment is provided by being incorporated in advance in the ROM 52 or the like.

  The object detection program executed by the object detection apparatus according to the present embodiment is a file in an installable format or an executable format, and is a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R. (Compact Disk Recordable), DVD (Digital Versatile Disk), and the like may be provided by being recorded on a computer-readable recording medium.

  Furthermore, the object detection program executed by the object detection apparatus according to the present embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. . The object detection program executed by the object detection apparatus according to the present embodiment may be configured to be provided or distributed via a network such as the Internet.

  The object detection program executed by the object detection device according to the present embodiment includes a module configuration including the above-described units (image input unit, parallax calculation unit, parameter calculation unit, candidate rectangle generation unit, determination unit, output unit) As the actual hardware, the CPU 51 reads out and executes the object detection program from the ROM 52, so that the respective units are loaded onto the main storage device, and the respective units are generated on the main storage device. Yes.

It is a figure which shows an example of the common visual field of a stereo camera. It is a figure which shows an example of the image image | photographed when a pedestrian enters into the advancing direction of a moving body from the right side. It is a block diagram which shows the structure of the object detection apparatus which concerns on this Embodiment. It is a figure which shows the relationship between an object detection apparatus and the object to detect. It is a figure which shows an example of the stereo camera coordinate system used by this Embodiment. It is a schematic diagram for demonstrating an example of the calculation method of parallax. It is a figure which shows an example of the monocular process area | region of each image on either side. It is a figure which shows an example of the candidate rectangle produced | generated in a stereo process area | region. It is a schematic diagram for demonstrating the evaluation method of a rectangle. It is a figure which shows the example of the candidate rectangle produced | generated by the monocular process area | region on each image on either side. It is a figure which shows the position in the real space of the produced | generated candidate rectangle stereo camera. It is a flowchart which shows the whole flow of the object detection process in this Embodiment. It is a block diagram which shows the structure of the object detection apparatus which concerns on the modification 1 of this Embodiment. It is a block diagram which shows the structure of the object detection apparatus which concerns on the modification 2 of this Embodiment. It is a figure which shows an example of the production | generation method of the candidate rectangle in the modification 2. It is a block diagram which shows the structure of the object detection apparatus which concerns on the modification 3 of this Embodiment. It is explanatory drawing which shows the hardware constitutions of the object detection apparatus which concerns on this Embodiment.

Explanation of symbols

102 parallax calculation unit 103 parameter calculation unit 104a region calculation unit 104b first generation unit 104c second generation unit 105 determination unit

Claims (11)

An object detection device for detecting an object existing on a reference plane,
A parallax calculator that calculates parallax between images captured by each of a plurality of imaging devices;
An area calculation unit that calculates a common area of the plurality of images and a non-common area other than the common area based on the parallax;
A rectangle in the common area, in which the uniformity of the parallax at each point included in the rectangle is greater than a predetermined threshold, is generated as a candidate rectangle that is a candidate for detecting the object. A first generator that
A second generation unit that generates a rectangle in the non-common area as the candidate rectangle;
A determination unit that determines whether or not the generated object rectangle includes an object to be detected;
An object detection apparatus comprising:   The uniformity is determined in advance by a difference between the parallax with respect to a point on the reference plane included in a rectangle and the parallax calculated for each point that does not exist on the reference plane included in the rectangle. The object detection apparatus according to claim 1, wherein the number of points is smaller than the difference threshold. A steering angle detector that detects a steering angle of a moving body on which the plurality of imaging devices are mounted;
The second generation unit estimates a traveling direction of the mobile body based on the steering angle, and generates the candidate rectangle generated for the non-common region existing in the estimated traveling direction among the non-common regions. The object detection apparatus according to claim 1, wherein the number is increased from the number of the candidate rectangles generated in the non-common area existing in a direction different from the estimated traveling direction. The plurality of images are time-series images;
The second generation unit corresponds to the candidate rectangles generated from the plurality of images captured at the second time before the first time with respect to the plurality of images captured at the first time. The object detection apparatus according to claim 1, wherein a rectangle on the reference plane of the non-common area is generated as the candidate rectangle.   The first generation unit is a rectangle on the reference plane of the common area, and the uniformity of the parallax of each point included in the rectangle on the reference plane of the common area is greater than a predetermined threshold value The object detection apparatus according to claim 1, wherein a rectangle having the largest area among the rectangles is generated as the candidate rectangle.   The first generation unit is a rectangle having a predetermined size with each point on the reference plane of the common area as a midpoint of a lower side, and is included in the rectangle on the reference plane of the common area The object detection apparatus according to claim 1, wherein a rectangle having a parallax uniformity at each point larger than a predetermined threshold is generated as the candidate rectangle.   The said 2nd production | generation part produces | generates the rectangle of the predetermined magnitude | size which makes each point on the said reference plane of the said non-common area | region the midpoint of a lower side as said candidate rectangle, The said rectangle is characterized by the above-mentioned. The object detection apparatus described.   The said 2nd production | generation part produces | generates the rectangle of several magnitude | size which makes each point on the said reference planes of the said non-common area | region the midpoint of a lower side as said candidate rectangle. Object detection device.   The second generation unit is generated by the determination unit by the first generation unit when the determination unit determines that an object to be detected is included in the candidate rectangle generated by the first generation unit. The object detection apparatus according to claim 1, wherein more candidate rectangles are generated than when the candidate rectangle does not include an object to be detected. An object detection method for detecting an object existing on a reference plane,
A parallax calculating step in which a parallax calculating unit calculates parallax between images captured by each of the plurality of imaging devices;
An area calculation step in which an area calculation unit calculates a common area of the plurality of images and a non-common area other than the common area based on the parallax;
Rectangle candidates for detecting the object by the first generation unit that is a rectangle in the common area and in which the parallax uniformity of each point included in the rectangle is larger than a predetermined threshold value A first generation step for generating as a candidate rectangle,
A second generation step in which a second generation unit generates a rectangle in the non-common area as the candidate rectangle;
A determination step for determining whether the determination unit includes an object to be detected in the generated candidate rectangle;
An object detection method comprising: Computer
A parallax calculator that calculates parallax between images captured by each of a plurality of imaging devices;
An area calculation unit that calculates a common area of the plurality of images and a non-common area other than the common area based on the parallax;
A rectangle in the common area, in which the uniformity of the parallax at each point included in the rectangle is greater than a predetermined threshold, is generated as a candidate rectangle that is a candidate for detecting the object. A first generator that
A second generation unit that generates a rectangle in the non-common area as the candidate rectangle;
A determination unit that determines whether or not the generated object rectangle includes an object to be detected;
Object detection program to function as

Images