JP2018185694A - Moving object determination device, and moving object determination program - Google Patents

Abstract

An object of the present invention is to accurately determine whether an object included in a captured image is a moving object regardless of a positional relationship between a host vehicle and surrounding objects. A moving object determination apparatus includes an image feature extraction unit that extracts a feature point from an image, and a first optical flow calculation unit that calculates an optical flow indicating the extracted feature point for each feature point. A vehicle trajectory calculation unit 50 that calculates a travel trajectory of the host vehicle 8, a feature point three-dimensional position calculation unit 60 that calculates a spatial feature point 14 for each feature point 9 from the optical flow and the travel trajectory of the host vehicle 8, The second optical flow calculator 70 calculates the optical flow of the spatial feature point 14 corresponding to the feature point 9 for each feature point 9 using the travel locus of the host vehicle 8 and the spatial feature point 14, and the feature point 9. The moving object determining unit 80 that compares the optical flow with the optical flow of the spatial feature point 9 and determines whether the extracted optical flow is the optical flow of the moving object. Provided. [Selection diagram] Fig. 1

Description

  The present invention relates to a moving object determination device and a moving object determination program.

  Conventionally, a method has been proposed in which an object is photographed with a camera and whether the object contained in the photographed image is a moving object or a stationary object is determined (see, for example, Patent Document 1).

JP 2010-286963 A

  In the moving object detection device described in Patent Document 1, a feature point included in a predetermined area at the bottom corresponding to a road surface of an image is selected as a reference feature point belonging to the road surface structure, and is automatically based on the selected reference feature point. A reference attitude change amount of the vehicle is calculated. Then, an arbitrary feature point is selected as a comparison feature point from the remaining feature points excluding the reference feature point in the image, and the comparison posture change amount of the host vehicle is calculated based on the selected comparison feature point. The moving object detection device maps the reference posture change amount and the comparison posture change amount to a three-dimensional coordinate, and when the coordinate position of the reference posture change amount and the coordinate position of the comparison posture change amount are separated by a distance r or more, the comparative posture The comparison feature point used to calculate the amount of change is determined as a feature point associated with the moving object.

  However, in the moving object detection device described in Patent Document 1, since the determined range of the image is set to an area corresponding to the road surface, the set area may not necessarily be the road surface depending on the situation. For example, when a camera is attached in front of the host vehicle, when the vehicle traveling in front approaches the host vehicle, the vehicle traveling in front is shown in the predetermined area at the bottom of the captured image. There is. In this case, since the reference feature point selected from the predetermined region is not a feature point of the stationary object, the precondition of the determination method of selecting the reference feature point from the stationary object is not satisfied. Therefore, it is difficult for the moving object detection device described in Patent Document 1 to detect a moving object.

  In addition, depending on the mounting position of the camera and the resolution of the camera to be used, it may be difficult to select feature points on the road surface with high accuracy. Even if the feature points on the road surface can be selected with high accuracy, the feature points in a limited range of the road surface often have similar features, and there may be little variation in the feature points. As described above, since the moving object detection device described in Patent Document 1 detects a moving object using a distribution of biased feature points, the reference posture change amount of the host vehicle may not be accurately calculated. In addition, if the reference attitude change amount of the host vehicle cannot be calculated with high accuracy, the detection accuracy of the moving object also decreases.

  The present invention has been made in view of the problems described above, and accurately determines whether or not an object included in a captured image is a moving object regardless of the positional relationship between the host vehicle and surrounding objects. An object of the present invention is to provide a moving object determination device.

  In order to achieve the above object, the moving object determination device according to claim 1 includes an extraction unit that extracts a feature point from each image photographed in time series by a photographing device attached to a moving body. A first calculation unit that tracks a corresponding feature point in each image among the feature points extracted by the extraction unit, and calculates a time series of the position of the tracked feature point for each feature point; A second calculation unit that calculates a travel locus of the moving body using a physical quantity related to the moving speed and motion of the moving body, a time series of the positions of feature points calculated by the first calculation unit, and the second Using the travel trajectory of the moving body calculated by the calculation unit, a third calculation unit that calculates the position of the extracted feature point in the three-dimensional space for each feature point, and the second calculation unit The travel locus of the mobile body and the calculated by the third calculation unit Using the position of the spatial feature point representing the feature point in the dimensional space, the position of the spatial feature point in each of the images is calculated, and the feature point corresponding to the spatial feature point represented in each of the images A fourth calculation unit that calculates a time series of positions for each feature point corresponding to the spatial feature point, a time series of feature point positions calculated by the first calculation unit, and a corresponding feature point A determination unit that compares the time series of the positions of the spatial feature points calculated by the fourth calculation unit to determine, for each feature point, whether or not the object represented by the feature point is a moving object.

  According to a second aspect of the present invention, the image processing apparatus includes a detection unit that detects a moving object region from each of the images, and the determination unit includes a feature point determined to represent a moving object. The feature points that are not included in any of the moving object regions detected from the above are re-determined not to be feature points representing a moving object.

  According to a third aspect of the present invention, the image processing apparatus includes a detection unit that detects a moving object region from any one of the images, and the determination unit performs the detection for each moving object region detected by the detection unit. The ratio of the feature point representing the moving object to the feature point included in the moving object area is calculated, and the feature point included in the moving object area where the ratio is less than a predetermined ratio is not a feature point representing the moving object. And the feature point included in the moving object region in which the ratio is equal to or greater than a predetermined ratio is determined again as a feature point representing a moving object.

  According to a fourth aspect of the present invention, the determination unit determines that a feature point that has been determined to represent the moving object region in the past is a feature point that represents a moving object.

  In the invention according to claim 5, the determination unit obtains the characteristic obtained in the previous determination as the time series of the position of the feature point when the variation degree of the physical quantity related to the motion of the moving body is larger than a predetermined threshold. Using the time series of point positions, it is determined for each feature point whether the object represented by the feature point is a moving object.

  In the invention according to claim 6, the determination unit includes a time series of the position of the feature point calculated by the first calculation unit and the spatial feature calculated by the fourth calculation unit corresponding to the feature point. Whether or not the object represented by the feature point is a moving object is determined for each feature point using the distance from the point position in time series.

  The moving object determination program according to claim 7, wherein a computer is extracted by each of an extraction unit that extracts a feature point from each image photographed in time series by a photographing device attached to a moving body, and the extraction unit extracts A first calculation unit that tracks a corresponding feature point in each of the images and calculates a time series of the position of the tracked feature point for each feature point; a moving speed of the moving object; Using a physical quantity related to the moving direction, a second calculation unit that calculates a travel locus of the moving body, a time series of the positions of feature points calculated by the first calculation unit, and a calculation by the second calculation unit A third calculation unit that calculates the position of the extracted feature point in the three-dimensional space for each feature point using the travel locus of the mobile unit, and the travel of the mobile unit calculated by the second calculation unit The trajectory and the 3 calculated by the third calculation unit Using the position of the spatial feature point representing the feature point in the original space, the position of the spatial feature point in each of the images is calculated, and the feature point corresponding to the spatial feature point represented in each of the images is calculated. A fourth calculation unit that calculates a time series of positions for each feature point corresponding to the spatial feature point, a time series of feature point positions calculated by the first calculation unit, and a corresponding feature point The time series of the position of the spatial feature point calculated by the fourth calculation unit is compared to function as a determination unit that determines for each feature point whether or not the object represented by the feature point is a moving object.

  According to the present invention, it is possible to accurately determine whether or not an object included in a captured image is a moving object regardless of the positional relationship between the host vehicle and surrounding objects.

It is a figure which shows the structural example of the moving object determination apparatus which concerns on 1st Embodiment. It is the schematic diagram explaining the calculation method of the three-dimensional position of a feature point. It is the schematic diagram explaining the reprojection of the feature point. It is the schematic diagram explaining the determination method of the locus | trajectory of a mobile body. It is a figure which shows the principal part structural example of the electric system of a moving object determination apparatus. It is a flowchart which shows an example of the flow of the moving object determination process which concerns on 1st Embodiment. It is a figure which shows an example of an optical flow. It is a figure which shows an example of the determination result of a moving object. It is a figure which shows the structural example of the moving object determination apparatus which concerns on 2nd Embodiment. It is a flowchart which shows an example of the flow of the moving object determination process which concerns on 2nd Embodiment. It is a figure which shows an example of a vehicle image area | region. It is a figure which shows an example of the determination result of a moving object. It is a figure which shows the structural example of the moving object determination apparatus which concerns on 3rd Embodiment. It is a flowchart which shows an example of the flow of the moving object determination process which concerns on 3rd Embodiment. It is a figure which shows the structural example of the moving object determination apparatus which concerns on 4th Embodiment. It is a flowchart which shows an example of the flow of the moving object determination process which concerns on 4th Embodiment. It is a flowchart which shows an example of the flow of a moving vehicle area | region determination process. It is a figure which shows an example of the determination result of a vehicle image area. It is a figure which shows an example of the determination result of a moving object.

  Hereinafter, the present embodiment will be described with reference to the drawings. In addition, the same code | symbol is provided to the component and process with the same function throughout all the drawings, and the overlapping description is abbreviate | omitted.

<First Embodiment>
FIG. 1 is a diagram illustrating a configuration example of a moving object determination device 1 according to the first embodiment. As illustrated in FIG. 1, the moving object determination device 1 includes an image information acquisition unit 10, an image feature extraction unit 20, a first optical flow calculation unit 30, an IMU information acquisition unit 40, and a vehicle trajectory calculation unit 50. Furthermore, the moving object determination device 1 includes a feature point three-dimensional position calculation unit 60, a second optical flow calculation unit 70, and a moving object determination unit 80.

  The image information acquisition unit 10 is a periphery of the own vehicle, which is taken by a camera (not shown) that is an example of an imaging device attached to a vehicle (hereinafter, may be referred to as “own vehicle”) on which the moving object determination device 1 is mounted. Get the image.

  A camera (not shown) captures images in time series and stores the captured images as image data in a storage device such as a RAM described later. Note that the form of the image acquired by the image information acquisition unit 10 may be a still image or a moving image, and if the image content around the host vehicle can be analyzed, a visible image, an infrared image, and Any of the distance images may be used. Moreover, the image information acquisition part 10 may acquire an image from one camera attached to the own vehicle, or may acquire an image from several cameras.

  Furthermore, there is no restriction | limiting in the attachment position of the camera in the own vehicle, For example, a camera can be attached to the position which image | photographs the image of the omnidirectional centering on a vehicle advancing direction, a vehicle advancing direction, or the own vehicle. The host vehicle is an example of a moving body.

  The image feature extraction unit 20 extracts, from each of the images acquired by the image information acquisition unit 10, a plurality of feature points indicating a characteristic pixel pattern different from other regions in the image.

  A known method is used as the feature point extraction method in the image feature extraction unit 20, and there is no restriction on the feature point extraction method to be applied. The image feature extraction unit 20 is a SIFT (Scale-Invariant Feature Transform) method having an internal property (robustness) that attempts to exclude the influence of external factors such as enlargement, rotation, and brightness change in feature point extraction. , And SURF (Speed Up Robust Feature) method can be used. Further, the image feature extraction unit 20 can use a Harris method that is easier to implement than other feature point extraction methods and a FAST (Features from Accelerated Segment Test) method that requires a small amount of calculation.

  Note that the image feature extraction unit 20 may extract a line segment (feature line segment) or a surface (feature plane) indicating an image feature instead of or together with feature point extraction. As an example, it is assumed that the image feature extraction unit 20 extracts feature points. The image feature extraction unit 20 is an example of an extraction unit.

  The first optical flow calculation unit 30 tracks each feature point that is extracted by the image feature extraction unit 20 and moves with time as the host vehicle travels, and indicates a feature point that indicates the same location (also a corresponding feature point). The optical flow expressed as a time series of positions) is calculated for each feature point. For tracking corresponding feature points, a known method such as Lucas-Kanade method can be used, and the feature point tracking method is not particularly limited. The first optical flow calculation unit 30 is an example of a first calculation unit. Hereinafter, the optical flow may be abbreviated as “OF (Optical Flow)”. In addition, when “optical flow” is simply described, it indicates an optical flow obtained by tracking the feature points 9 extracted by the image feature extraction unit 20.

  The IMU information acquisition unit 40 acquires, from a vehicle speed sensor and a gyro sensor (not shown) attached to the host vehicle, a vehicle speed that represents the moving speed of the host vehicle and an angular velocity that represents the motion of the host vehicle in time series.

  The gyro sensor measures respective angular velocities in the pitch direction, the roll direction, and the yaw direction.

  Although there is no restriction on the acquisition interval of the vehicle speed and angular velocity of the host vehicle by the IMU information acquisition unit 40, the movement state of the host vehicle can be analyzed in detail as the acquisition interval is shortened. In addition, there is no restriction | limiting also about the acquisition method of the vehicle speed and angular velocity of the own vehicle by the IMU information acquisition part 40, even if it acquires the vehicle speed and angular velocity of the own vehicle from the wiring connected to the vehicle speed sensor and the gyro sensor, respectively, it uses radio | wireless. May be obtained. Further, the IMU information acquisition unit 40 does not need to directly acquire the vehicle speed and the angular velocity of the host vehicle from the vehicle speed sensor and the gyro sensor, but is another device (for example, a sensor management device) that accumulates measurement values of the vehicle speed sensor and the gyro sensor. You may make it acquire from.

  Note that IMU is an abbreviation for Inertial Measurement Unit.

  The vehicle trajectory calculation unit 50 calculates the travel trajectory of the host vehicle using the vehicle speed and angular velocity of the host vehicle acquired by the IMU information acquisition unit 40.

If the vehicle speed and angular velocity measurement interval is Δt, the yaw direction, that is, the angular velocity in the azimuth angle direction is ω, the angular velocity in the pitch direction is ψ, the azimuth angle at time t is θ ^t , and the pitch angle at time t is φ ^t The azimuth angle θ ^{t + 1} and the pitch angle φ ^{t + 1} at ( ^{t + 1} ) are expressed by equations (1) and (2), respectively.

If the vehicle speed at time (t + 1) is V ^{t + 1} and the position of the host vehicle at time t is x _t = (X ^t , Y ^t , Z ^t ), the three-dimensional space of the host vehicle at time (t + 1). A position x _{t + 1} = (X ^{t + 1} , Y ^{t + 1} , Z ^{t + 1} ) in (real space) is expressed by equation (4).

  The vehicle trajectory calculation unit 50 sequentially calculates the position of the host vehicle at each time based on the formulas (1) to (3), and connects the calculated positions in time series, thereby Is calculated.

  Note that the initial values of the azimuth angle θ, the pitch angle φ, and the position of the host vehicle are “0”. The vehicle trajectory calculation unit 50 is an example of a second calculation unit.

  The feature point three-dimensional position calculation unit 60 uses the optical flow for each feature point calculated by the first optical flow calculation unit 30 and the travel locus of the host vehicle calculated by the vehicle locus calculation unit 50 to use the feature points in the real space. (The three-dimensional position of the feature point) is calculated for each feature point.

FIG. 2 is a schematic diagram illustrating a method for calculating a three-dimensional position of a feature point in the feature point three-dimensional position calculation unit 60. Note that the feature point three-dimensional position calculation unit 60 calculates the three-dimensional position of the feature point for each feature point. FIG. 2 illustrates a method for calculating the three-dimensional position of the feature point by focusing on one feature point. Yes. In FIG. 2, the optical flow at time t _{1 to} t ₄ and the travel locus of the host vehicle are used, but the time range used for calculating the three-dimensional position of the feature point is an example, and four times t ₁ to t _{1 are used} . it goes without saying that not limited to t _4.

The feature point three-dimensional position calculation unit 60 calculates the position of the image at the same time at which the position is represented by the optical flow from each position of the own vehicle 8 at each time t _{1 to} t ₄ represented by the travel locus of the own vehicle 8. A straight line 12 representing the line of sight when the feature point 9 is viewed is set in real space. The position where each straight line 12 set in the real space is most concentrated is set as a feature point position in the real space of the same feature point 9 represented by the optical flow. Hereinafter, the feature point in the real space corresponding to the feature point 9 is referred to as “spatial feature point 14”.

  Thus, the calculation for calculating the spatial feature point 14 from the feature point 9 included in the image at each time is called “back projection of the feature point 9”.

A three-dimensional rotation matrix having the position of the host vehicle 8 at the time t represented by x _t , the azimuth angle θ ^t , the pitch angle φ ^t , and the roll angle ρ ^t as an element at the time t is represented by R _t , If the position of the feature point m in the image at time t is X ^′ _{t, m} and the camera parameter is K, the three-dimensional position X _m of the spatial feature point 14 corresponding to the feature point _m is expressed by equation (4). The

  Here, “I” is a 3 × 3 unit square matrix, and “m” is an index of feature points respectively assigned to a plurality of feature points. The camera parameter is a parameter that represents the optical characteristics of a camera that captures an image, and includes elements such as focal length, pixel size, lens aberration, and image center.

If the angular velocity in the roll direction is ξ, the roll angle ρ ^{t + 1 at} time (t + 1) is expressed by equation (5).

Therefore, the three-dimensional rotation matrix R _t is obtained from the equations (1), (2), and (5). The feature point three-dimensional position calculation unit 60 is an example of a third calculation unit.

  The second optical flow calculation unit 70 uses the travel trajectory of the host vehicle 8 calculated by the vehicle trajectory calculation unit 50 and the three-dimensional position of the spatial feature point 14 calculated by the feature point three-dimensional position calculation unit 60. The position of the spatial feature point 14 is calculated for each image taken at each time. Then, the second optical flow calculation unit 70 calculates the time series of the positions of the feature points in the image corresponding to the spatial feature points 14, that is, the optical flow of the spatial feature points 14 for each spatial feature point 14.

  The calculation for calculating the position of the spatial feature point 14 corresponding to the feature point 9 in the image at each time is called “reprojection of the feature point 9”. Therefore, a feature point at a position corresponding to the spatial feature point 14 in each image may be referred to as a “reprojection feature point 16”.

FIG. 3 is a schematic diagram illustrating the reprojection of the feature point 9 in the second optical flow calculation unit 70. Note that the second optical flow calculation unit 70 calculates the position of the reprojection feature point 16 in the image for each feature point 9. In FIG. 3, the feature is focused on the spatial feature point 14 corresponding to one feature point 9. The reprojection of point 9 is described. Although calculates the optical flow of spatial feature point 14 at time t ₁ ~t ₄ in FIG. 3, the range of time for calculating the optical flow of the spatial feature point 14 is an example, the time t ₁ ~t ₄ It goes without saying that it is not limited to. However, in order to compare the optical flow and the optical flow of the corresponding spatial feature point 14 by the moving object determination unit 80 to be described later, the range of the optical flow of the spatial feature point 14 to be calculated is the feature point three-dimensional position calculation unit 60. It is preferable to match with the time range used when calculating the position of the spatial feature point 14.

The second optical flow calculation unit 70 sets a straight line 12 for each position of the host vehicle 8 at each time t _{1 to} t ₄ represented by the travel locus of the host vehicle 8 from the spatial feature point 14. There point (projected point) of each intersecting the image taken at the respective times _t 1 ~t _4, the position of the reprojection feature points 16 at each time _t 1 ~t _4. Then, the second optical flow calculation unit 70, by tracking the reprojection feature points 16 at each time t ₁ ~t ₄ in chronological obtain an optical flow of the spatial feature point 14.

Note that the position X ^″ _{t, m} of the reprojection feature point 16 at time t corresponding to the spatial feature point 14 indicated by the index m is expressed by equation (6).

  The second optical flow calculation unit 70 is an example of a fourth calculation unit.

  The moving object determination unit 80 calculates the optical flow calculated by the first optical flow calculation unit 30 and the optical flow of the spatial feature points 14 corresponding to the feature points 9 calculated by the second optical flow calculation unit 70 for each feature point 9. Compare to. Then, the moving object determination unit 80 determines for each feature point 9 whether or not the optical flow calculated by the first optical flow calculation unit 30 indicates the locus of the moving object.

  FIG. 4 is a schematic diagram for explaining a moving object trajectory determination method by the moving object determination unit 80.

  As shown in FIG. 4, the moving object determination unit 80 calculates the distance (deviation distance 18) between the optical flow 22 and the optical flow 24 of the spatial feature point 14 corresponding to the feature point 9 at each time.

  The moving object determination unit 80 calculates a divergence index value indicating the divergence degree between the optical flow 22 and the optical flow 24 of the spatial feature point 14 from the calculated divergence distance 18, and the divergence index value exceeds a predetermined divergence threshold value. Then, it is determined that the optical flow 22 is the locus of the moving object.

  As the divergence index value, for example, the variance value, standard deviation, average value, median value, maximum value, minimum value, and the like of the divergence distance 18 at each time can be used. Can be used. The divergence threshold is set to a value that allows the optical flow to be regarded as indicating the trajectory of the moving object when the divergence index value calculated from the divergence distance 18 for each time is equal to or greater than this value. The deviation threshold is a threshold set in advance, for example, by an experiment using an actual machine of the moving object determination device 1 or a computer simulation based on the design specifications of the moving object determination device 1. The moving object determination unit 80 is an example of a determination unit.

  Next, the operation of the moving object determination device 1 will be described. The moving object determination apparatus 1 described above can be configured using a computer 2.

  FIG. 5 is a diagram illustrating a configuration example of a main part of the electric system of the moving object determination device 1.

  The moving object determination device 1 includes a CPU (Central Processing Unit) 2A that executes determination processing. The moving object determination device 1 includes a ROM (Read Only Memory) 2B in which various programs and various parameters are stored in advance, and a RAM (Random Access Memory) 2C used as a work area when the CPU 2A executes the program. Prepare. The moving object determination device 1 further includes a nonvolatile memory 2D that retains data even when the computer 2 is turned off, and an input / output interface (I / O) 2E that connects an external module.

  The CPU 2A, ROM 2B, RAM 2C, nonvolatile memory 2D, and I / O 2E are connected to each other via an internal bus 2F. The I / O 2E includes, for example, a camera 3, a GPS (Global Positioning System) module 4, a vehicle speed sensor 5, and a gyro sensor. 6 is connected.

  The GPS module 4 is an example of a position measuring device that receives GPS radio waves transmitted from a plurality of GPS and represents the position of the host vehicle 8 by latitude and longitude.

  The vehicle speed sensor 5 is an example of a speed measuring device that monitors the rotational speed of the axle of the host vehicle 8 and measures the vehicle speed from the rotational speed of the axle.

  The gyro sensor 6 is an example of an inertial measurement device that measures angular velocities in the pitch direction, the roll direction, and the yaw direction of the host vehicle 8, for example. There is no restriction on the method of measuring the angular velocity in the gyro sensor 6, and a vibration method in which the angular velocity is measured by detecting a mechanical movement may be detected, or the displacement generated by the Coriolis force is detected as a capacitance displacement. An electrostatic capacitance method for measuring angular velocity may be used.

  The external module connected to the I / O 2E is an example, and is not limited to the camera 3, the GPS module 4, the vehicle speed sensor 5, and the gyro sensor 6. For example, a communication device that connects to a communication line such as the Internet and performs data communication with another external device may be connected to the I / O 2E.

  FIG. 6 is a flowchart illustrating an example of the flow of moving object determination processing in the moving object determination apparatus 1. A moving object determination program for defining the moving object determination process is stored in the ROM 2B in advance. For example, when the power of the moving object determination device 1 is turned on, the CPU 2A reads the moving object determination program from the ROM 2B and executes it. Thereafter, the CPU 2A executes a moving object determination process every time a certain period of time elapses.

  In step S10, the CPU 2A controls the camera 3 to start capturing an image. Since the image photographed by the camera 3 is stored in the RAM 2C, the CPU 2A acquires the image from the RAM 2C in time series.

  In step S20, the CPU 2A extracts a plurality of feature points 9 from each of the images acquired in step S10 using a known feature point extraction method, and associates the images with the coordinates of the extracted feature points 9 in the RAM 2C. Remember.

  In step S30, the CPU 2A calculates, for each feature point 9, an optical flow obtained by tracking the feature points 9 of each image extracted in step S20 along the time series. The CPU 2A stores the calculated optical flow in the RAM 2C.

  FIG. 7 is a diagram illustrating an example of the optical flow calculated in step S30. The optical flow 22 is obtained for each feature point 9 by tracking the feature points 9 of each image at time t (t = 1 to 5) extracted in step S20.

  In step S <b> 40, the CPU 2 </ b> A acquires the vehicle speed and angular velocity along the time series of the host vehicle 8 from the vehicle speed sensor 5 and the gyro sensor 6 over a predetermined time range (determination period), respectively. The CPU 2A acquires the angular velocity ω in the azimuth direction of the host vehicle 8, the angular velocity ψ in the pitch direction, and the angular velocity ξ in the roll direction. The CPU 2A stores the acquired vehicle speed and angular velocity in the RAM 2C.

In step S50, CPU 2A is the angular velocity ω of the azimuthal direction of the own vehicle 8 obtained in step S40, by using the orientation angle theta ^t at time t, the time (t + 1) the azimuth angle theta ^{t + 1} in (1) Calculated by the formula.

Further, CPU 2A calculates pitch angle φ ^{t + 1} at time (t + 1) using equation (2) using angular velocity ψ in the pitch direction of host vehicle 8 acquired in step S40 and pitch angle φ ^t at time t. To do.

Further, the CPU 2A uses the calculated azimuth angle θ ^{t + 1} and pitch angle φ ^{t + 1} at the time (t + 1) and the vehicle speed V ^{t + 1} at the time (t + 1) acquired in step S40 to obtain the time (t + 1). The position (X ^{t + 1} , Y ^{t + 1} , Z ^{t + 1} ) of the host vehicle 8 is calculated by the equation (3).

  The CPU 2A sequentially calculates the position of the host vehicle 8 at each time t within the determination period, thereby calculating the change in the position of the host vehicle 8 in the determination period, that is, the travel locus of the host vehicle 8. Note that Δt, which is the measurement interval of the vehicle speed and the angular velocity, is determined in advance, and may be stored in advance in, for example, the nonvolatile memory 2D.

  The CPU 2A stores each position of the host vehicle 8 in the determination period in the RAM 2C as a travel locus of the host vehicle 8.

In step S60, the CPU 2A uses the angular velocity ξ in the roll angle direction of the host vehicle 8 acquired in step S40 and the roll angle ρ ^t at time ^t to obtain the roll angle ρ ^{t + 1} at time (t + 1) as (5). Calculated by the formula.

CPU2A, within the decision period, the roll angle [rho ^t of the vehicle 8 sequentially calculated at each time t, and the roll angle [rho ^t of the vehicle 8 in a determined period, the vehicle in the determination period calculated in Step S50 The three-dimensional rotation matrix R _t of the host vehicle 8 is calculated using the azimuth angle θ ^t and the pitch angle φ ^t of 8.

Then, the CPU 2A performs the optical flow indicated by the camera parameter K of the camera 3, the three-dimensional rotation matrix R _{t of} the host vehicle 8, the travel locus x _t of the host vehicle 8 calculated in step S50, and the index m calculated in step S30. X _^'t, with _m, calculated for each feature point 9 by a three-dimensional position X _m (4) expression of spatial feature points 14 corresponding to the feature point 9.

  As the camera parameter K of the camera 3, for example, a value stored in advance in the nonvolatile memory 2D may be used.

CPU2A the calculated three-dimensional position _{X m} of the spatial feature point 14, in association with the optical flows used to calculate the spatial feature point 14, and stores the RAM2C for each feature point 9.

  In step S70, the CPU 2A selects one optical flow that has not been selected from the optical flows stored in the RAM 2C.

In step S80, CPU 2A acquires the three-dimensional position _{X m} of the spatial feature points 14 associated with the optical flow selected in step S70.

The CPU 2A acquires the three-dimensional position X _m of the acquired spatial feature point 14, the camera parameter K of the camera 3, the three-dimensional rotation matrix R _{t of} the host vehicle 8 calculated in step S60, and the travel of the host vehicle 8 calculated in step S50. Using the trajectory x _t , the position X ^″ _{t, m} of the reprojection feature point 16 in each image in the determination period corresponding to the spatial feature point 14 is calculated by the equation (6).

The CPU 2A selects the optical flow of the spatial feature point 14 obtained by tracking the position X ^″ _{t, m} of the reprojection feature point 16 in each image in the determination period along the time series in step S70. It is stored in the RAM 2C in association with the flow.

  In step S90, the CPU 2A calculates a divergence distance 18 between the optical flow selected in step S70 and the optical flow of the spatial feature point 14 associated with the feature point 9 calculated in step S80. Specifically, the CPU 2A calculates a deviation index value representing the degree of deviation from the deviation distance 18 between the selected optical flow and the optical flow of the spatial feature point 14. As an example, the CPU 2A calculates an average value of the divergence distances 18 at each time within the determination period as the divergence index value.

  In step S100, the CPU 2A determines whether or not the deviation index value calculated in step S90 exceeds the deviation threshold. When the deviation index value exceeds the deviation threshold value, the process proceeds to step S110.

  If the selected optical flow is an optical flow of a moving object, since the moving object is moving, the deviation distance 18 tends to increase. That is, when the deviation index value exceeds the deviation threshold, the optical flow selected in step S70 can be regarded as an optical flow of a moving object such as another vehicle traveling around the host vehicle 8, for example.

  Accordingly, in step S110, the CPU 2A stores the tag of the moving object OF in the RAM 2C in association with the optical flow selected in step S70. Here, the “moving object OF” is a tag indicating that the optical flow is an optical flow of a moving object.

  On the other hand, if the determination process in step S100 is negative, that is, if the deviation index value is equal to or less than the deviation threshold, the process proceeds to step S120 without executing the process in step S110.

  In step S120, the CPU 2A determines whether or not all the optical flows calculated in step S30 have been selected in step S70. If there is an unselected optical flow, the process proceeds to step S70. In step S70, the CPU 2A selects an optical flow that has not yet been selected, and repeatedly executes the processing in steps S70 to S120 to determine whether the optical flow is an optical flow of a moving object for each feature point 9. judge.

  Then, if it is determined whether or not all the optical flows calculated in step S30 are optical flows of the moving object, the determination process in step S120 is affirmative, and the moving object determination process illustrated in FIG. 6 is performed. Ends.

  In other words, an object with an optical flow determined to be an optical flow of a moving object is determined to be a moving object.

  FIG. 8 is a diagram illustrating an example of a determination result when the moving object determination apparatus 1 performs the moving object determination process illustrated in FIG. 6.

  In FIG. 8, an image 32 is an image obtained by integrating the optical flows calculated in step S30, and an image 34 is an image obtained by integrating the optical flow 24 of the spatial feature points 14 calculated in step S80. Further, the image 36 calculates the average value of the divergence distance 18 between the optical flow 22 and the optical flow 24 of the spatial feature point 14 for each optical flow, and represents a locus with a relatively long average value of the divergence distance 18 in black. It is the figure which represented the locus | trajectory in which the average value of the deviation distance 18 is comparatively short in gray. In the image 38, the optical flow determined to be an optical flow of a moving object in the optical flow 22 is represented by a black solid line, and the optical flow determined to be an optical flow of a stationary object is represented by a gray solid line. It is an image.

  As described above, according to the moving object determination device 1 according to the first embodiment, the feature point 9 is extracted without limiting the region of the feature point 9 extracted from the image, for example, at the lower part of the image. In addition, the moving object determination device 1 calculates the spatial feature point 14 corresponding to the feature point 9 and re-projects the spatial feature point 14 on the image, thereby calculating the optical flow of the spatial feature point 14. Then, the moving object determination device 1 determines whether the optical flow is an optical flow of the moving object based on the degree of deviation between the optical flow indicating the feature point 9 and the optical flow of the spatial feature point 14. Therefore, the moving object determination device 1 can determine whether or not the object included in the captured image is a moving object, regardless of the positional relationship between the host vehicle 8 and surrounding objects. Further, the moving object determination apparatus 1 determines whether or not the object included in the captured image is a moving object, as compared with the case of determining whether or not the object is a moving object only from the optical flow indicating the feature point 9. be able to.

Second Embodiment
The moving object determination device 1 according to the first embodiment determines whether the optical flow extracted from the image is the optical flow of the moving object based on the degree of deviation between the optical flow indicating the feature point 9 and the optical flow of the spatial feature point 14. Judged.

  In the second embodiment, a moving object determination apparatus 1A that can accurately determine whether an object included in an image is a moving object from an optical flow by narrowing down the range of moving objects from the image in advance will be described. .

  FIG. 9 is a diagram illustrating a configuration example of the moving object determination device 1A. The difference of the configuration of the moving object determination device 1A from the configuration of the moving object determination device 1 according to the first embodiment shown in FIG. 1 is that a moving object candidate determination unit 90 and a vehicle detection unit 100 are added. The part 80 is replaced with a moving object determination part 80A.

  The moving object candidate determination unit 90 performs the same processing as the moving object determination unit 80 of the moving object determination device 1 according to the first embodiment. That is, in the moving object determination device 1A, the moving object candidate determination unit 90 functions as the moving object determination unit 80.

  The moving object candidate determination unit 90 notifies the moving object determination unit 80A of a determination result for determining whether or not each of the optical flows calculated by the first optical flow calculation unit 30 is an optical flow of a moving object. Hereinafter, the optical flow determined by the moving object candidate determination unit 90 as the optical flow of the moving object is referred to as “moving object OF candidate”.

  The vehicle detection unit 100 detects an image region (vehicle image region) corresponding to a vehicle for each of the images acquired by the image information acquisition unit 10. A known detection method is used as a method for detecting the vehicle image region from the image. For example, by learning various vehicle images as teacher signals in advance, SVM (Support Vector Machine) that performs pattern recognition of whether the object included in the image is a vehicle, or parts that well represent the characteristics of the vehicle BOF (Bag Of Features) that detects a vehicle image region by determining whether or not a set of local feature amounts is included in an image using a set of local feature amounts is used. Further, a deep learning method using a neural network such as CNN (Convolutional Neural Network) may be used for detection of the vehicle image region.

  The vehicle detection unit 100 is an example of a detection unit, and the vehicle image region is an example of a moving object region.

  The moving object determination unit 80A uses the moving object OF candidate determined by the moving object candidate determination unit 90 and the vehicle image area detected by the vehicle detection unit 100 to determine which optical flow of the moving object OF candidates. It is determined whether or not the optical flow of the moving object.

  Specifically, the moving object determination unit 80A determines that the moving object OF candidate in which at least a part of the optical flow is included in the vehicle image area among the moving object OF candidates is the optical flow of the moving object. That is, the moving object determination unit 80A determines that the optical flow in which any part of the optical flow is not included in the vehicle image area among the moving object OF candidates is not the optical flow of the moving object. The fact that at least a part of the optical flow is included in the vehicle image area is referred to as “the optical flow is included in the vehicle image area”.

  In the moving object determination device 1A, the moving object determination unit 80A and the moving object candidate determination unit 90 are examples of a determination unit.

  Next, the operation of the moving object determination device 1A will be described. The configuration example of the main part of the electric system of the moving object determination device 1A is the same as the main configuration example of the electric system in the moving object determination device 1 according to the first embodiment shown in FIG.

  FIG. 10 is a flowchart illustrating an example of the flow of the moving object determination process in the moving object determination apparatus 1A. A moving object determination program that defines the moving object determination processing is stored in the ROM 2B in advance. For example, when the power of the moving object determination device 1A is turned on, the CPU 2A reads the moving object determination program from the ROM 2B and executes it. Thereafter, the CPU 2A executes a moving object determination process every time a certain period of time elapses.

  The moving object determination process in FIG. 10 is different from the moving object determination process according to the first embodiment shown in FIG. 6 in that steps S130 and S140 are added, and the processes in steps S10 to S120 are the same. It is.

  In step S130, the CPU 2A detects a vehicle image area from each of the images acquired in step S10 using a known extraction method. The CPU 2A stores the coordinate data of the detected vehicle image area in the RAM 2C in association with the detection source image.

  FIG. 11 is a diagram illustrating an example of the vehicle image area 42 detected from the image 52. In FIG. 11, the vehicle image area 42 is represented by a dotted rectangle.

  In step S140, the CPU 2A integrates the vehicle image area 42 associated with any of the images acquired in step S10 and the optical flow calculated in step S30 into the vehicle image area 42 in the optical flow. The included moving object OF candidate is determined as the optical flow of the final moving object.

  Therefore, the CPU 2A stores the tag of the moving object OF in the RAM 2C in association with only the moving object OF candidate finally determined to be the optical flow of the moving object among the moving object OF candidates, and stores the optical flow of the moving object in the RAM 2C. The tag of the moving object OF is removed from the moving object OF candidates determined not to be.

  Note that the method of determining the final optical flow of the moving object in step S140 is not limited to this. For example, among the optical flows extracted in step S30, all the optical flows included in the vehicle image area 42 are determined to be optical flows of moving objects, and other optical flows are determined not to be optical flows of moving objects. Good.

  The moving object determination process shown in FIG.

  FIG. 12 is a diagram illustrating an example of a determination result when the moving object determination process illustrated in FIG. 10 is executed by the moving object determination apparatus 1A.

  In FIG. 12, an image 52 is an image obtained by integrating the vehicle image areas 42, and an image 54 is an image obtained by integrating the optical flow. In the image 54, the black solid line represents the moving object OF candidate, and the gray solid line represents the optical flow determined as the optical flow of the stationary object by the moving object candidate determination unit 90. The image 56 is an image obtained by integrating the image 52 and the image 54.

  For example, the moving object determination apparatus 1A determines the moving object OF candidate included in the vehicle image area 42 among the moving object OF candidates shown in the image 56 as the optical flow of the moving object.

  Thus, according to the moving object determination device 1A according to the second embodiment, the optical flow of the moving object can be further narrowed down from the moving object OF candidates by detecting the vehicle image region 42 from the image. The optical flow of the moving object can be accurately determined.

<Third Embodiment>
The moving object determination device 1A according to the second embodiment determines the moving object OF candidate included in the vehicle image area 42 among the moving object OF candidates as the optical flow of the moving object.

  In the third embodiment, a moving object determination apparatus 1B that determines the reliability of a moving object OF candidate and prevents the moving object OF candidate with low reliability from being used for determination of the optical flow of the moving object will be described.

  FIG. 13 is a diagram illustrating a configuration example of the moving object determination device 1B. The configuration of the moving object determination device 1B is different from the configuration of the moving object determination device 1A according to the second embodiment illustrated in FIG. 9 in that a reliability determination unit 110 is added.

  The reliability determination unit 110 uses the pitch rate representing the degree of variation of the angular velocity ψ in the pitch direction among the angular velocities of the host vehicle 8 acquired by the IMU information acquisition unit 40 to determine the movement determined by the moving object candidate determination unit 90. The reliability of the object OF candidate is determined. The pitch rate is represented by a variation amount of the angular velocity ψ in a predetermined period such as a determination period.

  In the moving object determination device 1B, the moving object determination unit 80A, the moving object candidate determination unit 90, and the reliability determination unit 110 are examples of a determination unit.

  Next, the operation of the moving object determination device 1B will be described. The configuration example of the main part of the electric system of the moving object determination apparatus 1B is the same as the main configuration example of the electric system in the moving object determination apparatus 1 according to the first embodiment shown in FIG.

  FIG. 14 is a flowchart illustrating an example of a flow of moving object determination processing in the moving object determination apparatus 1B. A moving object determination program that defines the moving object determination process is stored in the ROM 2B in advance. For example, when the power of the moving object determination device 1B is turned on, the CPU 2A reads the moving object determination program from the ROM 2B and executes it. Thereafter, the CPU 2A executes a moving object determination process every time a certain period of time elapses.

  The moving object determination process of FIG. 14 is different from the moving object determination process according to the second embodiment shown in FIG. 10 in that steps S122 to S126 are added, and the other processes are the same.

  After the moving object OF candidate is obtained in steps S10 to S120, in step S122, the CPU 2A calculates the pitch rate in the determination period using the angular velocity ψ in the pitch direction of the host vehicle 8 acquired in step S40.

  When frequent fluctuations occur in the pitch direction of the host vehicle 8, the traveling locus of the host vehicle 8 calculated from the vehicle speed and the angular velocity due to differences in the installation positions of the vehicle speed sensor 5, the gyro sensor 6, and the camera 3, for example, The calculation accuracy of the deviation distance 18 from the optical flow calculated from the photographed image is lowered. That is, as the pitch rate increases, the ratio of the optical flow that actually indicates the optical flow of the moving object among the moving object OF candidates obtained in steps S10 to S120 decreases. There is a tendency for the reliability to represent the optical flow to decrease.

  Therefore, a variation threshold value is provided, and when the pitch rate is larger than this value, the variation threshold value is set in advance to a value that can be considered that the reliability of the moving object OF candidate is low.

  In step S124, the CPU 2A determines whether or not the pitch rate calculated in step S122 is larger than the variation threshold.

  The variation threshold value may be set by, for example, an experiment using the actual moving object determination device 1B or a computer simulation based on the design specifications of the moving object determination device 1B, and may be stored in advance in the nonvolatile memory 2D.

  When the pitch rate is less than or equal to the variation threshold, it is determined that the reliability of the moving object OF candidate is high. Therefore, in step S140, the CPU 2A determines the final optical flow of the moving object using the moving object OF candidate obtained in steps S10 to S120 and the vehicle image area 42 detected in step S130.

  On the other hand, when the pitch rate is larger than the fluctuation threshold, it is determined that the reliability of the moving object OF candidate is low. Therefore, the process proceeds to step S126, and in step S126, the CPU 2A acquires the moving object OF candidate (previous moving object OF candidate) obtained by the previously executed moving object determination process from the RAM 2C.

  In this case, in step S140, the CPU 2A determines the final optical flow of the moving object using the previous moving object OF candidate acquired in step S126 and the vehicle image area 42 detected in step S130.

  As described above, according to the moving object determination device 1B according to the third embodiment, the reliability of the moving object OF candidate is determined using the pitch rate of the host vehicle 8, and when it is determined that the reliability is low. Then, the optical flow of the moving object is determined using the previously acquired moving object OF candidate. Therefore, the moving object determination apparatus 1B can determine the optical flow of the moving object with higher accuracy than the case of determining the optical flow of the moving object without considering the reliability of the moving object OF candidate.

  Although the moving object determination apparatus 1B determines the reliability of the moving object OF candidate using the pitch rate, the method for determining the reliability of the moving object OF candidate is not limited to this. For example, the reliability of the moving object OF candidate may be determined using a roll rate that represents the degree of fluctuation of the angular velocity ξ in the roll direction. Similar to the pitch rate, the reliability of the moving object OF candidate decreases as the roll rate increases. Further, the reliability of the moving object OF candidate may be determined by combining the pitch rate and the roll rate.

<Fourth embodiment>
The moving object determination device 1B according to the third embodiment regards the vehicle included in the vehicle image area 42 detected by the vehicle detection unit 100 as a moving vehicle, and the moving object OF candidates included in the vehicle image area 42 are All were determined as optical flows of moving objects.

  However, the vehicle included in the vehicle image area 42 may include a stationary vehicle such as a parked vehicle.

  Therefore, in 4th Embodiment, the moving object OF candidate contained in the vehicle image area 42 containing the moving vehicle among the vehicle image areas 42 detected by the vehicle detection part 100 is determined as an optical flow of a moving object. The moving object determination device 1C will be described.

  FIG. 15 is a diagram illustrating a configuration example of the moving object determination device 1C. The configuration of the moving object determination device 1C is different from the configuration of the moving object determination device 1B according to the third embodiment shown in FIG. 13 in that a moving vehicle region determination unit 120 is added and the moving object determination unit 80A This is a point replaced with the determination unit 80B. With the addition of the moving vehicle area determination unit 120, the output destination of the vehicle image area 42 detected by the vehicle detection unit 100 is changed from the moving object determination unit 80A to the moving vehicle area determination unit 120.

  The moving vehicle area determination unit 120 moves the vehicle image area 42 using the moving object OF candidate determined as having high reliability by the reliability determination unit 110 and the vehicle image area 42 detected by the vehicle detection unit 100. It is determined for each vehicle image area 42 whether or not the vehicle image area 42 includes the vehicle in question. The moving vehicle area determination unit 120 then moves the vehicle image area 42 detected by the vehicle detection unit 100 to a vehicle image area 42 (stationary vehicle image area) that includes a stationary vehicle and a moving vehicle. Are included in the vehicle image region 42 (moving vehicle image region) including the moving vehicle image region, and the moving vehicle image region is output to the moving object determination unit 80B.

  The moving object determination unit 80B uses the moving object OF candidate determined to have high reliability by the reliability determination unit 110 and the moving vehicle image region obtained by the moving vehicle region determination unit 120, and the optical flow of the moving object. Determine.

  In the moving object determination device 1C, the moving object determination unit 80B, the moving object candidate determination unit 90, the reliability determination unit 110, and the moving vehicle area determination unit are examples of a determination unit.

  Next, the operation of the moving object determination device 1C will be described. The configuration example of the main part of the electric system of the moving object determination apparatus 1C is the same as the main configuration example of the electric system in the moving object determination apparatus 1 according to the first embodiment shown in FIG.

  FIG. 16 is a flowchart illustrating an example of a flow of moving object determination processing in the moving object determination apparatus 1C. The moving object determination program for defining the moving object determination process is stored in the ROM 2B in advance. For example, when the power of the moving object determination device 1C is turned on, the CPU 2A reads the moving object determination program from the ROM 2B and executes it. Thereafter, the CPU 2A executes a moving object determination process every time a certain period of time elapses.

  The moving object determination process in FIG. 16 is different from the moving object determination process according to the third embodiment shown in FIG. 14 in that step S135 is added, and the other processes are the same.

  After acquiring the moving object OF candidate with high reliability in steps S10 to S126 and acquiring the vehicle image area 42 from the image in step S130, in step S135, the CPU 2A executes a moving vehicle area determination process.

  FIG. 17 is a flowchart illustrating an example of the flow of the moving vehicle area determination process executed in step S135.

  In step S200, the CPU 2A selects one vehicle image area 42 that has not been selected from the vehicle image areas 42 of any of the images detected in step S130.

  In step S210, the CPU 2A integrates the optical flow extracted in step S30 and the vehicle image area 42 selected in step S200, and calculates the moving object OF candidate ratio in the vehicle image area 42 selected in step S200. The moving object OF candidate ratio is expressed as a ratio of the number of optical flows of the moving object OF candidate included in the vehicle image area 42 to the number of optical flows included in the selected vehicle image area 42.

  As the moving object OF candidate ratio decreases, the possibility that the vehicle included in the selected vehicle image area 42 is stationary increases.

  Therefore, when the moving object OF candidate ratio is less than this value, the selected vehicle image area 42 is provided with a stationary threshold value that can be regarded as the vehicle image area 42 corresponding to the stationary vehicle, and in step S220. The CPU 2A determines whether or not the calculated moving object OF candidate ratio is less than the stationary threshold.

  The stationary threshold value may be set by, for example, an experiment using the actual moving object determination apparatus 1C or a computer simulation based on the design specifications of the moving object determination apparatus 1C, and may be stored in advance in the nonvolatile memory 2D.

  If the determination process in step S220 is affirmative, the process proceeds to step S230.

  Since the moving object OF candidate ratio in the vehicle image area 42 selected in step S200 is less than the stationary threshold value, in step S230, the CPU 2A classifies the vehicle image area 42 into a stationary vehicle image area. Specifically, the CPU 2A stores the vehicle image area 42 selected in step S200 in association with the area tag whose value is set to “0” in the RAM 2C. The area tag is information indicating whether the vehicle image area 42 is a stationary vehicle image area or a moving vehicle image area. For example, when the value of the area tag is “0”, the area tag is a stationary vehicle image area. When the value of the area tag is “1”, it indicates a moving vehicle image area.

  On the other hand, when the determination process in step S220 is negative, the process proceeds to step S240.

  In this case, since the moving object OF candidate ratio in the vehicle image area 42 selected in step S200 is larger than the static threshold, in step S240, the CPU 2A classifies the vehicle image area 42 into the moving vehicle image area. Specifically, the CPU 2A associates an area tag whose value is set to “1” with the vehicle image area 42 selected in step S200 and stores it in the RAM 2C.

  In step S250, CPU 2A determines whether all vehicle image areas 42 included in the image have been selected in step S200. If there is an unselected vehicle image area 42, the process proceeds to step S200.

  In step S200, the CPU 2A selects a vehicle image area 42 that has not yet been selected, and repeatedly executes the processes in steps S200 to S250, so that the vehicle image area 42 is a stationary vehicle image area or a moving vehicle image area. Whether or not there is is determined for each vehicle image area 42.

  On the other hand, if the determination process in step S250 is affirmative, that is, if all the vehicle image areas 42 included in the image are selected, the moving vehicle area determination process shown in FIG. 17 is terminated.

  In step S140 of FIG. 16, the CPU 2A integrates the optical flow extracted in step S30 and the vehicle image area 42 detected in step S130, and is classified as a moving vehicle image area in step S135 out of the vehicle image areas 42. All the optical flows included in the vehicle image area 42 are determined as the optical flows of the final moving object.

  Note that the method of determining the final optical flow of the moving object in step S140 is not limited to this. For example, among the moving object OF candidates, only the optical flow included in the vehicle image area 42 classified as the moving vehicle image area in step S135 is determined as the optical flow of the moving object, and the other optical flows are the optical objects of the moving object. You may determine that it is not a flow. In addition, optical flows that have been included once in the vehicle image area 42 classified as a moving vehicle image area, regardless of whether or not they are included in the vehicle image area 42 at the current time. You may determine with the optical flow of a moving object.

  FIG. 18 is a diagram illustrating an example of the determination result of the vehicle image region 42 when the moving object region determination apparatus 1C executes the moving vehicle region determination process in step S135 of FIG.

  In FIG. 18, an image 58 shows the determination result of the moving vehicle area determination process, a solid rectangle 44 represents a moving vehicle image area, and a dotted rectangle 46 represents a stationary vehicle image area.

  FIG. 19 is a diagram illustrating an example of a determination result in which the optical flow included in the moving vehicle image region is the final optical flow of the moving object.

  In the image 62, a black solid line represents a moving object OF candidate, and a gray solid line represents an optical flow determined to be an optical flow of a stationary object. Further, in the image 64, the black solid line represents the optical flow of the moving object, and the gray solid line represents the optical flow of the stationary object.

  Even if an optical flow that is not a moving object OF candidate in the image 62 is included in the moving vehicle image area 44, it is determined as an optical flow of the moving object. Conversely, even if an optical flow that is a moving object OF candidate in the image 62 is not included in the moving vehicle image area 44, it is determined as an optical flow of a stationary object.

  As described above, according to the moving object determination device 1C according to the fourth embodiment, the moving object is detected using the vehicle image area 42 including the moving vehicle and the optical flow among the vehicle image areas 42 detected from the image. Determine the optical flow. Therefore, since the possibility that the optical flow included in the vehicle image area 42 corresponding to the actually stationary vehicle is erroneously determined as the optical flow of the moving object is reduced, the moving object determination device 1C The optical flow of the object can be accurately determined.

  The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention. For example, the execution order in the moving object determination process according to each embodiment may be changed without departing from the gist of the invention.

  In addition, the vehicle detection unit 100 in the moving object determination devices 1A, 1B, and 1C detects an image area corresponding to the vehicle from the image, but detects a moving object such as a pedestrian and a two-wheeled vehicle in addition to the vehicle. You may do it.

  In the moving object determination devices 1, 1A, 1B, and 1C according to the embodiments, the case where the moving object determination process is realized by software has been described. However, the present invention is not limited to this, and the moving object is determined. The determination process may be realized by a hardware configuration using, for example, an ASIC (Application Specific Integrated Circuit) or the like. In this case, the processing speed can be increased as compared with the case where software is used.

  In the moving object determination devices 1, 1A, 1B, and 1C according to each embodiment, the moving object determination program is installed in the ROM 2B. However, the present invention is not limited to this. The moving object determination program according to the present invention may be provided in a form recorded on a computer-readable storage medium. For example, a moving object determination program according to the present invention is recorded on an optical disc such as a CD (Compact Disc) -ROM and a DVD (Digital Versatile Disc) -ROM, or a USB (Universal Serial Bus) memory and a memory card. You may provide with the form recorded on the semiconductor memory. The moving object determination program according to the present invention may be acquired from an external device connected to a communication line.

1 (1A, 1B, 1C) ... Moving object determination device, 2 ... Computer, 2A ... CPU, 3 ... Camera, 4 ... GPS module, 5 ... Vehicle speed sensor, 6. ..Gyro sensor, 8 ... own vehicle, 9 ... feature point, 10 ... image information acquisition unit, 14 ... spatial feature point, 16 ... reprojection feature point, 18 ... divergence Distance, 20 ... Image feature extraction unit, 22 ... Optical flow, 24 ... Optical flow of spatial feature points, 30 ... First optical flow calculation unit, 40 ... IMU information acquisition unit, 42 ... Vehicle image area, 50 ... Vehicle trajectory calculation unit, 60 ... Feature point three-dimensional position calculation unit, 70 ... Second optical flow calculation unit, 80 (80A, 80B) ... Moving object Determination unit, 90... Moving object candidate determination unit, 100 · Vehicle detection section, 110 ... reliability determination unit, 120 ... mobile vehicle area determining unit

Claims (7)

An extraction unit for extracting feature points from each image photographed in time series with a photographing device attached to a moving body;
A first calculation unit that tracks a corresponding feature point in each image among the feature points extracted by the extraction unit, and calculates a time series of the position of the tracked feature point for each feature point;
A second calculation unit that calculates a travel locus of the moving body using a physical quantity related to a moving speed and motion of the moving body;
Using the time series of the position of the feature point calculated by the first calculation unit and the travel locus of the moving body calculated by the second calculation unit, the position of the extracted feature point in the three-dimensional space is characterized. A third calculation unit for calculating each point;
Using the travel locus of the moving body calculated by the second calculation unit and the position of the spatial feature point representing the feature point in the three-dimensional space calculated by the third calculation unit, the spatial feature point Calculating a position in each of the images, and calculating a time series of the positions of the feature points corresponding to the spatial feature points represented by the respective images for each feature point corresponding to the spatial feature points A calculation unit;
By comparing the time series of the feature point positions calculated by the first calculation unit with the time series of the spatial feature point positions calculated by the fourth calculation unit corresponding to the feature points, A determination unit that determines, for each feature point, whether the object represented by is a moving object;
A moving object determination apparatus comprising: A detection unit for detecting a moving object region from each of the images;
Of the feature points determined to represent the moving object, the determination unit that is not included in any of the moving object regions detected from the respective images by the detection unit is a feature point that represents the moving object The moving object determination device according to claim 1, wherein the determination is made again. A detection unit for detecting a moving object region from any one of the images;
The determination unit calculates, for each moving object region detected by the detection unit, a ratio of feature points representing a moving object to feature points included in the moving object region, and the ratio is less than a predetermined ratio. The feature point included in the moving object region is re-determined as not a feature point representing the moving object, and the feature point included in the moving object region in which the ratio is equal to or higher than a predetermined ratio is The moving object determination device according to claim 1, wherein the moving object determination device is re-determined as a feature point. The moving object determination device according to claim 2, wherein the determination unit determines that a feature point that has been determined to represent the moving object region in the past is a feature point that represents a moving object. The determination unit uses the time series of the feature point positions obtained in the previous determination as the time series of the feature point positions when the degree of variation of the physical quantity related to the motion of the moving body is greater than a predetermined threshold. The moving object determination device according to any one of claims 2 to 4, wherein, for each feature point, it is determined whether or not the object represented by the feature point is a moving object. The determination unit is a distance between a time series of the feature point positions calculated by the first calculation unit and a time series of the spatial feature point positions calculated by the fourth calculation unit corresponding to the feature points. The moving object determination device according to any one of claims 1 to 5, wherein, for each feature point, it is determined whether or not the object represented by the feature point is a moving object. Computer
An extraction unit for extracting feature points from each image photographed in time series with a photographing device attached to a moving body;
A first calculation unit that tracks a corresponding feature point in each image among the feature points extracted by the extraction unit, and calculates a time series of the position of the tracked feature point for each feature point;
A second calculation unit that calculates a travel locus of the moving body using physical quantities related to a moving speed and a moving direction of the moving body;
Using the time series of the position of the feature point calculated by the first calculation unit and the travel locus of the moving body calculated by the second calculation unit, the position of the extracted feature point in the three-dimensional space is characterized. A third calculation unit for calculating each point;
Using the travel locus of the moving body calculated by the second calculation unit and the position of the spatial feature point representing the feature point in the three-dimensional space calculated by the third calculation unit, the spatial feature point Calculating a position in each of the images, and calculating a time series of the positions of the feature points corresponding to the spatial feature points represented by the respective images for each feature point corresponding to the spatial feature points A calculation unit;
By comparing the time series of the feature point positions calculated by the first calculation unit with the time series of the spatial feature point positions calculated by the fourth calculation unit corresponding to the feature points, A moving object determination program for functioning as a determination unit for determining for each feature point whether or not the object represented by is a moving object.