CN102542256B - Advanced Warning System with Forward Collision Warning for Traps and Pedestrians - Google Patents

Abstract

The present invention relates to an advanced warning system and method for forward collision warning of traps and pedestrians using a camera that can be incorporated in a motor vehicle. The method acquires image frames at known intervals. A blob may be selected in at least one image frame. The optical flow between image frames of multiple image points of the plaque may be tracked. The image points may be fitted to at least one model. Based on the fitting of the image points to the at least one model, a Time To Collision (TTC) may be determined if a collision is expected. The image points may be fitted to a road surface model and portions thereof modeled as imaged from the road surface. It is determined that no collision is expected based on the fitting of the image points to the road surface model. The at least one model may also include a hybrid model, and the first portion of the image points may be modeled as imaged from the road surface, and the second portion thereof may be modeled as imaged from a substantially vertical object. The image points may be fitted to a vertical surface model and the image point portions may be modeled as imaged from a vertical object. The TTC may be determined based on a fit of the image points to the vertical surface model.

Description

Advanced Warning System with Forward Collision Warning for Traps and Pedestrians

background

1. Technical Background

The present invention relates to driver assistance systems that provide forward collision warning.

2. Description of related technologies

In recent years, camera-based driver assistance systems (driver assistance systems, DAS) have entered the market; the driver assistance systems include lane departure warning (lane departure warning, LDW), automatic high beam control (Automatic High-beam Control , AHC), pedestrian recognition and forward collision warning (forwardcollision warning, FCW).

Lane Departure Warning (LDW) systems are designed to warn of unintentional lane departures. Warns when vehicles pass or are about to pass lane markings. Driver intent is determined based on the use of turn signals, changes in the angle of the steering wheel, vehicle speed, and brake activation.

In image processing, the Moravec corner detection algorithm may be one of the earliest corner detection algorithms and defines corners as points with low self-similarity. The Moravec algorithm tests each pixel in the image to see if a corner exists by considering how similar a patch centered on a pixel is to nearby patches that mostly overlap. Similarity was measured by taking the sum of squared difference (SSD) between two patches. The smaller the number, the greater the similarity. An alternative method for detecting corners in an image is based on the method proposed by Harris and Stephens, which is a modification of the method proposed by Moravec. Harris and Stephens improved Moravec's corner detection algorithm by considering the differentiation of corner scores directly related to orientation rather than using Moravec's neighboring patches.

In computer vision, a widely used differential method for optical flow estimation was developed by Bruce D. Lucas and Takeo Kanade. The Lucas-Kanade method assumes that the optical flow is basically invariant in the local neighborhood of the pixel under consideration, and solves the basic optical flow equation for all pixels in the neighborhood by the least squares criterion. By combining information from several neighboring pixels, the Lucas-Kanade method is generally able to resolve the inherent ambiguity of the optical flow equation. This method is also insensitive to image noise compared to point-wise methods. On the other hand, since this method is a purely local method, it cannot provide flow information in the inner uniform region of the image.

overview

According to a feature of the invention, different methods are provided for signaling a forward collision warning which use a camera which can be installed in a motor vehicle. A plurality of image frames are acquired at known time intervals. An image patch can be selected in at least one image frame. The optical flow of multiple image points of the plaque can be tracked between image frames. The image points can be fitted to at least one model. Based on the fit of the image points, it can be determined whether a collision is expected and, if so, a time to collision (TTC) can be determined. The image points can be fitted to a road surface model, and a portion of the image points can be modeled as imaged from the road surface. It may be determined that no collision is expected based on the fit of the image points to the road surface model. The image points can be fitted to a vertical surface model, where a portion of the image points can be modeled as being imaged from the vertical object. The time to collision TTC can be determined based on the fit of the image points to a vertical surface model. The image points can be fitted to a mixture model where a first portion of the image points can be modeled as being imaged from the road surface and a second portion of the image points can be modeled as being imaged from a substantially vertical or upright object rather than lying across the road surface Object.

In the image frames, candidate images of pedestrians may be detected, wherein the blobs are selected to include candidate images of pedestrians. When the best fitting model is a vertical surface model, it can be verified that the candidate image is an image of an upright pedestrian rather than an object in the road. In the image frame, a vertical line may be detected, wherein the blob is selected to include the vertical line. When the best-fit model is a vertical surface model, it can be verified that the vertical lines are images of vertical objects rather than objects in the road surface.

In a different approach, a warning may be issued based on a time-to-collision being less than a threshold. In a different approach, the relative proportion of the plaque can be determined based on the optical flow between image frames, and the time to collision (TTC) can be determined responsive to the relative proportion and the time interval. This approach avoids object recognition in patches before determining relative proportions.

According to features of the invention, a system including a camera and a processor is provided. The system can be used to provide forward collision warning using a camera that can be installed in a motor vehicle. The system is also operable to acquire a plurality of image frames at known time intervals for selecting a plaque in at least one of the image frames; for tracking optical flow between the image frames of a plurality of image points of the plaque; For fitting the image points to at least one model and determining whether a collision is expected and, if so, a time to collision (TTC) based on the fit of the image points to the at least one model. The system can also be used to fit image points to a road surface model. It may be determined that no collision is expected based on the fit of the image points to the road surface model.

According to other embodiments of the invention, blobs in the image frame can be selected, which blobs can correspond to where the motor vehicle will be after a predetermined time interval. The patch can be monitored; if an object is imaged in the patch a forward collision warning is issued. Whether an object is substantially vertical, upright, or not in the roadway may be determined by tracking optical flow between image frames of multiple image points of the object in the patch. The image points can be fitted to at least one model. A portion of the image points can be modeled as being imaged from the object. Based on the fit of the image points to the at least one model, it is determined whether a collision is expected and, if so, a time to collision (TTC). A forward collision warning may be issued when the best fit model includes a vertical surface model. The image points can be fitted to a road surface model. It may be determined that no collision is expected based on the fit of the image points to the road surface model.

According to a feature of the invention, a system for providing forward collision warning in a motor vehicle is provided. The system includes a camera and a processor mountable in a motor vehicle. A camera can be used to acquire multiple image frames at known time intervals. The processor is operable to select a blob in the image frame that corresponds to where the motor vehicle will be after a predetermined time interval. If the object is imaged in the patch, a forward collision warning may be issued if the object is found to be upright and/or out of the road. The processor is also operable to track a plurality of image points of objects in the plaque between image frames and fit the image points to one or more models. The model may include a vertical object model, a road surface model, and/or a hybrid model including one or more image points assumed to be from the road surface and one or more image points from vertical objects not in the road surface. Based on the fit of the image points to the model, it is determined whether a collision is expected and, if so, a time to collision (TTC). The processor may be used to issue a forward collision warning based on the TTC being less than a threshold.

Brief description of the drawings

The invention is herein described, by way of example only, with reference to the accompanying drawings, in which:

Figures 1a and 1b schematically illustrate two images captured from a forward looking camera installed in a vehicle as the vehicle approaches a metal barrier, according to a feature of the invention.

Figure 2a illustrates a method for providing forward collision warning using a camera installed in a host vehicle according to a feature of the invention.

Fig. 2b shows further details of the step of determining time to collision shown in Fig. 2a according to a feature of the present invention.

Figure 3a shows an image frame of an upright surface (the back of a van) according to a feature of the invention.

Fig. 3c shows a mainly rectangular area of the road surface according to a feature of the invention.

Figure 3b shows the vertical movement δy of the point as a function of vertical image position (y) with respect to Figure 3a, according to a feature of the present invention.

Figure 3d shows the vertical movement δy of the point as a function of vertical image position (y) with respect to Figure 3c, according to a feature of the invention.

Figure 4a shows an image frame comprising an image of a metal barrier with horizontal lines and rectangular patches according to a feature of the invention.

Figures 4b and 4c show more details of the rectangular patch shown in Figure 4a according to a feature of the invention.

Figure 4d shows a graph of the vertical movement (δy) of a point versus the vertical point position (y) according to a feature of the invention.

Fig. 5 shows another example of a mirage in an image frame according to a feature of the present invention.

Figure 6 illustrates a method for providing a forward collision warning trap according to a feature of the invention.

Figures 7a and 7b show an example of a front collision trap warning triggered for a wall according to an exemplary feature of the invention.

Figure 7c shows an example of a forward collision trap warning triggered for a box according to an exemplary feature of the invention.

Figure 7d shows an example of a forward collision trap warning triggered for the side of a car according to an exemplary feature of the invention.

Figure 8a shows an example of an object with a distinct vertical line on a box, according to an aspect of the invention.

Figure 8b shows an example of an object with a distinct vertical line on a lamppost according to an aspect of the invention.

9 and 10 illustrate a system including a camera or an image sensor installed in a vehicle according to an aspect of the present invention.

A detailed description

Reference will now be made in detail to the features of the invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. The features are described below in order to explain the present invention by referring to the figures.

Before the features of the invention are explained in detail, it is to be understood that the invention is not limited in its application to the details of design and arrangement of parts set forth in the following description or shown in the drawings. The invention has other features or is capable of being practiced or carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting.

By way of introduction, embodiments of the present invention relate to forward collision warning (FCW) systems. According to US patent 7113867, the image of the vehicle in front is identified. The width of the vehicle can be used to detect changes in the scale or relative scale S between image frames, and the relative scale is used to determine the time of collision. Specifically, eg the width of the preceding vehicle has a length denoted by w(t1) and w(t2) in the first image and the second image respectively (as measured eg in pixels or millimeters). Then optionally, the relative ratio is S(t)=w(t2)/w(t1).

According to the teachings of US Patent 7113867, Forward Collision Warning (FCW) systems rely on the recognition of images of obstacles or objects, eg a vehicle ahead as identified in an image frame. In a forward collision warning system, as disclosed in US Patent 7113867, a proportional change in the size (eg width) of a detected object (eg vehicle) is used to calculate the time to collision (TTC). However, objects are first detected and segmented from the surrounding scene. This disclosure describes a system that uses relative scaling to determine time-to-collision TTC and likelihood of collision based on optical flow and, if necessary, issue a FCW warning. Looming phenomenon: The perceived image appears larger as the imaged object gets closer. According to different features of the invention, object detection and/or recognition may be performed, or object detection and/or recognition may be avoided.

Mirage phenomena have been extensively studied in biological systems. Mirages appear to be a very low-level visual attention mechanism in humans and trigger visceral responses. There have been various attempts in computer vision to detect mirages, and there are even silicon sensors designed to detect mirages in the case of pure translation.

Mirage detection, which includes translational movement and rotation, can be performed in real world environments with changing lighting conditions, complex scenes including multiple objects, and host vehicles.

As used herein, the term "relative scale" refers to an increase (or decrease) in the relative size of an image patch in one image frame and a corresponding image patch in a subsequent image frame.

Referring now to FIGS. 9 and 10 , a system 16 including a camera or image sensor 12 installed in a vehicle 18 is shown in accordance with an aspect of the present invention. An image sensor 12 imaging the forward field of view delivers images in real time, which are captured in a temporal sequence of image frames 15 . Image processor 14 may be used to process image frames 15 simultaneously and/or in parallel to serve many driver assistance systems. Driver assistance systems may be implemented using specific hardware circuits with on-board software and/or software control algorithms in memory 13 . Image sensor 12 may be monochrome or black and white, ie without color separation, or image sensor 12 may be color sensitive. By way of example in Figure 10, the image frame 15 is used to serve Pedestrian Warning (PW) 20, Lane Departure Warning (LDW) 21, Forward Collision Warning (FCW) 22 based on object detection and tracking according to the teachings of US Patent 7113867 , image mirage based forward collision warning (FCWL) 209 and/or FCW trap based (FCWT) 601 based forward collision warning 601 . Image processor 14 is used to process image frames 15 to detect mirage for images in the forward view of camera 12 for image mirage and FCWT 601 based forward collision warning 209 . Image mirage-based forward collision warning 209 and trap-based forward collision warning (FCWT) 601 can be performed in parallel with conventional FCW 22, as well as with other driver assistance functions, pedestrian detection (PW) 20, lane departure warning ( LDW)21, traffic sign detection and ego motion detection are performed in parallel. FCWT 601 can be used to verify conventional signals from FCW 22. The term "FCW signal" as used herein refers to a forward collision warning signal. The terms "FCW signal", "forward collision warning" and "warning" are used interchangeably herein.

Features of the invention are illustrated in Figures 1a and 1b which show examples of optical flow or mirages. As the vehicle 18 approaches the metal barrier 30, two captured images from the forward looking camera 12 mounted within the vehicle 18 are displayed. The image in FIG. 1 a shows the field of view and the guardrail 30 . The image in Figure 1b shows the same feature, where the vehicle 18 is much closer to the metal barrier 30, and if one looks at the small rectangle p 32 in the barrier (marked with dashed lines), one can see that in Figure 1b the horizontal line 34 appears to be approaching as the vehicle 18 The guardrail 30 is extended.

Reference is now made to Figure 2a, which illustrates a method 201 for providing forward collision warning 209 (FCWL 209) using camera 12 installed in host vehicle 18, in accordance with features of the present invention. Method 201 does not rely on object recognition of objects in the forward field of view of vehicle 18 . In step 203 a plurality of image frames 15 are acquired by the camera 12 . The time interval between captures of image frames is Δt. The blob 32 in the image frame 15 is selected in step 205 and the relative scale (S) of the blob 32 is determined in step 207 . In step 209, a time to collision (TTC) is determined based on the relative scale (S) and time interval (Δt) between frames 15 .

Reference is now made to Figure 2b which shows further details of the step 209 of determining time to collision shown in Figure 2a in accordance with a feature of the present invention. In step 211 , a plurality of image points in the patch 32 may be tracked between the image frames 15 . In step 213, the image points may be fitted to one or more models. The first model may be a vertical surface model, which may include objects such as pedestrians, vehicles, walls, bushes, trees or lampposts. The second model may be a road surface model, which takes into account the characteristics of the image points on the road surface. The hybrid model may include one or more image points from the road, and one or more image points from the upright object. A number of time-to-collision (TTC) can be calculated for a model that is assumed to include at least a fraction of the image points of an upright object. In step 215, the best fit of the image points to a road surface model, a vertical surface model or a hybrid model enables selection of a time-to-collision (TTC) value. A warning may be issued based on a time to collision (TTC) less than a threshold and when the best fit model is a vertical surface model or a hybrid model.

Optionally, step 213 may also include the detection of candidate images in the image frame 15 . Candidate images can be pedestrians or vertical objects such as vertical lines of lampposts. In the case of pedestrians or vertical lines, a patch 32 may be selected to include a candidate image. Once a patch 32 has been selected, it is possible to perform a verification that the candidate image is an image of an upright pedestrian and/or an image of a vertical line. This validation confirms that the candidate image is not an object in the road surface when the best fitting model is a vertical surface model.

Referring back to FIGS. 1a and 1b, the subpixel arrangement of the blob 32 from the first image shown in FIG. 1a to the second image shown in FIG. 1.08) (step 207). Assuming a time difference Δt=0.5 seconds between images, the time to collision (TTC) can be calculated with the following Equation 1 (step 209):

If the velocity of the vehicle 18 is known to be v (v=4.8m/s), the distance Z to the target can also be calculated with Equation 2 below:

Figures 3b and 3d show the vertical motion δy of the point as a function of the vertical image position (y), according to a feature of the invention. The vertical motion δy is zero at the horizontal line and negative below the horizontal line. The vertical motion δy of the point is shown in Equation 3 below.

Equation (3) is a linear model with respect to y and δy and actually has two variables. These two variables can be solved for using two points.

For vertical surfaces, since all points are equidistant, as in the distance in the image shown in Figure 3b, the motion is zero at the horizontal line ( _y0 ) and varies linearly with image position. For road surfaces, the lower the point is in the image, the closer it is (Z is smaller), as shown in Equation 4 below:

Therefore, the image motion δy does not simply increase at a linear rate, as shown in Equation 5 below and in the graph of Figure 3d.

Equation (5) is actually a constrained quadratic equation with two variables.

Again, two points can be used to solve for these two variables.

Reference is now made to FIGS. 3 a and 3 c showing different image frames 15 . In Figures 3a and 3c, two rectangular areas are shown with dashed lines. Figure 3a shows the upright surface (the back of the van). The square point is the point that is tracked (step 211 ) and the motion matches the motion model of the upright surface shown in the image of the image motion (δy) in Figure 3b compared to the point's height y (step 213). The motion of the triangular point in Figure 3a does not match the motion model of the upright surface. Reference is now made to Figure 3c, which shows a mainly rectangular area of the road surface. The square points are points that match the road surface model shown in the image of the image motion (δy) compared to the point's height y in Fig. 3d. The motion of the triangle points does not match the motion model of the road surface and is an outlier. In general, therefore, the task here is to determine which points belong to the model (and to which model) and which points are outliers, which can be performed by robust fitting methods as explained below.

Reference is now made to Figures 4a, 4b, 4c and 4d, which illustrate typical situations of a mixture of two motion models located in an image according to a feature of the present invention. FIG. 4 a shows an image frame 15 comprising an image of a metal fence 30 with horizontal lines 34 and a rectangular patch 32 a. Further details of the plaque 32a are shown in Figures 4b and 4c. FIG. 4 b shows a detail of a blob 32 a in a previous image frame 15 , and FIG. 4 c shows a detail of a blob 32 a in a subsequent image frame 15 as the vehicle 18 comes closer to the barrier 30 . In FIGS. 4c and 4d , some image points are shown as squares, triangles and circles on the upright obstacle 30 , while some image points are shown on the road ahead of the obstacle 30 . Tracking points within the rectangular area 32a shows that some points are in the lower part of the area 32a corresponding to the road model and some points are in the upper part of the area 32a corresponding to the upright surface model. Figure 4d shows a graph of the vertical motion of the dot (δy) versus the vertical dot position (y). In Fig. 4d, the restored model is illustrated as having two parts: a curved (parabolic) part 38a and a linear part 38b. The transition point between portions 38a and 38b corresponds to the bottom of upright surface 30 . This transition point is also marked by the horizontal dashed line 36 in Fig. 4c. In Figures 4b and 4c there are some points shown by triangles that are tracked but do not match the model, some tracked points that match the model are shown by squares and some points that are not well tracked are shown as circles.

Referring now to FIG. 5 , another example of a mirage in image frame 15 is shown. In image frame 15 of Fig. 5, there is no upright surface in patch 32b, only the clear road ahead, and the transition point between the two models is marked with dashed line 50 at the horizontal.

Motion model and time-to-collision (TTC) estimation

The estimation of motion model and time to collision (TTC) (step 215 ) assumes that an area 32 is provided, for example a rectangular area in the image frame 15 . Examples of rectangular areas are, for example, the rectangles 32a and 32b shown in FIGS. 3 and 5 . These rectangles may be selected based on detected objects such as pedestrians or based on the motion of host vehicle 18 .

1. Tracking point (step 211):

(a) The rectangular area 32 can be subdivided into 5x20 sub-rectangular grids.

(b) An algorithm can be performed for each sub-rectangle to find the corner point of the image, eg using the Harris and Stephens method, and this point can be tracked. Preferably using 5x5Harris points, consider the eigenvalues of the following matrix,

And find out two strong eigenvalues.

(c) Tracking can be performed by an exhaustive search for the best few square difference (SSD) matches in a rectangular search area with width W and height H. This exhaustive search is important at the beginning because it means that previous motions are not taken and the measurements from all sub-rectangles are statistically more independent. The search is followed by fine-tuning using optical flow estimation using, for example, the Lukas Kanade method. The Lukas Kanade method allows for sub-pixel motion.

2. Robust model fitting (step 213):

(a) Randomly select two or three points from 100 tracked points.

(b) The number of pairs selected (N _pairs ) depends on the vehicle speed (v), for example given by:

N _pairs = min(40, max(5, 50-v)) (7)

where the unit of v is m/s. The number of triplets (N _triplets ) is given by:

N _triples = 50-N _pairs (8)

(c) For two points, they can fit two models (step 213). One model assumes that the two points are on upright objects. The second model assumes that both points are on the road.

(d) For three points, they also fit two models. One model assumes that the upper two points are on upright objects and the third (lowermost) point is on the road. The second model assumes that the topmost point is on an upright object and the bottom two points are on a road.

The two models can be solved with respect to three points by solving the first model (Equation 3) using two points, and then solving the second model (Equation 5) with the result y ₀ and the third point.

(e) Each model in (d) gives a time-to-collision TTC value (step 215). Each model also gets a score based on how well the 98 other points fit the model. The score is given by the Sum of the Clipped Square of the Distance (SCSD) between the point's y-motion and the predicted model motion. SCSD values are transformed into a probability-like function:

where N is the number of points (N=98).

(f) Based on the TTC value, the speed of the vehicle 18 and assuming that these points are on stationary objects, the distance Z=v x TTC to these points can be calculated. From the x image coordinates of the distance of each image point, the lateral position in world coordinates can be calculated:

(g) The lateral position at time TTC is thus calculated. Binary lateral scores require that at least one of the points from a pair or triplet must be in the path of the vehicle 18 .

3. Scores for multiple frames: new models can be generated at each frame 15, each new model has its associated TTC and score. The 200 best (highest scoring) models may be retained from the previous 4 frames 15, where the scores are weighted as follows:

fraction( ⁿ ) = α nfraction(12)

where n=0.3 is the age of the fraction and α=0:95.

4. FCW Judgment: If any of the following three conditions occurs, a real FCW warning is issued:

(a) the TTC of the model with the highest score is below the TTC threshold and the score is greater than 0.75, and

(b) The TTC of the model with the highest score is below the TTC threshold and

(c)

3 and 4 have shown how to robustly provide FCW warnings for points within a given rectangle 32 . How the rectangle is defined depends on the application as shown by the other exemplary features of Figures 7a-7d and 8a, 8b.

General FCW pitfalls for stationary objects

Reference is now made to FIG. 6 , which illustrates a method 601 for providing a forward collision warning trap (FCWT) 601 in accordance with features of the present invention. In step 203 , a plurality of image frames 15 are obtained by the camera 12 . In step 605, a blob 32 in the image frame 15 is selected, which blob corresponds to where the motor vehicle 18 will be after a predetermined time interval. Plaque 32 is then monitored in step 607 . In decision step 609 , if a general object is imaged and detected in the plaque 32 , then in step 611 a forward collision warning is issued. Otherwise the capture of image frames continues with step 203 .

Figures 7a and 7b show examples of FCWT 601 warnings triggered against a wall 70 according to an exemplary feature of the invention; An example of an alert; and, an example of an alert triggered for boxes 74a and 74b according to an exemplary feature of the present invention is shown in FIG. 7c. Figures 7a-7d are examples of generic stationary objects that do not require prior class-based detection. The dashed rectangular area is defined as a W=1m wide target at a distance where the main vehicle will be after t=4s.

Z=vt (16)

where v is the velocity of the vehicle 18, H is the height of the camera 12, and w and y are the width and vertical position of the rectangle in the image, respectively. This rectangular area is an example of a FCW trap. If an object "falls" within this rectangular area, the FCW trap can generate a warning if the TTC is less than a threshold. Improve performance with several traps:

To increase the detection rate, FCW traps can be replicated in 5 regions with 50% overlap to produce a total trap area of 3m width.

The dynamic position of the FCW trap can be selected according to the yaw rate: the trap area 32 can be translated laterally based on the path of the vehicle 18 determined from the yaw rate sensor, the velocity of the vehicle 18 and the dynamic model of the host vehicle 18 .

FCW trap for verifying forward collision warning signal

Special classes of objects such as vehicles and pedestrians can be detected in the image 15 using pattern recognition techniques. These objects are then tracked over time according to the teachings of US Patent 7113867, and using changes in the ratio can generate the FCW 22 signal. However, it is important to verify the FCW 22 signal using independent techniques before issuing a warning. It may be especially important to use an independent technique, such as using method 209 (FIG. 2b), to verify the FCW 22 signal if the system 16 will activate the brakes. In a radar/vision fusion system, independent verification can come from the radar. In a vision-only system 16, independent validation comes from independent vision algorithms.

Detection of objects (eg pedestrians, vehicles ahead) is not a problem. A very high detection rate can be achieved with a very low error rate. It is a feature of the present invention to generate a reliable FCW signal without too many false alarms that would annoy the driver, or worse cause the driver to brake unnecessarily. One possible problem with conventional pedestrian FCW systems is to avoid false forward collision warnings because the number of pedestrians in the scene is large and the number of true forward collision situations is very small. Even a 5% error rate would mean that the driver would likely receive frequent false alerts, possibly never experiencing real warnings.

Pedestrian targets are especially challenging for FCW systems because the targets are non-rigid, which makes tracking difficult (as taught by US Patent 7113867), and scale changes in particular are subject to many disturbances. Therefore, the robust model (method 209) can be used to validate forward collision warning for pedestrians. The rectangular area 32 may be determined by the pedestrian detection system 20 . According to US Patent 7113867, only object tracking performed by FCW 22 can produce FCW signal, and robust FCW (method 209) gives a TTC smaller than one or more thresholds which may or may not be predetermined. The forward collision warning FCW 22 may have a different threshold than that used in the robust model (method 209 ).

One of the factors that may increase the number of false warnings is that pedestrians are often present on less structured roads where the driver's driving patterns can be quite erratic, including sharp turns and lane changes. Therefore the issuance of warnings may need to include some further constraints:

When a curb or lane marking is detected, the FCW signal is blocked if a pedestrian is on the far side of the curb and/or lane and none of the following conditions occur:

1. A pedestrian is crossing (or approaching very quickly) a lane marking or curb. For this, detection of pedestrian's feet may be important.

2. The host vehicle 18 is not crossing a lane marking or curb (eg, as detected by the LDW 21 system).

Driver's intentions are less predictable. If the driver is driving straight ahead, the turn signal is not activated and no additional lane markings are anticipated, it is reasonable to assume that the driver will continue straight ahead. Therefore, if there is a pedestrian in the path and the TTC is below the threshold, a FCW signal can be issued. However, if the driver is turning, it is equally possible that he/she will continue to turn or stop turning and keep going. Therefore, when detecting yaw rate, the FCW signal is only issued when it is assumed that the vehicle 18 will continue to turn at the same yaw angle with the pedestrian in the path, and if the vehicle is going straight and the pedestrian is in the path.

The concept of FCW trap 601 can be extended to objects that contain mostly vertical lines (or horizontal lines). A possible problem with using point-based techniques for such objects is that good Harris (corner) points are generally produced by intersecting vertical lines on the edge of the object with horizontal lines of the distant background. The vertical motion of these points will resemble the road surface in the distance.

Figures 8a and 8b show examples of objects with distinct vertical lines 82 on the lamp post 80 in Figure 8b and on the box 84 in Figure 8a. A vertical line 82 is detected in the trap area 32 . Detected straight lines 82 can be tracked between images. Robust estimation can be performed by pairing lines 82 frame by frame and computing a TTC model for each line pair, assuming a vertical object, and then giving a score based on the SCSD of the other lines 82 . Since the number of lines may be small, it is usual to test all combinations of possible pairs. Only use line pairs that have significant overlap. As far as horizontal lines are concerned, triplet lines give two models as when using points.

The indefinite articles "one (a)" and "one (an)" used herein, such as "an image" ("an image"), "a rectangular region" ("a rectangular region"), have "one or Multiple" means "one or more images" or "one or more rectangular areas".

While selected features of the invention have been shown and described, it should be understood that the invention is not limited to the described features. Rather, it should be realized that changes may be made in these characteristics without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims (17)

1. It is 1. a kind of to the method use the video camera that can be installed in a motor vehicle for determining method expected from front shock, Methods described includes：

   Multiple images frame is obtained by known time interval；

   Patch is selected at least one of described image frame；

   The light stream between the picture frame of the multiple images point of the patch is tracked to produce the light stream of tracking；

   For multiple models, at least one of the tracked light stream of described image point is fitted, to produce to the multiple mould Multiple fittings of type, wherein, the multiple model is selected from the group being made up of following item：(i) road surface model, wherein, described image A part for point is modeled as imaging from road surface, and a part for (ii) vertical surface model, wherein described image point is modeled as The object from substantial orthogonality, and (iii) mixed model are imaged, the Part I of wherein described image point is modeled as imaging Object of the imaging from substantial orthogonality is modeled as from the Part II on road surface, and described image point；

   Using at least a portion of described image point, to the light stream for being tracked and the fitting of each model score, it is each to produce Individual fraction；And

   There is a model for fraction by selection, it is determined that whether expection collides and determines collision time, the fraction pair Should be in described image point and the best fit of the light stream for being tracked.
2. 2. the method for claim 1, also includes：

   Best fit based on described image point with the light stream for being tracked for corresponding to the road surface model, it is determined that expection does not have Collision.
3. 3. the method for claim 1, also includes：

   It is optimal with the light stream for being tracked corresponding to the vertical surface model or the mixed model based on described image point Fitting, it is determined that contemplating that collision.
4. 4. method as claimed in claim 3, also includes：

   The candidate image of pedestrian is detected in the patch；And

   When best fit model is the vertical surface model, verify the candidate image be upright pedestrian image without It is the image of the object in road surface.
5. 5. method as claimed in claim 3, also includes：

   Vertical line is detected in described image frame, wherein selecting the patch with including the vertical line；

   When best fit model is the vertical surface model, verify the vertical line be vertical object image rather than The image of the object in road surface.
6. 6. the method for claim 1, also includes：

   Given a warning less than threshold value based on the collision time.
7. 7. method as claimed in claim 4, also includes：

   When the best fit model is road surface model, verify that the candidate image is the object in road surface and expection will not There is collision.
8. 8. a kind of for determining system expected from front shock, it includes：

   Video camera；And

   Processor in a motor vehicle can be installed；Wherein described system is configured to determine that front shock is expected,

   Wherein described processor is configured as obtaining multiple images frame by known time interval；

   Wherein described processor is configured as selecting patch at least one of described image frame；

   Wherein described processor is configured as multiple model followings between the picture frame of the multiple images point of the patch Light stream, wherein the multiple model is selected from the group that is made up of following item：(i) road surface model, wherein, one of described image point Divide and be modeled as imaging from road surface, (ii) vertical surface model a, part for wherein described image point is modeled as imaging from fact Vertical object in matter, and (iii) mixed model, the Part I of wherein described image point are modeled as imaging from road surface, And the Part II of described image point is modeled as object of the imaging from substantial orthogonality；

   Wherein, the processor is configured as being fitted at least one of the tracked light stream of described image point, is arrived with producing Multiple fittings of the multiple model；

   Wherein, the processor be configured with described image point at least a portion and to the light stream for being tracked and each The fitting of model is scored to produce each fraction；And

   Wherein, the processor is configured with least a portion of described image point, has fraction by selection Model determines whether expection collides and determine collision time, and the fraction corresponds to described image point and the light stream for being tracked Best fit.
9. 9. system as claimed in claim 8, wherein, the processor is configured as described image point being fitted to road surface mould Type；

   Wherein described processor is configured as the fitting with the road surface model based on described image point, it is determined that expection does not have and touches Hit.
10. 10. a kind of method expected from determination front shock, the method use video camera and the place that can be installed in a motor vehicle Reason device, methods described includes：

    Multiple images frame is obtained by known time interval；

    Patch in selection picture frame, the patch correspondence motor vehicle will the location of after a predetermined interval of time；

    The light stream between the picture frame of the multiple images point of the patch is tracked to produce the light stream of tracking；

    For multiple models, at least one of the tracked light stream of described image point is fitted, to produce to the multiple mould Multiple fittings of type, wherein, the multiple model is selected from the group being made up of following item：(i) road surface model, wherein, described image A part for point is modeled as imaging from road surface, and a part for (ii) vertical surface model, wherein described image point is modeled as The object from substantial orthogonality, and (iii) mixed model are imaged, the Part I of wherein described image point is modeled as imaging Object of the imaging from substantial orthogonality is modeled as from the Part II on road surface, and described image point；

    Using at least a portion of described image point, to the light stream for being tracked and the fitting of each model score, it is each to produce Individual fraction；And

    There is a model for fraction by selection, it is determined that whether expection collides and determines collision time, the fraction pair Should be in described image point and the best fit of the light stream for being tracked.
11. 11. methods as claimed in claim 10, also include：

    It is determined that whether the object being imaged in the patch includes the part of substantial orthogonality.
12. 12. methods as claimed in claim 11, also include：

    Described image point is fitted to road surface model；And

    Best fit based on described image point with the road surface model, it is determined that expection does not have collision.
13. 13. methods as claimed in claim 11, also include：

    When best fit model is vertical surface model or mixed model, front shock warning is sent.
14. 14. methods as claimed in claim 10, also include：

    The yaw rate of the motor vehicle is input into or calculated from described image frame；And

    The yaw rate based on motor vehicle dynamic on described image frame laterally translates the patch.
15. 15. is a kind of for determining system expected from front shock in a motor vehicle, and the system includes：

    Video camera, it can be arranged in the motor vehicle, and the video camera can be operated and obtained by known time interval Multiple images frame；

    Processor, it is configured as selecting the patch in picture frame, and the patch correspondence motor vehicle is by the predetermined time The location of behind interval；Wherein described processor is configured as the multiple images point in the patch for multiple model followings Picture frame between light stream to produce the light stream of tracking, wherein the multiple model is selected from the group being made up of following item：(i) road Surface model, wherein, a part for described image point is modeled as imaging from road surface, (ii) vertical surface model, wherein the figure A part for picture point is modeled as imaging from the object of substantial orthogonality, and (iii) mixed model, wherein described image point Part I is modeled as imaging from road surface, and the Part II of described image point is modeled as imaging from the right of substantial orthogonality As,

    Wherein, the processor is configured as being fitted at least one of the tracked light stream of described image point, is arrived with producing Multiple fittings of the multiple model；

    Wherein, the processor be configured with least a portion of described image point, to the light stream for being tracked and each mould The fitting of type is scored to produce each fraction；And

    Wherein, the processor is configured to selection, and there is the model of fraction to determine whether expection has collision and really Determine collision time, the fraction corresponds to the best fit of described image point and the light stream for being tracked.
16. 16. systems as claimed in claim 15, wherein the processor be additionally configured to when described image point with tracked When the best fit of light stream is the best fit of the vertical surface model or the mixed model, it is determined that in the patch Whether the object of middle imaging includes the part of substantial orthogonality.
17. 17. systems as claimed in claim 15, wherein the processor is configured as sending front portion less than threshold value based on TTC Conflict alert.