JP2008034981A - Image recognition device and method, pedestrian recognition device and vehicle controller - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To specify a candidate region highly likely to include an object to be determined previously from an image with high precision, by recognizing a marking such as a pattern or a symbol drawn on a road surface in the image; and to enhance recognition precision of the object in an image picked up by an on-vehicle camera. <P>SOLUTION: A road surface region is clipped by recognizing a white line (e.g. the center line painted on the road surface) in an input image picked up by means of a camera, and a region other than a road is specified from the road surface region in the input image based on reference data corresponding to the road. The input image specifying the region other than a road is converted nito a birds-eye view image, the road surface marking is recognized based on the reference data thereof, and the region other than the road marking is specified as a candidate region to recognize a pedestrian. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image recognition device, an image recognition method, a pedestrian recognition device, and a vehicle control device that perform image recognition of an image taken by a vehicle-mounted camera.

  Conventionally, a technique for performing image processing on an image captured by a vehicle-mounted camera has been disclosed for the purpose of avoiding a collision with an object that obstructs vehicle travel and performing an appropriate driving operation. ing. For example, in Patent Document 1, by generating a composite image viewed from a virtual viewpoint set above the vehicle, the shape is the same as the shape of road markings such as patterns and symbols drawn on the road surface. However, it is possible to provide the driver with images that are not affected by the mounting position and angle of the camera, which are factors that make them look different from each other, to avoid collisions with objects that obstruct vehicle travel, A technique for enabling an appropriate driving operation is disclosed.

  Similarly, for the purpose of avoiding a collision with an object that obstructs vehicle travel, an image taken by an in-vehicle camera is recognized, and there is an object to be discriminated from within the image. A technique for detecting a candidate area having a probability of being detected is disclosed. For example, by extracting and grasping the features of road surface display such as patterns and symbols drawn on the road surface running on a vehicle so that they can be recognized, those that cannot be recognized on the road surface are subject to discrimination. A technique for detecting a candidate area having a high probability that an object is present is disclosed.

WO00 / 64175

  However, the above-described conventional technique has a problem in that it may not be possible to accurately recognize a pattern or a symbol drawn on a road surface traveled by a vehicle.

  In other words, one of the conventional techniques described above generates a composite image viewed from a virtual viewpoint set above the vehicle, so that the same shape as a road surface display shape such as a pattern or a symbol drawn on the road surface can be obtained. Even if it is a shape, it provides the driver with an image that is not affected by the mounting position and angle of the camera, which is a factor that makes them look different from each other. There was a problem that it did not recognize drawn patterns and symbols.

  The other of the above conventional techniques generates data that extracts features of road markings such as patterns and symbols drawn on the road surface based on a plurality of images taken by a camera attached to a predetermined position of the vehicle. In addition, since the projected road marking in the image is grasped based on this data, even if it is the same shape, it looks different depending on the mounting position and mounting angle of the camera (for example, various shapes) It is difficult to extract road marking features without diluting them, and it may not be possible to accurately recognize patterns and symbols drawn on the road surface running on the vehicle. There is a problem in that it is impossible to detect a candidate area with a high probability that an object to be a target of an object exists.

  Therefore, the present invention has been made to solve the above-described problems of the prior art, and by accurately recognizing road markings such as patterns and symbols drawn on the road surface projected in the image, the image Image recognition apparatus capable of accurately identifying candidate regions having a high probability of existence of objects to be discriminated from within, and improving the recognition accuracy of objects in images taken by an in-vehicle camera An object of the present invention is to provide an image recognition method, a pedestrian recognition device, and a vehicle control device.

  In order to solve the above-mentioned problems and achieve the object, the invention according to claim 1 is a reference data storage means for storing reference data corresponding to an overhead view shape of a road marking displayed in an image photographed by a vehicle-mounted camera. And an image conversion means for converting the image into an overhead image in an overhead view, and a road marking in the overhead image obtained by the image conversion means based on the reference data stored in the reference data storage means Road marking recognition means for recognizing

  According to a second aspect of the present invention, in the above invention, a rotation reference object used as a reference for rotating the overhead image is detected from the road marking in the overhead image obtained by the image conversion means. A rotation reference object detection means for rotating, and a rotation correction means for rotating the road marking detected as the rotation reference object by the rotation reference object detection means so as to make the overhead view image coincide with the vehicle traveling direction. , Further provided.

  According to a third aspect of the present invention, an area that has not been recognized as a road marking by the road marking recognition means is identified as a candidate area to be subject to object recognition from the overhead image obtained by the image conversion means, It further comprises object recognition means for performing object recognition on an area corresponding to the candidate area in the original image before being changed to the overhead image by the image conversion means.

  The invention according to claim 4 is further characterized by further comprising a brightness adjusting means for adjusting the brightness difference between the road marking displayed in the image and the road surface to be a predetermined value.

  According to a fifth aspect of the present invention, there is provided a reference data storage step for storing reference data corresponding to a bird's eye shape of a road marking displayed in an image photographed by a vehicle-mounted camera, and an overhead image in a state where the image is overhead. And a road marking recognition step for recognizing a road marking in the overhead image obtained by the image conversion step based on the reference data stored in the reference data storage step. It is characterized by that.

  According to a sixth aspect of the present invention, there is provided a reference data storage means for storing reference data corresponding to a bird's-eye view shape of a road marking displayed in an image photographed by a vehicle-mounted camera, and a bird's-eye view image in a state where the image is bird's-eye view. An image conversion means for converting to a road marking recognition means for recognizing a road marking in the overhead image obtained by the image conversion means based on the reference data stored in the reference data storage means, and the image An area that is not recognized as a road marking by the road marking recognition means is specified as a candidate area for pedestrian recognition from the overhead image obtained by the conversion means, and is changed to an overhead image by the image conversion means. Pedestrian recognition means for performing pedestrian recognition in an area corresponding to the candidate area in the previous original image is provided.

  According to a seventh aspect of the present invention, there is provided a reference data storage means for storing reference data corresponding to a bird's-eye view shape of a road marking displayed in an image photographed by a vehicle-mounted camera, and a bird's-eye view image in a state where the image is bird's-eye view. An image conversion means for converting to a road marking recognition means for recognizing a road marking in the overhead image obtained by the image conversion means based on the reference data stored in the reference data storage means, and the image An area that is not recognized as a road marking by the road marking recognition means is specified as a candidate area for pedestrian recognition from the overhead image obtained by the conversion means, and is changed to an overhead image by the image conversion means. In the previous original image, pedestrian recognition means for performing pedestrian recognition in an area corresponding to the candidate area, and pedestrian recognition by the pedestrian recognition means If it, characterized by comprising notification control for notifying the caution or warning to the driver, and / or a control means for executing a vehicle control for controlling the running state of the vehicle, a.

  The invention according to claim 8 is based on a comparison result between conversion means for converting an input image including a road into an overhead image in a bird's-eye view, and reference data in which a road marking is previously overheadd, and the overhead image. And specifying means for specifying an area other than the road marking from the overhead image as a candidate area to be a target of pedestrian recognition.

  According to the invention of claim 1 or 5, the reference data corresponding to the bird's-eye view shape of the road marking projected in the image taken by the vehicle-mounted camera is stored, and based on the reference data, the taken image is stored. Recognizes road markings in the bird's-eye view image converted to a bird's-eye view, so the road markings displayed in the image (patterns and symbols drawn on the road surface, regardless of the shooting direction and shooting angle of the in-vehicle camera) Etc.) can be accurately recognized.

  According to the invention of claim 2, a rotation reference object as a reference for rotating the overhead image is detected from the road marking (for example, a white line such as a center line or an outer line is detected) and rotated. By rotating the road marking detected as the reference object, the rotation is corrected so that the overhead view image coincides with the vehicle traveling direction. Therefore, when the vehicle travels diagonally (for example, when traveling diagonally across the road) In order to make it easier to recognize the road markings displayed in the images taken in, etc., the overhead view image can be corrected by rotating, and the road markings projected in the images can be recognized more accurately. is there.

  According to the invention of claim 3, an area that is not recognized as a road marking is identified as a candidate area to be subject to object recognition from the overhead image, and the candidate in the original image before being changed to the overhead image. Since object recognition is performed for the region corresponding to the region, a highly probable candidate region in which an object to be discriminated (for example, a pedestrian, a quadruped animal such as a dog or cat) exists in the image is accurately identified in advance. It is possible to improve the recognition accuracy of an object in an image taken by an in-vehicle camera.

  According to the invention of claim 4, the brightness difference between the road marking displayed in the image and the road surface is adjusted to a predetermined value (that is, the white line such as the center line or the outer line is used as a reference). As a result, the road markings such as patterns and symbols drawn on the road surface are adjusted to be clear. It is possible to recognize more accurately.

  According to the invention of claim 6, the reference data corresponding to the overhead view shape of the road marking projected in the image taken by the vehicle-mounted camera is stored, and the taken image is obtained based on the reference data. The road marking in the bird's-eye view image converted to the bird's-eye view is recognized, and from the bird's-eye view image, an area that has not been recognized as a road marking is identified as a candidate area for pedestrian recognition and is changed to a bird's-eye view image. Pedestrian recognition is performed for a region corresponding to a candidate region in the original image before the image, so that a highly probable candidate region where a pedestrian to be identified exists from the image can be accurately identified in advance. It is possible to improve the pedestrian recognition accuracy in the image photographed with this camera.

  According to the seventh aspect of the invention, the reference data corresponding to the overhead view shape of the road marking projected in the image taken by the vehicle-mounted camera is stored, and the taken image is obtained based on the reference data. The road marking in the bird's-eye view image converted to the bird's-eye view is recognized, and from the bird's-eye view image, an area that has not been recognized as a road marking is identified as a candidate area for pedestrian recognition and is changed to a bird's-eye view image. When the pedestrian is recognized for the area corresponding to the candidate area in the original image before the pedestrian, and the pedestrian is recognized as a result, the notification control or the vehicle control is performed. As a result of improving the pedestrian recognition accuracy, appropriate notification control and vehicle control can be performed, and a collision with an object that becomes an obstacle to vehicle travel can be avoided.

  According to the invention of claim 8, an input image including a road is converted into a bird's-eye view image in a bird's-eye view, and a bird's-eye view image is obtained based on a comparison result between reference data and a bird's-eye view in which road markings are previously bird's-eye view. Since the area other than the road marking is identified as a candidate area for pedestrian recognition from inside, for example, data obtained by overlooking the road image including the road marking is stored in advance as reference data, and the reference data and the overhead image Can be identified from the difference extracted as a result of the comparison with the pedestrian, and can easily identify the candidate area without the need for road marking recognition to improve the accuracy of pedestrian recognition It is possible to improve.

  Exemplary embodiments of an image recognition device, an image recognition method, a pedestrian recognition device, and a vehicle control device according to the present invention will be described below in detail with reference to the accompanying drawings. In the following, a vehicle equipped with an image recognition device and a vehicle control device according to the present invention will be described as a first embodiment, and then another embodiment included in the present invention will be described.

  In the following first embodiment, the outline and features of the first embodiment, the configuration of the first embodiment, and the flow of processing will be described in order, and finally the effects of the first embodiment will be described.

[Outline and Features (Example 1)]
First, the outline and features according to the first embodiment will be described with reference to FIG. FIG. 1 is a diagram for explaining the outline and features according to the first embodiment.

  The first embodiment outlines performing image recognition of an image taken by a vehicle-mounted camera for the purpose of avoiding a collision with an object that becomes an obstacle to vehicle travel, but the road surface displayed in the image By accurately recognizing a sign (a pattern or symbol drawn on the road surface), a highly probable candidate area in which an object (for example, a pedestrian) to be identified exists is accurately identified in advance. The main feature is that it is possible to improve the recognition accuracy of an object in an image taken by an in-vehicle camera.

  Specifically explaining the main feature, as shown in FIG. 1, reference data corresponding to a bird's-eye view shape (a shape grasped in a bird's-eye view) projected on an image taken by an in-vehicle camera is shown. It is memorized beforehand. Here, the reference data refers to, for example, data generated by extracting a shape feature obtained in a state where a road marking projected in an image is seen from a bird's-eye view image collected as a sample.

  Then, a white line (for example, a center line painted on the road surface) in the input image photographed by the camera is recognized, the road surface area is cut out, and in the input image based on the reference data corresponding to the road The area outside the road is specified from among the road surface areas. Specifically, the vehicle control device specifies the position of the white line in the input image, cuts out the road surface area in the input image from the positional relationship with the recognized white line, and then based on the reference data corresponding to the road Identify areas other than roads in the area.

  Next, the input image specifying the area other than the road is converted into a bird's-eye view, road marking recognition is performed based on the road marking reference data, and areas other than the road marking are identified as candidate areas for pedestrian recognition. To do. Specifically, an area other than the road marking that has not been recognized as a road marking (symbol or pattern on the road surface) in the overhead view image is specified as a candidate area for pedestrian recognition.

  Subsequently, pedestrian recognition is performed on the area in the original image (input image before being converted into the overhead image) corresponding to the candidate area specified in the overhead image. Note that pedestrian recognition is performed using a technique such as pattern matching or a neural network.

  And notification control and vehicle travel control are performed according to the result of pedestrian recognition. Specifically, for example, in addition to the result of pedestrian recognition (for example, a determination result of “being a pedestrian”), distance information (for example, millimeter wave radar, optical radar, etc.) obtained from an object by a radar (for example, a pedestrian) The distance from the vehicle to the object) is used to determine the collision risk with the pedestrian, and vehicle travel control is performed as notification control or vehicle control according to the determination result.

  For this reason, the first embodiment recognizes the road markings (patterns and symbols drawn on the road surface) displayed in the image with high accuracy, as in the main features described above, so that the object to be discriminated from within the image. Can be identified with high accuracy in advance, and the recognition accuracy of an object in an image taken by an in-vehicle camera can be improved. In addition, the first embodiment can improve the accuracy of pedestrian recognition in an image taken by an in-vehicle camera. As a result, appropriate notification control and vehicle control can be performed, and an object that hinders vehicle travel can be obtained. It is possible to avoid such collisions.

[Configuration of Vehicle (Example 1)]
Next, the configuration of the vehicle on which the image recognition device and the vehicle control device according to the first embodiment are mounted will be described with reference to FIG. FIG. 2 is a block diagram illustrating a configuration of a vehicle on which the image recognition device and the vehicle control device according to the first embodiment are mounted. In the following, only processing units necessary for carrying out the present invention are shown, and description of other processing units is omitted.

  As shown in the figure, this vehicle includes a vehicle control device 10, an imaging device (for example, a monocular camera or a stereo camera) 11, a navigation unit 12, a radar unit 13, an in-vehicle notification unit 14, and an image recognition device 20. Yes. The vehicle control device 10 includes a notification control unit 15 and a vehicle control unit 16, and the image recognition device 20 includes a preprocessing unit 21, a storage unit 22, and a control unit (microcomputer) 23.

  The navigation (navigation) unit 12 communicates with a GPS (Global Positioning System) artificial satellite, and is a means for setting and guiding a travel route from the specified vehicle position and map data. The navigation unit 12 supplies various information useful for vehicle driving operations such as vehicle position information, road shape, road width, and inclination to the driver via an in-vehicle notification unit 14 described later.

  The radar unit 13 irradiates a radar (for example, a 76.5 GHz millimeter wave radar or an optical radar), and measures a distance, speed, direction, and the like with a vehicle or an obstacle (for example, a pedestrian) ahead. It is. Specifically, in Example 3, when the radar unit 13 receives a measurement command for the distance from the collision risk determination unit 23e described later to the pedestrian, the radar unit 13 recognizes the pedestrian's feet (ground plane), After acquiring the distance from the ground contact surface to the vehicle, the distance information (distance from the vehicle to the pedestrian) is output to the collision risk determination unit 23e.

  The in-vehicle notification unit 14 is means for notifying information from the navigation unit 12 or a notification control unit 15 described later, and includes a monitor, a speaker, and the like. For example, an instruction from the notification control unit 15 is received, and an image indicating the presence of a pedestrian is output to a monitor, or a message voice or an alarm sound is output from a speaker.

  The notification control unit 15 of the vehicle control device 10 is a processing unit that performs notification control according to the determination result of the collision risk received from the collision risk determination unit 23e described later. Specifically, for example, in the third embodiment, the notification control unit 15 has the collision risk received from the collision risk determination unit 23e based on a processing control table (see FIG. 3) held internally. In the case of “level 1”, the in-vehicle notification unit 14 is commanded to output an image indicating the presence of a pedestrian.

  The vehicle control unit 16 of the vehicle control device 10 is a processing unit that performs vehicle control according to a collision risk determination result received from a collision risk determination unit 23e described later. Specifically, for example, in the third embodiment, the vehicle control unit 16 determines that the collision risk received from the collision risk determination unit 23e is “based on a processing control table (see FIG. 3) held internally. When it is “level 2”, speed reduction by brake control is executed, and when the collision risk received from the collision risk determination unit 23e is “level 3”, collision avoidance by steering wheel control is executed. .

  The preprocessing unit 21 of the image recognition device 20 is a processing unit that performs preprocessing on an input image input from the imaging device 11 (for example, a monocular camera or a stereo camera), and includes a filter unit 21a and a contour extraction unit 21b. Composed. Among these, the filter unit 21a is a means for performing preprocessing (for example, sharpness, contrast adjustment, and saturation adjustment) for enhancing the contour of an object projected in an image. The contour extracting unit 21b is means for extracting the contour of an object in the image based on the filtering performed by the filter unit 21a.

  The storage unit 22 of the image recognition apparatus 20 is a storage unit that stores data and programs necessary for various processes performed by the control unit 23. In particular, as closely related to the present invention, the reference data storage unit 22a and the determination table 22b.

  The reference data storage unit 22a is a means for storing various types of information related to the road marking recognition processing of the road marking recognition unit 23c described later, and specifically, a shape obtained in a state where the road marking displayed in the image is looked down on. The reference data generated by extracting the above features from a bird's-eye view image collected as a sample is stored.

  The determination table 22b is a storage unit that stores a determination table used for collision risk determination processing in the collision risk determination unit 23e described in detail later. Specifically, as illustrated in FIG. 4, when the recognition result of “being a pedestrian” is not received from the pedestrian recognition unit 23 d described in detail later, the risk is “0”, while the pedestrian recognition is performed. When the recognition result “is a pedestrian” is received from the part 23d, the risk “level 1” at the distance “30 m or more” to the pedestrian and the risk “level 2” at the distance “10-30 m”. The distance “less than 10 m” is defined so as to have a risk level “level 3”.

  The control unit (microcomputer) 23 is a processing unit (microcomputer) that has a predetermined control program, a program that defines various processing procedures, and an internal memory for storing necessary data, and executes various processes using these. In particular, those closely related to the present invention include a white line recognition unit 23a, an image control unit 23b, a road marking recognition unit 23c, a pedestrian recognition unit 23d, and a collision risk determination unit 23e.

  Among these, the white line recognition unit 23a is a processing unit that recognizes a white line (for example, a white line such as a center line or an outer line) from an input image input from the imaging device 10 (for example, a monocular camera or a stereo camera). The recognition result is output to the image control unit 23b.

  The image control unit 23b is a processing unit that performs predetermined processing on an input image input from the imaging apparatus 10 (for example, a monocular camera or a stereo camera). Specifically, the recognition result received from the white line recognition unit 23a. Based on the above, the position of the white line in the input image is specified, and after the road surface area in the input image is cut out from the positional relationship with the white line, an area other than the road in the road surface area is determined based on the reference data corresponding to the road. The input image specifying the area other than the road is specified and converted into an overhead view. And the image control part 23b outputs a bird's-eye view image to the road marking recognition part 23c.

  Here, the principle of coordinate transformation will be briefly described. As illustrated in FIG. 5, an input image input from the in-vehicle imaging device 11 performs projective transformation of predetermined coordinates in the original drawing of the real space onto the projection plane. It is the image obtained by this. Accordingly, by performing inverse transformation on predetermined coordinates in the input image, it is possible to obtain an overhead view (overhead image) in a state where the original view of the real space is overhead.

  That is, for example, when the original drawing of the real space and the projection plane are not parallel, the coordinates (x, y) of the original drawing are converted into the coordinates (x ′, y ′) shown in the following expression on the projection plane. An overhead view image can be obtained by performing inverse transformation.

x ′ = f _x / z = f _x / (a ₀ x + b ₀ y + c ₀ )

y ′ = f _y / z = f _y / (a ₀ x + b ₀ y + c ₀ )

  The road surface display recognition unit 23c is a processing unit that recognizes road surface display (symbols and patterns on the road surface) for the overhead image received from the image control unit 23b. Specifically, the road surface display recognition unit 23c is stored in the reference data storage unit 22a. If the reference data is obtained in the bird's-eye view image and has a predetermined matching degree with the characteristics of the reference data, it is recognized as a road marking. Then, the road marking recognition unit 23c outputs the recognition result to the pedestrian recognition unit 23d.

  The pedestrian recognition unit 23d is a processing unit that identifies pedestrian recognition target areas from within the overhead view image according to the recognition result received from the road marking recognition unit 23c, and performs pedestrian recognition. Specifically, a candidate other than a road marking that has not been recognized as a road marking (a symbol or pattern on the road surface) in the overhead image is specified as a candidate area for pedestrian recognition, and a candidate specified in the overhead image Pedestrian recognition is performed on the area in the original image corresponding to the area (the input image before being converted into the overhead image). Note that pedestrian recognition is performed using a technique such as pattern matching or a neural network. And the pedestrian recognition part 23d outputs a recognition result to the collision risk determination part 23e.

  The collision risk determination unit 23e is a processing unit that determines the collision risk based on the recognition result received from the pedestrian recognition unit 23d. Specifically, upon receiving a recognition result from the pedestrian recognition unit 23d, the collision risk determination unit 23e reads a collision risk determination table (see FIG. 4) from the determination table 22b. When the recognition result “being a pedestrian” is received from the pedestrian recognition unit 23d, the collision risk determination unit 23e instructs the radar unit 13 to measure the distance to the pedestrian. Then, the collision risk determination unit 23e applies the recognition result of “being a pedestrian” received from the pedestrian recognition unit 23d and the distance information to the pedestrian obtained from the radar unit 13 to the collision risk determination table. The degree of risk (for example, “level 1”, “level 2”, etc.) is determined, and the determination result is output to the notification control unit 15 and the vehicle control unit 16.

[Processing by Microcomputer (Control Section) of Image Control Device and Vehicle Control Device (Example 1)]
Subsequently, processing performed by the microcomputer (control unit) and the vehicle control device of the image control device according to the first embodiment will be described with reference to FIG. FIG. 6 is a flowchart illustrating a flow of processing by the microcomputer (control unit) and the vehicle control device of the image control device according to the first embodiment. Note that the processing described below is repeatedly executed every time a predetermined image frame (for example, one frame) is input.

  As shown in the figure, when an image is input from the imaging device 11 (Yes in Step S601), the white line recognition unit 23a recognizes a white line (for example, a white line such as a center line or an outer line) from the input image (Step S601). In step S602, the recognition result is output to the image control unit 23b.

  Subsequently, the image control unit 23b specifies the position of the white line in the input image based on the recognition result received from the white line recognition unit 23a, and cuts out the road surface area in the input image from the positional relationship with the white line (step) S603). Next, the image control unit 23b specifies an area other than the road in the road surface area based on the reference data corresponding to the road (step S604), and converts the image into a state where the input image specifying the area other than the road is overlooked. (Step S605). And the image control part 23b outputs a bird's-eye view image to the road marking recognition part 23c.

  The road marking recognition unit 23c that has received the overhead image from the image control unit 23b performs road marking recognition processing (step S606). Specifically, the reference data stored in the reference data storage unit 22a is read, and if there is an object in the bird's-eye view image that has a predetermined matching degree with the characteristics of the reference data, it is recognized as a road marking. To do. Then, the road marking recognition unit 23c outputs the recognition result to the pedestrian recognition unit 23d.

  Subsequently, the pedestrian recognition unit 23d specifies a candidate area for pedestrian recognition from the overhead image according to the recognition result received from the road marking recognition unit 23c (step S607). Specifically, an area other than the road marking that has not been recognized as a road marking (symbol or pattern on the road surface) in the overhead view image is specified as a candidate area for pedestrian recognition.

  Next, the pedestrian recognition unit 23d performs pedestrian recognition on a region in the original image (input image before being converted into the overhead image) corresponding to the candidate region specified in the overhead image (step S608). Note that pedestrian recognition is performed using a technique such as pattern matching or a neural network. And the pedestrian recognition part 23d outputs a recognition result to the collision risk determination part 23e.

  Then, the collision risk determination unit 23e determines the collision risk based on the recognition result received from the pedestrian recognition unit 23d (step S609).

  Specifically, upon receiving a recognition result from the pedestrian recognition unit 23d, the collision risk determination unit 23e reads a collision risk determination table (see FIG. 4) from the determination table 22b. When the recognition result “being a pedestrian” is received from the pedestrian recognition unit 23d, the collision risk determination unit 23e instructs the radar unit 13 to measure the distance to the pedestrian. Then, the collision risk determination unit 23e applies the recognition result of “being a pedestrian” received from the pedestrian recognition unit 23d and the distance information to the pedestrian obtained from the radar unit 13 to the collision risk determination table. The degree of risk (for example, “level 1”, “level 2”, etc.) is determined, and the determination result is output to the notification control unit 15 and the vehicle control unit 16.

  When the determination result of the collision risk is received from the collision risk determination unit 23e, the notification control unit 15 and the vehicle control unit 16 respectively perform notification control and vehicle travel control according to the determination result (step S610).

  Specifically, the notification control unit 15 is, for example, when the collision risk received from the collision risk determination unit 23e is “level 1” based on the internally stored processing control table (see FIG. 3). Instructs the in-vehicle notification unit 14 to output an image indicating the presence of a pedestrian.

  Similarly, for example, when the collision risk received from the collision risk determination unit 23e is “level 2” based on the internally stored processing control table (see FIG. 3), the vehicle control unit 16 Then, speed reduction by brake control is executed, and when the collision risk received from the collision risk determination unit 23e is “level 3”, collision avoidance by handle control is executed.

[Effects on Example 1]
As described above, according to the first embodiment, the reference data corresponding to the bird's eye shape of the road marking displayed in the image taken by the vehicle-mounted camera is stored, and the image is taken based on the reference data. Recognizes road markings in the bird's-eye view image converted to the bird's-eye view, so the road markings shown in the image (patterns drawn on the road surface) are not affected by the shooting direction or shooting angle of the in-vehicle camera. Can be accurately recognized.

  In addition, according to the first embodiment, the reference data corresponding to the overhead view shape of the road marking displayed in the image taken by the in-vehicle camera is stored, and the taken image is looked down based on the reference data. Before recognizing the road marking in the bird's-eye view image converted into the state, and further identifying the area that was not recognized as a road marking from the bird's-eye view image as a candidate area for pedestrian recognition and before changing to the bird's-eye view image Since the pedestrian recognition is performed for the area corresponding to the candidate area in the original image, the highly probable candidate area where the pedestrian to be determined exists from the image can be accurately identified in advance. It is possible to improve the pedestrian recognition accuracy in the image photographed in (1).

  In addition, according to the first embodiment, the reference data corresponding to the overhead view shape of the road marking displayed in the image taken by the in-vehicle camera is stored, and the taken image is looked down based on the reference data. Before recognizing the road marking in the bird's-eye view image converted into the state, and further identifying the area that was not recognized as a road marking from the bird's-eye view image as a candidate area for pedestrian recognition and before changing to the bird's-eye view image When the pedestrian is recognized for the area corresponding to the candidate area in the original image and the pedestrian is recognized as a result, the notification control or the vehicle control is performed. Therefore, the pedestrian in the image photographed by the in-vehicle camera is used. As a result of improving the recognition accuracy, appropriate notification control and vehicle control can be performed, and a collision with an object that becomes an obstacle to vehicle travel can be avoided.

  Further, in the first embodiment, the case where the pedestrian recognition unit 23d specifies a candidate area that is a target of pedestrian recognition from the overhead image according to the result of the road marking recognition by the road marking recognition unit 23c is described. However, the present invention is not limited to this, and the candidate region may be specified by the background difference (interframe difference) method. For example, the previous frame image and the current frame image are each converted into a bird's-eye view image, and in some cases, rotation correction is performed to take the difference between the previous frame image and the current frame, and the difference is detected. The identified area is identified as a candidate area that is a target for pedestrian recognition.

  Thereby, it is possible to specify in advance with high accuracy a candidate area having a probability that an object (for example, a pedestrian) to be discriminated exists from the image also by the background subtraction method or the like.

  Further, in the above-described first embodiment, a case has been described in which a candidate region that is a target of pedestrian recognition is specified from the overhead image according to the recognition result received from the road marking recognition unit 23c, and pedestrian recognition is performed. The present invention is not limited to this. For example, an object other than a pedestrian that is specified in a bird's-eye view image and affects the vehicle driving (for example, a vehicle, an animal such as a dog or a cat, a tree) Etc.) may be performed.

  For this reason, a highly probable candidate area in which an object to be determined (for example, a pedestrian or a vehicle other than a pedestrian, an animal such as a dog or a cat, a tree, etc.) exists in the image is accurately identified in advance. It is possible to improve the recognition accuracy of an object in an image taken by an in-vehicle camera.

  Further, in the first embodiment, the road marking recognition unit 23c, when there is an item in the bird's-eye view image that has a predetermined matching degree with the characteristics of the reference data stored in the reference data storage unit 22a, However, the present invention is not limited to this.

  For example, a pattern corresponding to the reference data may be generated in advance in the bird's-eye view shape of the road marking, and the road marking may be recognized by pattern matching, or the bird's-eye view shape of the road marking corresponds to the reference data. Data may be learned in advance, and a road marking may be recognized by a neural network.

  According to this, by generating a pattern corresponding to the bird's-eye view shape of the road marking and recognizing the road marking on the bird's-eye view image, even if the same road surface shape looks different due to the influence of the camera mounting position and mounting angle Therefore, the number of patterns that had to be prepared can be greatly reduced, or the learning load can be reduced, so that the load on the equipment can be reduced and the road marking can be accurately displayed. It is possible to recognize.

  In the first embodiment, the case has been described in which the road marking in the road area is recognized by previously holding the reference data in the bird's-eye view shape of the road marking, but the present invention is not limited to this. Alternatively, reference data regarding the bird's-eye view shape of the road including the road marking may be generated and possessed in advance, and the road and the road marking may be recognized for the bird's-eye view image obtained by looking down the input image based on the reference data.

  By the way, road marking recognition is performed after performing rotation correction on the bird's-eye view image converted to the bird's-eye view state in the image control unit 23b of the vehicle control device of the first embodiment so as to coincide with the vehicle traveling direction. Also good. Therefore, in the following, the outline and features of the image recognition apparatus according to the second embodiment, the configuration and processing of the image recognition apparatus according to the second embodiment will be described in order, and finally the effects of the second embodiment will be described.

[Outline and Features of Image Recognition Apparatus (Example 2)]
First, the outline and features of the image recognition apparatus according to the second embodiment will be described with reference to FIG. FIG. 7 is a diagram for explaining the outline and features of the image recognition apparatus according to the second embodiment.

  The second embodiment is similar to the first embodiment described above in that it recognizes an image captured by an in-vehicle camera for the purpose of avoiding a collision with an object that becomes an obstacle to vehicle travel. However, the main feature is that the road markings displayed in the image are recognized more accurately.

  This main feature will be specifically described. As shown in FIG. 7, the second embodiment recognizes a white line (for example, a center line painted on the road surface) in the input image photographed by the camera, and the road surface. The area is cut out and converted into a bird's-eye view image in a bird's-eye view. Further, rotation correction is performed on the bird's-eye view image so as to coincide with the vehicle traveling direction.

  Specifically, for example, the white line recognized in the overhead image is adopted as a rotation reference object that serves as a reference for rotating the overhead image, and the white line adopted as the rotation reference object is rotated, whereby the overhead image is The rotation is corrected so as to match the traveling direction.

  Then, the vehicle control apparatus according to the second embodiment performs road marking recognition in the rotation corrected image obtained by rotating the overhead image based on the stored reference data.

  For this reason, the vehicle control apparatus according to the second embodiment includes an image captured when the vehicle travels diagonally (for example, when traveling across a road diagonally) as in the main feature described above. The overhead view image can be rotated and corrected so that the projected road marking can be easily recognized, and the road marking projected in the image can be recognized more accurately.

[Configuration of Image Recognition Device (Example 2)]
Next, as in the first embodiment, the configuration of the image recognition apparatus according to the second embodiment will be described with reference to FIG. The image recognition apparatus according to the second embodiment has basically the same configuration as the vehicle control apparatus according to the first embodiment, but differs in the points described below.

  That is, the image control unit 23b of the image recognition device 20 specifies the white line position in the input image based on the recognition result received from the white line recognition unit 23a, and determines the road surface area in the input image from the positional relationship with the white line. After cutting out, the area other than the road is identified within the road surface area based on the reference data corresponding to the road, and the input image identifying the area other than the road is not only converted to an overhead view, but also an overhead image Is corrected so as to coincide with the vehicle traveling direction.

  Specifically, the image control unit 23b adopts the white line specified in the overhead view image based on the recognition result of the white line recognition unit 23a as a rotation reference target that serves as a reference for rotating the overhead view image. Rotation correction is performed so that the bird's-eye view image matches the vehicle traveling direction. Then, the image control unit 23b outputs the image after the rotation correction to the road marking recognition unit 23c.

[Processing by Image Recognition Apparatus (Example 2)]
Next, processing performed by the control unit (microcomputer) of the image recognition apparatus according to the second embodiment will be described with reference to FIG. FIG. 8 is a flowchart illustrating a flow of processing performed by the control unit (microcomputer) of the image recognition apparatus according to the second embodiment. Note that the processing from step S801 to step S805 and the processing of step S807 are the same as the processing from step S601 to step S606 described with reference to FIG. Hereinafter, the process of step S806 will be described.

  As shown in the figure, based on the recognition result received from the white line recognition unit 23a, the position of the white line in the input image is specified and the road surface area in the input image is cut out from the positional relationship with the white line, For the overhead image obtained by identifying an area other than the road in the road surface area based on the corresponding reference data and converting the input image identifying the area other than the road into an overhead view, the image control unit 23b Rotation correction is performed so as to match (step S806).

  Specifically, the image control unit 23b adopts the white line specified in the overhead view image based on the recognition result of the white line recognition unit 23a as a rotation reference target that serves as a reference for rotating the overhead view image. Rotation correction is performed so that the bird's-eye view image matches the vehicle traveling direction. Then, the image control unit 23b outputs the image after rotation correction to the road marking recognition unit 23c, and the road marking recognition unit 23c performs road marking recognition on the image after rotation correction (step S807).

[Effect of Example 2]
As described above, according to the second embodiment, a rotation reference object as a reference for rotating the overhead image is detected from the road marking (for example, a white line such as a center line or an outer line is detected). By rotating the road marking detected as the rotation reference target, the rotation is corrected so that the overhead view image coincides with the vehicle traveling direction. Therefore, when the vehicle travels diagonally (for example, travels diagonally across the road) In order to make it easier to recognize the road markings displayed in the image taken in the case), the overhead image can be corrected by rotating, and the road markings projected in the image can be recognized more accurately. Is possible.

  Although the embodiments of the present invention have been described so far, the present invention may be implemented in various different forms other than the embodiments described above. Therefore, another embodiment included in the present invention will be described below.

(1) Contrast adjustment In the above embodiment, adjustment is made so that the luminance difference between the road marking displayed in the image and the road surface becomes a predetermined value (that is, white lines such as the center line and the outer line are adjusted). As a reference, adjustment may be made so that road markings such as patterns and symbols drawn on the road surface become clear).

  As a result, for example, the influence of paint density, weather, and the like on the road marking can be reduced, so that the road marking can be recognized more accurately.

(2) Notification control and vehicle control based on road marking recognition result In addition, in the first embodiment, when performing road marking recognition, the contents (for example, pause, speed limit, etc.) should be grasped. For example, when it is recognized that the content of the road marking is a temporary stop, the driver may be notified of the temporary stop by voice or the like, and the speed may be reduced by braking.

(3) Device Configuration The components of the vehicle control device 10 shown in FIGS. 2 and 8 are functionally conceptual and need not be physically configured as illustrated. That is, the specific form of dispersion / integration of the vehicle control device 10 is not limited to the illustrated one. For example, the white line recognition unit 23a, the image control unit 23b, and the road marking recognition unit 23c are integrated, or the image control 23b. Disperse all or part of the image conversion function and rotation correction function provided in the system, functionally or physically in any unit according to various loads and usage conditions, etc. Can be configured. Furthermore, each processing function (image conversion function, road marking recognition function, rotation correction function, and pedestrian recognition function) performed by the vehicle control device 10 is analyzed by the CPU and the CPU in part or in part. It can be realized by a program to be executed or as hardware by wired logic.

  As described above, the image recognition device, the image recognition method, the pedestrian recognition device, and the vehicle control device according to the present invention are performed by a vehicle-mounted camera for the purpose of avoiding a collision with an object that becomes an obstacle to vehicle travel. This is useful when recognizing a photographed image, and is particularly suitable for accurately recognizing road markings such as patterns and symbols drawn on the road surface projected in the image.

FIG. 3 is a diagram for explaining an overview and features according to the first embodiment. 1 is a block diagram illustrating a configuration of a vehicle on which an image recognition device and a vehicle control device according to a first embodiment are mounted. It is a figure which shows the structural example of a process control table. It is a figure which shows the structural example of a collision risk determination table. It is a figure for demonstrating the principle of coordinate transformation. 5 is a flowchart illustrating a flow of processing by a microcomputer (control unit) and a vehicle control device of the image control device according to the first embodiment. FIG. 10 is a diagram for explaining an overview and features of an image recognition apparatus according to a second embodiment. 12 is a flowchart illustrating a flow of processing by a microcomputer (control unit) of the image control apparatus according to the second embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Vehicle control apparatus 11 Imaging device 12 Navigation part 13 Radar part 14 In-vehicle notification part 15 Notification control part 16 Vehicle control part 17 Handle 18 Brake 20 Image recognition apparatus 21 Pre-processing part 21a Filter part 21b Contour extraction part 22 Storage part 22a Reference data Storage unit 22b Determination table 23 Control unit (microcomputer)
23a White line recognition unit 23b Image control unit 23c Road marking recognition unit 23d Pedestrian recognition unit 23e Collision risk determination unit

Claims (8)

Reference data storage means for storing reference data corresponding to an overhead view shape of a road marking projected in an image taken by an in-vehicle camera;
Image conversion means for converting the image into an overhead image in an overhead view;
Road marking recognition means for recognizing road markings in the overhead image obtained by the image conversion means based on the reference data stored by the reference data storage means;
An image recognition apparatus comprising: Rotation reference object detection means for detecting a rotation reference object as a reference for rotating the overhead image from the road marking from the overhead image obtained by the image conversion means;
Rotation correction means for rotating the road marking detected as the rotation reference object by the rotation reference object detection means so as to make the overhead view image coincide with the vehicle traveling direction;
The image recognition apparatus according to claim 1, further comprising:   An area that is not recognized as a road marking by the road marking recognition means is identified as a candidate area to be subject to object recognition from the overhead image obtained by the image conversion means, and is changed to an overhead image by the image conversion means. The image recognition apparatus according to claim 1, further comprising an object recognition unit that performs object recognition on a region corresponding to the candidate region in the original image before the detection.   4. The apparatus according to claim 1, further comprising a brightness adjusting unit configured to adjust a brightness difference between the road marking displayed in the image and the road surface to a predetermined value. Image recognition device. A reference data storage step for storing reference data corresponding to the overhead view shape of the road marking displayed in the image taken by the in-vehicle camera;
An image conversion step of converting the image into an overhead image in an overhead state;
A road marking recognition step for recognizing a road marking in the overhead image obtained by the image conversion step based on the reference data stored by the reference data storage step;
An image recognition method comprising: Reference data storage means for storing reference data corresponding to an overhead view shape of a road marking projected in an image taken by an in-vehicle camera;
Image conversion means for converting the image into an overhead image in an overhead view;
Road marking recognition means for recognizing road markings in the overhead image obtained by the image conversion means based on the reference data stored by the reference data storage means;
From the overhead image obtained by the image conversion means, an area that has not been recognized as a road marking by the road marking recognition means is identified as a candidate area for pedestrian recognition, and changed to an overhead image by the image conversion means Pedestrian recognition means for performing pedestrian recognition in an area corresponding to the candidate area in the original image before
A pedestrian recognition device comprising: Reference data storage means for storing reference data corresponding to an overhead view shape of a road marking projected in an image taken by an in-vehicle camera;
Image conversion means for converting the image into an overhead image in an overhead view;
Road marking recognition means for recognizing road markings in the overhead image obtained by the image conversion means based on the reference data stored by the reference data storage means;
From the overhead image obtained by the image conversion means, an area that has not been recognized as a road marking by the road marking recognition means is identified as a candidate area for pedestrian recognition, and changed to an overhead image by the image conversion means Pedestrian recognition means for performing pedestrian recognition in an area corresponding to the candidate area in the original image before
Control means for executing notification control for notifying a driver of warning or warning and / or vehicle control for controlling the running state of the vehicle when a pedestrian is recognized by the pedestrian recognition means;
A vehicle control device comprising: Conversion means for converting an input image including a road into a bird's-eye view image;
Based on a comparison result between the reference data in which the road marking is pre-viewed in advance and the bird's-eye view image, a specifying unit that specifies a region other than the road marking as a candidate region for pedestrian recognition from the overhead image,
A pedestrian recognition device comprising:

Images