JP2018063680A - Traffic signal recognition method and traffic signal recognition apparatus - Google Patents

Abstract

A traffic signal recognition method with improved traffic signal recognition accuracy is provided. In this traffic signal recognition method, a captured image is acquired by a camera 101 of a vehicle V10 (S11), a signal to be recognized facing the vehicle V10 is searched from a plurality of traffic signals (S12), and map information is obtained. Based on the state of the vehicle V10 and the state of the camera 101, an area including the signal to be recognized is calculated as an ROI from the captured images (S13), and the presence probability of the signal to be recognized at each position in the ROI is shown in advance. A probability distribution is calculated (S14), the contrast of the ROI image is updated according to the prior probability distribution, a contrast update image is generated (S16), and feature points are extracted from the contrast update image, thereby obtaining the inside of the ROI. The area of the lighted portion of the recognition target traffic light in is identified as a lighting area (S17), Based on recognizes the lighting color of the recognition target traffic (S18). [Selection] Figure 4

Description

  The present invention relates to a method and apparatus for recognizing traffic signals, for example for automatic driving of a vehicle.

  Currently, research on automatic driving of vehicles is being actively promoted in companies and universities. In order for a vehicle to automatically drive on a public road, it is necessary to recognize the lighting color of a traffic light installed on the public road as a traffic signal. In addition, a technology has been proposed in which a region of interest including the position of a traffic light is set for an image obtained by photographing with an in-vehicle camera, and a traffic signal is recognized from the region of interest using map information ( For example, refer nonpatent literature 1). By setting such a region of interest, it is possible to limit the image region where the recognition process is performed, reduce the recognition processing load, and improve the recognition processing speed.

N. Fairfield and C.I. Urmson: Traffic Light Mapping and Detection, Proceedings of the 2011 IEEE International Conference on Robotics and Automation, p. 5421-5426 (2011)

  However, even if the technique of Non-Patent Document 1 is used, there is a problem that the traffic signal recognition accuracy is low.

  Therefore, the present invention has been made in view of such problems, and an object of the present invention is to provide a traffic signal recognition method and a traffic signal recognition device with improved traffic signal recognition accuracy.

  In order to achieve the above object, a traffic signal recognition method according to an aspect of the present invention is a traffic signal recognition method for recognizing a lighting color of a traffic light installed on a road as a traffic signal, which is a camera mounted on a vehicle. Based on the image acquisition step of acquiring a captured image by shooting according to the map information indicating the position and orientation of the plurality of traffic lights installed around the vehicle, and the state of the vehicle, from among the plurality of traffic lights, Based on the search step for searching for the first traffic light facing the vehicle, and the map information, the state of the vehicle, and the state of the camera, an area including the first traffic light is selected from the captured image. A region of interest calculation step for calculating as a region of interest, and a prior probability distribution indicating the existence probability of the first traffic light at each position in the calculated region of interest A distribution calculating step for calculating, a contrast updating step for generating a contrast update image by updating the contrast of the image of the first region of interest according to the prior probability distribution, and feature points from the contrast update image. Based on the lighting region specifying step of specifying, as the lighting region, the region of the lighting portion of the first traffic light in the first region of interest by extracting, the first lighting region is based on the color of the specified lighting region. A recognition step of recognizing the lighting color of the traffic light.

  In order to achieve the above object, a traffic signal recognition apparatus according to an aspect of the present invention is a traffic signal recognition apparatus that recognizes a lighting color of a traffic light installed on a road as a traffic signal, and is a captured image by photographing. Based on the camera mounted on the vehicle, the map information indicating the positions and orientations of the plurality of traffic lights installed around the vehicle, and the state of the vehicle, the traffic lights Based on a search unit that searches for a first traffic light facing the vehicle, and the map information, the state of the vehicle, and the state of the camera, an area including the first traffic light is first selected from the captured image. A region-of-interest calculation unit that calculates a region of interest, a probability distribution calculation unit that calculates a prior probability distribution indicating the existence probability of the first traffic light at each position in the calculated region of interest By updating the contrast of the image of the first region of interest according to the prior probability distribution, an image processing unit that generates a contrast update image, and by extracting feature points from the contrast update image, the first A signal recognition unit that identifies an area of a lighting portion of the first traffic light in one region of interest as a lighting area, and recognizes a lighting color of the first traffic light based on the color of the identified lighting area; Is provided.

  In such a traffic signal recognition method and traffic signal recognition apparatus, the contrast update image is generated by updating the contrast of the image of the first region of interest according to the prior probability distribution. Therefore, in the contrast update image, it is possible to increase the contrast of pixel values at a position where the probability that the first traffic light exists is high and a position where the probability that the first traffic light exists is low. Since the lighting area is specified based on the contrast update image, the lighting corresponding to the lighting portion of the first traffic light is included even if other traffic lights other than the first traffic light are included in the first region of interest. A region can be accurately identified from its first region of interest. As a result, the lighting color of the first traffic light can be properly recognized, and the traffic signal recognition accuracy can be improved.

  Note that these comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and the system, method, integrated circuit, and computer program. And any combination of recording media.

  The traffic signal recognition method of the present invention can improve the traffic signal recognition accuracy.

FIG. 1A is a diagram illustrating an example of a captured image of a traffic light obtained by photographing with a monocular camera. FIG. 1B is a diagram illustrating another example of a captured image of a traffic light obtained by photographing with a monocular camera. FIG. 2 is a diagram illustrating an example in which a plurality of traffic lights exist in an ROI set for a captured image. FIG. 3 is a block diagram illustrating an example of a configuration of the traffic signal recognition device according to the embodiment. FIG. 4 is a flowchart illustrating an example of the traffic signal recognition method according to the embodiment. FIG. 5 is a diagram for explaining a search for a visible signal device and a recognition target signal device in the embodiment. FIG. 6 is a diagram for explaining the calculation of the ROI in the embodiment. FIG. 7 is a diagram for explaining the calculation of the map log odds distribution in the embodiment. FIG. 8 is a diagram illustrating an example of an RGB image, an S channel, a V channel, and an SV image in the embodiment. FIG. 9 is a diagram illustrating the types of feature points classified according to the determinant value det (H) of the Hessian matrix. FIG. 10 is a diagram illustrating the relationship between the range of the hue value and the state of the traffic light in the embodiment. FIG. 11 is a diagram illustrating an example of an RGB image, a prior log odds distribution, an observed log odds distribution, and a posterior log odds distribution in the embodiment. FIG. 12 is a diagram illustrating a traffic signal recognition result by each recognition method. FIG. 13 is a diagram illustrating an SV image, an improved SV image obtained by adaptive contrast improvement processing, and a prior log odds distribution in a typical running scene in the embodiment. FIG. 14 is a diagram showing the correct answer rate and the unrecognized rate according to the distance to the recognition target traffic signal in the traffic signal recognition method in the embodiment in comparison with other recognition methods.

(Knowledge that became the basis of the present invention)
Autonomous vehicles use systems that integrate sensor information such as Lidar (Light Detection and Ranging, Laser Imaging Detection and Ranging), millimeter waves, cameras, GNSS (Global Navigation Satellite System) and IMU (Inertial Measurement Unit). The following processing is performed in real time.

  (Process 1) Recognize moving objects, obstacles, and road features (white lines, curbs, traffic lights, etc.) around the vehicle from the sensor information.

  (Process 2) The current position of the vehicle in the map is estimated with high accuracy by matching the feature indicated in the digital map information with the sensor information.

  (Process 3) A safe traveling path according to traffic regulations is optimized using object information indicating the recognized moving object, the current position of the vehicle, and road information on the map.

  (Process 4) Smoothly control the vehicle speed and steering in accordance with the optimized travel path.

  As a method of reliably recognizing road features and the like in the above (Process 1), use of map information can be mentioned. Since road features such as white lines or traffic lights are generally fixed objects, misrecognition and reduction of processing costs can be expected by referring to map information prepared in advance.

  A traffic light is an important road feature in driving at intersections of automobiles. In the case of automatic driving, it is necessary to stop the vehicle in accordance with the recognized traffic signal state, so that it is required to recognize the traffic signal from a distance of 100 m or more.

  1A and 1B are diagrams illustrating an example of a captured image of a traffic light obtained by capturing with a monocular camera. Specifically, FIG. 1A shows a photographed image obtained by daytime photography, and FIG. 1B shows a photographed image obtained by nighttime photography.

  In order to make the subject and background easy to understand, FIGS. 1A and 1B show a color photographed image in which a traffic light that is lit red is projected as an image expressed in white and black. Moreover, in FIG. 1A and FIG. 1B, the lighting part currently lighted in red of the traffic light is shown as a white circle.

  In recognition of a traffic signal by image processing, misrecognition due to direct sunlight, day and night, and background is a problem. In addition, the recognition of a traffic light is recognition of the lighting color of a traffic light, ie, a traffic signal. For example, the daytime photographed image shown in FIG. 1A and the nighttime photographed image shown in FIG. 1B show a traffic light that lights in red, but these photographed images are greatly different as a whole image. Therefore, the recognition of a traffic signal by simple image processing may cause a false recognition of the lighting color of the traffic signal. In response to such problems, many image processing techniques have been proposed in the fields of automatic driving and advanced driving support systems. For example, the method includes the following steps 1 to 3.

  (Step 1) The position of the traffic light included in the captured image is limited, and a region of interest (ROI, Region Of Interest) is set.

  (Step 2) An image in which the lighting part of the traffic light is emphasized is created. That is, a lighting part and a lighting color are emphasized using HSV (Hue, Saturation, Value / Brightness) or a LAB image.

  (Step 3) The created image is identified by threshold judgment or machine learning to recognize the traffic light and the lighting color.

  Until now, a traffic signal recognition method using a monocular camera and map information has been proposed. In the proposed recognition method, not only the ROI is set based on the map information, but also the recognition accuracy is improved by using a classifier based on deep learning.

  Here, in the automatic driving, the recognized state of the traffic light (that is, the lighting color) is used to determine the state of the intersection scheduled to travel. Therefore, the traffic signal of the map information not only has position information but also is associated with each intersection and each stop line. Therefore, when recognizing a traffic light, it is important to recognize the traffic light corresponding to each ROI from the image.

  However, on urban roads, particularly straight roads where a plurality of intersections are lined up at a short distance, there are cases where a plurality of traffic lights are observed in the ROI of the captured image.

  FIG. 2 is a diagram illustrating an example in which a plurality of traffic lights exist in an ROI set for a captured image.

  Although the above-described proposed recognition method can be expected to recognize any one or all of the traffic lights, it is difficult to recognize only one traffic light corresponding to the ROI. That is, even if it is the recognition method using the recognizer by deep learning by setting ROI, it cannot be said that recognition accuracy is enough.

  Therefore, in the traffic signal recognition method and the traffic signal recognition apparatus according to an aspect of the present invention, the signal recognition accuracy is improved by defining a probability distribution indicating the probability that a signal to be recognized exists at each position in the ROI. improves.

  Hereinafter, embodiments of the traffic signal recognition method and the traffic signal recognition apparatus of the present invention will be specifically described with reference to the drawings.

  It should be noted that each of the embodiments described below shows a comprehensive or specific example. The numerical values, shapes, materials, constituent elements, arrangement positions and connecting forms of the constituent elements, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present invention. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements.

  Each figure is a mimetic diagram and is not necessarily illustrated strictly. Moreover, in each figure, the same code | symbol is attached | subjected about the same structural member.

(Embodiment)
[Device configuration]
FIG. 3 is a block diagram showing an example of the configuration of the traffic signal recognition apparatus in the present embodiment.

  Traffic signal recognition device 100 in the present embodiment is a device mounted on vehicle V10, and includes camera 101, vehicle state detection unit 102, map information holding unit 103, traffic light search unit 104, and ROI calculation unit. 105, a probability distribution calculation unit 106, an image processing unit 107, and a signal recognition unit 108.

  The vehicle state detection unit 102 detects the state of the vehicle V10. Specifically, the vehicle state detection unit 102 is equipped with a GNSS and INS (Inertial Navigation System) system, and acquires the position and posture angle of the vehicle V10 at, for example, 100 Hz. Further, the vehicle state detection unit 102 may be equipped with a three-dimensional Lidar and measure a surrounding three-dimensional point group at 10 Hz.

  The camera 101 captures an image in front of the vehicle V10 as a captured image. For example, the camera 101 captures a color captured image having a resolution of 1280 × 960 pixels at 7.5 Hz.

  The map information holding unit 103 holds map information. This map information shows the latitude, longitude, and direction of each traffic light. That is, this map information shows the position and direction of a plurality of traffic lights installed around the vehicle V10. This map information shows not only the position of each traffic signal but also the direction, so if you refer to this map information, you can check whether the traffic signal is a bidirectional type, You can determine if there is. In addition, the high-precision information of each of the plurality of traffic lights is not set in the map information from the viewpoint of maintenance cost.

  The traffic signal search unit 104, the ROI calculation unit 105, the probability distribution calculation unit 106, the image processing unit 107, and the signal recognition unit 108 are obtained by the state of the vehicle V10 detected by the vehicle state detection unit 102 and the photographing by the camera 101. Using the captured image and the map information stored in the map information storage unit 103, the lighting color of the traffic light is recognized as a traffic signal.

[Entire processing of traffic signal recognition method]
FIG. 4 is a flowchart showing an example of the traffic signal recognition method in the present embodiment.

  First, the camera 101 mounted on the vehicle V10 acquires a captured image at predetermined intervals by photographing as described above, and the vehicle state detection unit 102 determines the current position and posture angle of the vehicle V10, etc. It detects as the state of V10 (step S11).

  Next, the traffic signal search unit 104 refers to the map information held in the map information holding unit 103, based on the state of the vehicle V10 detected by the vehicle state detection unit 102 (that is, the current position and attitude angle). Search for traffic lights around the vehicle V10. At this time, the vehicle state detection unit 102 searches for a visible signal (Visible TS) and a recognition target signal (Target TS) facing the vehicle V10 from among a plurality of signals indicated by the map information. When a plurality of recognition target signals are found, the ROI calculation unit 105, the probability distribution calculation unit 106, the image processing unit 107, and the signal recognition unit 108 perform the following steps S13 to S21 for each recognition target signal i. Run repeatedly.

  That is, the ROI calculation unit 105 calculates a region including the recognition target signal device i from the captured images as the ROI based on the map information, the vehicle state, and the camera 101 state (step S13).

  Next, the probability distribution calculation unit 106 calculates a prior probability distribution indicating the existence probability of the recognition target signal device i at each position in the calculated ROI (step S14).

  Next, the image processing unit 107 generates an SV image by converting the color space of the RGB image that is the ROI image (step S15). The SV image is an image composed of an S channel and a V channel in the HSV color space.

  Next, the image processing unit 107 generates a contrast update image by updating the contrast of the ROI SV image according to the prior probability distribution (step S16). This process of updating the contrast is hereinafter referred to as an adaptive contrast improvement process, and the contrast update image is referred to as an improved SV image.

  Next, the signal recognizing unit 108 extracts the feature point from the improved SV image that is the contrast update image, thereby specifying the region of the lighting portion of the recognition target signal device i in the ROI as the lighting region (step S17). That is, the signal recognition unit 108 extracts the lighting region of the recognition target signal device i from the improved SV image by applying the SURF algorithm to the improved SV image.

  Next, the signal recognition unit 108 recognizes the lighting color of the recognition target signal device i based on the color of the specified lighting region (step S18). The color of the lighting area may be a color expressed by the RGB color space in the ROI lighting area, or may be a color expressed by the H (Hue) channel in the HSV color space. That is, the signal recognition unit 108 recognizes the lighting color of the recognition target signal device i by performing Hue filtering on the lighting region.

  Next, the signal recognition unit 108 determines whether or not the lighting color of the recognition target signal device i has been recognized (step S19). Here, when it is determined that the lighting color of the recognition target signal device i cannot be recognized (NO in step S19), the signal recognition unit 108 uses the machine learning to turn on the recognition target signal device i from the ROI image. A color is recognized (step S20).

  Next, the probability distribution calculation unit 106 calculates a posterior probability distribution for updating the prior probability distribution calculated in step S14 (step S21). After the posterior probability distribution is calculated, when the prior probability distribution is calculated again in step S14, the prior probability distribution reflecting the posterior probability distribution is calculated.

  And when the several recognition object signal apparatus is found by step S12, the signal recognition part 108 outputs the lighting color recognized with respect to each recognition object signal apparatus by the process of step S13-S21 as a signal state (step S22). .

  Hereinafter, each process of step S12-S21 is demonstrated in detail.

[Search for visible signal / recognized signal]
FIG. 5 is a diagram for explaining a search for a visible signal and a signal to be recognized.

Based on the distance between the current position of the vehicle V10 and the traffic signal and the posture angle, the traffic signal search unit 104 searches the map information for a plurality of traffic signals around the vehicle V10 and facing the vehicle V10. Then, as shown in FIG. 5, the traffic signal search unit 104 selects a traffic signal whose distance to the vehicle V10 is within dv [m] from among a plurality of traffic signals facing the vehicle V10 shown in the map information. Search as. Further, the traffic signal search unit 104 searches for a traffic signal having a distance to the vehicle V10 within d _T [m] (dv> d _T ) as a recognition target traffic signal from the plurality of traffic signals indicated in the map information. . Note that these thresholds dv and d _T is set by the terms of the resolution and the lens of the camera 101.

[Calculation of ROI]
FIG. 6 is a diagram for explaining calculation of ROI.

  The ROI calculation unit 105 calculates the ROI in the image for each visible light signal searched by the traffic signal search unit 104. That is, as shown in FIG. 6, the ROI calculation unit 105 calculates ROI1 for the traffic signal TS1, calculates ROI2 for the traffic signal TS2, calculates ROI3 for the traffic signal TS3, and calculates the ROI3 for the traffic signal TS4. ROI4 is calculated.

Specifically, the ROI calculation unit 105 uses the latitude and longitude indicated in the map information to determine the position of the traffic light in the absolute coordinate system as the three-dimensional position Xv = [x _v , y _v , z _{v in the} vehicle coordinate system. ] Convert to ^T. More specifically, the ROI calculation unit 105 converts the latitude and longitude of the traffic light into two-dimensional coordinates Xw = [x _w , y _w ] ^T in the plane rectangular coordinate system, and converts the two-dimensional coordinates Xw into the vehicle coordinate system. Convert to two-dimensional coordinates [x _v , y _v ] ^T. In addition, since the height of the traffic signal is not included in the map information, the ROI calculation unit 105 calculates the height z of the traffic signal in the vehicle coordinate system according to the following equation (1) from the height of the general traffic signal. _v is calculated.

In Expression (1), z ₀ is the reference height of the traffic light, φ is the pitch angle of the vehicle V10, and c _z is an arbitrary constant. Thereby, the ROI calculation part 105 obtains the position Xv of the traffic signal in the vehicle coordinate system. Next, the ROI calculation unit 105 determines the position of the vehicle coordinate system in the captured image (image coordinate system) according to the following formulas (2) to (6) using the internal parameters of the camera 101. v).

In the above formulas (2) to (6), Xs = [x _s , y _s , z _s , 1] ^T is a position in the sensor coordinate system, and Rvs is the same for converting from the vehicle coordinate system to the sensor coordinate system. The next transformation matrix, fx, fy, c _x , c _y , p ₁ and p ₂ are internal parameters indicating the state of the camera 101, respectively. From the above, the pixel position (u, v) of the traffic light in the image coordinate system is obtained. The ROI calculation unit 105 sets a rectangular ROI centered on the pixel position of this traffic light. Specifically, the ROI calculation unit 105 calculates the width w _roi and the height h _roi of the ROI based on the actual size of the traffic light according to the following equations (7) and (8).

In the above formula (7) and _{(8), s s [m} ] is the magnitude of the actual _{traffic, c roi} is an arbitrary constant for setting the size of the ROI, _{c h} is the pitch angle of the vehicle This is an arbitrary constant for enlarging the ROI in consideration of changes in

[Calculation of prior probability distribution]
The ROI set in the photographed image indicates the position where the traffic light exists. By using this ROI, the existence probability p _{t, i} (u, v) of the recognition target signal device i at the time t and the pixel position (u, v) is defined. Logarithmic odds l _{t, i} (u, v) are calculated from probabilities p _{t, i} (u, v) as shown in equation (9) below.

The probability distribution calculation unit 106 calculates a prior log odds distribution L _{t, i} ^Priority which is a probability distribution indicating the existence probability of the recognition target signal device i at each pixel position in the ROI as shown in the following equation (10). . That is, the prior log odds distribution L _{t, i} ^Prior is given as a linear sum of the a posteriori log odds distribution L _{t−1, i} ^{Post at} the previous time and the map log odds distribution L _{t, i} ^Map obtained from the map information. .

In the above equation (10), α is a decay constant, and the map log odds distribution L _{t, i} ^Map is a probability calculated from the positional relationship between the recognition target signal i and other recognition target signals and visible signals. Distribution (that is, map probability distribution).

In the present embodiment, such a prior log odds distribution L _{t, i} ^Prior , that is, a prior probability distribution, shows a higher existence probability as it is closer to the center of the ROI.

  FIG. 7 is a diagram for explaining the calculation of the map log odds distribution.

The probability distribution calculation unit 106 calculates the map log odds distribution L _{t, i} ^Map according to the following steps 1 to 5.

(Step 1) The probability distribution calculation unit 106 initializes the map log odds distribution L _{t, i} ^Map indicating the existence probability of the recognition target signal device i at each pixel position in the ROI of the recognition target signal device i to 0.

  (Step 2) The probability distribution calculation unit 106 classifies the ROI of the recognition target signal device i as a positive ROI.

  (Step 3) The probability distribution calculation unit 106 classifies the ROIs of other recognition target signals and visible signals as negative ROIs.

  (Step 4) The probability distribution calculation unit 106 sets a positive template that is a fixed probability distribution or a log odds distribution as the existence probability of the recognition target signal device i at each pixel position in the positive ROI. Furthermore, the probability distribution calculation unit 106 sets a negative template that is a fixed probability distribution or a log odds distribution as the existence probability of the recognition target signal device i at each pixel position in the negative ROI.

(Step 5) The probability distribution calculation unit 106 resizes each of the positive template and the negative template to the size of the ROI corresponding to the template. Further, the probability distribution calculation unit 106 arranges each of the positive template and the negative template at the position of the ROI corresponding to the template. At this time, the probability distribution calculation unit 106 adds the log odds of each pixel position of the positive template to the map log odds distribution L _{t, i} ^Map initialized to 0 in Step 1. Further, when there is a region that overlaps at least part of the positive ROI and the negative ROI, the probability distribution calculation unit 106 adds the log odds of the negative template in the region to the map log odds distribution L _{t, i} ^Map . .

  Here, the positive template and the negative template are defined using a Gaussian distribution, for example. However, logarithmic odds are a constant value in the vertical direction in the positive template in consideration of the variation of the pitch angle.

Thus, the probability distribution calculation unit 106 adjusts the positive template indicating the probability distribution in the two-dimensional space predetermined for the traffic signal to be recognized to the size of the positive ROI. Further, the probability distribution calculation unit 106 adjusts the negative template indicating the probability distribution in the two-dimensional space predetermined for the traffic lights not to be recognized to the size of the negative ROI. This positive template shows a higher existence probability as it is closer to the center in the horizontal direction of the two-dimensional space, and a negative template shows a lower existence probability as it is closer to the center in the horizontal direction and the vertical direction of the two-dimensional space. Then, the probability distribution calculation unit 106 adds the positive template and the negative template whose sizes are adjusted according to the arrangement of the positive ROI and the negative ROI. Thereby, the probability distribution calculation unit 106 calculates a map probability distribution indicating the existence probability of the recognition target signal device i at each position in the positive ROI as a map log odds distribution L _{t, i} ^Map . Further, the probability distribution calculation unit 106 calculates the prior log odds distribution L _{t, i} ^Prior based on the map log odds distribution L _{t, i} ^Map as shown in the above equation (10).

[SV image generation and adaptive contrast improvement processing]
The image processing unit 107 emphasizes the lighting region (that is, the contrast update image) in order to easily detect the region (lighting region) corresponding to the lighting portion of the recognition target signal device i from the image in the ROI. Create In general, since the lighted portion of the traffic light displayed in the captured image has high saturation and brightness, the image processing unit 107 creates a contrast update image using this characteristic.

  For example, the image processing unit 107 converts an RGB image of ROI into an HSV image, and creates an SV image by multiplying the S channel and the V channel in the HSV image. The image processing unit 107 generates a contrast updated image (that is, an improved SV image) by performing an adaptive contrast improvement process on the SV image.

  FIG. 8 is a diagram illustrating an example of an RGB image, an S channel, a V channel, and an SV image. In order to make the subject and the background easy to understand, FIG. 8A shows an RGB image on which a traffic light that is lit red is displayed as an image expressed in white and black. Moreover, in (a) of FIG. 8, the lighting part currently lighted in red of the traffic light is shown as a white circle.

  For example, the image processing unit 107 converts the RGB image shown in (a) of FIG. 8 into the S channel and V channel of the HSV image shown in (b) and (c) of FIG. Is multiplied to generate the SV image shown in FIG.

  The SV image shown in (d) of FIG. 8 is an image in which areas with high saturation and lightness are emphasized in white in the image. In a general scene, the lighting area of the recognition target signal device i can be extracted by directly using the SV image. However, an SV image in which the lighting area is sufficiently emphasized may not be obtained depending on the brightness of the entire SV image. In order to prevent such a case, it is conceivable to normalize the contrast of the SV image.

  However, by normalizing the contrast of the SV image, areas corresponding to features such as signs around the road or other lighting objects are also emphasized, and there is a possibility that misrecognition of traffic signals will be induced.

Therefore, in the present embodiment, the image processing unit 107 performs adaptive contrast improvement processing using the prior log odds distribution L _{t, i} ^Priority calculated by the probability distribution calculation unit 106 as an adjustment parameter. That is, the image processing unit 107 performs an adaptive contrast improvement process on the SV image, thereby improving the SV image in which the region corresponding to the lighting portion of the recognition target signal device (lighting region) is emphasized in white. Convert to image.

  Specifically, the image processing unit 107 updates the pixel value of the SV image according to the following equations (11) and (12) in the adaptive contrast improvement processing.

In the above equations (11) and (12), I (u, v) is the pixel value of the SV image that is the input image, and I ′ (u, v) is the updated pixel value. G _thresh and G are arbitrary constants. Therefore, in the adaptive contrast improvement processing, when the prior probability p _{t, i} ^Prior (u, v) is larger than g _thresh , the contrast (that is, the pixel value) of the pixel at the position (u, v) is increased, and g _thresh When it is smaller, it has the effect of reducing the contrast of the pixel at that position. Accordingly, the contrast of the pixel increases in the region where the existence probability is high, and the contrast decreases in the region where the existence probability is low, so that the feature away from the recognition target signal device i is not easily recognized. Further, in order to improve the recognition accuracy of a remote traffic signal, the image processing unit 107 applies a morphological operation when the distance between the vehicle V10 and the traffic signal i to be recognized is equal to or greater than d _far [m], and SV You may further emphasize the area | region corresponding to the lighting part in an image.

As described above, the image processing unit 107 converts the contrast of the ROI image expressed only by the saturation and lightness out of the hue, saturation, and lightness constituting the color space into the prior log odds distribution L _{t, i} ^Priority . Therefore, update.

[Recognition of signal to be recognized and recognition of its status]
When the improved SV image is obtained, the signal recognizing unit 108 extracts an area highlighted in white in the improved SV image from the improved SV image as a lighting area where a lighting portion of the recognition target signal device i exists. And the signal recognition part 108 recognizes a traffic signal from the color of the image of the lighting area.

  In the traffic signal recognition method using adaptive contrast improvement processing, the shape of the entire traffic light in the image affected by day and night is not considered so that the traffic signal can be recognized day and night.

  That is, the signal recognizing unit 108 extracts a feature point from the improved SV image, thereby extracting a circular white emphasized region (hereinafter referred to as a circular region). This circular area corresponds to the lighting area. In other words, the signal recognizing unit 108 extracts the feature point from the improved SV image, thereby specifying the area of the lighting portion of the recognition target signal device i in the ROI as the lighting area. And the signal recognition part 108 recognizes the state of the recognition target signal apparatus i from the color of the lighting area. The color of the lighting area may be the hue of the lighting area represented by the ROI HSV color space or the color of the lighting area represented by the RGB color space. The state of the recognition target signal i is the lighting color of the recognition target signal i, that is, a traffic signal.

  Specifically, the signal recognition unit 108 extracts the feature points of the circular area by applying the SURF algorithm. In the SURF algorithm, feature points such as edges can be extracted from changes in luminance of an image.

  FIG. 9 is a diagram illustrating the types of feature points classified according to the determinant value det (H) of the Hessian matrix.

As shown in FIG. 9, the types of feature points can be classified according to the determinant value det (H) of the Hessian matrix. Therefore, the signal recognizing unit 108 extracts feature points having det (H) equal to or greater than a certain value c _surf (> 0) as a circular region.

  Here, the radius r of the circular region can be calculated by the following equation (13) using the scale value σ of the feature point.

  r = (1.2 × 2.0σ) /9.0 (13)

On the other hand, when the diameter of the lighting part of the traffic light is d ₁ [m], the radius r ₁ of the lighting part of the traffic light at the distance z _s [m] is as shown in the following formula (14).

r ₁ = 0.5fxd ₁ / z _s (14)

As described above, if the improved SV image includes a plurality of circular regions, the signal recognition unit 108 has a circular region in which the radius r of the circular region is in the range of c _min r ₁ or more and c _max r ₁ or less. Extract regions. However, c _min and c _max are arbitrary constants. When there are a plurality of circular regions in the improved SV image corresponding to one ROI, the signal recognizing unit 108 recognizes the plurality of circular regions whose radius r is in the range from c _min r _{1 to} c _max r _1. Among the circular regions, a circular region having the largest det (H) is extracted.

  The signal recognition unit 108 treats the circular area thus extracted as a lighting area, and recognizes the state of the recognition target signal device i by evaluating the color (specifically, the Hue value) of the lighting area.

  FIG. 10 is a diagram illustrating the relationship between the range of Hue values and the state of traffic lights.

  The signal recognizing unit 108 holds, for example, a table shown in FIG. 10 in which a plurality of ranges of Hue values are associated with traffic light states (that is, lighting colors) corresponding to the plurality of ranges. Then, the signal recognition unit 108 calculates an average Hue value in the lighting area, and applies Hue filtering to the calculated average Hue value. That is, the signal recognition unit 108 recognizes the state associated with the range including the average hue value in the table shown in FIG. 10 as the state of the recognition target signal device i, that is, the lighting color of the recognition target signal device i. Thereby, a traffic signal is recognized.

[Recognition of lighting colors by machine learning]
As described above, the signal recognition unit 108 attempts to recognize a traffic signal using the improved SV image that is the result of the adaptive contrast improvement processing. However, depending on the sunshine conditions around the traffic signal to be recognized, the lit part may not appear bright, so that the recognition may not be performed properly. Therefore, if the traffic signal, that is, the lighting color cannot be recognized even by using the result of the adaptive contrast improvement processing, the signal recognition unit 108 recognizes the lighting color by a recognition method using machine learning. In this recognition method using machine learning, the signal recognition unit 108 finds the shape of the entire recognition target signal device i from the ROI image (for example, RGB image), and sets the lighting color of the lighting part included in the entire recognition target signal device i. recognize. This recognition method using machine learning is, for example, a non-patent document (D. Barnes, W. Maddern and I. Posner: Exploiting 3D Semantic Scene Priors for Online Traffic Light Interpretation, Proceedings of the 2015 IEEE IVS, p. 573-578. , (2015)), which uses a discriminator using Haar-like features and Adaboost. That is, in the algorithm, a rectangular frame corresponding to the shape of the entire signal to be recognized i is detected from the ROI image, and binarization processing is performed on the hue value in the area corresponding to each lighting part in the rectangular frame. To recognize the lighting color of the area having a high Hue value.

  In addition, since the shape of the traffic light in a photographed image becomes unclear at night, it may be difficult to recognize a traffic signal by a recognition method using such machine learning. Therefore, in the present embodiment, the signal recognition unit 108 first attempts to recognize a traffic signal by a recognition method using the result of the adaptive contrast improvement process, and as a result, when the traffic signal could not be recognized. Only traffic signals are recognized by a recognition method using machine learning.

[Calculation of posterior probability distribution]
When the state of the recognition target signal device i is recognized, the probability distribution calculation unit 106 updates the prior log odds distribution used for the recognition, and calculates the posterior log odds distribution that is the posterior probability distribution.

  FIG. 11 is a diagram illustrating an example of an RGB image, a prior log odds distribution, an observed log odds distribution, and a posterior log odds distribution. In order to make the subject and the background easy to understand, FIG. 11A shows a color photographed image in which the recognition target signal device i that lights in red and other traffic signals that light in green are projected. It shows as an image expressed by. Moreover, in (a) of FIG. 11, the lighting part currently lit in green and red of each traffic light is shown as a white circle.

As shown in FIG. 11C, the probability distribution calculation unit 106 observes the logarithmic odds distribution L _t that is an observation probability distribution indicating the existence probability of the recognition target signal device i at each pixel position with respect to the RGB image of ROI. _{, I} ^Obs . This observed log odds distribution L _{t, i} ^Obs is a probability distribution showing high log odds in the region corresponding to the entire recognition target signal device i whose lighting color is recognized in the ROI. As shown in the following equation (15), the probability distribution calculation unit 106 uses the observed log odds distribution L _{t, i} ^Obs as the prior log odds distribution L _{t, i} indicating the prior log odds of each pixel position in the ROI. ^The posterior log odds distribution L _{t, i} ^Post is calculated by adding to ^Priority .

For example, the probability distribution calculation unit 106 calculates the prior log odds distribution L _{t, i} ^Prior shown in FIG. 11B for the positive ROI of the RGB image shown in FIG. Then, the probability distribution calculation unit 106 generates an observed log odds distribution L _{t, i} ^Obs indicating high log odds at the position of the recognition target signal device i recognized in the positive ROI, as shown in FIG. To do. Next, the probability distribution calculation unit 106 adds the observed log odds distribution L _{t, i} ^Obs to the prior log odds distribution L _{t, i} ^Prior to add the posterior log odds distribution L _{t, i} shown in FIG. _i ^Post is calculated.

As shown in (d) of FIG. 11, in the posterior log odds distribution L _{t, i} ^Post , the log odds or probability of the recognized recognition target signal device i region is higher than its surroundings.

The probability distribution calculation unit 106 uses the posterior log odds distribution L _{t, i} ^Post calculated in this way, as shown in the above equation (10), the prior log odds distribution L _{t, i} ^Prior at the next timing. Is calculated.

That is, in the traffic signal recognition method according to the present embodiment, when the lighting color of the recognition target signal device i is recognized at time t, the closer to the lighting region specified for the recognition target signal device i in the ROI, the closer it is. showing high existence probability, observation log odds distribution _L ^t of the time ^t, to calculate the ^{i Obs,} the observation log odds distribution _L ^t, pre-log odds distribution of the ^{i Obs} time ^t _L ^t, by adding to ^{i prior} The posterior log odds distribution L _{t, i} ^Post at time t is calculated.

When the camera 101 further acquires a subsequent image that is a captured image at time (t + 1) following the captured image at time t, the ROI calculation unit 105 calculates the ROI again from the subsequent image. The probability distribution calculation unit 106 calculates the map log odds distribution L _{t + 1, i} ^Map at time (t + 1) for the ROI calculated again, and the posterior logarithmic odds at time t to the map log odds distribution L _{t + 1, i} ^Map. The prior log odds distribution L _{t, i} ^Prior at time t is updated to the prior log odds distribution L _{t + 1, i} ^Prior at time (t + 1) by weighted addition of the distributions L _{t, i} ^Post . Then, the image processing unit 107 generates an improved SV image for the recalculated ROI according to the updated prior log odds distribution, that is, the prior log odds distribution L _{t + 1, i} ^{Prior at} time (t + 1).

  Thus, by calculating the prior log odds distribution, the probability of the pixel position where the recognition target signal device i exists or the log odds can be increased more appropriately. As a result, the recognition accuracy of the lighting color by the signal recognition unit 108 can be improved.

[Evaluation experiment]
Hereinafter, an experiment for evaluating the recognition accuracy of traffic signals by the traffic signal recognition method according to the present embodiment will be described. In this evaluation experiment, a vehicle V10 equipped with the traffic signal recognition device 100 was run along a general road, and data indicating the traffic signal recognized by the traffic signal recognition device 100 at that time was collected.

  The ordinary road is 7.7 [km], and the vehicle V10 reciprocates on the ordinary road. At this time, the vehicle V10 passes through a place where 38 traffic lights are installed. The camera 101 of the traffic signal recognition apparatus 100 acquires 7,958 shot images by shooting while the vehicle V10 is traveling. These captured images were 4,832 images with a blue traffic light reflected, 111 images with a yellow traffic light reflected, and a red traffic light. It is composed of 3,015 images. From these captured images, the recognition accuracy of the traffic signal recognition method using machine learning in the present embodiment and the traffic signal recognition method not using machine learning in the present embodiment is evaluated. The traffic signal recognition method using machine learning in the present embodiment and the traffic signal recognition method not using machine learning in the present embodiment are hereinafter referred to as traffic signal recognition methods m1 and m2, respectively. In the evaluation, the recognition result by the two reference traffic signal recognition methods (hereinafter referred to as the reference recognition method Bm1 and the reference recognition method Bm2) and the recognition result by the traffic signal recognition methods m1 and m2 in the present embodiment Compare In the reference recognition method Bm1, the RGB image is converted into an SV image, and the lighting area of the traffic light is specified from the SV image without changing the contrast of the SV image. In the reference recognition method Bm2, the contrast of the SV image is normalized, and the lighting area of the traffic light is specified from the normalized SV image.

Here, the above parameters in this evaluation experiment are as follows: d _T = 150 [mm], d _V = 300 [mm], c _z = 0.25, c _roi = 3.0, c _h = 0.1, s. _s = 1.2 [mm], α = 0.7, d _l = 0.4 [mm], c _surf = 20,000, c _max = 2.5, c _min = 0.5, g _thresh = 0 .65, G = 10.0, d _far = 120 [mm]. Each parameter was determined based on the performance of the camera 101 and experience.

  FIG. 12 is a diagram illustrating a traffic signal recognition result by each recognition method.

  In FIG. 12, the correct answer rate (TP: True Positive) and the false recognition rate (FP: False Positive) of the traffic signal recognition methods m1 and m2 and the reference recognition methods Bm1 and Bm2 in the present embodiment, respectively. The unrecognized rate (FN: False Negative) is shown. The erroneous recognition rate FP includes an erroneous recognition rate that erroneously recognizes the lighting color of the traffic signal and an erroneous recognition rate that recognizes the background color as the lighting color. The correct answer rate is a rate at which the lighting color of the traffic light is correctly recognized, and the unrecognized rate is a rate at which the lighting color of the traffic signal is not recognized.

  From the recognition result shown in FIG. 12, the traffic signal recognition methods m1 and m2 in the present embodiment using the improved SV image are more accurate than the standard recognition methods Bm1 and Bm2 using the general SV image. Can be confirmed. Further, in the traffic signal recognition methods m1 and m2 in the present embodiment, it is possible to confirm the reduction of the unrecognized rate FN.

  On the other hand, in traffic signal recognition methods m1 and m2 in the present embodiment, erroneous recognition of lighting colors of 10% or more in yellow and red was confirmed. That is, it was confirmed that each of yellow and red was mistakenly recognized as the other color. However, in the traffic signal recognition method m1 in the present embodiment, since the shape of the entire traffic signal is recognized by machine learning, it is possible to reduce future misrecognition by considering the position of the lighting portion with respect to the overall shape of the traffic signal. .

  FIG. 13 is a diagram illustrating an SV image, an improved SV image obtained by adaptive contrast improvement processing, and a prior log odds distribution in a typical driving scene. Specifically, FIG. 13 shows an example in which an RGB image in each ROI of a plurality of consecutive captured images, an SV image corresponding to the RGB image, and adaptive contrast improvement processing are performed on the SV image. The obtained improved SV image and the prior log odds distribution used for the adaptive contrast improvement processing are shown. In order to make the subject and the background easy to understand, in FIG. 13, the RGB image on which the recognition target signal lighted in red is projected is shown as an image expressed in white and black.

  In a normal SV image, lighting objects (other traffic lights, etc.) other than the recognition target traffic light are also emphasized. However, in the improved SV image, the contrast is improved according to the existence probability of the recognition target signal device at each pixel position. Therefore, it is possible to suppress the lighting object other than the recognition target signal from being emphasized.

  FIG. 14 is a diagram showing the correct rate and the unrecognized rate according to the distance to the recognition target traffic signal in the traffic signal recognition method m2 in the present embodiment in comparison with the reference recognition method Bm2.

  As shown in FIG. 14, in the traffic signal recognition method m2 in the present embodiment, even if the recognition target traffic signal is far away, the lighting colors (green and red) are not overlooked as in the case of being near. Can be recognized correctly.

  As described above, in the traffic signal recognition method according to the present embodiment, the frequency of erroneous recognition and unrecognition can be suppressed even in a situation where many traffic lights are lined up.

[Summary]
As described above, the traffic signal recognition method according to the present embodiment is a traffic signal recognition method for recognizing the lighting color of a traffic light installed on a road as a traffic signal, and is obtained by photographing with the camera 101 mounted on the vehicle V10. Based on the image acquisition step (S11) for acquiring a captured image, map information indicating the position and orientation of a plurality of traffic lights installed around the vehicle V10, and the state of the vehicle V10, among the plurality of traffic lights, Based on the search step (S12) for searching for the first traffic light facing the vehicle V10 and its map information, the state of the vehicle V10 and the state of the camera, a region including the first traffic light is selected from the captured images. Region-of-interest calculation step (S13) to be calculated as one region of interest, and the existence probability of the first traffic light at each position in the calculated first region of interest A distribution calculating step (S14) for calculating a prior probability distribution to be shown, and a contrast updating step (S16) for generating a contrast update image by updating the contrast of the image of the first region of interest according to the prior probability distribution. A lighting area specifying step (S17) for specifying the area of the lighting portion of the first traffic light in the first region of interest as a lighting area by extracting feature points from the contrast update image, and the specified lighting area A recognition step (S18) for recognizing the lighting color of the first traffic light based on the color of the first traffic light.

The first traffic light is the above-described recognition target traffic light, and the region of interest is the above-described ROI. Further, the prior probability distribution may be, for example, the above-mentioned prior log odds distribution L _{t, i} ^Prior or the map log odds distribution L _{t, i} ^Map .

  Thereby, the contrast update image is generated by updating the contrast of the image of the first region of interest according to the prior probability distribution. Therefore, in the contrast update image, it is possible to increase the contrast of pixel values at a position where the probability that the first traffic light exists is high and a position where the probability that the first traffic light exists is low. Since the lighting area is specified based on the contrast update image, the lighting corresponding to the lighting portion of the first traffic light is included even if other traffic lights other than the first traffic light are included in the first region of interest. A region can be accurately identified from its first region of interest. As a result, the lighting color of the first traffic light can be properly recognized, and the traffic signal recognition accuracy can be improved.

  Further, the prior probability distribution may indicate a higher existence probability as it is closer to the center of the first region of interest.

  Thereby, it is possible to appropriately calculate the prior probability distribution for the first region of interest calculated so that the first traffic light is arranged in the center.

  In the contrast update step (S16), the pixel value at each position in the first region of interest is changed to a larger pixel value as the existence probability of the position indicated by the prior probability distribution is higher. The contrast of the image of the region of interest may be updated.

  Thereby, in the contrast update image, it is possible to appropriately increase the contrast of pixel values at a position where the probability that the first traffic light exists is high and a position where the probability that the first traffic light exists is low.

  In the search step (S12), a second traffic signal is further searched as a traffic signal facing the vehicle V10 from the plurality of traffic signals. In the region of interest calculation step (S13), the map information, the state of the vehicle V10, and Based on the state of the camera 101, a region including the second traffic light is further calculated as a second region of interest from the photographed image. In the distribution calculation step (S14), a predetermined traffic signal is determined in advance. The first template showing the probability distribution in the two-dimensional space is adjusted to the size of the first region of interest, and the first template showing the probability distribution in the two-dimensional space predetermined for the traffic lights not to be recognized. The two templates are adjusted to the size of the second region of interest, and the first and second templates respectively adjusted in size are arranged in the first and second regions of interest. Thus by adding, to calculate a map probability distribution showing a probability at each position in the first ROI may calculate a prior probability distribution on the basis of the map probability distribution.

  The first and second regions of interest correspond to a positive ROI and a negative ROI, for example, as shown in FIG. 7, and the first and second templates are, for example, as shown in FIG. Corresponds to a negative template.

  Thereby, even if the second traffic signal is included in the first region of interest, the prior probability that more accurately indicates the existence probability of the first traffic signal based on the map information, the state of the vehicle V10, and the state of the camera 101 Distribution can be calculated.

  The first template has a higher existence probability as it is closer to the center in the horizontal direction of the two-dimensional space, and the second template is lower as it is closer to the center in the horizontal direction and the vertical direction of the two-dimensional space. An existence probability may be indicated.

  Thereby, as shown in FIG. 7, the prior probability distribution which shows more accurately the existence probability of the first traffic light can be calculated.

  Further, the traffic signal recognition method further shows a higher probability of existence in the first region of interest as it is closer to the specified lighting region when the lighting color of the first traffic light is recognized in the recognition step (S18). It includes an a posteriori distribution calculating step (S21) for calculating an observation probability distribution and adding the observation probability distribution to the a priori probability distribution to calculate the a posteriori probability distribution. When a subsequent image, which is a subsequent captured image, is further acquired, the region of interest calculation step (S13) further calculates the first region of interest from the subsequent image again, and the distribution calculation step (S14) further again By calculating the map probability distribution for the calculated first region of interest and adding the posterior probability distribution to the map probability distribution Update the prior probability distribution, in contrast updating step (S16), further, the first generation of contrast updated image region of interest, may be performed according to the updated prior probability distribution calculated again.

  The observation probability distribution and the posterior probability distribution correspond to, for example, the probability log odds distribution and the posterior log odds distribution shown in FIG.

  Thereby, after the lighting color of the first traffic light is recognized, an observation probability distribution that indicates a higher probability of existence as it is closer to the lighting area corresponding to the lighting portion of the first traffic light is reflected in the prior probability distribution. . Therefore, since the past recognition result of the first traffic light is reflected in the existence probability of the first traffic light indicated by the prior probability distribution, the prior probability distribution more accurately indicates the presence probability of the first traffic light. Can be shown. By using the prior probability distribution, the lighting color recognition accuracy of the first traffic light can be further increased.

  Further, the traffic signal recognition method further uses the machine learning to obtain the first signal from the first region of interest when the lighting color of the first traffic light cannot be recognized in the recognition step (S18). A re-recognition step (S20) for recognizing the lighting color of the traffic light may be included.

  Thus, for example, even if the lighting color of the traffic light displayed in the photographed image is dark due to sunshine conditions or the like, even if the lighting color cannot be correctly recognized in the recognition step (S18), the machine learning does not The lighting color can be recognized.

  Further, in the contrast update step, the contrast of the image of the first region of interest expressed only by the saturation and lightness out of the hue, saturation and lightness constituting the color space may be updated.

  Thereby, the contrast update image which emphasized the lighting area corresponding to the lighting part of the 1st traffic light can be produced | generated, and a lighting area can be pinpointed more correctly.

  In the above embodiment, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software that realizes the traffic signal recognition apparatus 100 according to the above-described embodiment is a program that causes a computer to execute each step included in the flowchart shown in FIG.

  As described above, the traffic signal recognition method and the traffic signal recognition device according to one aspect of the present invention have been described based on the embodiment, but the present invention is not limited to this embodiment. Unless it deviates from the meaning of the present invention, various modifications conceived by those skilled in the art may be included in the scope of the present invention.

  For example, in the above embodiment, a rectangular ROI is calculated, but the shape of the ROI is not limited to a rectangular shape, and may be another shape (for example, a circle or a triangle).

  In the above embodiment, the SURF algorithm is used for extracting feature points. However, other algorithms such as SIFT (Scale-Invariant Feature Transform) may be used.

  The traffic signal recognition method according to an aspect of the present invention has an effect of improving the recognition accuracy of traffic signals. For example, a traffic signal recognition device or an automatic driving support device mounted on a vehicle for automatic driving. Etc.

DESCRIPTION OF SYMBOLS 100 Traffic signal recognition apparatus 101 Camera 102 Vehicle state detection part 103 Map information holding part 104 Traffic signal search part 105 ROI calculation part 106 Probability distribution calculation part 107 Image processing part 108 Signal recognition part V10 Vehicle

Claims (9)

A traffic signal recognition method for recognizing a lighting color of a traffic light installed on a road as a traffic signal,
An image acquisition step of acquiring a photographed image by photographing with a camera mounted on the vehicle;
Search for searching for a first traffic signal facing the vehicle from among the plurality of traffic signals based on map information indicating positions and directions of the traffic signals installed around the vehicle and a state of the vehicle. Steps,
A region-of-interest calculation step of calculating a region including the first traffic light as a first region of interest from the captured image based on the map information, the state of the vehicle, and the state of the camera;
A distribution calculating step of calculating a prior probability distribution indicating the existence probability of the first traffic light at each position in the calculated first region of interest;
A contrast update step of generating a contrast update image by updating the contrast of the image of the first region of interest according to the prior probability distribution;
A lighting area specifying step of specifying a lighting area of the first traffic light in the first region of interest as a lighting area by extracting feature points from the contrast update image;
A recognition step of recognizing a lighting color of the first traffic light based on the identified color of the lighting area. The traffic signal recognition method according to claim 1, wherein the prior probability distribution indicates a higher existence probability as being closer to a center of the first region of interest. In the contrast update step,
By changing the pixel value of each position in the first region of interest to a larger pixel value as the existence probability of the position indicated by the prior probability distribution is higher, the image of the first region of interest The traffic signal recognition method according to claim 1 or 2, wherein the contrast is updated. In the search step,
Search for a second traffic signal as a traffic signal facing the vehicle from among the plurality of traffic signals,
In the region of interest calculation step,
Based on the map information, the state of the vehicle, and the state of the camera, a region including the second traffic light is further calculated as a second region of interest from the captured image.
In the distribution calculating step,
Adjusting a first template indicating a probability distribution in a predetermined two-dimensional space for a traffic signal to be recognized to a size of the first region of interest;
Adjusting a second template indicating a probability distribution in a predetermined two-dimensional space for a traffic light that is not a recognition target to the size of the second region of interest;
By adding the first template and the second template, each of which has been adjusted in size, according to the arrangement of the first and second regions of interest, the probability at each position in the first region of interest is obtained. Calculate the map probability distribution shown,
The traffic signal recognition method according to claim 1, wherein the prior probability distribution is calculated based on the map probability distribution. The first template has a higher existence probability as it is closer to the center in the horizontal direction of the two-dimensional space, and the second template has a lower existence probability as it is closer to the center in the horizontal direction and the vertical direction of the two-dimensional space. The traffic signal recognition method according to claim 4. The traffic signal recognition method further includes:
When the lighting color of the first traffic light is recognized by the recognition step, an observation probability distribution indicating a higher existence probability is calculated in the first region of interest as it is closer to the specified lighting region, and the observation A posterior distribution calculating step of calculating a posterior probability distribution by adding a probability distribution to the prior probability distribution;
When a subsequent image that is a captured image subsequent to the captured image is further acquired by the image acquisition step,
In the region of interest calculation step,
Recalculating the first region of interest from the subsequent image;
In the distribution calculating step,
Calculating the map probability distribution for the first region of interest calculated again;
Updating the prior probability distribution by adding the posterior probability distribution to the map probability distribution;
In the contrast update step,
The traffic signal recognition method according to claim 4 or 5, wherein generation of the contrast update image for the first region of interest calculated again is performed according to the updated prior probability distribution. The traffic signal recognition method further includes:
Re-recognition of recognizing the lighting color of the first traffic light from the image of the first region of interest using machine learning when the lighting color of the first traffic light could not be recognized in the recognition step. The traffic signal recognition method according to claim 1, comprising steps. In the contrast update step,
The contrast of the image of the first region of interest expressed only by saturation and lightness out of hue, saturation and lightness constituting the color space is updated. Traffic signal recognition method. A traffic signal recognition device for recognizing a lighting color of a traffic light installed on a road as a traffic signal,
A camera mounted on the vehicle that captures the captured image by shooting,
Search for searching for a first traffic signal facing the vehicle from among the plurality of traffic signals based on map information indicating positions and directions of the traffic signals installed around the vehicle and a state of the vehicle. And
A region-of-interest calculator that calculates, as a first region of interest, a region including the first traffic light from the captured image based on the map information, the state of the vehicle, and the state of the camera;
A probability distribution calculating unit that calculates a prior probability distribution indicating the existence probability of the first traffic light at each position in the calculated first region of interest;
An image processing unit that generates a contrast update image by updating the contrast of the image of the first region of interest according to the prior probability distribution;
By extracting a feature point from the contrast update image, the lighting area of the first traffic light in the first region of interest is identified as a lighting area, and based on the color of the identified lighting area, A traffic signal recognition device comprising: a signal recognition unit that recognizes a lighting color of the first traffic light.