WO2024161469A1 - Traffic signal recognition method and traffic signal recognition device - Google Patents

Abstract

In the present invention, a traffic signal recognition method and a traffic signal recognition device assess the current display state for each traffic signal device in each of a plurality of images that are captured in the advancement direction of a vehicle, and acquire a result column for each traffic signal device, the result column being composed of a plurality of assessment results in chronological order. When there are traffic signal devices that operate in tandem among the plurality of traffic signal devices, the result column for each of the traffic signal devices that operate in tandem are integrated across the traffic signal devices that operate in tandem into an integrated column, and the current display state pertaining to the most numerous assessment results among the assessment results that constitute the integrated column is outputted as the current display state based on the traffic signal devices.

Description

Traffic light recognition method and traffic light recognition device

The present invention relates to a traffic light recognition method and a traffic light recognition device.

Patent Document 1 proposes a traffic light recognition technology that accumulates the traffic light color (current display state) for each frame across multiple frames, and performs a majority vote on the traffic light color results for multiple frames to determine the latest traffic light color.

International Publication No. 2019/177019

However, the technology described in Patent Document 1 involves majority voting on the results of the current signal status of a traffic light in multiple frames, which creates the problem of a delay between when the current signal status of a traffic light changes and when it is actually determined that the current signal status has changed.

The present invention was made in consideration of the above problems, and its purpose is to provide a traffic light recognition method and a traffic light recognition device that can reduce the delay between when the current signal state of a traffic light changes and when it is actually determined that the current signal state has changed.

In order to solve the above-mentioned problems, a traffic light recognition method and traffic light recognition device according to one embodiment of the present invention targets multiple images captured in the direction of vehicle travel, determines the current status of each traffic light for each image, and obtains a result sequence consisting of multiple determination results in chronological order for each traffic light. If there are interlocking traffic lights among the multiple traffic lights, the result sequences for each interlocking traffic light are integrated across the interlocking traffic lights to generate an integrated sequence, and the current status associated with the most frequent determination result among the determination results constituting the integrated sequence is output as the current status based on the traffic light.

The present invention makes it possible to reduce the delay between when the current signal state of a traffic light changes and when it is actually determined that the current signal state has changed.

FIG. 1 is a block diagram showing the configuration of a traffic light recognition device according to an embodiment of the present invention. FIG. 2 is a flowchart showing a processing procedure of the traffic light recognition device according to one embodiment of the present invention. FIG. 3 is a flowchart showing the output determination process. FIG. 4 is a diagram showing an example of a result sequence made up of a plurality of determination results in chronological order. FIG. 5 shows an example of a current status output based on a result sequence for one traffic light. FIG. 6A is a diagram showing an example of integrating two result strings. FIG. 6B is a diagram showing an example of integrating three result strings. FIG. 7 is a diagram showing an example of integrating two result strings taking into account the chronological order. FIG. 8 is a diagram showing an example of the output of the current status when the number of interlocking traffic signals is reduced. FIG. 9 is a diagram showing an example of the output of the current status when the number of interlocking traffic signals is increased. FIG. 10 is a flowchart showing the process of calculating the size of a subsequence. FIG. 11 is a diagram showing an example of merging two result strings by varying the size of the substrings. FIG. 12 is a diagram showing an example of complementation of the determination results in the result string.

Next, an embodiment of the present invention will be described in detail with reference to the drawings. In the description, the same parts will be given the same reference numerals and duplicate explanations will be omitted.

[Configuration of traffic light recognition device]
Fig. 1 is a block diagram showing the configuration of a traffic light recognition device according to this embodiment. As shown in Fig. 1, the traffic light recognition device according to this embodiment includes an imaging unit 71 and a controller 100. The traffic light recognition device may include an on-board sensor 73, a map information acquisition unit 75, and a vehicle control device 400. The controller 100 is connected to the imaging unit 71, the on-board sensor 73, the map information acquisition unit 75, and the vehicle control device 400 via a wired or wireless communication path.

Here, the imaging unit 71, the on-board sensor 73, and the vehicle control device 400 are mounted on a vehicle (not shown), but the map information acquisition unit 75 and the controller 100 may be mounted on the vehicle or installed outside the vehicle.

The imaging unit 71 captures images in the direction in which the vehicle is traveling. For example, the imaging unit 71 is a digital camera equipped with a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor), and captures images of the surroundings of the vehicle to obtain digital images of the surrounding area. The imaging unit 71 captures an image of a specified range around the vehicle by setting the focal length, the lens angle of view, the vertical and horizontal angles of the camera, etc.

The captured images captured by the imaging unit 71 are output to the controller 100 and stored in a storage unit (not shown) for a predetermined period of time. For example, the imaging unit 71 captures captured images at predetermined time intervals, and the captured images captured at the predetermined time intervals are stored in the storage unit as past images. The past images may be deleted after a predetermined period of time has passed since the capture of the past images.

The on-board sensor 73 is composed of an object detection sensor, such as a laser radar, a millimeter wave radar, or a camera, mounted on the vehicle, that detects objects present around the vehicle. The on-board sensor 73 may be equipped with a plurality of different types of object detection sensors.

The on-board sensor 73 detects the environment around the vehicle. For example, the on-board sensor 73 may detect moving objects including other vehicles, motorcycles, bicycles, and pedestrians, and stationary objects including stopped vehicles, and detect the position, attitude, size, speed, acceleration, deceleration, yaw rate, etc. of the moving and stationary objects relative to the vehicle. The on-board sensor 73 may output, as a detection result, the behavior of two-dimensional objects in a zenith diagram (also called a plan view) viewed from the air above the vehicle. The on-board sensor 73 may also detect signs (road signs and signs marked on the road surface) and guide rails that exist around the vehicle. In addition, the on-board sensor 73 may detect the rotational speed or difference in rotational speed of the wheels of the vehicle to detect the slipperiness of the road surface on the lane on which the vehicle is traveling.

In addition to the environment around the vehicle, the on-board sensor 73 detects the state of the vehicle. For example, the on-board sensor 73 may detect the vehicle's moving speed (forward/backward and left/right moving speed, turning speed), the steering angle of the wheels of the vehicle, and the speed of change of the steering angle.

In addition, the on-board sensor 73 may include a sensor that measures the absolute position of the vehicle, that is, the position, attitude, and speed of the vehicle relative to a predetermined reference point, using a position detection sensor that measures the absolute position of the vehicle, such as a GPS (Global Positioning System) or odometry.

The map information acquisition unit 75 acquires map information that indicates the structure of the road on which the vehicle is traveling. The map information acquired by the map information acquisition unit 75 includes road structure information such as absolute lane positions, lane connection relationships, and relative position relationships. The map information acquired by the map information acquisition unit 75 may also include facility information such as parking lots and gas stations. In addition, the map information may include traffic light position information, traffic light type, and the position of the stop line corresponding to the traffic light. The map information acquisition unit 75 may own a map database that stores map information, or may acquire map information from an external map data server by cloud computing. The map information acquisition unit 75 may also acquire map information using vehicle-to-vehicle communication or road-to-vehicle communication.

The vehicle control device 400 controls a vehicle (not shown) based on the traffic light recognition results obtained by the controller 100. For example, the vehicle control device 400 may be a device that automatically drives the vehicle along a predetermined driving route, or a device that assists the vehicle occupant in driving operations. Additionally, the vehicle control device 400 may be a notification device that notifies the vehicle occupant of the traffic light recognition results.

The controller 100 (an example of a control unit or processing unit) is a general-purpose microcomputer equipped with a CPU (central processing unit), memory, and input/output units. A computer program (traffic light recognition program) for functioning as part of the traffic light recognition device is installed in the controller 100. By executing the computer program, the controller 100 functions as multiple information processing circuits (140, 150, 160, 165, 170, 180) equipped in the traffic light recognition device.

Here, an example is shown in which the multiple information processing circuits (140, 150, 160, 165, 170, 180) of the traffic light recognition device are realized by software. However, it is also possible to configure the information processing circuits (140, 150, 160, 165, 170, 180) by preparing dedicated hardware for executing each of the information processes shown below. Also, the multiple information processing circuits (140, 150, 160, 165, 170, 180) may be configured by individual hardware. Furthermore, the information processing circuits (140, 150, 160, 165, 170, 180) may also be used as electronic control units (ECUs) used for other vehicle-related controls.

The controller 100 includes a detection area setting unit 140, a judgment unit 150, a judgment result storage unit 160, an integration unit 165, an output determination unit 170, and an output unit 180 as multiple information processing circuits (140, 150, 160, 165, 170, 180).

The detection area setting unit 140 sets a detection area corresponding to a traffic light on the image captured by the imaging unit 71. Here, "detection area" means an area in the image where a traffic light is estimated to be present. The detection area setting unit 140 sets a detection area for each traffic light, for one or more traffic lights estimated to be present in the image.

The position of the traffic light captured within the imaging range in the image can be estimated based on the imaging direction of the imaging unit 71, the position and posture of the vehicle at the time of imaging, and the position of the traffic light. The detection area setting unit 140 sets, for example, a part of the captured image that includes the estimated position of the traffic light in the image as the detection area.

The detection area setting unit 140 may extract, from among the traffic lights captured within the imaging range, those that correspond to the road on which the vehicle is traveling as target traffic lights, and set a detection area for each target traffic light, targeting only the extracted target traffic lights. For example, the detection area setting unit 140 extracts, as the target traffic light, a traffic light that corresponds to the road on which the vehicle is traveling, based on map information that indicates the position of the vehicle and the structure of the road on which the vehicle is traveling. This makes it possible to output the current indication status of the target traffic light based on the result sequence related to the target traffic light or the integrated sequence, through processing by the determination unit 150, determination result storage unit 160, integration unit 165, output determination unit 170, and output unit 180, which will be described later.

The detection area setting unit 140 also determines whether or not there is an "interlocking traffic light" among the target traffic lights. Here, "interlocking traffic lights" refers to multiple traffic lights that correspond to the same stop line. The detection area setting unit 140 may directly obtain the position of the stop line corresponding to the traffic lights from the map information, and determine whether or not multiple traffic lights are "multiple traffic lights" based on whether or not the positions of the stop lines are the same. The detection area setting unit 140 may also determine whether or not multiple traffic lights are "multiple traffic lights" based on position information of the traffic lights. The detection area setting unit 140 may also determine whether or not multiple traffic lights are "multiple traffic lights" based on position information of the multiple traffic lights detected by the in-vehicle device.

The determination unit 150 performs image processing on the detection area set for each traffic light, detects the traffic light within the detection area, and determines the current status of the traffic light. The determination unit 150 detects traffic lights by, for example, template matching. Template matching uses an image of a standard traffic light as a template, scans the detection area while shifting it by one pixel at a time, and calculates the correlation of the brightness distribution, for example. Then, when the correlation reaches the highest value, it detects that a traffic light is present at the position on the image where the template is located.

The "color signals" shown by traffic lights include "green light," "yellow light," and "red light." The meaning of a "color signal" is determined by traffic laws that vehicles must follow; for example, a "green light" means "you may proceed," a "red light" means "stop at the stop position," and a "yellow light" means "stop at the stop position, unless you are too close to the stop position to stop safely." In this way, a "green light" means "permission to start" for a vehicle, while a "red light" and a "yellow light" have meanings other than "permission to start" for a vehicle.

Such discrimination between "green light," "yellow light," and "red light" may be performed by assuming that the "color signal" with the highest brightness level among the three "color signals" is illuminated.

In addition, traffic lights are not limited to "color signals," but may also display "arrow signals" that indicate the direction vehicles are permitted to go at the intersection where the traffic lights are installed. Examples include "right turn signals," "straight ahead signals," and "left turn signals."

"Arrow signals" are not limited to "right turn signals," "straight ahead signals," and "left turn signals," and there are many variations depending on the structure of the intersection where the traffic lights are installed. The meaning of an "arrow signal" is determined by the traffic laws that vehicles must follow.

The determination unit 150 performs image processing on the detection area to determine the current status of the traffic light, such as the illumination status of the "color signal" or "arrow signal."

In addition, the image processing for traffic light detection in the determination unit 150 may utilize machine learning such as a support vector machine or a neural network. When detecting traffic lights, a learning database in which templates of traffic lights of different sizes are stored in advance can be prepared, and the recognition rate can be improved by referring to different learning databases depending on the distance to the traffic light.

The determination result storage unit 160 stores the current state of one or more traffic lights determined by the determination unit 150 for each traffic light. Specifically, the determination unit 150 sequentially performs a determination for each traffic light on multiple images in chronological order acquired by the imaging unit 71, and the determination result storage unit 160 acquires the obtained determination results from the determination unit 150. The determination result storage unit 160 then stores the multiple determination results in chronological order for each traffic light as a result sequence associated with the traffic light.

The structure of the result string stored by the judgment result storage unit 160 will be explained using Figure 4. Figure 4 is a diagram showing an example of a result string for one traffic light, consisting of multiple judgment results in chronological order. For the sake of simplicity, the judgment results obtained by the judgment unit 150 are assumed to be of four types: "green", "yellow", "red", and "unknown" (a judgment result in which the current display status is unknown). In Figure 4, the judgment results of "green", "yellow", "red", and "unknown" are represented by a square filled with white, a square filled with a diagonal stripe pattern, a square filled with a checkerboard pattern, and a square marked with a "?" mark, respectively. The same notation is used in the subsequent figures.

The reason why the determination unit 150 may determine the status as "unknown" is, for example, when the traffic light flickers due to a flicker phenomenon (when the traffic light is an LED lamp that flashes periodically, the brightness in the captured image changes depending on the timing of the image capture by the image capture unit 71) or a rolling shutter phenomenon (when the image capture method used by the image capture unit 71 causes the image capture timing to differ for each position in the captured image).

The number N of judgment results that constitute one result sequence associated with one traffic light and are stored by the judgment result storage unit 160 is preset.

The N judgment results that make up one result string associated with one traffic light are each assigned a subscript ranging from 0 to N-1. Here, the judgment result with subscript 0 is the most recent judgment result obtained by the judgment unit 150, and as the subscript assigned to the judgment result increases, it indicates that the judgment result is older.

When a new judgment result is input to the judgment result storage unit 160, the judgment result storage unit 160 increments the subscript assigned to the judgment result by 1 and assigns the subscript 0 to the new judgment result input to the judgment result storage unit 160. The judgment result with the subscript N is erased.

In this way, the judgment results with subscripts from 0 to N-1 are stored in the judgment result storage unit 160 as a result string.

In addition, the judgment result storage unit 160 may have a function of supplementing the judgment results. Specifically, when "unknown" is included in the judgment results constituting the result sequence stored in the judgment result storage unit 160, the judgment result of "unknown" may be replaced based on the judgment result immediately before the "unknown" and the judgment result immediately after the "unknown".

For example, in the result string shown in FIG. 4, the judgment result for subscript 2 is "unknown." In this case, the judgment result storage unit 160 refers to the judgment result for subscript 3, which is immediately before the judgment result for subscript 2, and the judgment result for subscript 1, which is immediately after the judgment result for subscript 2. The judgment results for subscript 1 and subscript 3 are both "yellow light." Therefore, the judgment result for subscript 2 may be "yellow light," which is the judgment result for subscript 3, which is immediately before the judgment result for subscript 2, rather than "unknown."

In this way, the judgment result storage unit 160 may extract, from among the judgment results constituting the result string, a judgment result that indicates that the current display state is unknown as the first judgment result. The judgment result storage unit 160 may then extract, from among the judgment results constituting the result string, excluding judgment results that indicate that the current display state is unknown, the judgment result immediately before the first judgment result and the judgment result immediately after the first judgment result as the second judgment result and the third judgment result, respectively, and may replace the first judgment result in the result string with the second judgment result when the second judgment result and the third judgment result are identical.

In addition, the judgment result storage unit 160 may replace the first judgment result in the result string with the judgment result immediately following the first judgment result if there is no judgment result immediately preceding the first judgment result among the judgment results constituting the result string, excluding the first judgment result. This makes it possible to complement the judgment results even if there is a judgment result at the beginning of the result string that indicates that the current display state is unknown.

In addition, the judgment result storage unit 160 may replace the first judgment result in the result string with the judgment result immediately preceding the first judgment result if there is no judgment result immediately following the first judgment result among the judgment results constituting the result string, excluding the first judgment result. This makes it possible to complement the judgment results even if there is a judgment result at the end of the result string indicating that the current display state is unknown.

In situations where the determination unit 150 cannot determine the current state of a traffic light due to flickering of the traffic light, the function of complementing the determination result by the determination result storage unit 160 described above is effective in improving the accuracy of traffic light recognition.

In addition, after the integrated sequence is generated by the integration unit 165 described below, the determination result storage unit 160 may determine whether or not an interlocking traffic signal has changed from a state in which there is an interlocking traffic signal to a state in which there is no interlocking traffic signal. If it is determined that an interlocking traffic signal has changed from a state in which there is an interlocking traffic signal to a state in which there is no interlocking traffic signal, the determination result storage unit 160 may replace the determination results constituting the result sequence related to the traffic signal with the determination results constituting the integrated sequence.

When one new judgment result is input to the judgment result storage unit 160, the judgment result storage unit 160 may increase the subscript assigned to the judgment result by 1 for the result string after it has been replaced with the judgment results that make up the integrated string, and assign the subscript 0 to the new judgment result input to the judgment result storage unit 160. The judgment result with the subscript N is erased.

This prevents the current signal status judgment result from changing too frequently in a very short step when the number of traffic lights on the image goes from multiple to one, and the majority decision based on the integrated column goes back to majority decision based on a single result column. As a result, the current signal status judgment result becomes more stable, making it easier to use the current signal status judgment result for vehicle control.

In the above explanation, the judgment results stored in the judgment result storage unit 160 are described as N, but this is not limited to this.

For example, the judgment result storage unit 160 may determine whether or not to store a judgment result based on the capture time of the image on which the judgment result is based, instead of the number of judgment results. More specifically, the judgment result storage unit 160 may erase judgment results corresponding to captured images captured a predetermined time or more after capture, so that the judgment results stored in the judgment result storage unit 160 are only those corresponding to images captured up to a predetermined time in the past from the present.

When the imaging unit 71 captures images at a predetermined cycle, the result is the same whether the number of judgment results stored in the judgment result storage unit 160 is limited based on the number N or limited based on a predetermined time.

On the other hand, when the imaging unit 71 captures images at unspecified timing that is not a specified cycle, the results may differ depending on whether the number is limited based on the number N or based on a specified time. However, the configuration of the judgment result storage unit 160 can be adopted regardless of whether the number of judgment results to be stored is limited based on the number N or based on a specified time.

The output determination unit 170 determines the output value based on the judgment results included in the result column for each traffic light or the judgment results included in the integrated column described below.

First, when there is no linked traffic light, for example when there is only one target traffic light, the output determination unit 170 sets the current indication state associated with the most frequent judgment result among the judgment results constituting the result string related to the traffic light as the output value. In other words, the output value is set to the current indication state associated with the judgment result determined by majority vote among the judgment results constituting the result string.

In addition, if the most common judgment result is not determined to be one, the current status related to the judgment result to be used preferentially when setting the output value may be set in advance. For example, to give priority to ensuring that the vehicle stops, the output value may be set preferentially in the following order: "red light," "yellow light," "green light," and "unknown."

The operation of the output determination unit 170 when the number of traffic lights on the image is one will be described with reference to FIG. 5. FIG. 5 is a diagram showing an example of the output of the current display state based on the result sequence related to one traffic light. FIG. 5 shows how the result sequence related to one traffic light transitions from step "t" to step "t+13".

Between step "t" and step "t+4", the majority of the judgment results in the result sequence are related to "red light", so the output determination unit 170 sets the output value to "red".

In step "t+5", there are the same number of judgment results related to "red light" and "green light", and it is not possible to determine which is the most common judgment result. Therefore, the output determination unit 170 prioritizes "red light" over "green light", and sets the output value to "red".

Between step "t+6" and step "t+11", the majority of the judgment results in the result sequence are related to "green light", so the output determination unit 170 sets the output value to "green".

Between step "t+12" and step "t+13", the majority of the judgment results in the result sequence are related to "yellow light", so the output determination unit 170 sets the output value to "yellow".

Furthermore, when there are interlocking traffic signals, the output determination unit 170 sets the current indication state associated with the most frequent judgment result among the judgment results that make up the integrated sequence, which will be described later, as the output value. In other words, the output value is set to the current indication state associated with the judgment result determined by majority vote among the judgment results that make up the integrated sequence. The method of determining the output value based on the integrated sequence, performed by the output determination unit 170, is the same as the method of determining the output value based on the result sequence described above.

In the above description of the output determination unit 170, it has been described that when there are no interlocking traffic lights, the output value is determined based on the result sequence, and when there are interlocking traffic lights, the output value is determined based on the integrated sequence. Alternatively, when there are no interlocking traffic lights, the integration unit 165 may overwrite the integrated sequence with the determination result of the result sequence. In this way, the output determination unit 170 may always determine the output value based on the integrated sequence regardless of the number of traffic lights.

When there are interlocking traffic signals, the integration unit 165 integrates the result sequences for each of the interlocking traffic signals across the interlocking traffic signals to generate an integrated sequence. For example, among the judgment results that constitute the result sequence related to one interlocking traffic signal, the integration unit 165 integrates a subsequence consisting of a predetermined number or less of consecutive judgment results, including the most recent judgment result, across the interlocking traffic signals to generate an integrated sequence.

In addition, the integration unit 165 may set the predetermined number as the "subsequence size calculation process" to a value equal to or less than the value obtained by dividing the number of judgment results constituting a result sequence related to one linked traffic light by the number of linked traffic lights.

The operation of the merging unit 165 that generates the merged string will be described with reference to Figures 6A and 6B. Figure 6A is a diagram showing an example of merging two result strings. Figure 6B is a diagram showing an example of merging three result strings.

FIG. 6A shows the transition of the result sequence related to the first traffic light and the result sequence related to the second traffic light from step "t" to step "t+6". FIG. 6B further shows the transition of the result sequence related to the third traffic light from step "t" to step "t+6".

In FIG. 6A, the integration unit 165 shows an example in which the predetermined number is set to 5 (a value obtained by dividing the number of judgment results constituting the result string by 2, which is the number of traffic lights). That is, in FIG. 6A, the substring in each step is composed of five judgment results with subscripts 0 to 4.

As shown in "Transition of integrated sequence" in FIG. 6A, the integration unit 165 generates an integrated sequence by simply concatenating a subsequence that is a part of the result sequence related to the first traffic light with a subsequence that is a part of the result sequence related to the second traffic light. For example, the judgment results of the subscripts 0 to 4 of the subsequence of the first traffic light are respectively used as the judgment results of the subscripts 0 to 4 of the integrated sequence. Also, the judgment results of the subscripts 0 to 4 of the subsequence of the second traffic light are respectively used as the judgment results of the subscripts 5 to 9 of the integrated sequence.

When the output decision unit 170 makes a decision based on the result sequence related to the first traffic light, the majority decision result from step "t" to step "t+4" will be "green," and the majority decision result from step "t+5" onwards will be "yellow." The same applies to the majority decision result based on the result sequence related to the second traffic light.

On the other hand, if the output determination unit 170 makes a judgment based on the integrated sequence, the majority result from step "t" to step "t+2" will be "green," and the majority result from step "t+3" onwards will be "yellow." Therefore, compared to when the judgment is made based on the result sequence, when the judgment is made based on the integrated sequence, the delay from when the current signal state of the traffic light changes to when it is actually determined that the current signal state has changed is reduced by two steps.

The effect of reducing the delay between when the current signal status of a traffic light changes and when it is actually determined that the current signal status has changed becomes greater the more traffic lights that contribute to the generation of the integrated sequence.

In FIG. 6B, the integration unit 165 shows an example in which the predetermined number is set to 3 (the largest integer that does not exceed the value obtained by dividing the number of judgment results that make up the result sequence by 3, which is the number of traffic lights). That is, in FIG. 6B, the subsequence in each step is made up of three judgment results with subscripts 0 to 2.

As shown in "Transition of integrated sequence" in FIG. 6B, the integration unit 165 generates an integrated sequence by directly concatenating a subsequence that is a part of the result sequence related to the first traffic light, a subsequence that is a part of the result sequence related to the second traffic light, and a subsequence that is a part of the result sequence related to the third traffic light. For example, the judgment results of subscripts 0 to 2 of the subsequence of the first traffic light are respectively used as the judgment results of subscripts 0 to 2 of the integrated sequence. Furthermore, the judgment results of subscripts 0 to 2 of the subsequence of the second traffic light are respectively used as the judgment results of subscripts 3 to 5 of the integrated sequence. Furthermore, the judgment results of subscripts 0 to 2 of the subsequence of the third traffic light are respectively used as the judgment results of subscripts 6 to 8 of the integrated sequence.

When the output determination unit 170 makes a judgment based on the integrated sequence, the majority result from step "t" to step "t+1" will be "green," and the majority result from step "t+2" onwards will be "yellow." Therefore, compared to when the judgment is made based on the result sequence, when the judgment is made based on the integrated sequence, the delay from when the current signal state of the traffic light changes to when it is actually determined that the current signal state has changed is reduced by three steps.

In the above-mentioned Figures 6A and 6B, an example was described in which a subsequence that is part of a result sequence is integrated to generate an integrated sequence. However, it is not necessary to set a subsequence and integrate it. For example, the integration unit 165 may integrate the result sequences as they are to generate an integrated sequence. When integrating two result sequences made up of determination results with subscripts 0 to 9, the integrated sequence will be made up of determination results with subscripts 0 to 19.

In addition, the integration unit 165 may generate an integrated sequence by arranging the determination results that constitute the integrated sequence in chronological order based on the chronological order of the determination results that constitute the result sequence related to the linked traffic signals.

The operation of the integrating unit 165 that generates an integrated sequence taking into account the chronological order will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of integrating two result sequences taking into account the chronological order.

The result sequence and subsequence for the first traffic light in Figure 7, and the result sequence and subsequence for the second traffic light in Figure 7 are the same as those shown in Figures 6A and 6B.

As shown in "Transition of integrated sequence" in Figure 7, the integration unit 165 generates an integrated sequence by combining a subsequence that is a part of the result sequence related to the first traffic light and a subsequence that is a part of the result sequence related to the second traffic light, taking into account the chronological order. For example, the determination results of the subscripts 0, 1, 2, 3, and 4 of the subsequence of the first traffic light are respectively set as the determination results of the subscripts 0, 2, 4, 6, and 8 of the integrated sequence. Also, the determination results of the subscripts 0, 1, 2, 3, and 4 of the subsequence of the second traffic light are respectively set as the determination results of the subscripts 1, 3, 5, 7, and 9 of the integrated sequence.

By generating the integrated sequence in this way while taking the chronological order into consideration, the output determination unit 170 can determine the output value based on the chronological order information in addition to the majority vote result. Furthermore, when replacing the determination results constituting the result sequence related to traffic lights with the determination results constituting the integrated sequence, it is possible to eliminate a situation in which the chronological order is lost in the determination results in the result sequence after replacement.

This prevents the current signal status judgment result from changing too frequently in a very short step when the number of traffic lights on the image goes from multiple to one, and the process returns from majority judgment based on the integrated sequence to majority judgment based on a single result sequence. As a result, the current signal status judgment result becomes more stable, making it easier to use the current signal status judgment result for vehicle control.

For example, Figure 8 is a diagram showing an example of the current status output when the number of linked traffic lights is reduced. Figure 8 shows how the result sequence related to the first traffic light transitions from step "t" to step "t+6". Also, it shows how the result sequence related to the second traffic light transitions from step "t" to step "t+3". At step "t+4", the second traffic light disappears from the image, and only the first traffic light is visible on the image.

In FIG. 8, the integration unit 165 shows an example in which the predetermined number is set to 5 (a value obtained by dividing the number of judgment results constituting the result string by 2, which is the number of traffic lights). That is, in FIG. 8, the substring in each step is composed of five judgment results with subscripts 0 to 4.

In step "t+4", the output decision unit 170 returns from majority decision based on the integrated sequence to majority decision based on the result sequence of the first traffic light.

Here, a problem arises if the judgment results constituting the result sequence for the first traffic light are not replaced with the judgment results constituting the integrated sequence. That is, in step "t+3", the majority judgment based on the integrated sequence is "yellow", whereas in step "t+4", the majority judgment based on the result sequence for the first traffic light is "green". In other words, in a very short number of steps, the judgment result of the current state of the traffic light changes frequently, such as "green" -> "yellow" -> "green".

On the other hand, if the judgment results constituting the result sequence for the first traffic light are replaced with the judgment results constituting the integrated sequence, the above problem does not occur. In the "Progression of the result sequence for the first traffic light after replacement" in Figure 8, the judgment results of the result sequence in step "t+3" are replaced with the judgment results of the integrated sequence in step "t+3". For example, in step "t+3", the judgment results of the subscripts 0 to 9 of the integrated sequence are respectively set as the judgment results of the subscripts 0 to 9 of the result sequence for the first traffic light.

As a result of replacing the judgment results that make up the integrated sequence in this way, in step "t+4", the majority decision based on the result sequence of the first traffic light can be "yellow". This makes it possible to prevent the judgment results of the current state of the traffic light from changing frequently in a very short step.

The effect of preventing frequent changes in the judgment results of the current signal state of a traffic light is greater when the integrated sequence is generated taking into account the chronological order, as shown in Figure 7.

In addition, the integrating unit 165 may determine whether or not the number of interlocking traffic lights has increased since the previous generation of the integrated sequence. When it is determined that the number of interlocking traffic lights has increased, the integrating unit 165 may generate an integrated sequence by integrating the result sequences for each interlocking traffic light other than the newly added interlocking traffic light until the number of determination results constituting the result sequence related to the newly added interlocking traffic light reaches or exceeds a predetermined threshold.

The integration unit 165 may set the predetermined threshold to a value equal to or smaller than the value obtained by dividing the number of judgment results constituting the result sequence related to one traffic light by the number of traffic lights.

The operation of the integration unit 165 when the number of traffic lights is increased will be described with reference to FIG. 9. FIG. 9 is a diagram showing an example of the output of the current status when the number of interlocking traffic lights is increased.

FIG. 9 shows how the result sequence related to the first traffic light and the result sequence related to the second traffic light transition from step "t" to step "t+6". However, assume that the second traffic light, which was not captured in the image in steps prior to step "t", is captured in the image from step "t" onwards. As the second traffic light was captured in the image in step "t", the judgment results that make up the result sequence related to the second traffic light, with the subscripts 1 to 9 excluding the judgment result with the subscript 0, are "unknown".

In FIG. 9, the integration unit 165 shows an example in which the predetermined threshold is set to 5 (a value obtained by dividing the number of judgment results that make up the result sequence by 2, which is the number of traffic lights).

In such a situation, the integrating unit 165 does not use the result sequence related to the second traffic light in generating the integrated sequence until the number of judgment results that make up the result sequence related to the second traffic light becomes equal to or greater than a predetermined threshold. In the example shown in FIG. 9, at the time of step "t+4", the number of judgment results that make up the result sequence related to the second traffic light becomes 5, which is equal to or greater than the predetermined threshold. Therefore, the integrating unit 165 does not use the result sequence related to the second traffic light in generating the integrated sequence from step "t" to step "t+3". On the other hand, the integrating unit 165 uses the result sequence related to the second traffic light in generating the integrated sequence from step "t+4" onwards.

In this way, the integration unit 165 may not use the result sequence for the newly added traffic light to generate the integrated sequence until a sufficient number of judgment results are accumulated in the result sequence for the newly added traffic light. This prevents a decrease in the accuracy of the majority decision based on the integrated sequence.

The output unit 180 outputs the output value determined by the output determination unit 170 as the "current indication state based on the traffic light." Here, the "current indication state based on the traffic light" can be said to be the result obtained by integrating the current indication states of one or more traffic lights.

For example, the "current status based on the traffic light" is output from the output unit 180 to the vehicle control device 400 and used to control the vehicle. Alternatively, the "current status based on the traffic light" may be output from the output unit 180 to a reporting device (not shown) and notified to the occupant via the reporting device.

[Processing procedure of traffic light recognition device]
Next, a process for traffic light recognition by the traffic light recognition device according to the present embodiment will be described with reference to the flowcharts of Figures 2 and 3. Figure 2 is a flowchart showing the process by the traffic light recognition device according to the present embodiment. Figure 3 is a flowchart showing an output determination process.

The traffic light recognition process shown in Figures 2 and 3 may be executed each time an image is captured by the imaging unit 71, or may be executed at each cycle in which image processing is performed on the detection area after an image is captured by the imaging unit 71.

In FIG. 2, in step S101, the controller 100 acquires an image of one or more traffic lights in the direction in which the vehicle is traveling.

In step S103, the controller 100 acquires the vehicle's position via the on-board sensor 73, etc.

In step S105, the controller 100 acquires the positions of one or more traffic lights based on the map information acquired from the map information acquisition unit 75 or the like.

In step S107, the detection area setting unit 140 extracts the traffic light corresponding to the road on which the vehicle is traveling as the target traffic light. Note that if all traffic lights reflected in the acquired image are to be treated as target traffic lights, the processing of steps S103, S105, and S107 may be omitted.

In step S109, the determination result storage unit 160 determines whether the state has changed from one in which there is an interlocking traffic light to one in which there is not.

If the state changes from one in which there is an interlocking traffic light to one in which there is no interlocking traffic light (YES in step S109), in step S111, the determination result storage unit 160 replaces the result sequence related to the traffic light with the integrated sequence. In other words, the determination result storage unit 160 replaces the determination results constituting the result sequence related to the traffic light with the determination results constituting the integrated sequence.

On the other hand, if the state of the interlocking traffic light has not changed from a state where there is an interlocking traffic light to a state where there is no interlocking traffic light (NO in step S109), step S111 is skipped and the process proceeds to step S113.

In step S113, the detection area setting unit 140 sets a detection area for each traffic light on the image.

In step S115, the determination unit 150 performs image processing on the detection area set for each traffic light to detect the traffic lights within the detection area, and determines the current status for each traffic light.

In step S117, the determination result storage unit 160 stores the result sequence for each traffic light.

In step S119, the output determination unit 170 performs the output determination process shown in FIG. 3.

In FIG. 3, in step S121, the output determination unit 170 determines whether or not there is a linked traffic light.

If there is an interlocking traffic light (YES in step S121), in step S123, the output determination unit 170 performs a "subsequence size calculation process."

In step S125, the integration unit 165 generates an integrated sequence based on the set subsequence.

In step S127, the output determination unit 170 determines the output value based on the integrated column.

On the other hand, if there is no linked traffic light (NO in step S121), in step S131, the output determination unit 170 determines the output value based on the result sequence of one traffic light.

After the processing of step S127 or step S131, in step S129, the output determination unit 170 outputs the output value to the output unit 180. After that, the output unit 180 outputs the output value determined by the output determination unit 170 as the "current indication state based on the traffic light."

[Modification of the traffic light recognition device]
A modified example of the traffic light recognition device will be described with reference to Figs.

In the above embodiment, the size of the subsequence (the number of judgment results that make up the subsequence) has been described as being common among multiple traffic lights. Also, the size of the subsequence has been described as being determined based on the number of judgment results that make up the result sequence of the traffic light, and is invariant.

However, the size of the subsequence may be varied based on the latest judgment result in the traffic light result sequence. For example, when there are interlocking traffic lights, the integration unit 165 may integrate a subsequence consisting of consecutive judgment results, including the latest judgment result, among the judgment results constituting the result sequence related to one interlocking traffic light across the interlocking traffic lights to generate an integrated sequence. In this case, the number of judgment results constituting a subsequence whose latest judgment result indicates something other than permission to start the vehicle may be smaller than the number of judgment results constituting a subsequence whose latest judgment result indicates permission to start the vehicle.

FIG. 10 is a flowchart showing the process of calculating the size of a subsequence. In step S201, the integration unit 165 selects one result sequence from among the result sequences of unselected traffic lights.

In step S203, the integration unit 165 determines whether the most recent judgment result (the judgment result with the subscript 0) among the judgment results that make up the selected result string is "green." In other words, the integration unit 165 determines whether it indicates permission to start the vehicle.

If the latest judgment result is "blue" (YES in step S203), in step S205, the integration unit 165 sets the first number as the size of the subsequence related to the selected result sequence.

On the other hand, if the latest judgment result is not "blue" (NO in step S203), in step S207, the integration unit 165 sets a second number as the size of the subsequence related to the selected result sequence. Here, the second number is smaller than the first number.

In step S209, the integration unit 165 determines whether there are any unselected result columns.

If there is an unselected result string (YES in step S209), return to step S201. If there is no unselected result string (NO in step S209), end the substring size calculation process.

In the above example, the size of the subsequence is set to the first number and the second number for the cases where the latest judgment result is "blue" and where the latest judgment result is not "blue", but this is not limiting. For example, the first number, second number, and third number may be set to correspond to the cases where the latest judgment result is "blue", "yellow", and "red", respectively. Here, the second number may be smaller than the first number, and the third number may be smaller than the second number.

FIG. 11 shows an example of integrating two result sequences by varying the size of the subsequences. FIG. 11 shows how the result sequence related to the first traffic light and the result sequence related to the second traffic light transition from step "t" to step "t+6."

Referring to the size of the subsequence related to the result sequence related to the first traffic light, at step "t", the most recent judgment result is "green", so the size of the subsequence is set to 5. On the other hand, between step "t+1" and step "t+6", the most recent judgment result is "yellow", so the size of the subsequence is set to 3.

Referring to the size of the subsequence related to the result sequence for the second traffic light, at step "t" and step "t+1", the most recent judgment result is "green", so the size of the subsequence is set to 5. On the other hand, from step "t+2" to step "t+6", the most recent judgment result is "yellow", so the size of the subsequence is set to 3.

As a result, the size of the integrated column (the number of judgment results that make up the integrated column) is 10 at step "t", 8 at step "t+1", and 6 from step "t+2" to step "t+6".

When the output determination unit 170 makes a decision based on the result sequence related to the first traffic light, the majority result from step "t" to step "t+4" will be "green", and the majority result from step "t+5" onwards will be "yellow". When the output determination unit 170 makes a decision based on the result sequence related to the second traffic light, the majority result from step "t" to step "t+5" will be "green", and the majority result from step "t+6" onwards will be "yellow".

In contrast, when the output determination unit 170 makes a judgment based on the integrated sequence generated as described above, the majority result from step "t" to step "t+2" will be "green," and the majority result from step "t+3" onwards will be "yellow." Therefore, compared to when the judgment is made based on the result sequence, when the judgment is made based on the integrated sequence, the delay from when the current signal state of the traffic light changes to when it is actually determined that the current signal state has changed is reduced by three steps.

In this way, the number of judgment results constituting the subsequence whose latest judgment result indicates something other than permission to start the vehicle is set to be smaller than the number of judgment results constituting the subsequence whose latest judgment result indicates permission to start the vehicle. As a result, the delay until a judgment other than permission to start the vehicle is made can be more effectively suppressed.

[Effects of the embodiment]
As described above in detail, the traffic light recognition method and traffic light recognition device according to this embodiment target a plurality of images captured in the direction of travel of the vehicle, determine the current status of each traffic light for each image, and obtain a result sequence consisting of a plurality of determination results in chronological order for each traffic light. If there are interlocking traffic lights among the plurality of traffic lights, the result sequences for each interlocking traffic light are integrated across the interlocking traffic lights to generate an integrated sequence, and the current status associated with the most frequent determination result among the determination results constituting the integrated sequence is output as the current status based on the traffic light.

This makes it possible to reduce the delay between when the current status of a traffic light changes and when it is actually determined that the current status has changed. In particular, it is possible to speed up the change in the judgment result compared to a majority vote result based on the judgment results that make up the result string. Also, it is possible to make a more accurate judgment of the current status based on the judgment results of the current status of multiple traffic lights.

The traffic light recognition method and traffic light recognition device according to this embodiment may also generate an integrated sequence by integrating a subsequence of a predetermined number or less of consecutive judgment results, including the most recent judgment result, among the judgment results that make up the result sequence related to one linked traffic light, across the linked traffic lights when there are linked traffic lights. This allows for more accurate determination of the current indication status based on the judgment results of the current indication status of multiple traffic lights.

Furthermore, the traffic light recognition method and traffic light recognition device according to this embodiment may, when there are interlocking traffic lights, set the predetermined number to a value equal to or less than the value obtained by dividing the number of determination results that make up the result sequence for one interlocking traffic light by the number of interlocking traffic lights. This makes the size of the integrated sequence (the number of determination results that make up the integrated sequence) equal to or less than the size of the result sequence (the number of determination results that make up the result sequence). As a result, it is possible to switch the determination result more quickly in the majority decision result based on the determination results that make up the integrated sequence than in the majority decision result based on the determination results that make up the result sequence.

The traffic light recognition method and traffic light recognition device according to this embodiment may also generate an integrated sequence by integrating, across all of the linked traffic lights, a subsequence of consecutive judgment results, including the most recent judgment result, among the judgment results constituting the result sequence related to one linked traffic light when there are linked traffic lights. The number of judgment results constituting the subsequence whose most recent judgment result indicates something other than permission to start the vehicle may be smaller than the number of judgment results constituting the subsequence whose most recent judgment result indicates permission to start the vehicle.

This makes it possible to more effectively reduce delays in making decisions other than those permitting the vehicle to start. The delay in making decisions other than those permitting the vehicle to start is smaller than the delay in making decisions permitting the vehicle to start. As a result, it is possible to more effectively determine the current status of traffic lights that are likely to affect vehicle driving control.

Furthermore, the traffic light recognition method and traffic light recognition device according to this embodiment may determine whether the number of interlocking traffic lights has increased since the previous generation of the integrated sequence. If it is determined that the number of interlocking traffic lights has increased, the integrated sequence may be generated by integrating the result sequences of the interlocking traffic lights other than the newly added interlocking traffic light until the number of determination results constituting the result sequence for the newly added interlocking traffic light reaches or exceeds a predetermined threshold. This prevents a decrease in the accuracy of the majority decision based on the integrated sequence due to a small number of determination results constituting the result sequence for the newly added traffic light.

The traffic light recognition method and traffic light recognition device according to this embodiment may also generate an integrated sequence by arranging the determination results that make up the integrated sequence in chronological order based on the chronological order of the determination results that make up the result sequence related to the linked traffic lights when there are linked traffic lights. This allows for more accurate determination of the current state based on chronological order information in addition to the majority vote result. Also, when replacing the determination results that make up the result sequence related to a traffic light with the determination results that make up the integrated sequence, it is possible to eliminate a situation in which the chronological order is lost in the determination results in the result sequence after replacement.

Furthermore, the traffic light recognition method and traffic light recognition device according to this embodiment may determine whether an interlocking traffic light has changed from a state in which it is present to a state in which it is not present after the generation of the integrated sequence. If it is determined that an interlocking traffic light has changed from a state in which it is present to a state in which it is not present, the determination result constituting the result sequence related to the traffic light may be replaced with the determination result constituting the integrated sequence. This prevents the determination result of the current status of the traffic light from changing frequently in a very short step, and as a result, it becomes easier to use the determination result of the current status for vehicle control.

The traffic light recognition method and traffic light recognition device according to this embodiment may also be configured to output, when there is one traffic light, the current indication state associated with the most frequent judgment result among the judgment results constituting the result string related to the traffic lights as the current indication state based on the traffic light, when there is one traffic light. This allows a judgment to be made using the judgment results constituting the result string when there is one traffic light.

Furthermore, in the traffic light recognition method and traffic light recognition device according to this embodiment, the interlocking traffic lights may be multiple traffic lights that correspond to the same stop line on the road on which the vehicle is traveling. This makes it possible to determine the current status of only the traffic lights that the vehicle should obey when traveling. As a result, erroneous recognition of the current status of traffic lights can be suppressed.

The traffic light recognition method and traffic light recognition device according to this embodiment may extract, from among the judgment results constituting the result string, a judgment result that indicates that the current display state is unknown as the first judgment result. From among the judgment results constituting the result string, excluding the first judgment result, the judgment result immediately before the first judgment result and the judgment result immediately after the first judgment result may be extracted as the second judgment result and the third judgment result, respectively. If the second judgment result and the third judgment result are identical, the first judgment result in the result string may be replaced with the second judgment result.

This makes it possible to reduce the time delay between when the actual signal state of the traffic light changes and when the output value changes, even in situations where the current signal state cannot be determined due to flickering of the traffic light, thereby reducing the free-running distance of the vehicle.

For example, as shown in FIG. 12, when determining the current state of a traffic light based on the determination results that constitute a result sequence, delays in determination can be suppressed. In FIG. 12, according to the result sequence before interpolation, the output determination unit 170 can set the output value to "yellow" for the first time at step "t+7". On the other hand, according to the result sequence after interpolation, the output determination unit 170 can set the output value to "yellow" for the first time at step "t+6". In this way, delays in determination can be suppressed.

Furthermore, the traffic light recognition method and traffic light recognition device according to this embodiment may replace the first judgment result in the result string with the judgment result immediately following the first judgment result if there is no judgment result immediately preceding the first judgment result among the judgment results constituting the result string, excluding the first judgment result. This makes it possible to complement the judgment results even if there is a judgment result at the beginning of the result string indicating that the current status is unknown.

The traffic light recognition method and traffic light recognition device according to this embodiment may also replace the first judgment result in the result string with the judgment result immediately preceding the first judgment result if there is no judgment result immediately following the first judgment result among the judgment results constituting the result string, excluding the first judgment result. This makes it possible to complement the judgment results even if there is a judgment result at the beginning of the result string that indicates that the current status is unknown.

Each of the functions described in the above embodiments may be implemented by one or more processing circuits. Processing circuits include programmed processors, electrical circuits, and even devices such as application specific integrated circuits (ASICs), and circuit components arranged to perform the described functions.

The contents of the present invention have been described above in accordance with the embodiments, but the present invention is not limited to these descriptions, and it will be obvious to those skilled in the art that various modifications and improvements are possible. The descriptions and drawings that form part of this disclosure should not be understood as limiting the present invention. Various alternative embodiments, examples, and operating techniques will be apparent to those skilled in the art from this disclosure.

The present invention naturally includes various embodiments not described here. Therefore, the technical scope of the present invention is determined only by the invention-specific matters related to the scope of the claims that are appropriate from the above explanation.

71 Imaging unit 73 Vehicle-mounted sensor 75 Map information acquisition unit 100 Controller 140 Detection area setting unit 150 Determination unit 160 Determination result storage unit 165 Integration unit 170 Output determination unit 180 Output unit 400 Vehicle control device

Claims (13)

A traffic light recognition method for controlling a controller connected to an imaging unit mounted on a vehicle, comprising:
The controller:
Using the imaging unit, one or more traffic lights in a traveling direction of the vehicle are imaged a plurality of times to obtain a plurality of images;
A result sequence is obtained for each of the traffic lights, the result sequence being made up of a plurality of judgment results in chronological order, the judgment results being obtained by judging the current state of each of the traffic lights for each of the images;
When there is an interlocking signal among the plurality of signals,
aggregating the result sequence for each of the interlocking traffic lights across the interlocking traffic lights to generate an aggregate sequence;
A traffic light recognition method, comprising the steps of: outputting a current indication state associated with the most frequent judgment result among the judgment results constituting the integrated sequence as a current indication state based on the traffic light. The controller, when the interlocking traffic light is present,
The traffic light recognition method according to claim 1, characterized in that, among the judgment results constituting the result sequence relating to one of the interlocking traffic lights, a subsequence consisting of a predetermined number or less of consecutive judgment results including the most recent judgment result is integrated across the interlocking traffic lights to generate the integrated sequence. The controller, when the interlocking traffic light is present,
The traffic light recognition method according to claim 2, characterized in that the predetermined number is set to a value equal to or less than a value obtained by dividing the number of judgment results constituting the result sequence relating to one of the interlocking traffic lights by the number of the interlocking traffic lights. The controller, when the interlocking traffic light is present,
generating the integrated sequence by integrating a subsequence of successive determination results including a latest determination result among the determination results constituting the result sequence related to one of the interlocking traffic lights across the interlocking traffic lights;
A traffic light recognition method as described in any one of claims 1 to 3, characterized in that the number of judgment results constituting the subsequence having the latest judgment result indicating something other than permission to start the vehicle is smaller than the number of judgment results constituting the subsequence having the latest judgment result indicating permission to start the vehicle. The controller:
determining whether the number of interlocking traffic lights has increased since the last generation of the integrated sequence;
The traffic light recognition method according to any one of claims 1 to 4, characterized in that when it is determined that the number of interlocking traffic lights has increased, the integrated sequence is generated by integrating the result sequences for each of the interlocking traffic lights other than the newly added interlocking traffic light until the number of determination results that constitute the result sequence related to the newly added interlocking traffic light becomes equal to or greater than a predetermined threshold. The controller, when the interlocking traffic light is present,
A traffic light recognition method as described in any one of claims 1 to 5, characterized in that the integrated sequence is generated by arranging the judgment results that constitute the integrated sequence in chronological order based on the chronological order of the judgment results that constitute the result sequence related to the linked traffic lights. The controller:
determining whether the interlocking traffic signal has changed from a present state to a non-present state after the generation of the integrated sequence;
A traffic light recognition method as described in any one of claims 1 to 6, characterized in that when it is determined that the linked traffic light has changed from a present state to a absent state, the judgment result constituting the result sequence relating to the traffic light is replaced with the judgment result constituting the integrated sequence. When the number of the traffic lights is 1, the controller
A traffic light recognition method as described in any one of claims 1 to 7, characterized in that the current indication status associated with the most frequent judgment result among the judgment results constituting the result sequence related to the traffic light is output as the current indication status based on the traffic light. The traffic light recognition method according to any one of claims 1 to 8, characterized in that the interlocking traffic lights are a plurality of traffic lights corresponding to the same stop line on the road on which the vehicle is traveling. The controller:
extracting a determination result indicating that the current state is unknown as a first determination result from among the determination results constituting the result sequence;
extracting, from among the judgment results constituting the result sequence, a judgment result immediately before the first judgment result and a judgment result immediately after the first judgment result as a second judgment result and a third judgment result, respectively;
A traffic light recognition method according to any one of claims 1 to 9, characterized in that when the second judgment result and the third judgment result are identical, the first judgment result in the result sequence is replaced with the second judgment result. The controller:
extracting a determination result indicating that the current state is unknown as a first determination result from among the determination results constituting the result sequence;
A traffic light recognition method according to any one of claims 1 to 9, characterized in that if there is no judgment result immediately preceding the first judgment result among the judgment results constituting the result sequence, excluding the first judgment result, the first judgment result in the result sequence is replaced with the judgment result immediately following the first judgment result. The controller:
extracting a determination result indicating that the current state is unknown as a first determination result from among the determination results constituting the result sequence;
A traffic light recognition method according to any one of claims 1 to 9, characterized in that if there is no judgment result immediately following the first judgment result among the judgment results constituting the result sequence, excluding the first judgment result, the first judgment result in the result sequence is replaced with the judgment result immediately preceding the first judgment result. A traffic light recognition device including an imaging unit and a controller mounted on a vehicle,
The controller:
Using the imaging unit, one or more traffic lights in a traveling direction of the vehicle are imaged a plurality of times to obtain a plurality of images;
A result sequence is obtained for each of the traffic lights, the result sequence being made up of a plurality of judgment results in chronological order, the judgment results being obtained by judging the current state of each of the traffic lights for each of the images;
When there is an interlocking signal among the plurality of signals,
aggregating the result sequence for each of the interlocking traffic lights across the interlocking traffic lights to generate an aggregate sequence;
A traffic light recognition device characterized by outputting the current indication status associated with the most frequent judgment result among the judgment results constituting the integrated sequence as the current indication status based on the traffic light.

Images