JP7787011B2 - Vehicle control system and vehicle data collection method - Google Patents

Description

The present invention relates to a vehicle control system and a vehicle data collection method.

In recent years, efforts have been made to collect driving data from vehicles and use it as training data for autonomous driving AI (Artificial Intelligence) in order to prevent accidents involving self-driving vehicles. By distributing trained AI over the air (OTA), it is possible to constantly update vehicle control systems to the latest version.

Patent Document 1 describes a method for collecting driving data as follows: "A recording device includes an acquisition unit, a setting unit, and a selection unit. The acquisition unit acquires biometric information of the vehicle driver. The setting unit sets the importance of the vehicle's driving information based on the biometric information acquired by the acquisition unit. The selection unit selects the driving information to be stored in the storage unit based on the importance set by the setting unit" (see abstract).

Japanese Patent Application Laid-Open No. 2018-195193

The information that should be obtained from vehicles as training data for autonomous driving AI is driving information when there is a discrepancy between humans and the AI in their perception and judgment of driving conditions. If there is no discrepancy between humans and the AI, the AI is already able to perceive and judge in the same way as humans, and new training is not necessarily required.

However, with the method in Patent Document 1, driving information is recorded when the driver senses danger, so it is not possible to determine whether there is a discrepancy in perception or judgment between humans and the AI. As a result, there is a possibility that the collected information may contain information that is unnecessary as training data. Furthermore, since the discrepancy in perception and judgment between humans and the AI is expected to decrease as the level of autonomous driving increases, there is a possibility that the proportion of information that does not require training may increase when driving information is obtained using the method in Patent Document 1, particularly in a highly autonomous driving environment.

The present invention solves the problems of the prior art described above and provides a vehicle control system and vehicle data collection method that enable the efficient collection of data that is useful for training the AI responsible for recognition in self-driving vehicles.

In order to solve the above-mentioned problems, the present invention provides a vehicle control system comprising: a vehicle control determination unit that generates control signals for controlling the vehicle using data acquired from sensors mounted on the vehicle; a near-miss detection unit that detects near-misses felt by occupants from biochemical changes in the vehicle while driving the vehicle or from the vehicle's control signals, and outputs a near-miss detection signal; a judgment mismatch detection unit that detects a mismatch between the control signal generated by the vehicle control determination unit and the near-miss detection signal at the time the near-miss was detected, based on data acquired from the sensors mounted on the vehicle, the control signal generated by the vehicle control determination unit, and the near-miss detection signal detected by the near-miss detection unit; and a data storage unit that stores data acquired from the sensors mounted on the vehicle and the control signal generated by the vehicle control determination unit corresponding to the time the mismatch was detected by the judgment mismatch detection unit.

In order to solve the above-mentioned problems, the present invention provides a vehicle data collection method for collecting vehicle data in a vehicle control system equipped with a vehicle control determination unit, a near-miss detection unit, a judgment mismatch detection unit, and a data storage unit. The vehicle control determination unit creates a control signal for controlling the vehicle using data acquired from a sensor attached to the vehicle, the near-miss detection unit detects a near-miss felt by the occupant from biochemical changes in the occupant while driving the vehicle or from the vehicle's control signal, the judgment mismatch detection unit detects a mismatch between the control signal and the near-miss detection signal at the time the near-miss was detected based on the data acquired from the sensor attached to the vehicle, the control signal created by the vehicle control determination unit, and the near-miss detection signal detected by the near-miss detection unit, and the data acquired from the sensor attached to the vehicle corresponding to the time the mismatch was detected by the mismatch detection unit and the control signal created by the control unit are stored in the data storage unit.

This invention makes it possible to efficiently collect data that is useful for training the AI responsible for recognition in self-driving vehicles. Furthermore, by utilizing the collected data to improve the AI's functions, the safety of the vehicle control system is improved. Issues, configurations, and effects other than those described above will become clear from the description of the embodiments below.

1 is a block diagram showing a schematic configuration of a vehicle control system according to a first embodiment of the present invention. FIG. 2 is a block diagram showing the configuration of a biological change detection unit according to the first embodiment of the present invention. FIG. 3 is a flowchart showing a processing flow in the vehicle control system according to the first embodiment of the present invention. FIG. 10 is a block diagram showing a schematic configuration of a vehicle control system according to a modified example of the first embodiment of the present invention. FIG. 10 is a front view of a display screen of a data output unit in the vehicle control system according to a modified example of the first embodiment of the present invention. FIG. 10 is a block diagram showing a schematic configuration when data is transmitted from a vehicle control system according to a second embodiment of the present invention to a cloud for processing. FIG. 10 is a block diagram showing a data flow when managing travel data transmitted from a vehicle control system according to a second embodiment of the present invention, creating learning data, and transmitting update data. FIG. 10 is a block diagram showing the internal configuration of a cloud that receives information from a vehicle control system according to a second embodiment of the present invention. FIG. 10 is a flowchart showing a processing flow in the cloud that receives information from the vehicle control system according to the second embodiment of the present invention. FIG. 10 is a flowchart showing a processing flow in a vehicle control system according to a second embodiment of the present invention. 10 is a table illustrating an example of learning data generated by a learning data generating unit according to a second embodiment of the present invention. FIG. 10 is a block diagram showing a configuration in which a vehicle control system according to a second embodiment of the present invention is implemented by an add-on device. FIG. 10 is a block diagram showing a schematic configuration of a vehicle control system according to a third embodiment of the present invention. FIG. 11 is a block diagram showing a configuration in which a biological change detection unit is realized by a bus in Example 3 of the present invention. FIG. 10 is a block diagram showing a schematic configuration of a vehicle control system according to a fourth embodiment of the present invention. FIG. 10 is a block diagram showing a schematic configuration of a vehicle control system according to a fifth embodiment of the present invention. FIG. 10 is a block diagram showing an example in which a vehicle control system according to a sixth embodiment of the present invention is applied to trucks traveling in a convoy.

This invention detects near misses (when a vehicle occupant senses danger) from changes in biometric information or vehicle control, and if the vehicle control system is unable to detect a vehicle control signal corresponding to the near miss at the time the near miss factor occurs, it determines that there is a mismatch in judgment between the occupant and the vehicle control system.When a mismatch in judgment occurs, the vehicle control signal and driving data are recorded, and learning data is created to improve the accuracy of the vehicle control system.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and some omissions and simplifications have been made as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.
In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

In this embodiment, we will explain a vehicle control system that saves driving data when a judgment mismatch of the present invention is detected. Figure 1 is a configuration diagram of this embodiment, Figure 2 is an example configuration of a biological change detection unit when this embodiment is implemented in a passenger car, and Figure 3 is a process flow diagram for saving data in this embodiment.

First, let us explain Figure 1. The vehicle control system 1 according to this embodiment includes a sensor group 2, a vehicle control determination unit 3, a buffer unit 5, a vehicle control detection unit 6, a biological change detection unit 7, a near-miss detection unit 8, a determination mismatch detection unit 9, a delay time setting unit 10, an information acquisition unit 11, and a data storage unit 12.

Sensor group 2 is a set of sensors used to collect information necessary for vehicle control, including GNSS (Global Navigation Satellite Systems) such as GPS (Global Positioning System), LiDAR (Light Detection and Ranging), radar, and cameras. The sensing results of each sensor (including RAW data from the camera) are input to vehicle control determination unit 3 and buffer unit 5.

The vehicle control determination unit 3 determines the vehicle's status based on input data from the sensor group 2, calculates the vehicle control amount, and transmits control data (vehicle control values) to the vehicle drive unit 4 and buffer unit 5. In addition to the amount of brake and accelerator pedal depression, the control data includes information used to determine vehicle control, such as the object recognition results and distance to the object, and the result of a determination as to whether the vehicle needs to take emergency avoidance measures (such as issuing a driver's attention or warning, or applying emergency braking). Here, the vehicle control determination unit 3 is configured with trained AI (artificial intelligence).

The vehicle driving unit 4 is composed of the brake, accelerator, steering, etc., and drives the vehicle based on the control data output from the vehicle control determination unit 3.

The buffer unit 5 temporarily stores information from the sensor group 2 and the vehicle control determination unit 3. When recording, the data is linked to time, and the information (input) from the sensor group 2 referenced by the vehicle control determination unit 3 and the control data (output) derived by the vehicle control determination unit 3 as a result are recorded together as driving data.

The vehicle control detection unit 6 detects when the vehicle is controlled significantly differently from normal and outputs this detected signal as a vehicle control detection signal. For example, by using time series analysis of operations such as braking, accelerating, and steering, it is possible to detect unusual behavior (such as sudden braking or abrupt steering).

The biometric change detection unit 7 acquires biometric information from vehicle occupants, and if the acquired biometric information changes significantly from normal biometric information, it detects this change and outputs it as a biometric change detection signal. Occupants include the vehicle driver, passengers in the front or back seats, and passengers in the case of buses, etc. For example, biometric information includes heart rate, blood pressure, facial expressions, etc., and abnormal behavior can be detected using time series analysis. In the case of facial expressions, detection is possible by learning human facial expressions in advance and installing a camera to acquire and analyze the facial expressions of occupants.

When the near-miss detection unit 8 detects a vehicle control detection signal from the vehicle control detection unit 6 and/or a biochange detection signal from the biochange detection unit 7, it determines that the occupant has detected danger, i.e., that a near-miss has occurred, and outputs a near-miss detection signal. Note that the near-miss detection unit 8 may also detect a wide range of near-misses based on either the vehicle control detection signal or the biochange detection signal, and output a near-miss detection signal.

When the judgment mismatch detection unit 9 detects a near-miss detection signal from the near-miss detection unit 8, it references the control data stored in the buffer unit 5 and references the control data (judgment results when a near-miss event occurred) from the time set in the delay time setting unit 10 back to the time when the near-miss detection signal was detected, as well as the time before and after that (for example, several seconds).

If the vehicle control judgment unit 3 fails to detect danger after referencing the control data stored in the buffer unit 5 a predetermined time prior to and including the time when the near miss was detected by the judgment mismatch detection unit 9 (if the vehicle control judgment unit 3 does not take any action such as issuing a warning signal to the driver or outputting control data such as emergency braking to the vehicle drive unit 4), it determines that there is a judgment mismatch between the human and the vehicle control system, and outputs to the information acquisition unit 11 a data acquisition trigger signal and information on the linking time (reference data time) of the data referenced in the buffer unit 5.

In response to a data acquisition trigger signal from the judgment mismatch detection unit 9, the information acquisition unit 11 stores the control data and input data corresponding to the reference data time from the data stored in the buffer unit 5 in the data storage unit 12. The data stored in this data storage unit 12 may be data corresponding only to the reference data time, or data corresponding to a predetermined time range from the reference data time.

The functions of each component of this embodiment may be implemented by being incorporated into the vehicle's central gateway or into the ECU (Electronic Control Unit). The functions of each component of this embodiment may also be implemented by a retrofit device.

Next, using Figure 2, we will explain how this embodiment is implemented in a passenger car. A passenger 130 rides in the vehicle 13. Biometric information from the passenger 130 is acquired from a biometric change detection unit 7, which is realized by a smartphone 7-11, a smartwatch 7-12, a camera 7-13, a seat with a heart rate measurement function 7-14, or the like. The acquired information is transmitted to the near-miss detection unit 8 by means of wireless communication, wired communication, or the like. The operation when a near-miss detection signal is detected by the near-miss detection unit 8 is as described in Figure 1.

The flow of data processing in the vehicle control system 1 according to this embodiment will be explained using Figure 3.

First, data detected by each sensor in the sensor group 2 while the vehicle 13 is being driven is acquired (S301), and this acquired data is input to the vehicle control determination unit 3 to determine the vehicle's condition and create vehicle control data (S302), which is also stored in the buffer unit 5.

In S302, the vehicle control determination unit 3 processes the input data from each sensor using trained AI (artificial intelligence) to determine the vehicle's condition and create vehicle control data. The control data created by the vehicle control determination unit 3 is sent to the vehicle drive unit 4 to control the vehicle (S303) and is also stored in the buffer unit 5 (S304).

On the other hand, if the vehicle control detection unit 6 detects that the vehicle 13 has been controlled significantly differently from normal while the vehicle 13 is being driven, the signal output from the vehicle control detection unit 6 is received by the near-miss detection unit 8 as a vehicle control detection signal (S305), and if the biometric information of the vehicle 130 occupant detected by the biometric change detection unit 7 changes significantly differently from normal biometric information, the signal output from the biometric change detection unit 7 is received by the near-miss detection unit 8 as a biometric change detection signal (S306). It is determined that a near-miss has occurred based on the vehicle control detection signal and the biometric change detection signal received by the near-miss detection unit 8, and a near-miss detection signal is output (S307). Furthermore, if either S305 or S306 is detected in S307, it may be determined that a near-miss has occurred.

Next, the judgment mismatch detection unit 9 receives the near-miss detection signal output from the near-miss detection unit 8 and extracts, from the signals stored in the buffer unit 5 in S304, sensor data from the sensor group 2 and control data from the vehicle control judgment unit 3 from a time preceding the time the near-miss detection signal was received by the predetermined delay time stored in the delay time setting unit 10, as well as before and after that time. If the vehicle control judgment unit 3 fails to detect a danger based on this extracted sensor data and control data (if the vehicle control judgment unit 3 does not take action such as issuing a driver's attention or outputting a warning signal, or outputting control data such as emergency braking to the vehicle drive unit 4), a judgment mismatch between the human and the vehicle control system is detected (S308), and the information acquisition unit 11 collects driving data at the time of this detected judgment mismatch (S309). Next, the collected driving data at the time of detection of the judgment mismatch is stored in the data storage unit 12 (S310).

In this embodiment, driving data is acquired only when there is a mismatch between the judgments made by humans and the vehicle control system, making it possible to efficiently collect data that is useful for training the AI responsible for recognition in autonomous vehicles.

[Modification]
4 shows the configuration of a vehicle control system 1-1 according to a modified example of the first embodiment. The same components as those in the vehicle control system 1 according to the first embodiment are assigned the same part numbers.

As shown in Figure 4, the vehicle control system 1-1 of this modified example differs from Example 1 in that a data output unit 14 is provided between the information acquisition unit 11 and the data storage unit 12, and information acquired by the information acquisition unit 11 is output to the data output unit 14, allowing the selection of data to be stored in the data storage unit 12.

That is, this modified example differs from Example 1 in that it adds a display unit 141 that displays information acquired by the information acquisition unit 11 on the screen, and a data output unit 14 that includes a save button 142 for selecting whether to save the displayed data and a delete button 143 for selecting whether to delete the displayed data, as shown in FIG. 5.

In addition to the effects described in Example 1, this modification makes it possible to delete data when a near-miss signal is detected due to factors other than a near-miss by displaying the information acquired by the information acquisition unit 11 on the data output unit 14 and selecting the data to be saved in the data saving unit 12. In other words, it is possible to reduce the amount of data saved in the data saving unit 12 and improve the accuracy of the saved information (the probability that the saved data is data from when a near-miss occurred).

In a second embodiment of the present invention, a method for collecting vehicle data and utilizing the collected data using the vehicle control system 1 described in the first embodiment will be described. Figure 6 is a block diagram showing the configuration of this embodiment, Figure 7 is a block diagram showing the data flow of this embodiment, Figure 8 is a block diagram showing the internal configuration of the cloud in this embodiment, Figure 9 is a flow diagram showing the processing flow in the cloud in this embodiment, and Figure 10 is a flow diagram showing the processing flow in the vehicle control system.

First, the overall configuration of a vehicle data collection system utilizing the vehicle control system 1 according to this embodiment will be described using Figure 6.

Vehicle 13 is equipped with vehicle control system 1 having the configuration and functions described in Example 1, and passenger 130 (see Figure 2) rides in it. While shown in simplified form in Figure 6, the configuration and operation of vehicle control system 1 are the same as those described in Example 1.

The transmission unit 15 transmits the driving data stored in the data storage unit 12 of the vehicle control system 1 at the time of detection of the judgment mismatch to the information collection unit 16 (e.g., a cloud server).

The information collection unit 16 collects and organizes the driving data stored in the data storage unit 12 at the time of the judgment mismatch detection, which is transmitted from the transmission unit 15, and the surrounding information 17 at the time of the judgment mismatch detection. The purpose of collecting the surrounding information 17 is because the cause of the judgment mismatch between humans (near-miss detection signal) and AI (control data) may lie in something other than the driving data. Even if the judgment mismatch detection unit 9 determines that there is a judgment mismatch between humans and AI, there is a possibility that it is not actually a judgment mismatch. Therefore, the information collection unit 16 performs processing to remove data that is not a judgment mismatch between humans and AI from the data received from the data storage unit 12. Details of this processing will be described later using Figure 9.

Data in which a mismatch in judgment between humans and AI has been confirmed through the above processing is sent to the learning data generation unit 18. The data storage unit 12 of the vehicle control system 1 contains time information and vehicle position information at the time of the mismatch in judgment between the occupant 130 and the vehicle control system 1, so it is possible to obtain and store surrounding information 17 at that time and position via the Internet, etc. The surrounding information 17 includes weather, temperature, traffic conditions, etc.

The learning data generation unit 18 is equipped with a defect database (defect DB) 19, which stores actual defect information collected and organized by the information collection unit 16. The defect database 19 also collects information from other vehicles 21, making it possible to integrate defect information. Using the information from the defect database 19, the learning data generation unit 18 creates learning data 20 for training the AI stored in the vehicle control determination unit 3. The created learning data 20 is provided to the system design department 22.

The system design department 22 utilizes the learning data 20 received from the learning data generation unit 18 to train the AI stored in the vehicle control determination unit 3 of the vehicle control system 1 and create update data. The created update data is distributed to each vehicle using OTA or the like and stored in the vehicle control determination unit 3.

According to this embodiment, the safety and reliability of the vehicle control system 1 can be improved by training the AI using data on mismatches in judgment between humans and AI collected by the vehicle control system 1, and updating the AI stored in the vehicle control judgment unit 3 by distributing update data.

In the configuration shown in Figure 6, the transmitter 15 is configured to be located outside the vehicle control system 1, but the transmitter 15 may also be incorporated into the vehicle control system 1 as a single unit that constitutes the vehicle control system 1.

Next, an overview of a service utilizing the present invention will be explained using Figure 7. Here, the players in the service include a personal vehicle purchaser 23, a vehicle provision service provider 24, a vehicle user 25 who receives the vehicle provision service from the vehicle provision service provider 24, a vehicle operation company 26, a data management department 27, and a system design department 28 (corresponding to the system design department 22 in Figure 6). In Figure 7, the information provided by each player is indicated by arrows, and the value each player receives from this service is described.

First, let us explain the flow of information. Personal vehicle purchasers 23 and vehicle operators 26 provide driving data to the data management department 27. Vehicle provision service providers 24 obtain driving data from vehicles used by vehicle users 25 and provide it to the data management department 27. The data management department 27 aggregates, analyzes, and processes the driving data provided by each player to create learning data 20. The created learning data 20 is provided to the system design department 28. The system design department 28 uses the provided learning data 20 to create update data for the vehicle control system 1 installed in each vehicle. The created update data is provided to personal vehicle purchasers 23, vehicle provision service providers 24, and vehicle operators 26 using means such as OTA.

Next, we will explain the value each player receives from the service. By receiving update data from the system design department 28, personal vehicle purchasers 23 can enjoy the value of improved safety for their vehicles. Furthermore, vehicle provision service providers 24 and vehicle operation companies 26 can enjoy the value of improved vehicle safety as well as improved vehicle uptime due to a reduction in vehicle malfunctions. Vehicle users 25 can receive improved vehicle provision services with reduced vehicle malfunctions as well as improved vehicle safety. The system design department 28 can improve its market competitiveness by utilizing the learning data 20 to improve its products. Furthermore, the provision of update data enables the provision of safe vehicles, which also improves customer trust.

Next, the configuration of the information collection unit 16 that collects information will be described using Figure 8. The information collection unit 16 includes a receiving unit 161 that receives information transmitted from the transmitting unit 15 described in Figure 6, a driving data acquisition unit 162, a mismatch check unit 163, a mismatch detection exclusion condition storage unit 164, a mismatch feature extraction unit 165, a surrounding environment information acquisition unit 166, a storage frequency designation unit 167, and a transmitting unit 168, all of which are connected by data lines 169.

The data organization flow on the cloud side having such a configuration will be described with reference to FIG.
First, the receiving unit 161 receives the driving data transmitted from the transmitting unit 15 and stored in the data storage unit 12 of the vehicle control system 1 when the judgment mismatch detection unit 9 detects a mismatch, and stores the data in the driving data acquisition unit 162 (S901).

Next, the mismatch check unit 163 checks whether there was actually a mismatch between the human and AI judgments for the driving data stored in the driving data acquisition unit 162 (S902).

This mismatch check unit 163 performs this automatically using methods such as extracting common features when detecting mismatches between humans and AI using mismatch detection exclusion conditions preset in the mismatch detection exclusion condition storage unit 164, or information collected in the defect database 19 of the learning data generation unit 18 using the mismatch feature extraction unit 165.

An example of a mismatch detection exclusion condition that is pre-set in the mismatch detection exclusion condition storage unit 164 is to determine that no near-misses have occurred if the camera's RAW data or recognition results do not show any objects that are likely to cause near-misses, such as vehicles, people, or animals.

Examples of features extracted by the mismatch feature extraction unit 165 include the time of the mismatch, the recognized object, and the weather.

If it is determined in S902 that a mismatch exists (YES in S902), information about the surrounding environment, such as weather, temperature, and traffic conditions, corresponding to the time when the mismatch between the human and AI was detected is collected (S903).

Next, before storing the information determined to contain a mismatch in S902 in the defect database 19, a final check is made by a human to determine whether or not there is a mismatch (S904). If it is determined that there is a mismatch (YES in S904), the data is stored in the defect database 19 (S905) and the process ends. This S904 step may be omitted if the accuracy of the mismatch check unit 163 in S902 is high, and the process may proceed directly from S903 to S905.

The learning data generation unit 18 creates learning data 20 based on data regarding mismatches stored in the defect database 19.

As an example of the learning data 20, an example of learning data 1100 shown in table format in Figure 11 is shown. In the learning data 1100, the following information is stored in association with one another: vehicle type 1101, information about the type of vehicle in which the near miss was detected; mismatch occurrence date and time 1102, information about the date and time when a mismatch in the near miss data occurred; vehicle control information 1103, such as the recognition result of the near miss detected by the near miss detection unit 8, vehicle control values output from the vehicle control determination unit 3, and vehicle situation judgment results obtained from an ECU (not shown); surrounding information 1104, such as weather, temperature, and traffic conditions obtained via the Internet; and information 1105 regarding the presence or absence of a mismatch determined by the mismatch check unit 163.

Here, the driving data received from the vehicle in S901 is a mixture of small amounts of data, such as location information and time information, and large amounts of data, such as camera RAW data, and depending on the frequency of data reception, this may put a strain on communication between the vehicle 13 and the information collection unit 16. For this reason, only small amounts of data are normally received, and if the possibility of a true mismatch increases in S902 or S904, the remaining large amounts of data may be requested from the vehicle to obtain more detailed information.

If it is determined in S902 that there is no mismatch (NO in S902), it is determined whether additional data is required (S906). If it is determined that additional data is required (YES in S906), the additional data is obtained from the data stored in the data storage unit 12 (S907), and the process proceeds to S902 again to determine whether there is a mismatch.

On the other hand, if it is determined that additional data is not necessary (NO in S906), the data acquired in S901 is erased (S910) and the process ends.

Furthermore, if it is determined in S904 that there is no mismatch (NO in S904), it is determined whether additional data is required (S908), and if it is determined that additional data is required (YES in S908), the additional data is obtained from the data stored in the data storage unit 12 (S909), and the process proceeds to S904 again to determine whether there is a mismatch.

On the other hand, if it is determined that additional data is not necessary (NO in S908), the data acquired in S901 is erased (S910) and the process ends.

Next, the data processing flow on the vehicle control system 1 side in this embodiment will be described using Figure 10. The processing from S1001 to S1009 is the same as the processing from S301 to S309 in the processing flow described in Example 1 using Figure 3, so the description will be omitted.

For the information collected in S1009 of FIG. 10, it is determined whether the frequency at which the information is stored in the data storage unit 12 has reached a preset frequency (S1010). If it is determined that the preset frequency has not yet been reached (NO in S1010), the information collected in S1009 is stored in the data storage unit 12 (S1011), and this stored data is sent to the information collection unit 16 (S1012).

Furthermore, if update data is available in the system design department 22 shown in Figure 6, the vehicle control determination unit 3 receives the update data (S1014), and uses this update data to make a vehicle control determination in S1002. By obtaining update data in this way and making a vehicle control determination, it is possible to maintain a high level of accuracy in near-miss determination, and maintain a high level of reliability in the determination.

When transmitting data to the information collection unit 16 configured in the cloud in S1012, basic vehicle information such as vehicle type, mileage, and years of use may also be transmitted in addition to the driving data stored in the buffer unit 5.

The storage frequency used for the determination in S1010 is the data stored in the storage frequency designation unit 167 of the information collection unit 16, but may be set as the number of times data is stored according to the mismatch occurrence situation. For example, if the data processing in the information collection unit 16 reveals that mismatch detections for the vehicle model are rare, the storage frequency is set low (such as saving once every 10 times). The storage frequency data stored in the storage frequency designation unit 167 may also be updatable via OTA, etc.

On the other hand, if it is determined in S1010 that the set frequency has been reached (YES), the information collected in S1009 is discarded (deleted) (S1013), and processing ends.

Next, using Figure 12, we will explain the hardware configuration when the present invention is implemented as a retrofit device on a vehicle 120. In relation to this embodiment, the hardware installed on the vehicle 120 to detect near-miss mismatches consists of a driving assistance system 1210, a telematics data collection device 1201, a near-miss detection device 1202, and a telematics control system 1203.

Data acquired in this configuration is transmitted to the cloud server 1220 via wireless or other means. Of these, the telematics data collection device 1201, near-miss detection device 1202, and telematics control system 1203 are retrofit devices 1200. The telematics data collection device 1201 is a device capable of collecting information about the vehicle, and can collect information directly from the CAN or from an OBD port, although either method may be used.

The near-miss detection device 1202 collects biometric information from a smartwatch, smartphone, etc., and if a mismatch is detected between the device and the telematics data collection device 1201, the data is transmitted from the telematics control system 1203 to the cloud server 1220 via wireless communication, etc. This configuration makes it possible to collect information from existing vehicles when a near-miss is detected.

In this embodiment, data regarding mismatches in judgment between humans and vehicle control detected in each vehicle is collected and learned in the cloud, and this learned data is sent to each vehicle and reflected in the vehicle control judgment unit, thereby improving the reliability of the vehicle control judgment unit and further reducing mismatches in judgment between humans and vehicle control.

As a third embodiment of the present invention, a case will be described in which a vehicle control system uses multiple pieces of biometric information to detect mismatches. Figure 13 is a configuration diagram of a vehicle control system 101 according to this embodiment. In the following explanation, the same components as those in the vehicle control system 1 described in embodiment 1 are assigned the same reference numerals, and differences will be mainly explained. Points that are not specifically explained are the same as in embodiment 1.

In this embodiment, instead of the biochange detection unit 7 of Example 1, the system is configured with multiple biochange detection units 7-1, ..., 7-n, and further includes a biochange aggregation unit 35 that receives information from the multiple biochange detection units 7-1, ..., 7-n. The biochange aggregation unit 35 is configured to output a biochange detection signal to the near-miss detection unit 8 when a biochange is detected by more than one or a majority of the multiple biochange detection units 7-1, ..., 7-n.

By adopting this configuration, according to this embodiment, a biochange detection signal is output to the near-miss detection unit 8 only when two or more of the multiple biochange detection units 7-1, ... 7-n detect a biochange, thereby increasing the probability that the mismatch detection timing is a near-miss compared to when a single piece of bioinformation is used.

Figure 14 shows an example of facial expression detection on a bus 131. As a specific example of this embodiment, a camera 7-21 that reads the facial expressions 7-24 of passengers 130 is installed on the bus 131 as a biometric change detection unit 7. The results of the facial expressions 7-24 of passengers 130 detected by the camera 7-21 are collected by a biometric change collection unit 35, and if it is estimated that a certain number of passengers 130 or more are feeling unsafe, a biometric change detection signal is output to an information collection unit 16 configured in the cloud. In addition to the camera 7-21, devices such as smartphones 7-22 and smartwatches 7-23 owned by passengers 130 may also be used as the biometric change detection unit 7. As in this embodiment, mismatch detection is possible even in vehicles without drivers, such as autonomous buses.

As a fourth embodiment of the present invention, a vehicle control system 102 will be described in which, instead of the delay time setting unit 10 of the vehicle control system 1 described in the first embodiment, a predetermined delay time width setting unit 36 is provided so that a width can be set in the predetermined delay time, as shown in FIG. 15 .

In the following explanation, the same components as those in the vehicle control system 1 described in Example 1 are given the same reference numerals, and differences will be mainly explained. Points that are not specifically explained are the same as in Example 1.

This embodiment is characterized by the provision of a predetermined delay time width setting unit 36 and an estimation unit 37 instead of the delay time setting unit 10 of the vehicle control system 1 described in Example 1. The predetermined delay time width setting unit 36 sets a certain width for the predetermined delay time used when the judgment mismatch detection unit 9 refers to the buffer unit 5 after the near-miss detection unit 8 outputs a near-miss detection signal.

After the near-miss detection unit 8 outputs a near-miss detection signal, the estimation unit 37 performs time-series analysis of the control data in the buffer unit 5 within a predetermined delay time, estimates the timing (data time) at which the control data output from the vehicle control judgment unit 3 changes significantly compared to other data, and outputs this as the reference data time to the judgment mismatch detection unit 9.

According to this embodiment, after a near-miss is detected, by searching for timing within a certain range, compared to going back a fixed delay time, it is possible to save driving data that is more reliably synchronized with the timing of the near-miss occurrence.

In Example 5, we will explain the operation of the vehicle control system 102 described in Example 4 when a delay adjustment unit that calibrates the delay time set by the predetermined delay time width setting unit 36 is provided.

Figure 16 is a configuration diagram of the vehicle control system 103 in this embodiment. In the following explanation, the same components as in the fourth embodiment are given the same reference numerals, and differences will be mainly explained. Points that are not specifically explained are the same as in the fourth embodiment.

The vehicle control system 103 according to this embodiment differs from the vehicle control system 103 described in Example 4 in that it includes a delay adjustment unit 38. In addition to the operations described in Example 4, the estimation unit 37 estimates the reaction speed of the biometric information provider from the timing of the near-miss detection signal output by the near-miss detection unit 8 and the estimated timing, and outputs the result to the delay adjustment unit 38. The delay adjustment unit 38 feeds back the adjusted delay time to the predetermined delay time width setting unit 36 based on the estimated reaction speed. The predetermined delay time width setting unit 36 changes the value of the delay time width based on the adjusted delay time.

In this embodiment, by estimating the biological reaction time and providing feedback to a predetermined delay time, it becomes possible to store driving data that is more reliably synchronized with the timing of near-miss incidents with each successive save.

In Example 6, we will explain the operation of trucks equipped with either vehicle control system 1 or 101, or 102 or 103 described in Examples 1 to 5, when they travel in a convoy.

Figure 17 is a diagram showing the configuration of trucks 39a, 39b, and 39c according to this embodiment when traveling in a convoy. In this embodiment, the leading truck 39a is equipped with a rear camera 40b, the middle truck 39b is equipped with a front camera 40a, and the trailing truck 39c is not equipped with a camera. Additionally, manned vehicles 41a, 41b, and 41c are located nearby, and a surveillance camera 42 is installed.

In this situation, trucks 39a, 39b, and 39c travel in a convoy. Convoy driving refers to the driving of multiple trucks controlled by vehicle-to-vehicle communication. Trucks 39a, 39b, and 39c may or may not have people on board, but in this embodiment, the lead truck a is described as being manned, while the middle truck 39b and the rear truck 39c are unmanned.

First, the leading truck 39a is manned, so the vehicle control system 1 can operate as described in Example 1. On the other hand, the middle truck 39b and the trailing truck 39c are unmanned, and so the biochemical change detection unit 7 in the vehicle control system 1 cannot acquire biochemical change information. Instead, biochemical information from the surrounding manned vehicles 41a, 41b, and 41c is used.

Specifically, one method is for a passenger in a manned vehicle 41a, 41b, or 41c traveling in the lane next to trucks 39a, 39b, or 39c to notice a dangerous situation in the unmanned center truck 39b or rear truck 39c, and receive information about a biological change as a near-miss detection signal via vehicle-to-vehicle communication.

When a near-miss detection signal is received from a nearby manned vehicle 41 via vehicle-to-vehicle communication, if the unmanned central truck 39b or rear truck 39c is unable to detect the danger, the data is saved.

As another method, if a passenger in the manned leading truck 39a notices a dangerous situation in the unmanned middle truck 39b or trailing truck 39c traveling behind them through a rearview mirror or the like, and a change in their biometric information is detected, this can be used as a near-miss detection signal.

Furthermore, a near-miss detection signal may be detected using the front camera 40a equipped on the central truck 39b. Specifically, a near-miss detection signal may be detected when there is a difference in the recognition results of camera images capturing the same position, such as when the recognition results of the rear camera 40b of the lead truck 39a detect a danger, but the image recognition results of the front camera 40a of the central truck 39b do not. In addition, images from a surveillance camera 42 capturing the area around the road on which the truck 39 is traveling may be used as a method of detecting the difference in the recognition results of similar camera images.

According to this embodiment, even when multiple vehicles in a platoon are driven unmanned, it is possible to detect near-miss detection signals corresponding to each vehicle, making it possible to store driving data synchronized with the timing of near-miss occurrences.

The invention made by the inventor has been specifically described above based on examples, but it goes without saying that the present invention is not limited to the above examples and can be modified in various ways without departing from the spirit of the invention. For example, the above examples have been described in detail to clearly explain the invention, and the invention is not necessarily limited to those that include all of the described configurations. Furthermore, it is possible to add, delete, or replace part of the configuration of each example with other configurations.

DESCRIPTION OF SYMBOLS 1, 1-1, 101, 102, 103... Vehicle control system 2... Sensor group 3... Vehicle control determination unit 4... Vehicle drive unit 5... Buffer unit 6... Vehicle control detection unit 7... Biological change detection unit 8... Near miss detection unit 9... Determination mismatch detection unit 10... Delay time setting unit 11... Information acquisition unit 12... Data storage unit 13, 120... Vehicle 14... Data output unit 15... Transmission unit 16... Information collection unit 17... Peripheral information 18... Learning data generation unit 19... Defect database 20... Learning data 22, 28... System design department 23... Personal vehicle purchaser 24... Vehicle provision service provider 25... Vehicle user 26... Vehicle operation company 27... Data management department 35... Biological change aggregation unit 36... Predetermined delay time width setting unit 37... Estimation unit 38... Delay adjustment unit 39a... Leading track 39b... Center track 39c: Rear truck 40a: Front camera 40b: Rear camera 41: Nearby manned vehicle 42: Surveillance camera 161: Receiving unit 162: Traveling data acquisition unit 163: Mismatch check unit 164: Mismatch detection exclusion condition storage unit 165: Mismatch feature extraction unit 166: Surrounding environment information acquisition unit 167: Storage frequency designation unit 168: Transmitter 1200: Retrofit device 1201: Telematics data collection device 1202: Near miss detection device 1203: Telematics control system 1210: Driving assistance system 1220: Cloud server.

Claims (11)

a vehicle control determination unit that generates a control signal for controlling the vehicle using data acquired from a sensor mounted on the vehicle;
a near-miss detection unit that detects a near-miss felt by a passenger in the vehicle from at least one of a biological change of the passenger in the vehicle while driving the vehicle and a control signal of the vehicle, and outputs a near-miss detection signal;
a judgment mismatch detection unit that detects a mismatch between the control signal and the near-miss at the timing when the near-miss is detected based on the data acquired from the sensor attached to the vehicle, the control signal created by the vehicle control judgment unit, and the near-miss detection signal detected by the near-miss detection unit; and
a data storage unit that stores the data acquired from the sensor mounted on the vehicle corresponding to the timing at which the mismatch is detected by the determination mismatch detection unit and the control signal created by the vehicle control determination unit ,
The vehicle control system is characterized in that the judgment mismatch detection unit detects the mismatch when the vehicle control judgment unit has not detected any danger at a time that is a predetermined time before the time when the occupant felt a near miss and before and after that time.
2. The vehicle control system of claim 1,
Further comprising a delay time setting unit,
The vehicle control system is characterized in that, when the near miss detection unit detects the near miss, the judgment mismatch detection unit detects the mismatch between the control signal and the timing at which the near miss was detected using the data acquired from the sensor attached to the vehicle at a time including before and after the time set back by the delay time setting unit and the control signal created by the vehicle control judgment unit. 3. The vehicle control system according to claim 1 or 2,
A vehicle control system further comprising: a buffer unit that stores the data acquired from the sensor mounted on the vehicle and the control signal created by the vehicle control judgment unit; and an information acquisition unit that acquires from the buffer unit the data and the control signal stored in the buffer unit, the data and the control signal corresponding to the timing at which the mismatch was detected by the judgment mismatch detection unit, and stores the data and the control signal acquired by the information acquisition unit corresponding to the timing at which the mismatch was detected in the data storage unit. 3. The vehicle control system according to claim 1 or 2,
The vehicle control system further comprises a transmitting unit that transmits the information about the mismatch stored in the data storage unit to an external device. 3. The vehicle control system according to claim 1 or 2,
The vehicle control system is characterized in that the vehicle control determination unit receives learning data created externally and creates the control signal using the received learning data. 3. The vehicle control system according to claim 1 or 2,
A vehicle control system characterized in that the near-miss detection unit, the judgment mismatch detection unit, and the data storage unit are configured as retrofit devices for the vehicle. A method for collecting vehicle data in a vehicle control system including a vehicle control determination unit, a near-miss detection unit, a mismatch detection unit, and a data storage unit,
a control signal for controlling the vehicle by the vehicle control determination unit using data acquired from a sensor mounted on the vehicle;
The near miss detection unit detects a near miss felt by the occupant from at least one of a biological change of the occupant riding in the vehicle while driving the vehicle or a control signal of the vehicle,
Detecting a mismatch between the control signal and the near miss at the timing when the near miss is detected by the mismatch detection unit from the data acquired from the sensor attached to the vehicle, the control signal created by the vehicle control determination unit, and the near miss detected by the near miss detection unit;
the data acquired from the sensor mounted on the vehicle corresponding to the timing at which the mismatch was detected by the mismatch detection unit and the control signal created by the vehicle control determination unit are stored in the data storage unit,
A vehicle data collection method characterized in that the mismatch is detected when the vehicle control judgment unit is unable to detect danger at a time a predetermined time prior to and around the time when the occupant felt a near miss . 8. The vehicle data collection method according to claim 7, further comprising:
the vehicle control system further includes a delay time setting unit;
A vehicle data collection method characterized in that, when the near miss detection unit detects the near miss, the mismatch detection unit detects the mismatch between the control signal and the timing at which the near miss was detected using the data acquired from the sensor attached to the vehicle at a time that is backdated by the delay time setting unit and the control signal created by the vehicle control determination unit. 9. The vehicle data collection method according to claim 7 or 8,
the vehicle control system further includes a buffer unit and an information acquisition unit;
a buffer unit that stores the data acquired from the sensor mounted on the vehicle and the control signal created by the vehicle control determination unit; an information acquisition unit that acquires from the buffer unit the data and the control signal that correspond to the mismatch detected by the mismatch detection unit among the data and the control signal stored in the buffer unit; and a data storage unit that stores the data and the control signal that correspond to the timing at which the mismatch was detected and acquired by the information acquisition unit. 9. The vehicle data collection method according to claim 7 or 8,
the vehicle control system further includes a transmitter;
The vehicle data collection method further comprises transmitting the information about the mismatch stored in the data storage unit from the transmission unit to an external device. 9. The vehicle data collection method according to claim 7 or 8,
The vehicle data collection method, wherein the vehicle control determination unit receives externally generated learning data, and generates the control signal using the received learning data.