JP2024162110A - Sensor state estimation device and sensor state estimation system - Google Patents

Abstract

To enable the estimation of the state of a peripheral monitoring sensor used in automatic driving while reducing the processing load and power consumption in a movable body that performs automatic driving.SOLUTION: A sensor state estimation device comprises: a recognition information storage unit 102 which stores recognition information being information on a peripheral environment of an own vehicle recognized from a sensing result of a peripheral monitoring sensor; a reference extraction unit 103 which extracts recognition information of a reference target being a reference for estimating a sensor state from the recognition information stored in the recognition information storage unit 102; and a sensor state estimation unit 106 which estimates the sensor state by comparing the recognition information of the reference target previously extracted from the reference extraction unit 103 with new recognition information of the reference target recognized from a new sensing result of the peripheral monitoring sensor and included in the peripheral environment of the own vehicle. The device uses a processor different from one used by an automatic driving ECU and is supplied with power from a power supply system different from one that supplies the power to the automatic driving ECU.SELECTED DRAWING: Figure 2

Description

This disclosure relates to a sensor state estimation device and a sensor state estimation system.

Patent Document 1 discloses a technology in which a degradation diagnosis unit provided in an autonomous driving control unit that controls autonomous driving evaluates the degradation of each sensor used in autonomous driving. The technology disclosed in Patent Document 1 evaluates the degradation of each sensor using the recognition results of object recognition processing executed based on data or signals from each sensor.

International Publication No. 2021/065559

Because autonomous driving control is complex, the processing load is large on the autonomous driving control unit during autonomous driving. Therefore, if the autonomous driving control unit performs processing related to deterioration evaluation during autonomous driving, there is a risk that the processing load on the autonomous driving control unit will become excessive. Examples of processing related to deterioration evaluation include processing to obtain information used for deterioration evaluation from a sensor and processing to evaluate deterioration. In response to this, it is also possible to perform processing related to deterioration evaluation while the vehicle is parked. However, because autonomous driving control is complex, the power required for processing by the autonomous driving control unit is also large. Therefore, even if processing related to deterioration evaluation is performed while the vehicle is parked, there is a risk that battery power consumption will increase.

One objective of this disclosure is to provide a sensor state estimation device and a sensor state estimation system that enable estimation of the state of a surrounding monitoring sensor used in autonomous driving with reduced processing load and power consumption in a vehicle performing autonomous driving.

The above object is achieved by a combination of features recited in the independent claims, and the subclaims define further advantageous embodiments of the disclosure. The reference characters in parentheses in the claims indicate a correspondence with the specific means described in the embodiments described below as one aspect, and do not limit the technical scope of the present disclosure.

In order to achieve the above object, the sensor state estimation device disclosed herein is a sensor state estimation device that estimates a sensor state, which is the state of a perimeter monitoring sensor that monitors the perimeter of a mobile body used for the automatic driving of a mobile body that performs automatic driving, and includes a recognition information storage unit (102) that stores recognition information, which is information on the surrounding environment of the mobile body recognized from the sensing result of the perimeter monitoring sensor, a reference extraction unit (103, 103a) that extracts recognition information of a reference target that is used as a reference for estimating the sensor state from the recognition information stored in the recognition information storage unit, and a sensor state estimation unit (106, 106a, 106b) that estimates the sensor state by comparing the recognition information of the reference target previously extracted from the reference extraction unit with new recognition information of the reference target included in the surrounding environment of the mobile body recognized from new sensing results of the perimeter monitoring sensor, and uses a calculation processing device different from that used by the automatic driving control unit (17) that controls the automatic driving, and is supplied with power from a power supply system different from that from which the automatic driving control unit receives power.

According to this, the sensor state estimation device including the recognition information storage unit, the reference extraction unit, and the sensor state estimation unit uses a different arithmetic processing device from that used by the automatic driving control unit that controls automatic driving. Therefore, even if the sensor state of the periphery monitoring sensor is estimated during automatic driving of the mobile body, the processing load of the automatic driving control unit due to the estimation of the sensor state is not excessive. In addition, the sensor state estimation device including the recognition information storage unit, the reference extraction unit, and the sensor state estimation unit receives power from a power supply system different from that from which the automatic driving control unit receives power. Therefore, it is possible to estimate the sensor state when the operation of the mobile body ends without operating the automatic driving control unit. Therefore, it is possible to suppress an increase in power consumption in the mobile body. As a result, it is possible to estimate the state of the periphery monitoring sensor used for automatic driving while reducing the processing load and power consumption in the mobile body performing automatic driving.

In order to achieve the above object, the sensor state estimation system of the present disclosure includes a surrounding environment recognition unit (14, 15) that recognizes the surrounding environment of the mobile body from the sensing results of a surrounding monitoring sensor that monitors the surroundings of the mobile body and is used for the autonomous driving of the autonomous driving mobile body, an autonomous driving control unit (17) that controls the autonomous driving of the mobile body based on the recognition information, which is information on the surrounding environment recognized by the surrounding environment recognition unit, and the aforementioned sensor state estimation device (10, 10a, 10b).

As a result, since the system includes the aforementioned sensor state estimation device, it is possible to estimate the state of the surrounding monitoring sensors used in autonomous driving while reducing the processing load and power consumption in the autonomous driving vehicle.

1 is a diagram showing an example of a schematic configuration of a mobile body system 1. FIG. 1 is a diagram illustrating an example of a schematic configuration of a sensor state estimation device 10. FIG. FIG. 2 is a diagram for explaining an example of a reference target; FIG. 2 is a diagram showing an example of a schematic configuration of a mobile body system 2. FIG. 2 is a diagram showing an example of a schematic configuration of a vehicle-side unit 1a. FIG. 1 is a diagram illustrating an example of a schematic configuration of a sensor state estimation device 10a. FIG. 2 is a diagram showing an example of a schematic configuration of a mobile body system 2b. 2 is a diagram showing an example of a schematic configuration of a vehicle-side unit 1b. FIG. FIG. 2 is a diagram illustrating an example of a schematic configuration of a sensor state estimation device 10b.

Several embodiments for disclosure will be described with reference to the drawings. For ease of explanation, parts in several embodiments that have the same functions as parts shown in the drawings used in the previous explanations may be given the same reference numerals and their explanations may be omitted. For parts given the same reference numerals, the explanations in other embodiments may be referred to.

<Overall configuration of mobile body system 1>
The mobile body system 1 can be used for a mobile body that performs automatic driving. As shown in FIG. 1, the mobile body system 1 includes a sensor state estimation device 10, a communication module 11, a locator 12, a map database (hereinafter, map DB) 13, an external camera 14, a LiDAR (Light Detection and Ranging/Laser Imaging Detection and Ranging) device 15, a vehicle control ECU 16, an automatic driving ECU 17, an HCU (Human Machine Interface Control Unit) 18, and a presentation device 19. For example, the sensor state estimation device 10, the communication module 11, the locator 12, the map DB 13, the external camera 14, the LiDAR device 15, the vehicle control ECU 16, the automatic driving ECU 17, and the HCU 18 may be configured to be connected to an in-vehicle LAN (see LAN in FIG. 1). The mobile body that uses the mobile body system 1 may be, for example, a vehicle. The mobile body using the mobile body system 1 is not limited to a vehicle as long as it is a mobile body, but the following description will be given using an example of an autonomously driven automobile. An autonomously driven automobile will be referred to as an autonomously driven vehicle hereinafter.

There can be multiple levels of automation for an autonomous vehicle (hereafter referred to as automation level), as defined by the SAE, for example. Automation levels are classified into LV0 to 5, for example, as follows:

LV0 is a level where the driver performs all driving tasks without intervention of the vehicle's system. The driving tasks may be referred to as dynamic driving tasks. Driving tasks include, for example, steering, acceleration/deceleration, and periphery monitoring. LV0 corresponds to so-called manual driving. LV1 is a level where the system assists with either steering or acceleration/deceleration. LV1 corresponds to so-called driving assistance. LV2 is a level where the system assists with both steering and acceleration/deceleration. LV2 corresponds to so-called partial driving automation. LV1 to LV2 are also considered to be part of automated driving. LV3 automated driving is a level where the system can perform all driving tasks under certain conditions, and the driver performs driving operations in an emergency. In LV3 automated driving, the driver is required to be able to respond quickly when the system requests a change of driving. LV3 corresponds to so-called conditional driving automation. LV4 automated driving is a level where the system can perform all driving tasks, except under some circumstances such as roads that cannot be handled and extreme environments. LV4 corresponds to so-called high driving automation. LV5 autonomous driving is a level where the system can perform all driving tasks in any environment. LV5 corresponds to so-called full driving automation.

The autonomous vehicle of this embodiment is assumed to perform autonomous driving at least at LV1 or higher. The autonomous vehicle of this embodiment may be capable of switching the automation level. The automation level may be configured to be switchable only between some of the levels LV0 to 5.

The communication module 11 transmits and receives information via wireless communication with a server device external to the vehicle. In other words, it performs wide-area communication. This server device may be a server device at a center or a server device in the cloud. The communication module 11 may transmit and receive information via wireless communication with other vehicles. In other words, it may perform vehicle-to-vehicle communication. The communication module 11 may transmit and receive information via wireless communication with a roadside unit installed on the roadside. In other words, it may perform road-to-vehicle communication. When performing road-to-vehicle communication, the communication module 11 may receive information about surrounding vehicles transmitted from the surrounding vehicles via the roadside unit.

The locator 12 includes a GNSS (Global Navigation Satellite System) receiver and an inertial sensor. The GNSS receiver receives positioning signals from multiple positioning satellites. The inertial sensor includes, for example, a gyro sensor and an acceleration sensor. The locator 12 sequentially measures the position of the vehicle (hereinafter, the vehicle position) on which the locator 12 is mounted, by combining the positioning signals received by the GNSS receiver with the measurement results of the inertial sensor. The vehicle position is represented, for example, in latitude and longitude coordinates. Note that the vehicle position may also be measured using a travel distance calculated from a signal sequentially output from a vehicle speed sensor mounted on the vehicle.

The map DB 13 is a non-volatile memory that stores map data. The map data may be map data used for route guidance in the navigation function, or may be high-precision map data. The high-precision map data is map data with higher precision than the map data used for route guidance in the navigation function. The high-precision map data includes information that can be used for automated driving, such as three-dimensional shape information of the road, information on the number of lanes, and information indicating the permitted traveling direction for each lane. In addition, the high-precision map data may include information on node points indicating the positions of both ends of road markings such as dividing lines. The locator 12 may be configured not to use a GNSS receiver by using three-dimensional shape information of the road. For example, the locator 12 may be configured to identify the vehicle position using three-dimensional shape information of the road and the detection results of a perimeter monitoring sensor. Examples of the perimeter monitoring sensor include an external camera 14 and a LiDAR device 15, which will be described later.

The communication module 11 may receive map data distributed from an external server device, for example, via wide area communication, and store the data in the map DB 13. In this case, the map DB 13 may be a volatile memory, and the communication module 11 may be configured to sequentially acquire map data for an area corresponding to the vehicle position.

The external camera 14 captures an image of a predetermined range of the outside world of the vehicle. The external camera 14 corresponds to a perimeter monitoring sensor. In FIG. 1, for convenience, a configuration in which the mobile body system 1 includes one external camera 14 is shown, but this is not necessarily limited to this. The mobile body system 1 may include multiple external cameras 14. For example, the mobile body system 1 may be configured to include four external cameras 14 so that the imaging ranges of the front, left side, right side, and rear of the vehicle are respectively provided.

The external camera 14 obtains information on the captured images as a sensing result. The external camera 14 performs image recognition processing on the captured images (hereinafter, captured images). The external camera 14 may recognize, for example, the position of an object relative to the vehicle's position, the size of the object, and the type of the object through image recognition processing. Therefore, the external camera 14 corresponds to the surrounding environment recognition unit. This recognition result corresponds to recognition information, which is information on the surrounding environment of the vehicle. The external camera 14 may sequentially output this recognition information to the in-vehicle LAN. The size of an object may be converted from the size in the captured image to the actual size based on the distance from the vehicle to the object. The size of an object may be the width, height, etc. of the object. In addition, if some components of an object can be detected by image recognition processing, the distance between these components may be recognized as the size. One example is the distance between the tail lights of a vehicle. Other examples include the size of characters on a signboard or road sign, and the size ratio of color coding. The type of object may be recognized, for example, by pattern recognition.

The external camera 14 also has a storage device 141. When the mobile system 1 includes multiple external cameras 14, each external camera 14 may have a storage device 141. The external camera 14 outputs the recognition information to the in-vehicle LAN one after another, and stores the recognition information in the storage device 141. The external camera 14 may be configured to store only recognition information about a predetermined specific object so as not to overload the memory capacity of the storage device 141. Examples of specific objects may be automobiles, guide signs, and warning signs. The storage device 141 may also store specification information about the specifications of the external camera 14, log data of the captured image, and the like, linked to the recognition information. Examples of the captured image log include the detection date and time, detection position, detection distance, reflection intensity, operating voltage, and weather at the time of detection.

The LiDAR device 15 is an optical sensor that emits light to a predetermined range around the vehicle and detects the light reflected by a target. The LiDAR device 15 also corresponds to a perimeter monitoring sensor. For convenience, FIG. 1 shows a configuration in which the mobile body system 1 includes one LiDAR device 15, but this is not necessarily limited to this. The mobile body system 1 may include multiple LiDAR devices 15. For example, the mobile body system 1 may be configured to include three LiDAR devices 15 so that the scanning ranges are respectively in front, to the left rear side, and to the right rear side of the vehicle.

The LiDAR device 15 sequentially obtains, as a sensing result, a scanning result based on a received signal obtained when receiving a reflected wave reflected by an object. The LiDAR device 15 performs object recognition processing on the point cloud obtained as the scanning result. The LiDAR device 15 may recognize, for example, the position of an object relative to the vehicle position, the size of the object, and the type of the object through the object recognition processing. Thus, the LiDAR device 15 corresponds to a surrounding environment recognition unit. This recognition result also corresponds to recognition information, which is information on the surrounding environment of the vehicle. The LiDAR device 15 may sequentially output this recognition information to the in-vehicle LAN. The size of the object may be converted from the size in the coordinate system of the point cloud to the actual size based on the distance from the vehicle to the object. The size of the object may be the width, height, etc. of the object. The type of the object may be recognized, for example, by pattern recognition.

The LiDAR device 15 also has a storage device 151. When the mobile system 1 includes multiple LiDAR devices 15, each LiDAR device 15 may have a storage device 151. The LiDAR device 15 sequentially outputs the recognition information to the in-vehicle LAN, and stores the recognition information in the storage device 151. The LiDAR device 15 may be configured to store only recognition information about a predetermined specific object so as not to overload the memory capacity of the storage device 151. Examples of specific objects include automobiles, guide signs, and warning signs. The storage device 151 may also store specification information about the specifications of the LiDAR device 15, log data of the scanning results, etc., linked to the recognition information. Examples of logs of the scanning results include the detection date and time, the detection position, the detection distance, the reflection intensity, the operating voltage, and the weather at the time of detection.

In this embodiment, the case where the perimeter monitoring sensors included in the mobile body system 1 are the external camera 14 and the LiDAR device 15 will be described as an example, but this is not necessarily limited to this. The perimeter monitoring sensor included in the mobile body system 1 may be only one of the external camera 14 and the LiDAR device 15. Furthermore, the perimeter monitoring sensor included in the mobile body system 1 may be other than the external camera 14 and the LiDAR device 15. Examples of perimeter monitoring sensors other than the external camera 14 and the LiDAR device 15 include millimeter wave radar, sonar, etc.

In this embodiment, the external camera 14 and the LiDAR device 15 are configured to recognize the position, size, and type of an object, but this is not necessarily limited to this. For example, the position, size, and type of an object may be recognized by an electronic control device other than the peripheral monitoring sensor based on the sensing result of the peripheral monitoring sensor. In this case, the recognition information may be stored in the storage device of this electronic control device. In other words, the functional block corresponding to the peripheral environment recognition unit and the functional block that stores the recognition information may be provided in an electronic control device other than the peripheral monitoring sensor. If the power supply system of the functional block corresponding to the peripheral environment recognition unit is different from that of the automatic driving ECU 17, the following may be used. The functional block corresponding to the peripheral environment recognition unit and the functional block that stores the recognition information may be provided within the housing of the automatic driving ECU 17. The configuration including the peripheral environment recognition unit, the automatic driving ECU 17, and the sensor state estimation device 10 corresponds to the sensor state estimation system. In this embodiment, the mobile system 1 corresponds to the sensor state estimation system.

The vehicle control ECU 16 is an electronic control device that controls the driving of the vehicle. Driving control includes acceleration/deceleration control and/or steering control. The vehicle control ECU 16 includes a steering ECU that controls steering, a power unit control ECU that controls acceleration/deceleration, and a brake ECU. The vehicle control ECU 16 controls driving by outputting control signals to each driving control device installed in the vehicle, such as an electronically controlled throttle, a brake actuator, and an EPS (Electric Power Steering) motor.

The autonomous driving ECU 17 is mainly composed of a computer equipped with, for example, a processor, memory, I/O, and a bus connecting these. This processor corresponds to an arithmetic processing device. The autonomous driving ECU 17 corresponds to an autonomous driving control unit. The autonomous driving ECU 17 operates by receiving power from a vehicle power supply. The vehicle power supply is a power supply that is turned on when the vehicle is running. If the vehicle is an engine vehicle, the ignition power supply corresponds to the vehicle power supply. Also, if the vehicle is an electrically powered vehicle such as an electric vehicle or hybrid vehicle, the system main relay corresponds to the vehicle power supply.

The autonomous driving ECU 17 executes processes related to autonomous driving by executing control programs stored in memory. The memory referred to here is a non-transitory tangible storage medium that non-temporarily stores computer-readable programs and data. The non-transitory tangible storage medium is realized by a semiconductor memory or a magnetic disk, for example. The autonomous driving ECU 17 has the following functional blocks: a situation recognition unit, an action determination unit, and a control execution unit.

The situation recognition unit recognizes the situation regarding the vehicle. The situation recognition unit recognizes the driving environment around the vehicle based on the recognition information successively obtained by the surrounding monitoring sensor. In addition to the recognition information from the surrounding monitoring sensor, the situation recognition unit may recognize the driving environment around the vehicle based on the vehicle position acquired from the locator 12 and map data acquired from the map DB 13. As an example, the situation recognition unit uses this information to generate a virtual space that reproduces the actual driving environment.

The behavior determination unit switches the control of driving operations between the driver and the vehicle's system. When the system has control of driving operations, the behavior determination unit determines a driving plan for driving the vehicle based on the recognition results of the situation recognition unit. The driving plan may determine the route to the destination and the behavior that the vehicle should take to reach the destination. Examples of behavior include going straight, turning right, turning left, changing lanes, etc.

When the control authority for driving operations is in the vehicle's system, the control execution unit executes acceleration/deceleration control and steering control of the vehicle in accordance with the driving plan determined by the action determination unit in cooperation with the vehicle control ECU 16. The control execution unit executes, for example, ACC (Adaptive Cruise Control), LTA (Lane Tracing Assist), PCS (Pre-Collision Safety), AEB (Automatic Emergency Braking), and LCA (Lane Change Assist) control.

HCU18 is mainly composed of a computer equipped with, for example, a processor, memory, I/O, and a bus connecting these. HCU18 executes various processes related to the interaction between the occupants and the vehicle's systems by executing control programs stored in the memory. The memory referred to here is a non-transitory tangible storage medium that non-temporarily stores computer-readable programs and data. The non-transitory tangible storage medium is realized by a semiconductor memory, a magnetic disk, or the like.

The presentation device 19 is provided in the vehicle and presents information to the interior of the vehicle. The presentation device 19 presents information according to instructions from the HCU 18. The presentation device 19 only needs to present information to at least the driver. The presentation device 19 may also present information to passengers other than the driver. Examples of the presentation device 19 include a display device and an audio output device. The display device presents information by displaying information. Examples of the display device that can be used include a meter MID (Multi Information Display), a CID (Center Information Display), a HUD (Head-Up Display), and an indicator. The meter MID is a display device provided in front of the driver's seat in the vehicle interior. As an example, the meter MID may be configured to be provided in a meter panel. The CID is a display device located in the center of the instrument panel of the vehicle. The HUD is provided in the vehicle interior, for example, in the instrument panel. The HUD projects a display image formed by a projector onto a predetermined projection area on the windshield as a projection member. The HUD may be configured to project the display image onto a combiner provided in front of the driver's seat instead of the windshield. The audio output device presents information by outputting audio. Examples of the audio output device include a speaker.

The sensor state estimation device 10 is mainly composed of a computer having, for example, a processor, a memory, an I/O, and a bus connecting these. This processor corresponds to an arithmetic processing device. The processor used by the sensor state estimation device 10 is a physically different processor from the processor used by the automatic driving ECU 17. In other words, the sensor state estimation device 10 uses an arithmetic processing device different from that used by the automatic driving ECU 17. The sensor state estimation device 10 executes various processes related to the estimation of the sensor state of the periphery monitoring sensor by executing a control program stored in the memory. The sensor state may be a state related to the presence or absence of an abnormality in the periphery monitoring sensor. The sensor state may be a state related to performance deterioration of the periphery monitoring sensor that is not a temporary abnormality. The sensor state estimation device 10 operates by receiving power supply from a power source other than the vehicle power source. In other words, the sensor state estimation device 10 receives power supply from a power source system other than that from which the automatic driving ECU 17 receives power supply. As an example, the sensor state estimation device 10 may be configured to operate by receiving power supply from a backup power source that can supply power even when the vehicle is parked. The sensor state estimation device 10 may be configured to be provided in a moving body. In this embodiment, the sensor state estimation device 10 is installed in an automobile. Details of the processing by the sensor state estimation device 10 will be described later.

<Overall configuration of sensor state estimation device 10>
Next, a schematic configuration of the sensor state estimation device 10 will be described with reference to Fig. 2. As shown in Fig. 2, the sensor state estimation device 10 includes a storage acquisition unit 101, a recognition information storage unit 102, a reference extraction unit 103, an environmental information recording unit 104, a matching acquisition unit 105, and a sensor state estimation unit 106 as functional blocks. Note that some or all of the functions executed by the sensor state estimation device 10 may be configured as hardware using one or more ICs or the like. Also, some or all of the functional blocks included in the sensor state estimation device 10 may be realized by a combination of software execution by a processor and hardware members.

The storage acquisition unit 101 acquires the recognition information stored in the storage device 141 of the external camera 14. The storage acquisition unit 101 acquires the recognition information stored in the storage device 151 of the LiDAR device 15. The storage acquisition unit 101 stores the acquired recognition information in the recognition information storage unit 102. The storage acquisition unit 101 may be configured to acquire the recognition information while the host vehicle is traveling. The storage acquisition unit 101 may be configured to acquire the recognition information while the host vehicle is parked.

The reference extraction unit 103 extracts the recognition information of the reference object as a reference for estimating the sensor state of the perimeter monitoring sensor. In this embodiment, the recognition information of the reference object for estimating the sensor state of the external camera 14 and the LiDAR device 15 is extracted. The reference extraction unit 103 extracts the recognition information of the reference object from the recognition information stored in the recognition information storage unit 102. The reference extraction unit 103 extracts the recognition information of the reference object from the recognition information sequentially stored in the recognition information storage unit 102, using a target that is constantly recognized at the same point as the reference target and extracting the recognition information of the reference object. The fact that the object is constantly recognized at the same point may be identified from statistical information on the position of the recognized object. For example, a target whose position variance is equal to or less than a threshold may be used as the reference target and the recognition information may be extracted. Examples of the recognition information in this case include the object position, reflection intensity, recognition distance, etc. The operating voltage of the perimeter monitoring sensor may be used as the recognition information. Examples of targets that are constantly recognized at the same point include specific signboards, road signs, etc. It is preferable to extract the reference objects not only from a specific range with respect to the vehicle, but also from all ranges in the left and right directions, upward, nearby, and distant directions. This improves the accuracy of estimating the sensor state. The example in FIG. 3 shows a case where a warning sign LWS in the left direction, a warning sign RWS in the right direction, and a guide sign RGS in the upward direction are extracted as reference objects with respect to the vehicle (HV). The recognition information of the reference objects can be extracted by taking logs over a span of several days to several years, for example.

The reference extraction unit 103 may also include a model identification unit 1031 and an installation identification unit 1032 as sub-functional blocks. The model identification unit 1031 identifies the vehicle model from the recognition information stored in the recognition information storage unit 102. In other words, it identifies the model of the vehicle included in the surrounding environment of the vehicle. The reference extraction unit 103 may then extract size information specific to each vehicle model identified by the model identification unit 1031 as recognition information of the reference target. The size information specific to each vehicle model may be extracted by referring to a database that previously stores the correspondence between vehicle models and size information. In this case, the size information may be, for example, the distance between the tail lamps of the vehicle. Other examples of the size information include the vehicle height and vehicle width. With the above configuration, even a moving vehicle can be used as a reference target.

The installation identification unit 1032 identifies road installations with established standard specifications from the recognition information stored in the recognition information storage unit 102. In other words, it identifies road installations with established standard specifications that are included in the surrounding environment of the vehicle. Examples of road installations with established standard specifications include road signs. The reference extraction unit 103 may then extract size information specific to each road installation identified by the installation identification unit 1032 as recognition information of the reference target. The size information specific to each road installation may be extracted by referring to a database that previously stores the correspondence between the type of road installation and size information. Examples of size information in this case include the size of characters in road signs and the size ratio of color coding. With the above configuration, it is possible to use road installations with established standard specifications as reference targets even before the same target is recognized multiple times at the same location.

When the reference extraction unit 103 extracts the recognition information of a reference target, it may record in the environmental information recording unit 104 the environmental information at the time when the reference target was sensed by the surrounding monitoring sensor. Environmental information is information about the environment. Examples of environmental information include information on the time of day, season, weather, etc. The reference extraction unit 103 may link the extracted recognition information of the reference target with the environmental information and record it in the environmental information recording unit 104.

The matching acquisition unit 105 acquires the recognition information obtained by the external camera 14 and the LiDAR device 15. That is, the matching acquisition unit 105 acquires the recognition information of the reference object included in the surrounding environment of the vehicle recognized from the sensing result of the surrounding monitoring sensor. The matching acquisition unit 105 may be configured to acquire the recognition information when the sensor state estimation unit 106 estimates the sensor state, for example. When the external camera 14 and the LiDAR device 15 sequentially output the recognition information, the matching acquisition unit 105 may acquire the recognition information sequentially output. When the external camera 14 and the LiDAR device 15 do not sequentially output the recognition information, the matching acquisition unit 105 may be as follows. The matching acquisition unit 105 may acquire the most recent recognition information stored in the storage devices 141 and 151, for example. This makes it possible to estimate the sensor state even when the external camera 14 and the LiDAR device 15 do not perform sensing when parking the vehicle.

The sensor state estimation unit 106 compares the recognition information of the reference object previously extracted from the reference extraction unit 103 with the new recognition information of the reference object acquired by the matching acquisition unit 105. The sensor state estimation unit 106 estimates the sensor state by performing this comparison. The recognition information of the reference object previously extracted from the reference extraction unit 103 is hereinafter referred to as matching reference recognition information. The new recognition information of the reference object acquired by the matching acquisition unit 105 is hereinafter referred to as new recognition information. The new recognition information is recognition information obtained by sensing after the matching reference recognition information was obtained by the perimeter monitoring sensor. With the above configuration, it is possible to estimate the sensor state with higher accuracy based on the change in recognition information due to the decrease in detection distance and the narrowing of the detection range caused by the deterioration of the perimeter monitoring sensor. As an example, if the absolute value of the value obtained by subtracting the detection distance as the matching reference recognition information from the detection distance as the new recognition information is equal to or greater than a set value, the deterioration of the sensor can be estimated.

It is preferable that the sensor state estimation unit 106 estimates the sensor state based on the fluctuation of the new recognition information relative to the matching reference recognition information for the past at least one day ago. The matching reference recognition information for the past at least one day ago is the recognition information of the reference target for the past at least one day ago extracted from the reference extraction unit 103. For example, the past older than one day ago may be one year ago or one month ago. With the above configuration, it becomes possible to more accurately estimate the sensor state, such as a long-term abnormality such as aging deterioration that is not a temporary abnormality.

It is preferable that the sensor state estimation unit 106 estimates the sensor state by correcting the matching reference recognition information to matching reference recognition information for the current environment. The sensor state estimation unit 106 may estimate and correct the matching reference recognition information for the current environment based on the difference between the current environment and the environment indicated by the environmental information stored in the environmental information recording unit 104. The current environment can be rephrased as the environment when a new sensing result is obtained by the perimeter monitoring sensor. As a specific example, the above prediction may be performed using a map or the like that stores in advance the tendency of changes in the recognition information, such as the tendency of reflection intensity for each weather, season, and time period. Then, the matching reference recognition information may be corrected by the amount of change in the environmental information due to the weather, season, and time period. According to this, even if the matching reference recognition information for the same environment as the current environment has not been obtained, it is possible to estimate and correct from the obtained matching reference recognition information. As a result, even if the matching reference recognition information for the same environment as the current environment has not been obtained, it is possible to estimate the sensor state with higher accuracy. Note that, if the matching reference recognition information for the same environment as the current environment has been obtained, the matching reference recognition information may be used to estimate the sensor state.

The sensor state estimation unit 106 may send an instruction to the HCU 18 to notify the HCU 18 of the estimated result of the sensor state. The estimated result of the sensor state is hereinafter referred to as a diagnosis result. The notification of the diagnosis result may be made from the presentation device 19 to the occupants of the vehicle. The notification from the presentation device 19 may include a notification on the navigation screen. The notification of the diagnosis result may be made by the terminal of the vehicle manager. The manager may be the owner of the vehicle, a maintenance person, etc. The terminal may be a PC or a multi-function mobile phone. When the diagnosis notification is made by the terminal of the vehicle manager, the following may be performed. The HCU 18 may transmit the diagnosis result via a network to an address registered as the terminal of the vehicle manager, and notify the diagnosis result on the display of the terminal. The diagnosis result may be notified to the vehicle dealer. In this case, the HCU 18 may transmit the diagnosis result via a network to an address registered as the call center of the vehicle dealer. The call center can then notify the dealer of the diagnosis results on the display of their terminal.

Summary of First Embodiment
According to the configuration of the first embodiment, as described above, the sensor state estimation device 10 can estimate the sensor state of the perimeter monitoring sensor with higher accuracy. In addition, the sensor state estimation device 10 uses a different arithmetic processing device from that used by the automatic driving ECU 17. Therefore, even if the sensor state of the perimeter monitoring sensor is estimated during automatic driving of the vehicle, the processing load of the automatic driving ECU 17 is not excessive due to the estimation of the sensor state. In other words, it is possible to suppress interference with the automatic driving process. In addition, the sensor state estimation device 10 receives power from a power supply system different from that from which the automatic driving ECU 17 receives power. Therefore, it is possible to estimate the sensor state when the vehicle is parked without operating the automatic driving ECU 17. Therefore, it is possible to suppress an increase in power consumption in the battery of the vehicle. As a result, it is possible to estimate the state of the perimeter monitoring sensor used for automatic driving by reducing the processing load and power consumption in the moving body performing automatic driving.

(Embodiment 2)
The configuration is not limited to that of the above-described embodiment, and may be that of the following embodiment 2. An example of the configuration of embodiment 2 will be described below with reference to the drawings.

<Overall configuration of mobile body system 2>
The mobile body system 2 shown in FIG. 4 includes a vehicle side unit 1a and a server device 3. The vehicle side unit 1a and the server device 3 communicate with each other via a network. The vehicle side unit 1a is provided in the vehicle HV. Details of the vehicle side unit 1a will be described later. The server device 3 may be a center server device or a cloud server device. The server device 3 corresponds to a storage device external to the mobile body. Details of the server device 3 will be described later. In this embodiment, the mobile body system 2 corresponds to a sensor state estimation system.

<General configuration of vehicle side unit 1a>
5, the vehicle-side unit 1a includes a sensor state estimation device 10a, a communication module 11a, a locator 12, a map DB 13, an external camera 14, a LiDAR device 15, a vehicle control ECU 16, an autonomous driving ECU 17, an HCU 18, and a presentation device 19. The vehicle-side unit 1a is similar to the mobile body system 1 except for some points, in that the vehicle-side unit 1a includes a sensor state estimation device 10a and a communication module 11a instead of the sensor state estimation device 10 and the communication module 11.

The communication module 11a is similar to the communication module 11 of embodiment 1, except for some differences in processing. The differences are described below. The communication module 11a communicates with the server device 3 via a network. That is, the communication module 11a transmits and receives information between the server device 3 and an external storage device. This communication module 11a corresponds to a transmitter/receiver. The communication module 11a may communicate with the server device 3 by cellular communication, for example. For the cellular communication, wireless communication conforming to 5G may be used, for example.

The communication module 11a transmits the recognition information of the reference object extracted by the sensor state estimation device 10a to the server device 3. The communication module 11a also transmits environmental information when the reference object is sensed by the surrounding monitoring sensor to the server device 3. This environmental information may be the same as that described in the first embodiment. The server device 3 records the transmitted recognition information of the reference object. The server device 3 may record the transmitted recognition information of the reference object and the environmental information in association with each other. In addition, the communication module 11a receives the recognition information of the reference object recorded in the server device 3 from the server device 3 as necessary. The communication module 11a may also be configured to receive the environmental information associated with the recognition information of the reference object from the server device 3.

<Overall configuration of sensor state estimation device 10a>
Next, a schematic configuration of the sensor state estimation device 10a will be described with reference to Fig. 6. As shown in Fig. 6, the sensor state estimation device 10a includes a storage acquisition unit 101, a recognition information storage unit 102, a reference extraction unit 103a, a matching acquisition unit 105, and a sensor state estimation unit 106a as functional blocks. The sensor state estimation device 10a includes the reference extraction unit 103a instead of the reference extraction unit 103. The sensor state estimation device 10a includes the sensor state estimation unit 106a instead of the sensor state estimation unit 106. The sensor state estimation device 10a does not include the environmental information recording unit 104. Except for these points, the sensor state estimation device 10a is similar to the sensor state estimation device 10 of the first embodiment.

The reference extraction unit 103a is similar to the reference extraction unit 103 of embodiment 1, except for some differences in processing. The differences are described below. The reference extraction unit 103a transmits recognition information of the extracted reference target to the server device 3 via the communication module 11a. The reference extraction unit 103a may also transmit environmental information at the time when the reference target is sensed by the surrounding monitoring sensor to the server device 3 via the communication module 11a.

The sensor state estimation unit 106a is similar to the sensor state estimation unit 106 of embodiment 1, except for some differences in processing. The differences are described below. The sensor state estimation unit 106a acquires the recognition information of the reference object recorded in the server device 3 from the server device 3 via the communication module 11a. This recognition information of the reference object is the recognition information of the reference object previously extracted by the reference extraction unit 103a. The sensor state estimation unit 106a compares this acquired recognition information of the reference object with new recognition information of the reference object acquired by the comparison acquisition unit 105. By performing this comparison, the sensor state estimation unit 106a estimates the sensor state in the same manner as described in embodiment 1.

When acquiring recognition information of a reference target from the server device 3, the sensor state estimation unit 106a may also acquire linked environmental information. The sensor state estimation unit 106a may estimate and correct matching reference recognition information for the current environment based on the difference between the current environment and the environment indicated by the acquired environmental information. The sensor state estimation unit 106a may estimate and correct matching reference recognition information for the current environment in the same manner as described in embodiment 1.

Summary of Second Embodiment
Even in the configuration of the second embodiment, the sensor state estimation device 10a can estimate the sensor state of the perimeter monitoring sensor with higher accuracy, similar to the sensor state estimation device 10. Moreover, the sensor state estimation device 10a uses a calculation processing device different from that used by the automatic driving ECU 17. Moreover, the sensor state estimation device 10a receives power from a power supply system different from that used by the automatic driving ECU 17. Therefore, it is possible to estimate the state of the perimeter monitoring sensor used for automatic driving with reduced processing load and power consumption in the moving body performing automatic driving. In the second embodiment, the recognition information of the reference object extracted from the reference extraction unit 103a is transmitted to the server device 3, stored, and retrieved as necessary. Therefore, it is possible to reduce the memory consumption on the vehicle side.
(Embodiment 3)
The configuration is not limited to that of the above-described embodiment, and may be that of the following embodiment 3. An example of the configuration of embodiment 3 will be described below with reference to the drawings.

<Overall configuration of mobile body system 2b>
The mobile body system 2b shown in FIG. 7 includes a vehicle side unit 1b and a server device 3b. The vehicle side unit 1b and the server device 3b communicate with each other via a network. The vehicle side unit 1a is provided in the vehicle HV. Details of the vehicle side unit 1b will be described later. The server device 3b includes a sensor state estimation device 10b. The server device 3b may be a center server device or a cloud server device. The server device 3b is provided outside the vehicle HV. Details of the server device 3b will be described later. In this embodiment, the mobile body system 2b corresponds to a sensor state estimation system.

<General configuration of vehicle-side unit 1b>
8, the vehicle-side unit 1b includes a communication module 11b, a locator 12, a map DB 13, an external camera 14, a LiDAR device 15, a vehicle control ECU 16, an autonomous driving ECU 17, an HCU 18, and a presentation device 19. The vehicle-side unit 1b is similar to the mobile body system 1 except that it includes a communication module 11b instead of the communication module 11 and does not include the sensor state estimation device 10.

The communication module 11b is similar to the communication module 11 of the first embodiment, except for some differences in processing. The differences will be described below. The communication module 11b communicates with the server device 3b via a network. The communication module 11b may communicate with the server device 3b by, for example, cellular communication. For the cellular communication, for example, wireless communication conforming to 5G may be used. The communication module 11b transmits the recognition information stored in the storage device 141 of the external camera 14 to the server device 3b. The communication module 11b transmits the recognition information stored in the storage device 151 of the LiDAR device 15 to the server device 3b. The communication module 11b may be configured to sequentially transmit the recognition information while the vehicle HV is traveling. The communication module 11b may be configured to transmit the recognition information obtained while the vehicle HV is traveling while parked.

The communication module 11b receives an instruction to notify the sensor state estimation result (i.e., the diagnosis result) transmitted from the sensor state estimation device 10b of the server device 3b. The instruction to notify the diagnosis result is hereinafter referred to as a notification instruction. The notification instruction received by the communication module 11b is sent to the HCU 18. Then, the HCU 18 may notify the diagnosis result. The notification of the diagnosis result may be performed in the same manner as described in the first embodiment. Note that, when the diagnosis notification is performed at the terminal of the vehicle manager and the vehicle dealer, a configuration may be adopted that does not involve the vehicle side unit 1b. For example, the sensor state estimation device 10b may transmit the diagnosis result to an address registered as the terminal of the vehicle manager, and the diagnosis result may be notified on the display of the terminal. For example, the sensor state estimation device 10b may transmit the diagnosis result to an address registered as the call center of the vehicle dealer. Then, the call center may notify the diagnosis result on the display of the terminal of the dealer.

<Schematic configuration of sensor state estimation device 10b>
Next, a schematic configuration of the sensor state estimation device 10b will be described with reference to FIG. 9. As shown in FIG. 9, the sensor state estimation device 10b includes a storage acquisition unit 101b, a recognition information storage unit 102, a reference extraction unit 103, an environmental information recording unit 104, a matching acquisition unit 105b, a sensor state estimation unit 106b, and a transmission/reception unit 107 as functional blocks. The sensor state estimation device 10b includes a storage acquisition unit 101b instead of the storage acquisition unit 101. The sensor state estimation device 10b includes a matching acquisition unit 105b instead of the matching acquisition unit 105. The sensor state estimation device 10b includes a sensor state estimation unit 106b instead of the sensor state estimation unit 106. The sensor state estimation device 10b includes a transmission/reception unit 107. Except for these points, the sensor state estimation device 10b is similar to the sensor state estimation device 10 of the first embodiment.

The processor used by the sensor state estimation device 10b is a processor provided in the server device 3b. In other words, the sensor state estimation device 10b uses a different calculation processing device from that used by the autonomous driving ECU 17. The sensor state estimation device 10b operates by receiving power from a power source different from the vehicle power source. The sensor state estimation device 10b operates by receiving power from the power source of the server device 3b. In other words, the sensor state estimation device 10b receives power from a power source system different from that which receives power from the autonomous driving ECU 17.

The transmission/reception unit 107 communicates with the communication module 11b via the network. That is, the transmission/reception unit 107 transmits and receives information to and from the communication module 11b. The storage acquisition unit 101b is similar to the storage acquisition unit 101 of the first embodiment, except for some differences in processing. The differences are described below. The storage acquisition unit 101b acquires the recognition information stored in the storage device 141 of the external camera 14 via the transmission/reception unit 107. The storage acquisition unit 101 acquires the recognition information stored in the storage device 151 of the LiDAR device 15 via the transmission/reception unit 107. That is, the recognition information storage unit 102 of the third embodiment acquires and stores the recognition information from the vehicle HV via communication.

The matching acquisition unit 105b is similar to the matching acquisition unit 105 of the first embodiment, except for some different processing. The following describes this difference. The matching acquisition unit 105 acquires the recognition information obtained by the external camera 14 and the LiDAR device 15 via the transmission/reception unit 107. That is, it acquires recognition information of a reference target included in the surrounding environment of the vehicle HV recognized from the sensing result of the surrounding monitoring sensor. The matching acquisition unit 105 may be configured to acquire recognition information, for example, when the sensor state estimation unit 106b estimates the sensor state. The matching acquisition unit 105b may acquire the most recent recognition information stored in the storage devices 141 and 151 via the transmission/reception unit 107. The matching acquisition unit 105b may acquire the recognition information sequentially output from the external camera 14 and the LiDAR device 15 via the transmission/reception unit 107.

The sensor state estimation unit 106b is similar to the sensor state estimation unit 106 of the first embodiment, except for some differences in processing. The differences are described below. The sensor state estimation unit 106b transmits an instruction to notify the estimation result of the sensor state (i.e., a notification instruction) to the communication module 11b via the transmission/reception unit 107.

Summary of the Third Embodiment
Even in the configuration of the third embodiment, the sensor state estimation device 10b can estimate the sensor state of the perimeter monitoring sensor with higher accuracy, similar to the sensor state estimation device 10. Moreover, the sensor state estimation device 10b uses a calculation processing device different from that used by the automatic driving ECU 17. Moreover, the sensor state estimation device 10b receives power from a power supply system different from that used by the automatic driving ECU 17. Therefore, it is possible to estimate the state of the perimeter monitoring sensor used for automatic driving with reduced processing load and power consumption in the moving body performing automatic driving. In the third embodiment, since the sensor state estimation device 10b is outside the vehicle HV, it is possible to estimate the state of the perimeter monitoring sensor used for automatic driving with further reduced processing load and power consumption in the moving body performing automatic driving.

Note that the present disclosure is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. The technical scope of the present disclosure also includes embodiments obtained by appropriately combining the technical means disclosed in different embodiments. The control unit and the method described in the present disclosure may be realized by a dedicated computer that constitutes a processor programmed to execute one or more functions embodied in a computer program. Alternatively, the device and the method described in the present disclosure may be realized by a dedicated hardware logic circuit. Alternatively, the device and the method described in the present disclosure may be realized by one or more dedicated computers that are configured by combining a processor that executes a computer program with one or more hardware logic circuits. Furthermore, the computer program may be stored in a computer-readable non-transient tangible recording medium as instructions executed by the computer.

1 Mobile system (sensor state estimation system), 2, 2b Mobile system (sensor state estimation system), 3 Server device (external storage device), 10, 10a, 10b Sensor state estimation device, 11a Communication module (transmitter/receiver), 14 External camera (surroundings monitoring sensor, surrounding environment recognition unit), 15 LiDAR device (surroundings monitoring sensor, surrounding environment recognition unit), 17 Autonomous driving ECU (autonomous driving control unit), 102 Recognition information storage unit, 103, 103a Reference extraction unit, 104 Environmental information recording unit, 106, 106a, 106b Sensor state estimation unit, 1031 Type identification unit, 1032 Installation identification unit

Claims (10)

A sensor state estimation device that estimates a sensor state, which is a state of a periphery monitoring sensor that is used for automatic driving of a moving body that performs automatic driving and monitors a periphery of the moving body, comprising:
a recognition information storage unit (102) for storing recognition information, which is information on the surrounding environment of the moving object recognized from the sensing result of the surrounding monitoring sensor;
a reference extraction unit (103, 103a) that extracts, from the recognition information stored in the recognition information storage unit, recognition information of a reference object that is used as a reference for estimating the sensor state;
a sensor state estimation unit (106, 106a, 106b) that estimates the sensor state by comparing recognition information of a reference object previously extracted by the reference extraction unit with new recognition information of the reference object included in the surrounding environment of the moving body recognized from a new sensing result of the surrounding monitoring sensor;
A sensor state estimation device that uses a processing device different from that used by an autonomous driving control unit (17) that controls the autonomous driving, and that receives power from a power supply system different from that from which the autonomous driving control unit receives power. The sensor state estimation device according to claim 1 ,
A transmitting/receiving unit (11a) for transmitting and receiving information to and from an external storage device (3) that is an external storage device of the mobile body,
the transmitting/receiving unit transmits recognition information of the reference object extracted by the reference extraction unit to the external storage device, and receives the recognition information of the reference object transmitted to the external storage device from the storage device;
The sensor state estimation unit is a sensor state estimation device that estimates the sensor state by comparing recognition information of a reference target previously extracted by the reference extraction unit with new recognition information of the reference target included in the surrounding environment of the moving body recognized from new sensing results of the surrounding monitoring sensor. The sensor state estimation device according to claim 1 ,
The reference extraction unit is a sensor state estimation device that extracts recognition information of a reference target from the recognition information sequentially stored in the recognition information storage unit, the reference target being a target that is constantly recognized at the same point. The sensor state estimation device according to claim 1 ,
The reference extraction unit has a model identification unit (1031) that identifies a vehicle model, which is a model of a vehicle included in the surrounding environment of the moving object, from the recognition information stored in the recognition information storage unit;
The reference extraction unit is configured to extract size information specific to each vehicle model identified by the model identification unit as recognition information of the reference object. The sensor state estimation device according to claim 1 ,
The criterion extraction unit has an installation identification unit (1032) that identifies a road installation having a standard specification that is included in the surrounding environment of the moving object from the recognition information stored in the recognition information storage unit,
The reference extraction unit is a sensor state estimation device that extracts size information specific to each of the road installations identified by the installation identification unit as recognition information of the reference object. The sensor state estimation device according to claim 1 ,
The sensor state estimation unit is a sensor state estimation device that estimates a sensor state based on fluctuations in new recognition information of a reference object included in the surrounding environment of the moving body, recognized from new sensing results of the surrounding monitoring sensor, relative to recognition information of the reference object extracted from the reference extraction unit for a period at least older than one day. The sensor state estimation device according to claim 1 ,
an environmental information recording unit (104) for recording environmental information, which is information about the environment when the reference object is sensed by the perimeter monitoring sensor, when the reference object recognition information is extracted by the reference extraction unit;
The sensor state estimation unit corrects the recognition information of the reference target previously extracted by the reference extraction unit to recognition information predicted in the environment when the new sensing result is obtained by the perimeter monitoring sensor, based on the difference between the environment when the new sensing result is obtained by the perimeter monitoring sensor and the environment indicated by the environmental information stored in the environmental information recording unit, thereby estimating the sensor state. The sensor state estimation device according to claim 2,
A sensor state estimation device provided in the moving body. The sensor state estimation device according to claim 1 ,
provided outside the moving body,
The recognition information storage unit is a sensor state estimation device that acquires the recognition information from the moving body via communication and stores the recognition information. A surrounding environment recognition unit (14, 15) that recognizes the surrounding environment of the moving body from a sensing result of a surrounding monitoring sensor that is used for the automatic driving of the moving body and monitors the surroundings of the moving body;
An automatic driving control unit (17) that controls automatic driving of the moving body based on recognition information, which is information on the surrounding environment recognized by the surrounding environment recognition unit;
A sensor state estimation system comprising: a sensor state estimation device (10, 10a, 10b) according to any one of claims 1 to 9.

Images