KR100949558B1 - Driver assistance system - Google Patents

Abstract

The present invention provides a driver assistance system having a structure generally usable for applications related to driver assistance, passenger safety and convenience, such as communication with an external system for diagnostic and maintenance purposes, and the like. A first interface (18) for sensing external signals received from; A second interface 20 for sensing internal signals of the vehicle 12; First subsystems 26 and 28 for fusing output signals of first interface 18 and second interface 20 to generate a description of the environment 10 of the vehicle; A second subsystem 30 for receiving the output signals of the first subsystem 26 to model the current state of the vehicle 12; A third subsystem 32 that models the behavior of the driver 14; A fourth subsystem 36 for interpreting the current landscape described by the first subsystem 28 to determine a feature state of the current exterior state of the vehicle 12; And a fifth sub that generates stimuli for the actuators 42 and / or the driver 14 of the vehicle 12 in accordance with the output signals of the second, third and fourth subsystems 30, 32, 36. Provided is a driver assistance system having systems 38 and 40.

Vehicle, environment, first interface, second interface, first subsystem, second subsystem, driver, third subsystem, fourth subsystem, actuator, fifth subsystem.

Description

Driver Assistance System {DRIVER ASSISTANCE SYSTEM}

Further aspects and advantages of the present invention will become more apparent from the accompanying claims and the description of the embodiments of the present invention.

1 is a functional block diagram of an embodiment of a driver assistance system according to the present invention.

2 is a detailed diagram of the information fusion, landscape representation, and interpretation subsystem of the embodiment of FIG.

Explanation of symbols on the main parts of the drawings

10: environment of the vehicle 12: vehicle

14: Driver 16: Passenger

18: Pre-operation unit for external sensing 20: Pre-operation unit for internal sensing

22: external sensing 24: internal sensing

26: information fusion module 28: landscape representation module

30: vehicle model subsystem 32: driver model subsystem

34: Human-machine interface (HMI) 36: landscape analysis module

38: Behavior Selection / Combination Subsystem 40: Behavior Generation Module

42: vehicle actuator 44: HMI

45: internal passenger sensing 46: landscape description module

48: Rule Selection Module 50: Visual External Sensing

52: extended external sensing 54: feature extraction unit

56: Create Object Hypothesis 58: Information Database

60: extension information extraction 62: conversion / mapping unit

64: property binding unit 66: object hypothesis fusion system

68: Tracking Module 70: Visual Object Classification

D-스케치" 표현 74: 2D 포인트 대상물 표현72: "

D-Sketch "Representation 74: 2D point object representation

76: attention / appropriate filtering

The present invention relates to a driver assistance system and a corresponding method and computer program.

Driver assistance systems are systems that control vehicles or intelligent vehicle technology aimed at increasing the convenience and safety of traffic personnel. Potential applications of such systems include lane departure warning, lane keeping, collision warning or prevention, adaptive cruise control, and low speed automation in crowded traffic situations.

Accordingly, the driver assistance system in the present invention not only "supports" the actions caused or performed by the driver, but also can voluntarily start or perform the control operation of the vehicle.

In general, all the above mentioned applications already require a number of separate sensors and electronic control units (ECUs) to control the sensors, analyze the sensor data and generate the appropriate output signals. For example, radar (RAdio Detecting And Ranging) and LIDAR (Light Detecting And Ranging) sensors can accurately measure the distance to an object being detected, but can be used for semantic information related to the object being detected. For example, it is not possible to provide which lane the vehicle is located and what kind of object is being sensed. This information must be collected separately by the video camera or camera.

As the trend towards increased safety and support for new convenience features for drivers and passengers increases, the complexity of each application, as well as the number of applications in modern vehicles, will increase further in the future. . Basically, a vehicle with multiple sensor systems means a corresponding number of electronic control units (ECUs), all of which carry out their work independently of the others.

For example, a technique is known in which a bus system is provided in a vehicle such as a controller area network (CAN) bus, and almost all or all electronic control units (ECUs) are connected to the can bus. . The canvas allows the exchange of data between connected electronic control units. However, simply adding multiple sensors and / or applications together and interconnecting them on a single bus means that a driver assistance system can be implemented that satisfies a very simple form in terms of reliability or stability and robustness. It turned out.

Nevertheless, it is aimed at limiting the costs for deploying these new and / or improved safety and convenience features while further increasing stability and providing more convenience. Thus, there is a desire to integrate separate, separate sensor systems to date into a single information system capable of actively controlling a vehicle.

Problems of the Prior Art

However, a general model for this behavior generation is not yet known. Conventional driver assistance systems have one or more sensors and accompanying sensor data processing procedures that are specifically implemented only for specific applications. To implement another application, the system must be reconfigured.

In view of the foregoing, it is an object of the present invention to provide a driver assistance system based on a flexible structure that can be easily adapted.

It is a further object of the present invention to provide a driver assistance system which provides a significant improvement in safety and convenience compared to conventional systems.

It is still another object of the present invention to provide a driver assistance system or method that enables sharing of limited resources, such as energy consumption, computing power and space, for example.

It is yet another object of the present invention to provide a driver assistance system that provides robust predictive capability and enables complex behavior.

It is yet another object of the present invention to provide a modular structure concept that allows for easy integration of new components such as, for example, aspects of sensors, new applications or improved internal computing units.

It is yet another object of the present invention to provide a driver assistance system that enables real time control and action selection.

The above objects are achieved by the features of the appended independent claims. Advantageous features are defined in the dependent claims.

According to a first aspect of the present invention, there is provided a driver assistance system having a first interface for sensing external signals received from an environment of a vehicle and a second interface for sensing internal signals of the vehicle. Also provided is a first subsystem for fusing output signals of a first interface and a second interface to produce a description of the environment of the vehicle and to generate internal signals of the vehicle. The second subsystem receives the output signals of the first subsystem to model the current state of the vehicle. The third subsystem is for modeling the driver's behavior. The fourth subsystem is configured to interpret the current scene described by the first subsystem to determine a characteristic state for the current exterior situation of the vehicle. A fifth subsystem is provided for generating stimuli for actuators and / or drivers of a vehicle in accordance with output signals of the first, second, third and fourth subsystems.

The fifth subsystem may be configured to use additional traffic rules depending on the current landscape.

The fifth subsystem may be configured to feed back a signal indicative of specific information related to the generated stimuli to the fourth subsystem. The fourth subsystem may be configured to interpret the current landscape according to the feedback signal of the fifth subsystem.

According to another aspect of the present invention, a driver assistance system having a first interface for sensing external signals received from an environment of a vehicle and a second interface for sensing internal signals of the vehicle are provided. Also provided is a first subsystem for fusing output signals of a first interface and a second interface to produce a description of the environment of the vehicle and to generate internal signals of the vehicle. The second subsystem is for modeling the current state of the vehicle by receiving the output signals of the first subsystem and the third subsystem is for modeling the driver's behavior. A fourth subsystem is provided that interprets the current landscape described by the first subsystem and generates stimuli for the actuators and / or the driver of the vehicle in accordance with the output signals of the second and third subsystems.

The fourth subsystem may be configured to use additional traffic rules depending on the current landscape.

The fourth subsystem may have a scene description subsystem that provides a global description of the exterior landscape, wherein the landscape description subsystem may be configured to represent a characteristic characteristic of the current exterior situation of the vehicle. Determine the general landscape by the output signal of the subsystem and output a signal indicative of the determined general landscape. A subsystem for generating stimuli for actuators and / or drivers of the vehicle may be configured to generate the stimuli in accordance with the output signal of the landscape description subsystem.

The landscape description subunit may provide its output signal to the traffic rule selection.

The fourth subsystem can be configured to also generate a stimulus for the third subsystem. The third subsystem may be configured to model the driver's behavior according to the input signal of the fourth subsystem and an input signal from a human machine interface representing the driver's behavior.

The subsystem for generating a stimulus for the actuator and / or the driver may be configured to further generate the stimuli in accordance with an input signal from passenger monitoring.

The first subsystem may provide an output signal indicative of the physical object description of the object to the behavior generation subsystem.

The first subsystem includes an information extraction and data mapping module for pre-computing sensor data delivered by the first interface, an information fusion module for fusing information meant by output signals of the first and second interfaces, and an object. It may be provided with an information database for storing the attributes and properties, and a landscape expression module for expressing the external landscape.

The information fusion module may provide an object signal indicative of the determined object to the information database, and the information database may provide an attribute signal indicative of the attribute and / or property of the object determined according to the object signal to the landscape representation module.

The information fusion module may have a feature submodule that determines hypothetical object properties from the feature extracted at the output of the first interface, binding the feature to the object and tracking the object. It may further comprise an object sub-module for.

The information database can provide the tracking module of the information fusion module with constraints on the determined possible motion of the object, thereby reducing ambiguity in the object tracking process. In addition, the output signal of the information database can provide the object tracking module with information for selecting an appropriate motion model.

The feature submodule may provide a signal representing the general landscape information to the landscape expression module, wherein the landscape expression module may be configured to determine the constraint condition based on the general landscape information.

The third subsystem for modeling the driver's behavior may provide an input signal related to the general mapping configuration to the landscape representation module.

In a system according to all aspects of the invention as described above, a system equipped with a vehicle is proposed.

According to another aspect of the present invention, a method for supporting a driver of a vehicle is proposed, which is characterized by the features and advantages of the system according to the invention as described above.

According to another aspect of the present invention, a method of filtering the expressed landscape is proposed to focus on object representations appropriate to the current context of the current scenario.

According to another aspect of the present invention, a computer software program product implementing the method as described above is proposed.

The program product may have firmware configured to flash with a programmable device of the vehicle.

According to another aspect of the present invention, a storage device, for example a CD-ROM or a DVD-ROM or the like, in which a computer software program product according to the present invention can be stored, is proposed.

Overall Aspects and Advantages of the Invention

In order to achieve the above object reliably, the present invention proposes a driver assistance system incorporating a priori separated multiple sensor systems to implement a centralized model of the vehicle environment. The model is achieved by analyzing data from different kinds of sensors. The common representation provides the basis for implementing the complex behavior of the system, such as generating warning messages for the driver and / or controlling commands for the various actuators of the vehicle to which the system is connected. To form.

The present invention proposes a general structure of a driver assistance system and in this respect differs from the prior art in which certain conventional sensor systems are separately connected to one another. In the system according to the invention, the sensors are not only assigned to a specific application, but several internal and external sensors are integrated into the system to achieve a centralized representation of the environment as described above. Thus, even new applications can be defined in the vehicle's sensor pool without special changes.

From the foregoing it will be clear that information fusion (data fusion) is an important aspect of an advanced driver assistance system. For example, the object information obtained by the image processing step should be integrated with the data generated from other sensors, such as, for example, radar (RADAR) data, to collectively describe the sensed environment.

The result of the fusion process, or landscape expression, is evaluated by symbolic information and action is created.

Since the driver assistance system according to the present invention operates independently of the driver and outputs information to the driver or acts on the actuators of the vehicle, the system may be referred to as an 'agent' throughout this document.

Some advantages of the present invention are summarized as follows.

The structure of the driver assistance system is based on a centralized representation of the environment and the state of the vehicle itself. Thus, the present invention allows saving of limited resources such as energy, computing power and space, for example. These resources are shared among the various supporting functions implemented within the common structure according to the present invention. In addition, according to the present invention, multiple control centers are coordinated or integrated to stabilize the overall system.

Representation is a form suitable for generating complex or high-level behavior because the agent in accordance with the present invention integrates and stabilizes multiple flows of information over time, thereby ensuring robustness.

Agents according to the present invention also provide immediate access to available information for high-level real-time control and action selection. Since it is technically impossible to interpret the landscape in every detail, perception is organized and supported by prediction and in this way enables complex behavior.

The model according to the invention discloses an open modular structure which allows for easy exchange of components (subsystems, modules, units) to achieve technical advances with minimal cost and effort. Finally, the model also allows for easy consideration of human factors.

Like reference numerals in the drawings denote like or similar components.

1 illustrates a functional block diagram of a particular embodiment of a driver assistance system according to the present invention. The system is connected to the environment 10, the vehicle 12 interacting with the environment 10, the driver 14 and potential additional passengers 16.

The driver assistance system is for example an autonomously operating software system (firmware, flashed) as an agent in the vehicle, i.e. as part of the vehicle 12, flashed on a programmable device. It may be implemented in the form such as. The programmable device may be a single central processing unit or may include a plurality of processors distributed throughout the vehicle 12.

1 illustrates in detail the interaction between the agent and the driver 14 according to the present invention. For the sake of brevity, another possible interaction with an external system or person, such as a service technician in a car shop, for example, is omitted.

First, the functional elements of the present system will be described. In principle, the agent has an input step (interface for input data), an operation step and an output step (interface for output data). Two pre-processing units (18, 20) form the input stage. These include external sensing 22 (eg, radar, Lidar, video, GPS, electronic traffic reporting, vehicle-vehicle-communication, digital maps, etc.) and internal sensing 24 (internal sensing). ) (Eg, speed, acceleration, yaw-rate, tilt, altitude, etc.). The units 18, 20 are connected with a first subsystem of the driver assistance system comprising an information fusion module 26, a scene representation module 28 and a knowledge database 58. The information database 58 is provided with object information of the information fusion module 26 and also outputs the attribute of the object to the landscape representation module 28.

The landscape representation module 28 stores all suitable objects (and attributes of objects provided from the information database 58) that are captured by the external sensing module 22. Every object includes a set of descriptions of its properties (eg, physical properties such as soft / hard) and its relative position to the ego-centered vehicle coordinate system. In addition, the landscape representation module 28 includes semantics in terms of object categories such as 'cars' or 'pedestrians' transmitted from the information fusion module 26, as well as somewhat abstract information such as traffic congestion in a particular area. Semantic information may also be stored.

Details thereof will be described with reference to FIG. 2.

The information fusion module 26 combines all the information provided from the line computing units 18 and 20 to determine a consistent description of the environment stored in the landscape representation module 28. Robustness of the landscape representation is achieved by temporary tracking of the information being fused, which is the current use transmitted by module 28 as well as the new sensor data provided by line computing modules 18 and 20. Need explanation of possible objects. Thus, modules 26 and 28 should be well coupled to each other. Detailed description thereof will be described with reference to FIG. 2.

Internally pre-computed sensing signals are also provided to a second subsystem, the vehicle model subsystem 30.

Vehicle model subsystem 30 is configured to estimate the internal condition of the vehicle based on internal measurements transmitted from module 24. It primarily acts as a state-space observer (eg, Kalman-Filter) to generate robust state estimates if some of the internal sensors fail. . In addition, the vehicle model subsystem 30 provides its own physical dynamics information to the agent, which allows the behavior selection module 38 to determine whether the scheduled behavior can be performed according to physical constraints.

For example, one of the important assessments is the mass of a vehicle, which can vary when carrying many passengers. The mass of the vehicle will have as much influence on the possible accelerations and decelerations and must be taken into account in collision avoidance manoeuvre.

In the third subsystem, the driver model subsystem 32 is connected to a human-machine interface (HMI) 34 operated by the driver 14.

The driver model subsystem 32 serves to determine the driver's intention by observing the driver's behavior from the human-machine interface (HMI) 34 and the current landscape as interpreted by the module 36. do. This intention, for example, requires that side control for lane-keep / lane-off warnings be deactivated to perform a lane-change operation if a turn signal is detected and detected by the landscape analysis module that there are no obstacles in that direction. This is important for selecting the optimal behavior during the behavior selection process in module 38.

Scenery representation module 28 is coupled to a scene interpretation module 36 that is coupled to an action selection / combination subsystem 38. This subsystem 38 is primarily connected to the output stage of the agent with a (low-level) behavior generation module 40, which (low-level) behavior generation module 40 is for example a throttle ( vehicle actuators 42 such as throttles, brakes, gear-shifting, actuators for steering and safety actuators such as airbags or belt-pretensioners, and the like. Connected.

The landscape analysis module 36 analyzes the landscape represented by the module 28 on the detected object. For example, landscape analysis module 36 generates lane information based on the detected lane-border and assigns traffic signals to the correct lane. It also determines the appropriate object, for example, the nearest vehicle ahead in the lane. The landscape analysis module 36 also guides the relevance filtering process in accordance with current work to organize perception and support prediction.

Next, the information processing process in the driver assistance system shown in FIG. 1 is demonstrated. The sensor 24 sensing the internal state of the vehicle may include sensors for speed, acceleration, yaw rate, and inclination. Sensor data is also communicated to the driver via an appropriate HMI 44.

The HMI 44 may also inform the driver 14 of a warning from the behavior selection / combination system 38.

For example, more detailed information may be provided to the support system, such as the speed of each wheel of the vehicle. The pre-computation module 20 serves as a unit for temporarily stabilizing and smoothing input sensor information. In addition, the pre-computation module 20 configures the sensor information for the next information fusion process. For example, the information may be configured to match the time step scheme of the computational step of the agent.

Sensors 22 that sense the environment 10 of the vehicle may, for example, carry video, radar and lidar data. In addition, data captured via radio information, vehicle-vehicle-communication or electronic maps may also be delivered by appropriate devices. The pre-computation unit 18 includes a number of different processes configured to pre-compute sensor data. The two units 18 and 20 deliver the pre-computed data to the information fusion module 26 where the external and internal data are fused. A neural network may be used for data fusion.

The data generated by the fusion process is passed to the landscape representation module 28, where the landscape representing the environment of the vehicle 12 is determined. A scene can, for example, represent a traffic situation. Scene representations generally include a background composed of detected objects such as, for example, cars, motorcycles and / or pedestrians, and the like such as streets, plants and / or the sky. A more detailed description of information fusion and landscape representation will be described below with reference to FIG. 2.

Data associated with the landscape representation is also conveyed as feedback to the information fusion module 26 which supports, for example, prediction at a later time step.

According to the present invention, perception is organized and supported by prediction. The driver subsystem 32 receives data from the driver 14 via the HMI 34. Thus, the module 32 may determine and predict the driver's intention, for example, a planned lane change operation on a multi-lane road (Autobahn, ...). The driver- or passenger-related data is also received from the internal sensing 45 of the vehicle 12. In particular, the internal sensing 45 may include a camera for observing the driver's eyes. Within the driver model subsystem 32, the visual data, along with other data, allows for inferring the driver's condition, such as evaluating whether the driver is awake / sleeping. The driver model 32 determines and expresses the driver's state for use by the agent, as described below.

The data of the landscape representation 28 is further processed within the landscape analysis module 36.

The landscape analysis module 36 outputs signals to the driver model subsystem 32. Appropriate data may include, for example, a representation of statements for things such as a distance to another object (another vehicle) that falls below a predetermined minimum value or (relative) speed of the vehicle above a predetermined value, and the like.

The landscape analysis module 36 also provides signals to a (global) scene description module 46. Here, an overview scenario or landscape description is recorded. The most important scenarios relate to 'urban' or extra-urban, "rural" environments, respectively. Additionally or alternatively, scenarios such as 'Freeway' and 'Highway' may be included. The general features extracted from the precomputation module 18 are passed directly to the landscape description module 46.

Data from the landscape analysis module 36 is fed back to the module 28 to guide the intention / relevance filtering process in the presented scenario, for example to focus on the appropriate object in the current situation. .

From the landscape depiction module 46, the data is passed to the (traffic) rule-selection module 48. The data may include data transmitted from the landscape analysis module 36, such as data associated with one sensed traffic signal or a complex of traffic signals, for example. Module 48 has a traffic signal list and other traffic rules as well as a representation of the general landscape description, each entry in the list being one or more related to rules such as, for example, maximum speed, lane changes, no parking, etc. Instruct the above traffic rules. Allocating all kinds of rules within the landscape to the general landscape and / or objects may be implemented as rule selection module 48.

The agent's subsystem 38 collects data from multiple subsystems and modules. Vehicle model 30 conveys data related to the determined state of vehicle 12. The driver model 32 conveys data related to the state of the driver 14. The landscape analysis module 36 delivers data related to the state variables of the expressed landscape. The landscape description module 46 conveys data related to general scenarios and weather conditions. The rule selection module 48 conveys data related to appropriate traffic rules for the future behavior of the vehicle 12 or the driver 14. The subsystem 38 should interpret the data to define the actions to be taken and to rank the required actions as much as possible according to certain criteria, such as for example safety. The relevant data is also fed back to the landscape analysis module 36.

The subsystem 38 may generate a warning message output through the HMI 44 to the driver. Suitable interfaces may include, for example, visual or audio interfaces such as the output of numbers or letters or voice messages to LEDs, displays or front shields.

Next, data related to one or more actions to be taken are passed to the action generation module 40, which is the output phase of the agent, which has driver-assisted assistance such as ACC, lane-keeping or collision avoidance functions, and the like. The module is located. Additional additional data related to the physical description of the object may be communicated from the landscape representation module 28. Additional data relating to the driver's intention is input from the driver 14 via the HMI 34. In module 40 the actions calculated by the top modules are integrated together. For example, the behavior signal structure may be implemented such that the behavior signal received from the driver 14 (HMI 34) has the highest priority.

The module 40 generates output commands and sends them to appropriate actuators 42, such as throttles, brakes, steering, gears, etc., or airbags, pre-tensioners, internal lighting, temperature control or window regulators, for example. And other actuators, including actuators associated with drivers and passengers.

Embodiments of the invention described herein provide for various interactions between the driver and the driving assistance agent. Appropriate driver behavior is represented by the driver model and has a direct impact on behavior generation. The agent assists the driver by alerting or, more generally, by the control of driver-related actuators and information messages generated by the landscape analysis module. Thus, the system according to the present invention is flexible enough to allow complex interactions and to provide support that does not have to protect itself.

2 is a detailed function of a first sub-system for fusing sensed information to generate a landscape representation, and a fourth sub-system for interpreting the current landscape and selecting an optimal action based on some meaningful information about the external landscape. It is a diagram showing.

The external sensing module 22 has, for example, a visual external sensing unit 50 and an extended external sensing unit 52. The visual external sensing unit 50 captures information sensed by projective sensor elements, such as cameras, radars, and LIDARs, while the extended external sensing unit 52 captures electronic maps. It receives information from non-projecting elements such as data and the like and information contained within the visual range, such as radio traffic reports or vehicle-vehicle-communication messages.

The visually captured data is passed to a feature extraction unit 54, which is an example of an inconsistency (depth) or radar / LIDAR of the edge and stereo-camera system of the camera image. ) Determine the characteristic characteristics of the sensor measurements, such as distance and speed data from the device. In addition, the general characteristics are extracted and provided to the general landscape description unit 46. Suitable general characteristics from camera-images can be determined from frequency-elements (Fourier-spectrum, Gabor-wavelets or power-density spectrum).

The characteristics conveyed by unit 54 are clustered and / or matched to the object-model to produce object hypothesis from each feature modalities by unit 56. . Object hypotheses are defined as potential candidates for real-world objects that must be further validated by additional information integration and consistency verification.

Both the object hypothesis itself and its corresponding characteristics are transformed by the unit 62 into the appropriate coordinate system (if necessary) and fed directly to the information fusion system 26. Fusion system 26 generates the object-hypothesis fused at unit 66 based on the feature binding process at unit 64. Different and unnecessary feature styles are merged to support the corresponding object hypothesis and eliminate inconsistencies in the sensed data. Object-hypothesis fusion and trait binding act in conjunction with each other in this fusion structure and can generally be performed by statistical fusion methods or neural networks. Algorithms that can be used for property binding are obvious to those skilled in the art and are not described here. Robustness to noise and uncertain information in the measurements is achieved by temporary tracking (unit 68) of both the property and the object hypothesis.

The fused object hypothesis is also validated by the visual object classification unit 70, which conveys a semantic description of the object in terms of the type of the object, for example 'car', 'pedestrian' or 'traffic signal'.

차원 스케치 형태로 저장된다. "

D 스케치"라는 용어는 2-D 돌출형 센서 요소만을 사용하여 경관의 실제적인 3-D 특징을 완벽히 탐색하는 것은 가능하지 않다는 것을 의미한다.The physical description of the object (position, velocity, acceleration) and the corresponding object type are determined by the unit 72 used to navigate the visual landscape in the near range within the landscape representation module 28.

It is stored in the form of a dimensional sketch. "

The term "D sketch" means that it is not possible to fully explore the actual 3-D features of the landscape using only 2-D raised sensor elements.

The object category conveyed by the unit 70 is linked with several attributes via the information database 58, for example, some attributes of the detected object may describe different risk-categories based on the solidity of the object. It may be. Other possible attributes may be a motion parameter or motion model for adjusting the tracking unit 68. Attributes are also represented in unit 72.

D 스케치 표현(72)의 특정 지역을 선-활성화(pre-activating) 시키는데 사용된다.Data from the extended external sensing module 52 is fed directly to the information extraction module 60, which performs map data analysis and report analysis from various messages, thereby providing a visual external sensing module 50. Extract relevant information about traffic conditions around the vehicle that cannot be captured by. The obtained information is mapped to the 2D-space point object representation of the unit 74 via the mapping unit 62. The representation in unit 74 is a kind of 'birdview' extensive environmental map, which guides attention and supports prediction in sensor-fusion module 26.

It is used to pre-activate a specific area of the D sketch representation 72.

Examples of data available at unit 74 include map-data information, such as curves to guide the lane detection process or general locations of incidents, such as to guide the lane detection process so that the convergence system can be pre-activated to focus on fixed object detection. There is.

D 스케치 표현에 축적된 데이터는 획득된 대상물 정보에 근거하여 현재 상황에서 무엇이 발생하고 있는가에 대한 일종의 '이해(understanding)'를 에이전트에 제공하는 경관 해석 모듈(36)에서 재차 평가된다. 경관을 '이해'하기 위한 몇 가지 해석은 탐지된 교통 신호를 다른 차선에 할당하거나 탐지된 장애물을 다른 차선에 할당하는 것 등이다. 어떤 것이 길이고 차량이 어디로 진행할 수 있는가에 대한 해석 또한 경관 해석 모듈(36)의 작업에 속한다.

The data accumulated in the D sketch representation is evaluated again in the landscape analysis module 36, which provides the agent with a kind of 'understanding' of what is happening in the current situation based on the acquired object information. Some interpretations to 'understand' the landscape include assigning detected traffic signals to other lanes or assigning detected obstacles to other lanes. The interpretation of what is the road and where the vehicle can go also belongs to the work of the landscape analysis module 36.

Module 36 is a decision / action selection module 38 which is a current landscape that contains continuous information such as the physical properties (distance, speed) of the landscape along with distributed semantic information such as the number of lanes, obstacles in the lane, etc. Provide some status. The decision system 38 obtains the optimal behavior by analyzing the state space of the scene based on this information and occupancy monitoring and transmitted by the driver-model. do.

The landscape analysis module 36 also conveys information in a general landscape description unit 46, ie traffic signals and lane-setting. The general landscape description unit 46 uses the general features delivered by the feature extraction unit 54 to determine the general state of the landscape as described above.

The general landscape description unit 46 also adjusts the visual object classification unit 70 according to different statistics occurring in different general landscapes. The general aspects of the landscape (city, subcenter, main street) are also provided as an information database 48 to determine the appropriate set of traffic rules valid for the current situation. These rules represent some constraints on the behavior selection module 38. In addition, module 38 guides an attention / relevance filtering module 76 to filter the appropriate object according to the task or action. The suitability filtering module 76 also receives the interpreted information from the module 36, which may only transmit objects that are in or adjacent to the lane, for example, for the vehicle.

Behavior selection information from unit 38 activates, deactivates, or combines different driver assistance modules 40 capable of controlling the vehicle based on the physical information of the object conveyed by landscape representation unit 28. Possible functions include longitudinal control (ACC, ACC stop or progress), lateral control (lane-hold, lane departure warning), collision prevention / mitigation, traffic rules warning, and so on.

The internal sensing module 24 is connected to a smoothing / stabilization pre-processing unit 20 as described above. The internal vehicle data is transformed / mapping unit 62 for modifying the mapping or for feature binding process, for example when using tilt angle information for projective transformation, decision / action selection model 38 and support module ( 40 may be used in the vehicle model 30 to provide internal status and dynamics information of the vehicle.

 According to the driver assistance system of the present invention, it is flexible to be easily adapted to the driver.

In addition, the driver assistance system of the present invention is safer and more convenient compared to the conventional system.

In addition, the present invention may share limited resources such as, for example, energy consumption, computing power and space.

In addition, the present invention provides robust predictive capability and enables complex behavior.

In addition, the present invention makes it easy to integrate new components such as, for example, aspects of sensors, new applications or improved internal computing units.

The present invention also enables real time control and action selection.

Explanation of symbols of drawing

(The translation of the drawings should be written in descending order from left to right.)

(Figure 1)

Sensor-fusion, representation and interpretation: system details

52 extended external sensing extended external sensing

-digital map

-traffic reports

-vehicle-vehicle-comm .. vehicle-vehicle communication

50 "visual" external sensing "visual" external sensing

camera

-radar, lidar radar, lidar

-...

22 external sensing External sensing

24 internal sensing internal sensing

velocity, acceleration

yaw-rate rate

tilt tilt

60 information extraction

map-data analysis

-report analysis

56 Building object hypothesis Create object hypothesis

data-clustering

model-matching

54 feature extraction feature extraction

edge, depth ... edge, depth ...

distance, velocity

-global features

62 transformation / mapping transformation / mapping

20 smoothing / stabilization smoothing / stabilization

correction correction

global scene description

weather condition weather condition

prediction prediction

70 visual object categorization

66 fusing object-hypothesis object hypothesis fusion

64 feature binding feature binding

68 tracking tracking

26 fusion system

58 knowledge base Information Database

attribute attribute

type type

object-position object position

74 2D point object representation 2D point object representation

lon-range, environment-map

prediction / attention prediction / attention

72 21/2 D-sketch representation 2 1/2 D-sketch representation

close-range, projective narrow-range protrusions

28 pseudo 3D world-representation Pseudo 3D world representation

global features

76 attention / relevance filtering

relevant object filtering

attention control

36 scene interpretation

traffic sign assignment

-road following

-targets in lane

traffic signs / lane-configuration traffic signs / lane-configuration

46 global scene description

urban / rural

kind of road

-weather

scene interpretation scene interpretation

task dependent scene analysis

"states" of the scene

traffic signs, road description traffic signs, road description

measurement

occupancy monitoring occupancy monitoring

32 driver-model

38 decision state space

state of the environment

states of the vehicle

states of the driver

constraints constraints

48 knowledge base information database

traffic rule selection

behavior control behavior control

30 vehicle model

activation / combination activation / combination

internal states internal states

40 assistance modules

longitudinal control

lateral control

collision mitigation collision mitigation

rule violation warning

-...

physical object description physical object description

(Figure 2)

Driver-agent-interaction model Driver-agent-interaction model

10 environment

24 internal sensing internal sensing

velocity velocity

accel. acceleration

yaw-rate rate

tilt slope

..

22 external sensing External sensing

Radar radar (radar)

LIDAR

Video Video

GPS GPS

traffic reports traffic reports

vehicle-vehicle-comm. Vehicle-vehicle communication

map, .. map ...

20 pre-processing linear computation

2. interface interface

Interface

18 pre-processing linear computation

stabilized mesurements

16 passenger passenger

44 HMI HMI

warning warning

1. subsystem subsystem

58 knowledge base Information Database

objects

attributes attribute

26 information fusion

28 scene representation

2. subsystem subsystem

30 vehicle model

global feature

45 interior sensing interior sensing

occupancy monitoring occupancy monitoring

3. subsystem subsystem

seat beat? Seat belt?

driver ok? Driver ok?

32 driver model driver model

14 driver

34 HMI HMI

driver states driver states

36 scene interpretation

4. sub-system subsystem

46 global scene description (urban / rural) general landscape description (urban / rural)

scenery state variables

38 action selection / combination system

rules rules

48 traffic rule selection

5. sub-system subsystem

40 lowlevel behavior generation

(control loops) control loops

physical object description physical description of the object

vehicle-states vehicle-states

42 actuators

throttle throttle

brake brake

gear gear

steering steering

..

airbags airbags

pretensioner pretensioner

12 vehicle

Claims (19)

A first interface 18 for sensing external signals received from the environment 10 of the vehicle 12; A second interface 20 for sensing internal signals of the vehicle 12; A first subsystem (26, 28) for fusing output signals of the first interface (18) and the second interface (20) to produce a description of the environment (10) of the vehicle; A second subsystem (30) for receiving the output signals of the second interface (20) to model the current state of the vehicle (12); A third subsystem 32 that models the behavior of the driver 14; And A fourth subsystem 36 for interpreting the current landscape described by the first subsystem 28 to determine the feature state of the current exterior state of the vehicle 12, The first subsystem 26, 28 is an information extraction and data mapping module for pre-computing sensor data delivered by the first interface 18, and outputs of the first and second interfaces 18, 20. An information fusion module 26 for fusing information meant by signals, an information database 58 for storing an object with related attributes, and a landscape representation module 28 for describing the landscape, The information fusion module 26 provides an object signal describing the determined object to the information database 58, and the information database 58 expresses an attribute signal representing an attribute of the object determined according to the object signal. Driver assistance system provided to module 28. The method of claim 1, A fifth sub that generates stimuli for the actuators 42 and / or the driver 14 of the vehicle 12 in accordance with the output signals of the second, third and fourth subsystems 30, 32, 36. System (38, 40) Driver assistance system further comprising. The method of claim 2, The fifth subsystem (38, 40) is configured to additionally use additional traffic rules (48) according to the current landscape. The method according to claim 2 or 3, The fifth subsystem 38 is configured to feed back signals to the fourth subsystem 36 indicative of special information associated with the generated stimulus, The fourth subsystem (36) is configured to interpret the current landscape according to the feedback signal of the fifth subsystem (38). delete The method according to claim 2 or 3, Configured to determine the general landscape by the output signal of the fourth subsystem 36 indicative of the characteristic state of the current external state of the vehicle 12 and output a signal representing the determined general landscape, providing a description of the general landscape. Further includes a landscape description subsystem 46, The fifth subsystem 38, 40, which generates stimuli for the actuators 42 and / or the driver 14 of the vehicle 12, generates the stimuli in accordance with the output signal of the landscape description subsystem 46. A driver assistance system configured to generate. The method of claim 6, The scenery description subsystem (46) provides the output signal to the traffic rule selection (48). The method according to any one of claims 1 to 3, The fourth subsystem 36 is configured to further generate a stimulus to the third subsystem 32, wherein the third subsystem 28 is configured to generate an input signal from the fourth subsystem 36 and a driver ( A driver assistance system configured to model the behavior of the driver 14 in accordance with an input signal from the human machine interface 34 indicative of the behavior of 14. The method according to claim 2 or 3, The fifth subsystem 38, 40 for generating stimuli for the actuators 42 and / or the driver 14 is further configured to generate the stimuli further in accordance with an input signal from the passenger monitoring 45. Driver assistance system. The method according to any one of claims 1 to 3, The first subsystem (28) is characterized in that it provides an output signal to the behavior generation module (40) indicative of the physical description of the object. The method according to any one of claims 1 to 3, Driver assistance system comprising a vehicle. Sensing (22) external signals of the vehicle (12); Sensing (24) internal signals of the vehicle (12); Fusing (26) the external and internal signals to produce a description of the environment (10) of the vehicle; Modeling (30) the current state of the vehicle (12) according to the vehicle model (30) and external and internal signals; Modeling the behavior of the driver 14 (32); And Interpreting the current landscape (36) to determine a feature state of the current exterior state of the vehicle 12 Including, Generating a description of the environment 10 of the vehicle 28, information extraction and data mapping step of pre-computing the sensor data according to the external signal, the object for which the information fusion module 26 is to determine the object to be determined Providing a signal to the information database 58 and the information database 58 providing an attribute signal to the landscape representation module 28 indicative of an attribute of the object determined according to the object signal, A method of supporting a driver (14) of a vehicle (12). The method of claim 12, Combining 38 the characteristic state of the landscape with the current state of the vehicle 12 and the behavior of the driver 14 to generate a stimulus for the actuator 42 and / or the driver 14 of the vehicle 12 ( 40) The method of supporting a driver (14) of the vehicle (12) further comprising. A data storage device having stored thereon a computer software program product which implements the method of claim 12 when operated in a computing device. The method of claim 14, The computer software program product A data storage device comprising firmware that is implemented by flashing a programmable device of a vehicle. delete delete delete delete

Images