DE102017216837A1 - Gesture and facial expression control for a vehicle - Google Patents

Abstract

The present invention relates to a vehicle having a plurality of devices and a human-machine interface (HMI) for mimicking and / or gesturing a function of a device of the vehicle, comprising a camera for detecting a specific occupant of the vehicle and a control unit connected to the camera.

Description

The present invention relates to a vehicle having a plurality of devices and a human-machine interface (HMI) for mimicking and / or gesturing a function of a device of the vehicle, comprising a camera for detecting a specific occupant of the vehicle and a control unit connected to the camera. Furthermore, the present invention relates to a method for actuating a function of a device of a vehicle by means of a facial expression and a gesture of a specific occupant of the vehicle.

Modern vehicles usually have a variety of different functions, some of which go far beyond merely controlling the vehicle. Accordingly, vehicles are being developed with the aim of enabling, in particular, the driver of a vehicle to operate and monitor all intended vehicle functions as simply as possible. For this purpose, the individual functions of the vehicle are generally made accessible to the driver via one or more man-machine interfaces.

Each man-machine interface thus allows a mostly bidirectional interaction of the driver or other occupants of the vehicle with the vehicle and in its simplest form, for example, a lever, a knob, a toggle switch, a push button or a pedal and an analog or digital display, include a control or warning light.

Increasingly, vehicles are being equipped with so-called driver assistance systems which assist and / or relieve the driver when controlling a vehicle and thereby reduce the risk of an accident for the vehicle or increase the safety of the vehicle occupants. For example, parking aids are available to assist the driver in parking the vehicle, ranging from simple warning distance sensors to rearview cameras with conveniently observable screens, to fully automated parking assistants, which autonomously reverse a vehicle without driver intervention.

Another example of advanced driver assistance systems are so-called brake assistants, which monitor the space in front of a moving vehicle and automatically decelerate the vehicle when the distance of the vehicle to a preceding vehicle, other road users, such as cyclists or pedestrians, or any other obstacle under one reduced depending on the speed of the vehicle dependent value.

Many traffic accidents of vehicles are due to an inattention of the respective driver. Driver inattention may be due to distraction, lack of concentration, tiredness or a microsleep. A driver assistance system for detecting an inattentiveness of the driver accordingly requires a highly specialized human-machine interface.

So revealed the DE 10 2015 011 522 A1 a driver assistance device for a vehicle, which comprises an image capture device with a camera integrated into the steering wheel of the vehicle. The camera is substantially aligned with the driver's head to capture parameters that are relevant to the driver's inattention or fatigue, such as a blink of the eye, an eyelid, a view, a head tilt, and / or head movements of the driver. Upon detection of inattention or fatigue of the driver, a corresponding warning can be triggered.

From the US 2011/0235919 A1 For example, a similar device is known, which includes a camera and a control unit for detecting open / closed eyes and can be used in a vehicle to detect a driver's microsleep. The device also analyzes, in addition to the degree of opening of the eyes, the entire facial expression of the driver in order to avoid erroneous detection of instantaneous sleep in the driver's facial expressions, which have closed or nearly closed eyes when awake.

On the other hand, driver assistance systems are also known which primarily aim at making operation of the functions of a vehicle more uniform and intuitive as well as at least partially allowing additional vehicle occupants in addition to the driver. For example, the additional vehicle occupants could be enabled by a corresponding assistance system to comfortably operate an entertainment system, such as a radio or a music system installed in the vehicle. Such driver assistance systems naturally require correspondingly complex human-machine interfaces.

The DE 10 2014 218 457 A1 discloses a portable memory unit having a processor for connecting to a data interface of a sensor of a driver assistance system of a vehicle. The memory unit can be connected to various sensors of the driver assistance system and, for interacting with vehicle occupants, comprises a human-machine interface, which has a A camera for optically monitoring a vehicle compartment, a light source for illuminating the vehicle compartment or for outputting light signals, a microphone for acoustically monitoring the vehicle compartment, a speaker for emitting acoustic signals and / or a radar sensor for detecting vehicle occupants in the vehicle compartment in the dark. With the camera, the gesture or facial expressions of a particular vehicle occupant can be detected to determine an actuated function of the vehicle. For this purpose, the Gestik- or Mimikerfassung must be activated by the particular vehicle occupant by voice command. If the transportable storage unit is integrated in a key of a vehicle, it may also be provided to allow starting of the vehicle only after a successful authentication of the driver by means of speech recognition and / or face recognition.

However, it is not yet possible that various vehicle occupants, d. H. the driver and at least the passenger or another passenger, simultaneously or temporally overlapping interact with a vehicle via a common human-machine interface.

The invention is therefore based on the object to provide a vehicle with an improved man-machine interface, which avoids the disadvantages described and allows a simultaneous or temporally overlapping interacting of various vehicle occupants with the vehicle. In addition, it is an object of the invention to provide a method for simultaneously or temporally overlapping actuation of functions of a vehicle by different vehicle occupants.

An object of the invention is a vehicle having a plurality of functional devices and a human-machine interface (HMI) for mimicking and gesturing a function of a device of the vehicle. The man-machine interface comprises a camera for detecting a specific occupant of the vehicle and a control unit connected to the camera.

In one embodiment, the control unit is configured to extract an occupant's facial expression and hand gesture from camera-captured image sequences of the occupant and to combine the extracted facial expression and the extracted hand gesture of the particular occupant to determine an operable function of the vehicle. If the vehicle already has a camera for detecting a vehicle compartment and a control unit, to create a vehicle according to the invention, only installation of special software modules in the control unit of the vehicle is required. The software modules may include an image processing module, an actuation module, and a machine learning engine that interact with one another via corresponding interfaces, respectively. In addition, data storage may be provided in the control unit, which can be read and / or written by the learning module.

A combination of facial expressions and gestures extends facial expressions in comparison to facial expressions or gestures, and allows a greater degree of differentiation and a lower error rate when interacting with devices of the vehicle. In particular, the human-machine interface according to the invention can also be suitable for the deaf. The differentiated expressions of their sign language are based on a combination of mimic and gestural utterances.

In addition, multiple occupants of a vehicle can be well distinguished by a combination of facial expressions and gestures to ensure an undisturbed interaction of a particular occupant, such as the driver with the vehicle, even if overlapping at the same time or overlapping other vehicle occupants interact gesturally with the vehicle.

In one embodiment, the man-machine interface is configured to recognize the occupant based on the occupant's facial expression and a device's actuated function based on the occupant's facial expression and / or hand movement. Facial expressions are particularly well suited to authenticate a particular occupant, d. H. to identify it, while hand movements allow intuitive functional interaction with the vehicle. The functional interaction with the vehicle can be further differentiated and improved by combining facial expressions and hand movements.

In one embodiment, the man-machine interface comprises a data store for storing an interaction profile of the particular occupant, comprising a plurality of associations each of a combination of a recognized facial expression and a detected hand gesture of the occupant to a function of a device. By means of a data memory, a learning function of the human-machine interface can be realized. During interaction of the particular occupant with the vehicle, the man-machine interface stores typical facial expressions and hand movements of the particular occupant from the data store. The stored facial expressions and gesture data significantly reduce the risk of misdirection by the particular occupant. Combined facial expressions and gestures data of each vehicle occupant can be summarized together with to be operated functions of devices in an individual interaction profile. In this case, the particular occupant of the facial expression and / or the hand movement may be free, with which he wants to operate a certain function of a particular device of the vehicle.

In a preferred embodiment, the vehicle has a communication unit connected to the control unit, which is designed for wireless communication with a server (automotive backend) arranged outside the vehicle in order to store an interaction profile of an occupant of the vehicle stored in a data memory of the server, in particular with the latter Consent to load into the control unit or store an interaction profile of an occupant of the vehicle, in particular with his consent, in the data memory of the server. In other words, a vehicle occupant using different vehicles need not re-train every man-machine interface to learn individual peculiarities of his mimetic or gestural utterances. As a result, a vehicle occupant may, for example, readily access his interaction profile generated by training the man-machine interface from all vehicles of a vehicle fleet. This can be useful in larger companies, but also in the context of car sharing. A server in the sense of this invention is also a computer center, a distributed computer center, a cloud or a computer cluster.

In one embodiment of the man-machine interface, the particular occupant is the driver of the vehicle and / or a passenger in the vehicle. The control of conventional vehicles requires extensive interaction of the driver with the vehicle. In such vehicles, therefore, the driver benefits most from the human-machine interface according to the invention. Nevertheless, for example, the man-machine interface allows the passenger or another passenger of the vehicle to operate entertainment devices of the vehicle, while the driver simultaneously operates safety-relevant devices of the vehicle. Vehicles with a highly automatic or fully automatic self-drive function reduce the interaction requirement for the driver, so that in such vehicles, the interaction of other occupants of the vehicle relative to the driver interaction receives more importance.

Another object of the invention is a method for mimic and gestural actuation of a function of a device of a vehicle by a specific occupant of the vehicle, in particular by means of a human-machine interface according to the invention.

In the method according to the invention, image sequences of the particular occupant are recorded continuously in real time by means of a camera. For this purpose, the camera detects an interior of the vehicle in which a driver and possibly other occupants of the vehicle are present. The captured image sequences are analyzed and facial expressions and hand movements of the particular occupant from the real-time images are separately extracted by means of software modules installed in the vehicle, respectively. In this case, the facial expressions and hand movements detected in the real-time images are assigned to the occupant of the vehicle because of the relative spatial arrangement in the real-time images.

According to the invention, the extracted facial expressions and hand movements of the occupant are combined with each other to determine the operation of the vehicle to be operated. Combining mimic and gestural utterances can reduce the risk of misoperation. Apart from that, in a combination mimic and gestural remarks of different occupants of the vehicle can be well distinguished.

Then the determined function of the vehicle is actuated. In other words, the man-machine interface interprets a combination of a recognized facial expression and a detected hand movement of a particular occupant of the vehicle as a command to the vehicle and automatically actuates a corresponding functions of a device of the vehicle.

In one embodiment of the method, the occupant is recognized by the facial expression and the function of the vehicle to be actuated based on the hand movement of the occupant. Facial expressions are particularly well suited to authenticate a particular occupant, d. H. to identify it, while hand movements allow the particular occupant to interact intuitively with the vehicle and more likely to address the vehicle function to be operated. But also different weights of facial expressions and hand movements do not leave the scope of the present invention.

In a preferred embodiment of the method, the separate extraction of the facial expressions and hand movements occurs due to a face and a hand corresponding visible and spatially separated skin areas of the occupant. Visible areas of the skin can usually be distinguished well from their body areas by their color. Generally, the face and hands are the only visible skin areas of a vehicle occupant while other areas of the body are covered by clothing. Accordingly, color contrasts between skin and clothing can be used to detect and extract a facial expression and a hand.

In a further embodiment of the method, the facial expressions are determined by means of a facial expression recognition algorithm and / or the hand movements by means of a gesture recognition algorithm. Such algorithms for recognizing facial expressions and / or gestures may be comprised by a software module and determine and associate facial proportions and / or hand positions or movements from the real-time images.

In one embodiment of the method, the facial expressions and hand movements of the occupant are processed by a machine learning algorithm to better recognize the particular occupant based on facial expressions and / or hand movements. Such an algorithm may be included in a software module. By means of machine learning, the mimic and gestural utterances of an occupant can be reliably determined despite an inevitable natural variability and an occupant of the vehicle can be determined with certainty on the basis of the recognized mimic and gestural utterances.

In one embodiment of the method, a plurality of associations, each of a combination of a recognized facial expression and a recognized hand movement of the occupant to a function of a device of the vehicle is summarized in an interaction profile. Such an interaction profile is associated with the particular occupant and includes typical facial expressions and hand movements of the particular occupant as well as device functions to be actuated therewith.

In a further embodiment of the method, the interaction profile is stored after approval of the particular occupant by means of a provided in the vehicle communication unit in a data store located outside the vehicle and / or loaded from a data store located outside the vehicle. When the interaction profile is stored outside the vehicle, it may be used by the particular occupant in different vehicles. A multiple training of the human-machine interface in each case in different vehicles can be omitted as a result. In the interaction profile, other data of the particular occupant may also be stored for use in different vehicles. In other words, this embodiment provides an occupant-centered method of operating vehicle functions that is advantageous over vehicle-centric methods, for example, for fleets of vehicles that are shared by multiple employees of a company.

• 1 in einer schematischen Darstellung eine Ausführungsform eines erfindungsgemäßen Fahrzeugs,
• 2 in einem Blockschaltbild eine Ausführungsform einer Steuerungseinheit eines erfindungsgemäßen Fahrzeugs und
• 3 in einem Flussdiagramm eine Ausführungsform des erfindungsgemäßen Verfahrens.

The invention is illustrated schematically by means of an embodiment of the vehicle and the method in the drawings and will be further described with reference to the drawings. It shows:

• 1 in a schematic representation of an embodiment of a vehicle according to the invention,
• 2 in a block diagram of an embodiment of a control unit of a vehicle according to the invention and
• 3 in a flowchart an embodiment of the method according to the invention.

The 1 shows a schematic representation of an embodiment of a vehicle according to the invention 10 , The vehicle 10 includes a plurality of devices 11 . 12 . 13 of which only three are shown for the sake of clarity. At the devices 11 . 12 . 13 it may be a vehicle lighting, a windshield wiper system, an air conditioner, a navigation device, an entertainment device or the like.

Further, the vehicle points 10 a man-machine interface 30 (Human Machine Interface, HMI). The man-machine interface 30 includes two cameras 31 . 32 , which are directed into the vehicle compartment to certain occupants 20 . 21 of the vehicle 10 capture. In the illustrated certain occupants 20 . 21 it is the driver 20 of the vehicle 10 and a passenger 21 For example, the passenger in the vehicle 10 , The man-machine interface 30 but can also be designed so that they only the driver 20 , only one particular passenger 21 or additionally also other unrepresented riders recorded. Furthermore, the man-machine interface includes 30 a display unit 33 on the particular inmates 20 . 21 For example, operating states of the devices 11 . 12 . 13 are displayed.

The man-machine interface 30 further comprises a control unit 40 , The control unit 40 is with the cameras 31 . 32 and the display unit 33 connected and configured to the specific occupant 20 . 21 one too Simultaneous mimic and / or gestural actuation of functions of the devices 11 . 12 . 13 to enable.

The vehicle 10 also has one with the control unit 40 connected communication unit 14 on. The communication unit 14 is for wirelessly communicating with a server 50 (Automotive backend) trained to work on a data store 51 of the server 50 To store data or from a data store 51 of the server 50 Load data. The server 50 is outside the vehicle 10 arranged and, for example, centrally with the manufacturer of the vehicle 10 or in a corporate headquarters with a fleet of vehicles 10 be arranged.

The 2 shows a block diagram of an embodiment of a control unit 40 a vehicle according to the invention 10 , In the control unit 40 is an image processing module 41 Installed. The image processing module 41 includes a facial expressions recognition algorithm and a gesture recognition algorithm and is configured to be from the cameras 31 . 32 captured image sequences based on a face and a hand corresponding visible and spatially separated skin areas of a particular occupant 20 . 21 a facial expression and a hand movement of the occupant 20 . 21 to extract.

In the control unit 40 is also an actuation module 42 Installed. The actuation module 42 is configured to perform a function of a device to be operated from a combination of an extracted facial expression and an extracted hand movement 11 . 12 . 13 to determine the function of the device 11 . 12 . 13 to operate and the operating state of the operated device 11 . 12 . 13 on the display unit 33 display.

Further, in the control unit 40 a learning module 43 (Machine Learning Engine) installed. The learning module 43 includes a machine learning algorithm and is configured to recognize extracted facial expressions and extracted hand movements, to learn and to a particular occupant 20 . 21 assigned. The combination of a recognized facial expression with a detected hand movement of a particular occupant 20 . 21 serves a differentiated and reliable determination of one of the occupant 20 . 21 desired function of a device to be operated 11 . 12 . 13 of the vehicle 10 , The learning module recognizes this 43 a certain occupant 20 . 21 based on the facial expression while performing a function of a device to be operated 11 . 12 . 13 of the vehicle 10 on the facial expression and the hand movement of the occupant 20 . 21 recognizes.

Furthermore, the control unit comprises 40 a first data store 44 which is for storing and / or reading an interaction profile of a particular occupant 20 . 21 is configured. An interaction profile of an occupant 20 . 21 includes a plurality of assignments each of a combination of a recognized facial expression and a detected hand movement of the occupant 20 . 21 to a function of a device 11 . 12 . 13 of the vehicle 10 , The learning module 43 can on the first data store 44 access to stored in it interaction profiles of inmates 20 . 21 of the vehicle 10 create and keep up to date. Alternatively or additionally, in the first data memory 44 from the data store 51 of the server 50 loaded interaction profiles are saved and updated locally.

The control unit 40 further includes a second data store 45 that is configured to the learning module 43 Provide reference data on facial expressions and hand movements. The reference data comes from a large number of people and forms a database for comparison purposes. Based on the reference data of the second data memory 45 is the recognition of extracted facial expressions and hand movements for the learning module 43 facilitated.

The 3 shows in a flow chart an embodiment of a method according to the invention 100 , During operation of the man-machine interface 30 become image sequences of the particular occupants 20 . 21 by means of the cameras 31 . 32 continuously recorded in real time. The captured image sequences are taken by the image processing module 41 analyzed on an ongoing basis. As a result of the analysis, facial expressions and hand movements of the particular occupant become 20 . 21 extracted separately from the real-time images 110 ,

The separate extract 110 a facial expression and a hand movement takes place on the basis of corresponding to a face and a hand visible and spatially separated skin areas of the respective occupant 20 . 21 , In this case, the facial expression by means of the image processing module 41 included facial expression recognition algorithm and hand movement by means of the image processing module 41 extracted gesture recognition algorithm.

Based on the extracted facial expression becomes the particular occupant 20 . 21 recognized 120 , A function to be operated on a device 11 . 12 . 13 is first recognized independently on the basis of the extracted facial expression 130 and detected by the extracted hand movement 140 , For a more differentiated and / or more reliable recognition of a function of a device to be actuated 11 . 12 . 13 then become the recognized facial expression and the recognized hand movement combined 150 , Subsequently, the car is automatically taken by the passenger 20 . 21 desired function of the device 11 . 12 . 13 actuated 160.

The facial expressions and hand movements of the occupant 20 . 21 be using the in the learning module 43 implemented algorithm for machine learning. This makes it the human-machine interface 30 possible, facial expressions and / or hand movements of a particular occupant 20 . 21 to learn the particular function of a device to be operated 11 . 12 . 13 of the vehicle 10 to recognize more and more reliable after appropriate training and thereby an inevitable natural variability of the facial expressions and gestures of the occupant 20 . 21 to tolerate.

A plurality of assignments each of a combination of a recognized facial expression and a detected hand movement of the particular occupant 20 . 21 to a function of a device 11 . 12 . 13 of the vehicle 10 becomes an interaction profile of the occupant 20 . 21 summarized and in the first data store 44 the control unit 40 saved.

In each case after approval of the particular occupant 20 . 21 can be an interaction profile from the first data store 44 by means of the communication unit 14 in the data store 51 of the server 50 stored or from the data store 51 of the server 50 in the first data store 44 getting charged.

The human-machine interface according to the invention 30 offers the inmates 20 . 21 of a vehicle 10 a completely new interaction experience, putting them out of their respective seating position in the vehicle 10 out a particularly comfortable, intuitive and reliable interaction with devices 11 . 12 . 13 of the vehicle 10 allowed. Next to the driver 20 can riders 21 - Also at the same time or overlapping with the driver 20 - certain devices 11 . 12 . 13 of the vehicle 10 mimic and gesticulate. The combination of facial expressions and gesture control allows misallocation of occupants 20 . 21 on the one hand and the functions of devices to be operated 11 . 12 . 13 On the other hand, significantly reduce or even completely exclude.

LIST OF REFERENCE NUMBERS

10

vehicle

11

device

12

device

13

device

14

communication unit

20

inmate

21

inmate

30

Human-machine interface

31

camera

32

camera

33

display unit

40

control unit

41

Image processing module

42

actuation module

43

learning module

44

data storage

45

data storage

50

server

51

data storage

100

method

110

Separate extraction of facial expressions and hand movements

120

Recognizing an occupant by facial expression

130

Recognize a function to be operated on the basis of a facial expression

140

Recognizing a function to be operated by means of a hand movement

150

Recognizing a function to be operated based on a combination of a facial expression and a hand movement

160

Actuation of a function

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102015011522 A1 [0007]
• US 2011/0235919 A1 [0008]
• DE 102014218457 A1 [0010]

Claims (11)

Vehicle (10) comprising a plurality of functional devices (11, 12, 13) and a man-machine interface (30) for mimic and / or gestural actuation of a function of a device (11, 12, 13), wherein the man-machine interface (30) comprises a camera (31, 32) for detecting a particular occupant (20, 21) of the vehicle (10) and a control unit (40) connected to the camera (31, 32) which configures is to extract a facial expression and a hand movement of the occupant (20, 21) from image sequences of the occupant (20, 21) captured by the camera (31, 32) and extract the extracted facial expression and the extracted hand movement of the particular occupant (20, 21) combine with each other to determine an operable function of a device (11, 12, 13). Vehicle after Claim 1 in which the man-machine interface (30) is configured to control the occupants (20, 21) based on the facial expression of the occupant (20, 21) and a function to be actuated by a device (11, 12, 13) on the basis of the facial expression and / or the hand movement of the occupant (20, 21) to recognize. Vehicle after one of the Claims 1 or 2 in that the man-machine interface (30) comprises a data memory (44) for storing an interaction profile of the particular occupant (11, 12) having a plurality of assignments each of a combination of a recognized facial expression and a detected hand movement of the occupant (20 , 21) to a function of a device (11, 12, 13). Vehicle after Claim 3 comprising a communication unit (14) connected to the control unit (40) and configured to communicate wirelessly with a server (50) located outside the vehicle (10), around a data memory (51) of the server (50) Interaction profile of an occupant (20, 21) of the vehicle (10), in particular with its consent to load in the control unit (40) or an interaction profile of an occupant (20, 21) of the vehicle (10), in particular with its consent, in the Store data memory (51) of the server (50). A vehicle according to any one of the preceding claims, wherein the particular occupant (20, 21) is the driver (20) of the vehicle (10) and / or a passenger (21) in the vehicle (10). Method (100) for mimicking and gesturing a function of a device (11, 12, 13) of a vehicle (10), in particular of a vehicle according to one of the preceding claims, by a specific occupant (20, 21) of the vehicle (10), in which Image sequences of the particular occupant (20, 21) are recorded continuously in real time by means of a camera (31, 32), the recorded image sequences are analyzed and from the real-time images by means of a control unit (40) of the vehicle (10) a facial expression and a hand movement of the particular occupant (20, 21) are separately extracted (110), - the extracted facial expressions and hand movements of the occupant (20, 21) are combined (150) to determine a function of a device (11, 12, 13) to be actuated, and - The device (11, 12, 13) according to the determined function is actuated (160). Method according to Claim 6 in which the occupant (20, 21) is recognized (120) on the facial expression and / or detects a function of a device (11, 12, 13) to be actuated based on the facial expression and / or the hand movement of the occupant (20, 21) (130, 140). Method according to one of Claims 6 or 7 in which the separate extraction of a facial expression and a hand movement in each case on the basis of corresponding to a face and a hand visible and spatially separated skin areas of the occupant (20, 21) and in which in particular the facial expression by means of a facial expression detection algorithm and / or the hand movement by means of a Gestikerkennungsalgorithmus be recognized. Method according to one of Claims 6 to 8th in which the facial expressions and hand movements of the occupant (20, 21) are processed by means of a learning module (43) to learn facial expressions and / or hand movements of a particular occupant (20, 21). Method according to one of Claims 6 to 9 in which a plurality of associations each of a combination of a recognized facial expression and a recognized hand movement of the occupant (20, 21) to a function of a device (11, 12, 13) of the vehicle (10) is summarized in an interaction profile, which in particular Consent of the particular occupant (20, 21) stored by means of a provided in the vehicle communication unit (14) in a data memory (51) of a vehicle outside the vehicle (10) server (50) and / or from the data memory (51) one outside the Vehicle (10) arranged server (50) is loaded. Method according to Claim 10 in which the interaction profile of the particular occupant (20, 21) in stored in the data memory (51) and loaded from the data memory (51) for selective use in a plurality of different vehicles (10), in particular vehicles (10) of a vehicle fleet of a company or a car-sharing pool.

Images