WO2026013006A1 - Method for controlling a vehicle - Google Patents

Abstract

The invention relates to a method with which a driver (100) can control an at least partially autonomous vehicle via speech input. The method comprises the following steps: capturing the speech input of the driver (100) via a microphone (303); identifying, via a speech recognition unit (140), whether the speech input is an actuator instruction to actuators (200) of the vehicle; and, if the speech input is identified as the actuator instruction: outputting a first action instruction to the driver (100) via a loudspeaker (103); checking, via the verification unit (150), a reaction of the driver (100) to the first action instruction being output; and, if the reaction of the driver (100) is assessed as correct by the verification unit (150), using the actuator instruction to control the actuators (200) of the vehicle.

Description

Procedures for controlling a vehicle

Field of invention

The invention relates to a method for controlling an at least partially autonomous vehicle by a driver, in particular by means of voice input. The invention further relates to a device, a vehicle, a use, and a non-volatile, computer-readable storage medium.

Background

Current technology includes voice input systems in vehicles that allow for the control of a variety of non-safety-critical vehicle functions. However, it would be a significant relief for the driver if a partially autonomous vehicle could also be controlled via voice input for safety-critical functions. This requires measures to verify the voice input.

Summary

The object of the invention is to provide a method by which a driver's voice input can be verified. This object is achieved by the subject matter of the independent claims. Further developments of the invention are described in the dependent claims and the following description.

One aspect concerns a procedure for controlling a partially autonomous vehicle by a driver using voice input, comprising the following steps: capturing the driver's voice input via a microphone; recognizing, using a speech recognition unit, whether the voice input is an actuator instruction to the vehicle's actuators; if the voice input is the actuator instruction: issuing, via a loudspeaker, an initial instruction to the driver; verifying, using the verification unit, the driver's response to the output of the initial instruction; If, by means of the verification unit, the driver's response is judged to be correct, the actuator instruction is used to control the vehicle's actuators.

A vehicle can be a road-bound vehicle, such as a passenger car, a truck, or a special-purpose vehicle. The vehicle can also be, for example, a boat. The vehicle is at least a partially autonomous vehicle, i.e., a vehicle of at least Level 1 according to SAE standard J3016. The speech input is a natural language input, meaning that in at least some embodiments, it may be permissible for not just a single term to be understood as an instruction to the vehicle's actuators—or to the vehicle control unit, which controls the actuators at least partially autonomously—but rather a multitude of terms may be permissible. For example, instead of just "slow down," terms such as "slower," "not so fast," etc., may also be considered actuator instructions, i.e., permissible speech inputs.

The driver's voice input is captured using a microphone. The microphone can be, for example, a dynamic microphone, condenser microphone, electret condenser microphone, piezoelectric microphone, and/or another type of microphone. The microphone can be configured as a multiple microphone array, such as a microphone field or array. The microphone may incorporate measures to improve voice quality, particularly through the use of multiple microphones. A microphone is considered particularly suitable for this method if it is capable of clearly capturing the driver's voice and also sound from a loudspeaker, as described below.

The speech recognition unit can use the signal from the microphone to determine whether the voice input is an actuator instruction to the vehicle's actuators. To do this, the speech recognition unit may not only have the ability to recognize words from the voice input, but it may also include a list or database containing valid actuator instructions and their respective meanings. Using the example above, the speech recognition unit recognizes, for instance, the words "slow down,""slower," and "not so fast" (a) as valid, i.e., as actuator instructions, and (b) assigns the meaning "slow down" to all of these words. Other examples of actuator instructions might be: "left,""right,""overtake,""accelerate," etc. If the words are not recognized as valid, the unit will not recognize them as actuator instructions. If the voice input is recognized as an actuator instruction, it can be ignored. In any case, words that are not recognized as actuator instructions are not forwarded to the vehicle's actuators (or to the vehicle control unit). The vehicle's actuators can include, for example, the powertrain, brakes, and steering. In some embodiments, the actuators can also include signaling (e.g., "flashing").

In some embodiments, more complex driving maneuvers, such as overtaking a vehicle in front or stopping the vehicle in the next parking space, are also possible actuator instructions.

In some implementations, an error message can be displayed if the voice input is recognized as similar but not valid. For example, in response to the voice input "decelerate," the response might be: "I didn't understand you."

If the voice input is recognized as an actuator instruction, an initial instruction can be issued to the driver. This initial instruction could be, for example, "Confirm the deceleration by pressing button A" or "Confirm the deceleration by nodding your head." The output can be via a loudspeaker. The loudspeaker can be, for example, a dynamic, electrostatic, piezoelectric loudspeaker, and/or another type of transducer. The loudspeaker can consist of one or more speakers, including, for example, separate speakers for different frequency ranges. The loudspeaker is preferably located near the driver's head or ears, for example, in a headrest.

The driver can react to the initial instruction. A reaction to the examples above could be, for instance, pressing button A or nodding the head. The verification unit can assess this reaction as correct. To do this, the verification unit can query one or more sensors. For example, pressing button A can be detected using a haptic sensor – such as a push button. Or the head nod can be detected using an interior camera designed to capture (at least) this predefined movement of the driver, along with an image recognition unit. The system is configured to evaluate the driver's movements captured by the interior camera, so that the driver's head nod is recognized as such by the verification unit. If any other driver reaction, or no reaction at all, is detected, it is not considered correct. A time window can be defined within which the driver's reaction is considered correct. This time window could, for example, encompass a few seconds after the initial action instruction is issued. If the correct driver reaction is not detected within this time window, it is also not considered correct. To ensure that a particular reaction does not become "habitual," the initial action instruction can be varied from case to case.

If the driver's reaction is judged to be correct by the verification unit, the actuator instruction can be used to control the vehicle's actuators or forwarded to the at least semi-autonomous vehicle control unit.

This provides a method for verifying a driver's voice input. This allows verification—and in at least some cases, confirmation—that the driver's voice input was indeed intended as an actuator command. Furthermore, it improves the quality of the vehicle's human-machine interface. In particular, potential misunderstandings—such as instructions misinterpreted by the speech recognition unit—can be significantly reduced or even eliminated. It also reduces errors caused by factors like incorrect language settings (e.g., Japanese instead of German). Additionally, it reduces the risk of the driver misunderstanding the instruction, for example, due to hearing impairment (e.g., after a loud concert) and/or excessive ambient noise (e.g., in a construction zone).

In some embodiments, the method includes the following further steps: outputting a control signal via the loudspeaker;

Capture, using the microphone, the control signal;

If the control signal is judged to be correct by the verification unit, the actuator instruction is used to control the vehicle's actuators. The control signal can be, for example, an inaudible audio signal with a characteristic modulation. This audio signal could be, for example, modulated near-ultrasound, e.g., in the range between 25 and 30 kHz. The modulation could be, for example, a predefined signal sequence. If this audio signal with the characteristic modulation is detected, the verification unit can assess the control signal as correct. If this audio signal is not detected, the actuator instruction can be ignored. The audio signal might not be detected, for example, if the ratio of the sound pressure level of the audio signal to the ambient noise falls below a predefined value. In this case, the ambient noise might be so loud that the speech recognition unit malfunctions, making it impossible to control the vehicle via voice input.

In some embodiments, haptic input from the driver is the driver's reaction to the output of the instruction. Haptic input can be determined by a haptic sensor. The haptic sensor can be, for example, a button, switch, rotary knob, pressure sensor, proximity sensor, and/or a capacitive touch sensor on a screen or input field. The haptic sensor can be located, for example, on the steering wheel or in the area of a center console. This can be a particularly intuitive input method because many vehicles use haptic sensors on the steering wheel or in the area of the center console for numerous types of input.

In some embodiments, a predefined movement of the driver is the driver's reaction to the issuance of the instruction. The predefined movement can, for example, include a gesture such as waving or nodding.

In some embodiments, a movement of the steering wheel by the driver is the driver's reaction to the issuing of the instruction. The movement of the steering wheel can be interpreted as an objection by the driver to the actuator instruction, for example, if the driver wants to steer manually and does not want to transfer steering control to the at least partially autonomous vehicle control unit. The driver's reaction is thus judged as correct if the driver does not move the steering wheel for a certain period of time after the instruction. Optionally, another "positive" (or correct) reaction from the driver may be present. In some embodiments, the driver's movement of a pedal is their response to the issued instruction. The pedal could be, for example, a brake or accelerator pedal, or possibly a clutch pedal. The movement of the pedal can also be interpreted as the driver's objection to the actuator instruction, for instance, if the driver intends to brake or accelerate themselves. The driver's response is therefore considered correct if the driver does not move the pedal.

In some embodiments, the method includes the following additional steps: detecting, using the microphone, an ambient noise level in addition to the control signal;

If, by means of the verification unit, the ambient noise level is judged to be relatively too loud compared to the control signal, the instruction to the driver will be output via the loudspeaker at an increased volume.

This can be advantageous if the ambient noise level is not yet so loud that the speech recognition unit has a high error rate, but high enough that at least some drivers have difficulty understanding the first instruction.

In some embodiments, the method includes the following further steps: issuing a second instruction to the driver via a loudspeaker;

Check, using the verification unit, the driver's reaction to the issuance of the second instruction;

If, by means of the verification unit, the driver's response is judged to be correct, the actuator instruction is used to control the vehicle's actuators.

This advantageously avoids situations where – for example, if the driver reacts too slowly – the actuator instruction is ignored and has to be repeated by the driver.

In some embodiments, the procedure includes the following additional step: If the verification unit does not judge the driver's response to the first instruction and/or the second instruction to be correct, it ignores the driver's verbal input. Ignoring the driver's voice input means that the potential actuator command is completely ignored and the entire process associated with that voice input is aborted. An error message may be displayed. If the potential actuator command is still to be executed, it must be entered again using voice commands.

One aspect concerns a device for controlling a vehicle by a driver using voice input. The device features:

An interface to a microphone designed to capture the driver's voice input;

A speech recognition unit that is set up to recognize whether the speech input is an actuator instruction;

An interface to a loudspeaker that is set up to issue an instruction to the driver when the voice input is the actuator instruction;

A verification unit, equipped to check the driver's response to the issuance of the instruction by means of a sensor; and

An interface to an at least semi-autonomous vehicle control unit, set up to control the vehicle's actuators based on the actuator instruction, if the verification unit judges the driver's reaction to be correct.

The device can therefore advantageously build upon existing interfaces already present in many vehicles. This can significantly simplify and/or reduce the cost of controlling a vehicle via voice input.

In some embodiments, the loudspeaker is still configured to output a control signal, the microphone is still configured to receive the control signal from the loudspeaker, and the vehicle control unit is still configured to control the vehicle's actuators based on the actuator command. However, this requires that the verification unit evaluates the control signal from the microphone as correct. In cases where the control signal uses low-frequency ultrasound, the loudspeaker can, for example, be implemented as an additional piezoelectric transducer. In some embodiments, the sensor is configured as at least one of the following devices: a haptic sensor configured to detect haptic input from the driver, wherein the haptic sensor is configured as a button, switch, rotary control, pressure sensor, proximity sensor, and/or as a capacitive touch sensor of a screen or input field; and/or an interior camera configured to detect a predefined movement of the driver, and an image recognition unit configured to evaluate the movement of the driver detected by the interior camera; and/or a steering wheel motion sensor configured to detect movement of a steering wheel by the driver; and/or a pedal motion sensor configured to detect movement of a pedal by the driver.

In some embodiments, the action prompt may require further voice input from the driver as confirmation, or a combination of voice input with another driver action that can be detected by a different sensor. The action prompt may include a keyword that the driver must repeat to confirm the actuator instruction, or it may require a mental effort from the driver, e.g., "Confirm acceleration like Captain Picard!" In the latter case, a well-known gesture and a keyword from the popular TV series "Star Trek" would be required to confirm the actuator instruction.

One aspect concerns a vehicle with a device as described above and/or below.

One aspect concerns the use of a device as described above and/or below for the safe control of a vehicle by a driver using voice input.

One aspect concerns a non-volatile, computer-readable storage medium containing a program which, when executed on a processor of a device as described above and/or below, instructs the device to perform the steps as described above and/or below. For further clarification, the invention is described with reference to embodiments illustrated in the figures. These embodiments are to be understood as examples only, and not as limitations.

Brief description of the characters

This shows:

Fig. 1 schematically shows a device for controlling a vehicle according to one embodiment;

Fig. 2 schematically shows a system for controlling a vehicle according to a further embodiment;

Fig. 3 shows a flowchart of a method according to one embodiment.

Detailed description of embodiments

Fig. 1 schematically shows a device for controlling a vehicle according to one embodiment. A driver 100 is shown using a microphone 303 to control the at least partially autonomous vehicle by means of voice input. The microphone 303 can be a single microphone; more typically, the microphone 303 is implemented as a plurality of microphones, e.g., as a microphone field or microphone array. The microphone 303 can include measures for improving speech quality, particularly by utilizing the plurality of microphones, e.g., implemented by means of a microcontroller. The microphone 303 can be a microphone that is "already present" in the vehicle (e.g., for implementing a hands-free system), so that only an interface to the microphone 303 needs to be provided, which supplies the microphone signal Ma, possibly optimized. The voice input (i.e., in the form of the microphone signal Ma) is directed to a speech recognition unit 140, which can recognize whether the voice input is an actuator instruction. The speech recognition unit 140 can not only recognize words from the spoken input, but it can also include a list or database (not shown) in which permissible actuator instructions and their respective meanings are stored. The actuator instruction Ft is then sent to a verification unit 150.

The verification unit 150 has an interface Sa to a

Speaker 103, which is used to issue an initial instruction to the

Driver 100 can be used. The first instruction will only be given then. Output is generated when the voice input is recognized as an actuator instruction. The first instruction might be, for example, "Confirm the actuator instruction by pressing button A" or "Confirm the actuator instruction by nodding your head." Pressing button A can be detected, for example, by a haptic input device, such as a button 111 located on a steering wheel 110. The head nod can be detected by an interior camera 120, which is configured to capture the driver's movements, together with an image recognition unit (not shown) configured to evaluate the driver's movements captured by the interior camera 120 and which provides a signal K. In this way, the driver's head nod can be recognized as a "head nod" by the verification unit 150. The verification unit 150 can further check whether signals P are present from the pedals 112 or whether signals L are present from the steering wheel 110. The signals P can be determined by a pedal sensor 113. The signals L from the steering wheel 110 can be determined by a steering wheel sensor 114. The movement of the steering wheel 110 and/or one of the pedals 112 can be interpreted as an objection by the driver to the actuator instruction, for example, if the driver wants to steer, brake, and/or accelerate himself and does not want to transfer these actions to at least a partially autonomous vehicle control unit.

The verification unit 150 checks the driver's 100 response to the output of the instruction using one of the aforementioned sensors. If the verification unit 150 assesses the driver's 100 response as correct, the actuator instruction is transmitted via interface Af to a partially autonomous vehicle control unit 190, which controls the vehicle's actuators 200 via interface Ak.

Fig. 2 schematically shows a system for controlling a vehicle according to a further embodiment. Identical reference numerals denote identical or similar elements as in Fig. 1. Fig. 2 shows a person 100 – hereinafter referred to as the “driver” 100 – sitting in a driver's seat 101. From the driver's seat 101, the driver 100 can control lateral and longitudinal user controls, e.g., with a steering wheel 110 and with pedals 112 for braking and accelerating. The steering wheel 110 has easily accessible haptic input instruments 114 for the driver 100, which offer various input options, e.g., input buttons or input fields. The signals B of the haptic input instruments 114 and signals L from the steering wheel are Signals 110 and P from the pedals 112 are routed to a verification unit 150. The system includes an interior camera 120. This camera can be used to detect driver reactions to commands Fa. The commands Fa are transmitted via a voice output 130 to one or more loudspeakers 103, which are advantageously located on or near the driver's head, e.g., in a headrest. The system includes a microphone array comprising several microphones 303, 304, which are positioned in the vehicle to optimally capture both the driver's voice input 100 and the loudspeaker 103 audio signals.

A sound evaluation unit 302 can be provided for processing the signals from the microphones 303 and 304. The sound evaluation unit 302 can separate the driver's voice input 100 from the audio signals of the loudspeaker 103 and can combine the microphone signals from 303 and 304, e.g., filter and/or delay them, so that they amplify the voice input and the loudspeaker signals and attenuate other sound sources inside and outside the vehicle, and output the combined microphone signal Ma to both an audio output unit 130 and a speech recognition unit 140. The audio output unit 130 generates the audio signal Sa. In addition to action requests Fa, this can generate an audio signal such that, for example, based on a characteristic modulation from the microphone signal Ma that is not perceptible to humans, it can be determined that the output acoustic signal Sa is the speech reproduction of a digital input signal Fa from the secure verification unit 150, e.g., an action request.

Furthermore, a ratio SN of a disturbing ambient noise in relation to the useful signal Sa can be calculated from the comparison of the microphone signal Ma with the input signal Fa, or, if the characteristic modulation in the microphone signal Ma is not detected, it can be set to such a poor value that this leads to a rejection of the actuator instruction in the verification unit 150.

Furthermore, the system features speech recognition 140, which detects the driver's responses Ft to requests from the verification unit 150 in the combined microphone signal Ma. From the driver's response 110 and/or from the sensor signals K, L, B, F, it can be determined: a) whether the driver's response 110 matches the action request Fa and thus reliable communication with the driver is ensured; or b) whether the driver's response to the question or request for action is unclear or absent; or c) whether the SN signal of the safe voice output indicates that the ambient noise is too loud for safe communication with the driver, or that the electroacoustic transmission link is disrupted, e.g., by a technical defect.

In case a), for example, a voice input from the driver Ft, detected via microphones 303, 304 and sound evaluation unit 302 and by speech recognition 140, can be interpreted as an actuator instruction Af for a vehicle control unit 190 for a predefined period. The at least semi-autonomous vehicle control unit 190 controls the actuators 200, e.g., the vehicle's drive system ("power train"), brakes, and steering, via control signals Ak.

In case b) or after the expiry of the period T, the action instruction Fa can be varied or repeated by the safe verification unit. If the driver responds positively, the system continues to be processed as in case a). If the driver responds incorrectly or not at all, the control system can be overridden by voice input, as in case b). The control system can be put into a so-called safe state, for example, by no longer accepting any actuator instructions from the voice recognition system, e.g., in case c), if the signal SN indicates a permanent defect in the electroacoustic transmission link. Furthermore, information can be output to the driver via another method, e.g., visually via a screen (not shown).

In case c), the safe speech output 130 can increase the audio volume of the signal Sa within a range that does not result in fright or overtaxing of the hearing due to excessively loud signals.

Before starting the journey, the verification unit 150 can establish reliable communication with the driver by issuing prompts via the voice output 130 (and the signals mentioned above). The driver can adjust the audio volume of the audio output signal Sa to a comfortable and easily understandable level through dialogue. The driver's comprehensibility can be checked – after prompts – for example, by tactilely entering a number combination. This can also be verified through a combination of... Other instructions for action, such as operating pedals, steering, or showing a gesture, ensure that audio intelligibility for the driver is maintained.

Fig. 3 shows a flowchart 400 for a method according to one embodiment. In step 402, a voice input from a driver 100 is detected by means of a microphone 303 (see, for example, Fig. 1). In step 404, a speech recognition unit 140 detects whether the voice input is an actuator instruction to actuators 200 of the vehicle. In step 406, it is determined whether the voice input is an actuator instruction to actuators 200 of the vehicle. If the voice input is not an actuator instruction, in step 408 the voice input is ignored and the method is terminated for this voice input. If the voice input is recognized as the actuator instruction, in step 410 a first action instruction is output to the driver 100 by means of a loudspeaker 103. In step 412, the verification unit 150 checks the driver's (100) response to the initial instruction, and a decision is made in step 414. If the driver's (100) response is deemed incorrect, the voice input is ignored in step 416, and the procedure for that input is terminated. If the driver's (100) response is deemed correct, the actuator instruction is used to control the vehicle's actuators (200) in step 416. After steps 408, 416, and/or 418, the procedure can be terminated and subsequently restarted in step 402. ...

Claims

1. Method for controlling a driver (100) of at least a semi-autonomous vehicle by means of speech input, comprising the steps of: capturing, by means of a microphone (303), the speech input of the driver (100); recognizing, by means of a speech recognition unit (140), whether the speech input is an actuator instruction to actuators (200) of the vehicle; if the speech input is recognized as the actuator instruction: issuing, by means of a loudspeaker (103), a first action instruction to the driver (100); checking, by means of the verification unit (150), a response of the driver (100) to the output of the first action instruction; if, by means of the verification unit (150), the response of the driver (100) is judged to be correct, using the actuator instruction to control the actuators (200) of the vehicle. 2. Method according to claim 1, comprising the further steps of: outputting a control signal by means of the loudspeaker (103); detecting the control signal by means of the microphone (303); when the control signal is judged to be correct by means of the verification unit (150), using the actuator instruction to control the actuators (200) of the vehicle. 3. Method according to one of the preceding claims, wherein a haptic input of the driver (100) is the driver's (100) reaction to the output of the instruction. 4. Method according to one of the preceding claims, wherein a predefined movement of the driver (100) is the driver's (100) reaction to the issuance of the instruction. 5. Method according to one of the preceding claims, wherein a movement of a steering wheel (110) by the driver (100) is the driver's (100) reaction to the issuance of the instruction to act. 6. Method according to one of the preceding claims, wherein a movement of a pedal (112) by the driver (100) is the driver's (100) reaction to the issuance of the instruction. 7. Method according to one of the preceding claims, comprising the further steps of: detecting, by means of the microphone (303), an ambient noise level in addition to the control signal; if, by means of the verification unit (150), the ambient noise level is judged to be relatively too loud compared to the control signal, outputting, by means of the loudspeaker (103), the instruction to the driver (100) at an increased volume. 8. Method according to one of the preceding claims, comprising the further steps of: issuing, by means of a loudspeaker (103), a second instruction to the driver (100); checking, by means of the verification unit (150), a reaction of the driver (100) to the issuing of the second instruction; if, by means of the verification unit (150), the reaction of the driver (100) is judged to be correct, using the actuator instruction to control the actuators (200) of the vehicle. 9. Method according to one of the preceding claims, comprising the further step: if the verification unit (150) does not judge the driver's (100's) response to the first instruction and/or the second instruction to be correct, ignore the driver's (100's) verbal input. 10. Device for controlling at least a semi-autonomous vehicle by a driver (100) by means of speech input, the device comprising: an interface to a microphone configured to capture the speech input of the driver (100); a speech recognition unit (140) configured to recognize whether the speech input is an actuator instruction; an interface to a loudspeaker (103) configured to issue an instruction to the driver (100) when the speech input is the actuator instruction; a verification unit (150) configured to check a response of the driver (100) to the output of the instruction by means of a sensor; and an interface to an at least semi-autonomous vehicle control unit (190) configured to control the actuators (200) of the vehicle based on the actuator instruction when the verification unit (150) judges the response of the driver (100) to be correct. 11. Device according to claim 10, wherein the loudspeaker (103) is further configured to output a control signal, wherein the microphone is further configured to receive the control signal from the loudspeaker (103), and wherein the vehicle control unit (190) is further configured to control the actuators (200) of the vehicle based on the actuator instruction, when the verification unit (150) judges the control signal from the microphone to be correct. 12. Device according to claim 10 or 11, wherein the sensor is configured as at least one of the following devices: as a haptic sensor (114) configured to detect haptic input from the driver (100), wherein the haptic sensor (114) is configured as a button, switch, rotary control, pressure sensor, proximity sensor, and/or as a capacitive touch sensor of a screen or input field; and/or as an interior camera (120) configured to detect a predefined movement of the driver (100), and an image recognition unit configured to evaluate the movement of the driver (100) detected by the interior camera (120); and/or as a steering wheel motion sensor (111) configured to detect movement of a steering wheel (110) by the driver (100); and/or as a pedal motion sensor (113) designed to detect movement of a pedal (112) by the driver (100); and/or as a speech sensor comprising a microphone (303) and a speech recognition unit (140) designed to detect verbal driver responses to instructions. 13. Vehicle with a device according to one of claims 10 - 12. 14. Use of a device according to one of claims 10 - 12 for safely controlling a vehicle by a driver (100) by means of speech input. 15. Non-volatile, computer-readable storage medium with a program stored therein which, when executed on a processor of a device according to one of claims 10-12, the The device instructs the user to perform the steps according to any one of claims 1-9.