DE102023001624A1 - Anomaly detection in the vehicle by analyzing noise sequences - Google Patents

Abstract

The invention relates to a system for detecting anomalies in events in a vehicle (1), the system being designed to receive information from at least one microphone in the vehicle (1) that detects a sound in the vehicle (1), and consequences of noises in to recognize the information about the sound, as well as to determine the time intervals between the noises and/or the number of noises and/or the order of the noises in each sequence and to compare them with a respective pre-stored comparison pattern, and if a deviation determined in the comparison exceeds a predetermined comparison condition to detect a possible anomaly in an event in the vehicle (1).

Description

The invention relates to a system for detecting anomalies in events in a vehicle and a method for generating a reference pattern for sequences of noises for use in a system for detecting anomalies in events in a vehicle.

In the prior art, for applications in the vehicle sector, apart from human speech, noises are often considered and evaluated as individual events, so that only a reduced class of error cases can be detected.

The DE 10 2019 124 862 A1 relates in this context to a method for operating a motor vehicle, wherein the motor vehicle comprises at least one sensor device, a vehicle door, a locking device for locking the vehicle door, and a control device, wherein the sensor device of the motor vehicle transmits image data relating to a person in the vicinity of the motor vehicle and /or sound data are recorded, with the control device controlling the locking device when an unlocking condition is met in order to unlock the vehicle door, wherein the fulfillment of the unlocking condition depends on the image data and/or sound data and/or the image data and/or sound data or processing data determined from them be transmitted to a processing device external to the vehicle and the fulfillment of the unlocking condition depends on a response message received from the processing device external to the vehicle as a response thereto.

The object of the invention is to be able to better recognize anomalies in events or actions of a user.

The invention results from the features of the independent claims. Advantageous developments and refinements are the subject matter of the dependent claims.

A first aspect of the invention relates to a system for detecting anomalies in events in a vehicle, the system being designed to receive information from at least one microphone in the vehicle that detects sound in the vehicle, and consequences of noises in the information about the sound and to determine the time intervals between the noises and/or the number of noises and/or the order of the noises per sequence and to compare them with a respective pre-stored comparison pattern, and if a deviation determined in the comparison exceeds a predetermined comparison condition, a to detect possible anomaly in an event in the vehicle.

In a vehicle, in particular a passenger car, certain events and noises associated with them often occur in a specific sequence or time interval, i.e. an expectation of sequences and time intervals of noises for various events and actions in the vehicle can be defined. Time intervals between noises that deviate from the expectation or from their sequence can be recognized as possible faults in the underlying events or actions.

The events in the vehicle can occur through the execution and functioning of vehicle subsystems, they can also be actions carried out manually by a user. The nature of events is such that they produce a succession of discrete sounds. The detection of anomalies in these events thus advantageously enables the detection of possible faults in the vehicle's subsystems, as well as possible faults in the actions of a user.

The sequences of noises are, in particular, sequences of individual noises in a finite number, which can be the same as one another, but alternatively can also differ from one another. In particular, if they deviate from each other, it is possible for the system to check their order. Also, if individual sounds differ from each other, the lengths of time between individual sounds can be checked individually without being limited only to the lengths of time between the first and second individual sounds, the second and third individual sounds, etc., which in some cases could lead to a lower reliability of the comparison.

It is an advantageous effect of the invention that user errors or functional errors in the vehicle are automatically detected in order to be able to increase safety in the vehicle. In addition, the range of functions and quality of the system can also be improved over the operating times of the vehicle, and not just designed once during manufacture.

Application examples are closing a door of the vehicle, manually adjusting a seat including disengaging and engaging, plugging in the mobile phone charging cable including the acknowledgment tone for charging, etc.;

In a further embodiment, the correct functioning of vehicle functions is monitored. For this purpose, a comparison model is created in advance, preferably by the manufacturer of the vehicle, for certain functions, for example for the correct opening and closing of a sunroof of the vehicle. The expectation, stored in transformed form in the comparison example, is in the correct order: start of the sunroof motor, opening time 15s, end stop noise, waiting time 400ms, latching noise when the safety catch engages. If one of the noises disappears while the vehicle is in operation, or if one or more periods of time deviate significantly from the comparison pattern, or if the sunroof does not fully engage because something is stuck, the comparison condition is exceeded and the message "Check sunroof" is displayed to the driver.

According to an advantageous embodiment, the system is designed to control an output unit of the vehicle to output information about the possible anomaly.

The driver can be informed of the detected anomaly, which can indicate an irregularity or a fault in a part of the vehicle or an object in the vehicle or incorrect operation of such a part or object; this can be done in particular via an acoustic output, a visual output, or textual feedback. This feedback ideally contains information about the sequence of noises and/or reasons for the anomaly detection and/or the deviation from reference patterns.

According to a further advantageous embodiment, the system is designed to use known states of vehicle elements to assign a sequence of noises to a specific one of the vehicle elements and to determine the comparison pattern depending on the assigned vehicle element or to carry out the comparison or not depending on the state of the assigned vehicle element.

The vehicle elements can be assemblies, components, parts or entire vehicle subsystems, but also objects in the vehicle itself that are not permanently connected to the rest of the vehicle, such as a child seat. A condition of the child seat can be, for example, whether or not it is present in the interior of the vehicle. If, for example, it is recognized, in particular with the help of an interior camera of the vehicle, that there is no child seat in the vehicle, but the characteristic click sequence of seat belt buckles of a 5-point seat belt is supposedly recognised, no anomaly detection is carried out or, if such is carried out, the result is switched to negative , since by knowing that there is no child seat in the vehicle interior, it is recognized in the sense of a plausibility analysis that the affirmation of an anomaly when closing a belt system of a child seat in the absence of the child seat is contradictory and therefore no anomaly can be detected.

According to a further advantageous embodiment, the system is designed to control the output unit of the vehicle to output the information about the possible anomaly of the associated vehicle element.

According to a further advantageous embodiment, the system is designed to determine the state of the associated vehicle element by means of a pressure sensor or a haptic sensor. For example, for the above example, the pressure sensor may be used to detect the presence of an object on a seat of the vehicle, such as a child seat.

According to a further advantageous embodiment, the system is designed to generate or adapt a previously stored comparison pattern by determining an existing sequence of noises in the sound following the input based on an input from a user and storing it transformed into a comparison pattern or based on the sequence Adjusts an existing reference pattern to noise after input.

Another possibility is to give a customer the opportunity to train and categorize sound sequences for a comparison pattern, in that the customer starts a function, for example with a voice command such as "listen now". He then manually triggers the sequence of sounds.

According to a further advantageous embodiment, the system is designed to record an input by the user regarding an association of a vehicle element with the sequence of noises following its input and to store the detected sequence of noises in association with the vehicle element.

A sequence of sounds is thus enriched by context information. Variables known in the vehicle can be used, such as speed, door locking status, location, time, object recognition via cameras in the vehicle (child seat, mobile phone, luggage, etc.). These then support the correct recognition of noise sequences. For example, the sequence of events at a kin The seat, especially its belt, is then only monitored for anomalies if a child in the child seat has already been detected by a camera beforehand. This can reduce the error rate of anomaly detection.

• - Erfassen mittels eines Mikrofons eines Fahrzeugs von Schall und Erzeugen von verarbeitbaren Informationen über den Schall;
• - Ermitteln von Folgen von Geräuschen in den Informationen über den Schall und Abspeichern der ermittelten Folgen; und
• - Ausführen einer statistischen Methode auf eine jeweilige gespeicherte Folge zur Erzeugung eines jeweiligen Vergleichsmusters, insbesondere eines aus:
  • Korrelationsanalyse, Cluster-Algorithmus, Unsupervised Learning;

Another aspect of the invention relates to a method for generating a comparison pattern for sequences of noises for use in a system for detecting anomalies in events in a vehicle by comparing recorded noises with the comparison pattern with regard to the time intervals between the noises and/or the number of noises and/or the order of the sounds per sequence of sounds, comprising the steps:

• - detecting sound by means of a microphone of a vehicle and generating processable information about the sound;
• - determining sequences of noises in the information about the sound and storing the determined sequences; and
• - Execution of a statistical method on a respective stored sequence to generate a respective comparison pattern, in particular one of:
  • correlation analysis, cluster algorithm, unsupervised learning;

According to a further advantageous embodiment, larger quantities of sequences of noises in vehicles, for example in a fleet of development vehicles, can also be collected and the comparative samples can be determined from them. In a manual or semi-manual step, the sequences of noises are then categorized, i.e. e.g. one sequence of noises is named "close sunroof", another as "open sunroof". Methods can now be applied that can map sequential processes, such as directed graphs, recurrent neural networks or other architectures from the field of machine learning. In a first step, the training process, a model is then learned that predicts the categories/names of sequences of sounds. In other words, the model translates the sequential input space into a latent space, which can be, for example, a vector representation. In a second step, the model is now calculated for all elements of the sequence of noises of the training set and thus the latent space is calculated. Cluster algorithms, for example, can now be used in this. In the third step, the model is executed in the vehicle and compared with the reference model for each sequence of noises. If a distance measure exceeds a threshold value as a specified comparison condition, then an anomaly is detected.

Alternatively, reference samples can also be created manually. The comparison samples can also be supplied by a manufacturer of the system or the manufacturer of the vehicle with the system.

Advantages and preferred developments of the proposed method result from an analogous and analogous transfer of the statements made above in connection with the proposed system.

Further advantages, features and details result from the following description, in which at least one exemplary embodiment is described in detail-if necessary with reference to the drawing. Identical, similar and/or functionally identical parts are provided with the same reference symbols.

• 1: Ein Fahrzeug mit einem System zum Erkennen von Anomalien in Ereignissen in dem Fahrzeug gemäß einem Ausführungsbeispiel der Erfindung.
• 2: Ein Verfahren zum Erzeugen eines Vergleichsmusters für Folgen von Geräuschen zur Verwendung in einem System der 1.

Show it:

• 1 : A vehicle having a system for detecting anomalies in events in the vehicle according to an embodiment of the invention.
• 2 : A method of generating a master pattern for sequences of sounds for use in a system of 1 .

The representations in the figures are schematic and not to scale.

1 FIG. 1 shows a vehicle 1 with a system for detecting noise anomalies that follow events such as actions by a user in a vehicle 1. FIG. A child's seat is placed on the rear seat of vehicle 1 and is detected by a pressure sensor 5 as being present. The belt system is designed as a five-point belt. The buckle between the child's legs has a sensor with four contacts. In this way, four straps are connected to the receiver, each of which snaps into place with a clicking sound. The clicking noises are recognized as individual noises in a sequence, which are determined in the form (click, 500ms, click, 700ms, click, 800ms, click) as a time sequence with the respective delay times between the individual noises. The system's algorithm now accesses the database with comparison patterns, which contains an entry in the form (click, 600ms, click, 600ms, click, 600ms, click) associated with the process of fastening the child seat belt. Since the comparison condition is met, no anomaly is detected. In a second situation, a belt has not latched correctly and a pattern appears in the shape (click1, 100ms, click, 700ms, click). This time the comparison condition has been exceeded and a warning is issued at an output unit 3: “Please check the seat belt in the child seat”. However, this comparison only takes place if the pressure sensor 5 (which can alternatively be used with a camera) detects the presence of the child seat.

2 shows a method for generating a master pattern for sequences of sounds for use in a system as in 1 described, but not necessarily limited thereto. In a first step of the method, sound is detected S1 by means of a microphone of a vehicle 1 and information about the sound that can be processed is generated; Furthermore, sequences of noises are determined S2 in the information about the sound and the sequences determined are stored; Finally, a statistical method is executed S3 on a respective stored sequence to generate a respective comparison pattern, in particular one of: correlation analysis, cluster algorithm, unsupervised learning; such a comparison pattern thus generated can adapt an already existing comparison pattern supplied by the manufacturer. If, however, no such comparison model has already been supplied by the manufacturer, for example because a new component in the vehicle interior was only introduced after the vehicle was purchased, a new comparison model can be entered into the database and further adapted by the user based on his input, e.g. new behavior of the user must be taken into account.

Although the invention has been illustrated and explained in more detail by means of preferred exemplary embodiments, the invention is not restricted by the disclosed examples and other variations can be derived therefrom by a person skilled in the art without departing from the protective scope of the invention. It is therefore clear that a large number of possible variations exist. It is also understood that the exemplary embodiments given are really only examples and should not be construed as limiting in any way the scope, applications or configuration of the invention. Rather, the preceding description and the description of the figures enable the person skilled in the art to concretely implement the exemplary embodiments, whereby the person skilled in the art, knowing the disclosed inventive concept, can make a variety of changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment. without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

Reference List

1

vehicle

3

output unit

5

pressure sensor

S1

Capture

S2

Determine

S3

Carry out

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102019124862 A1 [0003]

Claims (8)

System for detecting anomalies in events in a vehicle (1), the system being designed to receive information from at least one microphone in the vehicle (1) detecting a sound in the vehicle (1), and sequences of noises in the information about the to recognize sound and to determine the time intervals between the noises and/or the number of noises and/or the order of the noises per sequence and to compare them with a respective pre-stored reference pattern, and if a deviation determined in the comparison exceeds a specified comparison condition to detect an anomaly in an event in the vehicle (1). system after claim 1 , wherein the system is designed to control an output unit (3) of the vehicle (1) to output information about the possible anomaly. System according to one of the preceding claims, wherein the system is designed to assign a sequence of noises to a specific one of the vehicle elements using known states of vehicle elements and to determine the comparison pattern depending on the assigned vehicle element or to carry out the comparison depending on the state of the assigned vehicle element or not. system after the claims 2 and 3 , wherein the system is designed to control the output unit (3) of the vehicle (1) for outputting the information about the possible anomaly and the associated vehicle element. system according to one of the claims 3 until 4 , wherein the system is designed to determine the state of the associated vehicle element by means of a pressure sensor (5) or a haptic sensor, or by means of an interior sensor, in particular a camera, lidar unit, radar unit, or ultrasonic unit. System according to one of the preceding claims, wherein the system is designed to generate or adapt a previously stored comparison pattern by determining an existing sequence of noises in the sound following an input from a user and transforming it into a comparison pattern and saving it or based on it adapts an existing comparison pattern to the sequence of noises after the input. system after claim 6 , wherein the system is designed to detect an input by the user with regard to an affiliation of a vehicle element to the sequence of noises following its input and to store the detected sequence of noises assigned to the vehicle element. Method for generating a comparison pattern for sequences of noises for use in a system for detecting anomalies in events in a vehicle (1) by comparing recorded noises with the comparison pattern with regard to the time intervals between the noises and/or the number of noises and/or the Order of sounds per sequence of sounds, comprising the steps: - Detecting (S1) of sound by means of a microphone of a vehicle (1) and generating processable information about the sound; - determining (S2) sequences of noises in the information about the sound and storing the determined sequences; and - Execution (S3) of a statistical method on a respective stored sequence to generate a respective comparison pattern, in particular one of: correlation analysis, cluster algorithm, unsupervised learning;

Images