DE102023124815B3 - System for continuous acoustic monitoring of at least one technical installation - Google Patents

Abstract

The invention relates to a system (1) for in particular continuous acoustic monitoring of at least one technical installation (2), in particular a machine, comprising at least:
- at least one detection unit (3) with at least one acoustic sensor device (3a), which is arranged in the vicinity of the technical installation (2) to be monitored, wherein the detection unit (3) is designed to automatically store acoustic signals of the technical installation (2) detected by means of the sensor device (3a) as audio signal data and to transmit the stored audio signal data and/or data features extracted from at least a part of the stored audio signal data associated with the at least one technical installation (2) to a cloud-based analysis unit (5),
- the cloud-based analysis unit (5), which is communicatively connected to the at least one detection unit (3), and which is configured to analyze and process the system-associated audio signal data received from the at least one detection unit (3) and/or data features extracted from at least a portion of the system-associated audio signal data by means of artificial intelligence, in particular at least one machine learning algorithm and/or at least one neural network.

Description

The invention relates to a system for, in particular, continuous acoustic monitoring of at least one technical installation, in particular a machine.

Systems for monitoring machines and systems that record and analyze acoustic signals are well known in practice. Sound monitoring systems can be very effective because they can detect a wide range of problems. In particular, unusual noises that indicate faulty bearings or other mechanical problems can be detected. They can also enable rapid diagnosis, as listening to noises can be faster than analyzing vibrations or temperature measurements. Furthermore, acoustic monitoring systems can be very accurate and can detect noises that are inaudible to the human ear. This makes it possible to detect even the smallest deviations or irregularities in machine function. In addition, acoustic monitoring systems can be used on a wide range of machines and systems, regardless of size, shape or purpose. They can sometimes also be used in areas where temperature or vibration monitoring is not practical.

It is also known to use artificial intelligence, such as machine learning, deep learning or neural networks, to train algorithms to analyze spectrograms and recognize patterns in the data. These patterns can then be used to identify different types of sound signals (e.g. speech, music or ambient noise) or to detect anomalies or deviations from normal patterns. The acoustic signals are first recorded using microphones and converted into spectrograms. An example of an application of this technology is speech recognition, in which a computer program is trained to recognize human speech and convert it into text.

Recording acoustic signals through microphones and analyzing them using artificial intelligence technologies to create spectrograms is also used in various applications in manufacturing and production environments. In manufacturing, acoustic signals can be recorded from machines and equipment to detect anomalies or problems in real time. Furthermore, in production environments, acoustic signals can be recorded to monitor processes to detect deviations from normal operating conditions. In addition, acoustic signals can also be used to control production processes. For example, machines can be programmed to respond to certain sounds or sound patterns to perform certain actions, such as shutting down when unusual sounds or vibrations are detected.

The EP 3 816 742 A1 refers, among other things, to an electrical system with one or more electrical system components and a monitoring device for monitoring the electrical system. It is provided that the monitoring device has at least one acoustic sensor for detecting the noise of the electrical system to form an acoustic measurement signal and the monitoring device comprises an evaluation device connected to the at least one acoustic sensor, which evaluates the acoustic measurement signal and generates a warning signal if the acoustic measurement signal indicates a malfunction of the electrical system or at least one of the system components.

The US 2020/0393811 A1 concerns an automatic inspection system.

The WO 2010/ 057 661 A1 relates to a method and a device for monitoring a laser processing operation to be carried out on a workpiece and to a laser processing head with such a device.

The WO 2018/ 050 438 A2 reveals the use of comprehensive artificial intelligence in basic industry plants.

The known systems often require complex installation and adaptation to the specific machine or system to be monitored. A complicated setup procedure can mean that installations take longer and professional help may be needed to set up the system properly. If the system to be set up is complex, this can potentially result in high installation and configuration costs. Special and expensive directional microphones are often used, which must be precisely aligned. Systems that are not user-friendly or difficult to set up can in turn lead to users not using the systems correctly or not using them at all, which can significantly limit the effectiveness of the monitoring system. The sometimes high effort required to implement a system for predictive maintenance is currently largely caused by the phase of preparing or preprocessing data. Machine learning and data processing are therefore currently the most common. Due to the considerable effort, high costs and uncertain benefits involved, data-driven projects are not very common in small and medium-sized companies. Potential efficiency gains are initially offset by costs that are difficult to overlook, which are mostly due to poor data quality and data availability. The collection and pre-processing of data and the training of algorithms are among the most time- and resource-intensive tasks and can only be planned to a limited extent. Therefore, these techniques are currently only accessible to larger companies.

Confusing and difficult to understand user interfaces can cause additional problems, such as missing or unclear information, misinterpretation of data, difficulties in using the system and loss of trust, especially in an acoustic surveillance system based on artificial intelligence.

Proceeding from this, the present invention is based on the object of creating a system for, in particular, continuous acoustic monitoring of at least one technical system, which avoids the disadvantages of the prior art, is in particular easy to install and adaptable to various technical systems or machines, and is preferably also cost-effective and/or at least implementable within a foreseeable cost framework.

This object is achieved according to the invention by a system having the features mentioned in claim 1.

• - wenigstens eine Erfassungseinheit mit wenigstens einer akustischen Sensoreinrichtung, welche in der Nähe bzw. im Bereich der zu überwachenden technischen Anlage angeordnet oder anordenbar ist, wobei die Erfassungseinheit dazu eingerichtet und insbesondere derart ausgerichtet ist, automatisiert mittels der Sensoreinrichtung erfasste akustische Signale oder Geräusche der technischen Anlage als Audiosignaldaten zu speichern und die gespeicherten Audiosignaldaten und/oder aus wenigstens einem Teil der gespeicherten Audiosignaldaten extrahierte Datenmerkmale assoziiert mit der wenigstens einen technischen Anlage an eine Cloud-basierte Analyseeinheit zu übermitteln,
• - die Cloud-basierte Analyseeinheit, welche mit der wenigstens einen Erfassungseinheit kommunikationsverbunden ist, und welche dazu eingerichtet ist, die von der wenigstens einen Erfassungseinheit empfangenen anlagenassoziierten Audiosignaldaten und/oder aus wenigstens einem Teil der Audiosignaldaten extrahierte Datenmerkmale mittels künstlicher Intelligenz, nämlich wenigstens eines maschinellen Lernalgorithmus und/oder wenigstens eines neuronalen Netzes zu analysieren und zu verarbeiten,

• - wobei bei deren Nichtvorhandensein automatisiert, insbesondere von der Cloud-basierten Analyseeinheit eine für die anlagenassoziierte Speicherung und Verarbeitung der empfangenen Audiosignaldaten und/oder der Datenmerkmale geeignete Datenumgebung, insbesondere Datenbanken und/oder Dateien, insbesondere in einer der Cloud-basierten Analyseeinheit zugeordneten Cloud erzeugt werden, wobei eine Vorverarbeitung der empfangenen Audiosignaldaten und/oder der Datenmerkmale durchgeführt wird,
• - wobei mittels der empfangenen anlagenassoziierten Audiosignaldaten und/oder der Datenmerkmale, der wenigstens eine maschinelle Lernalgorithmus und/oder das wenigstens eine neuronale Netz, insbesondere von der Cloud-basierten Analyseeinheit, unbeaufsichtigt trainiert werden, um einen Normalzustand der technischen Anlage aufzuzeichnen bzw. aufzunehmen und eine Unterscheidung von abnormen Audiosignaldaten und Datenmerkmalen zu ermöglichen, und
• - wobei die Analyse von neuen bzw. neu empfangenen anlagenassoziierten Audiosignaldaten und/oder Datenmerkmalen, insbesondere von der Cloud-basierten Analyseeinheit, insbesondere in Abhängigkeit von bereits aufgezeichneten oder gespeicherten Audiosignaldaten und/oder Datenmerkmalen, mehrstufig, in drei unterschiedlichen Stufen erfolgt.

According to the invention, a system for continuous acoustic monitoring of at least one technical installation, in particular a machine, is proposed, which comprises at least:

• - at least one detection unit with at least one acoustic sensor device, which is arranged or can be arranged in the vicinity or in the area of the technical installation to be monitored, wherein the detection unit is set up and in particular oriented to automatically store acoustic signals or noises of the technical installation detected by means of the sensor device as audio signal data and to transmit the stored audio signal data and/or data features extracted from at least a part of the stored audio signal data associated with the at least one technical installation to a cloud-based analysis unit,
• - the cloud-based analysis unit, which is communicatively connected to the at least one detection unit and which is configured to analyze and process the system-associated audio signal data received from the at least one detection unit and/or data features extracted from at least a portion of the audio signal data by means of artificial intelligence, namely at least one machine learning algorithm and/or at least one neural network,

• - in the absence of such data, a data environment suitable for the system-associated storage and processing of the received audio signal data and/or the data features, in particular databases and/or files, are generated automatically, in particular by the cloud-based analysis unit, in particular in a cloud assigned to the cloud-based analysis unit, wherein a preprocessing of the received audio signal data and/or the data features is carried out,
• - wherein, by means of the received system-associated audio signal data and/or the data features, the at least one machine learning algorithm and/or the at least one neural network are trained unsupervised, in particular by the cloud-based analysis unit, in order to record or record a normal state of the technical system and to enable a distinction to be made between abnormal audio signal data and data features, and
• - wherein the analysis of new or newly received system-associated audio signal data and/or data features, in particular from the cloud-based analysis unit, in particular depending on already recorded or stored audio signal data and/or data features, is carried out in several stages, in three different stages.

By means of the system according to the invention, technical systems and installations can be monitored 24 hours a day, seven days a week, without the explicit need for personnel. By acoustically recording all signals in the relevant rooms and automatically detecting anomalies using artificial intelligence, the system according to the invention, as an all-in-one solution, takes on the role of a qualified maintenance engineer for the relevant technical systems and installations after a short training period. Since the solution according to the invention learns independently on the basis of the information available in the room or area to be monitored, a wide variety of technical systems and installations can be monitored. Systems are monitored. Customizing or the like is not necessary. This increases the availability of the technical systems and systems, reduces the susceptibility to errors and saves costs and time. This can improve process quality. Getting started with artificial intelligence is made much easier with the all-in-one solution according to the invention. Without any configuration, customizing or the initiation of a complex project, the system according to the invention can represent a simple introduction to artificial intelligence. The possibility of remote diagnosis allows the travel times of technicians to be optimized or minimized. There is no need for complicated installation of the monitoring system in the technical system or machine. The system according to the invention can detect a large number of different causes of failure. The system can continuously learn and adapt to the knowledge of specialist personnel. It is also conceivable that it can be used in environments in which people cannot or should not spend long periods of time. Notifications of possible anomalies or failures can be sent by email, for example.

In the context of the invention, a technical system is to be understood as a planned and systematic assembly of spatially related apparatus, devices and/or machines, including associated buildings or supporting structures, which can be connected to one another in terms of function, control technology or safety technology. The technical systems can include a large number of moving parts as well as electrical or electronic or optoelectronic devices.

The system according to the invention can be used as a plug-and-play system for the automated acoustic recording and analysis of machine and system noises without the need for complex installations or adaptations for different machines or systems. The system comprises a recording unit and a cloud-based artificial intelligence analysis platform (smart cloud). The recording unit, which is appropriately aligned and arranged or can be arranged near the technical system to be monitored, automatically records acoustic signals from the machine or system, extracts features and transmits them together with the recorded acoustic signals to the analysis unit or analysis platform. The recording unit can be a cost-effective alternative to expensive special equipment. This reduces the cost of the equipment, while at the same time making it possible to use the technology according to the invention in smaller production environments. In the cloud-based analysis platform, the appropriate databases or files for storing the recorded sound data and/or features are automatically created, the data is preprocessed for the downstream machine learning algorithms, the machine learning algorithms for anomaly detection and noise classification are trained, and the noises are analyzed. The analysis unit has a system for the automated analysis of acoustic signals, which is based on a multi-stage algorithm. First, a normal state is created from several audio files, which is used to train an algorithm that can distinguish between normal and abnormal signals. The user does not have to manually classify data at the beginning in order to obtain meaningful predictions. The system uses multi-stage anomaly detection, whereby states found by the unsupervised learning algorithm can serve as the basis for more efficient supervised learning. It is possible to issue special alarms based on the detected noises, which are sent, for example, by email. The system can be used in various application areas, such as machine diagnostics, safety monitoring and quality assurance.

The system according to the invention comprises a combination of a low-cost acquisition unit and a cloud-based artificial intelligence analysis unit or platform that automatically creates databases and trains machine learning algorithms. The system can be used for different machines or systems without complex installations or adaptations, making it a practical and user-friendly solution. The system according to the invention can be considered an innovative and effective solution that offers a simple and cost-efficient way to capture and analyze acoustic signals from machines and systems.

• - werden in einer ersten Stufe Anomalien erkannt, insbesondere wenn eine Abweichung von dem aufgezeichneten Normalzustand der wenigstens einen technischen Anlage vorliegt,
• - ist in einer zweiten Stufe ein Fehler der wenigstens einen technischen Anlage klassifizierbar, insbesondere ein Klappern oder eine Kavitation, und
• - ist in einer dritten Stufe ein Ausfall der wenigstens einen technischen Anlage vorhersagbar.

When analyzing the new or newly received system-associated audio signal data and/or data characteristics:

• - in a first stage, anomalies are detected, in particular if there is a deviation from the recorded normal state of the at least one technical installation,
• - in a second stage, a fault in the at least one technical system can be classified, in particular a rattling or cavitation, and
• - in a third stage, a failure of at least one technical system can be predicted.

The analysis of sound data takes place in three stages, which has the advantage that the time-consuming and costly collection and preprocessing of data by machine learning algorithms is significantly reduced. Every machine or system has its own background noise, which is why the algorithm must first learn the different sounds. To do this, however, a lot of audio data must be classified with the corresponding label (normal/abnormal). To minimize this effort, the normal state of the system is first recorded and an unsupervised learning algorithm is trained. This technique can then be used to detect deviations from the recorded normal state.

The sound analysis is structured in three stages, since sufficient data must be collected for each stage.

In the first stage, the normal state of the machine can be recorded. Any deviation represents an anomaly and may indicate a fault in the machine.

In the second stage, the system can not only detect an anomaly but also indicate to the relevant maintenance employee what type of anomaly (e.g. cavitation) is present.

In the third stage, it is possible to predict when (e.g. in a number of days) failures of technical equipment and systems are likely to occur.

A three-stage analysis can offer the possibility of simply entering the artificial intelligence in the first step and determining the states of the system (normal/abnormal), obtaining qualified information about errors in the second step and making predictions in the third step as to when corresponding technical systems and machines might fail.

According to the invention, a speech recognition algorithm is also present or is executed, in particular on the cloud-based analysis unit, which checks the audio signal data received from the at least one detection unit for the presence of speech components, wherein audio signal data in which speech components were recognized by the speech recognition algorithm are not stored or processed.

For data protection reasons, another machine learning algorithm can check the recorded sounds for possible speech. If speech is detected, these sounds are not saved. To ensure that people's conversations are not accidentally recorded, speech recognition can be integrated.

When training the at least one machine learning algorithm and/or at least one neural network, user inputs can additionally be taken into account, wherein the audio signal data to be classified are preferably displayed to the user in an audible manner, visually in the form of a spectrogram and/or with an indication of a preliminary analysis result automatically determined by the at least one machine learning algorithm and/or at least one neural network in order to support the decision-making process.

The predicted states can be provided to the user in a clear representation, as an audible audio file or spectrogram, preferably a melspectrogram, to support the user in making decisions. The system uses an active learning algorithm to use the user's domain knowledge and actively improve the algorithm. In addition to the melspectrograms and the audible audio data, the system can also use a machine learning algorithm which analyzes the provided noise and, in addition to being used as speech recognition, suggests to the user what the noise sounds like. This can further support the user in making decisions. In addition, it can be possible to adapt this machine learning algorithm for noise classification to the specific noises (e.g. cavitation, rattling) of the user's environment through transfer learning and sound examples.

The data features can be extracted from audio signal data or sections of audio signal data, each with a recording duration of five seconds. The audio signal data or sections can be collected and analyzed at intervals of, for example, 2 minutes.

The at least one acquisition unit or the cloud-based analysis unit can extract the data features from the audio signal data. This means that the at least one acquisition unit or the cloud-based analysis unit can each be configured to extract data features from at least a portion of the stored or received audio signal data.

The audio signal data and/or the extracted data features can be transmitted from the acquisition unit to the cloud-based analysis unit with a sensor ID and a facility or customer ID, in particular especially via Message Queuing Telemetry Transport (MQTT).

The acquisition unit can use MQTT to send, among other things, the acquired audio signal data, the extracted features, a sensor ID and customer ID. This enables the analysis platform to determine which customer the sensor or the sending acquisition unit belongs to and consequently in which database the received data must be stored. MQTT is a messaging protocol for restricted, low-bandwidth networks. Based on the sensor ID and customer ID, an algorithm on the analysis platform can determine whether this sensor already exists and which database the sensor belongs to. The creation of the necessary databases, the collection and storage of the data, the training and testing of the machine learning algorithms, the use of the latest machine learning algorithm adapted based on the user's feedback are all done automatically by the cloud-based artificial intelligence analysis platform. The user does not have to do anything else and can be notified by email as soon as an anomaly has appeared.

The system according to the invention can further comprise a visualization application which is communicatively connected to the cloud-based analysis unit and displays possible states of the at least one technical system to the user in a system-associated manner, in particular visually in the form of tables, dashboards or spectrograms and/or as audible audio signal data.

• - den Status der Überwachung im Zeitverlauf betrachten;
• - farblich codierte Anomalien im Zeitverlauf feststellen;
• - die aktiven Tonsignale, die zur Aufzeichnung der Anomalie führten, anhören und ansehen;
• - die detektierten Anomalien markieren (als Anomalie bestätigt oder es liegt keine Anomalie vor);
• - die bestätigten Anomalien bezeichnen (beispielsweise mit einem konkreten Fehler, der festgestellt wurde); und
• - die Daten für weitere Stufen verwenden.

The system according to the invention can provide a web interface or visualization application in which the result of the recording and the result of the anomaly detection can be displayed to the user. The web interface can be accessed by entering a corresponding URL and authentication. Each recording unit can have its own dashboard in which the results of the monitoring are visible. The user can use the dashboard in particular to:

• - view the status of monitoring over time;
• - identify color-coded anomalies over time;
• - listen to and view the active audio signals that led to the recording of the anomaly;
• - mark the detected anomalies (confirmed as anomaly or no anomaly);
• - identify the confirmed anomalies (for example, with a specific error that was detected); and
• - use the data for further stages.

As a result, the user of the system according to the invention can, in the first step, monitor all the machines/systems concerned with the detection unit (as a remote ear, so to speak), identify any abnormalities, mark and label any abnormalities found. Using artificial intelligence, the system can learn the usual noises of the machine/system, identify abnormal noises and mark them. Thanks to the knowledge and feedback of the users, the learning system can become increasingly more precise and then be able to clearly indicate when corresponding failures in the form of anomalies have been identified.

The user can be provided with possible states of their machine or system in a clear representation in the form of dashboards and tables, as well as audible audio files and spectrograms (especially Melspectrograms). Melspectrograms offer the great advantage that acoustic signals are processed in a way that is visually understandable for humans. This allows an additional assessment of the noise, since hearing deteriorates with age and is different for each person. By simply listening to the noise, some people may not notice anything and thus not recognize potential risks from, for example, shaft vibrations, imbalances, meshing frequencies of gears and shock pulses from gears or rolling bearings, which usually occur at higher frequencies. The states for anomaly detection (e.g. normal/abnormal) and for noise classifications (e.g. cavitation, rattling) can be efficiently entered in the interface and the system can be automatically adjusted to them by pressing a button.

Claim 7 specifies a method for continuous acoustic monitoring of at least one technical installation, in particular a machine, by means of a system according to the invention.

Advantageous embodiments and further developments of the invention are specified in the subclaims.

An embodiment of the invention is described in principle below with reference to the drawing.

• 1 Eine schematische Darstellung zur Verdeutlichung der Funktionsweise eines erfindungsgemäßen Systems;
• 2 eine schematische Darstellung zur Veranschaulichung des Sicherheitskonzepts und des Datenflusses des erfindungsgemäßen Systems;
• 3 ein Ablaufdiagramm zur Veranschaulichung eines erfindungsgemäßen Verfahrens, welches auf dem erfindungsgemäßen System abläuft;
• 4 eine schematische Darstellung einer Analyse von Sounddaten durch das erfindungsgemäße System in drei Stufen;
• 5 eine vereinfachte Darstellung eines ersten Teils einer Benutzerschnittstelle des erfindungsgemäßen Systems; und
• 6 eine vereinfachte Darstellung eines zweiten Teils der Benutzerschnittstelle des erfindungsgemäßen Systems.

It shows:

• 1 A schematic representation to illustrate the functioning of a system according to the invention;
• 2 a schematic representation to illustrate the security concept and the data flow of the system according to the invention;
• 3 a flow chart to illustrate a method according to the invention which runs on the system according to the invention;
• 4 a schematic representation of an analysis of sound data by the system according to the invention in three stages;
• 5 a simplified representation of a first part of a user interface of the system according to the invention; and
• 6 a simplified representation of a second part of the user interface of the system according to the invention.

In the figures, functionally identical elements are provided with the same reference symbols.

• - wenigstens eine Erfassungseinheit 3 mit wenigstens einer akustischen Sensoreinrichtung 3a (in 1 symbolisch als Mikrofon angedeutet), welche in der Nähe bzw. in einem Bereich bzw. in einem Raum 4 der zu überwachenden technischen Anlage 2 angeordnet oder anordenbar ist, wobei die Erfassungseinheit 3 dazu eingerichtet und insbesondere auch entsprechend ausgerichtet ist, automatisiert mittels der Sensoreinrichtung 3a erfasste akustische Signale der technischen Anlage 2 als Audiosignaldaten zu speichern und die gespeicherten Audiosignaldaten und/oder aus wenigstens einem Teil der gespeicherten Audiosignaldaten extrahierte Datenmerkmale assoziiert mit der wenigstens einen technischen Anlage 2 an eine Cloud-basierte Analyseeinheit 5 zu übermitteln,
• - die Cloud-basierte Analyseeinheit 5, welche mit der wenigstens einen Erfassungseinheit 3 kommunikationsverbunden ist (durch Kommunikationssymbole 6 angedeutet), und welche dazu eingerichtet ist, die von der wenigstens einen Erfassungseinheit 3 empfangenen anlagenassoziierten Audiosignaldaten und/oder aus wenigstens einem Teil der anlagenassoziierten Audiosignaldaten extrahierte Datenmerkmale mittels künstlicher Intelligenz, insbesondere wenigstens eines maschinellen Lernalgorithmus und/oder wenigstens eines neuronalen Netzes zu analysieren und zu verarbeiten,
• - wobei bei deren Nichtvorhandensein automatisiert eine für die anlagenassoziierte Speicherung und Verarbeitung der empfangenen Audiosignaldaten und/oder der Datenmerkmale geeignete Datenumgebung, insbesondere eine oder mehrere Datenbanken 10 (siehe 2) und/oder Dateien erzeugt werden, wobei eine Vorverarbeitung der empfangenen Audiosignaldaten und/oder der Datenmerkmale durchgeführt wird,
• - wobei mittels der empfangenen anlagenassoziierten Audiosignaldaten und/oder der Datenmerkmale, der wenigstens eine maschinelle Lernalgorithmus und/oder das wenigstens eine neuronale Netz vorzugsweise unbeaufsichtigt trainiert werden, insbesondere um einen Normalzustand der technischen Anlage 2 aufzuzeichnen bzw. in der wenigstens einen Datenbank abzuspeichern und eine Unterscheidung von abnormen Audiosignaldaten und Datenmerkmalen zu ermöglichen, und
• - wobei die Analyse von neuen bzw. neu empfangenen anlagenassoziierten Audiosignaldaten und/oder Datenmerkmalen, insbesondere in Abhängigkeit von bereits aufgezeichneten oder gespeicherten Audiosignaldaten und/oder Datenmerkmalen, mehrstufig, vorzugsweise in drei unterschiedlichen Stufen erfolgt.

1 shows a schematic representation to illustrate a system 1 according to the invention for in particular continuous acoustic monitoring of at least one technical system 2, in particular a machine, which at least comprises:

• - at least one detection unit 3 with at least one acoustic sensor device 3a (in 1 symbolically indicated as a microphone), which is arranged or can be arranged in the vicinity of or in an area or in a room 4 of the technical installation 2 to be monitored, wherein the detection unit 3 is set up and in particular also aligned accordingly to automatically store acoustic signals of the technical installation 2 detected by means of the sensor device 3a as audio signal data and to transmit the stored audio signal data and/or data features extracted from at least a part of the stored audio signal data associated with the at least one technical installation 2 to a cloud-based analysis unit 5,
• - the cloud-based analysis unit 5, which is communicatively connected to the at least one detection unit 3 (indicated by communication symbols 6), and which is set up to analyze and process the system-associated audio signal data received by the at least one detection unit 3 and/or data features extracted from at least a portion of the system-associated audio signal data by means of artificial intelligence, in particular at least one machine learning algorithm and/or at least one neural network,
• - in the absence of such a data environment, in particular one or more databases 10 (see 2 ) and/or files are generated, whereby a preprocessing of the received audio signal data and/or the data features is carried out,
• - wherein the at least one machine learning algorithm and/or the at least one neural network are preferably trained unsupervised by means of the received system-associated audio signal data and/or the data features, in particular in order to record a normal state of the technical system 2 or to store it in the at least one database and to enable a distinction to be made between abnormal audio signal data and data features, and
• - wherein the analysis of new or newly received system-associated audio signal data and/or data features, in particular depending on already recorded or stored audio signal data and/or data features, is carried out in several stages, preferably in three different stages.

A background noise from the monitored system 2 or machine is in 1 provided with the reference number 7. As can be seen from 1 As can also be seen, the system 1 comprises a visualization application 8, which is communicatively connected to the cloud-based analysis unit 5 (in 1 simplified by arrows 9) and system-associated to a user U (see 4 ) possible states of the technical installation 2, in particular visually in the form of tables, dashboards or spectrograms and/or as audible audio signal data (see also 5 and 6 ).

The system 1 according to the invention can be designed as a plug and play system. The detection unit 3 can be connected to the power grid, for example, via a power plug and placed in the room in which the technical system 2 or machine to be monitored is located and aligned accordingly. The background noise 7 of the technical system 2 should be audible at the location where the detection unit 3 is located, i.e. have a certain volume. The detection unit 3 is then connected to the Internet. A smartphone app can be used for this, which can be downloaded is loaded and the recording unit 3 can connect to the Internet. Such an app can connect to the recording unit 3 via Bluetooth and the user can enter the SSID and password of their WLAN. The recording unit 3 can then connect to the WLAN and the Internet. Alternatively, the recording unit 3 can be connected via a LAN cable. Finally, there is the option of connecting the recording unit 3 to the Internet via a USB port.

The detection unit 3 can also have an LTE module and an antenna to give the user U the possibility of using the system 1 according to the invention even if there is no possibility of connecting it directly to the Internet via WLAN or LAN. The detection unit 3 can have a single-board computer and a microphone module as sensor device 3a.

In 2 a security concept and a data flow of the system 1 according to the invention are illustrated. The detection unit 3 (which can also be referred to as a sensor box) or a client and the cloud-based analysis unit 5 or an associated server 5a as well as the other applications on the server 5a (a database 10, an MQTT broker 11 and a visualization tool 12) can each be protected with its own password and 256-bit encryption.

The cloud-based analysis unit 5 or the server 5a can have a speech recognition algorithm or an artificial intelligence-based speech recognition 13, which checks the audio signal data received from the at least one recording unit for existing speech components, whereby audio signal data in which speech components were recognized by the speech recognition algorithm are not stored or processed. A deep neural network can be used for the artificial intelligence-based speech recognition, which has been trained accordingly and thus has a high recognition rate. As soon as the audio data has been transmitted from the recording unit 3 to the MQTT broker 11, it is checked for human voices in the first step using the artificial intelligence-based speech recognition. Here, every single second of the recording can be analyzed for existing voices. If a voice is recognized, the audio data is not stored. This ensures that people's conversations are not accidentally recorded.

The visualization application 8 receives input signals from the artificial intelligence-based speech recognition (data without human voices) and the visualization tool 12. The visualization application 8 also has access to the database 10.

3 shows a further diagram to illustrate the system 1 according to the invention, wherein the process flow shown in the left part of the drawing is carried out automatically by the system 1 and the user U only has to perform the steps shown in the right part of the 3 must undertake the activities T1, T2 shown. An activity T1 to be carried out by the user U includes plugging in or establishing the power supply of the detection unit 3 of the system 1 and connecting it to the Internet. In a step S1, the sounds are recorded by the detection unit 3 audio signal data or raw data. Then, in a step S2, the detection unit 3 can extract data features from at least some or all of the audio signal data received and/or stored in step S1. In another embodiment, the extraction of the data features could also be carried out by the cloud-based analysis unit 5. The data features can be extracted from the audio signal data or raw data in sections of 5 seconds recording time each. The data can be collected and analyzed at intervals of two minutes each.

Subsequently, in a step S3, the audio signal data and/or the extracted data features can be transmitted from the acquisition unit 3 to the cloud-based analysis unit 5 together with a sensor ID and a system or customer ID, in particular via Message Queuing Telemetry Transport (MQTT).

In a step S4, the cloud-based analysis unit 5 checks whether the necessary databases and dashboards are available, and if they are not available, a data environment suitable for the system-associated storage and processing of the received audio signal data and the data features, in particular one or more databases 10 and/or files or dashboards, is automatically generated, wherein a preprocessing of the received audio signal data and/or the data features can be carried out.

In a step S5, the noises are then classified. If speech components are detected in the raw data or audio signal data, these are not stored in the database 10 for listening.

In a step S6, the extracted data features are stored in the database 10 for later Training of the machine learning algorithms.

After a period of, for example, seven days, all collected features can be automatically read out from the database 10 in a step S7 and the at least one machine learning algorithm can be trained with these data.

In a step S8, all necessary data are stored and the at least one machine learning algorithm for the multi-stage, in particular three-stage, sound analysis is trained, provided that sufficient data is available.

Finally, in a step S9, a prediction based on all available data, a visualization and a provision of audible audio data and spectrograms can be carried out, in particular on the visualization application 8 (in 3 simplified representation).

As from 3 As can be seen, the user U can provide feedback via the visualization application 8, which can be taken into account in step S8. In this way, additional user inputs can be included in the training of the at least one machine learning algorithm and/or the at least one neural network, wherein the audio signal data to be classified are preferably displayed to the user U in an audible manner, visually in the form of a spectrogram and/or with an indication of a preliminary analysis result determined automatically by the at least one machine learning algorithm or by the at least one neural network to support the decision-making process.

• - wobei von der wenigstens einen Erfassungseinheit 3 mittels der wenigstens einen akustischen Sensoreinrichtung 3a erfasste akustische Signale der technischen Anlage 2 als Audiosignaldaten gespeichert und die gespeicherten Audiosignaldaten und/oder aus wenigstens einem Teil der gespeicherten Audiosignaldaten extrahierten Datenmerkmale assoziiert mit der wenigstens einen technischen Anlage 2 an die Cloud-basierte Analyseeinheit 5 übermittelt werden,
• - wobei die von der wenigstens einen Erfassungseinheit 3 empfangenen anlagenassoziierten Audiosignaldaten und/oder aus wenigstens einem Teil der, insbesondere empfangenen anlagenassoziierten Audiosignaldaten extrahierte Datenmerkmale von der Cloud-basierten Analyseeinheit 5 mittels künstlicher Intelligenz, insbesondere wenigstens eines maschinellen Lernalgorithmus und/oder wenigsten eines neuronalen Netzes analysiert und verarbeitet werden,
• - wobei bei deren Nichtvorhandensein automatisiert eine für die anlagenassoziierte Speicherung und Verarbeitung der empfangenen Audiosignaldaten und/oder der Datenmerkmale geeignete Datenumgebung, insbesondere eine oder mehrere Datenbanken 10 und/oder Dateien erzeugt werden, wobei eine Vorverarbeitung der empfangenen Audiosignaldaten und/oder der Datenmerkmale durchgeführt wird,
• - wobei mittels der empfangenen anlagenassoziierten Audiosignaldaten und/oder der Datenmerkmale, der wenigstens eine maschinelle Lernalgorithmus oder das wenigstens eine neuronale Netz, vorzugsweise unbeaufsichtigt trainiert wird, insbesondere um einen Normalzustand der technischen Anlage 2 aufzuzeichnen und eine Unterscheidung von abnormen Audiosignaldaten und Datenmerkmalen zu ermöglichen, und
• - wobei die Analyse von neuen anlagenassoziierten Audiosignaldaten und/oder Datenmerkmalen, insbesondere in Abhängigkeit von bereits aufgezeichneten anlagenassoziierten Audiosignaldaten und/oder Datenmerkmalen, mehrstufig, vorzugsweise in drei unterschiedlichen Stufen, erfolgt.

According to the invention, a method for in particular continuous acoustic monitoring of the at least one technical installation 2, in particular machine, on the system 1 is proposed,

• - wherein acoustic signals of the technical installation 2 detected by the at least one detection unit 3 by means of the at least one acoustic sensor device 3a are stored as audio signal data and the stored audio signal data and/or data features extracted from at least a portion of the stored audio signal data are transmitted to the cloud-based analysis unit 5 associated with the at least one technical installation 2,
• - wherein the system-associated audio signal data received by the at least one detection unit 3 and/or data features extracted from at least a portion of the, in particular received, system-associated audio signal data are analyzed and processed by the cloud-based analysis unit 5 by means of artificial intelligence, in particular at least one machine learning algorithm and/or at least one neural network,
• - if they are not present, a data environment suitable for the system-associated storage and processing of the received audio signal data and/or the data features, in particular one or more databases 10 and/or files, is automatically generated, wherein a preprocessing of the received audio signal data and/or the data features is carried out,
• - wherein the at least one machine learning algorithm or the at least one neural network is trained, preferably unsupervised, by means of the received system-associated audio signal data and/or the data features, in particular in order to record a normal state of the technical system 2 and to enable a distinction to be made between abnormal audio signal data and data features, and
• - wherein the analysis of new system-associated audio signal data and/or data features, in particular depending on already recorded system-associated audio signal data and/or data features, is carried out in several stages, preferably in three different stages.

In order for the recording unit 3 to be able to automatically execute a program, corresponding scripts can be installed, which in particular include recording with the microphone and processing as well as sending the data to the cloud-based analysis unit 5.

The recording unit 3 can use MQTT to send, among other things, the audio signal data, the extracted data features, as well as a sensor ID and a customer ID. This tells the analysis platform which customer the sensor belongs to and therefore in which database 10 the data must be stored. Using the sensor ID and the customer ID, an algorithm on the analysis platform can determine whether this sensor already exists and which database 10 it belongs to.

The creation of the necessary databases 10, the collection and storage of the data, the training and testing of the machine learning algorithms, the use of the latest machine learning algorithm adapted by the feedback of the user U can be automated by the cloud-based artificial intelligence analysis. analysis platform or analysis unit 5. User U does not need to take any further steps and can be notified by email as soon as an anomaly is detected.

The system 1 can continuously listen to the MQTT broker 11. As soon as a message has arrived in the broker, it is read with an MQTT client, decoded and checked to see whether databases 10 have already been stored for the sensor. To do this, a corresponding database client and the IDs are used to check whether a table exists in the database 10. If this is not the case, the necessary databases 10 and tables are created. If a database 10 exists and the configurations for the machine learning algorithm are saved, these are loaded from the database 10, predictions are made and the data is saved together with the predictions.

A function for recording the normal state of the technical system 2 is called automatically after the databases 10 have been created. A background process is then started, which starts recording the normal state after, for example, seven days. If this function is called, all data for training the machine learning algorithms for anomaly detection can be retrieved from the database 10. The query filters the data according to the sensor ID and sorts it according to time. Training then takes place and the configurations are saved in the database 10 using another function.

The visualization application 8 or the user interface can be automatically generated on the cloud-based analysis unit 5 for each acquisition unit 3 or each sensor based on templates.

In addition, user U can immediately start monitoring the technical system 2 or machine without any knowledge of machine learning, since he does not have to configure anything on system 1 and the machine learning technology used eliminates the need for complex and time-consuming evaluation/classification of the sound data for meaningful predictions in anomaly detection. In addition, the clear user interface or visualization application (see also 5 and 6 ) a simple evaluation of the recorded sounds and feedback for the artificial intelligence can be carried out. At the same time, data protection is guaranteed by the speech recognition, as these sound files with speech content are not saved.

• - in einer ersten Stufe können Anomalien erkannt werden, insbesondere wenn eine Abweichung von dem aufgezeichneten Normalzustand der wenigstens einen technischen Anlage 2 vorliegt,
• - in einer zweiten Stufe kann ein Fehler der wenigstens einen technischen Anlage 2 klassifizierbar sein, insbesondere ein Klappern oder eine Kavitation, und/oder
• - in einer dritten Stufe kann ein Ausfall der wenigstens einen technischen Anlage 2 vorhersagbar sein.

The analysis of the new or newly received system-associated audio signal data and/or data features can be carried out in three stages:

• - in a first stage, anomalies can be detected, in particular if there is a deviation from the recorded normal state of the at least one technical installation 2,
• - in a second stage, a fault of the at least one technical installation 2 can be classified, in particular a rattling or a cavitation, and/or
• - in a third stage, a failure of at least one technical installation 2 can be predictable.

4 shows the three-stage analysis of the sound data. The user U can contribute his domain-specific knowledge through active feedback in the visualization application 8 and improve the predictions of the machine learning algorithm for anomaly detection. In addition, the audio classification can be adapted through feedback and transfer learning. This gives the user U suggestions based on the domain-specific sounds as to what the noise sounds like (e.g. cavitation, bearing damage, etc.) and can have corresponding automated messages sent to him by email as soon as the noise occurs again.

In a step S10, the cloud-based analysis unit 5 receives the audio signal data as input 14 and/or the extracted data features as input 15 from the acquisition unit 3. The raw data (or audio signal data) recorded by the microphone are used to create spectrograms for audio classification. The extracted data features are used in particular for easier anomaly detection and prediction in stage 3. In stage 1, in a step S11, audio classification is carried out using a deep neural network (in particular for speech recognition) with the audio signal data. In a step S12, unsupervised learning takes place and in a step S13, anomaly detection takes place, which also takes into account active learning in a step S14 based on feedback from the user U, which are stored as validated states 16 (anomaly, normal, error) in a database/table. In the first stage, the user U receives a message and a visualization in a step S15 on the visualization application 8. In addition, error classes 17 are stored in a database/table.

In stage 2, transfer learning for an audio classification can be carried out based on error classes 17. cation can be carried out in a step S16, which in turn is incorporated into the visualization S15. Historical data that has been recorded over a longer period of time is used as validated states in a supervised learning in a step S17 in level 3. Thus, in a step S18, a prediction can be made, for example of the type "in x days, error y will occur with a probability of z%", and a message can be issued in a step S19.

Machine learning algorithms such as "Random Forest" or "Isolation Forest" are used for anomaly detection in step S13 in the first stage. These measures allow the system to be scaled very well, as comparatively little computing power and memory is required. In order for the machine learning algorithm to recognize simple patterns in the audio data, the data is transformed into time and frequency domain features and passed to the machine learning algorithm as input.

The detection unit 3 can extract data features such as average values, maximum and minimum values, volume and/or frequency ranges from at least part of the received audio signal data. In addition, a Mel spectral analysis can be carried out using suitable libraries. The sampling rate can be 16,000 Hz and the number of Mel frequency bands 128. Each Mel frequency band is iterated through in a loop and the maximum of each frequency band is determined.

The size of the recorded audio signal data (raw data) can, for example, be reduced from 80,000 data points to a maximum of 200 data points using feature extraction at a sampling rate of 16,000 Hz for a recording of 5 seconds. This allows this data to be stored more quickly and in a more resource-efficient manner in the database 10 of the cloud-based analysis unit 5. By using data frames in which the extracted data features are stored (for example, line by line), faster predictions can be made. This is particularly necessary when monitoring a machine and/or a technical system 2 24/7 due to the large amounts of data.

On the analysis platform, a time series database can be used as database 10 to store the extracted data features of the audio signals.

Storing the extracted data features in the database 10 allows them to be subsequently used for analyses or predictive models to identify patterns or trends in the signal waveform.

The cloud-based analysis unit 5 can train an isolation forest learning algorithm for anomaly detection on the dataset from the database 10, performs predictions to detect anomalies in this dataset, and writes results to the time series database. The dataset can be randomly shuffled to ensure that the model is not influenced by the order of the data. The dataset is first scaled using a scaling method to ensure that all features are scaled to the same range of values. The dataset can be split into training and test datasets. An active learning algorithm is trained on the training data.

Furthermore, a "random forest" classifier and the "uncertainty sampling" strategy are used. The system can thus make targeted suggestions to the user U for classifying the data on the user interface of the visualization application 8. The user U can then check the determined states (normal, abnormal) on the user interface for plausibility. The machine learning algorithm is trained on the inputs/feedback of the user U and thus improves its predictions.

The analysis of sound data makes it possible to detect not only acute failures, but also gradual deterioration of machines and systems and to take timely measures. The method automates the process of data preparation and analysis by using machine learning and automated workflows. This minimizes manual work and enables fast and efficient analysis of data. A "random under-sampling" method can ensure that the model is not influenced by the dominance of a class "normal" or "anomaly", which increases the accuracy of the model. By using active learning, the model can be continuously improved by being supplied with new data by user U. This allows the model to respond better to changes in the data set and better detect anomalies in new data. The method can use the "isolation forest" method, which is particularly effective in detecting anomalies in large amounts of data. It can quickly and accurately identify anomalies in a data set. This machine learning algorithm requires few hyperparameters, making it an efficient method for anomaly detection. The method is scalable and can be applied to large data sets sets. It can also be applied to different types of data, such as audio data. The method also takes into account the temporal dependence of data by collecting and analyzing the data in a specific time window. This enables it to detect time-dependent anomalies that might go undetected in other methods.

For the speech recognition/audio classification based on artificial intelligence, a deep neural network can be used in the second stage. This can analyze a range of 0.96 seconds by default and jumps forward by 0.48 seconds at a time in order to examine the entire audio file. A prediction can be made for each time window.

First, the analysis platform checks whether a model adapted to the use case is stored. If this is not the case, the deep neural network is used by default. The 5-second audio data sections are normalized and entered as input into the model. For the inventive system 1, the top prediction for the respective analyzed area is always used and cached in a list. The noise class that appears most frequently in this list is used to classify the noise.

If the algorithm predicts an area with the class "silence", this class is not stored in the list. This ensures that short-term unsteady noises within the five-second period are taken into account.

The list is also checked to see if it contains any of the human sound classes. If it does, this five-second sound portion is not saved in the database for later listening and evaluation.

Abnormal noises are difficult to collect because they ideally do not appear. Therefore, the transfer learning method can be used in the system 1 according to the invention.

The pre-trained network already offers the advantage that the sounds can be classified well and the user thus receives reliable predictions to support the evaluation of the sound data. The pre-trained neural network of the system 1 according to the invention can also detect sounds that can indicate dangers, burglary or sabotage attempts, such as explosions, fire, broken glass, etc. The system 1 according to the invention can therefore also be used for security monitoring.

Transfer learning makes it possible to train a model even when only limited data is available. The pre-trained model can be used as a starting point to improve performance on a new dataset that contains only a limited number of examples, thus reducing the data overhead.

Using a pre-trained model saves time and resources because the model has already been trained on large datasets and its features are already mature. This significantly reduces the time required to train a model from scratch.

Transfer learning can improve the performance of a model, especially if it was trained on similar tasks or datasets. The pre-trained model can help the new model produce faster and better results.

Transfer learning can also help reduce overfitting because it forces the model to learn more general features that are relevant to multiple tasks, rather than focusing on the features that are specifically relevant to a single task.

The trained weights of the model are stored in a TFLITE file, which embodies a more compact representation of the model. This allows the model to be easily used on different devices and platforms, increasing scalability.

By storing the model and the “trigger” value in the database 10, the system 1 is able to respond quickly to new classes of errors and use a new classification model to improve future predictions.

Transfer learning also enables rapid adaptation to domain-specific noises.

In the third stage, the data already collected, especially from the time series database, is used to identify patterns in the data that have led to a certain class of error. The user receives a message in good time (in the best case a few days in advance) that, for example, the error "cavitation" will occur with 80% probability in four days with a probability of 89%. In the area of Remaining Useful Life (RUL), different machine learning methods can be used to predict time series data. The most common methods are autoregressive models, such as Recur rent Neural Networks (RNN) and Long Short-Term Memory (LSTM).

Although the models mentioned above can be very effective for predicting time series, there are also some problems that can arise when using these methods. Due to these problems, the approach is to solve the task as a simple classification task. Better results can be achieved by making the prediction as a classification task rather than a regression task (predicting an exact value, as in the methods mentioned above). Likewise, computationally intensive and complex to create deep learning algorithms can be dispensed with.

1. 1. Im ersten Schritt wird die Zeit bevor die Fehlerklasse „Broken“ aufgetaucht ist berechnet.
2. 2. Der Datensatz kann mit drei unterschiedlichen Labels gekennzeichnet werden: a) Label 0 => vier Tage bis „Broken“, b) Label 1 = zwischen zwei und vier Tagen bis „Broken“ und c) Label 2 = kleiner zwei Tage bis „Broken“.
3. 3. Der Datensatz wird durchgemischt und in Trainings- und Testdaten aufgeteilt.
4. 4. Ein maschinelles Lernen-Klassifikator wie zum Beispiel ein „Random Forest“ wird auf den Trainingsdatensatz trainiert.
5. 5. Auf dem Testdatensatz wird überprüft, ob der maschinelle Lernalgorithmus auf neue Daten gut verallgemeinern kann.

The procedure is as follows:

1. 1. In the first step, the time before the error class “Broken” appeared is calculated.
2. 2. The data set can be marked with three different labels: a) Label 0 => four days until “Broken”, b) Label 1 = between two and four days until “Broken” and c) Label 2 = less than two days until “Broken”.
3. 3. The dataset is shuffled and divided into training and test data.
4. 4. A machine learning classifier such as a random forest is trained on the training dataset.
5. 5. The test dataset is used to check whether the machine learning algorithm can generalize well to new data.

The average accuracy can be around 96% on a training dataset of a few examples for each label (label 0: 1,261 training data, label 1: 201 training data, label 2: 202 training data). Another advantage of using the random forest classifier is that it can output a list of values indicating how important each data feature is to building the model. This can be used to identify the relevant data features in a model and exclude irrelevant data features if necessary. This can help improve prediction accuracy and reduce model complexity.

The system could be beneficial for a variety of applications and users. Machine and plant operators, production facilities, maintenance and repair service providers, and security services could benefit from the automated analysis and classification system according to the invention.

The system 1 according to the invention automatically analyzes acoustic signals and classifies them as normal or abnormal without the user having to manually classify data.

The system 1 according to the invention can use unsupervised learning methods. Therefore, the user U does not need to preprocess the data in order to obtain good predictions. Through active learning, the user can quickly and easily adapt the artificial intelligence for anomaly detection to his knowledge. This significantly reduces the time required for data preparation.

System 1 features a multi-stage algorithm that combines unsupervised learning, active learning, and supervised learning to enable efficient anomaly detection.

The system can provide the user U with a clear representation of the predicted states in the form of dashboards, audible audio files and melspectrograms to assist him in making decisions. Additionally, the machine learning algorithm can suggest to the user U what the noise sounds like to help him make a decision. This makes it easy to make decisions quickly and accurately.

The system 1 may provide the ability to adapt the machine learning algorithm for sound classification to the environment-specific sounds encountered by the user U by using transfer learning and sound samples. This allows the system 1 to be adapted for different applications and use cases.

The system can create special alarms based on the detected sounds and send them via email to inform the user U of potential problems.

The system can be used in various application areas such as machine diagnostics, safety monitoring and quality assurance. This can reduce operating costs, increase efficiency and improve safety.

A faster collection and evaluation of sound data is possible. The system 1 enables the user U to quickly and easily collect and evaluate sound data, which speeds up the monitoring of the technical system 2 or machine. The three-stage structure of the system 1 in particular facilitates this process. process and makes the system accessible to laypeople.

The system 1 according to the invention represents an innovative and inventive solution for the automated analysis and classification of acoustic signals in various application areas. In particular, the use of unsupervised learning methods, active learning algorithms and transfer learning to improve the prediction accuracy without manual labeling of the data is a very innovative approach.

5 shows a first part 8a of a visualization application 8 or user interface.

As from 5 As can be seen, the desired recording unit 3 can be selected in a selection list 80. The associated data in the other components of the user interface are then automatically reloaded accordingly.

Interactive dashboards 81, 82 can be created with the visualization tool 12 and integrated into the user interface as frames.

In part 8a of the visualization application 8, the user U gets a quick overview of the status of the technical system 2 to be monitored within the last 24 hours via the dashboard 81. A “normal” status can be shown in green, a “possible anomaly” status in orange and an “anomaly” status in red. The different colors are shown in 5 indicated by different dashes. The diagram 81 is interactive and can update automatically every 5 seconds. In addition, the name of the respective state can be displayed by moving a mouse pointer (not shown) over the dashboard 81.

In the interactive dashboard 82, the states of the technical system 2 can be displayed in a time range selected using an input field 83. In the interactive dashboard 82, the time is plotted on the X-axis and a probability value for a state is plotted on the Y-axis. The height of the bars shown indicates how likely one of the states "normal", "possible anomaly" or "anomaly" is, whereby only a limited number of bars spaced far apart from one another are shown as an example and in a highly simplified manner. The coloring or hatching of the bars shown corresponds to that of the interactive dashboard 81. By clicking on the desired state in an area 82a or a legend of the dashboard 82, it is possible to display only this state or these bars on the interactive dashboard 82. This enables a quicker overview (not shown in detail).

The user U can use the playback area 84 to listen to the complete sound recordings up to a week in retrospect. To do this, the five-second audio signal data parts or audio signal data sections are combined into an audio file of a maximum of 10 minutes. The desired time period for the audio file can be visually marked in the dashboard 82 (see dashed area 82b of the dashboard 82). The audio file is automatically generated as soon as the user opens the web interface or sets the time period in the input field 83. An algorithm can automatically load all audio data available in this time period from the corresponding database 10 10 minutes backwards from the input "date to" and generate the audio file or audio format for playback.

In part 8b of the visualization application 8, the anomalies in the set time range (input field 83) can be displayed in a clear table. Each anomaly can be clearly assigned in the time range of the dashboard 82. There can be a row for each anomaly in the displayed table.

• - „Datum und Uhrzeit“, damit die Anomalie zugeordnet werden kann;
• - „Anomalie“, um eine Filterung nach dem Wahrscheinlichkeitswert zu ermöglichen;
• - „Grund“, um dem Benutzer eine erste Einschätzung zu geben, wie sich das Geräusch anhört;
• - „Audio, Anomalie (oben), normal (unten)“, hier kann sich der Benutzer Audiodateien beispielsweise mit einer Länge von fünf Sekunden anhören und hat den direkten Vergleich zwischen einer von der künstlichen Intelligenz vorhergesagten Anomalie und einem normalen Zustand;
• - „Spektrogramm Anomalie“, hier kann sich der Benutzer die Anomalie zusätzlich auch als Spektrogramm ansehen;
• - „Spektrogramm normal“, hierdurch wird ein Vergleich mit dem Spektrogramm der Anomalie ermöglicht;
• - „Feedback“, hier kann der Benutzer beispielsweise mittels eines Drop down-Menüs entscheiden, ob es wirklich eine Anomalie war oder der Zustand „normal“ vorliegt. Außerdem kann er in einem darunterliegenden Eingabefeld eine Fehlerklasse angeben, wie zum Beispiel Kavitation, Lagerschaden, etc.;
• - „Bewertet“, hier wird vermerkt, ob die Anomalie schon einmal bewertet wurde.

The table can have the following columns:

• - “Date and time” so that the anomaly can be assigned;
• - “Anomaly” to enable filtering by probability value;
• - “Reason” to give the user an initial assessment of what the noise sounds like;
• - “Audio, anomaly (top), normal (bottom)”, here the user can listen to audio files, for example, with a length of five seconds and has a direct comparison between an anomaly predicted by the artificial intelligence and a normal state;
• - “Spectrogram Anomaly”, here the user can also view the anomaly as a spectrogram;
• - “Normal spectrogram”, which allows a comparison with the anomaly spectrogram;
• - “Feedback”, here the user can decide, for example, using a drop-down menu, whether it was really an anomaly or whether the condition is “normal”. He can also enter an error in an input field below. class, such as cavitation, bearing damage, etc.;
• - “Rated”, this indicates whether the anomaly has already been rated.

In the 6 The spectrograms shown may be so-called Melspectrograms, which are only shown schematically in a very simplified manner. By clicking on the respective image, the Melspectrograms can optionally be displayed enlarged in a separate window (not shown in detail). This allows the user to examine the spectrograms more closely.

A melspectrogram is a representation of the frequency spectrum of sound or tone in relation to the human perception of loudness and pitch. It is often used to analyze audio or speech signals in speech recognition, music analysis or audio processing.

The Mel spectrogram is created by applying the Mel scale to the frequency spectrum of a signal. The Mel scale is a non-linear scale that better represents the human perception of loudness and pitch than the linear frequency scale. It is achieved by converting frequencies into Mel units, which are proportional to the logarithmic transformation of the frequency.

The resulting spectrogram is displayed as a 2D diagram, where the X-axis represents time and the Y-axis represents the frequency of the measurement. The color intensity in each area of the diagram indicates the intensity of the signal in that area (in 6 Melspectrograms are easy to visualize and understand. This makes them easy to interpret, which is a great advantage for many applications, such as audio analysis.

A pattern can often be immediately identified by briefly looking at the Melspectrograms. This allows for a faster assessment of the proposed anomaly, as the user U can jump directly to that point in time in the provided audio file and listen to the significant sounds.

In another part of the visualization application 8 (not shown), settings such as assigning a sensor name can be made. The settings can also be used to specify when an alarm is issued when an anomaly has occurred in the corresponding detection unit 3. It can also be defined how many consecutive anomalies must occur before the alarm is triggered. This has the advantage that false alarms caused by short-term noises such as the dropping of an aluminum sheet or other non-machine-specific noises can be avoided.

If the user has provided incorrect information and the anomaly detection system is making incorrect predictions, or if more data has been collected and user U wants to adjust the anomaly detection algorithm based on the data so far, this can also be easily done in the settings. For example, a function can retrieve the last 30 days of data from the database. It then trains an isolation forest algorithm and makes predictions on the dataset. The data validated by user U and the predictions of the isolation forest algorithm are displayed graphically in the user interface, for example in dashboard 82. The prediction accuracy of anomaly detection improves with more data, support from unsupervised learning (isolation forest), and the data already validated.

The system 1 can thus provide possible states of the technical system 2 or machine in a clear representation in the form of dashboards 81, 82 and tables. This facilitates the interpretation and analysis of the acoustic signals by the user U.

The states can be represented in three different colors or hatchings. This allows the user U to quickly see whether, for example, an anomaly has a low probability ("possible anomaly") that he can possibly ignore and focus on the anomalies with higher probabilities. The height of the bars in the dashboard 82 also makes it possible to quickly see how certain the machine learning algorithm is over time. The advantage of this representation is that the user can quickly see whether the machine learning algorithm is becoming less certain with more recent data. If the bars become lower, this can signal a higher uncertainty in the prediction. This allows the user to react more quickly to changes in the algorithm's prediction performance and, if necessary, intervene to improve the prediction accuracy. This can be particularly advantageous in applications such as monitoring machines or industrial plants, where the prediction accuracy can be critical to avoid downtime and damage.

System 1 can provide not only audio files, but also mel spectrograms. These offer the advantage that acoustic signals are processed in a way that is visually understandable for humans. By listening to and viewing the spectrograms, the user can make an additional assessment of the noise.

System 1 uses the Melspectrograms and the suggestions of the artificial intelligence to take into account how the noise might sound with a certain percentage probability, taking into account that hearing deteriorates with age and is different for each person. This means that potential risks from higher frequencies that are no longer perceptible for some people can still be identified.

The conditions for anomaly detection and sound classifications can be efficiently entered in the user interface. The system 1 can automatically adapt to the entered conditions by pressing a button or the like. This increases the efficiency of the system 1.

By clearly displaying the anomalies, listening to and viewing sounds and suggesting what the sound sounds like, a faster assessment/diagnosis of the anomaly by the user U is possible.

By listening to the last few days for up to 10 minutes at a time using the playback area 84, any anomalous noises found can be better understood. The ability to listen to the last 10 minutes of noise remotely provides the user with a time-saving way to monitor and diagnose anomalies, especially in equipment or machinery that is difficult to access.

Overall, the monitoring/labeling system for audio data offers an effective way to monitor acoustic signals and identify potential problems in machines or systems at an early stage. The visual processing of the acoustic signals and the consideration of age- and person-related differences make System 1 particularly user-friendly and powerful.

Overall, the system 1 according to the invention offers the following improvements.

System 1 can be used for different machines or systems without complex installations or adjustments. This saves time and costs for installation and implementation.

The system 1 can automatically record acoustic signals from the machines and equipment and extract features that are transmitted to the cloud-based analysis unit 5. This eliminates the need for manual recording and analysis. This saves time and labor and enables cost-effective monitoring of machines and equipment, which in turn leads to higher efficiency and productivity.

System 1 reduces equipment costs and its ease of use and automated analysis make it suitable for smaller production environments and enables cost-effective monitoring of machines and equipment. Its ease of use does not require any special technical knowledge. It can be used by any employee in a production environment and does not require any special training or the like.

The detection unit 3 can consist of a low-cost alternative to expensive specialised equipment, which reduces the cost of the equipment and offers the possibility of using the technology also in smaller production environments.

For data protection purposes, a machine learning algorithm can, among other things, check the sound fragments for possible speech components. If speech is detected, this fragment is not saved and processed further.

System 1 can be used in various industries such as manufacturing, automotive, food, chemical and other areas where machinery and equipment are used.

The system 1 is scalable so that it can be used for small and large production environments by simply adding additional acquisition units 3.

The transmission via MQTT technologies with the quality levels (QoS) as well as the possibility of connecting the recording unit 3 to the Internet with three different variants enables robust recording in many areas where the Internet is not so good or well developed or not available at all.

The three-stage analysis and the clear monitoring/labeling system offer the user many advantages over conventional technologies. The clear presentation of the analyzed noises and anomalies enables the Users can more easily understand what they sound like and what they look like in the spectrogram. The noise class prediction of the pre-trained deep neural network and the percentages for the probability also support the user in making decisions.

The feedback from user U allows system 1 to be continuously improved and thus make even better predictions. The labeling system makes it easier for the user to identify anomalies and classify noises, which in turn leads to faster and more effective monitoring.

Overall, the system 1 offers the user a user-friendly and effective way to analyze noise from machines and systems and to detect anomalies, even if no reference data is yet available to train the algorithms. The improved prediction accuracy and fast response time of the system 1 according to the invention help the user to quickly identify and solve potential problems, which can lead to greater operational reliability and a longer service life of the systems.

These are significant improvements over the state of the art. The system 1 according to the invention enables automated, fast, simple, reliable and cost-effective monitoring of machine and system noise, which can facilitate the work of maintenance technicians and engineers and improve the efficiency and productivity of the company.

list of reference symbols

1

system

2

technical system

3

registration unit

3a

acoustic sensor device

4

Space

5

cloud-based analysis unit

5a

server

6

communication symbols

7

background noise

8

visualization application

8a

first part of the visualization application

8b

second part of the visualization application

9

arrows

10

database

11

MQTT broker

12

visualization tool

13

artificial intelligence-based speech recognition

14, 15

inputs

16

validated states

17

error classes

80

selection list

81

Dashboard interactive

82

Dashboard interactive

82a

area with states

82b

selection area

83

input field time range

84

playback area for audio signal data

T1, T2

user's activities

S1 - S19

procedural steps

U

user

Claims (7)

System (1) for continuous acoustic monitoring of at least one technical installation (2), in particular a machine, comprising at least: - at least one detection unit (3) with at least one acoustic sensor device (3a), which is arranged in the vicinity of the technical installation (2) to be monitored, wherein the detection unit (3) is set up to automatically store acoustic signals of the technical installation (2) detected by the sensor device (3a) as audio signal data and to transmit the stored audio signal data and/or data features extracted from at least a portion of the stored audio signal data associated with the at least one technical installation (2) to a cloud-based analysis unit (5), - the cloud-based analysis unit (5), which is communicatively connected to the at least one detection unit (3), and which is set up to use artificial intelligence, namely at least one machine learning algorithm and/or at least one neural network, to process the installation-associated audio signal data received by the at least one detection unit (3) and/or data features extracted from at least a portion of the installation-associated audio signal data to analyse and process, - in the absence of which automated a data environment suitable for the system-associated storage and processing of the received audio signal data and/or the data features, in particular one or more databases (10) and/or files, are generated, wherein a preprocessing of the received audio signal data and/or the data features is carried out, - wherein the at least one machine learning algorithm and/or the at least one neural network are trained unsupervised by means of the received system-associated audio signal data and/or the data features in order to record a normal state of the technical system (2) and to enable a distinction to be made between abnormal audio signal data and data features, - wherein the analysis of new system-associated audio signal data and/or data features, in particular depending on already stored audio signal data and/or data features, takes place in three different stages, - wherein when analyzing the new system-associated audio signal data and/or data features: - in a first stage, anomalies are detected, in particular if there is a deviation from the recorded normal state of the at least one technical system (2), - in a second stage a fault of the at least one technical system (2) can be classified, in particular a rattling or a cavitation, and - in a third stage, a failure of the at least one technical system (2) can be predicted, and - wherein a speech recognition algorithm is also present which checks the audio signal data received by the at least one detection unit (3) for the presence of speech components, wherein audio signal data in which speech components were recognized by the speech recognition algorithm are not stored or processed. System (1) according to claim 1 , wherein during the training of the at least one machine learning algorithm or the at least one neural network, user inputs are additionally taken into account, wherein the audio signal data to be classified are preferably displayed to the user (U) in an audible manner, visually in the form of a spectrogram and/or with an indication of a preliminary analysis result automatically determined by the at least one machine learning algorithm or the at least one neural network in order to support the decision-making process. System (1) according to claim 1 or 2 , where the data features are extracted from audio signal data of five seconds of recording duration each. System (1) according to claim 1 , 2 or 3 , wherein the at least one detection unit (3) or the cloud-based analysis unit (5) extracts the data features from the at least part of the received and/or stored audio signal data. System (1) according to one of the Claims 1 until 4 , wherein the audio signal data and/or the extracted data features are transmitted from the acquisition unit (3) to the cloud-based analysis unit (5) with a sensor ID and a system or customer ID, in particular via Message Queuing Telemetry Transport, MQTT. System (1) according to one of the Claims 1 until 5 , which further comprises a visualization application (8) which is communicatively connected to the cloud-based analysis unit (5) and displays possible states of the at least one technical system (2) to the user in an associated manner, in particular visually in the form of tables, dashboards or spectrograms and/or as audible audio signal data. Method for continuous acoustic monitoring of at least one technical installation (2), in particular a machine, on a system (1) according to one of the Claims 1 until 6 , - wherein acoustic signals of the technical system (2) detected by the at least one detection unit (3) by means of the at least one acoustic sensor device (3a) are stored as audio signal data and the stored audio signal data and/or data features extracted from at least a portion of the stored audio signal data are transmitted to the cloud-based analysis unit (5) in association with the at least one technical system (2), - wherein the system-associated audio signal data received by the at least one detection unit (3) and/or data features extracted from at least a portion of the system-associated audio signal data are analyzed and processed by the cloud-based analysis unit (5) by means of artificial intelligence, namely at least one machine learning algorithm and/or at least one neural network, - wherein, if these are not present, a data environment suitable for the system-associated storage and processing of the received audio signal data and/or the data features, in particular one or more databases (10) and/or files, are automatically generated, wherein a preprocessing of the received audio signal data and/or the data features is carried out - wherein the at least one machine learning algorithm or the at least one neural network is trained unsupervised by means of the received system-associated audio signal data and/or the data features in order to achieve a normal state of the technical system (2) and to enable a differentiation of abnormal audio signal data and data features, and - wherein the analysis of new system-associated audio signal data and/or data features, in particular depending on already stored system-associated audio signal data and/or data features, is carried out in three different stages.

Images