WO2025026613A1 - Universal platform for converting information into audio signals - Google Patents

Abstract

The invention relates to a device for converting information into audio signals, having a plurality of inputs. At least one input is configured so as to receive information from one or more sensors, and at least one output of the device is configured so as to output an audio signal at the at least one output. An audio path of the device can be coupled in a configurable manner between the plurality of inputs and the at least one output in order to generate the audio signal at the at least one output on the basis of the information from the sensor or from the plurality of sensors.

Description

Description

title

Universal platform for converting information into audio signals

The present invention relates to devices, methods and systems for converting information into audio signals, in particular from sensor signals and/or sensor data into audio signals.

State of the art

Psychological research has shown that auditory feedback can play an important role in motor learning (Shams et al., Trends Cognit. Sci., 12(11), 411, 2008), whether in sports (Schaffertet al., Front. Psychol. 10(244), 2019) or in more general tasks such as learning gestures and other movements (Danna et al., Hum. Mov. Sci., 43, 216, 2015). Of particular interest is the way in which auditory feedback can be used to optimize performance, actions and/or movements.

The conversion of information into audio signals is generally referred to as sonification (D. Worrall, Sonification Design, Springer Nature, 2019). State-of-the-art methods for sonification, in particular of non-acoustic sensor signals and/or sensor data, into audio signals are known. In sports science, for example, research has been carried out in recent years into whether artificially amplified and/or generated acoustic feedback can improve a training process, especially for beginners. The results of the research indicate improvements in the training process when training is accompanied by suitable sonification.

The publication DE 102 97 168 T5 discloses the sonification of respiratory parameters to monitor human respiration and gas exchange during anesthesia and intensive care treatment. The publication DE 20 2006 000 757 Ul discloses a sonifier which converts a data stream into sound pressure waves which are perceived as audible music, at least by a person exercising.

The publication DE 10 2020 001 182 A1 discloses feedback to a vehicle driver in order to signal to him by sonification whether or not to reduce the speed of a vehicle.

DE 10 2021 204 036 A1 discloses the use of sonification in hearing aids to reduce the risk of falls and/or as a rehabilitation measure to support and improve a user’s self-movement.

However, the systems described in the literature and/or commercially available are designed for only a specific application or can only be used for a single project or a single sport, such as skiing (see Hasegawa et al. 2012, Realtime Sonification of the Center of Gravity for Skiing, in Proceedings of the 3 ^rd Augmented Human International Conference, Megeve, France, 2012) or rowing (see https://rowinginmotion.com [accessed July 11, 2023]).

disclosure of the invention

The invention provides an apparatus and a method for converting information into audio signals having the features of the independent claims

Preferred embodiments are the subject of the respective subclaims.

Devices and a method according to the invention have the advantage that they combine a large number of application possibilities in one device and/or in one method. Furthermore, they have the advantage that they enable different processing strategies for different tasks and/or sports and are adaptable to changing user preferences and new research findings, thus enabling flexible, application-specific and easily reconfigurable sonification of sensor data, in particular of Data from a sensor about movements of a user wearing one or more sensors directly or indirectly.

In other words, a universal platform for converting information into audio signals is provided that can be integrated into a device such as a body-worn device such as a smartwatch to support a variety of sports or other activities using different sonification strategies.

According to a first aspect, the invention relates to a device for converting information into audio signals with a plurality of inputs, wherein at least one input is configured to receive information from one or more sensors. At least one output of the device is configured to output an audio signal at the at least one output. An audio path of the device is configurable to be coupled between the plurality of inputs and the at least one output in order to generate the audio signal at the at least one output based on the information from the one or more sensors.

According to a further development, the at least one audio path has at least one submodule that can be configurably coupled between the plurality of inputs and the at least one output in order to generate the audio signal at the at least one output based on the information from the one or more sensors.

According to a development, the at least one submodule has an audio signal input and/or an audio data input; a control signal input and/or control data input; and an audio signal output and/or audio data output, wherein the audio signal input and/or audio data input and the audio signal output and/or audio data output can be configurably coupled between the plurality of inputs and the at least one output by means of signals or data that can be received via the control signal input and/or control data input. According to a further development, the at least one submodule is configurable to provide a function for generating signals or data that are output from the submodule via the audio signal and/or the audio data output, or to provide a function for mapping signals or data that are received from the submodule via the audio signal and/or the audio data input to signals or data that are output from the submodule via the audio signal and/or the audio data output.

According to a further development, the function provided is configurable by the signals or data received via the control signal input and/or control data input.

According to a further development, the at least one audio path has a plurality of identical submodules which can be configurably coupled between the plurality of inputs and the at least one output in order to generate the audio signal at the at least one output based on the information from the one or more sensors.

According to a second aspect, the invention relates to a system for converting information into audio signals. The system comprises a device for converting information into audio signals as described above and at least one inertial measuring unit as a sensor of the one or more sensors coupled to the device described above. The system further comprises a sonification library for selectively configuring the device for different applications of sonification.

According to a further development, the at least one inertial measuring unit is integrated into a device worn on the body and/or into a sports device

According to a further development, the inertial measuring unit integrated in a sports device can be wirelessly coupled to the device.

According to a third aspect, the invention relates to a method for configuring a device for converting information into audio signals as described above. The method comprises initializing an input of a plurality of of inputs for receiving information from one or more sensors. The method further comprises initializing at least one output for outputting an audio signal. The method further comprises configuring an audio path between the plurality of inputs and the at least one output to generate the audio signal at the at least one output based on the information from the one or more sensors.

Short description of the drawings

They show:

Figure 1 shows a device for converting information into audio signals with N inputs and M outputs according to an embodiment;

Figure 2 shows a device for converting information into audio signals with a plurality of audio paths with one or more submodules for processing data according to an embodiment; and

Figure 3 shows a generic structure of a submodule for processing data according to an embodiment.

In all figures, identical or functionally identical elements and devices are provided with the same reference symbols. The numbering of process steps serves the purpose of clarity and is generally not intended to imply a specific chronological order. In particular, several process steps can also be carried out simultaneously.

Description of the embodiments

Figure 1 shows a device 1000 for converting information into audio signals with N inputs and M outputs. The device 1000 may comprise four or fewer inputs. Via the N inputs, or a subset thereof, the device 1000 can receive information from sensors or other information sources. The reception can be wireless and/or wired. The reception can only be possible wirelessly. The information can include, for example, unprocessed sensor signals and/or sensor data, processed sensor signals and/or sensor data, or derived sensor signals and/or sensor data derived from unprocessed and/or processed sensor signals and/or sensor data. Each input n EN can be configured to receive specific information from a single sensor or other specific information source.

The device 1000 for converting information into audio signals may comprise at most two outputs m E M to enable either a mono or a stereo output. The frequencies of the output signals may be in the frequency range between 0 and 30 kHz, preferably in a frequency range between 20 and 20 kHz. The frequencies of input signals may be lower or higher than the frequencies of the corresponding output signals.

The device 1000 for converting information into audio signals can be configured to continuously receive information via one or more inputs. For example, the device 1000 can be configured to receive sensor signals and/or sensor data from at least one inertial measurement unit (IMU). An inertial measurement unit can be integrated, for example, in a device that is worn by a user, such as a golf club, another piece of sports equipment, or a device used to perform the sporting activity, such as a golf ball. In particular, the inertial measurement unit can be integrated in a smartwatch or a smartphone. The reception of the sensor signals and/or sensor data by the device 1000 can be wireless and/or wired.

One or more inertial measurement units can be used to derive

Position data, a path such as a trajectory, a curvature, a torsion, or other time domain data, or enable derivation of one or more of these based on the sensor signals.

The device 1000 may also be configured to receive other sensor signals and/or sensor data from other sensors, such as sensors for determining a relative position, e.g. by means of satellite technology and/or terrestrial mobile technology, inductive sensors, ultrasonic sensors, optical sensors, capacitive sensors, or magnetic sensors, etc.

The device 1000 may comprise a plurality of interconnectable submodules, as shown by way of example in Figure 2 and described in detail below with reference to Figures 2 and 3.

Figure 2 shows a device 2000 for converting information into audio signals with a plurality of audio paths, for example the audio path 2100 and the audio path 2200. The plurality of audio paths are linked to an input n E N. The input can also be linked to only one audio path.

An audio path may include one or more submodules. For example, an audio path 2100 may include at least one of submodule 2110, submodule 2120, submodule 2130, and/or submodule 2140. Submodule 2110 may be configured to generate a signal or data, such as sound data. In particular, submodule 2110 may be configured to generate white noise. White noise may be noise with a constant power density spectrum in a certain frequency range. Submodule 2120 may be a low-pass filter, a high-pass filter, or a combination thereof. Submodule 2130 may be a ring modulator. Submodules in an audio path, or a subset thereof, may be linked in parallel or in series.

An audio path 2200 may include a submodule 2210 and a summation node 2220, which is also summation node 2140 in the audio path 2100. The summation node may be connected to an output m EM of the device 2000 The device 2000 may have only two outputs or only one output.

A submodule may be configured to process one or more sensor signals and/or sensor data in different ways. In particular, a submodule may be configured to generate a sound, to filter amplitudes and/or frequencies of a signal in continuous form or in the form of discrete pulses.

The device 2000 can be configurable. In other words, the links of the inputs n E N of the device 2000 with one or more signal paths can be configurable. Alternatively or additionally, the links of the one or more submodules in a signal path can be configurable. For example, the links between the submodules, their order or the use of a specific submodule can be configurable. A user can thus specify how the inputs and/or the signal paths and/or the submodules of the respective signal paths are linked to one another. A user can also configure a function of a submodule, for example a function of how an input signal and/or input data of the submodule is processed and/or controlled in order to generate an output signal and/or output data of the submodule. Configuring the device 2000 can be possible during a runtime of the device 2000.

The configurability of the connections of the submodules and/or of their function enables the device 2000 for sonification to be used for a variety of different applications and in particular sports.

In conjunction with a system or components thereof, such as a body-worn mobile device, a wearer of the system or component may be directly sounded based on the received sensor signals and/or sensor data and the functions of the device 2000.

Sounding is also possible based on signals and/or data derived from previous processing or otherwise from sensor signals and/or sensor data. Examples of such signals and/or data are one-dimensional or multi-dimensional shape parameters such as curvature, torsion, etc. of e.g. a path, a trajectory, or other information such as position, inclination, etc. of e.g. an object.

Furthermore, sound is also based on signals and/or data about a similarity with respect to a reference, for example signals and/or data from which it can be directly determined or at least deduced how far a movement of a user or an object deviates from or is away from an ideal movement.

Alternatively or additionally, sound can also be emitted based on a determination that a user is performing a desired or undesired action or movement, or that the action or movement is correct or incorrect in a context.

Figure 3 shows a generic structure of a submodule 3120 for processing data as used in the device 2000. Without loss of generality (oBdA), the generic structure is described with respect to the submodule 2120 of Figure 2, embedded between the submodules 2110 and 2130. However, the generic structure of the submodule 3120 also corresponds to the other submodules in Figure 2, as described in detail below.

The submodules all have a common structure to facilitate their connection in an audio path (see Figure 2). Each submodule 3120 has an audio signal input and/or audio data input 3121, a control signal input and/or control data input 3122, and an audio signal output and/or audio data output 3123. The signals or data received via the control signal input and/or control data input 3122 can define in the submodule control parameters 3124 of a function 3125 which maps the signals or data received via the audio signal input and/or audio data input 3121 from the submodule 3120 to signals or data output via the audio signal output and/or audio data output 3123 from the submodule 3120.

Although each submodule of, for example, the device 2000 of Figure 2 has the same generic structure, it is not necessary that each submodule have all Elements of the generic structure are used. A submodule 2110 for generating a signal or data such as sound data can, for example, allow no signals or data to be received by the submodule 2110 and the sound data can therefore be generated by the function of the submodule itself, using the signals or data received via the control signal input and/or control data input from, for example, an inertial measuring unit. Furthermore, a submodule, for example a summation node 2140 or 2220 as shown in Figure 2, can not have or allow any controllable parameters in order to provide its function.

This allows, using generic programming techniques, a function to be designed for each submodule in each audio path of a device to specify how to convert information into audio signals. The device can therefore be used for a variety of applications and can be easily programmed by a user of the device for different applications.

The device can also be configured and/or continuously updated for a variety of applications, for example using a sonification library. A connection to a sonification library can be wireless and/or wired.

By means of an interface, a user can be enabled to develop new applications. In particular, the user can initialize the input or inputs of the device, i.e. cause the input or inputs to be used to receive information, add or remove submodules to audio paths, and set parameters to define the function of a submodule (parameterize). Furthermore, the user can add or remove audio paths to the final output of the audio signal.

Claims

claims 1. A device (1000; 2000) for converting information into audio signals, comprising: a plurality of inputs, wherein at least one input is configured to receive information from one or more sensors; at least one output, wherein the at least one output is configured to output an audio signal at the at least one output; and at least one audio path (2100, 2200), wherein the at least one audio path is configurable to be coupled between the plurality of inputs and the at least one output to generate the audio signal at the at least one output based on the information from the one or more sensors. 2. The device (1000; 2000) of claim 1, wherein the at least one audio path (2100, 2200) comprises at least one submodule (3120) configurably coupled between the plurality of inputs and the at least one output to generate the audio signal at the at least one output based on the information from the one or more sensors. 3. Device (1000; 2000) according to claim 2, wherein the at least one submodule (3120) has an audio signal and/or an audio data input (3121); a control signal input and/or control data input (3122); and an audio signal output and/or audio data output (3123), wherein the audio signal input and/or audio data input (3121) and the audio signal output and/or audio data output (3123) can be configurably coupled between the plurality of inputs and the at least one output by means of signals or data that can be received via the control signal input and/or control data input (3122). 4. The device (1000; 2000) of claim 3, wherein the at least one submodule (3120) is configurable to provide a function (3125) for generating signals or data output from the submodule (3120) via the audio signal and/or audio data output (3123), or to provide a function (3125) for mapping signals or data received from the submodule (3120) via the audio signal and/or audio data input (3121) to signals or data output from the submodule (3120) via the audio signal and/or audio data output (3123). 5. The device (1000; 2000) of claim 4, wherein the provided function (3125) is configurable by the signals or data received via the control signal input and/or control data input (3122). 6. The device (1000; 2000) of claim 1, wherein the at least one audio path (2100, 2200) comprises a plurality of identical submodules (3120) configurably coupled between the plurality of inputs and the at least one output to generate the audio signal at the at least one output based on the information from the one or more sensors. 7. A system for converting information into audio signals comprising: a device (1000; 2000) according to any one of the preceding claims; at least one inertial measurement unit as a sensor of the one or more sensors coupled to the device (1000; 2000); a sonification library for selectively configuring the Device (1000; 2000) for different applications of sonification. 8. System according to claim 7, wherein the at least one inertial measuring unit is integrated into a device worn on the body and/or into a sports device 9. System according to claim 8, wherein the inertial measuring unit integrated in a sports device can be wirelessly coupled to the device (1000; 2000). 10. A method for configuring a device (1000; 2000) according to any one of the preceding claims 1 to 6, comprising the steps: Initializing one of a plurality of inputs to receive information from one or more sensors; Initializing at least one output to output an audio signal; Configuring an audio path (2100, 2200) between the plurality of inputs and the at least one output to generate the audio signal at the at least one output based on the information from the one or more sensors.