WO2026019330A1 - Apparatus and method for a resonance modem - Google Patents

Abstract

A method for interaction between a first electronic device comprising a transducer of acoustic signals and a second electronic device, the method includes the steps of transmitting modulated information in an acoustic signal using the transducer within a known range of frequencies, in a sequence during a predetermined time window, monitoring the output signal from the sensor within said time window, assessing an impact of the acoustic signal from the sensor based on an identifiable pattern in the sensor output signal, and demodulating the information in the acoustic signal transmitted by the transducer. A system for interaction between a first electronic device and a second electronic device having a sensor having a mechanical resonance frequency with a frequency band, including a transducer of acoustic signals, a sensor, and an analysis unit.

Description

APPARATUS AND METHOD FOR A RESONANCE MODEM

INTRODUCTION

The present invention concerns a method and a system for interaction between a first electronic device comprising a transducer of acoustic signals and a second electronic device comprising a sensor having a mechanical resonance frequency with a frequency band.

BACKGROUND

In electronic devices transmitting an ultrasound output signal for a specific usecase, the frequency band of the ultrasound signal is important for signal to noise ratio (SNR) and use-case functionality and performance. In some electronic devices, the limited frequency band allocated to the ultrasound signal may interfere with other sensors operating in the same frequency band in the same or other electronic devices.

Gyroscopes in electronic devices usually operate in and around a resonance frequency. Thus, gyroscopes can experience interference from use-case specific ultrasound signals affecting their performance. Specifically, the gyroscopic sensors can display erroneous readings when the emitted ultrasound frequency or frequency band overlaps with the resonance frequency of the gyroscope. Although mitigation strategies such as damping/isolation, shielding, filtering, calibration, etc may be in use for the gyroscope, an ultrasound signal may still cause erroneous readings from the gyroscope. The impact of the ultrasound signal on the sensor data from the gyroscope depends on several factors including distance between the acoustic transmitter and the gyroscope, the amplitude of the ultrasound signal and the overlap in frequency ranges. Usually, the longer the distance between the speaker and the gyroscope is, the lower the amplitude and thereby the impact of the ultrasound signal is on the gyroscope data. Similarly, the lower the amplitude of the ultrasound signal, the lower the impact on the gyroscope data will be. The amount of overlap in the frequency range between the operating frequency of the gyroscope and the ultrasound signal will also matter. In general, the more overlap in the frequency range, the more impact on the gyroscope data. Since different use-cases may emit different ultrasound signals, the solution and specific thresholds or parameters can vary from use-case to use-case mandating use-case specific solutions.

WO2022041427A1 discloses a gyroscope resonance-based covert channel communication method. According to the method, a communication channel from a loudspeaker to a gyroscope is established; inter-axle characteristics during resonance of a gyroscope are discovered for the first time, and channel noise cancelled, and motion interference are eliminated on this basis, thus achieving a cross-protocol motion-robust covert channel communication method; and the motion robustness based on a motion sensor reuse system is achieved. The invention uses a low-cost loudspeaker and gyroscope to realize cross-protocol communication of the Internet of things, without additional peripherals and physical contact or fixed position requirements, can achieve high communication rate and accuracy, supports multi-channel/multi-user simultaneous transmission, and also satisfies convenience and practicability.

CN108632255A discloses a covert communication system based on random noise modulation. At a transmitter, host data is taken as a seed of a Gaussian noise generator, a correlation coefficient of joint normal distribution noise is modulated by a covert bit, and the joint normal distribution noise carrying the covert bit is linearly overlapped with output of a host transmitter digital modulator; and at a receiver, Gaussian noise the same as the transmitter is generated by taking the host data as the seed, then the Gaussian noise and the estimation of the joint normal distribution noise are correlated, and the covert bit can be restored through hard decision, so the covert transmission of the information is realized.

WO201 8128781 A1 discloses a network service that can detect an ultrasonic signal by an ultrasonic receptive component of a first device. The ultrasonic signal can be based on an ultrasonic output signal transmitted from an audio output component. In response to the detected ultrasonic signal, the network service can perform a network service operation.

US7480692B2 discloses a method of communicating with an electronic device. The method includes providing an electronic device having an audible sound receiving and generating sub-system including a microphone, transmitting from a source at least one acoustic signal encoded with information, receiving said at least one acoustic signal by said microphone and determining a spatial position, distance or movement of the microphone relative to the source, responsive to the received at least one signal. Therefore, this publication discloses simply the sending and receiving of acoustic signals between a speaker and a microphone.

As discussed above communication using acoustic signals is well known, but may also be vulnerable to noise and disturbances from the environment and from other devices. Under some situations there is a need for alternative ways to interact, for example communicate, or at least receive acoustic information which may supplement or improve the communication. The present invention is based on the fact that the gyros and other sensors, such as accelerometers, inertia measuring units (IMU), surface acoustic wave sensors or pressure sensors may be affected by a received acoustic or vibration signal from another device and is aimed at utilizing this for communication purposes or determining the proximity between two devices. This is achieved as defined in the accompanying claims.

SUMMARY OF THE INVENTION

The invention relates to a method for interaction between a first electronic device comprising a transducer of acoustic signals and a second electronic device comprising a sensor having a mechanical resonance frequency with a frequency band. The method comprises the step of transmitting modulated information in an acoustic signal using the transducer within a known range of frequencies, in a sequence during a predetermined time window. The method further comprises monitoring the output signal from the sensor within said time window. The method further comprises assessing an impact of the acoustic signal from the sensor based on an identifiable pattern in the sensor output signal. The method further comprises demodulating the information in the acoustic signal transmitted by the transducer.

Wherein the second device comprises an additional sensor, the demodulation may be performed based on signals from both sensors. The additional sensor may be a microphone.

The first and second devices may include a communication channel configured to transmit a signal from the first to the second device including the characteristics of the transmitted acoustic signal. The communication channel may be wireless.

The second device may be configured to transmit a signal representing the response of the sensor to the transmitted acoustic signal to the first device, the first device being configured to adapt the transmitted acoustic signal to the sensor response.

The invention further relates to a system for interaction between a first electronic device comprising a transducer of acoustic signals and a second electronic device comprising a sensor having a mechanical resonance frequency with a frequency band comprising a transducer of acoustic signals, configured to modulated information in an acoustic signal. The system further comprises a sensor having a mechanical resonance frequency with a frequency band, producing an output signal. The system further comprises an analysis unit including a processor adapted to assess an impact of the acoustic signal from the sensor based on an identifiable pattern in the sensor output signal, and further being adapted to demodulate the information in the acoustic signal transmitted by the transducer.

Wherein the second device also includes an additional sensor being affected by the received acoustic signal, the processor may use both the sensor output and the additional sensor output for demodulating the signal. The additional sensor may be a microphone. The first and second device may be connected through a communication channel, the first device being configured to transmit a set of characteristics representing the transmitted acoustic signal to the second device, the processor in the second device being configured to compare the transmitted acoustic signal with the sensor signal. The communication channel may be wireless.

The second device may be configured to transmit information representing the sensor output to the first device, the first device being configured to store the information and possibly to adapt the transmitted acoustic signal to the information from the second device.

The system wherein the first device may include a first processor and a data storage including information related to the second device, the processor in the first device being configured to adapt the transmitted acoustic signals to the characteristics of the sensor in the second device.

The system may include at least two second devices, the first device may include a first processor and a data storage including information related to known the at least two second devices, the processor in the first device being configured to adapt the transmitted acoustic signals to the characteristics of the sensor in the one of the at least two second devices.

The system may include at least two first devices, the at least two first devices may each include a first processor and a data storage including information related to the second device, the processor in the first device being configured to adapt the transmitted acoustic signals to the characteristics of the sensor in one of the at least two second devices

The present invention thus provides a possibility to communicate with a device that does not necessarily have a communication sensor, to determine a proximity between two devices without a proximity sensor, or to improve the signal transmission using sensors in addition to a microphone, providing a degree of redundancy in the signal transmission. BRIEF DESCRIPTION OF DRAWINGS

Examples of the invention are disclosed with reference to the following drawings, wherein:

Fig 1 illustrates the communication between a first and a second device according to the invention.

Fig. 2 shows the effect on the gyroscope data in an Android smartphone from an acoustic signal sweeping through the resonance frequency of the gyroscope.

Fig. 3 shows a possible pulse used by the first device to construct the acoustic signal it transmits.

Fig. 4 shows an acoustic signal consisting of a pattern of individual pulses with a timing known by the second device.

DETAILED DESCRIPTION

The present invention concerns an apparatus and method for detecting the impact of an acoustic signal transmitted by at least one transducer in a first electronic device on sensor data from a physical sensor in another second electronic device.

The transmitted acoustic signal can be constructed by the electronic device in a way that it will contain frequency components partly at the resonance frequency of at least one sensor in the other electronic device.

More specifically, the present invention relates to electronic devices with MEMS accelerometers and gyroscopes and, more particularly, to methods and apparatus for utilizing the resonance frequency of these MEMS devices to modulate information messages in acoustic signals transmitted close to the electronic device from another electronic device. An acoustic signal from a first device, such as a phone, can impact the gyroscope on a second device. When the first and second device are held relatively close to each other, the acoustic signal from one phone, the first device, can be used to transmit a signal impacting the gyroscope sensor on another phone, the second device. This enables communication between devices through acoustic signals through a separate channel other than a direct Bluetooth connection or Wi-Fi connection or Cloud-based backend-service or turning on the audio system in both devices.

Referring to the drawings Figure 1 shows an interference 17 between a transducer 13 transmitting acoustic signals 16 and a sensor 14, where the transducer 13 is positioned in a first device 18 and the sensor 14 is positioned in a second device 19. The transducer 13 of acoustic signals is configured to modulate information in an acoustic signal. The sensor has a mechanical resonance frequency with a frequency band. The graphical representation of the interference 17 encompasses interference caused by the emitted acoustic signals 16, or any vibrations that may reach the second device 19, or a coupling between the acoustic signals and the vibrations. In figure 1 both devices include processors 12a, 12b and microphones 15a, 15b, the devices 18,19 for example constituting laptops or mobile phones which can include a number of different sensors possibly being affected by the transmitted acoustic signals.

An analysis unit including a processor is adapted to assess an impact of the acoustic signal from the sensor based on an identifiable pattern in the sensor output signal. The analysis unit is further adapted to demodulate the information in the acoustic signal transmitted by the transducer. The transducer 13 is connected to an interface and processor 12a controlling the transmitted acoustic signal 16, thus having known characteristics, especially in terms of frequency bands, for example being used for communication purposes or for acoustic proximity measurements. The signal 16 from the transducer 13 may be received by a receiver or microphone 15b in the second device, but may also generate vibrations 17 propagating to the device and thus affecting the sensor 14. The second device includes a processor 12b being configured to register the signals from the sensor 14, and thus also results of the vibrations or acoustic signals affecting the sensor.

The second device may also include an additional sensor being affected by the received acoustic signal. The processor may use both the sensor output and the additional sensor output for demodulating the signal. If the second device 19 for example also includes a microphone 15b, the second processor 12b may process the signals individually or in combination, for example by evaluating if the signals from the sensor 14 correspond to the signal from the microphone 15b, thus being related to the same acoustic signals, or if the sensor 14 is affected by vibrations or movements from other sources.

The characteristics of the generated signal 16 may also be transmitted through a wired or wireless connection 20 through a separate communication channel such as Wi-Fi, which may increase the quality of the analysis in the second processor 12b even further. The result of this analysis may also be communicated to the first processor and used to improve the transmitted acoustic signal so as to improve the reception at the sensor 14 for communication purposes or, as discussed in the simultaneously filed application NO20240772 to reduce the interference at the sensor 14 caused by the acoustic signal 16. Both these processes may involve selecting time periods, frequency bands etc of the transmitted acoustic signal 16 by avoiding or choosing frequencies and time sequences that gives a coupling between the transmitted acoustic signal 16 and the resonance frequencies of the sensor 14.

The effect on the gyroscope data in an Android smartphone from an acoustic signal sweeping through the resonance frequency of the gyroscope is shown in Figure 2. The gyro-x 21 , gyro-y 22, and gyro-z 23 measurements present a sinusoidal behaviour across the whole spectrum with varying amplitude intensities. Between the areas defined by the dashed lines 24a and 24b, the gyroscope readings oscillate with their peak amplitude. Between the dashed lines 24c, 24a, and 24b, 24d the sinusoidal oscillating behaviour continues with smaller amplitudes. The sinusoidal behaviour appears to extend across the whole range, before 24c and after 24d, with even smaller amplitudes.

The interaction between the two devices may be seen as a communication, or a trigger for communication between said two devices, or for determination of a proximity between the two devices. This can be achieved through modulation of the transmitted acoustic signal and demodulation of the received acoustic signal. The demodulation may be realised through an authentication key, that is either predetermined, or received through a different channel of communication, e.g. WiFi, Bluetooth Low Energy (BLE), etc. Encryption using pseudorandom keys is another possibility, or used in connection with the predetermined or received authentication keys as an additional layer.

An exemplary modulated signal 31 is shown in Figure 3. This pulse may be used by the first device to construct the acoustic signal it transmits. The signal’s amplitude is modulated in this example.

In another exemplary modulated signal 41 , that of Figure 4, more complex modulation may be chosen for the signal. The signal 41 demonstrates a pattern of individual pulses. The timing between each pulse may become known to the second device as discussed above.

The core concept of this invention is that a first device will modulate information in an acoustic signal by controlling its frequency content (i.e. frequency, phase, amplitude) to assert an expected impact on the sensor event readings from a MEMS gyroscope in a second device. The acoustic signal is transmitted using at least one transducer 13 in the first device 18 to the second devices 19 that are nearby. If the first device knows the resonance frequency of the gyroscope sensor or another sensor 14 with a resonance frequency within the frequency range of the acoustic signal in a second device, it can create an acoustic signal in a way that when transmitted close to the second device, the resulting sensor data will contain an identifiable pattern. The pattern can be detected in the second device using signal processing algorithms and/or CNN or RNN ML models for simplicity. The second device can extract the sensor events from the MEMS gyroscope sensor and based on the expected impact demodulate the information transmitted by the first device in the acoustic signal. The information may be a coded signal from the first device, or a generic modulated protocol message sent from the first device to the second device.

The first device can use a keep-alive scheme where it sends out an acoustic signal at specific intervals agreed upon by the first and second devices. As long as the second device acknowledge the detection of the acoustic signal using a backchannel, the first device will know that the second device is still within detection distance. The acoustic signal can be varied frequently based on a schedule known by all the devices involved like codes in authenticator apps for 2- factor authentication. The backchannel may be a cloud-based service accessible to all the devices enabling communication between these devices. Using direct connections using Wi-Fi or Bluetooth Low Energy (BLE), to send information back to the first device is an alternative. Additionally, the second device can, if it includes a transducer that can send modulated information, use an acoustic channel to send information back to the first device. It can also use its vibration sensor and use haptics to vibrate in a specific pattern recognizable by the first device through capturing information using either a microphone or another transducer to pick up the coded haptic signal and thereby confirm close-range proximity.

The first device can dynamically change the amplitude of the acoustic signal and thereby the detection distance for any given second device. Since the design of the second device and the sensor placement and mounting will influence the resonance effect on the sensor events, the first device may have to get feedback about the resonance effect from the second device to be able to adjust the amplitude or characteristics of the acoustic signal. Ideally, the feedback which may include information about the detection (i.e. sensor event, raw sensor data from the second device to allow the first device to verify that the acoustic signal has be received correctly, etc) can be transmitted using a high-bandwidth back channel (e.g. Bluetooth, Wi-Fi, NFC). The acoustic signal can be constructed in different ways by relying on pulses of the resonance frequency as illustrated in the drawings. The acoustic signal could also use an up-chirp signal where the frequency of the pulse starts below the resonance frequency and ends up above the resonance frequency if the effect can be detected. Similarly, the acoustic signal may be a down-chirp where the frequency of the pulse starts above the resonance frequency and ends up below the resonance frequency if the effect can be detected. The start and stop frequency for these chirp signals should be adapted to the sensor in question to make sure that the detection rate is high.

Centered around the resonance frequency, the system can send a sine signal, maintaining it long enough for the gyroscope to detect and generate specific events. By modulating the signal, it is possible to transmit data, such as using a pulse for a bit value of 1 and silence for a bit value of 0. The detection of these signals can be influenced by whether the device is on a stable surface or handheld, impacting gyroscope readings. It may be useful to perform advanced signal processing to extract information about the abnormal sensor event readings from the gyroscope sensor.

In this system, the modulation can be based on existing modulation techniques such as frequency modulation, phase modulation, amplitude modulation, or pulse modulation or a combination of them. All these techniques are relevant for the aspects discussed here based on the acoustic signal transmitted by the first device. Depending on the detectability in the second device for individual sensors, the smallest entity that can be modulated is singular bit, multiple bits, nibble, byte or multi-byte words.

If the sensor, due to its design, has a plurality of the resonance frequencies, the acoustic signal should take advantage of the design and construct a combined signal that will include pulses based on a plurality of resonance frequencies. The detection can be based on correlation of the resonance effects at all or at least a subset of resonance frequencies. In a first example, a first device, such as a laptop transmits an acoustic signal to impact the gyroscope sensor in a second device, such as a smartphone. When the user approaches the laptop with the smartphone, the Bluetooth connection between the devices is reconnected. Once the Bluetooth connection is up, RSSI measurements allows the laptop to detect that the smartphone is in close proximity. The first device may start to send out the acoustic signal regularly or continuously once the second device is within Bluetooth range at the cost of power consumption. Alternatively, the first device may upon confirming Bluetooth connection and potentially getting information from the smartphone that the gyroscope readings are stable indicating that the phone is no longer handheld but laying on the table etc, the laptop will start transmitting the acoustic signal regularly or continuously. Once the Smartphone detects the acoustic signal correctly, close-range proximity between the laptop and smartphone is verified. By using the backchannel between the devices, the devices can transmit information with each other. If the second device is not detecting the acoustic signal, it may transmit information asking the first device to increase the amplitude for a short time. If the second device does still not detect the acoustic signal and the amplitude cannot be increased, the second device may transmit information letting the first device know that it is not in close-range proximity and that the process should be paused due to power saving reasons until either of the devices have been moved within range. Once the phone has been moved based for example IMU sensor monitoring, the second device may inform the first device that the process of detecting close-range proximity should be restarted.

If the smartphone has moved within range, the acoustic signal will suddenly impact the gyroscope, causing it to provide readings indicating close-range proximity to the second device. The smartphone can then send a message indicating the detection of the acoustic signal, allowing for various actions on either device to take place. Possible actions include transferring a call, exchanging application contexts, or other communications between the devices, screen sharing utilizing even the relative position of the devices. The first device can send out an acoustic signal of variable length. If the second device indicates that the sensor readings are inconclusive, the first device can keep transmitting a longer acoustic signal potentially with a higher amplitude until the second device has accepted the signal.

In another example, a second device, such as a smartphone, transmits information via Bluetooth, advertising that a device is looked for and that the smartphone is currently in a state where a first device, such as a laptop, can transmit an acoustic signal. The laptop, detecting the transmitted information as an advertisement and considering RSSI values of the Smartphone, will get information that the smartphone is on a flat surface and not being handheld. Upon confirmation, the laptop can modulate information in the acoustic signal allowing the Smartphone to demodulate the information by reading and analyzing the sensor events from the gyroscope sensor in the smartphone.

To implement the system described above, it is preferable to know the resonance frequency of the gyroscope. The resonance frequency of the relevant sensors in the smartphone can be detected during the manufacturing process and stored locally or in an accessible device database. Alternatively, the Smartphone app can detect the plurality of the resonance frequencies after being installed on the device. It can transmit its own acoustic signal sweeping through the relevant frequency ranges and monitor the sensor data from the plurality of sensors to detect abnormal readings. This requires the device to be in a state where the sensor readings are constant or where abnormal readings can easily be detected. What this means is sensor specific. As an example, abnormal sensor readings and the corresponding resonance frequency of a gyroscope can be detected when the smartphone is laying flat on a table or similar. Once the smartphone knows the resonance frequency of a sensor, it can advertise the information to the first device via an app-based scheme or through a backchannel like Bluetooth advertisements. The first device may include a first processor and a data storage including information related to the second device. The processor in the first device may be configured to adapt the transmitted acoustic signals to the characteristics of the sensor in the second device, which is the intended receiving device. The laptop for example can use a database of the resonance frequency range of different sensors. Combined with a database over sensors used in specific second devices, the first device can use the range of resonance frequency values for a particular sensor as starting point for any detection scheme to find the resonance frequency of a specific sensor in a specific second device. Alternatively, the laptop can sweep through frequency ranges to detect abnormal gyroscope readings, indicating the resonance frequency of the sensor in the second device.

In another example, a car can use an acoustic signal with a transducer mounted near the door handle. When the user approaches with a smartphone, the phone can pick up the acoustic signal, demodulate it using gyroscope readings, and respond with a secured Bluetooth code to unlock the car. This process ensures that the two devices are close and securely paired. The system can also work in reverse, with the smartphone sending the acoustic signal and the car detecting it through a gyroscope near the door handle.

In another example, the close-range proximity detection can be used to transfer information to allow the first and second devices to bootstrap a connection relationship between them. Since the acoustic signal can be used to modulate any information and the information can be demodulated in the second device by analyzing the sensor data readings, the first device can send information allowing the second device to bootstrap the setup of for example a Bluetooth pairing relationship or a Direct Wi-Fi connection between the two or more devices. The modulated information may be a pin code for Bluetooth pairing authentication process or a part of Wi-Fi password for a Direct Wi-Fi network.

In another example, consider a Bluetooth smart speaker that is already being controlled by a first user. A second user wants to add one of their songs to the smart speaker without taking over the Bluetooth channel. This can be achieved by having a sensitive gyroscope in the smart speaker and a feature in the music app on the second user's smartphone. When the second user approaches the smart speaker, they can press a button in their music app to transfer information about the song they want to play. The information is modulated using an acoustic signal sent from the smartphone, which is then picked up by the gyroscope in the smart speaker. The resonance frequency for this signal can be broadcasted via Bluetooth or determined through a scanning process to find the resonance frequency. This allows the transfer of information without needing a secure Bluetooth connection, relying instead on the proximity of a user to the smart speaker for security.

Another technique involves separating the acoustic signal modulating information utilizing the resonance frequency from other types of device movements by changing the frequency. By ramping up and down the frequency in the gyroscope data, the system can detect the length of the gyroscope impact. Knowledge of the sweeping frequency tempo allows for the determination of the period of the gyroscope impact, making it easier to use machine learning to distinguish this data from normal gyroscope movements.

The advantage of using a gyroscope at the resonance frequency to demodulate acoustic signals or detect specific acoustic signals is that it enables devices without audio systems to receive acoustic signals. Yet, these devices might have gyroscopes or similar sensors that are sensitive to acoustic signals at specific resonance frequencies. This allows close-range acoustic signal demodulation in devices without audio systems (e.g. stylus, game controllers, VR controllers, smart pens, motion sensing devices, 3D computer mice, etc).

Modem gyroscopes are also power-efficient compared to turning on a complete audio system. They can be connected to a low power sensor hub, allowing for sensor fusion with other types of sensors. The scheme can be used to detect the relative position of a smartphone and a laptop. This is possible if the laptop plays different acoustic signals from speakers located on different sides of the laptop. Different acoustic signals can be played concurrently or possibly alternately playing the same acoustic signal from each speaker. If a smartphone is placed on either the right or left side of the laptop, there will be a difference in the acoustic signals that the smartphone detects. Since the laptop speakers are usually placed on either side of a laptop, it allows the laptop and the smartphone to determine the relative position between the devices. If the smartphone can detect the acoustic signal from both speakers, one will be significantly louder, indicating the side on which the smartphone is placed. If both signals are equally strong, the smartphone may be located between or at an equal distance from both speakers. Spatial information about the relative position between the devices can be utilized to enhance the user experience on either or both devices. For instance, if a call or a running application is being transferred from the smartphone to the laptop, the visual transfer effect can enter the laptop screen from the left side if the smartphone is detected on the left.

In another example, the laptop can improve a detection rate by using a plurality of sensors. If the second device has more than one sensor having a mechanical resonance frequency with a frequency band, the first device could mix different acoustic signals before transmitting it out on the speaker, provoking abnormal readings for a plurality of sensors (e.g. gyroscope, accelerometer). Synchronized data from both sensors can be correlated to verify close-range proximity to the transmitting first device.

If the acoustic signals are coded or contain modulated information from the first device, it can be determined that the acoustic signal is from a specific first device, which is in a relationship with the second device, confirming close-range proximity between the devices. In another example, the first device includes a first processor and a data storage including information related to the second device. The processor in the first device is configured to adapt the transmitted acoustic signals to the characteristics of the sensor in the second device. The system may include at least two second devices, or a second device with two sensors having a mechanical resonance frequency with a frequency band. The processor can then be configured to adapt the transmitted acoustic signals to the characteristics of the sensor in one of the at least two second devices, or in one of the two sensors on a second device.

In another example, the first device can send out a plurality of acoustic signals, each one suitable for a specific set of sensors or a specific sensor in a second device. This makes it possible for multiple second devices to receive close-range proximity validation concurrently. Thus, one first device can send out several acoustic signals, each targeted at a specific second device, enabling close-range proximity detection for multiple second devices at the same time.

In another example, a plurality of first devices may be in close-range proximity of the same second device allowing them to optionally cooperate with the second device as a device group. Thus, with two co-located first devices, a singular second device may need to detect proximity from both first devices before performing any action. Each first device will send out its own acoustic signal preferably in sequence. The second device will wait until it detects close-range proximity from a plurality of first devices before signaling proximity to all the first devices simultaneously.

In another example, the first device can send out an acoustic signal detected by the second device once it gets close enough. The second device can then either send an acoustic reply to the first device or use a backchannel. When the second device starts to move away, such as when a smartphone is lifted from a table next to the laptop, it can use back channels like Bluetooth, Wi-Fi, or 5G to notify the first device that it is moving outside the proximity range. Acoustic signals and/or haptics can also be used to notify the first device that the second device is being moved. The second device itself can use IMU sensor data to detect the device movement. The first device can send out another acoustic signal to verify that the second device has not moved outside the close-range proximity detection zone.

If the second device includes multiple sensors, the first device can correlate abnormal readings from a plurality of sensors to determine if an abnormal event was created by the acoustic signals, thus avoiding false positives. This increases the robustness of the system, especially if multiple acoustic signals trigger abnormal readings from various sensors simultaneously, correlated with high- accuracy timestamps.

A benefit of using resonance frequency in this scheme is that the sampling rate does not need to be extremely high, positively impacting processing cycles, MIPS numbers, and power consumption. While microphones are typically sampled at high rates to function effectively, using resonance frequency allows for lower sampling rates, which can mitigate privacy concerns and other issues related to second devices.

One of the challenges addressed in this document is the variability in gyroscope and accelerometer sensors across different phones and second devices due to manufacturing differences. This means the resonance frequency is not known beforehand. Consequently, the first and second devices may need to determine the resonance frequency together during an initial relationship set up phase. This may for example be performed by monitoring the sensor output signal during a controlled transmitted acoustic signal, for example including a frequency sweep, as discussed in simultaneously filed NO20240772. One proposed solution is for the first device sweep through the supported frequency range in a controlled manner, playing each frequency for a period. The characteristics of the controlled signal may prestored or be transmitted to the second device through Wi-Fi communication or acoustic communication in order to compare the received signal with the transmitted acoustic signal. When the second device detects an abnormal gyroscope reading, it should immediately send a message to the first device. This message could be sent via a back channel, an acoustic message, a haptic message, Wi-Fi or another communication method to indicate the detection of an abnormal sensory event. The duplex communication between the first device and the second device can even be forwarded by a shared cloud service that knows the network address of both devices and can communicate with both entities. Once the first device gets the information that the resonance frequency was detected for a particular sensor, the first device can restart the frequency sweep with a smaller frequency range to get more accurate detection of the resonance frequency. This process can continue iteratively with shorter and shorter frequency ranges around the resonance frequency until the accuracy is acceptable to both device and are within configurable thresholds.

If the second device has multiple sensors with different resonance frequencies, each sensor that detects an abnormal reading should send a message indicating which sensor detected the abnormal reading. After receiving this information, the first device should record and store it. This allows the first device to iterate and fine-tune the frequency range based on previous feedback to accurately determine the resonance frequency of each sensor.

Once the resonance frequencies are identified, and if the second device remains close to the first device long enough for the process to complete, the first device can store this information. This data can be indexed using the Bluetooth MAC address or another unique identifier for that particular second device. The unique identifier could be the Bluetooth pairing relationship or another suitable identifier known to both devices. When the second device moves out of range and later returns within range, it can be re-identified through its unique ID. The stored resonance frequencies can then be used to detect proximity, facilitating the identification of the device.

Another possible approach is to perform similar testing during manufacturing and record the resonance frequencies of the plurality of sensors. Alternatively, upon installing the application on the smartphone, a calibration or resonance frequency detection phase can be conducted. The phone itself would send out an acoustic signal and determine the resonance frequency by performing the same process as the first device described above, but now conducted by the second device itself. It can playout the acoustic signal and read the sensor data synchronously to detect at which frequency the frequency sweeping acoustic signal was at when the sensor event readings were abnormal given the current orientation and movement of the smartphone.

In another example, the first device can be a tablet, or a smartphone and the second device can be a stylus with a gyroscope sensor inside. When the second device is put next to the first device or in a designated compartment inside or a holder next to the first device, the acoustic signal can be sent by the first device and detected by the second device.

In another example, the first device can be a laptop and the second device can be a tablet or a smartphone both of which have gyroscope sensors inside. When the second device is put next to the first device or in a designated holder next to the first device, the acoustic signal can be sent by the first device and detected by the second device. Once the acoustic signal has been detected and the close-range proximity has been confirmed, the devices can proceed with whatever actions should be triggered automatically in this scenario or potential action that needs instant visual feedback from users.

The above describe a method for interaction between a first electronic device comprising a transducer of acoustic signals and a second electronic device comprising a sensor having a mechanical resonance frequency with a frequency band comprises the step of transmitting modulated information in an acoustic signal using the transducer within a known range of frequencies, in a sequence during a predetermined time window. The method further comprises monitoring the output signal from the sensor within said time window. The method further comprises assessing an impact of the acoustic signal from the sensor based on an identifiable pattern in the sensor output signal. The method further comprises demodulating the information in the acoustic signal transmitted by the transducer.

In this method, the second device may also comprise an additional sensor, such as a microphone, so that the demodulation may be performed based on signals from both sensors.

The first and second devices may include a communication channel configured to transmit a signal from the first to the second device including the characteristics of the transmitted acoustic signal. The communication channel can be wireless.

Wherein the second device is configured to transmit a signal representing the response of the sensor to the transmitted acoustic signal to the first device, the first device may be configured to adapt the transmitted acoustic signal to the sensor response.

Having described example embodiments of the invention it will be apparent to those skilled in the art that other embodiments incorporating the concepts may be used. These and other non-limiting examples illustrated above are intended by way of example only and the actual scope of the invention is to be determined from the following claims.

Claims

P A T E N T C L A I M S

1 . A method for interaction between a first electronic device comprising a transducer of acoustic signals and a second electronic device comprising a sensor having a mechanical resonance frequency with a frequency band, the method comprising the following steps: transmitting modulated information in an acoustic signal using the transducer within a known range of frequencies, in a sequence during a predetermined time window, monitoring the output signal from the sensor within said time window assessing an impact of the acoustic signal from the sensor based on an identifiable pattern in the sensor output signal, and demodulating the information in the acoustic signal transmitted by the transducer.

2. Method according to claim 1 , wherein the second device also comprises an additional sensor, the demodulation being performed based on signals from both sensors.

3. Method according to claim 2, wherein the additional sensor is a microphone.

4. Method according to claim 1 , wherein the first and second devices include a communication channel configured to transmit a signal from the first to the second device including the characteristics of the transmitted acoustic signal.

5. Method according to claim 4, wherein the communication channel is wireless.

6. Method according to claim 4 or claim 5, wherein the second device is configured to transmit a signal representing the response of the sensor to the transmitted acoustic signal to the first device, the first device being configured to adapt the transmitted acoustic signal to the sensor response.

7. A system for interaction between a first electronic device comprising a transducer of acoustic signals and a second electronic device comprising a sensor having a mechanical resonance frequency with a frequency band, comprising: a transducer of acoustic signals, configured to modulate information in an acoustic signal; and a sensor having a mechanical resonance frequency with a frequency band, producing an output signal, an analysis unit including a processor adapted to assess an impact of the acoustic signal from the sensor based on an identifiable pattern in the sensor output signal, and further being adapted to demodulate the information in the acoustic signal transmitted by the transducer.

8. System according to claim 7, wherein the second device also includes an additional sensor being affected by the received acoustic signal, the processor using both the sensor output and the additional sensor output for demodulating the signal.

9. System according to claim 8, wherein the additional sensor is a microphone.

10. System according to claim 7, wherein the first and second device are connected through a communication channel, the first device being configured to transmit a set of characteristics representing the transmitted acoustic signal to the second device, the processor in the second device being configured to compare the transmitted acoustic signal with the sensor signal.

11 . System according to claim 10, wherein the communication channel is wireless.

12. System according to claim 10 or claim 11 , wherein the second device is configured to transmit information representing the sensor output to the first device, the first device being configured to store the information and to adapt the transmitted acoustic signal to the information from the second device.

13. System according to any one of the preceding claims, wherein the first device includes a first processor and a data storage including information related to the second device, the processor in the first device being configured to adapt the transmitted acoustic signals to the characteristics of the sensor in the second device.

14. System according to any one of the preceding claims, wherein the system includes at least two second devices, the first device includes a first processor and a data storage including information related to the at least two second devices, the processor in the first device being configured to adapt the transmitted acoustic signals to the characteristics of the sensor in one of the at least two second devices.