CN120066313A

CN120066313A - Ultrasonic touch and force input detection

Info

Publication number: CN120066313A
Application number: CN202510125913.4A
Authority: CN
Inventors: 西纳·阿克巴里; 唐浩晏; 莫·马格苏德尼亚; 陈曼嘉
Original assignee: Aosi Semiconductor Technology Co ltd
Current assignee: Aosi Semiconductor Technology Co ltd
Priority date: 2018-05-21
Filing date: 2019-05-21
Publication date: 2025-05-30
Also published as: WO2019226680A1; CN112154404A; CN112154404B

Abstract

A system includes an ultrasound input device coupled to a layer of material having an outer surface and one or more data processors. The ultrasound input device receives a set of reflected ultrasound signals associated with the transmitted signals, the set of reflected ultrasound signals associated with touch events between the object and the outer surface of the material layer. The one or more data processors are configured to determine energy signals comprising energy measurements over time, each energy measurement corresponding to a summed value obtained from a portion of the set of reflected ultrasound signals, extract feature information associated with the energy signals, determine inferences associated with the object based on the extracted feature information, and generate an output signal associated with the determined inferences.

Description

Ultrasonic touch and force input detection

The application is a divisional application of Chinese patent application with the application number 201980034705.6 and the application name of ultrasonic touch and force input detection, which is filed on the 5 th month 21 of 2019.

Cross Reference to Related Applications

The present application is non-transitory and requires U.S. provisional application serial No.62/674,317 entitled "ultrasonic touch and force input detection (ULTRASONIC TOUCH AND FORCE INPUT DETECTION)", U.S. provisional application serial No.62/725,697 entitled "ultrasonic touch and force input detection (ULTRASONIC TOUCH AND FORCE INPUT DETECTION)", U.S. provisional application serial No.62/751,053 entitled "ultrasonic touch feature extraction (ULTRASONIC TOUCH FEATURE EXTRACTION)", U.S. provisional application serial No. 62/784,615 entitled "ultrasonic touch sensor and system (ULTRASONIC TOUCH SENSOR AND SYSTEM)", U.S. provisional application serial No. 62/786 entitled "ultrasonic touch sensor and decision (ULTRASONIC TOUCH DETECTION AND DECISION)", U.S. provisional application serial No. 16/396,597 entitled "ultrasonic touch and force input detection (ULTRA SONIC TOUCH AND FORCE INPUT DETECTION)", and U.S. patent application serial No. 16/417,184 entitled "ultrasonic touch feature extraction (ULTRASONIC TOUCH FEATURE EXTRACTION)", which were filed on 10 months and 20 months 2019, 12, 26, and is entitled "ultrasonic touch feature extraction", which are all incorporated herein by reference.

Background

Capacitive, resistive, and inductive sensing are used in industrial, automotive, medical, and consumer applications to detect touch inputs. In Human Interface Devices (HIDs) such as trackpads and touch screens, capacitive technology is used to detect that touch input has grown rapidly. Consumer and industrial applications began to employ touch buttons and sliders using capacitive technology in devices such as mobile phones, TV controls, car dashboards, remote controls, or industrial controls. Capacitive sensing has proven to be more attractive than mechanical switches and rotary encoders in terms of appearance and reliability.

However, the use of capacitive, resistive, or inductive sensing limits the inventive industrial design due to challenges in touch input layout and system stacking. The conflicting priorities between design and robustness further complicate the design. It should also be noted that current input touch sensing methods cannot be implemented on metal surfaces. Furthermore, current sensing technologies have inherent properties that limit waterproof applications. Pressure sensing techniques using strain gauges have emerged as an alternative sensing technique for metal surface touch input. However, the measurement of deflection and strain is often unreliable, especially in metals. Such sensors are very susceptible to unwanted disturbances that cause surface deflections, and their sensitivity and performance are very dependent on the overall boundary conditions of the surface to which they are attached. Furthermore, the surface to which the sensor is attached must be sufficiently conformal (compliant) to deflect it sufficiently when touched by a person so that the sensor can detect it. An additional sensing layer (e.g., capacitive) is required to detect the x-y position of an input touch detected using a strain gauge. The increased complexity of touch input interface materials, the meaning of complex interfaces in industrial designs, waterproofing, and cost have become key challenges limiting the use of touch inputs in any environment and in any material. There is a need for improved systems and methods of detecting touch input to a human-machine interface (HMI).

Embodiments of the present invention address these and other problems, individually and collectively.

Disclosure of Invention

Some embodiments of the present disclosure relate to systems, methods, and devices related to ultrasonic touch and force input detection.

According to some embodiments, a method is provided. A transducer coupled to a first surface of a layer of material having a distance between the first surface and the second surface may emit an ultrasonic signal directed at the second surface. The transducer may then detect the reflected ultrasonic signal and then determine the amplitude of the reflected ultrasonic signal. The transducer may then determine that the amplitude exceeds a threshold associated with a portion of the ultrasonic signal penetrating the second surface. In the event that the amplitude exceeds a threshold, the transducer may generate a signal indicative of a touch input on the second surface.

According to other embodiments, a method is provided. A transducer coupled to a first surface of a layer of material having a distance between the first surface and the second surface may emit an ultrasonic signal directed at the second surface. The transducer may then detect the reflected ultrasonic signal. The method then includes determining an energy value associated with the reflected ultrasound signal. The method may further include determining that the energy value exceeds a threshold associated with penetration of a portion of the second surface by the ultrasonic signal. In the event that the energy value exceeds the threshold, the method may include generating a signal indicative of a touch input on the second surface.

Some embodiments of the present disclosure relate to systems, methods, and devices related to ultrasonic touch feature extraction. The system may include an ultrasound input device and one or more data processors.

The ultrasound input device may be coupled to a layer of material that may have an outer surface. The outer surface may be located opposite the ultrasound input device from the layer of material. An ultrasonic input device may be coupled to the layer of material to transmit a transmitted signal through the layer of material toward the outer surface and to receive a set of reflected ultrasonic signals associated with the transmitted signal. The set of reflected ultrasonic signals may include at least one reflected ultrasonic signal and may be associated with a touch event between the object and an outer surface of the material layer. Touch events may include, for example, an individual touching the exterior surface with their finger or other object (e.g., a stylus, etc.).

The one or more data processors may be configured to determine an energy signal associated with the set of reflected ultrasound signals and extract characteristic information associated with the energy signal. The one or more data processors may be further configured to determine an inference associated with the object based on the extracted feature information, and then generate an output signal associated with the determined inference.

According to some embodiments, a computer-implemented method is provided. The transmitted signal may be transmitted using an ultrasonic input device coupled to a layer of material having an outer surface. A set of reflected ultrasound signals may be received. The set of reflected ultrasound signals may be associated with the transmitted signal and may include at least one reflected ultrasound signal. The set of reflected ultrasonic signals may be associated with a touch event between the object and an outer surface of the material layer. An energy signal associated with the set of reflected ultrasound signals may be determined. The characteristic information associated with the energy signal may then be extracted. Subsequently, an inference can be determined. The inference can be associated with the object based on the extracted feature information. An output signal associated with the determined inference can then be generated.

Some embodiments of the present disclosure relate to systems, methods, and devices related to ultrasonic touch sensors and systems. The touch sensor may include an ultrasonic sensor layer and an integrated circuit layer.

The ultrasonic sensor layer may comprise an array of ultrasonic transducers. The array of ultrasound transducers may include one or more ultrasound transducers. The integrated circuit layer may be coupled to the ultrasonic transducer layer. The integrated circuit layer may include circuitry configured to drive an array of ultrasound transducers to generate ultrasound signals. The integrated circuit layer may also include circuitry configured to receive the reflected ultrasound signals using the array of ultrasound transducers and to generate energy signals associated with the received reflected ultrasound signals.

According to some embodiments, a method is provided that may be performed by a touch sensor or other suitable device. The method includes generating a drive signal in an integrated circuit coupled to a transmitting ultrasound transducer located in an array of ultrasound transducers coupled to the integrated circuit. The transmitting ultrasound transducer may be a piezoelectric micromachined ultrasound transducer. The transmitted ultrasound signal may then be generated by the transmitting ultrasound transducer in response to the drive signal. Generating the transmitted ultrasonic signal may include transmitting the transmitted ultrasonic signal through the layer of material along a longitudinal direction orthogonal to the outer surface of the layer of material, or in some embodiments within 20% of a normal to the outer surface of the layer of material. Subsequently, a set of reflected signals may be received at the receiving ultrasound transducer in an array of ultrasound transducers. The set of reflected signals may include one or more ultrasound signals associated with the transmitted ultrasound signals. The receiving ultrasonic transducer may be a piezoelectric micromachined ultrasonic transducer. Subsequently, the energy signal may be measured. The energy signal may be associated with a received set of reflected signals. The method may also include determining that a touch event has occurred at an outer surface of the material layer based on the measured energy signal.

Some embodiments of the present disclosure relate to systems, methods, and devices related to ultrasonic touch detection and decision making.

According to some embodiments, a method is provided that may be performed by a touch sensor device or other suitable device. The method includes receiving energy data associated with an ultrasonic input device coupled to a layer of material. The energy data may include current energy values and past energy values associated with reflected ultrasonic signals received at the ultrasonic input device in response to the ultrasonic input device transmitting signals through the layer of material toward an outer surface of the layer of material. The energy data may then be compared to threshold data to generate a current trigger value for triggering the data. In the event that the current energy value exceeds the current threshold of the threshold data, the trigger data may indicate the occurrence of a touch event. The threshold data may then be updated based on the energy data, the trigger data, and the threshold data. Updating the threshold data may include generating a subsequent threshold.

According to some embodiments, another method is provided that may be performed by a touch sensor device or other suitable device. The method includes receiving energy data associated with an ultrasonic input device coupled to a layer of material. The energy data may include current energy values and past energy values associated with reflected ultrasonic signals received at the ultrasonic input device in response to the ultrasonic input device transmitting signals through the layer of material toward an outer surface of the layer of material. The energy data may then be provided to a recurrent neural network to generate output data indicative of a touch event occurring at the outer surface of the material layer.

These and other embodiments of the invention are described in detail below. For example, other embodiments relate to systems, apparatuses, and computer-readable media associated with the methods described herein.

A better understanding of the nature and advantages of embodiments of the present invention may be gained with reference to the following detailed description and the accompanying drawings.

Drawings

Fig. 1 is a schematic diagram depicting the effect of touch force on reflected ultrasound signals in an ultrasound input system, in accordance with certain aspects of the present disclosure.

Fig. 2 is a schematic diagram depicting an ultrasound input system in a non-contact state and a contact state, in accordance with certain aspects of the present disclosure.

Fig. 3 is a schematic diagram depicting an ultrasound input device, in accordance with certain aspects of the present disclosure.

Fig. 4 is a cross-sectional view of two piezoelectric micromachined ultrasonic transducers bonded to a CMOS wafer in accordance with certain aspects of the present disclosure.

Fig. 5 is a set of schematic diagrams depicting an ultrasound input device coupled to various surfaces, in accordance with certain aspects of the present disclosure.

Fig. 6 is a schematic side view depicting an ultrasound input system with a common plate assembly, in accordance with certain aspects of the present disclosure.

Fig. 7 is a schematic diagram depicting an example ultrasound input system, in accordance with certain aspects of the present disclosure.

Fig. 8 is a schematic side view depicting an integrated ultrasound input device with an ultrasound sensor and an ASIC, in accordance with certain aspects of the present disclosure.

Fig. 9 is a schematic diagram of a set of combinations depicting two processing routes for generating an integrated ultrasound input device wafer in accordance with certain aspects of the present disclosure.

Fig. 10 is a set of schematic diagrams depicting a single integrated ultrasonic input device cut from a wafer, PCB mounted, and stacked mounted, in accordance with certain aspects of the present disclosure.

Fig. 11 is a schematic cross-sectional view of a consumer electronic product incorporating an integrated ultrasound input device in accordance with certain aspects of the present disclosure.

Fig. 12 is a set of schematic cross-sectional views comparing a non-integrated ultrasound input device with an integrated ultrasound input device, in accordance with certain aspects of the present disclosure.

Fig. 13A is a top view of a sensor array of an ultrasound input device according to certain aspects of the present disclosure.

Fig. 13B is a top view of an alternative sensor array of an ultrasound input device according to certain aspects of the present disclosure.

Fig. 14A is a schematic diagram of the sensor array of fig. 13A depicting the functionality of the individual transducers of the array, in accordance with certain aspects of the present disclosure. Fig. 14B is a schematic diagram of a sensor array depicting eight example configurations of individual transducers of a corner region of the sensor array, in accordance with certain aspects of the present disclosure. Fig. 14C is a schematic diagram of a sensor array depicting eight example configurations of individual transducers of the sensor array, in accordance with certain aspects of the present disclosure. Fig. 14D is a schematic diagram of a sensor array depicting eight example configurations of individual transducers of the sensor array, in accordance with certain aspects of the present disclosure. Fig. 14E is a schematic diagram of a sensor array depicting two example configurations of individual transducers of the sensor array, in accordance with certain aspects of the present disclosure. Fig. 14F is a schematic diagram of a sensor array depicting example configurations of individual transducers of sensor arrays of various sizes, in accordance with certain aspects of the present disclosure. Fig. 14G is a schematic diagram of a sensor array depicting twelve example configurations of individual transducers of the sensor array of size 8x8 transducers, in accordance with certain aspects of the present disclosure.

Fig. 15 is a set of graphs depicting energy measurements of transducers of a single sensor array operating at different frequencies, in accordance with certain aspects of the present disclosure.

Fig. 16 is a graph depicting a temperature behavior of an ultrasonic transducer with respect to operating frequency (behavior), in accordance with certain aspects of the present disclosure.

Fig. 17 is a graph depicting a frequency response with respect to a stack makeup (stack makeup) in accordance with certain aspects of the present disclosure.

Fig. 18 is a schematic diagram depicting a circuit for receiving and transmitting signals by an ultrasound transducer, wherein the circuit is in a transmitting state, in accordance with certain aspects of the present disclosure.

Fig. 19 is a schematic diagram depicting a circuit for receiving and transmitting signals by an ultrasound transducer, wherein the circuit is in a receiving state, in accordance with certain aspects of the present disclosure.

Fig. 20 is a schematic diagram depicting an isolation circuit for receiving and transmitting signals by an ultrasonic transducer, in accordance with certain aspects of the present disclosure.

Fig. 21 is a set of schematic side views depicting beamforming achieved through the use of an ultrasound transducer, in accordance with certain aspects of the present disclosure.

Fig. 22 is a set of graphs depicting the operational modes of a micromechanical ultrasound transducer according to certain aspects of the present disclosure as compared to a standard body transducer, depicted as an average displacement for different frequencies.

Fig. 23 is a set of schematic side views depicting lateral signal rejection (signal rejection) of a micromechanical ultrasound transducer in accordance with certain aspects of the present disclosure as compared to a standard body transducer.

Fig. 24 is a set of schematic side views depicting lateral signal rejection of a micromechanical ultrasound transducer, in accordance with certain aspects of the present disclosure.

Fig. 25 is a schematic flow chart for digitally processing ultrasonic signals transmitted and received by an ultrasonic input device according to certain aspects of the present disclosure.

Fig. 26 is a schematic flow diagram for processing ultrasonic signals transmitted and received by an ultrasonic input device using energy integration, according to certain aspects of the present disclosure.

Fig. 27 is a schematic example flowchart for processing ultrasonic signals transmitted and received by an ultrasonic input device using energy integration, according to certain aspects of the present disclosure.

Fig. 28 is a schematic flow chart for processing ultrasonic signals transmitted and received by an ultrasonic input device using energy integration by absolute value summation, according to certain aspects of the present disclosure.

Fig. 29 is a schematic flow chart of processing ultrasonic signals transmitted and received by an ultrasonic input device using energy integration through self-mixing and integration in accordance with certain aspects of the present disclosure.

Fig. 30 is a schematic circuit diagram depicting an analog integrator with a negative bias current circuit in accordance with certain aspects of the present disclosure.

FIG. 31 is a schematic flow chart for processing ultrasonic signals depicting the reduced impact of reflected ultrasonic signal time-of-flight variations on touch input detection within an energy measurement window, in accordance with certain aspects of the present disclosure.

FIG. 32 is a schematic diagram of a simplified flow for processing ultrasonic signals depicting the enhanced effect of reflected ultrasonic signal time-of-flight variations on touch input detection outside an energy measurement window, in accordance with certain aspects of the present disclosure.

FIG. 33 is a schematic diagram of a flow for processing ultrasonic signals depicting ultrasonic signals of minimal impact of reflected ultrasonic signal time-of-flight variations on touch input detection outside an energy measurement window with window shaping (window shaping) in accordance with certain aspects of the present disclosure.

Fig. 34 is a schematic circuit diagram depicting a window shaping circuit in accordance with certain aspects of the present disclosure.

Fig. 35 is a schematic diagram depicting a flow for processing an ultrasonic signal to detect touch input using an amplitude of a reflected ultrasonic signal, in accordance with certain aspects of the present disclosure.

Fig. 36 is a graph depicting a simplified example energy signal in accordance with certain aspects of the present disclosure.

FIG. 37 is a chart depicting reflected ultrasonic signal measurements made using an ultrasonic input device and showing techniques for improving touch input detection, in accordance with certain aspects of the present disclosure.

FIG. 38 is a chart depicting reflected ultrasound signal measurements made using an ultrasound input device and showing additional techniques for improving touch input detection, according to certain aspects of the present disclosure.

Fig. 39 is a set of graphs depicting temperature dependence of reflected ultrasound signals, in accordance with certain aspects of the present disclosure.

FIG. 40 is a set of graphs depicting time-of-flight temperature dependence of two frequency methods of detecting touch input, in accordance with certain aspects of the present disclosure.

FIG. 41 is a chart depicting reflected ultrasonic signal measurements across several frequencies using an ultrasonic input device and showing techniques for improving touch input detection, in accordance with certain aspects of the present disclosure.

Fig. 42 is a schematic plan view depicting a dual frequency PMUT with concentric circle design, in accordance with certain aspects of the present disclosure.

Fig. 43 is a schematic plan view depicting a multi-frequency ultrasound input device having a square design, in accordance with certain aspects of the present disclosure.

FIG. 44 is a set of three charts depicting example signals received by an ultrasound input system that are attributable to three different users, in accordance with certain aspects of the present disclosure.

FIG. 45 is a set of graphs depicting energy measurement signals associated with a human finger, a water droplet, and placing a device on a table (e.g., placing an object on a sensor).

FIG. 46 is a combined schematic and set of charts depicting how temperature can be utilized to further identify whether a human finger is initiating a touch event.

FIG. 47 is a combined schematic and chart depicting a finger touch and associated temperature information, in accordance with certain aspects of the present disclosure.

Fig. 48 is a schematic combined side view and signal diagram depicting ridges and valleys of a fingerprint initiating a touch event on an ultrasound input system, in accordance with certain aspects of the present disclosure.

Fig. 49 is a schematic diagram depicting example signals received by an ultrasound input system that are attributable to a glove initiated touch event and the same user initiating a touch event without a glove, in accordance with certain aspects of the present disclosure.

Fig. 50 is a flow chart depicting a process for extracting features from signals of an ultrasound input system, in accordance with certain aspects of the present disclosure.

FIG. 51 is a chart depicting a machine learning decision algorithm for improving touch detection, in accordance with certain aspects of the present disclosure.

FIG. 52 is a flow chart depicting a process for detecting a touch event, in accordance with certain aspects of the present disclosure.

FIG. 53 is a schematic diagram depicting an adaptive threshold scheme for identifying touch events, in accordance with certain aspects of the present disclosure.

FIG. 54 is an example graph depicting energy signals and adaptive thresholds associated with identifying touch events, in accordance with certain aspects of the present disclosure.

Fig. 55 is a schematic diagram depicting a generalized recurrent neural network, according to certain aspects of the present disclosure.

FIG. 56 is a schematic diagram depicting an example recurrent neural network for identifying trigger events, in accordance with certain aspects of the present disclosure.

FIG. 57 is a schematic diagram depicting an example environment for touch detection and state classification using a set of recurrent neural networks, in accordance with certain aspects of the present disclosure.

Fig. 58 is a schematic diagram depicting an electronic device with an ultrasound input device, in accordance with certain aspects of the present disclosure.

Fig. 59 is a schematic diagram depicting an automotive component having an ultrasonic input device, in accordance with certain aspects of the present disclosure.

Fig. 60 is a schematic diagram depicting a keypad using an ultrasonic input device, in accordance with certain aspects of the present disclosure.

Fig. 61 is a schematic diagram depicting a robotic arm using an ultrasound input device, in accordance with certain aspects of the present disclosure.

Fig. 62 is a schematic diagram depicting a piece of furniture using an ultrasonic input device, in accordance with certain aspects of the present disclosure.

Fig. 63 is a set of graphs depicting energy measurement signals of an ultrasonic input device exhibiting material detection, in accordance with certain aspects of the present disclosure.

FIG. 64 is a schematic view of a piezoelectric resonator array containing piezoelectric cantilevers that can be used in an ultrasound input device in accordance with certain aspects of the present disclosure.

Fig. 65 is a schematic diagram of a piezoelectric resonator array containing piezoelectric pillars that can be used in an ultrasonic input device, according to certain aspects of the present disclosure.

Detailed Description

A touch input solution for improving detection of touch input in an HMI is provided. The ultrasound input device may detect the presence of an object on any surface using a sensor located opposite the surface material. The ultrasonic input device enables the inventive design without damaging the product skin or the design material (e.g., material stack). Such an ultrasonic input device may be implemented in various devices, for example, input touch buttons, sliders, scroll wheels, and the like. Ultrasonic input devices may be deployed under surfaces comprising various materials to simplify industrial design and appearance. Further, a grid of ultrasound input device buttons may be implemented to create a keypad, mouse pad, or touch input at any location on any surface. The ultrasonic input device allows the HMI to perform touch input deployment on surfaces including wood, leather, glass, plastic, metal (e.g., aluminum or steel), ceramic, plastic, a combination of one or more materials, and the like.

In some cases, the ultrasound input device may include an ultrasound sensor coupled to a processor, such as an Application Specific Integrated Circuit (ASIC), to provide a fully integrated system on a chip (SOC) that may receive touch input through ultrasound detection. In some cases, the ultrasonic sensor and the processor (e.g., ASIC) may be produced in a single wafer. Fully integrated SOCs may provide many benefits, such as low cost, low profile form factor, improved signal-to-noise ratio, and improved freedom of sensor array design due to mass production via wafer level processes.

In some cases, the ultrasonic input device may include an ultrasonic sensor including a Micromachined Ultrasonic Transducer (MUT), such as a piezoelectric micromachined ultrasonic transducer (pMUT) or a capacitive micromachined ultrasonic transducer (cMUT). By using MUTs in an ultrasound input device as disclosed herein, a number of benefits may be realized, optionally as part of a fully integrated SOC. The use of MUTs may provide improved energy transfer regions because the MUTs (due to their unique and predictable bending mode shape) produce signals that propagate more primarily normal to the transducer surface (longitudinal waves normal to the surface) than other types of waves that propagate laterally. Because the predictable bending mode shape of the MUT is far away from other modes (e.g., bulk mode) over a large frequency range, it is also less likely to generate or receive other types of acoustic waves, such as shear or surface waves that may travel transverse or normal to the sensor surface. Thus, the MUT may achieve more pronounced transmission and sensing regions on the surface material, such as regions directly perpendicular to the MUT through the surface material. Furthermore, the use of MUTs may reduce or minimize the amount of power required to operate an ultrasound input device. For example, MUTs may be used with low parasitics, low drive voltages, and small device capacitances about three orders of magnitude lower than the capacitance of conventional piezoelectric ceramic ultrasound transducers.

The ultrasonic input device may detect patterns associated with touch inputs and distinguish between different types of touch inputs. The different types of touch inputs may vary between finger presses, palm presses, taps, touches and holds, or other such inputs. Each of the various types of touch inputs may have a recognizable and/or distinguishable pattern. In some cases, feedback from multiple sensors (such as multiple sensors arranged in an array) may be used to determine the type of touch input initiated. For example, a palm resting on an array of ultrasound input devices may register an identifiable pattern across multiple ultrasound input devices, and thus a processor coupled to the multiple ultrasound input devices may determine that a touch input is a palm rest and take appropriate action (e.g., reject a palm rest as a touch input or initiate an action based on a palm rest).

The ultrasonic input device may detect a pattern (pattern) associated with the touch input and distinguish between different users initiating the touch input. It has been found that different users of ultrasonic input devices will often produce identifiable and distinguishable signals when initiating touch input. For example, the signal measured from the touch input may vary based on the user's finger, such as the moisture content of the finger, the dimensions of the ridges and valleys of the fingerprint, and other mechanical properties of the individual finger. Further, some users may initiate touch inputs in a repeatable manner, which may be used to identify the user. For example, a first user may typically tap the input device quickly, while a second user may typically place their finger on the input device and then press. As another example, different users may generate different touch pressures, which may also be detected by monitoring the amount of ultrasound signal change. Such factors as the speed of the touch input and the manner of the touch input may be used to facilitate recognition of the user.

In some cases, it may be determined whether a touch event has occurred by comparing the energy signal from the ultrasonic transducer to a threshold. In some cases, the threshold may be updated dynamically or automatically in order to improve detection of touch events and rejection of false positives. The adaptive threshold may be updated based on the input energy signal and any combination of historical threshold data and trigger data (e.g., information regarding whether a touch event has occurred). These inputs may help update a threshold update function that may be used to filter the energy signal value to a new threshold. Thus, certain changes (e.g., slow changes) in the energy signal that are not generally indicative of a touch event may be tracked by the adaptive threshold, while changes (e.g., fast changes) that are indicative of a touch event may not be tracked by the adaptive threshold, which allows the energy signal to fall below the threshold and thus be indicative of a touch event.

In some cases, it may be determined whether a touch event has occurred by passing the energy signal into a recurrent neural network that has been trained based on training data. The recurrent neural network may convert the input energy signal into an output that indicates whether a touch event has occurred.

In some cases, the state of the sensor (e.g., a classification of the type of touch event, such as a press, tap, double-tap, hold, or other such type) may be determined by analyzing the trigger data. In some cases, the trigger data may be passed as input into a recurrent neural network that has been trained based on training data for a particular state. The recurrent neural network may convert the input trigger data to an output indicative of the sensor state.

In some cases, the ultrasonic input device may provide improvements in aesthetic features and reliability of touch input detection on capacitive and mechanical devices. Buttons may be implemented on a surface by defining button areas on the touch surface. The ultrasonic input device may be embedded/placed behind the surface and thus limit environmental exposure (including dust and moisture), as well as reduce manufacturing costs associated with creating special openings on the surface required for other sensors. The ultrasound input device may increase the flexibility of the button programmability options. For example, a user may define the functions of the buttons through a system controller that may be embedded on a shared Printed Circuit Board (PCB) along with the ultrasound input device. In some embodiments, the system controller may monitor user behavior to improve machine/system preferences and performance. An ultrasonic input device mechanically coupled to a surface but positioned away from the field of view, such as beneath or behind an opaque surface, may be used to provide hidden inputs that are not observable or easily discoverable to those who are not yet aware of their location. For example, the ultrasound input device may be placed under a sign (e.g., on a laptop computer or another surface or device), behind a wall, or under the surface of a piece of furniture.

The ultrasound input device may be low power and/or battery powered to operate for extended periods of time without requiring a direct connection to a mains power supply. The ultrasound input device may be or be incorporated into an internet of things (IOT) device that is capable of providing sensor data (e.g., button presses) to other devices on a local or remote network. In some cases, the use of MUTs can allow the ultrasound input device to operate at particularly low power requirements. In some cases, the ultrasound input device, which is a fully integrated SOC, may operate at low power and/or may provide IOT functionality.

I. Overview of the device

Embodiments of the present invention relate to an ultrasonic input device for detecting touch input. In particular, embodiments relate to an ultrasonic input device that includes a transducer coupled to a layer of material that provides a surface to receive touch input signals to a system. The ultrasound input device may be implemented using various layers of materials including wood, leather, glass, plastic, metal (e.g., aluminum, steel, etc.), stone, concrete, plasterboard, gypsum, paper, polymers, biological materials (e.g., tissue such as skin), combinations of one or more materials, and the like. The flexibility of material selection enables the use of ultrasonic input devices in a variety of applications including front and side buttons of mobile devices, steering wheels of vehicles, infotainment units, center console controls, mirrors, seats, door handles, windows, etc., internet of things devices, medical devices such as bed controls, blood pressure measurement devices, input detection for robots such as touch sensing for robotic fingers, and hidden input devices such as hidden within furniture or behind walls.

A. detecting touch input using ultrasonic signals

Fig. 1 is a schematic diagram depicting the effect of a touch on a reflected ultrasonic signal in an ultrasonic input system, in accordance with certain aspects of the present disclosure. The ultrasonic input may include a transducer 104 coupled to the material layer 102. The material layer 102 may be referred to as a stack and may incorporate one or more sub-layers of one or more materials. For example, the stack may be a single piece of glass, a piece of drywall, a laminated set of plastic and glass, or a plastic steering wheel wrapped in leather, or the like. The material layer 102 has a first (inner) surface 106 and a second (outer) surface 108. The layer of material may be characterized by a distance 110 between the first surface 106 and the second surface 108. The material layer 102 may be a cover material for a larger device integrated with the ultrasound input device. In some embodiments, the material layer 102 may form the body or a portion of the body of the device. In these embodiments, the first surface 106 may form an inner surface of the body and the second surface 108 may form an outer surface of the body. The second surface 108 may be considered external in that it is exposed to the environment. The first surface 106 may be considered internal, either because it is not the surface to detect contact, or because it is the surface to which the transducer 104 is acoustically coupled to the material layer 102. Fig. 1 shows an touchless ultrasound input device 120, an ultrasound input device 122 with a light touch, and an ultrasound input device 124 with a heavy touch.

The touch sensor is triggered based on the material acoustic properties of the touch surface (material layer 102) and the input object 112. Detection of the optical touch 122 depends on the extent of the reflected ultrasonic signal 114 in the material layer 102 relative to the absorbed ultrasonic signal 116 transmitted into the input object 112 through the second surface 108 of the material layer 102. As used herein, a reflected ultrasonic signal (e.g., reflected ultrasonic signal 114) may refer to a signal that has reflected off the second surface 108 of the material layer 102, and an absorbed ultrasonic signal (e.g., absorbed ultrasonic signal 116) may refer to a signal at least a portion of which has been absorbed by an input object 112 (e.g., a finger) contacting the second surface 108 of the material layer 102. Contact (e.g., based on pressure) of the input object 112 on the touch surface defines one or more contact areas 118 and an amount of reflection. The material layer 102 may be a single layer or may be composed of multiple layers of materials having different properties. For example, in some implementations, the material layer 102 may be a uniform and isotropic material. In other implementations, the material layer 102 may be a composite material layer composed of multiple layers of different materials. The threshold may be set based on the contact area 118 for the touch of the trigger button and the impedance difference between the input object 112 and the material layer 102, as well as the geometric and acoustic properties of the entire material stack of the material layer 102.

The size of the contact areas 118 and the space between the contact areas 118 may be indicative of the size and spacing of the ridges of the finger and the size and spacing of the valleys of the finger fingerprint. A particular change in size and/or spacing between contact areas 118 may be indicative of different fingers contacting material layer 102. For example, a young individual may have smaller valleys (e.g., smaller distances between contact areas 118) than an older individual. In some cases, the detected dimensions and/or spacing between contact areas 118 may be used to detect or infer that a user is in contact with material layer 102. Such inferences can be used to apply customization (e.g., having touch events cause different actions for different users or having different sensing thresholds), test permissions (e.g., allowing actions only if an identified user is initiating a touch event or the user touches the surface in the same particular manner as a "password"), or perform other rule-based actions using inferences.

Heavy touch 124 may be distinguished from light touch 122 by determining that transducer 104 receives less reflected signals or less unattenuated signals due to an increase in the number of absorbed ultrasonic signals 126. If the touch pressure increases, for example when the contact surface flattens, the ultrasound input device 100 and the input object 113 (e.g., a finger) will have a larger contact area 128. As shown in fig. 1, the larger contact area 128 increases the amount of the absorbed ultrasonic signal 126 that is transferred into the input object 113 through the second surface 108 of the material layer 102. In the case of a user's finger, a larger contact area 128 may indicate that the ridge of the user's finger is flattened against the second surface 108 of the (agains t) material layer 102. In some cases, where the input object 113 is not a finger or is covered by another material, the larger contact area 128 may be the result of the texture elements of the input object 113 being flattened against the second surface 108 of the material layer 102.

Fig. 2 is a schematic diagram depicting an ultrasound input system in a non-contact state and a contact state, in accordance with certain aspects of the present disclosure. Fig. 2 shows an ultrasonic input device 200 without touch (e.g., non-contact state) and an ultrasonic input device 250 with touch (e.g., contact state). The ultrasound input device includes a transducer 202 coupled to a layer of material 204. In this embodiment, the material layer 204 is shown as aluminum, but may be any material (e.g., glass, wood, leather, plastic, etc., or a composite formed from a combination of materials). The transducer 202 is coupled to a first (inner) surface 206 of the material layer 204. The second (outer) surface 208 of the material layer 204 is in contact with air or some other environment similar to the liquid acoustic impedance of a human finger.

For a contactless ultrasound input device 200, the transducer 202 emits an ultrasound signal 210A that is directed into the material layer 204 and toward the second surface 208. Air has an acoustic impedance that is approximately zero and causes the second surface 208 to reflect the reflected ultrasonic signal 212A, which is approximately 100% (e.g., equal to or greater than 90％、91％、92％、93％,94％、95％、96％、97％、98％、99％、99.1％、99.2％、99.3％、99.4％、99.5％,99.6％、99.7％、99.8％、99.81％、99.82％、99.83％、99.84％、99.85％、99.86％、99.87％,99.88％、99.89％、99.9％、99.91％、99.92％、99.93％、99.94％、99.95％、99.96％、99.97％、99.98％ and/or 99.99%) of the transmitted ultrasonic signal. The reflected ultrasonic signal 212A may itself reflect off of the first surface 206 to generate a reflected-transmitted signal 210B, which may be reflected off of the second surface 208 to cause a second reflected ultrasonic signal 212B. In the case of the composite material stack of 204, the signal reflected from 208 may be reflected multiple times within the composite stack itself, and such an echo train may be sensed by the transducer 202. In the case of composite materials, analysis of the received echo train formed by reflections between 206 and 208 and/or internal reflections within multiple layers of 204 may be used directly to identify the material stack of 204 and/or the environment (e.g., air). Such information may be used only to identify acoustic and/or geometric properties of the stack or as additional information for threshold tuning of the sensor calibration and detection algorithm. As depicted in fig. 2, the four reflected ultrasound signals 212A, 212B, 212C, 212D generate four corresponding reflected-transmitted signals 212B, 212C, 212D, 210E. Any number of reflected ultrasonic signals 212A, 212B, 212C, 212D, 212E and reflected-transmitted signals 212B, 212C, 212D, 210E may be obtained from the initially transmitted ultrasonic signal 210A until these signals become too weak to be reflected and/or detected. Graph 214 shows a first amplitude 216 corresponding to transmitted ultrasonic signal 210A and a set of subsequent amplitudes 218A, 218B, 218C, 218D, 218E corresponding to reflected ultrasonic signals 212A, 212B, 212C, 212D, 212E. The first subsequent amplitude 218A is less than the first amplitude 216 due to losses in the material layer 204. Each of the remaining subsequent amplitudes 218B, 218C, 218D, 218E is smaller than the previous subsequent amplitudes 218A, 218B, 218C, 218D due to losses in the material layer 204.

In some cases, one or more frequencies selected for use with an ultrasound input device may be selected to achieve small or minimal attenuation in a non-contact state, thereby achieving a large or maximum number of reflected ultrasound signals. In some cases, a set of reflected ultrasound signals 212A, 212B, 212C, 212D, 212E originating from a single transmitted ultrasound signal 210A may be referred to as a series of reflected signals. For illustration purposes, the various reflected ultrasonic signals 212A, 212B, 212C, 212D, 212E and reflected-transmitted signals 210B, 210C, 210D, 210E are depicted from left to right in fig. 2, however, it will be understood that these signals are separated in time and may not necessarily be spatially separated. The echo signals may be analyzed separately and/or combined or integrated with one another with another echo signal as a detection metric analysis.

For an ultrasound input device 250 with a touch, the input object 220 (in this case a finger) is in contact with the second surface 208 of the material layer 204. The local reflection loss from the area contacted by the object (e.g., finger ridge) depends on how much the touch input medium differs from the input object in acoustic impedance. For example, reflection loss (dB) may be expressed asWhere Z1 is the impedance of the material layer 204 and Z2 is the impedance of the input object 220. Once the input object 220 is in contact with the material layer 204, the emitted ultrasonic signal 210A is split into two parts. The first part, the echo, is the reflected ultrasonic signal 213A and is reflected back to the transducer. The second portion 222 is the transmission signal that penetrates into the input object 220. The reflected ultrasonic signal 213A may itself be reflected off the first surface 206 to generate a reflected-transmitted signal. The reflected-transmitted signal itself may be split into two parts, one of which is the second reflected ultrasonic signal 212B and the other of which is the second part 222 that penetrates into the input object 220. As depicted in fig. 2, the four reflected ultrasound signals 213A, 213B, 213C, 213D generate four corresponding reflected-transmitted signals. Any number of reflected ultrasonic signals 212A, 212B, 212C, 212D, 212E and reflected-transmitted signals may be generated from the initially transmitted ultrasonic signal 210A until these signals become too weak to be reflected and/or detected.

As shown in graph 224, the first amplitude 226 corresponds to the transmitted ultrasonic signal 210A. As the second portion 222 penetrates the input object 220, the first subsequent amplitude 228A corresponding to the reflected ultrasound signal 213A is reduced compared to a touchless ultrasound input device. Each of the remaining subsequent amplitudes 228B, 228C, 228D, 228E is smaller than the previous subsequent amplitudes 228A, 228B, 228C, 228D due to losses in the material layer 204 and internal multipath reflections in the case of the composite stack 204. For illustration purposes, the graph 224 depicts subsequent amplitudes 228A, 228B, 228C, 228D, 228E in solid lines that overlap with corresponding subsequent amplitudes 218A, 218B, 218C, 218D, 218E depicted in dashed lines. The total attenuation of the subsequent amplitudes 228A, 228B, 228C, 228D, 228E of the ultrasound input device in the contact state may be greater than the total attenuation of the subsequent amplitudes 218A, 218B, 218C, 218D, 218E of the ultrasound input device in the non-contact state. Further, the amount of attenuation between each of the subsequent amplitudes 228A, 228B, 228C, 228D, 228E of the ultrasound input device in the contact state may be greater than the amount of attenuation of the subsequent amplitudes 218A, 218B, 218C, 218D, 218E of the ultrasound input device in the non-contact state.

Notably, subsequent amplitudes 228A, 228B, 228C, 228D, 228E associated with a touch event from graph 224 decay faster than corresponding subsequent amplitudes 218A, 218B, 218C, 218D, 218E associated with no touch event from graph 214. In other words, the contrast between the subsequent amplitude of the touch event and the subsequent amplitude of the no-touch event increases with each subsequent reflection number n. In some cases, the ratio of the nth subsequent amplitude associated with the no touch event to the nth subsequent amplitude associated with the touch event may be Γ ⁿ：(1-Γⁿ), where Γ is the percentage of the signal reflected back from the second surface 208. For example, the ratio of subsequent amplitude 218A to subsequent amplitude 228A may be 100:90, the ratio of subsequent amplitude 218B to subsequent amplitude 228B may be 100:81, the ratio of subsequent amplitude 218C to subsequent amplitude 228C may be 100:72, the ratio of subsequent amplitude 218D to subsequent amplitude 228D may be 100:63, and the ratio of subsequent amplitude 218E to subsequent amplitude 228E may be 100:54.

B. ultrasonic touch input device

Fig. 3 illustrates an ultrasound input device in accordance with certain aspects of the present disclosure. The ultrasonic input device 300 may be attached to any surface to detect touch input. The ultrasonic input device 300 may include a sensor 302, such as a Piezoelectric Micromachined Ultrasonic Transducer (PMUT). PMUT transducers are piezoelectric ultrasonic transducers that include a membrane coupled to a thin piezoelectric membrane to sense and/or sense ultrasonic signals. The sensor 302 may be integrated on an Application Specific Integrated Circuit (ASIC), such as a CMOS (complementary metal oxide semiconductor) ASIC 304 (integrated), and formed on a substrate 306. ASIC 304 may include circuitry and/or modules that may be used to perform various processes disclosed herein, such as various analog and/or digital processes described with reference to at least fig. 25-41. For example, ASIC 304 may be used to drive sensor 302, detect the reflected ultrasound signal using sensor 302, and determine the amplitude associated with the reflected ultrasound signal (e.g., using various analog techniques such as summation and integration). In some cases, ASIC 304 may optionally determine a threshold value with which the determined amplitude may be compared to determine whether a touch event has occurred, in which case ASIC 304 may output a signal associated with the occurrence of the touch event.

In some cases, the circuitry of ASIC 304 may perform certain processes in an analog manner, such as signal rectification, integration, mixing, modification, accumulation, and the like. As used herein, an analog circuit may include any circuit capable of performing an action (e.g., rectifying, integrating, etc.) on an analog signal without first digitizing the analog signal. In one example, ASIC 304 may include analog circuitry capable of acquiring a received ultrasonic signal, rectifying the signal, and integrating at least a portion of the rectified signal to provide an integrated signal, such as described with reference to fig. 26, and in another example, ASIC 304 may include analog circuitry capable of acquiring a received ultrasonic signal, calculating an absolute value of the signal, and accumulating the absolute value to provide an accumulated signal, such as described with reference to fig. 28. In another example, ASIC 304 may include analog circuitry capable of acquiring a received ultrasonic signal, squaring the signal by self-mixing, and integrating the squared signal to provide an integrated signal, such as described with reference to fig. 29.

In some cases, different forms of ultrasound transducers may be used for the sensor 302, rather than the PMUT sensor. In some cases, the ultrasonic sensor may be formed using a deposited layer of piezoelectric material, such as aluminum nitride, lead zirconate titanate (PZT), or polyvinylidene fluoride (PVDF). In some cases, the ultrasonic sensor may be a Capacitive Micromachined Ultrasonic Transducer (CMUT). In some cases, the ultrasonic sensor may be a resonator array of a piezoelectric device (e.g., a piezoelectric cantilever or a piezoelectric column).

The substrate 306 may be bonded 310 to a flexible printed circuit/printed circuit board 308 (FPC/PCB) of a larger integrated device such as a mobile phone. In some embodiments, the contact area 312 on the sensor 302 may be bonded to the base contact 314. As shown, the size of the ultrasound input device 300 may be equal to or less than 1.5mm x 1.5mm x 0.5mm, although other sizes may be used. In some cases, FPC/PCB 308, to which substrate 306 is attached, may receive information associated with the detected amplitude of the reflected ultrasonic signal and perform some of the functions disclosed herein, such as determining a threshold value and/or determining when a touch event has occurred. However, in some cases, FPC/PCB 308 receives only signals associated with the occurrence of a touch event and, therefore, does not need to perform further analysis of the amplitude of the detected reflected ultrasonic signal to perform an action based on the touch event.

The integration of ASIC 304 and sensor 302 enables a small form factor, which results in placement of buttons or other functions in many space-constrained applications. For example, the mechanical buttons on the smart phone side can be easily replaced by the ultrasound input device 300 under the housing (casing). To implement a touch interface or other suitable function of the system, the ultrasound input device 300 may be bonded to the surface 316 using an adhesive 318.

Fig. 4 is a cross-sectional view of two piezoelectric micromachined ultrasonic transducers integrated into a CMOS wafer in accordance with certain aspects of the present disclosure. The device 400 shows a cross-sectional view of two PMUTs bonded to a CMOS wafer 402 that may be used in an ultrasound input device. Each PMUT may be formed on a MEMS wafer 401 bonded to a CMOS wafer 402. In this way, the PMUT may be coupled to the necessary processing electronics of the CMOS wafer 402. It should be appreciated that each PMUT may have an active piezoelectric layer 404 and first and second electrodes 403, 405. The first electrode 403 and the second electrode 405 may be electrically coupled to the piezoelectric layer 404.

In some embodiments, the PMUT may include a first contact 422 electrically coupled to the first electrode 403, a second contact 424 electrically coupled to the second electrode 405, and a third electrode 426 electrically coupled to the CMOS wafer 402. Application of an alternating voltage through the first electrode 403 and the second electrode 405 may cause movement (e.g., bending motion) of the piezoelectric layer 404, which may result in a generated acoustic wave. Also, the received acoustic wave that causes movement in the piezoelectric layer 404 may be sensed as a varying voltage across the first electrode 403 and the second electrode 405. One or more vias (vertical interconnect channels) 410 may be formed in the PMUT. Each contact may be wire bonded to the electronic board. In some embodiments, the PMUT may include a passivation layer 428 formed on the surface 420 and contacts. An adhesive coupling surface 430 on the surface 420 or the surface of the passivation layer 428 may be coupled to the material layer of the ultrasound input device.

In some embodiments, passive electrical layer 408 may include SiO ₂ or any other suitable passive layer. The active piezoelectric layer 404 may be aluminum nitride approximately 1 μm thick and the passive elastic layer may be monocrystalline silicon approximately 1 μm thick, although other dimensions and materials may be used. In some embodiments, the active piezoelectric layer 404 may be scandium-doped aluminum nitride. Alternatively, the active piezoelectric layer 404 may be another suitable piezoelectric ceramic, such as PZT. Both the top and bottom electrodes 406 may comprise molybdenum. To bond the PMUT to the top metal 412 of the CMOS wafer 402, fusion bonding via Through Silicon Vias (TSVs) as shown at vias 410 may be used. This approach results in significant parasitic reduction, which in turn results in improved signal integrity and lower power consumption.

In some embodiments, the chamber 414 may be formed with a vacuum or near vacuum to isolate the transducer from the processing electronics in the CMOS wafer 402. The sound generated by PMUTS will not propagate through the near vacuum of the chamber 414, minimizing reflections and interference that may be caused by the material interface with the CMOS wafer 402. The chamber 414 may cause the ultrasound 416 to travel away from the PMUT. Ultrasound 416 may travel through adhesive coupling surface 430 and into the material layer of the ultrasound input device. The material layer may reflect the ultrasound 416, causing a return echo to reflect back to the PMUT. The return echo travels through the adhesive coupling interface and is received by the PMUT.

In some embodiments, CMOS wafer 402 may be an Application Specific Integrated Circuit (ASIC) that includes one or more devices required to drive the transducer. The drive voltage for the PMUT array may be less than 4 volts. In some cases, the drive voltage may be less than 1.8 volts. In some cases, the drive voltage may be 4, 3.5, 3, 2.5, 2, 1.9, 1.8, 1.7, 1.6, or 1.5 volts or less. The ASIC may be manufactured to meet the size requirements associated with the size of the associated PMUT. In some embodiments, the ASIC may include one or more modules to receive the measured signals. The ASIC may be configured to further process the signal. For example, the ASIC may include one or more rectifiers to generate an absolute value signal by taking the absolute value of the received signal (which may be an alternating current). The ASIC may also include an integrator and an analog-to-digital converter (ADC) to convert the reflected ultrasonic signal into a digital representation of the reflected signal. Integration of the ASIC and PMUT further allows embedding the gain amplifier and ADC in the ASIC and eliminates a separate ADC sensor controller chip. This opens up space on the associated circuit board and reduces the implementation costs of the touch input sensor. In some embodiments, the ASIC may transmit the digital signals to at least one or more of a memory, a processor, and a remote device. In other embodiments, the ASIC may include one or more signal processing modules.

The PMUT array may be compatible with CMOS semiconductor processes. In some embodiments, the PMUT materials and dimensions may conform to semiconductor device and materials international (SEMI) standard specifications. Because the PMUT may conform to SEMI specifications, the transducer array may be used with existing CMOS semiconductor fabrication tools and methods. For example, one or more PMUTs may be formed using photolithographic techniques. In contrast, current piezoelectric ultrasound transducer arrays are formed using wafer saws that do not match the accuracy of photolithographic techniques. As a result, PMUTs can be smaller, operate at lower voltages, and have lower parasitics.

C. integration with a circuit board

Fig. 5 is a set of schematic diagrams 502, 504, 506, 508 depicting ultrasonic input devices 510, 512, 514, 516 coupled to various surfaces, in accordance with certain aspects of the present disclosure. Fig. 502 depicts an ultrasonic input device 510 coupled to a metal surface by an adhesive. Fig. 504 depicts an ultrasonic input device 512 coupled to a glass surface by an adhesive. Fig. 506 depicts an ultrasonic input device 514 coupled to a plastic surface by an adhesive. Fig. 508 depicts an ultrasonic input device 516 coupled to a wooden surface by an adhesive. Any suitable material may be used as the sensing surface, such as a non-porous material or a semi-porous material. Porous materials may be used for the sensing surface, but better results may be obtained with smaller pores, higher density and more uniform density.

Further, the ultrasound input devices 510, 512, 514, 516 may be coupled to the flexible PCB, such as on opposite sides of the ultrasound input devices 510, 512, 514, 516 coupled to the sensing surface. The ultrasonic input devices 510, 512, 514, 516 may serve as a mechanical coupling between the sensing surface and the PCB, where the PCB is not attached elsewhere to the sensing surface, although this is not always required. In some cases, a flexible PCB may be used.

The use of PCBs may allow additional components to be integrated with the ultrasound input devices 510, 512, 514, 516 to extend the functionality of the ultrasound input devices 510, 512, 514, 516, such as described with reference to fig. 6.

Fig. 6 is a schematic side view of an ultrasound input system 600 with a common plate assembly, according to certain aspects of the present disclosure. The ultrasound input system 600 may include an ultrasound input device 602 electrically coupled to a circuit board 610, and any number of common board assemblies 612. Each common board assembly may be electrically coupled to a circuit board 610. In some cases, the ultrasound input device 602 may be mechanically coupled to the circuit board 610, such as using an electrical coupling (e.g., a solder joint) or other mechanical support. In some cases, one, some, or all of the commoning plate assemblies 612 may be mechanically coupled to the circuit board 610. In some cases, the circuit board may be a printed circuit board, such as a flexible PCB, although this need not always be the case.

The entire ultrasound input system 600 may be contained within a single common housing, multiple housings, or may not be contained within a housing. In some cases, two or more common plate assemblies 612 may be contained within a single housing, with or without the ultrasound input device 602. In some cases, all of the commoning plate assemblies 612 may be located on the same side of the circuit board 610 as the ultrasound input device 602, although this need not always be the case. When located on the same side as the ultrasound input device 602, the commoning plate assembly 612 may be selected or designed to have a height that exceeds the height of the ultrasound input device 602.

In some cases, the ultrasound input system 600 may include a power component 604. The power component 604 can provide power to the ultrasound input device 602 and/or any other common plate component 612. Examples of power components 604 include batteries, transformers (e.g., transformers coupled to a mains), capacitors (e.g., supercapacitors), solar cells, fuel cells, and/or any other suitable power source.

In some cases, the ultrasound input system 600 may include a processor 606. The processor 606 may enable various processing functions to be performed within the ultrasound input system 600 based on signals received from the ultrasound input device 602. Examples of suitable processors 606 include microcontrollers, central processing units, or other suitable devices. The processor 606 may also be coupled to memory to access processing routines, to access stored data, and/or to store data.

In some cases, the ultrasound input system 600 may include a communication component 608. The communication component 608 can interact with the ultrasound input device 602 and/or the processor 606 to send signals to or receive signals from an external device. Examples of suitable communication components 608 include wireless radios (e.g., bluetooth, wiFi, zigbee, Z-Wave, etc.), audio devices (e.g., microphones or speakers), visual devices (e.g., cameras, lights, or displays), haptic devices (e.g., haptic feedback devices such as motors and vibrators), or other devices suitable for transmitting or receiving signals.

In some cases, the ultrasound input system 600 may include a common board assembly 612 that includes a power assembly 604, a processor 606, and a communication assembly 608. In some cases, the ultrasound input system 600 may include more or fewer common plate assemblies that include different types of assemblies.

D. Example System settings

Fig. 7 is a schematic diagram depicting an example ultrasound input system 700, in accordance with certain aspects of the present disclosure. The ultrasound input system 700 may include an ultrasound sensor 702 and a processor 722. The ultrasonic sensor 702 may be identical to the transducer 104 of fig. 1, and the processor 722 may be electrically coupled to the ultrasonic sensor 702, and may optionally be mechanically coupled to the ultrasonic sensor 702. In some cases, the processor 722 and the ultrasonic sensor 702 may be integrated into the same package, although this need not always be the case. Processor 722 may perform certain functions as disclosed herein, such as acquiring signals from ultrasonic sensor 702 and/or detecting a touch event. In some cases, an optional computing device 724 may be coupled to the processor 722 to exchange information, such as information related to touch events, information related to signals from the ultrasonic sensor 702, or information related to how the processor 722 interprets the signal information. A data store 726 may be coupled to the processor 722 for storing information, such as information related to how the processor 722 interprets the signal information. In some cases, optional computing device 724 may be coupled to a data store 728 that may store information, such as information related to how to interpret signal information from ultrasonic sensor 702 to determine touch events. Computing device 724 may be any suitable computing device, such as a desktop computer, laptop computer, server, smart phone, tablet, or any other suitable computing device. The computing device 724 may be coupled to the processor 722 by a wired or wireless connection. The computing device 724 may be coupled to the processor 722 through a local or remote connection.

In some cases, processor 722 may be an Application Specific Integrated Circuit (ASIC). In some cases, the ultrasonic sensor 702 may be a MUT. Processor 722 may be any suitable circuitry designed to be able to drive and receive one or more transducers of sensor 502. Processor 722 can drive transducers to send and receive ultrasonic signals to implement the touch sensing capabilities described herein. In some cases, the processor 722 may output a measured energy level (e.g., an energy signal) associated with the sensor 502, which may later be used to determine whether a touch event has occurred. In some cases, processor 722 may output a touch signal indicating the occurrence of a touch event. In this case, the processor 722 may perform the necessary processing to determine whether a touch event has occurred. In some cases, processor 722 may further perform the necessary processing to determine additional information associated with a touch event, such as whether the touch event was initiated by a bare or gloved finger, whether the touch event was initiated by a first user or a second user, or other aspects of the touch event. Such additional information may take the form of inferences and may have different confidence levels, although this need not always be the case. In some cases, the processor 722 may have the ability to process signals and identify the type of pattern (e.g., single click, double click, hold, etc.) that the user is entering. Such capability in processor 722 may be implemented by a hardware processing block or may be written as part of firmware in chip memory. In some cases, the processor 722 may have the ability to self-calibrate and tune its parameters for signal recognition and pattern recognition.

In some cases, the processor 722 may send energy signals and/or touch signals to the computing device 724. The computing device 724 may perform the necessary processing to determine whether a touch event has occurred and/or additional information associated with the touch event, such as whether the touch event was initiated by a bare or gloved finger, whether the touch event was initiated by a first user or a second user, or other aspects of the touch event.

In some cases, the data store 726 may store information related to how the processor 722 determined whether a touch event has occurred or other information associated with a touch event. In some cases, the data store 726 may store model information used by the processor 722 to process the energy signals and determine whether a touch event has occurred. In some cases, model information stored in data store 726 may be provided by and/or updated using computing device 724.

Fully integrated system on chip for ultrasonic touch input

Embodiments of the present disclosure allow for a fully integrated system-on-chip for ultrasonic touch input. For example, the integrated ultrasound input device may include an ultrasound sensor and an Application Specific Integrated Circuit (ASIC). Various production techniques may be used to create integrated ultrasonic input device wafers to allow for low profile dimensions and improved noise resistance and lower power.

A. integrated ultrasound input device overview

Fig. 8 is a schematic side view depicting an integrated 820 with an ultrasonic sensor 802 and an Application Specific Integrated Circuit (ASIC) 822, in accordance with certain aspects of the present disclosure. The ultrasonic sensor 802 may be comprised of one or more ultrasonic transducers arranged in an array. In some cases, the ultrasonic transducer is a MUT.

ASIC 822 may be any suitable circuitry designed to be capable of driving and receiving one or more transducers of ultrasonic sensor 802. ASIC 822 may drive transducers to send and receive ultrasonic signals to implement the touch sensing capabilities described herein. In some cases, ASIC 822 may output a measured energy level associated with ultrasonic sensor 802, which may later be used to determine whether a touch event has occurred. In some cases, ASIC 822 may output a touch signal indicating the occurrence of a touch event. In this case, ASIC 822 can perform the necessary processing to determine if a touch event has occurred. In some cases, ASIC 822 may further perform the necessary processing to determine additional information associated with the touch event, such as whether the touch event was initiated by a bare or gloved finger, whether the touch event was initiated by the first user or the second user, or other aspects of the touch event. Such additional information may take the form of inferences and may have different confidence levels, although this need not always be the case. In some cases, ASIC 822 may have the ability to process the signal and identify the type of pattern the user is inputting (e.g., single click, double click, hold, etc.). This capability in ASIC 822 may be implemented by a hardware processing block or may be written as part of firmware in chip memory. In some cases, ASIC 822 may have the ability to self-calibrate and tune its parameters for signal identification and pattern recognition.

The integrated ultrasound input device 820 may be fully or partially enclosed within a housing 824 to form an enclosure. The housing 824 may take the form of any suitable material, such as a cured resin. In some cases, housing 824 contains only ultrasonic sensor 802 and ASIC 822, as well as any electrical contacts necessary to couple ASIC 822 to external components. In some cases, housing 824 may contain additional components, such as additional sensors (e.g., thermal sensors, vibration sensors, or gyroscopes). In some cases, the material for the housing 824 may be selected to perform well as part of the stack of ultrasound input systems. For example, a material having maximum energy transmission in the frequency range associated with a particular ultrasound input device 820 may be used to maximize the signal. In some cases, additional materials may be used within the housing 824 or incorporated into the housing 824 itself to achieve the desired response to ultrasound propagation into the stack. For example, a window may be fitted into housing 824 adjacent to ultrasonic sensor 802 to provide a path for transmitting ultrasonic signals to and from ultrasonic sensor 802. The window may be made of an optically transparent, translucent or opaque material and may be selected to pass the ultrasound signal with little or no attenuation. Moreover, materials may be used in the stack to enhance acoustic matching between layers to facilitate transmitting and/or receiving signals.

In some cases, housing 824 may be applied after ultrasonic sensor 802 and ASIC 822 have been formed into a wafer and diced into individual chips. However, in some cases, housing 824 may be applied while ultrasonic sensor 802 and ASIC 822 are still part of a wafer containing many chips. Any suitable chip packaging method may be used to package the ultrasonic sensor 802 and the ASIC 822.

In some cases, other types of processors or circuits may be used instead of ASIC 822. For example, a general purpose programmable processor may be used in place of ASIC 822 while still achieving many of the benefits associated with integrated ultrasound input device 820. In some cases, ASIC 822 may receive power as an input, which may be used to power ASIC 822 itself and drive the transducers of ultrasonic sensor 802. In some cases, a general purpose programmable processor may be used to communicate between multiple chips with or without internal ASICs in master and slave form.

In some cases, the height of the package of the integrated ultrasound input device 820 may be about 500 microns or less. In some cases, the ultrasonic sensor 802 and ASIC 822 of the integrated ultrasonic input device 820 may have a combined height of about 150 microns or less.

B. Production technology

Fig. 9 is a schematic diagram of a set of combinations depicting two processing routes 926, 928 for generating an integrated ultrasound input device wafer 930, in accordance with certain aspects of the present disclosure. The first processing path 926 depicts generating the wafer 930 using monolithic techniques. The second process route 928 depicts the generation of the wafer 930 using wafer bonding techniques. Any suitable process may be used to generate wafer 930 including sensors and ASICs as described herein.

Beneath the first processing route 926, an ASIC wafer 932 is provided, and a sensor layer 934 is then built on the ASIC wafer 932 to produce a monolithic wafer 930 that includes both sensors and ASICs. This type of wafer level fabrication may allow for small form factors to be generated in an economical manner.

Below the second processing route 928, a sensor layer 934 is provided, and an ASIC wafer 932 is provided. The provided sensor layer 934 may then be bonded to the ASIC wafer 932 using any suitable wafer bonding technique, with or without an intervening layer.

The wafer 930 generated by the first processing route 926, the second processing route 928, or any other suitable processing route may include one or more instances of sensors and ASICs that may be used to create an integrated ultrasound input device.

Fig. 10 is a set of schematic diagrams depicting a single integrated ultrasonic input device 1020 cut from a wafer 1030, PCB mounted, and stacked mounted, in accordance with certain aspects of the present disclosure. Wafer 1030 may be wafer 1030 of fig. 10, wafer 1030 being cut or diced into pieces (e.g., die). Each wafer 1042 may contain a sensor 1002 and an ASIC 1022 for a single integrated ultrasound input device 1020. If the housing was not previously applied to the wafer 1030, each die 1042 may be packaged in the housing to create an integrated ultrasonic input device 1020.

Ultrasonic input device 1020 may be mounted on a Printed Circuit Board (PCB) 1036 or otherwise electrically coupled to any other necessary electronics. For example, in some cases, the ultrasound input device 1020 may be electrically coupled to a battery or other power source. In some cases, the ultrasound input device 1020 may be mounted on a PCB 1036 containing other electronic components 1038, such as a processor and a power supply.

The ultrasonic input device 1020 may be mounted to a substrate 1040. Substrate 1040 may be any combination of one or more materials through which ultrasonic signals may be transmitted to sensor 1002. The housing of the ultrasonic input device 1020 may be coupled to a base 1040. The combination of materials through which ultrasonic signals are transmitted from the outer surface of substrate 1040 to sensor 1002 may be referred to as a stack, which may include a housing of ultrasonic input device 1020. The ultrasonic input device 1020 may be coupled to the substrate 1040 using any suitable technique, including using an adhesive, mechanical coupling, active pressure, or any other suitable technique for acoustically coupling the ultrasonic input device 1020 and the substrate 1040.

C. low profile dimensions

Fig. 11 is a schematic cross-sectional view of a consumer electronic product 1100 including an integrated ultrasound input device 1120 in accordance with certain aspects of the present disclosure. The consumer electronic product 1100 may be a smart phone or any other suitable device. The integrated ultrasound input device 1120 may be attached to a substrate comprised of one or more layers of the display 1140 or to any other portion of the consumer electronic device, such as a frame or back surface (1144) which may be made of metal, plastic, or other material. Display 1140 may comprise a plurality of layers including a display layer, an illumination layer, a protective layer, a sensing layer, and other suitable layers. Through its coupling with the display 1140, the integrated ultrasound input device 1120 may be used to register touch events associated with the display 1140. However, in some cases, the integrated ultrasonic input device 1120 may be coupled to any surface of the consumer electronic product to detect touch events on opposite sides of the surface, such as the back side or side edges of the consumer electronic product.

As described herein, the integrated ultrasound input device 1120 may be formed to have a very small height, for example, equal to or less than 500 microns. Because of the low profile of the integrated ultrasound input device 1120, one or more of such integrated ultrasound input devices can be easily positioned within the consumer electronic product 1100, leaving sufficient space for other components. For example, the low profile of the integrated ultrasound input device 1120 may occupy only a small portion of the overall height of the consumer electronic product 1100, allowing more space for other components, such as a larger battery 1144 with greater capacity, or more open space for airflow. Furthermore, due to the physical properties behind the design and operation of the device, the integrated ultrasound input device can be operated in a small localized area for transmitting and receiving ultrasound information. Such local operation greatly improves the performance robustness of the device against sources of interference (such as touch or hold) caused outside the operating region.

D. Improved noise resistance and lower power

Fig. 12 is a set of schematic cross-sectional views comparing a non-integrated ultrasound input device 1200 with an integrated ultrasound input device 1220 in accordance with certain aspects of the present disclosure. The non-integrated ultrasound input device 1200 is more susceptible to noise due, at least in part, to the relatively long length of wire required to couple the ASIC and the sensor. For example, non-integrated ultrasound input device 1200 may have exposed electrical traces (ELECTRICAL TRACE). Not only is power used to transmit signals along the electrical trace, but the electrical trace may be further susceptible to interference. Thus, the overall signal-to-noise ratio of the non-integrated ultrasound input device is relatively low. If a higher signal-to-noise ratio is required, the ASIC must provide more power to drive the sensor, in which case the overall system will have a relatively higher power consumption.

In contrast, the integrated ultrasound input device 1220 of the present disclosure is an integrated chip packaged in a housing. The integrated ultrasound input device 1220 does not have large exposed traces or wires between the sensor and the ASIC. Thus, due at least in part to the minimal conductive traces between the sensor and the ASIC, there is little or no risk of interference and little or minimal energy drain in transmitting signals from the sensor to the ASIC. Thus, the integrated ultrasound input device 1220 can operate with improved signal-to-noise ratio and/or with improved power efficiency compared to a similar non-integrated ultrasound input device 1200.

III ultrasonic sensor design

The ultrasound input device may include a plurality of transducers, which may be configured as, for example, a sensor array. In some embodiments, multiple transducers may allow measurement of multiple frequencies. Furthermore, in other embodiments, multiple transducers may allow for separation of transmission and reception capabilities. For example, some transducers may be configured to transmit ultrasonic signals, while other transducers of the plurality of transducers may be configured to receive ultrasonic signals. In further embodiments, multiple transducers may allow beamforming.

A. Transducer array

Fig. 13A is a top view of a sensor array 1302 of an ultrasound input device according to certain aspects of the present disclosure. The sensor array 1302 can include one or more transducers 1350 (e.g., MUTs). In general, sensor array 1302 can have multiple transducers 1350. The sensor array 1302 of FIG. 13A is depicted as having 144 different transducers 1350 that pass through the sensor array 1302 approximately 1.2mm square, although other numbers of transducers 1350 and arrays of other sizes may be used. Various electrical traces in the sensor array 1302 can interconnect different transducers 1350 with the ASIC. Each transducer 1350 may be individually addressable. In some cases, the use of transducers 1350 for a particular purpose (e.g., as transmitters or receivers, or with certain particular frequencies) may be set or changed by the ASIC, so that each transducer 1350 may perform any particular function performed by any other transducer 1350 of the sensor array 1302. However, in some cases, one or more transducers 1350 may be specifically selected or configured to perform particular functions more efficiently or effectively. For example, some transducers 1350 may be designed to achieve improved transmission, while other transducers 1350 may be designed to achieve improved reception.

Fig. 13B is a top view of an alternative sensor array 1312 of an ultrasound input device in accordance with certain aspects of the present disclosure. The sensor array 1312 may include one or more transducers 1360. Sensor array 1312 depicts a sensor array that includes 36 ultrasonic transducers 1360. Various electrical traces in the sensor array 1312 may interconnect different transducers 1360 with the integrated circuit layers. One or more transducers 1360 of the sensor array 1312 may be transmitting ultrasound transducers. One or more of the transducers 1360 of the sensor array 1312 may be receiving ultrasound transducers. As described herein, the transducers 1360 of the sensor array 1312 may transmit and receive at any suitable frequency. The relative size of the transducer 1360 may be indicative of the frequencies that can be transmitted/received by the transducer.

Various electrical traces (not shown) in the sensor array 1312 may interconnect different transducers 1360 with the integrated circuit. The various electrical traces may interconnect the different transducers 1360 in any suitable manner. For example, electrical traces may connect transducers 1360 in horizontal and vertical grids. As another example, the electrical traces may connect transducers 1360 that are diagonally positioned with respect to each other.

Fig. 14A is a schematic diagram of the sensor array 1302 of fig. 13A depicting one example configuration of individual transducers of the sensor array 1302, in accordance with certain aspects of the present disclosure. In this example configuration, of the 144 different transducers in sensor array 1402, 60 are set to operate as low frequency transmitters, 8 are set to operate as low frequency receivers, 56 are set to operate as high frequency transmitters, and 20 are set to operate as high frequency receivers. The configuration depicted in fig. 14A may be particularly useful for sensing touch events using multiple ultrasonic frequencies to better identify environmental changes and/or to improve the operating frequency bandwidth of the device relative to real touch events, for example, to make the device more responsive over a wider frequency range.

Fig. 14A further illustrates a sensor array 1402 including four corner regions. The corner regions of the sensor array 1402 may include a plurality of transducers. For example, the sensor array 1402 includes four rotationally symmetric corner regions that primarily include (e.g., most of) low frequency transmitting ultrasound transducers that may surround the low frequency receiving ultrasound transducers. The corner region of the sensor array 1402 includes 16 ultrasonic transducers in a 4x4 array. However, it should be understood that the corner regions of the sensor array 1402 may include up to one-fourth of the total number of ultrasonic transducers included in the sensor array 1402. For example, a square sensor array comprising 81 transducers may comprise four corner regions. Each of the four corner regions may include a 1x1, 2x2, 3x3, or 4x4 transducer grid. In some cases, the sensor array 1402 may include a high frequency transmitting ultrasound transducer surrounding a low frequency transmitting ultrasound transducer, for example, as shown. In some implementations, the high frequency transmitting ultrasound transducer may not be on the diagonal of the corner region, but there may be a high frequency receiving ultrasound transducer, for example, as shown in fig. 14A. Furthermore, the central region may mainly comprise a low frequency transmitting ultrasound transducer. The central region may be surrounded by a high frequency transmitting ultrasound transducer. In some cases, the central region may include a transmitting ultrasound transducer. In other cases, the central region may include a receiving ultrasound transducer. The central region of the sensor array may have any suitable size, e.g., 1x1, 2x2, 3x3, 4x4, 5x5, 6x6, 7x7, etc.

In some cases, sensor array 1302 may have any number of transducers operating at any number of different frequencies. While the example configuration of fig. 13A may be useful in some circumstances, other configurations may be used. In some cases, a single type of sensor array 1302 may be manufactured in bulk and used with the same or different types of ASICs. For example, different types of ASICs may be configured to operate the same sensor array 1302 in different configurations (e.g., with more or fewer transmitters or receivers, different frequencies, or a greater or fewer number of different frequencies). In some cases, the same type of ASIC may also be programmed to operate in different configurations. In some cases, an integrated version of the transducer and/or ASIC may be used in conjunction with a non-integrated transducer and/or ASIC to achieve a particular purpose, such as increasing transmission power.

The sensor array depicted in fig. 13A and 14A may include one or more piezoelectric micromachined ultrasonic transducers, one or more capacitive micromachined ultrasonic transducers, one or more monolithic piezoelectric transducers, or one or more non-monolithic piezoelectric transducers. In some cases, the sensor array may include any suitable combination of the above transducers. Further, the sensor array may have any suitable size. For example, the sensor array may include an array of ultrasonic transducers of 2x2, 3x3, 5x5, 9x9, 16x16, etc. For example, sensor array 1312 depicts a sensor array of 6 ultrasound transducers by 6 ultrasound transducers.

Fig. 14B is a schematic diagram of a sensor array depicting eight example configurations of individual transducers of a corner region of the sensor array, in accordance with certain aspects of the present disclosure. The sensor array depicted in fig. 14B is 12x12 transducer in size and the corner region is 4x4 transducer in size, however, it should be understood that embodiments may include any suitable size sensor array and corner region.

Each corner region of the sensor arrays 1410-1424 may include a transmitting ultrasound transducer 1426 and a receiving ultrasound transducer 1425. The transmitting ultrasound transducer and the receiving ultrasound transducer may be arranged as in the sensor arrays 1410-1424. Thus, in various combinations, the receiving transducers may be diagonal to each other, where the diagonal may be in various positions and of various lengths. The receiving transducers may be in blocks (e.g., 2x 2) at various locations, and may be other shapes that include an odd number of receiving transducers.

For example, the transmitting ultrasound transducers in the sensor array 1410 may each transmit at the same frequency. Similarly, the receiving ultrasound transducers in the sensor array 1410 may each receive the same frequency, which may be the same frequency transmitted from the transmitting ultrasound transducer. The corner regions need not all be the same and may occur in various combinations, for example, a combination may be of one type selected from 1410, 1412, 1414, and 1416. The interior region may have various combinations shown in fig. 14C.

Fig. 14C is a schematic diagram of a sensor array depicting eight example configurations of individual transducers of the sensor array, in accordance with certain aspects of the present disclosure. The sensor array depicted in fig. 14C depicts transducers inside corner regions of the sensor array. The interior region may include a transducer between at least two corner regions. The transducers shown in the sensor array may include a transmitting ultrasound transducer 1446 and a receiving ultrasound transducer 1445. The transmitting ultrasound transducer and the receiving ultrasound transducer may be arranged as in sensor arrays 1430-1444. Any of the arrangements in fig. 14C may be used with any of the corner arrangements in fig. 14B.

As depicted, the majority may be transmitting transducers, but they may be minority. The receiving transducers may contact each other to form a loop, for example, as in sensor arrays 1430-1438. As an alternative, the receiving transducers may form disjoint groups, as in sensor arrays 1440-1444. In such disjoint groups, there may be an even or odd number of receiving transducers. These groups may all be the same or may vary.

Fig. 14D is a schematic diagram of a sensor array depicting eight example configurations of individual transducers of the sensor array, in accordance with certain aspects of the present disclosure. The sensor array depicted in fig. 14D depicts a sensor array comprising a different number of ultrasound transducers. For example, sensor array 1450 includes 36 transducers, while sensor array 1460 includes 64 transducers. The transmitting ultrasound transducer 1448 and the receiving ultrasound transducer 1447 may be arranged as in the sensor arrays 1450-1464.

Fig. 14E is a schematic diagram of a sensor array depicting two example configurations of individual transducers of the sensor array, in accordance with certain aspects of the present disclosure. Fig. 14E shows two example sensor arrays, which are 12x12 ultrasound transducers in size. For example, both sensor array 1465 and sensor array 1466 include 144 transducers. The transmitting ultrasound transducer 1492 and the receiving ultrasound transducer 1491 may be arranged as in the sensor arrays 1465-1466. In some implementations, the central region of sensor arrays 1465-1466 may not include an ultrasound transducer. The sensor array 1465 may then comprise, for example, 138 ultrasound transducers. However, it should be understood that the central region may be greater or less than the size of 16 transducers in a square. In some implementations, the ultrasound transducer may form a ring around a central region that does not include the ultrasound transducer. The sensor array, which does not include ultrasonic sensors, may include a central region for routing space.

In some cases, the ultrasound transducers in the sensor array may be grouped. For example, sensor array 1465 may include 8 sets of ultrasonic transducers, where each set may include 16 ultrasonic transducers included in a square shape. The central region of sensor array 1465 does not include a set of ultrasonic transducers. Each group of ultrasound transducers may not be contiguous with each other. For example, there may be a gap between two or more sets of ultrasound transducers. The groups may be horizontally, vertically, diagonally disjoint.

Fig. 14F is a schematic diagram of a sensor array depicting example configurations of individual transducers of sensor arrays of various sizes, in accordance with certain aspects of the present disclosure. Fig. 14F shows ten example sensor arrays of varying dimensions. For example, sensor array 1467 includes 16 ultrasonic transducers, sensor array 1468 includes 25 ultrasonic transducers, sensor array 1469 includes 36 ultrasonic transducers, sensor array 1470 includes 49 ultrasonic transducers, sensor array 1471 includes 64 ultrasonic transducers, sensor array 1472 includes 81 ultrasonic transducers, sensor array 1473 includes 100 ultrasonic transducers, sensor array 1474 includes 131 ultrasonic transducers, sensor array 1475 includes 144 ultrasonic transducers, and sensor array 1476 includes 169 ultrasonic transducers. The transmitting ultrasound transducer 1494 and the receiving ultrasound transducer 1493 may be arranged as in the sensor arrays 1467-1476. However, it should be understood that the configuration of the transmitting ultrasound transducer 1494 and the receiving ultrasound transducer 1493 may be any suitable arrangement as described herein.

Fig. 14G is a schematic diagram of a sensor array depicting twelve example configurations of individual transducers of the sensor array of transducers of size 8x8, in accordance with certain aspects of the present disclosure. For example, sensor arrays 1477-1488 include 81 transducers. The transmitting ultrasound transducer 1496 and the receiving ultrasound transducer 1495 may be arranged as in the sensor arrays 1477-1488.

In some implementations, the sensor array may include any suitable combination of the sensor array characteristics (e.g., areas, groups, arrangements, etc.) described herein and described with reference to fig. 13A-13B and 14A-14G. For example, the sensing array may include corner regions as depicted in sensor array 1420 of fig. 14B and interior and center regions as depicted in sensor array 1472 of fig. 14F. The arrangement of the ultrasonic transducers in the sensor array may be based on the application of the sensor array, the frequency of operation, size limitations, power constraints, etc.

The various embodiments provide a number of advantages. For example, different array sizes may be implemented depending on sensor area (physical size) constraints and power constraints. The total array size, the configuration of the transmitting and receiving ultrasound transducers (e.g., pmuts), and the size of the ultrasound transducers may be used to determine the transmitting and receiving acoustic aperture and beam shape. The transmit and receive acoustic aperture and beam shape can be modified using at least the above characteristics for which different stack thicknesses and materials and use of the sensor array can be selected to produce optimal performance given constraints (e.g., size, power, sampling frequency, supply voltage, process breakdown voltage, etc.).

B. Multi-frequency measurement

Fig. 15 is a set of graphs 1502, 1504, 1506 depicting energy measurements from transducers of a single sensor array operating at different frequencies, in accordance with certain aspects of the present disclosure. Graphs 1502, 1504, 1506 show energy measurements over time for a pair of touch events. Graph 1502 depicts an energy measurement of a transducer operating at 100kHz, graph 1504 depicts an energy measurement of a transducer operating at 1MHz, and graph 1506 depicts an energy measurement of a transducer operating at 10 MHz. It is apparent that measurements made at these different frequencies have different energy traces, particularly with respect to temperature drift.

Since the drop in energy measurement associated with the ultrasound transducer receiving the reflected ultrasound signal is used as a factor in identifying touch events, it may be desirable to find a technique that reduces any false touch events. As depicted in fig. 15, energy measurements at different frequencies react differently with respect to temperature changes (e.g., temperature changes that occur when heat is transferred from a finger to a substrate or from a substrate to air, or other such temperature changes). Thus, rather than simply relying on identifying the decline in energy measurements to infer a touch event, an ultrasonic touch input system may use energy measurements or other types of operational processes at multiple frequencies, such as different ultrasonic beam shapes, pulse numbers, etc., to confirm or reject an inference of a touch event. For example, a perceived energy drop in graph 1502 may not be registered as a touch event because a concurrent energy drop is not identified in graphs 1504 or 1506. However, once all three graphs 1502, 1504, 1506 register a concurrent energy drop, it can be assumed that a touch event has occurred.

Fig. 16 is a graph 1600 depicting temperature behavior of an ultrasonic transducer with respect to operating frequency, in accordance with certain aspects of the present disclosure. The graph 1600 includes four lines, each associated with an air signal (AIR SIGNAL) or a target signal at a first or second frequency. The air signal may refer to the energy measured when there is no touch event, while the target signal may refer to the energy measured when a touch event is occurring. The first and second frequencies may be any suitable different frequencies. Graph 1600 shows. For all signals, the overall signal strength decreases as the temperature increases. The chart 1600 also shows a different behavior of each frequency with respect to temperature, which can thus be utilized to help identify whether a touch event has occurred (e.g., to identify whether a change in energy measurement is associated with a touch event or only a temperature drift).

In an example, the first and second measurements may be made by a transducer operating at a first frequency, which results in measurements at point 1610 and line 1612. At this point, it may not be clear whether the measurement at line 1612 is associated with a touch event (e.g., moving from point 1610 to point 1614) or a temperature change (e.g., moving from point 1610 to point 1616). First and second measurements may also be made for transducers operating at a second frequency, which results in measurements at point 1618 and either line 1620 or line 1622. If a second measurement at a second frequency falls on line 1620, it can be inferred that the energy drop is associated with a temperature change from point 1618 to point 1626, and thus is unlikely to be associated with a touch event. However, if a second measurement at a second frequency falls on line 1622, it may be inferred that the energy drop is associated with a touch event because the energy drops from point 1618 to point 1624. Measurements at the first and second frequencies may be taken simultaneously, sequentially, or otherwise in close temporal proximity to each other (e.g., within milliseconds, tens of milliseconds, or hundreds of milliseconds of each other). Thus, by comparing the change in energy measurements over a period of time over multiple frequencies, it can be determined whether a touch event has occurred.

Although the graph 1600 has been described with reference to frequency-dependent energy changes due to temperature changes, such techniques may be used to identify and utilize frequency-dependent energy changes due to other environmental condition changes such as humidity.

Fig. 17 is a graph 1700 depicting frequency response with respect to a stacked structure, in accordance with certain aspects of the present disclosure. The graph 1700 shows three lines, each associated with a different stack. Each different stack may be composed of different materials or different combinations of materials. Because of inherent differences in each stack, each stack may have a unique response curve associated with the transmission frequency used by the ultrasound input device. The response curve may be a measure of energy, received signal peak, or any other figure of merit. As depicted in fig. 17, the frequency of providing the highest response for the stack/cover 1 is higher than the frequency of providing the highest response for the stack/cover 2, which itself is also higher than the frequency of providing the highest response for the stack/cover 3.

Thus, specific frequencies and stacked materials can be matched to provide optimal results. For example, given a set of known frequencies, the material from which the housing of the integrated ultrasound input device is made may be selected to maintain the highest possible energy measurement of the initially transmitted reflected ultrasound signal from the ultrasound input device. As another example, given a known stack or known material (e.g., a particular display or a particular type of wood from a consumer product manufacturer), the ultrasonic input device may be set to operate at a frequency that provides the highest possible energy. In some cases, the ultrasound input device may automatically detect the optimal frequency to use based on measuring multiple frequencies that are in close temporal proximity to each other.

C. Separate transmission and reception

Fig. 18 is a schematic diagram depicting a circuit 1800 for receiving and transmitting signals by an ultrasound transducer, the circuit being in a transmitting state. The circuit 1800 drives the ultrasonic transducer to transmit and receive signals and thus requires a high voltage switching circuit to separate the high voltage transmitter from the low voltage receiver. On transmission, the high voltage switch allows the high voltage transmitter circuit to drive the transducer while isolating the low voltage receiver. To move to the receive state, the switch must isolate the high voltage transmitter circuit and couple the transducer to the low voltage receiver circuit.

Fig. 19 is a schematic diagram depicting the circuit 1800 of fig. 18 for receiving and transmitting signals by an ultrasound transducer, the circuit being in a receiving state. When in the receive state, the high voltage switch isolates the high voltage transmitter circuit and couples the transducer to the low voltage receiver circuit. However, high voltage switches typically have a large capacitance that inherently attenuates the signal received at the transducer when it is conducted to the low voltage receiver. Thus, for example, an input voltage of 0.37 millivolts (370 microvolts) may be attenuated to less than 2 microvolts. Such parasitics can drastically reduce the available signal, thereby reducing the overall signal-to-noise ratio.

Fig. 20 is a schematic diagram depicting isolation circuits 2000, 2002 for receiving and transmitting signals by an ultrasound transducer, in accordance with certain aspects of the present disclosure. Unlike the circuit 1800 of fig. 18-19, the circuit 2000, 2002 of fig. 20 eliminates the need for a high voltage switch. Thus, the circuits 2000, 2002 may provide for efficient driving of the transmit transducer while also providing for efficient reception of the receive transducer. The circuit 2000 contains a high voltage transmitter circuit that directly drives a transducer configured as a transmitting transducer. The circuit 2002 contains a low voltage receiver circuit that receives signals directly from a transducer arranged to receive the transducer.

By separating the transmitting and receiving transducers, signal integrity may be improved, size may be reduced, and overall cost may be reduced. For example, by reducing or eliminating parasitics from electrical components (e.g., high voltage switches) embedded between the transducer and its low voltage receiver circuitry, signal integrity may be improved and power consumption may be improved. Since high voltage devices (e.g., high voltage switches) tend to be larger in size, the overall chip size may also be reduced. Thus, by eliminating these switches, and optionally some high voltage transmitter circuitry, the overall chip size and cost may be reduced.

D. Beamforming

Fig. 21 is a set of schematic side views 2100, 2102, 2104, 2106 depicting beamforming achieved through use of an ultrasound transducer, in accordance with certain aspects of the present disclosure.

The diagram 2100 depicts the beam pattern of a single ultrasound transducer, such as a standard piezoelectric transducer. The beam is wide and fixed by the sensor size and sensor topology. There is no ability to adjust the beam for the transducer of diagram 2100.

The diagram 2102 depicts a focused beam implemented by activating a particular transducer group. Using beamforming techniques, the activated transducer may focus the beam to a particular distance, which may improve the pressure sensitivity and accuracy of the ultrasonic sensor. For example, the focused beam may be used to provide fine point accuracy for touch events, as well as fine point accuracy for detecting other information associated with touch events, such as ridges and valleys of a user's fingerprint.

Fig. 2104 depicts a wide beam achieved by activating a particular transducer group. Using beamforming techniques, the activated transducer may focus the beam to a certain close distance to allow the beam to reach a certain point and spread out again before reaching the target distance. Such a wide beam may improve the overall coverage of the sensor and may be used to obtain more average measurements over a larger area. Such a broad beam may be used to reduce target position sensitivity, which may be advantageous where a degree of variability is expected or desired, such as providing a large touch sensitive area and/or additional touch sensitive area on or around the button.

As depicted in fig. 2104 and 2106, the beam can be adjusted as desired, and a tradeoff can be made between more focused transmission pressure on the target and a larger effective area with lower target sensitivity.

Fig. 2106 depicts a multi-receiver configuration of an activated transducer. In this configuration, one set of transmitting transducers may emit an ultrasonic signal that may be reflected and received at two or more sets of receiving transducers. For example, a first set of receiving transducers (e.g., one or more transducers) may be positioned to receive ultrasonic signals that have been reflected within a first zone, and a second set of receiving transducers may be positioned to receive ultrasonic signals that have been reflected within a second zone. As depicted in diagram 2106, the first region may be smaller and enclosed within the second region.

By performing beamforming using an array of ultrasound transducers, energy may be limited to a particular region of interest, and thus the ultrasound transducers may be less sensitive to regions outside the region of interest. Micromechanical ultrasound transducer for touch input

In some cases, the ultrasonic input device may include an ultrasonic sensor including a Micromachined Ultrasonic Transducer (MUT), such as a piezoelectric micromachined ultrasonic transducer (pMUT) or a capacitive micromachined ultrasonic transducer (cMUT). Other types of transducers besides pmuts and cmuts may include bulk piezoelectric transducers that are integrated (i.e., fabricated directly on CMOS) and non-integrated (i.e., fabricated separately and then assembled with an on-board CMOS chip or directly in communication with a microprocessor/microcontroller or Field Programmable Gate Array (FPGA) or any hardware with inter-integrated circuit (I2C) or Serial Peripheral Interface (SPI) communication capabilities). As described herein, a micromechanical ultrasonic transducer for touch input may allow for an improved energy sensing area. In addition, the MUT may also reduce the overall power consumption of the ultrasound input device.

A. improved energy sensing region

Fig. 22 is a set of graphs 2202, 2204 depicting the operational modes of a micromechanical ultrasound transducer according to certain aspects of the present disclosure as compared to a standard body transducer, depicted as average displacements for different frequencies. The graphs 2202, 2204 contain lines depicting average displacement over a frequency range of 0.5MHz to 5MHz, and axisymmetric cross-sectional visualization illustrations of transducer patterns.

Graph 2202 depicts the mode of operation of a standard body transducer (e.g., a standard piezoelectric transducer) operating from 0.5MHz to 5 MHz. During this relatively small frequency range, the number of peaks in the average displacement and the overall range of each of these peaks is significantly dependent on various combinations of body modes, shear modes, bending modes, surface acoustic modes, and other modes experienced by the body transducer. As a result, shear waves and surface acoustic waves in different directions can be generated in addition to the normal longitudinal waves of interest. Thus, sensors utilizing such a body transducer may have uncontrollable beam patterns, detrimental crosstalk, more multipath reflections from different angles from different modes, spurious modes and notches in the spectrum, less clean received signals, more energy wasted on undesired modes, and other such problems.

Instead, graph 2204 depicts a uniform and predictable bending mode shape that exists in a MUT (e.g., pMUT) over the same frequency span, and that is used to emit longitudinal sound waves in the normal direction toward the outer surface of the stack. As a result, the MUT is able to implement much improved performance over standard bulk transducers.

Due to the nature of the ultrasound input device, it is desirable to detect ultrasound reflections based on longitudinal sound waves (e.g., propagating in a direction orthogonal to the sensor). In the case of MUTs used as touch input ultrasound transducers, MUTs perform particularly well due to their inherent ability to perform bending mode displacements to generate such longitudinal acoustic waves without inadvertently generating many (if any) transverse or other undesirable waves. Thus, MUTs may be used for beamforming operations, such as those described herein, may be tightly packed into a sensor array, may be used with fewer filtering devices, and may use the same or less power to achieve a higher signal-to-noise ratio than would be the case with a standard bulk piezoelectric transducer.

Fig. 23 is a set of schematic side views 2302, 2304, 2306 depicting modes of operation of a standard body transducer for ultrasonic touch detection. When a standard bulk transducer is used for ultrasonic touch detection, driving the transducer to transmit a signal may cause the transducer to displace in multiple modes of operation, which may cause erroneous signals to be transmitted into the receiving medium (e.g., stack).

Graph 2302 depicts a longitudinal mode of operation in which the driving of the transducer initiates a longitudinal signal in a direction orthogonal to the sensor. However, the same or similar driving of the transducer in fig. 2302 may cause lateral displacement as depicted in fig. 2304. Such lateral displacement (e.g., due to a lateral mode of operation) may initiate a lateral signal that is carried into the receiving medium in a direction different from the normal of the sensor, or may result in undesired normal traveling shear waves. As a result, driving the body transducer may generate a signal as depicted in fig. 2306, where both normal and abnormal signals propagate from the body transducer. Since the sensing region (e.g., the region desired to be sensed) is typically located directly above the stack, an abnormal signal may cause interference with the signal received from the sensing region. Furthermore, the bulk transducer may be susceptible to physical topology of the region of the stack proximate to the sensing region, as different topologies may initiate different reflections of the abnormal signal, which may result in false positives or false negatives.

Fig. 24 is a set of schematic side views 2402, 2404 depicting lateral signal rejection of a micromechanical ultrasound transducer, according to certain aspects of the present disclosure. Figure 2402 is a close-up view of a single transducer of a MUT array. The transducer may be composed of multiple layers, including piezoelectric layers that, when excited, may initiate bending displacements to cause longitudinal waves to be transmitted in a direction normal to the sensor (e.g., a direction normal to the MUT surface).

Fig. 2404 depicts an ultrasound input device using a sensor with MUT. The ultrasonic input device is depicted as being coupled to an aluminum layer and a glass layer, but any other stacked configuration may be used. The nature of the MUT may allow ultrasonic signals to be transmitted in a direction orthogonal to the sensor while minimizing or eliminating any signals that would otherwise propagate in a direction that is non-orthogonal or substantially non-orthogonal to the sensor if a bulk transducer were used. Thus, using MUTs as transducers in an ultrasound input device may help focus energy into a desired sensing region and reduce sensitivity to false positives or false negatives due to false reflections.

B. Convenience of driving

In addition to the benefits of MUTs described above, MUTs may also reduce the overall power consumption of an ultrasound input device when used with an ultrasound input device. Since the power required to drive the transducer is proportional to its capacitance times the square of its voltage, the low capacitance level of the MUT array (e.g., picofarad magnitude) results in a much lower power consumption than the relatively higher capacitance level of an equivalent standard body transducer (e.g., nanofarad magnitude, three orders of magnitude greater than picofarad).

V. ultrasonic signal processing

The reflected ultrasound signals may be processed to generate an image and determine a range to the object. Embodiments described herein may process reflected ultrasonic signals to determine whether an object is in contact with a surface.

A. detecting touch input by digitizing reflected signals

Fig. 25 is a schematic flow chart 2500 for processing ultrasonic signals transmitted and received by an ultrasonic input device in accordance with certain aspects of the present disclosure. Flowchart 2500 includes transmitting and receiving ultrasound signals as shown in first graph 2502. The first graph 2502 shows analog measurements of a first signal 2503 for a transmitted ultrasonic signal associated with an ultrasonic input device and a set of subsequent signals 2504A, 2504B, 2504C, 2504D, 2504E for a set of reflected ultrasonic signals associated with the ultrasonic input device. The first signal 2503 and the subsequent signal 2504 may be measured using a high speed ADC 2506 to digitize the signals.

The output of high speed ADC 2506 is shown in a second graph 2508. Second graph 2508 includes a first digital representation 2510 of an emitted ultrasound signal and a subsequent digital representation 2512A, 2512B, 2512C, 2512D, 2512E of a reflected ultrasound signal associated with an ultrasound input device. The first digital representation 2510 and subsequent digital representations 2512A, 2512B, 2512C, 2512D, 2512E may be processed by a digital processing module 2514 embedded in and/or coupled to an ultrasound input device. The digital processing module 2514 may demodulate a digital representation of the data to extract touch input information. For example, the digital processing module may process one or more of the subsequent digital representations 2512A, 2512B, 2512C, 2512D, 2512E to determine that the amplitude of the second digital representation is below a threshold associated with an object contacting the surface of the ultrasound input device.

B. detecting touch input using energy integration

Fig. 26 is a schematic flow chart 2600 for processing ultrasonic signals transmitted and received by an ultrasonic input device using energy integration, according to certain aspects of the present disclosure. The flowchart 2600 includes transmitting and receiving an ultrasonic signal as shown in a first graph 2602. The first graph 2602 shows an analog measurement of an emitted ultrasound signal 2603 and a set of subsequent signals 2604A, 2604B, 2604C, 2604D, 2604E for a set of reflected ultrasound signals associated with an ultrasound input device. The flowchart 2600 may include an ultrasound input device having an analog circuit that includes a rectifier 2606 to rectify subsequent signals 2604A, 2604B, 2604C, 2604D, 2604E.

The second graph 2608 shows a first signal 2603 and a set of rectified signals 2610A, 2610B, 2610C, 2610D, 2610E, each corresponding to a respective reflected ultrasound signal in the set of reflected ultrasound signals. The rectified signals 2610A, 2610B, 2610C, 2610D, 2610E may be processed by an analog integrator 2612 to output a Direct Current (DC) signal 2613, as shown in a third graph 2614, which is proportional to the amplitude of the reflected ultrasonic signal. The DC signal 2613 may be determined using an energy measurement window 2616. The DC signal 2613 may represent an energy value associated with the energy of the received signal measured during an energy window 2616. The DC signal 2613 may be processed by a low speed ADC 2618. The DC signal 2613 is output by the rectifier 2606 and the integrator 2612 removes the need to generate a high frequency digital output and as a result, the low speed ADC can use lower power and can be fabricated on a smaller chip area.

Fig. 27 is a schematic, example flowchart 2700 for processing ultrasound signals transmitted and received by an ultrasound input device using energy integration, in accordance with certain aspects of the present disclosure. Flowchart 2700 includes transmitting and receiving ultrasound signals as shown in first graph 2702. The first graph 2702 shows an analog measurement of a first signal 2703 for a transmitted ultrasound signal and an analog measurement of a subsequent set of signals 2704A, 2704B, 2704C, 2704D, 2704E for a reflected set of ultrasound signals associated with an ultrasound input device. Flowchart 2700 may include an ultrasound input device with an analog summing or integration circuit 2720 and a summed voltage output 2722.

The second graph 2708 shows the first signal 2703 and a set of energy signals 2710A, 2710B, 2710C, 2710D, 2710E, which respectively correspond to the energy of the corresponding reflected ultrasound signals in the set of reflected ultrasound signals. For illustration purposes, the set of energy signals 2710A, 2710B, 2710C, 2710D, 2710E are depicted in solid lines, overlapping the set of subsequent signals 2704A, 2704B, 2704C, 2704D, 2704E from the first graph 2702 shown in dashed lines.

Summing or integrating circuit 2720 may receive the set of energy signals 2710A, 2710B, 2710C, 2710D, 2710E from within energy window 2716. The summing or integration circuit 2720 may generate a voltage output 2722 that is an analog value representing the summed/integrated energy within the energy measurement window 2716.

In some cases, an optional negative DC charging circuit 2724 may be applied to the summing or integrating circuit 2720 to counteract information not associated with a touch event. Since the touch event is identified based on differences between the signals received during the non-contact state and the signals received during the contact state, there is some information (e.g., baseline signals) within the set of subsequent signals 2704A, 2704B, 2704C, 2704D, 2704E that is not associated with these differences. Removing such baseline signals may result in a more efficient sampling range during analog-to-digital conversion. This correction is difficult to apply because removing such an analog baseline signal in the set of subsequent signals 2704A, 2704B, 2704C, 2704D, 2704E would require precise phase alignment. However, as depicted in fig. 27, an optional negative DC charging circuit 2724 applied to the summing or integrating circuit 2720 may counteract a particular amount of energy associated with the baseline signal or a portion thereof, thereby increasing the amount of active range available for analog-to-digital conversion. In this case, the voltage output 2722 may be proportional to the energy of the signal minus the energy of the negative DC charging circuit 2724.

Voltage output 2722 may be processed by low speed ADC 2718. The voltage output 2722 of the summed/integrated energy within the energy measurement window 2716 may eliminate the need to generate a high frequency digital output and, as a result, a low speed ADC may use less power and may be fabricated on a smaller chip area.

Fig. 28 is a schematic flow chart 2800 for processing ultrasonic signals transmitted and received by an ultrasonic input device using energy integration by absolute value summation, according to certain aspects of the present disclosure. Flowchart 2800 may be one technique for implementing flowchart 2700 of fig. 27. The flow chart 2800 includes transmitting and receiving an ultrasonic signal as shown in a first graph 2802. A first graph 2802 shows analog measurements of a first signal of an emitted ultrasound signal and a subsequent signal of a set of reflected ultrasound signals associated with an ultrasound input device. The first graph 2802 may depict voltage (e.g., V (t)) as a function of time. The first graph 2802 may be the first graph 2702 of fig. 27. Flowchart 2800 may include an ultrasound input device with analog sampling circuit 2806, absolute value circuit 2814, analog accumulator 2824, and summed voltage output 2828.

The set of subsequent signals from the first graph 2802 may pass through an analog sampling circuit 2806 to produce a sampled first signal 2810 and a set of sampled subsequent signals 2812A, 2812B, 2812C, 2812D, 2812E as depicted in the second graph 2808. The first signal may correspond to an initially transmitted ultrasonic wave. The second graph 2808 may depict voltage as a function of sample (e.g., V (n), where n is the number of samples). The sampled subsequent signals 2812A, 2812B, 2812C, 2812D, 2812E may be passed to an absolute value circuit 2814, which may generate a set of energy signals 2820A, 2820B, 2820C, 2820D, 2820E as depicted in a third graph 2816. Third graph 2816 may depict an absolute value of voltage (e.g., |v (n) |) as a function of sample. Absolute value circuit 2814 may pass all zero values or positive values of the subsequent signals 2812A, 2812B, 2812C, 2812D, 2812E of the set of samples and reverse the polarity of all negative values. The sampled first signal 2818 is also shown in the third graph 2816, and the sampled first signal 2818 may be similar to the sampled first signal 2810.

A switched capacitor analog accumulator 2824 may be used to sum the set of energy signals 2820A, 2820B, 2820C, 2820D, 2820E from within the energy window 2822. The switched capacitor analog accumulator may generate a voltage output 2828 that is an analog value representing the sum of the energies within the energy window 2822. In some cases, an analog integrator may be used instead of an accumulator.

In some cases, an optional negative timing DC charging circuit 2826 may be applied to the switched capacitor analog accumulator 2824 to counteract information not associated with a touch event. Since sampling circuit 2806 is clocked according to the sampling rate, optional negative-timing DC charging circuit 2826 may be clocked at the same rate to ensure that bias voltages are applied at appropriate intervals corresponding to samples of sampled subsequent signals 2812A, 2812B, 2812C, 2812D, 2812E. When an optional negative timing DC charging circuit 2826 is used, the voltage output 2828 may be proportional to the energy of the signal minus the energy of the negative timing DC charging circuit 2826.

The voltage output 2828 may be processed by the low speed ADC 2830. The voltage output 2828 of the summed energy within the energy measurement window 2822 may eliminate the need to generate a high frequency digital output and as a result, the low speed ADC may use lower power and may be fabricated on a smaller chip area.

Fig. 29 is a schematic flow chart 2900 for processing ultrasonic signals transmitted and received by an ultrasonic input device using energy integration through self-mixing and integration in accordance with certain aspects of the present disclosure. Flowchart 2900 may be one technique for implementing flowchart 2700 of fig. 27. Flowchart 2900 includes transmitting and receiving ultrasound signals, as shown in a first graph 2902. A first graph 2902 shows analog measurements of a first signal for transmitting an ultrasonic signal and a set of subsequent signals for a set of reflected ultrasonic signals associated with an ultrasonic input device. The first graph 2802 may depict voltage (e.g., V (t)) as a function of time. The first graph 2902 may be the first graph 2702 of fig. 27. Flowchart 2900 may include an ultrasonic input device having self-mixing circuit 2906, analog integrator circuit 2920, and integrated voltage output 2922.

The set of subsequent signals from the first graph 2902 may pass through the self-mixing circuit 2906 to generate a set of squared subsequent signals 2910A, 2910B, 2910C, 2910D, 2910E, as depicted in the second graph 2908. Self-mixing circuit 2906 may effectively self-multiply each analog value over time. As a result, the second graph 2908 may depict the square voltage (e.g., V ² (t)) as a function of time. Due to the nature of the square, and thus the self-mixing circuit 2906, the set of squared subsequent signals 2910A, 2910B, 2910C, 2910D, 2910E will always be positive.

The set of squared subsequent signals 2910A, 2910B, 2910C, 2910D, 2910E may be passed to an analog integrator circuit 2920. The analog integrator circuit 2920 may integrate the set of squared subsequent signals 2910A, 2910B, 2910C, 2910D, 2910E within the energy measurement window 2916 to generate an integrated voltage output 2922. The integrated voltage output 2922 may be an analog representation of the total energy over time within the energy measurement window 2916. In some cases, an accumulator may be used in place of the analog integrator circuit 2920.

In some cases, an optional negative bias current circuit 2924 may be applied to the analog integrator circuit 2920 to counteract information not associated with a touch event. The negative bias current circuit 2924 may constantly drain charge out of the analog integrator circuit 2920 during integration. When an optional negative bias current circuit 2924 is used, the voltage output 2922 may be proportional to the energy of the signal minus the energy of the negative bias current circuit 2924.

The voltage output 2922 may be processed by a low speed ADC 2926. The voltage output 2922 of the integrated energy within the energy measurement window 2916 may eliminate the need to generate a high frequency digital output, and as a result, the low speed ADC may use lower power and may be fabricated on a smaller chip area.

Fig. 30 is a schematic circuit diagram depicting an analog integrator 3000 with a negative bias current in accordance with certain aspects of the present disclosure. The analog integrator 3000 negative bias may be the analog integrator circuit 2920 of fig. 29 and the optional negative bias current circuit 2924.

Analog integrator 3000 may receive an input voltage (V _in) through resistor (R _in) to obtain an input current (I _in). The capacitor (C) may be charged by a charging current (I _f) to generate an integrated signal, which may feed the voltage output (V _out). The article (a) is an operational amplifier. A negative bias current (I _bias) may be applied at point X to drain charge out of analog integrator 3000, resulting in a reduced charge current (I _f). Thus, the charging current may be calculated as I _f＝I_in-I_bias.

C. energy measurement windowing

FIG. 31 is a schematic flow chart for processing ultrasonic signals depicting the reduced impact of time-of-flight variations on touch input detection within an energy measurement window, in accordance with certain aspects of the present disclosure. In an ultrasound imaging system or a proximity detection system, the precise time of flight is critical to determining the distance of an object in the field of view from the ultrasound transducer. In contrast to imaging and proximity systems, the distance to the first and second surfaces of the material layer in the ultrasound input device may be provided and touch input may be detected without regard to time-of-flight variations. Fig. 31 shows a first plot 3102 in which a first set of reflected ultrasonic signals 3104 is received beginning at a first time 3106, and a second plot 3108 in which a second set of reflected ultrasonic signals 3110 is received at a second time 3112. The first signal 3103 may be associated with the transmitted ultrasonic signal that occurs before the first time 3106 and the second time 3112 of the first graph 3102 and the second graph 3108, respectively. The first set of reflected ultrasonic signals 3104 pass through an energy accumulator or integrator circuit 3120 to generate an output voltage 3122 (e.g., V _sum1) that can be fed into the low speed ADC 3124 and processed to obtain an output value 3118 (e.g., 3000LSB, where LSB represents the least significant bit). The second set of reflected ultrasound signals 3110 is passed through an energy accumulator or integrator circuit 3120 to generate an output voltage 3123 (e.g., V _sum2) that can be fed to the low speed ADC 3124 and processed to obtain an output value 3119 (e.g., 3000LSB, where LSB represents the least significant bit). The output values 3118, 3119 may represent pulse reflected energy during the energy measurement window 3116 of the graphs 3102, 3108. Although the start times of the first and second sets of reflected ultrasonic signals 3104, 3110 are different (e.g., first time 3106 and second time 3112), the output values 3118, 3119 may be the same or substantially the same because the entire first set of reflected ultrasonic signals 3104 and the entire second set of reflected ultrasonic signals 3110 are each fitted (fit) within the energy measurement window 3116.

Thus, the ultrasound input device may be insensitive to time of flight at least to some extent (e.g., within an energy measurement window). In some cases, advanced windowing techniques such as those disclosed herein may further improve the insensitivity of the ultrasound input device to time of flight. As a result, the surface of the ultrasound input device (e.g., the material layer) need not be perfectly flat and/or the alignment of the ultrasound input device with respect to the material (e.g., the material layer) need not be exactly 90 ° (e.g., the angle between the propagation direction of the ultrasound transducer and the surface of the material layer). Furthermore, insensitivity to time of flight may allow for some degree of insensitivity to varying refractive indices (e.g., layers of material having a slightly non-uniform refractive index throughout) through which the ultrasound signal passes.

For example, as shown in fig. 26-29 and 31, the energy of the reflected ultrasonic signals (e.g., reflected echoes and standing waves) is summed or integrated over an energy measurement window. This energy is related to the condition of the touch input and thus can be used for input touch detection. The energy measurement window 3116 may be sized to include a pulse time of the ultrasonic signal and take into account variations in time of flight due to temperature, stack variations (e.g., variations in the materials comprising the ultrasonic input device), and the like. The energy measurement window 3116 may reduce errors due to time of flight variations. The ultrasonic touch device may determine an input touch contact based on a particular threshold.

FIG. 32 is a schematic simplified flow chart for processing ultrasonic signals depicting the enhanced effect of reflected ultrasonic signal time-of-flight variations on touch input detection outside an energy measurement window, in accordance with certain aspects of the present disclosure. Fig. 32 illustrates a first graph 3202 in which a first set of reflected ultrasound signals 3204 is received beginning at a first time 3206, and a second graph 3208 in which a second set of reflected ultrasound signals 3210 is received at a second time 3212. The first signal 3203 may be associated with a transmitted ultrasound signal that occurs before a first time 3206 and a second time 3212 of the first graph 3202 and the second graph 3208, respectively. The first set of reflected ultrasound signals 3204 may be processed as disclosed herein to obtain an output value 3218 (e.g., 3000 LSBs, where LSBs represent the least significant bits). The second set of reflected ultrasonic signals 3210 may be processed as disclosed herein to obtain an output value 3219 (e.g., 2500 LSBs, where LSBs represent the least significant bits). The output values 3218, 3219 may represent pulse reflected energy during an energy measurement window 3216 of the graphs 3202, 3208.

As shown in fig. 32, because substantially all of the first set of reflected ultrasound signals 3204 fit within energy measurement window 3216, but a smaller portion of the second set of reflected ultrasound signals 3210 fit within energy measurement window 3216, output value 3218 is greater than output value 3219. As depicted in fig. 32, the output values 3218, 3219 differ by 500LSB. If the reflected ultrasonic signal falls outside of the energy measurement window 3216, some of the measured pulses may be cut off from being measured and thus the ultrasonic input device may be susceptible to time-of-flight variations (e.g., variations that would cause a difference in the first time 3206 and the second time 3212).

FIG. 33 is a schematic flow diagram for processing ultrasonic signals depicting the minimal effect of reflected ultrasonic signal time-of-flight variations on touch input detection outside an energy measurement window with window shaping, in accordance with certain aspects of the present disclosure. Fig. 33 shows a first plot 3302, wherein a first set of reflected ultrasound signals 3304 is received beginning at a first time 3306, and a second plot 3308, wherein a second set of reflected ultrasound signals 3310 is received at a second time 3312. The first signal 3303 may be associated with an emitted ultrasound signal that occurs before a first time 3306 and a second time 3312 of the first graph 3302 and the second graph 3308, respectively. The first set of reflected ultrasound signals 3304 may be processed as disclosed herein to obtain output values 3318 (e.g., 2500 LSBs, where LSBs represent least significant bits). The second set of reflected ultrasound signals 3310 may be processed as disclosed herein to obtain output values 3319 (e.g., 2450 LSBs, where LSBs represent the least significant bits). The output values 3318, 3319 may represent pulse reflected energy during an energy measurement window 3316 of the graphs 3302, 3308.

Unlike fig. 32, energy measurement window envelope 3320 is used in conjunction with energy measurement window 3316. The energy measurement window envelope 3320 scales portions of the signal within the energy measurement window 3316 such that portions near the edge of the energy measurement window 3316 have less weight than portions near the center of the energy measurement window 3316. Thus, despite small variations near the ends of the energy measurement window 3316, the resulting output value will be based primarily on the signal measured within the center of the energy measurement window 3316. The energy measurement window envelope 3320 is depicted in fig. 33 as having a particular bell shape, although any suitable shape may be used, including symmetrical and asymmetrical shapes. The vertical extension of the energy measurement window envelope 3320 as depicted in fig. 33 may represent any suitable scaling, such as 0% to 100%. In some cases, the energy measurement window envelope 3320 may include a value that amplifies the signal near the center of the energy measurement window 3316 to more than 100% of the original signal, such as at that time.

As depicted in fig. 33, since the signals (e.g., the first set of reflected ultrasound signals 3304 and the second set of reflected ultrasound signals 3310) are weighted using the energy measurement window envelope 3320, portions of the signal closest to the center of the energy measurement window 3316 are given greater weight than portions closest to the edges of the energy measurement window 3316, thereby weakening (de-emphasizing) any portions resected by the beginning or end of the energy measurement window 3316. As a result, the output values 3318 and 3319 are closer than the output values 3218 and 3219 of fig. 32. As depicted in fig. 33, the output values 3318, 3319 differ by only 50LSB. Thus, as a result of the energy measurement window envelope 3320, the ultrasound input device may become less susceptible to time-of-flight variations.

Fig. 34 is a schematic circuit diagram depicting a window shaping circuit 3400 in accordance with certain aspects of the present disclosure. The window shaping circuit 3400 may generate an energy measurement window having an energy measurement window envelope (e.g., energy measurement window 3316 having energy measurement window envelope 3320 of fig. 33). With the addition of the adjustable capacitor 3402, the window shaping circuit 3400 may operate as a conventional analog accumulator circuit. The tunable capacitor 3402 may take any suitable form, such as a switching ladder of capacitors of different sizes. The selection of the capacitor size for the adjustable capacitor 3402 may cause the gain on the analog accumulator circuit to adjust over time. In some cases, the tunable capacitor 3402 may be driven by a clock 3404 or other source to determine when to change capacitance. In some cases, the adjustable capacitor 3402 may be used with an analog sampling circuit (such as analog sampling circuit 2806 of fig. 28), and the adjustable capacitor 3402 may vary with different sample numbers (e.g., n of V (n)).

Fig. 35 is a schematic diagram depicting a flow 3500 for processing an ultrasonic signal to detect touch input using an amplitude of a reflected ultrasonic signal, in accordance with certain aspects of the disclosure. Fig. 35 shows no touch input 3504 and an ultrasonic input device 3502 with a touch input 3506. A first graph 3508 associated with the ultrasound input device 3502 without the touch input 3504 shows the transmitted signal 3510 and the first set of reflected signals 3512. The first set of reflected signals 3512 may be processed to generate an output voltage 3530 (e.g., V _sum1) associated with the first set of reflected signals 3512, which may be provided to a low-speed ADC 3534 and further processed to generate a first output 3536. The first output 3536 may represent energy of a first set of reflected signals 3512 within an energy measurement window envelope 3516. The second graph 3520 shows a transmitted signal 3522 and a second set of reflected signals 3524. As disclosed herein, the second set of reflected signals 3524 can be processed to generate an output voltage 3532 (e.g., V _sum2), which can be provided to the low-speed ADC 3534 and further processed to generate a second output 3538. The second output 3538 may represent energy of a second set of reflected signals 3524 within the energy measurement window envelope 3516.

An energy measurement window envelope 3516 (e.g., an envelope similar to the energy measurement window envelope 3320 of fig. 33) may be applied to the first set of reflected signals 3512 and the second set of reflected signals 3524. In some embodiments, the energy measurement window envelope 3516 may be applied to the first set of reflected signals 3512 and the second set of reflected signals 3524 to attenuate signals at the edges of the energy measurement window envelope 3516.

The first output 3536 and the second output 3538 can be compared to determine whether a touch input (e.g., a touch event) has occurred. For example, if the second output 3538 is lower than the first output 3536 by a predetermined amount and/or if the second output 3538 is lower than a threshold, the ultrasound input device 3502 can generate a signal indicating that a touch input is present on a surface. Since the output voltages 3530, 3532 are indicative of the first output 3536 and the second output 3520, respectively, the output voltages 3530, 3532 can be used to determine whether a touch input has occurred. In some embodiments, only a single output, such as first output 3518, may be compared to a reference value. The reference value may be established at the time of manufacture and/or updated based on a background characteristic (such as temperature) measured by or transmitted to the device.

The techniques described with reference to fig. 35 may be used to generate output signals from the ultrasound input device 3502, although other techniques may be used. Any technique that can convert the signals associated with the first set of reflected signals 3512 or the second set of reflected signals 3524 into a measure of total energy can be used.

Fig. 36 is a graph 3600 depicting a simplified example energy signal 3614 in accordance with certain aspects of the present disclosure. When the ultrasound input system processes the input ultrasound signal received by the ultrasound transducer, the ultrasound input system may convert the ultrasound signal into an energy signal 3614. The energy signal 3614 may represent the overall energy associated with the input ultrasound signal. For example, as depicted in fig. 35, the signals depicted in graphs 3508 and 3520 may be converted to outputs 3536 and 3538. These outputs may be recorded, plotted or output as energy signals 3614 over time. The output 3536 associated with a touchless event can generally be considered as an overall higher region of the energy signal, while the output 3538 associated with a touch event can generally be considered as an overall lower region of the energy signal. It should be appreciated that the sustained output of the processed ultrasound signal may be used to generate an energy signal that may then be used to determine whether a touch event occurred at a point in time. The energy signal 3614 of fig. 36 is simplified for illustrative purposes only.

D. touch input error prevention

Fig. 37 is a chart 3700 depicting reflected ultrasonic signal measurements made using an ultrasonic input device and illustrating techniques for improving touch input detection in accordance with certain aspects of the present disclosure. The sensor readouts (e.g., DC signals or other sensor data) determined by the ultrasound input device may be measured continuously or at a particular frequency according to the present application. In some embodiments, the sensor readout may be measured at a frequency of 100 Hz. A single measurement 3702 may correspond to an energy measurement within an energy measurement window. One or more individual measurements may be used to determine the current state 3706. The current state may be defined by the current individual measurement 3702 or by a best fit line based on two or more individual measurements. In some embodiments, the best fit line may be calculated using a least squares method. Multiple individual measurements may be used to determine the moving average threshold 3704.

The current state 3706 and the moving average threshold 3704 may be used to detect a touch event. The moving average threshold 3704 may be used to determine a sudden signal drop that may trigger a touch input event. For example, the system may detect a "hand touch" effect only when a "rapid signal change" 3708 from the current state 3706 is detected. The rapid signal change 3708 may be associated with a sudden signal drop across all or many channels and may be considered a touch input event. The threshold for detecting the rapid signal change 3708 may be a moving average threshold 3704 (dynamic threshold) when no hand touch event is detected. In some embodiments, the rapid signal change 3708 may be a preprogrammed static threshold. The rapid signal change 3708 event may trigger a touch input event and cause the ultrasonic input device to generate a signal indicative of a touch input on a surface of the device. For the rapid signal change 3708 event, multiple measurements 3710 are made to ensure that the signal does actually drop and does not jump back up, such as to its original value. For example, a user's re-press may result in a declining sensor readout, but will still provide a continuous signal. During the multiple measurements 3710, if the signal quickly returns to a higher value (such as the value previously seen prior to the suspected touch event), the ultrasonic input device may recognize the temporary signal drop as a false touch event without classifying it as a touch event. The multiple measurements 3710 may occur in a very short time frame (e.g., on the order of tens or hundreds of milliseconds). In some embodiments, a "gradual signal change" may be considered a temperature change rather than a hand touch event, since the moving average will be adjusted with each individual measurement 3702 at a rate based on the number of measurements used to determine the moving average.

In some cases, threshold 3704 may be based on calculations other than moving average calculations. In some cases, threshold 3704 is merely a function of past history (e.g., historical measurements), such as a function of past x measurements. In some cases, past measurements may be weighted, such as where the most recent measurement has a greater weight than a measurement taken more recently. In this case, the response time of the ultrasound input device may be adjusted based on the weights of the past x measurements. For example, the Threshold may be calculated as a function of the historical value based on Threshold = f (X [ n-1], X [ n-2],. X [ n-m ]) (where X [ n ] is the nth sensor readout (or current sensor readout)). In another example, the Threshold may be calculated as a function of the weighted history value based on Threshold = w ₁X[n-1]+w₂X[n-2],…,w_m X [ n-m ] (where w _n is the weight parameter read by the nth sensor). In some cases, the weight parameters may be trained using machine learning, such as described in more detail herein.

In some cases, the determination may be made using the slope of a set of measurements (such as the slope of the current measurement and a certain number of past measurements) in addition to or instead of determining the rapid signal change 3708 based on the measurements themselves.

Fig. 38 is a chart 3800 depicting reflected ultrasound signal measurements made using an ultrasound input device and showing additional techniques for improving touch input detection, in accordance with certain aspects of the present disclosure. A portion of graph 3800 is depicted as graph 3700 of fig. 37. Graph 3800 shows that the signal may change over time due to various factors (such as temperature changes), however the ultrasound input device may be able to discern that these changes are not touch events. However, a sudden signal drop between successive measurements may indicate a touch event. Current state 3806 may be similar to current state 3706 of fig. 37. The moving average threshold 3804 is similar to the threshold 3704 of fig. 37. The threshold 3804 is based in part on a previously measured moving average of the current state 3806, such as a moving average of previous measurements that is offset by a given amount. This type of threshold 3804 may be referred to as a dynamic threshold, although other threshold techniques may be used.

At region 3816, a touch event occurs. When a touch event occurs, the current state 3806 drops rapidly. As depicted in the labeled section of chart 3800, various measurements 3802 are shown. Each measurement 3802 may be separated in time based on the measurement frequency. For example, each measurement 3802 may be separated by 0.01 seconds (e.g., at 100 Hz), although other frequencies may be used. A sudden drop may be detected between two or more consecutive measurements 3802. A touch event may be considered to have occurred when the sudden drop in current state 3806 falls below threshold 3804. Region 3817 depicts another touch event.

At regions 3818 and 3820, gradual changes in the temperature of the ultrasonic sensor and the temperature of the surface to which the sensor is coupled may cause a gradual change in the current state 3806. Because of the relatively slow change in current state 3806, threshold 3804 based on the moving average of current state 3806 will also change. Since the threshold 3804 is able to compensate for slow changes in the current state 3806, such as changes due to temperature changes, these slow changes in the current state 3806 do not exceed the threshold 3804 and therefore do not trigger a touch event. Furthermore, since the threshold 3804 is dynamically updated, the threshold 3804 can operate properly at different temperatures. In some cases, the changes in the current state 3806 due to temperature changes may even be greater than the contrast caused by an actual hand touch, but because these temperature changes are much slower than the changes in the current state 3806 due to touch events, they are not detected as touch events.

VI Multi-frequency touch detection

Fig. 39 is a graph depicting the temperature dependence of a reflected ultrasound signal, in accordance with certain aspects of the present disclosure. The reflected ultrasonic signals received by the ultrasonic input device may include the primary signal 3902 and any undesired signals 3904. The primary signal travels along a first path through the layer of material and is associated with a first time of flight (TOF), and any undesired signals 3904 travel along a second path through the layer of material and are associated with a second TOF. The speed of sound in the material layer depends on the temperature of the material layer. As the speed of sound changes as a result of temperature changes, the main signal 3902 and the undesired signal 3904 travel through different acoustic paths, and the associated first and second TOF's change by different amounts accordingly. This produces a net TOF difference Δt (T) 3906 between the main signal 3902 and the undesired signal 3904, which varies with temperature T. Which is then converted into a phase delay difference ΔΦ (T) between the main signal 3902 and the undesired signal 3904. And thus a different integrated signal strength difference Dout (T) is produced, as depicted by line 3910.

FIG. 40 is a set of graphs depicting TOF temperature dependence of two frequency methods of detecting touch input, according to certain aspects of the present disclosure. These graphs may be similar to the graph of fig. 39, in a multi-frequency ultrasound input device, different frequencies will have different temperature effects, resulting in different TOF for each signal. The multi-frequency ultrasonic input device may process a "finger touch" (e.g., a touch event) when a signal drop is detected in a threshold number of frequency channels. For example, two different methods may detect whether a finger touches the ultrasonic touch input device, and the device may only process touch events when both methods agree that a finger touch has been detected.

In a multi-frequency ultrasonic touch input device, a first signal 4002 at a first frequency and a second signal 4004 at a second frequency have different background and temperature drift characteristics. For example, when the temperature changes, the first signal 4002 and the second signal 4004 experience the same Δt (T). The same Δt (T) will translate into a different phase delay for each frequency due to different temperature drift characteristics. For example, the first signal 4002 will have a first phase delay Δφ+.1 (T) 4006 and the second signal 4004 will have a second phase delay Δφ+.2 (T) 4008. The resulting phase delay differences may result in two different ADC output value patterns at temperatures Dout+.1 (T) and Dout+.2 (T), respectively, as depicted by lines 4010, 4012.

Accordingly, signal drop can be measured in multiple frequencies to improve touch detection reliability and reduce false trigger detection. Touch input events can be handled if all frequency channels detect a sudden signal drop. Multiple measurements can occur very quickly (< 1 ms) to ensure that sudden signal drops are not due to temperature effects.

Multi-frequency ultrasonic touch input devices can avoid false triggers by reducing noise associated with environmental conditions. The touch input device can immediately execute a rapid pulse echo test to ensure that the touch event is true rather than false triggering due to noise. In some embodiments, multiple tests may occur within 1 ms.

Fig. 41 is a multi-part chart 4100 depicting reflected ultrasonic signal measurements across several frequencies using an ultrasonic input device and showing techniques for improving touch input detection in accordance with certain aspects of the disclosure. Different frequencies of the ultrasound signal may exhibit different variations due to temperature variations. Thus, by sensing using multiple ultrasonic frequencies, the ultrasonic input device may compare the suspected touch event to data from one or more other frequencies to ensure that the suspected touch event is confirmed by the one or more other frequencies. The use of multiple frequencies may reduce the error rate.

Line 4106 may represent an energy signal associated with a 100kHz frequency, line 4105 may represent an energy signal associated with a 1MHz frequency, and line 4107 may represent an energy signal associated with a 10MHz frequency. Line 4104 may represent a moving average threshold, such as threshold 3704 of fig. 37, which is depicted for illustration purposes only with respect to 100kHz frequencies, but there may be a corresponding threshold for each frequency used (e.g., 1MHz and 10 MHz). Although frequencies of 100kHz, 1MHz and 10MHz are used with respect to fig. 41, any other suitable frequency may be used. Although three different frequencies are used with respect to fig. 41, any number of different frequencies may be used, such as two or more than three. Touch events may be registered only if they are detected over all, most, or at least a threshold percentage of the different frequencies used for detection.

In some cases, instead of or in addition to driving the ultrasound input device at different frequencies, the ultrasound input device may drive an ultrasound array with different phase delays to generate different beam patterns. Since different beam patterns may have different temperature characteristics, different beam patterns may be used similarly to different frequencies to reduce errors and confirm suspected touch events.

Fig. 42 is a schematic plan view depicting a dual frequency PMUT 4200, according to certain aspects of the present disclosure. In some embodiments, a circular PMUT design may be fabricated to implement a multi-frequency transducer. The circular PMUT design may consist of multiple separate channels for transmission and reception per frequency. In some cases, multiple channels or transducers may be arranged concentrically. For example, the dual-frequency PMUT 4200 includes a first transmit/receive pair 4202 associated with a low frequency. The first transmit/receive pair 4202 may include a low frequency transmit loop 4204 and a low frequency receive loop 4206. The dual frequency PMUT 4200 also includes a second transmit/receive pair 4208 associated with high frequencies. Second transmit/receive pair 4208 may include a high frequency transmit loop 4210 and a low frequency receive loop 4212. In various embodiments, a circular PMUT design may include a range of frequencies from 2 to 10, and the frequency range may be from 1MHz to 10MHz. In some embodiments, frequencies less than 1MHz may be used, depending on the material layer and the particular application. A second PMUT array may be added for TOF measurements in the range of 1MHz-3 MHz. In some cases, the frequency range for any array may be from 30kHz to 50MHz.

Fig. 43 is a schematic plan view depicting a multi-frequency ultrasound input device 4300 having a square design, in accordance with certain aspects of the present disclosure. The square sensor design may consist of a square grid of multiple individual channels for transmission and reception per frequency. In some cases, one or more receive channels may be located between multiple transmit channels. In this case, the position of the receiving channel between the plurality of transmission channels can facilitate the reception and detection of the reflected signal. In an example, the multi-frequency ultrasound input device 4300 may include various low frequency transmitters 4302, low frequency receivers 4304, high frequency transmitters 4306, and high frequency receivers 4308. The square design may include a nesting pattern, such as the crisscross nesting pattern depicted in fig. 43, any other suitable pattern may be used. The various transmitters and receivers may be of any suitable frequency, such as between 30kHz and 50MHz, between 1MHz and 10MHz, or any other suitable range. It should be appreciated that the frequencies described with reference to fig. 43 may be applied to any suitable sensor array, for example, as described with reference to fig. 14A-14G.

VII feature extraction

Systems and methods according to embodiments may allow for extracting features from signals received, for example, by an ultrasound input device. The ultrasound input device is capable of extracting energy signals as well as features of physical characteristics.

A. Distinguishable energy signal

Fig. 44 is a set of three charts 4402, 4404, 4406 depicting example signals 4412, 4414, 4416 received by an ultrasound input system attributable to three different users, in accordance with certain aspects of the present disclosure. Each of the graphs 4402, 4404, 4406 depicts energy measurements over time associated with reflected signals detected by the ultrasound input device.

Signal 4412 of chart 4402 is an example of a dry finger that is pressed quickly with relatively little force. The dryness of the finger and the relatively small force show relatively little dip (dip) in the energy measurement during a touch event. The speed of the compressions is seen in the relatively short duration of the dip in the energy measurement.

Signal 4414 of chart 4404 is an example of a wet finger that is moderately pressed with a relatively heavy force. Both the wetness of the finger and the intensity of the compression can lead to a greater damping effect on the reflected signal and thus to a deeper dip in the energy measurement. The speed of the compression can be seen in a moderately wide dip in the energy measurement. Furthermore, the occurrence of the initial drop and the subsequent drop is more pronounced when the energy measurement is first sunk, which means that little time is taken to contact the surface before the full pressing force is initiated.

Signal 4416 of chart 4406 is an example of a touch event pattern in which the user taps the surface before pressing and initiating a full touch event. The initial dip in the energy measurement and the relatively long delay until the subsequent full dip indicate to the user to place the finger on the surface and wait a short time before pressing the finger.

While the signals 4412, 4414, 4416 may each be used to indicate a desired touch event due to the presence of sufficient dip in the energy measurement, each of the signals 4412, 4414, 4416 contains various features that are discernable. Examples of distinguishable features include the depth of the dip in the energy measurement, the width of the dip in the energy measurement, the presence of an initial dip prior to a subsequent and deeper dip in the energy measurement, the delay between an initial dip in the energy measurement and a subsequent and deeper dip, the rate of decrease and/or increase of the energy measurement into and out of the dip (e.g., the rate of change of the energy signal at the edge of the dip), or any other feature of the energy measurement.

By extracting various features from the energy measurement signal, different users can be distinguished or even identified to enable additional user-based advanced functionality. For example, after a training session, the ultrasound input system can distinguish between the first user and the second user due to a particular manner in which the user interacts with the ultrasound input device, such as the manner of touch (e.g., a quick tap or placement and pressing), the duration of the touch, the characteristics of the skin (e.g., the natural wetness or dryness of the finger), the intensity of the touch (e.g., a tap or a re-tap), or other characteristics that can be discerned from the measurement signals. Although characteristics can be distinguished from the energy measurement signals, they may not be easily perceived by a user because the energy measurement signals can be acquired at a high speed. Thus, the difference between the quick tap and the placement and press may be easily discernable from the energy measurement signal, but may not be discernable or readily discernable from a visual inspection of the touch action.

FIG. 45 is a set of graphs depicting energy measurement signals associated with a human finger, a water droplet, and placing a device on a table (e.g., placing an object on a sensor). For a human finger, the energy measurement signal inevitably has a slight movement or change, even during the duration of the touch event, which can be detected and identified to confirm that the human finger is initiating the touch event. For a droplet or drop, the energy measurement signal has certain characteristics, such as a sharp drop, followed by a generally steady signal that does not vary much, if any. Detection of these characteristics can be used to distinguish between actual intentional touch events and accidental contact by other objects, such as falling water. Placing a device or other object on a sensor (e.g., a table mounted sensor) may have an energy measurement signal of a particular characteristic, such as a relatively shallow drop, followed by a generally steady signal without much, if any, change.

Thus, a system as described herein may determine an energy signal associated with a set of reflected ultrasound signals. The system may then extract feature information associated with the energy signal and then determine an inference associated with the object based on the extracted feature information. Determining the inference can include using the characteristic information to determine whether the touch event is associated with a human finger or with a drip. For example, as shown in fig. 27, a drip (i.e., a water droplet) may cause a greater drop in the energy signal determined by the system than a human finger (i.e., a finger). The finger may have peaks and valleys (i.e., a fingerprint) that reduce the amount of surface area placed on the sensor and thus the amount of ultrasound signal absorbed by the subject.

Thus, a criterion of the amplitude of the energy signal (e.g., corresponding to a sharp drop) may be used to distinguish between a finger touch and a water droplet. Furthermore, the energy signal is more consistent over time than a human finger. Thus, criteria that the energy signal is within a specified range over a specified amount of time can be used to distinguish between a water droplet and a human finger. Such measurements may be performed using a change in the energy signal over time (e.g., standard deviation). Thus, the characteristic information may comprise the magnitude of the energy signal and/or a change in the energy signal. The inferred determination may include comparing the size and/or change to respective thresholds to determine whether the touch event is associated with a human finger or a drip.

FIG. 46 is a combined schematic and set of charts depicting how temperature can be utilized to further identify whether a human finger is initiating a touch event. The energy measurement signal output by the sensor (e.g., sensor chip and/or substrate) depends to some extent on the temperature of the sensor. As the temperature increases, the energy measurement signal tends to decrease.

Typically, the chip will be at room temperature (e.g., at or about 20 or 21 ℃) and the human finger will be at body temperature (e.g., at or about 30 ℃). When living tissue (e.g., a human finger) initiates a touch event, heat will be transferred between the tissue (e.g., finger) and the chip. When the finger is hotter, it may cause the chip temperature to rise slightly. Since the energy measurement signal as a whole depends in part on the temperature of the chip and/or substrate, temperature fluctuations of the chip and/or substrate can be detected as a potentially steady increase or decrease of the energy measurement signal over time. As depicted in the lower left chart of fig. 46, when a warm finger is placed on a cooler sensor, the heat transfer will cause the energy measurement signal to assume a generally downward slope. As depicted in the lower middle graph of fig. 46, when a cold finger is placed on a softer sensor, the heat transfer will cause the energy measurement signal to exhibit an overall upward slope (i.e., upward trend). However, as depicted in the lower right chart of fig. 46, when something other than living tissue (e.g., a finger) is placed on the sensor and the other object has a temperature at or near the same temperature of the sensor (e.g., both at room temperature), the lack of heat transfer will cause the energy measurement signal to exhibit a generally flat slope. In summary, this temperature effect on the energy measurement signal can be used to identify when something contacting the sensor is at or near body temperature, or at or near other temperatures. In some cases, the approximate temperature of the object initiating the touch event may be discerned by analyzing the general slope of the energy measurement signal.

In some cases, one or more temperature sensors may be used to measure the temperature of the chip and/or substrate. Knowledge of the temperature of the chip and/or substrate may help inform a determination of whether the object initiating the touch event is a human finger.

FIG. 47 is a combined schematic and chart depicting a finger touch and associated temperature information, in accordance with certain aspects of the present disclosure. In some cases, the ultrasound input system may include a temperature sensor, e.g., in-chip, on-chip, or near-chip. The temperature sensor may provide a temperature signal (e.g., a temperature sensor readout) associated with the temperature of the ultrasound input system. Typically, there will be minimal or no change in the temperature signal when no touch event is initiated, as the ultrasound input system will remain at or near ambient temperature, such as room temperature. However, if a human finger is used to initiate a touch event, an expected temperature change towards body temperature (e.g., a temperature rise from room temperature to body temperature) may occur. As depicted in the lower left chart of fig. 47, human finger touch may be detected or confirmed by identifying a change in the temperature signal toward body temperature (e.g., at or near 30 ℃). As depicted in the lower right chart of fig. 47, a touch event initiated by an object other than a human finger (e.g., a room temperature object) will not cause a change in the temperature of the ultrasound input system toward body temperature.

B. Distinguishable physical characteristics

Fig. 48 is a schematic combined side view 4802 and signal diagram 4804 depicting ridges 4806 and valleys 4808 of a fingerprint initiating a touch event on an ultrasound input system, in accordance with certain aspects of the present disclosure. When a user places a finger on a surface associated with the ultrasonic input device 4810, the ultrasonic input device 4810 is capable of detecting a portion of the user's fingerprint. Typically, the ultrasound input device 4810 may sense an area less than the entire fingerprint of the user, although this is not always required.

The ultrasonic input device 4810 may identify ridges 4806 and valleys 4808 of a user fingerprint (e.g., a portion of a user fingerprint). At the ridge 4806, the ultrasound input device 4810 will detect a decrease in the energy measurement of the reflected signal due to the damping effect of the meat of the ridge 4806. However, at the valley 4808, the same damping effect does not exist.

Thus, the ultrasonic input device 4810 measuring a finger as depicted in the schematic side view 4802 may generate a signal diagram 4804 showing ridges 4806 and valleys 4808. As can be seen in signal plot 4804, darker areas represent dips in the energy measurement of the reflected signal, while lighter areas represent signals closer to the baseline energy measurement. Although the entire fingerprint cannot be discerned from the field of view of the ultrasonic input device 4810, many ridges 4806 and valleys 4808 may be discerned. By measuring the widths of the ridges 4806 and valleys 4808, as well as the inter-valley and inter-ridge distances (e.g., inter-ridge distance 4812), the ultrasonic input device 4810 is able to distinguish one finger from another finger. In an example case, an adult's finger may exhibit ridges 4806 and valleys 4808 that are wider than a young person's finger. Thus, in a household with adults and children, the ultrasound input device 4810 is capable of discriminating between two users based on distinguishable physical characteristics of the user's finger, such as fingerprint characteristics. In some cases, the presence of repeating line patterns (e.g., the pattern of ridges 4806 and valleys 4808) may be used to confirm or determine whether the object initiating the touch event is a human finger.

In some cases, a discernible physical characteristic, such as a fingerprint, may be used with the discernible energy signal to further identify the user.

Fig. 49 is a schematic diagram depicting example reflected signals 4924, 4925 received by an ultrasound input system 4902 attributable to a glove 4908 and the same user initiating a touch event without a glove 4906, in accordance with certain aspects of the present disclosure. A first curve 4910 associated with an ultrasound input device 3502 having touch input from a user who is not wearing a glove 4906 shows a transmission signal 4922 and a first set of reflected signals 4924. The first set of reflected signals 4924 shows the characteristic decay of the reflected signals associated with a touch event. A second plot 4920 associated with the ultrasound input device 3502 having a touch input from a user wearing the glove 4908 shows the transmission signal 4922 and the second set of reflected signals 4925. The second set of reflected signals 4925 shows a characteristic decay of the reflected signals associated with the touch event that is somewhat similar to the first set of reflected signals 4925, but may have additional decay due to the presence of glove 4912. The first set of reflected signals 4924 may be processed to generate a first output voltage 4932. Similarly, the second set of reflected signals 4925 may be processed to generate a second output voltage 4933.

Thus, the ultrasound input system 4902 may distinguish between gloved hands and ungrooved hands. In some cases, depending on whether the user is wearing gloves, certain actions may or may not be available. For example, in a medical laboratory, certain functions associated with an ultrasound input system may not be available unless the user is wearing gloves to ensure proper protection in place.

C. Extracting and using features

Fig. 50 is a flow chart depicting 5000 for extracting features from signals of an ultrasound input system, in accordance with certain aspects of the present disclosure. The method shown in fig. 50 will be described in the context of a system including an ultrasound input device and one or more data processors that determine energy signals from touch events. However, it should be understood that the present invention may be applied to other cases.

At optional block 5002, the baseline signal may be received by an ultrasound input system. The baseline signal may be an energy measurement associated with a touchless event (e.g., when no user touches a surface coupled to the ultrasound input device). Removing such baseline signals may result in a more efficient sampling range during analog-to-digital conversion, e.g., as described herein with reference to at least fig. 27. For example, the ultrasound input system may transmit a first signal. Any suitable number of reflected ultrasonic signals and reflected-transmitted signals may then be measured by the ultrasonic input system. Based on characteristics of the received signal, it may be determined that the signal is not associated with a touch event (e.g., a finger touching the exterior surface). For example, the received signal may be indicative of a baseline signal associated with the over-the-air signal. Further example details of baseline signals are described herein.

At block 5004, the system may transmit the transmitted signal using an ultrasound input device. The ultrasonic input device may be coupled to a layer of material having an outer surface positioned opposite the layer of material of the ultrasonic input device. The emitted signal may be directed through the material layer towards the outer surface. As described in detail herein, any number of reflected ultrasonic signals and reflected-transmitted signals may be generated from an initially transmitted ultrasonic signal until the signal becomes too weak to be reflected and/or detected.

At block 5006, a signal associated with a touch event is received. For example, the system may receive a set of reflected ultrasound signals associated with the transmitted signals. The received signal may be a measure of energy associated with the reflected ultrasound. The signal received at block 5004 may depend on how the touch event was initiated (e.g., timing of the touch, manner of the touch, amount of force of the touch, physical characteristics of the object initiating the touch).

At block 5008, one or more data processors of the system may determine an energy signal associated with a set of reflected ultrasonic signals associated with a touch event between the object and an outer surface of a layer of material coupled to the ultrasonic input device.

As an example, referring to fig. 27, a flow chart 2700 includes transmitting and receiving ultrasonic signals, as shown in a first graph 2702. The first graph 2702 shows an analog measurement of a first signal 2703 for the transmitted ultrasound signal and an analog measurement of a subsequent set of signals 2704A, 2704B, 2704C, 2704D, 2704E of the reflected set of ultrasound signals associated with the ultrasound input device. Flowchart 2700 may include an ultrasound input device having an analog summing or integrating circuit 2720 and a summed voltage output 2722.

The second graph 2708 shows a first signal 2703 and a set of energy signals 2710A, 2710B, 2710C, 2710D, 2710E, each corresponding to the energy of a respective one of the set of reflected ultrasound signals. For illustration purposes, the set of energy signals 2710A, 2710B, 2710C, 2710D, 2710E are depicted in solid lines, overlapping a subsequent set of signals 2704A, 2704B, 2704C, 2704D, 2704E from the first graph 2702 shown in dashed lines.

At block 5010, after determining the energy signal associated with the set of reflected ultrasound signals, features may be extracted from the signals associated with the touch event. The extracted features may be any suitable characteristic of the signal, which may be discernable and/or capable of informing an inference. The one or more data processors may be configured to extract the characteristic information associated with the energy signal in any suitable manner.

In some embodiments, extracting the feature information may include identifying a pattern in the energy signal associated with a dip in energy (associated with a touch event). For example, when an individual places their finger on the system, particularly on an external surface, the individual's finger may absorb at least a portion of the emitted ultrasonic signal, thereby causing a dip in the energy measurement.

The pattern may be identified in any suitable manner described herein. For example, in some embodiments, identifying the pattern in the energy signal may include identifying a depth of the dip, a duration of the dip, a presence of a subsequent dip after the dip, a delay between the dip and another dip, and/or a rate of change of the energy signal at the edge of the dip (e.g., during finger drop (FINGER LAND) or removal). In other embodiments, identifying the pattern may include identifying a change in the energy signal attributable to a temperature drift in the material layer, as described in detail herein.

In some cases, extracting features (i.e., feature information) at block 5010 may include comparing the signal with stored history signal(s), such as to determine whether the signal received at block 5010 matches a stored signal associated with a particular user. In some cases, extracting features at block 5010 may include identifying patterns in the received signal, such as to identify that the received signal is attributable to a sharp tap or a placement and press action. In some cases, extracting the features at block 5010 may include measuring characteristics of the received signal. Any discernable characteristic of the received signal may be measured and used to make a determination or inference as to the source of the touch event.

At block 5012, inferences can be determined based on the extracted feature information. The one or more data processors may be configured to determine inferences associated with the object in any suitable manner based on the extracted feature information.

For example, in some embodiments, determining the inference may include estimating a relative temperature of the object based on an identified change in the energy signal attributable to temperature drift in the material layer. For example, an individual contacting the outer surface of the material layer may have a body temperature that is higher than the ambient temperature and/or the temperature of the material layer. As described herein, the determined energy signal may be affected by temperature and thus allow one or more data processors to determine an inference of temperature measurement and/or temperature drift (e.g., as measured by a temperature sensor as described below).

In other embodiments, one or more data processors may determine the inference by comparing the identified pattern to stored data. The stored data may be associated with a previous touch event of the outer surface. For example, a previous touch event of the outer surface may have been performed by the individual. As described herein, the current touch event may be compared to previous touch events to determine whether the current touch event is also performed by the individual.

In other embodiments, the one or more data processors may determine inferences by using the characteristic information to determine that the touch event is associated with a human finger, a bare human finger, a wet human finger, a dry human finger, and/or a gloved human finger. For example, as described herein, the determined energy signal may be affected by one or more characteristics of the individual finger(s) placed on the outer surface of the material layer. The one or more data processors may also determine inferences by using the characteristic information to determine a manner of touching (e.g., tapping, double-tapping, placing, pressing, etc.) of the touch event, a touch intensity associated with the touch event, and/or a physical characteristic of the object.

In some embodiments, determining the inference can include identifying that the object is associated with one of the plurality of users based on associating the touch event with a touch manner of the touch event, a touch strength associated with the touch event, and/or a physical characteristic of the object. The physical characteristic of the object may include a measurement associated with a portion of the fingerprint contacting the outer surface.

In some embodiments, the one or more data processors may determine additional signals associated with additional sensors associated with the ultrasound input device (e.g., the temperature sensor of fig. 29). The one or more data processors may then further use the additional signals to determine the inference. The additional sensors may include any suitable additional sensors associated with the ultrasound input device. For example, the additional sensors may include temperature sensors, pressure sensors, charge coupled devices, and the like.

For example, the system may include a temperature sensor. The temperature sensor may record, for example, the temperature of the external surface of the system over time. Because the human fingertip has a particular physical size and temperature range, the one or more data processors may determine that the touch event is caused by a human finger when the human touches the outer surface. As an illustrative example, the temperature sensor may record the temperature of at least a portion of the outer surface at predetermined intervals (e.g., 1ms, 0.1s, 1s, etc.). The temperature sensor may record the ambient temperature (e.g., 70°f). When a user touches the outer surface during a touch event, the system may record an energy signal, which may include, for example, dip. During a touch event, the temperature sensor may continue to measure the temperature of the outer surface. A human finger in contact with the outer surface may raise the temperature of the outer surface, thereby causing the temperature sensor to record an increase in temperature. For example, the human finger may be about 98°f. The temperature sensor may record a temperature between 70°f ambient temperature and 98°f human finger temperature because the finger will heat the outer surface and the temperature sensor.

The temperature measured by the temperature sensor may be an additional signal associated with an additional sensor (e.g., a temperature sensor) associated with the ultrasound input device. The one or more data processors may use the additional signal along with the energy signal to determine the inference. For example, the one or more data processors may determine that a dip in the energy signal and a temperature rise from ambient temperature to a higher temperature between ambient temperature and average human body temperature indicate that a touch event indicates that a human finger is touching the outer surface. In some cases, one or more data processors may use temperature data from a temperature sensor to determine whether a signal change is a result of a human touch or from another object in contact with an external surface (e.g., a table, pocket fabric, pen/stylus, etc.). For example, when contacted with a table, pocket fabric, pen/stylus, etc., the temperature sensor may not measure as much temperature increase as when contacted with a human finger.

In some cases, the temperature sensor may be a known (i.e., predetermined) distance from the finger. For example, the temperature sensor may be located on the opposite side of the outer surface from the finger. In this case, during processing of the additional signal associated with the additional sensor (e.g., temperature sensor), the heat transfer problem with known boundary conditions and initial values may be addressed to determine what the temperature at the outer surface is.

In some embodiments, the additional sensors may include pressure sensors and/or strain gauges. For example, a typical touch from a human finger may exert a particular force and strain on the outer surface that may be propagated to additional sensors. The pressure sensor and/or strain gauge may measure the force and/or strain applied by the finger into the system. The one or more data processors may determine that the force and/or strain measured by the pressure sensor and/or strain gauge is indicative of a typical force and/or strain of a touch of a finger. The one or more data processors may also determine whether the energy signal indicates a touch by a finger. If both the additional signal from the pressure sensor and/or the strain gauge and the energy signal are indicative of a touch by a finger, the one or more data processors may determine that the touch event is a touch by a finger.

In some cases, the additional sensor may include a strain gauge. The strain gauge may detect deflection of a surface associated with a touch event and may output an electrical signal. The stronger the touch event (e.g., the greater the force exerted on the outer surface by an object such as a finger), the greater the deflection exerted on the strain gauge. Thus, the strain gauge may output a larger electrical signal.

At block 5014, the one or more data processors may generate an output signal associated with the determined inference. The output signal may include any suitable output generated based on the determined inference. In some embodiments, the output signals may be indicative of particular actions that may be performed by one or more data processors and/or external devices.

In some embodiments, the one or more data processors may perform actions based on the extracted feature(s). The action may include any suitable process that may occur based on the output signal. In an example, if the extracted features are used to identify a particular use, the action performed may be authenticating or authorizing the user to access the resource. In another example, if multiple users have preset a customization for a particular ultrasound input system, the extracted feature information may be used to determine which user is interacting with the ultrasound input system and thus performing a customized action for that particular user. In some cases, performing the action may include allowing or denying access to the resource, such as denying access to a room or tool when the extracted feature indicates that the user is not wearing gloves (when gloves are needed).

Machine learning decision algorithm

FIG. 51 is a chart 5100 depicting a machine learning decision algorithm for improving touch detection in accordance with certain aspects of the present disclosure. As described with reference to fig. 37, the weight parameters may be used to drive various decisions regarding when a touch event is detected or not detected. In some cases, the machine learning method may consider the sensor output values and the slope between the sensor values and the previous sensor values to generate an inference that a touch event has occurred or has not occurred. The machine learning method may use a decision function (f), such as:

f＝w₀X[n]+w₁X[n-1]+w₂X[n-2]+…+w_mX[n-m]+w_s0S[n]+W_s1S[n-1]

+...+w_smS[n-m]

Where w _n and w _sn are weight parameters, X [ n ] is the current sensor output, X [ n-1] is the previous sensor output, X [ n-m ] is the mth previous sensor output, S [ n ] is the slope of the current sensor output (e.g., as compared to the immediately previous sensor output), S [ n-1] is the slope of the previous sensor output, and S [ n-m ] is the slope of the mth previous sensor output. In some cases, other parameters may be used in the decision function.

The weight parameters of the decision function may be trained on a corpus of data to generate decision boundaries between inputs that are considered touch events and inputs that are not considered touch events, as depicted in graph 5100. Thus, for any given sensor output and slope of the sensor output, a point on the graph 5100 can be identified and if the point falls above the decision boundary, those sensor outputs and slope of the sensor output can be considered to be indicative of a touch event.

IX. Smart touch event detection

Systems and methods according to embodiments may allow for a touch event detection framework. Embodiments allow adaptive thresholds for touch event detection. The adaptive threshold scheme may involve identifying touch events from energy signals of the sensors using a continuously adaptive threshold. Embodiments also allow recurrent neural networks for touch event detection and/or recurrent neural networks for state classification.

A. Universal touch event detection framework

Fig. 52 is a flow chart depicting a process 5200 for detecting a touch event, in accordance with certain aspects of the present disclosure. The process 5200 may be performed by any suitable device, including the processor 722 and/or the computing device 724 of fig. 7. In some cases, data from multiple sensors may be used for any of blocks 5202, 5204, 5206.

At block 5202, energy signal data is accessed. The energy signal data is signal data from the ultrasonic sensor that is indicative of an amount of energy sensed by the ultrasonic sensor during a period of time, such as the energy signal 3614 depicted and described with reference to fig. 36. Any suitable period of time may be used.

At block 5204, a touch event may be identified based on the energy signal. Identifying the touch event may include determining whether the touch event has occurred based on the energy signal. In some cases, identifying the touch event at block 5204 may include outputting a touch signal. The touch signal may indicate whether the associated energy signal is inferred to be associated with a touch event.

At optional block 5206, a status classification may be identified from the touch event data (e.g., touch signal from block 5204). In some cases, a state classification may be identified from the touch event data and the associated energy signal data. The state classification may be a classification associated with a touch event. Any suitable classification may be determined, such as the type of touch event that has occurred. Examples of suitable status classifications related to the type of touch event that has occurred include single click, double click, triple click, n-tap, hold (e.g., touch and hold), tap and hold (e.g., tap followed by touch and hold), press (e.g., longer than tap), double press, press and hold (e.g., press followed by touch and hold), hold and press (e.g., touch and hold for a duration followed by press), and grip (e.g., hold with more surface area or other characteristics). A state classification may be determined, which may be associated with the touch event based on the trigger value. Examples of suitable status classifications related to other information associated with a touch event may include whether a user is wearing gloves, whether a user appears older or younger (e.g., based on the distance between fingerprint ridges), whether a user appears to be a pre-identified user, or other such classifications.

As an example, another classification may include hydration and/or perspiration of a user's finger and/or body. The system may detect hydration and/or perspiration of the user, for example, by determining an ultrasonic signal absorption that is lower than a typical ultrasonic signal absorption of the user. When the user's finger is drier, the finger will absorb less of the ultrasonic signal. Thus, different levels of thresholds for amplitude and time variation may be used. For example, a wet finger may cause a more uniform drop in the energy signal than a dry finger. Thus, a criterion of the magnitude of the energy signal (e.g., corresponding to a sharp drop) may be used to distinguish between a dry finger and a wet finger. Furthermore, the energy signal will be more consistent with a wet finger over time than with a dry finger due to the presence of additional water in the wet finger. Thus, a criterion that the energy signal is within a specified range over a specified amount of time may be used to distinguish between a wet finger and a dry finger. Such measurements may be performed using a change in the energy signal over time (e.g., standard deviation). Thus, the characteristic information may comprise the magnitude of the energy signal and/or a change in the energy signal. The inferred determination may include comparing the size and/or change to respective thresholds to determine whether the touch event is associated with a wet finger or a dry finger.

In some cases, any number of classifications may be used, depending on the orientation and placement of the sensors. In some cases, the state classification may be trained such that the identification of the state classification at block 5206 may refer to training data or a model generated using the training data.

B. Adaptive threshold for touch event detection

Fig. 53 is a schematic diagram depicting an adaptive threshold scheme 5300 for identifying touch events, in accordance with certain aspects of the present disclosure. The adaptive threshold scheme 5300 can be partially or wholly executed on a processor coupled to the ultrasonic sensor, such as processor 722 of fig. 7. The method described in fig. 53 may be performed by an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or any other suitable device and/or controller described herein. This solution 5300 does not require machine training, however, one tuning for each new cover material to which the sensor is coupled may improve detection. The adaptive threshold scheme 5300 can operate outside of discrete time periods or frames such that at any time, the scheme 5300 can analyze current data from current observations and historical data from any number of past observations.

The adaptive threshold scheme 5300 involves identifying touch events from energy signals of the sensors using a persistent adaptive threshold. The threshold is a continuously tracked version of the energy signal that has been filtered by an adaptive threshold update function, the filter parameters of which are adaptively updated based on the energy signal, the historical threshold, and optionally trigger data (e.g., whether a touch event is occurring or has recently occurred). A sensor trigger (e.g., a touch event) may be identified whenever the energy signal exceeds an adaptive threshold (i.e., is less than the adaptive threshold, greater than the adaptive threshold, less than or equal to the adaptive threshold, or greater than or equal to the adaptive threshold). Alternatively, the scheme 5300 may analyze the current and past trigger histories (e.g., based on a particular number of recently observed touch event signals) to identify a current state classification, such as determining whether a touch event is a tap, press, hold, or other action. For example, analyzing the touch event signal (e.g., trigger history) may show how many times the energy signal falls below a threshold within a particular time frame, how far the energy signal falls below a threshold, how long the energy signal is below a threshold, and other such characteristics that may be used by the scheme to make a determination regarding the current state of the sensor.

The sensor data 5302 may include current and historical sensor data that is an energy signal (e.g., energy data) from a sensor such as the ultrasonic sensor 702 of fig. 7 or based on an energy signal (e.g., energy data) from a sensor such as the ultrasonic sensor 702 of fig. 7. The sensor data 5302 may be provided to an adaptive threshold update function 5304, which may use the data to generate threshold data 5306. The adaptive threshold update function 5304 may use only current sensor data or both current sensor data and historical sensor data, as well as any additional parameters, as the case may be. The adaptive threshold update function 5304 acts as a low pass filter that allows slow changes in the energy signal to have a significant effect on the adaptive threshold, while faster changes in the energy signal have minimal or negligible effect on the adaptive threshold. Thus, factors that cause slow changes in the energy signal, such as temperature changes in the room, can be automatically compensated in the adaptive threshold, while factors that cause fast changes in the energy signal, such as finger presses, will be correctly detected as exceeding the threshold.

The threshold data 5306 may represent a threshold above which the energy signal should be considered to have triggered a touch event (e.g., below the threshold). The condition analyzer 5308 can compare the sensor data 5302 (e.g., energy data) to the threshold data 5306 to determine whether a trigger event (e.g., touch event) has occurred. For example, when the current energy signal from sensor data 5302 drops below the current threshold in threshold data 5306, it may be assumed that a touch event has occurred. In some cases, the condition analyzer 5308 may also provide feedback to the adaptive threshold update function 5304 to update parameters of the adaptive threshold update function 5304. For example, the speed, degree, or number of times the energy signal falls below the threshold may be used to manipulate how the adaptive threshold update function 5304 generates the threshold data 5306 from the sensor data 5302. If a trigger event (e.g., a touch event) is detected, the condition analyzer 5308 may output one or more trigger values in the trigger data 5310. The trigger data 5310 may include current trigger data and historical trigger data indicating whether a trigger event (e.g., touch event) has occurred. For example, the threshold data may be updated based on the energy data, the trigger data, and the threshold data, wherein updating the threshold data includes generating a subsequent threshold.

In some cases, the trigger data 5310 may be used by the condition analyzer 5308 to further inform the condition analyzer 5308 of the decision as to whether a trigger event has occurred. For example, historical trigger data may be used by the condition analyzer 5308 to confirm or refute a possible trigger event. However, in some cases, the trigger data 5310 may be used by the condition analyzer 5308 to provide updated parameters to the adaptive threshold update function 5304 such that the adaptive threshold update function 5304 is further updated based on current and/or historical trigger data.

In some cases, the trigger data 5310 may be passed to a trigger analyzer 5312 to determine a sensor state 5314. Trigger analyzer 5312 may retrieve information from trigger data 5310 to determine whether the most recent trigger event is a touch, tap, or other such classification of trigger events. The trigger analyzer 5312 may then output its determination as the sensor state 5314. The sensor state 5314 may indicate not only a trigger event, but also a classification of the state associated with the trigger event. For example, while the trigger data 5310 may take the form of a representation (e.g., a binary signal) that indicates whether a trigger has occurred, the sensor state 5314 may also take the form of a representation of what state the sensor is in. Example sensor states include hold, tap, press, double-click, and the like. The classification of the sensor states may be selected from a predetermined list, each list having a different pattern of energy signals.

As used herein, various signals may be considered to include a plurality of data points including a current data point (e.g., the most recent data point) and any number of previous data points. As used herein, the term history data may include current data points and past data points. The data points may be represented as analog or digital signals.

FIG. 54 is an example graph 5400 depicting energy signals 5402 and adaptive thresholds 5404 associated with identifying touch events in accordance with certain aspects of the disclosure. As described with reference to fig. 53, the energy signal 5402 may be from any suitable sensor data 5302, such as the ultrasonic sensor 702 of fig. 7. The adaptive threshold 5404 may be threshold data 5306 generated from the energy signal 5402, such as described with reference to fig. 53.

As depicted in fig. 54, the gradual change in the intensity of the energy signal 5402 throughout the graph 5400 is reflected in the adaptive threshold 5404. In particular, as seen in the relatively constant segment when no rapid spike occurs, the average intensity of the energy signal 5402 steadily decreases over time, which then reflects itself steadily decreasing over time in the adaptive threshold 5404. However, the rapid changes associated with a touch event, depicted as rapid negative spikes in the energy signal 5402, are not fully reflected in the adaptive threshold 5404, which allows the energy signal 5402 to fall below the adaptive threshold 5404. The system can register a touch event whenever the energy signal 5402 falls below an adaptive threshold 5404. Based on various properties of the detected touch event, such as frequency, intensity, duration, and other such properties as disclosed herein, a determination may be made regarding classification of the touch event (e.g., state of the sensor), such as whether the sensor is being tapped, pressed, held, double-tapped, or otherwise manipulated.

C. recurrent neural network for touch event detection

Fig. 55 is a schematic diagram depicting a generalized recurrent neural network 5500, according to certain aspects of the present disclosure. Recurrent neural network 5500 is a data analysis technique that operates to convert input data 5502 into output data 5508. Recurrent neural network 5500 may be used to identify one or both of a touch event from the energy signal (e.g., as seen in block 5204 of fig. 52) and a state classification from the touch event and/or the energy signal (e.g., as seen in block 5206 of fig. 52). For example, the input data 5502 may be an energy signal or energy data from any suitable sensor, such as the ultrasonic sensor 702 of fig. 7, and the output data may be a trigger signal or a status classification signal. For example, the ultrasonic sensor may provide energy data to the recurrent neural network to generate output data indicative of the occurrence of a touch event.

Recurrent neural network 5500 may cause input data 5502 to cross any number of layers through any number of nodes until output data 5508 is generated. In some cases, the output data (e.g., output data 5508) may include state classification information associated with the touch event. One or more hidden layers 5504, 5506 may be located between the input data 5502 and the output data 5508. Within each hidden layer 5504, 5506, a node 5510 may process incoming data into outgoing data. In node 5510, any number of inputs may be received and processed (e.g., summed and passed through a function) to generate an output. In the example node 5514, three inputs (e.g., weighted versions of other layers, such as w ⁱ _k,1;wⁱ _k,2;wⁱ _k,m((i-1)) are received and summed and passed through a function to generate a single output (e.g., a ⁱ _k). In other words, in some cases, the output of a node may be a decision function of a linear combination of the outputs of the previous layers, optionally with additional feedback as described below with reference to the tapped delay line. In other cases, the inputs to node 5514 may be linearly combined and then passed to another function f. The function f may be an activation function, which may be linear or non-linear. For example, the activation function may include a sigmoid function, a hyperbolic tangent function, a rectified linear unit (ReLU) function, an identification function, and/or any suitable activation function. The activation function may constrain the output to a probabilistic form between any suitable bounds (e.g., 0 to 1, -1 to 1, -0.5 to 0.5, etc.). The output from node 5510 may then be passed to one, some, or all of the nodes in the subsequent layers, or in the case of a final layer, may be passed to the output and used to generate output data 5508 (e.g., by summation or other function) along with other outputs from that same layer.

As depicted in fig. 55, recurrent neural network 5500 may also utilize a tapped delay line 5512. Each tapped delay line 5512 may be used to provide inputs to one, some, or all nodes 5510 of a particular layer (e.g., layer 5504) that may receive a current output or delayed output from that layer or a subsequent layer (e.g., layer 5506 or output 5508) via the tapped delay line 5512. In some cases, the tapped delay line 5512 may also provide a delayed version of the input data 5502 as an input to each node 5510 of a particular layer. The example tapped delay line 5516 depicts a vector of inputs (e.g., a ^j) from layer j 5506 that are delayed and output as inputs to the various layers of layer i 5504 (e.g., wd ^i,j). In this way, output from subsequent layers (e.g., historical output) may inform earlier layers in recurrent neural network 5500. The tapped delay line 5512 may include data from any suitable length of time. For example, the tapped delay line 5512 may provide data from a single frame in the past or from multiple frames.

Recurrent neural network 5500 in fig. 55 is depicted as having a single input (e.g., input data 5502), hidden layer i 5504 containing m (i) different nodes, hidden layer j 5506 containing m (j) different nodes, a single output (e.g., output data 5508). In some cases, recurrent neural network 5500 used in accordance with certain aspects of the present disclosure may include any suitable number of inputs, layers, nodes, and outputs. By providing tagged sensor data to the system, the recurrent neural network 5500 may be pre-trained by supervised machine learning to allow the function of each node (e.g., weight value of each node) to be updated until the recurrent neural network 5500 performs as desired.

In some cases, the recurrent neural network may be trained using historical energy data associated with a plurality of historical touch events. The historical energy data may include previous energy data recorded and stored in a suitable memory and/or database. The plurality of historical touch events may include data regarding previous touch events. For example, the historical energy data may include a dip in energy that may be associated with a historical touch event (e.g., "0" indicates no touch and "1" indicates a touch). The historical touch events may include one or more of each of a set of state classifications. For example, a touch event "1" may be associated with a status classification "tap".

In some embodiments, the set of state classifications may be selected from a plurality of available state classifications by user input. The plurality of available state classifications may include, for example, a list of state classifications available for selection by a user. For example, in some cases, the plurality of available state classifications may include tap, double-tap, press, and hold. In other cases, the plurality of available status categories may include tap, press, double press, and grasp. The plurality of available state classifications may include any suitable combination of state classifications.

In other embodiments, the plurality of historical touch events may also include a plurality of non-touch events. A non-touch event may facilitate training of additional recurrent neural networks to reject false positive events. The non-touch event may include, for example, a touch event indicating no touch. For example, a non-touch event may be associated with a dip in the energy signal, but may be associated with an event where the drip touches the outer surface rather than through a finger, as described herein. In some cases, the user may be prompted to touch the outer surface against other objects (e.g., pens, fabrics, etc.) that the user wishes to classify as non-touch events. In this way, the user may provide non-touch event data associated with a situation where the user does not wish the device to determine a touch event. In some cases, the apparatus may provide output data from the recurrent neural network to an additional recurrent neural network to generate state classification information associated with the touch event.

Fig. 56 is a schematic diagram depicting an example recurrent neural network 5600 for identifying trigger events, in accordance with certain aspects of the present disclosure. The recurrent neural network 5600 may be particularly useful for efficiently and accurately detecting trigger events from ultrasonic energy signals.

At input 5602, sensor data can be provided to recurrent neural network 5600 in the form of an energy signal from an ultrasonic sensor, such as ultrasonic sensor 702 of fig. 7. The energy signal passes on both nodes 5610, 5612 of the hidden layer 15604 and on a tapped delay line 5618 that provides one or more delay signals to the nodes 5610, 5612 of the hidden layer 15604 based on the energy signal from the input 5602. For example, the tapped delay line 5618 may be set to provide the energy signal for the last three or four frames to the hidden layer 1 5604. Furthermore, the nodes 5610, 5612 of the hidden layer 15604 may take as additional input the output of a tapped delay line 5620 that may be configured to output a set number of past frames of the output 5608 of the recurrent neural network 5600. For example, the tapped delay line 5620 may be configured to provide an immediately preceding frame of data output via the output 5608 of the recurrent neural network 5600 as an input to the hidden layer 15604. The output from the nodes 5610, 5612 of the hidden layer 15604 may then be passed as input to the nodes 5614, 5616 of the hidden layer 2 5606. The outputs of the nodes 5614, 5616 of the hidden layer 2 5606 may then be passed to the output 5608 (e.g., combined and output) as trigger data.

In some cases, a strong combination of efficiency and accuracy for identifying trigger data from the ultrasonic energy signal may be to use a recurrent neural network 5600 having a first layer that receives some combination of sensor data, past sensor data, and past trigger data and a second layer that receives the output of the first layer. The output from the second layer may be used to generate a trigger data output.

Recurrent neural network 5600 may be trained in advance and/or by a user. Training of recurrent neural network 5600 may include providing energy signals appropriately labeled as touch events or not. The training data may be provided by a supplemental input device (e.g., a physical button or electrical contact) that simultaneously records the touch event as the ultrasonic sensor detects the energy signal associated with the touch event, or by associating the recorded energy signal with the touch event, such as by instructing the user to initiate the touch event at a particular time or at a particular cadence. Once the training data has been obtained, the recurrent neural network may be programmed or trained through supervised machine learning, which allows the function of each node (e.g., the weight value of each node) to be updated until the recurrent neural network 5600 performs as desired (e.g., accurately identifies the triggering event). In some cases, the recurrent neural network 5600 may be retrained each time an ultrasonic sensor is coupled to the new material stack.

The output 5608 may take the form of a number due to the nature of a typical recurrent neural network. Inferring the appropriate trigger data from the number may include applying a threshold to the actual output 5608 of the recurrent neural network 5600. For example, if recurrent neural network 5600 outputs a number between 0 and 1.0, a threshold may be set between the two numbers, if above which the output may be considered a triggering event (e.g., a touch event), and if at or below which the output may be considered a non-triggering event (e.g., no touch event), or vice versa. In this example, the threshold may be set to 0.5, so an output of 0.55 may be considered a touch event. In some cases, recurrent neural network 5600 may adjust its sensitivity rather than simply retraining the entire neural network by adjusting the threshold. Thus, to reduce the likelihood of false triggers (e.g., reduce sensitivity), the threshold may be moved from 0.5 to 0.6. Thus, the same 0.55 output will not be considered a touch event.

In some cases, an unsupervised machine learning model may analyze data that has not been labeled. The unsupervised machine learning model may include any suitable type of unsupervised machine learning model, such as clustering (e.g., k-means, hierarchical clustering, etc.), anomaly detection, and the like. The plurality of trigger values may be measured by the device while in use by the user. For example, the user may perform any suitable number of touch events that may be logged. In this regard, multiple trigger values from a touch event may not have been marked as, for example, a tap, hold, press, etc. The plurality of trigger values may include at least 0s and 1s, which indicate touches detected at a particular time. Data items including successive trigger values may be used to determine a state.

An unsupervised machine learning model may group multiple trigger values or data items created therefrom (e.g., using a clustering method). As an illustrative example, an unsupervised machine learning model may group data items similar to (0,0,0,1,1,1,0,0,0) into a first cluster. An unsupervised machine learning model may group data items similar to (0,0,1,1,0,0,1,1,0,0) to a second cluster. The unsupervised machine learning model may create any suitable number of clusters based on the plurality of trigger data.

The user may be prompted to provide supervised data (e.g., to provide a desired touch event). In some embodiments, recurrent neural network 5600 may further determine a classification of the cluster determined from the unsupervised machine learning model based on the supervised data as part of training recurrent neural network 5600. For example, clusters with data items similar to (0,0,0,1,1,1,0,0,0) may be marked as "flicks" and clusters with data items similar to (0,0,1,1,0,0,1,1,0,0) may be marked as "double-flicks".

D. recurrent neural network for state classification

Fig. 57 is a schematic diagram depicting an example environment 5700 for touch detection and state classification using a set of recurrent neural networks 5706, 5708, in accordance with certain aspects of the present disclosure. The environment 5700 illustrates a user interface 5702 that may be presented on a computing device, such as computing device 724 of fig. 7, for generating information related to how an ultrasonic sensor (e.g., ultrasonic sensor 702 of fig. 7) can interpret energy signals. The user interface 5702 allows a user to select those states to be detected and identified. The recurrent neural networks 5706, 5708 may then be trained to identify the selected state. However, it should be appreciated that recurrent neural networks 5706, 5708 are not limited to identifying a selected state. For example, recurrent neural networks 5706, 5708 may be trained to identify trigger outputs based on trigger data.

In some cases, a single recurrent neural network may be used to generate an output indicative of a state based on the received energy signal as an input. However, as depicted in fig. 57, the first recurrent neural network 5706 may receive the energy signal as input 5704 and output a trigger signal, which may then be passed as input to the second recurrent neural network 5708, which may then output the status signal as output 5710. In fig. 57, the output 5710 is depicted as a graph of a hypothetical feature space. In the hypothetical feature space, the different possible states are distinguishable based on their position in the hypothetical feature space. The hypothetical feature space can be depicted as two-dimensional, although it can be based on virtually any number of dimensions, including one-dimensional or more than two-dimensional. The output 5710 of the environment 5700 can indicate a particular state of the ultrasonic sensor, such as tapped, touched, pressed, double-tapped, or any other suitable state.

The environment 5700 as depicted in fig. 57 shows a single input and depicts four possible energy modes that can be put into the input. However, in some cases, a single environment 5700 can utilize multiple sensors to provide multiple energy signals to recurrent neural network(s).

Model information may be stored when training recurrent neural network(s) of environment 5700. In some cases, the model information may be stored locally at the sensor (e.g., on a data store associated with a processor driving the ultrasound transducer), although this need not always be the case. In some cases, the model information may be stored remotely (e.g., on a computing device separate from the sensor) or may be split, such as storing model information locally at the sensor for determining whether a trigger event has occurred, and storing model information remotely for determining a state of the sensor based on the trigger signal. Model information may be any information that may be used to generate, and optionally interpret, an output from an input energy signal. For example, the model information may include information about structures and weights found in any recurrent neural network(s) of the environment 5700.

During an example training session, a user may select a set of states to train into model information. As depicted in fig. 57, the selected states include single click, double click, and hold. When prompted to do so, the user may engage in each action associated with each state, generating an energy signal as input data. As the user is prompted to engage in a particular action, the environment 5700 can associate the detected energy signal as being associated with a particular state (e.g., a single click or double click). As depicted in fig. 57, the information associated with a single click is shown in green, a double click is shown in blue, a hold is shown in yellow, and an error event is shown in red. For example, an error event may be generated by prompting the user to touch the surroundings of the sensor rather than directly above the sensor, so that the algorithm is less susceptible to such undesirable erroneous inputs and thus is more sensitive to local inputs located immediately above the sensor. Training data may be collected from one or more users once or repeatedly until the recurrent neural network(s) are sufficiently trained.

In some cases, the training may be offline (e.g., performed on a set of test sensors and optimal network parameters written for all sensors for that particular application), or the user may be prompted to perform a training session by the user at system initialization (e.g., this may be similar to a fingerprint login on a phone). The method may also be a combination of the two methods. For example, the method may include offline training and some optimization during user use. In other cases, data (e.g., energy data, status data, trigger data, etc.) may be shared on the cloud or through other suitable communication channels to strengthen the training data database in order to improve network models, training, and optimization.

In some cases, recurrent neural networks may be particularly useful for time series data, and may be more easily optimized for different material stacks and different environmental conditions. In some cases, an environment with multiple recurrent neural networks may allow different types of useful information to be output from the sensor (e.g., from a processor driving the ultrasound transducer). For example, the sensors may output energy signals, trigger signals, and status information, each from a different point in the environment 5700. Thus, the same sensor can be mass produced and used quickly in a variety of different ways. While some customers may prefer to utilize the trigger signal, other customers may wish to utilize the status information. Therefore, the same mass-produced sensor can meet the requirements of different customers. Furthermore, if multiple sensors are installed in one unit, the host/client may also decide how to combine information from the sensor network into a binding event at action trigger and/or even at training, such an example may be a sliding bar or mouse pad. In general, in the case where a plurality of sensors are installed in one unit, information from the plurality of sensors can be used to enhance the performance of an algorithm and to improve its robustness.

X. application

Fig. 58 is a schematic diagram depicting an electronic device with an ultrasound input device, in accordance with certain aspects of the present disclosure. The electronic device 5800 may include a housing 5802, a screen 5804, one or more front buttons 5806, a pair of ultrasonic input devices 5808, and a separate ultrasonic input device 5810. The electronic device 5800 may include a processor, memory, and a network interface. In some embodiments, the ultrasound input device may be coupled to a processor of the electronic device 5800.

In some embodiments, the pair of ultrasonic input devices 5808 may define an input touch region 5812 to detect user input. For example, the user may contact the input touch area 5812 to adjust the volume, brightness, etc. of the electronic device. In some embodiments, an array of ultrasonic input devices may be positioned below a screen or other location such as a side or back of an electronic device to detect touch inputs and replace or enhance capacitive touch or force touch capabilities or mechanical buttons of the electronic device. A single ultrasonic input device 5810 may define an input touch region 5814 to detect user input. The input touch area 5814 may be configured to control device power, screen on/off, and the like.

In some embodiments, an ultrasonic input device may be used to detect touch input at each of the one or more front buttons 5806. The ultrasonic input device may replace capacitive sensing for detecting touch input on the fingerprint sensor. The ultrasound input device provides a low power solution to detect touch input on the fingerprint sensor. In some embodiments, one or more ultrasonic input devices may be positioned below the flag 5822 on the back surface 5820 of the housing 5802 to detect user input. They may also be placed under the sides of the electronic device instead of the commonly used side mechanical buttons, for example for power or volume.

Fig. 59 is a schematic diagram of a steering wheel 5902 having an ultrasonic input device 5904, according to certain aspects of the present disclosure. The ultrasonic input device 5904 may be used to form a touch input area on the steering wheel 5902 to detect touch input. The flexibility of the ultrasonic input device 5904 facilitates detection of touch input by a variety of materials used to make steering wheels, such as plastic, leather, wood, and the like. Cross section 5906 of steering wheel 5902 shows an ultrasonic input device coupled to surface 5908 to form touch input area 5910. The touch input area may be combined with multiple touch input areas for applications such as cruise control, infotainment input control, cellular communication control, volume, and driver detection systems. For example, the ultrasonic input device 5904 may be used in a driver detection system to determine whether the driver's hand is in contact with the steering wheel.

Fig. 60 is a schematic diagram of a keypad 6000 using an ultrasonic input device in accordance with certain aspects of the present disclosure. The shape and materials that can be used to design the underlying touch area of the ultrasound input device are limited only by the creativity of the designer. For example, a 12-key standard telephone keypad is shown in fig. 60. The keypad 6000 may include 12 ultrasonic input devices 6002 to form a touch zone 6004 for each key. As another example, the keypad 6000 may include 23 or fewer ultrasound input devices 6002.

Fig. 61 is a schematic diagram depicting a robotic arm using an ultrasound input device, in accordance with certain aspects of the present disclosure. The robotic arm 6100 may include a first finger 6102 and a second finger 6104. The ultrasonic input device may be implemented as a robotic finger input device. The first finger 6102 and the second finger 6104 may include a first ultrasound input device 6106 and a second ultrasound input device 6108, respectively. The first ultrasonic input device 6106 may form a contact region 6110 on the surface of the first finger 6102 and the second ultrasonic input device 6108 may form a second contact region 6112 on the second finger. The ultrasonic input devices improve the detection capabilities of the robotic arm as they can be integrated into a finger comprising any material. Furthermore, the ultrasound input device may detect touch input without requiring incisions and/or different materials integrated into the finger.

In some cases, the ultrasonic input device may identify the type of material being touched by analyzing the energy measurement signal. In some cases, the ultrasound input device may identify the elasticity of the object being grasped. For example, a less elastic object will generally absorb less ultrasound than a more elastic object, thus resulting in a generally higher energy measurement signal. In some cases, the determination regarding the elasticity of the object may be used to adjust the behavior of the robotic arm, such as adjusting the force of the robotic arm to grasp the object. In some cases, the ultrasound input device is capable of detecting a texture or other mechanical property of the object based on analyzing energy measurement signals associated with the object. In some cases, analysis of the energy measurement signals from the ultrasound input device may be combined with other inputs (such as machine vision) to confirm or make a determination as to the object with which the robotic arm is to interact or is interacting.

Fig. 62 is a schematic diagram depicting a piece of furniture 6202 using an ultrasonic input device 6204, in accordance with certain aspects of the present disclosure. The ultrasound input device 6204 may be coupled to the furniture 6202 in any suitable manner. Touching furniture 6202 by a user at or near the location of ultrasonic input device 6204 may be detected by ultrasonic input device 6204 (e.g., by ultrasonic touch sensor 6212). Upon detection of a touch, the ultrasonic input device 6204 may perform any preprogrammed function. For example, the communication module 6214 of the ultrasound input device 6204 may send a signal (e.g., a wireless signal) to the control module 6206 spaced apart from the ultrasound input device 6204 and/or the furniture 6202. The control module 6206 may control another device, such as a power switch 6208 coupled to the light bulb 6210. Thus, the light bulb 6210 may be turned on, turned off, or otherwise controlled upon pressing a location on the furniture 6202 at or adjacent to the ultrasound input device 6204. The device being controlled (e.g., the bulb 6210) may be in the same environment as the ultrasound input device 6204, although this need not always be the case. In some cases, the controlled device may be located in a nearby environment or even in a remote environment.

In some embodiments, a piece of furniture or IoT may be equipped with one or more of the ultrasound input devices, which may operate alone or in the form of a network of sensors that communicate with each other to perform multiple tasks. The sensors may also communicate with other sensors on other devices, either through the IoT device itself or through the general purpose programmable processor of the sensor, to exchange information.

The ultrasound input device 6204 according to certain aspects of the present disclosure may operate at very low power, such as from an internal battery 6216. Such battery-powered low-power operation may allow the ultrasound input device 6204 to be used in locations that are otherwise difficult to access or inconvenient. For example, the light switch may be incorporated into a desk or desk, or the television remote control may be incorporated into an armrest of a chair.

In some cases, the ultrasound input device 6204 may be positioned on the hidden surface 6218 so as to hide the ultrasound input device 6204 from view during normal operation. The hidden surface 6218 may be the underside of a table (e.g., furniture 6202), the inside of a piece of furniture, the inside of a wall, or any other suitable location that is hidden from view. Thus, the hidden ultrasound input device can only be actuated by those who know its location, for whom it will be hidden from view.

Fig. 63 is a set of graphs depicting energy measurement signals of an ultrasonic input device exhibiting material detection, in accordance with certain aspects of the present disclosure. A characteristic of the energy measurement signal, such as shape, duration, slope, or other characteristics, may be used to determine the material interacting with the ultrasound input device. For example, a bare human finger may cause a different energy measurement signal than a human finger wearing a plastic glove. The top graph depicts an example of an energy measurement signal from a bare human finger contacting an ultrasound input device, where the characteristics drop off rapidly and increase relatively rapidly back to the baseline signal when the finger is removed. However, the bottom graph depicts an example of an energy measurement signal from a human finger wearing a plastic glove. When the plastic glove is donned, the energy measurement signal has different characteristics than when the plastic glove is not donned. For example, when wearing plastic gloves, the energy measurement signal may increase back to the baseline signal discernably slower after the finger is removed. This relatively slow increase is due in part to the slight tackiness between the plastic glove and the interaction of the substrate of the ultrasound input device. Although a human finger may pull open more quickly, the plastic glove around the finger will tend to remain on the substrate for a short duration, which is discernable in the resulting energy measurement signal. This type of comparison may be used to determine the material from which the touch event originated.

XI additional piezoelectric array design

Fig. 64 is a schematic diagram of a piezoelectric resonator array 6400 that includes a piezoelectric cantilever 6402 that may be used in an ultrasonic input device, in accordance with certain aspects of the present disclosure. The piezoelectric resonator array 6400 may comprise a set of piezoelectric cantilevers 6402 on a substrate 6404. The piezoelectric resonator array 6400, when acoustically coupled to a material layer (e.g., material layer 102 of fig. 1), may operate at a particular acoustic resonance. When a touch event is occurring, the touch event may cause the piezoelectric resonator array 6400 to resonate differently. Such changes in acoustic resonance caused by touch events may be detected and used as sensor signals in an ultrasound input device, such as in place of PMUTs. In addition, the piezoelectric cantilever 6402 may be driven to bend and thus cause a transmitted signal.

Fig. 65 is a schematic diagram of a piezoelectric resonator array 6500 that includes piezoelectric pillars 6502 that can be used in an ultrasonic input device, in accordance with certain aspects of the present disclosure. The piezoelectric resonator array 6500 can comprise a set of piezoelectric pillars 6502 on a substrate 6504. When acoustically coupled to a material layer (e.g., material layer 102 of fig. 1), piezoelectric resonator array 6500 can operate at a particular acoustic resonance. When a touch event is occurring, the touch event may cause the piezoelectric resonator array 6500 to resonate differently. Such changes in acoustic resonance due to touch events may be detected and used as sensor signals in an ultrasound input device, such as in lieu of PMUTs. In addition, the piezoelectric pillars 6502 may be driven to bend and thus cause a transmitted signal. The piezoelectric pillars 6502 may be arranged in any suitable pattern, such as a hexagonal grid.

Aspects of the embodiments may be implemented in the form of control logic using hardware circuitry (e.g., application specific integrated circuits or field programmable gate arrays) in a modular or integrated manner and/or using computer software with a generally programmable processor. As used herein, a processor may include a single core processor, a multi-core processor on the same integrated chip, or multiple processing units and dedicated hardware on a single circuit board or networked. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will know and understand other ways and/or methods to implement embodiments of the invention using hardware and combinations of hardware and software.

Any of the software components or functions described in the present application may be implemented as software code that may be executed by a processor using any suitable computer language (e.g., java, C, C++, C#, objective-C, swift, or scripting language, such as Perl or Python using, for example, conventional or object-oriented techniques). The software code may be stored as a series of instructions or commands on a computer readable medium for storage and/or transmission. Suitable non-transitory computer readable media may include Random Access Memory (RAM), read Only Memory (ROM), magnetic media such as a hard disk drive or floppy disk, or optical media such as Compact Disk (CD) or DVD (digital versatile disk), flash memory, and the like. The computer readable medium may be any combination of such storage or transmission means.

Such programs may also be encoded and transmitted using carrier signals suitable for transmission over wired, optical, and/or wireless networks (including the internet) conforming to a variety of protocols. Thus, a computer readable medium may be established using a data signal encoded with such a program. The computer readable medium encoded with the program code may be packaged with a compatible device or provided separately from the other devices (e.g., downloaded over the internet). Any such computer-readable medium may reside on or within a single computer product (e.g., a hard drive, CD, or entire computer system), and may reside on or within different computer products within a system or network. The computer system may include a monitor, printer, or other suitable display for providing any of the results mentioned herein to the user.

Any of the methods described herein may be performed in whole or in part using a computer system that includes one or more processors that are configurable to perform a plurality of steps. Accordingly, embodiments may relate to a computer system configured to perform the steps of any of the methods described herein, potentially with different components performing the respective steps or the respective groups of steps. The steps of the methods herein, although presented as numbered steps, may be performed simultaneously or at different times or in a different order. Furthermore, portions of these steps may be used in combination with portions of other steps of other methods. Moreover, all or portions of the steps may be optional. Furthermore, any steps of any method may be performed using modules, units, circuits, or other means for performing the steps.

The particular details of the particular embodiments may be combined in any suitable manner without departing from the spirit and scope of embodiments of the invention. However, other embodiments of the invention may involve specific embodiments relating to each individual aspect or specific combinations of these individual aspects.

The foregoing description of the exemplary embodiments of the invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and many modifications and variations are possible in light of the above teaching.

Recitation of "a", "an", or "the" means "one or more", unless expressly recited to the contrary. Unless indicated to the contrary, the use of "or" means "exclusive or" rather than "exclusive or". References to a "first" component do not necessarily require the provision of a second component. Furthermore, reference to a "first" or "second" component does not limit the referenced component to a particular location unless explicitly stated. The term "based on" is intended to mean "based at least in part on".

All patents, patent applications, publications, and descriptions mentioned herein are incorporated by reference in their entirety for all purposes. It is not admitted to be prior art.

Claims

1. A system, comprising:

An ultrasound input device coupled to a layer of material having an outer surface located opposite the ultrasound input device, wherein the ultrasound input device is coupled to the layer of material to transmit a transmitted signal through the layer of material toward the outer surface and to receive a set of reflected ultrasound signals associated with the transmitted signal, wherein the set of reflected ultrasound signals includes at least one reflected ultrasound signal, and wherein the set of reflected ultrasound signals is associated with a touch event between an object and the outer surface of the layer of material, and

One or more data processors configured to:

determining energy signals, the energy signals comprising energy measurements over time, each energy measurement corresponding to a summed value obtained from a portion of the set of reflected ultrasound signals;

extracting feature information associated with the energy signal;

determining an inference associated with the object based on the extracted feature information, and

An output signal associated with the determined inference is generated.

2. The system of claim 1, wherein extracting the characteristic information comprises identifying a pattern in the energy signal associated with a dip in the energy measurement associated with the touch event.

3. The system of claim 2, wherein identifying the pattern comprises identifying one or more of a depth of the dip, a duration of the dip, a presence of a subsequent dip after the dip, a delay between the dip and another dip, and a rate of change of an energy signal at the dip edge.

4. The system of claim 2, wherein identifying the pattern comprises identifying a change in an energy signal attributable to temperature drift in the material layer, and wherein determining the inference comprises estimating a relative temperature of the object based on the identified change in the energy signal attributable to temperature drift in the material layer.

5. The system of claim 2, wherein determining the inference comprises comparing the pattern to stored data, wherein the stored data is associated with a previous touch event of the outer surface.

6. The system of claim 1, wherein determining the inference comprises using the characteristic information to determine that the touch event is associated with one selected from the group consisting of a bare human finger, a wet human finger, a dry human finger, and a gloved human finger.

7. The system of claim 1, wherein the characteristic information comprises a magnitude of the energy signal and/or a change in the energy signal, and wherein determining the inference comprises:

comparing at least one selected from the size of (a) and the change of (b) to a respective threshold to determine whether the touch event is associated with a human finger or a water drop.

8. The system of claim 1, wherein determining the inference comprises using the characteristic information to determine one or more of a touch manner of the touch event, a touch strength associated with the touch event, and a physical characteristic of the object.

9. The system of claim 8, wherein determining the inference further comprises identifying that the object is associated with one of a plurality of users based on associating the touch event with one or more of a touch manner of the touch event, a touch strength associated with the touch event, or a physical characteristic of the object.

10. The system of claim 8, wherein the physical characteristic of the object is a measurement associated with a portion of the fingerprint contacting the outer surface.

11. The system of claim 1, wherein the one or more data processors are further configured to:

determining an additional signal associated with an additional sensor associated with the ultrasound input device, wherein determining the inference further comprises using the additional signal.

12. The system of claim 1, wherein the energy signal represents energy of the set of reflected ultrasound signals occurring within an energy measurement window, and wherein determining the energy signal comprises determining the energy signal by integrating reflected ultrasound signals of the set of reflected ultrasound signals occurring within the energy measurement window.

13. The system of claim 12, wherein at least one of the one or more data processors comprises a summing circuit or an integrating circuit, wherein the summing circuit or the integrating circuit is configured to:

Generating the energy signal based on the set of reflected ultrasound signals, the energy signal comprising the energy measurements over time, wherein each energy measurement is a summation value of the set of reflected ultrasound signals within the energy measurement window.

14. A computer-implemented method, comprising:

Transmitting a transmitted signal using an ultrasonic input device coupled to a layer of material having an outer surface located opposite the ultrasonic input device from the layer of material, wherein the transmitted signal is directed toward the outer surface through the layer of material;

receiving a set of reflected ultrasonic signals associated with the transmitted signals, wherein the set of reflected ultrasonic signals includes at least one reflected ultrasonic signal, and wherein the set of reflected ultrasonic signals is associated with a touch event between an object and the outer surface of the material layer;

extracting feature information associated with the energy signal;

An output signal associated with the determined inference is generated.

15. The method of claim 14, wherein extracting the characteristic information includes identifying a pattern in an energy signal associated with a dip in the energy measurement associated with the touch event.

16. The method of claim 15, wherein identifying the pattern comprises identifying one or more of a depth of the dip, a duration of the dip, a presence of a subsequent dip after the dip, a delay between the dip and another dip, and a rate of change of an energy signal at the dip edge.

17. The method of claim 15, wherein identifying the pattern comprises identifying a change in an energy signal attributable to temperature drift in the material layer, and wherein determining the inference comprises estimating a relative temperature of the object based on the identified change in the energy signal attributable to temperature drift in the material layer.

18. The method of claim 15, wherein determining the inference comprises comparing the pattern to stored data, wherein the stored data is associated with a previous touch event of the outer surface.

19. The method of claim 14, wherein determining the inference comprises using the characteristic information to determine that the touch event is associated with one selected from the group consisting of a bare human finger, a wet human finger, a dry human finger, and a gloved human finger.

20. The method of claim 14, wherein determining the inference comprises using the characteristic information to determine one or more of a touch manner of the touch event, a touch strength associated with the touch event, or a physical characteristic of the object.

21. The method of claim 20, wherein determining the inference further comprises identifying that the object is associated with one of a plurality of users based on associating the touch event with one or more of a touch manner of the touch event, a touch strength associated with the touch event, or a physical characteristic of the object.

22. The method of claim 14, further comprising determining an additional signal associated with an additional sensor associated with the ultrasound input device, wherein determining the inference further comprises using the additional signal.

23. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to enable a data processing apparatus to perform operations comprising:

extracting feature information associated with the energy signal;

An output signal associated with the determined inference is generated.

24. The computer program product of claim 23, wherein extracting the characteristic information comprises identifying a pattern in an energy signal associated with a dip in the energy measurement associated with the touch event.

25. The computer program product of claim 24, wherein identifying the pattern comprises identifying one or more of a depth of the dip, a duration of the dip, a presence of a subsequent dip after the dip, a delay between the dip and another dip, and a rate of change of an energy signal at the dip edge.

26. The computer program product of claim 24, wherein identifying the pattern comprises identifying a change in an energy signal attributable to temperature drift in the layer of material, and wherein determining the inference comprises estimating a relative temperature of the object based on the identified change in the energy signal attributable to temperature drift in the layer of material.

27. The computer program product of claim 24, wherein determining the inference comprises comparing the pattern to stored data, wherein the stored data is associated with a previous touch event of the outer surface.

28. The computer program product of claim 23, wherein determining the inference comprises using the characteristic information to determine that the touch event is associated with one selected from the group consisting of a bare human finger, a wet human finger, a dry human finger, and a gloved human finger.

29. The computer program product of claim 24, wherein determining the inference comprises using the characteristic information to determine one or more of a manner of touch of the touch event, a strength of touch associated with the touch event, or a physical characteristic of the object.

30. The computer program product of claim 29, wherein determining the inference further comprises identifying that the object is associated with one of a plurality of users based on associating the touch event with one or more of a touch manner of the touch event, a touch strength associated with the touch event, or a physical characteristic of the object.