HK40077304B - Multiband limiter modes and noise compensation methods - Google Patents

Description

Multi-band limiter modes and noise compensation methods

Cross-references to related applications

This application claims priority to the following U.S. provisional application numbers: 62/945,292, filed December 9, 2019; 63/198,995, filed November 30, 2020; 62/945,303, filed December 9, 2019; 63/198,996, filed November 30, 2020; 63/198,997, filed November 30, 2020; 62/945,607, filed December 9, 2019; 63/198,998, filed November 30, 2020; and 63/198,999, filed November 30, 2020, each of which is incorporated herein by reference in its entirety.

Technical Field

This disclosure relates to systems and methods for noise compensation.

Background Technology

Audio and video devices (including, but not limited to, televisions and associated audio equipment) are widely deployed. Some of these devices are configured to implement noise compensation algorithms that attempt to compensate for noise in the environment. While existing systems and methods for noise compensation offer benefits, improved systems and methods are still desired.

Symbols and terms

Throughout this disclosure, including in the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used synonymously to refer to any sound-generating transducer (or a group of transducers) driven by a single loudspeaker feed. A typical set of headphones includes two loudspeakers. A loudspeaker can be implemented to include multiple transducers (e.g., a woofer and a tweeter), which can be driven by a single common loudspeaker feed or multiple loudspeaker feeds. In some examples, the loudspeaker feeds can undergo different processing in different circuit branches coupled to different transducers.

Throughout this disclosure, including in the claims, the expression “to” perform an operation on a signal or data (e.g., to filter, scale, transform, or apply gain to a signal or data) is used broadly to indicate performing an operation directly on a signal or data or on a processed version of a signal or data (e.g., a version of a signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

Throughout this disclosure, including in the claims, the term "system" is used broadly to refer to a device, system, or subsystem. For example, a subsystem implementing a decoder may be referred to as a decoder system, and a system including such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, wherein the subsystem generates M inputs, and the other X-M inputs are received from an external source) may also be referred to as a decoder system.

Throughout this disclosure, including in the claims, the term "processor" is used broadly to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) for performing operations on data (e.g., audio or video or other image data). Examples of processors include field-programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and/or otherwise configured to perform pipelined processing of audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

Throughout this disclosure, including in the claims, the terms "couples" or "coupled" are used to mean a direct or indirect connection. Therefore, if a first device is coupled to a second device, the connection can be achieved either through a direct connection or through an indirect connection via other devices and connections.

As used herein, a "smart device" is an electronic device capable of operating interactively and/or autonomously to some extent, typically configured to communicate with one or more other devices (or networks) via various wireless protocols such as Bluetooth, Zigbee, Near Field Communication, Wi-Fi, Li-Fi, 3G, 4G, and 5G. Some notable types of smart devices include smartphones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, phablets and tablets, smartwatches, smart bracelets, smart keychains, and smart audio devices. The term "smart device" can also refer to devices exhibiting properties of ubiquitous computing, such as artificial intelligence.

In this document, the term "intelligent audio device" is used to refer to an intelligent device, which is a single-purpose audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of a virtual assistant function). A single-purpose audio device is a device that includes or is coupled to at least one microphone (and optionally includes or is coupled to at least one speaker and/or at least one camera) and is largely or primarily designed to perform a single purpose (e.g., a television (TV)). For example, although a TV can generally play (and is considered capable of playing) audio from program material, in most instances, modern TVs run some kind of operating system on which applications (including TV-watching applications) run natively. In this sense, a single-purpose audio device with (multiple) speakers and (multiple) microphones is typically configured to run local applications and/or services to directly utilize said (multiple) speakers and (multiple) microphones. Some single-purpose audio devices may be configured to be combined to enable audio playback in a specific zone or user-configured area.

A common type of multipurpose audio device is an audio device that implements at least some aspects of virtual assistant functionality, although other aspects of the virtual assistant functionality may be implemented by one or more other devices, such as one or more servers, and the multipurpose audio device is configured to communicate with said one or more servers. Such a multipurpose audio device may be referred to herein as a "virtual assistant". A virtual assistant is a device (e.g., a smart speaker or voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally includes or is coupled to at least one speaker and/or at least one camera). In some examples, a virtual assistant may provide the ability to use multiple devices (different from the virtual assistant) for applications that are, in a sense, cloud-enabled or otherwise not fully implemented in or on top of the virtual assistant itself. In other words, at least some aspects of the virtual assistant functionality (e.g., speech recognition functionality) may (at least partially) be implemented by one or more servers or other devices, and the virtual assistant may communicate with said one or more servers or other devices via a network (e.g., the Internet). Virtual assistants may sometimes work together, for example, in a discrete and conditionally defined manner. For example, two or more virtual assistants may work together in the sense that one of them (e.g., the virtual assistant most certain that it has heard the wake word) responds to the wake word. In some implementations, the connected virtual assistants can form a constellation, which can be managed by a main application, which may be (or implement) the virtual assistants.

In this document, "wake word" is used broadly to refer to any sound (e.g., a human-spoken word or other sound) in which a smart audio device is configured to wake up in response to the detection ("hearing") of a sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, "wake up" means that the device enters a state of waiting (in other words, listening) for a sound command. In some instances, the term "wake word" as used herein may include more than one word, such as a phrase.

In this document, the term "wake word detector" refers to a device configured to continuously search for alignments between real-time sound (e.g., speech) features and a training model (or software including instructions for configuring the device to continuously search for alignments between real-time sound features and a training model). Typically, a wake word event is triggered whenever the wake word detector determines that the probability of detecting a wake word exceeds a predefined threshold. For example, this threshold might be a predetermined threshold adjusted to provide a reasonable trade-off between a false acceptance rate and a false rejection rate. Following a wake word event, the device may enter a state (which may be referred to as a "wake-up" state or an "attention" state) in which the device listens for commands and passes the received commands to a larger, more computationally intensive recognizer.

As used herein, the terms "program stream" and "content stream" refer to a collection of one or more audio signals and, in some instances, video signals, at least a portion of which are to be heard together. Examples include music anthologies, movie soundtracks, movies, television programs, audio portions of television programs, podcasts, live voice calls, synthesized speech responses from intelligent assistants, and so on. In some instances, a content stream may include multiple versions of at least a portion of an audio signal, such as the same dialogue in more than one language. In such instances, only one version of the audio data or a portion thereof is intended to be reproduced at a time (e.g., a version corresponding to a single language).

Summary of the Invention

At least some aspects of this disclosure can be implemented via one or more audio processing methods, including but not limited to content streaming processing methods. In some instances, the methods can be implemented at least in part by a control system and/or via instructions (e.g., software) stored on one or more non-transient media. Some such methods involve receiving a content stream comprising input audio data by a control system and via an interface system. Some such methods involve receiving at least one type of level adjustment instruction related to the playback of audio data by a control system and via an interface system. Some such methods involve controlling the level of input audio data by a control system based on at least one type of level adjustment instruction to produce level-adjusted audio data. Some such methods involve determining a multi-band limiter configuration by a control system at least in part based on at least one type of level adjustment instruction. Some such methods involve configuring a multi-band limiter by a control system according to the multi-band limiter configuration. Some such methods involve applying a multi-band limiter to level-adjusted audio data to produce multi-band-limited audio data. Some such methods involve providing multi-band-limited audio data to one or more audio reproduction transducers of an audio environment.

According to some implementations, at least one type of level adjustment indication may include a user input level adjustment indication received via user input and/or a noise compensation level adjustment indication received from a noise compensation module. In some examples, if receiving at least one type of level adjustment indication involves receiving a user input level adjustment indication, determining the multi-band limiter configuration may involve determining a tone preservation configuration. According to some examples, if receiving at least one type of level adjustment indication involves receiving a noise compensation level adjustment indication, determining the multi-band limiter configuration may involve changing the tone preservation function. In some instances, changing the tone preservation function may involve at least partially disabling the tone preservation function. According to some examples, the noise compensation level adjustment indication may correspond to the ambient noise level in the audio environment. In some such examples, changing the tone preservation function may involve changing the tone preservation function at least partially based on the ambient noise level. In some instances, changing the tone preservation function may be at least partially based on the ambient noise level. Some examples may involve reproducing multi-band limited audio data on one or more audio reproduction transducers in the audio environment to provide reproduced audio data. Some such examples may involve determining or estimating the masking effect of the ambient noise level on the reproduced audio data. In some instances, altering the timbre preservation feature can be based at least in part on masking effects. In other examples, the timbre preservation configuration can be band-dependent.

In some instances, receiving at least one type of level adjustment indication may involve receiving both a user-input level adjustment indication and a noise compensation level adjustment indication. In some such examples, determining a multi-band limiter configuration may involve determining a tone preservation configuration that is at least partially based on a weighted average of the user-input level adjustment indication and the noise compensation level adjustment indication.

Some examples may also involve causing a change in the operation of a noise compensation module when multi-band limited audio data causes one or more audio reproduction transducers in an audio environment to operate outside the linear range. In some instances, the control system may cause one or more audio reproduction transducers in the audio environment to operate outside the linear range, at least in part, based on a noise compensation level adjustment indication and/or noise estimation. In some examples, the change in the operation of the noise compensation module may involve altering the echo canceller function of the noise compensation module. According to some examples, the change in the operation of the noise compensation module may involve causing the noise compensation module to use only the silence playback interval as input to the noise estimator of the noise compensation module. The silence playback interval may be an example of an audio signal that is at or below a threshold level in at least one frequency band and/or at least one time interval. In some instances, the multi-band limited audio data that causes one or more audio reproduction transducers in the audio environment to operate outside the linear range may be based on a noise compensation level adjustment corresponding to a high ambient noise level in the audio environment.

According to some implementations, the noise compensation module can be a subsystem of the control system. In some examples, the level adjuster module of the control system can be configured to control the level of input audio data to produce leveled audio data. According to some such examples, a method can also involve providing multi-band limiter feedback from a multi-band limiter to the level adjuster module. In some instances, the multi-band limiter feedback can indicate the amount of limitation applied by the multi-band limiter to each of a plurality of frequency bands of the leveled audio data. Some examples can also involve the level adjuster module controlling the level of one or more of the plurality of frequency bands at least in part based on the multi-band limiter feedback.

Some alternative aspects of this disclosure can also be implemented via one or more audio processing methods, including but not limited to content streaming processing methods. In some instances, the methods may be implemented at least in part by a control system and/or via instructions (e.g., software) stored on one or more non-transient media. Some such methods involve: receiving a content stream comprising input audio data by a control system and via an interface system; and applying a multi-band limiter to the audio data or a processed version of the audio data by the control system to produce multi-band limited audio data. Some such methods involve: determining whether the multi-band limited audio data, when played back on one or more audio reproduction transducers in an audio environment, would cause one or more audio reproduction transducers to operate outside the linear range; and controlling whether an acoustic echo canceller updates one or more filter coefficients or a noise estimator updates a noise estimate by the control system based at least in part on whether the multi-band limited audio data would cause one or more audio reproduction transducers in the audio environment to operate outside the linear range. Some such methods involve providing multi-band limited audio data to one or more audio reproduction transducers in an audio environment.

In some examples, controlling whether an acoustic echo canceller updates one or more filter coefficients may involve controlling the acoustic echo canceller not to update one or more filter coefficients if the multi-band limited audio data would cause one or more audio reproduction transducers of the audio environment to operate outside the linear range. Some implementations may involve: receiving at least one type of level adjustment indication related to the playback of audio data by a control system; determining a multi-band limiter configuration by the control system and at least in part based on the at least one type of level adjustment indication; and configuring the multi-band limiter according to the multi-band limiter configuration by the control system. Some such examples may also involve controlling the level of input audio data by the control system based on at least one type of level adjustment indication to produce level-adjusted audio data. Applying a multi-band limiter to audio data or a processed version of audio data may involve applying a multi-band limiter to level-adjusted audio data. In some examples, at least one type of level adjustment indication may include a user input level adjustment indication received via user input and/or a noise compensation level adjustment indication received from a noise compensation module including the acoustic echo canceller. According to some examples, if receiving at least one type of level adjustment indication involves receiving a user-input level adjustment indication, then determining the multi-band limiter configuration may involve determining the tone preservation configuration.

According to some implementations, if receiving at least one type of level adjustment indication involves receiving a noise compensation level adjustment indication, determining the multiband limiter configuration may involve changing the tone preservation function. In some instances, changing the tone preservation function may involve at least partially disabling the tone preservation function. In some examples, the noise compensation level adjustment indication may correspond to the ambient noise level in the audio environment. In some such examples, changing the tone preservation function may involve changing the tone preservation function at least in part based on the ambient noise level. Some examples may also involve reproducing multiband-limited audio data on one or more audio reproduction transducers in the audio environment to provide reproduced audio data. Some such examples may also involve determining or estimating the masking effect of the ambient noise level on the reproduced audio data. Changing the tone preservation function may be at least in part based on the masking effect. In some examples, the tone preservation configuration may be band-dependent.

In some examples, receiving at least one type of level adjustment indication may involve receiving both a user-input level adjustment indication and a noise compensation level adjustment indication. In some such examples, determining the multi-band limiter configuration may involve determining a tone preservation configuration that is at least partially based on a weighted average of the user-input level adjustment indication and the noise compensation level adjustment indication.

According to some implementations, the control system may operate one or more audio reproduction transducers of an audio environment outside the linear range, at least in part, based on a noise compensation level adjustment indication and/or noise estimation. In some such examples, the control system may operate one or more audio reproduction transducers of an audio environment outside the linear range, at least in part, based on a noise compensation level adjustment corresponding to a high ambient noise level in the audio environment.

Some examples may involve causing one or more additional noise compensation module operations to change when multi-band limited audio data causes one or more audio reproduction transducers in the audio environment to operate outside the linear range. These additional noise compensation module operations may involve causing the noise compensation module to use only the silence playback interval as input to its noise estimator. The silence playback interval may be an instance of an audio signal at or below a threshold level in at least one frequency band and/or an instance of an audio signal at or below a threshold level during at least one time interval. In some embodiments, the noise compensation module may be a subsystem of a control system.

According to some examples, the level adjuster module of the control system can be configured to control the level of input audio data to produce leveled audio data. Some such examples may also involve providing multi-band limiter feedback from a multi-band limiter to the level adjuster module. The multi-band limiter feedback may, for example, indicate the amount of limitation applied by the multi-band limiter to each of the multiple frequency bands of the leveled audio data. Some such examples may also involve the level adjuster module controlling the level of one or more frequency bands of the multiple frequency bands, at least in part, based on the multi-band limiter feedback.

Some or all of the operations, functions, and/or methods described herein can be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include, for example, the memory devices described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Accordingly, some innovative aspects of the subject matter described herein can be implemented via one or more non-transitory media on which software is stored.

At least some aspects of this disclosure can be implemented via an apparatus. For example, one or more devices may be able to perform at least partially the methods disclosed herein. In some embodiments, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof.

Details of one or more embodiments of the subject matter described herein are set forth in the following figures and description. Other features, aspects, and advantages will become apparent from the description, figures, and claims. Note that the relative dimensions in the following figures may not be drawn to scale.

Attached Figure Description

Figures 1A, 1B, and 1C show examples of loudspeakers operating within a linear range.

Figures 1D, 1E, and 1F show examples of loudspeakers operating in a non-linear range.

Figures 1G, 1H, and 1I illustrate examples of multi-band limiter operation.

Figures 1J, 1K, and 1L illustrate examples of the operation of a tone-preserving multi-band limiter according to one embodiment.

Figure 1M shows an example of a noise compensation system.

Figure 1N shows a portion of a noise compensation system, which includes an example of a level adjuster configured to modify leveled audio data based on a compression feedback signal from a multi-band limiter.

Figure 10 shows a more detailed version of Figure 1N based on an example.

Figure 2A shows an example of a graphical user interface (GUI) for setting the threshold and other parameters of a multi-band limiter.

Figure 2B shows another example of the threshold and other parameters of a multi-band limiter.

Figure 2C is a graph showing an example of a threshold for a certain range of frequencies.

Figure 2D is a graph showing another example of a threshold for a certain range of frequencies.

Figure 2E is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure.

Figure 3 is a flowchart outlining an example of the disclosed method.

Figures 4A and 4B show examples of the tone preservation modifier module.

Figure 4C shows an example of a band isolation modifier.

Figure 5 is a flowchart outlining an example of another disclosed method.

Figure 6A is a graph showing an example of the time interval of amplifier overdrive.

Figure 6B shows an example of a signal that can be sent to an echo canceller corresponding to the curve in Figure 6A.

Figure 7 illustrates a system configured to control an automatic echo canceller (AEC) based at least in part on the amount of “overdrive” occurring in the system.

Figure 8 shows an example of a system configured to determine the overdrive quantity.

Figure 9 is a graph showing the noise estimation and output level based on an example.

Figure 10 shows an example floor plan of an audio environment, which in this example is a living space.

In the various figures, similar reference numerals and names indicate similar elements.

Detailed Implementation

The noise compensation algorithm is designed to compensate for noise in what is referred to as the “audio environment” in this paper. As used herein, the term “audio environment” is not limited to components of an audio system, such as audio reproduction transducers, amplifiers, etc. Instead, the term “audio environment” generally refers to the environment in which such components may reside and/or the environment in which one or more listeners may listen to the playback audio. In some examples, the audio environment may be a home audio environment. In such instances, the audio environment may correspond to one or more rooms in a home. In other examples, the audio environment may be an office environment, a car environment, a train or bus environment, a street or sidewalk environment, a park or other outdoor environment, or other types of environments.

Noise compensation methods can be designed, for example, to compensate for noise in the audio environment by adjusting the level of the output signal through one or more audio reproduction transducers, based at least in part on the amount of noise in the audio environment. One of the challenges in implementing noise compensation methods is the limited range of linear operation of audio reproduction transducers in the physical world.

Figures 1A, 1B, and 1C illustrate examples of a loudspeaker operating within its linear range. Figure 1A is a graph showing an example of a signal 100 provided by the amplifier output and supplied to loudspeaker 101. Figure 1B shows an example of a cross-section of loudspeaker 101. In this example, signal 100 is a sinusoidal input signal with a peak-to-peak range of 10 volts. When a signal within this range is supplied to loudspeaker 101, it causes a range of voice coil displacement, during which the voice coil 102 remains within a portion of a magnetic field with uniform or substantially uniform magnetic field strength generated by magnet 103. Therefore, the displacement of the voice coil 102 and the diaphragm 104 is within the linear range.

Figure 1C is a graph illustrating an example of the displacement of the voice coil 102 and diaphragm 104 when the loudspeaker 101 is driven by the signal 100. In this example, the displacement is sinusoidal and proportional to the input voltage of the signal 100. Therefore, the signal 100 does not produce distortion and the loudspeaker 101 operates within the linear range.

When audio devices, such as reproducing transducers, operate outside the linear range, the audio device will distort and may sound unpleasant. Figures 1D, 1E, and 1F illustrate examples of a loudspeaker operating in a nonlinear range. Figure 1D is a graph providing another example of a signal 100 output from an amplifier and supplied to loudspeaker 101. Figure 1E shows another example of a cross-section of loudspeaker 101. In this example, signal 100 is a sinusoidal input signal with a peak-to-peak range of 20 volts. When a signal within this range is supplied to loudspeaker 101, it causes a range of voice coil displacement during which the voice coil 102 does not remain within a portion of the magnetic field generated by magnet 103 where the magnetic field strength is uniform or substantially uniform. Therefore, the displacement of voice coil 102 and diaphragm 104 is in the nonlinear range. Another source of nonlinearity in loudspeaker 101 is the elasticity of the suspension. The periphery of diaphragm 104 is connected to the frame via suspension 105. As diaphragm 104 moves forward and backward, suspension 105 stretches to accommodate this movement. However, the elasticity of this material has its limits, and as the diaphragm 104 is driven more forcefully and its movement approaches these limits, the driving force causing the movement becomes less able to overcome the reaction force of the elastic suspension 105 as it attempts to return to its neutral position. Similar to magnetic nonlinearity, this situation also results in the output signal no longer being identical to the input.

Figure 1F is a graph illustrating an example of the displacement of the voice coil 102 and diaphragm 104 when the loudspeaker 101 is driven by the signal 100. In this example, the displacement is non-sinusoidal and not proportional to the input voltage of the signal 100. The output shown in Figure 1F is not proportional to the input voltage, but corresponds to both the input voltage and the effect of loudspeaker distortion. Therefore, in this example, the signal 100 produces distortion and the loudspeaker 101 operates in a non-linear range.

Echo cancellers, typically implemented via adaptive linear filters or through machine learning (e.g., via trained neural networks), are fundamental components of many noise compensation systems. While an echo canceller may adapt perfectly, it performs even worse when adapting to nonlinear systems (e.g., in response to a loudspeaker operating in a nonlinear range). Furthermore, extended loudspeaker operation in a nonlinear range is likely to damage the loudspeaker.

Multiband limiters allow for frequency-dependent control of a loudspeaker's dynamic range. Multiband limiters are typically configured to increase the sound pressure level a loudspeaker can produce while ensuring that the loudspeaker does not introduce nonlinear distortion.

Figures 1G, 1H, and 1I illustrate examples of multi-band limiter operation. Figure 1G is a graph providing an example of an audio signal 100 supplied to the multi-band limiter. In this example, the audio signal 100 corresponds to a “white” input spectrum, where the level of each band is the same.

Figure 1H illustrates an example of a multi-band limiter threshold for each of multiple frequency bands. In this example, each threshold is at or below the level of the input audio signal 100. For example, the multi-band limiter threshold may correspond to the capabilities of a particular loudspeaker (e.g., distortion profile) and may be implemented by or for that loudspeaker. In some such examples, the multi-band limiter threshold may be preset at the factory where the loudspeaker is manufactured.

Figure 1I is a graph illustrating an example of the output of a multi-band limiter with the thresholds shown in Figure 1H when the input audio signal 100 shown in Figure 1G is provided. In this example, because each threshold is at or below the level of the input audio signal 100, the output for each frequency band corresponds to the multi-band limiter threshold for that band. Therefore, in this example, the frequency content or timbre of the input audio signal 100 is not preserved.

As illustrated in the previous example, a multi-band limiter can significantly alter the spectral content or timbre of an input audio signal. Allowing a multi-band limiter to operate unrestricted can adversely affect the timbre of the output audio signal, potentially reducing the enjoyment of the music content.

Some multi-band limiters developed by the assignee can at least partially preserve the timbre of an input audio signal. Figures 1J, 1K, and 1L illustrate examples of operation of a timbre-preserving multi-band limiter according to one embodiment. The term "timbre preservation" can have various meanings as used herein. In a broad sense, a "timbre preservation" method is a method of at least partially preserving the frequency content or timbre of an input audio signal. Some timbre preservation methods can completely or almost completely preserve the frequency content of the input audio signal. A timbre preservation method may involve constraining the output signal levels of at least some frequency bands according to the output signal levels of at least some other frequency bands and/or an applied threshold. In some examples, a "timbre preservation" method may involve constraining the output signal levels of all non-isolated frequency bands at least to some extent. However, as described in more detail herein, in some examples, frequency bands may be completely isolated, while in other examples, frequency bands may be only partially isolated.

Figure 1J is a graph illustrating an example of an audio signal 100 provided to a multi-band limiter. In this example, as shown in Figure 1G, the level of the audio signal 100 is the same for each frequency band.

Figure 1K illustrates an example of a multi-band limiter threshold for each of multiple frequency bands. In this example, each threshold is at or below the level of the input audio signal 100. For example, the multi-band limiter threshold may correspond to the capability of a particular loudspeaker and may be implemented by or for that loudspeaker.

Figure 1L is a graph illustrating an example of the output of a timbre-preserving multiband limiter with the threshold shown in Figure 1K when the input audio signal 100 shown in Figure 1J is provided. In this example, the output for each band does not correspond to the multiband limiter threshold for that band. Instead, the timbre of the audio signal 100 is preserved by constraining the output signal level of each band to a minimum multiband limiter threshold. This example illustrates the extreme case of a 100% timbre-preserving multiband limiter. In most implementations, timbre preservation is not so extreme. For example, some timbre-preserving implementations may preserve the timbre in higher frequency bands while allowing at least partial isolation of some low-frequency bands. Alternatively or additionally, timbre-preserving methods may involve constraining the output signal level of a band according to less than the output signal levels of all other bands and/or an applied threshold (at least to some extent).

Figure 1M illustrates an example of a noise compensation system. In this example, noise compensation system 150 includes a level adjuster 152, a multi-band limiter 154, a loudspeaker 156, a microphone 157, and a noise estimator 159. According to this example (and other examples of noise compensation systems disclosed herein, including but not limited to those shown in Figures 1N and 1O), the level adjuster 152, multi-band limiter 154, and noise estimator 159 are implemented by a control system 110, which may be an instance of the control system 210 described below with reference to Figure 2E. According to some examples, the microphone 157 may also be controlled by and/or included as part of the control system 110. As noted elsewhere herein, the control system 110 may reside in a single device or multiple devices, depending on the specific implementation. In some examples, the level adjuster 152, multi-band limiter 154, and/or noise estimator 159 may be implemented via software, for example, according to instructions stored on one or more non-transient media.

As with the other figures provided in this disclosure, the types, quantities, and arrangements of elements shown in Figure 1M are merely examples. Other embodiments may include more, fewer, and/or different elements. For example, some embodiments may include multiple audio reproduction transducers. Some embodiments may include multiple microphones.

According to this embodiment, the noise compensation system 150 includes a microphone 157 configured to detect sound in the audio environment including the noise compensation system 150 and provide a corresponding microphone signal 158 to the noise estimator 159. The sound may include sound generated by the loudspeaker 156 and ambient noise in the audio environment (also referred to herein as ambient noise or background noise). As noted elsewhere in this document, the term "audio environment" is not intended to be limited to components of an audio system, such as audio reproduction transducers, amplifiers, etc. Instead, the term "audio environment" generally refers to the environment in which such components may reside and/or the environment in which one or more listeners may listen to the playback audio. In some examples, the audio environment may be a home audio environment. In such instances, the audio environment may correspond to one or more rooms in a home. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc.

In this example, noise estimator 159 is configured to estimate the level of background noise. According to this example, noise estimator 159 is configured to implement an echo canceller to reduce the likelihood that the audio data reproduced by loudspeaker 156 is part of the background noise estimate. In this example, noise estimator 159 is configured to receive multi-band limited audio data 155 output by multi-band limiter 154, which is also provided to loudspeaker 156. Multi-band limited audio data 155 is an example of what is referred to herein as a "loudspeaker reference," "loudspeaker reference," or "echo reference" for the echo canceller implemented by noise estimator 159. Implementing an echo canceller in noise estimator 159 prevents positive feedback loops based on the sound generated by loudspeaker 156. In this example, noise estimator 159 is configured to calculate a noise estimate of the ambient noise and provide noise estimator output 160 to level adjuster 152. In some examples, noise estimator output 160 will include a spectral noise estimate. For example, the noise estimator output 160 may include noise estimates for each of multiple frequency bands.

In this example, a level adjuster 152 is shown as receiving input audio data 151. In some instances, the input audio data 151 may correspond to a content stream including video data. Here, the level adjuster 152 is configured to control (e.g., raise, lower, or hold) the level of the input audio data 151. According to this example, the level adjuster 152 is configured to control the level of the input audio data 151 at least in part based on a noise level already measured using microphone 157. According to some examples, for example, the level adjuster 152 is configured to control the level of the input audio data 151 at least in part based on noise estimator output 160. Thus, the noise estimator output 160 is an example of what is referred to herein as a "level adjustment indication". More specifically, the noise estimator output 160 is an example of what is herein referred to as a "noise compensation level adjustment indication".

In this example, the level adjuster 152 is shown receiving user input 163 corresponding to level adjustment, which is another example of what is referred to herein as a level adjustment indication. More specifically, user input 163 is an example of what is referred to herein as a "user input level adjustment indication." It will be understood that the level adjuster 152 will not typically receive user input 163 continuously, but rather intermittently only when the user attempts to adjust the audio playback level by providing input, for example, via voice command (e.g., a voice command received by the control system 110 via microphone 157), via manual remote control, etc. The level adjuster 152 (or another element of the control system 110) may, for example, store a value in a memory device corresponding to the most recent user input 163. In this example, the level adjuster 152 is configured to control the level of input audio data 151 based on at least one type of level adjustment indication. Here, the level adjuster 152 is configured to provide level-adjusted audio data 153 to the multi-band limiter 154.

According to some examples, the level adjuster 152 may be configured to determine a noise compensation method based at least in part on the state of the noise estimator 159 and/or user input 163. Therefore, in some implementations, the level adjuster 152 may be configured to determine a noise compensation method based at least in part on the noise estimator output 160 and/or user input 163.

In some examples, the noise estimator 159 can determine which noise compensation method the level adjuster 152 should implement. In some such examples, the noise estimator output 160 can (e.g., via the noise estimator output 160 and/or via additional information) indicate to the level adjuster 152 which noise compensation method the level adjuster 152 should implement.

In some embodiments where the noise estimator 159 is a multi-band noise estimator, if the noise estimate has a set of unupdated frequency bands (e.g., in higher frequency bands) that have not been updated for a threshold amount of time (e.g., on the order of seconds, such as 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, etc.), then the noise estimator output 160 may indicate that the noise compensation method should switch to tone-preserving mode because the quality of the noise estimate in the unupdated frequency bands is lower, even though the quality of the noise estimate in the updated frequency bands may still be high. Alternatively or additionally, in some embodiments, the noise estimator may be configured to provide a quality metric or confidence score to the noise compensation block, which can use the quality metric or confidence score to determine which mode to be (or partially) in. For example, if the quality metric or confidence score indicates that the quality of the noise estimate is low, then the noise compensation block may determine that the noise compensation method should be in tone-preserving mode.

In some embodiments, the control system 110 (e.g., noise estimator 159) may be configured to provide a multi-band noise estimator function, which is described in International Publication No. WO2019/209973, filed April 24, 2019, entitled “Background Noise Estimation Using Gap Confidence,” in particular, the gap confidence value and its use are discussed on pages 16-18, and the aforementioned document is incorporated herein by reference.

According to some implementations, the frequency band within the timbre-preserving frequency range of the noise compensator, multi-band limiter, or both (e.g., the frequency range of the non-isolated band in Figure 2B described below) can be selected based on a quality metric for the noise estimation. For example, the quality metric for the noise estimation can correspond to the amount of time since the noise estimation of the frequency band was updated.

In some examples, the gain applied in one frequency band (e.g., via a noise compensator) may be unconstrained compared to the gain applied in another frequency band (e.g., compared to the gain applied in an adjacent frequency band). Therefore, according to this type of noise compensation method, the spectral content of the input audio signal is typically not preserved. Thus, this type of noise compensation method may be referred to herein as an "unconstrained" noise compensation method or a non-timbre-preserving noise compensation method.

According to some examples, a multi-band limiter 154 can be configured to apply compression to horizontally regulated audio data 153, at least in part, based on prior calibration or "adjustment" of the multi-band limiter 154, to prevent distortion in the loudspeaker 156 (and, in some instances, other audio reproduction transducers in the audio environment). In some such examples, the multi-band limited audio data 155 does not produce distortion in the loudspeaker 156, allowing the loudspeaker 156 to operate within a linear range. The adjustment can correspond to a multi-band limiter threshold for each of the multiple frequency bands. For example, the multi-band limiter threshold can correspond to the capability of the loudspeaker 156 (e.g., a distortion profile) and can be implemented by or for the loudspeaker. In some examples, the multi-band limiter threshold can be preset at the factory where the loudspeaker 156 is manufactured.

However, in some examples, the multi-band limiter 154 can be configured to apply compression to the level-adjusted audio data 153, which allows at least some distortion in the loudspeaker 156 and/or one or more other audio reproduction transducers of the audio environment. In such examples, one or more audio reproduction transducers of the audio environment can be allowed to operate outside the linear range, at least temporarily. In some such examples, the control system can allow one or more audio reproduction transducers of the audio environment to operate outside the linear range, at least in part, based on noise compensation level adjustment indications and/or noise estimation (e.g., noise estimator output 160 from noise estimator 159). Some such examples may also involve causing a change in the operation of the noise compensation module when the multi-band-limited audio data 155 causes one or more audio reproduction transducers of the audio environment to operate outside the linear range. Some detailed examples are disclosed elsewhere in this document.

According to some implementations, the control system 110 (e.g., level adjuster 152) can be configured to determine a multi-band limiter configuration based at least in part on one or more types of received level adjustment indications. In some such examples, if the received indication is a user-input level adjustment, the multi-band limiter configuration may be a tone-preserving configuration. Various examples of tone-preserving configurations are disclosed herein. In some such examples, determining the multi-band limiter configuration may involve altering the tone-preserving function of the multi-band limiter 154 if the received indication is a noise-compensated level adjustment.

In some such examples, the control system 110 (e.g., the level adjuster 152 or the multi-band limiter 154 itself) can configure the multi-band limiter 154 according to a determined multi-band limiter configuration. In some embodiments, the control system 110 (e.g., the level adjuster 152) can be configured to control the multi-band limiter 154 and/or the noise estimator 159 based at least in part on the output level and/or capability of the loudspeaker 156.

According to the example shown in Figure 1M, the level adjuster 152 is configured to control the multi-band limiter 154 via control signal 161. In some embodiments, if the received level adjustment instruction is a user-input level adjustment instruction, the tone preservation setting of the multi-band limiter 154 can be maintained, for example, as initially adjusted at the factory. Such an implementation helps ensure a pleasant experience for the user when adjusting the volume control.

According to some examples, if the received level adjustment instruction is a noise compensation level adjustment instruction, the tone preservation setting of the multi-band limiter 154 can be gradually turned off, for example, proportionally to the noise compensation level adjustment instruction. In some such examples, when noise is present, the played-out audio content can still be understood under noise, while the loss of fidelity is masked by the noise source.

In some implementations, if the control system 110 (e.g., leveling regulator 152) implements a tone-preserving noise compensation mode, the leveling regulator 152 can (e.g., via control signal 161) instruct the multi-band limiter 154 to also operate in tone-preserving mode and not allow the amplifier 156 to be overdriven. According to some examples, if the control system 110 implements an unconstrained noise compensation mode, the multi-band limiter 154 can also operate in a relatively less constrained mode (e.g., as described below with reference to Figures 2A and 2B) to maximize volume (e.g., to overdrive the amplifier 156).

Conversely, if the multi-band limiter 154 imposes a limit, in some embodiments, the control system 110 can make the noise compensation mode an unconstrained noise compensation mode (even if the noise compensation mode was previously a tone-preserving noise compensation mode), so that the volume can be maximized.

In the example shown in Figure 1M, the multi-band limiter 154 is configured to send an optional compression feedback signal 162 to the level adjuster 152. For example, the compression feedback signal 162 may indicate the amount of limitation that the multi-band limiter 154 applies to the leveled audio data 153. In some examples, the compression feedback signal 162 may indicate the amount of limitation that the multi-band limiter 154 applies to each of the multiple frequency bands of the leveled audio data 153.

Figure 1N illustrates a portion of a noise compensation system, including an example of a level adjuster configured to modify leveled audio data based on a compression feedback signal from a multi-band limiter. In this example, level adjuster 152 includes a bass enhancement module 167 and a noise compensation level adjuster 169. Figure 1N also indicates a multichannel stream 168 that has been processed by bass enhancement module 167 and, in this example, provided to noise compensation level adjuster 169. According to some examples, bass enhancement module 167 is configured to propagate bass from one amplifier to one or more other amplifiers. In some examples, bass enhancement module 167 is configured to determine, at least in part, how much input bass should be propagated to other speakers based on compression feedback signal 162.

In some implementations, the bass enhancement module 167 is configured to perform psychoacoustic bass enhancement (e.g., to perform virtual bass utilizing the absence of a fundamental phenomenon, such as as described below with reference to FIG10). According to some examples, the bass enhancement module 167 may be configured to determine how much virtual bass should be performed, at least in part, based on the compression feedback signal 162.

In some implementations, the bass enhancement module 167 is configured to receive noise estimation 160. The bass enhancement module 167 may use noise estimation 160, for example, to control the aggressiveness of bass propagation and/or virtual bass. In some such examples, if the noise level is high, the bass enhancement module 167 will be relatively more aggressive in propagating audio compared to a situation where volume limiting is caused by user input, even to the point of potentially propagating most or all of the human-audible spectrum instead of just bass frequencies (e.g., the bass enhancement module 167 propagates frequencies higher than bass frequencies to all loudspeakers). Further examples of “aggressiveness” are provided below. According to some implementations, the bass enhancement module 167 may also begin introducing virtual bass processing earlier than a situation where volume limiting is entirely due to user volume control.

Figure 10 shows a more detailed version of Figure 1N according to one example. As with the other figures provided in this disclosure, the types, numbers, and arrangements of elements shown in Figure 10 are merely illustrative. Other embodiments may include more, fewer, and/or different elements. In some alternative examples, the processing flow may differ from that shown in Figure 10, for example, in the case where noise compensation is used to control bass enhancement.

In the example shown in Figure 10, the level adjuster 152 includes an example of a bass enhancement module 178 and a noise compensation level adjuster 169. As noted elsewhere herein, according to some examples, the level adjuster 152 may be configured to determine a noise compensation method based at least in part on the state of the noise estimator 159 and/or user input 163. Thus, in some embodiments, the noise compensation level adjuster 169 may be configured to determine a noise compensation method based at least in part on the noise estimator output 160 and/or user input 163. Some such examples have been described above with reference to Figure 1M. Additional examples are disclosed in the corresponding descriptions of Figures 7 and 17 and U.S. Patent No. 8,090,120 (column 18, line 29 through 26, line 46, and column 34, line 41 through 35, line 11), which are incorporated herein by reference.

According to the example shown in Figure 10, input audio data 151 is shown as two input channels (e.g., stereo channels) CH1 and CH2. In Figure 10, noise estimation 160 (which may be provided by noise estimator 159 of Figure 1M) is provided to noise compensation level adjuster 169. In this example, compression feedback signal 162 provided to level adjuster 152 indicates the amount of limiting performed in multi-band limiter 154. According to some examples, compression feedback signal 162 may optionally be provided to both bass enhancement module 178 and noise compensation level adjuster 169. According to some such examples, compression feedback signal 162 may be provided to virtual bass (VB) block 171. If compression feedback signal 162 is provided to noise compensation level adjuster 169, in some examples, compression feedback signal 162 may be used to modify the overall predicted output level used within noise compensation level adjuster 169.

In some examples, the noise compensation level adjuster 169 optionally implements psychoacoustic volume control. According to some such examples, instead of applying a broadband gain that causes a change in gain across all frequency bands by the same amount—which may result in a change in the perceived spectrum—a specific loudness scaling factor can be associated with the volume control adjustment. In some such examples, the gain in each of the multiple frequency bands is changed by an amount that takes into account a human hearing model, such that ideally the perceived spectrum remains unchanged. Some relevant examples are disclosed in the “Time-Invariant and Frequency-Invariant Function Suitable for Volume Control” section of U.S. Patent No. 8,090,120 (column 26, line 48 through line 28, line 13), which is incorporated herein by reference.

Some psychoacoustic volume control implementations may involve mapping between the digital/electric domain and the acoustic domain (e.g., between decibels (dBFS) and sound pressure level (dBSPL) relative to full scale) within downstream processing components, taking into account gain and other factors. In some such examples, psychoacoustic volume control can be calibrated in a system region where the multi-band limiter 154 is not activated. This means that when the multi-band limiter 154 is activated, the digital-to-sound pressure level (SPL) mapping is typically incorrect (because the mapping is usually fixed). By instructing the multi-band limiter 154 to be limiting via the level adjuster 152 (e.g., to the noise compensation level adjuster 169), the digital-to-SPL mapping can be corrected, and therefore the required amount of noise compensation is not underestimated.

The block arrangement and operating sequence shown in Figure 10 arise from the fact that, according to some examples of psychoacoustic volume control implemented by the noise compensation level adjuster 169, the noise compensation level adjuster 169 can be combined with the loudness curve of the psychoacoustic bass enhancement to control the volume level relative to the noise estimate. Therefore, performing noise compensation block audio processing after bass enhancement block audio processing requires less conversion from the acoustic domain to the digital domain, in which the psychoacoustic system according to this example will operate.

Furthermore, in some systems where the noise compensation level adjuster 169 has exhausted its gain headroom, the noise compensation level adjuster 169 may instruct other blocks of the level adjuster 152 (such as blocks 167 and/or 171) to increase the amount of processing they perform to ensure that the noise compensation system 150 can reach its peak loudness. The exhaustion of the headroom by the noise compensation level adjuster 169 may be indicated by a multi-band limiter 154 that limits and provides a compression feedback signal 162 to the noise compensation level adjuster 169. In some cases, the noise compensation level adjuster 169 may exhaust its headroom by compensating in response to high levels of noise. Additionally, in some embodiments where the noise compensation level adjuster 169 controls other blocks of the level adjuster 152, the noise compensation level adjuster 169 may send a signal 161 to the multi-band limiter 154 indicating that the multi-band limiter 154 should stop operating in tone-preserving mode and/or should allow one or more amplifiers to be overdriven.

According to this example, the bass enhancement module 178 includes a bass extraction module 177, mixers 172 and 179, and a virtual bass (VB) block 171. In this example, the VB block 171 provides an output 168 to a noise compensation level adjuster 169. In some examples, each bass extraction module 177 may be implemented as a set of dynamic cross-filters that can be controlled at runtime. In some such examples, bass (e.g., the corresponding low-frequency band) can be extracted when the multi-band limiter 154 is limited to the low-frequency band.

According to some such implementations, the bass extraction module 177 can be configured to extract (from input channels CH1 and CH2) high-frequency content in a high-frequency range above the cutoff frequency (high-pass filtered signal 175) and low-frequency content in a low-frequency range below the cutoff frequency (bass-extracted audio 173). In some examples, the bass extraction module 177 can be configured to control the cutoff frequency at least in part based on a compression feedback signal 162. The cutoff frequency can be controlled by a limiting amount (as indicated by the compression feedback signal 162) performed in the multi-band limiter 154. In some examples, the limiting can be limited only to the low-frequency range (e.g., up to 500 Hz), but in alternative examples, the limiting can be across a wider frequency range or the entire frequency range. In some examples, the compression feedback signal 162 can indicate the amount of compression applied by the multi-band limiter 154 in each of at least two low-frequency bands within the low-frequency range (e.g., up to 500 Hz) (and the cutoff frequency can be determined by said compression amount). Alternatively, the compression feedback signal 162 may indicate the amount of compression applied by the multi-band limiter 154 in a wider or full frequency range (and the cutoff frequency may be determined by the amount of compression).

In this example, the bass-extracted audio 173 from both input channels CH1 and CH2 has been low-pass filtered and provided to mixer 172, which downmixes the bass-extracted audio 173 into a single channel. According to this example, mixer 172 provides the downmixed bass 174, to be propagated to both channels (in other words, mixed back to both channels), to mixer 179. In this example, mixer 179 mixes the downmixed bass 174 with the high-pass filtered signal 175 without bass extraction and outputs a modified channel 170. According to some examples, based on multi-band limiter behavior and noise estimation 160 (and/or based on the ratio of the gain corresponding to user input 163 to a gain based on noise compensation control), the modified channel 170 has enabled bass propagation through both channels.

In this example, noise estimation 160 may optionally be provided to bass extraction module 177 and VB block 171. In this embodiment, bass enhancement module 178 also considers noise estimation 160 (and/or the ratio of the system volume controlled by noise compensation level adjuster 169 to a volume corresponding to user control). According to some examples, if noise estimation 160 is high, the frequencies to be extracted will typically constitute a larger portion of the spectrum compared to when noise estimation is low. According to some such examples, bass enhancement module 178 may be configured to adjust the so-called “aggressiveness” of frequency extraction herein based on noise estimation 160. As used herein, the term “aggressiveness” refers to a parameter indicating the degree of bass volume enhancement.

In some such examples, the bass extraction module 177 can be configured to determine the cutoff frequency ("targeted_crossover" in the following formula):

targeted_crossover=total_gain_ratio*max_freq_limiting*aggressiveness (Equation 1)

In Equation 1, "aggressiveness" represents a parameter indicating the aggressiveness of bass volume enhancement. In some examples, the "aggressiveness" parameter can be adjusted by ear, for example, by the user or by the provider of the noise compensation system, to ensure that the system does not include too much or too little energy in the downmixed bass 174. According to some examples, linear interpolation of the "aggressiveness" parameter can be used to gradient between two "aggressiveness" settings (e.g., a high volume setting due to noise and another high volume setting due to user input).

In Equation 1, "max_freq_limiting" represents the maximum frequency covered by the frequency band limited in multi-band limiter 154. In some examples, "max_freq_limiting" may be determined by or derived directly from the highest frequency in the highest frequency band limited by multi-band limiter 154. In some implementations, "max_freq_limiting" may be limited to a range supported by bass extraction module 177.

In some examples,

"total_gain_ratio"=total_gain/max_possible_gain (Equation 2)

In Equation 2, "max_possible_gain" represents the sum of the maximum gains of each frequency band limited by the multi-band limiter 154 for all frequency bands that can be bass extracted by the bass extraction module 177 (or, in some embodiments, all frequency bands that can be limited in the multi-band limiter 154). In some examples, "max_possible_gain" can be the sum of the maximum gains that can be applied to all frequency bands that can be bass extracted by the multi-band limiter 154; in this sense, "max_possible_gain" can be the maximum integral of all gains that can be applied to all bins whose frequencies do not exceed the maximum cutoff frequency by the multi-band limiter 154.

In Equation 2, “total_gain” represents the sum of all gains applied to all frequency bands that can be extracted by bass (or, in some embodiments, all frequency bands that can be restricted) (e.g., as indicated by the compression feedback signal 162 for each frequency band).

In Equations 1 and 2, "total_gain_ratio" represents an indicator of the total amount of limitation imposed by the multi-band limiter 154 across all frequency bands from which bass extraction can be performed by the bass extraction module 177. In Equation 2, "total_gain_ratio" (via the "max_possible_gain" parameter) is normalized so that "total_gain_ratio" better indicates the total amount of limitation imposed over a variable number of frequency bands.

In some implementations, when the multi-band limiter 154 applies more restrictions (e.g., when the “total_gain_ratio” in Equations 1 and 2 increases), the cutoff frequency of each of filters 205 and 206 (“targeted_crossover” in Equation 1) can be increased to increase the applied bass boost. In some implementations, when the multi-band limiter 154 applies less restrictions (e.g., when the “total_gain_ratio” in Equations 1 and 2 decreases), the cutoff frequency can be decreased to reduce the applied bass boost. In some examples, the cutoff frequency can be smoothed by enabling and disabling (e.g., “targeted_crossover” in Equations 1 and 2) to ensure that the user does not notice sudden jumps in the translation.

According to this example, VB module 171 creates bass perception based on the absence of a fundamental phenomenon. According to some examples, VB module 171 can be configured to create enhanced bass perception by injecting a signal at the harmonics of the bass frequencies within the harmonics of the input signal. In some such examples, the number of injected harmonics and the amplitude of the harmonics can be determined by both the corresponding compression feedback signal 162 and the noise estimate 160 (or the ratio of the volume controlled by the noise compensation level adjuster 169 to the volume corresponding to the user input). In some examples, if the noise estimate 160 is high, the virtual bass volume (e.g., the number and amplitude of the harmonics) will be increased (e.g., by adjusting the aggressiveness in Equations 1 and 2) compared to a low noise estimate. In some implementations, the virtual bass volume can be determined as follows:

virtual_bass_gains=min_virtual_bass_gain+((1+0.01×A) ^{-limiter_gain -} 1)(Equation 3)

In Equation 3, "limiter_gain" represents the multiband limiter gain value for the lowest frequency band that multiband limiter 154 can provide to one or both of the VB module 171. In Equation 3, "A" represents a parameter indicating the aggressiveness of the virtual bass application (e.g., how much virtual bass is applied per multiband limiter gain). In one example, A = -25, but in alternative examples, A can be higher or lower. In Equation 3, "min_virtual_bass_gain" represents the minimum amount of virtual bass gain that can be applied. According to some examples, linear interpolation of the "aggressiveness" parameter can be used to gradient between two "aggressiveness" settings (e.g., a high volume setting due to noise and another high volume setting due to user input).

Returning to the example shown in Figure 1M, in this embodiment, the level adjuster 152 is configured to send an optional control signal 164 to the noise estimator 159. In some embodiments, the multi-band limiter 154 may be configured to send an optional control signal 167 to the noise estimator 159. As described in more detail elsewhere herein, in some instances, the level adjuster 152 may be configured to allow some distortion (e.g., to increase playback volume in the presence of ambient noise), causing one or more audio reproduction transducers in the audio environment to operate outside the linear range. In such an instance, the audio path 165 from the loudspeaker 156 to the microphone 157 may include at least some sound corresponding to the nonlinear distortion of the loudspeaker 156. At least in part because echo cancellers are typically based on linear algorithms, the echo canceller will not properly cancel the sound corresponding to the nonlinear distortion of the loudspeaker 156.

Therefore, according to some such examples, if the level adjuster 152 is configured with the multi-band limiter 154 to operate one or more audio reproduction transducers of the audio environment outside the linear range, the level adjuster 152 may also send a control signal 164 (or the multi-band limiter 154 may also send a control signal 167) to the noise estimator 159, which indicates that the noise estimator 159 should change its operating mode. In some such examples, the control signal 164 (or control signal 167) may indicate that the echo canceller implemented by the noise estimator 159 should only use the silence playback interval of the multi-band limited audio data 155 as input. In some such examples, the silence playback interval may be an instance of an audio signal at or below a threshold level in one or more frequency bands. Alternatively or additionally, in some examples, the silence playback interval may be an instance of an audio signal at or below a threshold level during a certain time interval. The silence playback interval may also be referred to herein as a “gap.”

In some implementations, if the level adjuster 152 is configured to cause one or more audio reproduction transducers in the audio environment to operate outside the linear range, the control signal 164 (or control signal 167) can indicate that one or more functions of the echo canceller should be disabled or suspended. For example, an echo canceller typically operates by updating adaptive filter coefficients for each of multiple frequency bands. In some such implementations, if multi-band limited audio data 155 would cause the loudspeaker 156 (or one or more other audio reproduction transducers in the audio environment) to operate outside the linear range, the control signal 164 (or control signal 167) can control the acoustic echo canceller not to update the filter coefficients.

Figure 2A illustrates an example of a graphical user interface (GUI) for setting thresholds and other parameters of a multi-band limiter. The GUI 200 can be presented on a display device, for example, according to software executed by a control system. Users can interact with the GUI 200 by providing user input, for example, via a touchscreen on the display presenting the GUI 200, via a mouse, via a keyboard, via voice commands, etc. As with other figures presented herein, the types and quantities of elements indicated, as well as specific values, are provided only as examples.

In this implementation, the y-axis indicates decibels ranging from 0 dB to -60 dB, and the x-axis indicates frequency in Hz. In this example, GUI 200 shows a set of example thresholds 201 for each of a plurality of frequency bands 217. According to this example, each threshold 201 is illustrated by a dot in a vertical line representing the corresponding frequency band 217. The center frequency of each frequency band 217 is indicated adjacent to the vertical line representing the frequency band 217. In some implementations, the threshold 201 is a level at which a signal is not allowed to exceed the corresponding frequency band. If the input level exceeds the threshold 201, a negative gain can be applied to limit the level to the threshold 201.

The level of threshold 201 can be related to the maximum input value that still has a linear output response in the corresponding frequency band when reproduced by the audio reproduction transducer. For example, a particular threshold 201 shown in Figure 2A can be preset according to the capabilities of a particular audio reproduction transducer. In this example, threshold 201 is typically lower in the lowest frequency band. This indicates that the particular audio reproduction transducer exhibits less distortion at low frequencies than at high frequencies. In this example, frequencies above 844Hz are not limited in terms of the maximum volume of this particular device.

According to this example, element 213 indicates the isolation setting for a specific frequency band. If a frequency band is set to be isolated, only the audio in that band affects the applied limiting gain. In the example shown in Figure 2A, no frequency band is isolated.

Figure 2B shows another example of the thresholds and other parameters of a multi-band limiter. Figure 2B illustrates an example of GUI 200, where four elements 213 indicate that four corresponding low-frequency bands are isolated. In some cases, it may be desirable to allow specific frequency bands, such as low-frequency bands, to operate in complete isolation without affecting tone preservation. For example, due to the small size of the loudspeakers, the fixed threshold for bass frequencies in some audio systems may be extremely low. Allowing these low-frequency bands to affect tone preservation calculations would result in a sharp decrease in the overall playback quality. In such cases, it may be desirable to allow these low-frequency bands to operate independently (as depicted in Figure 2B) and apply tone preservation methods to the remaining frequency bands.

In some alternative implementations, the frequency bands may be partially isolated (e.g., 25% isolation, 50% isolation, 75% isolation, etc.) rather than completely isolated or not isolated at all. In some such examples, the degree of frequency band isolation may be selected via an alternative version of GUI 200, which includes sliders or other virtual user input devices corresponding to one or more (e.g., each) of elements 213. In other examples, the degree of frequency band isolation may change automatically due to changing conditions, such as changes in the ambient noise level in the audio environment. Some examples are described below with reference to FIG4C.

In the examples shown in Figures 2A and 2B, dots 218 in some frequency bands indicate example adjustments for what is referred to herein as “overdrive” or operation outside the linear range of the audio reproduction transducer. Various examples are disclosed herein. Between dots 201 and 218, the audio reproduction transducer will operate outside the linear range. According to some examples, dot 218 represents a hard limit that the audio reproduction transducer will not be driven beyond. In some alternative implementations, dot 218 represents a soft limit, in some cases the audio reproduction transducer can be driven beyond the soft limit. In some implementations, such explicit overdrive adjustment may be optional, and in some alternative examples, such explicit overdrive adjustment may not be present. In some implementations, explicit overdrive adjustment may be a fixed value across all frequency bands (e.g., 3 dB added to threshold 201).

In the examples shown in Figures 2A and 2B, element 212 allows a user (e.g., an employee of a device manufacturer) to select a tone preservation setting. In these examples, the user interacts with element 212 by moving a slider portion 212a and/or by entering a value in window portion 212b. In some implementations, the tone preservation setting corresponds to the amount by which a signal in one frequency band can affect the gain applied to other frequency bands (e.g., one or more adjacent frequency bands, if these frequency bands are not isolated).

In some implementations, the tone preservation setting of 1.00 may correspond to a time-varying threshold _Tb [n], which is calculated as a function of all frequency band signals _xb [n] and all fixed thresholds _Lb on all non-isolated frequency bands, where b = 1…B:

T _b [n]=TPF({x _i [n],L _i |i=1...B})

For example, a fixed threshold _Lb can correspond to a threshold of 201. Thus, the gain _gb [n] for each frequency band can be calculated as _gb [n] = CF(xb[n], _Tb [n]).

For tone preservation settings less than 1.00, each threshold _Tb [n] can be calculated as a function of more than all frequency band signals _xb [n] and/or more than all fixed thresholds _Lb for non-isolated frequency bands. For example, in a tone preservation setting of 0.50, each threshold _Tb [n] can be calculated as a function of half of the frequency band signals _xb [n] and the fixed thresholds _Lb for non-isolated frequency bands.

In some examples, the time-varying threshold of a frequency band can be calculated based on its nearest neighboring non-isolated frequency band or a range of adjacent non-isolated frequency bands.

In some examples, if a non-isolated band receives a significant gain reduction due to being above its fixed threshold, the time-varying thresholds of other non-isolated bands will also decrease, resulting in some gain reduction. Because the time-varying thresholds of the bands decrease to below their respective fixed thresholds, the multi-band limiter 154 still reduces distortion while mitigating or otherwise preventing changes to the timbre.

In some examples, the control system (e.g., multi-band limiter 154) can be configured to calculate the average difference of the audio input signal in each band and its corresponding fixed threshold _Lb. The time-varying threshold in each band can then be the audio input signal level in that band minus the average difference.

Alternatively or additionally, the time-varying threshold can be smoothed over time, at least smoother than the gain _gb [n]. That is, the level of the audio input signal used to calculate the threshold can be smoother than the signal used to calculate the gain _gb [n] (e.g., _eb [n]). In one such example, a single-pole smoother with a longer time constant can be used to calculate a smoother energy signal _sb [n]:

In this case, the start-up and release times can be 10 times that of a conventional multi-band limiter. The smoothed energy signal can then be expressed in dB as follows:

S _{_b} [n] = 10log_₁₀(s _{_b}[n])

The difference (also expressed in dB) between the smoothed energy signal in each frequency band and the fixed threshold _Lb in each frequency band can be calculated as follows:

_Db [n]＝ _Sb [n] _－Lb

And the minimum value of these distances can be found across all frequency bands:

The weighted average of these differences across all frequency bands can then be calculated as follows, where β represents the weighting factor:

When β = 1, the true average of the differences is calculated, and when β > 1, larger differences have a greater impact on the average. In other words, the frequency bands with energies much higher than the threshold _Lb have a greater impact. In some examples, it has been found that β = 8 produces appropriate weighting. Finally, the threshold _Tb [n] can be calculated as the smoothed signal energy in the frequency band minus the average difference—if this threshold is less than the fixed threshold _Lb. Otherwise, according to some implementations, the time-varying threshold can remain equal to the fixed threshold, for example, as follows:

In some alternative implementations, the threshold can be calculated based on the maximum value of the distance _Db [n] instead of the weighted average:

Then, each threshold can be calculated by subtracting the maximum distance from the smoothed signal energy in the frequency band and adding a certain tolerance value D _tol —if this threshold is less than a fixed threshold:

The tolerance value D _tol can, for example, be designed to allow slight variations in the amount of compression applied to each frequency band. In one specific embodiment, it has been found that an actual value of D _tol = 12 dB allows for sufficient variation.

In the example shown in Figure 2B, threshold 201a is the lowest threshold for the unisolated frequency band. In some such examples, if 100% tone preservation is selected via element 212, no other unisolated frequency band can exceed this threshold.

Figure 2C is a graph illustrating an example of threshold values for a range of frequencies. In graph 200C of Figure 2C, the vertical axis represents amplitude, and the horizontal axis represents frequency. The threshold value for each frequency shown in graph 200C is represented by line segments 203C and 205C. Line segment 203C represents the threshold value for the isolation bands in the bass range. Line segment 203C is intended to roughly correspond to the threshold values for the four isolation bands in the bass range shown in Figure 2B.

Line segment 205C represents the excitation of the higher frequency bands (those above the four isolated bands in the bass range) of Figure 2B, to which a tone-preserving method has been applied. Here, the dashed line 204C indicates the boundary between the independent bands and the tone-preserving bands. In some examples, there may be more than one boundary between the independent or isolated bands and the tone-preserving bands.

In this example, it can be seen that although the thresholds of the higher frequency bands in Figure 2B are different, the output of the timbre preservation method has limited the excitation of all frequencies represented by line segment 205C to the lowest threshold of the higher frequency bands in Figure 2B, namely threshold 201a in Figure 2B.

In the example of curve 200C, the control system (e.g., the control system implementing the level adjuster 152 of Figure 1M) has determined the multi-band limiter configuration in response to a user input level adjustment instruction received via user input. For example, the user may have used a remote control device to adjust the volume. According to this embodiment, determining the multi-band limiter configuration involves determining the timbre preservation configuration because the received level adjustment instruction of at least one type is a user input level adjustment instruction. In this example, as shown in Figures 1G and 1J, the input audio corresponds to white noise at a level that would cause a limit in the multi-band limiter. In the examples shown in Figures 2C and 2D, the level adjuster has adjusted the input audio to that level based on one or more level adjustment instructions that may have been received via user input or via input from a noise estimator. It can be seen that for the frequency band corresponding to line segment 205C, the timbre is fully preserved in the output excitation.

Figure 2D is a graph illustrating another example of thresholds for a range of frequencies. In graph 200D of Figure 2D, the vertical axis represents amplitude, and the horizontal axis represents frequency. The threshold for each frequency shown in graph 200D is represented by line segments 203D and 205D. Line segment 203D represents the threshold for the isolation bands in the bass range. Line segment 203D is intended to roughly correspond to the thresholds for the four isolation bands in the bass range shown in Figure 2B. In this example, line segment 203D is intended to be the same as line segment 203C in Figure 2C.

In this example, line segment 205D represents the excitation of the higher frequency bands in Figure 2B (those bands above the four isolated bands in the bass range), to which tone preservation methods are typically applied. Here, dashed line 204D indicates the boundary between the isolated frequency bands and the bands to which tone preservation methods are typically applied. In some examples, there may be more than one boundary between the isolated frequency bands and the bands to which tone preservation methods are typically applied.

In the example of graph 200D, the control system (e.g., the control system implementing the level adjuster 152 of Figure 1M) has determined the multi-band limiter configuration in response to a noise compensation level adjustment instruction received from the noise compensation module (e.g., from the noise estimator 159 of Figure 1M). According to this example, determining the multi-band limiter configuration involves changing the tone preservation function because the received level adjustment instruction of at least one type is a noise compensation level adjustment instruction. In this embodiment, changing the tone preservation function involves at least partially disabling the tone preservation function.

According to this example, the noise compensation level adjustment indicator corresponds to the ambient noise level in the audio environment, and changing the tone preservation function involves altering the tone preservation function at least in part based on the ambient noise level. In this example, the noise compensation level adjustment indicator indicates and/or responds to a high ambient noise level. As can be seen in graph 200D, in this example, the frequency band of Figure 2B (corresponding to the frequency band of line segment 205D), which was previously adjusted to be non-independent and tone-preserved, becomes completely independent. In this example, the input audio corresponds to white noise at a level that would cause limitations in the multi-band limiter.

In some instances, one or more previously received user input level adjustment indications may have previously pushed the input level of one or more frequency bands into the restricted area of a multi-band limiter configuration. According to some such examples, the resulting multi-band limiter configuration may be a linear combination of 205D and 205C (e.g., a crossfade between the two). In some other implementations, the multi-band limiter configuration responding to a noise compensation level adjustment indication may override the tone preservation response to the user input level adjustment indication. Some examples are described below.

Figure 2E is a block diagram illustrating examples of components of an apparatus capable of implementing various aspects of this disclosure. As with other figures provided herein, the types and quantities of elements shown in Figure 2E are provided by way of example only. Other embodiments may include more, fewer, and/or different types and/or quantities of elements. According to some examples, apparatus 240 may be configured to perform at least some of the methods disclosed herein. In some embodiments, apparatus 240 may be or may include one or more components of a television, an audio system, a mobile device (such as a cellular phone), a laptop computer, a tablet device, a smart speaker, or another type of device. In some embodiments, apparatus 240 may be or may include a television control module. Depending on the specific embodiment, the television control module may be integrated into the television or may not be integrated into the television. In some embodiments, the television control module may be a separate device from the television, and in some instances may be sold separately from the television or as an add-on or optional device that may be included with the purchased television. In some embodiments, the television control module may be available from a content provider (such as a provider of television programs, movies, etc.).

According to some alternative implementations, device 240 may be or may include a server. In some such examples, device 240 may be or may include an encoder. Thus, in some instances, device 240 may be a device configured for use in an audio environment such as a home audio environment; however, in other instances, device 240 may be or may include a device configured for use in the “cloud,” such as a server.

In this example, device 240 includes an interface system 207 and a control system 210. In some embodiments, the interface system 207 may be configured to communicate with one or more other devices in the audio environment. In some examples, the audio environment may be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some embodiments, the interface system 207 may be configured to exchange control information and associated data with audio devices in the audio environment. In some examples, the control information and associated data may be related to one or more software applications being executed by device 240.

In some implementations, interface system 207 can be configured to receive or provide content streams. Content streams may include audio data. Audio data may include, but is not limited to, audio signals. In some instances, audio data may include spatial data such as channel data and/or spatial metadata.

Interface system 207 may include one or more network interfaces and/or one or more external device interfaces (such as one or more Universal Serial Bus (USB) interfaces). According to some embodiments, interface system 207 may include one or more wireless interfaces. Interface system 207 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and/or a gesture sensor system. In some examples, interface system 207 may include one or more interfaces between control system 210 and a memory system (such as the optional memory system 215 shown in FIG. 2E). However, in some instances, control system 210 may include a memory system. In some embodiments, interface system 207 may be configured to receive input from one or more microphones in the environment.

For example, control system 210 may include a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and/or discrete hardware components.

In some implementations, control system 210 may reside in more than one device. For example, in some implementations, a portion of control system 210 may reside in a device within the audio environment, and another portion of control system 210 may reside in a device outside the audio environment, such as a server, mobile device (e.g., a smartphone or tablet). In other examples, a portion of control system 210 may reside in a device within one of the environments described herein, and another portion of control system 210 may reside in one or more other devices within the audio environment. For example, control system functionality may be distributed across multiple smart audio devices in the audio environment, or may be shared by an orchestration device (such as a device that may be referred to herein as a smart home hub) and one or more other devices in the audio environment. In other examples, a portion of control system 210 may reside in a device implementing a cloud-based service (e.g., a server), and another portion of control system 210 may reside in another device implementing a cloud-based service (e.g., another server, storage device, etc.). In some examples, interface system 207 may also reside in more than one device.

In some implementations, the control system 210 may be configured to perform the methods disclosed herein at least in part. According to some examples, the control system 210 may be configured to implement methods for content streaming processing.

Some or all of the methods described herein can be executed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include, for example, the memory devices described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may reside, for example, in the optional memory system 215 and/or control system 210 shown in FIG. 2E. Therefore, various innovative aspects of the subject matter described herein can be implemented in one or more non-transitory media on which software is stored. For example, the software may include instructions for controlling at least one device to process content streams, encode content streams, decode content streams, etc. For example, the software may be executed by one or more components of a control system, such as the control system 210 of FIG. 2E.

In some examples, device 240 may include the optional microphone system 220 shown in FIG. 2E. The optional microphone system 220 may include one or more microphones. In some embodiments, the one or more microphones may be part of or associated with another device, such as a speaker in a speaker system, a smart audio device, etc. In some examples, device 240 may not include microphone system 220. However, in some such embodiments, device 240 may still be configured to receive microphone data from one or more microphones in an audio environment via interface system 210. In some such embodiments, cloud-based implementations of device 240 may be configured to receive microphone data or at least partially corresponding noise metrics from one or more microphones in an audio environment, or from one or more devices in an audio environment including at least one microphone, via interface system 210.

According to some embodiments, device 240 may include the optional loudspeaker system 225 shown in FIG. 2E. The optional loudspeaker system 225 may include one or more loudspeakers, which may also be referred to herein as “loudspeakers” or more generally as “audio reproduction transducers.” In some examples (e.g., cloud-based embodiments), device 240 may not include loudspeaker system 225. However, in some such examples, one or more other devices in the audio environment may implement loudspeaker system 225.

In some embodiments, device 240 may include the optional sensor system 230 shown in FIG. 2E. The optional sensor system 230 may include one or more touch sensors, gesture sensors, motion detectors, etc. According to some embodiments, the optional sensor system 230 may include one or more cameras. In some embodiments, the camera may be a stand-alone camera. In some examples, one or more cameras of the optional sensor system 230 may reside in a smart audio device, which may be a single-purpose audio device or a virtual assistant. In some such examples, one or more cameras of the optional sensor system 230 may reside in a television, mobile phone, or smart speaker. In some examples, device 240 may not include sensor system 230. However, in some such embodiments, device 240 may still be configured to receive sensor data from one or more sensors in the audio environment via interface system 210.

In some embodiments, device 240 may include the optional display system 235 shown in FIG. 2E. The optional display system 235 may include one or more displays, such as one or more light-emitting diode (LED) displays. In some instances, the optional display system 235 may include one or more organic light-emitting diode (OLED) displays. In some examples, the optional display system 235 may include one or more displays of a television. In other examples, the optional display system 235 may include a laptop display, a mobile device display, or another type of display. In some examples where device 240 includes display system 235, sensor system 230 may include a touch sensor system and/or gesture sensor system for proximity to one or more displays of display system 235. According to some such embodiments, control system 210 may be configured to control display system 235 to present one or more graphical user interfaces (GUIs).

According to some such examples, device 240 may be or may include a smart audio device. In some such embodiments, device 240 may be or may include a wake word detector. For example, device 240 may be or may include a virtual assistant.

Figure 3 is a flowchart outlining an example of the disclosed method. As with other methods described herein, the boxes of method 300 need not be performed in the indicated order. Furthermore, such a method may include more or fewer boxes than those shown and/or described.

Method 300 can be performed by a device or system such as the apparatus 240 shown in FIG. 2E and described above. In some examples, blocks of method 300 can be performed by one or more devices within an audio environment, such as an audio system controller or another component of an audio system, such as a smart speaker, television, television control module, mobile device, etc. In some embodiments, the audio environment can include one or more rooms in a home environment. In other examples, the audio environment can be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. However, in alternative embodiments, at least some blocks of method 300 can be performed by a device (such as a server) implementing a cloud-based service.

In this example, block 301 relates to receiving a content stream including input audio data by a control system and via an interface system (e.g., by control system 210 of FIG. 2E and via interface system 207 of the same figure). In some examples, the content stream may include video data corresponding to the audio data. In some embodiments, the control system and interface system may be control system 210 and interface system 207 shown in FIG. 2E and described above. In some embodiments, the control system may be control system 110 shown in FIG. 1M and described above. In some examples, block 301 may relate to the level adjuster 152 of FIG. 1M receiving input audio data 151. According to some embodiments, block 301 may relate to receiving an encoded content stream. In such embodiments, block 301 may relate to decoding the encoded content stream. For example, the content stream may correspond to movies, television programs, music, music videos, podcasts, etc.

In this embodiment, block 305 relates to receiving at least one type of level adjustment instruction in connection with the playback of audio data by a control system and via an interface system. In some instances, the at least one type of level adjustment instruction may include a user input level adjustment instruction received via user input (e.g., via a remote control, via a voice command, etc.). According to some embodiments, block 305 may relate to the level adjuster 152 of FIG1M receiving user input 163.

Alternatively or additionally, at least one type of level adjustment indication may include a noise compensation level adjustment indication received from the noise compensation module. According to some embodiments, block 305 may relate to the level adjuster 152 of FIG1M receiving the noise estimator output 160. The noise compensation level adjustment indication may, for example, be responsive to an ambient noise level detected by the noise compensation module, or detected by one or more microphones from which the noise compensation module is configured to receive microphone signals. According to some embodiments, block 305 may relate to the level adjuster 152 of FIG1M receiving the noise estimator output 160.

In this example, block 310 relates to a control system controlling the level of input audio data based on at least one type of level adjustment instruction to produce level-adjusted audio data. According to some embodiments, in block 310, the level adjuster 152 of FIG1M can generate level-adjusted audio data 153 based on user input 163 and/or noise estimator output 160.

According to this example, block 315 relates to determining a multiband limiter configuration by a control system and at least in part based on at least one type of level adjustment indication. According to some examples, the control system 110 shown in FIG1M can determine the multiband limiter configuration at least in part based on user input 163 and/or noise estimator output 160. In some such examples, the level adjuster 152 of FIG1M can determine the multiband limiter configuration at least in part based on user input 163 and/or noise estimator output 160.

According to some implementations, determining the multi-band limiter configuration may involve determining the tone preservation configuration if the level adjustment indication (e.g., if the only level adjustment indication) is a user-input level adjustment indication. In some instances, the tone preservation configuration may be band-dependent. For example, some bands may be partially or completely isolated. According to some examples, the level of a completely isolated band can be controlled independently without referencing the levels and/or thresholds of other bands.

In some examples, if at least one type of level adjustment indication is a noise compensation level adjustment indication, determining the multi-band limiter configuration may involve altering the tone preservation function. In some such examples, altering the tone preservation function may involve at least partially disabling the tone preservation function. In some implementations, the noise compensation level adjustment indication may correspond to the ambient noise level in the audio environment. In some such examples, the tone preservation function may be altered at least in part based on the ambient noise level.

According to some implementations, both a user input level adjustment indication and a noise compensation level adjustment indication can be received. In some such implementations, determining the multi-band limiter configuration may involve determining a partial tone preservation configuration, which is at least partially based on the average (e.g., a weighted average) of the multi-band limiter configuration corresponding to the user input level adjustment indication and the multi-band limiter configuration corresponding to the noise compensation level adjustment indication.

In this example, block 320 relates to configuring the multi-band limiter by the control system according to the multi-band limiter configuration. In some such examples, the level adjuster 152 of FIG1M can send a control signal 161 indicating the multi-band limiter configuration to be applied to the multi-band limiter 154. In some embodiments, if the received level adjustment indication is a user-input level adjustment indication, the tone preservation setting of the multi-band limiter 154 can be maintained, for example, as initially adjusted at the factory. Such an implementation can help ensure a pleasant experience for the user when adjusting the volume control.

According to some examples, if the received level adjustment instruction is a noise compensation level adjustment instruction, the tone preservation setting of the multi-band limiter 154 can be gradually turned off, for example, proportionally to the noise compensation level adjustment instruction. In some such examples, when noise is present, the played-out audio content can still be understood under noise, while the loss of fidelity is masked by the noise source.

According to this embodiment, block 325 relates to applying a multi-band limiter to leveled audio data to produce multi-band limited audio data. In some such examples, in block 325, the multi-band limiter 154 of FIG. 1M can produce multi-band limited audio data 155. In some embodiments, method 300 can relate to reproducing the multi-band limited audio data on one or more audio reproduction transducers in an audio environment to provide reproduced audio data. For example, method 300 can relate to reproducing the multi-band limited audio data 155 via loudspeaker 156 of FIG. 1M.

Figures 4A and 4B illustrate examples of tone preservation modifier modules. For example, the tone preservation modifier module can be implemented via the control system 110 of Figure 1M or the control system 210 of Figure 2E. In some examples, tone preservation modifier modules 403A and 403B can be implemented via the level adjuster 152 of Figure 1M. In some alternative embodiments, tone preservation modifier modules 403A and 403B can be implemented via the multi-band limiter 154 of Figure 1M or via another element of the control system 110 not shown in Figure 1M.

In the example shown in Figure 4A, the tone preservation modifier module 403A is configured to control the amount of tone preservation used at any given time. In this example, the tone preservation modifier module 403A is configured to control the amount of tone preservation based at least in part on noise estimation 402A. In this example, element 401A represents the original tone preservation setting of the multi-band limiter. For example, element 401A could represent the tone preservation setting indicated by element 212 in the examples shown in Figures 2A and 2B, such as the value in window portion 212b of Figure 2B.

Noise estimate 402A is an example of what is referred to herein as a noise compensation level adjustment indication. In some examples, noise estimate 402A may be the average, median, or maximum noise level over a range of audible frequencies or a subset of audible frequency bands. In some examples, noise estimate 402A may be a spectral noise estimate of the ambient noise provided to level adjuster 152, determined by noise estimator 159. In some instances, level adjuster 152 (or another component of the control system) may be configured to determine noise estimate 402A based on noise estimator output 160.

According to this example, when the noise estimate 402A indicates a high noise level, the tone preservation modifier module 403A is configured to control the tone preservation amount in the modified tone preservation amount 404A to be low. Conversely, when the noise estimate 402A indicates a low noise level, the tone preservation modifier module 403A is configured to control the tone preservation amount in the modified tone preservation amount 404A to be high (or not modified).

In the example shown in Figure 4B, the tone preservation modifier module 403B is configured to control the amount of tone preservation based at least in part on the noise estimation 402A and the user volume setting 463. The user volume setting 463 is an example of what is referred to herein as a user input level adjustment indication.

According to this example, the tone preservation modifier module 403B is configured to determine the modified tone preservation amount 404B based on the following expression:

Timbre retention = A * gain _user + B * gain _noisecomp (Equation 4)

In Equation 4, A represents the original tone retention amount, which corresponds to element 401A in this example. For example, A could represent the tone retention setting indicated by element 212 in the examples shown in Figures 2A and 2B, such as the value in window portion 212b of Figure 2B. Here, gain _user represents the gain applied by the user, i.e., the user volume setting 463 in the example shown in Figure 4B. In Equation 4, gain _noisecomp represents the noise compensation level adjustment indication, i.e., the noise estimate 402A in the example shown in Figure 4B. In the case of a broadband noise compensation system, gain _noisecomp can be a broadband gain. Alternatively, gain _noisecomp can be a weighted average gain applied by a multi-band noise compensation system. In Equation 4, B represents the tone retention value. The tone retention value B can, for example, indicate the minimum amount of tone retention allowed. The tone retention value B can indicate, for example, how much tone retention the system will have when the volume has been primarily adjusted by the noise compensator. The tone retention value B can, for example, be set by the manufacturer during device adjustment operations.

According to some implementations, the "gain" term in Equation 4 is not intended to represent the unmodified gain, but rather a proportion of the gain applied by the control system. For example, if 75% of the gain applied by the control system is based on the user volume setting 463, then gain _user will be 0.75. Accordingly, 25% of the gain will be based on the noise estimate 402A, so gain _noisecomp will be 0.25.

According to this example, if the noise estimate 402A is low, the amount of tone retention will be close to the original adjusted tone retention value (represented by the A gain term). In this example, if the user volume setting 463 is high and the noise estimate 402A is also high, the tone will be partially retained proportionally to the relative values of the user volume setting 463 and the noise estimate 402A. According to this example, if the noise estimate 402A is high and the user volume setting 463 is low, the B gain term in Equation 4 will dominate and the tone will not be retained.

Figure 4C illustrates an example of a band isolation modifier. Band isolation modifier 405 can be implemented, for example, via control system 210 of Figure 2E. In some embodiments, band isolation modifier 405 can be implemented via control system 110 of Figure 1M (e.g., via level adjuster 152 and/or multi-band limiter 154). In some embodiments, band isolation modifier 405 can be implemented as an alternative to embodiments including tone preservation modifier module 403A or tone preservation modifier module 403B. However, some embodiments may include band isolation modifier 405 and tone preservation modifier module 403A or tone preservation modifier module 403B, for example, to allow band isolation modification of low-frequency bands (such as the four low-frequency bands shown as fully isolated in Figure 2B).

In this example, the band isolation modifier 405 is shown to receive isolation settings 413 for each of a plurality of bands. In some instances, the plurality of bands may include all bands. In other examples, the plurality of bands may include bands that, conversely, apply tone preservation, such as the non-isolated bands of Figure 2B.

According to this example, the band isolation modifier 405 is also shown to receive an optional user volume setting 463 and a noise estimate 402A. In some examples, the noise estimate 402A may be targeted at a specific frequency band. In alternative examples, the noise estimate 402A may be targeted at a subset of frequency bands, for example, a subset of frequency bands whose isolation values may be modified by the band isolation modifier 405. In some examples, the noise estimate 402A may be targeted at all frequency bands.

In this example, the band isolation modifier 405 is configured to determine whether to modify the isolation value of a band, and if so, to generate a modified band isolation value 407 for that band. The band isolation value modification can be binary or non-binary, depending on the specific implementation. In the case of binary modification, in some examples, if the ambient noise level is high, the band isolation value can be converted from non-isolated to isolated. In some such examples, if the ambient noise level is low or the band is already isolated, the band isolation value may remain unchanged. In some examples, the ambient noise level may be based on the full spectrum. In other examples, the ambient noise level may be specific to the frequency band where isolation may be modified.

In some alternative implementations, the frequency bands may be partially isolated (e.g., 25%, 50%, 75%, etc.) rather than completely isolated or not isolated at all. In some such examples, the degree of isolation of the frequency bands may correspond to the ambient noise level in the audio environment. For example, the degree of isolation of the frequency bands may correspond to a weighting value used in a timbre preservation method to give selected frequency bands a lighter weight than unisolated frequency bands.

In some such examples, the weighting of the frequency band's influence on the tone preservation algorithm can, for example, correspond to (1-I), where I represents the degree of frequency band isolation. In one such example, if the frequency band is 75% isolated, then I equals 0.75, and the weighting of the frequency band's influence on the tone preservation algorithm will be 0.25. In another such example, if the frequency band is 100% isolated, then I equals 1.0, and the weighting of the frequency band's influence on the tone preservation algorithm will be 0.0: in other words, the threshold corresponding to the frequency band is not used in the tone preservation calculation.

Returning to Figure 3, in some examples, method 300 may involve a control system causing a change in the operation of a noise compensation module, for example, in response to predetermined inputs, indicators, and/or conditions. For example, as described above with reference to Figure 1M, in some embodiments, the level adjuster 152 may be configured to send a control signal 164 to the noise estimator 159. In some embodiments, the multi-band limiter 154 may be configured to send a control signal 167 to the noise estimator 159. In some examples, control signals 164 and/or 167 may cause a change in the operation of the noise compensation module (e.g., cause a change in the functionality of the noise estimator 159). The noise compensation module may be a subsystem or module of the control system, for example, as shown in Figure 1M.

Some such examples may involve a change in the operation of a noise compensation module when multi-band limited audio data (e.g., multi-band limited audio data 155 output by multi-band limiter 154) causes one or more audio reproduction transducers (e.g., loudspeaker 156) in an audio environment to operate outside their linear range. In some such instances, the control system may cause one or more audio reproduction transducers in the audio environment to operate outside their linear range, at least in part, based on a noise compensation level adjustment indication and/or an ambient noise estimate. For example, the multi-band limited audio data causing one or more audio reproduction transducers in the audio environment to operate outside their linear range may be based on a noise compensation level adjustment corresponding to a high ambient noise level in the audio environment.

According to some examples, changes to the operation of a noise compensation module can involve altering the echo canceller function of the noise compensation module. For example, a change to the operation of a noise compensation module can involve causing the noise compensation module to use only a "mute" playback interval as input to the noise estimator of the noise compensation module. A "mute" playback interval can be an instance of an audio signal that is at or below a threshold level (e.g., a predetermined threshold level) in at least one of a frequency band or time interval. In some implementations, a "mute" playback interval can be an instance of an audio reproduction transducer of an audio environment operating within its linear range.

As noted elsewhere herein, in some embodiments, the level adjuster module of the control system (e.g., level adjuster 152 of FIG. 1M) may be configured to control the level of the input audio data to produce level-adjusted audio data (e.g., level-adjusted audio data 153). In some such embodiments, method 300 may also involve providing multi-band limiter feedback from a multi-band limiter to the level adjuster module, for example, via a control signal 162 illustrated in FIG. 1M. According to some such examples, the multi-band limiter feedback may indicate the amount of limitation applied by the multi-band limiter to each of the multiple frequency bands of the level-adjusted audio data.

In some such implementations, method 300 may also involve controlling the level of one or more frequency bands among a plurality of frequency bands by a level adjuster module based at least in part on multi-band limiter feedback. In some such examples, method 300 may involve reducing the level of one or more frequency bands based at least in part on multi-band limiter feedback indicating that the level of one or more frequency bands or other frequency bands among a plurality of frequency bands is limited. In some examples, the level adjuster may be configured to modify the level-adjusted audio data according to a compression feedback signal from the multi-band limiter, as described above with reference to FIG. 1N.

Figure 5 is a flowchart outlining an example of another disclosed method. As with other methods described herein, the boxes of method 500 need not be executed in the indicated order. Furthermore, such a method may include more or fewer boxes than those shown and/or described.

Method 500 can be performed by a device or system such as the apparatus 240 shown in FIG. 2E and described above. In some examples, blocks of method 500 can be performed by one or more devices within an audio environment, such as an audio system controller or another component of an audio system, such as a smart speaker, television, television control module, mobile device, etc. In some embodiments, the audio environment can include one or more rooms in a home environment. In other examples, the audio environment can be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. However, in alternative embodiments, at least some blocks of method 500 can be performed by a device (such as a server) implementing a cloud-based service.

In this example, block 505 relates to receiving a content stream including input audio data by a control system and via an interface system (e.g., by control system 210 of FIG. 2E and via interface system 207 of the same figure). In some examples, the content stream may include video data corresponding to the audio data. In some embodiments, the control system and interface system may be control system 210 and interface system 207 shown in FIG. 2E and described above. In some embodiments, the control system and interface system may be control system 110 shown in FIG. 1M and described above. In some examples, block 505 may relate to the level adjuster 152 of FIG. 1M receiving input audio data 151. According to some embodiments, block 505 may relate to receiving an encoded content stream. In such embodiments, block 505 may relate to decoding the encoded content stream. For example, the content stream may correspond to movies, television programs, music, music videos, podcasts, etc.

According to this example, block 510 relates to the application of a multi-band limiter to audio data or a processed version of audio data by a control system to produce multi-band limited audio data. In some such examples, in block 510, the multi-band limiter 154 of Figure 1M can produce multi-band limited audio data 155.

In this example, box 515 relates to determining whether multi-band limited audio data, when played back on one or more audio reproduction transducers in an audio environment, would cause one or more audio reproduction transducers to operate outside their linear range. In some such examples, the control system may cause one or more audio reproduction transducers in the audio environment to operate outside their linear range, at least in part, based on at least one of a noise compensation level adjustment indication or a noise estimate. For example, multi-band limited audio data that causes one or more audio reproduction transducers in the audio environment to operate outside their linear range may be based on a noise compensation level adjustment corresponding to a high ambient noise level in the audio environment.

In some implementations, block 515 may relate to a data structure that indicates the maximum level of operation within the linear range for each of a plurality of frequency bands. In some such examples, these maximum linear range levels may correspond to dot 201 of Figures 2A and 2B. For example, the control system may retrieve the maximum linear range levels from memory. In some examples, block 515 may relate to a data structure that indicates the maximum level of operation within the nonlinear range for each of a plurality of frequency bands. In some such examples, these maximum nonlinear range levels may correspond to dot 218 of Figures 2A and 2B.

According to this example, box 520 relates to the control system controlling whether an acoustic echo canceller updates one or more filter coefficients based on whether multi-band-limited audio data would cause one or more audio reproduction transducers of the audio environment to operate outside the linear range. According to some such examples, controlling whether the acoustic echo canceller updates one or more filter coefficients may involve controlling the acoustic echo canceller not to update one or more filter coefficients if the multi-band-limited audio data would cause one or more audio reproduction transducers of the audio environment to operate outside the linear range. In some examples, box 520 may relate to the control system controlling whether a noise estimator updates a noise estimate based on whether multi-band-limited audio data would cause one or more audio reproduction transducers of the audio environment to operate outside the linear range. According to some such examples, the noise estimator may not be configured to implement an acoustic echo canceller.

For example, an acoustic echo canceller can typically operate by updating (e.g., periodically updating) adaptive filter coefficients for each of multiple frequency bands. In some instances, the acoustic echo canceller can be implemented by the noise estimator 159 of Figure 1M. In some such embodiments, if multi-band limited audio data 155 would cause the loudspeaker 156 (or one or more other audio reproduction transducers of the audio environment) to operate outside the linear range, control signal 164 (or control signal 167) can control the acoustic echo canceller of the noise estimator 159 not to update the filter coefficients. According to some embodiments, the noise compensation system 150 may include one or more devices configured as a wake-word detector. In some such embodiments, an echo-cancelled version of the microphone signal 158 may be provided to the wake-word detector. In some instances, the acoustic echo canceller can be implemented by a device configured as a wake-word detector. Not updating the filter coefficients when the multi-band limited audio data 155 causes one or more other audio reproduction transducers in the audio environment to operate outside the linear range can improve the performance of the wake word detector. This is at least in part because the echo-cancelled version of the microphone signal 158 provided to the wake word detector can more accurately correspond to the speech command provided to the wake word detector. Here, box 525 relates to providing multi-band limited audio data to one or more audio reproduction transducers in the audio environment.

In some examples, method 500 (e.g., block 510) may involve applying a multi-band limiter to leveled audio data, such as as described above with reference to Figure 3. In such examples, the leveled audio data is an example of the “processed version of audio data” mentioned in block 510. Such examples may involve the control system controlling the level of input audio data based on at least one type of leveling instruction to produce leveled audio data. Some such examples may involve: the control system receiving at least one type of leveling instruction in relation to playback of the audio data; the control system determining a multi-band limiter configuration based at least in part on the at least one type of leveling instruction; and the control system configuring the multi-band limiter according to the multi-band limiter configuration. In some examples, the at least one type of leveling instruction may include at least one of a user input leveling instruction received via user input or a noise compensation leveling instruction received from a noise compensation module including an acoustic echo canceller.

According to some implementations, determining the multi-band limiter configuration may involve determining the tone preservation configuration if the level adjustment indication (e.g., if the only level adjustment indication) is a user-input level adjustment indication. In some instances, the tone preservation configuration may be band-dependent. For example, some bands may be partially or completely isolated. The level of a completely isolated band can be controlled independently without referencing the levels and/or thresholds of other bands.

In some examples, if receiving at least one type of level adjustment indication is a noise compensation level adjustment indication, determining the multiband limiter configuration may involve altering the tone preservation function. In some such examples, altering the tone preservation function may involve at least partially disabling the tone preservation function. In some implementations, the noise compensation level adjustment indication may correspond to the ambient noise level in the audio environment. In some such examples, the tone preservation function may be altered at least in part based on the ambient noise level.

According to some implementations, both a user input level adjustment indication and a noise compensation level adjustment indication can be received. In some such implementations, determining the multi-band limiter configuration may involve determining a tone preservation configuration, which is at least partially based on the average (e.g., a weighted average) of the multi-band limiter configuration corresponding to the user input level adjustment indication and the multi-band limiter configuration corresponding to the noise compensation level adjustment indication.

In some examples, method 500 may involve reproducing multi-band limited audio data on one or more audio reproduction transducers in an audio environment to provide reproduced audio data.

According to some examples, changes to the operation of a noise compensation module can involve altering the alternative or additional echo canceller functionality of the noise compensation module. For example, a change to the operation of a noise compensation module can involve causing the noise compensation module to use only a "mute" playback interval as input to the noise estimator of the noise compensation module. A "mute" playback interval can be an instance of an audio signal that is at or below a threshold level (e.g., a predetermined threshold level) in at least one of a frequency band or a time interval.

In some such implementations, method 500 may also involve providing multi-band limiter feedback from the multi-band limiter to the level adjuster module, for example, via a control signal 162 illustrated in FIG1M. According to some such examples, the multi-band limiter feedback may indicate the amount of limitation applied by the multi-band limiter to each of the multiple frequency bands of the leveled audio data.

In some such implementations, method 500 may also involve controlling the level of one or more frequency bands among a plurality of frequency bands by a level adjuster module based at least in part on multi-band limiter feedback. In some such examples, method 500 may involve reducing the level of one or more frequency bands based at least in part on multi-band limiter feedback indicating that the level of one or more frequency bands or other frequency bands is limited.

Figure 6A is a graph illustrating an example of the time interval of loudspeaker overdrive. The elements of Figure 6A are as follows:

• 601: Amplitude axis of the echo reference, which may also be referred to herein as the "loudspeaker reference" or "amplifier reference" of the echo canceller. An example of an echo reference is the multi-band limited audio data 155 of Figure 1M;

• 602: Time axis of the echo reference. Time axis 602 has the same range as the curve in Figure 6B;

• 603: A portion of the content (audio data) played by an audio system in an audio environment. In this example, the audio data is represented as a sine wave, and the loudspeaker is not overdriven. Therefore, the acoustic echo canceller updates its filter coefficients during the corresponding time interval;

• 604: The start of the overdrive interval of the loudspeaker;

·605: A sine wave with a large amplitude, indicating the audio data that the multi-band limiter will restrict when the control system does not allow the loudspeaker to be overdriven;

·606: End of the speaker overdrive time interval;

·607: A small-amplitude sine wave representing audio data during the time interval between when the loudspeaker operates again within its linear range;

• 614: A limitation indicating that the speaker-to-microphone path is no longer linear within this specific frequency range. When threshold 614 of Figure 6A is exceeded, the audio path 165 from loudspeaker 156 to microphone 157 (see Figure 1M) may include at least some sound corresponding to nonlinear distortion of loudspeaker 156. In some examples, threshold 614 may correspond to a single threshold for a specific frequency band (e.g., threshold 201 of Figure 2A). Exceeding threshold 614, the echo canceller will have difficulty modeling the system because at least a portion of the reproduced sound that the echo canceller will be modeling includes nonlinear distortion. However, since the corresponding overdrive setting of Figure 6B is sustained for a short period of time in many instances, it may be acceptable for the echo canceller not to update the model during this period, as the coefficients most recently calculated before the overdrive interval will be more accurate. Similarly, because ambient noise included in the microphone signal provided to the noise estimator will include nonlinear distortion during the overdrive interval, in some embodiments, the control system will cause the noise estimator not to provide an updated noise estimate during the overdrive interval.

Figure 6B illustrates an example of a signal that can be sent to an echo canceller corresponding to the graph in Figure 6A. In this example, the signal in Figure 6B indicates when the control system implements an “overdrive” mode, during which the control system (e.g., the level adjuster 152 in Figure 1M) can be configured with a multi-band limiter to allow some distortion (e.g., to increase the playback volume in the presence of ambient noise), thereby causing one or more audio reproduction transducers in the audio environment to operate outside the linear range. The “overdrive” mode can vary depending on the specific implementation. For example, the amount of overdrive allowed during the “overdrive” mode can depend on the frequency and, in some examples, can be set during the adjustment process (e.g., the factory adjustment process). In some instances, the amount of overdrive allowed at lower frequencies can be greater than the amount allowed at higher frequencies because humans are less sensitive to distortion at lower frequencies compared to higher frequencies within the human audible frequency range. According to this example, the signal in Figure 6B indicates whether the acoustic echo canceller should update its filter coefficients.

For example, the signal of FIG6B can be sent to the echo canceller implemented by the noise estimator 159 of FIG1M. For example, the signal of FIG6B can be an example of signal 167 sent to the noise estimator 159 by the multi-band limiter 154. In an alternative example, the signal of FIG6B can be sent to the echo canceller by a broadband limiter or by the level adjuster 152 of FIG1M.

In this example, the elements of Figure 6B are as follows:

• 608: Overdrive signal axis. In this example, the overdrive signal is 1 when the loudspeaker is overdriven, and 0 when the loudspeaker is not overdriven;

·609: Timeline;

·610: Crossover time when the loudspeaker changes from linear drive to overdrive;

·611: The time interval of loudspeaker overdrive as indicated by the overdrive signal value 1;

·612: Crossover time when a loudspeaker transitions from overdrive to linear operation;

·613: The time interval of the amplifier's linear operation as indicated by the drive signal value 0.

Therefore, in this example, the signals provided to the echo canceller include an echo reference and an overdrive signal value indicating whether the loudspeaker is operating outside its linear region. When the overdrive signal value is high (e.g., 1), in some examples, the noise estimate of the noise estimator will not be updated to ensure that the noise estimate remains accurate. According to some implementations, when the overdrive signal value is high, the noise estimator can be made to stop or pause providing a noise estimate or provide the same unupdated noise estimate. In some implementations, the overdrive signal value consists of only a single bit for the entire spectrum and operates in the time domain. Such implementations have the advantage of not requiring a nonlinear model of the loudspeaker. In some implementations, the overdrive signal can be implemented by embedding additional bits as part of the audio stream itself (e.g., as the least significant bit).

In some alternative implementations, an overdrive signal can be provided on a per-band basis. In some such implementations, it is assumed that the distortion products are harmonic distortion or intermodulation distortion (e.g., using the harmonic series of the original content to predict harmonic distortion and the Volterra series to predict intermodulation distortion), and therefore the bands where distortion products occur should be predictable. According to some such examples, if the frequencies of the distortion products are known, the echo canceller will not update those frequencies when it is known that the loudspeaker is overdriven in the bands where those frequencies will be generated. In some examples, if the frequencies of the distortion products are known, the noise estimator will not provide updated noise estimates for those frequencies when it is known that the loudspeaker is overdriven in the bands where those frequencies will be generated.

Figure 7 illustrates a system configured to control an acoustic echo canceller (AEC) at least in part based on the amount of “overdrive” occurring in the system. In some related embodiments, a noise estimator can be controlled as a supplement to or alternative to the AEC, similar to the system shown in Figure 7. In some examples, the blocks of Figure 7 are implemented as instances of the control system 210 of Figure 2E or the control system 110 of Figure 1M. According to this embodiment, the elements of Figure 7 are as follows:

701A: Input signal for the multi-band limiter;

• 702A: A multi-band limiter, in this example, having an adjustment that includes a threshold for each of a plurality of frequency bands above which the loudspeaker will operate outside its linear region. In this example, the multi-band limiter 702A also implements the ability to drive above/override these thresholds in order to provide additional volume at the expense of increased loudspeaker distortion (e.g., during periods of high ambient noise levels).

• 703A: Overdrive quantity occurring in the system. In some examples, element 703A may correspond to the overdrive signal described above with reference to Figures 6A and 6B;

• 704A: AEC Controller. In this example, the AEC controller 704A is configured to instruct the AEC 707A whether its coefficients should be updated based on the amount of distortion to be produced by one or more loudspeakers;

• 705A: A signal indicating whether AEC 707A should update its coefficients;

• 706A: Echo Reference (or Loudspeaker Feed). An example is the multi-band limited audio data 155 of Figure 1M;

• 707A: AEC. In this example, AEC 707A is a linear echo canceller whose coefficients can be optionally updated based on signal 705A.

Figure 8 illustrates an example of a system configured to determine the overdrive amount. According to this implementation, the elements of Figure 8 are as follows:

• 801A: Input signal to the multi-band limiter. In some examples, input signal 801A may correspond to the leveled audio data 153 in Figure 1M;

• 802A: A loudspeaker model that may include limiter thresholds for each of multiple frequency bands, the limiter thresholds being based on the capabilities of one or more loudspeakers in an audio environment. In some implementations, loudspeaker model 802A may correspond to the least capable loudspeaker in the audio environment. In some instances, loudspeaker model 802A may be a more complex model, such as a harmonic distortion model or an intermodulation distortion model;

• 803A: Overdrive Determination Module. In some examples, the overdrive determination module 803A may be implemented as an instance of the control system 210 of Figure 2E or an instance of the control system 110 of Figure 1M. In some examples, the overdrive determination module 803A may be implemented as a multi-band limiter. In some examples, the overdrive determination module 803A is configured to determine the amount of overdrive that the system will allow to be provided to one or more loudspeakers in the loudspeaker feed signal to the audio environment. In some examples, the overdrive determination module 803A may be configured to determine the amount of overdrive based on a noise estimate: in some such examples, if the noise estimate is high, the system is allowed to overdrive under the assumption that any distortion caused by overdrive will be masked. In some examples, the overdrive determination module 803A may be configured to incorporate more complex models, such as a harmonic distortion model, to predict the amount of distortion that the loudspeaker will generate.

·804A: Overdrive settings to be used in multi-band limiters;

·805A: Indicator of ambient noise.

Figure 9 is a graph showing the noise estimation and output level based on an example. In this example, the limits and the amount of tone retention or overdrive are determined based on the background noise level. According to this example, if the background noise level is low, the system's tone retention is maintained at its maximum value. In this example, if the background noise level is high, tone retention is turned off to maximize the system's volume. In some implementations, the overdrive amount may be determined based on whether the background noise level is high.

Based on this example, the elements of Figure 9 are as follows:

901: Frequency axis in Hertz;

·902: The acoustic level of the system, in dB sound pressure level (SPL);

·903: Output level of the (multiple) audio reproduction transducers for playing audio;

• 904: Example of high SPL background noise estimation. Due to high SPL background noise estimation, in some implementations, the control system can determine or estimate a relatively large amount of acoustic masking that may occur in the audio.

• 905: Example of medium SPL background noise estimation. Due to medium SPL background noise estimation, in some implementations, the control system can determine or estimate some acoustic masking that may occur in the audio.

• 906: Example of low SPL background noise estimation. Due to low SPL background noise estimation, in some implementations, the control system can determine or estimate acoustic masking of audio that is virtually impossible to occur;

• 907: The difference between the high SPL background level and the acoustic level of the reproduced audio from the system. In this example, the control system will operate the multi-band limiter in a manner that disables timbre preservation in most frequency bands. According to some examples, the overdrive amount can be increased to ensure that the difference between the noise level and the audio signal remains at or above a threshold (e.g., ensuring that the difference 907 is at least 2dB, 3dB, 4dB, 5dB, etc.). In some such implementations, the noise estimator/AEC will be notified as described above with reference to Figure 7. In some such implementations, the control system may allow overdrive of (multiple) audio reproduction transducers because the control system has identified or estimated a relatively large amount of acoustic masking that may occur in the audio, and therefore the introduced distortion may not be audible.

• 908: The difference between the low SPL background level and the acoustic level of the audio played back by the system. In some such examples, the control system will operate the multiband limiter at the highest tone preservation level or at least a higher tone preservation level than during instances of high background noise. According to some examples, the "highest" tone preservation level may correspond to a baseline tone preservation setting (e.g., factory setting) for low-noise or noise-free conditions. In some implementations, the control system will not overdrive the audio reproduction transducers(s) because the control system has determined or estimated that acoustic masking of the audio will hardly or not occur at all, and therefore the introduced distortion will likely be audible.

• 909: The difference between the moderate SPL background level and the acoustic level of the audio reproduced by the system. In some implementations, the amount of timbre preservation will be between the amounts described in reference elements 908 and 907 above. In some such examples, overdriving the system would be unnecessary or undesirable because the control system has determined or estimated that only a moderate amount of acoustic masking of the audio will occur, and therefore the introduced distortion may be audible.

Figure 10 shows an example floor plan of an audio environment, which in this example is a living space. As with other figures provided herein, the types and numbers of elements shown in Figure 10 are provided by way of example only. Other implementations may include more, fewer, and/or different types and numbers of elements.

According to this example, environment 1000 includes a living room 1010 in the upper left, a kitchen 1015 in the lower center, and a bedroom 1022 in the lower right. The boxes and circles distributed across the living space represent a group of loudspeakers 1005a to 1005h, at least some of which in some embodiments may be smart speakers placed in locations convenient to the space, but not following any standard layout (arbitrary placement). In some examples, television 1030 may be configured to at least partially implement one or more of the disclosed embodiments. In this example, environment 1000 includes cameras 1011a to 1011e distributed throughout the environment. In some embodiments, one or more smart audio devices in environment 1000 may also include one or more cameras. The one or more smart audio devices may be single-purpose audio devices or virtual assistants. In some such examples, one or more cameras of the optional sensor system 130 may reside in or on a television 1030, a mobile phone, or in one or more smart speakers (such as loudspeakers 1005b, 1005d, 1005e, or 1005h). Although cameras 1011a to 1011e are not shown in each depiction of environment 1000 presented in this disclosure, in some embodiments, each environment 1000 may still include one or more cameras.

Some aspects of this disclosure include a system or apparatus configured (e.g., programmed) to perform one or more examples of the disclosed methods, and a tangible computer-readable medium (e.g., a disk) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, digital signal processor, or microprocessor programmed with software or firmware and/or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system including input devices, memory, and a processing subsystem programmed (and/or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereon.

Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to perform necessary processing on (multiple) audio signals, including execution of one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed system (or its elements) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor that may include input devices and memory), programmed with software or firmware and/or otherwise configured to perform any of a variety of operations, including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the system of the invention are implemented as general-purpose processors or DSPs configured (e.g., programmed) to perform one or more examples of the disclosed methods, and the system also includes other elements (e.g., one or more loudspeakers and/or one or more microphones). A general-purpose processor configured to perform one or more examples of the disclosed methods may be coupled to input devices (e.g., a mouse and/or keyboard), memory, and a display device.

Another aspect of this disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) that stores code (e.g., an encoder executable to perform one or more examples of the disclosed methods or steps) for performing the disclosed methods or steps.

While specific embodiments and applications of this disclosure have been described herein, it will be apparent to those skilled in the art that many changes can be made to the described embodiments and applications without departing from the scope of this disclosure as described and claimed herein. It should be understood that although certain forms of this disclosure have been shown and described, this disclosure is not limited to the specific embodiments described and shown or the specific methods described.

Claims (22)

1. An audio processing method, comprising: The system receives a content stream, including input audio data, via an interface system. The control system applies a multi-band limiter to the audio data or a processed version of the audio data to produce multi-band limited audio data. Determine whether, when the multi-band limited audio data is played back on one or more audio reproduction transducers, the multi-band limited audio data will cause the one or more audio reproduction transducers in the audio environment to operate outside the linear range. The control system controls the updating of one or more filter coefficients by the acoustic echo canceller or the updating of the noise estimate by the noise estimator based on determining whether the multi-band limited audio data would cause one or more audio reproduction transducers of the audio environment to operate outside the linear range; and The multi-band limited audio data is provided to one or more audio reproduction transducers in the audio environment; Specifically, controlling the updating of the one or more filter coefficients by the acoustic echo canceller involves: if the multi-band limited audio data causes the one or more audio reproduction transducers of the audio environment to operate outside the linear range, then controlling the acoustic echo canceller not to update the one or more filter coefficients; and Specifically, controlling the updating of the noise estimate by the noise estimator involves: if the multi-band limited audio data causes one or more audio reproduction transducers of the audio environment to operate outside the linear range, then controlling the noise estimator not to provide an updated noise estimate. 2. The audio processing method as described in claim 1, further comprising: The control system receives at least one type of level adjustment instruction related to the playback of the audio data; The multi-band limiter configuration is determined by the control system and at least in part based on the at least one type of level adjustment indication; and The multi-band limiter is configured by the control system according to the configuration of the multi-band limiter. 3. The audio processing method of claim 2, further comprising: controlling the level of the input audio data by the control system based on the at least one type of level adjustment indication to generate level-adjusted audio data, wherein applying the multi-band limiter to the audio data or a processed version of the audio data involves applying the multi-band limiter to the level-adjusted audio data. 4. The audio processing method of claim 2, wherein the at least one type of level adjustment indication includes at least one of: a user input level adjustment indication received via user input or a noise compensation level adjustment indication received from a noise compensation module including the acoustic echo canceller. 5. The audio processing method of claim 4, wherein if receiving the at least one type of level adjustment instruction involves receiving the user input level adjustment instruction, then determining the multi-band limiter configuration involves determining the timbre preservation configuration. 6. The audio processing method of claim 4, wherein if receiving the at least one type of level adjustment indication involves receiving the noise compensation level adjustment indication, then determining that the multi-band limiter configuration involves changing the timbre preservation function. 7. The audio processing method of claim 6, wherein changing the timbre preservation function involves at least partially disabling the timbre preservation function. 8. The audio processing method of claim 6, wherein the noise compensation level adjustment indication corresponds to the ambient noise level in the audio environment, and wherein changing the timbre preservation function involves changing the timbre preservation function at least in part based on the ambient noise level. 9. The audio processing method of claim 8, further comprising: reproducing the multi-band limited audio data on the one or more audio reproduction transducers in the audio environment to provide reproduced audio data, the method further comprising: determining the masking effect of the ambient noise level on the reproduced audio data, wherein changing the timbre preservation function involves changing the timbre preservation function at least in part based on the masking effect. 10. The audio processing method of claim 5, wherein the timbre preservation configuration depends on the frequency band. 11. The audio processing method of claim 4, wherein receiving the at least one type of level adjustment indication involves receiving both the user input level adjustment indication and the noise compensation level adjustment indication, and wherein determining the multi-band limiter configuration involves determining a timbre preservation configuration, the timbre preservation configuration being based at least in part on a weighted average of the user input level adjustment indication and the noise compensation level adjustment indication. 12. The audio processing method of any one of claims 1 to 11, wherein the control system causes the one or more audio reproduction transducers of the audio environment to operate outside the linear range, at least in part, based on a noise compensation level adjustment indication corresponding to a high ambient noise level in the audio environment. 13. The audio processing method of any one of claims 4 to 11, further comprising: causing an additional noise compensation module to change operation when the multi-band limited audio data causes one or more audio reproduction transducers of the audio environment to operate outside the linear range. 14. The audio processing method of claim 13, wherein the additional noise compensation module operation change involves: causing the noise compensation module to use only a silence playback interval as input to the noise estimator of the noise compensation module, the silence playback interval being an instance during which the audio reproduction transducer of the audio environment operates within its linear range. 15. The audio processing method as described in claim 4, wherein the noise compensation module is a subsystem of the control system. 16. The audio processing method of claim 3, wherein the level adjuster module of the control system is configured to control the level of the input audio data to generate the level-adjusted audio data, the method further comprising: providing multi-band limiter feedback from the multi-band limiter to the level adjuster module. 17. The audio processing method of claim 16, wherein the multi-band limiter provides feedback indicating the amount of limitation applied by the multi-band limiter to each of the plurality of frequency bands of the horizontally adjusted audio data. 18. The audio processing method of claim 17, further comprising: controlling the level of one or more of the plurality of frequency bands by the level adjuster module at least in part based on feedback from the multi-band limiter. 19. An audio processing apparatus, the apparatus comprising a processor and a memory, the memory being coupled to the processor and storing instructions for execution by the processor, wherein the processor is configured to perform the method as claimed in any one of claims 1 to 18. 20. An audio processing system comprising an interface system and a control system, the control system being configured to implement the method as claimed in any one of claims 1 to 18. 21. A computer-readable storage medium having software stored thereon, the software including instructions for controlling one or more processors to perform the method as described in any one of claims 1 to 18. 22. A computer program product comprising a computer program that, when executed by a processor, causes the processor to perform the method as described in any one of claims 1 to 18.