RU2854360C2 - Media type-based dereverberation - Google Patents

Abstract

FIELD: computing.

SUBSTANCE: invention relates to the field of computing for processing audio data. The technical result is achieved by the steps of: classifying the media type of the input audio signal as one of the group comprising at least: 1) speech; 2) music or 3) speech over music; determining the degree of reverberation in the input audio signal; wherein determining the degree of reverberation comprises a step of computing a two-dimensional spectrum of acoustic modulation frequencies of the input audio signal, wherein the degree of reverberation is based on the amount of energy in the high modulation frequency part of the two-dimensional spectrum of acoustic modulation frequencies; determining whether to perform dereverberation for the input audio signal, at least based on determining that the media type of the input audio signal is classified as speech, and based on said degree of reverberation; and in response to determining that dereverberation should be performed for the input audio signal, forming an output audio signal by performing dereverberation for the input audio signal.

EFFECT: improving speech intelligibility and improving music sound quality.

14 cl, 12 dwg

Description

Cross-references to related applications

[0001] This application claims priority from the following priority applications: International Application No. PCT/CN2021/080314, filed March 11, 2021, U.S. Provisional Patent Application No. 63/180,710, filed April 28, 2021, and European Patent Application No. 21174289.5, filed May 18, 2021.

Field of technology to which the invention relates

[0002] This invention relates to systems, methods and media for dereverberation. Furthermore, this invention relates to systems, methods and media for classifying an input audio signal.

State of the art

[0003] Audio devices such as headphones, speakers, etc. are widely used. People often listen to audio content (e.g., podcasts, radio broadcasts, television broadcasts, music videos, etc.), which may include mixed types of media content, such as speech, music, speech over music, etc. Such audio content may include reverberation. Performing reverberation suppression for audio content, in particular user-generated audio content, which includes mixed types of media content, can be difficult.

Designations and terms

[0004] Throughout this description, including in the claims, the terms "driver," "loudspeaker," and "audio data transducer" are used synonymously to refer to any sound-producing transducer (or set of transducers) driven by a single input signal to the speakers. A typical set of headphones includes two speakers. A speaker may be implemented in such a way that it includes a plurality of transducers (e.g., a low-frequency speaker and a high-frequency speaker), which may be driven by a single, common input signal to the speakers or a plurality of input signals to the speakers. In some examples, the input signal to the speakers may be subject to different processing in different branches of the circuit connected to different transducers.

[0005] Throughout this description, including in the claims, the expression "performing an operation "on" a signal or data" (e.g., filtering, scaling, transforming, or applying gain to a signal or data) is used in a broad sense to mean performing an operation directly on the signal or data or on a processed version of the signal or data (e.g., on a version of the signal that has been pre-filtered or pre-processed prior to performing the operation).

[0006] Throughout this description, including in the claims, the term "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a "decoder system," and a system including such a subsystem (e.g., a system that generates X output signals in response to a plurality of inputs, wherein the subsystem generates M inputs, and the other X-M inputs are received from an external source) may also be referred to as a "decoder system."

[0007] Throughout this description, including in the claims, the term "processor" is used in a broad sense to refer to a system or device programmed or otherwise configured (e.g., by software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include a field-programmable gate array (or other configurable integrated circuit or chipset), a digital signal processor programmed and/or otherwise configured to perform pipeline processing on audio data or other sound data, a programmable general-purpose processor or computer, and a programmable microprocessor integrated circuit or chipset.

[0008] Throughout this description, including in the claims, the term "connected" or "connected" is used to mean a direct or indirect connection. Thus, if a first device is connected to a second device, the connection may be via a direct connection or via an indirect connection through other devices and connections.

[0009] Throughout this description, including in the claims, the term "classifier" is used generally to mean an algorithm that predicts the class of an input. For example, as used herein, an audio signal may be classified as being associated with a particular type of media, such as speech, music, speech over music, etc. It should be understood that various types of classifiers may be used to implement the technologies described herein, such as decision trees, an adaptive boosting algorithm (Ada-boost), an extreme gradient boosting algorithm (XG-boost), random forests, a generalized method of moments (GMM), hidden Markov models (HMM), a naive Bayes approach, and/or various types of neural networks (e.g., a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a long short-term memory (LSTM), a gated recurrent unit (GRU), etc.).

Disclosure of invention

[0010] At least some aspects of the present invention may be implemented as methods. Some methods may include receiving an input audio signal. Some such methods may include classifying the media type of the input audio signal as one of a group comprising at least: 1) speech; 2) music; or 3) speech over music. Some such methods may include determining whether to perform dereverberation on the input audio signal based at least on the determination that the media type of the input audio signal is classified as speech. Some such methods may include generating an output audio signal by performing dereverberation on the input audio signal in response to the determination that dereverberation is to be performed on the input audio signal.

[0011] In some examples, the method may include determining the degree of reverberation in the input audio signal, wherein the determination of whether to perform reverberation for the input audio signal may be based on the degree of reverberation. In some examples, the degree of reverberation may be based on at least one of: 1) the reverberation time (RT60); or 2) the ratio of the loudness levels of the direct and reverberant sound (DRR); or a diffuseness assessment. In some examples, determining the degree of reverberation may include calculating a two-dimensional spectrum of the acoustic modulation frequencies of the input audio signal, wherein the degree of reverberation may be based on the amount of energy in the high-frequency portion of the modulation from the two-dimensional spectrum of the acoustic modulation frequencies. In some examples, determining the degree of reverberation may include calculating at least one of: 1) a ratio of the energy in the high-frequency portion of the two-dimensional spectrum of acoustic modulation frequencies to the energy across all modulation frequencies in the two-dimensional spectrum of acoustic modulation frequencies; or 2) a ratio of the energy in the high-frequency portion of the two-dimensional spectrum of acoustic modulation frequencies to the energy in the low-frequency portion of the two-dimensional spectrum of acoustic modulation frequencies.

[0012] In some examples, the method may include determining whether to perform reverberation on the input audio signal based on a determination that the amount of reverberation exceeds a threshold.

[0013] In some examples, the method may include classifying the media type of the input audio signal by dividing the input audio signal into two or more spatial components. According to some implementations, the two or more spatial components may comprise a center channel and a side channel. In some examples, the method may further include calculating the power of the side channel and classifying the side channel in response to determining that the power of the side channel exceeds a threshold. According to other implementations, the two or more spatial components comprise a diffuse component and a direct component. In some examples, classifying the media type of the input audio signal may include classifying each of the two or more spatial components as one of: 1) speech; 2) music; or 3) speech over music, wherein the media type of the input audio signal may be classified by combining the classifications of each of the two or more spatial components. In some examples, the input audio signal may be divided into two or more spatial components in response to determining that the input audio signal contains a stereophonic audio signal.

[0014] In some examples, the method may include classifying the media type of the input audio signal by separating the input audio signal into a vocal component and a non-vocal component. In some examples, the input audio signal may be separated into a vocal component and a non-vocal component in response to determining that the input audio signal contains one audio channel. In some examples, the method may further include classifying the vocal component as one of: 1) speech; or 2) non-speech. The method may further include classifying the non-vocal component as one of: 1) music; or 2) non-music. In some examples, the media type of the input audio signal may be classified by combining the classification of the vocal component and the classification of the non-vocal component.

[0015] In some examples, determining whether to perform dereverberation on an input audio signal may be based on the classification of a second input audio signal that precedes the input audio signal.

[0016] In some examples, the method may include receiving a third input audio signal. The method may further include determining that dereverberation should not be performed for the third input audio signal. The method may further include prohibiting the execution of the dereverberation algorithm for the third input audio signal in response to the determination that dereverberation should not be performed for the third input audio signal. In some examples, the determination that dereverberation should not be performed for the third input audio signal may be based at least in part on the classification of the media type of the third input audio signal. In some examples, the classification of the third input audio signal may be one of: 1) music; or 2) speech over music. In some examples, the determination that dereverberation should not be performed for the third input audio signal may be based at least in part on the determination that the degree of reverberation in the third input audio signal is below a threshold.

[0017] According to another aspect of the present invention, a method is provided for classifying an input audio signal as one of at least two media types, wherein the method comprises: receiving the input audio signal; dividing the input audio signal into two or more spatial components; and classifying each of the two or more spatial components as one of the at least two media types, wherein the media type of the input audio signal is classified by combining the classifications of each of the two or more spatial components.

[0018] In some examples, the two or more spatial components comprise a center channel and a side channel, and the method further comprises: calculating the power of the side channel; and classifying the side channel in response to determining that the power of the side channel exceeds a threshold.

[0019] In some examples, the two or more spatial components comprise a diffuse component and a direct component.

[0020] In some examples, the input audio signal is separated into two or more spatial components in response to a determination that the input audio signal comprises a stereo audio signal.

[0021] In some examples, classifying the media type of the input audio signal comprises dividing the input audio signal into a vocal component and a non-vocal component. In some examples, the input audio signal is divided into a vocal component and a non-vocal component in response to determining that the input audio signal contains one audio channel. In some examples, classifying the media type of the input audio signal comprises: classifying the vocal component as one of: 1) speech; or 2) non-speech; classifying the non-vocal component as one of: 1) music; or 2) non-music, wherein the media type of the input audio signal is classified by combining the classification of the vocal component and the classification of the non-vocal component.

[0022] Some or all of the operations, functions, and/or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-volatile media. Such non-volatile media may include memory devices, such as the memory devices described herein, including, but not limited to, random access memory (RAM), read-only memory (ROM), etc. Accordingly, some innovative aspects of the subject matter described herein may be implemented as one or more non-volatile media having stored software.

[0023] At least some aspects of the present invention may be implemented as a device. For example, one or more devices may be capable of at least partially implementing the methods disclosed herein. In some implementations, the device is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single- or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof.

[0024] The present invention provides various technical advantages. For example, by selectively performing dereverberation for specific types of input audio signals (e.g., for input audio signals classified as speech), speech intelligibility can be improved. Furthermore, by prohibiting dereverberation for other types of input audio signals (e.g., for input audio signals classified as music, speech over music, etc.), disadvantageous results of dereverberation, such as reduced sound quality, can be prevented for audio signals for which speech intelligibility enhancement is not required. The technical advantages of the present invention may be particularly useful for user-generated content, such as podcasts.

[0025] The details of one or more embodiments of the subject matter described in this detailed description are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. It should be noted that the relative sizes of the following drawings may not be to scale.

Brief description of the drawings

[0026] Figs. 1A and 1B illustrate representations of example audio signals that include reverberation.

[0027] Fig. 2 shows a block diagram of an example system for performing de-riverberation based on a media type in accordance with some implementations.

[0028] Fig. 3 shows an example of a process for performing de-embedding based on a media type in accordance with some implementations.

[0029] Fig. 4 shows an example of a process for spatially separating input audio signals in accordance with some implementations.

[0030] Fig. 5 shows an example of a process for separating audio input sources in accordance with some implementations.

[0031] Fig. 6 shows an example of a process for determining the degree of reverberation in accordance with some implementations.

[0032] Figs. 7A, 7B, 7C, and 7D show exemplary graphs of two-dimensional acoustic modulation frequency spectra of exemplary audio signals.

[0033] Fig. 8 shows a block diagram that illustrates examples of components of a device capable of implementing various aspects of the present invention.

[0034] Like reference numerals and symbols in the various drawings indicate like elements.

Implementation of the invention

[0035] Reverberation occurs when an audio signal is distorted by various reflections from various surfaces (e.g., walls, ceilings, floors, furniture, etc.). Reverberation can have a significant impact on sound quality and speech intelligibility. Accordingly, dereverberation of an audio signal that includes speech can be performed to improve speech intelligibility.

[0036] Sound arriving at a receiver (e.g., a human listener, a microphone, etc.) consists of direct sound, which includes sound directly from the source without any reflections, and reverberant sound, which includes sound reflected from various surfaces in the environment. Reverberant sound includes early reflections and late reflections. Early reflections may reach the receiver somewhat later than or parallel to the direct sound and, therefore, may be partially integrated into the direct sound. The integration of early reflections with the direct sound creates a spectral coloring effect that contributes to the perceived quality of the sound. Late reflections arrive at the receiver after the early reflections (e.g., more than 50-80 milliseconds after the direct sound). Late reflections may have a negative effect on speech intelligibility. Accordingly, dereverberation can be performed on an audio signal to reduce the effect of late reflections present in the audio signal in order to thereby improve speech intelligibility.

[0037] Fig. 1A shows an example of acoustic impulse responses in a reverberant environment. As illustrated, early reflections 102 may arrive at the receiver in parallel with or immediately after the direct sound. In contrast, late reflections 104 may arrive at the receiver after the early reflections 102.

[0038] Fig. 1B shows an example of a time-domain input audio signal 152 and a corresponding spectrogram 154. As illustrated in the spectrogram 154, early reflections can cause changes in the spectrogram 154, which are shown by spectral colorings 156.

[0039] Dereverberation can reduce sound quality, for example, by reducing the perceived loudness level, changing spectral color effects, and the like. Reduced sound quality can be particularly disadvantageous when dereverberation is performed on audio signals that primarily include music or speech over music. For example, the sound quality of an audio signal that primarily includes music or speech over music can be degraded without any improvement in speech intelligibility. As a more specific example, dereverberation can be suitable for processing low-quality speech content, such as user-generated content that is captured in long-range use cases. Continuing with this specific example, user-generated content, such as podcasts, can include both low-quality speech content and professionally generated musical content. In some cases, professionally generated musical content can include artificial reverberation. In such cases, applying dereverberation to mixed media content (e.g., including low-quality speech content and professionally generated music content with artificial reverberation) may introduce excessive reverberation suppression, which may degrade the audio quality.

[0040] In some implementations, dereverberation may be performed on an input audio signal based on the identification of the media type(s) associated with the input audio signal. For example, the input audio signal may be analyzed to determine whether the input audio signal is: 1) speech; 2) music; 3) speech over music; or 4) other. Examples of speech over music content may include podcast intros or outros, television show intros or outros, etc.

[0041] In some implementations, dereverberation may be performed for input audio signals that are identified as speech or primarily speech. On the other hand, dereverberation may be prohibited for input audio signals that are identified as music, primarily music, speech over music, or primarily speech over music. By prohibiting dereverberation for media types that do not represent speech or primarily speech, dereverberation may be performed for input audio signals that should significantly benefit from dereverberation (e.g., because the input audio signal primarily includes speech), while preventing a reduction in audio quality resulting from dereverberation when such dereverberation is not required to improve speech intelligibility.

[0042] In some implementations, an input audio signal may be classified as representing one of: 1) speech; 2) music; 3) speech over music; or 4) other, using various technologies. As used herein, "other" may mean noise, sound effects, speech over sound effects, and the like. For example, in some implementations, an input audio signal may be classified by splitting the input audio signal into two or more spatial components and classifying each spatial component as one of: 1) speech; 2) music; 3) speech over music; or 4) other. Continuing with this example, in some implementations, the classification of each spatial component may then be combined to form an aggregated classification for the input audio signal. As another example, in some implementations, an input audio signal may be classified by splitting the input audio signal into a vocal component and a non-vocal component. The vocal component may be classified as one of: 1) speech; or 2) not speech, and the non-vocal component may be classified as one of: 1) music; or 2) not music. Continuing with this example, in some implementations, the classification of each of the vocal component and the non-vocal component may then be combined to form an aggregated classification of the input audio signal. Although several classification methods have been described in the present description in the context of a reverberation suppression method, the classification methods according to the invention may be used in other contexts. In particular, the present invention relates to a method for classifying an input audio signal as one of at least two media types, comprising: receiving an input audio signal; dividing the input audio signal into two or more spatial components; and classifying each of the two or more spatial components as one of the at least two media types, wherein the media type of the input audio signal is classified by combining the classifications of each of the two or more spatial components.

[0043] In some implementations, an input audio signal that is classified as speech may be further analyzed to determine the amount of reverberation present in the input audio signal. In some such implementations, reverberation may be performed on input audio signals that are identified as having a greater than threshold amount of reverberation. The amount of reverberation may be identified using the ratio of the loudness levels of direct and reverberant sound (DRR) and/or using the reverberation time (RT) to 60 dB (e.g., RT60), and/or using a diffuseness measurement and/or other suitable reverberation metrics. It should be noted that the amount of reverberation may be a function of DRR, in which the amount of reverberation increases as DRR values decrease, and in which the amount of reverberation decreases as DRR values increase.

[0044] Additionally or alternatively, in some implementations, dereverberation may be performed on the input audio signal based on the media type classification of the preceding audio signal. In some implementations, the preceding audio signal may be a preceding frame or portion of audio content that precedes the input audio signal. In some implementations, the classification of the input audio signal may be adjusted based on the classification of the preceding audio signal such that the classifications of adjacent audio signals are effectively smoothed. The adjustment may be performed based on the confidence levels of each classification. Determining whether to perform dereverberation on the input audio signal at least in part based on the classification of the preceding audio signal makes it possible to prevent dereverberation from being applied in an intermittent manner, thereby improving the overall audio quality.

[0045] In some implementations, dereverberation may be performed on an input audio signal using various technologies. For example, in some implementations, dereverberation may be performed based on amplitude modulation of the input audio signal in various frequency bands. As a more specific example, in some embodiments, a time domain audio signal may be converted into a frequency domain signal. Continuing with this more specific example, the frequency domain signal may be divided into several frequency subbands, for example, by applying a filter bank to the frequency domain signal. Further continuing with this more specific example, amplitude modulation values may be determined for each frequency subband, and bandpass filters may be applied to the amplitude modulation values. In some implementations, the bandpass filter values may be selected based on the beat of human speech, for example, so that the center frequency of the bandpass filter exceeds the beat of human speech (e.g., in the range of 10-20 Hz, approximately 15 Hz, etc.). Continuing with this specific example, gains can be determined for each frequency subband based on a function of the amplitude modulation signal values and the bandpass-filtered amplitude modulation values. Gains can then be applied to each frequency subband. In some implementations, dereverberation can be accomplished using techniques described in U.S. Patent No. 9,520,140, which is hereby incorporated by reference in its entirety.

[0046] As another example, in some implementations, dereverberation may be performed by estimating the dereverberated signal using a deep neural network, a weighted prediction error method, a normalized variance delayed linear prediction method, a single-channel linear filter, a multi-channel linear filter, and the like. As yet another example, in some implementations, dereverberation may be performed by estimating a room response and performing a deconvolution operation on the input audio signal based on the room response.

[0047] It should be noted that the technologies described in this document for de-embedding based on the media type can be performed for various types or forms of audio content, including, but not limited to: podcasts, radio broadcasts, audio content associated with video conferences, audio content associated with television broadcasts or movies, etc. The audio content can be presented in real time or pre-recorded.

[0048] Fig. 2 shows a block diagram of an example system 200 that may be used to perform dereverberation based on an identified media type associated with an input audio signal in accordance with some implementations.

[0049] As illustrated, system 200 may include a media type classifier 202. Media type classifier 202 may receive an input audio signal. In some implementations, media type classifier 202 may classify the input audio signal as: 1) speech; 2) music; 3) speech over music; or 4) other.

[0050] In some implementations, in response to determining that the input audio signal does not represent speech or does not represent primarily speech (e.g., in response to determining that the input audio signal represents music, speech over music, or other), the media type classifier 202 may transmit the input audio signal without directing the input audio signal to the reverberation analyzer 204. On the other hand, in response to determining that the input audio signal represents speech or represents primarily speech, the media type classifier 202 may transmit the input audio signal to the reverberation analyzer 204.

[0051] In some implementations, the reverberation analyzer 204 may determine the degree of reverberation present in the input audio signal. In some implementations, the reverberation analyzer 204 may determine that dereverberation should be performed for the input audio signal in response to determining that the degree of reverberation exceeds a threshold. Thus, in some implementations, the reverberation analyzer 204 may further direct the input audio signal to the dereverberation component 206 in response to determining that the input audio signal is sufficiently reverberant. In contrast, in response to determining that the input audio signal is not sufficiently reverberant (for example, that the input audio signal includes relatively "dry" speech), the reverberation analyzer 204 may pass the input audio signal without directing the input audio signal to the dereverberation component 206, effectively prohibiting the performance of dereverberation for the input audio signal.

[0052] The dereverberation component 206 may receive, as input, an input audio signal that is determined to have reverberation that exceeds a threshold value and may generate a dereverberated audio signal. It should be understood that the dereverberation component 206 may perform any suitable reverberation suppression technology(s).

[0053] In some implementations, the media type classifier 202 classifies the media type of the input audio signal based on one or both of the spatial separation of the components of the input audio signal or the separation of the music sources of the components of the input audio signal.

[0054] For example, in some implementations, the media type classifier 202 may include a spatial information separator 208. The spatial information separator 208 may separate the input audio signal into two or more spatial components. Examples of the two or more spatial components may include a direct component and a diffuse component, a side channel and a center channel, and the like. In some implementations, the spatial information separator 208 may classify the media type of the input audio signal by separately classifying each of the two or more spatial components. In some implementations, the spatial information separator 208 may then form a classification for the input audio signal by combining the classifications for each of the two or more components, for example, by using a decision fusion algorithm. Examples of decision fusion algorithms that may be used to combine the classifications for each of the two or more components include Bayesian analysis, the Dempster-Shafer algorithm, fuzzy logic algorithms, and the like. It should be noted that technologies for classifying the media type based on the separation of spatial sources are shown in and described below with reference to Fig. 4.

[0055] As another example, in some implementations, the media type classifier 202 may include a music source separator 210. The music source separator 210 may separate the input audio signal into a vocal component and a non-vocal component. In some implementations, the music source separator 210 may then classify the vocal component as one of: 1) speech; or 2) not speech. In some implementations, the music source separator 210 may classify the non-vocal component as one of: 1) music; or 2) not music. In some implementations, the music source separator 210 may generate a classification of the input audio signal as one of: 1) speech; 2) music; 3) speech over music; or 4) other, based on the classifications of the vocal component and the non-vocal component. For example, in some implementations, the music source separator 210 may combine the classifications of the vocal component and the non-vocal component (e.g., by using a decision fusion-based algorithm). Examples of decision fusion algorithms that can be used to combine classifications for each of two or more components include Bayesian analysis, the Dempster-Shafer algorithm, fuzzy logic algorithms, etc.

[0056] In some implementations, the media type classifier 202 may determine whether the media type of the input audio signal should be classified using the spatial information separator 208 or by using the music source separator 210. For example, the media type classifier 202 may determine that the media type should be classified using the spatial information separator 208 in response to determining that the input audio signal is a stereo audio signal. As another example, the media type classifier 202 may determine that the media type should be classified using the music source separator 210 in response to determining that the input audio signal is a monochannel audio signal.

[0057] In the example of Fig. 2, the media type classifier 202 is used in the context of the system 200 for performing dereverberation. It should be emphasized that the media type classifier 202 can be used as a stand-alone system or can be used in other audio processing systems.

[0058] Fig. 3 shows an example of a process 300 for performing dereverberation for input audio signals based on a classification of media types in accordance with some implementations. In some implementations, the steps of the process 300 may be performed by a device (e.g., device 200 of Fig. 2). It should be noted that in some implementations, the steps of the process 300 may be performed in orders not shown in Fig. 3, and/or one or more steps of the process 300 may be performed substantially in parallel. Furthermore, it should be noted that in some implementations, one or more steps of the process 300 may be omitted.

[0059] At 302, process 300 may receive an input audio signal. The input audio signal may be recorded or may be real-time content. The input audio signal may include various types of audio content, such as speech, music, speech over music, and the like. Exemplary types of audio content may include podcasts, radio broadcasts, audio content associated with television broadcasts or movies, and the like.

[0060] At 304, process 300 may classify the media type of the input audio signal. For example, in some implementations, process 300 may classify the input audio signal as representing one of: 1) speech; 2) music; 3) speech over music; or 4) other.

[0061] In some implementations, process 300 may classify the media type of the input audio signal based on the separation of spatial components of the input audio signal. For example, in some implementations, process 300 may separate the input audio signal into two or more spatial components, such as a direct component and a diffuse component, a side channel and a center channel, etc. In some implementations, process 300 may then classify the media type of the audio content in each spatial component. In some implementations, process 300 may then classify the input audio signal by combining the classifications of each spatial component. It should be noted that more detailed techniques for classifying the media type of the input audio signal based on spatial separation are shown in Fig. 4 and are described below with reference thereto.

[0062] Additionally or alternatively, in some implementations, process 300 may classify the media type of the input audio signal based on the separation of music sources of the input audio signal. For example, in some implementations, process 300 may separate the input audio signal into a vocal component and a non-vocal component. In some implementations, process 300 may then classify the media type of the audio content in each of the vocal component and the non-vocal component. In some implementations, process 300 may then classify the input audio signal by combining the classifications of each of the vocal component and the non-vocal component. It should be noted that more detailed techniques for classifying the media type of the input audio signal based on the separation of music sources are shown in Fig. 5 and are described below with reference thereto.

[0063] At 306, the process 300 may determine whether to analyze the reverberation characteristics of the input audio signal. In some implementations, the process 300 may determine whether to analyze the reverberation characteristics based on the media type classification of the input audio signal determined at step 304. For example, in some implementations, the process 300 may determine that the reverberation characteristics should be analyzed ("Yes" at 306) in response to a determination that the media type classification of the input audio signal represents speech. On the other hand, in some implementations, the process 300 may determine that the reverberation characteristics should not be analyzed ("No" at 306) in response to a determination that the media type classification does not represent speech (e.g., that the media type classification represents music, speech over music, or other).

[0064] If, at 306, process 300 determines that the reverberation characteristics are not to be analyzed (“No” at 306), process 300 may terminate at 314.

[0065] On the other hand, if, at 306, the process 300 determines that the reverberation characteristics are to be analyzed (“Yes” at 306), the process 300 may determine the amount of reverberation in the input audio signal at 308.

[0066] In some implementations, the amount of reverberation may be calculated using an RT60 metric and/or a DRR metric associated with the input audio signal.

[0067] Additionally or alternatively, in some implementations, the process 300 may determine the degree of reverberation in the input audio signal based on the spectrogram information. For example, in some implementations, the process 300 may determine the degree of reverberation based on the energy at various modulation frequencies of the input audio signal. In particular, since non-reverberant speech may tend to have a peak in modulation frequency at a relatively low modulation frequency (e.g., 3 Hz, 4 Hz, etc.), and since reverberant speech may tend to have significant energy at higher modulation frequencies (e.g., 10 Hz, 20 Hz, 50 Hz, etc.), the process 300 may determine the degree of reverberation in the input audio signal based on the energy of the input audio signal at relatively high modulation frequencies (e.g., above 10 Hz, above 20 Hz, etc.).

[0068] It should be noted that more detailed techniques for determining the degree of reverberation based on spectrogram information are shown in Fig. 7 and described below with reference thereto.

[0069] At 310, the process 300 may determine whether to perform dereverberation on the input audio signal. In some implementations, the process 300 may determine whether to perform dereverberation based on the degree of reverberation determined at step 308. For example, in some implementations, the process 300 may determine that dereverberation should be performed ("Yes" at 310) in response to determining that the degree of reverberation exceeds a threshold. As another example, in some implementations, the process 300 may determine that dereverberation should not be performed ("No" at 310) in response to determining that the degree of reverberation is below a threshold.

[0070] In some implementations, as an addition or alternative, the process 300 may determine whether to perform dereverberation for the input audio signal based on the classification of the media types of the preceding audio signal. The preceding audio signal may correspond to a frame or portion of the audio content that precedes the input audio signal. It should be noted that the frame or portion of the audio content may have any suitable duration, for example, 10 milliseconds, 20 milliseconds, etc.

[0071] In some implementations, process 300 may determine whether to perform dereverberation on an input audio signal based on the media type classification of a preceding audio signal by adjusting the media type classification (e.g., as determined in step 304) based on the classification of the preceding audio signal. For example, in some implementations, the media type classification of the input audio signal may be adjusted based on the confidence level of the media type classification of the input audio signal and/or based on the confidence level of the media type classification of the preceding audio signal. As a more specific example, in the case in which the classification of the media types of the preceding audio signal is associated with a relatively high confidence level (e.g., more than 70%, more than 80%, etc.), and in which the classification of the media types of the input audio signal is associated with a relatively low confidence level (e.g., less than 30%, less than 20%, etc.), the classification of the media types of the input audio signal can be adjusted or modified such that it represents the classification of the media types of the preceding audio signal. It should be noted that the adjustment of the classification of the media types of the input audio signal can be performed one or more times. For example, the classification of the media types can be adjusted before the analysis of the reverberation characteristics in step 306. As another example, the classification of the media types can be adjusted after the degree of reverberation is determined in step 308.

[0072] If, at 310, process 300 determines that derivation should not be performed (“No” at 310), process 300 may terminate at 314.

[0073] On the other hand, if, at 310, the process 300 determines that dereverberation should be performed ("Yes" at 310), the process 300 can generate an output audio signal by performing dereverberation on the input audio signal. For example, in some implementations, dereverberation can be performed based on amplitude modulation of the input audio signal in different frequency bands. As a more specific example, dereverberation can be performed using the technologies described in U.S. Patent No. 9,520,140, which is hereby incorporated herein by reference in its entirety. As another example, in some implementations, dereverberation can be performed by estimating the dereverberated signal using a deep neural network, a multi-channel linear filter, and the like. As yet another example, in some implementations, dereverberation can be performed by estimating the room response and performing a deconvolution operation on the input audio signal based on the room response.

[0074] Process 300 may then terminate at 314.

[0075] It should be noted that after completion at 314, the output audio signal may be presented, for example, through speakers, headphones, etc. In some implementations, in cases in which the dereverberation of step 312 is not performed (for example, because the input audio signal is classified as music, speech over music, or other non-speech content), the output audio signal may be the original input audio signal. Alternatively, in some implementations, in cases in which the dereverberation of step 312 is not performed (for example, because the input audio signal is classified as speech, speech over music, or other non-speech content), a different dereverberation technology may be applied to the original input audio signal than the technology that is applied at 312.

[0076] In some implementations, in cases in which dereverberation is performed at step 312, the output audio signal may correspond to the dereverberated input audio signal.

[0077] In some implementations, the media type of the input audio signal may be classified based on the spatial separation of the components of the input audio signal. Exemplary components include a direct component and a diffuse component, a center channel and a side channel, and the like. In some implementations, each spatial component may be classified as one of: 1) speech; 2) music; 3) speech over music; or 4) other. In some implementations, the input audio signal may be classified based on a combination of the classification of each of the spatial components. In some implementations, two or more spatial components may be identified based on an upmix of the input audio signal. In some implementations, the classification of media types of the input audio signal based on the spatial separation of the components of the input audio signal may be performed in response to a determination that the input audio signal is a multi-channel audio signal (e.g., a stereo audio signal, a 5.1 audio signal, a 7.1 audio signal, and the like).

[0078] Fig. 4 shows an example of a process 400 for classifying the media type of an input audio signal based on the spatial separation of the components of the input audio signal in accordance with some implementations. It should be noted that the steps of the process 400 may be performed in various orders not shown in Fig. 4, and/or in some implementations, two or more steps of the process 400 may be performed substantially in parallel. In addition or alternatively, it should be noted that in some implementations, one or more steps of the process 400 may be omitted.

[0079] Process 400 may begin at 402 by receiving an input audio signal. In some implementations, the input audio signal may include two or more audio channels.

[0080] At 404, process 400 may perform an upmix of the input audio signal to increase the number of audio channels associated with the input audio signal. Process 400 may use various types of upmixing. For example, in some implementations, process 400 may perform an upmixing technique such as mixing left/right with middle/side channels. As another example, in some implementations, process 400 may perform an upmixing technique that converts stereo audio input into multi-channel content, such as 5.1, 7.1, and the like.

[0081] In some implementations, the input audio signal may be decomposed into a direct component and a diffuse component. For example, in some implementations, the direct component and the diffuse component may be identified based on inter-channel coherence. As a more specific example, in some implementations, the direct component and the diffuse component may be identified based on a coherence matrix analysis.

[0082] At 406, process 400 may obtain side and center channels from upmixed input audio signals. For example, in a case in which the upmixed input audio signal corresponds to mixed middle/side channels, the side channel may correspond to the mixed side channel, and the center channel may correspond to the mixed middle channel. As another example, in a case in which the upmixed input audio signal corresponds to a multi-channel upmix (e.g., 5.1, 7.1, etc.), the center channel may be received directly from the upmixed audio signal, and the side channel may be obtained by downmixing a left/right channel pair (e.g., left/right, left surround/right surround, etc.).

[0083] In a case in which the input audio signal is split into a direct component and a diffuse component, the center channel may correspond to the direct component, and the side channel may correspond to the diffuse component.

[0084] At 408, process 400 may determine whether the side channel input power exceeds a threshold value. Examples of threshold values may be -65 dB relative to full scale (dBFS), -68 dBFS, -70 dBFS, -72 dBFS, etc.

[0085] If it is determined at 408 that the side channel input power does not exceed the threshold value (“No” at 408), the process 400 may proceed to step 412.

[0086] On the other hand, if at 408 it is determined that the input power of the side channel exceeds the threshold value (“Yes” at 408), the process 400 may classify the side channel at 410 as one of: 1) speech; 2) music; 3) speech over music; or 4) other. In some implementations, the classification of the side channel may be associated with a confidence level. Examples of classifiers that may be used to classify the side channel include the k-nearest neighbor method, analogical reasoning, decision trees, naive Bayes and/or various types of neural networks (e.g., a convolutional neural network (CNN), etc.).

[0087] At 412, process 400 may classify the center channel as one of: 1) speech, 2) music; 3) speech over music; or 4) other. In some implementations, the classification of the center channel may be associated with a confidence level. Examples of classifiers that may be used to classify the center channel include the k-nearest neighbor method, analogical reasoning, decision trees, naive Bayes, and/or various types of neural networks (e.g., a convolutional neural network (CNN), etc.).

[0088] At 414, process 400 may classify the input audio signal as one of: 1) speech; 2) music; 3) speech over music; or 4) other, by combining the side channel classification (if any) with the center channel classification.

[0089] For example, in some implementations, the side channel classification and the center channel classification may be combined using a decision fusion algorithm. Examples of decision fusion algorithms that may be used to combine classifications for each of two or more components include Bayesian analysis, the Dempster-Shafer algorithm, fuzzy logic algorithms, and the like.

[0090] As another example, in some implementations, in response to the classification of the side channel as music, speech over music, or other, the input audio signal may be classified as "not speech," regardless of the classification of the center channel. As a more specific example, in a case in which the center channel is classified as "speech," and in which the side channel is classified as "music," the input audio signal may be classified as speech over music.

[0091] As yet another example, in some implementations, the side channel classification and the center channel classification may be combined based on confidence levels associated with the side channel classification and the center channel classification, respectively. As a more specific example, in some implementations, the side channel classification and the center channel classification may be combined in such a way that the classification of the spatial component associated with a higher confidence level is weighted more in the combination. As a specific example, in a case in which the center channel is classified as "speech" with a relatively high confidence level (e.g., more than 70%, more than 80%, etc.), and in which the side channel is classified as "music", "speech over music", or "other" with a relatively low confidence level (e.g., less than 30%, less than 20%, etc.), the input audio signal may be classified as speech. As another specific example, in a case in which the center channel is classified as "speech" with a relatively low confidence level (e.g., less than 30%, less than 20%, etc.), and in which the side channel is classified as "music", "speech over music", or "other" with a relatively high confidence level (e.g., greater than 70%, greater than 80%, etc.), the input audio signal may be classified as "speech over music" or "other".

[0092] It should be noted that in the case in which the side channel is not classified (e.g., because the input power of the side channel is below the threshold, as determined in step 408), the classification of the input audio signal may correspond to the classification of the center channel.

[0093] In some implementations, the input audio signal may be classified based on the separation of music sources of the input audio signal into a vocal component and a non-vocal component. The vocal component may then be classified as speech or non-speech, and the non-vocal component may be classified as music or non-music. In some implementations, the input audio signal may then be classified as one of: 1) speech; 2) music; 3) speech over music; or 4) another, based on a combination of the classifications of the vocal component and the non-vocal component. In some implementations, the input audio signal may be classified using the separation of music sources of the input audio signal in response to a determination that the input audio signal is a monochannel audio signal. Alternatively, in some implementations, the input audio signal may be classified using the separation of music sources in addition to the classification of the input audio signal based on the spatial separation of the components.

[0094] Fig. 5 shows an example of a process 500 for classifying an input audio signal based on separation of music sources in accordance with some implementations. It should be noted that the steps of the process 500 may be performed in various orders not shown in Fig. 5, and/or in some implementations, two or more steps of the process 500 may be performed substantially in parallel. Additionally or alternatively, it should be noted that in some implementations, one or more steps of the process 500 may be omitted.

[0095] Process 500 may begin at 502 by receiving an input audio signal. In some implementations, the input audio signal may be a single-channel audio signal.

[0096] At 504, the process 500 may separate the input audio signal into a vocal component and a non-vocal component. In some implementations, the vocal component and the non-vocal component may be identified using one or more trained machine learning models. Exemplary types of machine learning models that may be used to separate the input audio signal into a vocal component and a non-vocal component may include a deep neural network (DNN), a convolutional neural network (CNN), a long short-term memory (LSTM)-based network, a convolutional recurrent neural network (CRNN), a gated recurrent unit (GRU), a convolutional gated recurrent unit (CGRU), and the like.

[0097] At 506, process 500 may classify the vocal component as one of: 1) speech; or 2) not speech. In some implementations, the classification of the vocal component may be associated with a confidence level. Examples of classifiers that may be used to classify the vocal component include the k-nearest neighbor method, analogical reasoning, decision trees, naive Bayes, and/or various types of neural networks (e.g., a convolutional neural network (CNN), etc.).

[0098] At 508, process 500 may classify the non-vocal component as one of: 1) music; and 2) not music. In some implementations, the classification of the non-vocal component may be associated with a confidence level. Examples of classifiers that may be used to classify the non-vocal component include the k-nearest neighbor method, analogical reasoning, decision trees, naive Bayes, and/or various types of neural networks (e.g., a convolutional neural network (CNN), etc.).

[0099] At 510, the process 500 may classify the input audio signal as one of: 1) speech; 2) music; 3) speech over music; or 4) another, by combining the classification of the vocal component and the classification of the non-vocal component. For example, in some implementations, the classification of the vocal component may be combined with the classification of the non-vocal component using any suitable decision fusion algorithm(s) that combine classifications from two classifiers to form an aggregated classification of the input audio signal. Examples of decision fusion algorithms that may be used to combine classifications for each of two or more components include a Bayesian approach, a Dempster-Shafer approach, fuzzy logic algorithms, and the like.

[0100] As another example, in some implementations, the classification of the vocal component may be combined with the classification of the non-vocal component based on the confidence levels of the classification of the vocal component and the classification of the non-vocal component, respectively. As a more specific example, in some implementations, the classification of the vocal component and the classification of the non-vocal component may be combined such that the component associated with a higher confidence level is weighted more in the combination.

[0101] In some implementations, the amount of reverberation present in the input audio signal may be determined. In some implementations, the amount of reverberation may be calculated using the DRR. For example, in some implementations, the amount of reverberation may be inversely related to the DRR such that the amount of reverberation increases as the DRR values decrease, and such that the amount of reverberation decreases as the DRR values increase. In some implementations, the amount of reverberation may be calculated using the time duration required to reduce the sound pressure level by a fixed amount (e.g., 60 dB). For example, the amount of reverberation may be calculated using RT60, which indicates the time for reducing the sound pressure level by 60 dB. In some implementations, the DRR or RT60 associated with the input audio signal may be estimated using various algorithms or technologies, which may be based on signal processing and/or based on a machine learning model.

[0102] In some implementations, the amount of reverberation in the input audio signal may be calculated by estimating the diffuseness of the input audio signal. Fig. 6 shows an example of a process 600 for estimating the diffuseness of the input audio signal in accordance with some implementations. It should be noted that the steps of the process 600 may be performed in different orders not shown in Fig. 6, and/or in some implementations, two or more steps of the process 600 may be performed substantially in parallel. Additionally or alternatively, it should be noted that in some implementations, one or more steps of the process 600 may be omitted.

[0103] It should be noted that in some implementations, the amount of reverberation may be determined based on a combination of multiple metrics. The multiple metrics may include, for example, DRR, RT60, diffuseness rating, etc. In some implementations, the multiple metrics may be combined using various techniques, such as weighted averaging. In some implementations, one or more metrics may be scaled or normalized.

[0104] Process 600 may begin at 602 by receiving an input audio signal.

[0105] At 604, process 600 may calculate a two-dimensional spectrum of acoustic modulation frequencies of the input audio signal. The two-dimensional spectrum of acoustic modulation frequencies may indicate the energy present in the input audio signal as a function of the acoustic frequency and the modulation frequency.

[0106] At 606, the process 600 may determine the degree of dispersion of the input audio signal based on the energy in the high modulation frequency portion (e.g., for modulation frequencies greater than 6 Hz, greater than 10 Hz, etc.) of the two-dimensional spectrum of acoustic modulation frequencies. For example, in some implementations, the process 600 may calculate the ratio of the energy in the high modulation frequency portion to the energy across all modulation frequencies. As another example, in some implementations, the process 600 may calculate the ratio of the energy in the high modulation frequency portion to the energy in the low modulation frequency portion (e.g., for modulation frequencies below 10 Hz, below 20 Hz, etc.)

[0107] Figs. 7A, 7B, 7C and 7D show examples of two-dimensional acoustic modulation frequency spectra for different types of input speech signals. As illustrated, each two-dimensional acoustic modulation frequency shows the energy present in the input signal as a function of the acoustic frequency (as indicated on the Y-axis of each spectrum shown in Figs. 7A, 7B, 7C and 7D) and the modulation frequency (as indicated on the X-axis of each spectrum shown in Figs. 7A, 7B, 7C and 7D).

[0108] As shown in Fig. 7A, “clean” speech, which has little or no reverberation, may have a two-dimensional spectrum of acoustic modulation frequencies in which most of the energy is concentrated at relatively low modulation frequencies (e.g., less than 5 Hz, less than 10 Hz, etc.).

[0109] As shown in Fig. 7B, an input signal that includes both clean speech and early and late reflections with reverberation may have a two-dimensional spectrum of acoustic modulation frequencies in which the energy is spread across all modulation frequencies.

[0110] As shown in Fig. 7C, an input signal that includes both clean speech and early reflections with reverberation may have a two-dimensional acoustic modulation frequency spectrum in which the energy is generally concentrated at relatively low modulation frequencies (e.g., less than 5 Hz, less than 10 Hz). In other words, the two-dimensional acoustic modulation frequency spectrum for an input signal that includes clean speech and early reflections with reverberation (but without late reflections with reverberation) may be substantially similar to the two-dimensional acoustic modulation frequency spectrum of only clean speech.

[0111] As shown in Fig. 7D, an input signal that includes late reverberant reflections without clean speech or early reverberant reflections may have a two-dimensional spectrum of acoustic modulation frequencies in which the energy is spread across all modulation frequencies.

[0112] Accordingly, as illustrated in Figs. 7A, 7B, 7C and 7D, the diffuseness estimate may be calculated based on the ratio between the amount of energy at relatively high modulation frequencies and the total energy, or based on the relative ratio between the energy at relatively high modulation frequencies and the energy at relatively low modulation frequencies.

[0113] Fig. 8 is a block diagram that shows examples of components of a device capable of implementing various aspects of the present invention. Like other drawings provided herein, the types and numbers of elements shown in Fig. 8 are given by way of example only. Other implementations may include a greater number, a lesser number, and/or different types and numbers of elements. According to some examples, device 800 may be configured to implement at least some of the methods disclosed herein. In some implementations, device 800 may be or include a television receiver, one or more components of an audio system, a mobile device (such as a cellular phone), a laptop computer, a tablet device, a smart speaker, or another type of device.

[0114] According to some alternative implementations, the device 800 may be or include a server. In some such examples, the device 800 may be or include an encoder. Accordingly, in some cases, the device 800 may be a device that is configured to be used in an audio environment, such as a home audio environment, while in other cases, the device 800 may be a device that is configured to be used in the "cloud," such as on a server.

[0115] In this example, device 800 includes an interface system 805 and a control system 810. Interface system 805, in some implementations, may be configured to communicate with one or more other devices of the audio environment. The audio environment may, in some examples, be a home audio environment. In other examples, the audio environment may be another type of environment, for example, an office environment, an automobile environment, an environment on a train, an environment on a street or on a sidewalk, an environment in a park, etc. Interface system 805, in some implementations, may be configured to exchange control information and associated data with audio devices of the audio environment. The control information and associated data, in some examples, may be associated with one or more software applications that device 800 executes.

[0116] Interface system 805, in some implementations, may be configured to receive or provide a content stream. The content stream may include audio data. The audio data may include, but is not limited to, audio signals. In some cases, the audio data may include spatial data, such as channel data and/or spatial metadata. In some examples, the content stream may include video data and audio data corresponding to the video data.

[0117] The interface system 805 may include one or more network interfaces and/or one or more external device interfaces (for example, one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 805 may include one or more wireless interfaces. The interface system 805 may include one or more devices for implementing a user interface, for example, one or more microphones, one or more speakers, a display system, a touch sensor system, and/or a gesture sensor system. In some examples, the interface system 805 may include one or more interfaces between the control system 810 and a memory system, such as the optional memory system 815 shown in Fig. 8. However, in some cases, the control system 810 may include a memory system. The interface system 805, in some implementations, may be configured to receive input from one or more microphones in the environment.

[0118] The control system 810, for example, may include a single- or multi-chip general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete logic element or transistor logic, and/or discrete hardware components.

[0119] In some implementations, the control system 810 may reside in more than one device. For example, in some implementations, a portion of the control system 810 may reside in a device in one of the environments illustrated herein, and another portion of the control system 810 may reside in a device that is outside the environment, such as a server, a mobile device (e.g., a smartphone or tablet computer), etc. In other examples, a portion of the control system 810 may reside in a device in one environment, and another portion of the control system 810 may reside in one or more other devices in the environment. For example, the functionality of the control system may be distributed across multiple smart audio devices in the environment, or may be shared through an orchestration device (e.g., which may be referred to herein as a "smart home hub") and one or more other devices in the environment. In other examples, a portion of the control system 810 may reside in a device that implements a cloud service, such as a server, and another portion of the control system 810 may reside in another device that implements a cloud service, such as another server, a storage device, etc. The interface system 805, in some examples, may also reside in more than one device.

[0120] In some implementations, the control system 810 may be configured to implement at least some of the methods disclosed herein. According to some examples, the control system 810 may be configured to implement de-riverberation methods based on a classification of media types.

[0121] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more persistent media. Such persistent media may include storage devices, such as the storage devices described herein, including, but not limited to, random access memory (RAM), read-only memory (ROM), etc. The one or more persistent media, for example, may reside in the optional memory system 815 shown in Fig. 8 and/or in the control system 810. Accordingly, various innovative aspects of the subject matter described herein may be implemented as one or more persistent media having stored software. The software may, for example, include instructions for controlling at least one device to classify the media type of audio content, determine the degree of reverberation, determine whether dereverberation should be performed, perform dereverberation for the audio signal, etc. The software may, for example, be executed by one or more components of a control system, such as the control system 810 of Fig. 8.

[0122] In some examples, device 800 may include an optional microphone system 820, shown in Fig. 8. The optional microphone system 820 may include one or more microphones. In some implementations, the one or more microphones may be part of or associated with another device, such as a speaker system speaker, a smart audio device, etc. In some examples, device 800 may not include microphone system 820. However, in some such implementations, device 800 may nonetheless be configured to receive microphone data for one or more microphones in the audio environment via interface system 810. In some such implementations, a cloud implementation of device 800 may be configured to receive microphone data or a noise metric at least partially corresponding to the microphone data from one or more microphones in the audio environment via interface system 810.

[0123] According to some implementations, the device 800 may include an optional speaker system 825, as shown in Fig. 8. The optional speaker system 825 may include one or more speakers, which may also be referred to herein as "speakers" or, in general, as "audio playback transducers." In some examples (e.g., in cloud implementations), the device 800 may not include the speaker system 825. In some implementations, the device 800 may include headphones. The headphones may be connected or linked to the device 800 via a headphone jack or via a wireless connection (e.g., Bluetooth).

[0124] In some implementations, device 800 may include an optional sensor system 830, shown in Fig. 8. The optional sensor system 830 may include one or more touch sensors, gesture sensors, motion detectors, etc. According to some implementations, the optional sensor system 830 may include one or more cameras. In some implementations, the cameras may be stand-alone cameras. In some examples, one or more cameras of the optional sensor system 830 may reside in an audio device, which may be a single-purpose audio device or a virtual assistant. In some such examples, one or more cameras of the optional sensor system 830 may reside in a television receiver, a mobile phone, or a smart speaker. In some examples, device 800 may not include the sensor system 830. However, in some such implementations, the device 800 may still be configured to receive sensor data for one or more sensors in the audio environment via the interface system 810.

[0125] In some implementations, device 800 may include an optional display system 835, shown in Fig. 8. Optional display system 835 may include one or more displays, such as one or more light-emitting diode (LED) displays. In some cases, optional display system 835 may include one or more organic light-emitting diode (OLED) displays. In some examples, optional display system 835 may include one or more television receiver displays. In other examples, optional display system 835 may include a laptop display, a mobile device display, or another type of display. In some examples in which device 800 includes display system 835, sensor system 830 may include a touch sensor system and/or a gesture sensor system close to one or more displays of display system 835. According to some such implementations, the control system 810 may be configured to control the display system 835 so as to present one or more graphical user interfaces (GUIs).

[0126] According to some such examples, device 800 may be or include an intelligent audio device. In some such implementations, device 800 may be or include a wake word detector. For example, device 800 may be or include a virtual assistant.

[0127] Some aspects of the present invention include a system or device configured (e.g., programmed) to perform one or more examples of the disclosed methods, and a tangible computer-readable medium (e.g., a disk) that stores code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, digital signal processor, or microprocessor programmed by software or firmware and/or otherwise configured to perform any of a variety of operations on data, including an embodiment of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system including an input device, a memory device, and a processing subsystem that is programmed (and/or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data input thereto.

[0128] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) that is configured (e.g., programmed or otherwise configured) to perform the desired processing for the audio signal(s), which includes performing one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed systems (or elements thereof) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor that may include an input device and a memory device), which is programmed via software or firmware and/or otherwise configured to perform any of a variety of operations, including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the system according to the invention are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed methods, and the system also includes other elements (e.g., one or more speakers and/or one or more microphones). A general-purpose processor configured to perform one or more examples of the disclosed methods may be connected to an input device (e.g., a mouse and/or keyboard), a memory device, and a display device.

[0129] Another aspect of the present invention is a computer-readable medium (e.g., a disk or other tangible storage medium) that stores code for performing (e.g., code executable with the ability to execute) one or more examples of the disclosed methods or steps thereof.

[0130] While particular embodiments of the present invention and applications of the invention have been described herein, it will be apparent to those skilled in the art that many changes to the embodiments and applications described herein are possible without departing from the scope of the invention described and claimed herein. It should be understood that while particular forms of the invention have been shown and described, the invention should not be limited to the particular embodiments described and shown or the particular methods described.

[0131] Various aspects of the present invention can be determined from the following numbered exemplary embodiments (EEE):

EEE1. A method for suppressing reverberation, comprising:

- receiving input audio signal;

- classifying the media type of the input audio signal as one of a group comprising at least: 1) speech; 2) music; or 3) speech over music;

- determining whether to perform dereverberation on the input audio signal based at least on the determination that the media type of the input audio signal is classified as speech; and

- in response to determining that dereverberation is to be performed on the input audio signal, generating an output audio signal by performing dereverberation on the input audio signal.

EEE2. The method of EEE 1, further comprising determining a degree of reverberation in the input audio signal, wherein the determination of whether to perform reverberation on the input audio signal is based on the degree of reverberation.

EEE3. The method according to EEE 2, in which the degree of reverberation is based on the reverberation time (RT60), the direct to reverberant loudness ratio (DRR), a diffuseness rating, or any combination thereof.

EEE4. The method according to EEE 3, wherein determining the degree of reverberation comprises:

- calculation of a two-dimensional spectrum of acoustic modulation frequencies of the input audio signal, where the degree of reverberation is based on the amount of energy in the high-frequency part of the modulation from the two-dimensional spectrum of acoustic modulation frequencies.

EEE5. The method according to EEE 4, in which determining the degree of reverberation comprises calculating at least one of: 1) the ratio of the energy in the high-frequency portion of the modulation from the two-dimensional spectrum of acoustic modulation frequencies to the energy across all modulation frequencies in the two-dimensional spectrum of acoustic modulation frequencies; or 2) the ratio of the energy in the high-frequency portion of the modulation from the two-dimensional spectrum of acoustic modulation frequencies to the energy in the low-frequency portion of the modulation from the two-dimensional spectrum of acoustic modulation frequencies.

EEE6. The method according to EEE 4 or 5, wherein the determination of whether to perform reverberation on the input audio signal is based on a determination that the degree of reverberation exceeds a threshold value.

EEE17. The method according to any one of EEE 1-6, wherein classifying the media type of the input audio signal comprises separating the input audio signal into two or more spatial components.

EEE8. The method according to EEE 7, wherein the two or more spatial components comprise a central channel and a side channel.

EEE9. The method according to EEE 8, further comprising:

- calculation of the side channel power; and

- side channel classification in response to a determination that the side channel power exceeds a threshold.

EEE10. The method of EEE 7, wherein the two or more spatial components comprise a diffuse component and a direct component.

EEE11. The method according to any one of EEE 7-10, wherein classifying the media type of the input audio signal comprises classifying each of two or more spatial components as one of: 1) speech; 2) music; or 3) speech over music, wherein the media type of the input audio signal is classified by combining the classifications of each of the two or more spatial components.

EEE12. The method of any one of EEE 7-11, wherein the input audio signal is separated into two or more spatial components in response to determining that the input audio signal comprises a stereophonic audio signal.

EEE13. The method according to any one of EEE 1-6, wherein classifying the media type of the input audio signal comprises separating the input audio signal into a vocal component and a non-vocal component.

EEE14. The method of EEE 13, wherein the input audio signal is separated into a vocal component and a non-vocal component in response to a determination that the input audio signal comprises a single audio channel.

EEE15. The method according to EEE 13 or 14, wherein the classification of the media type of the input audio signal comprises:

- classification of the vocal component as one of: 1) speech; or 2) non-speech;

- classification of the non-vocal component as one of: 1) music; or 2) not music,

- in this case, the media type of the input audio signal is classified by combining the classification of the vocal component and the classification of the non-vocal component.

EEE16. The method according to any one of EEE 1-15, wherein the determination of whether to perform dereverberation on the input audio signal is based on the classification of a second input audio signal that precedes the input audio signal.

EEE17. The method according to any one of EEE 1-16, further comprising:

- receiving the third input audio signal;

- specifying that dereverberation should not be performed on the third input audio signal; and

- in response to determining that dereverberation should not be performed for the third input audio signal, prohibiting the execution of the dereverberation algorithm for the third input audio signal.

EEE18. The method of EEE 17, wherein the determination that dereverberation is not to be performed for the third input audio signal is based at least in part on a classification of the media type of the third input audio signal.

EEE19. The method of EEE 18, wherein the media type classification of the third input audio signal is one of: 1) music; or 2) speech over music.

EEE20. The method of any one of EEE 17-19, wherein the determination that reverberation is not to be performed for the third input audio signal is based at least in part on the determination that the amount of reverberation in the third input audio signal is below a threshold.

EEE21. A device capable of implementing the method according to any of EEE 1-20.

EEE22. A system capable of implementing the method of any of EEE 1-20.

EEE23. One or more non-volatile media having stored software, wherein the software includes instructions for controlling one or more devices to perform the method according to any of EEE 1-20.

EEE24. A method for classifying an input audio signal as one of at least two media types, comprising:

- receiving input audio signal;

- splitting the input audio signal into two or more spatial components; and

- classifying each of two or more spatial components as one of at least two media types,

- wherein the media type of the input audio signal is classified by combining the classifications of each of two or more spatial components.

EEE25. The method of EEE 24, wherein the two or more spatial components comprise a central channel and a side channel, and the method further comprises:

- calculation of the side channel power; and

- side channel classification in response to a determination that the side channel power exceeds a threshold.

EEE26. The method of EEE 24, wherein the two or more spatial components comprise a diffuse component and a direct component.

EEE29. The method of any one of EEE 24-26, wherein the input audio signal is separated into two or more spatial components in response to determining that the input audio signal comprises a stereophonic audio signal.

EEE28. The method according to any one of EEE 24-26, wherein classifying the media type of the input audio signal comprises separating the input audio signal into a vocal component and a non-vocal component.

EEE29. The method of EEE 28, wherein the input audio signal is separated into a vocal component and a non-vocal component in response to a determination that the input audio signal comprises a single audio channel.

EEE30. The method according to EEE 28 or 29, wherein the classification of the media type of the input audio signal comprises:

- classification of the vocal component as one of: 1) speech; or 2) non-speech;

- classification of the non-vocal component as one of: 1) music; or 2) not music,

- in this case, the media type of the input audio signal is classified by combining the classification of the vocal component and the classification of the non-vocal component.

EEE31. A system capable of implementing the method according to any of EEE 24-30.

EEE32. One or more non-volatile media having stored software, wherein the software includes instructions for controlling one or more devices so as to perform the method according to any of EEE 24-30.

Claims (30)

1. A method for suppressing reverberation, comprising the steps of: - receive (step 302) an input audio signal; - classifying (step 304) the media type of the input audio signal as one of a group comprising at least: 1) speech; 2) music; or 3) speech over music; - determine (step 308) the degree of reverberation in the input audio signal; wherein determining the degree of reverberation comprises a step in which a two-dimensional spectrum of acoustic modulation frequencies of the input audio signal is calculated (step 604), wherein the degree of reverberation is based on the amount of energy in the high-frequency portion of the modulation from the two-dimensional spectrum of acoustic modulation frequencies; - determining (step 310) whether to perform reverberation on the input audio signal, at least based on the determination that the media type of the input audio signal is classified as speech, and based on said degree of reverberation; and - in response to determining that dereverberation is to be performed for the input audio signal, an output audio signal is generated (step 312) by performing dereverberation for the input audio signal. 2. The method of claim 1, wherein the degree of reverberation is based on a reverberation time (RT60), a direct to reverberant loudness ratio (DRR), a diffuseness rating, or any combination thereof. 3. The method according to paragraph 2, in which determining the degree of reverberation comprises a step in which at least one of: 1) the ratio of the energy in the high-frequency portion of the modulation from the two-dimensional spectrum of acoustic modulation frequencies to the energy across all modulation frequencies in the two-dimensional spectrum of acoustic modulation frequencies; or 2) the ratio of the energy in the high-frequency portion of the modulation from the two-dimensional spectrum of acoustic modulation frequencies to the energy in the low-frequency portion of the modulation from the two-dimensional spectrum of acoustic modulation frequencies. 4. The method according to claim 2 or 3, wherein the determination of whether to perform reverberation on the input audio signal is based on the determination that the degree of reverberation exceeds a threshold value. 5. The method according to any one of claims 1 to 4, wherein classifying the media type of the input audio signal comprises the step of separating the input audio signal into two or more spatial components, and, if necessary, wherein the input audio signal is separated into two or more spatial components in response to determining that the input audio signal comprises a stereophonic audio signal. 6. The method according to claim 5, wherein the two or more spatial components comprise a central channel and a side channel, and, if necessary, the method further comprises the steps of: - calculate the side channel power; and - classifying (step 410) the side channel in response to determining that the power of the side channel exceeds a threshold value. 7. The method of claim 5, wherein the two or more spatial components comprise a diffuse component and a direct component. 8. The method according to any one of claims 5-7, in which classifying the media type of the input audio signal comprises the step of classifying (steps 410, 412) each of two or more spatial components as one of: 1) speech; 2) music, or 3) speech over music, wherein the media type of the input audio signal is classified by combining (step 414) the classifications of each of the two or more spatial components. 9. The method according to any one of claims 1-4, in which classifying the media type of the input audio signal comprises the step of separating (step 504) the input audio signal into a vocal component and a non-vocal component, and, if necessary, wherein the input audio signal is separated into the vocal component and the non-vocal component in response to determining that the input audio signal contains one audio channel. 10. The method according to claim 9, wherein classifying the media type of the input audio signal comprises the steps of: - classify (step 506) the vocal component as one of: 1) speech or 2) non-speech; - classify (step 508) the non-vocal component as one of: 1) music or 2) not music, - wherein the media type of the input audio signal is classified by combining (step 510) the classification of the vocal component and the classification of the non-vocal component. 11. The method according to any one of claims 1-10, wherein the determination of whether to perform dereverberation on the input audio signal is based on the classification of a second input audio signal that precedes the input audio signal. 12. The method according to any one of paragraphs 1-11, further comprising the steps of: - receive a third input audio signal; - specify that dereverberation should not be performed for the third input audio signal; and - in response to determining that dereverberation should not be performed for the third input audio signal, prohibiting the execution of the dereverberation algorithm for the third input audio signal, and, if necessary, wherein the determination that reverberation is not to be performed for the third input audio signal is based at least in part on: (a) a media type classification of the third input audio signal or (b) a determination that the amount of reverberation in the third input audio signal is below a threshold, wherein the media type classification of the third input audio signal is one of: 1) music or 2) speech over music. 13. A device for suppressing reverberation, configured to implement the method according to any of paragraphs 1-12. 14. A non-transitory medium on which software is stored, wherein the software includes instructions for controlling one or more devices to perform the method according to any one of paragraphs 1-12.