HK1222470B - Hybrid waveform-coded and parametric-coded speech enhancement - Google Patents

Description

Hybrid Waveform Coding and Parametric Coding for Speech Enhancement

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent Application No. 61/870,933, filed on August 28, 2013, U.S. Provisional Patent Application No. 61/895,959, filed on October 25, 2013, and U.S. Provisional Patent Application No. 61/908,664, filed on November 25, 2013, each of which is incorporated herein by reference in its entirety.

Technical Field

The present invention relates to audio signal processing and, more particularly, to enhancement of the speech content of an audio program relative to other content of the program, wherein the speech enhancement is "hybrid" in the sense that it includes waveform coding enhancement (or relatively more waveform coding enhancement) under some signal conditions and parametric coding enhancement (or relatively more parametric coding enhancement) under other signal conditions. Other aspects are the encoding, decoding, and rendering of audio programs that include data sufficient to enable such hybrid speech enhancement.

Background Art

In movies and television, dialogue and narration are often presented together with other non-speech audio such as music, effects or atmosphere from sporting events. In many cases, speech and non-speech sounds are captured and mixed together respectively under the control of a sound engineer. The sound engineer selects the level of speech relative to the level of non-speech in a manner suitable for most listeners. However, some listeners, such as those with hearing impairment, experience difficulty when understanding the speech content of an audio program (with the speech and non-speech mixing ratio determined by the engineer), and prefer to mix speech with higher relative levels.

A problem to be solved exists in enabling these listeners to increase the audibility of the speech content of an audio program relative to the audibility of the non-speech audio content.

One current approach is to provide the listener with two high-quality audio streams. One stream carries the primary content audio (primarily voice) and the other carries the secondary content audio (the remaining audio program, which excludes voice), and gives the user control over the mixing process. Unfortunately, this approach is impractical because it does not build on the current practice of transmitting fully mixed audio programs. In addition, this approach requires approximately twice the bandwidth of current broadcast practices because two independent audio streams—each at broadcast quality—must be delivered to the user.

Another speech enhancement method (referred to herein as "waveform coding" enhancement) is described in U.S. Patent Application Publication No. 2010/0106507 A1, assigned to Dolby Laboratories, Inc. and designating Hannes Muesch as the inventor, published on April 29, 2010. In waveform coding enhancement, the speech to background (non-speech) ratio of the original audio mix of speech and non-speech content (sometimes referred to as the main mix) is increased by adding a reduced-quality version (low-quality replica) of the clean speech signal that has been sent to the receiver along with the main mix to the main mix. In order to reduce bandwidth overhead, the low-quality replica is usually encoded at a very low bit rate. Due to the low bit rate encoding, encoding artifacts are associated with the low-quality replica, and when the low-quality replica is presented and auditioned alone, the encoding artifacts are clearly audible. Therefore, when auditioned alone, the low-quality replica has an annoying quality. Waveform coding enhancement attempts to hide coding artifacts by adding a low-quality copy to the main mix only during times when the level of non-speech components is high enough that the coding artifacts are masked by the non-speech components. As will be described in detail later, limitations of this approach include the following: the amount of speech enhancement is generally not constant over time, and audio artifacts can become audible when the background (non-speech) components of the main mix are weak or their frequency-amplitude spectra differ significantly from those of the coding noise.

According to waveform coding enhancement, an audio program (for delivery to a decoder for decoding and subsequent presentation) is encoded as a bitstream that includes a low-quality speech copy (or an encoded version thereof) as a sidestream of the main mix. The bitstream may include metadata indicating scaling parameters that determine the amount of waveform coding speech enhancement to be performed (i.e., the scaling parameters determine the scaling factor to be applied to the scaled low-quality speech copy before it is combined with the main mix, or the maximum value of such a scaling factor that will ensure masking of coding artifacts). When the current value of the scaling factor is 0, the decoder does not perform speech enhancement on the corresponding segment of the main mix. Although the current value of the scaling parameter (or the current maximum value that the scaling parameter can reach) is typically determined in the encoder (since the scaling parameter is typically generated by a computationally intensive psychoacoustic model), it can also be generated in the decoder. In the latter case, the metadata indicating the scaling parameter does not need to be sent from the encoder to the decoder, and instead, the decoder can determine the ratio of the power of the mixed speech content to the power of the mix based on the main mix, and implement a model that determines the current value of the scaling parameter in response to the current value of the power ratio.

Another approach for enhancing speech intelligibility in the presence of competing audio (background), referred to herein as "parametric coding" enhancement, is to segment the original audio program (typically an audio track) into time/frequency tiles and enhance the tiles based on the ratio of the power (or level) of their speech content to the power (or level) of the background content to achieve an enhancement of the speech component relative to the background. The basic concept of this approach is similar to that of guided spectral reduction noise suppression. In an extreme example of this approach, in which all tiles with an SNR (i.e., the ratio of the power or level of the speech component to the power or level of the competing sound content) below a predetermined threshold are completely suppressed, this approach has been shown to provide robust speech intelligibility enhancement. When applied to broadcast, the speech-to-background ratio (SNR) can be inferred by comparing the original audio mix (of speech and non-speech content) with the mixed speech component. The inferred SNR can then be converted into an appropriate set of enhancement parameters that are transmitted along with the original audio mix. At the receiver, these parameters can (optionally) be applied to the original audio mix to obtain a signal indicating enhanced speech. As will be described in detail later, parametric coding enhancement works best when the speech signal (the speech component of the mixture) dominates the background signal (the non-speech component of the mixture).

Waveform coding enhancement requires that a low-quality copy of the speech component of the delivered audio program be available at the receiver. To limit the data overhead incurred when sending this copy along with the main audio mix, this copy is encoded at a very low bit rate and exhibits coding distortions. When the level of non-speech components is high, these coding distortions are likely to be masked by the original audio. When the coding distortions are masked, the quality of the resulting enhanced audio is very good.

Parametric coding enhancement is based on parsing the main audio mix signal into time/frequency blocks and applying appropriate gain/attenuation to each of these blocks. When compared with the data rate enhanced by waveform coding, the data rate required for forwarding these gains to the receiver is low. However, due to the limited time-spectral resolution of the parameters, when speech is mixed with non-speech audio, the speech cannot be manipulated and will not affect the non-speech audio. Therefore, parametric coding enhancement of the speech content of the audio mix introduces modulation in the mixed non-speech content, and when the speech enhancement mix is played back, this modulation ("background modulation") can become annoying. When the speech-to-background ratio is very low, background modulation is most likely annoying.

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise stated, it should not be assumed that any approach described in this section is prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise stated, it should not be assumed that a problem identified with respect to one or more approaches has been recognized in any prior art based on this section.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example and not by way of limitation in the figures of the accompanying drawings and in which like reference numerals refer to similar elements, and in which:

1 is a block diagram of a system configured to generate prediction parameters for reconstructing speech content of a single-channel mixed-content signal having speech content and non-speech content.

2 is a block diagram of a system configured to generate prediction parameters for reconstructing speech content of a multi-channel mixed content signal (having speech content and non-speech content).

3 is a block diagram of a system including an encoder configured to perform an embodiment of the encoding method of the present invention to generate an encoded audio bitstream indicative of an audio program, and a decoder configured to decode the encoded audio bitstream and perform speech enhancement according to an embodiment of the method of the present invention.

4 is a block diagram of a system configured to render a multi-channel mixed content audio signal including by performing conventional speech enhancement thereon.

5 is a block diagram of a system configured to render a multi-channel mixed content audio signal including speech enhancement by performing conventional parametric coding thereon.

6 and 6A are block diagrams of systems configured to present a multi-channel mixed content audio signal including an embodiment of the speech enhancement method of the present invention performed thereon.

FIG7 is a block diagram of a system for performing an embodiment of the encoding method of the present invention using an auditory masking model.

8A and 8B illustrate an example process flow, and

FIG9 illustrates an example hardware platform upon which a computer or computing device as described herein may be implemented.

DETAILED DESCRIPTION

Described herein are example embodiments involving hybrid waveform coding and parametric coding speech enhancement. In the following description, for illustrative purposes, numerous specific details are set forth to provide a thorough understanding of the present invention. However, it will be appreciated that the present invention can be practiced without these specific details. In other examples, known structures and devices are not described in detail to avoid unnecessarily enclosing, obscuring, or confusing the present invention.

Example implementations are described herein according to the following overview:

1. General Overview

2. Symbols and terminology

3. Generation of prediction parameters

4. Voice enhancement operation

5. Voice Presentation

6. Middle/Side Representation

7. Example Processing Flow

8. Implementation mechanism - Hardware overview

9. Equivalents, extensions, alternatives and other options

1. General Overview

This overview provides a basic description of some aspects of embodiments of the present invention. It should be noted that this overview is not an extensive or exhaustive summary of various aspects of the embodiments. Furthermore, it should be noted that this overview is not intended to be understood as identifying any particularly notable aspects or elements of the embodiments, nor is it intended to be understood as delineating any scope of the present invention generally, or the embodiments in particular. This overview merely provides some concepts related to the example embodiments in a concise and simplified form, and should be understood as merely a conceptual prelude to the more detailed description of the example embodiments that will be described below. Note that although individual embodiments are discussed herein, any combination of some embodiments and/or embodiments discussed herein may be combined to form additional embodiments.

The inventors have realized that the respective advantages and disadvantages of parametric coding enhancement and waveform coding enhancement can offset each other, and have realized that conventional speech enhancement can be significantly improved by the following hybrid enhancement method, which uses parametric coding enhancement (or a blend of parametric coding enhancement and waveform coding enhancement) under some signal conditions and uses waveform coding enhancement (or a different blend of parametric coding enhancement and waveform coding enhancement) under other signal conditions. Typical embodiments of the hybrid enhancement method of the present invention provide more stable and better quality speech enhancement than can be achieved by either parametric coding enhancement or waveform coding enhancement alone.

In one class of embodiments, the method includes the steps of: (a) receiving a bitstream indicative of an audio program including speech having an unenhanced waveform and other audio content, wherein the bitstream includes audio data indicative of the speech content and the other audio content, waveform data indicative of a reduced-quality version of the speech (wherein the audio data has been generated by mixing the speech data with non-speech data, the waveform data typically including fewer bits than the speech data), wherein the reduced-quality version has a second waveform that is similar (e.g., at least substantially similar) to the unenhanced waveform and would have objectionable quality if listened to alone, and the bitstream includes parametric data, wherein the parametric data, together with the audio data, defines parametrically constructed speech, and the parametrically constructed speech is a parametrically reconstructed version of the speech that at least substantially matches (e.g., is a good approximation of) the speech; and (b) responding Performing speech enhancement on a bitstream in response to a mixing indicator to thereby generate data indicative of a speech enhanced audio program, including by combining the audio data with a combination of low-quality speech data determined from waveform data and reconstructed speech data, wherein the combination is determined by the mixing indicator (e.g., the combination has a sequence of states determined by a current sequence of values of the mixing indicator), generating the reconstructed speech data in response to at least some of the parameter data and at least some of the audio data, the speech enhanced audio program having fewer audible speech enhancement artifacts (e.g., better masked and thus having fewer audible speech enhancement artifacts when the speech enhanced audio program is presented and auditioned) as compared to a purely waveform-encoded speech enhanced audio program determined by combining only the low-quality speech data (which indicates a reduced-quality version of the speech) with the audio data or a purely parametric-encoded speech enhanced audio program determined from the parameter data and the audio data.

As used herein, "speech enhancement artifacts" (or "speech enhancement coding artifacts") refer to distortions (typically measurable distortions) of audio signals (indicative of speech signals and non-speech audio signals) caused by a representation of a speech signal (e.g., parametric data or a waveform-encoded speech signal together with a mixed content signal).

In some embodiments, a blend indicator (which may have a sequence of values, e.g., one for each of a sequence of bitstream segments) is included in the bitstream received in step (a). Some embodiments include generating the blend indicator (e.g., in a receiver that receives and decodes the bitstream) in response to the bitstream received in step (a).

It should be understood that the expression "mix indicator" is not intended to require that the mix indicator be a single parameter or value (or a single sequence of parameters or values) for each segment of the bitstream. Rather, it is contemplated that in some embodiments, the mix indicator (for a segment of the bitstream) may be a set of two or more parameters or values (e.g., for each segment, a parameter coding enhancement control parameter and a waveform coding enhancement control parameter), or a sequence of sets of parameters or values.

In some embodiments, the mixing indicator for each segment may be a sequence of values indicating the mixing per frequency band of the segment.

Waveform data and parameter data need not be provided for (e.g., included with) each segment of the bitstream, and speech enhancement need not be performed on each segment of the bitstream using both waveform data and parameter data. For example, in some cases, at least one segment may include only waveform data (and the combination determined by the blend indicator for each such segment may include only waveform data) and at least one other segment may include only parameter data (and the combination determined by the blend indicator for each such segment may include only reconstructed speech data).

It is generally conceivable that an encoder generates a bitstream by encoding (e.g., compressing) the audio data without applying the same encoding to the waveform data or parameter data. Thus, when the bitstream is delivered to a receiver, the receiver typically parses the bitstream to extract the audio data, waveform data, and parameter data (and the blend indicator, if delivered in the bitstream), but only decodes the audio data. The receiver typically performs speech enhancement on the decoded audio data (using the waveform data and/or parameter data) without applying the same decoding processing to the waveform data or parameter data as was applied to the audio data.

Typically, the combination of waveform data and reconstructed speech data (indicated by the mixing indicator) changes over time, where each combination state is related to the speech content and other audio content of the corresponding segment of the bitstream. The mixing indicator is generated so that the current combination state (of the waveform data and the reconstructed speech data) is at least partially determined by the signal characteristics of the speech content and other audio content in the corresponding segment of the bitstream (e.g., the ratio of the power of the speech content to the power of the other audio content). In some embodiments, the mixing indicator is generated so that the current combination state is determined by the signal characteristics of the speech content and other audio content in the corresponding segment of the bitstream. In some embodiments, the mixing indicator is generated so that the current combination state is determined by both the signal characteristics of the speech content and other audio content in the corresponding segment of the bitstream and the amount of coding artifacts in the waveform data.

Step (b) may include the steps of: performing waveform coded speech enhancement by combining (e.g., mixing or blending) at least some of the low-quality speech data with audio data for at least one segment of the bitstream; and performing parametric coded speech enhancement by combining the reconstructed speech data with the audio data for at least one segment of the bitstream. The combination of waveform coded speech enhancement and parametric coded speech enhancement is performed on at least one segment of the bitstream by blending both the low-quality speech data and the parametrically constructed speech for the segment with the audio data for the segment. Under some signal conditions, only one (but not both) of waveform coded speech enhancement and parametric coded speech enhancement is performed on the segment of the bitstream (or on each of more than one segment) (in response to the blending indicator).

In this document, the expression "SNR" (signal-to-noise ratio) will be used to indicate the power ratio (or level difference) of the speech content of a segment of an audio program (or the entire program) to the non-speech content of the segment or program, or the power ratio (or level difference) of the speech content of a segment of a program (or the entire program) to the entire (speech and non-speech) content of the segment or program.

In one class of embodiments, the present invention method implements "blind" temporal SNR-based switching between parametrically coded enhancement and waveform-coded enhancement of segments of an audio program. In this context, "blind" means that the switching is not perceptually guided by a complex auditory masking model (e.g., of the type to be described herein), but rather by a sequence of SNR values (hybrid indicators) corresponding to the segments of the program. In one embodiment of this class, hybrid coded speech enhancement is implemented by temporally switching between parametrically coded enhancement and waveform-coded enhancement, such that either parametric or waveform-coded enhancement (rather than both) is performed on each segment of the audio program for which speech enhancement is performed. Recognizing that waveform-coded enhancement performs optimally under low SNR conditions (for segments with low SNR values) and that parametric enhancement performs optimally under good SNR (for segments with high SNR values), the switching decision is typically based on the ratio of speech (dialogue) to the remaining audio in the original audio mix.

Implementations of "blind" temporal SNR-based switching typically include the following steps: segmenting the unenhanced audio signal (the original audio mix) into consecutive time slices (segments), and determining for each segment the SNR between the speech content of the segment and other audio content (or between the speech content and the total audio content); and for each segment, comparing the SNR to a threshold, and setting a parametric coding enhancement control parameter for the segment (i.e., the segment's mixing indicator indicates that parametric coding enhancement should be performed) when the SNR is greater than the threshold, or setting a waveform coding enhancement control parameter for the segment (i.e., the segment's mixing indicator indicates that waveform coding enhancement should be performed) when the SNR is not greater than the threshold. Typically, the unenhanced audio signal is delivered (e.g., transmitted) to a receiver along with the control parameters included as metadata, and the receiver performs (for each segment) speech enhancement of the type indicated by the segment's control parameters. Thus, the receiver performs parametric coding enhancement on each segment for which the control parameters are parametric coding enhancement control parameters, and the receiver performs waveform coding enhancement on each segment for which the control parameters are waveform coding enhancement control parameters.

If one is willing to incur the cost of transmitting (for each segment of the original audio mix) both waveform data (for implementing waveform-coded speech enhancement) and parametrically coded enhancement parameters for the original (unenhanced) mix, then a higher degree of speech enhancement can be achieved by applying both waveform-coded and parametric-coded enhancement to individual segments of the mix. Thus, in one class of embodiments, the inventive method implements "blind" temporal SNR-based blending between parametric-coded and waveform-coded enhancement of segments of an audio program. In this context, "blind" also means that the switching is not perceptually guided by a complex auditory masking model (e.g., of the type to be described herein), but rather by a sequence of SNR values corresponding to the segments of the program.

An implementation of "blind" temporal SNR-based mixing generally includes the following steps: segmenting the unenhanced audio signal (the original audio mix) into consecutive time slices (segments); determining, for each segment, the SNR between the speech content and other audio content of the segment (or between the speech content and the total audio content); and setting a mixing control indicator for each segment, wherein the value of the mixing control indicator is determined by (is a function of) the SNR of the segment.

In some embodiments, the method includes the step of determining (e.g., receiving a request for) a total amount of speech enhancement ("T"), the blending control indicator being a parameter α for each segment such that T = αPw + (1-α)Pp, where Pw is the waveform coded enhancement for the segment that would produce a predetermined total enhancement amount T if the waveform coded enhancement for the segment were applied to the unenhanced audio content of the segment using waveform data set for the segment (wherein the speech content of the segment has an unenhanced waveform, the waveform data for the segment indicates a degraded version of the speech content of the segment, the degraded version having a waveform that is similar (e.g., at least substantially similar) to the unenhanced waveform, and the degraded version of the speech content has an objectionable quality when presented and perceived alone), and Pp is the parametric coded enhancement that would produce a predetermined total enhancement amount T if the parametric coded enhancement were applied to the unenhanced audio content of the segment using parameter data set for the segment (wherein the parameter data for the segment, together with the unenhanced audio content of the segment, determine a parametric reconstructed version of the speech content of the segment). In some embodiments, the mixing control indicator for each of the segments is a set of such parameters including parameters for each frequency band of the relevant segment.

When the unenhanced audio signal is delivered (e.g., transmitted) to a receiver along with the control parameters as metadata, the receiver can (for each segment) perform the hybrid speech enhancement indicated by the control parameters for the segment. Alternatively, the receiver generates the control parameters based on the unenhanced audio signal.

In some embodiments, the receiver performs (for each segment of the unenhanced audio signal) a combination of parametric coded enhancement (by an amount determined by enhancement Pp scaled by a parameter α for the segment) and waveform coded enhancement (by an amount determined by enhancement Pw scaled by a value (1-α) for the segment) such that the combination of parametric coded enhancement and waveform coded enhancement produces a predetermined total enhancement amount:

T＝αPw+(1-α)Pp (1)

In another class of embodiments, the combination of waveform coded enhancement and parametric coded enhancement to be performed on each segment of the audio signal is determined by an auditory masking model. In some embodiments of this class, the optimal mix ratio of the mix of waveform coded enhancement and parametric coded enhancement to be performed on the segments of the audio program uses the highest amount of waveform coded enhancement that just prevents the coding noise from becoming audible. It should be understood that the availability of coding noise in the decoder is always in the form of a statistical estimate and cannot be determined precisely.

In some embodiments of this class, the blend indicator for each segment of the audio data indicates a combination of waveform coding enhancement and parametric coding enhancement to be performed on the segment, and the combination is at least substantially equal to a waveform coding maximization combination determined for the segment by an auditory masking model, wherein the waveform coding maximization combination specifies a maximum relative amount of waveform coding enhancement that ensures that coding noise (caused by the waveform coding enhancement) in the corresponding segment of the speech enhancement audio program is not objectionably audible (e.g., inaudible). In some embodiments, the maximum relative amount of waveform coding enhancement that ensures that coding noise in a segment of a speech enhanced audio program does not sound objectionable is a maximum relative amount that ensures that the combination of waveform coding enhancement and parametric coding enhancement to be performed (on the corresponding segment of audio data) generates a predetermined total amount of speech enhancement for the segment, and/or (wherein artifacts of the parametric coding enhancement are included in the evaluation performed by the auditory masking model) that enables coding artifacts (due to waveform coding enhancement) to be audible over artifacts of the parametric coding enhancement (when this is desirable) (e.g., where the audible coding artifacts (due to waveform coding enhancement) are less objectionable than the audible artifacts of the parametric coding enhancement).

The contribution of waveform coding enhancement in the hybrid coding scheme of the present invention can be increased while ensuring that the coding noise does not become objectionably audible (e.g., does not become audible) by using an auditory masking model to more accurately predict how the coding noise in a degraded speech replica (to be used to implement waveform coding enhancement) will be masked by the audio mix of the main program and selecting the mixing ratio accordingly.

Some embodiments using an auditory masking model include the steps of: segmenting an unenhanced audio signal (the original audio mixture) into consecutive time slices (segments); providing a degraded replica of the speech in each segment (for waveform coded enhancement) and parametric coded enhancement parameters for each segment (for parametric coded enhancement); for each segment, using the auditory masking model to determine the maximum amount of waveform coded enhancement that can be applied without coding artifacts becoming unpleasantly audible; and generating an indicator (for each segment of the unenhanced audio signal) of a combination of waveform coded enhancement (in an amount that does not exceed and at least substantially matches the maximum amount of waveform coded enhancement determined using the auditory masking model for the segment) and parametric coded enhancement such that the combination of waveform coded enhancement and parametric coded enhancement generates a predetermined total amount of speech enhancement for the segment.

In some embodiments, each indicator is included (eg, by an encoder) in a bitstream that also includes encoded audio data indicative of the unenhanced audio signal.

In some embodiments, the unenhanced audio signal is segmented into consecutive time slices and each time slice is segmented into frequency bands, and for each frequency band in each time slice, an auditory masking model is used to determine the maximum amount of waveform coding enhancement that can be applied without coding artifacts becoming unpleasantly audible, and an indicator is generated for each frequency band of each time slice of the unenhanced audio signal.

Optionally, the method further comprises the steps of performing (for each segment of the unenhanced audio signal) in response to the indicator for each segment a combination of waveform coded enhancement and parametric coded enhancement as determined by the indicator, such that the combination of waveform coded enhancement and parametric coded enhancement generates a predetermined total amount of speech enhancement for the segment.

In some embodiments, the audio content is encoded in an encoded audio signal of a reference audio channel configuration (or representation), such as a surround sound configuration, a 5.1 speaker configuration, a 7.1 speaker configuration, a 7.2 speaker configuration, or the like. The reference configuration may include audio channels such as stereo channels, left and right front channels, surround channels, speaker channels, object channels, or the like. One or more of the channels carrying the voice content may not be channels of a mid/side (M/S) audio channel representation. As used herein, an M/S audio channel representation (or simply M/S representation) includes at least a mid channel and a side channel. In an example embodiment, the mid channel represents the sum of the left and right channels (e.g., equally weighted, etc.), while the side channel represents the difference between the left and right channels, where the left and right channels can be considered as any combination of two channels, such as a front center channel and a front left channel.

In some embodiments, the speech content of a program may be mixed with non-speech content and may be distributed across two or more non-M/S channels in a reference audio channel configuration, such as left and right channels, left front and right front channels, etc. The speech content may, but is not required to, be represented at the phantom center in stereo content where the speech content is equally loud in both non-M/S channels, such as left and right channels, etc. Stereo content may include non-speech content that is not necessarily equally loud or even present in both channels.

In some methods, multiple sets of non-M/S control data, control parameters, etc. for speech enhancement corresponding to multiple non-M/S audio channels on which speech content is distributed are sent from an audio encoder to a downstream audio decoder as part of overall audio metadata. Each of the multiple sets of non-M/S control data, control parameters, etc. for speech enhancement corresponds to a specific audio channel of the multiple non-M/S audio channels on which speech content is distributed and can be used by a downstream audio decoder to control speech enhancement operations related to the specific audio channel. As used herein, a set of non-M/S control data, control parameters, etc. refers to control data, control parameters, etc. for speech enhancement operations in an audio channel of a reference configuration in which an audio signal as described herein is encoded.

In some embodiments, M/S speech enhancement metadata—in addition to or in lieu of one or more sets of non-M/S control data, control parameters, etc.—is transmitted from an audio encoder to a downstream audio decoder as part of the audio metadata. The M/S speech enhancement metadata may include one or more sets of M/S control data, control parameters, etc. for speech enhancement. As used herein, a set of M/S control data, control parameters, etc. refers to control data, control parameters, etc. used for speech enhancement operations in audio channels of an M/S representation. In some embodiments, the M/S speech enhancement metadata for speech enhancement is transmitted from an audio encoder to a downstream audio decoder along with the mixed content encoded in a reference audio channel configuration. In some embodiments, the number of sets of M/S control data, control parameters, etc. for speech enhancement in the M/S speech enhancement metadata may be less than the number of non-M/S audio channels in the reference audio channel representation over which the speech content in the mixed content is distributed. In some embodiments, even when the speech content in the mixed content is distributed across two or more non-M/S audio channels, such as a left channel and a right channel, in a reference audio channel configuration, only one set of M/S control data, control parameters, and the like for speech enhancement—e.g., corresponding to a center channel of an M/S representation—is sent by the audio encoder to the downstream decoder as M/S speech enhancement metadata. A single set of M/S control data, control parameters, and the like for speech enhancement can be used to implement speech enhancement operations for all of the two or more non-M/S audio channels, such as a left channel and a right channel. In some embodiments, a conversion matrix between the reference configuration and the M/S representation can be used to apply speech enhancement operations based on the M/S control data, control parameters, and the like for speech enhancement as described herein.

The techniques described herein can be used in situations where the speech content is panned at the phantom center of the left and right channels, where the speech content is not fully panned to the center (e.g., the left and right channels are not equally loud), etc. In an example, these techniques can be used in situations where a large percentage (e.g., 70+%, 80+%, 90+%, etc.) of the speech content's energy is in the middle signal or middle channel of the M/S representation. In another example, (e.g., spatial, etc.) transformations such as translations, rotations, etc. can be used to convert unequal speech content in a reference configuration into equivalent or substantially equivalent speech content in an M/S configuration. Rendering vectors, transformation matrices, etc. representing translations, rotations, etc. can be used as part of or in conjunction with speech enhancement operations.

In some embodiments (e.g., hybrid mode, etc.), a version of the speech content (e.g., a degraded version, etc.) is sent to a downstream audio decoder as only the mid channel signal or both the mid channel signal and the side channel signal in an M/S representation, along with the mixed content sent in a reference audio channel configuration that may have a non-M/S representation. In some embodiments, when the version of the speech content is sent to a downstream audio decoder as only the mid channel signal in an M/S representation, corresponding rendering vectors for signal portions in one or more non-M/S channels of a non-M/S audio channel configuration (e.g., a reference configuration, etc.) that are operated on (e.g., performed a conversion, etc.) the mid channel signal to generate, based on the mid channel signal, corresponding rendering vectors are also sent to the downstream audio decoder.

In some implementations, a dialog/speech enhancement algorithm (e.g., in a downstream audio decoder, etc.) that implements "blind" temporal SNR switching between parametric coding enhancement (e.g., independent channel dialog prediction, multi-channel dialog prediction, etc.) and waveform coding enhancement of segments of an audio program operates at least partially in the M/S representation.

The techniques described herein for implementing speech enhancement operations at least partially in the M/S representation can be used for independent channel prediction (e.g., in the mid channel, etc.), multi-channel prediction (e.g., in the mid channel and side channels, etc.), etc. These techniques can also be used to support speech enhancement for one dialog, two, or more dialogs simultaneously. Zero, one, or one or more additional sets of control parameters, control data, and the like, such as prediction parameters, gains, rendering vectors, etc., can be provided in the encoded audio signal as part of the M/S speech enhancement metadata to support additional dialogs.

In some embodiments, the semantics of the encoded audio signal (e.g., output from an encoder, etc.) support the transmission of an M/S flag from an upstream audio encoder to a downstream audio decoder. The M/S flag is present/set when a speech enhancement operation is to be performed, at least in part, using M/S control data, control parameters, etc. transmitted using the M/S flag. For example, when the M/S flag is set, the receiving audio decoder may first convert the stereo signals in non-M/S channels (e.g., from the left and right channels, etc.) into mid and side channels in an M/S representation before applying an M/S speech enhancement operation according to one or more speech enhancement algorithms (e.g., independent channel dialogue prediction, multi-channel dialogue prediction, waveform-based, waveform parameter mixing, etc.) using the M/S control data, control parameters, etc. received using the M/S flag. After performing the M/S speech enhancement operation, the speech-enhanced signals in the M/S representation may be converted back to non-M/S channels.

In some embodiments, the audio program whose speech content is to be enhanced according to the present invention includes speaker channels but does not include any object channels. In other embodiments, the audio program whose speech content is to be enhanced according to the present invention is an object-based audio program (typically a multi-channel object-based audio program) that includes at least one object channel and, optionally, at least one speaker channel.

Another aspect of the present invention is a system comprising: an encoder configured (e.g., programmed) to, in response to audio data indicating a program including speech content and non-speech content, perform any embodiment of the encoding method of the present invention to generate a bitstream comprising encoded audio data, waveform data, and parameter data (and optionally, a mixing indicator (e.g., mixing indicator data) for each segment of the audio data); and a decoder configured to parse the bitstream to recover the encoded audio data (and optionally, each mixing indicator) and decode the encoded audio data to recover the audio data. Alternatively, the decoder is configured to generate a mixing indicator for each segment of the audio data in response to the recovered audio data. The decoder is configured to perform hybrid speech enhancement on the recovered audio data in response to each mixing indicator.

Another aspect of the present invention is a decoder configured to perform any embodiment of the method of the present invention. In another class of embodiments, the present invention is a decoder comprising a buffer memory (buffer) storing (e.g., in a non-transient manner) at least one segment (e.g., frame) of a coded audio bitstream that has been generated by any embodiment of the method of the present invention.

Other aspects of the present invention include systems or devices (e.g., encoders, decoders, or processors) configured (e.g., programmed) to perform any embodiment of the method of the present invention and computer-readable media (e.g., disks) storing code for implementing any embodiment of the method of the present invention or its steps. For example, the system of the present invention may be or include a programmable general-purpose processor, digital signal processor, or microprocessor that is programmed and/or otherwise configured using software or firmware to perform any of a variety of operations on data including embodiments of the method of the present invention or its steps. Such a general-purpose processor may be or include a computer system that includes an input device, memory, and processing circuitry that is programmed (and/or otherwise configured) to perform embodiments of the method of the present invention (or its steps) in response to data asserted to the computer system.

In some embodiments, the mechanisms as described herein form part of a media processing system including, but not limited to, audio and video devices, flat-panel TVs, handheld devices, game consoles, televisions, home theater systems, tablets, mobile devices, laptop computers, notebook computers, cellular wireless telephones, e-book readers, point of sale, desktop computers, computer workstations, computer kiosks, various other types of terminals and media processing units, and the like.

Various modifications to the general principles and features and preferred embodiments described herein will be apparent to those skilled in the art. Therefore, the present disclosure is not intended to be limited to the embodiments shown, but is intended to be consistent with the broadest scope consistent with the principles and features described herein.

2. Symbols and terminology

Throughout this disclosure, including the claims, the terms "dialogue" and "speech" are used interchangeably as synonyms to refer to audio signal content perceived as a form of communication by a human (or a character in a virtual world).

Throughout this disclosure, including the claims, the expression "performing an operation on" a signal or data (e.g., filtering, scaling, converting, or applying a gain to the signal or data) is used broadly to mean performing the operation directly on the signal or data or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

Throughout this disclosure, including the claims, the expression "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and receives another X-M inputs from an external source) may also be referred to as a decoder system.

Throughout this disclosure, including the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., using software or firmware) to perform operations on data (e.g., audio, or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and/or otherwise configured to perform pipeline processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

Throughout this disclosure, including the claims, the expressions "audio processor" and "audio processing unit" are used interchangeably and, in a broad sense, refer to a system configured to process audio data. Examples of audio processing units include, but are not limited to, encoders (e.g., transcoders), decoders, codecs, pre-processing systems, post-processing systems, and bitstream processing systems (sometimes referred to as bitstream processing tools).

Throughout this disclosure, including the claims, the expression "metadata" refers to data that is separate and distinct from corresponding audio data (the audio content of the bitstream that also includes the metadata). Metadata is associated with audio data and represents at least one feature or characteristic of the audio data (e.g., what type of processing has been performed or should be performed on the audio data or the trajectory of an object indicated by the audio data). The association of metadata with the audio data is time-synchronized. Thus, current (most recently received or updated) metadata can indicate that the corresponding audio data simultaneously has the indicated features and/or includes the results of the indicated type of audio data processing.

Throughout this disclosure, including the claims, the terms "couples" or "coupled" are used to mean either a direct or indirect connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

Throughout this disclosure, including the claims, the following expressions have the following definitions:

- Speaker and loudspeaker are used synonymously to mean any transducer that produces sound. This definition includes loudspeakers implemented as multiple transducers (e.g., a woofer and a tweeter);

- Loudspeaker feed: audio signal to be applied directly to a loudspeaker, or to an amplifier and loudspeaker connected in series;

Channel (or "audio channel"): a single-channel audio signal. Typically, such a signal can be presented in such a way that it is equivalent to applying the signal directly to a loudspeaker at a desired or nominal position. The desired position can be static, as is typically the case with physical loudspeakers, or it can be dynamic;

- audio program: a set of one or more audio channels (at least one loudspeaker channel and/or at least one object channel) and further optionally associated metadata (e.g. metadata describing the desired spatial audio representation);

- Loudspeaker channels (or "loudspeaker feed channels"): audio channels associated with named loudspeakers (at desired or nominal positions) or associated with named loudspeaker zones within a defined loudspeaker configuration. Loudspeaker channels are presented in such a way as to be equivalent to applying audio signals directly to the named loudspeakers (at desired or nominal positions) or to the loudspeakers in the named loudspeaker zones;

- Object channel: An audio channel that indicates the sound emitted by an audio source (sometimes referred to as an audio "object"). Typically, an object channel determines a parametric audio source description (e.g., metadata indicating that a parametric audio source description is included in or provided with the object channel). The source description may determine the sound emitted by the source (as a function of time), the apparent location of the source as a function of time (e.g., three-dimensional spatial coordinates), and optionally at least one additional parameter characterizing the source (e.g., apparent source size or width);

- object-based audio program: an audio program comprising a set of one or more object channels (and optionally at least one loudspeaker channel) and optionally associated metadata (e.g., metadata indicating the trajectory of an audio object emitting the sound indicated by the object channel, or otherwise indicating a desired spatial audio representation of the sound indicated by the object channel, or metadata indicating the identity of at least one audio object that is the source of the sound indicated by the object channel); and

Rendering: The process of converting an audio program into one or more speaker feeds, or the process of converting an audio program into one or more speaker feeds and converting the speaker feeds into sound using one or more loudspeakers (in the latter case, rendering is sometimes referred to herein as being rendered "by" the loudspeakers). An audio channel may be rendered ordinarily ("at" a desired location) by applying a signal directly to a physical loudspeaker at the desired location, or one or more audio channels may be rendered using one of a number of virtualization techniques designed to be substantially equivalent (to a listener) to such an ordinary rendering. In the latter case, each audio channel may be converted into one or more speaker feeds to be applied to loudspeakers in known locations, generally different from the desired location, so that the sound emitted by the loudspeakers in response to the feeds will be perceived as emanating from the desired location. Examples of such virtualization techniques include binaural rendering via headphones (e.g., using Dolby Headphone processing to simulate up to 7.1 surround sound channels for a headphone wearer) and wave field synthesis.

3 , 6 and 7 , embodiments of the encoding, decoding and speech enhancement methods of the present invention and a system configured to implement the methods will be described.

3. Generation of prediction parameters

In order to perform speech enhancement (including hybrid speech enhancement according to embodiments of the present invention), access to the speech signal to be enhanced is required. If a speech signal is not available when speech enhancement is to be performed (separate from the mix of speech content and non-speech content of the hybrid signal to be enhanced), parametric techniques can be used to create a reconstruction of the available mixed speech.

A method for parametric reconstruction of speech content of a mixed-content signal (indicative of a mixture of speech content and non-speech content) is based on the speech power in each time-frequency block of the reconstructed signal and generates parameters according to the following formula:

Where pn _,b is a parameter (parameter-encoded speech enhancement value) of the partition, pn _,b has a time index n and a frequency band index b, the value Ds _,f represents the speech signal in time slot s and frequency bin f of the partition, the value Ms _,f represents the mixed content signal in the same time slot and frequency bin of the partition, and the sum is over all values of s and f across all partitions. The parameter pn _,b can be delivered (as metadata) with the mixed content signal itself to enable the receiver to reconstruct the speech content of each segment of the mixed content signal.

As depicted in FIG1 , each parameter p _n,b can be determined by: performing a time domain to frequency domain conversion on a mixed content signal (“mixed audio”) whose speech content is to be enhanced; performing a time domain to frequency domain conversion on a speech signal (the speech content of the mixed content signal); integrating the energy of each time-frequency block (for each time-frequency block having a time index n and a frequency band index b of the speech signal) over all time slots and frequency bins in the block; integrating the energy of the corresponding time-frequency block of the mixed content signal with respect to all time slots and frequency bins in the block; and dividing the result of the first integration by the result of the second integration to generate the parameter p _n,b for the block.

When each time-frequency block of the mixed content signal is multiplied by the parameters pn _,b of the block, the resulting signal has a spectral and temporal envelope similar to the speech content of the mixed content signal.

A typical audio program, such as a stereo or 5.1 channel audio program, includes multiple speaker channels. Typically, each channel (or each of a subset of channels) indicates speech content and non-speech content, and a mixed content signal determines each channel. The described parametric speech reconstruction method can be applied independently to each channel to reconstruct the speech content of all channels. The reconstructed speech signal (one for each of the channels) can be added to the corresponding mixed content channel signal using an appropriate gain for each channel to achieve the desired enhancement of the speech content.

The mixed content signals (channels) of a multi-channel program can be represented as a set of signal vectors, where each vector element is a collection of time-frequency blocks corresponding to a specific set of parameters, i.e., time slots (s) in a frame (n) and all frequency bins (f) in a parameter band (b). An example of such a set of vectors for a three-channel mixed content signal is:

Here, _ci represents channels. This example assumes three channels, but the number of channels can be any number.

Similarly, the speech content of a multi-channel program can be represented as a set of 1×1 matrices (where the speech content includes only one channel) Dn _,b . The multiplication of each matrix element of the mixed content signal with a scalar value produces the product of each sub-element with a scalar value. Therefore, the reconstructed speech value for each block is obtained by calculating the following formula for each n and b:

D _r,n,b =diag(P)·M _n,b (4)

Where P is a matrix whose elements are prediction parameters. The reconstructed speech (of all blocks) can also be expressed as:

D _r =diag(P)·M (5)

The content in the multiple channels of the multi-channel mixed content signal induces inter-channel coherence that can be used to make a better prediction of the speech signal. By using a minimum mean square error (MMSE) predictor (e.g., of conventional type), the channels can be combined with the prediction parameters to reconstruct the speech content using a minimum error according to a mean square error (MSE) criterion. As shown in FIG2 , assuming a three-channel mixed content input signal, such an MMSE predictor (operating in the frequency domain) iteratively generates a set of prediction parameters p _i (where index i is 1, 2, or 3) in response to the mixed content input signal and a single input speech signal indicating the speech content of the mixed content input signal.

The speech value reconstructed from each channel block of the mixed content input signal (each block with the same index n and index b) is a linear combination of the content (M _ci,n,b ) of each channel (i=1, 2, or 3) of the mixed content signal, controlled by the weight parameters of each channel. These weight parameters are the prediction parameters p _i for the blocks with the same index n and b. Therefore, the speech reconstructed from all slices of all channels of the mixed content signal is:

D _r ＝p ₁ ·M _c1 +p ₂ ·M _c2 +p ₃ ·M _c3 (6)

Or in the form of the following signal matrix:

D _r ＝PM (7)

For example, when speech is coherently present in multiple channels of a mixed content signal and background (non-speech) sounds are incoherent between channels, the additive combination of the channels will favor the energy of the speech. Compared to independent reconstruction of the channels, this will result in 3dB better speech separation for two channels. As another example, when speech content is present in one channel and background sounds are coherently present in multiple channels, the subtractive combination of the channels will (partially) eliminate the background sounds while preserving the speech.

In one class of embodiments, the method of the present invention includes the following steps: (a) receiving a bitstream indicative of an audio program including speech having an unenhanced waveform and other audio content, wherein the bitstream includes: unenhanced audio data indicative of the speech content and the other audio content; waveform data indicative of a reduced-quality version of the speech, wherein the reduced-quality version of the speech has a second waveform that is similar (e.g., at least substantially similar) to the unenhanced waveform and would have an objectionable quality if listened to alone; and parameter data, wherein the parameter data, together with the unenhanced audio data, determines parameters for recreating speech, and the parametrically reconstructed speech is a parametrically reconstructed version of the speech that at least substantially matches (e.g., is a good approximation of) the speech; and (b) performing speech blending on the bitstream in response to a blending indicator. The invention relates to a method for enhancing speech sound, thereby generating data indicative of a speech-enhanced audio program, comprising combining unenhanced audio data with a combination of low-quality speech data determined based on waveform data and reconstructed speech data, wherein the combination is determined by a mixing indicator (e.g., the combination has a state sequence determined by a current value sequence of the mixing indicator), the reconstructed speech data being generated in response to at least some of the parametric data and at least some of the unenhanced audio data, the speech-enhanced audio program having less audible speech-enhancement coding artifacts (e.g., better masked speech-enhancement coding artifacts) than a purely waveform-coded speech-enhanced audio program determined by combining only the low-quality speech data with the unenhanced audio data or a purely parametric-coded speech-enhanced audio program determined based on the parametric data and the unenhanced audio data.

In some embodiments, the blend indicator (which may have a sequence of values, e.g., one for each of the sequence of bitstream segments) is included in the bitstream received in step (a). In other embodiments, the blend indicator is generated in response to the bitstream (e.g., in a receiver that receives and decodes the bitstream).

It should be understood that the expression "mixing indicator" is not intended to mean a single parameter or value (or a single sequence of parameters or values) for each segment of the bitstream. Rather, it is contemplated that in some embodiments, the mixing indicator (of a segment of the bitstream) may be a set of two or more parameters or values (e.g., a parameter coding enhancement control parameter and a waveform coding enhancement control parameter for each segment). In some embodiments, the mixing indicator for each segment may be a sequence of values indicating that the frequency bands of each segment are mixed.

Waveform data and parameter data need not be provided for (e.g., included in) every segment of a bitstream, or need not be used to perform speech enhancement on every segment of a bitstream. For example, in some cases, at least one segment may include only waveform data (and the combination determined by the blend indicator for each such segment may include only waveform data) and at least one other segment may include only parameter data (and the combination determined by the blend indicator for each such segment may include only reconstructed speech data).

It is contemplated that in some embodiments, the encoder generates a bitstream including by encoding (e.g., compressing) the unenhanced audio data rather than the waveform data or parameter data. Thus, when the bitstream is delivered to a receiver, the receiver will parse the bitstream to extract the unenhanced audio data, waveform data, and parameter data (and the blend indicator, if delivered in the bitstream), but will decode only the unenhanced audio data. The receiver will perform speech enhancement on the decoded, unenhanced audio data (using the waveform data and/or parameter data) without applying the same decoding processing to the waveform data or parameter data as was applied to the audio data.

Typically, the combination of waveform data and reconstructed speech data (indicated by the mixing indicator) changes over time, with each combination state being related to the speech content and other audio content of the corresponding segment of the bitstream. The mixing indicator is generated such that the current combination state (of the waveform data and the reconstructed speech data) is determined by signal characteristics of the speech content and other audio content (e.g., the ratio of the power of the speech content to the power of the other audio content) in the corresponding segment of the bitstream.

Step (b) may include the steps of: performing waveform coded speech enhancement by combining (e.g., mixing or blending) at least some of the low-quality speech data with unenhanced audio data for at least one segment of the bitstream; and performing parametric coded speech enhancement by combining reconstructed speech data with the unenhanced audio data for at least one segment of the bitstream. The combination of waveform coded speech enhancement and parametric coded speech enhancement is performed on at least one segment of the bitstream by blending both the low-quality speech data and the reconstructed speech data for the segment with the unenhanced audio data for the segment. Under some signal conditions, only one (but not both) of waveform coded speech enhancement and parametric coded speech enhancement is performed on the segment of the bitstream (or on each of more than one segment) (in response to the blending indicator).

4. Voice enhancement operation

As used herein, "SNR" (signal-to-noise ratio) is used to refer to the ratio of the power (or level) of the speech component (i.e., speech content) of a segment of an audio program (or the entire program) to the power (or level) of the non-speech component (i.e., non-speech content) of the segment or program, or to the power (or level) of the entire (speech and non-speech) content of the segment or program. In some embodiments, the SNR is derived from an audio signal (to undergo speech enhancement) and a separate signal indicative of the speech content of the audio signal (e.g., a low-quality copy of the speech content that has been generated for use in waveform coding enhancement). In some embodiments, the SNR is derived from an audio signal (to undergo speech enhancement) and from parametric data (which has been generated for use in parametric coding enhancement of the audio signal).

In one class of embodiments, the present invention implements "blind" temporal SNR-based switching between parametric and waveform-coded enhancement for segments of an audio program. In this context, "blind" means that the switching is not perceptually guided by a complex auditory masking model (e.g., of the type described herein), but rather by a sequence of SNR values corresponding to the segments of the program (a blend indicator). In one embodiment of this class, hybrid coded speech enhancement is implemented by temporally switching between parametric and waveform-coded enhancement (in response to a blend indicator, e.g., generated in encoder subsystem 29 of FIG. 3 , indicating that only parametric or waveform-coded enhancement should be performed on the corresponding audio data), such that each segment of the audio program for which speech enhancement is performed is subjected to either parametric or waveform-coded enhancement (rather than both). Recognizing that waveform-coded enhancement performs best under low SNR conditions (for segments with low SNR values) and that parametric enhancement performs best under good SNR conditions (for segments with high SNR values), the switching decision is typically based on the ratio of speech (dialogue) to the remaining audio in the original audio mix.

An implementation scheme for implementing "blind" temporal SNR-based switching generally includes the following steps: dividing the unenhanced audio signal (original audio mixture) into consecutive time slices (segments), determining for each segment the SNR between the speech content of the segment and other audio content (or between the speech content and the total audio content); and for each segment, comparing the SNR with a threshold and setting a parametric coding enhancement control parameter for the segment when the SNR is greater than the threshold (i.e., the segment's mixing indicator indicates that parametric coding enhancement should be performed), or setting a waveform coding enhancement control parameter for the segment when the SNR is not greater than the threshold (i.e., the segment's mixing indicator indicates that waveform coding enhancement should be performed).

When the unenhanced audio signal is delivered (e.g., transmitted) to a receiver along with the control parameters included as metadata, the receiver may perform (for each segment) the type of speech enhancement indicated by the control parameters for the segment. Thus, the receiver performs parametric coding enhancement for each segment for which the control parameters are parametric coding enhancement control parameters, and performs waveform coding enhancement for each segment for which the control parameters are waveform coding enhancement control parameters.

If one is willing to incur the cost of transmitting (for each segment of the original audio mix) both waveform data (for implementing waveform-coded speech enhancement) and parametrically coded enhancement parameters for the original (unenhanced) mix, then a higher degree of speech enhancement can be achieved by applying both waveform-coded and parametric-coded enhancement to the individual components of the mix. Thus, in one class of embodiments, the present method implements "blind" temporal SNR-based mixing between parametric-coded and waveform-coded enhancements for segments of an audio program. Furthermore, "blind" in this context means that the switching is not perceptually guided by a complex auditory masking model (e.g., of the type to be described herein), but rather by a sequence of SNR values corresponding to the segments of the program.

Implementations of "blind" temporal SNR-based mixing generally include the following steps: segmenting an unenhanced audio signal (the original audio mix) into consecutive time slices (segments), and determining, for each segment, the SNR between the segment's speech content and other audio content (or between the speech content and the total audio content); determining (e.g., receiving a request for) an amount of speech enhancement ("T"); and setting a mixing control parameter for each segment, wherein the value of the mixing control parameter is determined by (is a function of) the segment's SNR.

For example, the mixing indicator for a segment of an audio program may be a mixing indicator parameter (or set of parameters) generated for the segment in subsystem 29 of the encoder of FIG. 3 .

The blending control indicator may be a parameter α for each segment such that T = αPw + (1-α)Pp, where Pw is a waveform coded enhancement of a waveform that would produce a predetermined total enhancement amount T if the waveform coded enhancement of the waveform were applied to the unenhanced audio content of the segment using the waveform data set for the segment (wherein the speech content of the segment has an unenhanced waveform, the waveform data for the segment indicates a reduced-quality version of the speech content of the segment, the reduced-quality version having a waveform that is similar (e.g., at least substantially similar) to the unenhanced waveform, and the reduced-quality version of the speech content has an objectionable quality when presented and perceived alone), and Pp is a parametric coded enhancement that would produce a predetermined total enhancement amount T if the parametric coded enhancement were applied to the unenhanced audio content of the segment using the parameter data set for the segment (wherein the parameter data for the segment together with the unenhanced audio content of the segment determine a parametric reconstructed version of the speech content of the segment).

When the unenhanced audio signal is delivered (e.g., sent) to a receiver along with the control parameters as metadata, the receiver can (for each segment) perform the hybrid speech enhancement indicated by the control parameters for the segment. Alternatively, the receiver generates the control parameters based on the unenhanced audio signal.

In some embodiments, the receiver performs (for each segment of the unenhanced audio signal) a combination of a parameter coded enhancement Pp (scaled by the parameter α of the segment) and a waveform coded enhancement Pw (scaled by the value (1-α) of the segment) such that the combination of the scaled parameter coded enhancement and the scaled waveform coded enhancement produces a predetermined total amount of enhancement as in expression (1) (T=αPw+(1-α)Pp).

An example of the relationship between the SNR of a segment and α is as follows: α is a non-decreasing function of the SNR, and α ranges from 0 to 1, with the value of α being 0 when the SNR of the segment is less than or equal to a threshold ("SNR_poor") and the value of α being 1 when the SNR is greater than or equal to a larger threshold ("SNR_high"). When the SNR is good, α is high, resulting in a large portion of parametric coding enhancement. When the SNR is poor, α is low, resulting in a large portion of waveform coding enhancement. The locations of the saturation points (SNR_poor and SNR_high) should be selected to adjust the specific implementation of both the waveform coding enhancement algorithm and the parametric coding enhancement algorithm.

In another class of embodiments, the combination of waveform coded enhancement and parametric coded enhancement to be performed on each segment of the audio signal is determined by an auditory masking model. In some embodiments of this class, an optimal mix ratio of the mix of waveform coded enhancement and parametric coded enhancement to be performed on the segments of the audio program uses the highest amount of waveform coded enhancement that just causes the coding noise to not become audible.

In the blind SNR-based mixing implementation described above, the mixing ratio of the segment is derived from the SNR, with the SNR being assumed to indicate the ability of the audio mix to mask the coding noise in the reduced-quality version (replica) of the speech to be used for waveform coding enhancement. The advantages of the blind SNR-based approach are its simplicity of implementation and the low computational load at the encoder. However, the SNR is an unreliable predictor of the extent to which the coding noise will be masked and the extent to which a large safety margin must be applied to ensure that the coding noise will always remain masked. This means that at least some of the time the level of the reduced-quality speech replica being mixed is lower than it could be, or if the margin is set more tightly, the coding noise may sometimes become audible. The contribution of waveform coding enhancement in the hybrid coding scheme of the present invention can be increased by ensuring that the coding noise does not become audible by using an auditory masking model that more accurately predicts how the coding noise in the reduced-quality speech replica is masked by the audio mix of the main program and selecting the mixing ratio accordingly.

A specific embodiment using an auditory masking model includes the steps of: segmenting an unenhanced audio signal (the original audio mixture) into consecutive time slices (segments), setting a reduced quality replica of the speech in each segment (for use in waveform coded enhancement) and parametric coded enhancement parameters for each segment (for use in parametric coded enhancement); determining, for each of the segments, using the auditory masking model, a maximum amount of waveform coded enhancement that can be applied without artifacts becoming audible; and generating a mixed indicator (for each segment of the unenhanced audio signal) of a combination of waveform coded enhancement (in an amount that does not exceed the maximum amount of waveform coded enhancement determined for the segment using the auditory masking model and preferably at least substantially matches the maximum amount of waveform coded enhancement determined for the segment using the auditory masking model) and parametric coded enhancement, such that the combination of waveform coded enhancement and parametric coded enhancement produces a predetermined total amount of speech enhancement for the segment.

In some embodiments, each such blend indicator is included (e.g., by an encoder) in a bitstream that also includes encoded audio data indicating the unenhanced audio signal. For example, subsystem 29 of encoder 20 of FIG3 can be configured to generate such a blend indicator, and subsystem 28 of encoder 20 can be configured to include the blend indicator in the bitstream to be output from encoder 20. For another example, the blend indicator can be generated (e.g., in subsystem 13 of encoder 14 of FIG7 ) based on a g _max (t) parameter generated by subsystem 14 of encoder 14 of FIG7 , and subsystem 13 of encoder 13 of FIG7 can be configured to include the blend indicator in the bitstream to be output from encoder 14 of FIG7 (or subsystem 13 can include the g _max (t) parameter generated by subsystem 14 in the bitstream to be output from encoder 14 of FIG7 , and a receiver that receives and parses the bitstream can be configured to generate the blend indicator in response to the g _max (t) parameter).

Optionally, the method further comprises the step of performing, in response to the mixing indicator for each segment (for each segment of the unenhanced audio signal), a combination of waveform coded enhancement and parametric coded enhancement determined by the mixing indicator, such that the combination of waveform coded enhancement and parametric coded enhancement generates a predetermined total amount of speech enhancement for the segment.

An example of an embodiment of the method of the present invention using an auditory masking model will be described with reference to FIG7 . In this example, a mixture A(t) of speech and background audio (unenhanced audio mixture) is determined (in element 10 of FIG7 ) and passed to an auditory masking model (implemented by element 11 of FIG7 ) that predicts a masking threshold θ(f, t) for each segment of the unenhanced audio mixture. The unenhanced audio mixture A(t) is also provided to an encoding element 13 for encoding for transmission.

The masking threshold value that is generated by the model is indicated as the function of the frequency and time auditory stimulation that any signal must exceed to become audible.Such masking models are well known in the art.The speech component s (t) of each segment of unenhanced audio mixture A (t) is encoded (with low bit rate audio encoder 15) to generate a reduced quality replica s ' (t) of the speech content of the segment.The reduced quality replica s ' (t) (compared with the original speech s (t), it comprises less bits) can be conceptualized as the sum of the original speech s (t) and coding noise n (t).Coding noise can be separated from the reduced quality replica for analysis by deducting (in element 16) time alignment speech signal s (t) from the reduced quality replica.Alternatively, coding noise can be directly available from audio encoder.

In element 17, the coding noise n is multiplied by a scaling factor g(t), and the scaled coding noise is passed to an auditory model (implemented by element 18) that predicts the auditory excitation N(f, t) generated by the scaled coding noise. Such excitation models are known in the art. In a final step, the auditory excitation N(f, t) is compared with the predicted masking threshold Θ(f, t), and a maximum scaling factor gmax(t) is found (in element 14) that ensures that the coding noise is masked, i.e., that N(f, t) < Θ( _f , t) for the maximum value of g(t) is ensured. If the auditory model is nonlinear, it may be necessary to iteratively perform the above operation (as shown in FIG2 ) by iterating the value g(t) applied to the coding noise n(t) in element 17; if the auditory model is linear, the above operation can be performed in a simple feedforward step. The resulting scaling factor g _max (t) is the maximum scaling factor that can be applied to the reduced quality speech copy s'(t) before it is added to the corresponding segment of the unenhanced audio mixture A(t) and coding artifacts in the scaled, reduced quality speech copy do not become audible in the mixture of the scaled, reduced quality speech copy g _max (t)*s'(t) and the unenhanced audio mixture A(t).

The FIG7 system further comprises an element 12 configured to generate (in response to the unenhanced audio mixture A(t) and the speech s(t)) parametrically coded enhancement parameters p(t) for performing parametrically coded speech enhancement on each segment of the unenhanced audio mixture.

The parametrically coded enhancement parameters p(t) for each segment of the audio program, as well as the reduced-quality speech replica s'(t) generated in encoder 15 and the factor _gmax (t) generated in element 14 are also set to encoding element 13. Element 13 generates an encoded audio bitstream indicating the unenhanced audio mix A(t), the parametrically coded enhancement parameters p(t), the reduced-quality speech replica s'(t), and the factor _gmax (t) for each segment of the audio program, and the encoded audio bitstream may be transmitted or otherwise delivered to a receiver.

In this example, speech enhancement is performed on each segment of an unenhanced audio mix A(t) (e.g., in a receiver to which the encoded output of element 13 has been delivered) as follows, applying a predetermined (e.g., required) total enhancement amount T using a scaling factor _gmax (t) for the segment. The encoded audio program is decoded to extract the unenhanced audio mix A(t), parametrically coded enhancement parameters p(t), a reduced-quality speech replica s'(t), and the factor _gmax (t) for each segment of the audio program. For each segment, a waveform coded enhancement Pw is determined to be a waveform coded enhancement that would produce a predetermined total enhancement amount T if applied to the unenhanced audio content of the segment using the reduced-quality speech replica s'(t) of the segment. A parametric coded enhancement Pp is determined to be a parametric coded enhancement that would produce a predetermined total enhancement amount T if applied to the unenhanced audio content of the segment using parameter data set for the segment (wherein the parameter data for the segment determines a parametric reconstructed version of the speech content of the segment with respect to the unenhanced audio content of the segment). For each segment, a combination of parametric coding enhancement (by an amount scaled by the segment's parameter _α2 ) and waveform coding enhancement (by an amount determined by the segment's value _α1 ) is performed such that the combination of parametric coding enhancement and waveform coding enhancement generates a predetermined total enhancement amount using the maximum amount of waveform coding enhancement allowed by the following model: T=( _α1 (Pw)+ _α2 (Pp), where the factor _α1 is the maximum value that does not exceed _gmax (t) _for the segment and enables the indicated equation (T=( _α1 (Pw)+ _α2 (Pp)) to be achieved, and the parameter _α2 is the minimum non-negative value that enables the indicated equation (T= _{( α1} (Pw)+ _α2 (Pp)) to be achieved.

In an alternative embodiment, parametric coding enhancement artifacts are included in the evaluation (performed by the auditory masking model) so that coding artifacts (due to waveform coding enhancement) become audible when they outweigh parametric coding enhancement artifacts.

In the variation to Fig. 7 embodiment (and the embodiment that is similar to the embodiment of Fig. 7 using the auditory masking model), be sometimes referred to as the multi-band division embodiment of auditory model guidance, the waveform coding that reduces the quality speech replica enhances the relationship between coding noise N (f, t) and masking threshold value Θ (f, t) and can be inconsistent across all frequency bands.For example, the spectrum characteristic of waveform coding enhancement coding noise can be to make masking noise be about to exceed masking threshold value in the first frequency region, and masking noise is far below masking threshold value in the second frequency region.In Fig. 7 embodiment, the maximum contribution of waveform coding enhancement is determined by the coding noise in the first frequency region, and the maximum scaling factor g that can be applied to the speech replica of reducing quality is determined by the coding noise in the first frequency region and masking characteristic.It is less than the maximum scaling factor g that can be applied only based on the situation of the second frequency region when the determination of maximum scaling factor.If the principle of time mixing is applied respectively in two frequency regions, then overall performance can be improved.

In a kind of embodiment of the multi-band division of auditory model guidance, unenhanced audio signal is divided into M continuous non-overlapping frequency bands and in each in M band, the principle of time mixing is applied independently (that is, the mixed speech enhancement of the mixing that uses waveform coding enhancement and parameter coding enhancement according to an embodiment of the present invention). Alternative realization divides spectrum into the low-frequency band below the cut-off frequency fc and the high-frequency band above the cut-off frequency fc. Always use waveform coding enhancement to enhance the low-frequency band, and always use parameter coding enhancement to enhance the high-frequency band. Cut-off frequency varies with time and always selects the highest possible cut-off frequency under the following constraint: the waveform coding enhancement coding noise at predetermined total speech enhancement amount T place is below the masking threshold. In other words, the maximum cut-off frequency at any moment is:

max(fc|T*N(f＜fc,t)＜Θ(f,t)) (8)

The above embodiment has assumed that the method that can be used to prevent waveform coding enhancement coding artifacts from becoming audible adjusts the mixing ratio (of waveform coding enhancement and parametric coding enhancement) or reduces the total enhancement amount. The alternative method controls the amount of waveform coding enhancement coding noise to generate a reduced quality speech replica through a variable allocation of bit rate. In the example of this alternative embodiment, a constant basic amount of parametric coding enhancement is applied and additional waveform coding enhancement is applied to achieve the desired (predetermined) total amount enhancement. The reduced quality speech replica is encoded using a variable bit stream, and the bit rate is selected as the lowest bit rate that keeps the waveform coding enhancement coding noise below the masking threshold of the parametric coding enhancement main audio.

In some embodiments, the audio program whose speech content is to be enhanced according to the present invention includes speaker channels, but does not include any object channels. In other embodiments, the audio program whose speech content is to be enhanced according to the present invention is an object-based audio program (typically a multi-channel object-based audio program) that includes at least one object channel and, optionally, at least one speaker channel.

Other aspects of the present invention include: an encoder configured to perform any embodiment of the encoding method of the present invention to generate an encoded audio signal in response to an audio input signal (e.g., in response to audio data indicating a multi-channel audio input signal); a decoder configured to decode such an encoded signal and perform speech enhancement on the decoded audio content; and a system including such an encoder and such a decoder. The system of Figure 3 is an example of such a system.

The system of FIG3 includes an encoder 20 configured (e.g., programmed) to perform an embodiment of the encoding method of the present invention to generate an encoded audio signal in response to audio data indicative of an audio program. Typically, the program is a multi-channel audio program. In some embodiments, the multi-channel audio program includes only speaker channels. In other embodiments, the multi-channel audio program is an object-based audio program that includes at least one object channel and, optionally, at least one speaker channel.

The audio data includes data indicating mixed audio content (a mixture of voice content and non-voice content) (identified as "mixed audio" data in Figure 3), and data indicating voice content of the mixed audio content (identified as "voice" data in Figure 3).

The speech data is converted from the time domain to the frequency domain (QMF) in stage 21, and the resulting QMF components are applied to the enhancement parameter generation element 23. The mixed audio data is converted from the time domain to the frequency domain (QMF) in stage 22, and the resulting QMF components are applied to element 23 and to the encoding subsystem 27.

The speech data is also provided to a subsystem 25 configured to generate waveform data indicating a low-quality copy of the speech data (sometimes referred to herein as a "degraded quality" or "low-quality" speech copy) for use in waveform-coded speech enhancement of the mixed (speech and non-speech) content determined by the mixed audio data. The low-quality speech copy comprises fewer bits than the original speech data and has an objectionable quality when presented and perceived alone, and when presented with speech indicating a waveform similar (e.g., at least substantially similar) to the waveform of the speech indicated by the original speech data. Methods for implementing subsystem 25 are known in the art. Examples include code-excited linear prediction (CELP) speech encoders such as AMR and G729.1, which typically operate at low bit rates (e.g., 20 kbps), or modern hybrid encoders such as MPEG Unified Speech and Audio Coding (USAC). Alternatively, a frequency domain encoder may be used, examples of which include Siren (G722.1), MPEG 2 Layer II/III, and MPEG AAC.

Mixed speech enhancement performed according to an exemplary embodiment of the present invention (e.g., in subsystem 43 of decoder 40) includes the following steps: performing (on waveform data) the inverse of the encoding performed (e.g., in subsystem 25 of encoder 20) to generate waveform data to recover a low-quality replica of the speech content of the mixed audio signal to be enhanced. The remaining steps of speech enhancement are then performed using the recovered low-quality replica of the speech (via parameter data and data indicative of the mixed audio signal).

Element 23 is configured to generate parameter data in response to data output from stages 21 and 22. The parameter data, together with the original mixed audio data, determines parametrically constructed speech that is a parametrically reconstructed version of the speech indicated by the original speech data (i.e., the speech content of the mixed audio data). The parametrically reconstructed version of the speech at least substantially matches the speech indicated by the original speech data (e.g., is a good approximation of the speech indicated by the original speech data). The parameter data determines a set of parametrically coded enhancement parameters p(t) for performing parametrically coded speech enhancement on each segment of the unenhanced mixed content determined by the mixed audio data.

The blend indicator generating element 29 is configured to generate a blend indicator ("BI") in response to data output from stages 21 and 22. It is contemplated that the audio program indicated by the bitstream output from the encoder 20 will be subjected to blended speech enhancement (e.g., in the decoder 40) to determine a speech-enhanced audio program, including by combining the unenhanced audio data of the original program with a combination of low-quality speech data (determined from the waveform data) and parametric data. The blend indicator determines such a combination (e.g., the combination having a sequence of states determined by a current sequence of values of the blend indicator) that the speech-enhanced audio program has fewer audible speech-enhancing coding artifacts (e.g., better masked speech-enhancing coding artifacts) than a purely waveform-encoded speech-enhanced audio program determined by combining only low-quality speech data with the unenhanced audio data or a purely parametric-encoded speech-enhanced audio program determined by combining only parametrically constructed speech with the unenhanced audio data.

In a variation of the embodiment of Figure 3, the mixing indicator used by the hybrid speech enhancement of the present invention is not generated in the encoder of the present invention (and is not included in the bitstream output from the encoder), but is instead generated in response to the bitstream output from the encoder (which bitstream includes waveform data and parameter data) (for example, in a variation of receiver 40).

It should be understood that the expression "mixed indicator" is not intended to mean a single parameter or value (or a single sequence of parameters or values) for each segment of the bitstream. Rather, it is contemplated that in some embodiments, the mixed indicator (of a segment of the bitstream) may be a set of two or more parameters or values (e.g., for each segment, a parameter coding enhancement control parameter and a waveform coding enhancement control parameter).

The encoding subsystem 27 generates encoded audio data indicative of the audio content of the mixed audio data (typically, a compressed version of the mixed audio data).The encoding subsystem 27 typically implements the inverse of the conversion performed in stage 22 and other encoding operations.

Formatting stage 28 is configured to assemble the parameter data output from element 23, the waveform data output from element 25, the mix indicator generated in element 29, and the encoded audio data output from subsystem 27 into an encoded bitstream indicative of the audio program. The bitstream (which, in some implementations, may be in E-AC-3 or AC-3 format) includes the unencoded parameter data, the waveform data, and the mix indicator.

The encoded audio bitstream (encoded audio signal) output from encoder 20 is provided to a delivery subsystem 30. Delivery subsystem 30 is configured to store the encoded audio signal generated by encoder 20 (e.g., to store data indicative of the encoded audio signal) and/or transmit the encoded audio signal.

The decoder 40 is coupled and configured (e.g., programmed) to: receive an encoded audio signal from the subsystem 30 (e.g., by reading or retrieving data indicating the encoded audio signal from a storage device in the subsystem 30, or receiving the encoded audio signal that has been sent by the subsystem 30); decode data indicating the mixed (speech and non-speech) audio content of the encoded audio signal; and perform mixed speech enhancement on the decoded mixed audio content. The decoder 40 is typically configured to generate and output a speech-enhanced decoded audio signal (e.g., to a rendering system, not shown in FIG3 ) indicating a speech-enhanced version of the mixed audio content input to the encoder 20. Alternatively, it includes such a rendering system coupled to receive the output of the subsystem 43.

The buffer 44 (buffer memory) of the decoder 40 stores (e.g., in a non-transitory manner) at least one segment (e.g., frame) of the encoded audio signal (bitstream) received by the decoder 40. In typical operation, a sequence of segments of the encoded audio bitstream is provided to the buffer 44 and is set from the buffer 44 to the deformatting stage 41.

The deformatting (parsing) stage 41 of the decoder 40 is configured to parse the encoded bit stream from the delivery subsystem 30 to extract parameter data (generated by element 23 of the encoder 20), waveform data (generated by element 25 of the encoder 20), mixing indicators (generated in element 29 of the encoder 20), and encoded mixed (speech and non-speech) audio data (generated in the encoding subsystem 27 of the encoder 20) from the encoded bit stream.

The encoded mixed audio data is decoded in a decoding subsystem 42 of the decoder 40, and the resulting decoded mixed (speech and non-speech) audio data is supplied to a mixed speech enhancement subsystem 43 (and optionally output from the decoder 40 without undergoing speech enhancement).

In response to control data (including the blend indicator) extracted from the bitstream by stage 41 (or generated in stage 41 in response to metadata included in the bitstream), and in response to parameter data and waveform data extracted by stage 41, a speech enhancement subsystem 43 performs mixed speech enhancement in accordance with an embodiment of the present invention on the decoded mixed (speech and non-speech) audio data from the decoding subsystem 42. The speech-enhanced audio signal output from subsystem 43 indicates a speech-enhanced version of the mixed audio content input to the encoder 20.

In various implementations of the encoder 20 of FIG. 3 , the subsystem 23 may generate any of the described examples of prediction parameters _p for each block of each channel of the mixed audio input signal for use in reconstructing a speech component of the decoded mixed audio signal (e.g., in the decoder 40 ).

Speech enhancement can be performed by mixing the speech signal with the decoded mixed audio signal (e.g., in subsystem 43 of decoder 40 of FIG. 3 ) using a _speech signal indicative of the speech content of the decoded mixed audio signal (e.g., a low-quality replica of the speech generated by subsystem 25 of encoder 20, or a reconstruction of the speech content generated using prediction parameters p i generated by subsystem 23 of encoder 20). The amount of speech enhancement can be controlled by applying a gain to the speech to be added (mixed). For a 6 dB enhancement, a 0 dB gain can be added to the speech (assuming the speech in the speech enhancement mix has the same level as the transmitted or reconstructed speech signal). The speech enhancement signal is:

_Me = M + g· _Dr (9)

In some embodiments, to obtain the speech enhancement gain G, the following mixed gain is applied:

g＝10 ^G/20 -1 (10)

In the case of independent channel speech reconstruction, the speech enhancement mixture _Me is obtained as:

_Me = M·(1+diag(P)·g) (11)

In the above example, the same energy is used to reconstruct the speech contribution in each channel of the mixed audio signal. When speech is already sent as a side signal (e.g., as a low-quality replica of the speech content of the mixed audio signal) or when speech is reconstructed using multiple channels (such as using an MMSE predictor), the speech enhancement mix requires speech presence information to mix speech that has the same distribution across different channels as the speech component already present in the mixed audio signal to be enhanced.

The rendering information can be set by the rendering parameter _ri of each channel. When there are three channels, the rendering information can be expressed as a rendering vector R having the following form.

The speech enhancement mix is:

_Me = M + R·g· _Dr (13)

In the presence of multiple channels, using the prediction parameters _pi to reconstruct the speech (to be mixed with each channel of the mixed audio signal), the previous equation can be written as:

_Me =M+R·g·P·M=(I+R·g·P)·M (14)

Where I is the identity matrix.

5. Voice Presentation

FIG4 is a block diagram of a speech rendering system that implements a conventional speech enhancement mix of the following form:

_Me = M + R·g· _Dr (15)

In Figure 4, the three-channel mixed audio signal to be enhanced is in (or converted into) the frequency domain. The frequency component of the left channel is set to the input of the mixing element 52, the frequency component of the center channel is set to the input of the mixing element 53, and the frequency component of the right channel is set to the input of the mixing element 54.

The speech signal to be mixed with the mixed audio signal (to enhance the mixed audio signal) may have been sent as a side signal (e.g., as a low-quality replica of the speech content of the mixed audio signal) or may have been reconstructed based on prediction parameters p _i sent together with the mixed audio signal. The speech signal is represented by frequency domain data (e.g., comprising frequency components generated by converting a time domain signal into the frequency domain), which are set to the input of the mixing element 51, where they are multiplied by a gain parameter g.

The output of element 51 is set to the rendering subsystem 50. Also set to the rendering subsystem 50 are the CLD (channel level difference) parameters, CLD ₁ and CLD ₂ , which have been sent along with the mixed audio signal. The CLD parameters (for each segment of the mixed audio signal) describe how the speech signal is mixed into the channels of that segment of the mixed audio signal content. CLD ₁ represents the panning coefficients for one pair of speaker channels (e.g., defining the panning of speech between the left and center channels), and CLD ₂ represents the panning coefficients for another pair of speaker channels (e.g., defining the panning of speech between the center and right channels). Therefore, the rendering subsystem 50 sets (to element 52) data indicating the R·g·D _r of the left channel (the speech content, scaled by the gain parameters and rendering parameters of the left channel), and this data is summed with the left channel of the mixed audio signal in element 52. The rendering subsystem 50 sets (to element 53) data indicative of the R·g·D _r of the center channel (the speech content, scaled by the gain parameter and rendering parameters of the center channel), and sums this data with the center channel of the mixed audio signal in element 53. The rendering subsystem 50 sets (to element 54) data indicative of the R·g·D _r of the right channel (the speech content, scaled by the gain parameter and rendering parameters of the right channel), and sums this data with the right channel of the mixed audio signal in element 54.

The outputs of elements 52, 53 and 54 are used to drive a left speaker L, a centre speaker C and a right speaker "Right", respectively.

FIG5 is a block diagram of a speech rendering system that implements a conventional speech enhancement mix of the following form:

_Me =M+R·g·P·M=(I+R·g·P)·M (16)

In Figure 5, the three-channel mixed audio signal to be enhanced is in (or converted into) the frequency domain. The frequency components of the left channel are set to the input of the mixing element 52, the frequency components of the center channel are set to the input of the mixing element 53, and the frequency components of the right channel are set to the input of the mixing element 54.

The speech signal to be mixed with the mixed audio signal is reconstructed (as indicated) based on the prediction parameters p _i sent along with the mixed audio signal. The speech from the first (left) channel of the mixed audio signal is reconstructed using the prediction parameters p ₁ , the speech from the second (center) channel of the mixed audio signal is reconstructed using _the prediction parameters p ₂ , and the speech from the third (right) channel of the mixed audio signal is reconstructed using the prediction parameters p 3. The speech signal is represented by frequency domain data, and these frequency components are set to the input of the mixing element 51, in which these frequency components are multiplied by the gain parameter g.

The output of element 51 is set to the rendering subsystem 55. Also set to the rendering subsystem are the CLD (channel level difference) parameters, CLD ₁ and CLD ₂ , which have been sent along with the mixed audio signal. The CLD parameters (for each segment of the mixed audio signal) describe how the speech signal is mixed into the channels of that segment of the mixed audio signal content. CLD ₁ represents the panning coefficients for one pair of speaker channels (e.g., defining the panning of speech between the left and center channels), and CLD ₂ represents the panning coefficients for another pair of speaker channels (e.g., defining the panning of speech between the center and right channels). Therefore, the rendering subsystem 55 sets (to element 52) data indicating the R·g·P·M of the left channel (the reconstructed speech content mixed with the left channel of the mixed audio content, scaled by the gain parameters and rendering parameters of the left channel, mixed with the left channel of the mixed audio content), and this data is summed with the left channel of the mixed audio signal in element 52. The rendering subsystem 55 sets (to element 53) data indicative of the R·g·P·M of the center channel (the reconstructed speech content mixed with the center channel of the mixed audio content, scaled by the gain parameter and rendering parameters of the center channel), and sums this data with the center channel of the mixed audio signal in element 53. The rendering subsystem 55 sets (to element 54) data indicative of the R·g·P·M of the right channel (the reconstructed speech content mixed with the right channel of the mixed audio content, scaled by the gain parameter and rendering parameters of the right channel), and sums this data with the right channel of the mixed audio signal in element 54.

The outputs of elements 52, 53 and 54 are used to drive a left speaker L, a centre speaker C and a right speaker "Right", respectively.

CLD (channel level difference) parameters are typically sent along with speaker channel signals (e.g., to determine the ratio between the levels at which different channels should be rendered). In some embodiments of the present invention, CLD parameters are used in novel ways (e.g., to pan the enhanced speech between speaker channels of a speech enhancement audio program).

In a typical embodiment, the rendering parameters r _i are (or indicate) the upmix coefficients of the speech, describing how the speech signal is mixed into the channels of the mixed audio signal to be enhanced. These coefficients can be effectively sent to the speech enhancer using the channel level difference parameters (CLD). One CLD represents the panning coefficients of two speakers. For example,

Where _β1 represents the gain of the speaker feed of the first speaker at the instant during the panning, and _β2 represents the gain of the speaker feed of the second speaker at the instant during the panning. When CLD=0, the panning is entirely directed to the first speaker, while when CLD approaches infinity, the panning is entirely towards the second speaker. Using CLD defined in dB range, a limited number of quantization levels can adequately describe the panning.

Using two CLDs, we can define panning between the three loudspeakers. The CLDs can be derived from the rendering coefficients as follows:

where is the normalized presentation coefficient such that

The presentation coefficients can then be reconstructed from the CLD using the following equation:

As noted elsewhere herein, waveform coded speech enhancement uses a low-quality copy of the speech content of the mixed content signal to be enhanced. The low-quality copy is typically encoded at a low bit rate and transmitted as a side signal along with the mixed content signal, and therefore, typically includes significant coding artifacts. Consequently, waveform coded speech enhancement provides good speech enhancement performance in situations with a low SNR (i.e., a low ratio between the speech indicated by the mixed content signal and all other sounds), but typically provides poor performance (i.e., results in undesirable audible coding artifacts) in situations with a high SNR.

In contrast, parametric coding speech enhancement provides good speech enhancement performance when the speech content (of the mixed content signal to be enhanced) is singled out (e.g., it is set to be the content of only the center channel in a multi-channel mixed content signal) or the mixed content signal otherwise has a high SNR.

Therefore, waveform coded speech enhancement and parametric coded speech enhancement have complementary properties. Based on the characteristics of the signal whose speech content is to be enhanced, one class of embodiments of the present invention mixes the two methods to exploit their properties.

FIG6 is a block diagram of a speech rendering system configured to perform hybrid speech enhancement in one embodiment of the present invention. In one implementation, subsystem 43 of decoder 40 of FIG3 implements the system of FIG6 (except for the three speakers shown in FIG6). Hybrid speech enhancement (mixing) can be described by the following equation:

_Me =R·g ₁ ·D _r +(I+R·g ₂ ·P)·M (23)

Where R· _g1 · _Dr is waveform coded speech enhancement of the type implemented by the conventional system of FIG4, R· _g2 ·P·M is parametric coded speech enhancement of the type implemented by the conventional system of FIG5, and the parameters _g1 and _g2 control the overall enhancement gain and the trade-off between the two speech enhancement methods. An example of the definition of the parameters _g1 and _g2 is:

g ₁ = α _c ·(10 ^G/20 -1) (24)

g ₂ =(1-α _c )·(10 ^G/20 -1) (25)

The parameter _αc defines the balance between the parametrically coded speech enhancement method and the waveform-coded speech enhancement method. When the value _αc = 1, only a low-quality copy of the speech is used for waveform-coded speech enhancement. When _αc = 0, the parametric coding enhancement mode makes the full contribution to the enhancement. _αc values between 0 and 1 blend the two methods. In some implementations, _αc is a broadband parameter (applied to all frequency bands of the audio data). The same principle can be applied within each frequency band, so that the blending can be optimized in a frequency-dependent manner using different values of the parameter _αc for each frequency band.

In Figure 6, the three-channel mixed audio signal to be enhanced is in (or converted into) the frequency domain. The frequency components of the left channel are set to the input of the mixing element 65, the frequency components of the center channel are set to the input of the mixing element 66, and the frequency components of the right channel are set to the input of the mixing element 67.

The speech signal to be mixed with the mixed audio signal (to enhance the mixed audio signal) includes: a low-quality replica of the speech content of the mixed audio signal (labeled "Speech" in FIG. 6 ) generated based on waveform data transmitted along with the mixed audio signal (e.g., as a side signal) (according to waveform coded speech enhancement), and a reconstructed speech signal (output from parametrically coded speech reconstruction element 68 of FIG. 6 ) reconstructed based on the mixed audio signal and prediction parameters p _i transmitted along with the mixed audio signal (according to parametrically coded speech enhancement). The speech signal is represented by frequency domain data (e.g., comprising frequency components generated by converting a time domain signal into the frequency domain). The frequency components of the low-quality speech replica are set to the input of mixing element 61, where they are multiplied by a gain parameter g ₂ . The frequency components of the parametrically reconstructed speech signal are set from the output of element 68 to the input of mixing element 62, where they are multiplied by a gain parameter g ₁ . In an alternative embodiment, the mixing performed to achieve speech enhancement is performed in the time domain rather than in the frequency domain as in the embodiment of FIG. 6 .

Summing element 63 sums the outputs of elements 61 and 62 to generate a speech signal to be mixed with the mixed audio signal, and this speech signal is set from the output of element 63 to rendering subsystem 64. Also set to rendering subsystem 64 are CLD (channel level difference) parameters, CLD ₁ and CLD ₂ , which have been sent along with the mixed audio signal. The CLD parameters (for each segment of the mixed audio signal) describe how the speech signal is mixed into the channels of that segment of the mixed audio signal content. CLD ₁ represents the panning coefficient for one pair of speaker channels (e.g., it defines the panning of speech between the left and center channels), and CLD ₂ represents the panning coefficient for another pair of speaker channels (e.g., it defines the panning of speech between the center and right channels). Thus, the rendering subsystem 64 sets (to element 52) data indicative of R· _g1 · _Dr +(R· _g2 ·P)·M for the left channel (the reconstructed speech content mixed with the left channel of the mixed audio content, scaled by the gain parameter and rendering parameters of the left channel, mixed with the left channel of the mixed audio content), and sums this data with the left channel of the mixed audio signal in element 52. The rendering subsystem 64 sets (to element 53) data indicative of R· _g1 · _Dr +(R· _g2 ·P)·M for the center channel (the reconstructed speech content mixed with the center channel of the mixed audio content, scaled by the gain parameter and rendering parameters of the center channel), and sums this data with the center channel of the mixed audio signal in element 53. The rendering subsystem 64 sets (to element 54 ) data indicative of R·g ₁ ·D _r +(R·g ₂ ·P)·M for the right channel (the reconstructed speech content mixed with the right channel of the mixed audio content, scaled by the gain parameter and rendering parameter for the right channel) and sums this data with the right channel of the mixed audio signal in element 54 .

The outputs of elements 52, 53 and 54 are used to drive a left speaker L, a centre speaker C and a right speaker "Right", respectively.

The system of FIG6 can implement temporal SNR-based switching when the parameter _αc is constrained to have a value of _αc =0 or a value of _αc =1. Such an implementation is particularly useful in the following strong bit rate constraint situation: either the low-quality speech replica data can be sent or the parameter data can be sent, but the low-quality speech replica data and the parameter data cannot be sent together. For example, in one such implementation, the low-quality speech replica is sent with the mixed audio signal (e.g., as a side signal) only in the segments where _αc =1, and the prediction parameters p _i are sent with the mixed audio signal (e.g., as a side signal) only in the segments where _αc =0.

The switch (implemented by elements 61 and 62 in this implementation of FIG6 ) determines whether waveform-coded or _parametric -coded enhancement is to be performed on each segment based on the ratio (SNR) between the speech content and all other audio content in the segment (which in turn determines the value of αc). Such an implementation may use a threshold value for the SNR to decide which method to select:

where τ is a threshold value (eg, τ may be equal to 0).

Some implementations of FIG. 6 use hysteresis to prevent rapid alternation between waveform coding enhancement mode and parametric coding enhancement mode when the SNR is around a threshold of several frames.

When the parameter α _c is allowed to have any real value in the range of 0 to 1 (including 0 and 1), the system of Figure 6 can achieve time-SNR-based mixing.

One implementation of the system of FIG6 uses two target values τ ₁ and τ ₂ (of the SNR of a segment of the mixed audio signal to be enhanced), above which one method (either waveform coding enhancement or parametric coding enhancement) is always considered to provide the best performance. Between these targets, interpolation is used to determine the value of the parameter α _c for the segment. For example, linear interpolation can be used to determine the value of the parameter α _c for the segment:

Alternatively, other suitable interpolation schemes may be used.When the SNR is not available, in many implementations the prediction parameters may be used to provide an approximation of the SNR.

In another class of embodiments, the combination of waveform-coded enhancement and parametric-coded enhancement to be performed on each segment of the audio signal is determined by an auditory masking model. In a typical embodiment of this class, the optimal blend ratio of the waveform-coded enhancement and parametric-coded enhancement to be performed on the segments of the audio program uses the highest amount of waveform-coded enhancement that just prevents the coding noise from becoming audible. An example of an embodiment of the method of the present invention using an auditory masking model is described herein with reference to FIG.

More generally, the following considerations relate to the following embodiments: use an auditory masking model to determine the combination (e.g., mixing) of waveform coding enhancement and parametric coding enhancement to be performed on each segment of an audio signal. In such an embodiment, the data indicating the mixture A(t) of the speech and background audio to be referred to as an unenhanced audio mix are set and processed according to an auditory masking model (e.g., a model implemented by element 11 of Fig. 7). The model predicts a masking threshold θ(f, t) for each segment of the unenhanced audio mix. The masking threshold for each time-frequency block of the unenhanced audio mix with time index n and band index b can be represented as θn _{, b} .

The masking threshold Θn _,b indicates how much distortion can be added without being audible for frame n and frequency band b. Let _εD,n,b be the coding error (i.e., quantization noise) of the low-quality speech replica (for waveform coding enhancement), and let εP _,n,b be the parameter prediction error.

Some embodiments in this class implement a hard switch to a method (waveform coded enhancement or parametric coded enhancement) that is best masked by the unenhanced audio mix content:

In many practical situations, the exact parameter prediction errors ε _P,n,b may not be available when generating the speech enhancement parameters, because these may be generated before the mixture of the unenhanced mixture is encoded. In particular, the parametric coding scheme can have a significant impact on the error in the parametric reconstruction of speech from the mixed content channels.

Therefore, some alternative implementations perform a hybrid in parametric coded speech enhancement (with waveform coded enhancement) when the coding artifacts in the low-quality speech replica (to be used for waveform coded enhancement) are not masked by the mixed content:

where _τa is the distortion threshold, beyond which only parametric coding enhancement is applied. When the overall distortion is greater than the overall masking potential, the solution starts a mix of waveform and parametric coding enhancement. In practice, this means that the distortion is already audible. Therefore, a second threshold with a value higher than 0 can be used. Alternatively, one can use a situation where one would rather focus on the unmasked time-frequency blocks rather than the average behavior.

Similarly, when the distortion (coding artifacts) in the low-quality speech copy (to be used for waveform coding enhancement) is too high, this method can be combined with an SNR-guided blending rule. The advantage of this method is that in very low SNR situations, when it produces more audible noise than the distortion of the low-quality speech copy, the parametric coding enhancement mode is not used.

In another embodiment, when a spectral hole is detected in each such time-frequency block, the type of speech enhancement performed on some time-frequency blocks deviates from the type of speech enhancement determined by the above-described example scheme (or similar scheme). For example, spectral holes can be detected by evaluating the energy in the corresponding block during parameter reconstruction, while the energy is zero in the low-quality speech copy (to be used for waveform coding enhancement). If this energy exceeds a threshold, it can be considered as relevant audio. In these cases, the parameter α _c of the block can be set to 0 (or, depending on the SNR, the parameter α _c of the block can be biased towards 0).

In some embodiments, the encoder of the present invention is capable of operating in any selected one of the following modes:

1. Independent Channel Parameters - In this mode, parameter sets are transmitted for each channel, including speech. Using these parameters, a decoder receiving an encoded audio program can perform parametric speech enhancement on the program to increase the speech in those channels by any amount. Example bit rates for transmitting parameter sets are 0.75 kbps to 2.25 kbps.

2. Multi-channel Speech Prediction - In this mode, multiple channels of the mixed content are combined in a linear combination to predict the speech signal. A parameter set is transmitted for each channel. Using these parameters, a decoder receiving the encoded audio program can perform parametric speech enhancement on the program. Additional position data is transmitted along with the encoded audio program to enable the enhanced speech to be rendered back into the mix. Example bit rates for transmitting the parameter sets and position data are 1.5 kbps to 6.75 kbps per dialogue.

3. Waveform Coded Speech - In this mode, a low-quality copy of the speech content of an audio program is transmitted separately and in parallel with the regular audio content (e.g., as a separate bitstream) by any suitable means. A decoder receiving the encoded audio program can perform waveform-coded speech enhancement on the program by mixing in the separate, low-quality copy of the speech content with the main mix. Typically, mixing the low-quality copy of the speech with a gain of 0 dB will enhance the speech by 6 dB, when the amplitude is doubled. In addition, for this mode, position data is transmitted so that the speech signal is correctly distributed among the relevant channels. An example bit rate for transmitting the low-quality copy of the speech and the position data is greater than 20 kbps per conversation.

4. Waveform Parametric Hybrid - In this mode, a low-quality copy of the audio program's speech content (used to perform waveform-coded speech enhancement on the program) and a parameter set for each channel containing speech (used to perform parametric-coded speech enhancement on the program) are both transmitted in parallel with the program's unenhanced mixed (speech and non-speech) audio content. As the bit rate of the low-quality copy of the speech is reduced, more coding artifacts become audible in the signal, and the bandwidth required for transmission is reduced. In addition, a hybrid indicator is transmitted that uses the low-quality copy of the speech and the parameter set to determine the combination of waveform-coded speech enhancement and parametric-coded speech enhancement to be performed on each segment of the program. At the receiver, hybrid speech enhancement is performed on the program, including by performing the combination of waveform-coded speech enhancement and parametric-coded speech enhancement determined by the hybrid indicator, thereby generating data indicating the speech-enhanced audio program. In addition, position data is transmitted along with the program's unenhanced mixed audio content to indicate where the speech signal is to be presented. An advantage of this approach is that the required receiver/decoder complexity can be reduced if the receiver/decoder discards the low-quality copy of the speech and only applies the parameter sets to perform parametric coding enhancement. An example bit rate for transmitting the low-quality copy of the speech, the parameter sets, the mixing indicator, and the position data is 8 to 24 kbps per conversation.

For practical reasons, the speech enhancement gain can be limited to a range of 0 to 12 dB. The encoder can be implemented to be able to further reduce the upper limit of this range by means of bitstream fields. In some embodiments, the syntax of the encoded program (output from the encoder) will support multiple simultaneous enhanceable conversations (in addition to the non-speech content of the program) so that each conversation can be reconstructed and presented separately. In these embodiments, in the latter mode, speech enhancement for simultaneous conversations (from multiple sources at different spatial locations) will be presented at a single location.

In some embodiments where the encoded audio program is an object-based audio program, one or more object clusters (out of a maximum total) can be selected for speech enhancement. CLD value pairs can be included in the encoded program for use by the speech enhancement and rendering system to shift the enhanced speech between the object clusters. Similarly, in some embodiments where the encoded audio program includes speaker channels in a conventional 5.1 format, one or more of the front speaker channels can be selected for speech enhancement.

Another aspect of the present invention is a method for decoding an encoded audio signal that has been generated according to an embodiment of the encoding method of the present invention and performing hybrid speech enhancement (eg, the method performed by the decoder 40 of FIG. 3 ).

The present invention can be implemented in hardware, firmware or software or a combination of the two (e.g., as a programmable logic array). Unless otherwise stated, the algorithm or processing included as part of the present invention is not inherently related to any particular computer or other device. Specifically, various general-purpose machines can be used together with the program written according to the teachings herein, or more conveniently, a more specialized device (e.g., an integrated circuit) that performs the required method steps can be constructed. Therefore, the present invention can be implemented with one or more computer programs executed on one or more programmable computer systems (e.g., a computer system that implements the encoder 20 of Figure 3 or the encoder of Figure 7 or the decoder 40 of Figure 3), each programmable computer system including at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage element), at least one input device or port, and at least one output device or port. Application code is applied to the input data to perform the functions described herein and generate output information. Output information is applied to one or more output devices in a known manner.

Each such program can be implemented in any desired computer language for communicating with a computer system, including machine language, assembly language, or high-level procedural language, logical language, or object-oriented programming language. In any case, the language can be a compiled language or an interpreted language.

For example, when implemented by a sequence of computer software instructions, the various functions and steps of the embodiments of the present invention may be implemented by a multi-threaded software instruction sequence running in appropriate digital signal processing hardware. In this case, the various devices, steps and functions of the embodiments may correspond to a portion of the software instructions.

Preferably, each such computer program is stored on or downloaded to a storage medium or device (e.g., solid-state memory or media, or magnetic or optical media) that can be read by a general or special purpose programmable computer to configure and operate the computer when the storage medium or device is read by a computer system that performs the processes described herein. The present invention system can also be implemented as a computer-readable storage medium configured with (i.e., storing) a computer program, wherein the storage medium so configured causes the computer system to operate in a specific and predefined manner to perform the functions described herein.

A number of embodiments of the present invention have been described. However, it will be appreciated that various modifications may be made without departing from the spirit and scope of the present invention. In light of the above teachings, numerous modifications and variations of the present invention are possible. It will be appreciated that, within the scope of the appended claims, the present invention may be practiced in other ways than as specifically described herein.

6. Middle/Side Representation

The audio decoder may perform speech enhancement operations as described herein based at least in part on the control data, control parameters, etc. in the M/S representation. The upstream audio encoder may generate the control data, control parameters, etc. in the M/S representation, and the audio decoder extracts the control data, control parameters, etc. in the M/S representation from the encoded audio signal generated by the upstream audio encoder.

In a parametric coding enhancement mode where speech content (e.g., one or more dialogues, etc.) is predicted from mixed content, the speech enhancement operation can be generally represented using a single matrix H as shown in the following expression:

Therein, the left hand side (LHS) represents a speech enhanced mixed content signal generated by operating the original mixed content signal on the right hand side (RHS) by the speech enhancement operation as represented by the matrix H.

For purposes of illustration, each of the speech enhancement mixed content signal (e.g., the LHS of Expression (30), etc.) and the original mixed content signal (e.g., the original mixed content signal operated by H in Expression (30), etc.) includes two component signals having speech enhancement mixed content and original mixed content in two channels _c1 and _c2 , respectively. The two channels _c1 and _c2 may be non-M/S audio channels (e.g., left front channel, right front channel, etc.) based on non-M/S representation. It should be noted that, in various embodiments, each of the speech enhancement mixed content signal and the original mixed content signal may also include component signals having non-speech content in channels other than the two non-M/S channels _c1 and _c2 (e.g., surround channels, low frequency effects channels, etc.). It should also be noted that, in various embodiments, each of the speech enhancement mixed content signal and the original mixed content signal may include component signals having speech content in one channel, two channels as shown in Expression (30), or more than two channels. The speech content as described herein may include one dialogue, two dialogues, or more dialogues.

In some embodiments, a speech enhancement operation as represented by H in expression (30) can be used for time slices (segments) of mixed content in which the SNR value between the speech content and other (e.g., non-speech, etc.) content is relatively high (e.g., as guided by an SNR-guided mixing rule, etc.).

As shown in the following expression, the matrix H can be rewritten/expanded to represent the augmentation operation in the M/S representation as a matrix _HMS multiplied on the right by the forward conversion matrix from non-M/S representation to M/S representation and on the left by the inverse of the forward conversion matrix (which includes a factor of 1/2):

Wherein, the example transformation matrix to the right of the matrix _HMS defines the mid-channel mixed content signal in the M/S representation as the sum of the two mixed content signals in the two channels _c1 and _c2 based on the forward transformation matrix, and defines the side channel mixed content signal in the M/S representation as the difference between the two mixed content signals in the two channels _c1 and _c2 . It should be noted that in various embodiments, other transformation matrices other than the example transformation matrix shown in expression (31) can also be used (e.g., assigning different weights to different non-M/S channels, etc.) to transform the mixed content signal from one representation to a different representation. For example, for dialogue enhancement, where the dialogue is not presented at the center of the phantom, but is shifted between two signals with unequal weights _λ1 and _λ2 . As shown in the following expression, the M/S transformation matrix can be modified to minimize the energy of the dialogue component in the side signal:

In an example embodiment, the matrix _HMS representing the augmentation operation in the M/S representation may be defined as a diagonalized (eg, Hermitian matrix, etc.) matrix as shown in the following expression:

Where _p1 and _p2 represent the mid-channel and side-channel prediction parameters, respectively. Each of the prediction parameters _p1 and _p2 can include a set of time-varying prediction parameters for the time-frequency partition of the corresponding mixed content signal in the M/S representation, to be used to reconstruct the speech content from the mixed content signal. For example, as shown in Expression (10), the gain parameter g corresponds to the speech enhancement gain G.

In some embodiments, the speech enhancement operation in the M/S representation is performed in a parametric channel-independent enhancement mode. In some embodiments, the speech enhancement operation in the M/S representation is performed using the predicted speech content in both the mid-channel signal and the side-channel signal, or using the predicted speech content in only the mid-channel signal. For illustrative purposes, the speech enhancement operation in the M/S representation is performed using the mixed content signal in only the mid-channel, as shown in the following expression:

The prediction parameter _p1 comprises a single set of prediction parameters for the time-frequency partition of the mixed content signal in the middle channel of the M/S representation, so as to be used to reconstruct the speech content from the mixed content signal in the middle channel only.

Based on the diagonalized matrix H _MS given in Expression (33), the speech enhancement operation in the parameter enhancement mode as represented by Expression (31) can be further reduced to the following expression, which provides a clear example of the matrix H in Expression (30):

In waveform parameter hybrid enhancement mode, the speech enhancement operation can be expressed in M/S representation using the following example expressions:

Wherein, _m1 and _m2 in the mixed content signal vector M represent the mid-channel mixed content signal (e.g., the sum of the mixed content signals in non-M/S channels such as the left and right front channels) and the side-channel mixed content signal (e.g., the difference between the mixed content signals in non-M/S channels such as the left and right front channels), respectively. Signal dc _,l represents the mid-channel dialogue waveform signal in the M/S-represented dialogue signal vector _Dc (e.g., an encoded waveform representing a degraded version of the dialogue in the mixed content). Matrix _Hd represents the speech enhancement operation in the M/S representation based on the dialogue signal dc _,l in the M/S-represented mid-channel and may include only one matrix element at the first row and first column (1×1). Matrix _Hp represents the speech enhancement operation in the M/S representation based on the dialogue reconstructed using the prediction parameter _p1 of the M/S-represented mid-channel. In some embodiments, for example, as depicted in expressions (23) and (24), gain parameters _g1 and _g2 together (e.g., after being applied to the dialog waveform signal and the reconstructed dialog, etc., respectively) correspond to a speech enhancement gain G. Specifically, parameter _g1 is applied in a waveform-coded speech enhancement operation associated with the dialog signal dc _,l in the middle channel of the M/S representation, while parameter g2 is applied in a parametric-coded speech enhancement operation associated with the mixed content signals _m1 and _m2 in the middle and side channels of the M/ _S representation. Parameters _g1 and _g2 control the overall enhancement gain and the balance between the two speech enhancement methods.

In non-M/S representation, the speech enhancement operation corresponding to the speech enhancement operation expressed using expression (35) can be expressed using the following expression:

Here, the mixed content signals _Mc1 and _Mc2 in the non-M/S channels, which are left-multiplied by the forward conversion matrix between the non-M/S representation and the M/S representation, can be used instead of the mixed content signals _m1 and _m2 in the M/S representation as shown in Expression (35). The inverse conversion matrix (with a factor of 1/2) in Expression (36) converts the speech-enhanced mixed content signal in the M/S representation as shown in Expression (35) back to the speech-enhanced mixed content signal in the non-M/S representation (e.g., the left front channel and the right front channel, etc.).

Additionally, optionally or alternatively, in some embodiments where no further QMF-based processing is performed after the speech enhancement operations, for efficiency reasons, some or all of the speech enhancement operations that combine the speech enhancement content based on the dialogue signal d _c,l with the speech enhancement mixed content based on the dialogue reconstructed by prediction (e.g., as represented by H _d , H _p transform, etc.) may be performed after the QMF synthesis filter bank in the time domain.

Prediction parameters for constructing/predicting speech content based on mixed content signals in one or both of the mid and side channels of the M/S representation may be generated based on one or more prediction parameter generation methods, including but not limited to any of the following methods: the independent channel dialogue prediction method as depicted in FIG1 , the multi-channel dialogue prediction method as depicted in FIG2 , etc. In some embodiments, at least one of the prediction parameter generation methods may be based on MMSE, gradient descent, one or more other optimization methods, etc.

In some embodiments, a "blind" temporal SNR-based switching approach as previously discussed can be used between parametrically coded enhancement data (e.g., relating to speech enhancement content based on a dialogue signal d _c,l , etc.) and waveform coded enhancement (e.g., relating to speech enhancement mixed content based on dialogue reconstructed by prediction, etc.) for a segment of an audio program in an M/S representation.

In some embodiments, the combination of waveform data (e.g., related to speech enhancement content based on the dialogue signal d _c,l, etc.) and reconstructed speech data (e.g., related to speech enhancement mixed content based on dialogue reconstructed by prediction, etc.) in the M/S representation (e.g., indicated by the previously discussed mixing indicator, the combination of g ₁ and g ₂ in expression (35), etc.) changes over time, where each combination state is related to the speech content and other audio content of the corresponding segment of the bitstream carrying the waveform data and the mixed content used in reconstructing the speech data. The mixing indicator is generated so that the current combination state (of the waveform data and the reconstructed speech data) is determined by the signal characteristics of the speech content and other audio content in the corresponding segment of the program (e.g., the ratio of the power of the speech content to the power of the other audio content, the SNR, etc.). The mixing indicator of a segment of the audio program can be a mixing indicator parameter (or parameter set) generated for the segment in the encoder subsystem 29 of Figure 3. An auditory masking model as previously discussed may be used to more accurately predict how the coding noise in the degraded speech replica in the dialogue signal vector Dc is masked by the main program's audio mix and select the mixing ratio accordingly.

Subsystem 28 of encoder 20 of FIG3 can be configured to include a blending indicator related to the M/S speech enhancement operation in the bitstream as part of the M/S speech enhancement metadata to be output from encoder 20. The blending indicator related to the M/S speech enhancement operation can be generated (e.g., in subsystem 13 of encoder 13 of FIG7 ) based on, among other things, a scaling factor g _max (t) related to coding artifacts in dialogue signal D _c . The scaling factor g _max (t) can be generated by subsystem 14 of encoder 14 of FIG7 . Subsystem 13 of encoder 13 of FIG7 can be configured to include the blending indicator in the bitstream to be output from encoder 13 of FIG7 . Additionally, optionally or alternatively, subsystem 13 can include the scaling factor g _max (t) generated by subsystem 14 in the bitstream to be output from encoder 13 of FIG7 .

In some embodiments, the unenhanced audio mix A(t) generated by operation 10 of FIG. 7 represents a mixed content signal vector (e.g., a time segment thereof, etc.) in a reference audio channel configuration. The parametrically coded enhancement parameters p(t) generated by element 12 of FIG. 7 represent at least a portion of the M/S speech enhancement metadata used to perform parametrically coded speech enhancement in the M/S representation with respect to each segment of the mixed content signal vector. In some embodiments, the reduced-quality speech replica s'(t) generated by encoder 15 of FIG. 7 represents a dialog signal vector in the M/S representation (e.g., with respect to a mid-channel dialog signal, a side-channel dialog signal, etc.).

In some embodiments, element 14 of Figure 7 generates the scaling factor _gmax (t) and provides it to encoding element 13. In some embodiments, element 13 generates, for each segment of the audio program, an encoded audio bitstream indicating a mixed content signal vector (e.g., unenhanced, etc.) in a reference audio channel configuration, M/S speech enhancement metadata, a dialog signal vector in M/S representation if applicable, and the scaling factor _gmax (t) if applicable, which can be sent to or otherwise delivered to a receiver.

When the unenhanced audio signal in the non-M/S representation is delivered (e.g., transmitted) to a receiver together with the M/S speech enhancement metadata, the receiver may convert each segment of the unenhanced audio signal in the M/S representation and perform the M/S speech enhancement operation indicated by the M/S speech enhancement metadata on the segment. If a speech enhancement operation is to be performed on the segment in a hybrid speech enhancement mode or in a waveform coding enhancement mode, the unenhanced mixed content signal vector in the non-M/S representation may be provided to the dialogue signal vector in the M/S representation of the segment of the program. If applicable, a receiver that receives and parses the bitstream may be configured to generate a mixing indicator in response to the scaling factor g _max (t) and determine the gain parameters g ₁ and g ₂ in Expression (35).

In some embodiments, the speech enhancement operation is performed at least partially in the M/S representation in a receiver to which the encoded output of element 13 is delivered. In one example, the gain parameters g1 and g2 in expression (35) corresponding to a predetermined (e.g., required) amount of enhancement can be applied to each segment of the unenhanced mixed content signal based at least in part on a blend indicator parsed from a bitstream received by the receiver. In another example, the gain parameters _{g1 and g2} in expression (35) corresponding to a predetermined (e.g., required) amount of enhancement can be applied to each segment of the unenhanced mixed content signal based at least in part on a blend indicator determined from a scaling factor _gmax ( _t ) of the segment parsed from a bitstream received by the receiver.

In some embodiments, element 23 of encoder 20 of FIG3 is configured to generate parameter data including M/S speech enhancement metadata (e.g., prediction parameters for reconstructing dialogue/speech content based on mixed content in the mid channel and/or side channel, etc.) in response to data output from stages 21 and 22. In some embodiments, blend indicator generation element 29 of encoder 20 of FIG3 is configured to generate a blend identifier "BI" that identifies a combination of parametric speech enhancement content (e.g., using gain parameter _g1 , etc.) and waveform-based speech enhancement content (e.g., using gain parameter _g1 , etc.) in response to data output from stages 21 and 22.

In a variation of the embodiment of Figure 3, the mixing indicator for M/S hybrid speech enhancement is not generated in the encoder (and the mixing indicator is not included in the bitstream output from the encoder), but instead is generated (for example, in a variation of the receiver 40) in response to the bitstream output from the encoder (the bitstream including waveform data in the M/S channel and M/S speech enhancement metadata).

The decoder 40 is coupled and configured (e.g., programmed) to: receive an encoded audio signal from the subsystem 30 (e.g., by reading or retrieving data indicating the encoded audio signal from a storage device in the subsystem 30, or receiving the encoded audio signal having been transmitted by the subsystem 30); decode data indicating a mixed (speech and non-speech) content signal vector in a reference audio channel configuration from the encoded audio signal; and perform a speech enhancement operation on the decoded mixed content in the reference audio channel configuration, at least in part, in an M/S representation. The decoder 40 may be configured to generate and output (e.g., to a rendering system, etc.) a speech-enhanced decoded audio signal indicating the speech-enhanced mixed content.

In some embodiments, some or all of the rendering systems depicted in Figures 4 to 6 can be configured to render speech-enhanced mixed content generated by M/S speech enhancement operations, at least some of which are operations performed in an M/S representation. Figure 6A shows an example rendering system configured to perform speech enhancement operations as expressed in Expression (35).

The rendering system of FIG6A can be configured to perform a parametric speech enhancement operation in response to determining that at least one gain parameter used in the parametric speech enhancement operation (e.g., g ₂ in expression (35) etc.) is non-zero (e.g., in hybrid enhancement mode, in parametric enhancement mode, etc.). For example, based on such a determination, the subsystem 68A of FIG6A can be configured to perform a transformation on the mixed content signal vector ("mixed audio (T/F)") distributed on the non-M/S channel to generate a corresponding mixed content signal vector distributed on the M/S channel. If appropriate, the transformation can use a forward transformation matrix. Prediction parameters (e.g., p ₁ , p ₂ , etc.) and gain parameters (e.g., g ₂ in expression (35) etc.) for the parametric enhancement operation can be applied to predict speech content based on the mixed content signal vector of the M/S channel and enhance the predicted speech content.

The rendering system of FIG6A can be configured to perform a waveform coding speech enhancement operation in response to determining that at least one gain parameter used in the waveform coding speech enhancement operation (e.g., _g1 in expression (35), etc.) is non-zero (e.g., in hybrid enhancement mode, in waveform coding enhancement mode, etc.). For example, based on such a determination, the rendering system of FIG6A can be configured to receive/extract a dialogue signal vector distributed on the M/S channel from the received encoded audio signal (e.g., a reduced version of the speech content present in the mixed content signal vector). The gain parameter used for the waveform coding enhancement operation (e.g., _g1 in expression (35), etc.) can be applied to enhance the speech content represented by the dialogue signal vector of the M/S channel. A user-definable enhancement gain (G) can be used to derive gain parameters _g1 and _g2 using mixing parameters that may or may not be present in the bitstream. In some embodiments, the mixing parameters to be used with the user-definable enhancement gain (G) to derive gain parameters _g1 and _g2 can be extracted from metadata in the received encoded audio signal. In some other embodiments, such mixing parameters may not be extracted from metadata in the received encoded audio signal, but may be derived by the recipient encoder based on the audio content in the received encoded audio signal.

In some embodiments, a combination of parametric enhanced speech content and waveform-encoded enhanced speech content in the M/S representation is asserted or input to subsystem 64A of FIG. 6A . Subsystem 64A of FIG. 6 can be configured to perform a transformation on the combination of enhanced speech content distributed across the M/S channels to generate an enhanced speech content signal vector distributed across the non-M/S channels. If appropriate, the transformation can utilize an inverse transformation matrix. The enhanced speech content signal vector for the non-M/S channels can be combined with a mixed content signal vector ("Mixed Audio (T/F)") distributed across the non-M/S channels to generate a speech-enhanced mixed content signal vector.

In some embodiments, the syntax of the encoded audio signal (e.g., output from encoder 20 of FIG. 3 , etc.) supports transmission of an M/S flag from an upstream audio encoder (e.g., encoder 20 of FIG. 3 , etc.) to a downstream audio decoder (e.g., decoder 40 of FIG. 3 , etc.). The M/S flag is presented/set by the audio encoder (e.g., element 23 of encoder 20 of FIG. 3 , etc.) when the receiving audio decoder (e.g., decoder 40 of FIG. 3 , etc.) performs a speech enhancement operation at least in part using M/S control data, control parameters, etc. transmitted along with the M/S flag. For example, when the M/S flag is set, the receiving audio decoder (e.g., decoder 40 of FIG. 3 , etc.) may first convert the stereo signals in the non-M/S channels (e.g., from the left and right channels, etc.) into the mid and side channels of the M/S representation before applying the M/S speech enhancement operation according to one or more of the speech enhancement algorithms (e.g., independent channel dialogue prediction, multi-channel dialogue prediction, waveform-based waveform parameter mixing, etc.) using the M/S control data, control parameters, etc. received along with the M/S flag. In the receiving audio decoder (e.g., decoder 40 of FIG. 3 , etc.), after performing the M/S speech enhancement operation, the speech enhancement signals in the M/S representation may be converted back into the non-M/S channels.

In some embodiments, speech enhancement metadata generated by an audio encoder as described herein (e.g., encoder 20 of FIG. 3 , element 23 of encoder 20 of FIG. 3 , etc.) may carry one or more specific flags indicating the presence of one or more sets of speech enhancement control data, control parameters, etc. for one or more different types of speech enhancement operations. The one or more sets of speech enhancement control data, control parameters, etc. for one or more different types of speech enhancement operations may, but are not limited to, only include sets of M/S control data, control parameters, etc. as M/S speech enhancement metadata. The speech enhancement metadata may also include a preference flag indicating which type of speech enhancement operation (e.g., M/S speech enhancement operation, non-M/S speech enhancement operation, etc.) is preferred for the audio content to be speech enhanced. The speech enhancement metadata may be delivered to a downstream decoder (e.g., decoder 40 of FIG. 3 , etc.) as part of the metadata delivered in an encoded audio signal including mixed audio content encoded for a non-M/S reference audio channel configuration. In some embodiments, only the M/S speech enhancement metadata, and not the non-M/S speech enhancement metadata, is included in the encoded audio signal.

Additionally, optionally or alternatively, the audio decoder (e.g., 40 of FIG. 3 ) may be configured to determine and perform a specific type of speech enhancement operation (e.g., M/S speech enhancement, non-M/S speech enhancement, etc.) based on one or more factors. These factors may include, but are not limited to, only one or more of the following: user input specifying a preference for a specific user-selected type of speech enhancement operation; user input specifying a preference for a system-selected type of speech enhancement operation; the capabilities of a specific audio channel configuration operated by the audio decoder; the availability of speech enhancement metadata for a specific type of speech enhancement operation; any encoder-generated preference flags for a type of speech enhancement operation, etc. In some embodiments, the audio decoder may implement one or more priority rules and, if there is a conflict between these factors, may request further user input, etc., to determine a specific type of speech enhancement operation.

7. Example Processing Flow

8A and 8B illustrate an example process flow. In some implementations, one or more computing devices or units in a media processing system may perform the process flow.

FIG8A shows an example process flow that can be implemented by an audio encoder as described herein (e.g., encoder 20 of FIG3 ). In block 802 of FIG8A , the audio encoder receives mixed audio content having a mixture of speech content and non-speech audio content in a reference audio channel representation, the mixed audio content being distributed across a plurality of audio channels of the reference audio channel representation.

In block 804, the audio encoder converts one or more portions of mixed audio content distributed over one or more non-mid/side (M/S) channels of a plurality of audio channels of the reference audio channel representation into one or more converted mixed audio content portions distributed over one or more M/S channels of the M/S audio channel representation.

In block 806 , the audio encoder determines M/S speech enhancement metadata for one or more transmixed audio content portions in the M/S audio channel representation.

In block 808 , the audio encoder generates an audio signal comprising the mixed audio content in the reference audio channel representation and M/S speech enhancement metadata for one or more converted mixed audio content portions in the M/S audio channel representation.

In an embodiment, the audio encoder is further configured to perform: generating a version of the speech content in the M/S audio channel representation separate from the mixed audio content; and outputting an audio signal encoded using the version of the speech content in the M/S audio channel representation.

In an embodiment, the audio encoder is further configured to perform: generating mixing indication data that enables a receiving audio decoder to apply speech enhancement to the mixed audio content using a specific amount combination of waveform-coded speech enhancement based on a version of the speech content in the M/S audio channel representation and parametric speech enhancement based on a reconstructed version of the speech content in the M/S audio channel representation; and outputting an audio signal encoded using the mixing indication data.

In an embodiment, the audio encoder is further configured to refrain from encoding one or more transmix audio content portions in the M/S audio channel representation as part of the audio signal.

FIG8B shows an example process flow that can be implemented by an audio decoder as described herein (e.g., decoder 40 of FIG3 ). In block 822 of FIG8B , the audio decoder receives an audio signal including mixed audio content in a reference audio channel representation and mid/side (M/S) speech enhancement metadata.

In block 824 of FIG. 8B , the audio decoder converts one or more portions of the mixed audio content distributed over one, two or more non-M/S channels of the plurality of audio channels of the reference audio channel representation into one or more converted mixed audio content portions distributed over one or more M/S channels of the M/S audio channel representation.

In block 826 of FIG. 8B , the audio decoder performs one or more M/S speech enhancement operations on the one or more converted mixed audio content portions in the M/S audio channel representation based on the M/S speech enhancement metadata to generate one or more enhanced speech content portions in the M/S representation.

In block 828 of FIG. 8B , the audio decoder combines the one or more converted mixed audio content portions in the M/S audio channel representation with the one or more enhanced speech contents in the M/S representation to generate one or more speech-enhanced mixed audio content portions in the M/S representation.

In an embodiment, the audio decoder is further configured to inversely convert the one or more speech enhanced mixed audio content parts in the M/S representation into one or more speech enhanced mixed audio content parts in the reference audio channel representation.

In an embodiment, the audio decoder is further configured to perform: extracting a version of the speech content in the M/S audio channel representation that is separate from the mixed audio content from the audio signal; and performing one or more speech enhancement operations on one or more parts of the version of the speech content in the M/S audio channel representation based on the M/S speech enhancement metadata to generate one or more second enhanced speech content parts in the M/S audio channel representation.

In an embodiment, the audio decoder is further configured to perform: determining mixing indication data for speech enhancement; and generating, based on the mixing indication data for speech enhancement, a specific amount combination of waveform coded speech enhancement based on a version of the speech content in the M/S audio channel representation and parametric speech enhancement based on a reconstructed version of the speech content in the M/S audio channel representation.

In an embodiment, the mixing indication data is generated based at least in part on one or more SNR values for one or more converted mixed audio content portions in the M/S audio channel representation. The one or more SNR values represent one or more of the following power ratios: a power ratio of speech content to non-speech audio content in the one or more converted mixed audio content portions in the M/S audio channel representation; or a power ratio of speech content to total audio content in the one or more converted mixed audio content portions in the M/S audio channel representation.

In an embodiment, a specific amount of combination of waveform-coded speech enhancement based on a version of the speech content in the M/S audio channel representation and parametric speech enhancement based on a reconstructed version of the speech content in the M/S audio channel representation is determined using the following auditory masking model, in which auditory masking model, waveform-coded speech enhancement based on a version of the speech content in the M/S audio channel representation represents the maximum relative speech enhancement amount among multiple combinations of waveform-coded speech enhancement and parametric speech enhancement that ensures that coding noise in the output speech-enhanced audio program does not sound annoying.

In an embodiment, at least a portion of the M/S speech enhancement metadata enables a recipient audio decoder to reconstruct a version of the speech content in the M/S representation from the mixed audio content in the reference audio channel representation.

In an embodiment, the M/S speech enhancement metadata comprises metadata related to one or more of a waveform coded speech enhancement operation in the M/S audio channel representation or a parametric speech enhancement operation in the M/S audio channel.

In an embodiment, the reference audio channel representation includes audio channels associated with surround speakers. In an embodiment, the one or more non-M/S channels of the reference audio channel representation include one or more of a center channel, a left channel, or a right channel, and the one or more M/S channels of the M/S audio channel representation include one or more of a mid channel or a side channel.

In an embodiment, the M/S speech enhancement metadata comprises a collection of individual speech enhancement metadata associated with a mid-channel represented by the M/S audio channel. In an embodiment, the M/S speech enhancement metadata represents a portion of the total audio metadata encoded in the audio signal. In an embodiment, the audio metadata encoded in the audio signal comprises a data field indicating the presence of the M/S speech enhancement metadata. In an embodiment, the audio signal is part of an audio-video signal.

In an embodiment, a device comprising a processor is configured to perform any of the methods as described herein.

In one embodiment, a non-transitory computer-readable storage medium includes software instructions that, when executed by one or more processors, cause any of the methods described herein to be performed. Note that although individual embodiments are discussed herein, any combination of embodiments and/or portions of embodiments discussed herein may be combined to form additional embodiments.

8. Implementation mechanism - Hardware overview

According to one embodiment, the technology described herein is implemented by one or more special-purpose computing devices. Special-purpose computing devices can be hard-wired to perform technology, or can include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are permanently programmed to perform technology, or can include one or more general-purpose hardware processors that are programmed to perform technology according to program instructions in firmware, memory, other storage devices, or a combination thereof. Such special-purpose computing devices can also combine customized hard-wired logic, ASICs or FPGAs with customized programming to implement technology. Special-purpose computing devices can be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other devices that merge hard-wiring and/or program logic to implement technology.

For example, Figure 9 is a block diagram illustrating a computer system 900 on which embodiments of the present invention may be implemented. The computer system 900 includes a bus 902 or other communication mechanism for transmitting information, and a hardware processor 904 coupled to the bus 902 for processing information. The hardware processor 904 may be, for example, a general-purpose microprocessor.

The computer system 900 also includes a main memory 906, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 902 for storing information and instructions to be executed by the processor 904. The main memory 906 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 904. When such instructions are stored in a non-transitory storage medium accessible to the processor 904, such instructions cause the computer system 900 to become a special-purpose machine, ie, a device dedicated to performing the operations specified in the instructions.

Computer system 900 also includes a read only memory (ROM) 908 or other static storage device coupled to bus 902 for storing static information and instructions for processor 904. A storage device 910, such as a magnetic or optical disk, is provided and coupled to bus 902 for storing information and instructions.

The computer system 900 may be coupled to a display 912, such as a liquid crystal display (LCD), via the bus 902 for displaying information to a computer user. An input device 914, including alphanumeric and other keys, is coupled to the bus 902 for communicating information and command selections to the processor 904. Another type of user input device is a cursor control 916, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to the processor 904 and for controlling cursor movement on the display 912. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position in a plane.

The computer system 900 can implement the techniques described herein using device-specific hardwired logic, one or more ASICs or FPGAs, firmware, and/or program logic that is combined with the computer system to render or program the computer system 900 into a special-purpose machine. According to one embodiment, the computer system 900 can perform the techniques described herein in response to the processor 904 executing one or more sequences of one or more instructions contained in the main memory 906. Such instructions can be read into the main memory 906 from another storage medium, such as the storage device 910. Execution of the sequences of instructions contained in the main memory 906 causes the processor 904 to perform the process steps described herein. In alternative embodiments, hardwired circuitry can be used in place of software instructions, or in combination with software instructions.

As used herein, the term "storage media" refers to any non-transitory medium that stores data and/or instructions that enable a machine to operate in a particular manner. Such storage media can include non-volatile media and/or volatile media. Non-volatile media include, for example, optical or magnetic disks such as storage device 910. Volatile media include dynamic memories such as main memory 906. Common forms of storage media include, for example, floppy disks, diskettes, hard disks, solid-state drives, magnetic tape or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM and EPROM, flash EPROM, NVRAM, any other memory chip, or a cassette tape.

Storage media are distinct from transmission media, but can be used in conjunction with them. Transmission media participate in the transmission of information between storage media. Examples of transmission media include coaxial cables, copper wire, and optical fiber, including the wiring that makes up bus 902. Transmission media can also take the form of acoustic or optical waves, such as those generated during radio wave and infrared data communications.

Various forms of media may be involved in: transmitting one or more sequences of one or more instructions to the processor 904 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 900 can receive data on the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector can receive the data carried in the infrared signal, and appropriate circuitry can place the data on the bus 902. The bus 902 carries the data to the main memory 906, from which the processor 904 retrieves the instructions and executes them. Before or after execution by the processor 904, the instructions received by the main memory 906 may optionally be stored on the storage device 910.

The computer system 900 also includes a communication interface 918 coupled to the bus 902. The communication interface 918 provides bidirectional data communication coupled to a network link 920 connected to a local network 922. For example, the communication interface 918 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, the communication interface 918 can be a local area network (LAN) card that provides a data communication connection to a compatible LAN. A wireless link can also be implemented. In any such implementation, the communication interface 918 sends and receives electrical signals, electromagnetic signals, or optical signals that carry digital data streams representing various types of information.

Network link 920 typically provides data communication through one or more networks to other data devices. For example, network link 920 may provide a connection through local network 922 to data equipment or a host computer 924 operated by an Internet Service Provider (ISP) 926. ISP 926, in turn, provides data communication services through the global packet data communication network now commonly referred to as the "Internet" 928. Local network 922 and Internet 928 both use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks, as well as the signals on network link 920 and through communication interface 918, that carry digital data to and from computer system 900, are example forms of transmission media.

Computer system 900 can send messages and receive data, including program code, through the network, network link 920, and communication interface 918. In the Internet example, server 930 can transmit the requested code for an application program through Internet 928, ISP 926, local network 922, and communication interface 918.

When code is received and/or stored in storage device 910 or other non-volatile storage for later execution, the received code may be executed by processor 904 .

9. Equivalents, extensions, alternatives and other options

In the foregoing description, embodiments of the present invention have been described with reference to numerous specific details that may vary depending on the implementation. Thus, the sole and exclusive indication of what the invention is and what the applicants intend is the set of claims that issue from this application, in the specific form in which such claims are set forth, including any subsequent corrections. Any definitions expressly set forth herein for terms included in such claims shall govern the meaning of such terms as used in the claims. Therefore, no limitation, element, property, feature, advantage, or attribute that is not expressly recited in a claim should in any way limit the scope of such claim. Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense.

Claims (34)

1. An audio signal processing method, comprising: Receive mixed audio content distributed across multiple audio channels in a reference audio channel representation, wherein the mixed audio content is a mixture of speech content and non-speech audio content; The mixed audio content distributed on two or more non-M/S channels of the plurality of audio channels represented by the reference audio channel is converted into the one or more converted mixed audio content portions distributed on one or more channels of the M/S audio channel representation, wherein the M/S audio channel representation includes at least a middle channel and a side channel, wherein the middle channel represents the weighted sum or unweighted sum of the two channels of the reference audio channel representation, and wherein the side channel represents the weighted difference or unweighted difference of the two channels of the reference audio channel representation; Metadata for speech enhancement used in the one or more transformed mixed audio content portions of the M/S audio channel representation; and Generate an audio signal, the audio signal including the mixed audio content and the metadata for speech enhancement of one or more transformed mixed audio content portions in the M/S audio channel representation; The method is performed by one or more computing devices. 2. The method according to claim 1, wherein the mixed audio content is represented in a non-M/S audio channel. 3. The method according to any one of claims 1 to 2, further comprising: Generate a version of the speech content separate from the mixed audio content in the M/S audio channel representation; and Output an audio signal encoded using the version of the speech content represented in the M/S audio channel. 4. The method according to claim 3, further comprising: Generate hybrid indication data, which indicates a specific combination of a first type of speech enhancement and a second type of speech enhancement to be generated by the receiver audio decoder, wherein the first type of speech enhancement is waveform-coded speech enhancement based on the version of the speech content in the M/S audio channel representation, and wherein the second type of speech enhancement is parametric speech enhancement based on a reconstructed version of the speech content in the M/S audio channel representation; and The output is an audio signal encoded using the mixed instruction data. 5. The method of claim 4, wherein at least a portion of the metadata for speech enhancement enables the receiver audio decoder to reconstruct a reconstructed version of the speech content in the M/S representation based on the mixed audio content in the reference audio channel representation. 6. The method according to any one of claims 4 to 5, wherein the mixing indication data is generated at least in part based on one or more SNR values for one or more converted mixed audio content portions of the M/S audio channel representation, wherein the one or more SNR values represent one or more power ratios of the following: the power ratio of speech content to non-speech audio content in the one or more converted mixed audio content portions of the M/S audio channel representation, or the power ratio of speech content to total audio content in the one or more converted mixed audio content portions of the M/S audio channel representation. 7. The method according to any one of claims 4 to 5, wherein a specific combination of the first type of speech enhancement and the second type of speech enhancement is determined using an auditory masking model, in which the first type of speech enhancement represents the maximum relative amount of speech enhancement among a plurality of combinations of the first type of speech enhancement and the second type of speech enhancement, which ensures that the encoded noise in the output speech-enhanced audio program does not sound unpleasant. 8. The method according to any one of claims 1 to 2, wherein at least a portion of the metadata for speech enhancement enables the receiver audio decoder to reconstruct a version of the speech content in the M/S representation based on the mixed audio content in the reference audio channel representation. 9. The method according to any one of claims 1 to 2, wherein the metadata for speech enhancement includes metadata relating to one or more of waveform-coded speech enhancement operations or parametric speech enhancement operations in the M/S audio channel representation based on the version of the speech content. 10. The method according to any one of claims 1 to 2, wherein the reference audio channel represents an audio channel including an audio channel associated with a surround speaker. 11. The method according to any one of claims 1 to 2, wherein the two or more non-M/S channels represented by the reference audio channel include two or more of a center channel, a left channel, or a right channel; and wherein the one or more M/S channels represented by the M/S audio channel include one or more of a center channel or a side channel. 12. The method according to any one of claims 1 to 2, wherein the metadata for speech enhancement comprises a single set of speech enhancement metadata relating to an intermediate channel represented by the M/S audio channel. 13. The method according to any one of claims 1 to 2, further comprising preventing the encoding of one or more converted mixed audio content portions in the M/S audio channel representation as part of the audio signal. 14. The method according to any one of claims 1 to 2, wherein the metadata for speech enhancement represents a portion of all audio metadata encoded in the audio signal. 15. The method according to any one of claims 1 to 2, wherein the audio metadata encoded in the audio signal includes a data field indicating the presence of the metadata for speech enhancement. 16. The method according to any one of claims 1 to 2, wherein the audio signal is a part of an audio-visual signal. 17. An audio signal processing method, comprising: Receive an audio signal, the audio signal including metadata for speech enhancement and mixed audio content in a reference audio channel representation, the mixed audio content having a mixture of speech content and non-speech audio content; One or more portions of the mixed audio content distributed on two or more non-M/S channels of a plurality of audio channels represented by the reference audio channel are converted into one or more portions of the converted mixed audio content distributed on one or more M/S audio channel representations of the M/S audio channel representation, wherein the M/S audio channel representation includes at least a middle channel and a side channel, wherein the middle channel represents a weighted or unweighted sum of two channels of the reference audio channel representation, and wherein the side channel represents a weighted or unweighted difference of two channels of the reference audio channel representation; Based on the metadata used for speech enhancement, one or more speech enhancement operations are performed on one or more transformed mixed audio content portions in the M/S audio channel representation to generate one or more enhanced speech content portions in the M/S representation; and The one or more converted mixed audio content portions in the M/S audio channel representation are combined with the one or more enhanced speech content portions in the M/S representation to generate one or more speech-enhanced mixed audio content portions in the M/S representation; The method is performed by one or more computing devices. 18. The method of claim 17, wherein the steps of conversion, execution, and combination are performed in a single operation, said single operation being performed on said one or more portions of the mixed audio content distributed on two or more non-M/S channels of a plurality of audio channels represented by the reference audio channel. 19. The method according to any one of claims 17 to 18, further comprising inversely converting the one or more speech-enhanced hybrid audio content portions in the M/S representation into one or more speech-enhanced hybrid audio content portions in the reference audio channel representation. 20. The method according to any one of claims 17 to 18, further comprising: Extract from the audio signal a version of the speech content separate from the mixed audio content in the M/S audio channel representation; and Based on at least a portion of the metadata used for speech enhancement, one or more speech enhancement operations are performed on one or more portions of the versions of the speech content in the M/S audio channel representation to generate one or more second enhanced speech content portions in the M/S audio channel representation. 21. The method of claim 20, further comprising: Determine the mixed indication data for speech enhancement; Based on the hybrid indication data used for speech enhancement, a specific combination of two types of speech enhancement is generated, wherein the first type of speech enhancement is waveform-coded speech enhancement based on the version of the speech content in the M/S audio channel representation, and the second type of speech enhancement is parametric speech enhancement based on the reconstructed version of the speech content in the M/S audio channel representation. 22. The method of claim 21, wherein the mixing indication data is generated at least in part by one of the upstream audio encoder that generates the audio signal or the receiver audio decoder that receives the audio signal based on one or more SNR values for one or more converted mixed audio content portions in the M/S audio channel representation, wherein the one or more SNR values represent one or more of the following power ratios: the power ratio of speech content to non-speech audio content in one or more converted mixed audio content portions in the M/S audio channel representation, or the power ratio of speech content to total audio content in one or more portions of the converted mixed audio content in the M/S audio channel representation or the mixed audio content in the reference audio channel representation. 23. The method of any one of claims 21 to 22, wherein the specific combination of the two types of speech enhancement is determined using an auditory masking model constructed by one of the upstream audio encoder that generates the audio signal or the receiver audio decoder that receives the audio signal, wherein the first type of speech enhancement is the maximum relative speech enhancement amount among a plurality of combinations of the first type of speech enhancement and the second type of speech enhancement, which ensures that the encoded noise in the output speech-enhanced audio program does not sound unpleasant. 24. The method of any one of claims 17 to 18, wherein at least a portion of the metadata for speech enhancement enables the receiver audio decoder to reconstruct a version of the speech content in the M/S representation based on the mixed audio content in the reference audio channel representation. 25. The method according to any one of claims 17 to 18, wherein the metadata for speech enhancement includes metadata relating to one or more of waveform-coded speech enhancement operations or parametric speech enhancement operations in the M/S audio channel representation based on the version of the speech content. 26. The method according to any one of claims 17 to 18, wherein the reference audio channel represents an audio channel relating to a surround speaker. 27. The method according to any one of claims 17 to 18, wherein the two or more non-M/S channels represented by the reference audio channel include one or more of a center channel, a left channel, or a right channel; and wherein the one or more M/S channels represented by the M/S audio channel include one or more of a center channel or a side channel. 28. The method according to any one of claims 17 to 18, wherein the metadata for speech enhancement comprises a single set of speech enhancement metadata relating to an intermediate channel represented by the M/S audio channel. 29. The method according to any one of claims 17 to 18, wherein the metadata for speech enhancement represents a portion of all audio metadata encoded in the audio signal. 30. The method according to any one of claims 17 to 18, wherein the audio metadata encoded in the audio signal includes a data field indicating the presence of the metadata for speech enhancement. 31. The method according to any one of claims 17 to 18, wherein the audio signal is a part of an audio-visual signal. 32. A media processing system configured to perform any one of the methods of claims 1 to 31. 33. An apparatus comprising a processor and configured to perform any one of the methods of claims 1 to 31. 34. A non-transitory computer-readable storage medium comprising software instructions that, when executed by one or more processors, cause to perform any one of the methods of claims 1 to 31.