HK40001584B - Audio encoder and decoder - Google Patents

Description

Audio encoders and decoders

This application is a divisional application of the invention patent application with application number 201480011081.3, application date April 4, 2014, and invention title "Audio Encoder and Decoder".

Cross-reference to related applications

This application claims priority to U.S. Provisional Patent Application No. 61/808680, filed April 5, 2013, the entire contents of which are incorporated herein by reference.

Technical Field

The disclosures herein generally relate to multichannel audio coding. In particular, it relates to encoders and decoders for hybrid coding that includes parametric coding and discrete multichannel coding.

Background Technology

In traditional multichannel audio coding, possible coding schemes include discrete multichannel coding or parametric coding such as MPEG Surround. The scheme used depends on the bandwidth of the audio system. Parametric coding methods are known to be scalable and efficient in terms of listening quality, making them particularly attractive for low bitrate applications. Discrete multichannel coding is often used in high bitrate applications. From a bandwidth efficiency perspective, existing distribution or processing formats and associated coding techniques can be improved, especially in applications with bitrates between low and high bitrates.

US7292901 (Kroon et al.) relates to a hybrid coding method in which a hybrid audio signal is formed by at least one downmixed spectral component and at least one unmixed spectral component. The method proposed in this application can improve the capabilities of applications with a certain bit rate, but further improvements may be needed to further enhance the efficiency of the audio processing system.

Summary of the Invention

According to one aspect of the present invention, a method for decoding time frames of an encoded audio bitstream in an audio processing system is provided, the method comprising:

For each time frame, M upmixed signals are received, wherein the M upmixed signals include spectral coefficients corresponding to frequencies higher than the first crossover frequency.

The M upmixed signals are the result of upmixing N frequency extension downmixed signals obtained by performing frequency reconstruction above the second crossover frequency in the reconstruction range into M upmixed signals for the time frame, wherein the second crossover frequency is higher than the first crossover frequency and the frequency reconstruction uses reconstruction parameters derived from the encoded audio bitstream.

For the time frame, another waveform encoded signal is extracted from the encoded audio bitstream, the other waveform encoded signal including spectral coefficients corresponding to a subset of frequencies higher than the first crossover frequency; and

For the time frame, the other waveform encoded signal is interleaved with one of the M upmixing signals to generate an interleaved signal.

Attached Figure Description

An exemplary embodiment will now be described with reference to the accompanying drawings, in which:

Figure 1 is a general block diagram of a decoding system according to an exemplary embodiment;

Figure 2 shows the first part of the decoding system in Figure 1;

Figure 3 shows the second part of the decoding system in Figure 1;

Figure 4 shows the third part of the decoding system in Figure 1;

Figure 5 is a general block diagram of an encoding system according to an exemplary embodiment;

Figure 6 is a general block diagram of a decoding system according to an exemplary embodiment;

Figure 7 shows the third part of the decoding system in Figure 6; and

Figure 8 is a general block diagram of an encoding system according to an exemplary embodiment.

All accompanying drawings are schematic and generally show only the parts necessary to illustrate the contents of this disclosure, while other parts may be omitted or are merely implied. Unless otherwise stated, the same reference numerals refer to the same parts in different drawings.

Detailed Implementation

Overview – Decoder

As used herein, an audio signal can be a pure audio signal, an audio-visual signal, or the audio portion of a multimedia signal, or any of these signals combined with metadata.

As used herein, downmixing multiple signals means, for example, combining these signals by forming a linear combination to obtain a smaller number of signals. The inverse operation of downmixing is called upmixing, which is performing the operation on a smaller number of signals to obtain a larger number of signals.

According to the first aspect, exemplary embodiments propose methods, apparatus, and computer program products for reconstructing multichannel audio signals based on input signals. The proposed methods, apparatus, and computer program products can generally have the same features and advantages.

According to an exemplary embodiment, a decoder is provided for reconstructing a multichannel audio processing system with M coded channels, where M > 2. The decoder includes a first receiving stage configured to receive N waveform-coded downmixed signals, each comprising spectral coefficients corresponding to frequencies between first and second cross-over frequencies, where 1 < N < M.

The decoder also includes a second receiving stage configured to receive M waveform-coded signals comprising spectral coefficients corresponding to frequencies up to a first crossover frequency, each of the M waveform-coded signals corresponding to a specific one of the M coded channels.

The decoder also includes a downmixing stage located downstream of the second receiver stage, the downmixing stage being configured to downmix M waveform-coded signals into N downmixed signals, each including spectral coefficients corresponding to frequencies up to the first crossover frequency.

The decoder also includes a first combining stage located downstream of the first receiving stage and the downmixing stage, the first combining stage being configured to combine each of the N downmixing signals received by the first receiving stage and a corresponding downmixing signal from the N downmixing signals from the downmixing stage into N combined downmixing signals.

The decoder also includes a high-frequency reconstruction stage located downstream of the first combination stage, the high-frequency reconstruction stage being configured to extend each of the N combined downmixed signals from the combination stage to a frequency range above the second crossover frequency by performing high-frequency reconstruction.

The decoder also includes an upmixing stage located downstream of the high-frequency reconstruction stage, the upmixing stage being configured to perform upmixing of N frequency spread signal parameters from the high-frequency reconstruction stage into M upmixed signals, each of the M upmixed signals corresponding to one of the M coded channels.

The decoder also includes a second combining stage located downstream of the upmixing stage and the second receiving stage, the second combining stage being configured to combine M upmixed signals from the upmixing stage with M waveform-coded signals received by the second receiving stage.

The M waveform-coded signals are pure waveform-coded signals without any added parameter signals; that is, they are unmixed discrete representations of the processed multichannel audio signal. The advantage of representing lower frequencies with these waveform-coded signals is that the human ear is more sensitive to the low-frequency components of an audio signal. By encoding this portion with better quality, the overall impression of the decoded audio can be improved.

The advantage of having at least two downmixing signals is that, compared to a system with only one downmixing channel, this embodiment provides an increased dimension of downmixing. Therefore, according to this embodiment, better decoded audio quality can be provided, which may be more significant than the bit rate benefit offered by a system with only one downmixing signal.

The advantage of using hybrid coding, which includes parametric downmixing and discrete multichannel coding, is that it can improve the quality of decoded audio signals at certain bit rates compared to using traditional parametric coding methods (i.e., MPEG surround with HE-AAC). At bit rates of approximately 72 kilobits per second (kbps), traditional parametric coding models may saturate; that is, the quality of the decoded audio signal is limited by the shortcomings of the parametric model, rather than by the insufficient number of bits available for coding. Therefore, for bit rates starting at approximately 72 kbps, it may be more advantageous to use bits for discrete waveform coding at lower frequencies. Meanwhile, the hybrid approach using parametric downmixing and discrete multichannel coding can improve the quality of decoded audio at certain bit rates, such as 128 kbps or lower, compared to methods that use all of these bits for waveform coding at lower frequencies and employ spectral band replication (SBR) for the remaining frequencies.

The advantage of having N waveform-coded downmixers that include only spectral data corresponding to frequencies between the first and second crossover frequencies is that the bit transmission rate required by the audio signal processing system can be reduced. Alternatively, the bits saved by using downmixers with bandpass filtering can be used to waveform-code lower frequencies; for example, the sampling frequency for those frequencies can be higher or the first crossover frequency can be increased.

As mentioned above, since the human ear is more sensitive to the low-frequency part of the audio signal, the high-frequency part of the audio signal, which has a frequency higher than the second crossover frequency, can be reconstructed through high-frequency reconstruction without reducing the perceived audio quality of the decoded audio signal.

Another advantage of this embodiment is that since the parameter upmixing performed in the upmixing stage only operates on the spectral coefficients corresponding to frequencies higher than the first crossover frequency, the complexity of upmixing is reduced.

According to another embodiment, the combination performed in the first combination level is performed in the frequency domain, which includes each of N waveform-coded downmixing signals with spectral coefficients corresponding to frequencies between the first and second crossover frequencies and a corresponding downmixing signal from N downmixing signals with spectral coefficients corresponding to frequencies up to the first crossover frequency, combining them into N combined downmixing signals.

The advantage of this embodiment is that the M waveform encoded signals and the N waveform encoded downmixed signals can be encoded by the waveform encoder using overlapping windowed transforms that independently window the M waveform encoded signals and the N waveform encoded downmixed signals, respectively, and can still be decoded by the decoder.

According to another embodiment, extending each of the N combined downmixed signals in the high-frequency reconstruction stage to a frequency range above the second crossover frequency is performed in the frequency domain.

According to another embodiment, the combination performed in the second combination step—that is, the combination of M upmixed signals including spectral coefficients corresponding to frequencies above the first crossover frequency and M waveform-coded signals including spectral coefficients corresponding to frequencies up to the first crossover frequency—is performed in the frequency domain. As mentioned above, the advantage of combining signals in the QMF domain is that independent windowing can be used for overlapping window transforms used to encode signals in MDCT.

According to another embodiment, the parametric upmixing of the combination of N frequency extensions performed in the upmixing stage to M upmixing signals is performed in the frequency domain.

According to another embodiment, downmixing M waveform-coded signals into N downmixed signals, including spectral coefficients corresponding to frequencies up to the first crossover frequency, is performed in the frequency domain.

According to the embodiment, the frequency domain is the quadrature mirror filter (QMF) domain.

According to another embodiment, the downmixing performed in the downmixing stage is performed in the time domain, wherein M waveform-coded signals are downmixed into N downmixed signals, each including spectral coefficients corresponding to frequencies up to a first crossover frequency.

According to another embodiment, the first crossover frequency depends on the bit transmission rate of the multi-channel audio processing system. This results in the available bandwidth being used to improve the quality of the decoded audio signal, since the portion of the audio signal with frequencies below the first crossover frequency is purely waveform-coded.

According to another embodiment, extending each of the N combined undermixed signals into a frequency range above the second crossover frequency by performing high-frequency reconstruction at the high-frequency reconstruction stage is performed using high-frequency reconstruction parameters. These high-frequency reconstruction parameters can be received by the decoder at the receiver stage and then sent to the high-frequency reconstruction stage. High-frequency reconstruction may, for example, include performing spectral band replication (SBR).

According to another embodiment, the parameter upmixing in the upmixing stage is performed using upmixing parameters. These upmixing parameters are received by the encoder, for example, at the receiver stage and sent to the upmixing stage. A decorrelated version of the combined downmixed signal with N frequency spreads is generated, and the combined downmixed signal with N frequency spreads and its decorrelated version undergo matrix operations. The parameters for the matrix operations are given by the upmixing parameters.

According to another embodiment, the N waveform-coded downmixed signals received in the first receiving stage and the M waveform-coded signals received in the second receiving stage are encoded using an overlapping window transform that independently windows the N waveform-coded downmixed signals and the M waveform-coded signals, respectively.

The advantage of this approach is that it allows for improved encoding quality, which in turn allows for improved quality of the decoded multi-channel audio signal. For example, if a transient is detected in a higher frequency band at a certain time point, the waveform encoder can encode that specific time frame with a shorter window sequence, while maintaining the default window sequence for lower frequency bands.

According to an embodiment, the decoder may include a third receiving stage configured to receive another waveform-coded signal comprising spectral coefficients corresponding to a subset of frequencies above a first crossover frequency. The decoder may also include an interleaving stage downstream of the upmixing stage. The interleaving stage may be configured to interleave the other waveform-coded signal with one of M upmixing signals. The third receiving stage may also be configured to receive a plurality of other waveform-coded signals, and the interleaving stage may further be configured to interleave these plurality of other waveform-coded signals with a plurality of M upmixing signals.

Its advantage is that certain parts of the frequency range above the first crossover frequency that are difficult to parametrically reconstruct from the downmixing signal can be provided in the form of waveform encoding so as to interleave with the parametrically reconstructed upmixing signal.

In one exemplary embodiment, the interleaving is performed by adding the other waveform-coded signal to one of the M upmixing signals. According to another exemplary embodiment, the step of interleaving the other waveform-coded signal with one of the M upmixing signals includes replacing one of the M upmixing signals with the other waveform-coded signal in a subset of frequencies above a first crossover frequency corresponding to the spectral coefficients of the other waveform-coded signal.

According to an exemplary embodiment, the decoder can also be configured to receive control signals, for example, via a third receiving stage. The control signals can instruct how the other waveform-encoded signal is interleaved with one of the M upmix signals, wherein the step of interleaving the other waveform-encoded signal with one of the M upmix signals is based on the control signals. Specifically, the control signals can instruct the frequency range and time range for which the other waveform-encoded signal is to be interleaved with one of the M upmix signals, such as one or more time/frequency blocks in the QMF domain. Accordingly, the interleaving can occur in one channel, both in time and frequency.

The advantage of doing this is that you can select the time and frequency ranges that do not suffer from aliasing or start/fade-out problems caused by the overlapping window transform used to encode the waveform signal.

Overview – Encoder

According to the second aspect, exemplary embodiments provide a method, apparatus, and computer program product for encoding multichannel audio signals based on input signals.

The proposed methods, devices, and computer program products can generally have the same characteristics and advantages.

The advantages of the features and settings described in the above overview of the decoder are generally effective for the corresponding features and settings used in the encoder.

According to an exemplary embodiment, an encoder is provided for encoding a multichannel audio processing system with M channels, where M>2.

The encoder includes a receiver stage configured to receive M signals corresponding to the M channels to be encoded.

The encoder also includes a first waveform encoding stage configured to receive M signals from a receiving stage and generate M waveform-coded signals by individually waveform encoding the M signals for a frequency range corresponding to frequencies up to a first crossover frequency, whereby the M waveform-coded signals include spectral coefficients corresponding to frequencies up to the first crossover frequency.

The encoder also includes a downmixing stage configured to receive M signals from the receiving stage and downmix these M signals into N downmixed signals, where 1 < N < M.

The encoder also includes a high-frequency reconstruction coding stage configured to receive N downmixed signals from the downmixing stage and subject these N downmixed signals to high-frequency reconstruction coding, thereby configuring the high-frequency reconstruction coding stage to extract high-frequency reconstruction parameters that enable high-frequency reconstruction of the N downmixed signals above a second crossover frequency.

The encoder also includes a parameter encoding stage configured to receive M signals from a receiving stage and N downmixing signals from a downmixing stage, and to parameter-encode the M signals for a frequency range corresponding to frequencies above a first crossover frequency. The parameter encoding stage is thus configured to extract upmixing parameters that enable the N downmixing signals to be upmixed into M reconstructed signals corresponding to the M channels for a frequency range above the first crossover frequency.

The encoder also includes a second waveform encoding stage configured to receive N downmixing signals from the downmixing stage and generate N waveform-coded downmixing signals by waveform encoding the N downmixing signals for a frequency range corresponding to frequencies between the first and second crossover frequencies, whereby the N waveform-coded downmixing signals include spectral coefficients corresponding to frequencies between the first and second crossover frequencies.

According to an embodiment, subjecting N downmixed signals to high-frequency reconstruction coding in the high-frequency reconstruction coding stage is performed in the frequency domain, preferably in the quadrature mirror filter (QMF) domain.

According to another embodiment, the parameter encoding of M signals in the parameter encoding stage is performed in the frequency domain, preferably in the quadrature mirror filter (QMF) domain.

According to another embodiment, generating M waveform-coded signals by individually waveform-coding M signals in the first waveform coding level includes applying an overlap window transform to the M signals, wherein at least two of the M signals use different overlap window sequences.

According to an embodiment, the encoder may further include a third waveform encoding level, which is configured to generate another waveform-encoded signal by waveform encoding one of the M signals for a frequency range corresponding to a subset of the frequency range above the first crossover frequency.

According to an embodiment, the encoder may include a control signal generation stage. The control signal generation stage is configured to generate a control signal that indicates how the other waveform encoded signal is reconstructed and interleaved with one of the M signals in the decoder. For example, the control signal may indicate the frequency range and time range of the interleaving of the other waveform encoded signal with one of the M signals.

Exemplary embodiments

Figure 1 is a general block diagram of a decoder 100 used to reconstruct M coded channels in a multi-channel audio processing system. The decoder 100 comprises three conceptual sections 200, 300, and 400, which will be explained in more detail with reference to Figures 2-4 below. In the first conceptual section 200, the encoder receives M waveform-coded signals and N waveform-coded downmixed signals representing the multi-channel audio signal to be decoded, where 1 < N < M. In the illustrated example, N is set to 2. In the second conceptual section 300, the M waveform-coded signals are downmixed and combined with the N waveform-coded downmixed signals. High-frequency reconstruction (HFR) is then performed on the combined downmixed signal. In the third conceptual section 400, the high-frequency reconstruction signal is upmixed, and the M waveform-coded signals are combined with the upmixed signals to reconstruct the M coded channels.

In the exemplary embodiment described in conjunction with Figures 2-4, the reconstruction of encoded 5.1 surround sound is depicted. Note that low-frequency effect signals are not mentioned in the described embodiment or in the figures. This does not mean that any low-frequency effects are ignored. Low-frequency effects (Lfe) are added to the reconstructed 5 channels in any suitable manner known to those skilled in the art. It should also be noted that the described decoder is equally well suited to other types of encoded surround sound, such as 7.1 or 9.1 surround sound.

Figure 2 illustrates a first conceptual portion 200 of the decoder 100 in Figure 1. The decoder includes two receiver stages 212 and 214. In the first receiver stage 212, the bitstream 202 is decoded and dequantized into two waveform-coded downmixed signals 208a-b. Each of these two waveform-coded downmixed signals 208a-b includes spectral coefficients corresponding to frequencies between a first crossover frequency _ky and a second crossover frequency _xx .

In the second receiver stage 212, the bitstream 202 is decoded and dequantized into five waveform-coded downmixed signals 210a-e. Each of these five waveform-coded downmixed signals 208a-e includes spectral coefficients corresponding to frequencies up to the first crossover frequency _kx .

By way of example, signals 210a-e include two channel pairs and one mono element for the center. The channel pairs can be, for example, a combination of left front and left surround signals and a combination of right front and right surround signals. Another example is a combination of left front and right front signals and a combination of left surround and right surround signals. These channel pairs can be encoded, for example, in a sum-and-difference format. All five signals 210a-e can be encoded using an overlapping window transform with independent windowing and can still be decoded by the decoder. This allows for improved encoding quality and therefore allows for improved quality of the decoded signal.

For example, the first crossover frequency _ky is 1.1 kHz. For example, the second crossover frequency _kx is in the range of 5.6-8 kHz. It should be noted that the first crossover frequency _ky can vary, even on a signal-by-signal basis. That is, the encoder can detect that a signal component in a particular output signal may not be faithfully reproduced by the stereo downmix signal 208a-b, and can increase the bandwidth of the relevant waveform encoded signal (i.e., 210a-e), i.e., the first crossover frequency _ky , for that particular moment in order to perform appropriate waveform encoding of the signal component.

As will be described later in this specification, the remaining stages of encoder 100 typically operate in the quadrature mirror filter (QMF) domain. For this reason, each of the signals 208a-b, 210a-e received by the first and second receiving stages 212, 214 in modified discrete cosine transform (MDCT) form is transformed to the time domain by applying inverse MDCT 216. Each signal is then transformed back to the frequency domain by applying QMF transform 218.

In Figure 3, five waveform encoded signals 210 are downmixed in downmixing stage 308 into two downmixed signals 310 and 312, each including spectral coefficients corresponding to frequencies up to the first crossover frequency _ky . These downmixed signals 310 and 312 can be formed by downmixing the low-pass multichannel signals 210a-e using the same downmixing scheme used in the encoder to create the two downmixed signals 208a-b shown in Figure 2.

Then, these two new downmixing signals 310 and 312 are combined with the corresponding downmixing signals 208a-b in the first combination stages 320 and 322 to form combined downmixing signals 302a-b. Therefore, each of the combined downmixing signals 302a-b includes spectral coefficients derived from downmixing signals 310 and 312 corresponding to frequencies up to the first crossover frequency _ky , and spectral coefficients derived from the two waveform-coded downmixing signals 208a-b received in the first receiving stage 212 corresponding to frequencies between the first crossover frequency _ky and the second crossover frequency _xx (shown in Figure 2).

The encoder also includes a high-frequency reconstruction (HFR) stage 314. The HFR stage is configured to extend each of the two combined downmixed signals 302a-b from the combining stage into a frequency range above a second crossover frequency _kx by performing high-frequency reconstruction. According to some embodiments, the performed high-frequency reconstruction may include performing spectral band replication (SBR). High-frequency reconstruction can be performed by using the high-frequency reconstruction parameters received by the HFR stage 314 in any suitable manner.

The output from the high-frequency reconstruction stage 314 consists of two signals 304a-b, including the downmixed signals 208a-b and the applied HFR extensions 316, 318. As described above, the HFR stage 314 performs high-frequency reconstruction based on the frequencies present in the input signals 210a-e from the second receiver stage 214 (shown in FIG. 2), which are combined with the two downmixed signals 208a-b. In short, the HFR ranges 316, 318 include the portion of the spectral coefficients from the downmixed signals 310, 312 that has been copied to the HFR ranges 316, 318. Therefore, portions of the five waveform-coded signals 210a-e will appear in the HFR ranges 316, 318 of the output 304 from the HFR stage 314.

It should be noted that the downmixing at downmixing stage 308 before high-frequency reconstruction stage 314 and the combination in the first combination stages 320, 322 can be performed in the time domain, i.e., after each signal is transformed to the time domain by applying the inverse modified discrete cosine transform (MDCT) 216 (shown in Figure 2). However, assuming that the waveform-coded signals 210a-e and the waveform-coded downmixing signals 208a-b can be encoded by the waveform encoder using an overlapping window transform with independent windowing, signals 210a-e and 208a-b may not be able to be seamlessly combined in the time domain. Therefore, a better-controlled scenario is obtained if the combination in at least the first combination stages 320, 322 is performed in the QMF domain.

Figure 4 illustrates the third and final conceptual section 400 of encoder 100. The output 304 from HFR stage 314 constitutes the input to upmixing stage 402. Upmixing stage 402 creates five signal outputs 404a-e by performing parametric upmixing on the frequency-spread signals 304a-e. For frequencies above the first crossover frequency _ky , each of the five upmixed signals 404a-e corresponds to one of the five coded channels in the encoded 5.1 surround sound. According to an exemplary parametric upmixing process, upmixing stage 402 first receives parametric mixing parameters. Upmixing stage 402 also generates decorrelated versions of the combined downmixed signals 304a-b of the two frequency spreads. Upmixing stage 402 further subjectes the combined downmixed signals 304a-b of the two frequency spreads and their decorrelated versions to matrix operations, where the parameters of the matrix operations are given by the upmixing parameters. Alternatively, any other parametric upmixing process known in the art can be applied. Applicable parameter upmixing processes are described, for example, in “MPEG Surround—The ISO/MPEG Standard for Efficient and Compatible Multichannel Audio Coding” (Herre et al., Journal of the Audio Engineering Society, Vol. 56, No. 11, November 2008).

Therefore, the outputs 404a-e from the upmixing stage 402 do not include frequencies below the first crossover frequency _ky . The remaining spectral coefficients corresponding to frequencies up to the first crossover frequency _ky exist in five waveform-coded signals 210a-e, which have been delayed by the delay stage 412 to match the timing of the upmixing signal 404.

The encoder 100 also includes second combination stages 416, 418. The second combination stages 416, 418 are configured to combine five upmixed signals 404a-e with five waveform encoded signals 210a-e received by the second receiving stage 214 (shown in FIG. 2).

Note that any existing Lfe signal can be added as a separate signal to the resulting combined signal 422. Each of the signals in 422 is then transformed to the time domain by applying the inverse QMF transform 420. Therefore, the output from the inverse QMF transform 414 is a fully decoded 5.1-channel audio signal.

Figure 6 shows a decoding system 100', a variant of the decoding system 100 of Figure 1. The decoding system 100' has conceptual sections 200', 300', and 400' corresponding to the conceptual sections 100, 200, and 300 of Figure 1. The difference between the decoding system 100' of Figure 6 and the decoding system of Figure 1 is that a third receiving stage 616 is present in the conceptual section 200', and an interleaving stage 714 is present in the third conceptual section 400'.

The third receiver stage 616 is configured to receive another waveform-coded signal. This other waveform-coded signal includes spectral coefficients corresponding to a subset of frequencies above the first crossover frequency. This other waveform-coded signal can be transformed to the time domain by applying inverse MDCT 216. It can then be transformed back to the frequency domain by applying QMF transform 218.

It should be understood that this additional waveform-coded signal can be received as a separate signal. However, this additional waveform-coded signal can also form part of one or more of the five waveform-coded signals 210a-e. In other words, this additional waveform-coded signal can, for example, be jointly encoded with one or more of the five waveform-coded signals 210a-e using the same MCDT transform. If so, then the third receiving stage 616 corresponds to the second receiving stage, that is, the additional waveform-coded signal is received together with the five waveform-coded signals 210a-e via the second receiving stage 214.

Figure 7 shows the third conceptual section 300' of the decoder 100' of Figure 6 in more detail. In addition to the high-frequency extended downmixed signals 304a-b and the five waveform-coded signals 210a-e, another waveform-coded signal 710 is also input to the third conceptual section 400'. In the illustrated example, this other waveform-coded signal 710 corresponds to the third of the five channels. This other waveform-coded signal 710 also includes spectral coefficients corresponding to frequency intervals starting from the first crossover frequency _ky . However, the form of the subset of frequency ranges above the first crossover frequency covered by this other waveform-coded signal 710 can, of course, vary in different embodiments. It should also be noted that multiple waveform-coded signals 710a-e can be received, with different waveform-coded signals corresponding to different output channels. The subset of frequency ranges covered by these multiple other waveform-coded signals 710a-e can vary among the different signals of these multiple other waveform-coded signals 710a-e.

The other waveform-encoded signal 710 can be delayed by delay stage 712 to match the timing of the upmix signal 404 output from upmix stage 402. The upmix signal 404 and the other waveform-encoded signal 710 are then input to interleaving stage 714. Interleaving stage 714 performs interleaving, that is, combines the upmix signal 404 and the other waveform-encoded signal 710 to generate an interleaved signal 704. In this example, interleaving stage 714 thus interleaves the third upmix signal 404c and the other waveform-encoded signal 710. Interleaving can be performed by adding the two signals together. However, typically, interleaving is performed by replacing the upmix signal 404 with the other waveform-encoded signal 710 within the frequency and time range of signal overlap.

Then, the interleaved signal 704 is input to the second combination stages 416, 418, where it is combined with the waveform-coded signals 201a-e to generate the output signal 722 in the same manner as described with reference to FIG4. Note that the order of the interleaving stage 714 and the second combination stages 416, 418 can be reversed, such that the combination is performed before the interleaving.

Furthermore, when the other waveform-coded signal 710 constitutes part of one or more of the five waveform-coded signals 210a-e, the second combination stages 416, 418 and the interleaving stage 714 can be combined into a single stage. Specifically, this combined stage will use the spectral components of the five waveform-coded signals 210a-e for frequencies up to the first crossover frequency _ky . For frequencies above the first crossover frequency, the combined stage will use an upmixed signal 404 interleaved with the other waveform-coded signal 710.

The interleaving stage 714 can operate under the control of a control signal. For this purpose, the decoder 100' can receive the control signal, for example via a third receiving stage 616, which indicates how to interleave the other waveform-encoded signal with one of the M upmixing signals. For example, the control signal can indicate the frequency range and time range to be interleaved between the other waveform-encoded signal 710 and one of the upmixing signals 404. For example, the frequency range and time range can be represented according to time/frequency blocks to be interleaved. The time/frequency block can be a time/frequency block about the time/frequency grid of the QMF domain where the interleaving occurs.

The control signals can use vectors, such as binary vectors, to indicate the time/frequency blocks to be interleaved. Specifically, there can be a first vector in the frequency direction to indicate the frequency at which interleaving is to be performed. This indication can be made, for example, by indicating logic 1 for the corresponding frequency interval in the first vector. There can also be a second vector in the time direction to indicate the time interval at which interleaving is to be performed. This indication can be made, for example, by indicating logic 1 for the corresponding time interval in the second vector. For this purpose, time frames are typically divided into multiple time slots, allowing time indications to be made subframe by subframe. A time/frequency matrix can be constructed by intersecting the first and second vectors. For example, the time/frequency matrix can be a binary matrix in which logic 1 is included for each time/frequency block indicated by logic 1 in the first and second vectors. The interleaving level 714 can then use the time/frequency matrix when performing interleaving, for example, such that for a time/frequency block in the time/frequency matrix, such as that indicated by logic 1, one or more of the upmixing signals 704 are replaced by the other waveform-coded signal 710.

Note that vectors can use schemes other than binary schemes to indicate time/frequency blocks to be interleaved. For example, a vector can use a first value such as 0 to indicate no interleaving and a second value to indicate interleaving with respect to a particular channel identified by the second value.

Figure 5 shows, by way of example, a general block diagram of an encoding system 500 for encoding a multi-channel audio processing system with M channels according to an embodiment.

In the exemplary embodiment depicted in Figure 5, encoding of 5.1 surround sound is described. Therefore, in the illustrated example, M is set to five. Note that low-frequency effect signals are not mentioned in the described embodiment or in the figures. This does not mean that any low-frequency effects are ignored. Low-frequency effects (Lfe) are added to bitstream 552 in any suitable manner known to those skilled in the art. It should also be noted that the described encoder is equally well suited for encoding other types of surround sound, such as 7.1 or 9.1 surround sound. In encoder 500, five signals 502, 504 are received at a receiver stage (not shown). Encoder 500 includes a first waveform encoding stage 506 configured to receive the five signals 502, 504 from the receiver stage and generate five waveform-encoded signals 518 by waveform encoding these five signals 502, 504 one by one. Waveform encoding stage 506 may, for example, cause each of the five received signals 502, 504 to undergo MDCT transformation. As discussed regarding the decoder, the encoder can choose to use an MDCT transform with independent windowing to encode each of the five signals 502, 504. This allows for improved encoding quality and therefore allows for improved quality of the decoded signal.

For a frequency range corresponding to frequencies up to the first crossover frequency, five waveform-coded signals 518 are waveform-coded. Therefore, the five waveform-coded signals 518 include spectral coefficients corresponding to frequencies up to the first crossover frequency. This can be achieved by subjecting each of the five waveform-coded signals 518 to a low-pass filter. The five waveform-coded signals 518 are then quantized 520 according to a psychoacoustic model. The psychoacoustic model is configured to be as accurate as possible, taking into account the available bit rate in the multi-channel audio processing system, to reproduce the encoded signal as perceived by the listener when decoded at the system's decoder side.

As discussed above, encoder 500 performs hybrid coding including discrete multichannel coding and parametric coding. As described above, for frequencies up to the first crossover frequency, discrete multichannel coding is performed on each of the input signals 502, 504 in waveform coding stage 506. For frequencies above the first crossover frequency, parametric coding is performed so that the five input signals 502, 504 can be reconstructed on the decoder side based on N downmixed signals. In the example illustrated in Figure 5, N is set to 2. Downmixing of the five input signals 502, 504 is performed in downmixing stage 534. Downmixing stage 534 advantageously operates in the QMF domain. Therefore, before being input to downmixing stage 534, the five signals 502, 504 are transformed to the QMF domain by QMF analysis stage 526. The downmixing stage performs linear downmixing on the five signals 502, 504 and outputs two downmixed signals 544, 546.

After the two downmixed signals 544 and 546 are transformed back to the time domain by undergoing an inverse QMF transform 554, they are received by a second waveform coding stage 508. The second waveform coding stage 508 generates two waveform-coded downmixed signals by waveform coding the two downmixed signals 544 and 546 for a frequency range corresponding to frequencies between the first and second crossover frequencies. The waveform coding stage 508 may, for example, subject each of the two downmixed signals to an MDCT transform. Therefore, the two waveform-coded downmixed signals include spectral coefficients corresponding to frequencies between the first and second crossover frequencies. Then, according to a psychoacoustic model, the two waveform-coded downmixed signals are quantized 522.

In order to reconstruct frequencies above the second crossover frequency on the decoder side, high-frequency reconstruction (HFR) parameters 538 are extracted from the two downmixed signals 544 and 546. These parameters are extracted at the HFR coding level 532.

To reconstruct five signals from two downmixed signals 544 and 546 at the decoder side, the parametric encoding stage 530 receives five input signals 502 and 504. These five signals 502 and 504 undergo parametric encoding for the frequency range corresponding to frequencies above the first crossover frequency. The parametric encoding stage 530 is then configured to extract an upmixing parameter 536 that allows the two downmixed signals 544 and 546 to be upmixed into five reconstructed signals corresponding to the five input signals 502 and 504 (i.e., the five channels in the encoded 5.1 surround sound) for the frequency range above the first crossover frequency. Note that the upmixing parameter 536 is extracted only for the frequency range above the first crossover frequency. This reduces the complexity of the parametric encoding stage 530 and the bit rate of the corresponding parameter data.

Note that downmixing 534 can be implemented in the time domain. In this case, QMF analysis level 526 should be downstream of downmixing level 534 and before HFR coding level 532, because HFR coding level 532 typically operates in the QMF domain. In this case, inverse QMF level 554 can be omitted.

The encoder 500 also includes a bitstream generation stage (i.e., a bitstream multiplexer) 524. According to an exemplary embodiment of the encoder 500, the bitstream generation stage is configured to receive five encoded and quantized signals 548, two parameter signals 536, 538, and two encoded and quantized downmixed signals 550. These signals are converted into a bitstream 552 by the bitstream generation stage 524 for further distribution in a multi-channel audio system.

In the described multichannel audio system, such as when streaming audio over the Internet, there is often a maximum available bit rate. Because the characteristics of each time frame of the input signals 502 and 504 are different, the exact same bit allocation cannot be used among the five waveform-coded signals 548 and the two downmixed waveform-coded signals 550. Furthermore, each individual signal 548 and 550 may require more or fewer allocated bits, allowing the signal to be reconstructed according to a psychoacoustic model. According to an exemplary embodiment, the first and second waveform coding stages 506 and 508 share a common bit reservoir. Depending on the characteristics of the signal to be encoded and the current psychoacoustic model, the available bits for each encoded frame are first allocated between the first and second waveform coding stages 506 and 508. Then, as described above, bits are allocated among the individual signals 548 and 550. When allocating the available bits, the number of bits used for the high-frequency reconstruction parameter 538 and the upmixing parameter 536 must, of course, be taken into account. Regarding the number of bits allocated in a specific time frame, care should be taken to adjust the psychoacoustic models used for the first and second waveform coding levels 506 and 508 to achieve a perceptually smooth transition around the first crossover frequency.

Figure 8 illustrates an alternative embodiment of the encoding system 800. The difference between the encoding system 800 of Figure 8 and the encoding system 500 of Figure 5 is that the encoder 800 is arranged to generate another waveform-encoded signal by waveform-encoded input signals 502, 504 for a frequency range corresponding to a subset of the frequency range above the first crossover frequency.

For this purpose, encoder 800 includes an interleaving detection stage 802. The interleaving detection stage 802 is configured to identify portions of input signals 502, 504 that were not well reconstructed by parametric reconstruction during encoding by parametric encoding stage 530 and high-frequency reconstruction encoding stage 532. For example, the interleaving detection stage 802 can compare the input signals 502, 504 with the parametric reconstructions of the input signals 502, 504 defined by parametric encoding stage 530 and high-frequency reconstruction encoding stage 532. Based on this comparison, the interleaving detection stage 802 can identify a subset 804 of the frequency range above a first crossover frequency to be waveform encoded. The interleaving detection stage 802 can also identify the time range within which the identified subset 804 of the frequency range above the first crossover frequency is to be waveform encoded. The identified frequency and time subsets 804, 806 can be input to the first waveform encoding stage 506. Based on the received frequency and time subsets 804 and 806, the first waveform encoding stage 506 generates another waveform encoded signal 808 by means of one or more of the waveform encoded input signals 502 and 504 for the time and frequency range identified by subsets 804 and 806. This other waveform encoded signal 808 can then be encoded and quantized by stage 520 and added to the bitstream 846.

The interleaving detection stage 802 may also include a control signal generation stage. The control signal generation stage is configured to generate a control signal 810 that indicates how the other waveform encoded signal should be parametrically re-interleaved with one of the input signals 502, 504 in the decoder. For example, as described with reference to FIG7, the control signal may indicate the frequency range and time range with which the other waveform encoded signal should be parametrically re-interleaved. The control signal may be added to the bitstream 846.

Equivalent, extended, substitute and others

After studying the above description, further embodiments of this disclosure will become apparent to those skilled in the art. While embodiments and examples are disclosed in this specification and accompanying drawings, this disclosure is not limited to these specific examples. Various modifications and variations may be made without departing from the scope of this disclosure as defined by the appended claims. Any reference numerals appearing in the claims should not be construed as limiting their scope.

Furthermore, based on a study of the accompanying drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing this disclosure. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" does not exclude a plurality. The mere fact that certain measures are stated in mutually different dependent claims does not indicate that a combination of these measures cannot be beneficial.

The systems and methods disclosed above can be implemented as software, firmware, hardware, or a combination thereof. In a hardware implementation, the division of tasks among the functional units mentioned above does not necessarily correspond to the division of physical units; rather, a physical component may have multiple functions, and a task may be performed collaboratively by several physical components. Some or all of the components may be implemented as software executed by a digital signal processor or microprocessor, or as hardware, or as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or temporary media). As is known to those skilled in the art, the term "computer storage medium" includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to: RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. In addition, those skilled in the art will know that communication media typically contain computer-readable instructions, data structures, program modules or other data in modulated data signals such as carrier waves or other transmission mechanisms, and include any information transmission medium.

Claims (12)

1. A method for decoding time frames of an encoded audio bitstream in an audio processing system, the method comprising: For each time frame, M upmixed signals (404) are received, wherein the M upmixed signals include spectral coefficients corresponding to frequencies higher than the first crossover frequency. The M upmixed signals are the result of upmixing N frequency extension downmixed signals obtained by performing frequency reconstruction above the second crossover frequency in the reconstruction range into M upmixed signals for the time frame, wherein the second crossover frequency is higher than the first crossover frequency and the frequency reconstruction uses reconstruction parameters derived from the encoded audio bitstream. For the time frame, another waveform encoded signal is extracted from the encoded audio bitstream, the other waveform encoded signal including spectral coefficients corresponding to a subset of frequencies higher than the first crossover frequency; and For the time frame, the other waveform encoded signal is interleaved with one of the M upmixing signals to generate an interleaved signal. 2. The method of claim 1, wherein the first crossover frequency depends on the bit transmission rate of the audio processing system. 3. The method of claim 1, wherein the interleaving comprises: (i) adding the other waveform encoded signal to one of the M upmixing signals, (ii) combining the other waveform encoded signal with one of the M upmixing signals, or (iii) replacing one of the M upmixing signals with the other waveform encoded signal. 4. The method of claim 1, wherein the frequency reconstruction is performed in the frequency domain. 5. The method of claim 1, further comprising receiving a control signal used during interleaving to generate an interleaved signal. 6. The method of claim 5, wherein the control signal indicates how to interleave the other waveform-coded signal with one of the M upmixing signals by specifying the frequency range or time range of the interleaving. 7. The method of claim 5, wherein the first value of the control signal indicates that interleaving is performed for the corresponding frequency region. 8. The method of claim 1, wherein the audio processing system is a hybrid decoder that performs waveform decoding and parameter decoding. 9. An audio decoder for decoding time frames of an encoded audio bitstream, the audio decoder comprising: An input terminal for receiving M upmixed signals (404) for a time frame, wherein the M upmixed signals include spectral coefficients corresponding to frequencies higher than a first crossover frequency. The M upmixed signals are the result of upmixing N frequency extension downmixed signals obtained by performing frequency reconstruction above the second crossover frequency in the reconstruction range into M upmixed signals for the time frame, wherein the second crossover frequency is higher than the first crossover frequency and the frequency reconstruction uses reconstruction parameters derived from the encoded audio bitstream. A demultiplexer is configured to extract, for the time frame, another waveform-coded signal from the encoded audio bitstream, the other waveform-coded signal comprising spectral coefficients corresponding to a subset of frequencies above the first crossover frequency; and An interleaver is used to interleave the other waveform-coded signal with one of the M upmixing signals for the time frame to produce an interleaved signal. 10. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, perform the method as described in any one of claims 1-8. 11. An apparatus for decoding time frames of an encoded audio bitstream in an audio processing system, comprising: Processor; and A memory storing instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-8. 12. An apparatus for decoding time frames of an encoded audio bitstream in an audio processing system, comprising units for performing each step of the method as described in any one of claims 1-8.