HK40102011A - Apparatus and method for stereo filling in multichannel coding - Google Patents

Description

Apparatus and methods for stereo fill in multichannel encoding

This application is a divisional application of the Chinese national application (application number: 201780023524.4, date of entry into the Chinese national phase: October 12, 2018) corresponding to the international application PCT/EP2017/053272 entitled "Apparatus and method for stereo fill in multichannel coding" filed on February 14, 2017.

Technical Field

This invention relates to audio signal encoding, and more particularly, to apparatus and methods for stereo filling in multichannel encoding.

Background Technology

Audio coding belongs to the field of compression and involves utilizing redundancy and uncorrelatedness in audio signals.

In MPEG USAC (see, for example [3]), joint stereo coding of two channels is performed using complex prediction, MPS2-1-2, or unified stereo with band-limited or full-band residual signals. MPEG surround (see, for example [4]) layer-wise combines one-to-two (OTT) and two-to-three (TTT) frames for joint coding of multichannel audio, regardless of whether residual signals are transmitted.

In MPEG-H, the four-channel elements are layered and applied with MPS2-1-2 stereo frames, followed by complex prediction/MS stereo frames, to construct a fixed 4×4 remix tree (see, for example, [1]).

AC4 (see, for example [6]) introduces new 3-channel, 4-channel, and 5-channel elements, which allow the transmitted channels to be remixed using only the transmitted mixing matrix and subsequent joint stereo coding information. In addition, previous publications have proposed using orthogonal transforms such as the Karhunen-Loeve transform (KLT) for enhanced multichannel audio coding (see, for example [7]).

For example, in 3D audio, speaker channels are distributed across several height levels, resulting in horizontal and vertical channel pairs. As defined in USAC, joint coding of only two channels is insufficient to account for the spatial and perceptual relationships between channels. MPEG surround is applied in additional pre-processing/post-processing steps to individually transmit residual signals when joint stereo coding is not possible, for example, to utilize the dependency between the left and right vertical residual signals. AC-4 introduced a dedicated N-channel element, which allows for efficient coding of joint coding parameters, but it was not suitable for general speaker setups with multiple channels proposed for new immersive playback scenarios (7.1+4, 22.2). The MPEG-H four-channel element is also limited to only four channels and cannot be dynamically applied to arbitrary channels, but only to a pre-configured and fixed number of channels.

The MPEG-H multichannel coding tool allows the generation of arbitrary trees of discrete coded stereo frames (i.e., jointly coded channel pairs), see [2].

A common problem in audio signal coding is caused by quantization (e.g., spectral quantization). Quantization can lead to spectral holes. For example, all spectral values in a particular frequency band can be set to zero on the encoder side as a result of quantization. For instance, the exact values of such spectral lines might be quite low before quantization, and then quantization might result in a situation where, for example, the spectral values of all spectral lines within a particular frequency band have been set to zero. When decoding, this can lead to undesirable spectral holes on the decoder side.

Modern frequency-domain speech/audio coding systems (e.g., the IETF's Opus/Celt codec [9], MPEG-4 (HE-)AAC [10], or particularly MPEG-D xHE-AAC (USAC) [11]) provide means of encoding audio frames using a long transform-long block or eight sequential short transform-short blocks, depending on the temporal stability of the signal. Furthermore, for low bit-rate coding, these schemes provide tools for reconstructing the frequency coefficients of a channel using pseudo-random noise or low-frequency coefficients from the same channel. In xHE-AAC, these tools are referred to as noise filling and spectral band duplication, respectively.

However, for very tonal or transient stereo inputs, individual noise padding and/or spectral band duplication limit the coding quality achievable at extremely low bit rates, primarily because many spectral coefficients from both channels need to be explicitly transmitted.

MPEG-H stereo fill is a parametric tool that improves the filling of spectral holes caused by quantization in the frequency domain by using downmixing of previous frames. Similar to noise fill, stereo fill operates directly in the MDCT domain of the MPEG-H core encoder, see [1], [5], [8].

However, the use of MPEG surround and stereo fill in MPEG-H is limited by fixed channel pairs, thus preventing the utilization of time-varying inter-channel dependencies.

The Multichannel Coding Tool (MCT) in MPEG-H allows for adaptation to various inter-channel dependencies, but stereo fill is not permitted due to the use of a single channel element in typical operating configurations. Existing techniques do not disclose perceptually optimized methods for generating downmixing of previous frames in the case of time-varying arbitrary jointly coded channel pairs. Combining MCT with noise fill as an alternative to stereo fill to fill spectral holes will result in noise artifacts, especially for tonal signals.

Summary of the Invention

The object of this invention is to propose an improved audio coding concept. This object is achieved by an apparatus for decoding according to an example embodiment of this application, an apparatus for encoding according to an example embodiment of this application, a method for decoding according to an example embodiment of this application, a method for encoding according to an example embodiment of this application, a computer program according to an example embodiment of this application, and by encoding multichannel signals according to an example embodiment of this application.

An apparatus is proposed for decoding an encoded multichannel signal of a current frame to obtain three or more current audio output channels. A multichannel processor is adapted to select two decoded channels from the three or more decoded channels according to a first multichannel parameter. Furthermore, the multichannel processor is adapted to generate a first set of two or more processed channels based on the selected channels. A noise filling module is adapted to identify one or more frequency bands within at least one of the selected channels where all spectral lines are quantized to zero, and is adapted to generate a mixed channel using an appropriate subset of the three or more previously decoded audio output channels according to auxiliary information, and is adapted to fill the spectral lines of the frequency bands within the mixed channel with noise generated using the spectral lines of the mixed channel.

According to an embodiment, an apparatus is provided for decoding a previously encoded multichannel signal of a previous frame to obtain three or more previous audio output channels and for decoding a currently encoded multichannel signal of a current frame to obtain three or more current audio output channels.

The device includes an interface, a channel decoder, a multichannel processor for generating the three or more current audio output channels, and a noise filling module.

The interface is adapted to receive the currently encoded multichannel signal and to receive auxiliary information including first multichannel parameters.

The channel decoder is adapted to decode the currently encoded multichannel signal of the current frame to obtain a set of three or more decoded channels for the current frame.

The multichannel processor is adapted to select a first selected pair of two decoded channels from the set of three or more decoded channels based on the first multichannel parameter.

Furthermore, the multichannel processor is adapted to generate a first set of two or more processed channels based on the first selected two decoded channel pairs to obtain an updated set of three or more decoded channels.

Before the multichannel processor generates the first pair of two or more processed channels based on the first selected two decoded channel pairs, the noise filling module is adapted to identify one or more frequency bands in which all spectral lines of the first selected two decoded channel pairs are quantized to zero for at least one of the two channels, and is adapted to generate a mixed channel using two or more, but not all, of the three or more previous audio output channels, and is adapted to fill the spectral lines of the one or more frequency bands in which all spectral lines of the mixed channel are quantized to zero with noise generated using the spectral lines of the mixed channel, wherein the noise filling module is adapted to select two or more previous audio output channels from the three or more previous audio output channels for generating the mixed channel according to the auxiliary information.

The specific concept of an embodiment of how a noise filling module can generate and fill noise is referred to as stereo filling.

Furthermore, an apparatus for encoding multichannel signals having at least three channels is proposed.

The apparatus includes an iterative processor adapted to, in a first iterative step, calculate interchannel correlation values between each pair of channels in the at least three channels, to select, in the first iterative step, a channel pair having the highest value or a value above a threshold, and to process the selected channel pair using a multichannel processing operation to derive initial multichannel parameters of the selected channel pair and derive the channel of the first processing.

The iterative processor is adapted to perform the calculation, selection, and processing in a second iterative step using at least one of the processed channels to derive other multichannel parameters and a second processed channel.

Furthermore, the device includes a channel encoder adapted to encode the channels obtained by the iterative processing performed by the iterative processor to obtain encoded channels.

Furthermore, the device includes an output interface adapted to generate an encoded multichannel signal having the encoded channels, the initial multichannel parameters, and the other multichannel parameters, and having information indicating whether the decoding device should fill the spectral lines of one or more frequency bands where all spectral lines are quantized to zero with noise generated based on previously decoded audio output channels that have previously been decoded by the decoding device.

Furthermore, a method is proposed for decoding a previously encoded multichannel signal of a previous frame to obtain three or more previous audio output channels and for decoding a currently encoded multichannel signal of the current frame to obtain three or more current audio output channels. The method includes:

- Receive the currently encoded multichannel signal and receive auxiliary information including the first multichannel parameters.

- Decode the currently encoded multichannel signal of the current frame to obtain a set of three or more decoded channels for the current frame.

- Select a first pair of two decoded channels from the set of three or more decoded channels according to the first multi-channel parameter.

- Generate a first set of two or more processed channels based on the first selected two decoded channel pairs to obtain an updated set of three or more decoded channels.

Before generating the first pair of two or more processed channels based on the first selected two decoded channel pairs, the following steps are performed:

- For at least one of the two channels of the first selected two decoded channel pairs, identify one or more frequency bands in which all spectral lines within the channel are quantized to zero, and generate a mixed channel using two or more, but not all, of the three or more previous audio output channels, and fill the spectral lines of the one or more frequency bands in which all spectral lines within the channel are quantized to zero with noise generated using the spectral lines of the mixed channel, wherein two or more previous audio output channels are selected from the three or more previous audio output channels to generate the mixed channel according to the auxiliary information.

Furthermore, a method for encoding multichannel signals having at least three channels is proposed. The method includes:

- In the first iteration step, the interchannel correlation value between each pair of channels in the at least three channels is calculated, which is used to select the channel pair with the highest value or a value higher than a threshold in the first iteration step, and the selected channel pair is processed using a multichannel processing operation to derive the initial multichannel parameters for the selected channel pair and derive the channel of the first processing.

- In the second iteration step, the calculation, selection, and processing are performed using at least one of the processed channels to derive other multichannel parameters and the second processed channel.

- The audio channels obtained from the iterative processing performed by the iterative processor are encoded to obtain encoded audio channels.

- Generate an encoded multichannel signal having the encoded channels, the initial multichannel parameters, and the other multichannel parameters, and having information indicating whether the decoding device needs to fill the spectral lines of one or more frequency bands where all spectral lines are quantized to zero with noise generated based on previously decoded audio output channels that have been previously decoded by the decoding device.

Furthermore, a computer program is proposed, wherein each of the computer programs is configured to implement one of the above methods when executed on a computer or signal processor, such that each of the above methods is implemented by one of the computer programs.

Furthermore, an encoded multichannel signal is proposed. The encoded multichannel signal includes encoded channels and multichannel parameters, as well as information indicating whether the decoding device needs to fill the spectral lines of one or more frequency bands where all spectral lines are quantized to zero with spectral data generated based on previously decoded audio output channels that have been previously decoded by the decoding device.

Attached Figure Description

In the following, embodiments of the present invention will be described in further detail with reference to the accompanying drawings, in which:

Figure 1a illustrates an apparatus for decoding according to one embodiment;

Figure 1b illustrates an apparatus for decoding according to another embodiment;

Figure 2 shows a block diagram of a parametric frequency domain decoder according to an embodiment of this application;

Figure 3 shows a schematic diagram illustrating the spectral sequence of the spectrum of the channels that form a multi-channel audio signal, in order to facilitate understanding of the description of the decoder in Figure 2.

Figure 4 shows a schematic diagram illustrating the current spectrum in the spectrum diagram shown in Figure 3 to help understand the description in Figure 2;

Figures 5a and 5b show block diagrams of a parametric frequency domain audio decoder according to an alternative embodiment, which uses downmixing of previous frames as the basis for interchannel noise filling.

Figure 6 shows a block diagram of a parametric frequency domain audio encoder according to one embodiment;

Figure 7 shows a schematic block diagram of an apparatus for encoding a multichannel signal having at least three channels according to one embodiment;

Figure 8 shows a schematic block diagram of an apparatus for encoding a multichannel signal having at least three channels according to one embodiment;

Figure 9 shows a schematic block diagram of a stereo frame according to one embodiment;

Figure 10 shows a schematic block diagram of an apparatus for decoding an encoded multichannel signal having encoded channels and at least two multichannel parameters according to one embodiment.

Figure 11 shows a flowchart of a method for encoding a multichannel signal having at least three channels according to one embodiment;

Figure 12 shows a flowchart of a method for decoding an encoded multichannel signal having encoded channels and at least two multichannel parameters according to one embodiment;

Figure 13 illustrates a system according to one embodiment;

Figure 14 illustrates the generation of the combined audio channels for a first frame in scenario (a) and the generation of the combined audio channels for a second frame following the first frame in scenario (b) according to one embodiment; and

Figure 15 illustrates a retrieval scheme for multi-channel parameters according to an embodiment.

In the following description, the same or equivalent reference numerals are used to denote the same or equivalent elements or elements having the same or equivalent functions.

Detailed Implementation

In the following description, numerous details are set forth to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail so as not to obscure embodiments of the invention. Furthermore, unless otherwise specifically indicated, features of the different embodiments described below can be combined with each other.

Before describing the decoding apparatus 201 of FIG1a, noise filling for multichannel audio encoding will first be described. In an embodiment, the noise filling module 220 of FIG1a may, for example, be configured to perform one or more of the techniques described below for noise filling for multichannel audio encoding.

Figure 2 illustrates a frequency domain audio decoder according to one embodiment of this application. The decoder is generally indicated by reference numeral 10 and includes a scaling factor band identifier 12, a dequantizer 14, a noise filler 16, an inverse transformer 18, a spectral extractor 20, and a scaling factor extractor 22. Optional additional elements that the decoder 10 may include cover a complex stereo predictor 24, an MS (middle-side) decoder 26, and an inverse time noise shaping (TNS) filter tool 28, two examples of which 28a and 28b are shown in Figure 2. Furthermore, a downmixing provider is shown in more detail below using reference numeral 30, and its outline is depicted.

The frequency-domain audio decoder 10 of Figure 2 is a parametric decoder that supports noise filling. It fills a scaling factor band with noise according to a scaling factor of a certain zero-quantization scaling factor band as a means of controlling the level of noise filled in that scaling factor band. Furthermore, the decoder 10 of Figure 2 represents a multi-channel audio decoder configured to reconstruct a multi-channel audio signal from the input data stream 30. However, Figure 2 focuses on the elements of the decoder 10 involved in reconstructing one of the multi-channel audio signals encoded into the data stream 30 and outputting this (output) channel at output 32. Reference numeral 34 indicates that the decoder 10 may include additional elements or may include some pipeline operation control responsible for reconstructing other channels of the multi-channel audio signal, wherein the following description indicates how the reconstruction of the channel of interest by the decoder 10 at output 32 interacts with the decoding of other channels.

The multichannel audio signal represented by data stream 30 may include two or more channels. In the following description of embodiments of this application, the focus is on the stereo case where the multichannel audio signal includes only two channels; however, in principle, the embodiments presented below can be readily adapted to alternative embodiments involving multichannel audio signals comprising more than two channels and their encoding.

As will become clearer from the following description of Figure 2, the decoder 10 in Figure 2 is a transform decoder. In other words, according to the encoding technique of decoder 10, the channels are encoded in the transform domain, for example, using an overlap transform of the channels. Furthermore, depending on the audio signal generating device, there exist time phase deviations between each other due only to small or decisive variations (during which the channels of the audio signal primarily represent the same audio content). These variations are, for example, different amplitudes and/or phases to represent audio scenes where the differences between channels allow the audio source of the audio scene to be virtually located relative to the virtual speaker positions associated with the output channels of the multi-channel audio signal. However, at some other time phases, the different channels of the audio signal may be more or less unrelated to each other and may even, for example, represent completely different audio sources.

To account for possible time-varying relationships between the channels of the audio signal, the audio codec of decoder 10 in Figure 2 allows for the use of time-varying measurements of different quantities to take advantage of inter-channel redundancy. For example, MS encoding allows switching between representing the left and right channels of the stereo audio signal as themselves, or representing them as a pair of M (center) and S (side) channels representing the downmixing of the left and right channels and their halving difference, respectively. In other words, there are spectrum diagrams of two channels that are continuously (in the spectral time sense) transmitted by data stream 30, but the meaning of these (transmitted) channels can change over time and relative to the output channels.

Complex stereo prediction (another tool for utilizing inter-channel redundancy) enables the prediction of frequency domain coefficients or spectral lines of another channel using common localization lines on the spectrum of one channel. More details on this will be described below.

To aid in understanding the subsequent description of Figure 2 and the components shown therein, Figure 3 illustrates, for an exemplary case of a stereo audio signal represented by data stream 30, a possible method for encoding sample values of the spectral lines of the two channels into data stream 30 for processing by the decoder 10 of Figure 2. Specifically, while the upper half of Figure 3 depicts a spectrum diagram 40 of the first channel of the stereo audio signal, the lower half illustrates a spectrum diagram 42 of the other channel of the stereo audio signal. Moreover, it is noteworthy that the “meaning” of spectrum diagrams 40 and 42 can change over time due to, for example, time-varying switching between the MS coding domain and the non-MS coding domain. In the first case, spectrum diagrams 40 and 42 relate to the M and S channels, respectively, while in the latter case, they relate to the left and right channels. The switching between the MS coding domain and the non-MS coding domain can be signaled in data stream 30.

Figure 3 illustrates how spectrograms 40 and 42 can be encoded into data stream 30 with a time-varying spectral time resolution. For example, the (transmit) channels can be subdivided in a time-aligned manner into a sequence of frames indicated by braces 44, which can be of equal length and adjacent to each other without overlapping. As previously stated, the spectral resolution represented by spectrograms 40 and 42 in data stream 30 can change over time. Initially, it is assumed that the spectral time resolution changes identically over time for spectrograms 40 and 42, but this simplified extension is also possible, as will become apparent from the following description. For example, the change in spectral time resolution can be signaled in data stream 30 in units of frame 44. In other words, the spectral time resolution changes in units of frame 44. The change in the spectral time resolution of spectrograms 40 and 42 is achieved by switching the number and length of the transforms used to describe spectrograms 40 and 42 within each frame 44. In the examples of Figure 3, frames 44a and 44b exemplarily illustrate frames in which the audio signals have been sampled for each channel using a long transform, resulting in the highest spectral resolution, where each channel has one spectral line sample value for each spectral line per frame. In Figure 3, small crosses within boxes indicate the sample values of the spectral lines, where these boxes are arranged in rows and columns and represent a spectral time grid, with each row corresponding to one spectral line and each column corresponding to a sub-interval of frame 44 corresponding to the shortest transform involved in forming the spectral diagrams 40 and 42. In particular, Figure 3 illustrates, for example, frame 44d, a frame that can alternately undergo successive transforms of shorter lengths, thereby resulting in several temporally subsequent spectra with reduced spectral resolution for frames such as frame 44d. Using eight short transforms for frame 44d as an example results in spectral time sampling of spectrograms 40 and 42 within frame 42d at spectral lines spaced apart from each other, such that only every seven spectral lines are filled, but with sample values of each of the eight transform windows of shorter length used to transform frame 44d, or transforms. For illustrative purposes, Figure 3 shows that other numbers of transforms for a frame are also possible, for example, using two transforms whose transform length is, for example, half the transform length of the long transforms used for frames 44a and 44b, thereby obtaining spectral time grids or sampling of spectrograms 40 and 42, where two spectral line sample values are obtained every other spectral line, one involving the first transform and the other involving the last transform.

The transform windows used for transforming, in which frames are subdivided, are illustrated below each spectrogram in Figure 3 using overlapping window lines. Temporal overlap is used, for example, for TDAC (Temporal Alias Cancellation) purposes.

While the embodiments described below can also be implemented in another manner, Figure 3 illustrates the case in which switching between different spectral time resolutions for an individual frame 44 is performed in such a way that for each frame 44, spectrum diagrams 40 and 42 obtain the same number of spectral line values indicated by the small crosses in Figure 3, the only difference being the way these lines are sampled in spectral time in the manner corresponding to the corresponding spectral time tiles of the corresponding frame 44, which span the time of the corresponding frame 44 in time and span the frequency from zero to the maximum frequency _fmax in the spectrum.

Using the arrows in Figure 3, Figure 3 illustrates, for frame 44d, how a similar spectrum can be obtained for all frames 44 by appropriately distributing the sample values of spectral lines belonging to the same spectral line but with short transform windows within a frame of one channel across the unoccupied (empty) spectral lines of that frame until the next occupied spectral line in the same frame. This resulting spectrum is referred to below as the “interleaved spectrum.” When interleaving n transforms of a frame of one channel, for example, the spectral line values that are commonly located on the n spectra of the subsequent n short transforms of the spectral lines follow each other before the set of spectral line values that are commonly located on the n spectra of the subsequent spectral lines follows one another. An intermediate form of interleaving is also feasible: instead of interleaving all the spectral line coefficients of a frame, it would be feasible to interleave only the spectral line coefficients of a suitable subset of the short transforms of frame 44d. In summary, whenever discussing the spectra of the frames of two channels corresponding to the spectrum diagrams 40 and 42, these spectra can refer to either interleaved or non-interleaved spectra.

To efficiently encode the spectral line coefficients representing spectrograms 40 and 42 via the data stream 30 sent to decoder 10, these spectral line coefficients are quantized. To control quantization noise in a spectral-temporal manner, the quantization order size is controlled via a scaling factor set in a certain spectral time grid. Specifically, within each spectral sequence of each spectrogram, spectral lines are grouped into consecutive, non-overlapping scaling factor groups on the spectrum. Figure 4 shows spectrum 46 of spectrogram 40 and the synchronous spectrum 48 of spectrogram 42 in its upper half. As shown, spectrograms 46 and 48 are subdivided into scaling factor bands along the spectral axis f to group the spectral lines into non-overlapping groups. The scaling factor bands are illustrated in Figure 4 by braces 50. For simplicity, it is assumed that the boundaries between the scaling factor bands coincide between spectrograms 46 and 48, but this is not necessarily the case.

That is, by encoding with data stream 30, both spectrograms 40 and 42 are subdivided into time series of spectra, and each of these spectra is further subdivided into scaling factor bands. For each scaling factor band, data stream 30 encodes or transmits information about the scaling factor corresponding to the corresponding scaling factor band. The spectral coefficients falling within the corresponding scaling factor band 50 are quantized using the corresponding scaling factor, or, considering decoder 10, they can be dequantized using the scaling factor of the corresponding scaling factor band.

Before returning to Figure 2 and its description, it is assumed below that the specially processed channel, that is, the channel whose decoding involves a specific element (other than 34) of the decoder in Figure 2, is the transmit channel of the spectrum diagram 40, which, as mentioned above, can represent one of the left and right channels, the M channel, or the S channel, wherein it is assumed that the multichannel audio signal encoded into data stream 30 is a stereo audio signal.

While spectral line extractor 20 is configured to extract spectral data from data stream 30, i.e., the spectral coefficients of frame 44, scale factor extractor 22 is configured to extract the corresponding scale factor for each frame 44. For this purpose, extractors 20 and 22 can use entropy decoding. According to one embodiment, scale factor extractor 22 is configured to sequentially extract, for example, the scale factor of spectrum 46 in FIG. 4, i.e., the scale factor of scale factor band 50, from data stream 30 using context-adaptive entropy decoding. The order of sequential decoding may follow a spectral order defined in the scale factor band, for example, from low to high frequencies. Scale factor extractor 22 can use context-adaptive entropy decoding and may determine the context of each scale factor based on scale factors already extracted in the spectral neighborhood of the currently extracted scale factor, such as depending on the scale factors of the immediately preceding scale factor band. Alternatively, scale factor extractor 22 may predictively decode the scale factor from data stream 30, for example, using differential decoding, while predicting the currently decoded scale factor based on any scale factors of previously decoded scale factors (e.g., immediately preceding scale factors). It is worth noting that the scaling factor extraction process is agnostic for scaling factor bands that belong to bands exclusively filled by zero-quantized spectral lines or those filled by at least one of them quantized to a non-zero value. Scaling factors belonging to scaling factor bands filled only by zero-quantized spectral lines can serve as both a basis for predicting subsequently decoded scaling factors that may belong to scaling factor bands filled by spectral lines (one of which is non-zero) and a basis for prediction based on previously decoded scaling factors that may belong to scaling factor bands filled by spectral lines (one of which is non-zero).

For completeness only, note that the spectral line extractor 20 extracts spectral line coefficients, similarly using entropy coding and/or predictive coding to fill the scaling factor band 50 with these coefficients. Entropy coding can use context adaptability based on spectral line coefficients in the spectral temporal neighborhood of the currently decoded spectral line coefficient; similarly, prediction can be a spectral prediction, temporal prediction, or spectral-temporal prediction of the currently decoded spectral line coefficient based on previously decoded spectral line coefficients in its spectral temporal neighborhood. To improve coding efficiency, the spectral line extractor 20 can be configured to perform decoding of spectral lines or line coefficients in tuples, collecting or grouping spectral lines along the frequency axis.

Therefore, at the output of the spectral line extractor 20, the spectral line coefficients are provided, for example, in units of a spectrum such as spectrum 46, which collects, for example, all spectral line coefficients of the corresponding frame, or alternatively, all spectral line coefficients of certain short transforms of the corresponding frame. At the output of the scaling factor extractor 22, the corresponding scaling factor of the corresponding spectrum is output instead.

The scaling factor band identifier 12 and the dequantizer 14 have spectral line inputs coupled to the output of the spectral line extractor 20, and the dequantizer 14 and the noise filler 16 have scaling factor inputs coupled to the output of the scaling factor extractor 22. The scaling factor band identifier 12 is configured to identify a so-called zero-quantization scaling factor band within the current spectrum 46, i.e., a scaling factor band in which all spectral lines are quantized to zero, such as scaling factor band 50c in FIG. 4, and the remaining scaling factor bands of the spectrum in which at least one spectral line is quantized to a non-zero value. Specifically, in FIG. 4, the shaded areas in FIG. 4 are used to indicate spectral line coefficients. As can be seen from this figure, in spectrum 46, all scaling factor bands (except scaling factor band 50b) have at least one spectral line whose spectral line coefficient is quantized to a non-zero value. It will become clear later that zero-quantization scaling factor bands such as 50d form the object of inter-channel noise filling, which will be described further below. Before proceeding, note that the scale factor band identifier 12 can be limited to an appropriate subset of scale factor bands 50, such as scale factor bands above a certain starting frequency 52. In Figure 4, this limits the identification process to scale factor bands 50d, 50e, and 50f.

Scale factor band identifier 12 informs noise filler 16 about these scale factor bands that serve as zero-quantization scale factor bands. Dequantizer 14 uses the scale factor associated with the input spectrum 46 to dequantize or scale the spectral line coefficients of the spectrum 46 according to the associated scale factor, i.e., the scale factor associated with scale factor band 50. In particular, dequantizer 14 uses the scale factor associated with the corresponding scale factor band to dequantize and scale the spectral line coefficients falling within the corresponding scale factor band. Figure 4 should be interpreted as illustrating the dequantization results of the spectral lines.

Noise filler 16 obtains information relating to the zero-quantization scaling factor band (which forms the object of the noise fill below), the dequantization spectrum, and the scaling factor of at least these scaling factor bands identified as zero-quantization scaling factor bands, as well as a signal notification obtained from the data stream 30 of the current frame indicating whether interchannel noise fill should be performed for the current frame.

The interchannel noise filling process described in the example below actually involves two types of noise filling: inserting the background noise 54, which involves all spectral lines that have been quantized to zero (and is independent of their potential members), into any zero-quantization scaling factor band, and the actual interchannel noise filling process. Although this combination is described below, it should be emphasized that the insertion of the background noise can be omitted according to alternative embodiments. Furthermore, the signal notifications concerning the start and stop of noise filling in the current frame and obtained from data stream 30 can be related only to interchannel noise filling, or can control a combination of the two noise filling types together.

Regarding the background noise insertion, the noise filler 16 can operate as follows. Specifically, the noise filler 16 can employ artificial noise generation, such as a pseudo-random number generator or some other random source, to fill the spectral lines with zero spectral coefficients. The level of the background noise 54 thus inserted at the zero-quantization spectral line can be set according to explicit signaling within the data stream 30 for the current frame or current spectrum 46. The “level” of the background noise 54 can be determined using, for example, root mean square (RMS) or energy measurements.

Therefore, the background noise insertion represents a pre-filling for these scaling factor bands (e.g., scaling factor band 50d in Figure 4) that have been identified as zero-quantization scaling factor bands. It also affects other scaling factor bands beyond the zero-quantization scaling factor bands, but the former are further subjected to the following inter-channel noise filling. As described below, the inter-channel noise filling process is used to fill the zero-quantization scaling factor bands up to the level controlled by the scaling factor of the corresponding zero-quantization scaling factor band. The former can be used directly for this purpose because all spectral lines of the corresponding zero-quantization scaling factor band are quantized to zero. Nevertheless, the data stream 30 may include additional signaling for each frame or each spectrum 46 containing parameters, which are typically applied to the scaling factors of all zero-quantization scaling factor bands of the corresponding frame or spectrum 46, and when applied to the scaling factors of the zero-quantization scaling factor bands via the noise filler 16, this results in a separate corresponding fill level for each zero-quantization scaling factor band. In other words, noise filler 16 can use the same modification function to modify the scaling factor of the corresponding scaling factor band for each zero-quantization scaling factor band of spectrum 46, using the aforementioned parameters included in data stream 30 for spectrum 46 of the current frame, in order to obtain the target filling level of the corresponding zero-quantization scaling factor band as measured in terms of energy or RMS. For example, the inter-channel noise filling process should fill the corresponding zero-quantization scaling factor band to the level achieved by (optionally) adding noise (in addition to the background noise 54).

Specifically, to perform inter-channel noise filling 56, the noise filler 16 obtains the spectral colocalization portion of the spectrum 48 of another channel after it has been largely or fully decoded, and copies the obtained portion of spectrum 48 to a zero-quantization scaling factor band. This portion is colocalized spectrally and scaled in such a way that the total noise level generated within the zero-quantization scaling factor band, obtained by integrating the spectral lines of the corresponding scaling factor band, is equal to the aforementioned target filling level obtained from the scaling factor of the zero-quantization scaling factor band. Through this measure, the tone of the noise filled into the corresponding zero-quantization scaling factor band is improved compared to artificially generated noise (e.g., the noise that forms the basis of the noise floor 54), and is also superior to uncontrolled spectral copying/replication 46 from extremely low frequency lines within the same spectrum 46.

More precisely, for a current frequency band such as 50d, the noise filler 16 locates a spectral colocalization portion within the spectrum 48 of another channel, scaling its spectral lines according to the zero-quantization scaling factor band 50d in a manner that optionally involves some additional offset or noise factor parameters of the current frame or spectrum 46 contained in the data stream 30, as just described, such that the result fills the corresponding zero-quantization scaling factor band 50d up to the desired level defined by the scaling factor of the zero-quantization scaling factor band 50d. In this embodiment, this means that the filling is performed additively relative to the noise floor 54.

According to a simplified embodiment, the resulting noise-filled spectrum 46 is directly input to the input of the inverse transformer 18 to obtain the time-domain portion of the corresponding channel audio time signal for each transform window to which the spectral line coefficients of spectrum 46 belong. These time-domain portions can then be combined using an overlap-add process (not shown in Figure 2). That is, if spectrum 46 is a non-interleaved spectrum whose spectral line coefficients belong to only one transform, the inverse transformer 18 performs this transform, producing a time-domain portion, and its beginning and end are subjected to an overlap-add process, where the preceding and following time-domain portions are obtained by inverse transforming the preceding and following inverse transforms to achieve, for example, time-domain aliasing cancellation. However, if spectrum 46 has been interleaved into the spectral line coefficients of more than one consecutive transform, the inverse transformer 18 performs a separate inverse transform on it so that each inverse transform yields a time-domain portion, and these time-domain portions are subjected to an overlap-add process between them according to the time order defined therein, as is the case for the preceding and following time-domain portions of other spectra or frames.

However, for completeness, it must be noted that further processing can be performed on the noise-filled spectrum. As shown in Figure 2, an inverse TNS filter can perform inverse TNS filtering on the noise-filled spectrum. That is, controlled by the TNS filter coefficients of the current frame or spectrum 46, the spectrum obtained so far is linearly filtered along the spectral direction.

With or without inverse TNS filtering, the complex stereo predictor 24 can treat the spectrum as the prediction residual of inter-channel prediction. More specifically, the inter-channel predictor 24 can use the spectral co-localization portion of another channel to predict spectrum 46 or at least a subset of its scaling factor band 50. The complex prediction process is illustrated in FIG4 with dashed box 58 regarding scaling factor band 50b. That is, data stream 30 may contain inter-channel prediction parameters that control, for example, which of the scaling factor bands 50 should be inter-channel predicted in this manner and which should not. Furthermore, the inter-channel prediction parameters in data stream 30 may also include complex inter-channel prediction factors applied by inter-channel predictor 24 to obtain inter-channel prediction results. These factors may be included in data stream 30 for each scaling factor band individually, or alternatively in data stream 30 for each group consisting of one or more scaling factor bands, wherein inter-channel prediction is activated or signaled to be activated in data stream 30 for each group.

As shown in Figure 4, the source of inter-channel prediction can be the spectrum 48 of another channel. More precisely, the source of inter-channel prediction can be the spectral colocalization portion of spectrum 48, which colocalizes to the scaling factor band 50b to extend and perform inter-channel prediction by estimating its imaginary part. The estimation of the imaginary part can be performed based on the spectral colocalization portion 60 of spectrum 48 itself, and/or the downmixing of the decoded channel from a previous frame (i.e., the frame immediately preceding the currently decoded frame to which spectrum 46 belongs) can be used. In practice, the inter-channel predictor 24 adds the prediction signal obtained as just described to the scaling factor band to which inter-channel prediction is to be performed, such as the scaling factor band 50b in Figure 4.

As noted in the preceding description, the channel to which spectrum 46 belongs can be an MS-coded channel or a speaker-related channel, such as the left or right channel of a stereo audio signal. Therefore, optionally, MS decoder 26 performs MS decoding on the optional inter-channel predicted spectrum 46, similarly performing addition or subtraction on each spectral line or spectrum 46 corresponding to the spectral line of the other channel corresponding to spectrum 48. For example, although not shown in Figure 2, spectrum 48 as shown in Figure 4 is obtained by part 34 of decoder 10 in a manner similar to the description above regarding the channel to which spectrum 46 belongs, and MS decoding module 26, in performing MS decoding, subjectes spectra 46 and 48 to spectral line addition or subtraction, where spectra 46 and 48 are at the same stage of the processing, meaning, for example, that both have already been obtained through inter-channel prediction, or both have just been obtained through noise filling or inverse TNS filtering.

Note that, optionally, MS decoding can be performed in units of, for example, scaling factor band 50, either individually activated by data stream 30 or globally involving the entire spectrum 46. In other words, MS decoding can be started or stopped using the corresponding signals in data stream 30, at, for example, frames or some finer spectral temporal resolution (e.g., scaling factor bands of spectra 46 and/or 48 individually for spectrum diagrams 40 and/or 42), where it is assumed that the same boundaries of the scaling factor bands for both channels are defined.

As shown in Figure 2, the inverse TNS filtering of the inverse TNS filter 28 can also be performed after any inter-channel processing, such as inter-channel prediction 58 or MS decoding by the MS decoder 26. Performance before or downstream of inter-channel processing can be fixed or controlled by the corresponding signal in each frame of the data stream 30, or at some other granularity level. Wherever inverse TNS filtering is performed, the corresponding TNS filter coefficients present in the data stream of the current spectrum 46 control the TNS filter, i.e., the linear prediction filter running along the spectral direction, to linearly filter the spectrum input to the corresponding inverse TNS filter modules 28a and/or 28b.

Therefore, the spectrum 46 reaching the input of the inverse transformer 18 may have already undergone further processing as described above. Similarly, the above description does not imply that all these optional tools are either present simultaneously or absent. These tools may exist partially or collectively within the decoder 10.

In any case, the spectrum generated at the input of the inverter represents the final reconstruction of the channel output signal and forms the basis for the aforementioned downmixing of the current frame, as described with respect to complex prediction 58, which serves as the basis for estimating the potential imaginary part of the next frame to be decoded. It can also be used for inter-channel prediction of the final reconstruction of another channel, rather than the channel involved in the elements in Figure 2 other than 34.

By combining the final spectrum 46 with the corresponding final version of spectrum 48, the downmixer 31 forms a corresponding downmixer. The latter, namely the corresponding final version of spectrum 48, forms the basis for the inter-channel prediction in predictor 24.

Figures 5a and 5b illustrate an alternative to Figure 2, where the basis for inter-channel noise filling is a downmixing representation of spectral lines co-located in the previous frame's spectrum. This allows the source of the complex inter-channel prediction to be used twice, both as a source for inter-channel noise filling and as a source for estimating the imaginary part in the complex inter-channel prediction, in the optional case of using complex inter-channel prediction. Figures 5a and 5b show a decoder 10, including a section 70 relating to the decoding of the first channel to which spectrum 46 belongs, and the internal structure of the aforementioned other section 34 relating to the decoding of another channel including spectrum 48. The same reference numerals are used for the internal elements of section 70 on one hand, and for 34 on the other. It can be seen that the structure is the same. At output 32, one channel of the stereo audio signal is output, and at the output of the inverse converter 18 of the second decoder section 34, another (output) channel of the stereo audio signal is generated, indicated by reference numeral 74. Similarly, the above embodiment can be readily adapted to cases using more than two channels.

Downmixer 31 is shared by portions 70 and 34 and receives spectra 48 and 46, which are temporally co-located in spectrograms 40 and 42, in order to form a downmix based on these spectra on a spectral line basis, possibly by averaging the sum at each spectral line by dividing the sum by the number of channels being downmixed (i.e., two channels in the cases of Figures 5a and 5b). At the output of downmixer 31, the downmix of the previous frame is obtained through this measurement. In this respect, it is noted that if the previous frame contains more than one spectrum in either of spectrograms 40 and 42, there are different possibilities regarding how downmixer 31 should operate in that case. For example, in this case, downmixer 31 may use the spectrum of the tail transform of the current frame, or it may use the interleaving result of all spectral line coefficients of the current frame in interleaved spectrograms 40 and 42. The delay element 74 shown in Figures 5a and 5b as connected to the output of the downmixer provider 31 indicates that the downmixing provided at the output of the downmixer provider 31 forms the downmixing of the previous frame 76 (see Figure 4, with respect to interchannel noise fill 56 and complex prediction 58, respectively). Therefore, the output of the delay element 74 is connected on one hand to the input of the interchannel predictor 24 of the decoder sections 34 and 70, and on the other hand to the input of the noise filler 16 of the decoder sections 70 and 34.

That is, although in Figure 2, the noise filler 16 receives the temporally co-located spectrum 48 of the final reconstruction of another channel in the same current frame as the basis for inter-channel noise filling, in Figures 5a and 5b, inter-channel noise filling is performed based on the downmixing of the previous frame provided by the downmixer provider 31. The manner in which inter-channel noise filling is performed remains unchanged. That is, the inter-channel noise filler 16 extracts the co-located portion of the spectrum from the corresponding spectrum of the other channel in the current frame (in the case of Figure 2), and from the final decoded spectrum obtained from the previous frame representing the downmixing of the previous frame (in the cases of Figures 5a and 5b), and adds the same "source" portion to the spectral line within the scaling factor band (e.g., 50d in Figure 4) to be scaled according to the target noise level determined by the scaling factor of the corresponding scaling factor band.

Concluding the above discussion of embodiments describing interchannel noise filling in an audio decoder, it will be apparent to those skilled in the art that certain preprocessing can be applied to the "source" spectrum without departing from the overall concept of interchannel filling before adding the spectral or temporally co-located portion of the "source" spectrum to the spectral lines of the "target" scaling factor band. In particular, it may be advantageous to apply filtering operations (e.g., spectral flattening or skew removal) to the spectral lines of the "source" region to be added to the "target" scaling factor band (50d in Figure 4) to improve the audio quality of the interchannel noise filling process. Similarly, and as an example of a largely (but not fully) decoded spectrum, the aforementioned "source" portion can be obtained from a spectrum that has not yet been filtered with an available inverse (i.e., synthesized) TNS filter.

Therefore, the above embodiments relate to the concept of inter-channel noise filling. Hereinafter, the possibility of applying the above-described inter-channel noise filling concept to existing codecs (i.e., xHE-AAC) in a semi-backward compatible manner is described. Specifically, a preferred embodiment of the above embodiments is described below, according to which a stereo filler tool is applied to an xHE-AAC-based audio codec in a semi-backward compatible signaling manner. By using the embodiments further described below, stereo filler of the transform coefficients in either of the two channels of an MPEG-D xHE-AAC (USAC)-based audio codec is feasible for certain stereo signals, thereby improving the encoding quality of certain audio signals, especially at low bit rates. The stereo filler tool is signaled in a semi-backward compatible manner, allowing a conventional xHE-AAC decoder to parse and decode the bitstream without noticeable audio errors or loss. As mentioned above, better overall quality can be obtained if the audio encoder can reconstruct the zero-quantized (non-transmitted) coefficients of either of the currently decoded channels using a combination of previously decoded/quantized coefficients from the two stereo channels. Therefore, in addition to spectral band copying (from low to high frequency channel coefficients) and noise filling in the audio encoder (especially xHE-AAC or its-based encoders) (from unrelated pseudo-random sources), it is desirable to allow this stereo filling (from previous to current channel coefficients).

To allow conventional xHE-AAC decoders to read and parse bitstreams encoded with stereo fill, the required stereo fill tool should be used in a semi-backward compatible manner: its presence should not cause conventional decoders to stop or even fail to start decoding. The readability of the bitstream within the xHE-AAC infrastructure also promotes market adoption.

In order to achieve the aforementioned desire for semi-backward compatibility with the stereo fill tool in the case of xHE-AAC or its potential derivatives, the following implementation relates to the functionality of stereo fill and its ability to be signaled by syntax in the data stream that is actually related to noise fill. The stereo fill tool will operate as described above. In a channel pair with a common window configuration, when the stereo fill tool is activated, the coefficients of the zero quantization scaling factor band are reconstructed as a substitute for noise fill (or, as described above, with noise fill added) by the sum or difference of the coefficients of the previous frame in either of the two channels (preferably the right channel). Stereo fill is performed similarly to noise fill. The signaling is completed by the noise fill signaling of xHE-AAC. The stereo fill is transmitted via 8-bit noise fill auxiliary information. This is feasible because the MPEG-D USAC standard [3] specifies that all 8 bits should be sent even if the noise level to be applied is zero. In this case, some noise fill bits can be reused for the stereo fill tool.

The semi-backward compatibility of bitstream parsing and playback performed by the conventional xHE-AAC decoder is ensured as follows. Stereo fill is signaled by including auxiliary information for the stereo fill tool and the zero noise level (i.e., the first three noise fill bits, all with zero values) after five non-zero bits (traditionally representing noise offset). Since the conventional xHE-AAC decoder ignores the 5-bit noise offset value when the 3-bit noise level is zero, the presence of the stereo fill tool signaling only affects noise fill in the conventional decoder: noise fill is disabled due to the first three bits being zero, and the rest of the decoding operation proceeds as expected. In particular, stereo fill is not performed because it operates similarly to a disabled noise fill process. Therefore, the conventional decoder still provides "graceful" decoding of the enhanced bitstream 30 because it does not need to mute the output signal or even abort decoding upon reaching a frame where stereo fill is initiated. However, naturally, compared to decoding by a suitable decoder capable of properly handling the new stereo fill tool, a correct and expected reconstruction of the stereo-filled line coefficients cannot be provided, resulting in a deterioration in the quality of the affected frames. Nevertheless, assuming the stereo fill tool is used as intended, i.e. only for low bitrate stereo input, the quality through the xHE-AAC decoder should be better than if affected frames are lost due to mute or cause other noticeable playback errors.

The following section will describe in detail how to build the stereo fill tool into the xHE-AAC codec as an extension.

When incorporated into the standard, the stereo fill tool can be described as follows. Specifically, this stereo fill (SF) tool will represent a new tool in the frequency domain (FD) portion of MPEG-H 3D audio. Based on the above discussion, the purpose of this stereo fill tool is to perform parametric reconstruction of MDCT spectral coefficients at a low bit rate, similar to what can already be achieved with noise fill according to Section 7.2 of the standard described in [3]. However, unlike noise fill which uses pseudo-random noise sources to generate MDCT spectral values for any FD channel, SF can also be used to reconstruct the MDCT value of the right channel of a jointly coded stereo channel pair using downmixing of the left and right MDCT spectra of the previous frame. According to the implementation described below, SF is signaled semi-backward compatiblely by noise fill auxiliary information that can be correctly resolved by a conventional MPEG-DUSAC decoder.

The tool can be described as follows. When SF is active in a joint stereo FD frame, the MDCT coefficients of the empty (i.e., completely zero-quantized) scaling factor band of the right (second) channel (e.g., 50d) are replaced by the sum or difference of the MDCT coefficients of the corresponding decoded left and right channels of the previous frame (if it is FD). If conventional noise padding is active for the second channel, a pseudo-random value is also added to each coefficient. The resulting coefficients for each scaling factor band are then scaled such that the RMS (root of the square of the average coefficients) of each band matches the value sent through the scaling factor of that band. See Section 7.3 of the standard in [3].

Some operational constraints can be provided for the use of the new SF tool in the MPEG-D USAC standard. For example, the SF tool can only be used in the right FD channel of a common FD channel pair, i.e., by sending the channel pair element of StereoCoreToolInfo() with common_window == 1. Furthermore, due to semi-backward compatible signaling, the SF tool can only be used when noiseFilling == 1 in the syntax container UsacCoreConfig(). If any channel in the pair is in LPDcore_mode, the SF tool cannot be used even if the right channel is in FD mode.

The following terms and definitions are used to more clearly describe the extensions to the standard described in [3].

Specifically, regarding data elements, the following new data elements have been introduced:

The stereo_filling binary flag indicates whether stereo filling is used in the current frame and channel.

In addition, a new help element has been introduced:

`noise_offset` is the noise fill offset, used to modify the scaling factor of the zero-quantization band (Section 7.2).

noise_level: Noise fill level, indicating the magnitude of the added spectral noise (Section 7.2).

downmix_prev[] The downmix (i.e., sum or difference) of the left and right channels of the previous frame.

sf_index[g][sfb] The scaling factor index of window group g and bandwidth sfb (i.e., the integer to be transmitted).

The standard decoding process will be extended as follows. Specifically, decoding of the jointly stereo encoded FD channels will be performed in three consecutive steps when the SF tool is activated:

First, the stereo_filling flag will be decoded.

`stereo_filling` does not represent an independent bitstream element, but is derived from the noise fill element `noise_offset` and `noise_level` in `UsacChannelPairElement()` and the `common_window` flag in `StereoCoreToolInfo()`. If `noiseFilling == 0` or `common_window == 0` or the current channel is the left (first) channel in the element, then `stereo_filling` is 0, and the stereo fill process ends. Otherwise,

if((noiseFilling!=0)&&(common_window!=0)&&(noise_level==0)){

stereo_filling=(noise_offset&16)/16;

noise_level=(noise_offset&14)/2;

noise_offset=(noise_offset&1)*16;

}

else{

stereo_filling = 0;

}

In other words, if noise_level == 0, then noise_offset contains the stereo_filling flag, followed by 4 bits of noise padding data, which are then rearranged. Because this operation changes the values of noise_level and noise_offset, it needs to be performed before the noise padding process in Section 7.2. Furthermore, the above pseudocode is not executed in the left (first) channel of UsacChannelPairElement() or any other element.

Then, downmix_prev will be calculated.

`downmix_prev[]` will perform spectral downmixing for stereo fill, identical to `dmx_re_prev[]` used for MDST spectral estimation in complex stereo prediction (see Section 7.7.2.3). This means...

• If any channel of the element and frame in which it performs the downmix (i.e., the frame before the currently decoded frame) uses core_mode == 1 (LPD) or the channel uses unequal transform lengths (split_transform == 1 or only block switching in one channel window_sequence == EIGHT_SHORT_SEQUENCE) or usacIndependencyFlag == 1, then all coefficients of downmix_prev[] must be zero.

• If the channel transformation length in the current element changes from the last frame to the current frame (i.e., split_transform == 0 before split_transform == 1, or window_sequence == EIGHT_SHORT_SEQUENCE before window_sequence != EIGHT_SHORT_SEQUENCE, and vice versa), then all downmix_prev[] coefficients must be zero during stereo fill.

• If transform splitting is applied to a channel in the previous or current frame, then downmix_prev[] represents downmixing with progressive interleaving. For more information, see the Transform Split Tool.

• If complex stereo prediction is not used in the current frame and elements, then pred_dir equals 0.

Therefore, the downmixing only needs to be calculated once for both tools, thus reducing complexity. The only difference between downmix_prev[] and dmx_re_prev[] in Section 7.7.2 is the behavior when complex stereo prediction is not currently used, or when it is active but use_prev_frame == 0. In this case, downmix_prev[] is calculated for stereo fill decoding according to Section 7.7.2.3, even though complex stereo prediction decoding does not require dmx_re_prev[] and therefore it is undefined/zero.

After that, stereo fill of the empty scale factor band will be performed.

If stereo_filling == 1, then after the noise filling process, the following procedure is performed in all initial empty scale factor bands sfb[] below max_sf__ste (i.e., all bands where all MDCT spectral lines are quantized to zero). First, the energy of a given sfb[] and the corresponding spectral lines in downmix_prev[] are calculated by the sum of squares of the spectral lines. Thus, a given sfbWidth contains the number of spectral lines for each sfb[].

if(energy[sfb]<sfbwidth[sfb]){/*noiselevelisn′t maximum, or bandstarts below noise-fill region*/

facDmx=sqrt((sfbwidth[sfb]-energy[sfb])/energy_dmx[sfb]);

factor = 0.0;

/*if the previous downmix isn’t empty, add the scaled downmix linesssuch that band reaches unity energy*/

for(index=swb_offset[sfb]; index<swb_offset[sfb+1]; index++){

spectrum[window][index]+=downmix_prev[window][index]*facDmx;

factor+=spectrum[window][index]*spectrum[window][index];

}

if((factor!=sfbWidth[sfb])&&(factor＞0)){/*unity energy isn’treached,so modify band＊/

factor=sqrt(sfbwidth[sfb]/(factor+1e-8));

for(index=swb_offset[sfb]; index<swb_offset[sfb+1]; index++){

spectrum[window][index]*=factor;

}

}

}

For the spectrum of each window group, a scaling factor is then applied to the resulting spectrum as described in Section 7.3, where the scaling factor for empty frequency bands is treated the same as the regular scaling factor.

An alternative to the aforementioned extensions to the xHE-AAC standard would be to use an implicit semi-backward compatible signaling approach.

The above implementation in the xHE-AAC code framework describes a method that uses a single bit in the bitstream to signal the decoder the use of a new stereo filler tool contained in stereo_filling, according to Figure 2. More precisely, this signaling (let's call it explicit semi-backward compatible signaling) allows the following conventional bitstream data (here, noise filler auxiliary information) to be used independently of the SF signaling notification: In this embodiment, the noise filler data is independent of the stereo filler information, and vice versa. For example, noise filler data consisting of all zeros (noise_level = noise_offset = 0) can be sent, while stereo_filling can be signaled with any possible value (a binary flag, 0 or 1).

In cases where strict independence between conventional bitstream data and the bitstream data of this invention is not required, and where the signaling of this invention is a binary decision, explicit transmission of signaling bits can be avoided. The binary decision can be signaled by the presence or absence of what can be termed implicit semi-backward compatible signaling. Again, taking the above embodiment as an example, the use of stereo filler can be sent simply by employing new signaling: if noise_level is zero and noise_offset is not zero, the stereo_filling flag is set to 1. If both noise_level and noise_offset are not zero, stereo_filling equals 0. When both noise_level and noise_offset are zero, the implicit signaling depends on the conventional noise filler signal. In this case, it is unclear whether conventional or new SF implicit signaling is used. To avoid this ambiguity, the value of stereo_filling must be defined beforehand. In this example, defining stereo_filling = 0 is appropriate if the noise filler data consists of all zeros, because this is what a conventional encoder without stereo filler capability signals when no noise filler is applied in the frame.

The remaining problem in the case of implicit semi-backward compatible signaling is how to simultaneously signal stereo_filling == 1 and no noise filling. As mentioned above, the noise filling data cannot be all zero, and if zero noise amplitude is required, noise_level((noise_offset&14)/2, as mentioned above) must be equal to 0. This leaves only noise_offset((noise_offset&1)*16, as mentioned above) being greater than 0 as a solution. However, even if noise_level is zero, noise_offset is still considered in the case of stereo filling when the scaling factor is applied. Fortunately, encoders can compensate for the fact that a noise_offset that may not be sent to zero by changing the affected scaling factor so that, when writing the bitstream, they include the offset undone in the decoder with noise_offset. This allows the implicit signaling in the above embodiments at the cost of a potential increase in the scaling factor data rate. Therefore, the stereo filling signaling in the pseudocode described above can be modified as follows to send 2 bits (4 values) of noise_offset instead of 1 bit using the saved SF signaling bits:

if((noiseFilling)&&(common_window)&&(noise_level==0)&&

(noise_offset＞0){

stereo_filling = 1;

noise_level=(noise_offset&28)/4;

noise_offset=(noise_offset&3)*8;

}

else{

stereo_filling = 0;

}

For completeness, Figure 6 illustrates a parametric audio encoder according to an embodiment of this application. First, the encoder of Figure 6, generally denoted by reference numeral 90, includes a transducer 92 for performing a transformation of the original, undistorted version of the audio signal reconstructed at output 32 of Figure 2. As described with respect to Figure 3, an overlapping transformation can be used, where switching between different transform lengths and corresponding transform windows occurs in frames. Different transform lengths and corresponding transform windows are shown in Figure 3 using reference numeral 104. Similar to Figure 2, Figure 6 focuses on the portion of the encoder 90 responsible for encoding one channel of the multi-channel audio signal, while the other channel domain portion of the decoder 90 is generally denoted by reference numeral 96 in Figure 6.

At the output of converter 92, the spectral lines and scaling factors are unquantized, and there is essentially no coding loss. The spectrogram output from converter 92 enters quantizer 98, which is configured to quantize the spectral lines of the spectrogram output from converter 92 spectrally, setting and using the initial scaling factor of the scaling factor band. That is, at the output of quantizer 98, the initial scaling factor and the corresponding spectral line coefficients are obtained, and a series of noise fillers 16′, optional inverse TNS filters 28a′, inter-channel predictors 24′, MS decoders 26′, and inverse TNS filters 28b′ are sequentially connected to provide encoder 90 of Figure 6 with the ability to obtain a reconstructed final version of the current spectrum as available on the decoder side at the input of the downmixer (see Figure 2). When using inter-channel prediction 24′ and/or using inter-channel noise filler in a version formed by downmixing the previous frame, encoder 90 also includes a downmixer provider 31′ to form a reconstructed final version of the spectrum of the channels of the multi-channel audio signal. Of course, to save computation, instead of the final version, downmixer provider 31′ can use the original, unquantized version of the spectrum of the channels to form the downmix.

Encoder 90 may use information related to the final version of the available reconstructed spectrum to perform inter-frame spectral prediction, such as performing the aforementioned possible version of inter-channel prediction using imaginary part estimation, and/or to perform rate control, i.e. to determine possible parameters in the rate control loop that are set in the best sense of rate/distortion by encoder 90 to ultimately encode into data stream 30.

For example, for each zero-quantization scaling factor band identified by identifier 12', one such parameter set in this prediction loop and/or rate control loop of encoder 90 is the scaling factor of the corresponding scaling factor band, which is only initially set by quantizer 98. In the prediction and/or rate control loop of encoder 90, the scaling factor of the zero-quantization scaling factor band is set in some psychoacoustic or rate/distortion-optimal sense to determine the aforementioned target noise level and optional modification parameters also transmitted to the decoder side by the data stream of the corresponding frame as described above. It should be noted that the scaling factor can be calculated using only its associated spectrum and channel (i.e., the "target" spectrum as previously described), or alternatively, it can be determined using the spectral lines of the "target" channel spectrum and, in addition, the spectral lines of the downmixed spectrum from the previous frame (i.e., the "source" spectrum as previously described) or the spectral lines of another channel spectrum obtained from downmixer provider 31'. Specifically, to stabilize the target noise level and reduce temporal level fluctuations in the decoded audio channels with inter-channel noise filling, the target scaling factor can be calculated using the relationship between the energy measurements of spectral lines in the "target" scaling factor band and the energy measurements of spectral lines co-located in the corresponding "source" region. Finally, as mentioned above, the "source" region can originate from the reconstructed final version of another channel or the downmixing of a previous frame, or, if reducing encoder complexity, from the downmixing of the initial unquantized version of that other channel or the initial unquantized version of the spectrum of a previous frame.

The following explains multichannel encoding and multichannel decoding according to embodiments. In embodiments, the multichannel processor 204 of the device 201 for decoding of FIG1a may be configured, for example, to perform one or more of the techniques described below regarding noisy multichannel decoding.

However, before describing multichannel decoding, multichannel encoding according to the embodiment will be explained with reference to Figures 7 to 9, and multichannel decoding will be explained with reference to Figures 10 and 12.

Now, referring to Figures 7 through 9 and Figure 11, the multi-channel encoding according to the embodiment will be explained:

Figure 7 shows a schematic block diagram of an apparatus (encoder) 100 for encoding a multi-channel signal 101 having at least three channels CH1 to CH3.

The device 100 includes an iterative processor 102, a channel encoder 104, and an output interface 106.

The iterative processor 102 is configured to calculate, in a first iteration step, interchannel correlation values between each pair of channels in at least three channels CH1 to CH3, to select the channel pair with the highest value or a value above a threshold, and to process the selected channel pair using multichannel processing operations to derive the multichannel parameter MCH_PAR1 of the selected channel pair and to derive the channels P1 and P2 of the first processing. Hereinafter, such processed channels P1 and P2 may also be referred to as combined channels P1 and P2, respectively. Furthermore, the iterative processor 102 is configured to perform calculation, selection, and processing in a second iteration step using at least one of the processed channels P1 or P2 to derive the multichannel parameter MCH_PAR2 and the channels P3 and P4 of the second processing.

For example, as shown in Figure 7, the iterative processor 102 may calculate in the first iterative step: the interchannel correlation value between a first pair of at least three channels CH1 to CH3, the first pair consisting of the first channel CH1 and the second channel CH2; the interchannel correlation value between a second pair of at least three channels CH1 to CH3, the second pair consisting of the second channel CH2 and the third channel CH3; and the interchannel correlation value between a third pair of at least three channels CH1 to CH3, the third pair consisting of the first channel CH1 and the third channel CH3.

In Figure 7, it is assumed that in the first iteration step, the third pair consisting of the first channel CH1 and the third channel CH3 includes the highest interchannel correlation value, such that the iteration processor 102 selects the third pair with the highest interchannel correlation value in the first iteration step and processes the selected channel pair (i.e., the third pair) using multichannel processing operation to derive the multichannel parameter MCH_PAR1 of the selected channel pair and derive the channels P1 and P2 of the first processing.

Furthermore, the iterative processor 102 can be configured in the second iteration step to calculate interchannel correlation values between each pair of at least three channels CH1 to CH3 and processed channels P1 and P2, in order to select the channel pair with the highest interchannel correlation value or a value above a threshold in the second iteration step. Thus, the iterative processor 102 can be configured not to select the channel pair selected in the first iteration step in the second iteration step (or in any other iteration step).

Referring to the example shown in Figure 7, the iterative processor 102 can also calculate the interchannel correlation value between the fourth channel pair consisting of the first channel CH1 and the first processed channel P1, the interchannel correlation value between the fifth channel pair consisting of the first channel CH1 and the second processed channel P2, the interchannel correlation value between the sixth channel pair consisting of the second channel CH2 and the first processed channel P1, the interchannel correlation value between the seventh channel pair consisting of the second channel CH2 and the second processed channel P2, the interchannel correlation value between the eighth channel pair consisting of the third channel CH3 and the first processed channel P1, the interchannel correlation value between the ninth channel pair consisting of the third channel CH3 and the second processed channel P2, and the interchannel correlation value between the tenth channel pair consisting of the first processed channel P1 and the second processed channel P2.

In Figure 7, it is assumed that in the second iteration step, the sixth channel pair consisting of the second channel CH2 and the first processed channel P1 includes the highest inter-channel correlation value, such that the iteration processor 102 selects the sixth channel pair in the second iteration step and uses multi-channel processing operation to process the selected channel pair (i.e., the sixth pair) to derive the multi-channel parameter MCH_PAR2 of the selected channel pair and derive the second processed channels P3 and P4.

The iterative processor 102 can be configured to select a channel pair only when the horizontal difference between the channel pairs is less than a threshold, which is less than 40dB, 25dB, 12dB, or less than 6dB. Therefore, the threshold of 25dB or 40dB corresponds to a rotation angle of 3 or 0.5 degrees.

The iterative processor 102 can be configured to compute a normalized integer correlation value, wherein the iterative processor 102 can be configured to select a channel pair when the integer correlation value is greater than, for example, 0.2 or preferably 0.3.

Furthermore, the iterative processor 102 can provide the channel encoder 104 with the channels obtained through multi-channel processing. For example, referring to FIG7, the iterative processor 102 can provide the channel encoder 104 with the third processed channel P3 and the fourth processed channel P4 obtained through multi-channel processing performed in the second iteration step, and the second processed channel P2 obtained through multi-channel processing performed in the first iteration step. Therefore, the iterative processor 102 can only provide the channel encoder 104 with those processed channels that were not (further) processed in subsequent iteration steps. As shown in FIG7, the first processed channel P1 is not provided to the channel encoder 104 because it was further processed in the second iteration step.

The channel encoder 104 can be configured to encode channels P2 to P4 obtained by iterative processing (or multi-channel processing) performed by the iterative processor 102 to obtain encoded channels E1 to E3.

For example, the channel encoder 104 can be configured to encode channels P2 to P4 obtained through iterative processing (or multi-channel processing) using mono encoders (or mono frames or mono tools) 120_1 to 120_3. The mono frames can be configured to encode channels such that fewer bits are required to encode channels with less energy (or smaller amplitude) compared to encoding channels with more energy (or higher amplitude). Mono frames 120_1 to 120_3 can be, for example, transform-based audio encoders. Furthermore, the channel encoder 104 can be configured to encode channels P2 to P4 obtained through iterative processing (or multi-channel processing) using a stereo encoder (e.g., a parametric stereo encoder or a lossy stereo encoder).

Output interface 106 can be configured to generate encoded multichannel signals 107 with encoded channels E1 to E3 and multichannel parameters MCH_PAR1 and MCH_PAR2.

For example, output interface 106 can be configured to generate an encoded multichannel signal 107 as a serial signal or serial bit stream, such that multichannel parameter MCH_PAR2 precedes multichannel parameter MCH_PAR1 in the encoded signal 107. Therefore, the decoder (an embodiment of which will be described later with reference to FIG10) will receive multichannel parameter MCH_PAR2 before multichannel parameter MCH_PAR1.

In Figure 7, the iterative processor 102 exemplarily performs two multichannel processing operations: the multichannel processing operation in the first iteration step and the multichannel processing operation in the second iteration step. Of course, the iterative processor 102 can also perform additional multichannel processing operations in subsequent iteration steps. Thus, the iterative processor 102 can be configured to execute iterative steps until an iteration termination criterion is reached. The iteration termination criterion can be a maximum number of iteration steps equal to or greater than the total number of channels of the multichannel signal 101 by 2, or wherein the iteration termination criterion is when the inter-channel correlation value does not have a value greater than a threshold, preferably greater than 0.2 or preferably 0.3. In another embodiment, the iteration termination criterion can be a maximum number of iteration steps equal to or greater than the total number of channels of the multichannel signal 101, or wherein the iteration termination criterion is when the inter-channel correlation value does not have a value greater than a threshold, preferably greater than 0.2 or preferably 0.3.

For illustrative purposes, the multi-channel processing operations performed by the iterative processor 102 in the first and second iteration steps are exemplarily shown in FIG. 7 by processing blocks 110 and 112. Processing blocks 110 and 112 can be implemented in hardware or software. For example, processing blocks 110 and 112 can be stereo blocks.

Therefore, the inter-channel signal dependencies can be utilized by applying known joint stereo coding tools in a layered manner. In contrast to previous MPEG methods, the signal pairs to be processed are not predetermined by fixed signal paths (e.g., stereo coding trees), but can be dynamically changed to adapt to the characteristics of the input signals. The input to the actual stereo frame can be (1) unprocessed channels, such as channels CH1 to CH3, (2) the output of the previous stereo frame, such as processed signals P1 to P4, or (3) a combination of unprocessed channels and the output of the previous stereo frame.

The processing inside the stereo frames 110 and 112 can be prediction-based (such as complex prediction frames in USAC) or KLT/PCA-based (the input channel is rotated in the encoder (e.g., by a 2×2 rotation matrix) to maximize energy compression, i.e., to concentrate the signal energy into one channel, and in the decoder, the rotated signal is re-transformed back to the original input signal direction).

In a possible implementation of encoder 100, (1) the encoder calculates the interchannel correlation between each channel pair, selects a suitable signal pair from the input signals, and applies a stereo tool to the selected channel; (2) the encoder recalculates the interchannel correlation between all channels (unprocessed channels and processed intermediate output channels), selects a suitable signal pair from the input signals, and applies a stereo tool to the selected channel; (3) the encoder repeats step (2) until the interchannel correlation of all channels is below a threshold or if the maximum number of transformations is applied.

As already mentioned, the signal pairs processed by encoder 100, or more precisely iterative processor 102, are not predetermined by a fixed signal path (e.g., a stereo coding tree), but can be dynamically changed to adapt to the characteristics of the input signal. Thus, encoder 100 (or iterative processor 102) can be configured to construct a stereo tree based on at least three channels CH1 to CH3 of the multichannel (input) signal 101. In other words, encoder 100 (or iterative processor 102) can be configured to construct the stereo tree based on inter-channel correlation (e.g., by calculating the inter-channel correlation value between each pair of at least three channels CH1 to CH3 in a first iteration step to select the channel pair with the highest value or a value above a threshold in the first iteration step, and by calculating the inter-channel correlation value between each pair of at least three channels and the previously processed channels in a second iteration step to select the channel pair with the highest value or a value above a threshold in the second iteration step). According to the one-step method, a correlation matrix containing the correlations of all possible processed channels in the previous iterations can be computed for each possible iteration.

As described above, the iterative processor 102 can be configured to derive multichannel parameters MCH_PAR1 for the selected channel pair in a first iteration step and MCH_PAR2 for the selected channel pair in a second iteration step. Multichannel parameter MCH_PAR1 may include an identifier (or signaling) of a first channel pair identifier (or index) of the channel pair selected in the first iteration step, while multichannel parameter MCH_PAR2 may include an identifier (or signaling) of a second channel pair identifier (or index) of the channel pair selected in the second iteration step.

The following describes the effective indexing of the input signal. For example, channel pairs can be efficiently signaled using a unique index for each channel pair based on the total number of channels. For example, the indexes for six-channel channel pairs can be shown in the following table:

For example, in the table above, index 5 can signal the channel pair consisting of the first and second channels. Similarly, index 6 can signal the channel pair consisting of the first and third channels.

The total number of possible channel pair indices for n channels can be calculated as follows:

numPairs＝numChannels*(numChannels-1)/2

Therefore, the number of bits required to signal a channel pair is:

numBits＝floor(log ₂ (numPairs-1))+1

Furthermore, encoder 100 can use channel masks. The configuration of a multi-channel tool can include a channel mask indicating which channel the tool is active for. Therefore, LFE (Low Frequency Effects/Enhanced Channels) can be removed from the channel pair index, allowing for more efficient encoding. For example, for an 11.1 setting, this reduces the number of channel pair indices from 12 * 11 / 2 = 66 to 11 * 10 / 2 = 55, allowing signaling with 6 bits instead of 7 bits. This mechanism can also be used to exclude channels intended for mono objects (e.g., multilingual tracks). During decoding of the channel mask, a channel map can be generated to allow remapping of channel pair indices to decoder channels.

Furthermore, the iterative processor 102 can be configured to derive multiple selected channel pair indications for the first frame, wherein the output interface 106 can be configured to include a hold indicator in the multichannel signal 107 for a second frame following the first frame, indicating that the second frame has the same multiple selected channel pair indications as the first frame.

The hold indicator or hold tree flag can be used to signal that no new tree has been sent, but the last stereo tree should be used. This can be used to avoid multiple transmissions of the same stereo tree configuration if channel-related attributes remain unchanged for a long period of time.

Figure 8 shows a schematic block diagram of stereo frames 110 and 112. Stereo frames 110 and 112 include input terminals for a first input signal I1 and a second input signal I2, and output terminals for a first output signal O1 and a second output signal O2. As shown in Figure 8, the correlation between the output signals O1 and O2 and the input signals I1 and I2 can be described by s-parameters S1 to S4.

The iterative processor 102 may use (or include) stereo frames 110, 112 to perform multichannel processing operations on the input channels and/or processed channels to derive (further) processed channels. For example, the iterative processor 102 may be configured to use rotating stereo frames 110, 112 based on general prediction or KLT (Karhunen-Loève transform).

A general-purpose encoder (or encoder-side stereo frame) can be configured to encode input signals I1 and I2 based on the following equation to obtain output signals O1 and O2:

In decoding channel masks, channel maps can be generated.

A universal decoder (or decoder-side stereo frame) can be configured to decode input signals I1 and I2 to obtain output signals O1 and O2 based on the following equation:

A prediction-based encoder (or encoder-side stereo frame) can be configured to encode input signals I1 and I2 to obtain output signals O1 and O2 based on the following equation:

Where p is the prediction coefficient.

A prediction-based decoder (or decoder-side stereo frame) can be configured to decode input signals I1 and I2 to obtain output signals O1 and O2 based on the following equation:

A KLT-based rotary encoder (or encoder-side stereo frame) can be configured to decode input signals I1 and I2 to obtain output signals O1 and O2 based on the following equation:

A KLT-based rotating decoder (or decoder-side stereo frame) can be configured to decode input signals I1 and I2 to obtain output signals O1 and O2 based on the following equation (inverse rotation):

The calculation of the rotation angle α based on KLT is described below.

The rotation angle α based on KLT can be defined as:

c_{_xy} are entries in the unnormalized correlation matrix, where c _₁₁ and c _₂₂ are the tract energies.

This can be achieved using the atan2 function, allowing for the differentiation between negative correlations in the numerator and negative energy differences in the denominator:

alpha=0.5*atan2(2*correlation[ch1][ch2], (correlation[ch1][ch1]-correlation[ch2][ch2]));

Furthermore, the iterative processor 102 can be configured to use frames of each channel comprising multiple frequency bands to calculate inter-channel correlations, thereby obtaining individual inter-channel correlation values for multiple frequency bands, wherein the iterative processor 102 can be configured to perform multi-channel processing on each of the multiple frequency bands, such that multi-channel parameters are obtained from each of the multiple frequency bands.

Therefore, the iterative processor 102 can be configured to compute stereo parameters in multi-channel processing, wherein the iterative processor 102 can be configured to perform stereo processing only in a frequency band, wherein the stereo parameters are above a threshold defined by a stereo quantizer (e.g., a KLT-based rotary encoder) at which quantization is zero. The stereo parameters can be, for example, MS on/off, rotation angle, or prediction coefficients.

For example, iterative processor 102 can be configured to calculate rotation angles in multi-channel processing, wherein iterative processor 102 can be configured to perform rotation processing only in a frequency band where the rotation angle is above a threshold defined by a rotation angle quantizer (e.g., a KLT-based rotation encoder) to which quantization is zero.

Therefore, encoder 100 (or output interface 106) can be configured to send transformation/rotation information as a parameter (full-band frame) of the complete spectrum or as multiple frequency-related parameters as part of the spectrum.

Encoder 100 can be configured to generate bitstream 107 based on the following table:

Table 1 - Syntax of mpegh3daExtElementConfig()

Table 2 - Syntax of MCCConfig()

Table 3 - Syntax of MultichannelCodingBoxBandWise()

Table 4 - Syntax of MultichannelCodingBoxFullband()

Table 5 - Syntax of MultichannelCodingFrame()

Table 6 - Values of usacExtElementType

Table 7 - Explanation of data blocks used for extended payload decoding

Figure 9 shows a schematic block diagram of the iterative processor 102 according to an embodiment. In the embodiment shown in Figure 9, the multi-channel signal 101 is a 5.1-channel signal with six channels: left channel L, right channel R, left surround channel Ls, right surround channel Rs, center channel C, and low-frequency effect channel LFE.

As shown in Figure 9, the iterative processor 102 does not process the LFE channel. This could be because the interchannel correlation value between the LFE channel and each of the other five channels L, R, Ls, Rs, and C is too small, or because the channel mask indicates that the LFE channel is not processed, as will be assumed below.

In the first iteration step, the iterative processor 102 calculates the interchannel correlation value between each pair of the five channels L, R, Ls, Rs, and C to select the channel pair with the highest value or a value above a threshold in the first iteration step. In Figure 9, assuming that the left channel L and the right channel R have the highest values, the iterative processor 102 processes the left channel L and the right channel R using a stereo frame (or stereo tool) 110 that performs multichannel operation processing to derive the first processed channel P1 and the second processed channel P2.

In the second iteration step, the iterative processor 102 calculates the interchannel correlation values between each pair of the five channels L, R, Ls, Rs, and C and the processed channels P1 and P2, in order to select the channel pairs with the highest values or values above a threshold in the second iteration step. In Figure 9, assuming that the left surround channel Ls and the right surround channel Rs have the highest values, the iterative processor 102 processes the left surround channel Ls and the right surround channel Rs using a stereo frame (or stereo tool) 112 to derive the third processed channel P3 and the fourth processed channel P4.

In the third iteration step, the iterative processor 102 calculates the interchannel correlation values between each pair of the five channels L, R, Ls, Rs, and C and the processed channels P1 to P4, in order to select the channel pairs with the highest values or values above a threshold in the third iteration step. In Figure 9, assuming that the first processed channel P1 and the third processed channel P3 have the highest values, the iterative processor 102 processes the first processed channel P1 and the third processed channel P3 using a stereo frame (or stereo tool) 114 to derive the fifth processed channel P5 and the sixth processed channel P6.

In the fourth iteration step, the iterative processor 102 calculates the interchannel correlation values between each pair of the five channels L, R, Ls, Rs, and C and the processed channels P1 to P6, in order to select the channel pairs with the highest values or values above a threshold in the fourth iteration step. In Figure 9, assuming that the fifth processed channel P5 and the center channel C have the highest values, the iterative processor 102 processes the fifth processed channel P5 and the center channel C using a stereo frame (or stereo tool) 115 to derive the seventh processed channel P7 and the eighth processed channel P8.

Stereo frames 110 to 116 can be MS stereo frames, i.e., center/side stereo frames configured to provide a center channel and side channels. The center channel can be the sum of the input channels of the stereo frame, while the side channels can be the difference between the input channels of the stereo frame. Furthermore, stereo frames 110 and 116 can be rotated frames or stereo prediction frames.

In Figure 9, the first processing channel P1, the third processing channel P3, and the fifth processing channel P5 can be the middle channel, while the second processing channel P2, the fourth processing channel P4, and the sixth processing channel P6 can be the side channels.

Furthermore, as shown in Figure 9, the iterative processor 102 can be configured to use the input channels L, R, Ls, Rs, and C, and (only) the middle channels P1, P3, and P5 of the processed channels in the second iteration step, and, if applicable, perform calculations, selections, and processing in any additional iteration steps. In other words, the iterative processor 102 can be configured not to use the side channels P1, P3, and P5 of the processed channels in the second iteration step, and, if applicable, in the calculations, selections, and processing in any additional iteration steps.

Figure 11 shows a flowchart of a method 300 for encoding a multichannel signal having at least three channels. Method 300 includes: step 302, calculating interchannel correlation values between each pair of at least three channels in a first iteration step, selecting a channel pair with the highest value or a value above a threshold in the first iteration step, and processing the selected channel pair using a multichannel processing operation to derive multichannel parameters MCH_PAR1 for the selected channel pair and derive a first-processed channel; step 304, performing calculation, selection, and processing in a second iteration step using at least one processed channel to derive multichannel parameters MCH_PAR2 and a second-processed channel; step 306, encoding the channels obtained through iterative processing performed by the iterative processor to obtain encoded channels; and step 308, generating an encoded multichannel signal with encoded channels and encoded first and multichannel parameters MCH_PAR2.

Multichannel decoding is explained below.

Figure 10 shows a schematic block diagram of an apparatus (decoder) 200 for decoding a multi-channel signal 107 encoded with encoded channels E1 to E3 and at least two multi-channel parameters MCH_PAR1 and MCH_PAR2.

The device 200 includes a channel decoder 202 and a multi-channel processor 204.

The channel decoder 202 is configured to decode the encoded channels E1 to E3 to obtain the decoded channels D1 to D3.

For example, the channel decoder 202 may include at least three mono decoders (or mono boxes or mono tools) 206_1 to 206_3, wherein each of the mono decoders 206_1 to 206_3 may be configured to decode one of at least three coded channels E1 to E3 to obtain the corresponding decoded channels E1 to E3. The mono decoders 206_1 to 206_3 may be, for example, transform-based audio decoders.

The multichannel processor 204 is configured to perform multichannel processing using a second pair of decoded channels identified by the multichannel parameter MCH_PAR2 to obtain processed channels, and is configured to perform further multichannel processing using a first pair of channels identified by the multichannel parameter MCH_PAR1, wherein the first pair of channels includes at least one processed channel.

As illustrated in Figure 10, the multichannel parameter MCH_PAR2 can indicate (or signal) that the second decoded channel pair consists of the first decoded channel D1 and the second decoded channel D2. Therefore, the multichannel processor 204 uses the second decoded channel pair (identified by the multichannel parameter MCH_PAR2) consisting of the first decoded channel D1 and the second decoded channel D2 and performs multichannel processing using the multichannel parameter MCH_PAR2 to obtain processed channels P1* and P2*. The multichannel parameter MCH_PAR1 can indicate the first decoded channel pair consisting of the first processed channel P1* and the third decoded channel D3. Therefore, the multichannel processor 204 uses the first decoded channel pair (identified by the multichannel parameter MCH_PAR1) consisting of the first processed channel P1* and the third decoded channel D3 and performs further multichannel processing using the multichannel parameter MCH_PAR1 to obtain processed channels P3* and P4*.

In addition, the multi-channel processor 204 can provide a third processing channel P3* as the first channel CH1, a fourth processing channel P4* as the third channel CH3, and a second processing channel P2* as the second channel CH2.

Assuming that the decoder 200 shown in Figure 10 receives the encoded multichannel signal 107 from the encoder 100 shown in Figure 7, then the first decoded channel D1 of the decoder 200 can be equivalent to the third processed channel P3 of the encoder 100, the second decoded channel D2 of the decoder 200 can be equivalent to the fourth processed channel P4 of the encoder 100, and the third decoded channel D3 of the decoder 200 can be equivalent to the second processed channel P2 of the encoder 100. Furthermore, the first processed channel P1* of the decoder 200 can be equivalent to the first processed channel P1 of the encoder 100.

Furthermore, the encoded multichannel signal 107 can be a serial signal, wherein the multichannel parameter MCH_PAR2 is received at the decoder 200 before the multichannel parameter MCH_PAR1. In this case, the multichannel processor 204 can be configured to process the decoded channels sequentially, wherein the decoder receives the multichannel parameters MCH_PAR1 and MCH_PAR2. In the example shown in FIG10, the decoder receives the multichannel parameter MCH_PAR2 before the multichannel parameter MCH_PAR1, and therefore performs multichannel processing using the second decoded channel pair identified by the multichannel parameter MCH_PAR2 (consisting of the first decoded channel P1* and the third decoded channel D3) before performing multichannel processing using the first decoded channel pair identified by the multichannel parameter MCH_PAR1.

In Figure 10, the multichannel processor 204 exemplarily performs two multichannel processing operations. For illustrative purposes, the multichannel processing operations performed by the multichannel processor 204 are shown in Figure 10 by processing blocks 208 and 210. Processing blocks 208 and 210 can be implemented in hardware or software. Processing blocks 208 and 210 can be, for example, stereo blocks, as discussed above with reference to encoder 100, which is, for example, a general-purpose decoder (or decoder-side stereo block), a prediction-based decoder (or decoder-side stereo block), or a KLT-based rotation decoder (or decoder-side stereo block).

For example, encoder 100 can use a KLT-based rotary encoder (or encoder-side stereo frame). In this case, encoder 100 can derive multichannel parameters MCH_PAR1 and MCH_PAR2, such that MCH_PAR1 and MCH_PAR2 include rotation angles. The rotation angles can be differentially encoded. Therefore, the multichannel processor 204 of decoder 200 can include a differential decoder for differentially encoding the rotation angles.

The device 200 may also include an input interface 212 configured to receive and process encoded multichannel signals 107 to provide encoded channels E1 to E3 to the channel decoder 202 and multichannel parameters MCH_PAR1 and MCH_PAR2 to the multichannel processor 204.

As mentioned earlier, a hold indicator (or hold tree flag) can be used to signal that no new tree has been sent, but the last stereo tree should be used. This can be used to avoid multiple transmissions of the same stereo tree configuration if channel-related attributes remain unchanged for a long time.

Therefore, when the encoded multichannel signal 107 includes multichannel parameters MCH_PAR1 and MCH_PAR2 for the first frame and a hold indicator for the second frame following the first frame, the multichannel processor 204 can be configured to perform multichannel processing or further multichannel processing in the second frame on the same second channel pair or the same first channel pair used in the first frame.

Multichannel processing and further multichannel processing may include stereo processing using stereo parameters, wherein for each scaling factor band or scaling factor band group of the decoded channels D1 to D3, a first stereo parameter is included in multichannel parameter MCH_PAR1 and a second stereo parameter is included in multichannel parameter MCH_PAR2. Thus, the first and second stereo parameters can be of the same type, such as rotation angle or prediction coefficients. Of course, the first and second stereo parameters can be of different types. For example, the first stereo parameter could be a rotation angle, while the second stereo parameter could be a prediction coefficient, and vice versa.

Furthermore, the multichannel parameters MCH_PAR1 and MCH_PAR2 may include a multichannel processing mask indicating which scale factor bands are multichannel processed and which are not. Thus, the multichannel processor 204 can be configured not to perform multichannel processing on the scale factor bands indicated by the multichannel processing mask.

The multichannel parameters MCH_PAR1 and MCH_PAR2 can both include channel pair identifiers (or indices), wherein the multichannel processor 204 can be configured to decode the channel pair identifiers (or indices) using predefined decoding rules or decoding rules indicated in the encoded multichannel signal.

For example, as described above with reference to encoder 100, the channel pairs can be efficiently signaled using a unique index for each pair based on the total number of channels.

In addition, the decoding rule can be a Huffman decoding rule, in which the multichannel processor 204 can be configured to perform Huffman decoding on the channel pair identifiers.

The encoded multichannel signal 107 may also include a multichannel processing enable indicator, which indicates only the subgroup of decoded channels that are allowed to undergo multichannel processing, and indicates at least one decoded channel that is not allowed to undergo multichannel processing. Thus, the multichannel processor 204 can be configured not to perform any multichannel processing on the at least one decoded channel that is not allowed to undergo multichannel processing as indicated by the multichannel processing enable indicator.

For example, when the multichannel signal is a 5.1 channel signal, the multichannel processing enable indicator can indicate that multichannel processing is only allowed for 5 channels, namely right R, left L, right surround Rs, left surround LS and center C, where the LFE channel is not allowed to be multichannel processed.

For the decoding process (decoding the channel pair indexes), the following C code can be used. Therefore, for all channel pairs, the number of channels that can be effectively processed (nChannels) and the number of channel pairs in the current frame (numPairs) are required.

To decode the prediction coefficients for non-band-by-band angles, the following C-code can be used.

To decode the prediction coefficients for non-band-wise KLT angles, the following C-code can be used.

To avoid floating-point differences in trigonometric functions across different platforms, the following lookup table must be used to directly convert angle indices to sin/cos:

tabIndexToSinAlpha[64]＝{

-1.000000f, -0.998795f, -0.995185f, -0.989177f, -0.980785f, -0.970031f, -0.956940f, -0.941544f,

-0.923880f, -0.903989f, -0.881921f, -0.857729f, -0.831470f, -0.803208f, -0.773010f, -0.740951f,

-0.707107f, -0.671559f, -0.634393f, -0.595699f, -0.555570f, -0.514103f, -0.471397f, -0.427555f,

-0.382683f, -0.336890f, -0.290285f, -0.242980f, -0.195090f, -0.146730f, -0.098017f, -0.049068f,

0.000000f, 0.049068f, 0.098017f, 0.146730f, 0.195090f, 0.242980f, 0.290285f, 0.336890f,

0.382683f, 0.427555f, 0.471397f, 0.514103f, 0.555570f, 0.595699f, 0.634393f, 0.671559f,

0.707107f, 0.740951f, 0.773010f, 0.803208f, 0.831470f, 0.857729f, 0.881921f, 0.903989f,

0.923880f, 0.941544f, 0.956940f, 0.970031f, 0.980785f, 0.989177f, 0.995185f, 0.998795f

}；

tabIndexToCosAlpha[64]＝{

0.000000f, 0.049068f, 0.098017f, 0.146730f, 0.195090f, 0.242980f, 0.290285f, 0.336890f,

0.382683f, 0.427555f, 0.471397f, 0.514103f, 0.555570f, 0.595699f, 0.634393f, 0.671559f,

0.707107f, 0.740951f, 0.773010f, 0.803208f, 0.831470f, 0.857729f, 0.881921f, 0.903989f,

0.923880f, 0.941544f, 0.956940f, 0.970031f, 0.980785f, 0.989177f, 0.995185f, 0.998795f,

1.000000f, 0.998795f, 0.995185f, 0.989177f, 0.980785f, 0.970031f, 0.956940f, 0.941544f,

0.923880f, 0.903989f, 0.881921f, 0.857729f, 0.831470f, 0.803208f, 0.773010f, 0.740951f,

0.707107f, 0.671559f, 0.634393f, 0.595699f, 0.555570f, 0.514103f, 0.471397f, 0.427555f,

0.382683f, 0.336890f, 0.290285f, 0.242980f, 0.195090f, 0.146730f, 0.098017f, 0.049068f

}；

For decoding multi-channel encoding, the following C-code can be used for KLT rotation methods.

For frequency band processing, the following C code can be used.

For applications involving KLT rotation, the following C code can be used.

Figure 12 shows a flowchart of a method 400 for decoding an encoded multichannel signal having encoded channels and at least two multichannel parameters MCH_PAR1 and MCH_PAR2. Method 400 includes: step 402, decoding the encoded channels to obtain decoded channels; step 404, performing multichannel processing using a second decoded channel pair identified by multichannel parameter MCH_PAR2 and using multichannel parameter MCH_PAR2 to obtain processed channels, and performing further multichannel processing using a first channel pair identified by multichannel parameter MCH_PAR1 and using multichannel parameter MCH_PAR1, wherein the first channel pair includes at least one processed channel.

The stereo fill in multichannel encoding according to the embodiments is explained below:

As already outlined, an undesirable effect of spectral quantization is that quantization may result in spectral holes. For example, as a result of quantization, all spectral values in a particular frequency band may be set to zero on the encoder side. For instance, the exact values of these spectral lines may have been relatively low before quantization, so quantization may result in a situation where, for example, the spectral values of all spectral lines within a particular frequency band have been set to zero. On the decoder side, this may result in undesirable spectral holes during decoding.

The Multichannel Coding Tool (MCT) in MPEG-H allows for adaptation to different inter-channel dependencies, but stereo fill is not allowed because mono elements are used in typical operating configurations.

As can be seen from Figure 14, the multichannel coding tool combines three or more channels encoded in a layered manner. However, how the multichannel coding tool (MCT) combines different channels during encoding varies from frame to frame depending on the current signal properties of the channels.

For example, in the case of Figure 14(a), in order to generate the first encoded audio signal frame, the Multichannel Coding Tool (MCT) can combine the first channel Ch1 and the second channel CH2 to obtain the first combined channel (processed channel) P1 and the second combined channel P2. Then, the MCT can combine the first combined channel P1 and the third channel CH3 to obtain the third combined channel P3 and the fourth combined channel P4. Then, the MCT can encode the second combined channel P2, the third combined channel P3, and the fourth combined channel P4 to generate the first frame.

Then, for example, in the case of Figure 14(b), in order to generate a second encoded audio signal frame after the first encoded audio signal frame (in time), the Multichannel Coding Tool (MCT) can combine the first channel CH1' and the third channel CH1' to obtain a first combined channel P1' and a second combined channel P2'. Then, the MCT can combine the first combined channel P1' and the second channel CH2' to obtain a third combined channel P3' and a fourth combined channel P4'. Then, the MCT can encode the second combined channel P2', the third combined channel P3', and the fourth combined channel P4' to generate the second frame.

As can be seen from Figure 14, the way the second, third, and fourth combined audio channels of the first frame are generated in the case of Figure 14(a) is significantly different from the way the second, third, and fourth combined audio channels of the second frame are generated in the case of Figure 14(b). This is because different channel combinations are used to generate the corresponding combined audio channels P2, P3, and P4, as well as P2', P3', and P4', respectively.

In particular, embodiments of the present invention are based on the following findings:

As can be seen in Figures 7 and 14, the combined channels P3, P4, and P2 (or P2', P3', and P4' in case (b) of Figure 14) are fed into the channel encoder 104. Furthermore, the channel encoder 104 may, for example, perform quantization such that the spectral values of channels P2, P3, and P4 can be set to zero due to quantization. Spectral samples with adjacent spectra can be encoded into spectral bands, where each spectral band may include multiple spectral samples.

The number of spectral samples in a frequency band can vary for different frequency bands. For example, a frequency band with a lower frequency range may include fewer spectral samples (e.g., 4 spectral samples) compared to a frequency band in a higher frequency range (which may include, for example, 16 frequency samples). For example, a Bark-scale critical band can define the frequency band used.

When all spectral samples in the frequency band are set to zero after quantization, particularly undesirable situations may occur. If this happens, stereo fill is recommended according to the present invention. Furthermore, the present invention is based on the finding that at least not only (pseudo)random noise should be generated.

As an alternative or supplement to adding (pseudo)random noise, according to an embodiment of the invention, if, for example, in the case of FIG14(b), all spectral values of the frequency band of channel P4' have been set to zero, then the combined channel generated in the same or similar manner as channel P3' would be a very suitable basis for generating noise to fill the frequency band that has been quantized to zero.

However, according to an embodiment of the invention, it is preferable not to use the spectral values of the P3' combined channel at the current frame/current time point as the basis for filling the frequency band of the P4' combined channel (which only includes spectral values of zero), because the combined channels P3' and P4' are both generated based on channels P1' and P2', so using the P3' combined channel at the current time point would result in only a translation.

For example, if P3' is the middle channel of P1' and P2' (e.g., P3' = 0.5 * (P1' + P2')) and P4' is a side channel of P1' and P2' (e.g., P4' = 0.5 * (P1' - P2')), then introducing the attenuated spectral value of P3' into the frequency band of P4' will simply result in a shift.

Conversely, it would be preferable to use the channels from previous time points to generate spectral values for filling the spectral holes in the current P4' combined channels. According to the findings of the present invention, the channel combinations from previous frames corresponding to the P3' combined channels of the current frame would be an ideal basis for generating spectral samples for filling the spectral holes in P4'.

However, the combined channel P3 generated in the case of Figure 14(a) of the previous frame does not correspond to the combined channel P3' of the current frame because the combined channel P3 of the previous frame was generated in a different manner than the combined channel P3' of the current frame.

According to the findings of an embodiment of the invention, an approximation of the P3' combined channel should be generated on the decoder side based on the reconstructed channel of the previous frame.

Figure 14(a) illustrates the encoder scenario, where channels CH1, CH2, and CH3 are encoded for a previous frame by generating E1, E2, and E3. The decoder receives channels E1, E2, and E3 and reconstructs the encoded channels CH1, CH2, and CH3. Some coding loss may have occurred; however, the generated approximate channels CH1*, CH2*, and CH3* will be very similar to the original channels CH1, CH2, and CH3, so CH1*≈CH1, CH2*≈CH2, and CH3*≈CH3. According to an embodiment, the decoder keeps the channels CH1*, CH2*, and CH3* generated for the previous frame in a buffer to use them for noise padding in the current frame.

Figure 1a, which shows a device 201 for decoding according to an embodiment, will now be described in more detail:

The apparatus 201 of FIG1a is adapted to decode a previously encoded multichannel signal of a previous frame to obtain three or more previous audio output channels, and is configured to decode a currently encoded multichannel signal 107 of the current frame to obtain three or more current audio output channels.

The device includes an interface 212, a channel decoder 202, a multichannel processor 204 for generating three or more current audio output channels CH1, CH2, CH3, and a noise filling module 220.

Interface 212 is adapted to receive the currently encoded multichannel signal 107 and to receive auxiliary information including the first multichannel parameter MCH_PAR2.

The channel decoder 202 is adapted to decode the currently encoded multichannel signal of the current frame to obtain a set of three or more decoded channels D1, D2, D3 of the current frame.

The multichannel processor 204 is adapted to select a first selected pair of two decoded channels D1, D2 from a set of three or more decoded channels D1, D2, D3 according to a first multichannel parameter MCH_PAR2.

As an example, this is shown in Figure 1a by two channels D1, D2 being fed into the (optional) processing box 208.

Furthermore, the multichannel processor 204 is adapted to generate a first set of two or more processed channels P1*, P2* based on the first selected two decoded channel pairs D1, D2, to obtain an updated set of three or more decoded channels D3, P1*, P2*.

In this example, two channels D1 and D2 are fed into (optional) box 208 to generate two processed channels P1* and P2* from the two selected channels D1 and D2. Then, the updated set of three or more decoded channels includes the remaining unmodified channel D3, and also includes P1* and P2* already generated from D1 and D2.

Before the multichannel processor 204 generates a first pair of two or more processed channels P1*, P2* based on the first selected two decoded channel pairs D1, D2, the noise filling module 220 is adapted to identify at least one channel of the two channels of the first selected two decoded channel pairs D1, D2, wherein one or more frequency bands in which all spectral lines are quantized to zero, and is adapted to generate a mixed channel using two or more, but not all, of three or more previous audio output channels, and is adapted to fill the spectral lines in one or more frequency bands in which all spectral lines are quantized to zero with noise generated using the spectral lines of the mixed channel, wherein the noise filling module 220 is adapted to select two or more previous audio output channels from three or more previous audio output channels for generating the mixed channel according to auxiliary information.

Therefore, the noise filling module 220 analyzes whether there are frequency bands with only zero values in the spectrum, and further fills the found empty frequency bands with the generated noise. For example, the frequency band may have, for example, 4, 8, or 16 spectral lines, and the noise filling module 220 fills the generated noise when all spectral lines of the frequency band have been quantized to zero.

The noise filling module 220, which specifies how noise is generated and filled, may employ a particular concept in an embodiment referred to as stereo filling.

In the embodiment of FIG1a, the noise filling module 220 interacts with the multi-channel processor 204. For example, in the embodiment, when the noise filling module wants to process two channels, for example, through a processing frame, it feeds these channels to the noise filling module 220, and the noise filling module 220 checks whether the frequency bands have been quantized to zero, and if so, fills these frequency bands.

In another embodiment shown in Figure 1b, the noise filling module 220 interacts with the channel decoder 202. For example, when the channel decoder has decoded the encoded multichannel signal to obtain three or more decoded channels D1, D2, and D3, the noise filling module can, for example, check whether the frequency bands have been quantized to zero, and, if detected, fill those frequency bands. In this embodiment, the multichannel processor 204 can ensure that all spectral holes have been closed beforehand by filling noise.

In another embodiment (not shown), the noise filling module 220 may interact with both the channel decoder and the multichannel processor. For example, when the channel decoder 202 generates the decoded channels D1, D2, and D3, the noise filling module 220 may have just checked whether they have been quantized to zero after the channel decoder 202 has generated the frequency bands, but when the multichannel processor 204 actually processes these channels, it may only generate noise and fill the corresponding frequency bands.

For example, random noise and computationally inexpensive operations can be inserted into any frequency band that has been quantized to zero, but the noise filling module can only fill in the noise generated from the previously generated audio output channel when the multi-channel processor 204 actually processes it. However, in this embodiment, before inserting random noise, the presence of spectral holes should be detected, and this information should be stored in memory, because after the insertion of random noise, each frequency band will then have a non-zero spectral value due to the insertion of random noise.

In one embodiment, random noise is inserted into the frequency band that has been quantized to zero, in addition to the noise generated based on the previous audio output signal.

In some embodiments, interface 212 may be adapted, for example, to receive the currently encoded multichannel signal 107, and to receive auxiliary information including a first multichannel parameter MCH_PAR2 and a second multichannel parameter MCH_PAR1.

The multichannel processor 204 may, for example, be adapted to select a second selected pair of two decoded channels P1*, D3 from an updated set of three or more decoded channels D3, P1*, P2* according to a second multichannel parameter MCH_PAR1, wherein at least one channel P1* of the second selected pair of decoded channels (P1*, D3) is one of the channels P1*, P2* of the first pair of two or more processed channels P1*, P2*.

The multichannel processor 204 may, for example, be adapted to generate a second set of two or more processed channels P3*, P4* based on the second selected two decoded channel pairs P1*, D3, to further update the updated set of three or more decoded channels.

An example of this embodiment can be seen in Figures 1a and 1b, in which (optional) processing block 210 receives channel D3 and processed channel P1* and processes them to obtain processed channels P3* and P4*, such that the set of further updates of the three decoded channels includes the unprocessed block 210 modified P2* as well as the generated P3* and P4*.

Processing blocks 208 and 210 are labeled as optional in Figures 1a and 1b. This indicates that although processing blocks 208 and 210 can be used to implement the multichannel processor 204, various other possibilities exist regarding exactly how the multichannel processor 204 is implemented. For example, instead of using different processing blocks 208, 210 for each different processing of two (or more) channels, the same processing blocks can be used again, or the multichannel processor 204 can implement the processing of two channels without using processing blocks 208, 210 at all (as sub-units of the multichannel processor 204).

According to another embodiment, the multichannel processor 204 may, for example, be adapted to generate a first set of two or more processed channels P1*, P2* by generating a first set of exactly two processed channels P1*, P2* based on the first selected two decoded channel pairs D1, D2. The multichannel processor 204 may, for example, be adapted to replace the first selected two decoded channel pairs D1, D2 in a set of three or more decoded channels D1, D2, D3 with the first set of exactly two processed channels P1*, P2* to obtain an updated set of three or more decoded channels D3, P1*, P2*. The multichannel processor 204 may, for example, be adapted to generate a second set of two or more processed channels P3*, P4* by generating a second set of exactly two processed channels P3*, P4* based on the second selected two decoded channel pairs P1*, D3. Furthermore, the multichannel processor 204 may, for example, be adapted to replace the second selected two decoded channel pairs P1*, D3 in the updated set of three or more decoded channels D3, P1*, P2* with exactly two processed channels P3*, P4* of the second set, in order to further update the updated set of three or more decoded channels.

In this embodiment, exactly two processed channels are generated from two selected channels (e.g., the two input channels of processing block 208 or 210), and these exactly two processed channels replace the selected channels in a set of three or more decoded channels. For example, processing block 208 of multichannel processor 204 replaces the selected channels D1 and D2 with P1* and P2*.

However, in other embodiments, upmixing can be performed in device 201 for decoding, and more than two processed channels can be generated from two selected channels, or all selected channels can be left unremoved from the updated set of decoded channels.

Another issue is how to generate the mixed channels used to generate the noise produced by the noise filling module 220.

According to some embodiments, the noise filling module 220 may be adapted, for example, to generate a mixed channel using exactly two of three or more previous audio output channels as two or more of three or more previous audio output channels; wherein, the noise filling module 220 may be adapted, for example, to select exactly two previous audio output channels from three or more previous audio output channels based on auxiliary information.

Using only two of the three or more previously output channels helps reduce the computational complexity of calculating the mixed channels.

However, in other embodiments, two or more channels from the previous audio output channels are used to generate the mixed channel, but the number of previous audio output channels considered is less than the total number of three or more previous audio output channels.

In an embodiment that considers only two channels from the previous output channels, the mixed channel can be calculated, for example, as follows:

In an embodiment, the noise filling module 220 is adapted to be based on a formula

Or based on formula

A mixed channel is generated using exactly two previous audio output channels, where D _ch is the mixed channel; where is the first channel of the exactly two previous audio output channels; where is the second channel of the exactly two previous audio output channels, which is different from the first channel of the exactly two previous audio output channels, and where d is a real positive scalar.

In typical cases, the middle channel can be a suitable hybrid channel. This method calculates the hybrid channel as the middle channel between the two previous audio output channels under consideration.

However, in some situations, when applied, such as when the number of mixed channels is close to zero, it may be preferable to use a mixed signal. Therefore, side channels (for out-of-phase input signals) are used.

According to the alternative method, the noise filling module 220 is suitable for use based on the formula.

Or based on formula

A mixed channel is generated using exactly two previous audio output channels, where is the mixed channel; is the first channel of the exactly two previous audio output channels; is the second channel of the exactly two previous audio output channels, which is different from the first channel of the exactly two previous audio output channels, and α is the rotation angle.

This method calculates the mixed channel by rotating the two previously considered audio output channels.

The rotation angle α can be within the following range, for example: -90° < α < 90°.

In an embodiment, the rotation angle may be within the range of 30° < α < 60°.

Furthermore, in typical cases, the channel can be a suitable mixed channel. This method calculates the mixed channel as the middle channel between the two previous audio output channels under consideration.

However, in some situations, when applied, such as when the number of mixed channels is close to zero, it may be preferable to use as the mixed signal.

According to a particular embodiment, the auxiliary information may be, for example, current auxiliary information assigned to the current frame, wherein interface 212 may be, for example, adapted to receive previous auxiliary information assigned to a previous frame, wherein the previous auxiliary information includes a previous angle; wherein interface 212 may be, for example, adapted to receive current auxiliary information including the current angle, and wherein noise filling module 220 may be, for example, adapted to use the current angle of the current auxiliary information as the rotation angle α, and adapted not to use the previous angle of the previous auxiliary information as the rotation angle α.

Therefore, in this embodiment, even if the mixed channel is calculated based on the previous audio output channel, the current angle sent in the auxiliary information is still used as the rotation angle, rather than the previously received rotation angle, even though the mixed channel is calculated based on the previous audio output channel, which was generated based on the previous frame.

Another aspect of some embodiments of the present invention relates to a scaling factor.

For example, the frequency band could be a scaling factor band.

According to some embodiments, before the multichannel processor 204 generates a first pair of two or more processed channels P1*, P2* based on the first selected two decoded channel pairs (D1, D2), the noise filling module (220) may, for example, be adapted to identify one or more scaling factor bands for at least one of the two channels of the first selected two decoded channel pairs D1, D2, which are one or more frequency bands in which all spectral lines are quantized to zero, and may, for example, be adapted to generate a mixed channel using said two or more, but not all, channels of three or more previous audio output channels, and is adapted to fill the spectral lines of the one or more scaling factor bands in which all spectral lines are quantized to zero with noise generated using the spectral lines of the mixed channel, according to the scaling factor of each of the one or more scaling factor bands in which all spectral lines are quantized to zero.

In these embodiments, the scaling factor may, for example, be assigned to each scaling factor band, and the scaling factor may be taken into account when generating noise using a mixed channel.

In a particular embodiment, the receiving interface 212 may, for example, be configured to receive a scaling factor for each of the one or more scaling factor bands, and the scaling factor for each of the one or more scaling factor bands indicates the energy of the spectral line of the scaling factor band before quantization. The noise filling module 220 may, for example, be adapted to generate noise for each of the one or more scaling factor bands in which all spectral lines are quantized to zero, such that after adding noise to a frequency band, the energy of the spectral line corresponds to the energy indicated by the scaling factor of the scaling factor band.

For example, a mixing channel can indicate the spectral values of the four spectral lines of the scaling factor band in which noise should be inserted, and these spectral values can be, for example: 0.2; 0.3; 0.5; 0.1.

The energy of the scaling factor band for the mixed channels can be calculated, for example, as follows:

(0.2)'+(0.3)+(0.5)'+(0.1)2＝0.39

However, the scaling factor of the scaling factor band of the channel that should be filled with noise can be, for example, only 0.0039.

The attenuation factor can be calculated as follows:

Therefore, in the example above,

In this embodiment, each spectral value of the scaling factor band used as noise in the mixed channel is multiplied by an attenuation factor:

Therefore, each of the four spectral values of the scaling factor band in the example above is multiplied by the attenuation factor to obtain the attenuated spectral value:

0.2 * 0.01 = 0.002

0.3 * 0.01 = 0.003

0.5 * 0.01 = 0.005

0.1·0.01＝0.001

These attenuated spectral values can then be inserted into the scaling factor band of the channel to be filled with noise.

The above example also applies to logarithmic values by replacing the above operations with corresponding logarithmic operations, such as replacing multiplication with addition.

In addition to the description of the specific embodiments provided above, other embodiments of the noise filling module 220 are applicable to one, some, or all of the concepts described with reference to Figures 2 through 6.

Another aspect of the embodiments of the present invention addresses the problem of selecting an information channel from a previous audio output channel to generate a mixed channel to obtain noise to be inserted.

According to an embodiment, the means of the noise filling module 220 can be adapted, for example, to select exactly two previous audio output channels from three or more previous audio output channels according to a first multi-channel parameter MCH_PAR2.

Therefore, in this embodiment, the first multichannel parameter that controls which channel is selected for processing also controls which channel in the previous audio output channel is used to generate the mixed channel to generate the noise to be inserted.

In an embodiment, the first multichannel parameter MCH_PAR2 may, for example, indicate two decoded channels D1, D2 from a set of three or more decoded channels; and the multichannel processor 204 is adapted to select a first selected pair of decoded channels D1, D2 from a set of three or more decoded channels D1, D2, D3 by selecting the two decoded channels D1, D2 indicated by the first multichannel parameter MCH_PAR2. Furthermore, the second multichannel parameter MCH_PAR1 may, for example, indicate two decoded channels P1*, D3 from an updated set of three or more decoded channels. The multichannel processor 204 may, for example, be adapted to select a second selected pair of decoded channels P1*, D3 from an updated set of three or more decoded channels D3, P1*, P2* by selecting the two decoded channels P1*, D3 indicated by the second multichannel parameter MCH_PAR1.

Therefore, in this embodiment, the channel selected for the first processing (e.g., the processing of processing block 208 in FIG1a or FIG1b) depends not only on the first multichannel parameter MCH_PAR2. Furthermore, the two selected channels are explicitly specified in the first multichannel parameter MCH_PAR2.

Similarly, in this embodiment, the channel selected for the second processing (e.g., the processing of processing block 210 in FIG1a or FIG1b) depends not only on the second multichannel parameter MCH_PAR1. Furthermore, the two selected channels are explicitly specified in the second multichannel parameter MCH_PAR1.

The embodiments of the present invention describe a complex indexing scheme for multi-channel parameters, which is explained with reference to FIG15.

Figure 15(a) shows the encoding of the five channels on the encoder side, namely the left channel, right channel, center channel, left surround channel, and right surround channel. Figure 15(b) shows the decoding of the encoded channels E0, E1, E2, E3, and E4 to reconstruct the left channel, right channel, center channel, left surround channel, and right surround channel.

Assuming the index is assigned to each of the five channels: left channel, right channel, center channel, left surround channel, and right surround channel, that is...

In Figure 15(a), on the encoder side, the first operation performed could be, for example, mixing channel 0 (left channel) and channel 3 (left surround channel) in processing block 192 to obtain two processing channels. It can be assumed that one of the processing channels is the center channel and the other is a side channel. However, other ideas for forming the two processing channels can also be applied, for example, determining the two processing channels by performing a rotation operation.

Now, the two generated channels for processing acquire the same indices as the channels used for processing. That is, the first channel in the processed channels has index 0, and the second channel in the processed channels has index 3. The determined multichannel parameters for this processing can be, for example, (0; 3).

The second operation performed on the encoder side could be, for example, mixing channel 1 (right channel) and channel 4 (right surround channel) in processing box 194 to obtain two further processed channels. Similarly, the two further processed channels obtain the same index as the channel used for processing. That is, the first channel in the further processed channels has index 1, and the second channel in the processed channels has index 4. The determined multichannel parameters for this processing could be, for example, (1; 4).

The third operation performed on the encoder side could be, for example, mixing processed channel 0 and processed channel 1 in processing box 196 to obtain two additional processed channels. Similarly, these two generated processed channels obtain the same index as the channel used for processing. That is, the first channel in the further processed channels has index 0, and the second channel in the processed channels has index 1. The determined multi-channel parameters for this processing could be, for example, (0; 1).

The encoded channels E0, E1, E2, E3, and E4 are distinguished by their indices; that is, E0 has index 0, E1 has index 1, E2 has index 2, and so on.

The three operations on the encoder side yield three multi-channel parameters:

(0；3), (1；4), (0；1).

Since the decoding device must perform encoder operations in reverse order, the order of the multichannel parameters can be reversed, for example, when sending multichannel parameters to the decoding device, to obtain the multichannel parameters:

(0；1), (1；4), (0；3).

For the device used for decoding, (0; 1) can be called the first multi-channel parameter, (1; 4) can be called the second multi-channel parameter, and (0; 3) can be called the third multi-channel parameter.

On the decoder side shown in Figure 15(b), upon receiving the first multichannel parameters (0; 1), the decoding apparatus concludes that, as the first processing operation on the decoder side, channels 0 (E0) and 1 (E1) should be processed. This is performed in box 296 of Figure 15(b). Both generated processed channels inherit the indices of the channels E0 and E1 used to generate them; therefore, the generated processed channels also have indices 0 and 1.

Upon receiving the second multichannel parameters (1; 4), the decoding apparatus concludes that, as a second processing operation on the decoder side, channels 1 and 4 (E4) should be processed. This is performed in box 294 of Figure 15(b). Both generated processed channels inherit the indices of channels 1 and 4 used to generate them; therefore, the generated processed channels also have indices 1 and 4.

Upon receiving the third multichannel parameters (0; 3), the decoding apparatus concludes that, as the third processing operation on the decoder side, channels 0 and 3 (E3) should be processed. This is performed in box 292 of Figure 15(b). Both generated processed channels inherit the indices of channels 0 and 3 used to generate them; therefore, the generated processed channels also have indices 0 and 3.

As a result of the processing by the device used for decoding, the left channel (index 0), right channel (index 1), center channel (index 2), left surround channel (index 3), and right surround channel (index 4) were reconstructed.

Let's assume that on the decoder side, due to quantization, all values of channel E1 (index 1) within a certain scaling factor band have been quantized to zero. When the device used for decoding wants to process in box 296, it expects noise-filled channel 1 (channel E1).

As already outlined, the embodiment now uses two previous audio output signals to fill the spectral holes of channel 1 with noise.

In a particular embodiment, if the channel to be operated on has a scaling factor band that is quantized to zero, the two previous audio output channels are used to generate noise with the same index numbers as the two channels to be processed. In this example, if a spectral hole in channel 1 is detected before processing in processing block 296, the previous audio output channels with index 0 (previous left channel) and index 1 (previous right channel) are used to generate noise to fill the spectral hole in channel 1 on the decoder side.

Since the index is always inherited from the processed channels, it can be assumed that the previous output channel will serve to generate the actual processing channel participating in the decoder side, if the previous audio output channel will be the current audio output channel. Therefore, a good estimate of the scaling factor band that is quantized to zero can be achieved.

According to an embodiment, the apparatus may, for example, be adapted to assign identifiers from a set of identifiers to each of three or more previous audio output channels, such that each of the three or more previous audio output channels is assigned to exactly one identifier from the set of identifiers, and that each identifier from the set of identifiers is assigned to exactly one of the three or more previous audio output channels. Furthermore, the apparatus may, for example, be adapted to assign identifiers from the set of identifiers to each of three or more decoded channels, such that each of the three or more decoded channels is assigned to exactly one identifier from the set of identifiers, and that each identifier from the set of identifiers is assigned to exactly one of the three or more decoded channels.

Furthermore, the first multichannel parameter MCH_PAR2 may, for example, indicate a first pair of two identifiers from a set of three or more identifiers. The multichannel processor 204 may, for example, be adapted to select the first selected pair of two decoded channels D1, D2 from a set of three or more decoded channels D1, D2 by selecting two decoded channels D1, D2 assigned to the two identifiers of the first pair of two identifiers.

The device may, for example, be adapted to assign the first identifier of the two identifiers in the first pair of two identifiers to the first processed channel in the first group of exactly two processed channels P1*, P2*. Furthermore, the device may, for example, be adapted to assign the second identifier of the two identifiers in the first pair of two identifiers to the second processed channel in the first group of exactly two processed channels P1*, P2*.

The set of identifiers can be, for example, an index set, such as a set of non-negative integers (e.g., a set including the identifiers 0; 1; 2; 3 and 4).

In a particular embodiment, the second multichannel parameter MCH_PAR1 may, for example, indicate a second pair of two identifiers from a set of three or more identifiers. The multichannel processor 204 may, for example, be adapted to select a second selected pair of two decoded channels P1*, D3 from an updated set of three or more decoded channels D3, P1*, P2* by selecting two decoded channels (D3, P1*) assigned to the two identifiers of the second pair of two identifiers. Furthermore, the device may, for example, be adapted to assign the first identifier of the two identifiers of the second pair of two identifiers to the first processed channel of the second set of exactly two processed channels P3*, P4*. Furthermore, the device may, for example, be adapted to assign the second identifier of the two identifiers of the second pair of two identifiers to the second processed channel of the second set of exactly two processed channels P3*, P4*.

In a particular embodiment, the first multichannel parameter MCH_PAR2 may, for example, indicate the first pair of two identifiers in a set of three or more identifiers. The noise filling module 220 may, for example, be adapted to select exactly two previous audio output channels from three or more previous audio output channels by selecting two previous audio output channels of the two identifiers assigned to the first pair of two identifiers.

As already outlined, Figure 7 illustrates an apparatus 100 according to an embodiment for encoding a multichannel signal 101 having at least three channels (CH1:CH3).

The device includes an iterative processor 102 adapted to calculate, in a first iterative step, interchannel correlation values between each pair of at least three channels (CH:CH3), for selecting, in the first iterative step, channel pairs having the highest value or having a value above a threshold, and for processing the selected channel pairs using multichannel processing operations 110, 112 to derive initial multichannel parameters MCH_PAR1 for the selected channel pairs and derive channels P1, P2 of the first processing.

The iterative processor 102 is adapted to perform calculations, selections, and processing in a second iterative step using at least one processed channel P1 to derive additional multichannel parameters MCH_PAR2 and second processed channels P3 and P4.

In addition, the device includes a channel encoder adapted to encode the channels (P2:P4) obtained by the iterative processing performed by the iterative processor 104 to obtain encoded channels (E1:E3).

In addition, the device includes an output interface 106, which is adapted to generate an encoded multichannel signal 107 having encoded channels (E1:E3), initial multichannel parameters, and additional multichannel parameters MCH_PAR1 and MCH_PAR2.

In addition, the device includes an output interface 106 adapted to generate an encoded multi-channel signal 107 to include information indicating whether the decoding device should fill the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero with noise generated based on previously decoded audio output channels that have been previously decoded by the decoding device.

Therefore, the encoding device can signal to the decoding device whether the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero should be filled with noise generated based on previously decoded audio output channels that have been previously decoded by the decoding device.

According to the embodiment, each of the initial multichannel parameters and the additional multichannel parameters MCH_PAR1 and MCH_PAR2 indicates exactly two channels, each of which is one of the encoded channels (E1:E3) or one of the first or second processed channels P1, P2, P3, P4 or one of at least three channels (CH1:CH3).

Output interface 106 may, for example, be adapted to generate an encoded multichannel signal 107 such that information indicating whether a decoding device should fill the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero includes, for each of the initial and additional multichannel parameters MCH_PAR1, MCH_PAR2, indicating whether, for at least one of exactly two channels indicated by the parameters in the initial and additional multichannel parameters MCH_PAR1, MCH_PAR2, the decoding device should fill the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero with spectral data generated based on previously decoded audio output channels, wherein the previously decoded audio output channels were previously used by the decoding device for decoding.

The following describes a specific embodiment in which this information is sent using the hasStereoFilling[pair] value, which indicates whether stereo fill should be applied to the currently processed MCT channel pair.

Figure 13 illustrates the system according to an embodiment.

The system includes an encoding device 100 as described above, and a decoding device 201 according to one of the above embodiments.

The decoding device 201 is configured to receive the encoded multi-channel signal 107 generated by the encoding device 100 from the encoding device 100.

In addition, coded multi-channel signals 107 are provided.

Encoded multi-channel signals include

- Encoded channels (E1:E3), and

- Multi-channel parameters MCH_PAR1, MCH_PAR2, and

- Indicates whether the device used for decoding should fill the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero with spectral data generated based on previously decoded audio output channels, wherein the previously decoded audio output channels were previously used by the device for decoding.

According to an embodiment, the encoded multichannel signal may include, for example, two or more multichannel parameters as multichannel parameters MCH_PAR1 and MCH_PAR2.

Each of two or more multichannel parameters MCH_PAR1, MCH_PAR2 may, for example, indicate exactly two channels, each of which is one of the encoded channels (E1:E3) or one of the multiple processed channels P1, P2, P3, P4 or one of at least three initial (e.g., unprocessed) channels (CH:CH3).

Information indicating whether a device for decoding should fill spectral lines in one or more frequency bands where all spectral lines are quantized to zero may, for example, include, for each of two or more multichannel parameters MCH_PAR1, MCH_PAR2, indicating whether, for at least one of exactly two channels indicated by the parameters MCH_PAR1, MCH_PAR2, the device for decoding should fill spectral lines in one or more frequency bands where all spectral lines are quantized to zero with spectral data generated based on previously decoded audio output channels, wherein the previously decoded audio output channels were previously used by the device for decoding.

As further outlined below, a particular embodiment is described in which this information is sent using the hasStereoFilling[pair] value, which indicates whether stereo fill should be applied to the currently processed MCT channel pair.

The general concepts and specific embodiments are described in more detail below.

The embodiment implements a parameterized low bit rate coding mode, which has the flexibility to use arbitrary stereo trees (a combination of stereo fill and MCT).

Inter-channel signal dependencies are utilized by applying known joint stereo coding tools in a layered manner. For lower bit rates, embodiments extend MCT to use a combination of discrete stereo coding frames and stereo fill frames. Thus, semi-parametric coding can be applied to channels with similar content (i.e., channel pairs with the highest correlation), while different channels can be encoded individually or via non-parametric representations. Therefore, the MCT bitstream syntax is extended to be able to signal whether stereo fill is allowed and where it is active.

The embodiment implements the generation of the previous downmixer for arbitrary stereo fill pairs.

Stereo fill relies on using downmixing from previous frames to improve the filling of spectral holes caused by quantization in the frequency domain. However, combined with MCT, the set of jointly encoded stereo pairs is now allowed to be time-varying. Therefore, two jointly encoded channels may not have been jointly encoded in a previous frame, i.e., when the tree configuration has changed.

To estimate the previous downmix, the previously decoded output channels are preserved and processed using inverse stereo operations. For a given stereo frame, this is done using the parameters of the current frame and the decoded output channels of the previous frame corresponding to the channel indices of the processed stereo frame.

If, for example, the previous output channel signal becomes unavailable due to an independent frame (a frame that can be decoded without considering previous frame data) or a change in transform length, the previous channel buffer for the corresponding channel is set to zero. Therefore, a non-zero previous downmixer can still be calculated as long as at least one previous channel signal is available.

If the MCT is configured to use a prediction-based stereo frame, the specified inverse MS operation is performed with stereo fill, preferably using one of the following two equations based on the prediction direction flag (pred_dir in MPEG-H syntax) to calculate the previous downmixer.

Where d is any real positive scalar.

If MCT is configured to use a rotation-based stereo frame, the previous downmixer is calculated using a rotation with a negative rotation angle.

Therefore, for the following rotation:

The inverse rotation is calculated as follows:

Wherein, is the previously output channel and the expected previous downmixer.

The example demonstrates the application of stereo fill in MCT.

The application of stereo fill in a single stereo frame is described in [1] and [5]. For a single stereo frame, stereo fill is applied to the second channel of a given MCT channel pair.

Specifically, the differences in stereo fill in MCT are as follows:

The MCT tree configuration extends each frame by one signaling bit so that it can signal whether stereo fill is allowed in the current frame.

In a preferred embodiment, if stereo fill is permitted in the current frame, an additional bit is sent for each stereo frame to activate the stereo fill in that frame. This is a preferred embodiment because it allows the encoder side to control which frames should be used to apply stereo fill in the decoder.

In the second embodiment, if stereo fill is allowed in the current frame, then stereo fill is allowed in all stereo frames, and no additional bits are sent for each individual stereo frame. In this case, the decoder controls the selective application of stereo fill in each MCT frame.

The following describes another concept and detailed embodiments:

The implementation improves the quality of low bit rate multichannel operating points.

In frequency domain (FD) encoded channel pairs (CPEs), the MPEG-H 3D audio standard allows the use of the stereo filler tool described in subsection 5.5.5.4.9 of [1] to perceptually improve the filling of spectral holes caused by very coarse quantization in the encoder. This tool has proven particularly useful for two-channel stereo encoded at medium and low bit rates.

The Multichannel Coding Tool (MCT) described in Section 7 of [2] is introduced, which implements a flexible, signal-adaptive definition of co-coded channel pairs on a per-frame basis to take advantage of time-varying inter-channel dependencies in multichannel settings. The advantages of MCT are particularly significant when used for efficient dynamic co-coding of multichannel settings, where each channel resides in its individual mono element (SCE), because unlike the traditional CPE+SCE(+LFE) configuration which must be established a priori, it allows co-channel coding to be concatenated and/or reconfigured from one frame to the next.

The current drawback of encoding multichannel surround sound without using a CPE is that the joint stereo tools available only in the CPE—predictive M/S coding and stereo fill—cannot be utilized, which is particularly disadvantageous at low to medium bit rates. MCT can replace the M/S tools, but it cannot currently replace the stereo fill tools.

The implementation allows the use of the stereo fill tool within MCT channel pairs by extending the MCT bitstream syntax with corresponding signaling bits and by extending the application of stereo fill to any channel pair regardless of its channel element type.

For example, some embodiments can implement stereo fill signaling in MCT, as follows:

In CPE, the use of the stereo fill tool is signaled in the FD noise fill information of the second channel, as described in subsection 5.5.5.4.9.4 of [1]. When using MCT, each channel can be a “second channel” (due to the possibility of cross-element channel pairs). Therefore, it is proposed to explicitly signal the stereo fill by using an additional bit for each channel pair encoded by MCT. When stereo fill is not used in any channel pair of a particular MCT “tree” instance, to avoid needing the additional bit, the presence of the above additional bit is signaled for each channel pair by using two currently reserved entries [2] of the MCTSignalingType element in MultichannelCodingFrame().

A detailed description is provided below.

Some embodiments may implement the following prior downmixing calculation, for example:

Stereo fill in the CPE is achieved by adding the corresponding MDCT coefficients from the downmix of the previous frame to fill certain "empty" scaling factor bands in the second channel. These coefficients are scaled according to the transmitted scaling factor of the corresponding frequency band (which would otherwise be unused because the band is completely quantized to zero). The weighted summation process, controlled by the scaling factor band of the target channel, can be used in the same way in the case of the MCT. The source spectrum of the stereo fill, i.e., the downmix of the previous frame, must be calculated differently than in the CPE, especially because the MCT "tree" configuration may be time-varying.

In MCT, the previous downmix can be derived from the decoded output channels of the last frame (stored after MCT decoding) using the MCT parameters of a given joint channel pair in the current frame. For channel pairs applying predictive M/S-based joint coding, the previous downmix, as in CPE stereo fill, is equal to the sum or difference of the appropriate channel spectra, depending on the direction indicator of the current frame. For stereo pairs using joint coding based on Karhunen-Loève rotation, the previous downmix represents the inverse rotation calculated using the rotation angle of the current frame. Again, a detailed description is provided below.

Complexity assessments indicate that stereo fill in MCT, as a low-to-medium bit-rate tool, is not expected to increase worst-case complexity when measured at low/medium and high bit-rates. Furthermore, using stereo fill generally aligns with a larger number of spectral coefficients quantized to zero, thereby reducing the algorithmic complexity of the context-based arithmetic decoder. Assuming a maximum of N/3 stereo fill channels in an N-channel surround configuration, and using an additional 0.2 WMOPS per stereo fill operation, the peak complexity increases by only 0.4 WMOPS for 5.1 channels and 0.8 WMOPS for 11.1 channels when the encoder sampling rate is 48 kHz and the IGF tool only operates above 12 kHz. This equates to less than 2% of the total decoder complexity.

The implementation of the MultichannelCodingFrame() element is as follows:

According to some embodiments, stereo fill in MCT can be implemented as follows:

Similar to the IGF stereo fill in the channel pair element described in subsection 5.5.5.4.9 of [1], the stereo fill in the Multichannel Coding Tool (MCT) uses the downmixing of the output spectrum of the previous frame to fill the “empty” scaling factor band (fully quantized to zero) at or above the noise fill start frequency.

When stereo fill is activated in an MCT combined channel pair (hasStereoFilling[pair]≠0 in Table AMD4.4), all “empty” scaling factor bands in the noise-filled region of the second channel of that channel pair (i.e., starting at or above noiseFillingStartOffset) are filled to a specific target energy using the downmixing of the corresponding output spectrum from the previous frame (after MCT application). This is done after FD noise fill (see subsection 7.2 in ISO/IEC 23003-3:2012) and before the scaling factor and MCT combined stereo application. All output spectra after MCT processing are saved for potential stereo fill in the next frame.

Operational constraints might include, for example, the cascading execution of any subsequent MCT stereo pairs that do not support stereo fill algorithms (hasStereoFilling[pair]≠0) in the empty frequency bands of the second channel if the second channel is the same. Within the channel pair element, according to subsection 5.5.5.4.9 of [1], the IGF stereo fill activated in the second (residual) channel takes precedence over—and therefore disables—the application of any subsequent MCT stereo fill in the same channel of the same frame.

Terms and definitions can be defined, for example, as follows:

hasStereoFilling[pair] indicates the use of MCT channel centering to stereo fill in the currently processed channel.

ch1, ch2 are the channel indices of the currently processed MCT channel pair.

spectral_data[][] The spectral coefficients of the current MCT channel relative to the middle channel.

spectrum_data_prev[][] The output spectrum after MCT processing in the previous frame.

`downmix_prev[][]` is the downmixer with an estimate of the output channels of the previous frame at the given index of the MCT channels being processed.

num_swb is the total number of scaling factor bands, see section 6.2.9.4 of ISO/IEC 23003-3.

ccfl coreCoderFrameLength, transformation length, see section 6.1 of ISO/IEC 23003-3.

noiseFillingStartOffset: Noise fill start line, defined according to the CCFL definition in Table 109 of ISO/IEC 23003-3.

For spectral whitening in IGF_WhiteningLevel, see section 5.5.5.4.7 of ISO/IEC 23008-3.

See section 7.2 of ISO/IEC 23003-3 for the noise-filled seed used by seed[] randomSign().

For some specific embodiments, the decoding process can be described, for example, as follows:

MCT stereo fill is performed using four consecutive operations, as described below:

Step 1: Prepare the spectrum of the second channel for the stereo fill algorithm

If the stereo fill indicator hasStereoFilling[pair] for a given MCT channel pair is zero, stereo fill is not used, and the following steps are not performed. Otherwise, if a scaling factor was previously applied to the second channel spectrum spectral_data[ch2] of the channel pair, the scaling factor application is undone.

Step 2: Generate the previous downmix spectrum for the given MCT channel pair

The previous downmix is estimated based on the output signal of the previous frame stored after the MCT processing is applied, spectral_data_prev[][]. If the previous output channel signal is unavailable, for example due to a single frame (indepFlag>0), a change in transform length, or core_mode==1, the previous channel buffer for the corresponding channel should be set to zero.

For the predicted stereo pair, i.e., MCT SignalingType == 0, the previous downmix is calculated based on the previous output channel as downmix_prev[][] defined in step 2 of subsection 5.5.5.4.9.4 of [1], where spectrum[window][] is represented by spectral_data[][window].

For a rotating stereo pair, i.e., MCT SignalingType == 1, the previous downmixer is calculated based on the previous output channel by reversing the rotation operation defined in subsection 5.5.X.3.7.1 of [2].

Use L = spectral_data_prev[ch1][], R = spectral_data_prev[ch2][], and dmx = downmix_prev[] from the previous frame, and use Idx and nSamples from the current frame and the MCT pair.

Step 3: Perform stereo fill algorithm in the empty frequency band of the second channel.

Stereo fill is applied to the second channel of the MCT pair, as in step 3 of subsection 5.5.5.4.9.4 of [1], where spectrum[window] is represented by spectral_data[ch2][window] and max_sfb_ste is given by num_swb.

Step 4: Adaptive synchronization of scaling factor application and noise-filled seed.

Following step 3 in subsection 5.5.5.4.9.4 of [1], a scaling factor is applied to the resulting spectrum, as in 7.3 of ISO/IEC 23003-3, where the scaling factor for empty frequency bands is treated like a regular scaling factor. In the case of an undefined scaling factor, for example because it is above max_sfb, its value should be zero. If IGF is used, and igf_WhiteningLevel is equal to 2 in any second channel patch, and neither channel uses the eight short transforms, the spectral energy of the two channels in the MCT channel pair is calculated in the range from index noiseFillingStartOffset to index ccfl/2-1 before executing decode_mct(). If the calculated energy of the first channel is more than 8 times greater than the energy of the second channel, the seed [ch2] of the second channel is set to be equal to the seed [ch1] of the first channel.

Although some aspects have been described in the context of the apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of method steps also represent a description of a corresponding block or item or feature of the corresponding apparatus. Some or all of the method steps may be performed by (or using) hardware devices, such as microprocessors, programmable computers, or electronic circuits. In some embodiments, one or more of the most important method steps may be performed with such a device.

According to certain implementation requirements, embodiments of the invention can be implemented in hardware or software, or at least partially in hardware or at least partially in software. This implementation can be performed using a digital storage medium having electronically readable control signals stored thereon, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory, which cooperates with (or is capable of cooperating with) a programmable computer system to perform the corresponding methods. Therefore, the digital storage medium can be computer-readable.

Some embodiments of the invention include a data carrier having electronically readable control signals that are capable of cooperating with a programmable computer system to perform one of the methods described herein.

Typically, embodiments of the present invention can be implemented as a computer program product having program code operable to perform one of these methods when the computer program product is run on a computer. This program code may, for example, be stored on a machine-readable medium.

Other embodiments include a computer program stored on a machine-readable medium for performing one of the methods described herein.

In other words, embodiments of the method of the present invention are therefore computer programs having program code that, when the computer program is run on a computer, execute one of the methods described herein.

Therefore, another embodiment of the method of the present invention is a data carrier (or digital storage medium, or computer-readable medium) including a computer program recorded thereon for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is generally tangible and/or non-transitory.

Therefore, another embodiment of the method of the present invention represents a data stream or signal sequence for performing one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection, such as via the Internet.

Another embodiment includes a processing device, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

Another embodiment includes a computer on which a computer program for performing one of the methods described herein is installed.

Another embodiment of the invention includes an apparatus or system configured to transmit (e.g., electronically or optically) to a receiver a computer program for performing one of the methods described herein. The receiver may be, for example, a computer, mobile device, storage device, etc. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the method is preferably performed by any hardware device.

The apparatus described herein may be implemented using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

The methods described herein can be performed using hardware devices, computers, or a combination of hardware devices and computers.

The above embodiments are merely illustrative of the principles of the invention. It should be understood that modifications and variations of the arrangements and details described herein will be readily apparent to those skilled in the art. Therefore, the invention is intended to be limited only by the scope of the appended claims and not by the specific details presented in the manner of describing and interpreting the embodiments herein.

Implementation 1: An apparatus (201) for decoding a previously encoded multichannel signal of a previous frame to obtain three or more previous audio output channels and for decoding a currently encoded multichannel signal (107) of the current frame to obtain three or more current audio output channels.

The device (201) includes an interface (212), a channel decoder (202), a multichannel processor (204) for generating the three or more current audio output channels, and a noise filling module (220).

The interface (212) is adapted to receive the currently encoded multi-channel signal (107) and to receive auxiliary information including a first multi-channel parameter (MCH_PAR2).

The channel decoder (202) is adapted to decode the currently encoded multichannel signal of the current frame to obtain a set of three or more decoded channels (D1, D2, D3) of the current frame.

The multi-channel processor (204) is adapted to select a first selected pair of two decoded channels (D1, D2) from the set of three or more decoded channels (D1, D2) according to the first multi-channel parameter (MCH_PAR2).

The multi-channel processor (204) is adapted to generate a first set of two or more processed channels (P1*, P2*) based on a first selected pair of the two decoded channels (D1, D2) to obtain an updated set of three or more decoded channels (D3, P1*, P2*).

Before the multichannel processor (204) generates a first pair of channels of the two or more processed channels (P1*, P2*) based on a first selected pair of the two decoded channels (D1, D2), the noise filling module (220) is adapted to identify one or more frequency bands in which all spectral lines of the first selected pair of the two decoded channels (D1, D2) are quantized to zero, and is adapted to generate a mixed channel using two or more, but not all, of the three or more previous audio output channels, and is adapted to fill the spectral lines of the one or more frequency bands in which all spectral lines of the mixed channel are quantized to zero with noise generated using the spectral lines of the mixed channel, wherein the noise filling module (220) is adapted to select two or more previous audio output channels from the three or more previous audio output channels for generating the mixed channel according to the auxiliary information.

Implementation Method 2: The apparatus (201) according to Implementation Method 1,

The noise filling module (220) is adapted to generate the mixed channel by using exactly two of the three or more previous audio output channels as the two or more previous audio output channels among the three or more previous audio output channels;

The noise filling module (220) is adapted to select exactly two previous audio output channels from the three or more previous audio output channels based on the auxiliary information.

Implementation method 3: The apparatus (201) according to implementation method 2,

The noise filling module (220) is adapted to be based on the following equation

Or based on the following equation

The mixed channel is generated using exactly two of the previous audio output channels.

Wherein, D _ch is the mixed audio channel.

Here, is the first channel of exactly two previously output audio channels.

Here, is the second channel of the exactly two previous audio output channels, and the second channel is different from the first channel of the exactly two previous audio output channels.

Where d is a real positive scalar.

Implementation method 4: The apparatus (201) according to implementation method 2,

The noise filling module (220) is adapted to be based on the following equation

Or based on the following equation

The mixed channel is generated using exactly two of the previous audio output channels.

Wherein, is the mixed audio channel,

Here, is the first channel of exactly two previously output audio channels.

Here, is the second channel of the exactly two previous audio output channels, and the second channel is different from the first channel of the exactly two previous audio output channels.

Where α is the rotation angle.

Implementation method 5: The apparatus (201) according to implementation method 4,

The auxiliary information refers to the current auxiliary information assigned to the current frame.

The interface (212) is adapted to receive previous auxiliary information allocated to a previous frame, wherein the previous auxiliary information includes a previous angle.

The interface (212) is adapted to receive the current assistance information including the current angle, and

The noise filling module (220) is adapted to use the current angle of the current auxiliary information as the rotation angle α, and is also adapted not to use the previous angle of the previous auxiliary information as the rotation angle α.

Embodiment 6: The apparatus (201) according to any one of Embodiments 2 to 5, wherein the noise filling module (220) is adapted to select exactly two previous audio output channels from the three or more previous audio output channels according to the first multi-channel parameter (MCH_PAR2).

Embodiment 7: The apparatus (201) according to any one of Embodiments 2-6,

The interface (212) is adapted to receive the currently encoded multi-channel signal (107) and to receive the auxiliary information including the first multi-channel parameter (MCH_PAR2) and the second multi-channel parameter (MCH_PAR1).

The multichannel processor (204) is adapted to select a second selected pair of two decoded channels (P1*, D3) from the updated set of three or more decoded channels (D3, P1*, P2*) according to the second multichannel parameter (MCH_PAR1), wherein at least one channel (P1*) in the second selected pair of the two decoded channels (P1*, D3) is one of the first pair of channels of the two or more processed channels (P1*, P2*), and

The multi-channel processor (204) is adapted to generate a second set of two or more processed channels (P3*, P4*) based on a second selected pair of the two decoded channels (P1*, D3) to further update the set of the updated three or more decoded channels.

Implementation method 8: The apparatus (201) according to implementation method 7,

The multi-channel processor (204) is adapted to generate a first set of two or more processed channels (P1*, P2*) by generating a first set of exactly two processed channels (P1*, P2*) based on a first selected pair of the two decoded channels (D1, D2).

The multichannel processor (204) is adapted to replace the first selected pair of two decoded channels (D1, D2) in the set of three or more decoded channels (D1, D2, D3) with exactly two processed channels (P1*, P2*) from the first set to obtain the updated set of three or more decoded channels (D3, P1*, P2*).

The multi-channel processor (204) is adapted to generate a second set of two or more processed channels (P3*, P4*) by generating a second set of exactly two processed channels (P3*, P4*) based on a second selected pair of the two decoded channels (P1*, D3), and

The multichannel processor (204) is adapted to replace the second selected pair of two decoded channels (P1*, D3) in the set of three or more decoded channels (D3, P1*, P2*) with exactly two processed channels (P3*, P4*) of the second set to further update the set of three or more decoded channels.

Embodiment 9: The apparatus (201) according to Embodiment 8,

Wherein, the first multichannel parameter (MCH_PAR2) indicates two decoded channels (D1, D2) in the set of three or more decoded channels;

The multichannel processor (204) is adapted to select a first selected pair of the two decoded channels (D1, D2) from the set of three or more decoded channels (D1, D2) by selecting the two decoded channels (D1, D2) indicated by the first multichannel parameter (MCH_PAR2);

Wherein, the second multichannel parameter (MCH_PAR1) indicates two decoded channels (P1*, D3) in the set of three or more decoded channels after the update;

The multichannel processor (204) is adapted to select a second selected pair of the two decoded channels (P1*, D3) from the updated set of three or more decoded channels (D3, P1*, P2*) by selecting the two decoded channels (P1*, D3) indicated by the second multichannel parameter (MCH_PAR1).

Implementation 10: The apparatus (201) according to Implementation 9,

The device (201) is adapted to assign an identifier from an identifier set to each of the three or more previous audio output channels, such that each of the three or more previous audio output channels is assigned exactly one identifier from the identifier set, and that each identifier from the identifier set is assigned to exactly one of the three or more previous audio output channels.

The device (201) is adapted to assign an identifier from the identifier set to each channel in the set of three or more decoded channels (D1, D2, D3), such that each channel in the set of three or more decoded channels is assigned exactly one identifier from the identifier set, and such that each identifier in the identifier set is assigned to exactly one channel in the set of three or more decoded channels (D1, D2, D3).

Wherein, the first multichannel parameter (MCH_PAR2) indicates the first pair of two identifiers in a set of three or more identifiers.

The multichannel processor (204) is adapted to select a first selected pair of two decoded channels (D1, D2) from a set of three or more decoded channels (D1, D2) by selecting two decoded channels (D1, D2) of the two identifiers assigned to the first pair of two identifiers.

The device (201) is adapted to assign the first identifier of the two identifiers of the first pair of two identifiers to the first processing channel (P1*, P2*) of the first group of exactly two processing channels, and the device (210) is adapted to assign the second identifier of the two identifiers of the first pair of two identifiers to the second processing channel (P1*, P2*) of the first group of exactly two processing channels.

Implementation method 11: The apparatus (201) according to implementation method 10,

Wherein, the second multi-channel parameter (MCH_PAR1) indicates the second pair of two identifiers in the set of three or more identifiers.

The multichannel processor (204) is adapted to select a second selected pair of the two decoded channels (P1*, D3) from the updated set of three or more decoded channels (D3, P1*, P2*) by selecting two decoded channels (D3, P1*) of the two identifiers assigned to the second pair of two identifiers.

The device (201) is adapted to assign the first identifier of the two identifiers of the second pair of two identifiers to the first processing channel of the second set of exactly two processing channels (P3*, P4*), and the device (201) is adapted to assign the second identifier of the two identifiers of the second pair of two identifiers to the second processing channel of the second set of exactly two processing channels (P3*, P4*).

Implementation method 12: The apparatus (201) according to implementation method 10 or 11,

Wherein, the first multi-channel parameter (MCH_PAR2) indicates the first pair of two identifiers in the set of three or more identifiers, and

The noise filling module (220) is adapted to select exactly two previous audio output channels from three or more previous audio output channels by selecting two previous audio output channels of the two identifiers assigned to the first pair of two identifiers.

Implementation 13: The apparatus (201) according to any one of the preceding embodiments, wherein, before the multichannel processor (204) generates a first pair of channels of the two or more processed channels (P1*, P2*) based on a first selected pair of the two decoded channels (D1, D2), the noise filling module (220) is adapted to identify one or more scaling factor bands in which all spectral lines of the first selected pair of the two decoded channels (D1, D2) are quantized to zero, the one or more scaling factor bands being the one or more frequency bands, and is adapted to generate the mixed channel using two or more but not all of the three or more previous audio output channels, and is adapted to fill the spectral lines of the one or more scaling factor bands in which all spectral lines of the mixed channel are quantized to zero with noise generated using the spectral lines of the mixed channel, according to the scaling factor of each of the one or more scaling factor bands in which all spectral lines of the mixed channel are quantized to zero.

Embodiment 14: The apparatus (201) according to Embodiment 13,

The receiving interface (212) is configured to receive the scaling factor of each scaling factor band in the one or more scaling factor bands, and

Wherein, the scaling factor of each scaling factor band in the one or more scaling factor bands indicates the energy of the spectral line of the scaling factor band before quantization, and

The noise filling module (220) is adapted to generate the noise for each of the one or more scaling factor bands in which all spectral lines are quantized to zero, such that after the noise is added to one of the frequency bands, the energy of the spectral line corresponds to the energy indicated by the scaling factor of the scaling factor band.

Embodiment 15: An apparatus (100) for encoding a multichannel signal (101) having at least three channels (CH1:CH3), wherein the apparatus comprises:

An iterative processor (102) is adapted to calculate, in a first iterative step, the interchannel correlation value between each pair of channels in the at least three channels (CH1:CH3) for selecting, in the first iterative step, the channel pair having the highest value or a value above a threshold, and for processing the selected channel pair using multichannel processing operations (110, 112) to derive the initial multichannel parameters (MCH_PAR1) of the selected channel pair and derive the channels (P1, P2) of the first processing.

The iterative processor (102) is adapted to perform the calculation, the selection and the processing in the second iterative step using at least one of the processing channels (P1) to derive other multi-channel parameters (MCH_PAR2) and the second processing channels (P3, P4).

A channel encoder, adapted to encode the channels (P2:P4) obtained by the iterative processing performed by the iterative processor (104) to obtain encoded channels (E1:E3); and

An output interface (106) is adapted to generate an encoded multichannel signal (107) having the encoded channels (E1:E3), the initial multichannel parameters, and the other multichannel parameters (MCH_PAR1, MCH_PAR2), and having information indicating whether the decoding device needs to fill the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero with noise generated based on previously decoded audio output channels that have been previously decoded by the decoding device.

Implementation method 16: The apparatus (100) according to implementation method 15,

Wherein, each of the initial multichannel parameters and the other multichannel parameters (MCH_PAR1, MCH_PAR2) indicates exactly two channels, each of the exactly two channels being one of the encoded channels (E1:E3), one of the channels of the first processing, one of the channels of the second processing (P1, P2, P3, P4), or one of the at least three channels (CH1:CH3).

The output interface (106) is adapted to generate the encoded multichannel signal (107) such that the information indicating whether the device for decoding needs to fill the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero includes information indicating whether, for each of the initial multichannel parameters and the other multichannel parameters (MCH_PAR1, MCH_PAR2), for at least one of exactly two channels indicated by the parameters in the initial multichannel parameters and the other multichannel parameters (MCH_PAR1, MCH_PAR2), the device for decoding needs to fill the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero with spectral data generated based on previously decoded audio output channels that have previously been decoded by the device for decoding.

Implementation method 17: A system comprising:

The encoding device (100) according to embodiment 15 or 16, and

According to any one of embodiments 1 to 14, the device (201) for decoding,

The decoding device (201) is configured to receive the encoded multi-channel signal (107) generated by the encoding device (100) from the encoding device (100).

Implementation 18: A method for decoding a previously encoded multichannel signal of a previous frame to obtain three or more previous audio output channels, and for decoding a currently encoded multichannel signal (107) of the current frame to obtain three or more current audio output channels, wherein the method includes:

Receive the currently encoded multichannel signal (107), and receive auxiliary information including the first multichannel parameter (MCH_PAR2);

Decode the currently encoded multichannel signal of the current frame to obtain a set of three or more decoded channels (D1, D2, D3) of the current frame;

Based on the first multichannel parameter (MCH_PAR2), a first selected pair of two decoded channels (D1, D2) is selected from the set of three or more decoded channels (D1, D2).

Based on the first selected pair of the two decoded channels (D1, D2), a first set of two or more processed channels (P1*, P2*) is generated to obtain an updated set of three or more decoded channels (D3, P1*, P2*).

Before generating the first pair of channels (P1*, P2*) of the two or more processed channels based on the first selected pair of the two decoded channels (D1, D2), the following steps are performed:

For the two decoded channels (D1, D2), at least one channel of the first selected pair is used to identify one or more frequency bands in which all spectral lines are quantized to zero, and a mixed channel is generated using two or more, but not all, of the three or more previous audio output channels, and the spectral lines of the one or more frequency bands in which all spectral lines are quantized to zero are filled with noise generated using the spectral lines of the mixed channel, wherein two or more previous audio output channels are selected from the three or more previous audio output channels to generate the mixed channel according to the auxiliary information.

Implementation 19: A method for encoding a multichannel signal (101) having at least three channels (CH1:CH3), wherein the method includes:

In the first iteration step, the interchannel correlation value between each pair of channels in the at least three channels (CH1:CH3) is calculated to select the channel pair with the highest value or a value above a threshold in the first iteration step, and the selected channel pair is processed using multichannel processing operations (110, 112) to derive the initial multichannel parameters (MCH_PAR1) of the selected channel pair and derive the channels (P1, P2) of the first processing.

In the second iteration step, the calculation, selection and processing are performed using at least one channel (P1) of the processed channels to derive other multichannel parameters (MCH_PAR2) and the channels (P3, P4) of the second processing.

The audio channels (P2:P4) obtained by the iterative processing performed by the iterative processor (104) are encoded to obtain encoded audio channels (E1:E3); and

Generate an encoded multichannel signal (107) having the encoded channels (E1:E3), the initial multichannel parameters and the other multichannel parameters (MCH_PAR1, MCH_PAR2), and having information indicating whether the decoding device needs to fill the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero with noise generated based on previously decoded audio output channels that have been previously decoded by the decoding device.

Implementation 20: A computer program, when executed on a computer or signal processor, for implementing the method according to Implementation 18 or 19.

Implementation method 21: An encoded multi-channel signal (107), comprising:

Encoded audio channels (E1:E3),

Multichannel parameters (MCH_PAR1, MCH_PAR2); and

The information indicates whether the device used for decoding needs to fill the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero with noise generated based on the previously decoded audio output channel, which has been previously decoded by the device used for decoding.

Implementation method 22: The multi-channel signal (107) encoded according to implementation method 21,

The encoded multichannel signal includes two or more multichannel parameters (MCH_PAR1, MCH_PAR2) as the multichannel parameters (MCH_PAR1, MCH_PAR2).

Wherein, each of the two or more multi-channel parameters (MCH_PAR1, MCH_PAR2) indicates exactly two channels, each of the exactly two channels being one of the encoded channels (E1:E3), one of the multiple processed channels (P1, P2, P3, P4), or one of at least three initial channels (CH:CH3).

The information indicating whether the decoding device needs to fill the spectral lines of one or more frequency bands where all spectral lines are quantized to zero includes information indicating whether, for each of the two or more multichannel parameters (MCH_PAR1, MCH_PAR2), for at least one of the exactly two channels indicated by the parameter of the two or more multichannel parameters, the decoding device needs to fill the spectral lines of one or more frequency bands where all spectral lines are quantized to zero with spectral data generated based on previously decoded audio output channels that have previously been decoded by the decoding device.

References

[1]ISO/IEC international standard 23008-3:2015, "Information technology-High efficiency coding and media delivery in heterogeneousenvironments-Part 3:3Daudio," March 2015

[2]ISO/IEC amendment 23008-3:2015/PDAM3, "Information technology-Highefficiency coding and media delivery in heterogeneous environments-Part 3: 3Daudio, Amendment 3: MPEG-H 3D Audio Phase 2," July 2015

[3]International Organization for Standardization,ISO/IEC 23003-3:2012,″Information Technology-MPEG audio-Part 3:Unified speech and audiocoding,″Geneva,Jan.2012

[4]ISOIIEC 23003-1:2007-Information technology-MPEG audiotechnologies Part1:MPEG Surround

[5] C.R.Helmrich, A.Niedermeier, S.Bayer, B.Edler, "Low-Complexity Semi-Parametric Joint-Stereo Audio Transform Coding," in Proc.EUSIPCO, Nice, September 2015

[6]ETSI TS 103 190 V1.1.1(2014-04)-Digital Audio Compression(AC-4)Standard

[7]Yang, Dai and Ai, Hongmei and Kyriakakis, Chris and Kuo, C.-C.Jay, 2001: Adaptive Karhunen-Loeve Transform for Enhanced Multichanne t AudioCoding, http://iict.usc.edu/pubs/Adaptive%20Karhunen-Loeve%20Transform%20for%20Enhanced%20Multichannel%20Audio%20Coding.pdf

[8]European Patent Application, Publication EP 2 830 060 A1: "Noisefilling in multichannel audio coding", published on 28January 2015

[9]Internet Engineering Task Force(IETF),RFC 6716,″Definition of theOpus Audio Codec,″Int.Standard,Sep.2012.Available online at:http://tools.ietf.org/html/rfc6716

[10]International Organization for Standardization, ISO/IEC 14496-3:2009, "Information Technology-Coding of audio-visual objects-Part 3:Audio," Geneva, Switzerland, Aug.2009

[11] M.Neuendorf et al., "MPEG Unified Speech and Audio Coding-The ISO/MPEG Standard for High-Efficiency Audio Coding of All Content Types," in Proc.132 ^nd AES Convention, Budapest, Hungary, Apr.2012. Also to appear in the Journal of the AES, 2013

Claims (22)

1. An apparatus (201) for decoding a previously encoded multichannel signal of a previous frame to obtain three or more previous audio output channels and for decoding a currently encoded multichannel signal (107) of the current frame to obtain three or more current audio output channels. The device (201) includes an interface (212), a channel decoder (202), a multichannel processor (204) for generating the three or more current audio output channels, and a noise filling module (220). The interface (212) is adapted to receive the currently encoded multi-channel signal (107) and to receive auxiliary information including a first multi-channel parameter (MCH_PAR2). The channel decoder (202) is adapted to decode the currently encoded multichannel signal of the current frame to obtain a set of three or more decoded channels (D1, D2, D3) of the current frame. The multi-channel processor (204) is adapted to select a first selected pair of two decoded channels (D1, D2) from the set of three or more decoded channels (D1, D2) according to the first multi-channel parameter (MCH_PAR2). The multi-channel processor (204) is adapted to generate a first set of two or more processed channels (P1*, P2*) based on a first selected pair of the two decoded channels (D1, D2) to obtain an updated set of three or more decoded channels (D3, P1*, P2*). Before the multichannel processor (204) generates a first pair of channels of the two or more processed channels (P1*, P2*) based on a first selected pair of the two decoded channels (D1, D2), the noise filling module (220) is adapted to identify one or more frequency bands in which all spectral lines of the first selected pair of the two decoded channels (D1, D2) are quantized to zero, and is adapted to generate a mixed channel using two or more, but not all, of the three or more previous audio output channels, and is adapted to fill the spectral lines of the one or more frequency bands in which all spectral lines of the mixed channel are quantized to zero with noise generated using the spectral lines of the mixed channel, wherein the noise filling module (220) is adapted to select two or more previous audio output channels from the three or more previous audio output channels for generating the mixed channel according to the auxiliary information. 2. The apparatus (201) according to claim 1, The noise filling module (220) is adapted to generate the mixed channel by using exactly two of the three or more previous audio output channels as the two or more previous audio output channels among the three or more previous audio output channels; The noise filling module (220) is adapted to select exactly two previous audio output channels from the three or more previous audio output channels based on the auxiliary information. 3. The apparatus (201) according to claim 2, The noise filling module (220) is adapted to be based on the following equation Or based on the following equation The mixed channel is generated using exactly two of the previous audio output channels. Wherein, D _ch is the mixed audio channel. Here, is the first channel of exactly two previously output audio channels. Here, is the second channel of the exactly two previous audio output channels, and the second channel is different from the first channel of the exactly two previous audio output channels. Where d is a real positive scalar. 4. The apparatus (201) according to claim 2, The noise filling module (220) is adapted to be based on the following equation Or based on the following equation The mixed channel is generated using exactly two of the previous audio output channels. Wherein, is the mixed audio channel, Here, is the first channel of exactly two previously output audio channels. Here, is the second channel of the exactly two previous audio output channels, and the second channel is different from the first channel of the exactly two previous audio output channels. Where α is the rotation angle. 5. The apparatus (201) according to claim 4, The auxiliary information refers to the current auxiliary information assigned to the current frame. The interface (212) is adapted to receive previous auxiliary information allocated to a previous frame, wherein the previous auxiliary information includes a previous angle. The interface (212) is adapted to receive the current assistance information including the current angle, and The noise filling module (220) is adapted to use the current angle of the current auxiliary information as the rotation angle α, and is also adapted not to use the previous angle of the previous auxiliary information as the rotation angle α. 6. The apparatus (201) according to any one of claims 2 to 5, wherein the noise filling module (220) is adapted to select exactly two previous audio output channels from the three or more previous audio output channels according to the first multichannel parameter (MCH_PAR2). 7. The apparatus (201) according to any one of claims 2-6, The interface (212) is adapted to receive the currently encoded multi-channel signal (107) and to receive the auxiliary information including the first multi-channel parameter (MCH_PAR2) and the second multi-channel parameter (MCH_PAR1). The multichannel processor (204) is adapted to select a second selected pair of two decoded channels (P1*, D3) from the updated set of three or more decoded channels (D3, P1*, P2*) according to the second multichannel parameter (MCH_PAR1), wherein at least one channel (P1*) in the second selected pair of the two decoded channels (P1*, D3) is one of the first pair of channels of the two or more processed channels (P1*, P2*), and The multi-channel processor (204) is adapted to generate a second set of two or more processed channels (P3*, P4*) based on a second selected pair of the two decoded channels (P1*, D3) to further update the set of the updated three or more decoded channels. 8. The apparatus (201) according to claim 7, The multi-channel processor (204) is adapted to generate a first set of two or more processed channels (P1*, P2*) by generating a first set of exactly two processed channels (P1*, P2*) based on a first selected pair of the two decoded channels (D1, D2). The multichannel processor (204) is adapted to replace the first selected pair of two decoded channels (D1, D2) in the set of three or more decoded channels (D1, D2, D3) with exactly two processed channels (P1*, P2*) from the first set to obtain the updated set of three or more decoded channels (D3, P1*, P2*). The multi-channel processor (204) is adapted to generate a second set of two or more processed channels (P3*, P4*) by generating a second set of exactly two processed channels (P3*, P4*) based on a second selected pair of the two decoded channels (P1*, D3), and The multichannel processor (204) is adapted to replace the second selected pair of two decoded channels (P1*, D3) in the set of three or more decoded channels (D3, P1*, P2*) with exactly two processed channels (P3*, P4*) of the second set to further update the set of three or more decoded channels. 9. The apparatus (201) according to claim 8, Wherein, the first multichannel parameter (MCH_PAR2) indicates two decoded channels (D1, D2) in the set of three or more decoded channels; The multichannel processor (204) is adapted to select a first selected pair of the two decoded channels (D1, D2) from the set of three or more decoded channels (D1, D2) by selecting the two decoded channels (D1, D2) indicated by the first multichannel parameter (MCH_PAR2); Wherein, the second multichannel parameter (MCH_PAR1) indicates two decoded channels (P1*, D3) in the set of three or more decoded channels after the update; The multichannel processor (204) is adapted to select a second selected pair of the two decoded channels (P1*, D3) from the updated set of three or more decoded channels (D3, P1*, P2*) by selecting the two decoded channels (P1*, D3) indicated by the second multichannel parameter (MCH_PAR1). 10. The apparatus (201) according to claim 9, The device (201) is adapted to assign an identifier from an identifier set to each of the three or more previous audio output channels, such that each of the three or more previous audio output channels is assigned exactly one identifier from the identifier set, and that each identifier from the identifier set is assigned to exactly one of the three or more previous audio output channels. The device (201) is adapted to assign an identifier from the identifier set to each channel in the set of three or more decoded channels (D1, D2, D3), such that each channel in the set of three or more decoded channels is assigned exactly one identifier from the identifier set, and such that each identifier in the identifier set is assigned to exactly one channel in the set of three or more decoded channels (D1, D2, D3). Wherein, the first multichannel parameter (MCH_PAR2) indicates the first pair of two identifiers in a set of three or more identifiers. The multichannel processor (204) is adapted to select a first selected pair of two decoded channels (D1, D2) from a set of three or more decoded channels (D1, D2) by selecting two decoded channels (D1, D2) of the two identifiers assigned to the first pair of two identifiers. The device (201) is adapted to assign the first identifier of the two identifiers of the first pair of two identifiers to the first processing channel (P1*, P2*) of the first group of exactly two processing channels, and the device (210) is adapted to assign the second identifier of the two identifiers of the first pair of two identifiers to the second processing channel (P1*, P2*) of the first group of exactly two processing channels. 11. The apparatus (201) according to claim 10, Wherein, the second multi-channel parameter (MCH_PAR1) indicates the second pair of two identifiers in the set of three or more identifiers. The multichannel processor (204) is adapted to select a second selected pair of the two decoded channels (P1*, D3) from the updated set of three or more decoded channels (D3, P1*, P2*) by selecting two decoded channels (D3, P1*) of the two identifiers assigned to the second pair of two identifiers. The device (201) is adapted to assign the first identifier of the two identifiers of the second pair of two identifiers to the first processing channel of the second group of exactly two processing channels (P3*, P4*), and the device (201) is adapted to assign the second identifier of the two identifiers of the second pair of two identifiers to the second processing channel of the second group of exactly two processing channels (P3*, P4*). 12. The apparatus (201) according to claim 10 or 11, Wherein, the first multi-channel parameter (MCH_PAR2) indicates the first pair of two identifiers in the set of three or more identifiers, and The noise filling module (220) is adapted to select exactly two previous audio output channels from three or more previous audio output channels by selecting two previous audio output channels of the two identifiers assigned to the first pair of two identifiers. 13. The apparatus (201) according to any one of the preceding claims, wherein, before the multichannel processor (204) generates a first pair of channels of the two or more processed channels (P1*, P2*) based on a first selected pair of the two decoded channels (D1, D2), the noise filling module (220) is adapted to identify one or more scaling factor bands in which all spectral lines of the first selected pair of the two decoded channels (D1, D2) are quantized to zero, the one or more scaling factor bands being the one or more frequency bands, and is adapted to generate the mixed channel using two or more, but not all, of the three or more previous audio output channels, and is adapted to fill the spectral lines of the one or more scaling factor bands in which all spectral lines of the mixed channel are quantized to zero with noise generated using the spectral lines of the mixed channel, according to the scaling factor of each of the one or more scaling factor bands in which all spectral lines of the mixed channel are quantized to zero. 14. The apparatus (201) according to claim 13, The receiving interface (212) is configured to receive the scaling factor of each scaling factor band in the one or more scaling factor bands, and Wherein, the scaling factor of each scaling factor band in the one or more scaling factor bands indicates the energy of the spectral line of the scaling factor band before quantization, and The noise filling module (220) is adapted to generate the noise for each of the one or more scaling factor bands in which all spectral lines are quantized to zero, such that after the noise is added to one of the frequency bands, the energy of the spectral line corresponds to the energy indicated by the scaling factor of the scaling factor band. 15. An apparatus (100) for encoding a multichannel signal (101) having at least three channels (CH1:CH3), wherein the apparatus comprises: An iterative processor (102) is adapted to calculate, in a first iterative step, the interchannel correlation value between each pair of channels in the at least three channels (CH1:CH3) for selecting, in the first iterative step, the channel pair having the highest value or a value above a threshold, and for processing the selected channel pair using multichannel processing operations (110, 112) to derive the initial multichannel parameters (MCH_PAR1) of the selected channel pair and derive the channels (P1, P2) of the first processing. The iterative processor (102) is adapted to perform the calculation, the selection and the processing in the second iterative step using at least one of the processing channels (P1) to derive other multi-channel parameters (MCH_PAR2) and the second processing channels (P3, P4). A channel encoder, adapted to encode the channels (P2:P4) obtained by the iterative processing performed by the iterative processor (104) to obtain encoded channels (E1:E3); and The output interface (106) is adapted to generate an encoded multichannel signal (107) having the encoded channels (E1:E3), the initial multichannel parameters and the other multichannel parameters (MCH_PAR1, MCH_PAR2), and having information indicating whether the decoding device needs to fill the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero with noise generated based on previously decoded audio output channels that have been previously decoded by the decoding device. 16. The apparatus (100) according to claim 15, Wherein, each of the initial multichannel parameters and the other multichannel parameters (MCH_PAR1, MCH_PAR2) indicates exactly two channels, each of the exactly two channels being one of the encoded channels (E1:E3), one of the channels of the first processing, one of the channels of the second processing (P1, P2, P3, P4), or one of the at least three channels (CH1:CH3). The output interface (106) is adapted to generate the encoded multichannel signal (107) such that the information indicating whether the device for decoding needs to fill the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero includes information indicating whether, for each of the initial multichannel parameters and the other multichannel parameters (MCH_PAR1, MCH_PAR2), for at least one of exactly two channels indicated by the parameters in the initial multichannel parameters and the other multichannel parameters (MCH_PAR1, MCH_PAR2), the device for decoding needs to fill the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero with spectral data generated based on previously decoded audio output channels that have previously been decoded by the device for decoding. 17. A system comprising: The encoding apparatus (100) according to claim 15 or 16, and The device (201) for decoding according to any one of claims 1 to 14, The decoding device (201) is configured to receive the encoded multi-channel signal (107) generated by the encoding device (100) from the encoding device (100). 18. A method for decoding a previously encoded multichannel signal of a previous frame to obtain three or more previous audio output channels, and for decoding a currently encoded multichannel signal (107) of a current frame to obtain three or more current audio output channels, wherein the method comprises: Receive the currently encoded multichannel signal (107), and receive auxiliary information including the first multichannel parameter (MCH_PAR2); Decode the currently encoded multichannel signal of the current frame to obtain a set of three or more decoded channels (D1, D2, D3) of the current frame; Based on the first multichannel parameter (MCH_PAR2), a first selected pair of two decoded channels (D1, D2) is selected from the set of three or more decoded channels (D1, D2). Based on the first selected pair of the two decoded channels (D1, D2), a first set of two or more processed channels (P1*, P2*) is generated to obtain an updated set of three or more decoded channels (D3, P1*, P2*). Before generating the first pair of channels (P1*, P2*) of the two or more processed channels based on the first selected pair of the two decoded channels (D1, D2), the following steps are performed: For the two decoded channels (D1, D2), at least one channel of the first selected pair is used to identify one or more frequency bands in which all spectral lines are quantized to zero, and a mixed channel is generated using two or more, but not all, of the three or more previous audio output channels, and the spectral lines of the one or more frequency bands in which all spectral lines are quantized to zero are filled with noise generated using the spectral lines of the mixed channel, wherein two or more previous audio output channels are selected from the three or more previous audio output channels to generate the mixed channel according to the auxiliary information. 19. A method for encoding a multichannel signal (101) having at least three channels (CH1:CH3), wherein the method comprises: In the first iteration step, the interchannel correlation value between each pair of channels in the at least three channels (CH1:CH3) is calculated to select the channel pair with the highest value or a value above a threshold in the first iteration step, and the selected channel pair is processed using multichannel processing operations (110, 112) to derive the initial multichannel parameters (MCH_PAR1) of the selected channel pair and derive the channels (P1, P2) of the first processing. In the second iteration step, the calculation, selection and processing are performed using at least one channel (P1) of the processed channels to derive other multichannel parameters (MCH_PAR2) and the channels (P3, P4) of the second processing. The audio channels (P2:P4) obtained by the iterative processing performed by the iterative processor (104) are encoded to obtain encoded audio channels (E1:E3); and Generate an encoded multichannel signal (107) having the encoded channels (E1:E3), the initial multichannel parameters and the other multichannel parameters (MCH_PAR1, MCH_PAR2), and having information indicating whether the decoding device needs to fill the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero with noise generated based on previously decoded audio output channels that have been previously decoded by the decoding device. 20. A computer program, when executed on a computer or signal processor, for carrying out the method according to claim 18 or 19. 21. An encoded multichannel signal (107), comprising: Encoded audio channels (E1:E3), Multichannel parameters (MCH_PAR1, MCH_PAR2); and The information indicates whether the device used for decoding needs to fill the spectral lines of one or more frequency bands in which all spectral lines are quantized to zero with noise generated based on the previously decoded audio output channel, which has been previously decoded by the device used for decoding. 22. The encoded multi-channel signal (107) according to claim 21, The encoded multichannel signal includes two or more multichannel parameters (MCH_PAR1, MCH_PAR2) as the multichannel parameters (MCH_PAR1, MCH_PAR2). Wherein, each of the two or more multi-channel parameters (MCH_PAR1, MCH_PAR2) indicates exactly two channels, each of the exactly two channels being one of the encoded channels (E1:E3), one of the multiple processed channels (P1, P2, P3, P4), or one of at least three initial channels (CH:CH3). The information indicating whether the decoding device needs to fill the spectral lines of one or more frequency bands where all spectral lines are quantized to zero includes information indicating whether, for each of the two or more multichannel parameters (MCH_PAR1, MCH_PAR2), for at least one of the exactly two channels indicated by the parameter of the two or more multichannel parameters, the decoding device needs to fill the spectral lines of one or more frequency bands where all spectral lines are quantized to zero with spectral data generated based on previously decoded audio output channels that have previously been decoded by the decoding device.