HK1218589B - Coding of audio scenes - Google Patents

Description

Encoding of audio scenes

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent Application No. 61/827,246, filed May 24, 2013, which is hereby incorporated by reference in its entirety.

Technical Field

The invention disclosed herein generally relates to the field of audio encoding and decoding. In particular, the invention relates to encoding and decoding of audio scenes comprising audio objects.

Background Art

There are audio coding systems for parametric spatial audio coding. For example, MPEG Surround describes a system for parametric spatial coding of multi-channel audio. MPEG SAOC (Spatial Audio Object Coding) describes a system for parametric coding of audio objects.

On the encoder side, these systems typically downmix the channels/objects into a downmix, which is typically a mono (one channel) or stereo (two channels) downmix, and extract side information that describes the properties of the channels/objects through, for example, level differences and cross-correlations. The downmix and side information are then encoded and sent to the decoder. On the decoder side, the channels/objects are reconstructed or approximated from the downmix under the control of the parameters of the side information.

A drawback of these systems is that the reconstruction is often mathematically complex and often relies on assumptions about the nature of the audio content that are not explicitly described by the parameters sent as side information. Such assumptions can include, for example, that channels/objects are considered uncorrelated unless cross-correlation parameters are sent, or that the downmix of channels/objects is generated in a specific way. Furthermore, the mathematical complexity and the need for additional assumptions increase significantly as the number of channels in the downmix increases.

Furthermore, the required assumptions are inherently reflected in the algorithmic details of the processing applied on the decoder side. This means that considerable intelligence must be included on the decoder side. This is a disadvantage because it is difficult to upgrade and improve the algorithm when the decoder is, for example, installed in a consumer device where upgrading is difficult or even impossible.

BRIEF DESCRIPTION OF THE DRAWINGS

Hereinafter, example embodiments will be described in more detail with reference to the accompanying drawings, in which:

FIG1 is a schematic diagram of an audio encoding/decoding system according to an example embodiment;

2 is a schematic diagram of an audio encoding/decoding system with a legacy decoder according to an example embodiment;

FIG3 is a schematic diagram of an encoding side of an audio encoding/decoding system according to an example embodiment;

FIG4 is a flowchart of an encoding method according to an example embodiment;

FIG5 is a schematic diagram of an encoder according to an example embodiment;

FIG6 is a schematic diagram of a decoder side of an audio encoding/decoding system according to an example embodiment;

FIG7 is a flowchart of a decoding method according to an example embodiment;

FIG8 is a schematic diagram of a decoder side of an audio encoding/decoding system according to an example embodiment; and

FIG9 is a schematic diagram of time-frequency transform performed at a decoder side of an audio encoding/decoding system according to an example embodiment.

All the figures are schematic and generally show only parts which are necessary to elucidate the invention, while other parts may be omitted or merely suggested. Unless otherwise indicated, the same reference numerals in the different figures denote the same components.

DETAILED DESCRIPTION

In view of the above, it is an object to provide an encoder and a decoder, and related methods that provide a less complex and more flexible reconstruction of audio objects.

I. Overview - Encoder

According to a first aspect, example embodiments propose an encoding method, an encoder and a computer program product for encoding. The proposed method, encoder and computer program product may generally have the same features and advantages.

According to an example embodiment, a method for encoding time-frequency blocks of an audio scene including at least N audio objects is provided. The method includes: receiving the N audio objects; generating M downmix signals based on the at least N audio objects; generating a reconstruction matrix using matrix elements, the reconstruction matrix enabling reconstruction of the at least N audio objects from the M downmix signals; and generating a bitstream including the M downmix signals and at least some of the matrix elements of the reconstruction matrix.

The number N of audio objects may be equal to or greater than 1. The number M of downmix signals may be equal to or greater than 1.

This method generates a bitstream that includes at least some of the matrix elements of the reconstruction matrix as side information, along with M downmix signals. By including the matrix elements of the reconstruction matrix in the bitstream, minimal intelligence is required on the decoder side. For example, complex computations of the reconstruction matrix based on the transmitted object parameters and additional assumptions are not required on the decoder side. Consequently, the mathematical complexity on the decoder side is significantly reduced. Furthermore, because the complexity of this method is independent of the number of downmix signals used, flexibility regarding the number of downmix signals is increased compared to prior art methods.

As used herein, an audio scene generally refers to a three-dimensional audio environment that includes audio units associated with positions in three-dimensional space that can be rendered for playback on an audio system.

As used herein, an audio object refers to a unit of an audio scene. An audio object typically includes an audio signal and additional information such as the object's position in three-dimensional space. This additional information is typically used to optimally render the audio object on a given playback system.

As used herein, a downmix signal refers to a signal that is a combination of at least N audio objects. Other signals of the audio scene, such as the bed channels (described below), may also be combined into the downmix signal. For example, M downmix signals may correspond to the rendering of the audio scene for a given speaker configuration, such as a standard 5.1 configuration. The number of downmix signals, denoted by M herein, is typically (but not necessarily) less than the sum of the number of audio objects and bed channels, which explains why M downmix signals are referred to as a downmix.

Audio encoding/decoding systems typically divide the time-frequency space into time-frequency blocks, for example by applying a suitable filter bank to the input audio signal. A time-frequency block generally refers to a portion of the time-frequency space corresponding to a time interval and a frequency subband. A time interval may typically correspond to the duration of a time frame used in the audio encoding/decoding system. A frequency subband may typically correspond to one or several adjacent frequency subbands defined by the filter bank used in the encoding/decoding system. When a frequency subband corresponds to several adjacent frequency subbands defined by the filter bank, this allows for uneven frequency subbands during the decoding of the audio signal, for example, wider frequency subbands for higher frequencies in the audio signal. In the case of a wide-band audio encoding/decoding system operating over the entire frequency range, the frequency subbands of a time-frequency block may correspond to the entire frequency range. The above method discloses encoding steps for encoding an audio scene during one such time-frequency block. However, it should be understood that the method may be repeated for each time-frequency block of the audio encoding/decoding system. Furthermore, it should be understood that several time-frequency blocks may be encoded simultaneously. Typically, adjacent time-frequency blocks may overlap slightly in time and/or frequency. For example, the temporal overlap may be equivalent to a linear interpolation of the elements of the reconstruction matrix in time, i.e., from one time interval to the next. However, the present disclosure is directed to other components of the encoding/decoding system, and any temporal and/or frequency overlap between adjacent time-frequency blocks is left to those skilled in the art.

According to an example embodiment, M downmix signals are arranged in a first field of a bitstream using a first format, and matrix elements are arranged in a second field of the bitstream using a second format, thereby allowing a decoder that only supports the first format to decode and play back the M downmix signals in the first field and discard the matrix elements in the second field. This has the advantage that the M downmix signals in the bitstream are backward compatible with legacy decoders that do not implement audio object reconstruction. In other words, the legacy decoder can still decode and play back the M downmix signals of the bitstream, for example, by mapping each downmix signal to a channel output of the decoder.

According to an example embodiment, the method may further include the step of receiving position data corresponding to each of the N audio objects, wherein M downmix signals are generated based on the position data. The position data typically associates each audio object with a position in three-dimensional space. The position of the audio object may change over time. By using the position data when downmixing the audio objects, the audio objects are mixed into the M downmix signals in such a way that, for example, if the M downmix signals are listened to on a system with M output channels, the audio objects sound as if they were approximately located at their respective positions. This is advantageous, for example, when the M downmix signals are to be backward compatible with legacy decoders.

According to an exemplary embodiment, the matrix elements of the reconstruction matrix are time-varying and frequency-varying. In other words, the matrix elements of the reconstruction matrix can be different for different time-frequency blocks. In this way, excellent flexibility in the reconstruction of audio objects is achieved.

According to an example embodiment, the audio scene also includes multiple bed channels. This is common, for example, in cinema audio applications where the audio content includes bed channels in addition to audio objects. In this case, M downmix signals can be generated based on at least N audio objects and the multiple bed channels. A bed channel generally refers to an audio signal corresponding to a fixed position in three-dimensional space. For example, a bed channel can correspond to one of the output channels of an audio encoding/decoding system. In this way, the bed channel can be interpreted as having a relative position in three-dimensional space that is the same as the position of one of the output speakers of the audio encoding/decoding system. Therefore, the bed channel can be associated with a label that simply indicates the position of the corresponding output speaker.

When the audio scene comprises a bed channel, the reconstruction matrix may comprise matrix elements enabling reconstruction of the bed channel from the M downmix signals.

In some cases, an audio scene may include a large number of objects. To reduce the complexity and amount of data required to represent the audio scene, the audio scene may be simplified by reducing the number of audio objects. Therefore, if the audio scene initially includes K audio objects, where K>N, the method may further include the steps of receiving the K audio objects and reducing the K audio objects to N audio objects by clustering the K audio objects into N clusters and representing each cluster with an audio object.

To simplify the scenario, the method may further include receiving position data corresponding to each of the K audio objects, wherein clustering the K objects into N clusters is based on position distances between the K objects given by the position data of the K audio objects. For example, audio objects that are located close to each other in three-dimensional space may be clustered together.

As described above, the exemplary embodiments of the method are flexible with respect to the number of downmix signals used. Specifically, the method can be advantageously used when there are more than two downmix signals, i.e., when M is greater than two. For example, five or seven downmix signals, corresponding to a conventional 5.1 or 7.1 audio setup, can be used. This is advantageous because, in contrast to prior art systems, the mathematical complexity of the proposed encoding principle remains the same regardless of the number of downmix signals used.

To further improve the reconstruction of the N audio objects, the method may further include: forming L auxiliary signals from the N audio objects; including matrix elements in a reconstruction matrix that enables reconstruction of at least the N audio objects from the M downmix signals and the L auxiliary signals; and including the L auxiliary signals in the bitstream. Thus, the auxiliary signals act as helper signals that, for example, can capture aspects of the audio objects that are difficult to reconstruct from the downmix signals. The auxiliary signals may also be based on bed channels. The number of auxiliary signals may be equal to or greater than one.

According to an exemplary embodiment, the auxiliary signal may correspond to an audio object of particular importance, such as an audio object representing a conversation. Thus, at least one of the L auxiliary signals may be identical to one of the N audio objects. This allows the important objects to be presented with higher quality than if they had to be reconstructed based solely on the M downmix channels. In practice, the audio content provider may have prioritized and/or marked some of the audio objects as being audio objects that are preferably included separately as auxiliary objects. Furthermore, this makes modifications/processing of these objects prior to presentation less prone to artifacts. As a compromise between bit rate and quality, a mixture of two or more audio objects may also be sent as the auxiliary signal. In other words, at least one of the L auxiliary signals may be formed as a combination of at least two of the N audio objects.

According to an example embodiment, the auxiliary signal represents the signal dimensions of the audio objects that were lost during the generation of the M downmix signals, for example because the number of independent objects is typically greater than the number of downmix channels, or because the positions of two objects are associated such that the two objects are mixed into the same downmix signal. An example of the latter case is when two objects are separated only vertically but share the same position when projected onto a horizontal plane, meaning that the two objects would typically be presented as the same downmix channel in a standard 5.1 surround speaker setup, where all speakers are located on the same horizontal plane. Specifically, the M downmix signals span a hyperplane in the signal space. By forming a linear combination of the M downmix signals, only audio signals that lie within the hyperplane can be reconstructed. To improve the reconstruction, auxiliary signals that do not lie within the hyperplane can be included, thereby also enabling the reconstruction of signals that do not lie within the hyperplane. In other words, according to an example embodiment, at least one of the multiple auxiliary signals does not lie within the hyperplane spanned by the M downmix signals. For example, at least one of the multiple auxiliary signals may be orthogonal to the hyperplane spanned by the M downmix signals.

According to an example embodiment, there is provided a computer readable medium comprising computer code instructions adapted to perform any of the methods of the first aspect when run on a device having processing capabilities.

According to an example embodiment, an encoder for encoding time-frequency blocks of an audio scene including at least N audio objects is provided, the encoder comprising: a receiving component configured to receive the N audio objects; a downmix generation component configured to receive the N audio objects from the receiving component and generate M downmix signals based on the at least N audio objects; an analysis component configured to generate a reconstruction matrix using matrix elements, the reconstruction matrix enabling reconstructing the at least N audio objects from the M downmix signals; and a bitstream generation component configured to receive the M downmix signals from the downmix generation component and the reconstruction matrix from the analysis component, and generate a bitstream including the M downmix signals and at least some of the matrix elements of the reconstruction matrix.

II. Overview - Decoder

According to a second aspect, example embodiments propose a decoding method, a decoding apparatus and a computer program product for decoding. The proposed method, apparatus and computer program product may generally have the same features and advantages.

The advantages relating to the features and arrangements presented in the above overview of the encoder may generally be valid for the corresponding features and arrangements of the decoder.

According to an example embodiment, a method for decoding time-frequency blocks of an audio scene including at least N audio objects is provided, the method comprising the steps of: receiving a bitstream including M downmix signals and at least some of the matrix elements of a reconstruction matrix; generating a reconstruction matrix using the matrix elements; and reconstructing the N audio objects from the M downmix signals using the reconstruction matrix.

According to an example embodiment, M downmix signals are arranged in a first field of a bitstream using a first format, and matrix elements are arranged in a second subsegment of the bitstream using a second format, thereby allowing a decoder that only supports the first format to decode and play back the M downmix signals in the first field and discard the matrix elements in the second subsegment.

According to an example embodiment, the matrix elements of the reconstruction matrix are time-varying and frequency-varying.

According to an example embodiment, the audio scene further comprises a plurality of bed channels, and the method further comprises reconstructing the bed channels from the M downmix signals using a reconstruction matrix.

According to an example embodiment, the number M of downmix signals is greater than two.

According to an example embodiment, the method further comprises: receiving L auxiliary signals formed by N audio objects; and reconstructing the N audio objects based on the M downmix signals and the L auxiliary signals using a reconstruction matrix, wherein the reconstruction matrix comprises matrix elements enabling reconstruction of at least N audio objects based on the M downmix signals and the L auxiliary signals.

According to an example embodiment, at least one of the L auxiliary signals is identical to one of the N audio objects.

According to an example embodiment, at least one of the L auxiliary signals is a combination of N audio objects.

According to an example embodiment, the M downmix signals span a hyperplane, and wherein at least one of the plurality of auxiliary signals is not located in the hyperplane spanned by the M downmix signals.

According to an example embodiment, at least one of the plurality of auxiliary signals not located in the hyperplane is orthogonal to the hyperplane spanned by the M downmix signals.

As mentioned above, audio coding/decoding systems usually work in the frequency domain. Therefore, audio coding/decoding systems use filter banks to perform time-frequency transforms of audio signals. Different types of time-frequency transforms can be used. For example, M downmix signals can be represented with respect to a first frequency domain and a reconstruction matrix can be represented with respect to a second frequency domain. In order to reduce the computational burden of the decoder, it is advantageous to select the first frequency domain and the second frequency domain in a smart way. For example, the first frequency domain and the second frequency domain can be selected to be the same frequency domain, such as a modified discrete cosine transform (MDCF) domain. In this way, it is possible to avoid transforming the M downmix signals from the first frequency domain to the time domain and then to the second frequency domain in the decoder. Alternatively, the first frequency domain and the second frequency domain can be selected in the following manner: the transform from the first frequency domain to the second frequency domain can be implemented together, so that there is no need to pass through the time domain between the first frequency domain and the second frequency domain.

The method may further include receiving position data corresponding to the N audio objects, and rendering the N audio objects using the position data to create at least one output audio channel. In this manner, the reconstructed N audio objects are mapped to output channels of the audio encoder/decoder system based on their positions in three-dimensional space.

Preferably, presentation is performed in the frequency domain. In order to reduce the computational burden of the decoder, it is preferably selected in a smart way about the frequency domain of the reconstructed audio object to be presented. For example, if a reconstruction matrix is represented about the second frequency domain corresponding to the second filter group, and presentation is performed in the third frequency domain corresponding to the third filter group, then the second filter group and the third filter group are preferably selected to be at least partially the same filter group. For example, the second filter group and the third filter group can include a quadrature mirror filter (QMF) domain. Alternatively, the second frequency domain and the third frequency domain can include an MDCT filter group. According to an exemplary embodiment, the third filter group can be composed of a series of filter groups, such as a QMF filter group, followed by a Nyquist filter group. If so, at least one of the filter groups of the sequence (the first filter group of the sequence) is identical to the second filter group. In this way, it can be said that the second filter group and the third filter group are at least partially the same filter group.

According to an example embodiment, there is provided a computer readable medium comprising computer code instructions adapted to perform any of the methods of the second aspect when run on a device having processing capabilities.

According to an example embodiment, a decoder for decoding time-frequency blocks of an audio scene including at least N audio objects is provided, the decoder including: a receiving component configured to receive a bitstream including M downmix signals and at least some of the matrix elements of a reconstruction matrix; a reconstruction matrix generating component configured to receive the matrix elements from the receiving component and generate a reconstruction matrix based on the matrix elements; and a reconstruction component configured to receive the reconstruction matrix from the reconstruction matrix generating component and reconstruct the N audio objects according to the M downmix signals using the reconstruction matrix.

III. Example Embodiments

1 illustrates an encoding/decoding system 100 that encodes/decodes an audio scene 102. The encoding/decoding system 100 includes an encoder 108, a bitstream generating component 110, a bitstream decoding component 118, a decoder 120, and a renderer 122.

The audio scene 102 is represented by one or more audio objects 106a (such as N audio objects), i.e., audio signals. The audio scene 102 may also include one or more bed channels 106b, i.e., signals that directly correspond to one of the output channels of the renderer 122. The audio scene 102 is also represented by metadata, including position information 104. The position information 104 is used, for example, by the renderer 122 when rendering the audio scene 102. The position information 104 may associate the audio objects 106a, and possibly the bed channels 106b, with spatial positions in three-dimensional space as a function of time. The metadata may also include other types of data useful for rendering the audio scene 102.

The encoding portion of the system 100 includes an encoder 108 and a bitstream generation component 110. The encoder 108 receives the audio objects 106a, the bed channels 106b (if present), and metadata including position information 104. Based on this, the encoder 108 generates one or more downmix signals 112, such as M downmix signals. For example, the downmix signals 112 may correspond to the channels [Lf Rf Cf Ls Rs LFE] of a 5.1 audio system. ("L" stands for left, "R" stands for right, "C" stands for center, "f" stands for front, "s" stands for surround, and "LFE" stands for low frequency effects).

The encoder 108 also generates side information. The side information comprises a reconstruction matrix. The reconstruction matrix comprises matrix elements 114 that enable reconstruction of at least the audio object 106a from the downmix signal 112. The reconstruction matrix may also enable reconstruction of the bed channel 106b.

The encoder 108 transmits the M downmix signals 112 and at least some of the matrix elements 114 to a bitstream generation component 110. The bitstream generation component 110 generates a bitstream 116 including the M downmix signals 112 and at least some of the matrix elements 114 by performing quantization and encoding. The bitstream generation component 110 also receives metadata including the position information 104 for inclusion in the bitstream 116.

The decoding portion of the system includes a bitstream decoding component 118 and a decoder 120. The bitstream decoding component 118 receives the bitstream 116 and performs decoding and dequantization to extract M downmix signals 112 and side information including at least some matrix elements 114 of the reconstruction matrix. The M downmix signals 112 and matrix elements 114 are then input to the decoder 120, which generates a reconstruction 106' of the N audio objects 106a and possibly the bed channels 106b based on the downmix signals 112 and the matrix elements 114. Thus, the reconstruction 106' of the N audio objects is an approximation of the N audio objects 106a and possibly the bed channels 106b.

For example, if the downmix signal 112 corresponds to the channels [Lf Rf Cf Ls Rs LFE] of a 5.1 configuration, the decoder 120 can reconstruct the object 106' using only the full-band channels [Lf Rf Cf Ls Rs], thereby ignoring the LFE. The same applies to other channel configurations. The LFE channel of the downmix 112 can be sent (substantially unmodified) to the renderer 122.

The reconstructed audio object 106' and the position information 104 are then input to a renderer 122. Based on the reconstructed audio object 106' and the position information 104, the renderer 122 renders an output signal 124 having a format suitable for playback on a desired speaker or headphone configuration. Typical output formats are a standard 5.1 surround setup (3 front speakers, 2 surround speakers, and 1 low frequency effects (LFE) speaker) or a 7.1+4 setup (3 front speakers, 4 surround speakers, 1 LFE speaker, and 4 overhead speakers).

In some embodiments, the original audio scene may include a large number of audio objects. Processing a large number of audio objects comes at the cost of high computational complexity. Furthermore, the amount of side information (position information 104 and reconstruction matrix elements 114) to be embedded in the bitstream 116 depends on the number of audio objects. Typically, the amount of side information increases linearly with the number of audio objects. Therefore, to save computational complexity and/or to reduce the bitrate required to encode the audio scene, it is advantageous to reduce the number of audio objects before encoding. To this end, the audio encoder/decoder system 100 may also include a scene simplification module (not shown) disposed upstream of the encoder 108. The scene simplification module takes as input the original audio objects and, possibly, the audio bed channels, and performs processing to output audio objects 106a. The scene simplification module reduces the number of original audio objects, for example, K, to a more appropriate number N of audio objects 106a by performing clustering. More specifically, the scene simplification module organizes the K original audio objects and, possibly, the audio bed channels, into N clusters. Typically, clusters are defined based on the spatial proximity of the K original audio objects/audio bed channels in the audio scene. In order to determine the spatial proximity, the scene simplification module may take as input the position information of the original audio objects/bed channels. When the scene simplification module has formed N clusters, it proceeds to represent each cluster with an audio object. For example, the audio object representing a cluster may be formed as the sum of the audio objects/bed channels that form part of the cluster. More specifically, the audio content of the audio objects/bed channels may be added to generate the audio content of the representative audio object. Furthermore, the positions of the audio objects/bed channels in the clusters may be averaged to give the position of the representative audio object. The scene simplification module includes the position of the representative audio object in the position data 104. Furthermore, the scene simplification module outputs the representative audio object that constitutes the N audio objects 106a of FIG. 1 .

The M downmix signals 112 may be arranged using a first format in a first field of a bitstream 116. The matrix elements 114 may be arranged using a second format in a second field of the bitstream 116. In this manner, a decoder that supports only the first format can decode and play back the M downmix signals 112 in the first field and discard the matrix elements 114 in the second field.

1 supports a first format and a second format. More precisely, the decoder 120 is configured to interpret the first format and the second format, which means that it is able to reconstruct the object 106 ′ based on the M downmix signals 112 and the matrix elements 114 .

FIG2 illustrates an audio encoder/decoder system 200. The encoding portion 108, 110 of the system 200 corresponds to the encoding portion of FIG1. However, the decoding portion of the audio encoder/decoder system 200 differs from the decoding portion of the audio encoder/decoder system 100 of FIG1. The audio encoder/decoder system 200 includes a legacy decoder 230 that supports the first format but not the second format. Consequently, the legacy decoder 230 of the audio encoder/decoder system 200 is unable to reconstruct the audio object/bed channels 106a to 106b. However, because the legacy decoder 230 supports the first format, it can still decode the M downmix signals 112 to generate an output 224, which is a channel-based representation suitable for direct playback via a corresponding multi-channel speaker setup, such as a 5.1 representation. This property of the downmix signal is known as backward compatibility, meaning that legacy decoders that do not support the second format, i.e., cannot interpret the side information including the matrix elements 114, can also decode and play back the M downmix signals 112.

The operation of the encoder side of the audio encoding/decoding system 100 will now be described in more detail with reference to the flowcharts of FIG. 3 and FIG. 4 .

Figure 4 illustrates in more detail the encoder 108 and the bitstream generation component 110 of Figure 1. The encoder 108 has a receiving component (not shown), a downmix generation component 318 and an analysis component 328.

In step E02, the receiving component of the encoder 108 receives N audio objects 106a and, if present, bed channels 106b. The encoder 108 may also receive position data 104. Using vector notation, the N audio objects may be represented by a vector S = [S1S2...SN] ^T , and the bed channels by a vector B. The N audio objects and bed channels may be collectively represented by a vector A = [B ^T S ^T ] ^T .

In step E04, the downmix generation component 318 generates M downmix signals 112 based on the N audio objects 106a and the bed channels 106b (if present). Using vector notation, the M downmix signals can be represented by a vector D=[D1 D2 ... DM] ^T comprising the M downmix signals. Generally, a downmix of multiple signals is a combination of signals, such as a linear combination of the signals. For example, the M downmix signals can correspond to a specific speaker configuration, such as the speaker configuration [Lf Rf Cf Ls Rs LFE] in a 5.1 speaker configuration.

The downmix generation component 318 can use the position information 104 when generating the M downmix signals, so that the objects are combined into different downmix signals based on their positions in three-dimensional space. This is particularly relevant when the M downmix signals themselves correspond to a specific speaker configuration, as in the example above. For example, the downmix generation component 318 can derive a representation matrix Pd (corresponding to the representation matrix used in the renderer 122 in FIG. 1 ) based on the position information and use this representation matrix to generate the downmix according to D = pd * [B ^T S ^T ] ^T.

The N audio objects 106a and the bed channel 106b (if present) are also input to the analysis component 328. The analysis component 328 typically operates on time-frequency blocks of the input audio signals 106a, 106b. To this end, the N audio objects 106a and the bed channel 106b can be fed through a filter bank, i.e. a QMF bank, which performs a time-to-frequency transform on the input audio signals 106a, 106b. In particular, the filter bank 338 is associated with a plurality of frequency subbands. The frequency resolution of the time-frequency block corresponds to one or more of these frequency subbands. The frequency resolution of the time-frequency block can be non-uniform, i.e. it can vary with frequency. For example, a low frequency resolution can be used for high frequencies, which means that a time-frequency block in the high frequency range can correspond to several frequency subbands defined by the filter bank 338.

In step E06, the analysis component 328 generates a reconstruction matrix, denoted herein by R1. The generated reconstruction matrix consists of a plurality of matrix elements. The reconstructed matrix R1 enables the (approximate) reconstruction of the N audio objects 106a and possibly also the bed channels 106b in the decoder from the M downmix signals 112.

The analysis component 328 can take different approaches to generating the reconstruction matrix. For example, a minimum mean square error (MMSE) prediction method can be used that takes as input the N audio objects 106a/audio bed channels 106b and the M downmix signals 112. The method can be described as a method that aims to derive a reconstruction matrix that minimizes the mean square error of the reconstructed audio objects/audio bed channels. In particular, the method uses a candidate reconstruction matrix to reconstruct the N audio objects/audio bed channels and compares the audio objects/audio bed channels with the input audio objects 106a/audio bed channels 106b with respect to the mean square error. The candidate reconstruction matrix that minimizes the mean square error is selected as the reconstruction matrix, and its matrix elements 114 are the output of the analysis component 328.

The MMSE method requires estimating correlation matrices and covariance matrices for the N audio objects 106a/bed channels 106b and the M downmix signals 112. According to the above-described method, these correlation matrices and covariance matrices are measured based on the N audio objects 106a/bed channels 106b and the M downmix signals 112. In an alternative model-based approach, the analysis component 328 takes as input the position data 104 instead of the M downmix signals 112. By making certain assumptions, such as assuming that the N audio objects are uncorrelated, and using this assumption in conjunction with the downmix rules applied in the downmix generation component 318, the analysis component 328 can calculate the required correlation matrices and covariance matrices required to perform the above-described MMSE method.

The elements 114 of the reconstruction matrix and the M downmix signals 112 are then input to the bitstream generation component 110. In step E108, the bitstream generation component 110 quantizes and encodes the M downmix signals 112 and at least some of the matrix elements 114 of the reconstruction matrix, and arranges them in a bitstream 116. In particular, the bitstream generation component 110 may arrange the M downmix signals 112 in a first field of the bitstream 116 using a first format. Furthermore, the bitstream generation component 110 may arrange the matrix elements 114 in a second field of the bitstream 116 using a second format. As previously described with reference to FIG. 2 , this allows a legacy decoder that only supports the first format to decode and play back the M downmix signals 112 and discard the matrix elements 114 in the second field.

FIG5 illustrates an alternative embodiment of the encoder 108. Compared to the encoder shown in FIG3 , the encoder 508 of FIG5 also enables one or more auxiliary signals to be included in the bitstream 116. To this end, the encoder 508 includes an auxiliary signal generation component 548. The auxiliary signal generation component 548 receives the audio objects 106a/bed channels 106b and generates one or more auxiliary signals 512 based on the audio objects 106a/bed channels 106b. The auxiliary signal generation component 548 can, for example, generate the auxiliary signals 512 as combinations of the audio objects 106a/bed channels 106b. The auxiliary signals are represented by the vector C=[CI C2 ...CL] ^T , which can be generated as C=Q*[B ^T S ^T ] ^T , where Q is a matrix that can be both time- and frequency-varying. This includes situations where the auxiliary signal is equal to one or more of the audio objects, as well as situations where the auxiliary signal is a linear combination of the audio objects. For example, the auxiliary signal can represent a particularly important object, such as dialogue.

The auxiliary signals 512 serve to improve the reconstruction of the audio objects 106a/bed channels 106b at the decoder. More specifically, at the decoder side, the audio objects 106a/bed channels 106b can be reconstructed based on the M downmix signals 112 and the L auxiliary signals 512. Therefore, the reconstruction matrix will include matrix elements 114 that enable the reconstruction of the audio objects/bed channels based on the M downmix signals 112 and the L auxiliary signals.

Thus, the L auxiliary signals 512 can be input to the analysis component 328 so that the L auxiliary signals 512 are taken into account when generating the reconstruction matrix. The analysis component 328 can also send control signals to the auxiliary signal generation component 548. For example, the analysis component 328 can control which audio objects/bed channels are included in the auxiliary signals and how they are included. In particular, the analysis component 328 can control the selection of the Q matrix. This control can be based on the MMSE method described above, for example, so that the auxiliary signals can be selected so that the reconstructed audio objects/bed channels are as close as possible to the audio objects 106a/bed channels 106b.

The operation of the decoder side of the audio encoding/decoding system 100 will now be described in more detail with reference to the flowcharts of FIG. 6 and FIG. 7 .

6 illustrates in more detail the bitstream decoding component 118 and the decoder 120 of FIG1 . The decoder 120 includes a reconstruction matrix generation component 622 and a reconstruction component 624 .

In step D02, the bitstream decoding component 118 receives the bitstream 116. The bitstream decoding component 118 decodes and dequantizes the information in the bitstream 116 to extract the M downmix signals 112 and at least some matrix elements 114 in the reconstruction matrix.

The reconstruction matrix generation component 622 receives the matrix elements 114 and proceeds in step D04 to generate a reconstruction matrix 614. The reconstruction matrix generation component 622 generates the reconstruction matrix 614 by arranging the matrix elements 114 at appropriate positions in the matrix. If all the matrix elements of the reconstruction matrix are not received, the reconstruction matrix generation component 622 can, for example, insert zeros to replace the missing elements.

The reconstruction matrix 614 and the M downmix signals are then input to a reconstruction component 624. The reconstruction component 624 then reconstructs the N audio objects and, if applicable, the bed channels in step D06. In other words, the reconstruction component 624 generates an approximation 106' of the N audio objects 106a/bed channels 106b.

For example, the M downmix signals may correspond to a specific speaker configuration, such as the configuration of speakers [Lf Rf Cf Ls Rs LFE] in a 5.1 speaker configuration. If so, the reconstruction component 624 may base the reconstruction of the object 106' solely on the downmix signals corresponding to the full-band channels of the speaker configuration. As explained above, the band-limited signal (low-frequency LFE signal) may be sent to the renderer substantially unmodified.

The reconstruction component 624 typically operates in the frequency domain. More specifically, the reconstruction component 624 operates on individual time-frequency blocks of the input signal. Therefore, before being input to the reconstruction component 624, the M downmix signals 112 typically undergo a time-to-frequency transform 623. The time-to-frequency transform 623 is typically the same as or similar to the transform 338 applied at the encoder side. For example, the time-to-frequency transform 623 can be a QMF transform.

To reconstruct the audio object/bed channel 106', the reconstruction component 624 applies matrix operations. More specifically, using the notation introduced previously, the reconstruction component 624 can generate an approximation A' of the audio object/bed channel as A' = R1 * D. The reconstruction matrix R1 can vary in time and frequency. Therefore, the reconstruction matrix can differ between different time-frequency blocks processed by the reconstruction component 624.

The reconstructed audio objects/bed channels 106 ′ are typically transformed back to the time domain 625 before being output from the decoder 120 .

FIG8 illustrates the situation when the bitstream 116 additionally includes auxiliary signals. Compared to the embodiment of FIG7 , the bitstream decoding component 118 now additionally decodes one or more auxiliary signals 512 from the bitstream 116. The auxiliary signals 512 are input to a reconstruction component 624, where they are included in the reconstruction of the audio objects/bed channels. More specifically, the reconstruction component 624 generates the audio objects/bed channels by applying the matrix operation A′=R1*[D ^T C ^T ] ^T.

9 illustrates different time-frequency transforms used at the decoder side of the audio encoding/decoding system 100 of FIG 1. The bitstream decoding component 118 receives the bitstream 116. The decoding and dequantization component 918 decodes and dequantizes the bitstream 116 to extract the position information 104, the M downmix signals 112, and the matrix elements 114 of the reconstruction matrix.

At this stage, the M downmix signals 112 are typically represented in a first frequency domain, corresponding to a first set of time-frequency filter banks, denoted herein by T/ _FC and F/ _TC for transforming from the time domain to the first frequency domain and from the first frequency domain to the time domain, respectively. Typically, the filter banks corresponding to the first frequency domain can implement overlapping window transforms, such as MDCT and inverse MDCT. The bitstream decoding component 118 may include a transform component 901 that transforms the M downmix signals 112 into the time domain using the filter banks F/ _TC .

The decoder 120, and in particular the reconstruction component 624, typically processes signals with respect to a second frequency domain. The second frequency domain corresponds to a second set of time-frequency filter banks, represented herein by T/ _FU and F/ _TU, for transforming from the time domain to the second frequency domain and from the second frequency domain to the time domain. Therefore, the decoder 120 may include a transform component 903 that transforms the M downmix signals 112 represented in the time domain to the second frequency domain using the filter bank T/ _{FU .} When the reconstruction component 624 has reconstructed the object 106' based on the M downmix signals by performing processing in the second frequency domain, the transform component 905 may transform the reconstructed object 106' back to the time domain using the filter bank F/TU.

The renderer 122 typically processes the signal with respect to a third frequency domain. The third frequency domain corresponds to a third set of time-frequency filter banks, denoted herein by T/ _FR and F/ _TR, for transforming from the time domain to the third frequency domain and from the third frequency domain to the time domain, respectively. Thus, the renderer 122 may include a transform component 907 for transforming the reconstructed audio object 106' from the time domain to the third frequency domain using the filter bank T/ _FR . When the renderer 122 has rendered the output channels 124 via the rendering component 922, the transform component 909 may transform the output channels to the time domain using the filter bank F/ _TR .

As is apparent from the above description, the decoder side of the audio encoding/decoding system includes many time-frequency transformation steps. However, if the first frequency domain, the second frequency domain and the third frequency domain are selected in a certain way, some of the time-frequency transformation steps may become redundant.

For example, some of the first frequency domain, the second frequency domain, and the third frequency domain may be selected to be the same, or may be implemented together to go directly from one frequency domain to another frequency domain without passing through the time domain between them. An example of the latter is a situation where the second frequency domain and the third frequency domain differ only in that the transform component 907 in the renderer 122 uses a Nyquist filter bank to improve the frequency resolution at low frequencies in addition to using the QMF filter bank common to the two transform components 905 and 907. In this case, the transform components 905 and 907 may be implemented together in the form of a Nyquist filter bank, thereby saving computational complexity.

In another example, the second frequency domain and the third frequency domain are the same. For example, the second frequency domain and the third frequency domain can both be QMF frequency domains. In this case, transform components 905 and 907 are redundant and can be removed, thereby saving computational complexity.

According to another example, the first frequency domain and the second frequency domain may be the same. For example, the first frequency domain and the second frequency domain may both be MDCT domains. In this case, the first transform component 901 and the second transform component 903 may be removed, thereby saving computational complexity.

Equivalents, extensions, alternatives and others

Other embodiments of the present disclosure will become apparent to those skilled in the art after studying the above description. Although this specification and the drawings disclose embodiments and examples, the present disclosure is not limited to these specific examples. Many modifications and variations may be made without departing from the scope of the present disclosure as defined by the appended claims. Any reference signs appearing in the claims are not to be construed as limiting their scope.

Furthermore, variations to the disclosed embodiments may be understood and effected by those skilled in the art in practicing the present disclosure, based on a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude plural reference. The fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

The systems and methods disclosed above can be implemented as software, firmware, hardware, or a combination thereof. In hardware implementations, the division of tasks between the functional units mentioned in the above description does not necessarily correspond to the division of physical units; on the contrary, a physical component can have multiple functions, and a task can be performed jointly by several physical components. Some or all components can be implemented as software executed by a digital signal processor or microprocessor, or can be implemented as hardware or an application-specific integrated circuit. Such software can be distributed on a computer-readable medium that can include computer storage media (or non-transitory media) and communication media (or transient media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile media, removable media, and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage devices, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism, and includes any information delivery media.

Claims (33)

1. A method for encoding a time-frequency block of an audio scene comprising at least N audio objects, the method comprising: Receive the N audio objects; Generate M downmixed signals based on at least the N audio objects; A reconstruction matrix is generated from matrix elements used to reconstruct at least N audio objects based on the M downmixed signals, wherein the approximations of the at least N audio objects can be obtained as linear combinations of the at least M downmixed signals, and the matrix elements of the reconstruction matrix serve as coefficients in the linear combination; and Generate a bitstream, the bitstream comprising at least some of the matrix elements of the M downmixed signals and the matrix elements of the reconstruction matrix. 2. The method of claim 1, wherein the M downmixed signals are arranged in a first field of the bitstream using a first format, and the matrix elements are arranged in a second field of the bitstream using a second format, thereby allowing a decoder that only supports the first format to decode and replay the M downmixed signals in the first field, and discarding the matrix elements in the second field. 3. The method according to any one of the preceding claims further comprises the step of: receiving position data corresponding to each of the N audio objects, wherein the M downmix signals are generated based on the position data. 4. The method according to claim 1 or 2, wherein the matrix elements of the reconstructed matrix are time-varying and frequency-varying. 5. The method according to claim 1 or 2, wherein the audio scene further includes a plurality of audio bed channels, wherein the M downmix signals are generated based on at least the N audio objects and the plurality of audio bed channels. 6. The method of claim 5, wherein the reconstruction matrix comprises matrix elements for reconstructing the bed channel based on the M downmixing signals, wherein an approximation of the N audio objects and the bed channel can be obtained as a linear combination of at least the M downmixing signals, and the matrix elements of the reconstruction matrix serve as coefficients in the linear combination. 7. The method according to claim 1 or 2, wherein the audio scene initially includes K audio objects, where K>N, and the method further includes the steps of: receiving the K audio objects and reducing the K audio objects to the N audio objects by clustering the K audio objects into N clusters and representing each cluster with an audio object. 8. The method of claim 7, further comprising the step of: receiving location data corresponding to each of the K audio objects, wherein the K objects are clustered into N clusters based on the location distances between the K objects given by the location data of the K audio objects. 9. The method according to claim 1 or 2, wherein the number M of the downmixed signals is greater than 2. 10. The method according to claim 1 or 2, further comprising: The N audio objects form L auxiliary signals; The matrix elements used to reconstruct at least N audio objects based on the M downmix signals and the L auxiliary signals are included in the reconstruction matrix, wherein the approximation of the at least N audio objects can be obtained as a linear combination of the M downmix signals and the L auxiliary signals, and the matrix elements of the reconstruction matrix serve as coefficients in the linear combination; and The L auxiliary signals are included in the bit stream. 11. The method of claim 10, wherein at least one of the L auxiliary signals is identical to one of the N audio objects. 12. The method of claim 10, wherein at least one of the L auxiliary signals is formed as a combination of at least two audio objects among the N audio objects. 13. The method of claim 10, wherein the M downmixing signals span a hyperplane, and wherein at least one of the L auxiliary signals is not located in the hyperplane spanned by the M downmixing signals. 14. The method of claim 13, wherein at least one of the L auxiliary signals is orthogonal to the hyperplane spanned by the M downmixing signals. 15. A computer-readable medium comprising computer code instructions adapted to perform the method according to any one of claims 1 to 14 when operated on a processing device. 16. An encoder for encoding a time-frequency block of an audio scene comprising at least N audio objects, the encoder comprising: A receiving component, configured to receive the N audio objects; A downmixing generation unit is configured to receive the N audio objects from the receiving unit and to generate M downmixing signals based on at least the N audio objects; An analysis component is configured to generate a reconstruction matrix from matrix elements used to reconstruct at least the N audio objects based on the M downmixed signals, wherein an approximation of the at least N audio objects can be obtained as a linear combination of the at least M downmixed signals, and the matrix elements of the reconstruction matrix serve as coefficients in the linear combination; and A bitstream generation unit is configured to receive the M downmixed signals from the downmixing generation unit and the reconstruction matrix from the analysis unit, and to generate a bitstream comprising at least some of the matrix elements of the M downmixed signals and the reconstruction matrix. 17. A method for decoding a time-frequency block of an audio scene comprising at least N audio objects, the method comprising the steps of: Receive a bit stream comprising at least some matrix elements of M downmixed signals and a reconstruction matrix; The reconstructed matrix is generated using the matrix elements; and The reconstruction matrix is used to reconstruct the N audio objects based on the M downmixed signals, wherein at least the approximations of the N audio objects can be obtained as linear combinations of at least the M downmixed signals, and the matrix elements of the reconstruction matrix serve as coefficients in the linear combination. 18. The method of claim 17, wherein the M downmixed signals are arranged in a first field of the bitstream using a first format, and the matrix elements are arranged in a second field of the bitstream using a second format, thereby allowing a decoder that only supports the first format to decode and replay the M downmixed signals in the first field, and discarding the matrix elements in the second field. 19. The method according to any one of claims 17 to 18, wherein the matrix elements of the reconstructed matrix are time-varying and frequency-varying. 20. The method according to any one of claims 17 to 18, wherein the audio scene further includes a plurality of audio bed channels, the method further includes reconstructing the audio bed channels using the reconstruction matrix based on the M downmixing signals, wherein an approximation of the N audio objects and the audio bed channels can be obtained as a linear combination of at least the M downmixing signals, the matrix elements of the reconstruction matrix serving as coefficients in the linear combination. 21. The method according to any one of claims 17 to 18, wherein the number M of downmixed signals is greater than 2. 22. The method according to any one of claims 17 to 18, further comprising: Receive L auxiliary signals formed by the N audio objects; The reconstruction matrix is used to reconstruct the N audio objects based on the M downmixed signals and the L auxiliary signals, wherein at least the approximations of the N audio objects are obtained as linear combinations of the M downmixed signals and the L auxiliary signals, and the matrix elements of the reconstruction matrix are used as coefficients in the linear combination. 23. The method of claim 22, wherein at least one of the L auxiliary signals is identical to one of the N audio objects. 24. The method of claim 22, wherein at least one of the L auxiliary signals is a combination of the N audio objects. 25. The method of claim 22, wherein the M downmixing signals span a hyperplane, and wherein at least one of the L auxiliary signals is not located in the hyperplane spanned by the M downmixing signals. 26. The method of claim 25, wherein at least one of the L auxiliary signals not located in the hyperplane is orthogonal to the hyperplane spanned by the M downmixing signals. 27. The method according to any one of claims 17 to 18, wherein the M downmixed signals are represented with respect to a first frequency domain, and wherein the reconstruction matrix is represented with respect to a second frequency domain, the first frequency domain and the second frequency domain being the same frequency domain. 28. The method of claim 27, wherein the first frequency domain and the second frequency domain are improved discrete cosine transform (MDCT) domains. 29. The method according to any one of claims 17 to 18, further comprising: receiving position data corresponding to the N audio objects, and The location data is used to render the N audio objects to create at least one output audio channel. 30. The method of claim 29, wherein the reconstruction matrix is represented with respect to a second frequency domain corresponding to a second filter group, and the rendering is performed in a third frequency domain corresponding to a third filter group, wherein the second filter group and the third filter group are at least partially the same filter group. 31. The method of claim 30, wherein the second filter bank and the third filter bank comprise a quadrature mirror filter (QMF) filter bank. 32. A computer-readable medium comprising computer code instructions adapted, when operated on a processing device, to perform the method according to any one of claims 17 to 31. 33. A decoder for decoding a time-frequency block of an audio scene comprising at least N audio objects, the decoder comprising: A receiving unit configured to receive a bit stream comprising at least some matrix elements of a matrix including M downmixed signals and a reconstruction matrix; A reconstruction matrix generation component is configured to receive the matrix elements from the receiving component and generate the reconstruction matrix based on the matrix elements; and A reconstruction component is configured to receive the reconstruction matrix from the reconstruction matrix generation component and reconstruct the N audio objects based on the M downmixing signals using the reconstruction matrix, wherein at least the approximations of the N audio objects are obtained as linear combinations of at least the M downmixing signals, and the matrix elements of the reconstruction matrix serve as coefficients in the linear combination.