HK1232013B - Methods and devices for processing audio data - Google Patents

Description

Method and apparatus for processing audio data

This application claims the rights of:

U.S. Provisional Patent Application No. 62/020,348, entitled “REDUCING CORRELATION BETWEEN HOA BACKGROUND CHANNELS,” filed on July 2, 2014; and

U.S. Provisional Patent Application No. 62/060,512, entitled “REDUCING CORRELATION BETWEEN HOA BACKGROUND CHANNELS,” filed October 6, 2014,

The entire contents of each of which are incorporated herein by reference.

Technical Field

The present invention relates to audio data, and more particularly, to the coding of high-order ambisonic audio data.

Background Art

A higher-order ambisonic (HOA) signal (typically represented by a plurality of spherical harmonic coefficients (SHCs) or other layered elements) is a three-dimensional representation of a sound field. An HOA or SHC representation can represent the sound field in a manner independent of the local speaker geometry used to play back the multi-channel audio signal reproduced from the SHC signal. An SHC signal can also facilitate backward compatibility because the SHC signal can be reproduced in a well-known and widely adopted multi-channel format (e.g., a 5.1 audio channel format or a 7.1 audio channel format). The SHC representation can thus achieve a better representation of the sound field, which also accommodates backward compatibility.

Summary of the Invention

In general, techniques are described for coding higher-order ambisonic audio data. The higher-order ambisonic audio data may include at least one higher-order ambisonic (HOA) coefficient corresponding to a spherical harmonic basis function having an order greater than one. Techniques are also described for reducing correlation between higher-order ambisonic (HOA) background channels.

In one aspect, a method includes obtaining a decorrelated representation of ambisonic coefficients having at least a left signal and a right signal, the ambisonic coefficients having been extracted from a plurality of higher-order ambisonic coefficients and representing background components of a sound field described by the plurality of higher-order ambisonic coefficients, wherein at least one of the plurality of higher-order ambisonic coefficients is associated with a spherical basis function having an order greater than one; and generating a speaker feed based on the decorrelated representation of the ambisonic coefficients.

In another aspect, a method includes applying a decorrelation transform to ambisonic coefficients to obtain decorrelated representations of the ambisonic coefficients, the ambisonic HOA coefficients having been extracted from a plurality of higher-order ambisonic coefficients and representing background components of a sound field described by the plurality of higher-order ambisonic coefficients, wherein at least one of the plurality of higher-order ambisonic coefficients is associated with a spherical basis function having an order greater than one.

In another aspect, a device for compressing audio data includes one or more processors configured to: obtain a decorrelated representation of ambisonic coefficients having at least a left signal and a right signal, the ambisonic coefficients having been extracted from a plurality of higher-order ambisonic coefficients and representing background components of a sound field described by the plurality of higher-order ambisonic coefficients, wherein at least one of the plurality of higher-order ambisonic coefficients is associated with a spherical basis function having an order greater than one; and generate a speaker feed based on the decorrelated representation of the ambisonic coefficients.

In another aspect, a device for compressing audio data includes one or more processors configured to: apply a decorrelation transform to ambisonic reverberation coefficients to obtain decorrelated representations of the ambisonic reverberation coefficients, the ambisonic reverberation coefficients having been extracted from a plurality of higher-order ambisonic reverberation coefficients and representing background components of a sound field described by the plurality of higher-order ambisonic reverberation coefficients, wherein at least one of the plurality of higher-order ambisonic reverberation coefficients is associated with a spherical basis function having an order greater than one.

In another aspect, a device for compressing audio data includes: means for obtaining a decorrelated representation of ambisonic coefficients having at least a left signal and a right signal, the ambisonic coefficients having been extracted from a plurality of higher-order ambisonic coefficients and representing background components of a sound field described by the plurality of higher-order ambisonic coefficients, wherein at least one of the plurality of higher-order ambisonic coefficients is associated with a spherical basis function having an order greater than one; and means for generating speaker feeds based on the decorrelated representation of the ambisonic coefficients.

In another aspect, a device for compressing audio data includes: means for applying a decorrelation transform to ambisonic reverberation coefficients to obtain a decorrelated representation of the ambisonic reverberation coefficients, the ambisonic reverberation coefficients having been extracted from a plurality of higher-order ambisonic reverberation coefficients and representing background components of a sound field described by the plurality of higher-order ambisonic reverberation coefficients, wherein at least one of the plurality of higher-order ambisonic reverberation coefficients is associated with a spherical basis function having an order greater than one; and means for storing the decorrelated representation of the ambisonic reverberation coefficients.

In another aspect, a computer-readable storage medium is encoded with instructions that, when executed, cause one or more processors of an audio compression device to: obtain a decorrelated representation of ambisonic coefficients having at least a left signal and a right signal, the ambisonic coefficients having been extracted from a plurality of higher-order ambisonic coefficients and representing background components of a sound field described by the plurality of higher-order ambisonic coefficients, wherein at least one of the plurality of higher-order ambisonic coefficients is associated with a spherical basis function having an order greater than one; and generate a speaker feed based on the decorrelated representation of the ambisonic coefficients.

In another aspect, a computer-readable storage medium is encoded with instructions that, when executed, cause one or more processors of an audio compression device to apply a decorrelation transform to ambisonic reverberation coefficients to obtain decorrelated representations of the ambisonic reverberation coefficients, the ambisonic reverberation coefficients having been extracted from a plurality of higher-order ambisonic reverberation coefficients and representing background components of a sound field described by the plurality of higher-order ambisonic reverberation coefficients, wherein at least one of the plurality of higher-order ambisonic reverberation coefficients is associated with a spherical basis function having an order greater than one.

The details of one or more aspects of the technology are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the technology will be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a diagram illustrating spherical harmonic basis functions with various orders and sub-orders.

2 is a diagram illustrating a system that may perform various aspects of the techniques described in this disclosure.

3 is a block diagram illustrating one example of an audio encoding device shown in the example of FIG. 2 in greater detail, which may perform various aspects of the techniques described in this disclosure.

FIG. 4 is a block diagram illustrating the audio decoding apparatus of FIG. 2 in more detail.

5 is a flowchart illustrating exemplary operation of an audio encoding device in performing various aspects of the vector-based synthesis techniques described in this disclosure.

6A is a flowchart illustrating exemplary operation of an audio decoding device performing various aspects of the techniques described in this disclosure.

6B is a flowchart illustrating exemplary operation of an audio encoding device and an audio decoding device performing the coding techniques described in this disclosure.

DETAILED DESCRIPTION

The evolution of surround sound has made many output formats available for entertainment today. Examples of these consumer surround sound formats are mostly "channel" based because they implicitly specify the feeds to the loudspeakers in specific geometric coordinates. Consumer surround sound formats include the popular 5.1 format (which includes the following six channels: left front (FL), right front (FR), center or front center, left rear or left surround, right rear or right surround, and low frequency effects (LFE)), the developing 7.1 format, various formats that include height speakers, such as the 7.1.4 format, and the 22.2 format (for example, for use with ultra-high-definition television standards). Non-consumer formats can include any number of loudspeakers (in symmetrical and asymmetrical geometric arrangements), which are often referred to as "surround arrays." One example of such an array includes 32 loudspeakers positioned at coordinates on the corners of a truncated icosahedron.

The input to future MPEG encoders will be in one of three possible formats: (i) traditional channel-based audio (as discussed above), intended to be played by loudspeakers at pre-specified positions; (ii) object-based audio, which involves discrete pulse-code modulation (PCM) data for individual audio objects with associated metadata containing their position coordinates (and other information); and (iii) scene-based audio, which involves representing the sound field using coefficients of spherical harmonic basis functions (also called "spherical harmonic coefficients" or SHC, "higher-order ambisonics" or HOA, and "HOA coefficients"). The future MPEG encoder is described in more detail in the document entitled “Call for Proposals for 3D Audio” by the International Organization for Standardization/International Electrotechnical Commission (ISO)/(IEC) JTC1/SC29/WG11/N13411, published in Geneva, Switzerland in January 2013 and available at http://mpeg.chiariglione.org/sites/default/files/files/standards/parts/docs/w13411.zip.

There are various channel-based "surround sound" formats on the market. They range, for example, from 5.1 home theater systems, which have had the greatest success in bringing stereo sound to the living room, to the 22.2 system developed by NHK (Nippon Hoso Kyokai or Japan Broadcasting Corporation). Content creators (e.g., Hollywood studios) would like to produce a movie soundtrack once, without having to expend the effort of remixing it for each speaker configuration. Recently, Standards Developing Organizations have been considering approaches that provide encoding into a standardized bitstream, and subsequent decoding, that is adaptable and agnostic to the speaker geometry (and number) and acoustic conditions at the playback location (relating to the reproducer).

To provide content creators with this flexibility, a hierarchical element set can be used to represent the sound field. The hierarchical element set may refer to a set of elements in which the elements are ordered so that a base set of lower-order elements provides a complete representation of the modeled sound field. As the set is expanded to include higher-order elements, the representation becomes more detailed, thereby increasing the resolution.

An example of a hierarchical element set is a set of spherical harmonic coefficients (SHCs). The following expression demonstrates the description or representation of a sound field using SHCs:

The expression shows that the pressure p _i at any point {r _r , θ _r } in the sound field at time t can be uniquely represented by SHC . Here, c is the speed of sound (approximately 343 m/s), {r _r , θ _r } is a reference point (or observation point), j _n (·) is a spherical Bessel function of order n, and is a spherical harmonic basis function of order n and suborder m. It can be recognized that the term in square brackets is a frequency domain representation of the signal (i.e., S(ω, r _r , θ r ), which can be approximated by various time-frequency transforms (e.g., discrete Fourier transform (DFT), discrete cosine transform (DCT), or wavelet transform). Other examples of hierarchical sets include sets of wavelet transform coefficients and other sets of coefficients of multiresolution basis functions. High-order ambisonic reverberation signals are processed by truncating the high orders so that only the zeroth and first orders remain. Due to the energy loss of the high-order coefficients, some energy compensation is usually performed on the remaining signal.

Various aspects of the present invention are directed to reducing correlation between background signals. For example, the techniques of the present invention can reduce or potentially eliminate correlation between background signals expressed in the HOA domain. A potential advantage of reducing correlation between background HOA signals is reduced noise demasking. As used herein, the expression "noise demasking" may refer to attributing audio objects to locations that do not correspond to them in the spatial domain. In addition to reducing potential issues associated with noise demasking, the encoding techniques described herein can also generate output signals representing left and right audio signals (e.g., signals that together form a stereo output). A decoding device can then decode the left and right audio signals to obtain a stereo output, or can mix the left and right audio signals to obtain a mono output. Furthermore, in scenarios where the encoded bitstream represents a purely horizontal layout, the decoding device can implement the various techniques of the present invention to decode only the horizontal component decorrelated HOA background signals. By limiting the decoding process to the horizontal component decorrelated HOA background signals, the decoder can implement the techniques to save computational resources and reduce bandwidth consumption.

Figure 1 is a diagram illustrating spherical harmonic basis functions from zeroth order (n=0) to fourth order (n=4). As can be seen, for each order, there is an extension of a number of sub-orders m, which are shown in the example of Figure 1 for ease of illustration but not explicitly annotated.

SHC can be physically acquired (e.g., recorded) by various microphone array configurations or, alternatively, derived from a channel-based or object-based description of the sound field. SHC represents scene-based audio, where the SHC can be input to an audio encoder to obtain encoded SHC, which can facilitate more efficient transmission or storage. For example, a fourth-order representation involving (1+4) ^² (2⁵, and therefore fourth-order) coefficients can be used.

As mentioned above, SHC can be derived from microphone recordings using a microphone array. Various examples of how SHC can be derived from a microphone array are described in Poletti, M., "Three-Dimensional Surround Sound Systems Based on Spherical Harmonics," J. Audio Eng. Soc., Vol. 53, No. 11, November 2005, pp. 1004-1025.

To illustrate how SHC can be derived from an object-based description, consider the following equation. The coefficients of the sound field corresponding to an individual audio object can be expressed as:

where i is a spherical Hankel function of order n (of the second kind), and { _rs , _θs ,} is the position of the object. Knowing the frequency-varying source energy of the object, g(ω) (e.g., using time-frequency analysis techniques such as performing a Fast Fourier Transform on the PCM stream) allows each PCM object and corresponding position to be converted into an SHC. Furthermore, it can be shown (due to the linear and orthogonal decomposition above) that the coefficients for each object are additive. In this way, many PCM objects can be represented by coefficients (e.g., as the sum of the coefficient vectors for the individual objects). Essentially, the coefficients contain information about the sound field (pressure as a function of 3D coordinates), and the above scenario represents the transformation from an individual object to a representation of the entire sound field near the observation point { _rs , _θr ,}. The remaining figures are described below in the context of object-based and SHC-based audio coding.

FIG2 is a diagram illustrating a system 10 that can perform various aspects of the techniques described in this disclosure. As shown in the example of FIG2 , the system 10 includes a content creator device 12 and a content consumer device 14. Although described in the context of a content creator device 12 and a content consumer device 14, the techniques may be implemented in any context in which the SHCt (also referred to as HOA coefficients) or any other layered representation of a sound field is encoded to form a bitstream representing audio data. Furthermore, the content creator device 12 may represent any form of computing device capable of implementing the techniques described in this disclosure, including a handset (or cellular phone), a tablet computer, a smartphone, or a desktop computer (to provide a few examples). Likewise, the content consumer device 14 may represent any form of computing device capable of implementing the techniques described in this disclosure, including a handset (or cellular phone), a tablet computer, a smartphone, a set-top box, or a desktop computer (to provide a few examples).

The content creator device 12 may be operated by a movie studio or other entity that can produce multi-channel audio content for consumption by an operator of a content consumer device (e.g., content consumer device 14). In some examples, the content creator device 12 may be operated by an individual user who would like to compress the HOA coefficients 11. Content creators typically produce audio content along with video content. The content consumer device 14 may be operated by an individual. The content consumer device 14 may include an audio playback system 16, which may refer to any form of audio playback system capable of reproducing SHC for playback as multi-channel audio content.

The content creator device 12 includes an audio editing system 18. The content creator device 12 obtains a live recording 7 and audio objects 9 in various formats (including directly as HOA coefficients), which the content creator device 12 can edit using the audio editing system 18. The microphone 5 can capture the live recording 7. The content creator can reproduce the HOA coefficients 11 from the audio objects 9 during the editing process, listening to the reproduced speaker feeds to try to identify various aspects of the sound field that need further editing. The content creator device 12 can then edit the HOA coefficients 11 (potentially indirectly by manipulating different ones of the audio objects 9 from which the source HOA coefficients can be derived in the manner described above). The content creator device 12 can employ the audio editing system 18 to generate the HOA coefficients 11. The audio editing system 18 represents any system capable of editing audio data and outputting the audio data as one or more source spherical harmonic coefficients.

When the editing process is complete, the content creator device 12 can generate a bitstream 21 based on the HOA coefficients 11. That is, the content creator device 12 includes an audio encoding device 20, which represents a device configured to encode or otherwise compress the HOA coefficients 11 according to various aspects of the techniques described in this disclosure to generate the bitstream 21. The audio encoding device 20 can generate the bitstream 21 for transmission across a transmission channel (which can be a wired or wireless channel, a data storage device, or the like) (as an example). The bitstream 21 can represent an encoded version of the HOA coefficients 11 and can include a main bitstream and another side bitstream (which can be referred to as side channel information).

2 as being transmitted directly to the content consumer device 14, the content creator device 12 may output the bitstream 21 to an intermediary device located between the content creator device 12 and the content consumer device 14. The intermediary device may store the bitstream 21 for later delivery to the content consumer device 14 that may request the bitstream. The intermediary device may comprise a file server, a network server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smartphone, or any other device capable of storing the bitstream 21 for later retrieval by an audio decoder. The intermediary device may reside in a content delivery network that is capable of streaming the bitstream 21 (and possibly in conjunction with transmitting a corresponding video data bitstream) to a subscriber (e.g., the content consumer device 14) requesting the bitstream 21.

Alternatively, the content creator device 12 may store the bitstream 21 on a storage medium, such as a compact disc, a digital video disc, a high-definition video disc, or other storage medium, most of which are capable of being read by a computer and thus may be referred to as computer-readable storage medium or non-transitory computer-readable storage medium. In this context, a transmission channel may refer to a channel by which content stored on the medium is transmitted (and may include retail stores and other store-based delivery mechanisms). Thus, in any case, the techniques of this disclosure should not be limited to the example of FIG. 2 in this regard.

As further shown in the example of FIG2 , the content consumer device 14 includes an audio playback system 16. The audio playback system 16 may represent any audio playback system capable of playing back multi-channel audio data. The audio playback system 16 may include a plurality of different reproducers 22. The reproducers 22 may each provide for a different form of reproduction, wherein the different forms of reproduction may include one or more of various ways of performing Vector Basis Amplitude Shifting (VBAP) and/or one or more of various ways of performing sound field synthesis. As used herein, "A and/or B" means "A or B," or "both A and B."

The audio playback system 16 may further include an audio decoding device 24. The audio decoding device 24 may represent a device configured to decode the HOA coefficients 11′ from the bitstream 21, wherein the HOA coefficients 11′ may be similar to the HOA coefficients 11, but may differ due to lossy operations (e.g., quantization) and/or transmission via a transmission channel. The audio playback system 16 may obtain the HOA coefficients 11′ after decoding the bitstream 21 and reproduce the HOA coefficients 11′ to output a loudspeaker feed 25. The loudspeaker feed 25 may drive one or more loudspeakers (not shown in the example of FIG. 2 for ease of illustration).

To select an appropriate reproducer, or in some examples, generate an appropriate reproducer, the audio playback system 16 may obtain loudspeaker information 13 indicating the number of loudspeakers and/or the spatial geometric arrangement of the loudspeakers. In some examples, the audio playback system 16 may obtain the loudspeaker information 13 using a reference microphone and drive the loudspeakers in a manner that dynamically determines the loudspeaker information 13. In other examples, or in conjunction with dynamically determining the loudspeaker information 13, the audio playback system 16 may prompt a user to interface with the audio playback system 16 and input the loudspeaker information 13.

The audio playback system 16 may then select one of the audio renderers 22 based on the loudspeaker information 13. In some instances, when none of the audio renderers 22 are within a certain threshold similarity measure (in terms of loudspeaker geometry) to the loudspeaker geometry specified in the loudspeaker information 13, the audio playback system 16 may generate one of the audio renderers 22 based on the loudspeaker information 13. The audio playback system 16 may, in some instances, generate one of the audio renderers 22 based on the loudspeaker information 13 without first attempting to select an existing one of the audio renderers 22. The one or more speakers 3 may then play back the reproduced loudspeaker feed 25.

FIG3 is a block diagram illustrating in greater detail an example of the audio encoding device 20 shown in the example of FIG2 that can perform various aspects of the techniques described in this disclosure. The audio encoding device 20 includes a content analysis unit 26, a vector-based synthesis method unit 27, a direction-based synthesis method unit 28, and a decorrelation unit 40′. Although briefly described below, more information about the audio encoding device 20 and various aspects of compressing or otherwise encoding HOA coefficients can be found in International Patent Application Publication No. WO 2014/194099, filed May 29, 2014, entitled “INTERPOLATION FOR DECOMPOSED REPRESENTATIONS OF A SOUND FIELD.”

The content analysis unit 26 represents a unit configured to analyze the content of the HOA coefficients 11 to identify whether the HOA coefficients 11 represent content generated from a live recording or content generated from an audio object. The content analysis unit 26 can determine whether the HOA coefficients 11 are generated from a recording of an actual sound field or from an artificial audio object. In some examples, when the framed HOA coefficients 11 are generated from a recording, the content analysis unit 26 passes the HOA coefficients 11 to a vector-based decomposition unit 27. In some examples, when the framed HOA coefficients 11 are generated from a synthesized audio object, the content analysis unit 26 passes the HOA coefficients 11 to a direction-based synthesis unit 28. The direction-based synthesis unit 28 can represent a unit configured to perform direction-based synthesis of the HOA coefficients 11 to generate a direction-based bitstream 21.

As shown in the example of Figure 3, the vector-based decomposition unit 27 may include a linear reversible transform (LIT) unit 30, a parameter calculation unit 32, a reordering unit 34, a foreground selection unit 36, an energy compensation unit 38, a psychoacoustic audio decoder unit 40, a bitstream generation unit 42, a sound field analysis unit 44, a coefficient reduction unit 46, a background (BG) selection unit 48, a space-time interpolation unit 50 and a quantization unit 52.

The linear invertible transform (LIT) unit 30 receives HOA coefficients 11 in the form of HOA channels, each of which represents a block or frame of coefficients associated with a given order or sub-order of a spherical basis function (which may be denoted as HOA[k], where k may indicate the current frame or block of samples). The matrix of HOA coefficients 11 may have dimensions D: M×(N+1) ^² .

The LIT unit 30 may represent a unit configured to perform a form of analysis known as singular value decomposition. Although described with respect to SVD, the techniques described in this disclosure may be performed on any similar transform or decomposition of a set that provides linearly uncorrelated, energy-dense outputs. Moreover, references to "sets" in this disclosure are generally intended to refer to non-zero sets (unless specifically stated to the contrary) and are not intended to refer to the classical mathematical definition of a set that includes the so-called "empty set." Alternative transforms may include principal component analysis, commonly referred to as "PCA." Depending on the context, PCA may be referred to by several different names, such as, to name a few, the discrete Kärnen-Loewe transform, the Hotelling transform, proper orthogonal decomposition (POD), and eigenvalue decomposition (EVD). Characteristics of such operations that contribute to the fundamental goal of compressing audio data are "energy compression" and "decorrelation" of multi-channel audio data.

In any case, for purposes of example, assuming that the LIT unit 30 performs singular value decomposition (which may also be referred to as "SVD"), the LIT unit 30 may transform the HOA coefficients 11 into sets of two or more transformed HOA coefficients. A "set" of transformed HOA coefficients may include a vector of transformed HOA coefficients. In the example of FIG. 3 , the LIT unit 30 may perform SVD on the HOA coefficients 11 to generate so-called V matrices, S matrices, and U matrices. In linear algebra, SVD may represent the factorization of a y-by-z real or complex matrix X (where X may represent multi-channel audio data, such as the HOA coefficients 11) in the following form:

X＝USV*

U may represent a y-by-y real or complex unitary matrix, where the y columns of U are referred to as the left singular vectors of the multi-channel audio data. S may represent a y-by-z rectangular diagonal matrix with non-negative real numbers on the diagonal, where the diagonal values of S are referred to as the singular values of the multi-channel audio data. V* (which may denote the conjugate transpose of V) may represent a z-by-z real or complex unitary matrix, where the z columns of V* are referred to as the right singular vectors of the multi-channel audio data.

In some examples, the V* matrix in the SVD mathematical expression mentioned above is labeled as the conjugate transpose of the V matrix to reflect that SVD can be applied to matrices including complex numbers. When applied to matrices including only real numbers, the complex conjugate of the V matrix (or in other words, the V* matrix) can be considered as the transpose of the V matrix. For ease of explanation below, it is assumed that the HOA coefficients 11 include real numbers, and the result is that the V matrix is output via SVD rather than the V* matrix. In addition, although labeled as the V matrix in this disclosure, references to the V matrix should be understood to refer to the transpose of the V matrix where appropriate. Although assumed to be the V matrix, the described technique can be applied in a similar manner to the HOA coefficients 11 having complex coefficients, where the output of the SVD is the V* matrix. Therefore, in this regard, the described technique should not be limited to providing only the application of SVD to generate the V matrix, but may include applying SVD to the HOA coefficients 11 having complex components to generate the V* matrix.

In this manner, the LIT unit 30 may perform SVD on the HOA coefficients 11 to output a US[k] vector 33 having dimensions D: M×(N+1) ² (which may represent a combined version of the S vector and the U vector) and a V[k] vector 35 having dimensions D: (N+1) ² ×(N+1) ^2. Individual vector elements in the US[k] matrix may also be referred to as _XPS (k), and individual vectors in the V[k] matrix may also be referred to as v(k).

Analysis of the U, S, and V matrices can show that these matrices carry or represent the spatial and temporal characteristics of the basic sound field, represented above by X. Each of the N vectors in U (which is M samples long) can represent a normalized, separated audio signal that varies with time (for a time period represented by M samples), is orthogonal to one another, and has been decoupled from any spatial characteristics (which can also be referred to as directional information). The spatial characteristics representing spatial shape and position (r, θ, ) can alternatively be represented by individual i-th vectors v ⁽ⁱ⁾ (k) in the V matrix (each having a length of (N+1) ^² ). The individual elements of each v ⁽ⁱ⁾ (k) vector can represent HOA coefficients, which describe the shape (including width) and position of the sound field of the associated audio object. The vectors in both the U and V matrices are normalized so that their root mean square energy is equal to one. The energy of the audio signal in U is thus represented by the diagonal elements in S. Multiplying U and S forms US[k] (having individual vector elements _XPS (k)), thus representing the audio signal having energy. The ability of the SVD decomposition to decouple the audio time signal (in U), its energy (in S), and its spatial characteristics (in V) can support various aspects of the techniques described in this disclosure. Additionally, the model of synthesizing the elementary HOA[k] coefficients X by vector multiplication of US[k] and V[k] yields the term "vector-based decomposition" used throughout this document.

Although described as being performed directly on the HOA coefficients 11, the LIT unit 30 may apply a linear reversible transform to the derived terms of the HOA coefficients 11. For example, the LIT unit 30 may apply SVD to the power spectral density matrix derived from the HOA coefficients 11. By performing SVD on the power spectral density (PSD) of the HOA coefficients rather than the coefficients themselves, the LIT unit 30 may potentially reduce the computational complexity of performing the SVD in terms of one or more of processor cycles and memory space, while achieving the same source audio coding efficiency as if the SVD were applied directly to the HOA coefficients.

The parameter calculation unit 32 represents a unit configured to calculate various parameters, such as a correlation parameter (R), directional characteristic parameters (θ, r), and energy characteristic (e). Each of the parameters of the current frame may be denoted as R[k], θ[k], r[k], and e[k]. The parameter calculation unit 32 may perform energy analysis and/or correlation (or so-called cross-correlation) on the US[k] vector 33 to identify these parameters. The parameter calculation unit 32 may also determine the parameters of the previous frame, where the parameters of the previous frame may be denoted as R[k-1], θ[k-1], r[k-1], and e[k-1] based on the previous frame having the US[k-1] vector and the V[k-1] vector. The parameter calculation unit 32 may output the current parameters 37 and the previous parameters 39 to the reordering unit 34.

The parameters calculated by the parameter calculation unit 32 may be used by the reordering unit 34 to reorder the audio objects to represent their natural evaluation or continuity over time. The reordering unit 34 may compare each of the parameters 37 from the first US[k] vector 33 with each of the parameters 39 of the second US[k-1] vector 33 in terms of order. The reordering unit 34 may reorder the various vectors within the US[k] matrix 33 and the V[k] matrix 35 based on the current parameters 37 and the previous parameters 39 (using, as an example, the Hungarian algorithm) to output the reordered US[k] matrix 33′ (which may be mathematically designated as ) and the reordered V[k] matrix 35′ (which may be mathematically designated as ) to the foreground sound (or dominant sound (PS)) selection unit 36 (“foreground selection unit 36”) and the energy compensation unit 38.

The soundfield analysis unit 44 may represent a unit configured to perform a soundfield analysis on the HOA coefficients 11 in order to potentially achieve the target bitrate 41. The soundfield analysis unit 44 may determine the total number of psychoacoustic decoder instantiations (which may be a function of the total number of ambient or background channels (BG _TOT )) and the number of foreground channels (or in other words, dominant channels) based on the analysis and/or based on the received target bitrate 41. The total number of psychoacoustic decoder instantiations may be denoted as numHOATransportChannels.

Again to potentially achieve the target bit rate 41, the sound field analysis unit 44 may also determine the total number of foreground channels (nFG) 45, the minimum order of the background (or in other words, ambient) sound field ( _NBG or alternatively, MinAmbHOAorder), the corresponding number of actual channels representing the minimum order of the background sound field (nBGa = (MinAmbHOAorder + 1) ² ), and the index (i) of the additional BG HOA channel to be sent (which may be collectively labeled as background channel information 43 in the example of FIG. 3 ). The background channel information 42 may also be referred to as ambient channel information 43. Each of the channels remaining from numHOATransportChannels - nBGa may be an "additional background/ambient channel," an "active vector-based dominant channel," an "active direction-based dominant signal," or "completely inactive." In one aspect, the channel type may be a syntax element indicated by two bits ("ChannelType") (e.g., 00: direction-based signal; 01: vector-based dominant signal; 10: additional ambient signal; 11: inactive signal). The total number of background or ambient signals, nBGa, may be given by (MinAmbHOAorder+1) ² + index 10 (in the above example) as the number of times the channel type appears in the bitstream of the frame.

The sound field analysis unit 44 may select the number of background (or in other words, ambient) channels and the number of foreground (or in other words, dominant) channels based on the target bit rate 41, thereby selecting more background and/or foreground channels when the target bit rate 41 is relatively high (e.g., when the target bit rate 41 is equal to or greater than 512 Kbps). In one aspect, in the header portion of the bitstream, numHOATransportChannels may be set to 8 and MinAmbHOAorder may be set to 1. In this scenario, at each frame, four channels may be dedicated to representing the background or ambient portion of the sound field, while the other four channels may vary with the channel type on a frame-by-frame basis, such as either being used as additional background/ambient channels or foreground/dominant channels. The foreground/dominant signal may be one of a vector-based or a direction-based signal, as described above.

In some examples, the total number of vector-based dominant signals for a frame may be given by the number of times the ChannelType index is 01 in the bitstream of the frame. In the above aspect, for each additional background/ambient channel (e.g., corresponding to ChannelType 10), corresponding information may be provided as to which of the possible HOA coefficients (other than the first four) are represented in the channel. For fourth-order HOA content, the information may be indices indicating HOA coefficients 5 to 25. The first four ambient HOA coefficients 1 to 4 may always be sent when minAmbHOAorder is set to 1, and thus the audio encoding device may only need to indicate one of the additional ambient HOA coefficients with indices 5 to 25. Therefore, the information may be sent using a 5-bit syntax element (for fourth-order content), which may be designated "CodedAmbCoeffIdx". In any case, the sound field analysis unit 44 outputs the background channel information 43 and the HOA coefficient 11 to the background (BG) selection unit 36, outputs the background channel information 43 to the coefficient reduction unit 46 and the bitstream generation unit 42, and outputs the nFG 45 to the foreground selection unit 36.

The background selection unit 48 may represent a unit configured to determine background or ambient HOA coefficients 47 based on background channel information, such as the background sound field (N _BG ) and the number (nBGa) and index (i) of additional BG HOA channels to be sent. For example, when N _BG is equal to one, the background selection unit 48 may select the HOA coefficients 11 for each sample of the audio frame having an order equal to or less than one. In this example, the background selection unit 48 may then select the HOA coefficients 11 having an index identified by one of the indices (i) as the additional BG HOA coefficients, where nBGa to be specified in the bitstream 21 is provided to the bitstream generation unit 42 to enable the audio decoding device (e.g., the audio decoding device 24 shown in the examples of Figures 2 and 4) to parse the background HOA coefficients 47 from the bitstream 21. The background selection unit 48 may then output the ambient HOA coefficients 47 to the energy compensation unit 38. The ambient HOA coefficients 47 may have dimensions D: M×[(N _BG +1) ² +nBGa]. The ambient HOA coefficients 47 may also be referred to as “ambient HOA coefficients 47 ,” where each of the ambient HOA coefficients 47 corresponds to a separate ambient HOA channel 47 to be encoded by the psychoacoustic audio coder unit 40 .

The foreground selection unit 36 may represent a unit configured to select a reordered US[k] matrix 33′ and a reordered V[k] matrix 35′ representing foreground or distinct components of the sound field based on nFG 45 (which may represent one or more indices identifying a foreground vector). The foreground selection unit 36 may output an nFG signal 49 (which may be represented as reordered US[k] _{1, ..., nFG} 49, FG _{1, ..., nfG} [k] 49, or 49) to the psychoacoustic audio decoder unit 40, where the nFG signal 49 may have dimensions D: M×nFG, and each represents a mono audio object. The foreground selection unit 36 can also output the reordered V[k] matrix 35' (or v ^(1..nFG) (k)35') corresponding to the foreground components of the sound field to the spatial-temporal interpolation unit 50, where the subset corresponding to the foreground components in the reordered V[k] matrix 35' can be represented as a foreground V[k] matrix 51 _k with dimension D: ((N+1) ² ×nFG) (which can be mathematically represented as).

The energy compensation unit 38 may represent a unit configured to perform energy compensation on the ambient HOA coefficients 47 to compensate for the energy loss due to the removal of each of the HOA channels by the background selection unit 48. The energy compensation unit 38 may perform an energy analysis on one or more of the reordered US[k] matrix 33', the reordered V[k] matrix 35', the nFG signal 49, the foreground V[k] vector _51k , and the ambient HOA coefficients 47, and then perform energy compensation based on the energy analysis to produce energy-compensated ambient HOA coefficients 47'. The energy compensation unit 38 may output the energy-compensated ambient HOA coefficients 47' to the decorrelation unit 40'. The decorrelation unit 40' may then implement the techniques of the present disclosure to reduce or eliminate correlation between background signals of the HOA coefficients 47' to form one or more decorrelated HOA coefficients 47". The decorrelation unit 40' may output the decorrelated HOA coefficients 47" to the psychoacoustic audio decoder unit 40.

The spatial-temporal interpolation unit 50 may represent a unit configured to receive the foreground V[k] vector 51 _k of the kth frame and the foreground V[k-1] vector 51 _k-1 of the previous frame (hence the k-1 notation) and perform spatial-temporal interpolation to generate an interpolated foreground V[k] vector. The spatial-temporal interpolation unit 50 may recombines the nFG signal 49 with the foreground V[k] vector 51 _k to recover the reordered foreground HOA coefficients. The spatial-temporal interpolation unit 50 may then divide the reordered foreground HOA coefficients by the interpolated V[k] vector to generate the interpolated nFG signal 49'. The spatial-temporal interpolation unit 50 may also output the foreground V[k] vector 51 _k , which is used to generate the interpolated foreground V[k] vector so that an _audio decoding device, such as the audio decoding device 24, can generate the interpolated foreground V[k] vector and thereby recover the foreground V[k] vector 51 _k . The foreground V[k] vector 51 _k used to generate the interpolated foreground V[k] vector is labeled as the remaining foreground V[k] vector 53. To ensure that the same V[k] and V[k-1] are used at the encoder and decoder (to create the interpolated vector V[k]), quantized/dequantized versions of the vectors may be used at the encoder and decoder. The spatio-temporal interpolation unit 50 may output the interpolated nFG signal 49' to the psychoacoustic audio decoder unit 46 and the interpolated foreground V[k] vector 51 _k to the coefficient reduction unit 46.

The coefficient reduction unit 46 may represent a unit configured to perform coefficient reduction on the remaining foreground V[k] vector 53 based on the background channel information 43 to output a reduced foreground V[k] vector 55 to the quantization unit 52. The reduced foreground V[k] vector 55 may have a dimension D: [(N+1) ^2- (N _BG +1) ² -BG _TOT ]×nFG. The coefficient reduction unit 46 may, in this regard, represent a unit configured to reduce the number of coefficients in the remaining foreground V[k] vector 53. In other words, the coefficient reduction unit 46 may represent a unit configured to eliminate coefficients in the foreground V[k] vector (forming the remaining foreground V[k] vector 53) that have little or almost no directional information. In some examples, the coefficients of the distinct or (in other words) foreground V[k] vector corresponding to the first-order and zeroth-order basis functions (which may be designated as N _BG ) provide little directional information and, therefore, may be removed from the foreground V vector (by a process that may be referred to as "coefficient reduction"). In this example, greater flexibility may be provided to identify not only coefficients corresponding to _NBG from the set [( _NBG +1) ² +1, (N+1) ² ] but also additional HOA channels (which may be denoted by the variable TotalOfAddAmbHOAChan).

The quantization unit 52 may represent a unit configured to perform any form of quantization to compress the reduced foreground V[k] vectors 55 to produce a coded foreground V[k] vector 57, thereby outputting the coded foreground V[k] vector 57 to the bitstream generation unit 42. In operation, the quantization unit 52 may represent a unit configured to compress the spatial components of the soundfield (i.e., one or more of the reduced foreground V[k] vectors 55 in this example). The quantization unit 52 may perform any of the following 12 quantization modes, as indicated by the quantization mode syntax element labeled "NbitsQ":

NbitsQ value Quantization mode type

0-3: Reserved

4: Vector Quantization

5: Scalar Quantization without Huffman Decoding

6: 6-bit scalar quantization with Huffman decoding

7: 7-bit scalar quantization with Huffman decoding

8: 8-bit scalar quantization with Huffman decoding

… …

16: 16-bit scalar quantization with Huffman decoding

Quantization unit 52 may also perform a predicted version of any of the aforementioned types of quantization modes, in which the differences between the elements of the V vector of the previous frame (or weights when performing vector quantization) and the elements of the V vector of the current frame (or weights when performing vector quantization) are determined. Quantization unit 52 may then quantize the differences between the elements or weights of the current and previous frames rather than the values of the elements of the V vector of the current frame itself.

Quantization unit 52 may perform multiple forms of quantization on each of the reduced foreground V[k] vectors 55 to obtain multiple coded versions of the reduced foreground V[k] vectors 55. Quantization unit 52 may select one of the coded versions of the reduced foreground V[k] vectors 55 as the coded foreground V[k] vector 57. In other words, quantization unit 52 may select one of an unpredicted vector quantized V vector, a predicted vector quantized V vector, an unHuffman coded scalar quantized V vector, and a Huffman coded scalar quantized V vector for use as the output switched quantized V vector based on any combination of the criteria discussed in this disclosure. In some examples, quantization unit 52 may select a quantization mode from a set of quantization modes including a vector quantization mode and one or more scalar quantization modes, and quantize the input V vector based on (or in accordance with) the selected mode. Quantization unit 52 may then provide a selected one of the following to bitstream generation unit 52 for use as the coded foreground V[k] vector 57: an unpredicted vector quantized V vector (e.g., in terms of weight values or bits indicating weight values), a predicted vector quantized V vector (e.g., in terms of error values or bits indicating error values), a unHuffman coded scalar quantized V vector, and a Huffman coded scalar quantized V vector. Quantization unit 52 may also provide a syntax element indicating a quantization mode (e.g., an NbitsQ syntax element) and any other syntax elements used to dequantize or otherwise reconstruct the V vector.

The decorrelation unit 40′ included within the audio encoding device 20 may represent a single or multiple instances of a unit configured to apply one or more decorrelation transforms to the HOA coefficients 47′ to obtain decorrelated HOA coefficients 47″. In some examples, the decorrelation unit 40′ may apply a UHJ matrix to the HOA coefficients 47′. In various examples of the present disclosure, the UHJ matrix may also be referred to as a “phase-based transform.” Applying a phase-based transform may also be referred to herein as “phase-shift decorrelation.”

The Ambisonics UHJ format is a development of an Ambisonics surround sound system designed to be compatible with both mono and stereo media. The UHJ format encompasses a system hierarchy in which the recorded sound field is reproduced with a degree of accuracy that varies depending on the available channels. In various instances, UHJ is also referred to as the "C-format." The abbreviations indicate some of the sources incorporated into the system: U from Universal (UD-4); H from Matrix H; and J from System 45J.

UHJ is a layered system for encoding and decoding directional sound information within ambisonics. Depending on the number of available channels, the system can carry more or less information. UHJ is fully compatible with both stereo and mono. Up to four channels (L, R, T, Q) can be used.

In one form, 2-channel (L, R) UHJ, horizontal (or "planar") surround information can be carried by quadrature stereo signal channels (CD, FM, or digital radio, etc.), which can be recovered at the listening end using a UHJ decoder. Summing the two channels produces a compatible mono signal that can be a more accurate representation of the two-channel version than a conventional "panpotted mono" source. If a third channel (T) is available, then when decoded via a 3-channel UHJ decoder, the third channel can be used to produce improved localization accuracy for the planar surround effect. The third channel may not necessarily have the full audio bandwidth for this purpose, leading to the possibility of a so-called "2.5-channel" system, in which the third channel is limited in bandwidth. In one example, the limit may be 5 kHz. The third channel can be broadcast via FM radio, for example, using phase quadrature modulation. Adding a fourth channel (Q) to the UHJ system allows encoding of full surround sound at a height n (sometimes referred to as periphony) with the same degree of accuracy as the 4-channel B-format.

2-channel UHJ is a format commonly used for the distribution of ambisonic recordings. 2-channel UHJ recordings can be transmitted via all orthogonal stereo channels and can use any orthogonal 2-channel media without modification. UHJ is stereo-compatible, as the listener can perceive a stereo image without decoding, but it is significantly wider than conventional stereo (e.g., so-called "super stereo"). The left and right channels can also be summed for an extremely high degree of mono compatibility. Playback through a UHJ decoder reveals surround sound capabilities.

An example mathematical representation of the decorrelation unit 40' applying the UHJ matrix (or phase-based transform) is as follows:

UHJ Code:

S＝(0.9397*W)+(0.1856*X);

D=imag(hilbert((-0.3420*W)+(0.5099*X)))+(0.6555*Y);

T=imag(hilbert((-0.1432*W)+(0.6512*X)))-(0.7071*Y);

Q = 0.9772*Z;

Conversion of S and D to left and right:

Left = (S + D) / 2

Right = (S-D)/2

According to some implementations of the above calculation, assumptions regarding the above calculation may include the following: the HOA background channel is a first-order ambisonic reverberation, the FuMa is normalized, and the ambisonic reverberation channel numbering order is W(a00), X(a11), Y(a11-), Z(a10).

In the calculations listed above, decorrelation unit 40' may perform scalar multiplications of various matrices with constant values. For example, to obtain the S signal, decorrelation unit 40' may perform scalar multiplications of the W matrix with the constant value 0.9397 (e.g., via scalar multiplications) and the X matrix with the constant value 0.1856. As also illustrated in the calculations listed above, decorrelation unit 40' may apply a Hilbert transform (denoted by the "Hilbert()" function in the UHJ encoding above) when obtaining each of the D and T signals. The "imag()" function in the UHJ encoding above indicates that the imaginary number (in the mathematical sense) of the result of the Hilbert transform is obtained.

Another example mathematical representation of the decorrelation unit 40' applying the UHJ matrix (or phase-based transform) is as follows:

UHJ Code:

S＝(0.9396926*W)+(0.151520536509082*X);

D=imag(hilbert((-0.3420201*W)+(0.416299273350443*X)))+(0.535173990363608*Y);

T＝0.940604061228740*(imag(hilbert((-0.1432*W)+(0.531702573500135*X)))-(0.577350269189626*Y));

Q = Z;

Conversion of S and D to left and right:

Left = (S + D) / 2;

Right = (S-D)/2;

In some example implementations of the above calculations, assumptions regarding the above calculations may include the following: the HOA background channel is 1st-order ambisonics, and N3D (or "full 3D") normalized to the ambisonic channel numbering order W(a00), X(a11), Y(a11-), Z(a10). Although described herein with respect to N3D normalization, it should be understood that the example calculations may also be applied to SN3D-normalized (or "Schmidt semi-normalized") HOA background channels. N3D and SN3D normalizations may differ in the scaling factors used. An example representation of N3D normalization relative to SN3D normalization is expressed as follows:

An example of the weighting coefficients used in SN3D normalization is expressed as follows:

In the calculations listed above, decorrelation unit 40' may perform scalar multiplications of various matrices with constant values. For example, to obtain the S signal, decorrelation unit 40' may perform scalar multiplications of the W matrix with the constant value 0.9396926 (e.g., via scalar multiplications) and the X matrix with the constant value 0.151520536509082. As also illustrated in the calculations listed above, decorrelation unit 40' may apply a Hilbert transform (denoted by the "Hilbert()" function or phase-shift decorrelation in the UHJ encoding above) when obtaining each of the D and T signals. The "imag()" function in the UHJ encoding above indicates the imaginary number (in the mathematical sense) of the result of the Hilbert transform.

The decorrelation unit 40' may perform the calculations listed above so that the resulting S and D signals represent the left and right audio signals (or in other words, the stereo audio signal). In some such scenarios, the decorrelation unit 40' may output the T and Q signals as part of the decorrelated HOA coefficients 47", but the T and Q signals may not be processed by a decoding device receiving the bitstream 21 when the T and Q signals are reproduced to a stereo speaker geometry (or in other words, a stereo speaker configuration). In an example, the HOA coefficients 47' may represent a sound field that would be reproduced on a mono audio reproduction system. The decorrelation unit 40' may output the S and D signals as part of the decorrelated HOA coefficients 47", and the decoding device receiving the bitstream 21 may combine (or "mix") the S and D signals to form an audio signal that is to be reproduced and/or output in a mono audio format. In these examples, the decoding device and/or the reproduction device may recover the mono audio signal in various ways. One example is by mixing the left and right signals (represented by the S and D signals). Another example is by applying a UHJ matrix (or phase-based transform) to decode the W signal (discussed in more detail below with respect to FIG. 5 ). By applying a UHJ matrix (or phase-based transform) to generate an intrinsic left signal and an intrinsic right signal in the form of an S signal and a D signal, decorrelation unit 40 ′ can implement the techniques of this disclosure to provide potential advantages and/or potential improvements over techniques that apply other decorrelation transforms (e.g., pattern matrices described in the MPEG-H standard).

In various examples, the decorrelation unit 40' may apply different decorrelation transforms based on the bit rate of the received HOA coefficients 47'. For example, in a scenario where the HOA coefficients 47' represent a four-channel input, the decorrelation unit 40' may apply the UHJ matrix (or phase-based transform) described above. More specifically, based on the HOA coefficients 47' representing a four-channel input, the decorrelation unit 40' may apply a 4×4 UHJ matrix (or phase-based transform). For example, the 4×4 matrix may be orthogonal to the four-channel input of the HOA coefficients 47'. In other words, in instances where the HOA coefficients 47' represent a smaller number of channels (e.g., four), the decorrelation unit 40' may apply the UHJ matrix as the selected decorrelation transform to decorrelate the background signal of the HOA signal 47' to obtain decorrelated HOA coefficients 47".

Following this example, if the HOA coefficients 47′ represent a larger number of channels (e.g., nine), the decorrelation unit 40′ may apply a decorrelation transform other than the UHJ matrix (or phase-based transform). For example, in a scenario where the HOA coefficients 47′ represent a nine-channel input, the decorrelation unit 40′ may apply a pattern matrix (e.g., as described in the MPEG-H standard) to decorrelate the HOA coefficients 47′. In an example where the HOA coefficients 47′ represent a nine-channel input, the decorrelation unit 40′ may apply a 9×9 pattern matrix to obtain decorrelated HOA coefficients 47″.

Then, various components of the audio encoding device 20 (e.g., the psychoacoustic audio decoder 40) may perceptually decode the decorrelated HOA coefficients 47″ according to AAC or USAC. The decorrelation unit 40′ may apply a phase-shift decorrelation transform (e.g., a UHJ matrix or a phase-based transform in the case of a four-channel input) to optimize the AAC/USAC decoding for HOA. In instances where the HOA coefficients 47′ (and thereby, the decorrelated HOA coefficients 47″) represent audio data to be reproduced on a stereo reproduction system, the decorrelation unit 40′ may apply the techniques of the present invention to improve or optimize compression based on (or being optimized for) relatively oriented stereo audio data.

It will be understood that the decorrelation unit 40′ may apply the techniques described herein in scenarios where the energy-compensated HOA coefficients 47′ include foreground channels, as well as in scenarios where the energy-compensated HOA coefficients 47′ do not include any foreground channels. As an example, in scenarios where the energy-compensated HOA coefficients 47′ include zero (0) foreground channels and four (4) background channels (e.g., lower/smaller bit rate scenarios), the decorrelation unit 40′ may apply the techniques and/or calculations described above.

In some examples, the decorrelation unit 40′ may cause the bitstream generation unit 42 to signal one or more syntax elements as part of the vector-based bitstream 21 that indicate that the decorrelation unit 40′ applies a decorrelating transform to the HOA coefficients 47′. By providing this indication to the decoding device, the decorrelation unit 40′ may enable the decoding device to perform a reciprocal decorrelating transform on the audio data in the HOA domain. In some examples, the decorrelation unit 40′ may cause the bitstream generation unit 42 to signal a syntax element that indicates which decorrelating transform to apply, such as a UHJ matrix (or other phase-based transform) or a pattern matrix.

The decorrelation unit 40' may apply a phase-based transform to the energy-compensated ambient HOA coefficients 47'. The phase-based transform for the first O _MIN HOA coefficient sequence of _CAMB (k-1) is defined as follows

Where the coefficient d is defined as in Table 1, and the signal frames S(k-2) and M(k-2) are defined as follows

S(k-2)＝A ₊₉₀ (k-2)+d(6)·c _{AMB, 2} (k-2)

M(k-2)＝d(4)·c _{AMB, 1} (k-2)+d(5)·c _{AMB, 4} (k-2)

And A ₊₉₀ (k-2) and B ₊₉₀ (k-2) are frames of +90 degree phase shifted signals A and B, defined as follows

Thus a phase-based transformation of the first O _MIN HOA coefficient sequence for CP _,AMB (k-1) is defined.The described transformation may introduce a delay of one frame.

In the above, x _AMB,LOW,1 (k-2) to x _AMB,LOW,4 (k-2) may correspond to decorrelated ambient HOA coefficients 47". In the above equations, the varying C _AMB,1 (k) variable indicates the HOA coefficient corresponding to the k-th frame having a spherical basis function with (order:sub-order) of (0:0), which may also be referred to as the 'W' channel or component. The varying C _AMB,2 (k) variable indicates the HOA coefficient corresponding to the k-th frame having a spherical basis function with (order:sub-order) of (1:-1), which may also be referred to as the 'Y' channel or component. The varying C _AMB,3 (k) variable indicates the HOA coefficient corresponding to the k-th frame having a spherical basis function with (order:sub-order) of (1:0), which may also be referred to as the 'Z' channel or component. The varying C _AMB,4 The (k) variable designates the HOA coefficient corresponding to the kth frame with a spherical basis function of (order:suborder) of (1:1), which may also be referred to as the 'X' channel or component. _CAMB,1 (k) to _CAMB,3 (k) may correspond to the ambient HOA coefficients 47'.

Table 1 below illustrates an example of coefficients that may be used by decorrelation unit 40 to perform a phase-based transform.

n d(n) 0 0.34202009999999999 1 0.41629927335044281 2 0.14319999999999999 3 0.53170257350013528 4 0.93969259999999999 5 0.15152053650908184 6 0.53517399036360758 7 0.57735026918962584 8 0.94060406122874030 9 0.500000000000000

Table 1 Coefficients for phase-based transformation

In some examples, various components of the audio encoding device 20 (e.g., the bitstream generation unit 42) may be configured to transmit only first-order HOA representations for lower target bit rates (e.g., a target bit rate of 128K or 256K). According to some such examples, the audio encoding device 20 (or its components, such as the bitstream generation unit 42) may be configured to discard higher-order HOA coefficients (e.g., coefficients having an order greater than first order (or in other words, N>1)). However, in examples where the audio encoding device 20 determines that the target bit rate is relatively high, the audio encoding device 20 (e.g., the bitstream generation unit 42) may separate the foreground channel from the background channel and may allocate bits (e.g., in a larger amount) to the foreground channel.

The psychoacoustic audio decoder unit 40 included in the audio encoding device 20 may represent multiple instances of a psychoacoustic audio decoder, each of which is used to encode a different audio object or HOA channel for each of the decorrelated HOA coefficients 47″ and the interpolated nFG signal 49′ to produce encoded ambient HOA coefficients 59 and an encoded nFG signal 61. The psychoacoustic audio decoder unit 40 may output the encoded ambient HOA coefficients 59 and the encoded nFG signal 61 to the bitstream generation unit 42.

The bitstream generation unit 42 included in the audio encoding device 20 represents a unit that formats data to conform to a known format (which may be a format known to the decoding device), thereby generating a vector-based bitstream 21. In other words, the bitstream 21 may represent encoded audio data that has been encoded in the manner described above. In some examples, the bitstream generation unit 42 may represent a multiplexer that can receive the coded foreground V[k] vector 57, the encoded ambient HOA coefficients 59, the encoded nFG signal 61, and the background channel information 43. The bitstream generation unit 42 may then generate the bitstream 21 based on the coded foreground V[k] vector 57, the encoded ambient HOA coefficients 59, the encoded nFG signal 61, and the background channel information 43. In this manner, the bitstream generation unit 42 may thereby specify the vector 57 in the bitstream 21 to obtain the bitstream 21. The bitstream 21 may include a primary or main bitstream and one or more side channel bitstreams.

Although not shown in the example of FIG3 , the audio encoding device 20 may further include a bitstream output unit that switches the bitstream output from the audio encoding device 20 based on whether the current frame is to be encoded using direction-based synthesis or vector-based synthesis (e.g., switches between a direction-based bitstream 21 and a vector-based bitstream 21). The bitstream output unit may perform the switching based on a syntax element output by the content analysis unit 26 indicating whether direction-based synthesis is to be performed (as a result of detecting that the HOA coefficients 11 are generated from a synthesized audio object) or vector-based synthesis is to be performed (as a result of detecting that the HOA coefficients are recorded). The bitstream output unit may specify the correct header syntax to indicate the switching or current encoding for the current frame and the corresponding one in the bitstream 21.

In addition, as mentioned above, the sound field analysis unit 44 may identify BG _TOT ambient HOA coefficients 47, which may change from frame to frame (but sometimes the BG _TOT may remain constant or the same across two or more adjacent (in time) frames). Changes in the BG _TOT may result in changes in the coefficients expressed in the reduced foreground V[k] vector 55. Changes in the BG _TOT may result in changes in the background HOA coefficients (which may also be referred to as "ambient HOA coefficients") from frame to frame (but again, the BG _TOT may sometimes remain constant or the same across two or more adjacent (in time) frames). The changes typically result in energy changes in various aspects of the sound field, which are represented by the addition or removal of additional ambient HOA coefficients and the corresponding removal of coefficients from the reduced foreground V[k] vector 55 or the addition of coefficients to the reduced foreground V[k] vector 55.

Thus, the soundfield analysis unit 44 may further determine when the ambient HOA coefficients change from frame to frame and generate a flag or other syntax element indicating the change in the ambient HOA coefficients used to represent the ambient component of the soundfield (wherein the change may also be referred to as a "transition" of the ambient HOA coefficients or a "transition" of the ambient HOA coefficients. In particular, the coefficient reduction unit 46 may generate a flag (which may be denoted as an AmbCoeffTransition flag or an AmbCoeffIdxTransition flag), providing the flag to the bitstream generation unit 42 so that the flag may be included in the bitstream 21 (possibly as part of the side channel information).

In addition to specifying the ambient coefficient transition flag, the coefficient reduction unit 46 may also modify the manner in which the reduced foreground V[k] vector 55 is generated. In one example, upon determining that one of the ambient HOA ambient coefficients is in transition during the current frame, the coefficient reduction unit 46 may specify a vector coefficient (which may also be referred to as a "vector element" or "element") for each of the V vectors of the reduced foreground V[k] vector 55 that corresponds to the ambient HOA coefficient that is in transition. Furthermore, the ambient HOA coefficient that is in transition may be added to or removed from the total BG _TOT number of background coefficients. Thus, the resulting change in the total number of background coefficients affects whether the ambient HOA coefficient is included in the bitstream _, and whether the corresponding element of the V vector is included for the specified V vector in the bitstream in the second and third configuration modes described above. More information regarding how coefficient reduction unit 46 may specify reduced foreground V[k] vectors 55 to overcome energy changes is provided in U.S. application Ser. No. 14/594,533, filed Jan. 12, 2015, entitled “TRANSITIONING OF AMBIENT HIGHER-ORDERAMBISONIC COEFFICIENTS.”

Thus, the audio encoding device 20 may represent an example of a device for compressing audio, the device being configured to apply a decorrelation transform to ambisonic reverberation coefficients to obtain a decorrelated representation of the ambisonic reverberation coefficients, the ambisonic reverberation coefficients having been extracted from a plurality of higher-order ambisonic reverberation coefficients and representing background components of a sound field described by the plurality of higher-order ambisonic reverberation coefficients, wherein at least one of the plurality of higher-order ambisonic reverberation coefficients is associated with a spherical basis function having an order greater than one. In some examples, to apply the decorrelation transform, the device is configured to apply a UHJ matrix to the ambisonic reverberation coefficients.

In some examples, the device is further configured to normalize the UHJ matrix according to N3D (full three-dimensional) normalization. In some examples, the device is further configured to normalize the UHJ matrix according to SN3D normalization (Schmidt semi-normalization). In some examples, the ambisonic reverberation coefficients are associated with spherical basis functions of order zero or order one, and to apply the UHJ matrix to the ambisonic reverberation coefficients, the device is configured to perform a scalar multiplication of the UHJ matrix for at least a subset of the ambisonic reverberation coefficients. In some examples, to apply a decorrelation transform, the device is configured to apply a mode matrix to the ambisonic reverberation coefficients.

According to some examples, to apply the decorrelation transform, the device is configured to obtain the left signal and the right signal from the decorrelated ambisonic reverberation coefficients. According to some examples, the device is further configured to signal the decorrelated ambisonic reverberation coefficients and the one or more foreground channels. According to some examples, to signal the decorrelated ambisonic reverberation coefficients and the one or more foreground channels, the device is configured to signal the decorrelated ambisonic reverberation coefficients and the one or more foreground channels in response to determining that the target bit rate meets or exceeds a predetermined threshold.

In some examples, the device is further configured to signal decorrelated ambisonic reverberation coefficients without signaling any foreground channels. In some examples, to signal decorrelated ambisonic reverberation coefficients without signaling any foreground channels, the device is configured to signal decorrelated ambisonic reverberation coefficients without signaling any foreground channels in response to determining that a target bit rate is below a predetermined threshold. In some examples, the device is further configured to signal an indication that a decorrelation transform has been applied to the ambisonic reverberation coefficients. In some examples, the device further includes a microphone array configured to capture the audio data to be compressed.

4 is a block diagram illustrating in greater detail the audio decoding device 24 of FIG 2. As shown in the example of FIG 4, the audio decoding device 24 may include an extraction unit 72, a direction-based reconstruction unit 90, a vector-based reconstruction unit 92, and a re-correlation unit 81.

Although described below, more information regarding the audio decoding device 24 and various aspects of decompressing or otherwise decoding HOA coefficients may be obtained in International Patent Application Publication No. WO 2014/194099, filed on May 29, 2014, entitled “INTERPOLATION FOR DECOMPOSED REPRESENTATIONS OF A SOUND FIELD.”

The extraction unit 72 may represent a unit configured to receive the bitstream 21 and extract various encoded versions of the HOA coefficients 11 (e.g., a direction-based encoded version or a vector-based encoded version). The extraction unit 72 may determine, from the above description, syntax elements indicating whether the HOA coefficients 11 are encoded via various direction-based versions or vector-based versions. When direction-based encoding is performed, the extraction unit 72 may extract the direction-based version of the HOA coefficients 11 and syntax elements associated with the encoded version (which are represented as direction-based information 91 in the example of FIG. 4 ), thereby passing the direction-based information 91 to the direction-based reconstruction unit 90. The direction-based reconstruction unit 90 may represent a unit configured to reconstruct the HOA coefficients in the form of HOA coefficients 11′ based on the direction-based information 91. The arrangement of the bitstream and syntax elements within the bitstream is described below.

When the syntax element indicates that the HOA coefficients 11 are encoded using vector-based synthesis, the extraction unit 72 may extract the coded foreground V[k] vectors 57 (which may include the coded weights 57 and/or indices 63 or scalar-quantized V vectors), the encoded ambient HOA coefficients 59, and the corresponding audio objects 61 (which may also be referred to as encoded nFG signals 61). The audio objects 61 each correspond to one of the vectors 57. The extraction unit 72 may pass the coded foreground V[k] vectors 57 to the V-vector reconstruction unit 74 and provide the encoded ambient HOA coefficients 59 and the encoded nFG signal 61 to the psychoacoustic decoding unit 80.

The V-vector reconstruction unit 74 may represent a unit configured to reconstruct the V-vector from the encoded foreground V[k] vector 57. The V-vector reconstruction unit 74 may operate in an inverse manner to the quantization unit 52.

The psychoacoustic decoding unit 80 may operate in a reciprocal manner to the psychoacoustic audio decoder unit 40 shown in the example of FIG3 , so as to decode the encoded ambient HOA coefficients 59 and the encoded nFG signal 61 and thereby generate energy-compensated ambient HOA coefficients 47′ and an interpolated nFG signal 49′ (which may also be referred to as an interpolated nFG audio object 49′). The psychoacoustic decoding unit 80 may pass the energy-compensated ambient HOA coefficients 47′ to the re-correlation unit 81 and pass the nFG signal 49′ to the foreground formulation unit 78. The re-correlation unit 81 may then apply one or more re-correlation transforms to the energy-compensated ambient HOA coefficients 47′ to obtain one or more re-correlated HOA coefficients 47″ (or correlated HOA coefficients 47″), and may pass the correlated HOA coefficients 47″ to the HOA coefficient formulation unit 82 (optionally via the fade unit 770).

Similar to the description above, the re-correlation unit 81, relative to the decorrelation unit 40 ′ of the audio encoding device 20 , may implement the techniques of this disclosure to reduce the correlation between background channels of the energy-compensated ambient HOA coefficients 47 ′, thereby reducing or mitigating noise demasking. In instances where the re-correlation unit 81 applies a UHJ matrix (e.g., an inverse UHJ matrix) as the selected re-correlation transform, the re-correlation unit 81 may improve compression and conserve computational resources by reducing data processing operations. In some examples, the vector-based bitstream 21 may include one or more syntax elements indicating the application of a decorrelation transform during encoding. Including such syntax elements in the vector-based bitstream 21 may enable the re-correlation unit 81 to perform a mutually inverse decorrelation (e.g., correlation or re-correlation) transform on the energy-compensated HOA coefficients 47 ′. In some examples, a signal syntax element may indicate which decorrelation transform to apply, such as a UHJ matrix or a pattern matrix, thereby enabling the re-correlation unit 81 to select an appropriate re-correlation transform to apply to the energy-compensated HOA coefficients 47 ′.

In an instance where the vector-based reconstruction unit 92 outputs the HOA coefficients 11′ to a reproduction system comprising a stereo system, the re-correlation unit 81 may process the S signal and the D signal (e.g., the intrinsic left signal and the intrinsic right signal) to produce re-correlated HOA coefficients 47″. For example, because the S signal and the D signal represent the intrinsic left signal and the intrinsic right signal, the reproduction system may use the S signal and the D signal as two stereo output streams. In an instance where the reconstruction unit 92 outputs the HOA coefficients 11′ to a reproduction system comprising a mono audio system, the reproduction system may combine or mix the S signal and the D signal (as represented in the HOA coefficients 11′) to obtain a mono audio output for playback. In the instance of a mono audio system, the reproduction system may add the mixed mono audio output to one or more foreground channels (if any foreground channels are present) to produce an audio output.

In contrast to some existing UHJ-capable encoders, the signal is processed with a phase-amplitude matrix to recover a B-format-like signal set. In most cases, the signal will actually be B-format, but in the case of two-channel UHJ, there's insufficient information available to reconstruct a correct B-format signal, resulting in a signal with characteristics similar to a B-format signal. This information is then passed to the amplitude matrix that generates the speaker feeds via a set of Shelf filters. This improves the decoder's accuracy and performance in smaller listening environments (it can be omitted in larger applications). Ambisonics is designed to fit the requirements of real rooms (e.g., living rooms) and practical speaker placement: many such rooms are rectangular, so the base system is designed to decode four loudspeakers in a rectangle, with sideband lengths between 1:2 (width twice length) and 2:1 (length twice width), thus meeting the requirements of most such rooms. Layout controls are typically provided to allow the decoder to be configured for loudspeaker placement. Layout control is an aspect of ambisonic playback that differs from other surround sound systems: the decoder can be configured specifically for the size and layout of the speaker array. Layout control can take the form of a knob, a 2-way (1:2, 2:1), or a 3-way (1:2, 1:1, 2:1) switch. Four speakers are the minimum required for horizontal surround decoding, and while a four-speaker layout can be suitable for many listening environments, larger spaces may require more speakers to provide full surround positioning.

Examples of calculations that the re-correlation unit 81 may perform for applying a UHJ matrix (eg, an inverse UHJ matrix or an inverse phase-based transform) as the re-correlation transform are listed below:

UHJ decoding:

Conversion of left and right to S and D:

S＝left+right

D = left-right

W＝(0.982*S)+0.197.*imag(hilbert((0.828*D)+(0.768*T)));

X=(0.419*S)-imag(hilbert((0.828*D)+(0.768*T)));

Y＝(0.796*D)-0.676*T+imag(hilbert(0.187*S));

Z＝(1.023*Q);

In some example implementations of the above calculations, assumptions regarding the above calculations may include the following: the HOA background channel is first-order ambisonics, the FuMa is normalized, and the ambisonics channels are numbered in the order W(a00), X(a11), Y(a11-), Z(a10).

Examples of calculations that the re-correlation unit 81 may perform for applying the UHJ matrix (or the inverse phase-based transform) as the re-correlation transform are listed below:

UHJ decoding:

Conversion of left and right to S and D:

Conversion of left and right to S and D:

S = left + right;

D = left-right;

h1=imag(hilbert(1.014088753512236*D+T));

h2=imag(hilbert(0.229027290950227*S));

W＝0.982*S+0.160849826442762*h1;

X＝0.513168101113076*S-h1；

Y＝0.974896917627705*D-0.880208333333333*T+h2;

Z＝Q;

In some implementations of the above calculations, assumptions regarding the above calculations may include the following: the HOA background channel is 1st-order ambisonics, N3D (or "full 3D") normalized, and in the ambisonic channel numbering order W(a00), X(a11), Y(a11-), Z(a10). Although described herein with respect to N3D normalization, it should be understood that the example calculations may also be applied to HOA background channels that are SN3D normalized (or "Schmidt semi-normalized"). As described above with respect to FIG. 4 , N3D and SN3D normalizations may differ in the scaling factors used. An example representation of the scaling factors used in N3D normalization is described above with respect to FIG. 4 . An example representation of the weighting coefficients used in SN3D normalization is described above with respect to FIG.

In some examples, the energy-compensated HOA coefficients 47' may represent audio data that is only horizontally arranged, e.g., without any vertical channels. In these examples, the re-correlation unit 81 may not perform the above calculations on the Z signal, as the Z signal represents vertical audio data. Alternatively, in these examples, the re-correlation unit 81 may perform the above calculations only on the W, X, and Y signals, as the W, X, and Y signals represent horizontal data. In some examples where the energy-compensated HOA coefficients 47' represent audio data to be reproduced on a monophonic audio reproduction system, the re-correlation unit 81 may derive only the W signal from the above calculations. More specifically, because the resulting W signal represents monophonic audio data, the W signal may provide all the necessary data where the energy-compensated HOA coefficients 47' represent data to be reproduced in a monophonic audio format, or where the reproduction system comprises a monophonic audio system.

Similar to as described above with respect to the decorrelation unit 40′ of the audio encoding device 20, in an example, the recorrelation unit 81 may apply a UHJ matrix (or an inverse UHJ matrix or an inverse phase-based transform) in scenarios where the energy-compensated HOA coefficients 47′ include a smaller number of background channels, but may apply a pattern matrix or an inverse pattern matrix (e.g., as described in the MPEG-H standard) in scenarios where the energy-compensated HOA coefficients 47′ include a larger number of background channels.

It will be understood that the re-correlation unit 81 may apply the techniques described herein in scenarios where the energy-compensated HOA coefficients 47' include foreground channels, as well as in scenarios where the energy-compensated HOA coefficients 47' do not include any foreground channels. As an example, in scenarios where the energy-compensated HOA coefficients 47' include zero (0) foreground channels and eight (8) background channels (e.g., lower/smaller bit rate scenarios), the re-correlation unit 81 may apply the techniques and/or calculations described above.

Various components of the audio decoding device 24, such as the re-correlation unit 81, may be configured to determine which of the two processing methods to apply for decorrelation, such as a syntax element, such as a flag, UsePhaseShiftDecorr. In the example where the decorrelation unit 40' uses a spatial transform for decorrelation, the re-correlation unit 81 may determine that the UsePhaseShiftDecorr flag is set to a value of zero.

In the case where the re-correlation unit 81 determines that the UsePhaseShiftDecorr flag is set to a value of one, the re-correlation unit 81 may determine that re-correlation is to be performed using a phase-based transform. If the flag UsePhaseShiftDecorr has a value of 1, then the following process is applied to reconstruct the first four coefficient sequences of the ambient HOA component

where coefficient c is defined in Table 1 below and A ₊₉₀ (k) and B ₊₉₀ (k) are frames of +90 degree phase shifted signals A and B, defined as follows

A(k)＝c(0)·[c _I,AMB,1 (k)-c _I,AMB,2 (k)],

B(k)=c(1)·[c _I,AMB,1 (k)+c _I,AMB,2 (k)].

Table 2 below illustrates example coefficients that decorrelation unit 40' may use to implement the phase-based transform.

n c(n) 0 1.0140887535122356 1 0.22902729095022714 2 0.981999999999999998 3 0.16084982644276205 4 0.51316810111307576 5 0.97489691762770481 6 -0.880208333333333337

Table 2 Coefficients of phase-based transformation

In the above equations, the varying _CAMB,1 (k) variable denotes the HOA coefficients corresponding to the k-th frame of the spherical basis function with (order:sub-order) of (0:0), which may also be referred to as the 'W' channel or component. The varying _CAMB,2 (k) variable denotes the HOA coefficients corresponding to the k-th frame of the spherical basis function with (order:sub-order) of (1:-1), which may also be referred to as the 'Y' channel or component. The varying _CAMB,3 (k) variable denotes the HOA coefficients corresponding to the k-th frame of the spherical basis function with (order:sub-order) of (1:0), which may also be referred to as the 'Z' channel or component. The varying _CAMB,4 (k) variable denotes the HOA coefficients corresponding to the k-th frame of the spherical basis function with (order:sub-order) of (1:1), which may also be referred to as the 'X' channel or component. C _AMB,1 (k) to C _AMB,3 (k) may correspond to ambient HOA coefficients 47 ′.

The notation [C _I,AMB,1 (k) + C _I,AMB,2 (k)] above denotes a term that may alternatively be referred to as 'S', which is equivalent to the left channel plus the right channel. The C _I,AMB,1 (k) variable denotes the left channel produced as a result of UHJ encoding, while the C _I,AMB,2 (k) variable denotes the right channel produced as a result of UHJ encoding. The subscript 'I' notation denotes that the corresponding channel has been decorrelated from the other ambient channels (e.g., by applying a UHJ matrix or a phase-based transform). The notation [C _I,AMB,1 (k) - C _I,AMB,2 (k)] denotes a term that is referred to throughout this disclosure as 'D', which represents the left channel minus the right channel. _{The C I,AMB,3} (k) variable denotes a term that is referred to throughout this disclosure as the variable 'T'. _{The C I,AMB,4} (k) variable denotes a term that is referred to throughout this disclosure as the variable 'Q'.

The A ₊₉₀ (k) notation designates the positive 90 degree phase shift of c(0) times S (also designated by the variable 'h1' throughout this disclosure). The B ₊₉₀ (k) notation designates the positive 90 degree phase shift of c(1) times D (also designated by the variable 'h2' throughout this disclosure).

The spatial-temporal interpolation unit 76 may operate in a manner similar to that described above with respect to the spatial-temporal interpolation unit 50. The spatial-temporal interpolation unit 76 may receive the reduced foreground V[k] vector 55 _k and perform spatial-temporal interpolation on the foreground V[k] vector 55 _k and the reduced foreground V[k-1] vector 55 _k-1 to produce an interpolated foreground V[k] vector 55 _k ″. The spatial-temporal interpolation unit 76 forwards the interpolated foreground V[k] vector 55 _k ″ to the fade unit 770.

The extraction unit 72 may also output a signal 757 indicating when one of the ambient HOA coefficients is in transition to a fade unit 770, which may then determine which of the SHC _BG 47′ (where the SHC _BG 47′ may also be labeled “ambient HOA channel 47′” or “ambient HOA coefficient 47′”) and the elements of the interpolated foreground V[k] vector 55 _k ″ are to be faded in or out. In some examples, the fade unit 770 may operate in opposite manners for each of the elements of the ambient HOA coefficient 47′ and the interpolated foreground V[k] vector 55 _k ″. That is, the fade unit 770 may perform a fade-in or fade-out or both fade-in or fade-out on the corresponding ambient HOA coefficients in the ambient HOA coefficients 47 _′ , while performing a fade-in or fade-out or both fade-in and fade-out on the corresponding elements in the elements of the interpolated foreground V[k] vector 55 k ″. The fade unit 770 may output the adjusted ambient HOA coefficients 47″ to the HOA coefficient formulation unit 82 and the adjusted foreground V[k] vector 55 _k ″′ to the foreground formulation unit 78. In this regard, the fade unit 770 represents a unit configured to perform fade operations on various aspects of the HOA coefficients or their derivatives (e.g., in the form of the ambient HOA coefficients 47′ and the elements of the interpolated foreground V[k] vector 55 _k ″).

The foreground development unit 78 may represent a unit configured to perform matrix multiplication on the adjusted foreground V[k] vector _55k ″′ and the interpolated nFG signal 49′ to produce the foreground HOA coefficients 65. In this regard, the foreground development unit 78 may combine the audio object 49′ (which is another way to represent the interpolated nFG signal 49′) with the vector _55k ″′ to reconstruct the foreground (or in other words, dominant) aspects of the HOA coefficients 11′. The foreground development unit 78 may perform matrix multiplication of the interpolated nFG signal 49′ and the adjusted foreground V[k] vector _55k ″′.

The HOA coefficient formulation unit 82 may represent a unit configured to combine the foreground HOA coefficients 65 with the adjusted ambient HOA coefficients 47″ in order to obtain the HOA coefficients 11′. The apostrophe notation reflects that the HOA coefficients 11′ may be similar to, but not identical to, the HOA coefficients 11. Differences between the HOA coefficients 11 and 11′ may be due to losses due to transmission over a lossy transmission medium, quantization, or other lossy operations.

UHJ is a matrix transformation method that has been used to create a two-channel stereo stream from first-order ambisonic content. UHJ has been used in the past to transmit stereo or horizontal surround-only content via FM transmitters. However, it should be understood that UHJ is not limited to use in FM transmitters. In the MPEG-H HOA coding scheme, the HOA background channel can be preprocessed with a pattern matrix to convert the HOA background channel into orthogonal points in the spatial domain. The transformed channel is then perceptually coded using USAC or AAC.

The techniques of this disclosure generally involve using the UHJ transform (or phase-based transform) in applications for decoding HOA background channels rather than using this pattern matrix. Both approaches ((1) transform into the spatial domain via the pattern matrix, (2) the UHJ transform) generally involve reducing the correlation between the HOA background channels, which can cause the (potentially undesirable) effect of noise demasking within the decoded sound field.

Thus, in an example, the audio decoding device 24 may represent a device configured to: obtain a decorrelated representation of ambisonic reverberation coefficients having at least a left signal and a right signal, the ambisonic reverberation coefficients having been extracted from a plurality of higher-order ambisonic reverberation coefficients and representing background components of a sound field described by the plurality of higher-order ambisonic reverberation coefficients, wherein at least one of the plurality of higher-order ambisonic reverberation coefficients is associated with a spherical basis function having an order greater than one; and generate a speaker feed based on the decorrelated representation of the ambisonic reverberation coefficients. In some examples, the device is further configured to apply a recorrelation transform to the decorrelated representation of the ambisonic reverberation coefficients to obtain a plurality of correlated ambisonic reverberation coefficients.

In some examples, to apply the recorrelation transform, the device is configured to apply an inverse UHJ matrix (or a phase-based transform) to the ambisonics coefficients. According to some examples, the inverse UHJ matrix (or the inverse phase-based transform) has been normalized according to N3D (full three-dimensional) normalization. According to some examples, the inverse UHJ matrix (or the inverse phase-based transform) has been normalized according to SN3D normalization (Schmidt semi-normalization).

According to some examples, the ambisonic reverberation coefficients are associated with spherical basis functions of order zero or order one, and to apply an inverse UHJ matrix (or an inverse phase-based transform), the device is configured to perform a scalar multiplication of the UHJ matrix on a decorrelated representation of the ambisonic reverberation coefficients. In some examples, to apply a recorrelation transform, the device is configured to apply an inverse mode matrix to the decorrelated representation of the ambisonic reverberation coefficients. In some examples, to generate the speaker feeds, the device is configured to generate a left speaker feed based on the left signal and a right speaker feed based on the right signal, the left speaker feed and the speaker feed being output by a stereo reproduction system.

In some examples, to generate the speaker feeds, the device is configured to use the left signal as the left speaker feed and the right signal as the right speaker feed without applying a recorrelation transform to the right and left signals. According to some examples, to generate the speaker feeds, the device is configured to mix the left and right signals for output by a mono audio system. According to some examples, to generate the speaker feeds, the device is configured to combine correlated ambisonics coefficients with one or more foreground channels.

According to some examples, the device is further configured to determine that no foreground channel is available for combination with the correlated ambisonic reverberation coefficients. In some examples, the device is further configured to determine that the sound field is to be output via a monophonic audio reproduction system, and to decode at least a subset of decorrelated higher-order ambisonic reverberation coefficients comprising data for output by the monophonic audio reproduction system. In some examples, the device is further configured to obtain an indication that the decorrelated representation of the ambisonic reverberation coefficients is decorrelated by a decorrelation transform. According to some examples, the device further includes a loudspeaker array configured to output speaker feeds generated based on the decorrelated representation of the ambisonic reverberation coefficients.

FIG5 is a flowchart illustrating exemplary operation of an audio encoding device (e.g., the audio encoding device 20 shown in the example of FIG3 ) in performing various aspects of the vector-based synthesis techniques described in this disclosure. Initially, the audio encoding device 20 receives the HOA coefficients 11 ( 106 ). The audio encoding device 20 may invoke the LIT unit 30 , which may apply the LIT to the HOA coefficients to output transformed HOA coefficients (e.g., in the case of SVD, the transformed HOA coefficients may include the US[k] vector 33 and the V[k] vector 35 ) ( 107 ).

The audio encoding device 20 may then call the parameter calculation unit 32 to perform the above-described analysis to identify various parameters for any combination of the US[k] vector 33, the US[k-1] vector 33, the V[k] and/or the V[k-1] vector 35 in the manner described above. That is, the parameter calculation unit 32 may determine at least one parameter based on the analysis of the transformed HOA coefficients 33/35 (108).

The audio encoding device 20 may then call a reordering unit 34 that reorders the transformed HOA coefficients (again, in the context of SVD, which may refer to the US[k] vector 33 and the V[k] vector 35) based on the parameters to produce reordered transformed HOA coefficients 33'/35' (or in other words, the US[k] vector 33' and the V[k] vector 35'), as described above (109). The audio encoding device 20 may also call a sound field analysis unit 44 during any of the foregoing operations or subsequent operations. As described above, the sound field analysis unit 44 may perform sound field analysis on the HOA coefficients 11 and/or the transformed HOA coefficients 33/35 to determine the total number of foreground channels (nFG) 45, the order of the background sound field ( _NBG ), and the number (nBGa) and index (i) of additional BG HOA channels to be sent (which may be collectively labeled as background channel information 43 in the example of FIG. 3) (110).

The audio encoding device 20 may also invoke a background selection unit 48. The background selection unit 48 may determine background or ambient HOA coefficients 47 based on the background channel information 43 (112). The audio encoding device 20 may further invoke a foreground selection unit 36, which may select a reordered US[k] vector 33' and a reordered V[k] vector 35' representing the foreground or distinct components of the soundfield based on nFG 45 (which may represent one or more indices identifying the foreground vectors) (113).

The audio encoding device 20 may invoke the energy compensation unit 38. The energy compensation unit 38 may perform energy compensation on the ambient HOA coefficients 47 to compensate for the energy loss due to the removal of each of the HOA coefficients by the background selection unit 48 (114), and thereby generate energy-compensated ambient HOA coefficients 47'.

The audio encoding device 20 may also call the space-time interpolation unit 50. The space-time interpolation unit 50 may perform space-time interpolation on the reordered transformed HOA coefficients 33'/35' to obtain an interpolated foreground signal 49' (which may also be referred to as "interpolated nFG signal 49'") and remaining foreground directional information 53 (which may also be referred to as "V[k] vector 53") (116). The audio encoding device 20 may then call the coefficient reduction unit 46. The coefficient reduction unit 46 may perform coefficient reduction on the remaining foreground V[k] vector 53 based on the background channel information 43 to obtain reduced foreground directional information 55 (which may also be referred to as reduced foreground V[k] vector 55) (118).

The audio encoding device 20 may then call the quantization unit 52 to compress the reduced foreground V[k] vector 55 in the manner described above and produce a decoded foreground V[k] vector 57 (120). The audio encoding device 20 may also call the decorrelation unit 40' to apply phase-shift decorrelation to reduce or eliminate correlation between the background signal of the HOA coefficients 47', thereby forming one or more decorrelated HOA coefficients 47" (121).

The audio encoding device 20 may also call the psychoacoustic audio decoder unit 40. The psychoacoustic audio decoder unit 40 may psychoacoustically decode each vector of the energy-compensated ambient HOA coefficients 47' and the interpolated nFG signal 49' to generate encoded ambient HOA coefficients 59 and encoded nFG signal 61 (122). The audio encoding device may then call the bitstream generation unit 42. The bitstream generation unit 42 may generate the bitstream 21 based on the coded foreground directional information 57, the coded ambient HOA coefficients 59, the coded nFG signal 61, and the background channel information 43 (124).

FIG6A is a flowchart illustrating exemplary operation of an audio decoding device, such as the audio decoding device 24 shown in the example of FIG4 , in performing various aspects of the techniques described in this disclosure. Initially, the audio decoding device 24 may receive the bitstream 21 ( 130 ). Upon receiving the bitstream, the audio decoding device 24 may invoke the extraction unit 72 . Assuming for purposes of discussion that the bitstream 21 indicates that vector-based reconstruction is to be performed, the extraction unit 72 may parse the bitstream to retrieve the information mentioned above, thereby passing the information to the vector-based reconstruction unit 92 .

In other words, the extraction unit 72 can extract the decoded foreground directional information 57 (which, again, may also be referred to as the decoded foreground V[k] vector 57), the decoded ambient HOA coefficients 59 and the decoded foreground signal (which may also be referred to as the decoded foreground nFG signal 59 or the decoded foreground audio object 59) from the bitstream 21 in the manner described above (132).

The audio decoding device 24 may further invoke the dequantization unit 74. The dequantization unit 74 may entropy decode and dequantize the coded foreground directional information 57 to obtain reduced foreground directional information _55k (136). The audio decoding device 24 may invoke the re-correlation unit 81. The re-correlation unit 81 may apply one or more re-correlation transforms to the energy-compensated ambient HOA coefficients 47′ to obtain one or more re-correlated HOA coefficients 47″ (or correlated HOA coefficients 47″), and may pass the correlated HOA coefficients 47″ to the HOA coefficient formulation unit 82 (optionally, through the fade unit 770) (137). The audio decoding device 24 may also call the psychoacoustic decoding unit 80. The psychoacoustic audio decoding unit 80 may decode the encoded ambient HOA coefficients 59 and the encoded foreground signal 61 to obtain energy-compensated ambient HOA coefficients 47′ and an interpolated foreground signal 49′ (138). The psychoacoustic decoding unit 80 may pass the energy-compensated ambient HOA coefficients 47′ to the fade unit 770 and pass the nFG signal 49′ to the foreground formulation unit 78.

The audio decoding device 24 may next invoke the spatio-temporal interpolation unit 76. The spatio-temporal interpolation unit 76 may receive the reordered foreground directional information 55 _k ′ and perform spatio-temporal interpolation on the reduced foreground directional information 55 _k /55 _k-1 to generate interpolated foreground directional information 55 _k ″ (140). The spatio-temporal interpolation unit 76 may forward the interpolated foreground V[k] vector 55 _k ″ to the fade unit 770.

The audio decoding device 24 may call a fade unit 770. The fade unit 770 may receive or otherwise obtain (e.g., from the extraction unit 72) a syntax element (e.g., an AmbCoeffTransition syntax element) that indicates when the energy-compensated ambient HOA coefficients 47' are in transition. The fade unit 770 may fade the energy-compensated ambient HOA coefficients 47' in or out based on the transition syntax element and the maintained transition state information, thereby outputting the adjusted ambient HOA coefficients 47" to the HOA coefficient formulation unit 82. The fade unit 770 may also fade out or in corresponding one or more elements of the interpolated foreground V[k] vector _55k " based on the syntax element and the maintained transition state information, thereby outputting the adjusted foreground V[k] vector _55k '" to the foreground formulation unit 78 (142).

The audio decoding device 24 may call the foreground formulation unit 78. The foreground formulation unit 78 may perform a matrix multiplication of the nFG signal 49′ and the adjusted foreground direction information 55 _k ″′ to obtain the foreground HOA coefficients 65 (144). The audio decoding device 24 may also call the HOA coefficient formulation unit 82. The HOA coefficient formulation unit 82 may add the foreground HOA coefficients 65 to the adjusted ambient HOA coefficients 47″ to obtain the HOA coefficients 11 ′ (146).

6B is a flowchart illustrating exemplary operation of an audio encoding device and an audio decoding device performing the decoding techniques described in the present disclosure. FIG6B is a flowchart illustrating an example encoding and decoding process 160 according to one or more aspects of the present disclosure. Although process 160 can be performed by a variety of devices, for ease of discussion, process 160 is described herein with respect to the audio encoding device 20 and audio decoding device 24 described above. The encoding section and the decoding section of process 160 are demarcated using the dashed line in FIG6B. Process 160 may begin with one or more components of the audio encoding device 20 (e.g., foreground selection unit 36 and background selection unit 48) generating a foreground channel 164 and a first-order HOA background channel 166 from an HOA input using HOA spatial coding (162). Subsequently, decorrelation unit 40′ may apply a decorrelation transform (e.g., in the form of a phase-based decorrelation transform or matrix) to the energy-compensated ambient HOA coefficients 47′. More specifically, the audio encoding device 20 may apply a UHJ matrix or a phase-based decorrelation transform (eg, by scalar multiplication) to the energy-compensated ambient HOA coefficients 47 ′ ( 168 ).

In some examples, if the decorrelation unit 40' determines that the HOA background channel includes a smaller number of channels (e.g., four), the decorrelation unit 40' may apply a UHJ matrix (or a phase-based transform). Conversely, in these examples, if the decorrelation unit 40' determines that the HOA background channel includes a larger number of channels (e.g., nine), the audio encoding device 20 may select a decorrelation transform other than the UHJ matrix (e.g., a pattern matrix described in the MPEG-H standard) and apply the decorrelation transform to the HOA background channel. By applying the decorrelation transform (e.g., the UHJ matrix) to the HOA background channel, the audio encoding device 20 may obtain a decorrelated HOA background channel.

As shown in FIG6B , the audio encoding device 20 (e.g., by invoking the psychoacoustic audio decoder unit 40) may apply temporal encoding (e.g., by applying AAC and/or USAC) to the decorrelated HOA background signal (170) as well as to any foreground channels (166). It should be appreciated that in some scenarios, the psychoacoustic audio decoder unit 40 may determine that the number of foreground channels may be zero (i.e., in these scenarios, the psychoacoustic audio decoder unit 40 may not obtain any foreground channels from the HOA input). Because AAC and/or USAC may not be optimized for or otherwise well-suited to stereo audio data, the decorrelation unit 40′ may apply a decorrelation matrix to reduce or eliminate correlation between the HOA background channels. The reduced correlation shown in the decorrelated HOA background channels provides a potential advantage in mitigating or eliminating noise demasking during the AAC/USAC temporal encoding stage, since AAC and USAC may not be optimized for stereo audio data.

Then, the audio decoding device 24 may perform temporal decoding of the encoded bitstream output by the audio encoding device 20. In an example of process 160, one or more components of the audio decoding device 24 (e.g., the psychoacoustic decoding unit 80) may perform temporal decoding for the foreground channel (if any foreground channel is included in the bitstream) (172) and the background channel (174), respectively. In addition, the recordination unit 81 may apply a recordination transform to the temporally decoded HOA background channel. As an example, the recordination unit 81 may apply a decorrelation transform to the decorrelation unit 40' in a reciprocal manner. For example, as described in the specific example of process 160, the recordination unit 81 may apply a UHJ matrix or a phase-based transform to the temporally decoded HOA background signal (176).

In some examples, if the re-correlation unit 81 determines that the time-decoded HOA background signal includes a smaller number of channels (e.g., four), the re-correlation unit 81 may apply a UHJ matrix or a phase-based transform. Conversely, in these examples, if the re-correlation unit 81 determines that the time-decoded HOA background channel includes a larger number of channels (e.g., nine), the re-correlation unit 81 may select a decorrelation transform other than the UHJ matrix (e.g., a pattern matrix described in the MPEG-H standard) and apply the decorrelation transform to the HOA background channel.

Additionally, HOA coefficient formulation unit 82 may perform HOA spatial decoding of the associated HOA background channels and any available decoded foreground channels (178). HOA coefficient formulation unit 82 may then reproduce the decoded audio signal (180) to one or more output devices, such as speakers and/or headphones (including but not limited to output devices with stereo or surround sound capabilities).

The aforementioned techniques can be performed for any number of different contexts and audio ecosystems. Several example contexts are described below, but the techniques should not be limited to these example contexts. An example audio ecosystem may include audio content, a movie studio, a music studio, a game audio studio, channel-based audio content, a decoding engine, a game audio soundtrack (stem), a game audio decoding/rendering engine, and a delivery system.

Movie studios, music studios, and game audio studios can receive audio content. In some instances, the audio content can represent the output of acquired content. A movie studio can output channel-based audio content (e.g., in 2.0, 5.1, and 7.1), for example, by using a digital audio workstation (DAW). A music studio can output channel-based audio content (e.g., in 2.0 and 5.1), for example, by using a DAW. In either case, a decoding engine can receive and encode channel-based audio content based on one or more codecs (e.g., AAC, AC3, Dolby True HD, Dolby Digital Plus, and DTS Master Audio) for output by a delivery system. A game audio studio can output one or more game audio soundtracks, for example, by using a DAW. A game audio decoding/reproduction engine can decode the audio soundtracks and/or reproduce the audio soundtracks into channel-based audio content for output by a delivery system. Another example context in which the techniques may be performed includes an audio ecosystem that may include broadcast recorded audio objects, professional audio systems, capture on consumer devices, HOA audio formats, on-device reproduction, consumer audio, TV, and accessories, and automotive audio systems.

Broadcast recorded audio objects, professional audio systems, and capture on consumer devices can all be encoded using the HOA audio format for their output. In this way, the HOA audio format can be used to encode audio content into a single representation that can be played back using on-device reproduction, consumer audio, TV, accessories, and car audio systems. In other words, a single representation of the audio content can be played back on a universal audio playback system (i.e., as opposed to requiring a specific configuration such as 5.1, 7.1, etc.) (e.g., audio playback system 16).

Other examples of contexts in which the techniques may be implemented include audio ecosystems that may include acquisition elements and playback elements. The acquisition elements may include wired and/or wireless acquisition devices (e.g., intrinsic microphones), on-device surround sound capture, and mobile devices (e.g., smartphones and tablets). In some examples, the wired and/or wireless acquisition devices may be coupled to the mobile device via a wired and/or wireless communication channel.

According to one or more techniques of this disclosure, a mobile device may be used to acquire a sound field. For example, the mobile device may acquire the sound field via wired and/or wireless acquisition device and/or on-device surround sound capture (e.g., multiple microphones integrated into the mobile device). The mobile device may then decode the acquired sound field into HOA coefficients for playback by one or more of the playback elements. For example, a user of the mobile device may record a live event (e.g., a meeting, conference, game, concert, etc.) (capturing the sound field of the live event) and decode the recording into HOA coefficients.

The mobile device may also use one or more of the playback elements to play back the HOA decoded sound field. For example, the mobile device may decode the HOA decoded sound field and output a signal to the one or more of the playback elements that causes the one or more of the playback elements to reproduce the sound field. As an example, the mobile device may output the signal to one or more speakers (e.g., a speaker array, a sound bar, etc.) using a wireless and/or wireless communication channel. As another example, the mobile device may use a docking solution to output the signal to one or more docking stations and/or one or more docked speakers (e.g., a sound system in a smart car and/or home). As another example, the mobile device may use headphone reproduction to output the signal to a set of headphones (e.g., to create realistic binaural sound).

In some examples, a particular mobile device may acquire a 3D sound field and play back the same 3D sound field at a later time. In some examples, a mobile device may acquire a 3D sound field, encode the 3D sound field into an HOA, and transmit the encoded 3D sound field to one or more other devices (e.g., other mobile devices and/or other non-mobile devices) for playback.

Yet another context in which the techniques may be implemented includes an audio ecosystem, which may include audio content, a game studio, decoded audio content, a rendering engine, and a delivery system. In some examples, a game studio may include one or more DAWs that may support editing of HOA signals. For example, the one or more DAWs may include HOA plug-ins and/or tools that may be configured to operate (e.g., work) with one or more game audio systems. In some examples, a game studio may output a new soundtrack format that supports HOA. In any case, the game studio may output the decoded audio content to a rendering engine that may reproduce the sound field for playback by a delivery system.

The techniques may also be performed for an exemplary audio acquisition device. For example, the techniques may be performed for an eigenmicrophone, which may include multiple microphones collectively configured to record a 3D sound field. In some examples, the multiple microphones of the eigenmicrophone may be located on the surface of a substantially spherical sphere having a radius of approximately 4 cm. In some examples, the audio encoding device 20 may be integrated into the eigenmicrophone so as to output the bitstream 21 directly from the microphone.

Another exemplary audio acquisition context may include a production truck that may be configured to receive signals from one or more microphones (eg, one or more eigenmicrophones).The production truck may also include an audio encoder, such as audio encoder 20 of FIG.

In some instances, the mobile device may also include multiple microphones that are collectively configured to record a 3D sound field. In other words, the multiple microphones may have X, Y, Z diversity. In some examples, the mobile device may include a microphone that can be rotated to provide X, Y, Z diversity relative to one or more other microphones of the mobile device. The mobile device may also include an audio encoder, such as audio encoder 20 of FIG. 3 .

The ruggedized video capture device can be further configured to record a 3D sound field. In some examples, the ruggedized video capture device can be attached to a helmet of a user participating in an activity. For example, the ruggedized video capture device can be attached to a user's helmet while the user is whitewater rafting. In this way, the ruggedized video capture device can capture a 3D sound field representing the action around the user (e.g., water crashing behind the user, another whitewater rafter speaking in front of the user, etc.).

The techniques can also be performed for accessory-enhanced mobile devices that can be configured to record 3D sound fields. In some examples, the mobile device may be similar to the mobile device discussed above, with one or more accessories added. For example, an intrinsic microphone can be attached to the mobile device mentioned above to form an accessory-enhanced mobile device. In this way, the accessory-enhanced mobile device can capture a higher-quality version of the 3D sound field than using only the sound capture components integrated into the accessory-enhanced mobile device.

An example audio playback device that can perform various aspects of the techniques described in this disclosure is discussed further below. In accordance with one or more techniques of this disclosure, speakers and/or soundbars can be arranged in any arbitrary configuration when playing back a 3D soundfield. Furthermore, in some examples, a headphone playback device can be coupled to decoder 24 via a wired or wireless connection. In accordance with one or more techniques of this disclosure, a single, universal representation of a soundfield can be used to reproduce the soundfield on any combination of speakers, soundbars, and headphone playback devices.

A number of different example audio playback environments may also be suitable for performing various aspects of the techniques described in this disclosure. For example, the following environments may be suitable environments for performing various aspects of the techniques described in this disclosure: a 5.1 speaker playback environment, a 2.0 (e.g., stereo) speaker playback environment, a 9.1 speaker playback environment with full-height front speakers, a 22.2 speaker playback environment, a 16.0 speaker playback environment, an automotive speaker playback environment, and a mobile device with an earbud playback environment.

According to one or more techniques of the present invention, a single universal representation of a sound field can be utilized to reproduce the sound field on any of the aforementioned playback environments. In addition, the techniques of the present invention enable a reproducer to reproduce the sound field from the universal representation for playback on playback environments other than the environments described above. For example, if design considerations prohibit the proper placement of speakers according to a 7.1 speaker playback environment (e.g., if it is impossible to place a right surround speaker), then the techniques of the present invention enable the reproducer to compensate with the other six speakers, so that playback can be achieved on a 6.1 speaker playback environment.

In addition, a user can watch a sports game while wearing a headset. According to one or more techniques of the present invention, a 3D sound field of a sports game can be obtained (for example, one or more eigenmicrophones can be placed in and/or around a baseball field), HOA coefficients corresponding to the 3D sound field can be obtained and transmitted to a decoder, the decoder can reconstruct the 3D sound field based on the HOA coefficients and output the reconstructed 3D sound field to a reproducer, and the reproducer can obtain an indication of the type of playback environment (for example, a headset) and reproduce the reconstructed 3D sound field as a signal that causes the headset to output a representation of the 3D sound field of the sports game.

In each of the various examples described above, it should be understood that the audio encoding device 20 may perform a method, or otherwise include means for performing each step of the method that the audio encoding device 20 is configured to perform. In some examples, these means may include one or more processors. In some examples, the one or more processors may represent specialized processors configured with the aid of instructions stored on a non-transitory computer-readable storage medium. In other words, various aspects of the techniques in each of the set of encoding examples may provide a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause the one or more processors to perform the method that the audio encoding device 20 has been configured to perform.

In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media such as data storage media. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and/or data structures for implementing the techniques described in this disclosure. A computer program product may include computer-readable media.

Likewise, in each of the various examples described above, it should be understood that the audio decoding device 24 may perform a method or otherwise include means for performing each step of the method that the audio decoding device 24 is configured to perform. In some examples, the means may include one or more processors. In some examples, the one or more processors may represent specialized processors configured with the aid of instructions stored on a non-transitory computer-readable storage medium. In other words, various aspects of the technology in each of the set of encoding examples may provide a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause the one or more processors to perform the method that the audio decoding device 24 has been configured to perform.

By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are actually directed to non-transitory tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks generally reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above are also intended to be included within the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor," as used herein, may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated into a combined codec. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure can be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., a chipset). Various components, modules, or units are described herein to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by distinct hardware units. Indeed, as described above, the various units may be combined in a codec hardware unit in conjunction with appropriate software and/or firmware, or provided by a collection of interoperating hardware units, including one or more processors as described above.

Having described various aspects of the technology, it is intended that these and other aspects of the technology be within the scope of the following claims.

Claims (28)

1. A method for processing audio data, comprising: A decorrelation representation of an ambient stereo reverberation coefficient having at least one left signal and one right signal is obtained, the ambient stereo reverberation coefficient having been extracted from a plurality of higher-order stereo reverberation coefficients and representing the background component of the sound field described by the plurality of higher-order stereo reverberation coefficients, the decorrelation representation of the ambient stereo reverberation coefficient having been decorrelated using a phase-based transform, wherein at least one of the plurality of higher-order stereo reverberation coefficients is associated with a spherical basis function having an order of one or zero; The recorrelation transformation is applied to the decorrelation representation of the environmental stereo reverberation coefficients to obtain multiple correlated environmental stereo reverberation coefficients; and The loudspeaker feed is generated based on the plurality of associated ambient stereo reverberation coefficients obtained from the decorrelation representation of the ambient stereo reverberation coefficients. 2. The method of claim 1, wherein applying the recorrelation transform comprises applying a phase-based inverse transform to the environmental stereo reverberation coefficient. 3. The method according to claim 2, wherein the phase-based inverse transform has been normalized according to full three-dimensional normalization. 4. The method of claim 2, wherein the phase-based inverse transform has been normalized according to Schmitt semi-normalization. 5. The method of claim 2, wherein the ambient stereo reverberation coefficients are associated with spherical basis functions having order zero or order one, and wherein applying the phase-based inverse transform comprises performing a scalar multiplication of the phase-based transform on the decorrelation representation of the ambient stereo reverberation coefficients. 6. The method of claim 1, further comprising obtaining the decorrelation representation of the environmental three-dimensional reverberation coefficient as an indication of decorrelation through a decorrelation transformation. 7. The method of claim 1, further comprising obtaining one or more spatial components defining the spatial characteristics of the foreground component of the sound field, said spatial components being defined in the spherical harmonic domain and generated by performing decomposition on said plurality of higher-order stereo reverberation coefficients. Generating the loudspeaker feed involves combining the associated ambient stereo reverberation coefficient with one or more foreground channels obtained based on the one or more spatial components. 8. A method for compressing audio data, comprising: A phase-based decorrelation transform is applied to the ambient stereo reverberation coefficients to obtain a decorrelation representation of the ambient stereo reverberation coefficients, which have been extracted from a plurality of higher-order stereo reverberation coefficients and represent the background component of the sound field described by the plurality of higher-order stereo reverberation coefficients, wherein at least one of the plurality of higher-order stereo reverberation coefficients is associated with a spherical basis function having an order of one or zero. 9. The method of claim 8, further comprising normalizing the phase-based transform according to full three-dimensional normalization. 10. The method of claim 8, further comprising normalizing the phase-based transform according to Schmitt semi-normalization. 11. The method of claim 8, wherein the ambient stereo reverberation coefficients are associated with spherical basis functions having order zero or order one, and wherein applying the phase-based transformation to the ambient stereo reverberation coefficients comprises performing a scalar multiplication of the phase-based transformation on at least a subset of the ambient stereo reverberation coefficients. 12. The method of claim 8, further comprising signaling an indication that the decorrelation transformation has been applied to the ambient stereo reverberation coefficient. 13. An apparatus for processing audio data, the apparatus comprising: A memory configured to store at least a portion of the audio data to be processed; and One or more processors, configured to: A decorrelation representation of an ambient stereo reverberation coefficient having at least one left signal and one right signal is obtained, the ambient stereo reverberation coefficient having been extracted from a plurality of higher-order stereo reverberation coefficients and representing the background component of the sound field described by the plurality of higher-order stereo reverberation coefficients, the decorrelation representation of the ambient stereo reverberation coefficient having been decorrelated using a phase-based transform, wherein at least one of the plurality of higher-order stereo reverberation coefficients is associated with a spherical basis function having an order of one or zero; The recorrelation transformation is applied to the decorrelation representation of the environmental stereo reverberation coefficients to obtain multiple correlated environmental stereo reverberation coefficients; and The loudspeaker feed is generated based on the decorrelation representation of the ambient stereo reverberation coefficient. 14. The apparatus of claim 13, wherein, in order to generate the loudspeaker feed, the one or more processors are configured to generate a left loudspeaker feed based on the left signal and a right loudspeaker feed based on the right signal, the left loudspeaker feed and the loudspeaker feed being output by a stereo reproduction system. 15. The apparatus of claim 13, wherein, in order to generate the loudspeaker feed, the one or more processors are configured to use the left signal as the left loudspeaker feed and the right signal as the right loudspeaker feed without applying the recorrelation transform to the right signal and the left signal. 16. The apparatus of claim 13, wherein, in order to generate the loudspeaker feed, the one or more processors are configured to mix the left signal and the right signal for output by a mono audio system. 17. The apparatus of claim 13, wherein, in order to generate the loudspeaker feed, the one or more processors are configured to combine associated ambient stereo reverberation coefficients with one or more foreground channels. 18. The apparatus of claim 13, wherein the one or more processors are further configured to determine that no foreground channel is available for combination with the associated ambient stereo reverberation coefficients. 19. The apparatus of claim 13, wherein the one or more processors are further configured to: The sound field will be output via a mono audio reproduction system; and Decoding is performed on at least one subset of the decorrelated ambient stereo reverberation coefficients containing data output from the mono audio reproduction system. 20. The apparatus of claim 13, wherein the one or more processors are further configured to obtain the decorrelationd representation of the ambient stereo reverberation coefficients as an indication of decorrelation via a decorrelation transformation. 21. The apparatus of claim 13, further comprising a loudspeaker configured to output the loudspeaker feed generated based on the decorrelation representation of the ambient stereo reverberation coefficient. 22. An apparatus for compressing audio data, the apparatus comprising: A memory configured to store at least a portion of the audio data to be compressed; and One or more processors, configured to: A phase-based decorrelation transform is applied to the ambient stereo reverberation coefficients to obtain a decorrelation representation of the ambient stereo reverberation coefficients, which have been extracted from a plurality of higher-order stereo reverberation coefficients and represent the background component of the sound field described by the plurality of higher-order stereo reverberation coefficients, wherein at least one of the plurality of higher-order stereo reverberation coefficients is associated with a spherical basis function having an order of one or zero. 23. The apparatus of claim 22, wherein the one or more processors are further configured to signal the decorrelated ambient stereo reverberation coefficients and one or more foreground channels. 24. The apparatus of claim 22, wherein, in order to signal the decorrelated ambient stereo reverberation coefficients and one or more foreground channels, the one or more processors are configured to signal the decorrelated ambient stereo reverberation coefficients and one or more foreground channels in response to determining that a target bit rate meets or exceeds a predetermined threshold. 25. The apparatus of claim 22, wherein the one or more processors are further configured to signal the decorrelated ambient stereo reverberation coefficients without signaling any foreground channel. 26. The apparatus of claim 25, wherein, in order to signal the decorrelated ambient stereo reverberation coefficients without signaling any foreground channel, the one or more processors are configured to signal the decorrelated ambient stereo reverberation coefficients without signaling any foreground channel in response to determining that the target bit rate is below a predetermined threshold. 27. The apparatus of claim 26, wherein the one or more processors are further configured to signal an indication that the decorrelation transformation has been applied to the ambient stereo reverberation coefficients. 28. The apparatus of claim 22, further comprising a microphone configured to capture the audio data to be compressed.