HK1224073B - Indicating frame parameter reusability for coding vectors - Google Patents

Description

Indicates the reusability of frame parameters for decoding vectors

This application claims the benefit of the following U.S. provisional applications:

U.S. Provisional Application No. 61/933,706, filed January 30, 2014, entitled “COMPRESSION OF DECOMPOSED REPRESENTATIONS OF A SOUND FIELD”;

U.S. Provisional Application No. 61/933,714, filed January 30, 2014, entitled “COMPRESSION OF DECOMPOSED REPRESENTATIONS OF A SOUND FIELD”;

U.S. Provisional Application No. 61/933,731, filed January 30, 2014, entitled “INDICATING FRAME PARAMETER REUSABILITY FOR DECODING SPATIAL VECTORS”;

U.S. Provisional Application No. 61/949,591, filed on March 7, 2014, entitled “IMMEDIATE PLAY-OUTFRAME FOR SPHERICAL HARMONIC COEFFICIENTS”;

U.S. Provisional Application No. 61/949,583, filed on March 7, 2014, entitled “FADE-IN/FADE-OUT OF DECOMPOSED REPRESENTATIONS OF A SOUND FIELD”;

U.S. Provisional Application No. 61/994,794, filed May 16, 2014, entitled “CODING V-VECTORS OF A DECOMPOSED HIGHER ORDER AMBISONICS (HOA) AUDIO SIGNAL”;

U.S. Provisional Application No. 62/004,147, filed May 28, 2014, entitled “INDICATING FRAME PARAMETER REUSABILITY FOR DECODING SPATIAL VECTORS”;

U.S. Provisional Application No. 62/004,067, filed May 28, 2014, entitled “IMMEDIATE PLAY-OUT FRAME FOR SPHERICAL HARMONIC COEFFICIENTS AND FADE-IN/FADE-OUT OF DECOMPOSED REPRESENTATIONS OF A SOUND FIELD”;

U.S. Provisional Application No. 62/004,128, filed May 28, 2014, entitled “CODING V-VECTORS OF A DECOMPOSED HIGHER ORDER AMBISONICS (HOA) AUDIO SIGNAL”;

U.S. Provisional Application No. 62/019,663, filed July 1, 2014, entitled “CODING V-VECTORS OF A DECOMPOSED HIGHER ORDER AMBISONICS (HOA) AUDIO SIGNAL”;

U.S. Provisional Application No. 62/027,702, filed July 22, 2014, entitled “CODING V-VECTORS OF A DECOMPOSED HIGHER ORDER AMBISONICS (HOA) AUDIO SIGNAL”;

U.S. Provisional Application No. 62/028,282, filed July 23, 2014, entitled “CODING V-VECTORS OF A DECOMPOSED HIGHER ORDER AMBISONICS (HOA) AUDIO SIGNAL”;

U.S. Provisional Application No. 62/029,173, filed July 25, 2014, entitled “IMMEDIATE PLAY-OUT FRAME FOR SPHERICAL HARMONIC COEFFICIENTS AND FADE-IN/FADE-OUT OF DECOMPOSED REPRESENTATIONS OF A SOUND FIELD”;

U.S. Provisional Application No. 62/032,440, filed August 1, 2014, entitled “CODING V-VECTORS OF A DECOMPOSED HIGHER ORDER AMBISONICS (HOA) AUDIO SIGNAL”;

U.S. Provisional Application No. 62/056,248, filed on September 26, 2014, entitled “SWITCHED V-VECTOR QUANTIZATION OF A HIGHER ORDER AMBISONICS (HOA) AUDIOSIGNAL”; and

U.S. Provisional Application No. 62/056,286, filed September 26, 2014, entitled “PREDICTIVE VECTOR QUANTIZATION OF A DECOMPOSED HIGHER ORDER AMBISONICS (HOA) AUDIO SIGNAL”; and

U.S. Provisional Application No. 62/102,243, filed January 12, 2015, entitled “TRANSITIONING OF AMBIENT HIGHER-ORDER AMBISONIC COEFFICIENTS,”

Each of the aforementioned U.S. provisional applications is incorporated herein by reference as if set forth in its respective entirety.

Technical Field

The present disclosure relates to audio data, and more particularly, to the coding of high-order ambisonic audio data.

Background Art

A higher-order ambisonic (HOA) signal (often represented by a plurality of spherical harmonic coefficients (SHCs) or other hierarchical elements) is a three-dimensional representation of a sound field. An HOA or SHC representation can represent the sound field in a manner that is independent of the local speaker geometry used to play back the multi-channel audio signal rendered from the SHC signal. An SHC signal can also facilitate backward compatibility because the SHC signal can be rendered in a well-known and widely adopted multi-channel format (e.g., a 5.1 audio channel format or a 7.1 audio channel format). The SHC representation thus enables a better representation of the sound field, which also accommodates backward compatibility.

Summary of the Invention

In general, techniques for coding higher-order ambisonic audio data are described. The higher-order ambisonic audio data may include at least one spherical harmonic coefficient corresponding to a spherical harmonic basis function having an order greater than one.

In one aspect, an efficient bit usage method includes obtaining a bitstream including vectors representing orthogonal spatial axes in a spherical harmonic domain. The bitstream further includes an indicator as to whether to reuse at least one syntax element from a previous frame indicating information used in compressing the vectors.

In another aspect, a device configured to perform efficient bit usage includes one or more processors configured to obtain a bitstream comprising vectors representing orthogonal spatial axes in a spherical harmonic domain. The bitstream further includes an indicator as to whether at least one syntax element from a previous frame indicating information used in compressing the vectors is reused. The device also includes a memory configured to store the bitstream.

In another aspect, a device configured to perform efficient bit usage includes means for obtaining a bitstream comprising vectors representing orthogonal spatial axes in a spherical harmonic domain. The bitstream further includes an indicator as to whether at least one syntax element from a previous frame indicating information used in compressing the vectors is reused. The device also includes means for storing the indicator.

On the other hand, a non-transitory computer-readable storage medium has instructions stored thereon that, when executed, cause one or more processors to obtain a bitstream comprising vectors representing orthogonal spatial axes in a spherical harmonic domain, wherein the bitstream further comprises an indicator of whether to reuse at least one syntax element indicating information from a previous frame to be used in compressing the vectors.

The details of one or more aspects of the technology are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the technology will be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating spherical harmonic basis functions with various orders and sub-orders.

2 is a diagram illustrating a system that may perform various aspects of the techniques described in this disclosure.

3 is a block diagram illustrating in greater detail an example of the audio encoding device shown in the example of FIG. 2 that may perform various aspects of the techniques described in this disclosure.

FIG. 4 is a block diagram illustrating the audio decoding apparatus of FIG. 2 in more detail.

5A is a flowchart illustrating exemplary operation of an audio encoding device performing various aspects of the vector-based synthesis techniques described in this disclosure.

5B is a flowchart illustrating exemplary operation of an audio encoding device in performing various aspects of the coding techniques described in this disclosure.

6A is a flowchart illustrating exemplary operation of an audio decoding device in performing various aspects of the techniques described in this disclosure.

6B is a flowchart illustrating exemplary operation of an audio decoding device in performing various aspects of the coding techniques described in this disclosure.

7 is a diagram illustrating in greater detail a frame of a bitstream that may specify a compressed spatial component.

8 is a diagram illustrating in greater detail a portion of a bitstream that may specify a compressed spatial component.

DETAILED DESCRIPTION

The evolution of surround sound has made many output formats available for entertainment today. Examples of these consumer surround sound formats are mostly "channel" in nature because they implicitly specify the feeds to the loudspeakers with certain geometric coordinates. Consumer surround sound formats include the popular 5.1 format (which includes the following six channels: left front (FL), right front (FR), center or front center, left rear or left surround, right rear or right surround, and low-frequency effects (LFE)), the developing 7.1 format, various formats that include height speakers, such as the 7.1.4 format, and the 22.2 format (for example, for use with ultra-high-definition television standards). Non-consumer formats can span any number of speakers (in symmetrical and asymmetrical geometric arrangements) and are often referred to as "surround arrays." An example of such an array includes 32 loudspeakers positioned at coordinates on the corners of a truncated icosohedron.

The input to future MPEG encoders will be in one of three possible formats: (i) traditional channel-based audio (as discussed above), intended to be played through loudspeakers at pre-specified positions; (ii) object-based audio, which involves discrete pulse-code modulation (PCM) data for a single audio object with associated metadata containing its position coordinates (and other information); and (iii) scene-based audio, which involves representing the sound field using coefficients of spherical harmonic basis functions (also known as "spherical harmonic coefficients" or SHCs, "higher-order ambisonics" or HOA, and "HOA coefficients"). Such future MPEG encoders may be described in more detail in the document entitled "Call for Proposals for 3D Audio" of the International Organization for Standardization/International Electrotechnical Commission (ISO)/(IEC) JTC1/SC29/WG11/ N13411 , published in Geneva, Switzerland in January 2013 and available at http://mpeg.chiariglione.org/sites/default/files/files/standards/parts/docs/w13411.zip .

There are various formats based on "surround sound" channels in the market. For example, they range from the 5.1 home theater system, which has achieved the greatest success in bringing stereo sound to the living room, to the 22.2 system developed by the Japan Broadcasting Corporation or NHK. Content creators (for example, Hollywood studios) will want to produce the soundtrack of a movie once, without spending the effort to remix it for each speaker configuration. In recent years, standard development organizations have been considering ways to provide encoding and subsequent decoding (which can be adaptive and agnostic to the speaker geometry (and number) and acoustic conditions at the playback location (involving the renderer)) into a standardized bitstream.

To provide content creators with this flexibility, a set of hierarchical elements can be used to represent the sound field. The set of hierarchical elements may refer to a set of elements in which the elements are ordered so that a basic set of low-order elements provides a complete representation of the modeled sound field. As the set is expanded to include higher-order elements, the representation becomes more detailed, thereby increasing the resolution.

An example of a set of hierarchical elements is a set of spherical harmonic coefficients (SHCs). The following expression demonstrates the description or representation of the sound field using SHCs:

The expression shows that the pressure p _i at any point in the sound field at time t can be uniquely represented by the SHC. Here, c is the speed of sound (~343 m/s), is the reference point (or observation point), j _n (·) is the nth-order spherical Bessel function, and is the nth-order and mth-order spherical harmonic basis functions. It can be recognized that the terms in square brackets are frequency domain representations of the signal (i.e., ) that can be approximated by various time-frequency transforms, such as discrete Fourier transform (DFT), discrete cosine transform (DCT), or wavelet transform. Other examples of hierarchical groups include arrays of wavelet transform coefficients and other arrays of multiresolution basis function coefficients.

Figure 1 is a diagram illustrating spherical harmonic basis functions from zeroth order (n=0) to fourth order (n=4). As can be seen, for each order, there is an expansion of m sub-orders, which are shown in the example of Figure 1 for ease of illustration but not explicitly mentioned.

SHC can be physically acquired (e.g., recorded) using various microphone array configurations or, alternatively, derived from a channel-based or object-based description of the sound field. SHC represents scene-based audio, where the SHC can be input to an audio encoder to obtain encoded SHC, which can facilitate more efficient transmission or storage. For example, a fourth-order representation involving (1+4) ^² (2⁵, and therefore fourth-order) coefficients can be used.

As mentioned above, SHC can be derived from microphone recordings using a microphone array. Various examples of how SHC can be derived from a microphone array are described in Poletti, M., "Three-Dimensional Surround Sound Systems Based on Spherical Harmonics," J. Audio Eng. Soc., Vol. 53, No. 11, November 2005, pp. 1004-1025.

To illustrate how SHC can be derived from an object-based description, consider the following equation. The coefficients of the sound field corresponding to an individual audio object can be expressed as:

where i is the nth-order spherical Hankel function (of the second kind), and ω is the position of the object. Knowing the frequency-dependent source energy of the object, g(ω) (e.g., using time-frequency analysis techniques, such as performing a Fast Fourier Transform on the PCM stream) allows us to convert each PCM object and corresponding position into an SHC. Furthermore, it can be shown (because of the linear and orthogonal decomposition described above) that the coefficients of each object are additive. In this way, many PCM objects can be represented by coefficients (e.g., as the sum of the coefficient vectors for the individual objects). Essentially, the coefficients contain information about the sound field (pressure in terms of 3D coordinates), and the above scenario represents a transformation from an individual object to a representation of the entire sound field near the observation point. The remaining figures are described below in the context of object-based and SHC-based audio coding.

FIG2 is a diagram illustrating a system 10 that can perform various aspects of the techniques described in this disclosure. As shown in the example of FIG2 , system 10 includes a content creator device 12 and a content consumer device 14. Although described in the context of content creator device 12 and content consumer device 14, the techniques can be implemented in any context in which the SHC (which may also be referred to as HOA coefficients) or any other hierarchical representation of a sound field is encoded to form a bitstream representing audio data. Furthermore, content creator device 12 can represent any form of computing device capable of implementing the techniques described in this disclosure, including a handset (or cellular phone), a tablet, a smartphone, or a desktop computer (to provide a few examples). Similarly, content consumer device 14 can represent any form of computing device capable of implementing the techniques described in this disclosure, including a handset (or cellular phone), a tablet, a smartphone, a set-top box, or a desktop computer (to provide a few examples).

The content creator device 12 may be operated by a film studio or other entity that can produce multi-channel audio content for consumption by operators of content consumers (e.g., content consumer device 14). In some examples, the content creator device 12 may be operated by an individual user who wishes to compress the HOA coefficients 11. Often, the content creator produces audio content along with the video content. The content consumer device 14 may be operated by an individual. The content consumer device 14 may include an audio playback system 16, which may refer to any form of audio playback system capable of presenting SHC for playback as multi-channel audio content.

The content creator device 12 includes an audio editing system 18. The content creator device 12 obtains a field recording 7 and audio objects 9 in various formats (including directly as HOA coefficients), which the content creator device 12 can edit using the audio editing system 18. The content creator can render the HOA coefficients 11 from the audio objects 9 during the editing process, listening to the rendered speaker feeds in an attempt to identify various aspects of the sound field that require further editing. The content creator device 12 can then edit the HOA coefficients 11 (possibly indirectly by manipulating different ones of the audio objects 9 from which the source HOA coefficients can be derived in the manner described above). The content creator device 12 can generate the HOA coefficients 11 using the audio editing system 18. The audio editing system 18 represents any system capable of editing audio data and outputting the audio data as one or more source spherical harmonic coefficients.

When the editing process is complete, the content creator device 12 can generate a bitstream 21 based on the HOA coefficients 11. That is, the content creator device 12 includes an audio encoding device 20, which represents a device configured to encode or otherwise compress the HOA coefficients 11 according to various aspects of the techniques described in this disclosure to generate the bitstream 21. The audio encoding device 20 can generate the bitstream 21 for transmission, as an example, across a transmission channel (which can be a wired or wireless channel, a data storage device, or the like). The bitstream 21 can represent an encoded version of the HOA coefficients 11 and can include a main bitstream and another side bitstream (which can be referred to as side channel information).

Although described in greater detail below, the audio encoding device 20 may be configured to encode the HOA coefficients 11 based on either vector-based synthesis or directional-based synthesis. To determine whether to perform a vector-based decomposition method or a directional-based decomposition method, the audio encoding device 20 may determine, at least in part based on the HOA coefficients 11, whether the HOA coefficients 11 were generated via a natural recording of a sound field (e.g., a field recording 7) or artificially (i.e., synthetically) generated from an audio object 9, such as a PCM object (as an example). When the HOA coefficients 11 are generated from an audio object 9, the audio encoding device 20 may encode the HOA coefficients 11 using a directional-based decomposition method. When the HOA coefficients 11 are captured live (e.g., using eigenmike), the audio encoding device 20 may encode the HOA coefficients 11 based on a vector-based decomposition method. The above distinction represents one example of how either a vector-based or directional-based decomposition method may be deployed. Other situations may exist where either or both of the decomposition methods may be used for natural recordings, artificially generated content, or a mixture of both (hybrid content). Furthermore, it is also possible to use both methods simultaneously for decoding a single time frame of HOA coefficients.

Assuming for purposes of illustration that the audio encoding device 20 determines that the HOA coefficients 11 were captured live or otherwise represent a live recording (e.g., live recording 7), the audio encoding device 20 may be configured to encode the HOA coefficients 11 using a vector-based decomposition method involving the application of a linear reversible transform (LIT). One example of a linear reversible transform is called "singular value decomposition" (or "SVD"). In this example, the audio encoding device 20 may apply the SVD to the HOA coefficients 11 to determine a decomposed version of the HOA coefficients 11. The audio encoding device 20 may then analyze the decomposed version of the HOA coefficients 11 to identify various parameters that may facilitate reordering the decomposed version of the HOA coefficients 11. The audio encoding device 20 may then reorder the decomposed version of the HOA coefficients 11 based on the identified parameters, where, as described in further detail below, this reordering may improve coding efficiency given that the transform may reorder the HOA coefficients across a frame of HOA coefficients (where a frame may include M samples of the HOA coefficients 11 and, in some examples, M is set to 1024). After reordering the decomposed versions of the HOA coefficients 11, the audio encoding device 20 may select the decomposed version of the HOA coefficients 11 that represents the foreground (or, in other words, the specific, dominant, or prominent) component of the sound field. The audio encoding device 20 may designate the decomposed version of the HOA coefficients 11 that represents the foreground component as an audio object and associated directional information.

The audio encoding device 20 may also perform sound field analysis on the HOA coefficients 11 in order to at least partially identify HOA coefficients 11 representing one or more background (or, in other words, ambient) components of the sound field. The audio encoding device 20 may perform energy compensation on the background components given the following circumstances: in some instances, the background components may only include a subset of any given sample of the HOA coefficients 11 (e.g., HOA coefficients 11 corresponding to zero-order and first-order spherical basis functions, rather than HOA coefficients 11 corresponding to second-order or higher-order spherical basis functions). In other words, when performing order reduction, the audio encoding device 20 may amplify (e.g., add energy/subtract energy) the remaining background HOA coefficients in the HOA coefficients 11 to compensate for the change in overall energy caused by performing order reduction.

The audio encoding device 20 may then perform a form of psychoacoustic encoding (e.g., MPEG Surround, MPEG-AAC, MPEG-USAC, or other known forms of psychoacoustic encoding) on each of the HOA coefficients 11 representing each of the background components and the foreground audio objects. The audio encoding device 20 may perform a form of interpolation on the foreground directional information and then perform order reduction on the interpolated foreground directional information to produce reduced-order foreground directional information. In some examples, the audio encoding device 20 may further perform quantization on the reduced-order foreground directional information, thereby outputting the coded foreground directional information. In some cases, quantization may include scalar/entropy quantization. The audio encoding device 20 may then form a bitstream 21 to include the encoded background components, the encoded foreground audio objects, and the quantized directional information. The audio encoding device 20 may then transmit or otherwise output the bitstream 21 to the content consumer device 14.

2 as being transmitted directly to the content consumer device 14, the content creator device 12 may output the bitstream 21 to an intermediate device positioned between the content creator device 12 and the content consumer device 14. The intermediate device may store the bitstream 21 for later delivery to the content consumer device 14 that may request the bitstream. The intermediate device may comprise a file server, a web server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smartphone, or any other device capable of storing the bitstream 21 for later retrieval by an audio decoder. The intermediate device may reside in a content delivery network that is capable of streaming the bitstream 21 (and possibly in conjunction with transmitting a corresponding video data bitstream) to a subscriber (e.g., the content consumer device 14) requesting the bitstream 21.

Alternatively, the content creator device 12 may store the bitstream 21 on a storage medium such as a compact disc, a digital versatile disc, a high-definition video disc, or other storage medium, most of which are capable of being read by a computer and thus may be referred to as computer-readable storage medium or non-transitory computer-readable storage medium. In this context, a transmission channel may refer to those channels through which the content stored on the medium is transmitted (and may include retail stores and other store-based delivery mechanisms). In any case, the technology of this disclosure should not be limited to the example of FIG. 2 in this regard.

As further shown in the example of FIG2 , the content consumer device 14 includes an audio playback system 16. The audio playback system 16 may represent any audio playback system capable of playing back multi-channel audio data. The audio playback system 16 may include a number of different renderers 22. The renderers 22 may each provide a different form of presentation, wherein the different forms of presentation may include one or more of various ways of performing vector-based amplitude shifting (VBAP) and/or one or more of various ways of performing sound field synthesis. As used herein, "A and/or B" means "A or B," or "both A and B."

The audio playback system 16 may further include an audio decoding device 24. The audio decoding device 24 may represent a device configured to decode HOA coefficients 11′ from the bitstream 21, wherein the HOA coefficients 11′ may be similar to the HOA coefficients 11, but differ due to lossy operations (e.g., quantization) and/or transmission through a transmission channel. That is, the audio decoding device 24 may dequantize the foreground directional information specified in the bitstream 21 while also performing quality decoding on the foreground audio objects specified in the bitstream 21 and the encoded HOA coefficients representing the background components. The audio decoding device 24 may further perform interpolation on the decoded foreground directional information and then determine the HOA coefficients representing the foreground components based on the decoded foreground audio objects and the interpolated foreground directional information. The audio decoding device 24 may then determine the HOA coefficients 11′ based on the determined HOA coefficients representing the foreground components and the decoded HOA coefficients representing the background components.

The audio playback system 16 may obtain the HOA coefficients 11' after decoding the bitstream 21 and present the HOA coefficients 11' to output the loudspeaker feed 25. The loudspeaker feed 25 may drive one or more loudspeakers (which are not shown in the example of FIG. 2 for ease of illustration).

To select an appropriate renderer, or in some cases, generate an appropriate renderer, the audio playback system 16 may obtain loudspeaker information 13 indicating the number of loudspeakers and/or the spatial geometric arrangement of the loudspeakers. In some cases, the audio playback system 16 may obtain the loudspeaker information 13 using a reference microphone and driving the loudspeakers in a manner that dynamically determines the loudspeaker information 13. In other cases, or in conjunction with the dynamic determination of the loudspeaker information 13, the audio playback system 16 may prompt the user to interface with the audio playback system 16 and input the loudspeaker information 13.

The audio playback system 16 may then select one of the audio renderers 22 based on the loudspeaker information 13. In some cases, when none of the audio renderers 22 are within a certain threshold similarity metric (in terms of loudspeaker geometry) to the one specified in the loudspeaker information 13, the audio playback system 16 may generate that one of the audio renderers 22 based on the loudspeaker information 13. In some cases, the audio playback system 16 may generate one of the audio renderers 22 based on the loudspeaker information 13 without first attempting to select an existing one of the audio renderers 22.

FIG3 is a block diagram illustrating in greater detail an example of the audio encoding device 20 shown in the example of FIG2 that can perform various aspects of the techniques described in this disclosure. The audio encoding device 20 includes a content analysis unit 26, a vector-based decomposition unit 27, and a direction-based decomposition unit 28. Although briefly described below, more information regarding the audio encoding device 20 and various aspects of compressing or otherwise encoding HOA coefficients can be found in International Patent Application Publication No. WO 2014/194099, filed May 29, 2014, entitled “INTERPOLATION FOR DECOMPOSED REPRESENTATIONS OF A SOUND FIELD.”

The content analysis unit 26 represents a unit configured to analyze the content of the HOA coefficients 11 to identify whether the HOA coefficients 11 represent content generated from a live recording or from an audio object. The content analysis unit 26 may determine whether the HOA coefficients 11 are generated from a recording of an actual sound field or from an artificial audio object. In some cases, when the frame HOA coefficients 11 are generated from a recording, the content analysis unit 26 passes the HOA coefficients 11 to a vector-based decomposition unit 27. In some cases, when the frame HOA coefficients 11 are generated from a synthesized audio object, the content analysis unit 26 passes the HOA coefficients 11 to a direction-based synthesis unit 28. The direction-based synthesis unit 28 may represent a unit configured to perform direction-based synthesis of the HOA coefficients 11 to generate a direction-based bitstream 21.

As shown in the example of Figure 3, the vector-based decomposition unit 27 may include a linear reversible transform (LIT) unit 30, a parameter calculation unit 32, a reordering unit 34, a foreground selection unit 36, an energy compensation unit 38, a psychoacoustic audio decoder unit 40, a bitstream generation unit 42, a sound field analysis unit 44, a coefficient reduction unit 46, a background (BG) selection unit 48, a space-time interpolation unit 50 and a quantization unit 52.

The linear invertible transform (LIT) unit 30 receives HOA coefficients 11 in the form of HOA channels, each channel representing a block or frame of coefficients associated with a given order or sub-order of a spherical basis function (which may be denoted as HOA[k], where k may represent the current frame or block of samples). The matrix of HOA coefficients 11 may have dimensions D: M×(N+1) ^² .

That is, LIT unit 30 may represent a unit configured to perform a form of analysis known as singular value decomposition. Although described with respect to SVD, the techniques described in this disclosure may be performed with respect to any similar transform or decomposition that provides an energy-dense output of a set of linearly uncorrelated numbers. Furthermore, references to "groups" in this disclosure are generally intended to refer to non-zero groups (unless specifically stated to the contrary) and are not intended to refer to the classical mathematical definition of a group, which includes the so-called "null group."

An alternative transformation may include principal component analysis, often referred to as "PCA". PCA refers to a mathematical procedure that uses an orthogonal transformation to convert the observations of a set of possibly correlated variables into a set of linearly uncorrelated variables called principal components. Linearly uncorrelated variables refer to variables that do not have a linear statistical relationship (or dependence) with each other. Principal components can be described as having a small degree of statistical correlation with each other. In any case, the number of so-called principal components is less than or equal to the number of original variables. In some instances, the transformation is defined as follows: the first principal component has the largest possible variance (or, in other words, takes into account as much variability in the data as possible), and each subsequent component has the highest possible variance (under the constraint that the successive components are orthogonal to the previous components (which can be restated as being uncorrelated with the previous components)). PCA can perform a form of order reduction, which can result in compression of the HOA coefficients 11 with respect to the HOA coefficients 11. Depending on the context, PCA may be referred to by several different names, such as discrete Karhunen-Loeve transform, Hotelling transform, Proper Orthogonal Decomposition (POD), and Eigenvalue Decomposition (EVD), to name a few. Properties of such operations that facilitate the fundamental goal of compressing audio data are "energy compression" and "decorrelation" of multi-channel audio data.

In any case, assuming for purposes of example that the LIT unit 30 performs singular value decomposition (which again may be referred to as "SVD"), the LIT unit 30 may transform the HOA coefficients 11 into two or more sets of transformed HOA coefficients. The "array" of transformed HOA coefficients may include a vector of transformed HOA coefficients. In the example of FIG. 3 , the LIT unit 30 may perform SVD on the HOA coefficients 11 to generate so-called V matrices, S matrices, and U matrices. In linear algebra, SVD may represent the factorization of a y-by-z real or complex matrix X (where X may represent multi-channel audio data, such as the HOA coefficients 11) as follows:

X＝USV ^*

U may represent a y-by-y real or complex identity matrix, where the y columns of U are referred to as the left singular vectors of the multi-channel audio data. S may represent a y-by-z rectangular diagonal matrix with non-negative real numbers on the diagonal, where the diagonal values of S are referred to as the singular values of the multi-channel audio data. V* (which may represent the conjugate transpose of V) may represent a z-by-z real or complex identity matrix, where the z columns of V* are referred to as the right singular vectors of the multi-channel audio data.

Although described in the present invention as applying the technique to multi-channel audio data including the HOA coefficients 11, the technique can be applied to any form of multi-channel audio data. In this manner, the audio encoding device 20 can perform singular value decomposition on the multi-channel audio data representing at least a portion of the sound field to generate a U matrix representing the left singular vectors of the multi-channel audio data, an S matrix representing the singular values of the multi-channel audio data, and a V matrix representing the right singular vectors of the multi-channel audio data, and represent the multi-channel audio data as a function of at least a portion of one or more of the U matrix, the S matrix, and the V matrix.

In some examples, the V* matrix in the SVD mathematical expression mentioned above is expressed as the conjugate transpose of the V matrix to reflect that SVD can be applied to matrices including complex numbers. When applied to matrices including only real numbers, the complex conjugate of the V matrix (or, in other words, the V* matrix) can be considered as the transpose of the V matrix. For ease of explanation below, it is assumed that the HOA coefficients 11 include real numbers, resulting in the output of the V matrix via SVD rather than the V* matrix. In addition, although expressed as a V matrix in this disclosure, references to the V matrix should be understood to refer to the transpose of the V matrix when appropriate. Although assumed to be a V matrix, the described techniques can be applied in a similar manner to HOA coefficients 11 having complex coefficients, where the output of the SVD is the V* matrix. Therefore, in this regard, the described techniques should not be limited to providing only the application of SVD to generate the V matrix, but may include applying SVD to HOA coefficients 11 having complex components to generate the V* matrix.

In any case, LIT unit 30 may perform a block-by-block SVD on each block (which may be referred to as a frame) of higher-order ambisonic (HOA) audio data (where the ambisonic audio data includes blocks or samples of HOA coefficients 11 or any other form of multi-channel audio data). As mentioned above, the variable M may be used to represent the length (in number of samples) of the audio frame. For example, when the audio frame includes 1024 audio samples, M is equal to 1024. Although described with respect to typical values of M, the techniques of this disclosure should not be limited to typical values of M. LIT unit 30 may therefore perform a block-by-block SVD on a block of HOA coefficients 11 having M times (N+1) ² HOA coefficients, where N again represents the order of the HOA audio data. By performing the SVD, LIT unit 30 may generate a V matrix, an S matrix, and a U matrix, where each of the matrices may represent the respective V, S, and U matrices described above. In this manner, the linear reversible transform unit 30 may perform SVD on the HOA coefficients 11 to output a US[k] vector 33 having dimensions D: M×(N+1) ² (which may represent a combined version of the S vector and the U vector), and a V[k] vector 35 having dimensions D: (N+1) ² ×(N+1) ^2. Individual vector elements in the US[k] matrix may also be referred to as _XPS (k), and individual vectors in the V[k] matrix may also be referred to as v(k).

Analysis of the U, S, and V matrices reveals that the matrices carry or represent the spatial and temporal characteristics of the underlying sound field, denoted above by X. Each of the N vectors in U (of length M samples) can represent a normalized, separated audio signal in terms of time (for a time period denoted by M samples), which are orthogonal to one another and decoupled from any spatial characteristics (which can also be referred to as directional information). The spatial characteristics representing spatial shape and position width can instead be represented by individual i-th vectors v ⁽ⁱ⁾ (k) in the V matrix (each having length (N+1) ^² ). The individual elements of each ^v(i) (k) vector can represent the HOA coefficients describing the shape and direction of the sound field for the associated audio object. The vectors in both the U and V matrices are normalized so that their root mean square energy is equal to unity. The energy of the audio signal in U is therefore represented by the diagonal elements in S. Multiplying U and S forms US[k] (with individual vector elements _XPS (k)), thus representing the audio signal with true energy. The ability to perform SVD decomposition to decouple the audio time signal (in U), its energy (in S), and its spatial characteristics (in V) can support various aspects of the techniques described in this disclosure. Additionally, the model of synthesizing the basis HOA[k] coefficients X by vector multiplication of US[k] and V[k] leads to the term "vector-based decomposition" used throughout this document.

Although described as being performed directly on the HOA coefficients 11, the LIT unit 30 may apply a linear reversible transform to the derivatives of the HOA coefficients 11. For example, the LIT unit 30 may apply an SVD to the power spectral density matrix derived from the HOA coefficients 11. The power spectral density matrix may be represented as a PSD and is obtained via matrix multiplication of the transpose of hoaFrame to hoaFrame, as outlined in the pseudocode below. The hoaFrame notation refers to a frame of the HOA coefficients 11.

After applying SVD (svd) to the PSD, the LIT unit 30 may obtain an S[k] ² matrix (S_squared) and a V[k] matrix. The S[k] ² matrix may represent the square of the S[k] matrix, so the LIT unit 30 may apply a square root operation to the S[k] ² matrix to obtain the S[k] matrix. In some cases, the LIT unit 30 may perform quantization on the V[k] matrix to obtain a quantized V[k] matrix (which may be represented as a V[k]' matrix). The LIT unit 30 may obtain the U[k] matrix by first multiplying the S[k] matrix by the quantized V[k]' matrix to obtain the SV[k]' matrix. The LIT unit 30 may then obtain the pseudo-inverse (pinv) of the SV[k]' matrix and then multiply the HOA coefficients 11 by the pseudo-inverse of the SV[k]' matrix to obtain the U[k] matrix. The foregoing scenario may be represented by the following pseudo-code:

PSD = hoaFrame'*hoaFrame;

[V,S_squared]=svd(PSD,’econ’);

S = sqrt(S_squared);

U = hoaFrame*pinv(S*V');

By performing SVD on the power spectral density (PSD) of the HOA coefficients rather than the coefficients themselves, the LIT unit 30 can potentially reduce the computational complexity of performing the SVD in terms of one or more of processor cycles and memory space, while achieving the same source audio coding efficiency as if the SVD were applied directly to the HOA coefficients. That is, the PSD-type SVD described above can potentially be less computationally demanding because the SVD is performed on an F*F matrix (where F is the number of HOA coefficients) as compared to an M*F matrix (where M is the frame length, i.e., 1024 or greater). By being applied to the PSD rather than the HOA coefficients 11, the complexity of the SVD can now be approximately O(L ³ ) (where O(*) represents the Big O notation for computational complexity commonly used in computer science) as compared to O(M*L ² ) when applied to the HOA coefficients 11.

In this regard, the LIT unit 30 may perform decomposition on the higher-order ambisonic audio data 11 or otherwise decompose the higher-order ambisonic audio data 11 to obtain vectors (e.g., the aforementioned V-vectors) representing orthogonal spatial axes in the spherical harmonic domain. The decomposition may include SVD, EVD, or any other form of decomposition.

Parameter calculation unit 32 represents a unit configured to calculate various parameters, such as a correlation parameter (R), a directional property parameter, and an energy property (e). Each of the parameters for the current frame may be represented as R[k] and e[k]. Parameter calculation unit 32 may perform energy analysis and/or correlation (or so-called cross-correlation) on US[k] vector 33 to identify the parameters. Parameter calculation unit 32 may also determine parameters for the previous frame, where the previous frame parameters may be represented as R[k-1], θ[k-1], r[k-1], and e[k-1] based on the previous frame having US[k-1] vector and V[k-1] vector. Parameter calculation unit 32 may output current parameters 37 and previous parameters 39 to reordering unit 34.

SVD decomposition does not guarantee that the audio signal/object represented by the p-th vector in the US[k-1] vectors 33 (which may be represented as a US[k-1][p] vector (or, alternatively, as _XPS ^(p) (k-1))) will be the same audio signal/object represented by the p-th vector in the US[k] vectors 33 (which may also be represented as a US[k][p] vector 33 (or, alternatively, as _XPS ^(p) (k))) (advanced in time). The parameters calculated by the parameter calculation unit 32 may be used by the reordering unit 34 to reorder the audio objects to represent their natural evaluation or continuity over time.

That is, the reordering unit 34 may compare each of the parameters 37 from the first US[k] vector 33 with each of the parameters 39 for the second US[k-1] vector 33 on a round-by-round basis. The reordering unit 34 may reorder the various vectors within the US[k] matrix 33 and the V[k] matrix 35 based on the current parameters 37 and the previous parameters 39 (as an example, using the Hungarian algorithm) to output the reordered US[k] matrix 33′ (which may be mathematically represented as ) and the reordered V[k] matrix 35′ (which may be mathematically represented as ) to the foreground sound (or dominant sound—PS) selection unit 36 (“foreground selection unit 36”) and the energy compensation unit 38.

The soundfield analysis unit 44 may represent a unit configured to perform soundfield analysis on the HOA coefficients 11 in order to potentially achieve the target bitrate 41. Based on the analysis and/or based on the received target bitrate 41, the soundfield analysis unit 44 may determine the total number of psychoacoustic decoder instances (which may be a function of the total number of ambient or background channels (BG _TOT )) and the number of foreground channels (or, in other words, dominant channels). The total number of psychoacoustic decoder instances may be represented as numHOATransportChannels.

Again, to possibly achieve the target bit rate 41, the sound field analysis unit 44 may also determine the total number of foreground channels (nFG) 45, the minimum order of the background (or in other words, ambient) sound field ( _NBG or, alternatively, MinAmbHoaOrder), the corresponding number of actual channels representing the minimum order of the background sound field (nBGa = (MinAmbHoaOrder + 1) ² ), and the index (i) of the additional BG HOA channel to be sent (which may be collectively denoted as background channel information 43 in the example of FIG. 3 ). The background channel information 42 may also be referred to as ambient channel information 43. Each of the channels remaining after numHOATransportChannels-nBGa may be an "additional background/ambient channel," an "active vector-based dominant channel," an "active direction-based dominant signal," or "completely inactive." In one aspect, the channel type may be indicated by two bits in the "ChannelType" syntax element: (e.g., 00: direction-based signal; 01: vector-based dominant signal; 10: additional ambient signal; 11: inactive signal). The total number of background or ambient signals, nBGa, may be given by (MinAmbHOAorder+1) ^² + the number of times index 10 (in the above example) appears as a channel type in the bitstream for that frame.

In any case, the sound field analysis unit 44 may select the number of background (or in other words, ambient) channels and the number of foreground (or in other words, dominant) channels based on the target bit rate 41, thereby selecting more background and/or foreground channels when the target bit rate 41 is relatively high (e.g., when the target bit rate 41 is equal to or greater than 512 Kbps). In one aspect, in the header segment of the bitstream, numHOATransportChannels may be set to 8 and MinAmbHOAorder may be set to 1. In this scenario, at each frame, four channels may be dedicated to representing the background or ambient portion of the sound field, while the other four channels may vary in channel type from frame to frame - for example, used as additional background/ambient channels or foreground/dominant channels. The foreground/dominant signal may be one of a vector-based or a direction-based signal, as described above.

In some cases, the total number of vector-based dominant signals for a frame can be given by the number of times the ChannelType index is 01 in the bitstream of the frame. In the above aspect, for each additional background/ambient channel (e.g., corresponding to ChannelType 10), corresponding information can be provided indicating which of the possible HOA coefficients (except the first four) is present in the channel. For fourth-order HOA content, the information can be an index indicating HOA coefficients 5 to 25. The first four ambient HOA coefficients 1 to 4 can always be sent when minAmbHOAorder is set to 1, so the audio encoding device may only need to indicate one of the additional ambient HOA coefficients with an index of 5 to 25. Therefore, the information can be sent using a 5-bit syntax element (for fourth-order content), which can be denoted as "CodedAmbCoeffIdx".

For illustration, assume that minAmbHOAorder is set to 1 and additional ambient HOA coefficients with index 6 are sent via the bitstream 21 (as an example). In this example, minAmbHOAorder 1 indicates that the ambient HOA coefficients have indices 1, 2, 3, and 4. The audio encoding device 20 may select the ambient HOA coefficients because the ambient HOA coefficients have indices less than or equal to (minAmbHOAorder+1) ² or 4 (in this example). The audio encoding device 20 may specify the ambient HOA coefficients associated with indices 1, 2, 3, and 4 in the bitstream 21. The audio encoding device 20 may also specify the additional ambient HOA coefficient with index 6 in the bitstream as additionalAmbientHOAchannel with ChannelType 10. The audio encoding device 20 may specify the indices using the CodedAmbCoeffIdx syntax element. As a practice, the CodedAmbCoeffIdx element may specify all indices from 1 to 25. However, because minAmbHOAorder is set to 1, the audio encoding device 20 may not specify any of the first four indices (because it is known that the first four indices will be specified in the bitstream 21 via the minAmbHOAorder syntax element). In any case, because the audio encoding device 20 specifies five ambient HOA coefficients via minAmbHOAorder (for the first four coefficients) and CodedAmbCoeffIdx (for the additional ambient HOA coefficients), the audio encoding device 20 may not specify corresponding V-vector elements associated with the ambient HOA coefficients with indices 1, 2, 3, 4, and 6. Therefore, the audio encoding device 20 may specify the V-vector via elements [5, 7:25].

In a second aspect, all foreground/dominant signals are vector-based signals.In this second aspect, the total number of foreground/dominant signals may be given by nFG = numHOATransportChannels - [(MinAmbHoaOrder+1) ² + each of additionalAmbientHOAchannels].

The sound field analysis unit 44 outputs the background channel information 43 and the HOA coefficient 11 to the background (BG) selection unit 36 , outputs the background channel information 43 to the coefficient reduction unit 46 and the bitstream generation unit 42 , and outputs the nFG 45 to the foreground selection unit 36 .

The background selection unit 48 may represent a unit configured to determine background or ambient HOA coefficients 47 based on background channel information, such as the background sound field (N _BG ) and the number (nBGa) and index (i) of additional BG HOA channels to be sent. For example, when N _BG is equal to one, the background selection unit 48 may select the HOA coefficients 11 for each sample of the audio frame having an order equal to or less than one. In this example, the background selection unit 48 may then select the HOA coefficients 11 having an index identified by one of the indices (i) as the additional BG HOA coefficients, where nBGa to be specified in the bitstream 21 is provided to the bitstream generation unit 42 to enable an audio decoding device (e.g., the audio decoding device 24 shown in the examples of Figures 2 and 4) to parse the background HOA coefficients 47 from the bitstream 21. The background selection unit 48 may then output the ambient HOA coefficients 47 to the energy compensation unit 38. The ambient HOA coefficients 47 may have dimension D: M×[(N _BG +1) ² +nBGa]. The ambient HOA coefficients 47 may also be referred to as “ambient HOA coefficients 47,” where each of the ambient HOA coefficients 47 corresponds to a separate ambient HOA channel 47 to be encoded by the psychoacoustic audio coder unit 40.

The foreground selection unit 36 may represent a unit configured to select the reordered US[k] matrix 33′ and the reordered V[k] matrix 35′ representing the foreground or distinctive components of the sound field based on nFG 45 (which may represent one or more indices identifying the foreground vectors). The foreground selection unit 36 may output an nFG signal 49 (which may be represented as reordered US[k] _{1, ..., nFG} 49, FG _{1, ..., nfG} [k] 49, or 49) to the psychoacoustic audio decoder unit 40, where the nFG signal 49 may have dimensions D: M×nFG and each represents a single channel-audio object. The foreground selection unit 36 may also output the reordered V[k] matrix 35' (or v ^(1..nFG) (k)35') corresponding to the foreground components of the sound field to the spatial-temporal interpolation unit 50, where a subset of the reordered V[k] matrix 35' corresponding to the foreground components may be represented as a foreground V[k] matrix 51 _k (which may be mathematically represented as), which has dimension D: (N+1) ² ×nFG.

The energy compensation unit 38 may represent a unit configured to perform energy compensation on the ambient HOA coefficients 47 to compensate for the energy loss due to the removal of each of the HOA channels by the background selection unit 48. The energy compensation unit 38 may perform an energy analysis on one or more of the reordered US[k] matrix 33′, the reordered V[k] matrix 35′, the nFG signal 49, the foreground V[k] vector 51 _k , and the ambient HOA coefficients 47, and then perform energy compensation based on the energy analysis to generate energy-compensated ambient HOA coefficients 47′. The energy compensation unit 38 may output the energy-compensated ambient HOA coefficients 47′ to the psychoacoustic audio decoder unit 40.

The spatial-temporal interpolation unit 50 may represent a unit configured to receive the foreground V[k] vector 51 _k of the k-th frame and the foreground V[k-1] vector 51 _k-1 of the previous frame (hence the k-1 notation) and perform spatial-temporal interpolation to generate an interpolated foreground V[k] vector. The spatial-temporal interpolation unit 50 may recombines the nFG signal 49 with the foreground V[k] vector 51 _k to recover the reordered foreground HOA coefficients. The spatial-temporal interpolation unit 50 may then divide the reordered foreground HOA coefficients by the interpolated V[k] vector to generate the interpolated nFG signal 49 ′. The spatial-temporal interpolation unit 50 may also output the foreground V[k] vector 51 _k used to generate the interpolated foreground V[k] vector so that an audio decoding device (e.g., the audio decoding device 24) can generate the interpolated foreground V[k] vector and thereby recover the foreground V[k] vector 51 _k . The foreground V[k] vector 51 _k used to generate the interpolated foreground V[k] vector is denoted as the residual foreground V[k] vector 53. To ensure that the same V[k] and V[k-1] are used at the encoder and decoder (to create the interpolated vector V[k]), quantized/dequantized versions of the vectors may be used at the encoder and decoder.

In operation, the spatio-temporal interpolation unit 50 may interpolate one or more subframes of a first audio frame from a first decomposition of a portion of a first plurality of HOA coefficients 11 included in the first frame (e.g., the foreground V[k] vector 51 _k ) and a second decomposition of a portion of a second plurality of HOA coefficients 11 included in the second frame (e.g., the foreground V[k] vector 51 _k-1 ) to generate decomposed interpolated spherical harmonic coefficients for the one or more subframes.

In some examples, the first decomposition includes a first foreground V[k] vector _51k representing right singular vectors of the portion of HOA coefficients 11. Similarly, in some examples, the second decomposition includes a second foreground V[k] vector _51k representing right singular vectors of the portion of HOA coefficients 11.

In other words, 3D audio based on spherical harmonics can be a parametric representation of a 3D pressure field in terms of orthogonal basis functions on a sphere. The higher the order N of the representation, the higher the spatial resolution can be, and often the larger the number of spherical harmonic (SH) coefficients (a total of (N+1) ² coefficients). For many applications, bandwidth compression of the coefficients may be required so that they can be efficiently transmitted and stored. The techniques targeted in this disclosure can provide a frame-based dimensionality reduction process using singular value decomposition (SVD). SVD analysis can decompose each frame of coefficients into three matrices U, S, and V. In some examples, the techniques can treat some of the vectors in the US[k] matrix as foreground components of the underlying sound field. However, when treated in this way, the vectors (in the US[k] matrix) are discontinuous between frames, even if they represent the same specific audio component. When these components are fed through a transform audio decoder, these discontinuities can lead to noticeable artifacts.

In some aspects, spatiotemporal interpolation can rely on the observation that the V matrix can be interpreted as orthogonal spatial axes in the spherical harmonic domain. The U[k] matrix can represent the projection of the spherical harmonic (HOA) data onto the basis functions, where discontinuities can be attributed to the orthogonal spatial axes (V[k]), which change every frame and are therefore themselves discontinuous. This differs from some other decompositions, such as the Fourier transform, where, in some instances, the basis functions are constant across frames. In these terms, SVD can be considered a matching pursuit algorithm. The spatiotemporal interpolation unit 50 can perform interpolation to maintain continuity between the basis functions (V[k]) from frame to frame, possibly by interpolating between frames.

As mentioned above, interpolation can be performed with respect to samples. This is generalized in the above description when a subframe includes a set of single samples. In both the case of interpolation over samples and over subframes, the interpolation operation can take the form of the following equation:

In the above equation, interpolation can be performed from a single V-vector v(k-1) with respect to a single V-vector v(k), which in one aspect can represent the V-vectors from adjacent frames k and k-1. In the above equation, l represents the resolution at which interpolation is performed, where l can indicate an integer number of samples and l=1, ..., T (where T is the length of samples over which interpolation is performed and over which the output interpolated vector is desired, and the length also indicates the l of the output generation vector of the process). Alternatively, l can indicate a subframe consisting of multiple samples. When, for example, a frame is divided into four subframes, l can include values of 1, 2, 3, and 4 for each of the subframes. The value of l can be signaled via the bitstream as a field called "CodedSpatialInterpolationTime" so that the interpolation operation can be repeated in the decoder. w(l) can include the value of the interpolation weight. When interpolation is linear, w(l) can vary linearly and monotonically between 0 and 1 depending on l. In other cases, w(l) may vary between 0 and 1 in a nonlinear but monotonic manner (e.g., a quarter cycle of a raised cosine) depending on l. The function w(l) can be indexed between several different function possibilities and signaled in the bitstream as a field called "SpatialInterpolationMethod" so that the same interpolation operation can be repeated by the decoder. When w(l) has a value close to 0, the output may be heavily weighted or influenced by v(k-1). When w(l) has a value close to 1, it ensures that the output is heavily weighted and influenced by v(k-1).

The coefficient reduction unit 46 may represent a unit configured to perform coefficient reduction on the remaining foreground V[k] vector 53 based on the background channel information 43 to output a reduced foreground V[k] vector 55 to the quantization unit 52. The reduced foreground V[k] vector 55 may have a dimension D: [(N+1) ^2- ( _NBG +1) ² - _BGTOT ]×nFG.

In this regard, the coefficient reduction unit 46 may represent a unit configured to reduce the number of coefficients of the remaining foreground V[k] vector 53. In other words, the coefficient reduction unit 46 may represent a unit configured to eliminate coefficients in the foreground V[k] vector that have little or no directional information (which form the remaining foreground V[k] vector 53). As described above, in some examples, the coefficients of the unique or (in other words) foreground V[k] vector corresponding to the first-order and zero-order basis functions (which may be represented as N _BG ) provide little directional information and, therefore, may be removed from the foreground V-vector (via a process that may be referred to as "coefficient reduction"). In this example, greater flexibility may be provided such that not only the coefficients corresponding to N _BG are identified from the set [(N _BG +1) ² +1, (N+1) ² ] but also additional HOA channels (which may be represented by the variable TotalOfAddAmbHOAChan). The sound field analysis unit 44 may analyze the HOA coefficients 11 to determine BG _TOT , which may identify not only ( _NBG +1) ² but also TotalOfAddAmbHOAChan, which may be collectively referred to as background channel information 43. The coefficient reduction unit 46 may then remove the coefficients corresponding to ( _NBG +1) ² and TotalOfAddAmbHOAChan from the remaining foreground V[k] vector 53 to generate a smaller dimensional V[k] matrix 55 of size ((N+1) ² -(BG _TOT )×nFG, which may also be referred to as a reduced foreground V[k] vector 55.

In other words, as mentioned in WO 2014/194099, coefficient reduction unit 46 may generate syntax elements for side channel information 57. For example, coefficient reduction unit 46 may specify a syntax element in the header of an access unit (which may include one or more frames) that indicates which of a plurality of configuration modes is selected. Although described as being specified on a per-access-unit basis, coefficient reduction unit 46 may specify the syntax element on a per-frame basis or any other periodic or aperiodic basis (e.g., once for the entire bitstream). In any case, the syntax element may include two bits that indicate which of the three configuration modes is selected for specifying the set of non-zero coefficients of the reduced foreground V[k] vector 55 to represent the directional aspect of the unique component. The syntax element may be denoted as "CodedVVecLength." In this way, coefficient reduction unit 46 may signal or otherwise specify in the bitstream which of the three configuration modes is used to specify the reduced foreground V[k] vector 55 in the bitstream 21.

For example, three configuration modes may be presented in the syntax table for VVecData (referenced later in this document). In this example, the configuration modes are as follows: (Mode 0) the full V-vector length is transmitted in the VVecData field; (Mode 1) the elements of the V-vector associated with the minimum number of coefficients for the ambient HOA coefficients and all elements of the V-vector containing the additional HOA channels are not transmitted; and (Mode 2) the elements of the V-vector associated with the minimum number of coefficients for the ambient HOA coefficients are not transmitted. The syntax table for VVecData illustrates these modes in conjunction with switch and case statements. Although described with respect to three configuration modes, the techniques should not be limited to three configuration modes and may include any number of configuration modes, including a single configuration mode or a plurality of modes. WO 2014/194099 provides a different example with four modes. Coefficient reduction unit 46 may also specify flag 63 as another syntax element in side channel information 57.

The quantization unit 52 may represent a unit configured to perform any form of quantization to compress the reduced foreground V[k] vector 55 to produce a coded foreground V[k] vector 57, thereby outputting the coded foreground V[k] vector 57 to the bitstream generation unit 42. In operation, the quantization unit 52 may represent a unit configured to compress the spatial components of the sound field (i.e., in this example, one or more of the reduced foreground V[k] vectors 55). For purposes of example, it is assumed that the reduced foreground V[k] vector 55 includes two row vectors, each column having fewer than 25 elements due to coefficient reduction (which implies a fourth-order HOA representation of the sound field). Although described with respect to two row vectors, any number of vectors may be included in the reduced foreground V[k] vector 55, up to (n+1) ² , where n represents the order of the HOA representation of the sound field. Furthermore, although described below as performing scalar and/or entropy quantization, the quantization unit 52 may perform any form of quantization that results in compression of the reduced foreground V[k] vector 55.

The quantization unit 52 may receive the reduced foreground V[k] vector 55 and perform a compression scheme to produce a coded foreground V[k] vector 57. The compression scheme may generally relate to any conceivable compression scheme for compressing elements of a vector or data and should not be limited to the examples described in more detail below. As an example, the quantization unit 52 may perform a compression scheme that includes one or more of: transforming a floating point representation of each element of the reduced foreground V[k] vector 55 into an integer representation of each element of the reduced foreground V[k] vector 55, uniform quantization of the integer representation of the reduced foreground V[k] vector 55, and sorting and coding of the quantized integer representation of the remaining foreground V[k] vector 55.

In some examples, several of the one or more processes of the compression scheme may be dynamically controlled by parameters to achieve, or nearly achieve (as an example), a target bit rate 41 for the resulting bitstream 21. Given that each of the reduced foreground V[k] vectors 55 is orthogonal to one another, each of the reduced foreground V[k] vectors 55 may be independently coded. In some examples, as described in more detail below, each element of each reduced foreground V[k] vector 55 may be coded using the same coding mode (defined by various sub-modes).

As described in WO 2014/194099, the quantization unit 52 may perform scalar quantization and/or Huffman encoding to compress the reduced foreground V[k] vector 55, thereby outputting a coded foreground V[k] vector 57 (which may also be referred to as side channel information 57). The side channel information 57 may include syntax elements used to code the remaining foreground V[k] vector 55.

Furthermore, although described with respect to a scalar form of quantization, quantization unit 52 may perform vector quantization or any other form of quantization. In some cases, quantization unit 52 may switch between vector quantization and scalar quantization. During scalar quantization described above, quantization unit 52 may calculate the difference between two consecutive V-vectors (e.g., consecutive from frame to frame) and code the difference (or, in other words, the residual). This scalar quantization may represent a form of predictive coding based on a previously specified vector and difference signal. Vector quantization does not involve this difference coding.

In other words, quantization unit 52 may receive an input V-vector (e.g., one of the downscaled foreground V[k] vectors 55) and perform different types of quantization to select a type of quantization type to be used for the input V-vector. As an example, quantization unit 52 may perform vector quantization, scalar quantization without Huffman coding, and scalar quantization with Huffman coding.

In this example, quantization unit 52 may vector quantize the input V-vector according to a vector quantization mode to generate a vector quantized V-vector. The vector quantized V-vector may include vector quantized weight values representing the input V-vector. In some examples, the vector quantized weight values may be represented as one or more quantization indices pointing to quantization codewords (i.e., quantization vectors) in a quantization codebook of quantization codewords. When configured to perform vector quantization, quantization unit 52 may decompose each of the reduced foreground V[k] vectors 55 into a weighted sum of codevectors based on code vectors 63 ("CV 63"). Quantization unit 52 may generate weight values for each of the selected code vectors in code vectors 63.

Quantization unit 52 may then select a subset of the weight values to generate a selected subset of weight values. For example, quantization unit 52 may select the Z largest magnitude weight values from the set of weight values to generate the selected subset of weight values. In some examples, quantization unit 52 may further reorder the selected weight values to generate the selected subset of weight values. For example, quantization unit 52 may reorder the selected weight values based on magnitude starting with the highest magnitude weight value and ending with the lowest magnitude weight value.

When performing vector quantization, quantization unit 52 may select a Z-component vector from a quantization codebook to represent the Z weight values. In other words, quantization unit 52 may vector-quantize the Z weight values to generate a Z-component vector representing the Z weight values. In some examples, Z may correspond to the number of weight values selected by quantization unit 52 to represent a single V-vector. Quantization unit 52 may generate data indicating the Z-component vector selected to represent the Z weight values and provide this data to bitstream generation unit 42 as coded weights 57. In some examples, the quantization codebook may include a plurality of indexed Z-component vectors, and the data indicating the Z-component vector may be an index value in the quantization codebook that points to the selected vector. In such examples, the decoder may include a similarly indexed quantization codebook to decode the index value.

Mathematically, each of the reduced foreground V[k] vectors 55 can be represented based on the following expression:

Where Ω _j represents the jth code vector in a set of code vectors ({Ω _j }), ω _j represents the jth weight in a set of weights ({ω _j }), V corresponds to the V-vector represented, decomposed, and/or decoded by V-vector decoding unit 52, and J represents the number of weights and the number of code vectors used to represent V. The right side of expression (1) may represent a weighted sum of code vectors including the set of weights ({ω _j }) and the set of code vectors ({Ω _j }).

In some examples, quantization unit 52 may determine the weight value based on the following equation:

where represents the transpose of the kth code vector in a set of code vectors ({Ω _k }), V corresponds to the V-vector represented, decomposed, and/or decoded by quantization unit 52, and ω _k represents the kth weight in a set of weights ({ω _k }).

Consider an example where 25 weights and 25 code vectors are used to represent the V-vector V _FG . This decomposition of V _FG can be written as:

where Ω _j represents the jth code vector in a set of code vectors ({Ω _j }), ω _j represents the jth weight in a set of weights ({ω _j }), and V _FG corresponds to the V-vector represented, decomposed, and/or decoded by quantization unit 52.

In the instance where the group code vectors ({Ω _j }) are orthogonal, the following expressions may apply:

In these examples, the right side of equation (3) can be simplified as follows:

where ω _k corresponds to the kth weight in the weighted sum of the code vectors.

For the example weighted sum of codevectors used in equation (3), quantization unit 52 may calculate weight values for each of the weights in the weighted sum of codevectors using equation (5) (similar to equation (2)) and may express the resulting weights as:

{ω _k } _{k＝1，…，25} (6)

Consider an example where quantization unit 52 selects the five largest weight values (i.e., weights with the largest or absolute values). The subset of weight values to be quantized can be represented as:

A weighted sum of codevectors to estimate the V-vector may be formed using a subset of weight values and their corresponding codevectors, as shown in the following expression:

Wherein Ω _j represents the j-th code vector in the subset of code vectors ({Ω _j }), represents the j-th weight in the subset of weights (), and corresponds to the estimated V-vector, which corresponds to the V-vector decomposed and/or decoded by quantization unit 52. The right side of expression (1) may represent a weighted sum of code vectors including a set of weights () and a set of code vectors ({Ω _j }).

Quantization unit 52 may quantize a subset of the weight values to produce quantized weight values, which may be represented as:

A weighted sum of codevectors representing quantized versions of the estimated V-vectors may be formed using the quantized weight values and their corresponding codevectors, as shown in the following expression:

Wherein Ω _j represents the j-th code vector in the subset of code vectors ({Ω _j }), represents the j-th weight in the subset of weights (), and corresponds to the estimated V-vector, which corresponds to the V-vector decomposed and/or decoded by quantization unit 52. The right side of expression (1) may represent a weighted sum of the subset of code vectors including a set of weights () and a set of code vectors ({Ω _j }).

An alternative restatement of the foregoing (which is largely equivalent to the one described above) can be as follows. V-vectors can be coded based on a set of predefined code vectors. To code a V-vector, each V-vector is decomposed into a weighted sum of code vectors. The weighted sum of code vectors consists of k pairs of predefined code vectors and associated weights:

Where _Ωj represents the jth code vector in a set of predefined code vectors ({ _Ωj }), _ωj represents the jth real-valued weight in a set of predefined weights ({ _ωj }), k corresponds to the index of the addend (which can be up to 7), and V corresponds to the coded V-vector. The choice of k depends on the encoder. If the encoder selects the weighted sum of two or more code vectors, then the total number of predefined code vectors that the encoder can select is (N+1) ² , which are derived as HOA expansion coefficients from Tables F.3 to F.7 of the 3D audio standard (entitled "Information technology - High efficiency coding and media delivery in heterogeneous environments - Part 3: 3D audio", ISO/IEC JTC 1/SC 29/WG 11, dated July 25, 2014, and identified by document number ISO/IEC DIS 23008-3). When N is 4, the table with 32 predefined directions in Appendix F.5 of the above-referenced 3D audio standard is used. In all cases, the absolute value of the weight ω is vector-quantized with respect to the predefined weighting values found in the first k+1 columns of the table in Table F.12 of the above-referenced 3D audio standard and signaled by the associated row number index.

Decode the positive and negative signs of the weight ω as:

In other words, after signaling the value k, the V-vector is encoded by k+1 indices pointing to the k+1 predefined code vectors {Ω _j }, an index pointing to the k quantized weights in the predefined weighted codebook, and k+1 digital sign values s _j :

If the encoder selects a weighted sum of code vectors, a codebook derived from Table F.8 of the aforementioned 3D audio standard is used in conjunction with the absolute weight values in Table F.11 of the aforementioned 3D audio standard, both of which are shown below. Furthermore, the numerical signs of the weight values ω may be individually coded. Quantization unit 52 may signal which of the aforementioned codebooks described in Tables F.3 through F.12 is used to code the input V-vector using the codebook index syntax element (hereinafter referred to as "CodebkIdx"). Quantization unit 52 may also scalar quantize the input V-vector to produce an output scalar quantized V-vector without Huffman coding the scalar quantized V-vector. Quantization unit 52 may further scalar quantize the input V-vector according to a Huffman coded scalar quantization mode to produce a Huffman coded scalar quantized V-vector. For example, quantization unit 52 may scalar quantize the input V-vector to produce a scalar quantized V-vector, and Huffman code the scalar quantized V-vector to produce an output Huffman coded scalar quantized V-vector.

In some examples, quantization unit 52 may perform a form of predicted vector quantization. Quantization unit 52 may identify whether vector quantization is predicted (as identified by one or more bits indicating a quantization mode, e.g., an NbitsQ syntax element) by specifying one or more bits in the bitstream 21 indicating whether prediction for vector quantization is performed (e.g., a PFlag syntax element).

To illustrate predicted vector quantization, quantization unit 42 may be configured to receive weight values (e.g., weight value magnitudes) corresponding to a code vector-based decomposition of a vector (e.g., a v-vector), generate predictive weight values based on the received weight values and based on reconstructed weight values (e.g., weight values reconstructed from one or more previous or subsequent audio frames), and vector quantize the set of predictive weight values. In some cases, each weight value in a set of predictive weight values may correspond to a weight value included in the code vector-based decomposition of a single vector.

Quantization unit 52 may receive the weight value and a weighted, reconstructed weight value obtained from a previous or subsequent decoding of the vector. Quantization unit 52 may generate a predictive weight value based on the weight value and the weighted, reconstructed weight value. Quantization unit 52 may subtract the weighted, reconstructed weight value from the weight value to generate a predictive weight value. The predictive weight value may alternatively be referred to as, for example, a residual, a prediction residual, a residual weight value, a weight value difference, an error, or a prediction error.

The weight value may be represented as | _wi,j |, which is the magnitude (or absolute value) of the corresponding weight value wi _,j . Thus, the weight value may alternatively be referred to as a weight value magnitude or as a weight value magnitude. The weight value wi _,j corresponds to the jth weight value from an ordered subset of weight values for the i-th audio frame. In some examples, the ordered subset of weight values may correspond to a subset of weight values in a code vector-based decomposition of a vector (e.g., a v-vector), which is sorted based on the magnitude of the weight values (e.g., sorted from largest magnitude to smallest magnitude).

The weighted reconstructed weight values may include entries corresponding to the magnitudes (or absolute values) of the corresponding reconstructed weight values. The reconstructed weight values correspond to the jth reconstructed weight value from the ordered subset of reconstructed weight values for the (i-1)th audio frame. In some examples, the ordered subset (or set) of reconstructed weight values may be generated based on quantized predictive weight values corresponding to the reconstructed weight values.

Quantization unit 42 also includes a weighting factor α _j . In some examples, α _j =1, in which case the weighted reconstructed weight value may be reduced to . In other examples, α _j ≠1. For example, α _j may be determined based on the following equation:

where I corresponds to the number of audio frames used to determine α _j . As shown in the previous equation, in some examples, the weighting factor may be determined based on a plurality of different weight values from a plurality of different audio frames.

Furthermore, when configured to perform predicted vector quantization, quantization unit 52 may generate predictive weight values based on the following equation:

where e _i,j corresponds to the predictive weight value of the jth weight value from the ordered subset of weight values for the i-th audio frame.

Quantization unit 52 generates quantized predictive weight values based on the predictive weight values and a predicted vector quantization (PVQ) codebook. For example, quantization unit 52 may vector quantize the predictive weight values in conjunction with other predictive weight values generated for the vector to be coded or for the frame to be coded to generate quantized predictive weight values.

Quantization unit 52 may vector-quantize predictive weight values 620 based on a PVQ codebook. The PVQ codebook may include a plurality of M-component candidate quantization vectors, and quantization unit 52 may select one of the candidate quantization vectors to represent the Z predictive weight values. In some examples, quantization unit 52 may select a candidate quantization vector from the PVQ codebook that minimizes quantization error (e.g., minimizes a least square error).

In some examples, the PVQ codebook may include multiple entries, wherein each of the entries includes a quantization codebook index and a corresponding M-component candidate quantization vector. Each of the indices in the quantization codebook may correspond to a respective one of a plurality of M-component candidate quantization vectors.

The number of components in each of the quantization vectors may depend on the number of weights selected to represent a single v-vector (i.e., Z). In general, for a codebook with Z-component candidate quantization vectors, quantization unit 52 may simultaneously quantize the Z predictive weight value vectors to produce a single quantized vector. The number of entries in the quantization codebook may depend on the bit rate used to quantize the weight value vectors.

When quantization unit 52 quantizes the predictive weight value vector, quantization unit 52 may select a Z-component vector from the PVQ codebook to be the quantized vector representing the Z predictive weight values. The quantized predictive weight value may be represented as a j-th component of the Z-component quantized vector for the i-th audio frame, which may further correspond to a vector quantized version of the j-th predictive weight value for the i-th audio frame.

When configured to perform predicted vector quantization, the quantization unit 52 may also generate a reconstructed weight value based on the quantized predictive weight value and the weighted reconstructed weight value. For example, the quantization unit 52 may add the weighted reconstructed weight value to the quantized predictive weight value to generate a reconstructed weight value. The weighted reconstructed weight value may be the same as the weighted reconstructed weight value described above. In some examples, the weighted reconstructed weight value may be a weighted and delayed version of the reconstructed weight value.

The reconstructed weight values may be represented as their magnitudes (or absolute values) corresponding to the corresponding reconstructed weight values. The reconstructed weight values correspond to the j-th reconstructed weight value from the ordered subset of reconstructed weight values for the (i-1)-th audio frame. In some examples, quantization unit 52 may separately encode data indicating the signs of the predictively coded weight values, and the decoder may use this information to determine the signs of the reconstructed weight values.

Quantization unit 52 may generate a reconstructed weight value based on the following equation:

wherein αj corresponds to the quantized predictive weight value of the jth weight value from the ordered subset of weight values for the i-th audio frame (e.g., the jth component of the M-component quantization vector), αj corresponds to the magnitude of the reconstructed weight value of the jth weight value from the ordered subset of weight values for the (i-1)-th audio frame, and _αj corresponds to the weighting factor of the jth weight value from the ordered subset of weight values.

The quantization unit 52 may generate a delayed reconstructed weight value based on the reconstructed weight value.For example, the quantization unit 52 may delay the reconstructed weight value by an audio frame to generate the delayed reconstructed weight value.

Quantization unit 52 may also generate a weighted reconstructed weight value based on the delayed reconstructed weight value and the weighting factor. For example, quantization unit 52 may multiply the delayed reconstructed weight value by the weighting factor to generate a weighted reconstructed weight value.

Similarly, quantization unit 52 may generate a weighted reconstructed weight value based on the delayed reconstructed weight value and the weighting factor. For example, quantization unit 52 may multiply the delayed reconstructed weight value by the weighting factor to generate a weighted reconstructed weight value.

In response to selecting a Z-component vector from the PVQ codebook that is to be the quantized vector for the Z predictive weight values, in some examples, quantization unit 52 may code an index (from the PVQ codebook) corresponding to the selected Z-component vector (rather than coding the selected Z-component vector itself). The index may indicate a set of quantized predictive weight values. In such examples, decoder 24 may include a codebook similar to the PVQ codebook and may decode the index indicating the quantized predictive weight values by mapping the index to the corresponding Z-component vector in the decoder codebook. Each of the components in the Z-component vector may correspond to a quantized predictive weight value.

Quantizing a vector (e.g., a V-vector) scalar-wise can involve quantizing each of the components of the vector individually and/or independently of the other components. For example, consider the following example V-vector:

V＝[0.23 0.31 -0.47 … 0.85]

To scalar quantize this example V vector, each of the components may be individually quantized (i.e., scalar quantized). For example, if the quantization step size is 0.1, the 0.23 component may be quantized to 0.2, the 0.31 component may be quantized to 0.3, and so on. The scalar quantized components may collectively form a scalar quantized V-vector.

In other words, quantization unit 52 may perform uniform scalar quantization on all elements of a given vector in the reduced foreground V[k] vector 55. Quantization unit 52 may identify a quantization step size based on the value of the NbitsQ syntax element, which may be represented as a quantization step size. Quantization unit 52 may dynamically determine this NbitsQ syntax element based on the target bit rate 41. The NbitsQ syntax element may also identify a quantization mode, as referenced in the ChannelSideInfoData syntax table reproduced below, while also identifying a step size (for scalar quantization purposes). That is, quantization unit 52 may determine the quantization step size based on this NbitsQ syntax element. As an example, quantization unit 52 may determine the quantization step size (denoted as a "delta" or "Δ" in this disclosure) to be equal to 2 ^{16 - NbitsQ} . In this example, when the value of the NbitsQ syntax element is 6, the delta is equal to 2 ¹⁰ and there are 2 ⁶ levels of quantization. In this regard, for a vector element v, the quantized vector element _vq is equal to [v/Δ], and ^-2NbitsQ-1 < _vq < ^2NbitsQ-1 .

Quantization unit 52 may then perform classification and residual coding of the quantized vector elements. As an example, quantization unit 52 may, for a given quantized vector element _vq , identify the category to which this element corresponds (by determining the category identifier cid) using the following equation:

Quantization unit 52 may then Huffman code this class index cid while also identifying the sign bit that indicates whether _vq is positive or negative. Quantization unit 52 may then identify the residual in this class. As an example, quantization unit 52 may determine this residual according to the following equation:

Residue = |v _q |-2 ^cid-1

Quantization unit 52 may then block code this residue with cid-1 bits.

In some examples, when coding the cid, quantization unit 52 may select different Huffman codebooks for different values of the NbitsQ syntax element. In some examples, quantization unit 52 may provide different Huffman coding tables for NbitsQ syntax element values of 6, ..., 15. Furthermore, quantization unit 52 may include five different Huffman codebooks for each of the different NbitsQ syntax element values in the range of 6, ..., 15, for a total of 50 Huffman codebooks. In this regard, quantization unit 52 may include multiple different Huffman codebooks to accommodate coding of the cid in a number of different statistical contexts.

For illustration, quantization unit 52 may include, for each of the NbitsQ syntax element values, a first Huffman codebook for coding vector elements one through four, a second Huffman codebook for coding vector elements five through nine, and a third Huffman codebook for coding vector elements nine and above. These first three Huffman codebooks may be used when the reduced foreground V[k] vector 55 to be compressed is not predicted from a corresponding reduced foreground V[k] vector 55 that is temporally subsequent in the reduced foreground V[k] vector 55 and does not represent spatial information of a synthetic audio object (e.g., an audio object originally defined by a pulse code modulated (PCM) audio object). When this one of the reduced foreground V[k] vectors 55 is predicted from a corresponding reduced foreground V[k] vector 55 that is temporally subsequent in the reduced foreground V[k] vectors 55, the quantization unit 52 may additionally include, for each of the NbitsQ syntax element values, a fourth Huffman codebook for coding that one of the reduced foreground V[k] vectors 55. When this one of the reduced foreground V[k] vectors 55 represents a synthetic audio object, the quantization unit 52 may also include, for each of the NbitsQ syntax element values, a fifth Huffman codebook for coding that one of the reduced foreground V[k] vectors 55. Various Huffman codebooks may be developed for each of such different statistical contexts, i.e., in this example, unpredicted and non-synthesized contexts, predicted contexts, and synthetic contexts.

The following table illustrates the Huffman table selection and the bits to be specified in the bitstream to enable the decompression unit to select the appropriate Huffman table:

Pred Mode HT Information HT table 0 0 HT5 0 1 HT{1,2,3} 1 0 HT4 1 1 HT5

In the preceding table, the prediction mode ("Pred Mode") indicates whether prediction is performed for the current vector, and the Huffman table ("HT Information") indicates additional Huffman codebook (or table) information used to select one of Huffman tables 1 to 5. The prediction mode can also be represented as the PFlag syntax element discussed below, while the HT Information can be represented by the CbFlag syntax element discussed below.

The following table further illustrates this Huffman table selection process (given various statistical contexts or situations).

Record synthesis No Pred HT{1,2,3} HT5 With Pred HT4 HT5

In the previous table, the "Recorded" column indicates the decoding context when the vector represents a recorded audio object, while the "Synthesized" column indicates the decoding context when the vector represents a synthetic audio object. The "Without Pred" row indicates the decoding context when prediction is not performed on the vector elements, while the "With Pred" row indicates the decoding context when prediction is performed on the vector elements. As shown in this table, the quantization unit 52 selects HT{1,2,3} when the vector represents a recorded audio object and does not perform prediction on the vector elements. The quantization unit 52 selects HT5 when the audio object represents a synthetic audio object and does not perform prediction on the vector elements. The quantization unit 52 selects HT4 when the vector represents a recorded audio object and performs prediction on the vector elements. The quantization unit 52 selects HT5 when the audio object represents a synthetic audio object and performs prediction on the vector elements.

Quantization unit 52 may select one of the following for use as the output switched-quantized V-vector based on any combination of criteria discussed in this disclosure: an unpredicted vector-quantized V-vector, a predicted vector-quantized V-vector, a non-Huffman-coded scalar-quantized V-vector, and a Huffman-coded scalar-quantized V-vector. In some examples, quantization unit 52 may select a quantization mode from a set of quantization modes including a vector quantization mode and one or more scalar quantization modes, and quantize the input V-vector based on (or in accordance with) the selected mode. Quantization unit 52 may then provide a selected one of the following to bitstream generation unit 52 for use as the coded foreground V[k] vector 57: an unpredicted vector quantized V-vector (e.g., in terms of weight values or bits indicating weight values), a predicted vector quantized V-vector (e.g., in terms of error values or bits indicating error values), a unHuffman coded scalar quantized V-vector, and a Huffman coded scalar quantized V-vector. Quantization unit 52 may also provide a syntax element indicating a quantization mode (e.g., an NbitsQ syntax element), and any other syntax elements used to dequantize or otherwise reconstruct the V-vector (as discussed in more detail below with respect to the examples of Figures 4 and 7).

The parasitic audio decoder unit 40 included in the audio encoding device 20 may represent multiple executions of a parasitic audio decoder, each of which is used to encode a different audio object or HOA channel for each of the energy-compensated ambient HOA coefficients 47′ and the interpolated nFG signal 49′ to generate encoded ambient HOA coefficients 59 and encoded nFG signal 61. The parasitic audio decoder unit 40 may output the encoded ambient HOA coefficients 59 and the encoded nFG signal 61 to the bitstream generation unit 42.

The bitstream generation unit 42 included in the audio encoding device 20 represents a unit that formats data to conform to a known format (which may be a format known to the decoding device), thereby generating a vector-based bitstream 21. In other words, the bitstream 21 may represent the encoded audio data encoded in the manner described above. In some examples, the bitstream generation unit 42 may represent a multiplexer that may receive the coded foreground V[k] vector 57, the encoded ambient HOA coefficients 59, the encoded nFG signal 61, and the background channel information 43. The bitstream generation unit 42 may then generate the bitstream 21 based on the coded foreground V[k] vector 57, the encoded ambient HOA coefficients 59, the encoded nFG signal 61, and the background channel information 43. In this manner, the bitstream generation unit 42 may thereby specify the vector 57 in the bitstream 21 to obtain the bitstream 21, as described in more detail below with respect to the example of FIG. 7. The bitstream 21 may include a primary or main bitstream and one or more side channel bitstreams.

Although not shown in the example of FIG3 , the audio encoding device 20 may also include a bitstream output unit that switches the bitstream output from the audio encoding device 20 (e.g., switches between a direction-based bitstream 21 and a vector-based bitstream 21) based on whether the current frame is to be encoded using direction-based synthesis or vector-based synthesis. The bitstream output unit may perform the switching based on a syntax element output by the content analysis unit 26 indicating whether direction-based synthesis is to be performed (as a result of detecting that the HOA coefficients 11 are generated from a synthesized audio object) or vector-based synthesis is to be performed (as a result of detecting that the HOA coefficients are recorded). The bitstream output unit may specify the correct header syntax to indicate the switching or current encoding for the current frame and the corresponding bitstream in the bitstream 21.

In addition, as mentioned above, the sound field analysis unit 44 can identify BG _TOT ambient HOA coefficients 47, which can change on a frame-by-frame basis (but often the BG _TOT can remain constant or the same across two or more adjacent (in time) frames). Changes in _the BG _TOT can result in changes in the coefficients expressed in the reduced foreground V[k] vector 55. Changes in the BG _TOT can result in background HOA coefficients (which can also be referred to as "ambient HOA coefficients") that change on a frame-by-frame basis (but again, often the BG _TOT can remain constant or the same across two or more adjacent (in time) frames). The changes often result in changes in energy in terms of the sound field represented by the addition or removal of additional ambient HOA coefficients and the corresponding removal of coefficients from the reduced foreground V[k] vector 55 or the addition of coefficients to the reduced foreground V[k] vector 55.

Thus, the soundfield analysis unit (soundfield analysis unit 44) may further determine when the ambient HOA coefficients change from frame to frame and generate a flag or other syntax element (in terms of the ambient component used to represent the soundfield) indicating the change in the ambient HOA coefficients (wherein the change may also be referred to as a "transition" of the ambient HOA coefficients or as a "transition" of the ambient HOA coefficients. In detail, the coefficient reduction unit 46 may generate a flag (which may be denoted as an AmbCoeffTransition flag or an AmbCoeffIdxTransition flag) and provide the flag to the bitstream generation unit 42 so that the flag can be included in the bitstream 21 (possibly as part of the side channel information).

In addition to specifying the ambient coefficient transition flag, coefficient reduction unit 46 may also modify the manner in which the reduced foreground V[k] vector 55 is generated. In one example, when it is determined that one of the ambient HOA environment coefficients is in transition in the current frame, coefficient reduction unit 46 may specify a vector coefficient (which may also be referred to as a "vector element" or "element") for each of the V-vectors of the reduced foreground V[k] vector 55 that corresponds to the ambient HOA coefficient in transition. Likewise, the ambient HOA coefficient in transition may be added to or removed from the total BG _TOT number of background coefficients. Thus, the resulting change in the total number of background coefficients affects whether the ambient HOA coefficients are included or not included in the bitstream _, and whether the corresponding elements of the V-vectors are included for the V-vector specified in the bitstream in the second and third configuration modes described above. More information on how coefficient reduction unit 46 may specify scaled-down foreground V[k] vectors 55 to overcome changes in energy is provided in U.S. application Ser. No. 14/594,533, filed Jan. 12, 2015, entitled “TRANSITIONING OF AMBIENT HIGHER_ORDER AMBISONIC COEFFICIENTS.”

FIG4 is a block diagram illustrating the audio decoding device 24 of FIG2 in greater detail. As shown in the example of FIG4, the audio decoding device 24 may include an extraction unit 72, a directionality-based reconstruction unit 90, and a vector-based reconstruction unit 92. Although described below, more information regarding the audio decoding device 24 and various aspects of decompressing or otherwise decoding HOA coefficients can be found in International Patent Application Publication No. WO 2014/194099, filed May 29, 2014, entitled “NTERPOLATION FOR DECOMPOSED REPRESENTATIONS OF A SOUND FIELD.”

The extraction unit 72 may represent a unit configured to receive the bitstream 21 and extract various encoded versions of the HOA coefficients 11 (e.g., a direction-based encoded version or a vector-based encoded version). The extraction unit 72 may determine the syntax elements mentioned above that indicate whether the HOA coefficients 11 are encoded via various direction-based versions or vector-based versions. When direction-based encoding is performed, the extraction unit 72 may extract the direction-based version of the HOA coefficients 11 and the syntax elements associated with the encoded version (which are represented as direction-based information 91 in the example of FIG. 4 ), and pass the direction-based information 91 to the direction-based reconstruction unit 90. The direction-based reconstruction unit 90 may represent a unit configured to reconstruct the HOA coefficients in the form of HOA coefficients 11 ′ based on the direction-based information 91. The bitstream and the arrangement of syntax elements within the bitstream are described in more detail below with respect to the examples of FIG. 7A to 7J .

When the syntax element indicates that the HOA coefficients 11 are encoded using vector-based synthesis, the extraction unit 72 may extract the coded foreground V[k] vectors 57 (which may include the coded weights 57 and/or indices 63 or scalar-quantized V-vectors), the encoded ambient HOA coefficients 59, and the corresponding audio objects 61 (which may also be referred to as encoded nFG signals 61). The audio objects 61 each correspond to one of the vectors 57. The extraction unit 72 may pass the coded foreground V[k] vectors 57 to the V-vector reconstruction unit 74 and provide the encoded ambient HOA coefficients 59 and the encoded nFG signal 61 to the psychoacoustic decoding unit 80.

To extract the coded foreground V[k] vector 57, extraction unit 72 may extract syntax elements according to the following ChannelSideInfoData (CSID) syntax table.

Table - Syntax of ChannelSideInfoData(i)

The semantics for the preceding table are as follows.

This payload holds side information for the ith channel. The size and data of the payload depends on the type of channel.

ChannelType[i] This element stores the type of the i-th channel defined in Table 95.

ActiveDirsIds[i] This element indicates the direction of the active directional signal using the index of 900 predefined uniformly distributed points from Annex F.7. Codeword 0 is used to signal the end of the directional signal.

PFlag[i] The prediction flag associated with the vector-based signal of the i-th channel.

CbFlag[i] Codebook flag for Huffman decoding of scalar quantized V-vectors associated with the vector-based signal of the i-th channel.

CodebkIdx[i] Signaling the vector-based signal associated with the i-th channel A specific codebook is used to dequantize the vector quantized V-vector.

NbitsQ[i] This index identifies the Huffman table used for Huffman decoding of the data associated with the vector-based signal for the i-th channel. Codeword 5 specifies the use of a uniform 8-bit dequantizer. The two MSBs 00 specify the reuse of NbitsQ[i], PFlag[i], and CbFlag[i] data from the previous frame (k-1).

bA,bB The msb (bA) and second msb (bB) of the NbitsQ[i] field.

uintC The codeword for the remaining two bits of the NbitsQ[i] field.

NumVecIndices The number of vectors used to dequantize the vector quantized V-vector.

AddAmbHoaInfoChannel(i) This payload holds information for additional ambient HOA coefficients.

Based on the CSID syntax table, extraction unit 72 may first obtain a ChannelType syntax element indicating the type of channel (e.g., where a value of 0 signals a direction-based signal, a value of 1 signals a vector-based signal, and a value of 2 signals an additional ambient HOA signal). Based on the ChannelType syntax element, extraction unit 72 may switch between three conditions.

Focusing on case 1 to illustrate an example of the techniques described in this disclosure, extraction unit 72 may obtain the most significant bits of an NbitsQ syntax element (i.e., the bA syntax element in the example CSID syntax table above) and the second most significant bits of an NbitsQ syntax element (i.e., the bB syntax element in the example CSID syntax table above). The (k)[i] of NbitsQ(k)[i] may represent obtaining the NbitsQ syntax element for the k-th frame of the i-th transport channel. The NbitsQ syntax element may represent one or more bits indicating a quantization mode used to quantize the spatial components of the soundfield represented by the HOA coefficients 11. The spatial components may also be referred to in this disclosure as V-vectors or as coded foreground V[k] vectors 57.

In the example CSID syntax table above, the NbitsQ syntax element may include four bits to indicate one of 12 quantization modes used to compress the vector specified in the corresponding VVecData field (when values zero to three for the NbitsQ syntax element are reserved or not used). The 12 quantization modes include the following modes indicated below:

0-3: Reserved

4: Vector Quantization

5: Scalar quantization without Huffman decoding

6: 6-bit scalar quantization with Huffman decoding

7: 7-bit scalar quantization with Huffman decoding

8: 8-bit scalar quantization with Huffman decoding

… …

16: 16-bit scalar quantization with Huffman decoding

In the above, the value of the NbitsQ syntax element from 6 to 16 indicates not only that scalar quantization with Huffman coding is to be performed, but also indicates the quantization step size of the scalar quantization. In this regard, the quantization mode may include a vector quantization mode, a scalar quantization mode without Huffman coding, and a scalar quantization mode with Huffman coding.

Returning to the example CSID syntax table described above, extraction unit 72 may combine the bA syntax element and the bB syntax element, where this combination may be additive, as shown in the example CSID syntax table described above. The combined bA/bB syntax element may represent an indicator as to whether at least one syntax element from a previous frame indicating information used when compressing a vector is reused. Extraction unit 72 next compares the combined bA/bB syntax element to a value of zero. When the combined bA/bB syntax element has a value of zero, extraction unit 72 may determine that the quantization mode information for the current k-th frame of the i-th transport channel (i.e., the NbitsQ syntax element indicating the quantization mode in the example CSID syntax table described above) is the same as the quantization mode information for the k-1-th frame of the i-th transport channel. In other words, when set to a value of zero, the indicator indicates that the at least one syntax element from the previous frame is reused.

Extraction unit 72 similarly determines that the prediction information for the current k-th frame of the i-th transport channel (i.e., the PFlag syntax element indicating whether prediction is performed during vector quantization or scalar quantization in this example) is the same as the prediction information for the k-1-th frame of the i-th transport channel. Extraction unit 72 may also determine that the Huffman codebook information for the current k-th frame of the i-th transport channel (i.e., the CbFlag syntax element indicating the Huffman codebook used to reconstruct the V-vector) is the same as the Huffman codebook information for the k-1-th frame of the i-th transport channel. Extraction unit 72 may also determine that the vector quantization information for the current k-th frame of the i-th transport channel (i.e., the CodebkIdx syntax element indicating the vector quantization codebook used to reconstruct the V-vector and the NumVecIndices syntax element indicating the number of code vectors used to reconstruct the V-vector) is the same as the vector quantization information for the k-1-th frame of the i-th transport channel.

When the combined bA/bB syntax element does not have a value of zero, extraction unit 72 may determine that the quantization mode information, prediction information, Huffman codebook information, and vector quantization information for the kth frame of the i-th transport channel are not the same as those for the k-1th frame of the i-th transport channel. Therefore, extraction unit 72 may obtain the least significant bit of the NbitsQ syntax element (i.e., the uintC syntax element in the example CSID syntax table above) to combine the bA, bB, and uintC syntax elements to obtain the NbitsQ syntax element. Based on this NbitsQ syntax element, extraction unit 72 may obtain the PFlag, CodebkIdx, and NumVecIndices syntax elements when the NbitsQ syntax element signals vector quantization, or the PFlag and CbFlag syntax elements when the NbitsQ syntax element signals scalar quantization with Huffman coding. In this manner, extraction unit 72 may extract the aforementioned syntax elements used to reconstruct the V-vector and pass these syntax elements to vector-based reconstruction unit 72.

Extraction unit 72 may then extract the V-vector from the k-th frame of the i-th transport channel. Extraction unit 72 may obtain the HOADecoderConfig container application, which includes a syntax element denoted as CodedVVecLength. Extraction unit 72 may parse the CodedVVecLength from the HOADecoderConfig container application. Extraction unit 72 may obtain the V-vector according to the following VVecData syntax table.

VVec(k)[i] This vector is the V-vector of the kth HOAframe() for the i-th channel.

VVecLength This variable indicates the number of vector elements to be read.

VVecCoeffId This vector contains the indices of the transmitted V-vector coefficients.

VecVal An integer value between 0 and 255.

aVal Temporary variable used during decoding of VVectorData.

huffVal The Huffman codeword to be Huffman decoded.

SgnVal This sign is the coded sign value used during decoding.

intAddVal This symbol is an additional integer value used during decoding.

NumVecIndices is the number of vectors used to dequantize the vector quantized V-vector.

WeightIdx Index in WeightValCdbk used to dequantize the vector quantized V-vector.

nBitsW The field size used to read WeightIdx to decode the vector quantized V-vector.

WeightValCbk Codebook containing vectors of positive real-valued weight coefficients. Necessary only if NumVecIndices > 1. WeightValCdbk is provided with 256 entries.

WeightValPredCdbk Codebook containing the vector of predictive weight coefficients. Necessary only if NumVecIndices > 1. WeightValPredCdbk is provided with 256 entries.

WeightValAlpha Predictive decoding coefficient used for predictive decoding mode of V-vector quantization.

VvecIdx Index into the VecDict used to dequantize the vector quantized V-vector.

nbitsIdx Field size used to read VvecIdx to decode the vector quantized V-vector.

WeightVal is the real-valued weighting coefficient used to decode the vector quantized V-vector.

In the aforementioned syntax table, extraction unit 72 may determine whether the value of the NbitsQ syntax element is equal to four (or, in other words, signaling the use of vector dequantization to reconstruct the V-vector). When the value of the NbitsQ syntax element is equal to four, extraction unit 72 may compare the value of the NumVecIndices syntax element to the value of one. When the value of NumVecIndices is equal to one, extraction unit 72 may obtain the VecIdx syntax element. The VecIdx syntax element may represent one or more bits indicating the index of the VecDict used to dequantize the vector quantized V-vector. Extraction unit 72 may individualize the VecIdx array, with the zeroth element set to the value of the VecIdx syntax element plus one. Extraction unit 72 may also obtain the SgnVal syntax element. The SgnVal syntax element may represent one or more bits indicating the coded sign value used during decoding of the V-vector. Extraction unit 72 may individualize the WeightVal array, with the zeroth element set according to the value of the SgnVal syntax element.

When the value of the NumVecIndices syntax element is not equal to a value of one, extraction unit 72 may obtain a WeightIdx syntax element. The WeightIdx syntax element may represent one or more bits indicating an index into the WeightValCdbk array used to dequantize the vector-quantized V-vector. The WeightValCdbk array may represent a codebook containing a vector of positive real-valued weighting coefficients. Extraction unit 72 may then determine nbitsIdx based on the NumOfHoaCoeffs syntax element specified in the HOAConfig container application (specified at the beginning of the bitstream 21 as an example). Extraction unit 72 may then iterate over NumVecIndices, obtaining VecIdx syntax elements from the bitstream 21 and setting a VecIdx array element with each obtained VecIdx syntax element.

Extraction unit 72 does not perform the PFlag syntax comparison involving determining a tmpWeightVal variable value that is not relevant to extracting syntax elements from bitstream 21. Therefore, extraction unit 72 may next obtain the SgnVal syntax element for use in determining the WeightVal syntax element.

When the value of the NbitsQ syntax element is equal to five (signaling the use of scalar dequantization without Huffman decoding to reconstruct the V-vector), extraction unit 72 iterates from 0 to VVecLength, setting the aVal variable to the VecVal syntax element obtained from bitstream 21. The VecVal syntax element may represent one or more bits indicating an integer between 0 and 255.

When the value of the NbitsQ syntax element is equal to or greater than six (signaling the use of NbitsQ-bit scalar dequantization with Huffman decoding to reconstruct the V-vector), extraction unit 72 iterates from 0 to VVecLength, obtaining one or more of the huffVal, SgnVal, and intAddVal syntax elements. The huffVal syntax element may represent one or more bits indicating a Huffman codeword. The intAddVal syntax element may represent one or more bits indicating an additional integer value used during decoding. Extraction unit 72 may provide these syntax elements to vector-based reconstruction unit 92.

The vector-based reconstruction unit 92 may represent a unit configured to perform operations reciprocal to those described above with respect to the vector-based synthesis unit 27 in order to reconstruct the HOA coefficients 11′. The vector-based reconstruction unit 92 may include a V-vector reconstruction unit 74, a spatio-temporal interpolation unit 76, a foreground formulation unit 78, a psychoacoustic decoding unit 80, an HOA coefficient formulation unit 82, a fade unit 770, and a reordering unit 84. The dashed line of the fade unit 770 indicates that the fade unit 770 may be an optional unit for inclusion in the vector-based reconstruction unit 92.

The V-vector reconstruction unit 74 may represent a unit configured to reconstruct a V-vector from the encoded foreground V[k] vector 57. The V-vector reconstruction unit 74 may operate in a manner that is reciprocal to that of the quantization unit 52.

In other words, the V-vector reconstruction unit 74 may operate according to the following pseudo code to reconstruct the V-vector:

According to the aforementioned pseudocode, the V-vector reconstruction unit 74 may obtain the NbitsQ syntax element for the kth frame of the i-th transport channel. When the NbitsQ syntax element is equal to four (which again signals that vector quantization is being performed), the V-vector reconstruction unit 74 may compare the NumVecIndicies syntax element with one. As described above, the NumVecIndicies syntax element may represent one or more bits indicating the number of vectors used to dequantize the vector quantized V-vector. When the value of the NumVecIndicies syntax element is equal to one, the V-vector reconstruction unit 74 may then iterate from 0 to the value of the VVecLength syntax element, setting the idx variable to VVecCoeffId and setting the VVecCoeffId-th V-vector element (v ⁽ⁱ⁾ _{VVecCoeffId[m]} (k)) to WeightVal multiplied by the VecDict entry identified by [900][VecIdx[0]][idx]. In other words, when the value of NumVvecIndicies is equal to one, the vector codebook HOA expansion coefficients are derived from the codebook of 8×1 weight values shown in Table F.8 in conjunction with Table F.11.

When the value of the NumVecIndicies syntax element is not equal to one, the V-vector reconstruction unit 74 may set the cdbLen variable, which is a variable representing the number of vectors, to 0. The cdbLen syntax element indicates the number of entries in the dictionary or codebook for code vectors (where this dictionary is represented as "VecDict" in the pseudocode above and represents a codebook with cdbLen codebook entries containing a vector of HOA expansion coefficients used to decode the vector quantized V-vector). When the number of HOA coefficients 11 (represented by "N") is equal to four, the V-vector reconstruction unit 74 may set the cdbLen variable to 32. The V-vector reconstruction unit 74 may then iterate from 0 to 0, setting the TmpVVec array to zero. During this iteration, v-vector reconstruction unit 74 may also iterate from 0 to the value of the NumVecIndecies syntax element, setting the mth entry of the TempVVec array equal to the jth WeightVal multiplied by the [cdbLen][VecIdx[j]][m] entry of VecDict.

The V-vector reconstruction unit 74 may derive WeightVal according to the following pseudo code:

In the aforementioned pseudocode, the V-vector reconstruction unit 74 may iterate from 0 up to the value of the NumVecIndices syntax element, first determining whether the value of the PFlag syntax element is equal to 0. When the PFlag syntax element is equal to 0, the V-vector reconstruction unit 74 may determine the tmpWeightVal variable, thereby setting the tmpWeightVal variable equal to the [CodebkIdx][WeightIdx] entry of the WeightValCdbk codebook. When the value of the PFlag syntax element is not equal to 0, the V-vector reconstruction unit 74 may set the tmpWeightVal variable equal to the [CodebkIdx][WeightIdx] entry of the WeightValPredCdbk codebook plus the WeightValAlpha variable multiplied by tempWeightVal for the k-1th frame of the i-th transport channel. The WeightValAlpha variable may refer to the alpha value mentioned above, which may be statically defined at the audio encoding and decoding devices 20 and 24. V-vector reconstruction unit 74 may then obtain WeightVal based on the SgnVal syntax element obtained by extraction unit 72 and the tmpWeightVal variable.

In other words, the V-vector reconstruction unit 74 may derive weight values for each corresponding code vector used to reconstruct the V-vector based on a weight value codebook (denoted as "WeightValCdbk" for unpredicted vector quantization and "WeightValPredCdbk" for predicted vector quantization, both of which may represent multidimensional tables indexed based on one or more of a codebook index (denoted as the "CodebkIdx" syntax element in the aforementioned VVectorData(i) syntax table) and a weight index (denoted as the "WeightIdx" syntax element in the aforementioned VVectorData(i) syntax table). This CodebkIdx syntax element may be defined in a portion of the side channel information, as shown in the ChannelSideInfoData(i) syntax table below.

The remaining vector quantization portion of the pseudocode above involves calculating FNorm to normalize the elements of the V-vector, and then calculating the V-vector element (v ⁽ⁱ⁾ _{VVecCoeffId[m]} (k)) as TmpVVec[idx] multiplied by FNorm. The V-vector reconstruction unit 74 can obtain the idx variable based on VVecCoeffID.

When NbitsQ is equal to 5, uniform 8-bit scalar dequantization is performed. In contrast, NbitsQ values greater than or equal to 6 may result in the application of Huffman decoding. The cid value mentioned above may be equal to the two least significant bits of the NbitsQ value. The prediction mode is denoted as PFlag in the above syntax table, while the Huffman table information bits are denoted as CbFlag in the above syntax table. The remaining syntax specifies how decoding occurs in a manner substantially similar to that described above.

The psychoacoustic decoding unit 80 can operate in a manner reciprocal to the psychoacoustic audio decoder unit 40 shown in the example of FIG3 to decode the encoded ambient HOA coefficients 59 and the encoded nFG signal 61 and thereby generate energy-compensated ambient HOA coefficients 47′ and interpolated nFG signal 49′ (which may also be referred to as interpolated nFG audio object 49′). The psychoacoustic decoding unit 80 can pass the energy-compensated ambient HOA coefficients 47′ to the fade unit 770 and pass the nFG signal 49′ to the foreground formulation unit 78.

The spatial-temporal interpolation unit 76 may operate in a manner similar to that described above with respect to the spatial-temporal interpolation unit 50. The spatial-temporal interpolation unit 76 may receive the reduced foreground V[k] vector 55 _k and perform spatial-temporal interpolation on the foreground V[k] vector 55 _k and the reduced foreground V[k-1] vector 55 _k-1 to generate an interpolated foreground V[k] vector 55 _k ”. The spatial-temporal interpolation unit 76 may transfer the interpolated foreground V[k] vector 55 _k ” to the fade unit 770.

The extraction unit 72 may also output a signal 757 indicating when one of the ambient HOA coefficients is in transition to a fade unit 770, which may then determine which of the SHC _BG 47′ (wherein the SHC _BG 47′ may also be represented as “ambient HOA channel 47′” or “ambient HOA coefficient 47′”) and the elements of the interpolated foreground V[k] vector 55 _k ” to fade in or out. In some examples, the fade unit 770 may operate inversely with respect to each of the elements of the ambient HOA coefficient 47′ and the interpolated foreground V[k] vector 55 _k ”. That is, the fade unit 770 may perform a fade-in or fade-out or both fade-in or fade-out with respect to the corresponding ambient HOA coefficients in the ambient HOA coefficients 47 _′ , while performing a fade-in or fade-out or both fade-in and fade-out with respect to the corresponding interpolated foreground V[k] vectors in the elements of the interpolated foreground V[k] vector 55 k ”. The fade unit 770 may output the adjusted ambient HOA coefficients 47″ to the HOA coefficient formulation unit 82 and the adjusted foreground V[k] vector 55 _k ”′ to the foreground formulation unit 78. In this regard, the fade unit 770 represents a unit configured to perform fade operations with respect to various aspects of the HOA coefficients or their derivatives (e.g., in the form of the ambient HOA coefficients 47′ and the elements of the interpolated foreground V[k] vector 55 _k ”).

The foreground development unit 78 may represent a unit configured to perform matrix multiplication on the adjusted foreground V[k] vector _55k ' and the interpolated nFG signal 49' to produce the foreground HOA coefficients 65. In this regard, the foreground development unit 78 may combine the audio object 49' (which is another way of representing the interpolated nFG signal 49') and the vector _55k ' to reconstruct the foreground (or in other words, dominant) aspects of the HOA coefficients 11'. The foreground development unit 78 may perform a matrix multiplication of the interpolated nFG signal 49' by the adjusted foreground V[k] vector _55k '.

The HOA coefficient formulation unit 82 may represent a unit configured to combine the foreground HOA coefficients 65 into the adjusted ambient HOA coefficients 47″ in order to obtain the HOA coefficients 11′. The apostrophe notation reflects that the HOA coefficients 11′ may be similar to but not identical to the HOA coefficients 11. The difference between the HOA coefficients 11 and 11′ may result from losses due to transmission over a lossy transmission medium, quantization, or other lossy operations.

5A is a flowchart illustrating exemplary operation of an audio encoding device (e.g., the audio encoding device 20 shown in the example of FIG. 3 ) in performing various aspects of the vector-based synthesis techniques described in this disclosure. Initially, the audio encoding device 20 receives the HOA coefficients 11 (106). The audio encoding device 20 may invoke the LIT unit 30, which may apply the LIT with respect to the HOA coefficients to output transformed HOA coefficients (e.g., in the case of SVD, the transformed HOA coefficients may include the US[k] vector 33 and the V[k] vector 35) (107).

The audio encoding device 20 may then call the parameter calculation unit 32 to perform the above-described analysis to identify various parameters with respect to any combination of the US[k] vector 33, the US[k-1] vector 33, the V[k] and/or the V[k-1] vector 35 in the manner described above. That is, the parameter calculation unit 32 may determine at least one parameter based on the analysis of the transformed HOA coefficients 33/35 (108).

The audio encoding device 20 may then call the reordering unit 34, which reorders the transformed HOA coefficients (again, in the context of SVD, which may refer to the US[k] vector 33 and the V[k] vector 35) based on the parameters to produce reordered transformed HOA coefficients 33'/35' (or, in other words, the US[k] vector 33' and the V[k] vector 35'), as described above (109). During any of the foregoing operations or subsequent operations, the audio encoding device 20 may also call the soundfield analysis unit 44. As described above, the soundfield analysis unit 44 may perform soundfield analysis on the HOA coefficients 11 and/or the transformed HOA coefficients 33/35 to determine the total number of foreground channels (nFG) 45, the order of the background soundfield ( _NBG ), and the number (nBGa) and index (i) of additional BG HOA channels to be sent (which may be collectively represented as background channel information 43 in the example of FIG. 3) (109).

The audio encoding device 20 may also invoke a background selection unit 48. The background selection unit 48 may determine background or ambient HOA coefficients 47 based on the background channel information 43 (110). The audio encoding device 20 may further invoke a foreground selection unit 36, which may select a reordered US[k] vector 33' and a reordered V[k] vector 35' representing a foreground or unique component of the sound field based on nFG 45 (which may represent one or more indices identifying a foreground vector) (112).

The audio encoding device 20 may invoke the energy compensation unit 38. The energy compensation unit 38 may perform energy compensation on the ambient HOA coefficients 47 to compensate for energy loss due to removal of various ones of the HOA coefficients by the background selection unit 48 (114), and thereby generate energy-compensated ambient HOA coefficients 47'.

The audio encoding device 20 may also call the space-time interpolation unit 50. The space-time interpolation unit 50 may perform space-time interpolation on the reordered transformed HOA coefficients 33'/35' to obtain an interpolated foreground signal 49' (which may also be referred to as "interpolated nFG signal 49'") and residual foreground directional information 53 (which may also be referred to as "V[k] vector 53") (116). The audio encoding device 20 may then call the coefficient reduction unit 46. The coefficient reduction unit 46 may perform coefficient reduction on the residual foreground V[k] vector 53 based on the background channel information 43 to obtain reduced foreground directional information 55 (which may also be referred to as reduced foreground V[k] vector 55) (118).

The audio encoding device 20 may then invoke the quantization unit 52 to compress the downscaled foreground V[k] vector 55 in the manner described above and generate a coded foreground V[k] vector 57 (120).

The audio encoding device 20 may also invoke the psychoacoustic audio decoder unit 40. The psychoacoustic audio decoder unit 40 may psychoacoustically decode each vector of the energy-compensated ambient HOA coefficients 47′ and the interpolated nFG signal 49′ to generate encoded ambient HOA coefficients 59 and encoded nFG signal 61. The audio encoding device may then invoke the bitstream generation unit 42. The bitstream generation unit 42 may generate the bitstream 21 based on the coded foreground directional information 57, the coded ambient HOA coefficients 59, the coded nFG signal 61, and the background channel information 43.

FIG5B is a flowchart illustrating exemplary operation of an audio encoding device performing the decoding techniques described in this disclosure. The bitstream generation unit 42 of the audio encoding device 20 shown in the example of FIG3 may represent an example unit configured to perform the techniques described in this disclosure. The bitstream generation unit 42 may determine whether the quantization mode of the frame is the same as the quantization mode of the previous frame in time (which may be denoted as "second frame") (314). Although described with respect to the previous frame, the techniques may be performed with respect to subsequent frames in time. A frame may include a portion of one or more transport channels. The portion of the transport channel may include ChannelSideInfoData (formed according to the ChannelSideInfoData syntax table) and a payload (e.g., the VVectorData field 156 in the example of FIG7). Other examples of a payload may include an AddAmbientHOACoeffs field.

When the quantization modes are the same ("yes" 316), bitstream generation unit 42 may specify a portion of the quantization mode in bitstream 21 (318). The portion of the quantization mode may include a bA syntax element and a bB syntax element, but not a uintC syntax element. The bA syntax element may represent bits indicating the most significant bit of the NbitsQ syntax element. The bB syntax element may represent bits indicating the second most significant bit of the NbitsQ syntax element. Bitstream generation unit 42 may set the value of each of the bA and bB syntax elements to 0, thereby signaling that the quantization mode field (i.e., as an example, the NbitsQ field) in bitstream 21 does not include a uintC syntax element. This signaling of zero-valued bA and bB syntax elements also indicates that the NbitsQ value, PFlag value, CbFlag value, and CodebkIdx value from the previous frame are used as the corresponding values for the same syntax elements for the current frame.

When the quantization modes are not the same ("No" 316), bitstream generation unit 42 may specify one or more bits in bitstream 21 that indicate the entire quantization mode (320). That is, bitstream generation unit 42 may specify bA, bB, and uintC syntax elements in bitstream 21. Bitstream generation unit 42 may also specify quantization information based on the quantization mode (322). This quantization information may include any information about quantization, such as vector quantization information, prediction information, and Huffman codebook information. As an example, vector quantization information may include one or both of a CodebkIdx syntax element and a NumVecIndices syntax element. As an example, prediction information may include a PFlag syntax element. As an example, Huffman codebook information may include a CbFlag syntax element.

In this regard, the techniques described herein may enable the audio encoding device 20 to be configured to obtain a bitstream 21 comprising a compressed version of the spatial components of the sound field. The spatial components may be generated by performing vector-based synthesis on a plurality of spherical harmonic coefficients. The bitstream may further include an indicator of one or more bits from a previous frame indicating whether to reuse information from a header field specifying information used in compressing the spatial components.

In other words, the techniques may enable the audio encoding device 20 to be configured to obtain a bitstream 21 including vectors 57 representing orthogonal spatial axes in the spherical harmonic domain. The bitstream 21 may further include an indicator from a previous frame regarding whether to reuse at least one syntax element indicating information used in compressing (e.g., quantizing) the vector (e.g., a bA/bB syntax element of an NbitsQ syntax element).

FIG6A is a flowchart illustrating exemplary operation of an audio decoding device (e.g., audio decoding device 24 shown in FIG4 ) in performing various aspects of the techniques described in this disclosure. Initially, audio decoding device 24 may receive bitstream 21 (130). Upon receiving the bitstream, audio decoding device 24 may invoke extraction unit 72. Assuming for purposes of discussion that bitstream 21 indicates that vector-based reconstruction is to be performed, extraction unit 72 may parse the bitstream to retrieve the information mentioned above, passing the information to vector-based reconstruction unit 92.

In other words, the extraction unit 72 can extract the decoded foreground directional information 57 (which, again, may also be referred to as the decoded foreground V[k] vector 57), the decoded ambient HOA coefficients 59 and the decoded foreground signal (which may also be referred to as the decoded foreground nFG signal 59 or the decoded foreground audio object 59) from the bitstream 21 in the manner described above (132).

The audio decoding device 24 may further call a dequantization unit 74. The dequantization unit 74 may entropy decode and dequantize the coded foreground directional information 57 to obtain reduced foreground directional information _55k (136). The audio decoding device 24 may also call a psychoacoustic decoding unit 80. The psychoacoustic audio decoding unit 80 may decode the encoded ambient HOA coefficients 59 and the encoded foreground signal 61 to obtain energy-compensated ambient HOA coefficients 47' and an interpolated foreground signal 49' (138). The psychoacoustic decoding unit 80 may pass the energy-compensated ambient HOA coefficients 47' to the fade unit 770 and pass the nFG signal 49' to the foreground formulation unit 78.

The audio decoding device 24 may next call the spatio-temporal interpolation unit 76. The spatio-temporal interpolation unit 76 may receive the reordered foreground directional information 55 _k ′ and perform spatio-temporal interpolation on the reduced foreground directional information 55 _k /55 _k-1 to generate interpolated foreground directional information 55 _k ″(140). The spatio-temporal interpolation unit 76 may transfer the interpolated foreground V[k] vector 55 _k ″ to the fade unit 770.

The audio decoding device 24 may invoke a fade unit 770. The fade unit 770 may receive or otherwise obtain a syntax element (e.g., an AmbCoeffTransition syntax element) that indicates when the energy-compensated ambient HOA coefficients 47′ are in transition (e.g., from the extraction unit 72). The fade unit 770 may fade the energy-compensated ambient HOA coefficients 47′ in or out based on the transition syntax element and the maintained transition state information, thereby outputting the adjusted ambient HOA coefficients 47″ to the HOA coefficient formulation unit 82. The fade unit 770 may also fade out or in the corresponding one or more elements in the interpolated foreground V[k] vector _55k ″ based on the syntax element and the maintained transition state information, thereby outputting the adjusted foreground V[k] vector _55k ′″ to the foreground formulation unit 78 (142).

The audio decoding device 24 may call the foreground formulation unit 78. The foreground formulation unit 78 may perform a matrix multiplication of the nFG signal 49′ by the adjusted foreground direction information 55 _k ″′ to obtain the foreground HOA coefficients 65 (144). The audio decoding device 24 may also call the HOA coefficient formulation unit 82. The HOA coefficient formulation unit 82 may add the foreground HOA coefficients 65 to the adjusted ambient HOA coefficients 47″ to obtain the HOA coefficients 11′ (146).

FIG6B is a flowchart illustrating exemplary operation of an audio decoding device performing the coding techniques described in this disclosure. Extraction unit 72 of audio encoding device 24 shown in the example of FIG4 may represent an example unit configured to perform the techniques described in this disclosure. Bitstream extraction unit 72 may obtain bits (362) indicating whether the quantization mode of a frame is the same as the quantization mode of a temporally previous frame (which may be denoted as "second frame"). Furthermore, although described with respect to a previous frame, the techniques may be performed with respect to a temporally subsequent frame.

When the quantization modes are the same ("YES" 364), extraction unit 72 may obtain a portion of the quantization mode from bitstream 21 (366). The portion of the quantization mode may include the bA syntax element and the bB syntax element, but not the uintC syntax element. Extraction unit 42 may also set the values of the NbitsQ value, PFlag value, CbFlag value, CodebkIdx value, and NumVertIndices value for the current frame to be the same as the values of the NbitsQ value, PFlag value, CbFlag value, CodebkIdx value, and NumVertIndices value set for the previous frame (368).

When the quantization modes are not the same ("No" 364), extraction unit 72 may obtain one or more bits indicating the entire quantization mode from bitstream 21. That is, extraction unit 72 obtains the bA, bB, and uintC syntax elements from bitstream 21 (370). Extraction unit 72 may also obtain one or more bits indicating quantization information based on the quantization mode (372). As mentioned above with respect to FIG. 5B, the quantization information may include any information related to quantization, such as vector quantization information, prediction information, and Huffman codebook information. As an example, the vector quantization information may include one or both of the CodebkIdx syntax element and the NumVecIndices syntax element. As an example, the prediction information may include the PFlag syntax element. As an example, the Huffman codebook information may include the CbFlag syntax element.

In this regard, the techniques described herein may enable the audio decoding device 24 to be configured to obtain a bitstream 21 comprising a compressed version of the spatial components of the sound field. The spatial components may be generated by performing vector-based synthesis on a plurality of spherical harmonic coefficients. The bitstream may further include an indicator of one or more bits from a previous frame indicating whether to reuse information from a header field specifying information used in compressing the spatial components.

In other words, the techniques may enable the audio decoding device 24 to be configured to obtain a bitstream 21 including vectors 57 representing orthogonal spatial axes in the spherical harmonic domain. The bitstream 21 may further include an indicator from a previous frame regarding whether to reuse at least one syntax element indicating information used in compressing (e.g., quantizing) the vector (e.g., a bA/bB syntax element of an NbitsQ syntax element).

FIG7 is a diagram illustrating example frames 249S and 249T specified according to various aspects of the techniques described in this disclosure. As shown in the example of FIG7 , frame 249S includes ChannelSideInfoData (CSID) fields 154A-154D, HOAGainCorrectionData (HOAGCD) fields, VVectorData fields 156A and 156B, and a HOAPredictionInfo field. CSID field 154A includes a uintC syntax element ("uintC") 267 set to a value of 10, a bb syntax element ("bB") 266 set to a value of 1, a bA syntax element ("bA") 265 set to a value of 0, and a ChannelType syntax element ("ChannelType") 269 set to a value of 01.

The uintC syntax element 267, the bB syntax element 266, and the bA syntax element 265 together form an NbitsQ syntax element 261, where the bA syntax element 265 forms the most significant bit, the bB syntax element 266 forms the second most significant bit, and the uintC syntax element 267 forms the least significant bit of the NbitsQ syntax element 261. As mentioned above, the NbitsQ syntax element 261 may represent one or more bits indicating a quantization mode (e.g., one of a vector quantization mode, a scalar quantization mode without Huffman coding, and a scalar quantization mode with Huffman coding) used to encode the high-order ambisonics audio data.

The CSID syntax element 154A also includes the PFlag syntax element 300 and the CbFlag syntax element 302 referenced above in the various syntax tables. The PFlag syntax element 300 may represent one or more bits indicating whether the coded elements of the spatial components of the soundfield represented by the HOA coefficients 11 of the first frame 249S (where again, the spatial components may refer to V-vectors) are predicted from a second frame (e.g., in this example, the previous frame). The CbFlag syntax element 302 may represent one or more bits indicating Huffman codebook information, which may identify which of the Huffman codebooks (or, in other words, tables) is used to encode the elements of the spatial components (or, in other words, V-vector elements).

7 , each of which is set to the corresponding values of 0, 0, and 01. Each of CSID fields 154C and 154D includes a ChannelType field 269 having a value of 3 (11 ₂ ). Each of CSID fields 154A-154D corresponds to a respective transport channel of transport channels 1, 2, 3, and 4. In practice, each CSID field 154A-154D indicates whether the corresponding payload is a direction-based signal (when the corresponding ChannelType is equal to zero), a vector-based signal (when the corresponding ChannelType is equal to one), additional ambient HOA coefficients (when the corresponding ChannelType is equal to two), or a null value (when ChannelType is equal to three).

In the example of FIG7 , frame 249S includes two vector-based signals (when the given ChannelType syntax element 269 is equal to 1 in CSID fields 154A and 154B) and two null values (when the given ChannelType 269 is equal to 3 in CSID fields 154C and 154D). Furthermore, the prediction used by the audio encoding device 20, as indicated by the PFlag syntax element 300, is set to one. Furthermore, the prediction, as indicated by the PFlag syntax element 300, refers to a prediction mode indication indicating whether prediction is performed on corresponding compressed spatial components of the compressed spatial components v1 through vn. When the PFlag syntax element 300 is set to one, the audio encoding device 20 may use prediction by taking the difference between vector elements from the previous frame and corresponding vector elements of the current frame for scalar quantization, or the difference between weights from the previous frame and corresponding weights of the current frame for vector quantization.

The audio encoding device 20 also determines that the value of the NbitsQ syntax element 261 for the CSID field 154B for the second transport channel in frame 249S is the same as the value of the NbitsQ syntax element 261 for the CSID field 154B for the second transport channel of the previous frame (e.g., frame 249T in the example of FIG. 7 ). Therefore, the audio encoding device 20 specifies a value of zero for each of the bA syntax element 265 and the bB syntax element 266 to signal that the value of the NbitsQ syntax element 261 for the second transport channel in the previous frame 249T is reused for the NbitsQ syntax element 261 for the second transport channel in frame 249S. Therefore, the audio encoding device 20 can avoid specifying the uintC syntax element 267 for the second transport channel in frame 249S, as well as the other syntax element identified above.

FIG8 is a diagram illustrating an example frame of one or more channels of at least one bitstream according to the techniques described herein. Bitstream 450 includes frames 810A through 810H, each of which may include one or more channels. Bitstream 450 may be an example of bitstream 21 shown in the example of FIG7 . In the example of FIG8 , audio decoding device 24 maintains state information, thereby updating the state information to determine how to decode the current frame k. Audio decoding device 24 may utilize state information from configuration 814 and frames 810B through 810D.

In other words, the audio encoding device 20 may include, for example, a state machine 402 within the bitstream generation unit 42 that maintains state information for encoding each of the frames 810A-810E because the bitstream generation unit 42 may specify syntax elements for each of the frames 810A-810E based on the state machine 402.

The audio decoding device 24 may likewise include, for example, a similar state machine 402 within the bitstream extraction unit 72, which outputs syntax elements based on the state machine 402 (some of which are not explicitly specified in the bitstream 21). The state machine 402 of the audio decoding device 24 may operate in a manner similar to that of the state machine 402 of the audio encoding device 20. Thus, the state machine 402 of the audio decoding device 24 may maintain state information, thereby updating the state information based on the configuration 814 (and, in the example of FIG. 8 , the decoding of frames 810B through 810D). Based on the state information, the bitstream extraction unit 72 may extract frame 810E based on the state information maintained by the state machine 402. The state information may provide a number of implicit syntax elements that the audio encoding device 20 may utilize when decoding various transport channels of frame 810E.

The aforementioned techniques can be performed with respect to any number of different contexts and audio ecosystems. Several example contexts are described below, but the techniques should be limited to these example contexts. An example audio ecosystem may include audio content, a film studio, a music studio, a game audio studio, channel-based audio content, a decoding engine, game audio stems, a game audio decoding/rendering engine, and a delivery system.

Film studios, music studios, and game audio studios may receive audio content. In some examples, the audio content may represent acquired output. A film studio may output channel-based audio content (e.g., in 2.0, 5.1, and 7.1), for example, using a digital audio workstation (DAW). A music studio may output channel-based audio content (e.g., in 2.0 and 5.1), for example, using a DAW. In either case, a decoding engine may receive and encode the channel-based audio content based on one or more codecs (e.g., AAC, AC3, Dolby True HD, Dolby Digital Plus, and DTS Master Audio) for output by a delivery system. A game audio studio may output one or more game audio stems, for example, using a DAW. A game audio decoding/rendering engine may decode the audio stems and/or render the audio stems into channel-based audio content for output by a delivery system. Another example context in which the techniques may be performed includes an audio ecosystem that may include broadcast recorded audio objects, professional audio systems, consumer on-device capture, HOA audio formats, on-device presentation, consumer audio, TV and accessories, and automotive audio systems.

Broadcast recorded audio objects, professional audio systems, and capture on consumer devices can all use the HOA audio format to encode their output. In this way, the HOA audio format can be used to encode audio content into a single representation that can be played back using on-device presentation, consumer audio, TV and accessories, and car audio systems. In other words, a single representation of the audio content can be played back on a universal audio playback system (i.e., as opposed to requiring a specific configuration such as 5.1, 7.1, etc.) (e.g., audio playback system 16).

Other examples of contexts in which the techniques may be implemented include audio ecosystems that may include acquisition elements and playback elements. The acquisition elements may include wired and/or wireless acquisition devices (e.g., Eigen microphones), on-device surround sound capturers, and mobile devices (e.g., smartphones and tablets). In some examples, the wired and/or wireless acquisition devices may be coupled to the mobile devices via wired and/or wireless communication channels.

According to one or more techniques of this disclosure, a mobile device may be used to acquire a sound field. For example, the mobile device may acquire the sound field via a wired and/or wireless acquisition device and/or an on-device surround sound capturer (e.g., multiple microphones integrated into the mobile device). The mobile device may then decode the acquired sound field into HOA coefficients for playback by one or more of the playback elements. For example, a user of the mobile device may record (capture the sound field) a live event (e.g., a rally, conference, game, concert, etc.) and decode the recording into HOA coefficients.

The mobile device may also utilize one or more of the playback elements to playback the HOA decoded sound field. For example, the mobile device may decode the HOA decoded sound field and output a signal to one or more of the playback elements that causes the one or more playback elements to recreate the sound field. As an example, the mobile device may utilize a wireless and/or wireless communication channel to output the signal to one or more speakers (e.g., a speaker array, a sound bar, etc.). As another example, the mobile device may utilize a docking solution to output the signal to one or more docking stations and/or one or more docked speakers (e.g., a sound system in a smart car and/or home). As another example, the mobile device may utilize a headphone presentation to output the signal to a set of headphones (e.g., to create actual binaural sound).

In some examples, a particular mobile device may acquire a 3D sound field and replay the same 3D sound field at a later time. In some examples, a mobile device may acquire a 3D sound field, encode the 3D sound field into an HOA, and transmit the encoded 3D sound field to one or more other devices (e.g., other mobile devices and/or other non-mobile devices) for playback.

Yet another context in which the techniques may be performed includes an audio ecosystem that may include audio content, a game studio, decoded audio content, a rendering engine, and a delivery system. In some examples, a game studio may include one or more DAWs that may support editing of HOA signals. For example, the one or more DAWs may include HOA plug-ins and/or tools that may be configured to operate (e.g., work) with one or more game audio systems. In some examples, a game studio may output a new stem format that supports HOA. In any case, the game studio may output the decoded audio content to a rendering engine that may render the sound field for playback by the delivery system.

The techniques may also be performed with respect to an exemplary audio acquisition device. For example, the techniques may be performed with respect to an Eigen microphone, which may include multiple microphones collectively configured to record a 3D sound field. In some examples, the multiple microphones of the Eigen microphone may be located on the surface of a substantially spherical ball having a radius of approximately 4 cm. In some examples, the audio encoding device 20 may be integrated into the Eigen microphone so as to output the bitstream 21 directly from the microphone.

Another exemplary audio acquisition context may include a production truck that may be configured to receive signals from one or more microphones (eg, one or more Eigen microphones). The production truck may also include an audio encoder, such as audio encoder 20 of FIG.

In some cases, the mobile device may also include multiple microphones that are collectively configured to record a 3D sound field. In other words, the multiple microphones may have X, Y, Z diversity. In some examples, the mobile device may include a microphone that can be rotated to provide X, Y, Z diversity with respect to one or more other microphones of the mobile device. The mobile device may also include an audio encoder, such as audio encoder 20 of FIG. 3 .

The ruggedized video capture device can be further configured to record a 3D sound field. In some examples, the ruggedized video capture device can be attached to a helmet of a user participating in an activity. For example, the ruggedized video capture device can be attached to a user's helmet while the user is whitewater rafting. In this way, the ruggedized video capture device can capture a 3D sound field representing the action around the user (e.g., water crashing behind the user, another whitewater rafter speaking in front of the user, etc.).

The techniques may also be performed with respect to an accessory-enhanced mobile device that can be configured to record a 3D sound field. In some examples, the mobile device may be similar to the mobile device discussed above, with one or more accessories added. For example, an Eigen microphone may be attached to the mobile device mentioned above to form an accessory-enhanced mobile device. In this way, the accessory-enhanced mobile device can capture a higher-quality version of the 3D sound field (compared to the case of using only the sound capture components that are integral to the accessory-enhanced mobile device).

Example audio playback devices that can perform various aspects of the techniques described in this disclosure are discussed further below. According to one or more techniques of this disclosure, speakers and/or soundbars can be arranged in any arbitrary configuration while still reproducing a 3D sound field. Furthermore, in some examples, a headphone playback device can be coupled to decoder 24 via a wired or wireless connection. According to one or more techniques of this disclosure, a single, universal representation of a soundfield can be utilized to render the soundfield on any combination of speaker, soundbar, and headphone playback devices.

Several different example audio playback environments may also be suitable for performing various aspects of the techniques described in this disclosure. For example, the following environments may be suitable environments for performing various aspects of the techniques described in this disclosure: a 5.1 speaker playback environment, a 2.0 (e.g., stereo) speaker playback environment, a 9.1 speaker playback environment with full-height front speakers, a 22.2 speaker playback environment, a 16.0 speaker playback environment, an automotive speaker playback environment, and a mobile device with an over-the-ear headphone playback environment.

According to one or more techniques of this disclosure, a single universal representation of a sound field can be utilized to render the sound field on any of the aforementioned playback environments. Additionally, the techniques of this disclosure enable a renderer to render the sound field from the universal representation for playback on playback environments other than the environments described above. For example, if design considerations prohibit proper placement of speakers according to a 7.1 speaker playback environment (e.g., if placement of a right surround speaker is not possible), the techniques of this disclosure enable the renderer to compensate by using the other six speakers so that playback can be achieved on a 6.1 speaker playback environment.

In addition, a user can watch a sports game while wearing a headset. According to one or more techniques of the present invention, a 3D sound field of a sports game can be obtained (for example, one or more Eigen microphones can be placed in and/or around a baseball field), HOA coefficients corresponding to the 3D sound field can be obtained and transmitted to a decoder, which can reconstruct the 3D sound field based on the HOA coefficients and output the reconstructed 3D sound field to a renderer, which can obtain an indication of the type of playback environment (for example, a headset), and present the reconstructed 3D sound field so that the headset outputs a signal representing the 3D sound field of the sports game.

In each of the various scenarios described above, it should be understood that the audio encoding device 20 may perform a method or otherwise include means for performing each step of the method that the audio encoding device 20 is configured to perform. In some cases, the device may include one or more processors. In some cases, the one or more processors may represent specialized processors configured with the aid of instructions stored on a non-transitory computer-readable storage medium. In other words, various aspects of the techniques in each of the array encoding examples may provide a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause the one or more processors to perform the method that the audio encoding device 20 has been configured to perform.

In one or more examples, the described functions can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media such as data storage media. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and/or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

Likewise, in each of the various scenarios described above, it should be understood that the audio decoding device 24 may perform a method or otherwise include means for performing each step of the method that the audio decoding device 24 is configured to perform. In some cases, the means may include one or more processors. In some cases, the one or more processors may represent specialized processors configured with the aid of instructions stored on a non-transitory computer-readable storage medium. In other words, various aspects of the technology in each of the array encoding examples may provide a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause the one or more processors to perform the method that the audio decoding device 24 has been configured to perform.

By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transitory, tangible storage media. As used herein, disk and optical disk include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), magnetic disk, and Blu-ray disc, where magnetic disks typically reproduce data magnetically, while optical discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor," as used herein, may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated into a combined codec. Moreover, the techniques may be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units. Rather, as described above, the various units may be combined with appropriate software and/or firmware in a codec hardware unit or provided by a collection of interoperable hardware units, including one or more processors as described above.

Various aspects of the technology have been described. These and other aspects of the technology are within the scope of the following claims.

Claims (52)

1. An efficient bit usage method, the method comprising: A compressed version of the bitstream is obtained, comprising spatial components of the sound field, the spatial components of which are represented by vectors representing orthogonal spatial axes in the spherical harmonic domain, wherein the bitstream further comprises an indicator of whether to reuse a syntax element from the previous frame that indicates a prediction mode, the prediction mode indicating whether to perform prediction with respect to the vector. 2. The method of claim 1, wherein the syntax element is a first syntax element and the indicator includes one or more bits of a second syntax element, the second syntax element indicating the quantization mode used when compressing the vector. 3. The method of claim 2, wherein when set to zero, the one or more bits of the second syntax element indicate reuse of the first syntax element from the previous frame. 4. The method according to claim 2, wherein the quantization mode includes vector quantization mode. 5. The method according to claim 2, wherein the quantization mode includes a scalar quantization mode without Huffman decoding. 6. The method of claim 2, wherein the quantization mode includes a scalar quantization mode with Huffman decoding. 7. The method of claim 2, wherein the indicator comprises the most significant bit of the second syntax element and the second most significant bit of the second syntax element. 8. The method of claim 1, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating a Huffman table used when compressing the vector. 9. The method of claim 1, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating a category identifier that identifies the compression category corresponding to the vector. 10. The method of claim 1, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating whether the elements of the vector are positive or negative. 11. The method of claim 1, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating the number of code vectors used when compressing the vector. 12. The method of claim 1, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating the vector quantization codebook used when compressing the vector. 13. The method of claim 1, wherein the compressed version of the vector in the bit stream is represented at least partially using Huffman codes to represent the residual values of the elements of the vector. 14. The method of claim 1, further comprising: Decompose the high-order stereo reverberation audio data to obtain the vector; and Specify the vector in the bit stream to obtain the bit stream. 15. The method of claim 1, further comprising: Obtain the audio object corresponding to the vector from the bitstream; and The audio object is combined with the vector to reconstruct high-order stereo reverberant audio data. 16. The method of claim 1, wherein the compression of the vector comprises the quantization of the vector. 17. An apparatus configured to perform efficient bit usage, the apparatus comprising: One or more processors configured to obtain a compressed version of a bitstream comprising spatial components of a sound field, the spatial components of which are represented by vectors representing orthogonal spatial axes in a spherically harmonic domain, wherein the bitstream further comprises an indicator regarding whether to reuse a syntax element from a previous frame indicating a prediction mode, the prediction mode indicating whether to perform prediction with respect to the vector; and A memory configured to store the bit stream. 18. The apparatus of claim 17, wherein the syntax element is a first syntax element and the indicator includes one or more bits of a second syntax element, the second syntax element indicating the quantization mode used when compressing the vector. 19. The apparatus of claim 18, wherein when set to a zero value, the one or more bits of the second syntax element indicate reuse of the first syntax element from the previous frame. 20. The apparatus of claim 18, wherein the quantization mode includes a vector quantization mode. 21. The apparatus of claim 18, wherein the quantization mode comprises a scalar quantization mode without Huffman decoding. 22. The apparatus of claim 18, wherein the quantization mode comprises a scalar quantization mode with Huffman decoding. 23. The apparatus of claim 18, wherein the indicator comprises the most significant bit of the second syntax element and the second most significant bit of the second syntax element. 24. The apparatus of claim 17, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating a Huffman table used when compressing the vector. 25. The apparatus of claim 17, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating a category identifier that identifies the compression category corresponding to the vector. 26. The apparatus of claim 17, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating whether the elements of the vector are positive or negative. 27. The apparatus of claim 17, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating the number of code vectors used when compressing the vector. 28. The apparatus of claim 17, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating a vector quantization codebook used when compressing the vector. 29. The apparatus of claim 17, wherein the compressed version of the vector in the bit stream is represented at least partially using Huffman codes to represent residual values of the elements of the vector. 30. The apparatus of claim 17, wherein the one or more processors are further configured to decompose high-order stereo reverberation audio data to obtain the vector, and to specify the vector in the bitstream to obtain the bitstream. 31. The apparatus of claim 17, wherein the one or more processors are further configured to obtain an audio object corresponding to the vector from the bit stream, and to combine the audio object with the vector to reconstruct higher-order stereo reverberation audio data. 32. The apparatus of claim 17, wherein the compression of the vector comprises the quantization of the vector. 33. An apparatus configured to perform efficient bit usage, the apparatus comprising: A means for obtaining a compressed version of a bitstream comprising spatial components of a sound field, the spatial components of which are represented by vectors representing orthogonal spatial axes in a spherically harmonic domain, wherein the bitstream further comprises an indicator regarding whether to reuse a syntax element from a previous frame indicating a prediction mode, the prediction mode indicating whether to perform prediction with respect to the vector; and A means for storing the indicator. 34. The apparatus of claim 33, wherein the syntax element is a first syntax element and the indicator includes one or more bits of a second syntax element, the second syntax element indicating the quantization mode used when compressing the vector. 35. The apparatus of claim 34, wherein when set to a zero value, the one or more bits of the second syntax element indicate reuse of the first syntax element from the previous frame. 36. The apparatus of claim 34, wherein the quantization mode includes a vector quantization mode. 37. The apparatus of claim 34, wherein the quantization mode comprises a scalar quantization mode without Huffman decoding. 38. The apparatus of claim 34, wherein the quantization mode comprises a scalar quantization mode with Huffman decoding. 39. The apparatus of claim 34, wherein the indicator comprises the most significant bit of the second syntax element and the second most significant bit of the second syntax element. 40. The apparatus of claim 33, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating a Huffman table used when compressing the vector. 41. The apparatus of claim 33, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating a category identifier that identifies the compression category corresponding to the vector. 42. The apparatus of claim 33, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating whether the elements of the vector are positive or negative. 43. The apparatus of claim 33, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating the number of code vectors used when compressing the vector. 44. The apparatus of claim 33, wherein the syntax element is a first syntax element and the indicator indicates whether a second syntax element from the previous frame is reused, the second syntax element indicating the vector quantization codebook used when compressing the vector. 45. The apparatus of claim 33, wherein the compressed version of the vector in the bit stream is represented at least partially using Huffman codes to represent residual values of the elements of the vector. 46. The apparatus of claim 33, further comprising: A means for decomposing high-order stereo reverberation audio data to obtain the vector; and A means for specifying the vector in the bit stream to obtain the bit stream. 47. The apparatus of claim 33, further comprising: A means for obtaining an audio object corresponding to the vector from the bit stream; and A means for combining the audio object with the vector to reconstruct higher-order stereo reverberation audio data. 48. The apparatus of claim 33, wherein the compression of the vector comprises the quantization of the vector. 49. A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to perform the following operations: A compressed version of a bitstream is obtained, comprising spatial components of a sound field, the spatial components of which are represented by vectors representing orthogonal spatial axes in a spherical harmonic domain, wherein the bitstream further comprises an indicator of at least one syntax element indicating whether a prediction mode from a previous frame is reused, the prediction mode indicating whether prediction is performed with respect to the vector. 50. An apparatus configured to perform efficient bit usage, the apparatus comprising: One or more processors configured to obtain a compressed version of a bitstream comprising spatial components of a sound field, the spatial components of which are represented by vectors representing orthogonal spatial axes in a spherically harmonic domain, wherein the bitstream further includes indicators regarding whether syntax elements from the previous frame are reused, the syntax elements indicating the Huffman table used when compressing the vectors; and A memory configured to store the bit stream. 51. An apparatus configured to perform efficient bit usage, the apparatus comprising: One or more processors configured to obtain a compressed version of a bitstream comprising spatial components of a sound field, the spatial components of which are represented by vectors representing orthogonal spatial axes in a spherically harmonic domain, wherein the bitstream further comprises an indicator regarding whether syntax elements from a previous frame are reused, the syntax elements indicating the vector quantization codebook used when compressing the vectors; and A memory configured to store the bit stream. 52. An apparatus configured to perform efficient bit usage, the apparatus comprising: One or more processors configured to obtain a compressed version of a bitstream comprising spatial components of a sound field, the spatial components of which are represented by vectors representing orthogonal spatial axes in a spherically harmonic domain, wherein the bitstream further comprises an indicator regarding whether a syntax element from a previous frame is reused, the syntax element indicating the quantization mode used when compressing the vector, the indicator comprising one or more bits of the syntax element; and A memory configured to store the bit stream.