HK1248910B - System and method for capturing, encoding, distributing, and decoding immersive audio - Google Patents

Description

Systems and methods for capturing, encoding, distributing, and decoding immersive audio

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Patent Application No. 62/110,211, filed January 30, 2015, entitled “System and Method for Capturing and Encoding a 3-D Audio Soundfield,” both of which are incorporated herein by reference in their entireties.

Background Art

As dedicated recording devices become more portable and more affordable, and as recording capabilities become more prevalent in everyday devices (such as smartphones), the capture of audio content (often in combination with video) has become increasingly common. The quality of video capture has continued to improve and has surpassed the quality of audio capture. Video capture on modern mobile devices is typically high-resolution and DSP-intensive, but the accompanying audio content is typically captured in mono with low fidelity and little additional processing.

To capture spatial cues, many existing audio recording technologies employ at least two microphones. As a general rule, recording a 360-degree horizontal surround audio scene requires at least three audio channels, while recording a three-dimensional audio scene requires at least four audio channels. While multi-channel audio capture is used for immersive audio recording, currently available more popular consumer audio delivery technologies and distribution frameworks are limited to transmitting two-channel audio. In standard two-channel stereo reproduction, the stored or transmitted left and right audio channels are intended to be played back directly on left and right loudspeakers or headphones, respectively.

In order to play back an immersive audio recording, it may be necessary to render the recorded spatial audio information in a variety of playback configurations. These playback configurations include headphones, front soundbar speakers, front discrete speaker pairs, 5.1 horizontal surround speaker arrays, and three-dimensional speaker arrays including height channels. Regardless of the playback configuration, it is desirable to reproduce a spatial audio scene for the listener that is a substantially accurate representation of the captured audio scene. In addition, it would be advantageous to provide an audio storage or transmission format that is agnostic to a particular playback configuration.

One such configuration-insensitive format is the B-format. The B-format includes the following signals: (1) W—a pressure signal corresponding to the output of an omnidirectional microphone; (2) X—front-to-back directional information corresponding to the output of a forward-pointing "figure-of-eight" microphone; (3) Y—side-to-side directional information corresponding to the output of a left-pointing "figure-of-eight" microphone; and (4) Z—up-to-down directional information corresponding to the output of an upward-pointing "figure-of-eight" microphone.

B-format audio signals can be spatially decoded for immersive audio playback on headphones or flexible loudspeaker configurations. B-format signals can be obtained directly or derived from a standard near-coincident microphone arrangement comprising omnidirectional and/or bidirectional microphones or unidirectional microphones. In particular, the 4-channel A-format is obtained from a tetrahedral arrangement of cardioid microphones and can be converted to B-format via a 4×4 linear matrix. In addition, the 4-channel B-format can be converted to a two-channel ambisonic UHJ format that is compatible with standard 2-channel stereo reproduction. However, the two-channel ambisonic UHJ format is not sufficient to enable faithful three-dimensional immersive audio or horizontal surround reproduction.

Other methods have been proposed for encoding multiple audio channels representing surround or immersive sound scenes into a reduced-data format for storage and/or distribution, which can then be decoded to enable faithful reproduction of the original audio scene. One such method is time-domain phase-amplitude matrix encoding/decoding. The encoder in this method linearly combines input channels with specific amplitude and phase relationships into a smaller set of encoded channels. The decoder combines the encoded channels with specific amplitude and phase to attempt to restore the original channels. However, due to the reduced intermediate channel count, the spatial localization fidelity of the reproduced audio scene may be lost compared to the original audio scene.

A method for improving the spatial localization fidelity of the reproduced audio scene is frequency-domain phase-amplitude matrix decoding, which decomposes a matrix-encoded two-channel audio signal into a time-frequency representation. The method then spatializes each time-frequency component separately. The time-frequency decomposition provides a high-resolution representation of the input audio signal in which the individual sources are represented more discretely than in the time domain. As a result, the method can improve the spatial fidelity of the subsequently decoded signal when compared to time-domain matrix decoding.

Another approach to data reduction for multi-channel audio representations is spatial audio coding. In this approach, the input channels are combined into a reduced-channel format (possibly even mono), and some side information about the spatial characteristics of the audio scene is also included. Parameters in the side information can be used to decode the reduced-channel format into a multi-channel signal that faithfully approximates the original audio scene spatially.

The phase-amplitude matrix coding and spatial audio coding methods described above are often used to encode multi-channel soundtracks created in recording studios. Furthermore, they sometimes require that the audio signal encoded with reduced channels be a viable listening alternative to the fully decoded version. This is to ensure that direct playback is an option and eliminates the need for a custom decoder.

Sound field coding is a similar attempt at spatial audio coding that focuses on capturing and encoding the "live" audio scene and accurately reproducing it through a playback system. Existing methods for sound field coding rely on specific microphone configurations to accurately capture directional sources. Moreover, they rely on various analysis techniques to properly handle directional and diffuse sources. However, the microphone configurations required for sound field coding are often impractical for consumer devices. Modern consumer devices often have significant design constraints imposed on the number and location of microphones, which can result in configurations that do not match the requirements for current sound field coding methods. Sound field analysis methods are also often computationally intensive and lack the scalability to support lower complexity implementations.

Summary of the Invention

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Embodiments of sound field coding systems and methods relate to the processing of audio signals, and more specifically to the capture, encoding, and reproduction of three-dimensional (3-D) audio sound fields. Embodiments of the systems and methods are used to capture a 3-D sound field representing an immersive audio scene. The capture is performed using an arbitrary microphone array configuration. For efficient storage and distribution, the captured audio is encoded into a common spatially encoded signal (SES) format. In some embodiments, the method for spatially decoding the SES format for reproduction is insensitive to the microphone array configuration used to capture the audio in the 3-D sound field.

Currently, there is no end-to-end system that enables the flexible capture, distribution, and reproduction of immersive audio recordings encoded in a universal digital audio format that is compatible with standard two-channel and multi-channel reproduction systems. In particular, because employing standard multi-channel microphone array configurations is impractical in consumer mobile devices (such as smartphones or cameras), there is a need for methods for spatially encoding two-channel or multi-channel immersive audio signals from flexible multi-channel microphone array configurations that are compatible with conventional playback systems.

Embodiments of the system and method include processing multiple microphone signals by selecting a microphone configuration having multiple microphones for capturing a 3-D sound field. The microphones are used to capture sound from at least one audio source. The microphone configuration defines a microphone directivity for each of the multiple microphones used in the audio capture. The microphone directivity is defined relative to a reference direction.

Embodiments of the system and method also include selecting a virtual microphone configuration comprising a plurality of microphones. The virtual microphone configuration is used to encode spatial information about the position of an audio source relative to a reference direction. The system and method also include calculating spatial coding coefficients based on the microphone configuration and the virtual microphone configuration. The spatial coding coefficients are used to convert the microphone signals into spatially coded signals (SES). The SES includes virtual microphone signals, wherein the virtual microphone signals are obtained by combining the microphone signals using the spatial coding coefficients.

It should be noted that alternative embodiments are possible, and the steps and elements discussed herein can be changed, added, or eliminated depending on the specific embodiment. Without departing from the scope of the present invention, these alternative embodiments include alternative steps and alternative elements that can be used and structural changes that can be made.

BRIEF DESCRIPTION OF THE DRAWINGS

Referring now to the drawings, wherein like reference numerals designate corresponding parts throughout:

FIG1 is an overview block diagram of an embodiment of a sound field coding system according to the present invention.

2A is a block diagram illustrating details of the capture component, encoding component, and distribution component of the embodiment of the sound field encoding system shown in FIG. 1 .

2B is a block diagram illustrating an embodiment of a portable capture device having microphones arranged in a non-standard configuration.

3 is a block diagram showing details of the decoding and playback components of the embodiment of the soundfield encoding system shown in FIG. 1 .

Fig. 4 shows a general block diagram of an embodiment of a sound field coding system according to the present invention.

FIG5 is a block diagram depicting an embodiment of a system similar to that described in FIG4 in greater detail, where T=2.

FIG. 6 is a block diagram illustrating the spatial decoder and renderer shown in FIG. 5 in more detail.

FIG. 7 is a block diagram illustrating a spatial encoder in a case with T=2 transmission signals and without auxiliary information.

FIG. 8 is a block diagram illustrating an alternative embodiment of the spatial encoder shown in FIG. 7 .

FIG. 9A shows a specific example embodiment of a spatial encoder in which an A-format signal is captured and converted to a B-format from which a 2-channel spatially encoded signal is derived.

FIG9B shows directivity patterns of the B-format W component, X component, and Y component in the horizontal plane.

FIG. 9C shows directivity patterns of three supercardioid virtual microphones derived by combining the B-format W component, X component, and Y component.

FIG. 10 illustrates an alternative embodiment of the system shown in FIG. 9A , in which the B-format signal is converted to a 5-channel surround sound signal.

11 illustrates an alternative embodiment of the system shown in FIG. 9A , in which the B-format signal is converted to a Directional Audio Coding (DirAC) representation.

FIG. 12 is a block diagram depicting an embodiment of a system similar to the system described in FIG. 11 in more detail.

FIG13 is a block diagram illustrating yet another embodiment of a spatial encoder that transforms a B-format signal into the frequency domain and encodes it into a 2-channel stereo signal.

FIG14 is a block diagram illustrating an embodiment of a spatial encoder in which an input microphone signal is first decomposed into a direct component and a diffuse component.

15 is a block diagram illustrating an embodiment of a spatial encoding system and method including a wind noise detector.

FIG. 16 shows a system for capturing N microphone signals and converting them to an M-channel format suitable for editing before spatial encoding.

FIG17 illustrates an embodiment of a system and method by which a captured audio scene is modified as part of the spatial decoding process.

18 is a flow chart illustrating the general operation of an embodiment of a capture component of a soundfield encoding system according to the present invention.

DETAILED DESCRIPTION

In the following description of embodiments of sound field coding systems and methods, reference is made to the accompanying drawings. These drawings show, by way of illustration, specific examples of how embodiments of the systems and methods may be implemented. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the claimed subject matter.

I. System Overview

Embodiments of the sound field coding systems and methods described herein are used to capture a sound field representing an immersive audio scene using an arbitrary microphone array configuration. For efficient storage and distribution, the captured audio is encoded into a universal spatial encoding signal (SES) format. In a preferred embodiment of the present invention, the method for spatially decoding the SES format for reproduction is insensitive to the microphone array configuration used. Storage and distribution can be implemented using existing methods for two-channel audio (e.g., commonly used digital media distribution or streaming networks). The SES format can be played back on a standard two-channel stereo reproduction system, or alternatively, reproduced with high spatial fidelity on a flexible playback configuration (if an appropriate SES decoder is available). The SES coding format enables the following spatial decoding: the spatial decoding is configured to achieve faithful reproduction of the original immersive audio scene in various playback configurations (e.g., headphones or surround sound systems).

Embodiments of the sound field coding systems and methods provide flexible and scalable techniques for capturing a three-dimensional sound field with an arbitrary microphone configuration and encoding the three-dimensional sound field. This differs from existing methods in that no specific microphone configuration is required. In addition, the SES coding format described herein is feasible for high-quality two-channel playback that does not require a spatial decoder. This differs from other three-dimensional sound field coding methods (such as Ambisonics B-format or DirAC) in that they generally do not involve providing faithful immersive 3-D audio playback directly from the encoded audio signal. Moreover, these coding methods may not be able to provide high-quality playback without including auxiliary information in the encoded signal. For embodiments of the systems and methods described herein, the auxiliary information is optional.

Capture, encoding and distribution systems

FIG1 is an overview block diagram of an embodiment of a sound field coding system 100. System 100 includes a capture component 110, a distribution component 120, and a playback component 130. In the capture component, an input microphone or preferably a microphone array receives an audio signal. Capture component 110 receives microphone signals 135 from various microphone configurations. By way of example, these configurations include mono, stereo, 3-microphone surround, 4-microphone periphonic (such as high-fidelity stereo B format) or any microphone configuration. First symbol 138 shows that any one of the microphone signal formats can be selected as input. Microphone signal 135 is input to audio capture component 140. In some embodiments of system 100, audio capture component 140 processes microphone signal 135 to remove undesirable ambient noise (such as static background noise or wind noise).

The captured audio signals are input to a spatial encoder 145. These audio signals are spatially encoded into a spatially encoded signal (SES) format suitable for subsequent storage and distribution. The SES is then passed to the storage/transmission component 150 of the distribution component 120. In some embodiments, the storage/transmission component 150 encodes the SES with an audio waveform encoder (such as MP3 or AAC) to reduce storage requirements or transmission data rates without modifying the spatial cues encoded in the SES. In the distribution component 120, the audio is stored or provided to a playback device via a distribution network.

In playback component 130, various playback devices are depicted. As depicted by second symbol 152, any one of the playback devices can be selected. Figure 1 shows a first playback device 155, a second playback device 160, and a third playback device 165. For the first playback device 155, SES is spatially decoded for optimal playback through headphones. For the second playback device 160, SES is spatially decoded for optimal playback through a stereo system. For the third playback device 165, the SES signal is spatially decoded for optimal playback through a multi-channel loudspeaker system. In general use scenarios, as will be understood by those skilled in the art and as shown in the following figures, audio capture, distribution, and playback can occur in conjunction with video.

FIG2A is a block diagram illustrating the details of the capture component 110 of the sound field coding system 100 shown in FIG1 . In the capture component 110 , the recording device supports both a four-microphone array connected to a first audio capture subcomponent 200 and a two-microphone array connected to a second audio capture subcomponent 210 . The outputs of the first audio capture subcomponent 200 and the second audio capture subcomponent 210 are provided to a first spatial encoder subcomponent 220 and a second spatial encoder subcomponent 230 , respectively, where they are encoded into a spatially encoded signal (SES) format. It should be noted that embodiments of the system 100 are not limited to two-microphone or four-microphone arrays. In other cases, other microphone configurations will be similarly supported using appropriate spatial encoders. In some embodiments, an audio bitstream encoder 240 encodes the SES generated by the first spatial encoder subcomponent 220 or by the second spatial encoder subcomponent 230. The encoded signal output from the encoder 240 is packaged into an audio bitstream 250.

In some embodiments, video is included in the capture component 110. As shown in FIG2A , the video capture component 260 captures a video signal, and the video encoder 270 encodes the video signal to produce a video bitstream. An A/V multiplexer 280 multiplexes an audio bitstream 250 with an associated video bitstream. The multiplexed audio and video bitstreams are stored or transmitted in the storage/transmission component 150 of the distribution component 120. The bitstream data can be temporarily stored as a data file on a capture device, on a local media server, or on a computer network and made available for transmission or distribution.

In some embodiments, the first audio capture subcomponent 200 captures an audiophile B-format signal, and the SES encoding performed by the first spatial encoder subcomponent 220 performs conventional B-format to UHJ two-channel stereo encoding, as described, for example, in "Ambisonics in multichannel broadcasting and video," by Michael Gerzon, JAES, Vol. 33, No. 11, pp. 859–871, November 1985. In an alternative embodiment, the first spatial encoder subcomponent 220 performs frequency-domain spatial encoding of the B-format signal to a two-channel SES, which, unlike the two-channel UHJ format, can preserve three-dimensional spatial audio cues. In yet another embodiment, the microphones connected to the first audio capture subcomponent 200 are arranged in a non-standard configuration.

FIG2B is a diagram illustrating an embodiment of a portable capture device 201 having microphones arranged in a non-standard configuration. The portable capture device 201 in FIG2B includes microphones 202, 203, 204, and 205 for audio capture and a camera 206 for video capture. In portable devices (such as smart phones), the positioning of the microphones on the device 201 may be subject to industrial design considerations or other constraints. Due to such constraints, microphones 202, 203, 204, and 205 may be configured in a manner that is not a standard microphone configuration (such as a recording microphone configuration recognized by those skilled in the art). In fact, the configuration may be specific to a particular capture device. FIG2B merely provides an example of such a device-specific configuration. It should be noted that various other embodiments are possible and are not limited to this particular microphone configuration. In addition, embodiments of the present invention are applicable to any microphone configuration.

In an alternative embodiment, only two microphone signals are captured (by the second audio capture subcomponent 210) and spatially encoded (by the second spatial encoder subcomponent 230). This limitation to two microphone channels may occur, for example, when there is a product design decision to minimize the manufacturing cost of the device. In this case, the fidelity of the spatial information encoded in the SES may be correspondingly compromised. For example, the SES may lack up-to-down or front-to-back distinction cues. However, in a preferred embodiment of the present invention, for the same original captured sound field, the left-to-right distinction cues encoded in the SES generated from the second spatial encoder subcomponent 230 are substantially equivalent to the left-to-right distinction cues encoded in the SES generated from the first spatial encoder subcomponent 220 (as perceived by a listener in a standard two-channel stereo playback configuration). Therefore, regardless of the capture microphone array configuration, the SES format remains compatible with standard two-channel stereo reproduction.

In some embodiments, the first spatial encoder subcomponent 220 also generates spatial audio auxiliary information or metadata that is included in the SES. This auxiliary information, in some embodiments, is derived from frequency-domain analysis of the inter-channel relationships between the captured microphone signals. Such spatial audio auxiliary information is incorporated into the audio bitstream by the audio bitstream encoder 240 and subsequently stored or transmitted so that it can be optionally retrieved and utilized in the playback component to optimize spatial audio reproduction fidelity.

More generally, in some embodiments, the digital audio bitstream generated by the audio bitstream encoder 240 is formatted to include a two-channel or multi-channel backward-compatible audio downmix signal and optional extensions (referred to herein as "side information") that may include metadata and additional audio channels. Examples of such audio encoding formats are described in U.S. patent application US2014-0350944 A1, entitled "Encoding and reproduction of three dimensional audio soundtracks," which is incorporated herein by reference in its entirety.

While it is often useful to perform spatial encoding before multiplexing the audio and video (for legacy and compatibility purposes) as depicted in FIG2A , in other embodiments, the original captured multi-channel audio signal can be multiplexed with the video "as is," and SES encoding can occur at some later stage in the delivery chain. For example, spatial encoding (including optional auxiliary information extraction) can be performed offline on a network-based computer. This approach can allow for more advanced signal analysis calculations than would be possible if the spatial encoding calculations were implemented on the original recording device processor.

In some embodiments, the two-channel SES encoded by the audio bitstream encoder 240 contains spatial audio cues captured in the original sound field. In some embodiments, the audio cues are in the form of inter-channel amplitude and phase relationships (within the fidelity limits imposed by the geometry of the microphone array and the number of microphones) that are substantially insensitive to the particular microphone array configuration employed on the capture device. The two-channel SES can be later decoded by extracting the encoded spatial audio cues and rendering an audio signal that is optimal for reproducing the spatial cues representing the original audio scene via the available playback device.

FIG3 is a block diagram illustrating the details of the playback component 130 of the sound field coding system 100 shown in FIG1 . The playback component 130 receives a media bitstream from the storage/transmission component 150 of the distribution component 120. In embodiments where the received bitstream includes both an audio bitstream and a video bitstream, these bitstreams are demultiplexed by an A/V demultiplexer 300. The video bitstream is provided to a video decoder 310 for decoding and playback on a monitor 320. The audio bitstream is provided to an audio bitstream decoder 330, which restores the original encoded SES accurately or in a form that preserves the spatial clues encoded in the SES. For example, in some embodiments, the audio bitstream decoder 330 includes an audio waveform decoder that is reciprocal to the audio waveform encoder optionally included in the audio bitstream encoder 240.

In some embodiments, the decoded SES output from the decoder 330 includes a two-channel stereo signal compatible with standard two-channel stereo reproduction. This signal can be provided directly to a legacy playback system 340 (such as a pair of loudspeakers) without further decoding or processing (except for digital-to-analog conversion and amplification of the respective left and right audio signals). As previously described, the backward-compatible stereo signal included in the SES enables it to provide a viable reproduction of the original captured audio scene on the legacy playback system 340. In alternative embodiments, the legacy playback system 340 can be a multi-channel playback system (such as a 5.1 or 7.1 surround sound reproduction system) and the decoded SES provided by the audio bitstream decoder 330 can include a multi-channel signal directly compatible with the legacy playback system 340.

In embodiments where the decoded SES is provided directly to a two-channel or multi-channel legacy playback system 340, any auxiliary information included in the audio bitstream (such as additional metadata or audio waveform channels) can simply be ignored by the audio bitstream decoder 330. Thus, the entire playback component 130 can be a legacy audio or A/V playback device, such as any existing mobile phone or computer. In some embodiments, the capture component 110 and the distribution component 120 are backward compatible with any legacy audio or video media playback device.

In some embodiments, an optional spatial audio decoder is applied to the SES output from the audio bitstream decoder 330. As shown in Figure 3, an SES headphone decoder 350 performs SES decoding for headphone output and playback by headphones 355. An SES stereo decoder 360 performs SES decoding to generate a stereo loudspeaker output to a stereo loudspeaker playback system 365. An SES multichannel decoder 370 performs SES decoding to generate a multichannel loudspeaker output to a multichannel loudspeaker playback system 375. Each of these SES decoders performs a decoding algorithm specifically customized for corresponding playback configuration. An embodiment of the playback component 30 includes one or more of the above-mentioned SES decoders for any playback configuration. Regardless of the playback configuration, these SES decoders do not require information about the original capture or recording configuration. For example, in some embodiments, the SES decoder includes an Ambisonics UHJ to B-format decoder followed by a B-format spatial decoder customized for a particular playback configuration, as described, for example, in Michael Gerzon, “Ambisonics in multichannel broadcasting and video,” JAES, Vol. 33, No. 11, pp. 859–871, November 1985.

By way of example, in embodiments supporting headphone playback, the SES is decoded by an SES headphone decoder 350 to output a binaural signal that reproduces the encoded audio scene. This is achieved by decoding the embedded spatial audio cues and applying appropriate directional filtering, such as head-related transfer functions (HRTFs). In some embodiments, this may involve a UHJ to B-format decoder followed by a binaural transcoder. The decoder may also support head tracking so that the orientation of the reproduced audio scene can be automatically adjusted during headphone playback to continuously compensate for changes in the listener's head orientation, thereby enhancing the listener's illusion of being immersed in the originally captured sound field.

As an example of an embodiment of the playback component 130 connected to a two-channel loudspeaker system (such as a stand-alone loudspeaker or a loudspeaker built into a laptop or tablet computer, a television, or a soundbar housing), the SES is first spatially decoded by an SES stereo decoder 360. In some embodiments, the decoder 360 comprises an SES decoder equivalent to the SES headphone decoder 350, whose binaural output signals may be further processed by appropriate crosstalk cancellation circuitry to provide a faithful reproduction of the spatial cues encoded in the SES (tailored for the specific two-channel loudspeaker playback configuration).

As an example of an embodiment of the playback component 130 connected to a multi-channel loudspeaker system, the SES is first spatially decoded by the SES multi-channel decoder 370. The configuration of the multi-channel loudspeaker playback system 375 can be a standard 5.1 or 7.1 surround sound system configuration or any arbitrary surround sound or immersive 3D configuration including, for example, height channels (such as a 22.2 system configuration).

The operations performed by the SES multi-channel decoder 370 may include reformatting the two-channel or multi-channel signal included in the SES. This reformatting is done to faithfully reproduce the spatial audio scene encoded in the SES according to the loudspeaker output layout and the optional additional metadata or auxiliary information included in the SES. In some embodiments, the SES includes a two-channel or multi-channel UHJ or B-format signal, and the SES multi-channel decoder 370 includes a spatial decoder optimized for a specific playback configuration.

In other embodiments where the SES includes a backward-compatible two-channel stereo signal that is feasible for standard two-channel stereo playback, alternative two-channel encoding/decoding schemes can be employed to overcome the known limitations of the UHJ encoding/decoding methods in terms of spatial audio fidelity. For example, to achieve improved spatial cue resolution and preserve three-dimensional information, the SES encoder can also utilize a two-channel frequency-domain phase-amplitude encoding method that can perform spatial encoding in multiple frequency bands. In addition, optional metadata extraction in the SES encoder and the combination of such spatial encoding methods enable further improvement in the fidelity and accuracy of the reproduced audio scene relative to the originally captured sound field.

In some embodiments, the SES decoder resides on a playback device with a default playback configuration that best suits the assumed listening scenario. For example, headphone reproduction may be an assumed listening scenario for a mobile device or camera, so the SES decoder may be configured to use headphones as the default decoding format. As another example, a 7.1 multichannel surround system may be an assumed playback configuration for a home theater listening scenario, so the SES decoder resident on the home theater device may be configured to use 7.1 multichannel surround as the default playback configuration.

II. System Details and Alternative Embodiments

The system details of various embodiments of the sound field coding system 100 and method will now be discussed. It should be noted that only a few of the several ways in which components, systems, and codecs can be implemented are described in detail below. Many variations of those shown and described herein are possible.

Flexible immersive audio capture and spatial encoding embodiments

FIG4 illustrates a general block diagram of an embodiment of a spatial encoder and decoder in sound field coding system 100. Referring to FIG4 , N audio signals are captured by N microphones to obtain N microphone signals. Each of the N microphones has a directivity pattern that characterizes its response as a function of frequency and direction relative to a reference direction. In spatial encoder 410, the N signals are combined into T signals such that each of the T signals has a specified directivity pattern associated with it.

In some embodiments, the spatial encoder 410 also generates auxiliary information S, represented by the dashed line in FIG4 . In some embodiments, the auxiliary information S includes spatial audio metadata and/or additional audio waveform signals. The T signals and the optional auxiliary information S form a spatially encoded signal (SES). The SES is transmitted or stored for subsequent use or distribution. In a preferred embodiment, T is less than N, so that the encoding of the N microphone signals into T transmitted signals reduces the amount of data required to represent the audio scene captured by the N microphones.

In some preferred embodiments, the auxiliary information S consists of spatial cues stored at a lower data rate than the data rate of the T audio transmission signals. This means that including the auxiliary information S generally does not significantly increase the overall SES data rate. The spatial decoder and renderer 420 converts the SES into Q playback signals optimized for a target playback system (not shown). The target playback system can be headphones, a two-channel loudspeaker system, a five-channel loudspeaker system, or some other playback configuration.

It should be noted that in Figure 4, without loss of generality, the number of transmitted signals T is depicted as 2. Other design choices for the number of transmitted channels are included within the scope of the present invention. For example, in some embodiments, T can be chosen to be 1. In these embodiments, the transmitted signal can be a monophonic downmix of the N captured signals, and some spatial side information S can be included in the SES to encode spatial cues representing the captured sound field. In other embodiments, T can be chosen to be greater than 2. When T is greater than 1, it is not necessary to include spatial cues in the side information S, as the spatial cues can be encoded in the T audio signals themselves. By way of example, the spatial cues can be mapped to the inter-channel amplitude and phase differences between the T transmitted signals.

FIG5 is a block diagram depicting in more detail an embodiment of a system 100 similar to the system described in FIG4 , where T=2. In these embodiments, N microphone signals are input into a spatial encoder 410. Spatial cues are encoded into T transmission signals by the spatial encoder 410 and the auxiliary information S can be omitted altogether. In some embodiments, as previously described in conjunction with FIG1 and FIG2 , a two-channel SES is perceptually encoded using a standard waveform encoder (such as MP3 or AAC), easily distributed via available digital distribution media or networks and broadcast infrastructure, and played back directly in a standard two-channel stereo configuration (using headphones or loudspeakers). In such embodiments, an important advantage is that the encoding and transmission system supports playback via commonly available 2-channel stereo systems without the need for spatial decoding and rendering processes.

Some embodiments of the system 100 include a single microphone (N=1). It should be noted that in these embodiments, spatial information will not be captured because there is no spatial diversity in the microphone signal. In these cases, pseudo-stereo techniques (such as those described, for example, in Orban's "A Rational Technique for Synthesizing Pseudo-Stereo From Monophonic Sources," JAES 18(2) (1970)) can be employed in the spatial encoder 410 to generate a 2-channel SES from a monophonic captured audio signal that is suitable for producing an artificial spatial impression when played back directly through a standard stereo reproduction system.

Some embodiments of the system 100 include a spatial decoder and renderer 420. In some preferred embodiments, the function of the spatial decoder and renderer 420 is to optimize the spatial fidelity of the reproduced audio scene for the specific playback configuration in use. For example, the spatial decoder and renderer 420 provides one or more of the following: (a) two output channels optimized for immersive 3-D audio reproduction in headphone playback (e.g., using HRTF-based virtualization techniques); (b) two output channels optimized for immersive 3-D audio reproduction in playback through two loudspeakers (e.g., using virtualization and crosstalk cancellation techniques); and (c) five output channels optimized for immersive 3-D audio or surround sound reproduction in playback through five loudspeakers. These are representative examples of reproduction formats. In some embodiments, as explained in more detail below, the spatial decoder and renderer 420 is configured to provide a playback signal optimized for reproduction through any arbitrary reproduction system.

FIG6 is a block diagram illustrating an embodiment of the spatial decoder and renderer 420 shown in FIG4 and FIG5 in more detail. As shown in FIG6, the spatial decoder and renderer 420 includes a spatial decoder 600 and a renderer 610. The SES shown without loss of generality includes T=2 channels and optional auxiliary information S. The decoder 600 first decodes the SES into P audio signals. In an exemplary embodiment, the decoder 600 outputs a 5-channel matrix decoded signal. The P audio signals are then processed to form Q playback signals optimized for the playback configuration of the reproduction system. In an exemplary embodiment, the SES is a 2-channel UHJ encoded signal, the decoder 600 is a conventional high-fidelity stereo UHJ to B-format converter, and the renderer 610 further decodes the B-format signal for the Q-channel playback configuration.

FIG7 is a block diagram illustrating SES capture and encoding with T=2 transmitted signals and no auxiliary information. In these embodiments, the spatial encoder 410 is designed to encode N microphone signals into a stereo signal. As explained above, the choice of T=2 is compatible with common perceptual audio waveform encoders (such as AAC or MP3), audio distribution media, and reproduction systems. The N microphones can be overlapping microphones, nearly overlapping microphones, or non-overlapping microphones. The microphones can be built into a single device (such as a camera, smartphone, field recorder, or accessory for such a device). In addition, the N microphone signals can be synchronized across multiple homogeneous or heterogeneous devices or device accessories.

In some embodiments, the T=2 transmission channels are encoded to simulate coincident virtual microphone signals, since coincidence (time alignment of the signals) is advantageous for facilitating high-quality spatial decoding. In embodiments using non-coincident microphones, provision of time alignment based on analysis of the directions of arrival and application of corresponding compensation can be incorporated into the SES encoder. In alternative embodiments, the stereo signal can be derived to correspond to binaural or non-coincident microphone recording signals, depending on the spatial audio reproduction use case and application associated with the intended decoder.

FIG8 is a block diagram illustrating an embodiment of the spatial encoder 410 shown in FIG4 to FIG7. As shown in FIG8, N microphone signals are input to a spatial analyzer and converter 800, where the N microphone signals are first converted into an intermediate format consisting of M signals. These M signals are then encoded into two channels by a renderer 810 for transmission. The embodiment shown in FIG8 is advantageous when the intermediate M-channel format is more suitable for processing by the renderer 810 than the N microphone signals. In some embodiments, the conversion to M intermediate channels can incorporate the analysis of the N microphone signals. Furthermore, in some embodiments, the spatial conversion process 800 can include multiple conversion steps and intermediate formats.

Details of specific embodiments

FIG9A illustrates a specific example embodiment of the spatial encoder 410 and method shown in FIG7 , in which an A-format microphone signal is captured. The original 4-channel A-format microphone signal can be easily converted to a high-fidelity stereo B-format signal (W, X, Y, Z) by an A-format to B-format converter 900. Alternatively, a microphone that directly provides a B-format signal can be used, in which case the A-format to B-format converter 900 is not required.

Various virtual microphone directivity patterns can be formed from the B-format signal. In this embodiment, the B-format to supercardioid converter block 910 converts the B-format signal into a set of three supercardioid microphone signals formed using these equations:

For example, the design parameters are set to: θ _S = π and p = 0.33. W is the omnidirectional pressure signal in B format, X is the front-to-back figure-of-eight signal in B format, and Y is the left-right figure-of-eight signal in B format. The Z signal (up-down figure-of-eight) in B format is not used in this conversion. V _L is the virtual left microphone signal corresponding to a hypercardioid with a directivity pattern steered to -60 degrees (in radians) in the horizontal plane, _VR is the virtual right microphone signal corresponding to a hypercardioid with a directivity pattern steered to +60 degrees (in radians) in the horizontal plane, and V _S is the virtual surround microphone signal corresponding to a hypercardioid with a directivity pattern steered to +180 _degrees (in radians) in the horizontal plane. The parameter p = 0.33 is selected based on the desired directivity of the virtual microphone signals.

Figure 9B shows the directivity patterns of the B-format components on a linear scale. Plot 920 shows the directivity pattern of the omnidirectional W component. Plot 930 shows the directivity pattern of the front-to-back X component, where 0 degrees is the forward direction. Plot 940 shows the directivity pattern of the left-right Y component.

FIG9C illustrates the directivity patterns of the supercardioid virtual microphones in this embodiment on a dB scale. Plot 950 shows the directivity pattern of V _L with the virtual microphone steered to -60 degrees. Plot 960 shows the directivity pattern of _VR with the virtual microphone steered to +60 degrees. Plot 970 shows the directivity pattern of _VS with the virtual microphone steered to +180 degrees.

The spatial encoder 410 converts the resulting 3-channel supercardioid signal (V _L , _VR , V _S ) produced by the converter 910 into a two-channel SES. This is achieved by using the following phase-amplitude matrix encoding equation:

_LT = aV _L + jbV _S

_RT = aVR _{- jbVS}

Where _LT denotes the encoded left channel signal, _RT denotes the encoded right channel signal, j denotes a 90-degree phase shift, a and b are 3:2 matrix encoding weights, and _VR , _VL , and _VS are the left channel virtual microphone signal, the right channel virtual microphone signal, and the surround channel virtual microphone signal, respectively. In some embodiments, the 3:2 matrix encoding weights can be selected such that a=1, and this preserves the total power of the three-channel signals ( _VL , _VR , _VS ) in the encoded SES. As will be clear to those skilled in the art upon reading this, the above matrix encoding equation has the effect of converting the set of three virtual microphone directivity patterns associated with the three-channel signals ( _VL , _VR , _VS ) shown in FIG9C into a pair of complex-valued virtual microphone directivity patterns associated with the two-channel SES ( _LT , _RT ).

The embodiment depicted in FIG9A and described above implements a low-complexity spatial encoder that can be suitable for low-power devices and applications. Note that, within the scope of the present invention, alternative directivity patterns for the intermediate 3-channel representation can be formed from the B-format signal. The resulting two-channel SES is suitable for spatial decoding using a phase-amplitude matrix decoder (such as the spatial decoder 600 shown in FIG6).

FIG10 illustrates a specific example embodiment of the spatial encoder 410 and method shown in FIG7 , in which a B-format signal is converted into a 5-channel surround sound signal (L, R, C, _LS , _RS ). Note that L denotes the front left channel, R denotes the front right channel, C denotes the front center channel, _LS denotes the left surround channel, and _RS denotes the right surround channel. Similar to FIG9A , an A-format microphone signal is input to an A-format to B-format converter 1000 and converted into a B-format signal. This 4-channel B-format signal is processed by a B-format to multi-channel format converter 1010, which in some embodiments is a multi-channel B-format decoder. Next, the spatial encoder converts the 5-channel surround sound signal generated by the converter 1010 into a two-channel SES by using the following phase-amplitude matrix coding equation in an embodiment:

L _T ＝a ₁ L+a ₂ R+a ₃ C+ja ₄ L _s -ja ₅ R _s

R _T =a ₂ L+a ₁ R+a ₃ C-ja ₅ Ls+ja ₄ R _s

Where _LT and _RT denote the left and right SES signals, respectively, output by the spatial encoder. In some embodiments, the matrix coding coefficients may be selected such that _a1 = 1 and _a2 = 0, and alternative sets of matrix coding coefficients may be used depending on the desired spatial distribution of the front and surround channels in the two-channel encoded signal. As in the spatial encoder embodiment of FIG9A , the resulting two-channel SES is suitable for spatial decoding by a phase-amplitude matrix decoder (such as the spatial decoder 600 shown in FIG6 ).

In the embodiment shown in FIG10 , the B-format signal is converted to a 5-channel center surround format. However, it will be apparent that any horizontal surround or 3D center multi-channel format may be used within the scope of the present invention. In these cases, the operation of the converter 1010 and the spatial encoder 410 can be easily configured based on the assumed set of directions assigned to the various center channels.

FIG11 illustrates a specific example embodiment of the spatial encoder 410 and method shown in FIG7 , in which a B-format signal is converted to a Directional Audio Coding (DirAC) representation. Specifically, as shown in FIG11 , an A-format microphone signal is input to an A-format to B-format converter 1100. The resulting B-format signal is converted to a DirAC-encoded signal by a B-format to DirAC format converter 1110, as described, for example, in Pulkki, “Spatial Sound Reproduction with Directional Audio Coding,” JAES, Vol. 55, No. 6, pp. 503-516, June 2007. The spatial encoder 410 then converts the DirAC-encoded signal into a two-channel SES. In one embodiment, the conversion is accomplished by converting the frequency-domain DirAC waveform data into a two-channel representation, such as obtained by the method described in Jot's "Two-Channel Matrix Surround Encoding for Flexible Interactive 3-D Audio Reproduction," presented at the 125th AES Convention in October 2008. The resulting SES is suitable for spatial decoding by a phase-amplitude matrix decoder, such as the spatial decoder 600 shown in FIG6 .

DirAC encoding involves a frequency domain analysis that distinguishes between direct and diffuse components of the sound field. In a spatial encoder according to the present invention (such as spatial encoder 410), two-channel encoding is performed within a frequency domain representation to fully exploit the DirAC analysis. This results in a higher degree of spatial fidelity than would be achieved using conventional time-domain phase-amplitude matrix coding techniques (such as those used in the spatial encoder embodiments described in conjunction with FIG9A and FIG10).

FIG12 is a block diagram illustrating an embodiment of the conversion of an A-format microphone signal to an SES in greater detail. As shown in FIG12 , the A-format microphone signal is converted to a B-format signal using an A-format to B-format converter 1200. The B-format signal is converted to the frequency domain using a time-frequency transform 1210. Transform 1210 is at least one of a short-time Fourier transform, a wavelet transform, a subband filter bank, or some other operation that converts a time-domain signal into a time-frequency representation. Next, a B-format to DirAC format converter 1220 converts the B-format signal to a _DirAC format signal. The DirAC signal is input to the spatial encoder 410 and spatially encoded into a two-channel SES, which is still represented in the frequency domain. The signal is converted back to the time domain using a frequency-time transform 1240, which is the inverse transform of the time-frequency transform 1210, or an approximation of the inverse transform if a perfect inverse transform is not possible or feasible. It should be noted that in order to improve the fidelity of the spatial coding, both direct time-to-frequency transform and inverse time-to-frequency transform may be incorporated in any of the encoder embodiments according to the present invention.

FIG13 is a block diagram illustrating another embodiment of a spatial encoder 410 that transforms a B-format signal into the frequency domain prior to spatial encoding. Referring to FIG13 , an A-format microphone signal is input to an A-format to B-format converter 1300. The resulting signal is converted from the time domain to the frequency domain using a time-frequency converter 1310. The signal is encoded using a B-format-based encoder 1320. In one embodiment, the SES is a two-channel stereo signal encoded according to the following equation:

L _T ＝a _L W+b _L X+c _L Y+d _L Z

R _T =a _R W+b _R X+c _R Y+d _R Z

where the coefficients (a _L , b _L , c _L , d _L ) are time- and frequency-dependent coefficients determined from the frequency-domain 3-D dominant directions computed from the B-format signal (W, X, Y, Z) such that if the sound field consists of a single sound source S at a 3-D position, the resulting coded signal is given by:

Where k _L and k _R are complex factors that uniquely map the left/right channel amplitude and phase differences to 3-D positions. Example mapping formulas for this purpose are presented, for example, in Jot's "Two-Channel Matrix Surround Encoding for Flexible Interactive 3-D Audio Reproduction," presented at the 125th AES Convention in October 2008. Such 3-D encoding can also be performed for other channel formats. The encoded signal is transformed from the frequency domain to the time domain using a frequency-to-time converter 1330.

An audio scene can consist of discrete sound sources (such as speakers or musical instruments) or diffuse sounds (such as rain, applause, or reverberation). Some sounds can be partially diffuse, such as the rumble of a large engine. In a spatial encoder, it can be beneficial to process discrete sounds (those that arrive at the microphone from different directions) differently than diffuse sounds.

FIG14 is a block diagram illustrating an embodiment of a spatial encoder 410 in which an input microphone component signal is first decomposed into a direct component and a diffuse component. The direct component and the diffuse component are then encoded separately so as to preserve the different spatial characteristics of the direct component and the diffuse component. An example method for direct/diffuse decomposition of a multichannel audio signal is described, for example, in Thompson et al., “Direct-Diffuse Decomposition of Multichannel Signals Using a System of Pairwise Correlations,” presented at the 133rd AES Convention (October 2012). It should be understood that the direct/diffuse decomposition can be used in conjunction with the various spatial coding systems described earlier.

In an outdoor setting, the audio signal captured by a microphone may be damaged by wind noise. In some cases, wind noise may seriously affect the signal quality on one or more microphones. In these and other situations, it is beneficial to include a wind noise detection module. Figure 15 is a block diagram showing an embodiment of a system 100 and method including a wind noise detector. As shown in Figure 15, N microphone signals are input to an adaptive spatial encoder 1500. A wind noise detector 1510 provides an estimate of the wind noise energy or energy ratio in each microphone. Severely damaged microphone signals can be adaptively excluded from the channel combination used in the encoder. On the other hand, partially damaged microphones can be reduced in weight in the coding combination to control the amount of wind noise in the coded signal. In some cases (such as when capturing fast-moving outdoor action scenes), adaptive coding based on wind noise detection can be configured to convey at least a certain portion of wind noise in the coded audio signal.

Adaptive encoding can also be useful to account for occlusion of one or more microphones from the acoustic environment (e.g., by a user's finger or accumulated dust on the device). In the case of occlusion, the microphone provides poor signal capture, and spatial information derived from the microphone signal may be misleading due to low signal levels. Detection of an occlusion condition can be used to exclude the blocked microphone from the encoding process.

In some embodiments, it may be desirable to perform editing operations on the audio scene before encoding the signal for storage or distribution. Such editing operations may include amplifying or reducing the size of a sound source, removing unwanted sound components (such as background noise), and adding sound objects to the scene. FIG16 illustrates a system for capturing N microphone signals and converting them into an M-channel format suitable for editing.

Specifically, N microphone signals are input to a spatial analyzer and converter 1600. The resulting M-channel signal output by converter 1600 is provided to an audio scene editor 1610, which is controlled by the user to implement desired modifications to the scene. After the modifications are made, the scene is spatially encoded by a spatial encoder 1620. For illustrative purposes, diagram 1620 shows a two-channel SES format. Alternatively, the N microphone signals can be provided directly to the editing tool.

In embodiments where the capture device is configured to provide only a two-channel SES format, the SES may be decoded into a multi-channel format suitable for editing and then re-encoded for storage or distribution. Because the additional decoding/encoding process may introduce some degradation in spatial fidelity, it is preferable to enable editing operations on the multi-channel format prior to two-channel spatial encoding. In some embodiments, the device may be configured to output a two-channel SES simultaneously with an M-channel format or N microphone signals intended for editing.

In some embodiments, SES can be imported into a non-linear video editing suite and manipulated with respect to traditional stereo film capture. The spatial integrity of the resulting content will remain intact post-editing, provided that no spatially deleterious audio processing effects are applied to the content. SES decoding and reformatting can also be applied as part of a video editing suite. For example, if the content is being burned to DVD or Blu-ray Disc, multi-channel speaker decoding and reformatting can be applied and the result encoded in a multi-channel format for subsequent multi-channel playback. Alternatively, the audio content can be authored "as is" for traditional stereo playback on any compatible playback hardware. In this case, SES decoding can be applied on the playback device if the appropriate reformatting algorithm exists on the device.

FIG17 illustrates an embodiment of a system and method by which a captured audio scene is modified as part of the decoding process. More specifically, N microphone signals are encoded into a SES by a spatial encoder 1700. In some embodiments, the SES includes side information S. The SES is stored, transmitted, or both. A spatial decoder 1710 decodes the encoded SES, and a renderer 1720 provides Q playback signals. The decoder 1710 uses scene modification parameters to modify the audio scene.

In some preferred embodiments, scene modifications occur at points in the decoding process where the modifications can be efficiently implemented. For example, in virtual reality applications using headphones for audio rendering, it is crucial to update the spatial cues of the sound scene in real time based on the user's head movements, so that the perceived localization of sound objects matches the perceived localization of their visual counterparts. To achieve this, a head tracking device is used to detect the orientation of the user's head. The virtual audio rendering is then continuously updated based on these estimates so that the reproduced sound scene appears independent of the listener's head movements.

The estimation of head position can be incorporated into the decoding process of the spatial decoder 1710, so that the renderer 1720 reproduces a stable audio scene. This is equivalent to rotating the scene before decoding or rendering to a rotated intermediate format (the P channels output by the spatial decoder) before virtualization. In embodiments where auxiliary information is included in the SES, such scene rotation can include manipulation of the spatial metadata included in the auxiliary information.

Other interesting modifications that can be supported during spatial decoding include warping the width of the audio scene and audio zoom. In some embodiments, the decoded audio signal can be spatially warped to match the field of view of the original video recording. For example, if the original video uses a wide-angle lens, the audio scene can be stretched across similar angular arcs to better match the audio cues and visual cues. In some embodiments, the audio can be modified to zoom into or out of the spatial region of interest; audio zoom can be combined with video zoom modifications.

In some embodiments, the decoder can modify the spatial characteristics of the decoded signal to guide or emphasize the decoded signal at a specific spatial location. This can allow the salience of certain auditory events (such as, for example, conversations) to be increased or decreased. In some embodiments, this can be facilitated by using a speech detection algorithm.

III. Operation Overview

Embodiments of the sound field coding system 100 and method use an arbitrary microphone array configuration to capture a sound field representing an immersive audio scene. The captured audio is encoded using a generic SES format that is insensitive to the microphone array configuration used.

FIG18 is a flow chart illustrating the general operation of an embodiment of the capture component 110 of the sound field coding system 100 shown in FIG1-17. The operation begins by selecting a microphone configuration comprising a plurality of microphones (block 1800). These microphones are used to capture sound from at least one audio source. The microphone configuration defines a microphone directivity pattern for each microphone relative to a reference direction. Additionally, a virtual microphone configuration comprising a plurality of virtual microphones is selected (block 1810).

The method calculates spatial coding coefficients based on the microphone configuration and the virtual microphone configuration (block 1820). The spatial coding coefficients are used to convert microphone signals from the plurality of microphones into spatially coded signals (block 1830). The output of the system 100 is a spatially coded signal (block 1840). This signal contains encoded spatial information about the position of the audio source relative to a reference direction.

As described above, various other embodiments of the system 100 and method are disclosed herein. By way of example, and not limitation, referring again to FIG. 7 , the spatial encoder 410 can be generalized from an N:2 spatial encoder to an N:T spatial encoder. Furthermore, within the scope of the present invention, various other embodiments can be implemented for an encoder that generates a two-channel SES ( _LT , _RT ) compatible with a phase-amplitude matrix decoder configured for immersive audio reproduction in flexible playback configurations and direct two-channel stereo playback. In embodiments using a standard microphone configuration (such as an Ambisonics A or B format), the two-channel encoding equations can be specified based on a specified directivity pattern for the microphone format.

More generally, in embodiments where microphones may be placed in non-standard configurations due to device design constraints or the adhoc nature of a network of devices, a derivation of a spatially coded signal may be formed by combining microphone signals based on relative microphone positioning and measured or estimated directivities of the microphones. These combinations may be formed to optimally achieve a prescribed directivity pattern suitable for two-channel SES encoding. Given the directivity patterns of N microphones mounted on a respective recording device or accessory (where a directivity pattern is a complex amplitude factor characterizing the response of a microphone as a function of frequency f and 3-D position), a set of coefficients k _Ln (f) and k _Rn (f) may be optimized for each microphone at each frequency to form a virtual microphone directivity pattern for the left and right SES channels:

Therein, coefficient optimization is performed to minimize an error criterion between the resulting left and right virtual microphone directivity patterns and the prescribed left and right directivity patterns for each encoded channel.

In some embodiments, the microphone responses can be combined to accurately form a prescribed virtual microphone directivity pattern, in which case the equations can hold in the above expressions. For example, in the embodiments described in conjunction with FIG9B and FIG9C , the B-format microphone responses are combined to accurately achieve the prescribed virtual microphone response. In some embodiments, coefficient optimization can be implemented using an optimization method such as least squares approximation.

The two-channel SES encoding equation is then given by:

Wherein _LT (f, t) and _RT (f, t) denote the frequency domain representation of the left SES channel and the right SES channel, respectively, and _Sn (f, t) denotes the frequency domain representation of the nth microphone signal.

Similarly, in some embodiments according to FIG. 4 , optimal directivity patterns for T virtual microphones corresponding to T coded signals may be formed, where T is not equal to 2. In the embodiment according to FIG. 8 , optimal directivity patterns for M virtual microphones may be formed corresponding to M channels in an intermediate format, where each channel in the intermediate format has a specified directivity pattern; the M channels in the intermediate format are then encoded into two channels. In other embodiments, the M intermediate channels may be encoded into T channels, where T is not equal to 2.

From the description of the various embodiments above, it should be understood that the present invention can be used to encode any microphone format; and further, if the microphone format provides a directionally selective response, the spatial encoding/decoding can preserve the directionality selectivity. Other microphone formats that can be incorporated into the capture and encoding system include, but are not limited to, XY stereo microphones and non-coincident microphones, which can be time-aligned based on frequency-domain spatial analysis to support matrix encoding and decoding.

From the description of the frequency domain operations incorporated in the various embodiments above, it should be understood that frequency domain analysis can be implemented in conjunction with any of the embodiments in order to improve the spatial fidelity of the encoding process; in other words, frequency domain processing will result in a decoded scene that more accurately matches the captured scene than a purely time domain approach, at the expense of the additional computations of performing the time-frequency transform, frequency domain analysis, and inverse transform following spatial encoding.

IV. Exemplary Operating Environment

According to this document, many other variations except those described herein will be clear.For example, depending on the embodiment, some actions, events or functions of any one of the methods and algorithms described herein can be performed in different orders, can be added, merged or omitted together (so that not all described actions or events are necessary for practicing the method and algorithm). Moreover, in certain embodiments, actions or events can be performed simultaneously (such as by multithreading, interrupt processing or multiple processors or processor cores or on other parallel architectures), rather than sequentially. In addition, different tasks or processes can be performed by different machines and computing systems that can run together.

The various illustrative logic blocks, modules, methods, and algorithmic processes and sequences described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, and process actions have been generally described above with respect to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The described functions can be implemented in varying ways for each specific application, but such implementation decisions should not be interpreted as departing from the scope of this document.

The various illustrative logical blocks and modules described in conjunction with the embodiments disclosed herein may be implemented or executed by a machine, such as a general-purpose processor, a processing device, a computing device having one or more processing devices, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. General-purpose processors and processing devices may be microprocessors, but in an alternative embodiment, the processor may be a controller, a microcontroller or a state machine, a combination thereof, etc. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

Embodiments of the sound field coding systems and methods described herein are operable within many types of general-purpose or special-purpose computing system environments or configurations. Generally, the computing environment may include any type of computer system, including but not limited to a computer system based on one or more microprocessors, a mainframe computer, a digital signal processor, a portable computing device, a personal organizer, an equipment controller, a computing engine within an appliance, a mobile phone, a desktop computer, a mobile computer, a tablet computer, a smartphone, and an appliance with an embedded computer, etc.

Such computing devices are typically found in devices with at least some minimum computing power, including but not limited to personal computers, server computers, handheld computing devices, laptop or mobile computers, communication devices (such as cellular phones and PDAs), multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, audio or video media players, etc. In some embodiments, the computing device will include one or more processors. Each processor can be a specialized microprocessor, such as a digital signal processor (DSP), a very long instruction word (VLIW), or other microcontroller, or can be a conventional central processing unit (CPU) with one or more processing cores (including a core based on a specialized graphics processing unit (GPU) in a multi-core CPU).

The process actions of the methods, processes, or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly in hardware, in software modules executed by a processor, or in any combination thereof. The software modules may be contained in a computer-readable medium that can be accessed by a computing device. Computer-readable media include both volatile and non-volatile media that are removable, non-removable, or some combination thereof. Computer-readable media is used to store information, such as computer-readable or computer-executable instructions, data structures, program modules, or other data. By way of example, and not limitation, computer-readable media may include computer storage media and communication media.

Computer storage media includes, but is not limited to, computer or machine readable media or storage devices, such as Blu-ray discs (BDs), digital versatile discs (DVDs), compact discs (CDs), floppy disks, tape drives, hard drives, optical drives, solid-state memory devices, RAM memory, ROM memory, EPROM memory, EEPROM memory, flash memory or other memory technology, magnetic cassettes, magnetic tape, disk storage, or other magnetic storage devices, or any other device that can be used to store the desired information and which can be accessed by one or more computing devices.

The software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, hard disk, removable disk, CD-ROM or any other form of non-transitory computer readable storage medium, medium or physical computer storage known in the art. An exemplary storage medium can be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium can be integrated with the processor. The processor and the storage medium can reside in an application specific integrated circuit (ASIC). The ASIC can reside in a user terminal. Alternatively, the processor and the storage medium can reside in a user terminal as discrete components.

As used in this document, the phrase "non-transitory" means "persistent or long-term." The phrase "non-transitory computer-readable medium" includes any and all computer-readable media, with the sole exception of transitory propagating signals. By way of example, and not limitation, this includes non-transitory computer-readable media such as register memory, processor cache, and random access memory (RAM).

The retention of information (such as computer-readable or computer-executable instructions, data structures, program modules, etc.) can also be achieved by using various communication media or other transmission mechanisms or communication protocols that encode one or more modulated data signals, electromagnetic waves (such as carrier waves), and include any wired or wireless information delivery mechanism. Generally speaking, these communication media refer to such signals, one or more of the characteristics of which are set or changed in such a way as to encode information or instructions in the signal. For example, communication media include wired media (such as a direct wired connection or wired network that carries one or more modulated data signals) and wireless media (such as acoustic, radio frequency (RF), infrared, laser, and other wireless media for sending, receiving, or both sending and receiving one or more modulated data signals or electromagnetic waves). Combinations of any of the above should also be included within the scope of communication media.

In addition, one of the software, programs, computer program products, or any combination thereof, or portions thereof, that implement some or all of the various embodiments of the sound field coding systems and methods described herein may be stored, received, sent, or read from any desired combination of computer or machine-readable media or storage devices and communication media in the form of computer-executable instructions or other data structures.

The embodiments of the sound field coding systems and methods described herein may be further described in the general context of computer-executable instructions (such as program modules) being executed by a computer device. Generally speaking, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The embodiments described herein may also be practiced in a distributed computing environment, in which tasks are performed by one or more remote processing devices, or within a cloud of one or more devices linked by one or more communication networks. In a distributed computing environment, program modules may be located in both local computer storage media and remote computer storage media (including media storage devices). Furthermore, the aforementioned instructions may be implemented in part or in whole as hardware logic circuits, which may or may not include a processor.

As used herein, conditional language (including, for example, "can," "might," "may," "for example," etc.), unless expressly stated otherwise or understood otherwise within the context in which it is used, is generally intended to convey that some embodiments include and other embodiments do not include certain features, elements, and/or states. Thus, such conditional language is generally not intended to imply that features, elements, and/or states are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining, with or without author input or prompting, whether such features, elements, and/or states are included in or will be performed in any particular embodiment. The terms "including," "comprising," "having," and the like are synonymous and are used inclusively in an open-ended manner and do not exclude additional elements, features, actions, operations, and the like. Furthermore, the term "or" is used in its inclusive sense (and not its exclusive sense) such that when used, for example, to connect a list of elements, the term "or" means one, some, or all of the elements in the list.

While the above detailed description has shown, described, and pointed out novel features as applied to various embodiments, it will be understood that various omissions, substitutions, and changes in the form and details of the devices or algorithms shown may be made without departing from the scope of the present disclosure. As will be appreciated, certain embodiments of the invention described herein may be implemented in a form that does not provide all of the features and benefits set forth herein, as some features may be used or practiced separately from other features.

Furthermore, although the subject matter has been described in language specific to structural features and methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims (11)

1. A method for processing signals from multiple capture microphones, comprising: Select a capture microphone configuration having a plurality of capture microphones for capturing sound from at least one audio source, the capture microphone configuration defining the capture microphone directivity of each of the plurality of capture microphones relative to a reference direction; Select a virtual microphone configuration with multiple virtual microphones, the virtual microphone configuration being used to encode spatial information about the position of the at least one audio source relative to a reference direction, the virtual microphone configuration defining the directivity of each of the multiple virtual microphones relative to the reference direction; Spatial coding coefficients are calculated based on the captured microphone configuration and the virtual microphone configuration; and The multiple captured microphone signals are converted into spatially encoded signals that include virtual microphone signals; Each of the virtual microphone signals is obtained by capturing microphone signals using spatial coding coefficients; The capture microphone directivity is a complex amplitude factor, which characterizes the microphone's response as it varies with the frequency and 3D position of at least one audio source. 2. The method of claim 1, wherein the spatial information is encoded in one of the following forms: (a) inter-channel amplitude; and (b) phase difference. 3. The method of claim 2, further comprising selecting a virtual microphone configuration having a plurality of virtual microphones, the virtual microphone configuration being used to encode spatial information about the position of the audio source relative to a reference direction. 4. The method of claim 1, wherein the plurality of captured microphone signals are A-format microphone signals, further comprising converting the A-format microphone signals into B-format microphone signals. 5. The method of claim 4, further comprising forming a virtual microphone directional pattern from a B-format microphone signal. 6. The method of claim 5, further comprising using the following equation to form a virtual microphone directional pattern: Where _θL , _θR , _θS and p are design parameters, W is the omnidirectional pressure signal in B format, X is the front-back figure-eight signal in B format, Y is the left-right figure-eight signal in B format, _VL is the virtual left microphone signal in the horizontal plane, _VR is the virtual right microphone signal in the horizontal plane corresponding to the supercardioid, and _VS is the virtual surround microphone signal in the horizontal plane corresponding to the supercardioid. 7. The method of claim 6, further comprising selecting design parameter p based on the desired directivity of the virtual microphone signal. 8. A method for processing an audio signal comprising signals from multiple capture microphones, comprising: Select a capture microphone configuration having multiple capture microphones for capturing sound from an audio source, the capture microphone configuration defining the capture microphone directivity of each of the multiple capture microphones relative to a reference direction; Spatial coding coefficients are calculated based on the capture microphone configuration; and Spatial coding coefficients are used to convert the plurality of captured microphone signals into spatially coded signals, wherein the spatially coded signals are two-channel spatially coded signals that carry coded spatial information about the position of the audio source relative to a reference direction; The capture microphone directivity is a complex amplitude factor, which characterizes the microphone's response as it varies with the frequency and 3D position of the audio source. 9. The method of claim 8, wherein the spatially encoded signal is a phase amplitude spatially encoded signal. 10. A method for processing signals from multiple capture microphones, comprising: Select a capture microphone configuration having multiple capture microphones for capturing sound from an audio source, the capture microphone configuration defining the capture microphone directivity of each of the multiple capture microphones relative to a reference direction; Spatial coding coefficients are calculated based on the capture microphone configuration; and Spatial coding coefficients are used to convert the plurality of captured microphone signals into spatially coded signals, wherein the spatially coded signals are spatially coded signals having at least two channels and carrying coded spatial information about the position of the audio source relative to a reference direction; The capture microphone directivity is a complex amplitude factor, which characterizes the microphone's response as it varies with the frequency and 3D position of the audio source. 11. The method of claim 10, wherein the spatial information is partially transmitted in the form of location audio metadata.