HK1232011B - Audio processing unit and method for decoding encoded audio bitstream - Google Patents

Description

Audio processing unit and method for decoding coded audio bit stream

This application is a divisional application of the invention patent application with the application date of June 12, 2014, application number "201480008799.7", and invention name "Audio encoder and decoder using program information or substream structure metadata".

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent Application No. 61/836,865, filed June 19, 2013, the entire contents of which are incorporated herein by reference.

Technical Field

The present invention relates to audio signal processing, and more particularly, to encoding and decoding of an audio data bitstream having metadata indicating a substream structure and/or program information related to the audio content indicated by the bitstream. Some embodiments of the present invention generate or decode audio data in one of the formats known as Dolby Digital (AC-3), Dolby Digital Plus (Enhanced AC-3 or E-AC-3), or Dolby E.

Background Art

Dolby, Dolby Digital, Dolby Digital Plus, and Dolby E are trademarks of Dolby Laboratories, Inc. Dolby Laboratories provides proprietary implementations of AC-3 and E-AC-3 known as Dolby Digital and Dolby Digital Plus, respectively.

Audio data processing units typically operate in a blind fashion and do not pay attention to the processing history of the audio data that occurred before the data was received. This can work in a processing framework where a single entity performs all audio data processing and encoding for various target media rendering devices, while the target media rendering devices perform all decoding and rendering of the encoded audio data. However, this blind processing does not work well (or at all) when multiple audio processing units are scattered or placed in series (i.e., chained) across a diverse network and are expected to perform their corresponding types of audio processing optimally. For example, some audio data may be encoded for a high-performance media system and may need to be converted into a simplified form suitable for mobile devices along the media processing chain. Therefore, the audio processing unit may unnecessarily perform a type of processing that has already been performed on the audio data. For example, a volume leveling unit may perform processing on an input audio segment, regardless of whether the same or similar volume leveling has been performed on the input audio segment before. Therefore, even when it is unnecessary, the volume leveling unit may perform leveling. This unnecessary processing may also lead to the degradation and/or elimination of specific features when rendering the content of the audio data.

Summary of the Invention

In one class of embodiments, the present invention is an audio processing unit capable of decoding a coded bitstream, the coded bitstream including substream structure metadata and/or program information metadata (optionally including other metadata, such as loudness processing state metadata) in at least one segment of at least one frame of the bitstream, and audio data in at least one other segment of the frame. Herein, substream structure metadata (or "SSM") refers to metadata of a coded bitstream (or a collection of coded bitstreams) that indicates a substream structure of the audio content of the coded bitstream, and "program information metadata" (or "PIM") refers to metadata of a coded audio bitstream that indicates at least one audio program (e.g., two or more audio programs), wherein the program information metadata indicates at least one attribute or characteristic of the audio content of at least one of the programs (e.g., metadata indicating a type or parameter of processing performed on the audio data of the program, or metadata indicating which channels of the program are active channels).

In a typical case (e.g., where the encoded bitstream is an AC-3 or E-AC-3 bitstream), program information metadata (PIM) indicates program information that cannot actually be carried in other parts of the bitstream. For example, the PIM may indicate the processing applied to the PCM audio before encoding (e.g., AC-3 or E-AC-3 encoding), which frequency bands of the audio program have been encoded using a specific audio coding technique, and the compression profile used to create dynamic range compression (DRC) data in the bitstream.

In another class of embodiments, the method includes the step of multiplexing the encoded audio data with the SSM and/or PIM in each frame (or at least each of some frames) of the bitstream. In a typical decoding, the decoder extracts the SSM and/or PIM from the bitstream (including by parsing and demultiplexing the SSM and/or PIM and the audio data), and processes the audio data to generate a stream of decoded audio data (and, in some cases, also performs adaptive processing of the audio data). In some embodiments, the decoded audio data and the SSM and/or PIM are forwarded from the decoder to a post-processor that is configured to perform adaptive processing on the decoded audio data using the SSM and/or PIM.

In one class of embodiments, the encoding method of the present invention generates an encoded audio bitstream (e.g., an AC-3 or E-AC-3 bitstream) comprising audio data segments (e.g., all or some of segments AB0 to AB5 of the frame shown in FIG. 4 , or segments AB0 to AB5 of the frame shown in FIG. 7 ). The audio data segments include the encoded audio data and metadata segments (including an SSM and/or PIM, and optionally other metadata) time-division multiplexed with the audio data segments. In some embodiments, each metadata segment (sometimes referred to herein as a "container") comprises a metadata segment header (optionally including other mandatory or "core" elements) and one or more metadata payloads following the metadata segment header. If present, the SIM is included in one of the metadata payloads (identified by the payload header and generally having a first type of format). If present, the PIM is included in another of the metadata payloads (identified by the payload header and generally having a second type of format). Similarly, each other type of metadata (if present) is included in another of the metadata payloads (identified by the payload header and generally having a format specific to the metadata type). The exemplary format allows convenient access to the SSM, PIM, or other metadata at times other than during decoding of the bitstream (e.g., by a post-processor after decoding, or by a processor configured to identify metadata without performing a full decoding of the encoded bitstream), and allows convenient and efficient error detection and correction during decoding of the bitstream (e.g., substream identification). For example, without access to the SSM in the exemplary format, a decoder may incorrectly identify the correct number of substreams associated with a program. One metadata payload in the metadata segment may include the SSM, another metadata payload in the metadata segment may include the PIM, and optionally, at least one other metadata payload in the metadata segment may include other metadata (e.g., loudness processing state metadata or "LPSM").

According to one embodiment, an audio processing unit is provided, comprising: a buffer memory; and at least one processing subsystem coupled to the buffer memory, wherein the buffer memory stores at least one frame of an encoded audio bitstream, the frame including program information metadata or substream structure metadata in at least one metadata segment of at least one reserved field of the frame and audio data in at least one other segment of the frame, wherein the processing subsystem is coupled and configured to use the metadata of the bitstream to perform at least one of bitstream generation, audio data decoding, or adaptive processing of the audio data, or to use the metadata of the bitstream to perform at least one of authentication or verification of at least one of the audio data or metadata of the bitstream. The metadata segment includes at least one metadata payload, the metadata payload including: a header; and, following the header, at least a portion of the program information metadata or at least a portion of the substream structure metadata. Furthermore, the reserved field is selected from the group consisting of a skip field, an addbsi field, an ancillary data field, or a combination thereof.

According to another embodiment, a method for decoding a coded audio bitstream is provided, the method comprising the steps of: receiving a coded audio bitstream comprising metadata and audio data; and extracting the metadata or the audio data from the coded audio bitstream, wherein the metadata is or comprises program information metadata or substream structure metadata. The coded audio bitstream comprises a series of frames and indicates at least one audio program, the program information metadata and the substream structure metadata indicate the program, each of the frames comprises at least one audio data segment, each audio data segment comprises at least a portion of the audio data, each frame in at least a subset of the frames comprises a metadata segment, each metadata segment comprises at least a portion of the program information metadata and at least a portion of the substream structure metadata, wherein the metadata segment is located in a reserved field selected from the group consisting of a skip field, an addbsi field, an ancillary data field, or a combination thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a block diagram of an embodiment of a system that may be configured to perform embodiments of the method of the present invention.

FIG. 2 is a block diagram of an encoder as an embodiment of the audio processing unit of the present invention.

3 is a block diagram of a decoder as an embodiment of an audio processing unit of the present invention and a post-processor coupled to the decoder as another embodiment of the audio processing unit of the present invention.

FIG. 4 is a diagram of an AC-3 frame including divided segments.

5 is a diagram of a synchronization information (SI) segment of an AC-3 frame including segments divided therein.

6 is a diagram of a bit stream information (BSI) section of an AC-3 frame including divided segments.

FIG. 7 is a diagram of an E-AC-3 frame including divided segments.

Figure 8 is a diagram of a metadata segment of an encoded bit stream generated according to an embodiment of the present invention, including a metadata segment header, the metadata segment header including a container synchronization word (identified as "Container Sync" in Figure 8) and version and key ID values, followed by multiple metadata payloads and protection bits.

Symbols and terminology

Throughout this disclosure, including the claims, expressions that “perform an operation on” a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) are used broadly to refer to performing the operation directly on the signal or data, or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on the signal).

Throughout this disclosure, including the claims, the expression "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and the other X-M inputs are received from external sources) may also be referred to as a decoder system.

Throughout this disclosure, including the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., using software or firmware) to perform operations on data (e.g., audio data or video data or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and/or otherwise configured to perform pipeline processing on audio data or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

Throughout this disclosure, including the claims, the expressions "audio processor" and "audio processing unit" are used interchangeably and broadly to refer to a system configured to process audio data. Examples of audio processing units include, but are not limited to, encoders (e.g., transcoders), decoders, codecs, pre-processing systems, post-processing systems, and bitstream processing systems (sometimes referred to as bitstream processing tools).

Throughout this disclosure, including the claims, the expression "metadata" (of an encoded audio bitstream) refers to data that is separate and distinct from the corresponding audio data of the bitstream.

Throughout this disclosure, including the claims, the expression "substream structure metadata" (or "SSM") denotes metadata of a coded audio bitstream (or set of coded audio bitstreams) that indicates the substream structure of the audio content of the coded bitstream.

Throughout this disclosure, including the claims, the expression "program information metadata" (or "PIM") means metadata for an encoded audio bitstream that is indicative of at least one audio program (e.g., two or more audio programs), wherein the metadata indicates at least one attribute or characteristic of the audio content of at least one of the programs (e.g., metadata indicating the type or parameters of processing performed on the program's audio data, or metadata indicating which channels of the program are active channels).

Throughout this disclosure, including the claims, the expression "processing state metadata" (e.g., as in the expression "loudness processing state metadata") refers to metadata associated with audio data of a bitstream (of an encoded audio bitstream) that indicates the processing state of the corresponding (associated) audio data (e.g., what type of processing has been performed on the audio data) and typically also indicates at least one feature or characteristic of the audio data. The association of the processing state metadata with the audio data is time-synchronous. Thus, current (most recently received or updated) processing state metadata indicates that the corresponding audio data also includes the results of the indicated type of audio data processing. In some cases, the processing state metadata may include some or all of the processing history and/or parameters used in and/or derived from the indicated type of processing. Additionally, the processing state metadata may include at least one feature or characteristic of the corresponding audio data that has been calculated or extracted from the audio data. The processing state metadata may also include other metadata that is not related to or derived from any processing of the corresponding audio data. For example, third-party data, tracking information, identifiers, proprietary or standards information, user annotation data, user preference data, etc. may be added by a specific audio processing unit for delivery to other audio processing units.

Throughout this disclosure, including the claims, the expression "loudness processing state metadata" (or "LPSM") refers to processing state metadata that indicates the loudness processing state of corresponding audio data (e.g., what type of loudness processing has been performed on the audio data) and typically also indicates at least one feature or characteristic of the corresponding audio data (e.g., loudness). Loudness processing state metadata may include data that is not (i.e., when considered alone) loudness processing state metadata (e.g., other metadata).

Throughout this disclosure including the claims, the expression "channel" (or "audio channel") refers to a single-channel audio signal.

Throughout this disclosure, including the claims, the expression "audio program" refers to a collection of one or more audio channels and optionally also associated metadata (e.g., metadata describing the desired spatial audio representation, and/or PIM, and/or SSM, and/or LPSM, and/or program boundary metadata).

Throughout this disclosure, including the claims, the expression "program boundary metadata" refers to metadata of a coded audio bitstream, wherein the coded audio bitstream indicates at least one audio program (e.g., two or more programs), and the program boundary metadata indicates the position of at least one boundary (start and/or end) of at least one of the audio programs in the bitstream. For example, the program boundary metadata (of the coded audio bitstream indicating the audio program) may include metadata indicating the position of the start of the program (e.g., the start of the "N"th frame of the bitstream, or the "M"th sample position of the "N"th frame of the bitstream), and additional metadata indicating the position of the end of the program (e.g., the start of the "J"th frame of the bitstream, or the "K"th sample position of the "J"th frame of the bitstream).

Throughout this disclosure, including the claims, the terms "couple" or "coupled" are used to mean either a direct or indirect connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

DETAILED DESCRIPTION

A typical audio data stream includes both audio content (e.g., one or more channels of audio content) and metadata indicating at least one characteristic of the audio content. For example, in an AC-3 bitstream, there are several audio metadata parameters specifically intended to alter the sound of a program delivered to a listening environment. One of these metadata parameters is the DIALNORM parameter, which is intended to indicate the average level of dialogue in an audio program and is used to determine the audio playback signal level.

During playback of a bitstream that includes a series of different audio program segments (each with different DIALNORM parameters), the AC-3 decoder uses the DIALNORM parameters for each segment to perform a type of loudness processing in which the AC-3 decoder modifies the playback level or loudness so that the perceived loudness of the dialogue for the series of segments is at a consistent level. Each encoded audio segment (item) in a series of encoded audio items will (usually) have different DIALNORM parameters, and the decoder will scale the levels of each item in the item so that the playback level or loudness of the dialogue for each item is the same or very similar, although this may require that different amounts of gain be applied to different items in the item during playback.

DIALNORM is typically set by the user rather than automatically generated, however, there is a default DIALNORM value if the user does not set a value. For example, a content creator might perform loudness measurement using a device external to the AC-3 encoder and then pass that result (indicating the loudness of the spoken dialogue of the audio program) to the encoder to set the DIALNORM value. Thus, the content creator is reliant on correctly setting the DIALNORM parameter.

There are several different reasons why the DIALNORM parameter in an AC-3 bitstream may be incorrect. First, if the DIALNORM value is not set by the content creator, each AC-3 encoder has a default DIALNORM value used during bitstream generation. This default value may differ significantly from the actual dialogue loudness of the audio. Second, even if the content creator measures loudness and sets the DIALNORM value accordingly, a loudness measurement algorithm or meter that does not conform to the recommended AC-3 loudness measurement method may have been used, resulting in an incorrect DIALNORM value. Third, even if the AC-3 bitstream was created using the DIALNORM value correctly measured and set by the content creator, the AC-3 bitstream may have been altered to an incorrect value during the bitstream's transmission and/or storage. For example, this is not uncommon in television broadcast applications where an AC-3 bitstream is decoded, modified, and then re-encoded using incorrect DIALNORM metadata information. Consequently, the DIALNORM value included in the AC-3 bitstream may be incorrect or inaccurate, potentially negatively impacting the quality of the listening experience.

Furthermore, the DIALNORM parameter does not indicate the loudness processing state of the corresponding audio data (e.g., what type of loudness processing has been performed on the audio data). Loudness processing state metadata (in the format in which it is provided in some embodiments of the present invention) helps facilitate adaptive loudness processing of audio bitstreams and/or verification of the validity of the loudness processing state and loudness of audio content in a particularly efficient manner.

Although the present invention is not limited to use with AC-3 bitstreams, E-AC-3 bitstreams, or Dolby E bitstreams, for convenience it will be described in embodiments that generate, decode, or otherwise process such bitstreams.

The AC-3 encoded bitstream includes metadata and 1 to 6 channels of audio content. The audio content is audio data that has been compressed using perceptual audio coding. The metadata includes several audio metadata parameters intended to change the sound of the program being delivered to the listening environment.

Each frame of an AC-3 encoded audio bitstream contains audio content and metadata about 1536 samples of digital audio. For a 48kHz sampling rate, this represents 32 milliseconds of digital audio, or 31.25 frames per second of audio.

Each frame of the E-AC-3 coded audio bitstream contains audio data and metadata for 256, 512, 768, or 1536 samples of digital audio, depending on whether the frame contains 1, 2, 3, or 6 blocks of audio data, respectively. For a sampling rate of 48kHz, this represents 5.333, 10.667, 16, or 32 milliseconds of digital audio, respectively, or an audio rate of 189.9, 93.75, 62.5, or 31.25 frames per second, respectively.

As shown in Figure 4, each AC-3 frame is divided into parts (segments), including: a synchronization information (SI) part containing (as shown in Figure 5) a synchronization word (SW) and the first of two error correction words (CRC1); a bit stream information (BSI) part containing most of the metadata; 6 audio blocks (AB0 to AB5) containing data compressed audio content (and may also include metadata); a useless bit segment (W) (also called a "skip field") containing any unused bits remaining after the compressed audio content; an auxiliary (AUX) information part that may contain more metadata; and the second of the two error correction words (CRC2).

As shown in Figure 7, each E-AC-3 frame is divided into parts (segments), including: a synchronization information (SI) part containing the synchronization word (SW) (as shown in Figure 5); a bit stream information (BSI) part containing most of the metadata; 6 audio blocks (AB0 to AB5) containing data compressed audio content (and may also include metadata); a useless bit segment (W) (also called a "skip field") containing any unused bits remaining after the compressed audio content (although only one useless bit segment is shown, a different useless bit segment or skip field segment may typically follow each audio block); an auxiliary (AUX) information part that may contain more metadata; and an error correction word (CRC).

In an AC-3 (or E-AC-3) bitstream, there are several audio metadata parameters that are specifically intended to change the sound of a program delivered to a listening environment. One of the metadata parameters is the DIALNORM parameter, which is included in the BSI segment.

As shown in Figure 6, the BSI segment of the AC-3 frame includes a 5-bit parameter ("DIALNORM") indicating the DIALNORM value of the program. If the audio coding mode ("acmod") of the AC-3 frame is 0, a 5-bit parameter ("DIALNORM2") indicating the 5-bit DIALNORM value of the second audio program carried in the same AC-3 frame is included, indicating the use of a dual mono channel or "1+1" channel configuration.

The BSI segment also includes a flag ("addbsie") indicating the presence (or absence) of additional bitstream information following the "addbsie" bits, a parameter ("addbsil") indicating the length of any additional bitstream information following the "addbsil" value, and up to 64 bits of additional bitstream information following the "addbsil" value ("addbsi").

The BSI segment includes other metadata values not specifically shown in FIG. 6 .

According to one class of embodiments, an encoded bitstream indicates multiple substreams of audio content. In some cases, the substreams indicate the audio content of a multi-channel program, and each of the substreams indicates one or more of the program's channels. In other cases, the multiple substreams of the encoded audio bitstream indicate the audio content of several audio programs, typically a "main" audio program (which may be a multi-channel program) and at least one other audio program (e.g., a program that provides commentary about the main audio program).

The encoded audio bitstream indicating at least one audio program needs to include at least one "independent" substream of audio content. The independent substream indicates at least one channel of the audio program (for example, the independent substream may indicate the five full-range channels of a conventional 5.1-channel audio program). In this document, this audio program is referred to as the "main" program.

In some types of implementations, an encoded audio bitstream indicates two or more audio programs (a "main" program and at least one other audio program). In such a case, the bitstream includes two or more independent substreams: a first independent substream indicating at least one channel of the main program; and at least one other independent substream indicating at least one channel of another audio program (a program different from the main program). Each independent substream can be decoded independently, and the decoder can be operated to decode only a subset (not all) of the independent substreams of the encoded bitstream.

In a typical example of an encoded audio bitstream indicating two independent substreams, one of the independent substreams indicates standard format speaker channels of a multi-channel main program (e.g., left, right, center, left surround, and right surround full-range speaker channels of a 5.1-channel main program), while the other independent substream indicates a single-channel audio commentary about the main program (e.g., a director's commentary about a movie, where the main program is the movie's soundtrack). In another example of an encoded audio bitstream indicating multiple independent substreams, one of the independent substreams indicates standard format speaker channels of a multi-channel main program (e.g., a 5.1-channel main program) including dialogue in a first language (e.g., one of the speaker channels of the main program may indicate dialogue), while each of the other independent substreams indicates a single-channel translation (into a different language) of the dialogue.

Optionally, the coded audio bitstream indicating a primary program (and optionally at least one other audio program) includes at least one "slave" substream of audio content. Each slave substream is associated with an independent substream of the bitstream and indicates at least one additional channel of the program (e.g., primary program) whose content is indicated by the associated independent substream (i.e., the slave substream indicates at least one channel of the program that is not indicated by the associated independent substream, while the associated independent substream indicates at least one channel of the program).

In the example of an encoded bitstream including an independent substream (indicating at least one channel of a main program), the bitstream also includes a dependent substream (associated with the independent substream) indicating one or more additional speaker channels for the main program. Such additional speaker channels are in addition to the main program channels indicated by the independent substream. For example, if the independent substream indicates the left, right, center, left surround, and right surround full-range speaker channels of a 7.1-channel main program, the dependent substream may indicate the other two full-range speaker channels of the main program.

According to the E-AC-3 standard, an E-AC-3 bitstream must indicate at least one independent substream (e.g., a single AC-3 bitstream) and can indicate up to 8 independent substreams. Each independent substream of an E-AC-3 bitstream can be associated with up to 8 dependent substreams.

An E-AC-3 bitstream includes metadata indicating the substream structure of the bitstream. For example, the "chanmap" field in the Bitstream Information (BSI) portion of an E-AC-3 bitstream determines the channel mapping for the program channels indicated by the bitstream's subordinate substreams. However, metadata indicating the substream structure is conventionally included in an E-AC-3 bitstream in a format that facilitates access and use only by an E-AC-3 decoder (during decoding of an encoded E-AC-3 bitstream) and is not convenient for access and use after decoding (e.g., by a post-processor) or before decoding (e.g., by a processor configured to identify the metadata). Furthermore, there is a risk that a decoder might incorrectly identify substreams of a conventional E-AC-3 encoded bitstream using the conventionally included metadata, and prior to the present invention, it was not known how to include substream structure metadata in an encoded bitstream (e.g., an encoded E-AC-3 bitstream) in a format that allows convenient and efficient detection and correction of errors in substream identification during decoding of the bitstream.

An E-AC-3 bitstream may also include metadata about the audio content of an audio program. For example, an E-AC-3 bitstream indicating an audio program includes metadata indicating the minimum and maximum frequencies at which spectral spreading (and channel coupling coding) was used to encode the program's content. However, such metadata is typically included in the E-AC-3 bitstream in a format that facilitates access and use only by an E-AC-3 decoder (during decoding of an encoded E-AC-3 bitstream); it is not convenient for access and use after decoding (e.g., by a post-processor) or before decoding (e.g., by a processor configured to recognize the metadata). Furthermore, such metadata is not included in the E-AC-3 bitstream in a format that allows for convenient identification of such metadata during decoding of the bitstream and for efficient error detection and error correction.

According to an exemplary embodiment of the present invention, a PIM and/or SSM (and optionally other metadata, such as loudness processing state metadata or "LPSM") is embedded in one or more reserved fields (or slots) of a metadata segment of an audio bitstream, which also includes audio data in other segments (the audio data segments). Typically, at least one segment of each frame of the bitstream includes a PIM or SSM, and at least one other segment of the frame includes corresponding audio data (i.e., audio data whose data structure is indicated by the SSM and/or whose at least one characteristic or attribute is indicated by the PIM).

In one class of embodiments, each metadata segment is a data structure (sometimes referred to herein as a container) that can contain one or more metadata payloads. Each payload includes a header that provides an explicit indication of the type of metadata present in the payload, wherein the header includes a specific payload identifier (or payload configuration data). The order of the payloads within the container is undefined, so that the payloads can be stored in any order and a parser must be able to parse the entire container to extract relevant payloads while ignoring irrelevant or unsupported payloads. FIG8 (described below) illustrates the structure of such a container and the payloads within the container.

Communicating metadata (e.g., SSM and/or PIM and/or LPSM) in an audio data processing chain is particularly useful when two or more audio processing units need to work cooperatively with each other throughout the processing chain (or content lifecycle). Without metadata included in the audio bitstream, for example, when two or more audio codecs are utilized in the chain and single-ended volume is applied more than once during the bitstream path (or rendering point of the audio content of the bitstream) of a media consumption device, several media processing issues, such as quality, level, and spatial degradation, can arise.

According to some embodiments of the present invention, loudness processing state metadata (LPSM) embedded in an audio bitstream can be authenticated and verified, for example, to enable a loudness adjustment entity to verify whether the loudness of a particular program is within a specified range and whether the corresponding audio data itself has not been modified (thereby ensuring compliance with applicable regulations). The loudness value included in the data block containing the loudness processing state metadata can be read to verify this without recalculating the loudness. In response to the LPSM, a management structure can determine that the corresponding audio content complies with loudness legal and/or regulatory requirements (e.g., rules promulgated under the Commercial Loudness Mitigation Act, also known as the "CALM" method) (as indicated by the LPSM) without needing to calculate the loudness of the audio content.

FIG1 is a block diagram of an exemplary audio processing chain (audio data processing system) in which one or more of the system's elements may be configured according to embodiments of the present invention. The system includes the following elements coupled together as shown: a preprocessing unit, an encoder, a signal analysis and metadata correction unit, a transcoder, a decoder, and the preprocessing unit. In variations of the illustrated system, one or more of the elements may be omitted, or additional audio data processing units may be included.

In some implementations, the pre-processing unit of FIG. 1 is configured to receive as input PCM (time domain) samples comprising audio content and output processed PCM samples. The encoder can be configured to receive as input the PCM samples and output an encoded (e.g., compressed) audio bitstream indicating the audio content. Data indicating the bitstream of audio content is sometimes referred to herein as "audio data." If the encoder is configured according to an exemplary embodiment of the present invention, the audio bitstream output from the encoder includes the PIM and/or SSM (optionally also including loudness processing state metadata and/or other metadata) as well as the audio data.

The signal analysis and metadata correction unit of FIG1 can receive one or more coded audio bitstreams as input and determine (e.g., verify) whether the metadata (e.g., processing state metadata) in each coded audio bitstream is correct by performing signal analysis (e.g., using program boundary metadata in the coded audio bitstream). If the signal analysis and metadata correction unit finds that the included metadata is invalid, it typically replaces the erroneous value with the correct value obtained from the signal analysis. Thus, each coded audio bitstream output from the signal analysis and metadata correction unit can include corrected (or uncorrected) processing state metadata as well as the coded audio data.

The code converter of Fig. 1 can receive coded audio bitstream as input, and in response (for example, by decoding the input stream and re-encoding the decoded stream with a different coding format) output modified (for example, differently encoded) audio bitstream. If the code converter is configured according to an exemplary embodiment of the present invention, the audio bitstream output from the code converter includes SSM and/or PIM (usually also including other metadata) and coded audio data. The metadata may have been included in the input bitstream.

The decoder of Figure 1 can receive an encoded (e.g., compressed) audio bitstream as input and output (in response) a stream of decoded PCM audio samples. If the decoder is configured according to an exemplary embodiment of the present invention, then in typical operation, the output of the decoder is or includes any one of the following:

a stream of audio samples, and at least one corresponding stream of SSM and/or PIM (and typically other metadata) extracted from the input coded bitstream; or

a stream of audio samples, and a corresponding stream of control bits determined from the SSM and/or PIM (and typically other metadata, such as LPSM) extracted from the input coded bitstream; or

A stream of audio samples but no corresponding stream of metadata or control bits determined based on the metadata. In the latter case, the decoder may extract the metadata from the input coded bitstream and perform at least one operation (e.g., validation) on the extracted metadata even if the extracted metadata or control bits determined based on the metadata are not output.

By configuring the post-processing unit of FIG1 according to an exemplary embodiment of the present invention, the post-processing unit is configured to receive a decoded PCM audio sample stream and perform post-processing (e.g., volume leveling of the audio content) using the SSM and/or PIM (typically with other metadata, such as LPSM) received with the samples, or based on control bits determined by the metadata received with the samples. The post-processing unit is also typically configured to render the post-processed audio content for playback by one or more speakers.

Typical embodiments of the present invention provide an enhanced audio processing chain in which audio processing units (e.g., encoders, decoders, transcoders, and pre-processing and post-processing units) modify their respective processing to be applied to audio data based on a contemporaneous state of the media data indicated by metadata respectively received by the audio processing units.

The audio data input to any audio processing unit of the FIG1 system (e.g., the encoder or transcoder of FIG1 ) may include the SSM and/or PIM (optionally, other metadata) along with the audio data (e.g., encoded audio data). The metadata may have been included in the input audio by another element of the FIG1 system (or another source, not shown in FIG1 ) according to an embodiment of the present invention. The processing unit that receives the input audio (with metadata) may be configured to perform at least one operation on the metadata (e.g., validation) or respond to the metadata (e.g., adaptive processing of the input audio), and may also typically include the metadata, a processed version of the metadata, or control bits determined based on the metadata in its output audio.

Typical embodiments of the audio processing unit (or audio processor) of the present invention are configured to perform adaptive processing of audio data based on the state of the audio data indicated by metadata corresponding to the audio data. In some embodiments, the adaptive processing is (or includes) loudness processing (if the metadata indicates that loudness processing or processing similar to loudness processing has not yet been performed on the audio data), rather than (and does not include) loudness processing (if the metadata indicates that such loudness processing or processing similar to loudness processing has already been performed on the audio data). In some embodiments, the adaptive processing is or includes metadata verification (e.g., performed in a metadata verification subunit) to ensure that the audio processing unit performs other adaptive processing of the audio data based on the state of the audio data indicated by the metadata. In some embodiments, the verification determines the reliability of metadata associated with the audio data (e.g., included in a bitstream with the audio data). For example, if the metadata is verified to be reliable, the results from a previously performed audio processing can be reused and a new execution of the same type of audio processing can be avoided. On the other hand, if the metadata is found to have been tampered with (or is otherwise unreliable), then a type of media processing purportedly previously performed (as indicated by the unreliable metadata) may be repeated by the audio processing unit, and/or other processing may be performed by the audio processing unit on the metadata and/or audio data. If the unit determines that the metadata is valid (e.g., based on a match between the extracted encryption value and the reference encryption value), the audio processing unit may also be configured to signal to other audio processing units downstream in the enhanced media processing chain that the metadata (e.g., present in the media bitstream) is valid.

FIG2 is a block diagram of an encoder (100) as an embodiment of an audio processing unit of the present invention. Any component or element of the encoder 100 can be implemented in hardware or software or a combination of hardware and software as one or more processes and/or one or more circuits (e.g., ASICs, FPGAs, or other integrated circuits). The encoder 100 includes a frame buffer 110, an analyzer 111, a decoder 101, an audio state validator 102, a loudness processing stage 103, an audio stream selection stage 104, an encoder 105, a filler/formatter stage 107, a metadata generation stage 106, a dialog loudness measurement subsystem 108, and a frame buffer 109, connected as shown. The encoder 100 typically also includes other processing elements (not shown).

The encoder 100 (which is a transcoder) is configured to convert an input audio bitstream (e.g., one of an AC-3 bitstream, an E-AC-3 bitstream, or a Dolby E bitstream) into an encoded output audio bitstream (e.g., another of an AC-3 bitstream, an E-AC-3 bitstream, or a Dolby E bitstream) by performing adaptive and automatic loudness processing using loudness processing state metadata included in the input bitstream. For example, the encoder 100 can be configured to convert an input Dolby E bitstream (which is a format commonly used in production and broadcast equipment, but not in consumer equipment that receives broadcast audio programs) into an encoded output audio bitstream in an AC-3 or E-AC-3 format (suitable for broadcast to consumer equipment).

The system of FIG2 also includes an encoded audio transmission subsystem 150 (which stores and/or transmits the encoded bitstream output from encoder 100) and a decoder 152. The encoded audio bitstream output from encoder 100 can be stored by subsystem 150 (e.g., in a DVD or Blu-ray Disc format), transmitted by subsystem 150 (which can implement a transmission line or network), or stored and transmitted by subsystem 150. Decoder 152 is configured to decode the encoded audio bitstream (generated by encoder 100) received via subsystem 150 by extracting metadata (PIM and/or SSM, and optionally loudness processing state metadata and/or other metadata) from each frame of the bitstream (and optionally also extracting program boundary metadata from the bitstream) and generating decoded audio data. Typically, decoder 152 is configured to perform adaptive processing on the decoded audio data using the PIM and/or SSM and/or LPSM (optionally also using program boundary metadata), and/or forward the decoded audio data and metadata to a post-processor configured to perform adaptive processing on the decoded audio data using the metadata. Typically, decoder 152 includes a buffer that stores (eg, in a non-transitory manner) an encoded audio bitstream received from subsystem 150 .

Various implementations of the encoder 100 and decoder 152 are configured to perform different embodiments of the method of the present invention.

The frame buffer 110 is a buffer memory coupled to receive an encoded input audio bitstream. In operation, the buffer 110 stores (e.g., in a non-transient manner) at least one frame of the encoded audio bitstream, and a sequence of frames of the encoded audio bitstream is set from the buffer 110 to the analyzer 111.

The analyzer 111 is coupled and configured to extract the PIM and/or SSM, as well as loudness processing state metadata (LPSM), and optionally also program boundary metadata (and/or other metadata) from each frame of the encoded input audio including such metadata, and to provide at least the LPSM (and optionally also program boundary metadata and/or other metadata) to the audio state validator 102, the loudness processing stage 103, the stage 106, and the subsystem 108 to extract audio data from the encoded input audio and to provide the audio data to the decoder 101. The decoder 101 of the encoder 100 is configured to decode the audio data to generate decoded audio data, and to provide the decoded audio data to the loudness processing stage 103, the audio stream selection stage 104, the subsystem 108, and typically also to the state validator 102.

The state validator 102 is configured to authenticate and verify the LPSM (and optionally other metadata) set thereto. In some embodiments, the LPSM is (or is included in) a data block that is already included in the input bitstream (e.g., according to embodiments of the present invention). The block may include a cryptographic hash (a hash-based message authentication code or "HMAC") for processing the LPSM (and optionally other metadata) and/or the underlying audio data (provided to the validator 102 from the decoder 101). In these embodiments, the data block may be digitally signed so that downstream audio processing units can relatively easily authenticate and verify the processing state metadata.

For example, HMAC is used to generate a digest, and the protection value included in the bitstream of the present invention may include the digest. The digest may be generated as follows with respect to the AC-3 frame:

1. After the AC-3 data and LPSM are encoded, the frame data bytes (concatenated frame data #1 and frame data #2) and the LPSM data bytes are used as input to the hash function HMAC. Other data that may be present in the ancillary data field is not considered for computing the digest. Such other data may be bytes that are neither AC-3 data nor LSPSM data. The protection bits included in the LPSM are not considered for computing the HMAC digest.

2. After the digest is calculated, it is written to the field reserved for protection bits in the bitstream.

3. The final step in generating a complete AC-3 frame is the calculation of the CRC checksum. This is written at the end of the frame and takes into account all the data belonging to the frame, including the LPSM bits.

Other encryption methods, including but not limited to any one of one or more non-HMAC encryption methods, may be used for verification of the LPSM and/or other metadata (e.g., in the verifier 102) to ensure secure transmission and reception of the metadata and/or the underlying audio data. For example, verification (using such an encryption method) may be performed in each audio processing unit of an embodiment of the present invention that receives an audio bitstream to determine whether the metadata and corresponding audio data included in the bitstream have undergone (and/or have been generated by) a specific processing (indicated by the metadata) and have not been modified after such specific processing has been performed.

The state validator 102 sets control data to the audio stream selection stage 104, metadata generator 106, and dialog loudness measurement subsystem 108 to indicate the results of the validation operation. In response to the control data, the stage 104 may select (and pass to the encoder 105):

the adaptively processed output of the loudness processing stage 103 (e.g., when the LPSM indicates that the audio data output from the decoder 101 has not undergone a particular type of loudness processing, and the control bit from the validator 102 indicates that the LPSM is valid); or

Audio data output from decoder 102 (e.g., when the LPSM indicates that the audio data output from decoder 101 has been subjected to a particular type of loudness processing to be performed by stage 103, and the control bit from validator 102 indicates that the LPSM is valid).

The stage 103 of the encoder 100 is configured to perform adaptive loudness processing on the decoded audio data output from the decoder 101 based on one or more audio data characteristics indicated by the LPSM extracted by the decoder 101. The stage 103 may be an adaptive transform-domain real-time loudness and dynamic range control processor. The stage 103 may receive user input (e.g., a user target loudness/dynamic range value or a dialogue normalization value), or other metadata input (e.g., one or more types of third-party data, tracking information, identifiers, proprietary or standards information, user annotation data, user preference data, etc.), and/or other input (e.g., from a fingerprinting process), and use such input to process the decoded audio data output from the decoder 101. Stage 103 may perform adaptive loudness processing on decoded audio data (output from decoder 101) indicating a single audio program (represented by program boundary metadata extracted by analyzer 111), and may reset the loudness processing in response to receiving decoded audio data (output from decoder 101) indicating a different audio program indicated by program boundary metadata extracted by analyzer 111.

When the control bit from validator 102 indicates that the LPSM is invalid, the dialog loudness measurement subsystem 108 may be operative to use the LPSM (and/or other metadata) extracted by decoder 101 to determine the loudness of a segment of decoded audio (from decoder 101) representing dialogue (or other speech). When the control bit from validator 102 indicates that the LPSM is valid, operation of the dialog loudness measurement subsystem 108 may be disabled when the LPSM indicates a previously determined loudness of a segment of dialogue (or other speech) for the decoded audio (from decoder 101). Subsystem 108 may perform loudness measurement on decoded audio data representing a single audio program (as indicated by program boundary metadata extracted by analyzer 111), and may reset loudness processing in response to receiving decoded audio data representing a different audio program as indicated by such program boundary metadata.

There are useful tools (e.g., the Dolby LM100 Loudness Meter) for conveniently and easily measuring the level of dialogue in audio content. Some embodiments of the APU of the present invention (e.g., stage 108 of encoder 100) are implemented to include such a tool (or perform the functionality of such a tool) to measure the average dialogue loudness of the audio content of an audio bitstream (e.g., the decoded AC-3 bitstream provided from decoder 101 of encoder 100 to stage 108).

If stage 108 is implemented to measure the true average dialogue loudness of the audio data, the measurement may include a step of separating segments of audio content that primarily contain speech. The primarily speech audio segments are then processed according to a loudness measurement algorithm. For audio data decoded from an AC-3 bitstream, the algorithm may be a standard K-weighted loudness measurement (according to the international standard ITU-R BS1770). Alternatively, other loudness measures (e.g., those based on psychoacoustic models of loudness) may be used.

Separation of speech segments is not necessary to measure the average dialogue loudness of audio data. However, it improves the accuracy of the measurement and generally provides more favorable results from the listener's perspective. Because not all audio content contains dialogue (speech), loudness measurements of the entire audio content can provide a sufficient approximation of the dialogue level for audio where speech is present.

The metadata generator 106 generates (and/or passes to the stage 107) metadata to be included by the stage 107 in the encoded bitstream to be output from the encoder 100. The metadata generator 106 may pass the LPSM (and optionally the LIM and/or PIM and/or program boundary metadata and/or other metadata) extracted by the encoder 101 and/or the analyzer 111 to the stage 107 (e.g., when the control bit from the validator 102 indicates that the LPSM and/or other metadata are valid), or generate new LIM and/or PIM and/or LPSM and/or program boundary metadata and/or other metadata and set the new metadata to the stage 107 (e.g., when the control bit from the validator 102 indicates that the metadata extracted by the decoder 101 is invalid), or may set a combination of the metadata extracted by the decoder 101 and/or the analyzer 111 and the newly generated metadata to the stage 107. Metadata generator 106 may include the loudness data generated by subsystem 108 and at least one value indicating the type of loudness processing performed by subsystem 108 in an LPSM, setting the LPSM to stage 107 for inclusion in the encoded bitstream to be output from encoder 100 .

The metadata generator 106 may generate control bits (which may consist of or include a hash-based message authentication code or "HMAC") for at least one of decryption, authentication, or verification of the LPSM (and optionally other metadata) to be included in the encoded bitstream and/or in the elementary audio data to be included in the encoded bitstream. The metadata generator 106 may provide such protection bits to the stage 107 for inclusion in the encoded bitstream.

In typical operation, the dialogue loudness measurement subsystem 108 processes the audio data output from the decoder 101 to generate loudness values (e.g., gated and ungated dialogue loudness values) and dynamic range values in response to the audio data. In response to these values, the metadata generator 106 may generate loudness processing state metadata (LPSM) for inclusion (by the stuffer/formatter 107) in the encoded bitstream to be output from the encoder 100.

Additionally, optionally, or alternatively, subsystems 106 and/or 108 of encoder 100 may perform additional analysis of the audio data to generate metadata indicative of at least one characteristic of the audio data for inclusion in the encoded bitstream to be output from stage 107 .

Encoder 105 encodes the audio data output from selection stage 104 (eg, by performing compression thereon), and sets the encoded audio to stage 107 for inclusion in an encoded bitstream to be output from stage 107 .

Stage 107 multiplexes the encoded audio from encoder 105 and metadata (including PIM and/or SSM) from generator 106 to generate an encoded bitstream to be output from stage 107, preferably such that the encoded bitstream has a format specified by preferred embodiments of the invention.

The frame buffer 109 is a buffer memory that stores (e.g., in a non-transitory manner) at least one frame of the encoded audio bitstream output from stage 107, and then a series of frames of the encoded audio bitstream are set from the buffer 109 as output from the encoder 100 to the transmission system 150.

The LPSM generated by metadata generator 106 and included in the encoded bitstream by stage 107 generally indicates a loudness processing state of the corresponding audio data (e.g., what type of loudness processing has been performed on the audio data) and the loudness of the corresponding audio data (e.g., measured dialogue loudness, gated and/or ungated loudness, and/or dynamic range).

As used herein, "gating" of loudness and/or level measurements performed on audio data refers to a specific level or loudness threshold above which calculated values exceeding the threshold are included in the final measurement (e.g., short-term loudness values below -60 dBFS are omitted from the final measurement). Absolute gating refers to a fixed level or loudness, while relative gating refers to a value that is dependent on the current "ungated" measurement value.

In some implementations of encoder 100, the encoded bitstream buffered in memory 109 (and output to delivery system 150) is an AC-3 bitstream or an E-AC-3 bitstream and includes an audio data segment (e.g., segments AB0 to AB5 of a frame shown in FIG4 ) and a metadata segment, wherein the audio data segment indicates audio data and at least some of the metadata segments each include a PIM and/or SSM (and optionally other metadata). Stage 107 inserts the metadata segment (including the metadata) into the bitstream in the following format. Each of the metadata segments including the PIM and/or SSM is included in a useless bit segment of the bitstream (e.g., the useless bit segment "W" shown in FIG4 or FIG7 ), or in an "addbsi" field of a bitstream information ("BSI") segment of a frame of the bitstream, or in an auxiliary data field at the end of a frame of the bitstream (e.g., the AUX segment shown in FIG4 or FIG7 ). A frame of a bitstream may include one or two metadata segments, each metadata segment including metadata, and if a frame includes two metadata segments, one may be present in the addbsi field of the frame and the other in the AUX field of the frame.

In some embodiments, each metadata segment (sometimes referred to herein as a "container") inserted by stage 107 has a format that includes a metadata segment header (optionally including other mandatory or "core" elements) and one or more metadata payloads following the metadata segment header. If present, the SIM is included in one of the metadata payloads (identified by the payload header and generally having a first type of format). If present, the PIM is included in another of the metadata payloads (identified by the payload header and generally having a second type of format). Similarly, each other type of metadata (if present) is included in another of the metadata payloads (identified by the payload header and generally having a format specific to the type of metadata). The exemplary format enables convenient access to the SSM, PIM, and other metadata at times other than during decoding (e.g., by a post-processor after decoding, or by a processor configured to identify metadata without performing a full decoding of the encoded bitstream), and allows for convenient and efficient error detection and correction during decoding of the bitstream (e.g., substream identification). For example, without access to the SSM in the exemplary format, a decoder may incorrectly identify the correct number of substreams associated with a program. One metadata payload in the metadata segment may include the SSM, another metadata payload in the metadata segment may include the PIM, and optionally, at least one other metadata payload in the metadata segment may include other metadata (e.g., loudness processing state metadata or "LPSM").

In some embodiments, a substream structure metadata (SSM) payload included (by stage 107) in a frame of an encoded bitstream (e.g., an E-AC-3 bitstream indicating at least one audio program) includes an SSM in the following format:

A payload header, typically including at least one identification value (e.g., a 2-bit value indicating the SSM format version, and optionally length, period, count, and substream association values); and following the header:

independent substream metadata indicating the number of independent substreams for the program indicated by the bitstream; and

Dependent substream metadata indicating whether each independent substream of the program has at least one associated dependent substream (i.e., whether at least one dependent substream is associated with each independent substream) and, if so, the number of dependent substreams associated with each independent substream of the program.

It is contemplated that an independent substream of an encoded bitstream may indicate a set of speaker channels for an audio program (e.g., the speaker channels for a 5.1 speaker channel audio program), and that each of one or more dependent substreams (associated with the independent substream, as indicated by the dependent substream metadata) may indicate a target channel for the program. However, an independent bitstream of an encoded bitstream typically indicates a set of speaker channels for a program, and each dependent substream associated with the independent substream (as indicated by the dependent substream metadata) indicates at least one additional speaker channel for the program.

In some embodiments, a program information metadata (PIM) payload included (by stage 107) in a frame of an encoded bitstream (e.g., an E-AC-3 bitstream indicating at least one audio program) has the following format:

A payload header, typically including at least one identification value (e.g., a value indicating the PIM format version, and optionally length, period, count, and substream association values); and, following the header, a PIM in the following format:

Active channel metadata indicating each silent channel and each non-silent channel of an audio program (i.e., which channels of the program contain audio information and which channels, if any, contain only silence (typically with respect to the duration of a frame)). In embodiments where the encoded bitstream is an AC-3 or E-AC-3 bitstream, the active channel metadata in a frame of the bitstream may be combined with additional metadata of the bitstream (e.g., the audio coding mode ("acmod") field of the frame and, if present, the chanmap field in the frame or associated dependent substream frame) to determine which channels of the program contain audio information and which channels contain silence. The "acmod" field of an AC-3 or E-AC-3 frame indicates the number of full-range channels of the audio program indicated by the audio content of the frame (e.g., whether the program is a 1.0 channel mono program, a 2.0 channel stereo program, or a program including L, R, C, Ls, Rs full-range channels), or whether the frame indicates two independent 1.0 channel mono programs. The "chanmap" field of an E-AC-3 bitstream indicates the channel mapping of the dependent substream indicated by the bitstream. Active channel metadata can help to implement upmixing (in a post-processor) downstream of the decoder, for example to add audio to a channel containing silence at the output of the decoder;

Downmix processing state metadata indicating whether the program was downmixed (before or during encoding) and, if so, the type of downmix that was applied. The downmix processing state metadata may aid in implementing upmixing (in a post-processor) downstream of a decoder, for example, to upmix the program's audio content using parameters that best match the type of downmix that was applied. In embodiments where the encoded bitstream is an AC-3 or E-AC-3 bitstream, the downmix processing state metadata may be combined with the audio coding model ("acmod") field of the frame to determine the type of downmix (if any) applied to the program's channels;

Upmix processing state metadata indicating whether the program was upmixed (e.g., from a smaller number of channels) before or during encoding, and if so, the type of upmix that was applied. The upmix processing state metadata can aid in implementing downmixing (in a post-processor) downstream of a decoder, such as downmixing the program's audio content in a manner consistent with the type of upmix applied to the program (e.g., Dolby Pro Logic, or Dolby Pro Logic II Movie mode, or Dolby Pro Logic II Music mode, or Dolby Professional upmixer). In embodiments where the encoded bitstream is an E-AC-3 bitstream, the upmix processing state metadata can be combined with other metadata (e.g., the value of a frame's "strmtyp" field) to determine the type of upmix (if any) applied to the program's channels. the value of the "strmtyp" field (in the BSI field of a frame of the E-AC-3 bitstream) indicates whether the audio content of the frame belongs to an independent stream (which identifies a program) or an independent substream (comprising multiple substreams or a program associated with multiple substreams), and thus can be encoded independently of any other substream indicated by the E-AC-3 bitstream, or whether the audio content of the frame belongs to a dependent substream (of a program comprising multiple substreams or associated with multiple substreams), and thus must be decoded in conjunction with the independent substream with which it is associated; and

Pre-processing status metadata indicating whether pre-processing was performed on the audio content of the frame (prior to encoding of the audio content to generate the encoded bitstream), and if so, the type of pre-processing that was performed on the audio content of the frame.

In some implementations, the pre-processing state metadata indicates:

whether surround attenuation is applied (e.g., whether the surround channels of an audio program are attenuated by 3dB before encoding),

whether to apply a 90° phase shift (e.g., to the surround channels Ls and Rs of the audio program before encoding),

Whether to apply a low-pass filter to the LFE channel of the audio program before encoding.

During generation, whether to monitor the level of the program's LFE channel and, if so, the monitored level of the LFE channel relative to the level of the program's full-range audio channels,

whether dynamic range compression should be performed (e.g., in a decoder) on each block of the program's decoded audio content, and if so, the type (and/or parameters) of dynamic range compression to be performed on each block of the program's decoded audio content (e.g., the type of pre-processing state metadata may indicate which of the following compression profile types was assumed by the encoder to generate the dynamic range compression control values included in the encoded bitstream: film standard, film mild, music standard, music mild, or speech. Alternatively, the type of pre-processing state metadata may indicate that heavy dynamic range compression ("compr" compression) should be performed on each frame of the program's decoded audio content in a manner determined by the dynamic range compression control values included in the encoded bitstream),

Whether spectral spreading and/or channel coupling coding is used to encode program content for a particular frequency range, and if spectral spreading and/or channel coupling coding is used to encode program content for a particular frequency range, the minimum and maximum frequencies of the frequency components of the content on which spectral spreading coding is performed, and the minimum and maximum frequencies of the frequency components of the content on which channel coupling coding is performed. This type of pre-processing state metadata information can be helpful in performing equalization downstream of the decoder (in a post-processor). Both channel coupling information and spectral spreading information help to optimize quality during transcoding operations and applications. For example, the encoder can optimize its behavior (including adaptation of pre-processing steps such as headphone virtualization, upmixing, etc.) based on parameters such as the state of spectral spreading and channel coupling information. Moreover, the encoder can dynamically modify its coupling parameters and spectral spreading parameters to match optimal values and/or modify its coupling and spectral spreading parameters to optimal values based on the state of the incoming (and authenticated) metadata, and

Whether dialogue enhancement adjustment range data is included in the encoded bitstream, and if so, the range of adjustments that may be obtained during performance of dialogue enhancement processing (e.g., in a post-processor downstream of a decoder) to adjust the level of dialogue content relative to the level of non-dialogue content in the audio program.

In some implementations, additional pre-processing state metadata (eg, metadata indicating headset-related parameters) is included (by stage 107 ) in the PIM payload of the encoded bitstream to be output from encoder 100 .

In some implementations, an LPSM payload included (by stage 107) in a frame of an encoded bitstream (e.g., an E-AC-3 bitstream indicating at least one audio program) includes an LPSM in the following format:

A header (typically comprising a synchronization word identifying the start of the LPSM payload, followed by at least one identification value, such as the LPSM format version, length, period, count, and substream association value shown in Table 2 below); and

After the header:

at least one dialogue indication value (e.g., parameter “Dialogue Channel” of Table 2) indicating whether the corresponding audio data indicates dialogue or does not indicate dialogue (e.g., which channels of the corresponding audio data indicate dialogue);

at least one loudness adjustment compliance value of the indicated set indicating whether the corresponding audio content complies with loudness adjustment (e.g., parameter “loudness adjustment type” of Table 2);

at least one loudness processing value indicating at least one type of loudness processing that has been performed on the corresponding audio data (e.g., one or more of the parameters "Dialogue Gating Loudness Correction Flag," "Loudness Correction Type," of Table 2); and

At least one loudness value indicating at least one loudness (e.g., peak or average loudness) characteristic of corresponding audio data (e.g., one or more of the parameters “ITU Relative Gated Loudness,” “ITU Speech Gated Loudness,” “ITU (EBU 3341) Short-Term 3s Loudness,” and “True Peak” of Table 2).

In some implementations, each metadata segment containing a PIM and/or SSM (and optionally other metadata) includes a metadata segment header (and optionally additional core elements), and following the metadata segment header (or metadata segment header and other core elements) at least one metadata payload segment having the following format:

The payload header, which typically includes at least one identification value (e.g., SSM or PIM format version, length, period, count, and substream association value), and

An SSM or PIM (or another type of metadata) follows the payload header.

In some implementations, each of the metadata segments (sometimes referred to herein as "metadata containers" or "containers") inserted by stage 107 into the useless bit segment/skip field segment (or "addbsi" field or ancillary data field) of a frame of the bitstream has the following format:

A metadata segment header (typically including a synchronization word identifying the start of the metadata segment, followed by an identification value such as version, length, period, extended element count, and substream association value as shown in Table 1 below); and

at least one protection value (e.g., the HMAC digest and audio fingerprint value of Table 1) following the metadata segment header that facilitates at least one of decryption, authentication, or verification of at least one of the metadata segment or metadata of corresponding audio data; and

Also following the metadata segment header are metadata payload identification ("ID") values and payload configuration values that identify the type of metadata in each following metadata payload and indicate at least one aspect of the configuration (e.g., size) of each such payload.

Each metadata payload is followed by the corresponding payload ID value and payload configuration value.

In some embodiments, each of the metadata fields in the useless bits field (or ancillary data field or "addbsi" field) of a frame has three levels of structure:

a high-level structure (e.g., a metadata section header) that includes a flag indicating whether the useless bits (or ancillary data or addbsi) field includes metadata, at least one ID value indicating what type of metadata is present, and typically also a value indicating how many bits of metadata (e.g., of each type) are present (if any metadata is present). One type of metadata that may be present is PIM, another type of metadata that may be present is SSM, and still other types of metadata that may be present are LPSM, and/or program boundary metadata, and/or media search metadata;

an intermediate hierarchical structure comprising data associated with each identified type of metadata (e.g., a metadata payload header, a protection value, and a payload ID value and a payload configuration value for each identified type of metadata); and

A low-level structure that includes a metadata payload for each identified type of metadata (e.g., if PIM is identified as being present, a series of PIM values, and/or if that other type of metadata is identified as being present, metadata values of another type (e.g., SSM or LPSM)).

In this way, data values in the three hierarchical structures can be nested. For example, the protection value for each payload identified by the high-level structure and the middle-level structure (e.g., each PIM, or SSM, or other data payload) can be included after the payload (and thus after the metadata payload header of the payload), or the protection value for all metadata payloads identified by the high-level structure and the middle-level structure can be included after the final metadata payload in the metadata segment (and thus after the metadata payload headers of all payloads in the metadata segment).

In one example (described with reference to the metadata segment or "container" in FIG8 ), the metadata segment header identifies four metadata payloads. As shown in FIG8 , the metadata segment header includes a container synchronization word (identified as "Container Sync"), as well as version and key ID values. Following the metadata segment header are the four metadata payloads and protection bits. The payload ID value and payload configuration (e.g., payload size) value of the first payload (e.g., PIM payload) follow the metadata segment header, the first payload itself follows the ID and configuration values, the payload ID value and payload configuration (e.g., payload size) value of the second payload (e.g., SSM payload) follows the first payload, the second payload itself follows these ID and configuration values, the payload ID value and payload configuration (e.g., payload size) value of the third payload (e.g., LPSM payload) follows the second payload, the third payload itself follows these ID and configuration values, the payload ID value and payload configuration (e.g., payload size) value of the fourth payload follows the third payload, the fourth payload itself follows these ID and configuration values, and protection values (identified as "protection data" in FIG. 8 ) regarding all or some of the payloads (or regarding the high-level structure and the intermediate-level structure and all or some of the payloads) follow the last payload.

In some embodiments, if the decoder 101 receives an audio bitstream having a cryptographic hash generated according to an embodiment of the present invention, the decoder is configured to analyze and retrieve the cryptographic hash based on data blocks determined by the bitstream, wherein the blocks include metadata. The verifier 102 can use the cryptographic hash to verify the received bitstream and/or associated metadata. For example, if the verifier 102 finds that the metadata is valid based on a match between a reference cryptographic hash and a cryptographic hash retrieved from the data block, then the processor 103 can be prohibited from operating on the corresponding audio data and the selection stage 104 can be caused to pass the (unchanged) audio data. In addition, other types of encryption techniques can be used, optionally or alternatively, instead of the cryptographic hash-based approach.

The encoder 100 of FIG2 can determine (in response to the LPSM extracted by the decoder 101 and optionally also in response to program boundary metadata) that the post-processing/pre-processing unit has performed a type of loudness processing on the audio data to be encoded (in elements 105, 106, and 107), and can therefore create (in generator 106) loudness processing state metadata that includes specific parameters for and/or derived from the previously performed loudness processing. In some implementations, the encoder 100 can create metadata indicating the processing history of the audio content (and include it in the encoded bitstream output from the encoder) as long as the encoder knows the type of processing that has been performed on the audio content.

3 is a block diagram of a decoder (200) and a post-processor (300) coupled to the decoder (200), which is an embodiment of an audio processing unit of the present invention. The post-processor (300) is also an embodiment of an audio processing unit of the present invention. Any of the components or elements of the encoder 200 and the post-processor 300 can be implemented as one or more processes and/or one or more circuits (e.g., ASICs, FPGAs, or other integrated circuits) in hardware, software, or a combination of hardware and software. The decoder 200 includes a frame buffer 201, an analyzer 205, an audio decoder 202, an audio state verification stage (verifier) 203, and a control bit generation stage 204, connected as shown. Typically, the decoder 200 also includes other processing elements (not shown).

The frame buffer 201 (buffer memory) stores (eg, in a non-transitory manner) at least one frame of the encoded audio bitstream received by the decoder 200. A sequence of frames of the encoded audio bitstream is set from the buffer 201 to the analyzer 205.

The analyzer 205 is coupled and configured to extract the PIM and/or SSM (and optionally other metadata, e.g., LPSM) from each frame of the encoded input audio, set at least some of the metadata (e.g., LPSM and program boundary metadata, if any is extracted, and/or the PIM and/or SSM) to the audio state validator 203 and stage 204, set the extracted metadata as output (e.g., to the post-processor 300), extract audio data from the encoded input audio, and set the extracted audio data to the decoder 202.

The encoded audio bitstream input to the decoder 200 may be one of an AC-3 bitstream, an E-AC-3 bitstream, or a Dolby E bitstream.

The system of FIG3 also includes a post-processor 300. Post-processor 300 includes a frame buffer 301 and other processing elements (not shown) including at least one processing element coupled to buffer 301. Frame buffer 301 stores (e.g., in a non-transitory manner) at least one frame of a decoded audio bitstream received by post-processor 300 from decoder 200. The processing elements of post-processor 300 are coupled and configured to receive a series of frames of the decoded audio bitstream output from buffer 301 and adaptively process them using metadata output from decoder 200 and/or control bits output from stage 204 of decoder 200. Generally, post-processor 300 is configured to perform adaptive processing on the decoded audio data using metadata from decoder 200 (e.g., to perform adaptive loudness processing on the decoded audio data using LPSM values and, optionally, program boundary metadata, where the adaptive processing may be based on a loudness processing state and/or one or more audio data characteristics indicated by an LPSM indicating audio data of a single audio program).

Various implementations of the decoder 200 and the post-processor 300 are configured to perform different embodiments of the method of the present invention.

The audio decoder 202 of the decoder 200 is configured to decode the audio data extracted by the analyzer 205 to generate decoded audio data, and set the decoded audio data as output (eg, to the post-processor 300 ).

The state validator 203 is configured to authenticate and verify metadata set thereto. In some embodiments, the metadata is (or is included in) a data block that has been included in an input bitstream (e.g., according to an embodiment of the present invention). The block may include a cryptographic hash (hash-based message authentication code or "HMAC") for processing the metadata and/or the base audio data (provided to the validator 203 from the analyzer 205 and/or decoder 202). The data block can be digitally signed in these embodiments so that downstream audio processing units can relatively easily authenticate and verify the processing state metadata.

Other encryption methods, including but not limited to any one of one or more non-HMAC encryption methods, may be used for metadata verification (e.g., in the verifier 203) to ensure secure transmission and reception of metadata and/or underlying audio data. For example, verification (using such an encryption method) may be performed in each audio processing unit that receives an embodiment of the audio bitstream of the present invention to determine whether the metadata and corresponding audio data included in the bitstream have undergone (and/or resulted from) a specific processing (indicated by the metadata) and have not been modified after such specific processing has been performed.

The state validator 203 sets the control data to the control bit generator 204 and/or sets the control data as an output (e.g., to the post-processor 300) to indicate the result of the validation operation. In response to the control data (and optionally other metadata extracted from the input bitstream), the stage 204 may generate (and set to the post-processor 300):

a control bit indicating that the decoded audio data output from the decoder 202 has been subjected to a particular type of loudness processing (when the LPSM indicates that the audio data output from the decoder 202 has been subjected to the particular type of loudness processing and the control bit from the validator 203 indicates that the LPSM is valid); or

A control bit indicating that the decoded audio data output from the decoder 202 should be subjected to a particular type of loudness processing (e.g., when the LPSM indicates that the audio data output from the decoder 202 has not been subjected to the particular type of loudness processing, or when the LPSM indicates that the audio data output from the decoder 202 has been subjected to the particular type of loudness processing but the control bit from the validator 203 indicates that the LPSM is invalid).

Alternatively, the decoder 200 sets the metadata extracted from the input bit stream by the decoder 202 and the metadata extracted from the input bit stream by the analyzer 205 to the post-processor 300, and the post-processor 300 performs adaptive processing on the decoded audio data using the metadata, or performs verification of the metadata, and then performs adaptive processing on the decoded audio data using the metadata if the verification indicates that the metadata is valid.

In some embodiments, if decoder 200 receives an audio bitstream generated according to an embodiment of the present invention using cryptographic hashing, the decoder is configured to analyze and retrieve cryptographic hashes from data blocks identified by the bitstream, the blocks comprising loudness processing state metadata (LPSM). Validator 203 can use the cryptographic hashes to validate the received bitstream and/or associated metadata. For example, if validator 203 finds that the LPSM is valid based on a match between a reference cryptographic hash and a cryptographic hash retrieved from the data block, then a signal can be sent to a downstream audio processing unit (e.g., post-processor 300, which may be or include a volume leveling unit) to pass the audio data of the bitstream (unchanged). Additionally, optionally or alternatively, other types of cryptographic techniques can be used in place of cryptographic hash-based approaches.

In some implementations of decoder 200, the encoded bitstream received (and buffered in memory 201) is an AC-3 bitstream or an E-AC-3 bitstream and includes an audio data segment (e.g., segments AB0 to AB5 of a frame shown in FIG. 4 ) and a metadata segment, wherein the audio data segment indicates audio data and at least some of the metadata segments each include a PIM or SSM (or other metadata). Decoder stage 202 (and/or analyzer 205) is configured to extract metadata from the bitstream. Each metadata segment, including the PIM and/or SSM (and optionally other metadata), is included in a useless bit field of a frame of the bitstream, in an "addbsi" field of a bitstream information ("BSI") segment of a frame of the bitstream, or in an auxiliary data field at the end of a frame of the bitstream (e.g., the AUX segment shown in FIG. 4 ). A frame of the bitstream may include one or two metadata segments, wherein each metadata segment includes metadata, and if a frame includes two metadata segments, one may be present in the addbsi field of the frame and the other in the AUX field of the frame.

In some embodiments, each metadata segment (sometimes referred to herein as a "container") of a bitstream buffered in buffer 201 has a format that includes a metadata segment header (optionally including other mandatory or "core" elements) and one or more metadata payloads following the metadata segment header. If present, the SIM is included in one of the metadata payloads (identified by the payload header and generally having a first type of format). If present, the PIM is included in another of the metadata payloads (identified by the payload header and generally having a second type of format). Similarly, other types of metadata (if present) are included in another of the metadata payloads (identified by the payload header and generally having a format specific to the type of metadata). This exemplary format enables convenient access to the SSM, PIM, and other metadata at times other than during decoding (e.g., by post-processor 300 after decoding, or by a processor configured to identify metadata without performing a full decoding of the encoded bitstream), and allows for convenient and efficient error detection and correction during decoding of the bitstream (e.g., substream identification). For example, without access to the SSM in the exemplary format, decoder 200 may incorrectly identify the correct number of substreams associated with a program. One metadata payload in the metadata segment may include the SSM, another metadata payload in the metadata segment may include the PIM, and optionally, at least one other metadata payload in the metadata segment may include other metadata (e.g., loudness processing state metadata or "LPSM").

In some embodiments, a substream structure metadata (SSM) payload included in a frame of a coded bitstream (e.g., an E-AC-3 bitstream indicating at least one audio program) buffered in buffer 201 includes an SSM in the following format:

A payload header, typically including at least one identification value (e.g., a 2-bit value indicating the SSM format version, and optionally length, period, count, and substream association values); and

After the header:

independent substream metadata indicating the number of independent substreams for the program indicated by the bitstream; and

Dependent substream metadata indicating whether each independent substream of the program has at least one dependent substream associated with it, and if so, the number of dependent substreams associated with each independent substream of the program.

In some embodiments, the program information metadata (PIM) payload included in a frame of an encoded bitstream (e.g., an E-AC-3 bitstream indicating at least one audio program) buffered in buffer 201 has the following format:

A payload header, typically including at least one identification value (e.g., a value indicating the PIM format version, and optionally length, period, count, and substream association values); and, following the header, a PIM in the following format:

active channel metadata for each silent channel and each non-silent channel of an audio program (i.e., which channels of the program contain audio information and which channels, if any, contain only silence (typically with respect to the duration of a frame)). In embodiments where the coded bitstream is an AC-3 or E-AC-3 bitstream, the active channel metadata in a frame of the bitstream may be combined with additional metadata of the bitstream (e.g., the audio coding mode ("acmod") field of the frame and, if present, the chanmap field in the frame or associated dependent substream frames) to determine which channels of the program contain audio information and which channels contain silence;

Downmix processing state metadata that indicates whether the program was downmixed (before or during encoding), and if so, the type of downmix that was applied. The downmix processing state metadata may facilitate upmixing downstream of the decoder (in the post-processor 300), for example, to upmix the program's audio content using parameters that best match the type of downmix that was applied. In embodiments where the encoded bitstream is an AC-3 or E-AC-3 bitstream, the downmix processing state metadata may be combined with the frame's audio coding model ("acmod") field to determine the type of downmix (if any) applied to the program's channels;

Upmix processing state metadata that indicates whether the program was upmixed (e.g., from a smaller number of channels) before or during encoding, and if so, the type of upmix that was applied. The upmix processing state metadata can aid in implementing downmixing (in a post-processor) downstream at a decoder, for example, by downmixing the program's audio content in a manner consistent with the type of upmix applied to the program (e.g., Dolby Pro Logic, or Dolby Pro Logic II Movie mode, or Dolby Pro Logic II Music mode, or Dolby Professional upmixer). In embodiments where the encoded bitstream is an E-AC-3 bitstream, the upmix processing state metadata can be combined with other metadata (e.g., the value of a frame's "strmtyp" field) to determine the type of upmix (if any) applied to the program's channels. the value of the "strmtyp" field (in the BSI field of a frame of the E-AC-3 bitstream) indicates whether the audio content of the frame belongs to an independent stream (which identifies a program) or an independent substream (comprising multiple substreams or a program associated with multiple substreams), and thus can be encoded independently of any other substream indicated by the E-AC-3 bitstream, or whether the audio content of the frame belongs to a dependent substream (of a program comprising multiple substreams or associated with multiple substreams), and thus must be decoded in conjunction with the independent substream with which it is associated; and

Pre-processing status metadata that indicates whether pre-processing was performed on the audio content of the frame (prior to encoding of the audio content to generate the encoded bitstream), and if so, the type of pre-processing that was performed.

In some implementations, the pre-processing state metadata indicates:

whether surround attenuation is applied (e.g., whether the surround channels of the audio program are attenuated by 3dB before encoding),

whether a 90° phase shift is applied (e.g. to the surround channels Ls and Rs of the audio program before encoding),

Whether to apply a low-pass filter to the LFE channel of the audio program before encoding,

During generation, whether to monitor the level of the program's LFE channel, and if so, the monitored level of the LFE channel relative to the level of the program's full-range audio channels,

whether dynamic range compression should be performed (e.g., in a decoder) on each block of the program's decoded audio, and if so, the type (and/or parameters) of dynamic range compression to be performed (e.g., the type of pre-processing state metadata may indicate which of the following compression profile types was assumed by the encoder to generate the dynamic range compression control values included in the encoded bitstream: Film Standard, Film Mild, Music Standard, Music Mild, or Speech. Alternatively, the type of pre-processing state metadata may indicate that heavy dynamic range compression ("compr" compression) should be performed on each frame of the program's decoded audio content in a manner determined by the dynamic range compression control values included in the encoded bitstream),

Whether spectral spreading and/or channel coupling coding is used to encode the content of a program in a specific frequency range, and if spectral spreading and/or channel coupling coding is used to encode the content of a program in a specific frequency range, the minimum and maximum frequencies of the frequency components of the content on which spectral spreading coding is performed, and the minimum and maximum frequencies of the frequency components of the content on which channel coupling coding is performed. This type of pre-processing state metadata information can be helpful in performing equalization downstream of the decoder (in a post-processor). Both channel coupling information and spectral spreading information also help to optimize quality during transcoding operations and applications. For example, the encoder can optimize its behavior (including adaptation of pre-processing steps such as headphone virtualization, upmixing, etc.) based on the state of parameters (such as spectral spreading and channel coupling information). Moreover, the encoder can dynamically modify its coupling and spectral spreading parameters to match optimal values and/or modify its coupling and spectral spreading parameters to optimal values based on the state of the incoming (and authenticated) metadata, and

Whether dialogue enhancement adjustment range data is included in the encoded bitstream, and if so, the adjustment range available during performance of dialogue enhancement processing (e.g., in a post-processor downstream of a decoder) to adjust the level of dialogue content relative to the level of non-dialogue content in the audio program.

In some embodiments, the LPSM payload included in a frame of an encoded bitstream (e.g., an E-AC-3 bitstream indicating at least one audio program) buffered in the buffer 201 includes an LPSM in the following format:

A header (typically comprising a synchronization word identifying the start of the LPSM payload, followed by at least one identification value, such as the LPSM format version, length, period, count, and substream association value indicated in Table 2 below); and

After the header:

at least one dialogue representation value (e.g., parameter “Dialogue Channel” of Table 2) indicating whether the corresponding audio data indicates dialogue or does not indicate dialogue (e.g., which channels of the corresponding audio data indicate dialogue);

at least one loudness adjustment compliance value of the indicated set indicating whether the corresponding audio content complies with loudness adjustment (e.g., parameter “loudness adjustment type” of Table 2);

at least one loudness processing value indicating at least one type of loudness processing that has been performed on the corresponding audio data (e.g., one or more of the parameters "Dialogue Gating Loudness Correction Flag," "Loudness Correction Type," of Table 2); and

At least one loudness value indicating at least one loudness (e.g., peak or average loudness) characteristic of corresponding audio data (e.g., one or more of the parameters “ITU Relative Gated Loudness,” “ITU Speech Gated Loudness,” “ITU (EBU 3341) Short-Term 3s Loudness,” and “True Peak” of Table 2).

In some implementations, the analyzer 205 (and/or the decoder stage 202) is configured to extract each metadata segment having the following format from the useless bits segment or "addbsi" field or the auxiliary data segment of a frame of the bitstream:

A metadata segment header (typically including a synchronization word that identifies the start of the metadata segment, followed by identification values such as version, length, period, extended element count, and substream association value); and

at least one protection value (e.g., the HMAC digest and audio fingerprint value of Table 1) following the metadata segment header that facilitates at least one of decryption, authentication, or verification of at least one of the metadata segment or metadata of corresponding audio data; and

Also following the metadata segment header are metadata payload identification ("ID") values and payload configuration values that identify the type of metadata in each following metadata payload and represent at least one aspect of the configuration (e.g., size) of each such payload.

Each metadata payload segment (preferably having the format specified above) is followed by a corresponding metadata payload ID value and metadata configuration values.

More generally, the coded audio bitstream generated by the preferred embodiment of the present invention has a structure that provides a mechanism for marking metadata elements and sub-elements as core (mandatory) or extended (optional) elements or sub-elements. This enables the data rate of the bitstream (including its metadata) to be scalable to a large number of applications. The core (mandatory) elements of the preferred bitstream syntax should also be able to signal the presence of extended (optional) elements associated with the audio content (in-band) and/or remotely (out-of-band).

Core elements are required to be present in every frame of the bitstream. Some sub-elements of core elements are optional and may be present in any combination. Extension elements are not required to be present in every frame (to limit bitrate overhead). Thus, extension elements may be present in some frames and not in others. Some sub-elements of extension elements are optional and may be present in any combination, however, some sub-elements of extension elements may be mandatory (i.e., if the extension element is present in a frame of the bitstream).

In one class of embodiments, an encoded audio bitstream is generated (e.g., by implementing an audio processing unit of the present invention) comprising a series of audio data segments and metadata segments. The audio data segments indicate audio data, at least some of the metadata segments each comprise a PIM and/or SSM (and optionally at least one other type of metadata), and the audio data segments are time-division multiplexed with the metadata segments. In a preferred embodiment of this class, each of the metadata segments has a preferred format as described herein.

In a preferred format, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment including the SSM and/or PIM in the metadata segment is included (e.g., by stage 107 of a preferred implementation of encoder 100) as additional bitstream information in an "addbsi" field (shown in FIG. 6) of a bitstream information ("BSI") segment of a frame of the bitstream, or in an auxiliary data field of a frame of the bitstream, or in a useless bit segment of a frame of the bitstream.

In a preferred format, each of the frames includes a metadata segment (also sometimes referred to herein as a metadata container or container) in the useless bit field (or addbsi field) of the frame. The metadata segment has the mandatory elements (collectively referred to as "core elements") shown in Table 1 below (and may include the optional elements shown in Table 1). At least some of the required elements shown in Table 1 are included in the metadata segment header of the metadata segment, but some may be included in other locations in the metadata segment:

Table 1

In the preferred format, each metadata segment containing an SSM, PIM, or LPSM (in the useless bit segment or addbsi or auxiliary data field of a frame of the coded bitstream) contains a metadata segment header (and optionally additional core elements), and one or more metadata payloads following the metadata segment header (or metadata segment header and other core elements). Each metadata payload includes a metadata payload header (indicating the specific type of metadata (e.g., SSM, PIM, or LPSM)) included in the payload, followed by metadata of the specific type. In general, the metadata payload header includes the following values (parameters):

A payload ID (identifying the type of metadata, e.g., SSM, PIM, or LPSM) following the metadata segment header (which may include the values specified in Table 1);

The payload configuration value following the payload ID (usually indicating the size of the payload);

and optionally also including additional payload configuration values (e.g., an offset value indicating the number of audio samples from the start of the frame to the first audio sample to which the payload relates, and a payload priority value, e.g., indicating conditions under which a payload may be discarded).

Typically, payload metadata has one of the following formats:

The metadata of the payload is an SSM, comprising independent substream metadata indicating the number of independent substreams of the program indicated by the bitstream; and dependent substream metadata indicating whether each independent substream of the program has at least one dependent substream associated therewith, and if each independent substream of the program has at least one dependent substream associated therewith, the number of dependent substreams associated with each independent substream of the program;

The metadata of the payload is a PIM, including active channel metadata indicating which channels of the audio program contain audio information and which channels, if any, contain only silence (typically for the duration of a frame); downmix processing state metadata indicating whether the program was downmixed (before or during encoding), and if the program was downmixed, the type of downmixing applied; upmix processing state metadata indicating whether the program was upmixed (e.g., from a smaller number of channels) before or during encoding, and if the program was upmixed, the type of upmixing applied; and preprocessing state metadata indicating whether preprocessing was performed on the audio data of the frame (before encoding to generate the audio content of the coded bitstream), and if so, the type of preprocessing performed on the audio data of the frame; or

The metadata of the payload is LPSM having a format as indicated in the following table (Table 2):

Table 2

In another preferred format of an encoded bitstream generated according to the present invention, the bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment (e.g., by stage 107 of a preferred implementation of encoder 100) that includes a PIM and/or SSM (and optionally at least one other type of metadata) is included in any of the following: a useless bit segment of a frame of the bitstream; or an "addbsi" field of a bitstream information ("BSI") segment of a frame of the bitstream (as shown in FIG6); or an auxiliary data field at the end of a frame of the bitstream (e.g., the AUX segment shown in FIG4). A frame may include one or two metadata segments, each of which includes a PIM and/or SSM, and (in some embodiments) if a frame includes two metadata segments, one may be present in the addbsi field of the frame and the other in the AUX field of the frame. Each metadata segment preferably has the format specified above with reference to Table 1 above (i.e., includes the core elements specified in Table 1, followed by a payload ID value (identifying the type of metadata in each payload of the metadata segment) and a payload configuration value, and each metadata payload). Each metadata segment including an LPSM preferably has the format specified above with reference to Tables 1 and 2 above (i.e., includes the core elements specified in Table 1, followed by a payload ID (identifying the metadata as an LPSM) and a payload configuration value, followed by a payload (LPSM data having the format indicated in Table 2)).

In another preferred format, the encoded bitstream is a Dolby E bitstream, and each of the metadata segments including the PIM and/or SSM (and optionally other metadata) is the first N sample positions of a Dolby E guard band interval. A Dolby E bitstream including such a metadata segment including the LPSM preferably includes a value indicating the LPSM payload length signaled in the Pd word of the SMPTE 337M preamble (the SMPTE 337M Pa word repetition frequency preferably remains the same as the associated video frame rate).

In a preferred format, where the encoded bitstream is an E-AC-3 bitstream, each metadata segment in the metadata segment that includes the PIM and/or SSM (and optionally also the LPSM and/or other metadata) is included (e.g., by stage 107 of a preferred implementation of encoder 100) as additional bitstream information in an "addbsi" field of a useless bit segment or bitstream information ("BSI") segment of a frame of the bitstream. Additional aspects of encoding an E-AC-3 bitstream using the LPSM in this preferred format are described below:

1. During the generation of the E-AC-3 bitstream, while the E-AC-3 encoder (which inserts the LPSM value into the bitstream to be generated) is "active", for each generated frame (synchronization frame), the bitstream shall include the metadata block (including LPSM) carried in the addbsi field (or useless bit field) of the frame. It is required that the bits carrying the metadata block shall not increase the encoder bit rate (frame length);

2. Each metadata block (including LPSM) should contain the following information:

Loudness correction type flag: where "1" indicates that the loudness of the corresponding audio data is corrected upstream of the encoder, while "0" indicates that the loudness is corrected by a loudness corrector embedded in the encoder (e.g., the loudness processor 103 of the encoder 100 of FIG. 2 );

Speech channels: Indicates which source channels contain speech (in the previous 0.5 seconds). If no speech is detected, this should be indicated as such;

Speech Loudness: Indicates the integrated speech loudness of each corresponding audio channel including speech (in the previous 0.5 seconds);

ITU Loudness: Indicates the combined ITU BS.1770-3 loudness of each corresponding audio channel; and

Gain: The inverted loudness composite gain in the decoder (to indicate reversibility);

3. When the E-AC-3 encoder (which inserts LPSM values into the bitstream) is "active" and is receiving AC-3 frames with the "trust" flag, the loudness controller in the encoder (e.g., loudness processor 103 of encoder 100 of FIG. 2 ) should be bypassed. The "trusted" source dialogue normalization and DRC values should be passed (e.g., by generator 106 of encoder 100 ) to the E-AC-3 encoder components (e.g., stage 107 of encoder 100 ). LPSM block generation continues, and the loudness correction type flag is set to "1." The loudness controller bypass sequence must be synchronized to the start of the decoded AC-3 frame where the "trust" flag appears. The loudness controller bypass sequence should be implemented as follows: the leveler amount control is reduced from a value of 9 to a value of 0 over 10 audio block periods (i.e., 53.3 milliseconds), and the leveler returns to the end. The meter control is placed in bypass mode (this operation should result in a seamless transition). The term "trusted" bypass by the conditioner implies that the source bitstream's dialogue normalization value is also reused at the encoder's output. (For example, if the "trusted" source bitstream has a dialogue normalization value of -30, then the encoder's output should use -30 for the output dialogue normalization value).

4. When the E-AC-3 encoder (which inserts LPSM values into the bitstream) is "active" and is receiving AC-3 frames without the "trust" flag, the loudness controller embedded in the encoder (e.g., loudness processor 103 of encoder 100 of FIG. 2 ) should be active. LPSM block generation continues, and the loudness correction type flag is set to "0." The loudness controller activation sequence should be synchronized to the start of the decoded AC-3 frame where the "trust" flag disappears. The loudness controller activation sequence should be implemented as follows: the Leveler Amount control is increased from a value of 0 to a value of 9 across 1 audio block period (e.g., 5.3 milliseconds), and the Leveler Return End Meter control is placed in "active" mode (this operation should result in a seamless transition and include a return end meter integrated reset); and

5. During encoding, the Graphical User Interface (GUI) should indicate the following parameters to the user: "Input Audio Program: [Trusted/Untrusted]" - the state of this parameter is based on the presence of a "Trusted" flag within the input signal; and "Real-time Loudness Correction: [Enabled/Disabled]" - the state of this parameter is based on whether the loudness controller embedded in the encoder is active.

When decoding an AC-3 or E-AC-3 bitstream that has LSPM (in the preferred format) included in the useless bit segment or skip field segment or the "addbsi" field of the bitstream information ("BSI") segment of each frame of the bitstream, the decoder should parse the LPSM block data (in the useless bit segment or addbsi field) and pass all extracted LPSM values to the graphical user interface (GUI). The set of extracted LPSM values is refreshed at each frame.

In another preferred format of an encoded bitstream generated according to the present invention, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment including the PIM and/or SSM (and optionally also the LPSM and/or other metadata) in the metadata segment (e.g., by stage 107 of a preferred implementation of encoder 100) is included in the useless bits segment or AUX segment of a frame of the bitstream or as additional bitstream information in the "addbsi" field (shown in FIG. 6) of the bitstream information ("BSI") segment. In this format (which is a variation of the format described above with reference to Tables 1 and 2), each field in the addbsi (or AUX or useless bits) field containing the LPSM contains the following LPSM values:

The core elements specified in Table 1 are followed by a payload ID (identifying the metadata as LPSM) and a payload value, followed by a payload (LPSM data) having the following format (similar to the mandatory elements shown in Table 2 above):

LPSM payload version: a 2-bit field indicating the version of the LPSM payload;

dialchan: A 3-bit field indicating the left, right, and/or center channel of the corresponding audio data containing spoken dialogue. The bit assignment of the dialchan field may be as follows: bit 0, indicating the presence of dialogue in the left channel, is stored in the most significant bit of the dialchan field; and bit 2, indicating the presence of dialogue in the center channel, is stored in the least significant bit of the dialchan field. Each bit of the dialchan field is set to "1" if the corresponding channel contains spoken dialogue during the first 0.5 seconds of the program;

loudregtyp: A 4-bit field that indicates which loudness adjustment standard the program loudness complies with. Setting the "loudregtyp" field to "0000" indicates that the LPSM does not indicate loudness adjustment compliance. For example, one value of this field (e.g., 0000) may indicate that no loudness adjustment standard compliance is indicated, another value of this field (e.g., 0001) may indicate that the program's audio data complies with the ATSC A/85 standard, and another value of this field (e.g., 0010) may indicate that the program's audio data complies with the EBU R128 standard. In this example, if this field is set to any value other than "0000", the loudcorrdialgat and loudcorrtyp fields should follow in the payload;

loudcorrdialgat: A 1-bit field indicating whether dialogue gating correction has been applied. If the program's loudness has been corrected using dialogue gating, the value of the loudcorrdialgat field is set to "1". Otherwise, it is set to "0";

loudcorrtyp: A 1-bit field indicating the type of loudness correction applied to the program. If the program's loudness has been corrected using an infinite look-ahead (file-based) loudness correction process, the value of the loudcorrtyp field is set to "0". If the program's loudness has been corrected using a combination of real-time loudness measurement and dynamic range control, the value of this field is set to "1";

loudrelgate: A 1-bit field indicating whether the relative gated program loudness (ITU) is present. If the loudrelgate field is set to "1", then the 7-bit ituloudrelgat field should follow in the payload;

loudrelgat: A 7-bit field indicating the relative gated program loudness (ITU). This field indicates the integrated loudness of the audio program measured according to ITU-R BS.1770-3 without any gain adjustment due to dialogue normalization and dynamic range compression (DRC) being applied. Values from 0 to 127 are interpreted as -58 LKFS to +5.5 LKFS in steps of 0.5 LKFS;

loudspchgate: A 1-bit field indicating whether speech gated loudness data (ITU) is present. If the loudspchgate field is set to "1", the payload shall be followed by a 7-bit loudspchgat field;

loudspchgate: A 7-bit field indicating the loudness of the speech-gated program. This field indicates the integrated loudness of the entire corresponding audio program measured according to equation (2) of ITU-R BS.1770-3 without any gain adjustment due to dialogue normalization and dynamic range compression being applied. Values from 0 to 127 are interpreted as -58 LKFS to +5.5 LKFS in steps of 0.5 LKFS;

loudstrm3e: A 1-bit field indicating whether short-term (3 seconds) loudness data is present. If this field is set to "1", the payload shall be followed by a 7-bit loudstrm3s field;

loudstrm3s: A 7-bit field indicating the ungated loudness of the first 3 seconds of the corresponding audio program measured according to ITU-R BS.1771-1 without any gain adjustment due to dialogue normalization and dynamic range compression being applied. Values from 0 to 256 are interpreted as -116 LKFS to +11.5 LKFS in steps of 0.5 LKFS;

TruePKE: A 1-bit field indicating whether true peak loudness data is present. If the TruePKE field is set to "1", then the 8-bit TruePK field shall follow in the payload; and

truepk: An 8-bit field indicating the true peak sample value of the program as measured according to Annex 2 of ITU-R BS.1770-3 without any gain adjustment due to dialogue normalization and dynamic range compression being applied. Values from 0 to 256 are interpreted as -116LKFS to +11.5LKFS in steps of 0.5LKFS.

In some embodiments, the core elements of a metadata segment in a useless bit field or ancillary data (or "addbsi") field of a frame of an AC-3 bitstream or an E-AC-3 bitstream include a metadata segment header (typically including an identification value, such as a version), and following the metadata segment header: a value indicating whether the metadata of the metadata segment includes fingerprint data (or other protection value), a value indicating whether external data (related to the audio data corresponding to the metadata of the metadata segment) is present, a payload ID value and a payload configuration value for each type of metadata identified by the core element (e.g., PIM and/or SSM and/or LPSM and/or a type of metadata), and a protection value for at least one type of metadata identified by the metadata segment header (or other core elements of the metadata segment). The metadata payload of the metadata segment follows the metadata segment header and (in some cases) is nested within the core elements of the metadata segment.

The embodiments of the present invention can be implemented in hardware, firmware, or software, or a combination of hardware and software (e.g., as a programmable logic array). Unless otherwise specified, the algorithm or processing included as part of the present invention does not inherently relate to any specific computer or other device. Specifically, various general-purpose machines can be used using programs written according to the teachings herein, or can be more convenient to construct more specific devices (e.g., integrated circuits) to perform required method steps. Thus, the present invention can be implemented by one or more computer programs executed on one or more programmable computer systems (e.g., the implementation of any one of the elements of FIG. 1 , or the encoder 100 (or elements of an encoder) of FIG. 2 , or the decoder (or elements of a decoder) of FIG. 3 , or the post-processor (or elements of a post-processor) of FIG. 3 ), each programmable computer system including at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. Output information is applied to one or more output devices in a known manner.

Each such program can be implemented in any desired computer language (including machine, assembly or high-level procedural, logical or object-oriented programming languages) to communicate with a computer system. In any case, the language can be a compiled language or an interpreted language.

For example, when implemented by a sequence of computer software instructions, the various functions and steps of the embodiments of the present invention may be implemented by a multi-threaded sequence of software instructions running in appropriate digital signal processing hardware, in which case the various devices, steps and functions of the embodiments may correspond to portions of the software instructions.

Each such computer program is preferably stored on or downloaded to a storage medium or device (e.g., solid-state memory or media, magnetic media, or optical media) readable by a general or special purpose programmable computer, and when the storage medium or device is read by a computer system to perform the processes described herein, is used to configure and operate the computer. The system of the present invention may also be implemented as a computer-readable storage medium configured with (e.g., storing) a computer program, wherein the storage medium so configured causes the computer system to operate in a specific and predefined manner to perform the functions described herein.

A number of embodiments of the present invention have been described. However, it will be appreciated that various modifications may be made without departing from the spirit and scope of the present invention. In light of the above teachings, numerous modifications and variations of the present invention are possible. It will be appreciated that, within the scope of the appended claims, the present invention may be practiced otherwise than as specifically described herein.

In addition, the present invention also includes the following embodiments:

(1) An audio processing unit comprising:

buffer memory; and

at least one processing subsystem coupled to the buffer memory, wherein the buffer memory stores at least one frame of an encoded audio bitstream, the frame including program information metadata or substream structure metadata in at least one metadata segment of at least one skip field of the frame and audio data in at least one other segment of the frame, wherein the processing subsystem is coupled and configured to perform at least one of generation of the bitstream, decoding of the bitstream, or adaptive processing of audio data of the bitstream using the metadata of the bitstream, or to perform at least one of authentication or verification of at least one of the audio data or metadata of the bitstream using the metadata of the bitstream,

The metadata segment includes at least one metadata payload, and the metadata payload includes:

header; and

Following the header, at least a portion of the program information metadata or at least a portion of the substream structure metadata.

(2) The audio processing unit of (1), wherein the encoded audio bitstream indicates at least one audio program, and the metadata segment includes a program information metadata payload, the program information metadata payload including:

Program information metadata header; and

Following the program information metadata header is program information metadata indicating at least one attribute or characteristic of the audio content of the program, the program information metadata including active channel metadata indicating each non-silent channel and each silent channel of the program.

(3) The audio processing unit according to (2), wherein the program information metadata further includes at least one of the following metadata:

downmix processing status metadata indicating whether the program is downmixed and, if so, the type of downmixing applied to the program;

upmixing process status metadata indicating whether the program is upmixed and, if so, the type of upmixing applied to the program;

pre-processing status metadata indicating whether pre-processing was performed on the audio content of the frame, and if so, the type of pre-processing performed on the audio content; or

Spectrum spreading processing or channel coupling metadata indicating whether spectrum spreading processing or channel coupling is applied to the program and, if so, the frequency range in which the spectrum spreading or channel coupling is applied.

(4) The audio processing unit of (1), wherein the coded audio bitstream indicates at least one audio program having at least one independent substream of audio content, and the metadata segment includes a substream structure metadata payload, the substream structure metadata payload including:

Substream structure metadata payload header; and

Following the substream structure metadata payload header are independent substream metadata indicating the number of independent substreams of the program and dependent substream metadata indicating whether each independent substream of the program has at least one associated dependent substream.

(5) The audio processing unit according to (1), wherein the metadata segment includes:

metadata segment header;

at least one protection value following the metadata segment header, used for at least one of decryption, authentication, or verification of at least one of the program information metadata, or the substream structure metadata, or the audio data corresponding to the program information metadata or the substream structure metadata; and

A metadata payload identification value and a payload configuration value follow the metadata segment header, wherein the metadata payload follows the metadata payload identification value and the payload configuration value.

(6) The audio processing unit of (5), wherein the metadata segment header includes a synchronization word identifying the start of the metadata segment and at least one identification value following the synchronization word, and the header of the metadata payload includes at least one identification value.

(7) The audio processing unit according to (1), wherein the encoded audio bitstream is an AC-3 bitstream or an E-AC-3 bitstream.

(8) The audio processing unit according to (1), wherein the buffer memory stores the frames in a non-transitory manner.

(9) The audio processing unit according to (1), wherein the audio processing unit is an encoder.

(10) The audio processing unit according to (9), wherein the processing subsystem includes:

a decoding subsystem configured to receive an input audio bitstream and extract input metadata and input audio data from the input audio bitstream;

an adaptive processing subsystem coupled and configured to perform adaptive processing on the input audio data using the input metadata, thereby generating processed audio data; and

An encoding subsystem is coupled and configured to generate the encoded audio bitstream in response to the processed audio data, including by including the program information metadata or the substream structure metadata in the encoded audio bitstream, and setting the encoded audio bitstream to the buffer memory.

(11) The audio processing unit according to (1), wherein the audio processing unit is a decoder.

(12) The audio processing unit according to (11), wherein the processing subsystem is a decoding subsystem coupled to the buffer memory and configured to extract the program information metadata or the substream structure metadata from the encoded audio bitstream.

(13) The audio processing unit according to (1), comprising:

a subsystem coupled to the buffer memory and configured to: extract the program information metadata or the substream structure metadata from the encoded audio bitstream, and extract the audio data from the encoded audio bitstream; and

A post-processor is coupled to the subsystem and configured to perform adaptive processing on the audio data using at least one of the program information metadata or the substream structure metadata extracted from the encoded audio bitstream.

(14) The audio processing unit according to (1), wherein the audio processing unit is a digital signal processor.

(15) An audio processing unit according to (1), wherein the audio processing unit is a preprocessor configured to extract the program information metadata or the substream structure metadata and the audio data from the encoded audio bit stream, and perform adaptive processing on the audio data using at least one of the program information metadata or the substream structure metadata extracted from the encoded audio bit stream.

(16) A method for decoding a coded audio bitstream, the method comprising the following steps:

receiving an encoded audio bitstream; and

Extracting metadata and audio data from the coded audio bitstream, wherein the metadata is or includes program information metadata and substream structure metadata,

The encoded audio bitstream includes a series of frames and indicates at least one audio program, the program information metadata and the substream structure metadata indicate the program, each of the frames includes at least one audio data segment, each of the audio data segment includes at least a portion of the audio data, each frame in at least a subset of the frames includes a metadata segment, and each of the metadata segments includes at least a portion of the program information metadata and at least a portion of the substream structure metadata.

(17) The method of (16), wherein the metadata segment includes a program information metadata payload, the program information metadata payload including:

Program information metadata header; and

The program information metadata header is followed by program information metadata indicating at least one attribute or characteristic of audio content of the program, the program information metadata including active channel metadata indicating each non-muted channel and each muted channel of the program.

(18) The method according to (17), wherein the program information metadata further includes at least one of the following metadata:

downmix processing status metadata indicating whether the program is downmixed and, if so, the type of downmixing applied to the program;

upmixing process status metadata indicating whether the programme is upmixed and, if so, the type of upmixing applied to the programme; or

Pre-processing status metadata indicating whether pre-processing has been performed on the audio content of the frame, and, if so, the type of pre-processing performed on the audio content.

(19) The method of (16), wherein the coded audio bitstream indicates at least one audio program having at least one independent substream of audio content, and the metadata segment includes a substream structure metadata payload, the substream structure metadata payload including:

Substream structure metadata payload header; and

Following the substream structure metadata payload header are independent substream metadata indicating the number of independent substreams of the program and dependent substream metadata indicating whether each independent substream of the program has at least one associated dependent substream.

(20) The method according to (16), wherein the metadata segment includes:

metadata segment header;

at least one protection value following the metadata segment header for at least one of decryption, authentication, or verification of at least one of the program information metadata or the substream structure metadata or the audio data corresponding to the program information metadata and the substream structure metadata; and

Following the metadata segment header is a metadata payload including the at least a portion of the program information metadata and the at least a portion of the sub-stream structure metadata.

(21) The method according to (16), wherein the encoded audio bitstream is an AC-3 bitstream or an E-AC-3 bitstream.

(22) The method according to (16), further comprising the steps of:

Adaptive processing is performed on the audio data using at least one of the program information metadata or the substream structure metadata extracted from the encoded audio bitstream.

Claims (20)

1. An audio processing unit, comprising: At least one processing subsystem is operable to access at least one frame of an encoded audio bitstream, the frame including program information metadata or substream structure metadata in at least one metadata segment of at least one reserved field of the frame and audio data in at least one other segment of the frame, and the processing subsystem is operable to perform at least one of generating the bitstream, decoding the audio data, or adaptive processing of the audio data using the metadata of the bitstream, or to perform at least one of authenticating or verifying at least one of the audio data or metadata of the bitstream using the metadata of the bitstream. The metadata segment includes at least one metadata payload, which includes: Header; and Following the header, at least a portion of the program information metadata or at least a portion of the substream structure metadata, and The reserved field is selected from a group consisting of the skip field, the addbsi field, the auxiliary data field, or a combination thereof. 2. The audio processing unit according to claim 1, wherein the encoded audio bitstream indicates at least one audio program, and the metadata segment includes a program information metadata payload, the program information metadata payload comprising: Program information metadata header; and Following the program information metadata header is program information metadata indicating at least one attribute or characteristic of the audio content of the program, the program information metadata including metadata indicating each non-mute channel and each mute channel of the program. 3. The audio processing unit according to claim 2, wherein the program information metadata further includes at least one of the following metadata: Downmixing status metadata, which indicates whether the program has been downmixed, and if the program has been downmixed, the type of downmixing applied to the program; Upmixing status metadata, which indicates whether the program is upmixed, and if the program is upmixed, the type of upmixing applied to the program; Preprocessing status metadata, indicating whether preprocessing was performed on the audio content of the frame, and if so, the type of preprocessing performed on the audio content; or Spectrum spreading or channel coupling metadata, indicating whether spectrum spreading or channel coupling has been applied to the program, and the frequency range in which spectrum spreading or channel coupling has been applied if spectrum spreading or channel coupling has been applied to the program. 4. The audio processing unit according to claim 1, wherein the encoded audio bitstream indicates at least one audio program of at least one independent substream having audio content, and the metadata segment includes a substream structure metadata payload, the substream structure metadata payload comprising: Substream structure metadata payload header; and Following the substream structure metadata payload header are independent substream metadata indicating the number of independent substreams of the program, and dependent substream metadata indicating whether each independent substream of the program has at least one associated dependent substream. 5. The audio processing unit according to claim 1, wherein the metadata segment comprises: Metadata segment header; At least one protection value following the metadata segment header, used for at least one of the decryption, authentication, or verification of the program information metadata, or the substream structure metadata, or the audio data corresponding to the program information metadata or the substream structure metadata; and The metadata payload identifier value and the payload configuration value follow the metadata segment header, wherein the metadata payload follows the metadata payload identifier value and the payload configuration value. 6. The audio processing unit according to claim 5, wherein the metadata segment header includes a synchronization word identifying the start of the metadata segment and at least one identification value following the synchronization word, and the header of the metadata payload includes at least one identification value. 7. The audio processing unit according to claim 1, wherein the encoded audio bitstream is an AC-3 bitstream or an E-AC-3 bitstream. 8. The audio processing unit according to claim 1, wherein the frame is stored in a buffer memory in a non-transitory manner. 9. The audio processing unit according to claim 1, wherein the audio processing unit is an encoder. 10. The audio processing unit according to claim 9, wherein the processing subsystem comprises: A decoding subsystem capable of operating to receive an input audio bitstream and extract input metadata and input audio data from the input audio bitstream; An adaptive processing subsystem capable of operating to perform adaptive processing on the input audio data using the input metadata, thereby generating processed audio data; and An encoding subsystem capable of operating in response to the processed audio data, including generating the encoded audio bitstream by including the program information metadata or the substream structure metadata in the encoded audio bitstream. 11. The audio processing unit according to claim 1, wherein the audio processing unit is a decoder. 12. The audio processing unit of claim 11, wherein the processing subsystem is a decoding subsystem capable of operating to extract the program information metadata or the substream structure metadata from the encoded audio bitstream. 13. The audio processing unit according to claim 1, comprising: A subsystem capable of operating to: extract the program information metadata or the substream structure metadata from the encoded audio bitstream, and extract the audio data from the encoded audio bitstream; and A post-processor, coupled to the subsystem, is operable to perform adaptive processing on the audio data using at least one of the program information metadata extracted from the encoded audio bitstream or the substream structure metadata. 14. The audio processing unit according to claim 1, wherein the audio processing unit is a digital signal processor. 15. The audio processing unit of claim 1, wherein the audio processing unit is a preprocessor, the preprocessor being operable to extract the program information metadata or the substream structure metadata and the audio data from the encoded audio bitstream, and to perform adaptive processing on the audio data using at least one of the program information metadata or the substream structure metadata extracted from the encoded audio bitstream. 16. A method for decoding an encoded audio bitstream, the method comprising the following steps: Receive encoded audio bitstreams including metadata and audio data; and Extract the metadata or the audio data from the encoded audio bitstream, wherein the metadata is or includes program information metadata or substream structure metadata. The encoded audio bitstream comprises a series of frames and indicates at least one audio program. The program information metadata and the substream structure metadata indicate the program. Each frame includes at least one audio data segment, and each audio data segment includes at least a portion of the audio data. Each frame in at least a subset of the frames includes a metadata segment, and each metadata segment includes at least a portion of the program information metadata and at least a portion of the substream structure metadata. The metadata segment is located in a reserved field, which is selected from a group consisting of a skip field, an addbsi field, an auxiliary data field, or a combination thereof. 17. The method of claim 16, wherein the metadata segment includes a program information metadata payload, the program information metadata payload comprising: Program information metadata header; and The program information metadata following the program information metadata header indicates at least one attribute or characteristic of the audio content of the program, and the program information metadata includes metadata indicating each non-mute channel and each mute channel of the program. 18. The method of claim 17, wherein the program information metadata further comprises at least one of the following metadata: Downmixing status metadata, which indicates whether the program has been downmixed, and if the program has been downmixed, the type of downmixing applied to the program; Upmixing status metadata, indicating whether the program is upmixed, and if so, the type of upmixing applied to the program; or Preprocessing status metadata, which indicates whether preprocessing was performed on the audio content of the frame, and if so, the type of preprocessing performed on the audio content. 19. The method of claim 16, wherein the encoded audio bitstream indicates at least one audio program of at least one independent substream having audio content, and the metadata segment includes a substream structure metadata payload, the substream structure metadata payload comprising: Substream structure metadata payload header; and Following the substream structure metadata payload header are independent substream metadata indicating the number of independent substreams of the program and dependent substream metadata indicating whether each independent substream of the program has at least one associated dependent substream. 20. The method of claim 16, wherein the encoded audio bitstream is an AC-3 bitstream or an E-AC-3 bitstream.