HK40017428B - Audio processing unit, method performed by an audio processing unit and storage medium - Google Patents

Description

Audio processing unit, method executed by audio processing unit, and storage medium

This application is a divisional application of the invention patent application filed on July 31, 2013, with application number "201310329128.8" and invention title "Audio Encoder and Decoder Using Program Information or Substream Structure Metadata".

Technical Field

This invention relates to audio signal processing, and more specifically, to the encoding and decoding of audio data bitstreams having metadata indicating substream structure and/or program information relating to the audio content indicated by the bitstream. Some embodiments of the invention generate or decode audio data in one of the formats known as Dolby Digital (AC-3), Dolby Digital+ (Enhanced AC-3 or E-AC-3), or Dolby E.

Background Technology

Dolby, Dolby Digital, Dolby Digital+, and Dolby E are trademarks of Dolby Laboratories, Inc. Dolby Laboratories provides proprietary implementations of AC-3 and E-AC-3, respectively known as Dolby Digital and Dolby Digital+.

Audio data processing units typically operate blindly and are not concerned with the processing history of audio data that occurred before the data was received. This can work in a processing framework where a single entity performs all audio data processing and encoding for various target media rendering devices, while the target media rendering devices perform all decoding and rendering of the encoded audio data. However, this blind processing does not work well (or not at all) when multiple audio processing units are scattered or chained across diverse networks and are expected to optimally perform their respective types of audio processing. For example, some audio data may be encoded for high-performance media systems and may need to be converted into a simplified form suitable for mobile devices along the media processing chain. Therefore, audio processing units may unnecessarily perform the type of processing on the audio data that has already been performed. For example, a leveling unit may perform processing on an input audio segment regardless of whether the same or similar leveling has been performed on the input audio segment before. Thus, the leveling unit may perform leveling even when it is unnecessary. This unnecessary processing may also lead to the degradation and/or elimination of specific features when rendering the content of the audio data.

Summary of the Invention

This invention discloses an audio processing unit, comprising: one or more processors; a memory coupled to the one or more processors and configured to store instructions, which, when executed by the one or more processors, cause the one or more processors to perform operations, the operations including: receiving an encoded audio bitstream comprising an audio program, the encoded audio bitstream comprising encoded audio data of a set of one or more audio channels and metadata associated with the set of audio channels, wherein the metadata includes loudness processing status metadata, and wherein the loudness processing status metadata includes metadata indicating the loudness of the audio program; decoding the encoded audio data to obtain decoded audio data of the set of audio channels; obtaining the loudness processing status metadata from the metadata of the encoded audio bitstream; and performing adaptive loudness processing on the decoded audio data of the set of audio channels based on the loudness processing status metadata; wherein the metadata further includes program information metadata, the program information metadata indicating a compression profile for creating dynamic range compressed (DRC) data in the bitstream, wherein the compression profile is a music standard compression profile.

The present invention also discloses a method executed by an audio processing unit, comprising: receiving an encoded audio bitstream including an audio program, the encoded audio bitstream including encoded audio data of a set of one or more audio channels and metadata associated with the set of audio channels, wherein the metadata includes loudness processing status metadata, and wherein the loudness processing status metadata includes metadata indicating the loudness of the audio program; decoding the encoded audio data to obtain decoded audio data of the set of audio channels; obtaining the loudness processing status metadata from the metadata of the encoded audio bitstream; and performing adaptive loudness processing on the decoded audio data of the set of audio channels based on the loudness processing status metadata; wherein the metadata further includes program information metadata, the program information metadata indicating a compression profile for creating dynamic range compressed DRC data in the bitstream, wherein the compression profile is a music standard compression profile.

The present invention further discloses a non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause one or more processors to perform operations, the operations including: receiving an encoded audio bitstream comprising an audio program, the encoded audio bitstream comprising encoded audio data of a set of one or more audio channels and metadata associated with the set of audio channels, wherein the metadata includes loudness processing status metadata, and wherein the loudness processing status metadata includes metadata indicating the loudness of the audio program; decoding the encoded audio data to obtain decoded audio data of the set of audio channels; obtaining the loudness processing status metadata from the metadata of the encoded audio bitstream; and performing adaptive loudness processing on the decoded audio data of the set of audio channels based on the loudness processing status metadata; wherein the metadata further includes program information metadata indicating a compression profile for creating dynamically range compressed (DRC) data in the bitstream, wherein the compression profile is a music standard compression profile.

In one embodiment, the present invention is an audio processing unit capable of decoding an encoded bitstream comprising substream structure metadata and/or program information metadata (optionally including other metadata, such as loudness processing status metadata) in at least one segment of at least one frame of the bitstream, and audio data in at least one other segment of the frame. Hereinafter, substream structure metadata (or “SSM”) refers to metadata of an encoded bitstream (or a collection of encoded bitstreams) indicating the substream structure of the audio content of the encoded bitstream, and “program information metadata” (or “PIM”) refers to metadata of an encoded audio bitstream indicating at least one audio program (e.g., two or more audio programs), wherein the program information metadata indicates at least one attribute or characteristic of the audio content of at least one of the programs (e.g., metadata indicating the type or parameters of processing performed on the audio data of the program, or metadata indicating which channels of the program are active channels).

In typical cases (e.g., where the encoded bitstream is AC-3 or E-AC-3), Program Information Metadata (PIM) indicates program information that cannot actually be carried in other parts of the bitstream. For example, PIM may indicate the processing applied to the PCM audio prior to encoding (e.g., AC-3 or E-AC-3 encoding), which frequency bands of the audio program have been encoded using specific audio coding techniques, and the compression profile used to create Dynamic Range Compression (DRC) data in the bitstream.

In another type of implementation, the method includes the step of multiplexing encoded audio data with SSM and/or PIM in each frame (or at least each of some frames) of the bitstream. In typical decoding, the decoder extracts the SSM and/or PIM from the bitstream (including by analyzing and demultiplexing the SSM and/or PIM along with the audio data) and processes the audio data to generate a stream of decoded audio data (and in some cases, performs adaptive processing of the audio data). In some implementations, the decoded audio data, along with the SSM and/or PIM, is forwarded from the decoder to a post-processor configured to perform adaptive processing on the decoded audio data using the SSM and/or PIM.

In one embodiment, the encoding method of the present invention generates an encoded audio bitstream (e.g., an AC-3 or E-AC-3 bitstream) comprising audio data segments (e.g., segments AB0 to AB5 of the frame shown in FIG. 4 or all or some of segments AB0 to AB5 of the frame shown in FIG. 7), the audio data segments comprising encoded audio data and metadata segments (including SSM and/or PIM, optionally including other metadata) time-division multiplexed with the audio data segments. In some embodiments, each metadata segment (sometimes referred to herein as a “container”) has a metadata segment header (optionally including other mandatory or “core” elements) and one or more metadata payloads following the metadata segment header. If present, the SIM is included in one of the metadata payloads (identified by a payload header and typically having a first type of format). If present, the PIM is included in another of the metadata payloads (identified by a payload header and typically having a second type of format). Similarly, each other type of metadata (if present) is included in another of the metadata payloads (identified by a payload header and typically having a metadata-specific format). The exemplary format allows convenient access to the SSM, PIM, or other metadata at times other than during the decoding of the bitstream (e.g., by a post-processor after decoding, or by a processor configured to identify metadata without performing full decoding of the encoded bitstream), and allows for convenient and efficient error detection and correction during the decoding of the bitstream (e.g., substream identification). For example, without accessing the SSM in the exemplary format, the decoder might incorrectly identify the correct number of substreams associated with the program. One metadata payload in a metadata segment may include the SSM, another metadata payload in a metadata segment may include the PIM, and optionally, at least one other metadata payload in a metadata segment may include other metadata (e.g., loudness processing status metadata or "LPSM").

Attached Figure Description

Figure 1 is a block diagram of an embodiment of a system that can be configured to perform an embodiment of the method of the present invention.

Figure 2 is a block diagram of an encoder as an embodiment of the audio processing unit of the present invention.

Figure 3 is a block diagram of a decoder as an embodiment of the audio processing unit of the present invention and a post-processor coupled to the decoder as another embodiment of the audio processing unit of the present invention.

Figure 4 is a diagram of an AC-3 frame that includes the segments it is divided into.

Figure 5 is a diagram of the synchronization information (SI) segment of an AC-3 frame, which is divided into segments.

Figure 6 is a diagram of the Bit Stream Information (BSI) segment of an AC-3 frame, which is divided into segments.

Figure 7 is a diagram of an E-AC-3 frame that includes the segments it is divided into.

Figure 8 is a diagram of a metadata segment of an encoded bitstream including a metadata segment header generated according to an embodiment of the present invention. The metadata segment header includes a container synchronization word (identified as "container synchronization" in Figure 8) and a version and key ID value, followed by multiple metadata payloads and protection bits.

Symbols and terms

Throughout this disclosure, including the claims, the expression “to” a signal or data (e.g., to filter, scale, transform, or apply gain to the signal or data) is used broadly to mean directly performing an operation on the signal or data, or on a processed version of the signal or data (e.g., on a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on the signal).

Throughout this disclosure, including the claims, the term "system" is used broadly to refer to a device, system, or subsystem. For example, a subsystem implementing a decoder can be called a decoder system, and a system including such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, wherein the subsystem generates M inputs and the other X-M inputs are received from an external source) can also be called a decoder system.

Throughout this disclosure, including the claims, the term "processor" is used broadly to refer to a system or apparatus that is programmable or otherwise configurable (e.g., using software or firmware) to perform operations on data (e.g., audio data, video data, or other image data). Examples of processors include field-programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and/or otherwise configured to perform pipelined processing of audio data or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

Throughout this disclosure, including the claims, the expressions "audio processor" and "audio processing unit" are used interchangeably and broadly to refer to a system configured to process audio data. Examples of audio processing units include, but are not limited to, encoders (e.g., code converters), decoders, codecs, preprocessing systems, post-processing systems, and bitstream processing systems (sometimes referred to as bitstream processing tools).

Throughout this disclosure, including the claims, the expression “metadata” (of encoded audio bitstreams) refers to data that is separate from and distinct from the corresponding audio data of the bitstream.

Throughout this disclosure, including the claims, the expression “substream structure metadata” (or “SSM”) represents metadata of an encoded audio bitstream (or set of encoded audio bitstreams) that indicates the substream structure of the audio content of the encoded bitstream.

Throughout this disclosure, including the claims, the expression “Program Information Metadata” (or “PIM”) represents metadata of an encoded audio bitstream that indicates at least one audio program (e.g., two or more audio programs), wherein the metadata indicates at least one attribute or characteristic of the audio content of at least one of the programs (e.g., metadata indicating the type or parameters of processing performed on the audio data of the program, or metadata indicating which channels of the program are active channels).

Throughout this disclosure, including the claims, the expression "processing status metadata" (e.g., as in the expression "loudness processing status metadata") refers to metadata associated with audio data of a bitstream (encoded audio bitstream), indicating the processing status of the corresponding (associated) audio data (e.g., what type of processing has been performed on the audio data), and generally also indicating at least one feature or characteristic of the audio data. The association between the processing status metadata and the audio data is time-synchronized. Thus, the current (latest received or updated) processing status metadata indicates the corresponding audio data and includes the result of audio data processing of the indicated type. In some cases, the processing status metadata may include some or all of the processing history and/or parameters used in and/or obtained from the indicated type of processing. Additionally, the processing status metadata may include at least one feature or characteristic of the corresponding audio data that has been calculated or extracted from the audio data. The processing status metadata may also include other metadata unrelated to any processing of the corresponding audio data or not obtained from any processing of the corresponding audio data. For example, third-party data, tracking information, identifiers, ownership or standard information, user annotation data, user preference data, etc., can be added by specific audio processing units to be passed to other audio processing units.

Throughout this disclosure, including the claims, the expression “loudness processing status metadata” (or “LPSM”) refers to processing status metadata, which indicates the loudness processing status of the corresponding audio data (e.g., what type of loudness processing has been performed on the audio data) and typically also indicates at least one feature or characteristic of the corresponding audio data (e.g., loudness). Loudness processing status metadata may include data that is not (i.e., when considered alone) loudness processing status metadata (e.g., other metadata).

Throughout this disclosure, including the claims, the term "channel" (or "audio channel") refers to a single-channel audio signal.

Throughout this disclosure, including the claims, the expression “audio program” means a collection of one or more audio channels and optionally also means associated metadata (e.g., metadata describing the desired spatial audio representation, and/or PIM, and/or SSM, and/or LPSM, and/or program boundary metadata).

Throughout this disclosure, including the claims, the expression "program boundary metadata" represents metadata of an encoded audio bitstream, wherein the encoded audio bitstream indicates at least one audio program (e.g., two or more programs), and the program boundary metadata indicates the position of at least one boundary (start and/or end) of at least one of the audio programs in the bitstream. For example, the program boundary metadata (of the encoded audio bitstream indicating the audio programs) may include metadata indicating the position of the start of the program (e.g., the start of the "N"th" frame of the bitstream, or the "M"th sample position of the "N"th frame of the bitstream), and additional metadata indicating the position of the end of the program (e.g., the start of the "J"th frame of the bitstream, or the "K"th sample position of the "J"th frame of the bitstream).

Throughout this disclosure, including the claims, the terms "coupled" or "coupled" are used to indicate a direct or indirect connection. Thus, if a first device is coupled to a second device, the connection can be a direct connection or an indirect connection via other devices and connections.

Detailed Implementation

A typical audio data stream consists of both audio content (e.g., one or more channels of audio content) and metadata indicating at least one characteristic of the audio content. For example, in an AC-3 bitstream, there are several audio metadata parameters specifically intended to alter the sound of the program being transmitted to the listening environment. One of these metadata parameters is the DIALNORM parameter, which is intended to indicate the average level of dialogue in the audio program and is used to determine the audio playback signal level.

During playback of a bitstream comprising a series of different audio program segments (each with different DIALNORM parameters), the AC-3 decoder performs a type of loudness processing using the DIALNORM parameters of each segment. In this loudness processing, the AC-3 decoder modifies the playback level, or loudness, so that the perceived loudness of the dialogue across the series of segments is at a consistent level. Each coded audio segment (project) in a series of coded audio projects will (typically) have different DIALNORM parameters, and the decoder will scale the level of each project within the project so that the playback level or loudness of the dialogue in each project is the same or very similar, although this requires applying different amounts of gain to the different projects within the project during playback.

DIALNORM is typically set by the user rather than automatically generated; however, a default DIALNORM value exists if the user does not set one. For example, content creators can use an external device to measure loudness using an AC-3 encoder and then transmit that result (indicating the loudness of spoken dialogue in an audio program) to the encoder to set the DIALNORM value. Thus, it depends on the content creator setting the DIALNORM parameter correctly.

There are several different reasons why the DIALNORM parameter in an AC-3 bitstream might be incorrect. First, if the DIALNORM value is not set by the content creator, then each AC-3 encoder has a default DIALNORM value used during bitstream generation. This default value may differ significantly from the actual loudness of the audio dialogue. Second, even if the content creator measures loudness and sets the DIALNORM value accordingly, an incorrect DIALNORM value may have been produced using a loudness measurement algorithm or meter that does not conform to recommended AC-3 loudness measurement methods. Third, even if the AC-3 bitstream was created using a DIALNORM value correctly measured and set by the content creator, the AC-3 bitstream may have been altered to an incorrect value during bitstream transmission and/or storage. This is not uncommon, for example, in television broadcasting applications that decode, modify, and then re-encode the AC-3 bitstream using incorrect DIALNORM metadata information. Therefore, the DIALNORM value included in the AC-3 bitstream may be incorrect or inaccurate, and thus may negatively impact the quality of the listening experience.

Furthermore, the DIALNORM parameter does not indicate the loudness processing status of the corresponding audio data (e.g., what type of loudness processing has been performed on the audio data). Loudness processing status metadata (in the format provided in some embodiments of the invention) helps to facilitate, in a particularly efficient manner, adaptive loudness processing of audio bitstreams and/or verification of the loudness processing status and loudness validity of audio content.

Although the present invention is not limited to the use of AC-3 bitstreams, E-AC-3 bitstreams or Dolby E bitstreams, for convenience, it will be described in embodiments of generating, decoding or otherwise processing such bitstreams.

The AC-3 encoded bitstream comprises 1 to 6 channels of metadata and audio content. The audio content is audio data that has been compressed using perceptual audio coding. The metadata includes several audio metadata parameters intended to modify the sound of the program transmitted to the listening environment.

Each frame of an AC-3 encoded audio bitstream contains audio content and metadata about 1536 samples of digital audio. At a sampling rate of 48kHz, this translates to 32 milliseconds of digital audio, or 31.25 frames per second of audio.

Depending on whether a frame contains 1, 2, 3, or 6 blocks of audio data, each frame of an E-AC-3 encoded audio bitstream contains audio data and metadata of 256, 512, 768, or 1536 samples of digital audio. For a sampling rate of 48 kHz, this represents 5.333, 10.667, 16, or 32 milliseconds of digital audio, or 189.9, 93.75, 62.5, or 31.25 frames per second, respectively.

As shown in Figure 4, each AC-3 frame is divided into sections, including: a synchronization information (SI) section containing a synchronization word (SW) and the first error correction word (CRC1) of the two error correction words (as shown in Figure 5); a bitstream information (BSI) section containing most of the metadata; six audio blocks (AB0 to AB5) containing compressed audio content (and may also include metadata); a useless bit field (W) containing any unused bits remaining after the compressed audio content (also known as a "skip field"); an auxiliary (AUX) information section that may contain more metadata; and the second error correction word (CRC2) of the two error correction words.

As shown in Figure 7, each E-AC-3 frame is divided into sections, including: a Synchronization Information (SI) section containing a Synchronization Word (SW) (as shown in Figure 5); a Bit Stream Information (BSI) section containing most of the metadata; six audio blocks (AB0 to AB5) containing compressed audio content (and may also include metadata); a Useless Bit Field (W) containing any unused bits remaining after the compressed audio content (also known as a "skip field") (although only one Useless Bit Field is shown, different Useless Bit Fields or Skip Fields are usually present after each audio block); an Auxiliary (AUX) Information section that may contain more metadata; and an Error Correction Word (CRC).

In an AC-3 (or E-AC-3) bitstream, there are several audio metadata parameters specifically designed to alter the sound of the program transmitted to the listening environment. One of these metadata parameters is the DIALNORM parameter, which is included in the BSI segment.

As shown in Figure 6, the BSI segment of an AC-3 frame includes a 5-bit parameter (“DIALNORM”) indicating the DIALNORM value of the program. If the audio coding mode (“acmod”) of the AC-3 frame is 0, it includes a 5-bit parameter (“DIALNORM2”) indicating the DIALNORM value of a second audio program carried in the same AC-3 frame, indicating the use of a dual-single channel or a “1+1” channel configuration.

The BSI segment also includes a flag (“addbsie”) indicating the presence (or absence) of additional bitstream information after the “addbsie” bit, a parameter (“addbsil”) indicating the length of any additional bitstream information after the “addbsil” value, and additional bitstream information (“addbsi”) up to 64 bits after the “addbsil” value.

The BSI segment includes other metadata values not specifically shown in Figure 6.

According to one implementation, the encoded bitstream indicates multiple substreams of audio content. In some cases, the substreams indicate the audio content of a multi-channel program, and each substream indicates one or more channels of the program. In other cases, the multiple substreams of the encoded audio bitstream indicate the audio content of several audio programs—typically a “main” audio program (which may be a multi-channel program) and at least one other audio program (e.g., a program providing commentary on the main audio program).

An encoded audio bitstream indicating at least one audio program needs to include at least one “independent” substream containing audio content. An independent substream indicates at least one channel of the audio program (e.g., an independent substream could indicate the five full-range channels of a typical 5.1-channel audio program). In this document, this audio program is referred to as the “main” program.

In some implementations, the encoded audio bitstream indicates two or more audio programs (a "main" program and at least one other audio program). In such cases, the bitstream comprises two or more independent substreams: a first independent substream indicating at least one channel of the main program; and at least one other independent substream indicating at least one channel of another audio program (a program different from the main program). Each independent substream can be decoded independently, and the decoder can operate to decode only a subset (not all) of the independent substreams of the encoded bitstream.

In a typical example of an encoded audio bitstream indicating two independent substreams, one independent substream indicates a standard format speaker channel for a multi-channel master program (e.g., left, right, center, left surround, and right surround full-range speaker channels for a 5.1-channel master program), while the other independent substream indicates a single-channel audio commentary on the master program (e.g., a director's commentary on a film, where the master program is the film's soundtrack). In another example of an encoded audio bitstream indicating multiple independent substreams, one independent substream indicates a standard format speaker channel for a multi-channel master program (e.g., a 5.1-channel master program) that includes dialogue in the first language (e.g., one of the speaker channels of the master program could indicate the dialogue), while each of the other independent substreams indicates a single-channel translation of the dialogue (translated into a different language).

Optionally, the encoded audio bitstream indicating the main program (and optionally at least one other audio program) includes at least one “subordinate” substream of audio content. Each subordinate substream is associated with an independent substream of the bitstream and indicates at least one additional channel of the program (e.g., the main program) whose content is indicated by the associated independent substream (i.e., the subordinate substream indicates at least one channel of the program that is not indicated by the associated independent substream, while the associated independent substream indicates at least one channel of the program).

In an example of an coded bitstream that includes an independent substream (indicating at least one channel of the main program), the bitstream also includes subordinate substreams (associated with the independent substream) that indicate one or more additional speaker channels of the main program. Such additional speaker channels are extra to the main program channel indicated by the independent substream. For example, if the independent substream indicates the left, right, center, left surround, and right surround full-range speaker channels of a 7.1 channel main program, then the subordinate substreams could indicate two other full-range speaker channels of the main program.

According to the E-AC-3 standard, an E-AC-3 bitstream must indicate at least one independent substream (e.g., a single AC-3 bitstream) and can indicate up to eight independent substreams. Each independent substream of an E-AC-3 bitstream can be associated with up to eight subordinate substreams.

E-AC-3 bitstreams include metadata indicating the substream structure of the bitstream. For example, the “chanmap” field in the Bitstream Information (BSI) section of an E-AC-3 bitstream determines the channel mapping of program channels indicated by subordinate substreams of the bitstream. However, the metadata indicating the substream structure is conventionally included in E-AC-3 bitstreams in a format that facilitates access and use only by the E-AC-3 decoder (during decoding of the encoded E-AC-3 bitstream); it is not convenient to access and use after decoding (e.g., by a post-processor) or before decoding (e.g., by a processor configured to identify metadata). Moreover, there is a risk that the decoder may incorrectly identify substreams of a conventional E-AC-3 encoded bitstream using the conventionally included metadata, and prior to this invention, it was unknown how to include substream structure metadata in an encoded bitstream (e.g., an encoded E-AC-3 bitstream) in such a format that errors in substream identification can be conveniently and efficiently detected and corrected during bitstream decoding.

E-AC-3 bitstreams may also include metadata about the audio content of an audio program. For example, an E-AC-3 bitstream indicating an audio program may include metadata indicating the minimum and maximum frequencies at which spectral spreading processing (and channel-coupled coding) has been used to encode the program's content. However, such metadata is typically included in the E-AC-3 bitstream in a format that facilitates access and use solely by the E-AC-3 decoder (during decoding of the encoded E-AC-3 bitstream); it is not convenient to access and use after decoding (e.g., by a post-processor) or before decoding (e.g., by a processor configured to recognize metadata). Furthermore, such metadata is not included in the E-AC-3 bitstream in a format that allows for convenient and efficient error detection and correction during the decoding of the bitstream.

According to a typical embodiment of the invention, PIM and/or SSM (and optionally other metadata, such as loudness processing status metadata or "LPSM") are embedded in one or more reserved fields (or slots) of a metadata segment of an audio bitstream that also includes audio data in other segments (audio data segments). Typically, at least one segment of each frame of the bitstream includes a PIM or SSM, and at least one other segment of the frame includes corresponding audio data (i.e., audio data whose data structure is indicated by the SSM and/or at least one of its characteristics or attributes is indicated by the PIM).

In one implementation, each metadata segment is a data structure (sometimes referred to herein as a container) that may contain one or more metadata payloads. Each payload includes a header to provide an explicit indication of the type of metadata present in the payload, wherein the header includes a specific payload identifier (or payload configuration data). The order of payloads within the container is undefined, allowing payloads to be stored in any order, and the analyzer must be able to analyze the entire container to extract relevant payloads while ignoring irrelevant or unsupported payloads. Figure 8 (described below) illustrates the structure of such a container and the payloads within it.

Communication metadata (e.g., SSM and/or PIM and/or LPSM) in the audio data processing chain is particularly useful when two or more audio processing units need to work collaboratively across the processing chain (or content lifecycle). In cases where metadata is not included in the audio bitstream, such as when two or more audio codecs are used in the chain and single-ended volume is applied more than once during the bitstream path of the media-consuming device (or the rendering point of the audio content of the bitstream), several media processing problems, such as quality, level, and spatial degradation, can occur.

According to some embodiments of the invention, loudness processing status metadata (LPSM) embedded in the audio bitstream can be authenticated and verified, for example, to enable a loudness adjustment entity to demonstrate whether the loudness of a particular program is within a specified range and whether the corresponding audio data itself has not been modified (thereby ensuring compliance with applicable regulation). Loudness values included in data blocks containing loudness processing status metadata can be read out for verification without recalculating the loudness. In response to the LPSM, a regulatory structure can determine that the corresponding audio content complies with (as indicated by the LPSM) the legal and/or regulatory requirements for loudness (e.g., rules published under the Commercial Advertising Loudness Mitigation Act, also known as the "CALM" Act) without needing to calculate the loudness of the audio content.

Figure 1 is a block diagram of an exemplary audio processing chain (audio data processing system), in which one or more elements of the system can be configured according to embodiments of the present invention. The system includes the following elements coupled together as shown: a preprocessing unit, an encoder, a signal analysis and metadata correction unit, a code converter, a decoder, and a post-processing unit. In variations of the system shown, one or more elements are omitted, or additional audio data processing units are included.

In some implementations, the preprocessing unit of Figure 1 is configured to receive PCM (temporal domain) samples including audio content as input and output processed PCM samples. The encoder can be configured to receive PCM samples as input and output an encoded (e.g., compressed) audio bitstream indicating the audio content. The data indicating the audio content bitstream is sometimes referred to herein as "audio data." If the encoder is configured according to a typical embodiment of the invention, the audio bitstream output from the encoder includes PIM and/or SSM (optionally also including loudness processing status metadata and/or other metadata) and audio data.

The signal analysis and metadata correction unit in Figure 1 can receive one or more encoded audio bitstreams as input and determine (e.g., verify) whether the metadata (e.g., processing status metadata) in each encoded audio bitstream is correct by performing signal analysis (e.g., using program boundary metadata in the encoded audio bitstreams). If the signal analysis and metadata correction unit finds that the included metadata is invalid, it typically replaces the erroneous value with the correct value obtained from the signal analysis. Thus, each encoded audio bitstream output from the signal analysis and metadata correction unit can include corrected (or uncorrected) processing status metadata as well as encoded audio data.

The code converter of Figure 1 can receive an encoded audio bitstream as input and, in response (e.g., by decoding the input stream and re-encoding the decoded stream in a different encoding format), output a modified (e.g., differently encoded) audio bitstream. If the code converter is configured according to a typical embodiment of the invention, the audio bitstream output from the code converter includes SSM and/or PIM (and typically other metadata) as well as encoded audio data. The metadata may have already been included in the input bitstream.

The decoder of Figure 1 can receive an encoded (e.g., compressed) audio bitstream as input and output (as a response) a decoded PCM audio sample stream. If the decoder is configured according to a typical embodiment of the invention, then in typical operation, the decoder's output is or includes any of the following:

An audio sample stream, and at least one corresponding stream of SSM and/or PIM (and often other metadata) extracted from the input encoded bitstream; or

The audio sample stream, and the corresponding stream of control bits determined based on the SSM and/or PIM extracted from the input encoded bit stream (usually along with other metadata, such as LPSM); or

An audio sample stream, but without metadata or a corresponding stream of control bits determined from the metadata. In the latter case, the decoder can extract metadata from the input encoded bit stream and perform at least one operation (e.g., verification) on the extracted metadata, even if there is no output of the extracted metadata or control bits determined from the metadata.

By configuring the post-processing unit of FIG1 according to a typical embodiment of the invention, the post-processing unit is configured to receive a decoded PCM audio sample stream and perform post-processing (e.g., volume leveling of the audio content) on it using the SSM and/or PIM (and typically other metadata, such as LPSM) received with the samples, or based on control bits determined according to the metadata received with the samples. The post-processing unit is also typically configured to render the post-processed audio content for playback by one or more speakers.

Typical embodiments of the present invention provide an enhanced audio processing chain in which audio processing units (e.g., encoders, decoders, code converters, and preprocessing and postprocessing units) modify their respective processing to be applied to the audio data based on the concurrent state of the media data indicated by metadata received by the audio processing units respectively.

Audio data input to any audio processing unit of the system of Figure 1 (e.g., the encoder or code converter of Figure 1) may include SSM and/or PIM (optionally including other metadata) and audio data (e.g., encoded audio data). This metadata may be included in the input audio via another element (or another source, not shown in Figure 1) of the system of Figure 1 according to embodiments of the invention. The processing unit receiving the input audio (with metadata) may be configured to perform at least one operation on the metadata (e.g., verification), or in response to the metadata (e.g., adaptive processing of the input audio), and typically also includes the metadata, a processed version of the metadata, or control bits determined based on the metadata in its output audio.

A typical implementation of the audio processing unit (or audio processor) of the present invention is configured to perform adaptive processing of audio data based on the state of the audio data indicated by metadata corresponding to the audio data. In some implementations, the adaptive processing is (or includes) loudness processing (if the metadata indicates that loudness processing or a similar process has not yet been performed on the audio data), but not (and does not include) loudness processing (if the metadata indicates that such loudness processing or a similar process has already been performed on the audio data). In some implementations, the adaptive processing is or includes (e.g., performed in a metadata verification subunit) metadata verification to ensure that the audio processing unit performs additional adaptive processing of the audio data based on the state of the audio data indicated by the metadata. In some implementations, the verification determines the reliability of the metadata associated with the audio data (e.g., included in a bitstream containing the audio data). For example, if the verified metadata is reliable, the result from a previously performed audio processing can be reused and new performance of the same type of audio processing can be avoided. On the other hand, if the metadata is found to have been tampered with (or otherwise unreliable), then a type of media processing that was allegedly previously performed (such as that indicated by the unreliable metadata) can be repeated by the audio processing unit, and/or other processing can be performed on the metadata and/or audio data by the audio processing unit. If the unit determines that the metadata is valid (e.g., based on a match between the extracted encrypted value and a reference encrypted value), the audio processing unit can also be configured to signal other audio processing units downstream in the enhanced media processing chain that the metadata (e.g., present in the media bitstream) is valid.

Figure 2 is a block diagram of an encoder (100) as an embodiment of the audio processing unit of the present invention. Any component or element of the encoder 100 may be implemented in hardware or software or a combination of hardware and software as one or more processes and/or one or more circuits (e.g., ASIC, FPGA or other integrated circuits). The encoder 100 includes a frame buffer 110, an analyzer 111, a decoder 101, an audio state verifier 102, a loudness processing stage 103, an audio stream selection stage 104, an encoder 105, a filler/formatter stage 107, a metadata generation stage 106, a dialogue loudness measurement subsystem 108, and a frame buffer 109, connected as shown. The encoder 100 typically also includes other processing elements (not shown).

Encoder 100 (a code converter) is configured to convert an input audio bitstream (e.g., one of an AC-3 bitstream, an E-AC-3 bitstream, or a Dolby E bitstream) into an encoded output audio bitstream (e.g., another of an AC-3 bitstream, an E-AC-3 bitstream, or a Dolby E bitstream) by performing adaptive and automatic loudness processing using loudness processing state metadata included in the input bitstream. For example, encoder 100 can be configured to convert an input Dolby E bitstream (in a format typically used in production and broadcast equipment, but not in consumer devices receiving already broadcast audio programs) into an encoded output audio bitstream in AC-3 or E-AC-3 format (suitable for broadcast to consumer devices).

The system of Figure 2 also includes an encoded audio transmission subsystem 150 (which stores and/or transmits the encoded bitstream output from encoder 100) and a decoder 152. The encoded audio bitstream output from encoder 100 may be stored by subsystem 150 (e.g., in DVD or Blu-ray disc format), transmitted by subsystem 150 (which may implement a transmission line or network), or both stored and transmitted by subsystem 150. Decoder 152 is configured to decode the encoded audio bitstream (generated by encoder 100) received via subsystem 150 by extracting metadata (PIM and/or SSM, and optionally loudness processing status metadata and/or other metadata) (and optionally program boundary metadata from the bitstream) from each frame of the bitstream and generating decoded audio data. Typically, decoder 152 is configured to perform adaptive processing on the decoded audio data using PIM and/or SSM and/or LPSM (optionally also using program boundary metadata), and/or forward the decoded audio data and metadata to a postprocessor configured to perform adaptive processing on the decoded audio data using metadata. Typically, decoder 152 includes a buffer that stores (e.g., in a non-transient manner) the encoded audio bitstream received from subsystem 150.

Various implementations of encoder 100 and decoder 152 are configured to perform different embodiments of the method of the present invention.

Frame buffer 110 is a buffer memory coupled to receive an encoded input audio bitstream. In operation, buffer 110 stores (e.g., in a non-transient manner) at least one frame of the encoded audio bitstream, and a sequence of frames of the encoded audio bitstream is set from buffer 110 to analyzer 111.

The analyzer 111 is coupled and configured to extract PIM and/or SSM, loudness processing state metadata (LPSM), and optionally program boundary metadata (and/or other metadata) from each frame of the encoded input audio, including such metadata. The LPSM (and optionally program boundary metadata and/or other metadata) is set to at least the audio state verifier 102, loudness processing level 103, level 106, and subsystem 108 to extract audio data from the encoded input audio and set the audio data to the decoder 101. The decoder 101 of the encoder 100 is configured to decode the audio data to generate decoded audio data and set the decoded audio data to the loudness processing level 103, audio stream selection level 104, subsystem 108, and typically also to the state verifier 102.

The state verifier 102 is configured to authenticate and verify the LPSM (optionally other metadata) set thereto. In some embodiments, the LPSM is (or is included in) a data block that has already been included in the input bitstream (e.g., according to embodiments of the invention). The block may include a cryptographic hash (a hash-based message authentication code or "HMAC") for processing the LPSM (optionally other metadata) and/or the basic audio data (provided from decoder 101 to verifier 102). In these embodiments, the data block may be digitally labeled so that downstream audio processing units can relatively easily authenticate and verify the processing state metadata.

For example, HMAC is used to generate a digest, and the protection value included in the bitstream of this invention may include this digest. This digest can be generated with respect to an AC-3 frame as follows:

1. After the AC-3 data and LPSM are encoded, the frame data bytes (connected frame data #1 and frame data #2) and LPSM data bytes are used as input to the hash function HMAC. Other data that may exist within the auxiliary data field for digest calculation is not considered. Such other data can be bytes that are neither part of the AC-3 data nor the LPSM data. The guard bits included in the LPSM for HMAC digest calculation can be disregarded.

2. After the digest is calculated, it is written into the field reserved for the protection bits in the bit stream.

3. The final step in generating a complete AC-3 frame is the calculation of the CRC checksum. This is written at the end of the frame and takes into account all data belonging to that frame, including the LPSM bit.

Other encryption methods, including but not limited to any one of one or more non-HMAC encryption methods, can be used for the verification of LPSM and/or other metadata (e.g., in verifier 102) to ensure the secure transmission and reception of metadata and/or basic audio data. For example, verification (using such encryption methods) can be performed in each audio processing unit receiving the audio bitstream of the present invention to determine whether the metadata and corresponding audio data included in the bitstream have undergone (and/or been generated) specific processing (indicated by the metadata) and whether they have not been modified after such specific processing was performed.

The state verifier 102 sets control data to the audio stream selection level 104, the metadata generator 106, and the dialogue loudness measurement subsystem 108 to represent the result of the verification operation. In response to the control data, level 104 can be selected (and passed to encoder 105):

The adaptively processed output of loudness processing level 103 (e.g., when the LPSM indicator shows that the audio data output from decoder 101 has not undergone a specific type of loudness processing, and a control bit from verifier 102 indicates that LPSM is active); or

Audio data output from decoder 102 (e.g., when LPSM indicates that the audio data output from decoder 101 has undergone a specific type of loudness processing to be performed by stage 103, and a control bit from verifier 102 indicates that LPSM is valid).

Stage 103 of encoder 100 is configured to perform adaptive loudness processing on the decoded audio data output from decoder 101 based on one or more audio data characteristics indicated by LPSM extracted by decoder 101. Stage 103 may be an adaptive transform domain real-time loudness and dynamic range control processor. Stage 103 may receive user input (e.g., user-targeted loudness/dynamic range values or dialogue normalization values), or other metadata input (e.g., one or more types of third-party data, tracking information, identifiers, ownership or standard information, user annotation data, user preference data, etc.) and/or other input (e.g., from fingerprint recognition processing), and use such input to process the decoded audio data output from decoder 101. Level 103 can perform adaptive loudness processing on decoded audio data (output from decoder 101) indicating a single audio program (represented by program boundary metadata extracted by analyzer 111), and can reset the loudness processing in response to receiving decoded audio data (output from decoder 101) indicating a different audio program (represented by program boundary metadata extracted by analyzer 111).

When the control bit from verifier 102 indicates that the LPSM is invalid, the dialogue loudness measurement subsystem 108 can operate to determine the loudness of a segment of decoded audio (from decoder 101) representing dialogue (or other speech) using the LPSM (and/or other metadata) extracted by decoder 101. When the control bit from verifier 102 indicates that the LPSM is valid, the operation of the dialogue loudness measurement subsystem 108 can be disabled when the LPSM indicates a previously determined loudness of a segment of dialogue (or other speech) in the decoded audio (from decoder 101). Subsystem 108 can perform loudness measurements on decoded audio data representing a single audio program (indicated by program boundary metadata extracted by analyzer 111) and can reset loudness processing in response to receiving decoded audio data representing a different audio program indicated by such program boundary metadata.

Useful tools (e.g., a Dolby LM100 loudness meter) exist for convenient and easy measurement of the level of dialogue in audio content. Some embodiments of the APU of the present invention (e.g., stage 108 of encoder 100) are implemented to include such a tool (or perform the function of such a tool) to measure the average dialogue loudness of the audio content of an audio bitstream (e.g., the decoded AC-3 bitstream from decoder 101 of encoder 100 to decoded stage 108).

If level 108 is implemented to measure the true average dialogue loudness of audio data, the measurement may include the step of separating audio segments that primarily contain speech. The audio segments that are primarily speech are then processed according to a loudness measurement algorithm. For audio data decoded from an AC-3 bitstream, this algorithm may be a standard K-weighted loudness measurement (according to the international standard ITU-R BS 1770). Alternatively, other loudness measurements may be used (e.g., those based on a psychoacoustic model of loudness).

Speech segment separation is not necessary for measuring the average dialogue loudness of audio data. However, it improves the accuracy of the measurement and generally provides more satisfactory results from listener perception. Because not all audio content contains dialogue (speech), a loudness measurement of the entire audio content can provide a sufficient approximation of the dialogue level of the audio where speech already exists.

Metadata generator 106 generates (and/or passes to level 107) the encoded bitstream to be included by level 107 in the output from encoder 100. Metadata generator 106 may pass LPSM (optionally also LIM and/or PIM and/or program boundary metadata and/or other metadata) extracted by encoder 101 and/or analyzer 111 to level 107 (e.g., when a control bit from verifier 102 indicates that LPSM and/or other metadata is valid), or generate new LIM and/or PIM and/or LPSM and/or program boundary metadata and/or other metadata and set the new metadata to level 107 (e.g., when a control bit from verifier 102 indicates that metadata extracted by decoder 101 is invalid), or set a combination of metadata extracted by decoder 101 and/or analyzer 111 and newly generated metadata to level 107. Metadata generator 106 can include loudness data generated by subsystem 108 and at least one value indicating the type of loudness processing performed by subsystem 108 in the LPSM, setting the LPSM to level 107 for inclusion in the encoded bitstream to be output from encoder 100.

Metadata generator 106 can generate control bits (which may consist of or include a hash-based message authentication code or "HMAC") for at least one of the decryption, authentication, or verification of LPSM (optionally other metadata) in the underlying audio data to be included in the encoded bitstream and/or the underlying audio data to be included in the encoded bitstream. Metadata generator 106 can provide such protection bits to stage 107 for inclusion in the encoded bitstream.

In typical operation, the dialogue loudness measurement subsystem 108 processes the audio data output from the decoder 101 to generate loudness values (e.g., gated and ungated dialogue loudness values) and dynamic range values in response to the audio data. In response to these values, the metadata generator 106 can generate loudness processing state metadata (LPSM) for inclusion (by the filler/formatter 107) in the encoded bitstream to be output from the encoder 100.

Additionally, or alternatively, subsystems 106 and/or 108 of encoder 100 may perform additional analysis of the audio data to generate metadata indicating at least one characteristic of the audio data for inclusion in the encoded bitstream output from slave level 107.

Encoder 105 encodes the audio data output from selection stage 104 (e.g., by compressing it) and sets the encoded audio to stage 107 for inclusion in the encoded bitstream to be output from stage 107.

Stage 107 multiplexes the encoded audio from encoder 105 and metadata (including PIM and/or SSM) from generator 106 to generate an encoded bitstream to be output from stage 107, preferably such that the encoded bitstream has a format specified by a preferred embodiment of the present invention.

Frame buffer 109 is a buffer memory for storing (e.g., in a non-transient manner) at least one frame of the encoded audio bitstream output from stage 107, and then a series of frames of the encoded audio bitstream are set from buffer 109 as output from encoder 100 to transmission system 150.

The LPSM generated by metadata generator 106 and included in the encoded bitstream by level 107 typically indicates the loudness processing status of the corresponding audio data (e.g., what type of loudness processing has been performed on the audio data) and the loudness of the corresponding audio data (e.g., measured dialogue loudness, gated and/or ungated loudness, and/or dynamic range).

In this paper, "gating" of loudness and/or level measurements performed on audio data refers to a specific level or loudness threshold in which calculated values exceeding a threshold are included in the final measurement (e.g., ignoring short-term loudness values below -60 dBFS in the final measurement). Gating of absolute values refers to a fixed level or loudness, while gating of relative values refers to values that depend on the currently "ungated" measurement.

In some implementations of encoder 100, the encoded bitstream cached in memory 109 (and output to transmission system 150) is an AC-3 bitstream or an E-AC-3 bitstream, and includes audio data segments (e.g., segments AB0 to AB5 of the frame shown in FIG. 4) and metadata segments, wherein the audio data segments indicate audio data, and each of at least some of the metadata segments includes PIM and/or SSM (and optionally other metadata). Stage 107 inserts the metadata segments (including metadata) into the bitstream in the following format. Each metadata segment including PIM and/or SSM is included in a useless bit segment of the bitstream (e.g., the useless bit segment “W” shown in FIG. 4 or FIG. 7), or in the “addbsi” field of the bitstream frame’s bitstream information (“BSI”) segment, or in an auxiliary data field at the end of the bitstream frame (e.g., the AUX segment shown in FIG. 4 or FIG. 7). A frame of a bitstream may include one or two metadata segments, each containing metadata, and if the frame includes two metadata segments, one may be in the addbsi field of the frame and the other in the AUX field of the frame.

In some implementations, each metadata segment inserted by level 107 (sometimes referred to herein as a “container”) has a format including a metadata segment header (optionally including other mandatory or “core” elements) and one or more metadata payloads following the metadata segment header. If present, the SIM is included in one of the metadata payloads (identified by a payload header and typically having a first-type format). If present, the PIM is included in another of the metadata payloads (identified by a payload header and typically having a second-type format). Similarly, each other type of metadata (if present) is included in another of the metadata payloads (identified by a payload header and typically having a format specific to the type of metadata). The exemplary format enables convenient access to SSM, PIM, and other metadata at times other than during decoding (e.g., by a post-processor after decoding, or by a processor configured to identify metadata without performing full decoding on the encoded bitstream) and allows for convenient and efficient error detection and correction during the decoding of the bitstream (e.g., substream identification). For example, if the SSM is not accessed in the exemplary format, the decoder may incorrectly identify the correct number of substreams associated with the program. One metadata payload in a metadata segment may include the SSM, another metadata payload in a metadata segment may include the PIM, and optionally, at least one other metadata payload in a metadata segment may include other metadata (e.g., loudness processing status metadata or "LPSM").

In some implementations, (by level 107) the substream structure metadata (SSM) payload included in frames of encoded bitstreams (e.g., E-AC-3 bitstreams indicating at least one audio program) comprises an SSM in the following format:

The payload header typically includes at least one identification value (e.g., a 2-bit value indicating the SSM format version, and optionally, length, period, count, and substream associated values); and following the header:

Independent substream metadata indicating the number of independent substreams of a program indicated by a bitstream; and

Sub-stream metadata indicates whether each independent sub-stream of the program has at least one associated sub-stream (i.e., whether at least one sub-stream is associated with each independent sub-stream), and if so, the number of sub-streams associated with each independent sub-stream of the program.

The expectation is that an independent substream of the encoded bitstream can indicate the set of speaker channels of an audio program (e.g., speaker channels of a 5.1 speaker channel audio program), and each of one or more subordinate substreams (as associated with the independent substream and indicated by the subordinate substream metadata) can indicate a target channel of the program. However, the independent bitstream of the encoded bitstream typically indicates the set of speaker channels of the program, and each subordinate substream associated with the independent substream (as indicated by the subordinate substream metadata) indicates at least one additional speaker channel of the program.

In some implementations, the Program Information Metadata (PIM) payload included in the frame of the encoded bitstream (e.g., an E-AC-3 bitstream indicating at least one audio program) (by level 107) has the following format:

The payload header typically includes at least one identification value (e.g., a value indicating the PIM format version, and optionally, length, period, count, and substream associated values); and the PIM in the following format following the header:

This indicates the active channel metadata for each silent and non-silent channel of an audio program (i.e., which channels of the program contain audio information and which channels, if any, contain only silence (typically related to the duration of the frame)). In implementations where the encoded bitstream is AC-3 or E-AC-3, the active channel metadata in the frames of the bitstream can be combined with additional metadata of the bitstream (e.g., the frame's audio encoding mode (“acmod”) field, and, if present, the chanmap field in the frame or associated subordinate substream frames) to determine which channels of the program contain audio information and which channels contain silence. The “acmod” field of an AC-3 or E-AC-3 frame indicates the number of full-range channels of the audio program indicated by the frame's audio content (e.g., whether the program is a 1.0-channel single-channel program, a 2.0-channel stereo program, or a program including L, R, C, Ls, Rs full-range channels), or the frame indicates two separate 1.0-channel single-channel programs. The “chanmap” field of an E-AC-3 bitstream indicates the channel mapping of the subordinate substream indicated by the bitstream. Active channel metadata can help enable downstream upmixing of the decoder (in the post-processor), for example, to add audio to the channel containing silence at the decoder's output;

This is downmixing processing state metadata indicating whether the program is downmixed (before or during encoding) and, if so, the type of downmixing applied. Downmixing processing state metadata can facilitate downstream upmixing (in the post-processor) of the decoder, for example, upmixing the program's audio content with parameters that best match the type of downmixing applied. In implementations where the encoded bitstream is AC-3 or E-AC-3, the downmixing processing state metadata can be combined with the frame's audio coding model ("acmod") field to determine the type of downmixing (if any) applied to the program's channels.

This is upmixing processing state metadata indicating whether the program is upmixed (e.g., from a smaller number of channels) before or during encoding, and if so, the type of upmixing applied. Upmixing processing state metadata can facilitate downstream downmixing (in the post-processor) of the decoder, for example, downmixing the program's audio content in a manner consistent with the type of upmixing applied to the program (e.g., Dolby Pro Logic, or Dolby Pro Logic II Cinema Mode, or Dolby Pro Logic II Music Mode, or Dolby Pro Upmixer). In implementations where the encoded bitstream is an E-AC-3 bitstream, the upmixing processing state metadata can be combined with other metadata (e.g., the value of the "strmtyp" field of a frame) to determine the type of upmixing (if any) applied to the channels of the program. The value of the “strmtyp” field (in the BSI field of a frame of an E-AC-3 bitstream) indicates whether the audio content of the frame belongs to an independent stream (which defines a program) or an independent substream (including multiple substreams or programs associated with multiple substreams), thus allowing it to be encoded independently of any other substream indicated by the E-AC-3 bitstream, or whether the audio content of the frame belongs to a subordinate substream (including multiple substreams or programs associated with multiple substreams), thus requiring it to be decoded in conjunction with its associated independent substream; and

Preprocessing status metadata, which indicates whether preprocessing was performed on the audio content of the frame (before encoding of the audio content to generate the encoded bitstream), and if so, the type of preprocessing performed.

In some implementations, preprocessed state metadata indicates:

Whether to apply surround attenuation (e.g., whether the surround channels of the audio program are attenuated by 3dB before encoding),

Whether to apply a 90° phase shift (e.g., to the surround channels Ls and Rs of the audio program before encoding),

Before encoding, should a low-pass filter be applied to the LFE channel of the audio program?

During generation, whether the LFE channel level of the program is monitored, and if so, the monitored LFE channel level relative to the program's full-range audio channels.

Whether dynamic range compression should be performed (e.g., in the decoder) on each block of the program's decoded audio content, and if so, the type (and/or parameters) of dynamic range compression to be performed (e.g., the type of preprocessing state metadata could indicate which of the following compression profile types is assumed by the encoder to generate dynamic range compression control values included in the encoded bitstream: cinematic standard, cinematic ray, music standard, music ray, or speech. Alternatively, the type of preprocessing state metadata could indicate that re-dynamic range compression (“compr” compression) should be performed on each frame of the program's decoded audio content in a manner determined by the dynamic range compression control values included in the encoded bitstream).

Whether spectral spreading and/or channel coupling coding is used to encode program content within a specific frequency range, and if spectral spreading and/or channel coupling coding is used, the minimum and maximum frequencies of the frequency components of the content for which spectral spreading coding is performed, and the minimum and maximum frequencies of the frequency components of the content for which channel coupling coding is performed. This type of preprocessing state metadata information can help perform equalization (in post-processing) downstream in the decoder. Both channel coupling information and spectral spreading information help optimize quality during code conversion operations and applications. For example, the encoder can optimize its behavior (including adaptation of preprocessing steps such as headset virtualization, upmixing, etc.) based on the state of parameters such as spectral spreading and channel coupling information. Moreover, the encoder can dynamically modify its coupling and spectral spreading parameters to match optimal values and/or modify its coupling and spectral spreading parameters to optimal values based on the state of incoming (and certified) metadata.

Whether the dialogue enhancement adjustment range data is included in the encoded bitstream, and if so, the range of adjustment that can be obtained during the execution of dialogue enhancement processing (e.g., downstream of the decoder's post-processor) that adjusts the level of dialogue content relative to the level of non-dialogue content in the audio program.

In some implementations, additional preprocessed state metadata (e.g., metadata indicating parameters related to the headset) is included in the PIM payload of the encoded bitstream to be output from encoder 100 (by stage 107).

In some implementations, (by level 107) the LPSM payload included in frames of the encoded bitstream (e.g., an E-AC-3 bitstream indicating at least one audio program) includes LPSM in the following format:

The header (typically includes a synchronization word identifying the start of the LPSM payload, followed by at least one identification value, such as the LPSM format version, length, period, count, and substream association value shown in Table 2 below); and

Following the masthead:

At least one dialogue indication value (e.g., the parameter "Dialogue Channels" in Table 2) that indicates whether the corresponding audio data indicates dialogue or not (e.g., which channels of the corresponding audio data indicate dialogue).

Indicates whether the corresponding audio content conforms to at least one loudness adjustment conformity value of the indicated set (e.g., the parameter "loudness adjustment type" in Table 2);

Indicates at least one loudness processing value of at least one type of loudness processing that has been performed on the corresponding audio data (e.g., one or more of the parameters “Dialogue Gating Loudness Correction Flag” and “Loudness Correction Type” in Table 2); and

At least one loudness value (e.g., peak or average loudness) indicating at least one loudness characteristic of the corresponding audio data (e.g., one or more of the parameters “ITU relative gated loudness”, “ITU speech gated loudness”, “ITU (EBU 3341) short 3s loudness” and “true peak” in Table 2).

In some implementations, each metadata segment containing PIM and/or SSM (and optionally other metadata) includes a metadata segment header (and optionally additional core elements), and at least one metadata payload segment following the metadata segment header (or metadata segment header and other core elements) with the following format:

The payload header typically includes at least one identification value (e.g., SSM or PIM format version, length, period, count, and substream association value), and

The SSM or PIM (or another type of metadata) following the payload header.

In some implementations, each of the metadata segments (sometimes referred to herein as "metadata containers" or "containers") in the useless bit segments/skip field segments (or "addbsi" fields or auxiliary data fields) of the frames inserted into the bitstream by level 107 has the following format:

Metadata segment header (typically includes a synchronization word identifying the start of the metadata segment, followed by an identification value, such as version, length, period, extended element count, and substream association value as shown in Table 1 below); and

At least one protected value (e.g., the HMAC digest and audio fingerprint value in Table 1) following the metadata segment header, which is helpful for the decryption, authentication, or verification of at least one of the metadata segments or corresponding audio data; and

Also following the metadata segment header is an identifier for the type of metadata in each of the following metadata payloads and an indication of at least one aspect of the configuration (e.g., size) of each such payload, including a metadata payload identifier (“ID”) value and a payload configuration value.

Each metadata payload follows the corresponding payload ID value and payload configuration value.

In some implementations, each of the metadata segments in the frame's unused bit fields (or auxiliary data fields or "addbsi" fields) has a three-level structure:

The higher-level structure (e.g., metadata segment header) includes a flag indicating whether a useless bit (or auxiliary data or addbsi) field includes metadata, at least one ID value indicating what type of metadata exists, and usually a value indicating how many bits of metadata (for each type) are present (if the metadata exists). One type of metadata that can exist is PIM, another type is SSM, and other types of metadata that can exist are LPSM, and/or program boundary metadata, and/or media search metadata;

The intermediate hierarchy includes data associated with the metadata of each identified type (e.g., metadata payload header, protection value, and payload ID and payload configuration values for the metadata of each identified type); and

The low-level structure includes metadata payloads for each identified type of metadata (e.g., a series of PIM values if PIM is identified as present, and/or metadata values for another type (e.g., SSM or LPSM) if the other type of metadata is identified as present).

Data values in these three hierarchical structures can be nested. For example, the protection value for each payload (e.g., each PIM, or SSM, or other data payload) identified by the high-level and intermediate-level structures can be included after the payload (and thus after the metadata payload header of the payload), or the protection values for all metadata payloads identified by the high-level and intermediate-level structures can be included after the final metadata payload in the metadata segment (and thus after the metadata payload header of all payloads in the metadata segment).

In one example (refer to Figure 8, which will describe the metadata segment or "container"), the metadata segment header identifies four metadata payloads. As shown in Figure 8, the metadata segment header includes the container synchronization word (identified as "container synchronization"), along with the version and key ID values. Following the metadata segment header are the four metadata payloads and protection bits. The payload ID and payload configuration (e.g., payload size) values for the first payload (e.g., PIM payload) are after the metadata segment header, and the first payload itself is after the ID and configuration values. The payload ID and payload configuration (e.g., payload size) values for the second payload (e.g., SSM payload) are after the first payload, and the second payload itself is after these ID and configuration values. The payload ID and payload configuration (e.g., payload size) values for the third payload (e.g., LPSM payload) are after the second payload, and the third payload itself is after these ID and configuration values. The payload ID and payload configuration (e.g., payload size) values for the fourth payload are after the third payload, and the fourth payload itself is after these ID and configuration values. The protection values (identified as "protection data" in Figure 8) for all or some of the payloads (or for high-level and intermediate-level structures and all or some of the payloads) are after the last payload.

In some implementations, if decoder 101 receives an audio bitstream with a cryptographic hash generated according to an embodiment of the invention, the decoder is configured to analyze and retrieve the cryptographic hash based on data blocks determined by the bitstream, wherein said blocks include metadata. Verifier 102 can use the cryptographic hash to verify the received bitstream and/or associated metadata. For example, if verifier 102 finds the metadata to be valid based on a match between a reference cryptographic hash and a cryptographic hash retrieved from a data block, then processor 103 can be prevented from operating on the corresponding audio data, and selection level 104 can pass through the (unaltered) audio data. Alternatively or alternatively, other types of encryption techniques can be used instead of the cryptographic hash-based method.

The encoder 100 of Figure 2 can determine (in response to the LPSM extracted by the decoder 101 and optionally also in response to program boundary metadata) that the post-processing/pre-processing unit has (in elements 105, 106, and 107) performed a type of loudness processing on the audio data to be encoded, and therefore can (in generator 106) create loudness processing status metadata that includes specific parameters for the previously performed loudness processing and/or based on the previously performed loudness processing. In some implementations, the encoder 100 can create metadata indicating the processing history of the audio content (and include it in the encoded bitstream output from the encoder) as long as the encoder knows the type of processing that has been performed on the audio content.

Figure 3 is a block diagram of a decoder (200) and a post-processor (300) coupled to the decoder (200) according to an embodiment of the audio processing unit of the present invention. The post-processor (300) is also an embodiment of the audio processing unit of the present invention. Any component or element of the encoder 200 and the post-processor 300 may be implemented in hardware, software, or a combination of hardware and software as one or more processes and/or one or more circuits (e.g., ASIC, FPGA, or other integrated circuits). The decoder 200 includes a frame buffer 201, a analyzer 205, an audio decoder 202, an audio state verification stage (verifier) 203, and a control bit generation stage 204 connected as shown. Typically, the decoder 200 also includes other processing elements (not shown).

Frame buffer 201 (buffer memory) stores (e.g., in a non-transient manner) at least one frame of the encoded audio bitstream received by decoder 200. The frame sequence of the encoded audio bitstream is set from buffer 201 to analyzer 205.

The analyzer 205 is coupled and configured to extract PIM and/or SSM (optionally other metadata, such as LPSM) from each frame of the encoded input audio, set at least some of the metadata (e.g., LPSM and program boundary metadata, if either is extracted, and/or PIM and/or SSM) to the audio state verifier 203 and level 204, set the extracted metadata as (e.g., to postprocessor 300) output, extract audio data from the encoded input audio, and set the extracted audio data to the decoder 202.

The encoded audio bitstream input to decoder 200 can be one of AC-3 bitstream, E-AC-3 bitstream, or Dolby E bitstream.

The system of Figure 3 also includes a post-processor 300. The post-processor 300 includes a frame buffer 301 and other processing elements (not shown) including at least one processing element coupled to the buffer 301. The frame buffer 301 stores (e.g., in a non-transient manner) at least one frame of the decoded audio bitstream received by the post-processor 300 from the decoder 200. The processing element coupled to the post-processor 300 is configured to receive a series of frames of the decoded audio bitstream output from the buffer 301 and to adaptively process them using metadata output from the decoder 200 and/or control bits output from stage 204 of the decoder 200. Typically, the post-processor 300 is configured to perform adaptive processing on the decoded audio data using metadata from the decoder 200 (e.g., adaptive loudness processing on the decoded audio data using LPSM values and optionally also using program boundary metadata, wherein the adaptive processing may be based on loudness processing status, and/or one or more audio data characteristics indicated by the LPSM indicating the audio data of a single audio program).

Various implementations of the decoder 200 and the post-processor 300 are configured to perform different embodiments of the method of the present invention.

The audio decoder 202 of the decoder 200 is configured to decode the audio data extracted by the analyzer 205 to generate decoded audio data, and to set the decoded audio data as (e.g., for the post-processor 300) output.

The state verifier 203 is configured to authenticate and verify the metadata set thereto. In some embodiments, the metadata is (or is included in) a data block that has already been included in the input bitstream (e.g., according to an embodiment of the invention). The block may include a cryptographic hash (hash-based message authentication code or "HMAC") for processing the metadata and/or basic audio data (provided from analyzer 205 and/or decoder 202 to verifier 203). The data block may be digitally labeled in these embodiments so that downstream audio processing units can relatively easily authenticate and verify the processing state metadata.

Other encryption methods, including but not limited to any one of one or more non-HMAC encryption methods, can be used for metadata verification (e.g., in verifier 203) to ensure the secure transmission and reception of metadata and/or basic audio data. For example, verification (using such encryption methods) can be performed in each audio processing unit receiving the audio bitstream of the present invention to determine whether the metadata and corresponding audio data included in the bitstream have undergone (and/or been generated by) a specific process (indicated by the metadata) and have not been modified after such specific processing has been performed.

The status verifier 203 sets control data to the control bit generator 204, and/or sets control data as output (e.g., to the post-processor 300) to indicate the result of the verification operation. In response to the control data (and optionally other metadata extracted from the input bitstream), level 204 can generate (and set to the post-processor 300):

A control bit indicating that the decoded audio data output from decoder 202 has undergone a specific type of loudness processing (when LPSM indicates that the audio data output from decoder 202 has undergone that specific type of loudness processing, and the control bit from verifier 203 indicates that LPSM is valid); or

A control bit that indicates that the decoded audio data output from decoder 202 should undergo a specific type of loudness processing (e.g., when LPSM indicates that the audio data output from decoder 202 has not undergone a specific type of loudness processing, or when LPSM indicates that the audio data output from decoder 202 has undergone that specific type of loudness processing but the control bit from verifier 203 indicates that LPSM is invalid).

Alternatively, decoder 200 sets metadata extracted from the input bitstream by decoder 202 and metadata extracted from the input bitstream by analyzer 205 to postprocessor 300, and postprocessor 300 performs adaptive processing on the decoded audio data using the metadata, or performs metadata verification, and then performs adaptive processing on the decoded audio data using the metadata if the verification indicates that the metadata is valid.

In some implementations, if decoder 200 receives an audio bitstream generated according to an embodiment of the invention using cryptographic hashing, the decoder is configured to analyze and retrieve cryptographic hashes from data blocks determined from the bitstream, said blocks including loudness processing state metadata (LPSM). Verifier 203 can use the cryptographic hashes to verify the received bitstream and/or associated metadata. For example, if verifier 203 finds the LPSM valid based on a match between a reference cryptographic hash and a cryptographic hash retrieved from the data block, then the audio data of the (unaltered) bitstream can be signaled to a downstream audio processing unit (e.g., a post-processor 300 that may be or include a volume leveling unit). Alternatively, or alternatively, other types of encryption techniques can be used instead of the cryptographic hash-based method.

In some implementations of decoder 200, the received (and cached in memory 201) encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and includes audio data segments (e.g., segments AB0 to AB5 of the frame shown in Figure 4) and metadata segments, wherein the audio data segments indicate audio data, and each of at least some of the metadata segments includes PIM or SSM (or other metadata). Decoder stage 202 (and/or analyzer 205) is configured to extract metadata from the bitstream. Each metadata segment that includes PIM and/or SSM (optionally also includes other metadata) is included in a useless bit segment of the frame of the bitstream, or in the “addbsi” field of the Bitstream Information (“BSI”) segment of the frame of the bitstream, or in an auxiliary data field at the end of the frame of the bitstream (e.g., the AUX segment shown in Figure 4). A frame of the bitstream may include one or two metadata segments, wherein each metadata segment includes metadata, and if the frame includes two metadata segments, one may exist in the addbsi field of the frame and the other may exist in the AUX field of the frame.

In some implementations, each metadata segment (sometimes referred to herein as a “container”) of the bitstream cached in buffer 201 has a format including a metadata segment header (optionally including other mandatory or “core” elements) and one or more metadata payloads following the metadata segment header. If present, the SIM is included in one of the metadata payloads (identified by a payload header and typically having a first-type format). If present, the PIM is included in another of the metadata payloads (identified by a payload header and typically having a second-type format). Similarly, other types of metadata (if present) are included in another of the metadata payloads (identified by a payload header and typically having a format specific to the type of metadata). This exemplary format enables convenient access to the SSM, PIM, and other metadata at times other than during decoding (e.g., by a post-processor 300 after decoding, or by a processor configured to identify metadata without performing full decoding of the encoded bitstream), and allows for convenient and efficient error detection and correction during the decoding of the bitstream (e.g., for substream identification). For example, if the SSM is not accessed in the exemplary format, the decoder 200 may incorrectly identify the correct number of substreams associated with the program. One metadata payload in a metadata segment may include the SSM, another metadata payload in a metadata segment may include the PIM, and optionally, at least one other metadata payload in a metadata segment may include other metadata (e.g., loudness processing status metadata or "LPSM").

In some implementations, the substream structure metadata (SSM) payload in frames containing encoded bitstreams (e.g., E-AC-3 bitstreams indicating at least one audio program) buffered in buffer 201 includes an SSM in the following format:

The payload header typically includes at least one identification value (e.g., a 2-bit value indicating the SSM format version, and optionally length, period, count, and substream association values); and

Following the masthead:

Independent substream metadata indicating the number of independent substreams of a program indicated by a bitstream; and

Sub-stream metadata indicates whether each independent sub-stream of a program has at least one associated sub-stream, and if so, the number of associated sub-streams of each independent sub-stream of the program.

In some implementations, the Program Information Metadata (PIM) payload included in frames of the encoded bitstream (e.g., an E-AC-3 bitstream indicating at least one audio program) buffered in buffer 201 has the following format:

The payload header typically includes at least one identification value (e.g., a value indicating the PIM format version, and optionally length, period, count, and substream association values); and following the header, the PIM in the following format:

The active channel metadata for each silent channel and each non-silent channel of the audio program (i.e., which channels of the program contain audio information and which channels (if any) contain only silence (typically related to the duration of the frame)). In implementations where the encoded bitstream is an AC-3 or E-AC-3 bitstream, the active channel metadata in the frames of the bitstream can be combined with additional metadata of the bitstream (e.g., the frame's audio encoding mode ("acmod") field, and, if present, the chanmap field in the frame or associated subordinate substream frames) to determine which channels of the program contain audio information and which channels contain silence;

Downmixing processing status metadata indicates whether the program is downmixed (before or during encoding), and if so, the type of downmixing applied. This downmixing processing status metadata can facilitate downstream upmixing by the decoder (in post-processor 300), for example, upmixing the program's audio content with parameters that best match the type of downmixing applied. In implementations where the encoded bitstream is AC-3 or E-AC-3, the downmixing processing status metadata can be combined with the frame's audio coding model ("acmod") field to determine the type of downmixing (if any) applied to the program's channels.

Upmixing processing state metadata indicates whether the program was upmixed (e.g., from a smaller number of channels) before or during encoding, and if so, the type of upmixing applied. This upmixing processing state metadata can facilitate downstream downmixing (in the post-processor) of the decoder, for example, downmixing the program's audio content in a manner consistent with the type of upmixing applied to the program (e.g., Dolby Pro Logic, or Dolby Pro Logic II Cinema Mode, or Dolby Pro Logic II Music Mode, or Dolby Pro Upmixer). In implementations where the encoded bitstream is an E-AC-3 bitstream, the upmixing processing state metadata can be combined with other metadata (e.g., the value of the "strmtyp" field of a frame) to determine the type of upmixing (if any) applied to the channels of the program. The value of the “strmtyp” field (in the BSI field of a frame of an E-AC-3 bitstream) indicates whether the audio content of the frame belongs to an independent stream (which defines a program) or an independent substream (including multiple substreams or programs associated with multiple substreams), thus allowing it to be encoded independently of any other substream indicated by the E-AC-3 bitstream, or whether the audio content of the frame belongs to a subordinate substream (including multiple substreams or programs associated with multiple substreams), thus requiring decoding in conjunction with its associated independent substream; and

Preprocessing status metadata, which indicates whether preprocessing was performed on the audio content of the frame (before encoding the audio content to generate the encoded bitstream), and if so, the type of preprocessing performed.

In some implementations, preprocessed state metadata indicates:

Whether surround attenuation was applied (e.g., whether the surround channels of the audio program were attenuated by 3dB before encoding).

Whether (e.g., whether a 90° phase shift was applied to the surround channels Ls and Rs of the audio program before encoding)

Was a low-pass filter applied to the LFE channel of the audio program before encoding?

During generation, is the level of the program's LFE channel monitored, and if so, is the monitored level of the LFE channel relative to the levels of the program's full-range audio channels?

Whether dynamic range compression should be performed (e.g., in the decoder) on each block of the decoded audio of the program, and if so, the type (and/or parameters) of dynamic range compression to be performed (e.g., the type of preprocessing state metadata could indicate which of the following compression profile types is assumed by the encoder to generate the dynamic range compression control values included in the encoded bitstream: cinematic standard, cinematic ray, music standard, music ray, or speech. Alternatively, the type of preprocessing state metadata could indicate that re-dynamic range compression (“compr” compression) should be performed on each frame of the decoded audio content of the program in a manner determined by the dynamic range compression control values included in the encoded bitstream).

Whether spectral spreading and/or channel coupling coding is used to encode the content of a program within a specific frequency range, and if spectral spreading and/or channel coupling coding is used, the minimum and maximum frequencies of the frequency components of the content for which spectral spreading coding is performed, and the minimum and maximum frequencies of the frequency components of the content for which channel coupling coding is performed. This type of preprocessing state metadata information can help perform equalization (in post-processing) downstream in the decoder. Both channel coupling information and spectral spreading information also help optimize quality during code conversion operations and applications. For example, the encoder can optimize its behavior (including adaptation of preprocessing steps such as headset virtualization, upmixing, etc.) based on the state of parameters such as spectral spreading and channel coupling information. Moreover, the encoder can dynamically modify its coupling and spectral spreading parameters to match optimal values and/or modify its coupling and spectral spreading parameters to optimal values based on the state of incoming (and certified) metadata.

Whether the dialogue enhancement adjustment range data is included in the encoded bitstream, and if so, the adjustment range that can be obtained during the execution of dialogue enhancement processing (e.g., downstream of the decoder's post-processor) that adjusts the level of dialogue content relative to the level of non-dialogue content in the audio program.

In some implementations, the LPSM payload in a frame including an encoded bitstream (e.g., an E-AC-3 bitstream indicating at least one audio program) buffered in buffer 201 includes an LPSM in the following format:

The header (typically includes a synchronization word identifying the start of the LPSM payload, followed by at least one identification value, such as the LPSM format version, length, period, count, and substream association value indicated in Table 2 below); and

Following the masthead:

At least one dialogue representation value (e.g., the parameter "Dialogue Channels" in Table 2) that indicates whether the corresponding audio data indicates dialogue or not (e.g., which channels of the corresponding audio data indicate dialogue).

Indicates whether the corresponding audio content conforms to at least one loudness adjustment conformity value of the indicated set (e.g., the parameter "loudness adjustment type" in Table 2);

At least one loudness processing value indicating at least one type of loudness processing has been performed on the corresponding audio data (e.g., one or more of the parameters “Dialogue Gating Loudness Correction Flag” and “Loudness Correction Type” in Table 2); and

At least one loudness value (e.g., peak or average loudness) indicating at least one loudness characteristic of the corresponding audio data (e.g., one or more of the parameters “ITU relative gated loudness”, “ITU speech gated loudness”, “ITU (EBU 3341) short 3s loudness” and “true peak” in Table 2).

In some implementations, the analyzer 205 (and/or decoder stage 202) is configured to extract each metadata segment with the following format from the useless bit segments or “addbsi” fields or auxiliary data segments of the frames of the bitstream:

Metadata segment header (typically includes a synchronization word identifying the start of the metadata segment, followed by identification values such as version, length, period, extended element count, and substream association value); and

At least one protected value following the metadata segment header that is helpful for the decryption, authentication, or verification of at least one of the metadata segments or corresponding audio data (e.g., the HMAC digest and audio fingerprint value in Table 1); and

Also following the metadata segment header are metadata payload identifiers (“ID”) values and payload configuration values that identify the type of metadata in each of the following metadata payloads and represent at least one aspect of the configuration (e.g., size) of each such payload.

Each metadata payload segment (preferably in the format specified above) follows the corresponding metadata payload ID value and metadata configuration value.

More generally, the encoded audio bitstream generated by the preferred embodiments of the present invention has a structure that provides a mechanism for marking metadata elements and sub-elements as core (mandatory) or extended (optional) elements or sub-elements. This allows the data rate of the bitstream (including its metadata) to be scaled to a wide range of applications. The core (mandatory) elements of the preferred bitstream syntax should also be able to signal the presence of extended (optional) elements associated with the audio content at (in-band) and/or remote (out-of-band) locations.

The core element is required to be present in every frame of the bitstream. Some child elements of the core element are optional and can exist in any combination. The extension element is not required to be present in every frame (to limit the total bitrate overhead). Therefore, the extension element can exist in some frames but not in others. Some child elements of the extension element are optional and can exist in any combination; however, some child elements of the extension element can be mandatory (i.e., if the extension element is present in a frame of the bitstream).

In one type of implementation, the generation (e.g., by implementing the audio processing unit of the present invention) comprises an encoded audio bitstream including a series of audio data segments and metadata segments. The audio data segments indicate audio data, each of at least some of the metadata segments includes PIM and/or SSM (and optionally at least one other type of metadata), and the audio data segments are time-division multiplexed with the metadata segments. In a preferred embodiment of this type, each of the metadata segments has a preferred format as described herein.

In a preferred format, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment in the metadata segment, including SSM and/or PIM, is included (e.g., by stage 107 of a preferred implementation of encoder 100) as the “addbsi” field of the Bitstream Information (“BSI”) segment of the frame of the bitstream (shown in FIG. 6), or in the auxiliary data field of the frame of the bitstream, or in the unused bit segments of the frame of the bitstream as additional bitstream information.

In the preferred format, each frame includes a metadata segment (sometimes referred to herein as a metadata container or container) within a frame's unused bit field (or addbsi field). The metadata segment has the mandatory elements shown in Table 1 below (collectively referred to as "core elements") (and may include the optional elements shown in Table 1). At least some of the required elements shown in Table 1 are included in the metadata segment header of the metadata segment, but some may be included in other locations within the metadata segment:

Table 1

In the preferred format, each metadata segment containing an SSM, PIM, or LPSM (in a useless bit segment or addbsi or auxiliary data field of a frame of the encoded bitstream) includes a metadata segment header (and optionally additional core elements), and one or more metadata payloads following the metadata segment header (or metadata segment header and other core elements). Each metadata payload includes a metadata payload header (indicating the specific type of metadata (e.g., SSM, PIM, or LPSM)) included in the payload, followed by the metadata of that specific type. Typically, the metadata payload header includes the following values (parameters):

The payload ID (identifying the type of metadata, such as SSM, PIM, or LPSM) following the metadata segment header (which may include the values specified in Table 1);

The payload configuration value following the payload ID (usually indicating the size of the payload);

It may also optionally include additional payload configuration values (e.g., an offset value indicating the number of audio samples from the beginning of the frame to the first audio sample involved in the payload, and a payload priority value, e.g., indicating the conditions under which the payload may be dropped).

Typically, payload metadata has one of the following formats:

The payload metadata is SSM, which includes independent substream metadata indicating the number of independent substreams of the program indicated by the bitstream; and dependent substream metadata indicating whether each independent substream of the program has at least one dependent substream associated with it, and if each independent substream of the program has at least one dependent substream associated with it, the number of dependent substreams associated with each independent substream of the program.

The payload metadata is PIM, including metadata indicating which channels of the audio program contain audio information and which channels (if any) contain only active channels with silence (typically related to the duration of a frame); downmixing processing status metadata, indicating whether the program is downmixed (before or during encoding), and if so, the type of downmixing applied; upmixing processing status metadata, indicating whether the program is upmixed (e.g., from a smaller number of channels) before or during encoding, and if so, the type of upmixing applied; and preprocessing status metadata, indicating whether preprocessing was performed on the audio data of the frame (before encoding the audio content that generates the encoded bitstream), and if so, the type of preprocessing performed; or

The payload metadata is an LPSM, which has the format indicated in the table below (Table 2):

Table 2

In another preferred format of the encoded bitstream generated according to the invention, the bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment (e.g., level 107 of a preferred implementation of encoder 100) including PIM and/or SSM (optionally including at least one other type of metadata) is included in any of the following: a useless bit segment of the frame of the bitstream; or the “addbsi” field of the bitstream information (“BSI”) segment of the frame of the bitstream (shown in FIG. 6); or an auxiliary data field at the end of the frame of the bitstream (e.g., the AUX segment shown in FIG. 4). A frame may include one or two metadata segments, each of which includes PIM and/or SSM, and (in some embodiments) if the frame includes two metadata segments, one may exist in the addbsi field of the frame and the other in the AUX field of the frame. Each metadata segment preferably has a format specified above, referring to Table 1 above (i.e., including the core element specified in Table 1, followed by a payload ID value (identifying the type of metadata in each payload of the metadata segment) and a payload configuration value, and each metadata payload). Each metadata segment including LPSM preferably has a format specified above, referring to Tables 1 and 2 above (i.e., including the core element specified in Table 1, followed by a payload ID (identifying the metadata as LPSM) and a payload configuration value, followed by the payload (LPSM data with the format indicated in Table 2)).

In another preferred format, the encoded bitstream is a Dolby E bitstream, and each metadata segment, including PIM and/or SSM (optionally including other metadata), represents the first N sample location of the Dolby E guard band interval. A Dolby E bitstream including such metadata segments containing LPSM preferably includes a value indicating the LPSM payload length signaled in the Pd word of the SMPTE 337M presync signal (the SMPTE 337M Pa word repetition frequency is preferably maintained at the same as the associated video frame rate).

In the preferred format, where the encoded bitstream is an E-AC-3 bitstream, each metadata segment (e.g., level 107 of the preferred implementation of encoder 100) including PIM and/or SSM (optionally also including LPSM and/or other metadata) is included with additional bitstream information in the "addbsi" field of the useless bit segment or bitstream information ("BSI") segment of the frame as the bitstream. Further aspects of encoding the E-AC-3 bitstream using LPSM in this preferred format are described below:

1. During the generation of the E-AC-3 bitstream, although the E-AC-3 encoder (which inserts LPSM values into the bitstream) is "active," for each generated frame (synchronization frame), the bitstream should include a metadata block (including LPSM) carried in the addbsi field (or unused bit field) of the frame. It is required that the bits carrying the metadata block should not increase the encoder bit rate (frame length);

2. Each metadata block (including LPSM) should contain the following information:

Loudness correction type flag: where “1” indicates that the loudness of the corresponding audio data is corrected upstream of the encoder, while “0” indicates that the loudness is corrected by a loudness corrector embedded in the encoder (e.g., loudness processor 103 of encoder 100 in Figure 2).

Voice Channels: Indicates which source channels contain voice (in the previous 0.5 seconds). This should be indicated if no voice is detected.

Speech loudness: Indicates the overall speech loudness of each corresponding audio channel, including the speech (in the preceding 0.5 seconds);

ITU loudness: Indicates the overall ITU BS.1770-3 loudness for each corresponding audio channel; and

Gain: The loudness composite gain of the inverted frequency in the decoder (to indicate reversibility);

3. When the E-AC-3 encoder (which inserts LPSM values into the bitstream) is "active" and is receiving AC-3 frames with the "trust" flag, the loudness controller in the encoder (e.g., loudness processor 103 of encoder 100 in Figure 2) should be bypassed. The "trusted" source dialogue normalization and DRC values should be passed (e.g., by generator 106 of encoder 100) to the E-AC-3 encoder components (e.g., stage 107 of encoder 100). LPSM block generation continues, and the loudness correction type flag is set to "1". The loudness controller bypass sequence must be synchronized to the beginning of the decoded AC-3 frame where the "trust" flag appears. The loudness controller bypass sequence should be implemented as follows: the leveler quantity control decreases from value 9 to value 0 across 10 audio block cycles (i.e., 53.3 milliseconds), and the leveler return end meter control is placed in bypass mode (this operation should result in a seamless transition). The term "trusted" bypass in the regulator implies that the dialogue normalization value of the source bitstream is still reused at the output of the encoder. (For example, if the "trusted" source bitstream has a dialogue normalization value of -30, then the encoder output should use -30 for the output dialogue normalization value.)

4. When the E-AC-3 encoder (which inserts LPSM values into the bitstream) is "active" and is receiving AC-3 frames without the "trust" flag, the loudness controller embedded in the encoder (e.g., loudness processor 103 of encoder 100 in Figure 2) should be active. LPSM block generation continues, and the loudness correction type flag is set to "0". The loudness controller activation sequence should be synchronized to the beginning of the decoded AC-3 frame in which the "trust" flag disappears. The loudness controller activation sequence should be implemented as follows: the leveler quantity control increases from value 0 to value 9 across one audio block period (e.g., 5.3 ms), and the leveler return-to-end meter control is placed in "active" mode (this operation should result in a seamless transition and include a return-to-end meter integrated reset); and 5. During encoding, the graphical user interface (GUI) should indicate the following parameters to the user: "Input audio program: [trusted/untrusted]"—the state of this parameter is based on the presence of the "trusted" flag within the input signal; and "Real-time loudness correction: [enabled/disabled]"—the state of this parameter is based on whether the loudness controller embedded in the encoder is active.

When decoding an AC-3 or E-AC-3 bitstream that includes LSPM (in the preferred format) in the "addbsi" field of the unused bit field, skip field, or Bitstream Information ("BSI") segment in each frame of the bitstream, the decoder should analyze the LPSM block data (in the unused bit field or addbsi field) and pass all extracted LPSM values to the graphical user interface (GUI). The set of extracted LPSM values is refreshed in each frame.

In another preferred format of the encoded bitstream generated according to the invention, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment (e.g., level 107 of a preferred implementation of encoder 100) including PIM and/or SSM (optionally also including LPSM and/or other metadata) is included as additional bitstream information in the unused bit segment or AUX segment of the frame of the bitstream or as the “addbsi” field (shown in FIG. 6) of the bitstream information (“BSI”) segment. In this format (a variation of the format described above with reference to Tables 1 and 2), each field in the addbsi (or AUX or unused bit) field containing LPSM contains the following LPSM value:

The core elements specified in Table 1 are followed by the payload ID (identifying metadata as LPSM) and the payload value, followed by the payload (LPSM data) in the following format (similar to the mandatory elements shown in Table 2 above):

LPSM payload version: A 2-bit field indicating the version of the LPSM payload;

dialchan: A 3-bit field indicating the left, right, and/or center channel containing the corresponding audio data for spoken dialogue. The bit allocation of the dialchan field can be as follows: bit 0, indicating the presence of dialogue in the left channel, is stored in the most significant bit of the dialchan field; while bit 2, indicating the presence of dialogue in the center channel, is stored in the least significant bit of the dialchan field. If the corresponding channel contains spoken dialogue during the first 0.5 seconds of the program, each bit of the dialchan field is set to "1".

loudregtyp: A 4-bit field indicating which loudness adjustment standard the program loudness conforms to. Setting the "loudregtyp" field to "0000" indicates that the LPSM does not indicate loudness adjustment conformance. For example, one value of this field (e.g., 0000) could indicate no loudness adjustment standard conformance, another value (e.g., 0001) could indicate that the program's audio data conforms to the ATSC A/85 standard, and yet another value (e.g., 0010) could indicate that the program's audio data conforms to the EBU R128 standard. In this example, if this field is set to any value other than "0000", the payload should subsequently include the loudcorrdialgat and loudcorrtyp fields.

loudcorrdialgat: A 1-bit field indicating whether dialogue gating correction has been applied. If dialogue gating correction has been applied to the program's loudness, the value of the loudcorrdialgat field is set to "1". Otherwise, it is set to "0".

loudcorrtyp: A 1-bit field indicating the type of loudness correction applied to the program. If the program's loudness has already been corrected using infinite lead (file-based) loudness correction processing, the value of the loudcorrtyp field is set to "0". If the program's loudness has already been corrected using a combination of real-time loudness measurement and dynamic range control, the value of this field is set to "1".

loudrelgate: A 1-bit field indicating the presence of the relative gated program loudness (ITU). If the loudrelgate field is set to "1", the payload should then be followed by a 7-bit ituloudrelgate field;

loudrelgat: A 7-bit field indicating the relative gated program loudness (ITU). This field indicates the overall loudness of the audio program as measured according to ITU-R BS.1770-3, without any gain adjustment, due to the applied dialogue normalization and dynamic range compression (DRC). Values from 0 to 127 are interpreted as -58 LKFS to +5.5 LKFS in 0.5 LKFS steps.

loudspchgate: A 1-bit field indicating the presence of speech gated loudness data (ITU). If the loudspchgate field is set to "1", the payload should then be followed by a 7-bit loudspchgate field;

loudspchgate: A 7-bit field indicating the loudness of the speech gated program. This field indicates the overall loudness of the entire corresponding audio program, measured according to Formula (2) of ITU-R BS.1770-3, without any gain adjustment, due to the applied dialogue normalization and dynamic range compression. Values from 0 to 127 are interpreted as -58LKFS to +5.5LKFS in 0.5LKFS steps;

loudstrm3e: A 1-bit field indicating the presence of short-term (3-second) loudness data. If this field is set to "1", the payload should be followed by a 7-bit loudstrm3s field;

loudstrm3s: A 7-bit field indicating the ungated loudness of the first 3 seconds of the corresponding audio program, measured according to ITU-R BS.1771-1 without any gain adjustment, due to the applied dialogue normalization and dynamic range compression. Values from 0 to 256 are interpreted as -116LKFS to +11.5LKFS in 0.5LKFS steps;

`truepke`: A 1-bit field indicating the presence of true peak loudness data. If the `truepke` field is set to "1", the payload should then be followed by an 8-bit `truepke` field; and

truepk: An 8-bit field indicating the true peak sample value of the program as measured according to Annex 2 of ITU-R BS.1770-3 without any gain adjustment, due to the applied dialogue normalization and dynamic range compression. Values from 0 to 256 are interpreted as -116LKFS to +11.5LKFS in 0.5LKFS steps.

In some implementations, the core element of a metadata segment in the unused bit field or auxiliary data (or "addbsi") field of an AC-3 bitstream or E-AC-3 bitstream frame includes a metadata segment header (typically including an identification value, e.g., version), and following the metadata segment header are: a value indicating whether the metadata of the metadata segment includes fingerprint data (or other protection values); a value indicating the presence of external data (related to the audio data corresponding to the metadata of the metadata segment); a payload ID value and a payload configuration value for each type of metadata (e.g., PIM and/or SSM and/or LPSM and/or one type of metadata) identified by the core element; and a protection value for at least one type of metadata identified by the metadata segment header (or other core elements of the metadata segment). The metadata payload of the metadata segment follows the metadata segment header and (in some cases) is nested within the core element of the metadata segment.

Embodiments of the present invention can be implemented in hardware, firmware, or software, or a combination of hardware and software (e.g., as a programmable logic array). Unless otherwise specified, the algorithms or processes included as part of the present invention do not inherently involve any particular computer or other device. Specifically, various general-purpose machines can be used with programs written in accordance with the teachings herein, or more specific devices (e.g., integrated circuits) can be constructed to perform the required method steps. Thus, the present invention can be implemented as one or more computer programs executing on one or more programmable computer systems (e.g., any implementation of the elements of FIG. 1, or the encoder 100 (or elements of the encoder) of FIG. 2, or the decoder (or elements of the decoder) of FIG. 3, or the post-processor (or elements of the post-processor) of FIG. 3), each programmable computer system including at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices in a known manner.

Each such program can be implemented in any desired computer language (including machine, assembly, or high-level procedural, logical, or object-oriented programming languages) to communicate with a computer system. In any case, the language can be a compiled language or an interpreted language.

For example, when implemented by a sequence of computer software instructions, the various functions and steps of embodiments of the present invention can be implemented by a sequence of multi-threaded software instructions running in appropriate digital signal processing hardware. In this case, the various means, steps and functions of the embodiments can correspond to parts of the software instructions.

Each such computer program is preferably stored or downloaded to a storage medium or device readable by a general-purpose or special-purpose programmable computer (e.g., solid-state memory or medium, magnetic medium, or optical medium) for configuring and operating the computer when the storage medium or device is read by a computer system to perform the processes described herein. The system of the present invention can also be implemented as a computer-readable storage medium configured (e.g., storing) a computer program, wherein such a configuration causes the computer system to operate in a specific and predefined manner to perform the functions described herein.

Numerous embodiments of the invention have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the invention. In view of the foregoing teachings, numerous modifications and variations of the invention are possible. It should be understood that the invention can be practiced differently than those specifically described herein, within the scope of the appended claims.

The present invention also includes the following solutions:

Solution 1. An audio processing unit, comprising:

Buffer memory; and

At least one processing subsystem coupled to the buffer memory, wherein the buffer memory stores at least one frame of an encoded audio bitstream, the frame including program information metadata or substream structure metadata in at least one metadata segment of at least one skip field of the frame and audio data in at least one other segment of the frame, wherein the processing subsystem is coupled and configured to perform at least one of the following using the metadata of the bitstream: generation of the bitstream, decoding of the bitstream, or adaptive processing of the audio data of the bitstream; or to perform at least one of the following using the metadata of the bitstream: authentication or verification of at least one of the audio data or metadata of the bitstream.

The metadata segment includes at least one metadata payload, which includes:

Header; and

Following the header, at least a portion of the program information metadata or at least a portion of the substream structure metadata.

Solution 2. The audio processing unit according to Solution 1, wherein the encoded audio bitstream indicates at least one audio program, and the metadata segment includes a program information metadata payload, the program metadata payload including:

Program information metadata header; and

Following the program information metadata header is program information metadata indicating at least one attribute or characteristic of the audio content of the program, the program information metadata including metadata indicating each non-mute channel and each mute channel of the program.

Option 3. The audio processing unit according to Option 2, wherein the program information metadata further includes one of the following:

Downmixing status metadata, which indicates whether the program has been downmixed, and if the program has been downmixed, the type of downmixing applied to the program;

Upmixing status metadata, which indicates whether the program is upmixed, and if the program is upmixed, the type of upmixing applied to the program;

Preprocessing status metadata, indicating whether preprocessing was performed on the audio content of the frame, and if so, the type of preprocessing performed on the audio content; or

Spectrum spreading or channel coupling metadata, indicating whether spectrum spreading or channel coupling has been applied to the program, and the frequency range in which spectrum spreading or channel coupling has been applied if spectrum spreading or channel coupling has been applied to the program.

Solution 4. The audio processing unit according to Solution 1, wherein the encoded audio bitstream indicates at least one audio program of at least one independent substream having audio content, and the metadata segment includes a substream structure metadata payload, the substream structure metadata payload including:

Substream structure metadata payload header; and

Following the substream structure metadata payload header are independent substream metadata indicating the number of independent substreams of the program, and dependent substream metadata indicating whether each independent substream of the program has at least one associated dependent substream.

Option 5. The audio processing unit according to Option 1, wherein the metadata segment includes:

Metadata segment header;

At least one protection value following the metadata segment header, used for at least one of the decryption, authentication, or verification of the program information metadata, or the substream structure metadata, or the audio data corresponding to the program information metadata or the substream structure metadata; and

The metadata payload identifier value and the payload configuration value follow the metadata segment header, wherein the metadata payload follows the metadata payload identifier value and the payload configuration value.

Solution 6. The audio processing unit according to Solution 5, wherein the metadata segment header includes a synchronization word identifying the start of the metadata segment and at least one identification value following the synchronization word, and the header of the metadata payload includes at least one identification value.

Scheme 7. The audio processing unit according to Scheme 1, wherein the encoded audio bitstream is an AC-3 bitstream or an E-AC-3 bitstream.

Option 8. The audio processing unit according to Option 1, wherein the buffer memory stores the frame in a non-transitory manner.

Solution 9. The audio processing unit according to Solution 1, wherein the audio processing unit is an encoder.

Solution 10. The audio processing unit according to Solution 9, wherein the processing subsystem includes:

A decoding subsystem configured to receive an input audio bitstream and extract input metadata and input audio data from the input audio bitstream;

An adaptive processing subsystem, coupled to and configured to perform adaptive processing on the input audio data using the input metadata, thereby generating processed audio data; and

An encoding subsystem, coupled to and configured to respond to the processed audio data, includes generating the encoded audio bitstream by including the program information metadata or the substream structure metadata in the encoded audio bitstream, and setting the encoded audio bitstream to the buffer memory.

Solution 11. The audio processing unit according to Solution 1, wherein the audio processing unit is a decoder.

Option 12. The audio processing unit according to Option 11, wherein the processing subsystem is a decoding subsystem coupled to the buffer memory and configured to extract the program information metadata or the substream structure metadata from the encoded audio bitstream.

Solution 13. The audio processing unit according to Solution 1 includes:

A subsystem, coupled to the buffer memory and configured to: extract the program information metadata or the substream structure metadata from the encoded audio bitstream, and extract the audio data from the encoded audio bitstream; and

A post-processor, coupled to the subsystem, is configured to perform adaptive processing on the audio data using at least one of the program information metadata extracted from the encoded audio bitstream or the substream structure metadata.

Option 14. The audio processing unit according to Option 1, wherein the audio processing unit is a digital signal processor.

Solution 15. The audio processing unit according to Solution 1, wherein the audio processing unit is a preprocessor, the preprocessor being configured to extract the program information metadata or the substream structure metadata and the audio data from the encoded audio bitstream, and to perform adaptive processing on the audio data using at least one of the program information metadata or the substream structure metadata extracted from the encoded audio bitstream.

Solution 16. A method for decoding an encoded audio bitstream, the method comprising the following steps:

Receive encoded audio bitstream; and

Metadata and audio data are extracted from the encoded audio bitstream, wherein the metadata is or includes program information metadata and substream structure metadata.

The encoded audio bitstream comprises a series of frames and indicates at least one audio program. The program information metadata and the substream structure metadata indicate the program. Each of the frames includes at least one audio data segment, and each audio data segment includes at least a portion of the audio data. Each frame in at least a subset of the frames includes a metadata segment, and each metadata segment includes at least a portion of the program information metadata and at least a portion of the substream structure metadata.

Solution 17. The method according to Solution 16, wherein the metadata segment includes a program information metadata payload, and the program information metadata payload includes:

Program information metadata header; and

The program information metadata following the program information metadata header indicates at least one attribute or characteristic of the audio content of the program, and the program information metadata includes metadata indicating each non-mute channel and each mute channel of the program.

Option 18. The method according to Option 17, wherein the program information metadata further includes at least one of the following:

Downmixing status metadata, which indicates whether the program has been downmixed, and if the program has been downmixed, the type of downmixing applied to the program;

Upmixing status metadata, indicating whether the program is upmixed, and if so, the type of upmixing applied to the program; or

Preprocessing status metadata, which indicates whether preprocessing was performed on the audio content of the frame, and if so, the type of preprocessing performed on the audio content.

Option 19. The method according to Option 16, wherein the encoded audio bitstream indicates at least one audio program of at least one independent substream having audio content, and the metadata segment includes a substream structure metadata payload, the substream structure metadata payload comprising:

Substream structure metadata payload header; and

Following the substream structure metadata payload header are independent substream metadata indicating the number of independent substreams of the program and dependent substream metadata indicating whether each independent substream of the program has at least one associated dependent substream.

Option 20. The method according to Option 16, wherein the metadata segment includes:

Metadata segment header;

At least one protection value following the metadata segment header is used for at least one of the decryption, authentication, or verification of the program information metadata, the substream structure metadata, or the audio data corresponding to the program information metadata and the substream structure metadata; and

The metadata payload following the metadata segment header includes at least a portion of the program information metadata and at least a portion of the substream structure metadata.

Option 21. The method according to Option 16, wherein the encoded audio bitstream is an AC-3 bitstream or an E-AC-3 bitstream.

Option 22. The method according to Option 16 further includes the step of:

Adaptive processing is performed on the audio data using at least one of the program information metadata or the substream structure metadata extracted from the encoded audio bitstream.

Claims (3)

1. An audio processing unit, comprising: One or more processors; A memory coupled to the one or more processors and configured to store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations including: Receive an encoded audio bitstream comprising an audio program, the encoded audio bitstream comprising encoded audio data of a set of one or more audio channels and metadata associated with the set of audio channels, wherein the metadata includes loudness processing status metadata, and wherein the loudness processing status metadata includes metadata indicating the loudness of the audio program; The encoded audio data is decoded to obtain the decoded audio data of the set of audio channels; The loudness processing status metadata is obtained from the metadata of the encoded audio bitstream; and Adaptive loudness processing is performed on the decoded audio data of the set of audio channels based on the loudness processing state metadata; The metadata also includes program information metadata, which indicates a compression profile for creating Dynamic Range Compressed (DRC) data in the bitstream, wherein the compression profile is a music standard compression profile. 2. A method executed by an audio processing unit, comprising: Receive an encoded audio bitstream comprising an audio program, the encoded audio bitstream comprising encoded audio data of a set of one or more audio channels and metadata associated with the set of audio channels, wherein the metadata includes loudness processing status metadata, and wherein the loudness processing status metadata includes metadata indicating the loudness of the audio program; The encoded audio data is decoded to obtain the decoded audio data of the set of audio channels; The loudness processing status metadata is obtained from the metadata of the encoded audio bitstream; and Adaptive loudness processing is performed on the decoded audio data of the set of audio channels based on the loudness processing state metadata; The metadata also includes program information metadata, which indicates a compression profile for creating Dynamic Range Compressed (DRC) data in the bitstream, wherein the compression profile is a music standard compression profile. 3. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: Receive an encoded audio bitstream comprising an audio program, the encoded audio bitstream comprising encoded audio data of a set of one or more audio channels and metadata associated with the set of audio channels, wherein the metadata comprises loudness processing status metadata, and wherein the loudness processing status metadata comprises metadata indicating the loudness of the audio program. The encoded audio data is decoded to obtain the decoded audio data of the set of audio channels; The loudness processing status metadata is obtained from the metadata of the encoded audio bitstream; and Adaptive loudness processing is performed on the decoded audio data of the set of audio channels based on the loudness processing state metadata; The metadata also includes program information metadata, which indicates a compression profile for creating Dynamic Range Compressed (DRC) data in the bitstream, wherein the compression profile is a music standard compression profile.