HK1248913B - Audio encoder and decoder with program loudness and boundary metadata - Google Patents

Description

Audio encoders and decoders utilizing program loudness and boundary metadata

This application is a divisional application of PCT application with international application number PCT/US2014/011672 filed on January 15, 2014, and invention name “Audio encoder and decoder using program loudness and boundary metadata”, which entered the Chinese national phase on April 13, 2015, and has national application number 201480002687.0.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent Application No. 61/754,882, filed January 21, 2013, and U.S. Provisional Patent Application No. 61/824,010, filed May 16, 2013, the entire contents of each of which are hereby incorporated by reference herein.

Technical Field

The present invention relates to audio signal processing, and more particularly, to encoding and decoding audio data bitstreams using metadata indicating the loudness processing state of the audio content and the location of audio program boundaries indicated by the bitstream. Some embodiments of the present invention generate or decode audio data in one of the formats known as AC-3, Enhanced AC-3 or E-AC-3, or Dolby E.

Background Art

"Dolby," "Dolby Digital," "Dolby Digital Plus," and "Dolby E" are trademarks of Dolby Laboratories, Inc. Dolby Laboratories provides proprietary implementations of AC-3 and E-AC-3 known as "Dolby Digital" and "Dolby Digital Plus," respectively.

Audio data processing units typically operate in a blind manner, oblivious to the processing history of the audio data that occurred before the data was received. This may work within a processing framework where a single entity performs all audio data processing and encoding for a variety of target media rendering devices, while the target media rendering devices also perform all decoding and rendering of the encoded audio data. However, this blind processing does not work well (or at all) in situations where multiple audio processing units are spread across a variety of networks or placed in series (i.e., chained) and are expected to optimally perform their respective types of audio processing. For example, some audio data may be encoded for use in high-performance media systems and may have to be converted along the media processing chain into a reduced form suitable for mobile devices. As a result, an audio processing unit may not have to perform a certain type of audio processing on audio data that has already been processed. For example, a volume leveling unit may perform processing on an input audio clip regardless of whether the same or similar volume adjustment has previously been performed on the input audio clip. Therefore, the volume leveling unit may perform adjustments when not necessary. This unnecessary processing may also result in the removal and/or degradation of specific features when rendering the content of the audio data.

A typical stream of audio data includes both audio content (e.g., one or more channels of audio content) and metadata representing at least one characteristic of the audio content. For example, in an AC-3 bitstream, there are several audio metadata parameters that are specifically intended to be used to alter the sound of a program delivered to a listening environment. One of the metadata parameters is a "DIALNORM" parameter that is intended to represent the average level of a conversation in which an audio program occurs and is used to determine the audio playback signal level.

During playback of a bitstream that includes a sequence of different audio program segments (each with different DIALNORM parameters), an AC-3 decoder uses the DIALNORM parameters for each segment to perform a type of loudness processing in which it modifies the playback level or loudness so that the perceived loudness of the session for the sequence of segments is at a constant level. Each encoded audio segment (item) in a sequence of encoded audio items will (usually) have different DIALNORM parameters, and the decoder can scale the level of each of the items so that the playback level or loudness of the session for each item is the same or very similar, although this may require applying different amounts of gain to the different items during playback.

DIALNORM is typically set by the user and is not automatically generated, although there is a default DIALNORM value if the user does not set any value. For example, the content creator may perform loudness measurement using a device external to the AC-3 encoder and then transmit the results (representing the loudness of the spoken dialogue of the audio program) to the encoder to set the DIALNORM value. Therefore, there is a reliance on the content creator to correctly set the DIALNORM parameter.

There are several different reasons why the DIALNORM parameter in an AC-3 bitstream may be incorrect. First, if the content creator does not set the DIALNORM value, each AC-3 encoder has a default DIALNORM value that is used during bitstream generation. This default value may be quite different from the actual conversational loudness level of the audio. Second, even if the content creator measures loudness and sets the DIALNORM value accordingly, it may use a loudness measurement algorithm or meter that does not adhere to the recommended AC-3 loudness measurement method, resulting in an incorrect DIALNORM value. Third, even if the AC-3 bitstream has been generated using the DIALNORM value correctly measured and set by the content creator, it may have been changed to an incorrect value during the transmission and/or storage of the bitstream. For example, in television broadcast applications, it is not uncommon for an AC-3 bitstream to be decoded, modified, and then re-encoded using incorrect DIALNORM metadata information. As a result, the DIALNORM value included in the AC-3 bitstream may be incorrect or inaccurate, which may negatively impact the quality of the listening experience.

Furthermore, the DIALNORM parameter does not indicate the loudness processing state of the corresponding audio data (e.g., what type of loudness processing has been performed on the audio data). Prior to the present invention, audio bitstreams have not included metadata in a format of the type described in the present disclosure that indicates the loudness processing state of the audio content of an audio bitstream (e.g., the type of loudness processing applied), or the loudness processing state and loudness of the audio content of the bitstream. Loudness processing state metadata in such a format is used to facilitate verification of the effectiveness of adaptive loudness processing of an audio bitstream and/or the loudness processing state and loudness of the audio content in a particularly efficient manner.

While the present invention is not limited to use with AC-3 bitstreams, E-AC-3 bitstreams, or Dolby E bitstreams, for convenience it will be described in embodiments in which it generates, decodes, or processes such bitstreams including loudness processing state metadata.

The AC-3 encoded bitstream includes metadata and one to six channels of audio content. The audio content is audio data that has been compressed using perceptual audio coding. The metadata includes several audio metadata parameters intended to change the sound of the program delivered to the listening environment.

The details of AC-3 (also known as Dolby Digital) encoding are well known and are set forth in a number of published references, including:

ATSC Standard A52/A:Digital Audio Compression Standard(AC-3),RevisionA,Advanced Television Systems Committee,20Aug.2001; and

U.S. Patents 5,583,962, 5,632,005, 5,633,981, 5,727,119, and 6,021,386, all of which are hereby incorporated by reference in their entirety.

The details of Dolby Digital Plus (E-AC-3) encoding are explained in “Introduction to Dolby Digital Plus, an Enhancement to the Dolby Digital Coding System”, AES Convention Paper 6191, 117 ^th AES Convention, October 28, 2004.

Details of Dolby E coding are explained in “Efficient Bit Allocation, Quantization, and Coding in an Audio Distribution System”, AES Preprint 5068, 107th AES ^conference , August 1999, and “Professional Audio Coder Optimized for Use with Video”, AES Preprint 5033, ^107th AES Conference August 1999.

Each frame of the AC-3 encoded audio bitstream contains audio content and metadata for 1536 samples of digital audio. For a sampling rate of 48kHz, this represents 32 milliseconds of digital audio or a rate of 31.25 frames per second of audio.

Each frame of the E-AC-3 encoded audio bitstream contains 256, 512, 768, or 1536 samples of digital audio content and metadata, depending on whether the frame contains one, two, three, or six audio data blocks. For a sampling rate of 48kHz, this represents 5.333, 10.667, 16, or 32 milliseconds of digital audio, or audio rates of 189.9, 93.75, 62.5, or 31.25 frames per second, respectively.

As shown in Figure 4, each AC-3 frame is divided into segments (segments), including: a synchronization information (SI) segment, which contains (as shown in Figure 5) a synchronization word (SW) and the first of two error correction words (CRC1); a bit stream information (BSI) segment, which contains most of the metadata; one to six audio blocks (AB0 to AB5), which contain data-compressed audio content (which may also include metadata); a wasted bit segment (W), which contains any unused bits left after the audio content is compressed; an auxiliary (AUX) information segment, which may contain more metadata; and the second of two error correction words (CRC2). The wasted bit segment (W) may also be called a "skip field."

As shown in Figure 7, each E-AC-3 frame is divided into segments (segments), including: a synchronization information (SI) segment, which contains (as shown in Figure 5) a synchronization word (SW); a bit stream information (BSI) segment, which contains most of the metadata; between one and six audio blocks (AB0 to AB5), which contain data-compressed audio content (which may also include metadata); a wasted bit segment (W), which contains any unused bits left after the audio content is compressed (although only one wasted bit segment is shown, each audio block can usually be followed by a different wasted bit segment); an auxiliary (AUX) information segment, which can contain more metadata; and an error correction word (CRC). The wasted bit segment (W) can also be called a "skip field."

In an AC-3 (or E-AC-3) bitstream, there are several audio metadata parameters that are specifically intended for use in changing the sound of a program delivered to a listening environment. One of the metadata parameters is the DIALNORM parameter, which is included in the BSI section.

As shown in Figure 6, the BSI section of the AC-3 frame includes a five-bit parameter ("DIALNORM") that indicates the DIALNORM value used for the program. If the audio coding mode ("acmod") of the AC-3 frame is "0," indicating the use of a dual-single or "1+1" channel configuration, a five-bit parameter ("DIALNORM2") is included to indicate the DIALNORM value of the second audio program carried in the same AC-3 frame.

The BSI segment also includes: a flag ("addbsie") for indicating the presence (or absence) of additional bitstream information following the "addbsie" bit, a parameter ("addbsil") for indicating the length of any additional bitstream information following the "addbsil" value, and up to 64 bits of additional bitstream information ("addbsi") following the "addbsil" value.

The BSI segment includes other metadata values not specifically shown in FIG. 6 .

Summary of the Invention

In one embodiment, the present invention is an audio processing unit comprising a buffer memory, an audio decoder, and a parser. The buffer memory stores at least one frame of an encoded audio bitstream. The encoded audio bitstream includes audio data and a metadata container. The metadata container includes a header, one or more metadata payloads, and protection data. The header includes a synchronization word that identifies the beginning of the container. The one or more metadata payloads describe an audio program associated with the audio data. The protection data is located after the one or more metadata payloads. The protection data can also be used to verify the integrity of the metadata container and one or more payloads within the metadata container. The audio decoder is coupled to the buffer memory and is capable of decoding the audio data. The parser is coupled to or integrated with the audio decoder and is capable of parsing the metadata container.

In a typical embodiment, the method includes receiving an encoded audio bitstream, wherein the encoded audio bitstream is segmented into one or more frames. Audio data is extracted from the encoded audio bitstream along with a metadata container. The metadata container includes a header followed by one or more metadata payloads, which are followed by protection data. Finally, the integrity of the container and the one or more metadata payloads is verified using the protection data. The one or more metadata payloads may include a program loudness payload containing data indicating the measured loudness of an audio program associated with the audio data.

Program loudness metadata payloads embedded in audio bitstreams according to exemplary embodiments of the present invention, referred to as Loudness Processing State Metadata ("LPSM"), can be authenticated and verified, e.g., enabling a loudness adjustment entity to verify that the loudness of a particular program is within a specified range and that the corresponding audio data itself has not been modified (thereby ensuring compliance with applicable regulations). Instead of recalculating the loudness, the loudness values included in the data blocks containing the loudness processing state metadata can be read for verification. In response to the LPSM, a regulatory agency can determine whether the corresponding audio content complies with loudness regulations and/or regulatory requirements (e.g., regulations promulgated under the Commercial Advertisement Loudness Mitigation Act, also known as the "CALM" Act) (as indicated by the LPSM) without having to calculate the loudness of the audio content.

Compliance with some loudness regulations and/or regulatory requirements (e.g., those enacted under the CALM Act) requires that loudness measurements be based on overall program loudness. Overall program loudness requires that loudness measurements—either at the session level or at the full mix level—be made on the entire audio program. Therefore, in order for program loudness measurements (e.g., at various stages in the broadcast chain) to verify compliance with typical legal requirements, it is crucial that the measurements be derived using knowledge of what audio data (and metadata) defines the entire audio program, and this typically requires knowledge of the start and end positions of the program (e.g., during processing of the bitstream indicating the audio program sequence).

According to an exemplary embodiment of the present invention, an encoded audio bitstream indicates at least one audio program (e.g., an audio program sequence), and program boundary metadata and an LPSM included in the bitstream enable program loudness measurements to be reset at the end of a program, thereby providing an automated way to measure overall program loudness. Exemplary embodiments of the present invention include program boundary metadata in the encoded audio bitstream in an efficient manner that enables accurate and robust determination of at least one boundary between consecutive audio programs indicated by the bitstream. Exemplary embodiments enable accurate and robust determination of program boundaries, even in scenarios where bitstreams indicating different programs are spliced together (to generate the inventive bitstream) in a manner that enables truncating one or both of the spliced bitstreams (and thereby discarding program boundary metadata already included in at least one of the pre-spliced bitstreams).

In a typical embodiment, the program boundary metadata in a frame of the inventive bitstream is a program boundary marker indicating a frame count. Typically, the marker indicates the number of frames between the current frame (including the marked frame) and the program boundary (the start or end of the current audio program). In some preferred embodiments, the program boundary marker is inserted at the beginning and end of each bitstream segment indicating a single program (i.e., in a frame that occurs within some predetermined number of frames after the start of the segment, and in a frame that occurs within some predetermined number of frames before the end of the segment) in a symmetrical and efficient manner, so that when two such bitstream segments are concatenated (so as to indicate a sequence of two programs), the program boundary metadata can appear on both sides of the boundary between the two programs (i.e., symmetrically).

In order to limit the increase in data rate caused by including program boundary metadata in a coded audio bitstream (which may indicate an audio program or sequence of audio programs), in a typical embodiment, program boundary markers are only inserted in a subset of the frames of the bitstream. Typically, the boundary marker insertion rate is a non-increasing function of the increasing separation of each of the frames of the bitstream (in which the marker is inserted) from the program boundary nearest to each of those frames, where the "boundary marker insertion rate" represents the average ratio of the number of frames that include program boundary markers (frames indicating a program) to the number of frames that do not include program boundary markers (frames indicating a program), where the average is a sliding average over a number (e.g., a relatively small number) of consecutive frames of the coded audio bitstream. In one class of embodiments, the boundary marker insertion rate is a logarithmically decreasing function of increasing distance (at each marker insertion location) from the nearest program boundary, and for each marker-containing frame that includes one of the markers, the size of the marker in the marker-containing frame is equal to or greater than the size of each marker in frames that are closer to the program boundary than the marker-containing frame (i.e., the size of the program boundary marker in each marker-containing frame is a non-decreasing function of increasing separation of the marker-containing frame from the nearest program boundary).

Another aspect of the present invention is an audio processing unit (APU) configured to perform any embodiment of the method of the present invention. In another class of embodiments, the present invention is an APU that includes a buffer memory (buffer) that stores (e.g., in a non-transitory manner) at least one frame of an encoded audio bitstream that has been generated by any embodiment of the method of the present invention. Examples of APUs include, but are not limited to, encoders (e.g., transcoders), decoders, codecs, pre-processing systems (pre-processors), post-processing systems (post-processors), audio bitstream processing systems, and combinations of such elements.

In another class of embodiments, the present invention is an audio processing unit (APU) configured to generate an encoded audio bitstream comprising audio data segments and metadata segments, wherein the audio data segments represent audio data and at least some of the metadata segments each comprise loudness processing state metadata (LPSM) and, optionally, program boundary metadata. Typically, at least one such metadata segment in a frame of the bitstream comprises: at least one segment of the LPSM indicating whether a first type of loudness processing has been performed on the audio data of the frame (i.e., the audio data in at least one audio data segment of the frame); and at least one other segment of the LPSM indicating loudness of at least some of the audio data of the frame (e.g., a session loudness of at least some of the audio data of the frame representing a session). In one embodiment of this class, the APU is an encoder configured to encode input audio to generate encoded audio, the audio data segments comprising the encoded audio. In a typical embodiment of this class, each metadata segment has a preferred format as described herein.

In some embodiments, each metadata segment in a metadata segment of a coded bitstream (in some embodiments, an AC-3 bitstream or an E-AC-3 bitstream) that includes an LPSM (e.g., an LPSM and program boundary metadata) is included in a wasted bits segment (e.g., a wasted bits segment W of the type shown in FIG. 4 or 7 ) of a skip field segment of a frame of the bitstream. In other embodiments, each metadata segment in a metadata segment of a coded bitstream (in some embodiments, an AC-3 bitstream or an E-AC-3 bitstream) that includes an LPSM (e.g., an LPSM and program boundary metadata) is included as additional bitstream information in an “addbsi” field of a bitstream information (“BSI”) segment of a frame of the bitstream or in an auxiliary data field (e.g., an AUX segment of the type shown in FIG. 4 or 7 ) at the end of a frame of the bitstream. Each metadata segment including an LPSM has a format specified herein with reference to Tables 1 and 2 below (i.e., it includes the core elements shown in Table 1 or a variation thereof, the core elements or a variation thereof followed by a payload ID (identifying the metadata as an LPSM) and a payload size value, the payload ID and payload size value followed by a payload (LPSM data having the format shown in Table 2 or a variation thereof as described herein)). In some embodiments, a frame may include one or two metadata segments, each of the one or two metadata segments including an LPSM, and if a frame includes two metadata segments, one metadata segment may appear in the addbsi field of the frame and the other metadata segment may appear in the AUX field of the frame.

In one class of embodiments, the present invention is a method comprising encoding audio data to generate an AC-3 or E-AC-3 encoded audio bitstream, said method being implemented by including, in a metadata segment (of at least one frame of the bitstream), LPSM and program boundary metadata, and optionally other metadata for the audio program to which the frame belongs. In some embodiments, each such metadata segment is included in the addbsi field of a frame or the auxdata field of a frame. In other embodiments, each such metadata segment is included in a wasted bits segment of a frame. In some embodiments, each metadata segment containing the LPSM and program boundary metadata comprises a core header (and optionally additional core elements), and, following the core header (or the core header and other core elements), an LPSM payload (or container) segment having the following format:

header, which typically includes at least one identification value (e.g., LPSM format version, length, period, count, and substream association value, as shown in Table 2 proposed herein), and

Following the header is the LPSM and program boundary metadata. The program boundary metadata may include a program boundary frame count, an encoded value, and (in some cases) an offset value. The encoded value (e.g., an "offset_exist" value) indicates whether the frame contains only a program boundary frame count or both a program boundary frame count and an offset value. The LPSM may include:

At least one session indication value indicating whether the corresponding audio data indicates a session or does not indicate a session (e.g., which channels of the corresponding audio data indicate a session). The session indication value may indicate whether a session exists in any combination of channels or in all channels of the corresponding audio data;

at least one loudness adjustment compliance value indicating whether the corresponding audio data complies with the indicated loudness rule set;

at least one loudness processing value indicating at least one type of loudness processing that has been performed on the corresponding audio data; and

At least one loudness value indicating at least one loudness (eg, peak loudness or average loudness) characterizing corresponding audio data.

In other embodiments, the coded bitstream is a bitstream that is not an AC-3 bitstream or an E-AC-3 bitstream, and each of the metadata segments including the LPSM (and optionally also program boundary metadata) is included in a segment (or field, or time slot) of the bitstream reserved for storing additional data. Each metadata segment including the LPSM can have a format similar to or the same as the format indicated below with reference to Tables 1 and 2 herein (i.e., it includes core elements similar to or the same as those shown in Table 1, followed by a payload ID (identifying the metadata as an LPSM) and a payload size value, followed by a payload (LPSM data having a format similar to or the same as the format shown in Table 2 or a variation of Table 2 as described herein)).

In some embodiments, a coded bitstream includes a sequence of frames, each frame including a bitstream information ("BSI") segment and an auxdata field or time slot (e.g., the coded bitstream is an AC-3 bitstream or an E-AC-3 bitstream), wherein the bitstream information ("BSI") segment includes an "addbsi" field (sometimes referred to as a segment or time slot), and each frame includes an audio data segment (e.g., segments AB0-AB5 of the frame shown in FIG. 4 ) and a metadata segment, wherein the metadata segment represents the audio data, and at least some of the metadata segments each include loudness processing state metadata (LPSM) and optionally program boundary metadata. The LPSM is present in the bitstream in the following format. Each metadata segment including the LPSM is included in the "addbsi" field of a BSI segment of a frame of the bitstream, or in the auxdata field of a frame of the bitstream, or in a wasted bits segment of a frame of the bitstream. Each metadata segment including the LPSM includes an LPSM payload (or container) segment having the following format:

header (usually including at least one identification value, such as LPSM format version, length, period, count, and substream association value as shown in Table 2 below); and

Following the header is the LPSM and, optionally, program boundary metadata. The program boundary metadata may include a program boundary frame count, an encoded value, and (in some cases) an offset value. The encoded value (e.g., an "offset_exist" value) indicates whether the frame contains only a program boundary frame count or both a program boundary frame count and an offset value. The LPSM may include:

At least one session indication value (e.g., parameter "session channel" of Table 2), which indicates whether the corresponding audio data indicates a session or does not indicate a session (e.g., which channel of the corresponding audio data indicates a session). The session indication value may indicate whether a session exists in any combination of channels or all channels of the corresponding audio data;

At least one loudness adjustment compliance value (e.g., parameter "Loudness Adjustment Type" of Table 2), which indicates whether the corresponding audio data complies with the indicated set of loudness adjustments;

At least one loudness processing value (e.g., one or more of the parameters "Session-gated Loudness Correction Flag" and "Loudness Correction Type" of Table 2) indicating at least one type of loudness processing that has been performed on the corresponding audio data; and

At least one loudness value (e.g., one or more of the parameters "ITU Relative Gated Loudness," "ITU Speech Gated Loudness," "ITU (EBU 3341) Short-Term 3s Loudness," and "True Peak" of Table 2) indicating at least one loudness (e.g., peak or average loudness) characteristic of corresponding audio data.

In any embodiment of the present invention that focuses on, uses, or generates at least one loudness value representing corresponding audio data, the loudness value may be indicative of at least one loudness measurement characteristic used to process the loudness and/or dynamic range of the audio data.

In some implementations, each metadata segment in the "addbsi" field or auxdata field or wasted bits segment of a frame of a bitstream has the following format:

Core header (typically includes a synchronization word identifying the start of the metadata segment, followed by identification values, such as the core element version, length and period, extension element count, and substream association value as shown in Table 1 below); and

At least one protection value following the core header (e.g., an HMAC digest and an audio fingerprint value, wherein the HMAC digest may be a 256-bit HMAC digest (using the SHA-2 algorithm) calculated based on the audio data of the entire frame, the core element, and all extended elements, as shown in Table 1, and used for at least one of decryption, authentication, or verification of at least one of the loudness processing state metadata or the corresponding audio data); and

If the metadata segment includes an LPSM payload, an LPSM payload identification (ID) and an LPSM payload size value also follow the core header, identifying the following metadata as an LPSM payload and indicating the size of the LPSM payload. An LPSM payload segment (preferably having the format described above) follows the LPSM payload ID and LPSM payload size value.

In some embodiments of the type described in the preceding paragraphs, each metadata segment in the auxdata field (or "addbsi" field or wasted bits segment) of a frame has a three-layer structure:

A high-level structure that includes: a flag indicating whether the auxdata (or addbsi) field contains metadata; at least one ID value indicating what type of metadata is present; and typically also a value indicating how many bits of metadata (e.g., of each type of metadata) are present, if metadata is present. One type of metadata that may be present is LPSM, another type of metadata that may be present is program boundary metadata, and another type of metadata that may be present is media studies metadata.

a middle-level structure including core elements of metadata for each identified type (e.g., metadata for each identified type such as a core header of the type described above, a protection value, and a payload ID and payload size value); and

A low-level structure, including each payload for a core element (e.g., an LPSM payload if the core element identifies it as present, and/or another type of metadata payload if the core element identifies it as present).

Data values in such a three-layer structure can be nested. For example, protection values for an LPSM payload and/or another metadata payload identified by a core element can be included after a payload identified by a core element (and thus after a core header of the core element). In one example, the core header can identify the LPSM payload and the other metadata payload, a payload ID and payload size value for a first payload (e.g., an LPSM payload) can follow the core header, the first payload itself can follow the ID and size values, a payload ID and payload size value for a second payload can follow the first payload, the second payload itself can follow these ID and size values, and a protection value for one or both of the two payloads (or the core element value and one or both of the two payloads) can follow the final payload.

In some embodiments, a core element of a metadata segment in an auxdata field (or "addbsi" field or wasted bit segment) of a frame includes a core header (typically including an identification value, such as a core element version), and after the core header includes: a value indicating whether fingerprint data is included for the metadata of the metadata segment, a value indicating whether external data (related to the audio data corresponding to the metadata of the metadata segment) is present, a payload ID and payload size value for each type of metadata identified by the core element (e.g., LPSM and/or metadata of a type other than LPSM), and a protection value for at least one type of metadata identified by the core element. The metadata payload of the metadata segment follows the core header and (in some cases) is nested within the value of the core element.

In another preferred format, the encoded bitstream is a Dolby E bitstream, and each of the metadata segments including the LPSM (and optionally also program boundary metadata) is included in the first N sample positions of the Dolby E guard band interval.

In another type of embodiment, the present invention is an APU (e.g., a decoder) coupled and configured to receive an encoded audio bitstream comprising audio data segments and metadata segments, wherein the audio data segments represent audio data and at least some of the metadata segments each comprise loudness processing state metadata (LPSM) and optionally program boundary metadata, and the APU is coupled and configured to extract the LPSM from the bitstream, generate decoded audio data in response to the audio data, and perform at least one adaptive loudness processing operation on the audio data using the LPSM. Some embodiments of this type further include a post-processor coupled to the APU, wherein the post-processor is coupled and configured to perform at least one adaptive loudness processing operation on the audio data using the LPSM.

In another type of embodiment, the present invention is an audio processing unit (APU) comprising a buffer memory (buffer) and a processing subsystem coupled to the buffer. The APU is coupled to receive an encoded audio bitstream comprising audio data segments and metadata segments, wherein the audio data segments represent audio data and at least some of the metadata segments each comprise loudness processing state metadata (LPSM) and optionally program boundary metadata. The buffer stores at least one frame of the encoded audio bitstream (e.g., in a non-transitory manner), and the processing subsystem is configured to extract the LPSM from the bitstream and perform at least one adaptive loudness processing operation on the audio data using the LPSM. In a typical embodiment of this type, the APU is one of an encoder, a decoder, and a post-processor.

In some implementations of the method of the present invention, the generated audio bitstream is one of an AC-3 encoded bitstream, an E-AC-3 bitstream, or a Dolby E bitstream, including loudness processing state metadata and other metadata (e.g., DIALNORM metadata parameters, dynamic range control metadata parameters, and other metadata parameters). In some other implementations of the method, the generated audio bitstream is another type of encoded bitstream.

Various aspects of the present invention include systems or devices configured (or programmed) to perform any embodiment of the method of the present invention, and computer-readable media (e.g., disks) storing (e.g., in a non-transitory manner) code for implementing any embodiment of the method of the present invention or its steps. For example, the system of the present invention can be or include a programmable general-purpose processor, digital signal processor, or microprocessor programmed with software or firmware, and/or configured to perform any of a variety of operations on data, including embodiments of the method of the present invention or its steps. Such a general-purpose processor can be or include a computer system that includes an input device, memory, and processing circuitry that is programmed (and/or configured) to perform embodiments of the method (or steps) of the present invention in response to data transmitted thereto.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a block diagram of an embodiment of a system that may be configured to perform an embodiment of the method of the present invention;

FIG2 is a block diagram of an encoder as an embodiment of an audio processing unit of the present invention;

3 is a block diagram of a decoder as an embodiment of an audio processing unit of the present invention and a post-processor coupled to the decoder as another embodiment of the audio processing unit of the present invention;

FIG4 is a diagram of an AC-3 frame, including the segments into which it is divided;

FIG5 is a diagram of a synchronization information (SI) segment of an AC-3 frame, including the segments into which it is divided;

FIG6 is a diagram of the bit stream information (BSI) segment of an AC-3 frame, including the segments into which it is divided;

FIG7 is a diagram of an E-AC-3 frame, including the segments into which it is divided;

FIG8 is a diagram of a frame of an encoded audio bitstream including program boundary metadata having a format according to an embodiment of the present invention;

9 is a diagram of other frames of the encoded audio bitstream of FIG. 9 , some of which include program boundary metadata having a format according to an embodiment of the present invention;

FIG10 is a diagram of two coded audio bitstreams: a bitstream (IEB) in which a program boundary (labeled “Boundary”) is aligned with a transition between two frames of the bitstream, and another bitstream (TB) in which a program boundary (labeled “TrueBoundary”) is offset by 512 samples from the transition between two boundaries of the bitstream; and

FIG11 is a set of graphics illustrating four encoded audio bitstreams. The bitstream at the top of FIG11 (labeled “Scene 1”) indicates a first audio program (P1) including program boundary metadata, followed by a second audio program (P2) also including program boundary metadata; a second bitstream (labeled “Scene 2”) indicates the first audio program (P1) including program boundary metadata, followed by a second audio program (P2) that does not include program boundary metadata; a third bitstream (labeled “Scene 3”) indicates a truncated first audio program (P1) including program boundary metadata, which has been spliced with an entire second audio program (P2) including program boundary metadata; and a fourth bitstream (labeled “Scene 4”) indicates a truncated first audio program (P1) including program boundary metadata and a truncated second audio program (P2) that includes program boundary metadata and has been spliced with a portion of the first audio program.

Symbols and naming

Throughout this disclosure, including in the claims, statements that refer to “performing an operation on” a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) are used broadly to mean performing the operation directly on the signal or data, or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to refer to an apparatus, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to a plurality of inputs, where the subsystem generates M inputs and the other X-M inputs are received from external sources) may also be referred to as a decoder system.

Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (using software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and/or otherwise configured to perform pipeline processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

Throughout this disclosure, including in the claims, the expressions "audio processor" and "audio processing unit" are used interchangeably and are used in a broad sense to refer to a system configured to process audio data. Examples of audio processing units include, but are not limited to, encoders (e.g., transcoders), decoders, codecs, pre-processing systems, post-processing systems, and bitstream processing systems (sometimes referred to as bitstream processing tools).

Throughout this disclosure, including in the claims, the term "processing state metadata" (e.g., in the term "loudness processing state metadata") refers to data that is separate and distinct from corresponding audio data (and the audio content of an audio data stream, including the processing state metadata). Processing state metadata is associated with audio data and indicates the loudness processing state of the corresponding audio data (e.g., what type of processing has been performed on the audio data) and typically also indicates at least one feature or characteristic of the audio data. The association of processing state metadata with the audio data is time-synchronous. Thus, current (most recently received or updated) processing state metadata indicates the corresponding audio data and also includes the results of the indicated type of processing on the audio data. In some cases, the processing state metadata may include some or all of the processing history and/or parameters used in and/or derived from the indicated type of processing. Additionally, the processing state metadata may include at least one feature or characteristic of the corresponding audio data that has been calculated from or extracted from the audio data. Processing state metadata may also include other metadata that is not related to or derived from any processing of the corresponding audio data. For example, third-party data, tracking information, identifiers, proprietary or standard information, user annotation data, user preference data, etc. can be added by a specific audio processing unit to be passed to other audio processing units.

Throughout this disclosure, including in the claims, the expression "loudness processing state metadata" (or "LPSM") refers to processing state metadata that indicates the loudness processing state of corresponding audio data (e.g., what type of loudness processing has been performed on the audio data) and typically also indicates at least one feature or characteristic of the corresponding audio data (e.g., loudness). Loudness processing state metadata may include data (e.g., other metadata) that is not (i.e., when considered alone) loudness processing state metadata.

Throughout this disclosure, including in the claims, the expression "channel" (or "audio channel") denotes a monophonic audio signal.

Throughout this disclosure, including in the claims, the expression "audio program" refers to a set of one or more audio channels and optionally also to associated metadata (e.g. metadata describing the expected spatial audio presence, and/or LPSM, and/or program boundary metadata).

Throughout this disclosure, including in the claims, the expression "program boundary metadata" refers to metadata of a coded audio bitstream, wherein the coded audio bitstream indicates at least one audio program (e.g., two or more audio programs), and the program boundary metadata indicates the position in the bitstream of at least one boundary (start and/or end) of at least one of the audio programs. For example, the program boundary metadata (of the coded audio bitstream indicating the audio program) may include metadata indicating the position of the start of the program (e.g., the start of the Nth frame of the bitstream, or the Mth sample position of the Nth frame of the bitstream), and additional metadata indicating the position of the end of the program (e.g., the start of the Jth frame of the bitstream, or the Kth sample position of the Jth frame of the bitstream).

Throughout this disclosure, including in the claims, the terms "couple" or "coupled" are used to indicate either a direct connection or an indirect connection. Thus, if a first device couples to a second device, that connection may be a direct connection or an indirect connection via other devices and connections.

DETAILED DESCRIPTION

According to an exemplary embodiment of the present invention, a payload of program loudness metadata, referred to as loudness processing state metadata (LPSM), and optionally program boundary metadata, is embedded in one or more reserved fields (or time slots) of a metadata segment of an audio bitstream that also includes audio data in other segments (audio data segments). Typically, at least one segment of each frame of the bitstream includes the LPSM, and at least one other segment of the frame includes corresponding audio data (i.e., audio data whose loudness processing state and loudness are indicated by the LPSM). In some embodiments, the amount of LPSM data can be sufficiently small to be carried without affecting the bit rate allocated for carrying the audio data.

Transmitting loudness processing state metadata in an audio data processing chain is particularly useful when two or more audio processing units need to work in tandem with each other throughout the processing chain (or content lifecycle). If loudness processing state metadata is not included in the audio bitstream, for example, when two or more audio codecs are used in the chain and single-ended volume adjustment is applied more than once during the bitstream's journey to a media consumption device (or the rendering point of the bitstream's audio content), serious media processing issues such as degradation in quality, level, and spatial quality can arise.

FIG1 is a block diagram of an exemplary audio processing chain (audio data processing system), wherein one or more of the elements of the system may be configured according to embodiments of the present invention. The system includes the following elements coupled together as shown: a preprocessing unit, an encoder, a signal analysis and metadata correction unit, a transcoder, a decoder, and the preprocessing unit. In variations of the illustrated system, one or more of these elements may be omitted, or additional audio data processing units may be included.

In some implementations, the pre-processing unit of FIG. 1 is configured to accept as input PCM (time-domain) samples comprising audio content and output processed PCM samples. The encoder can be configured to accept as input PCM samples and output an encoded bitstream (e.g., compressed) representing the audio content. The data representing the bitstream of audio content is sometimes referred to herein as "audio data." If the encoder is configured according to typical embodiments of the present invention, the audio bitstream output from the encoder includes loudness processing state metadata (and typically other metadata, optionally including program boundary metadata) as well as the audio data.

The signal analysis and metadata correction unit of FIG1 can accept one or more coded audio bitstreams as input and determine (e.g., verify) whether the processing state metadata in each coded audio bitstream is correct by performing signal analysis (e.g., using program boundary metadata in the coded audio bitstream). If the signal analysis and metadata correction unit finds that the included metadata is invalid, it typically replaces the erroneous value with the correct value obtained based on the signal analysis. Thus, each coded audio bitstream output from the signal analysis and metadata correction unit can include corrected (or uncorrected) processing state metadata as well as the coded bitstream audio data.

The transcoder of FIG1 can accept an encoded audio bitstream as input and output a modified (e.g., differently encoded) audio bitstream accordingly (e.g., by decoding the input stream and re-encoding the decoded stream in a different encoding format). If the transcoder is configured according to an exemplary embodiment of the present invention, the audio bitstream output from the transcoder includes loudness processing state metadata (and typically other metadata) as well as the encoded bitstream audio data. The metadata is already included in the bitstream.

The decoder of Figure 1 can accept an encoded (e.g., compressed) audio bitstream as input and (correspondingly) output a stream of decoded PCM audio samples. If the decoder is configured according to an exemplary embodiment of the present invention, the output of the decoder in typical operation is or includes any of the following:

a stream of audio samples and a corresponding stream of loudness processing state metadata (and typically other metadata) extracted from the input coded bitstream; or

a stream of audio samples and corresponding control bits determined according to loudness processing state metadata (and typically other metadata) extracted from the input coded bitstream; or

A stream of audio samples without a corresponding stream of processing state metadata or control bits determined based on the processing state metadata. In this last case, the decoder can extract loudness processing state metadata (and/or other metadata) from the input coded bitstream and perform at least one operation (e.g., validation) on the extracted metadata, even though it does not output the extracted metadata or control bits determined based thereon.

By configuring the post-processing unit of FIG. 1 according to an exemplary embodiment of the present invention, the post-processing unit is configured to accept a stream of decoded PCM audio samples and perform post-processing (i.e., volume adjustment of the audio content) thereon using loudness processing state metadata (and typically other metadata) received with the samples or control bits received with the samples (which are determined by the decoder based on the loudness processing state metadata and typically other metadata). The post-processing unit is also typically configured to render the post-processed audio content for playback by one or more speakers.

Typical embodiments of the present invention provide an enhanced audio processing chain in which audio processing units (e.g., encoder, decoder, transcoder, pre-processing unit, and post-processing unit) adapt their respective processing to be applied to audio data according to a contemporaneous state of metadata indicated by loudness processing state metadata respectively received by the audio processing units.

Audio data input to any audio processing unit of the system of FIG. 1 (e.g., the encoder or transcoder of FIG. 1 ) may include loudness processing state metadata (and optionally other metadata) along with audio data (e.g., encoded audio data). According to embodiments of the present invention, this metadata may already be included in the input audio by another element of the system of FIG. 1 (or another source not shown in FIG. 1 ). A processing unit that receives the input audio (with metadata) may be configured to perform at least one operation on the metadata (e.g., validation) or to perform at least one operation in response to the metadata (e.g., adaptive processing of the input audio), and is typically also configured to include the metadata, a processed version of the metadata, or control bits determined based on the metadata in its output audio.

Typical embodiments of an audio processing unit (or audio processor) of the present invention are configured to perform adaptive processing on audio data based on the state of the audio data indicated by loudness processing state metadata corresponding to the audio data. In some embodiments, the adaptive processing is (or includes) loudness processing (if the metadata indicates that loudness processing or similar processing has not yet been performed on the audio data), and is not (or does not include) loudness processing (if the metadata indicates that such loudness processing or similar processing has already been performed on the audio data). In some embodiments, the adaptive processing is or includes metadata verification (e.g., metadata verification performed in a metadata verification subunit) to ensure that the audio processing unit performs other adaptive processing on the audio data based on the state of the audio data indicated by the loudness processing state metadata. In some embodiments, the verification determines the reliability of the loudness processing state metadata associated with the audio data (e.g., included in a bitstream containing the audio data). For example, if the metadata is verified to be reliable, results from previously performed audio processing of a certain type can be reused, and new execution of the same type of audio processing can be avoided. On the other hand, if the metadata is found to have been tampered with (or is unreliable), the type of media processing purportedly performed previously may be repeated by the audio processing unit (as indicated by the unreliable metadata), and/or other processing may be performed by the audio processing unit on the metadata and/or audio data. The audio processing unit may also be configured to indicate to other audio processing units downstream in the enhanced media processing chain that the loudness processing state metadata (e.g., loudness processing state metadata present in the media bitstream) is valid if the audio processing unit determines that the processing state metadata is valid (e.g., based on a match between the extracted cryptographic value and the reference cryptographic value).

FIG2 is a block diagram of an encoder (100) as an embodiment of an audio processing unit of the present invention. Any component or element of the encoder 100 can be implemented as one or more processes and/or one or more circuits (e.g., ASICs, FPGAs, or other integrated circuits) using hardware, software, or a combination of hardware and software. The encoder 100 includes a frame buffer 110, a parser 111, a decoder 101, an audio state validator 102, a loudness processing stage 103, an audio stream selection stage 104, an encoder 105, a filler/formatter stage 107, a metadata generation stage 106, a session loudness measurement subsystem 108, and a frame buffer 109, connected as shown. Typically, the encoder 100 also includes other processing elements (not shown).

The encoder 100 (acting as a transcoder) is configured to convert an input audio bitstream (e.g., one of an AC-3 bitstream, an E-AC-3 bitstream, or a Dolby E bitstream) into an encoded output audio bitstream (e.g., another of an AC-3 bitstream, an E-AC-3 bitstream, or a Dolby E bitstream) including loudness processing state metadata by performing adaptive and automatic loudness processing using the loudness processing state metadata included in the input bitstream. For example, the encoder 100 can be configured to convert an input Dolby E bitstream (a format typically used in production and broadcast facilities rather than in consumer devices that receive audio programs that have been broadcast to them) into an encoded output audio bitstream in AC-3 or E-AC-3 format (suitable for broadcast to user devices).

The system of FIG2 also includes an encoded audio delivery subsystem 150 (which stores and/or delivers the encoded bitstream output from the encoder 100) and a decoder 152. The encoded audio bitstream output from the encoder 100 may be stored by the subsystem 150 (e.g., in the form of a DVD or Blu-ray disc), or transmitted by the subsystem 150 (which may implement a transmission link or network), or may be stored and transmitted by the subsystem 150. The decoder 152 is configured to decode the encoded audio bitstream (generated by the encoder 100) received via the subsystem 150, including loudness processing state metadata, by extracting loudness processing state metadata (LPSM) from each frame of the bitstream (and optionally also extracting program boundary metadata from the bitstream) and generating decoded audio data. Typically, decoder 152 is configured to perform adaptive loudness processing on the decoded audio data using the LPSM (and optionally also using program boundary metadata), and/or forward the decoded audio data and the LPSM to a post-processor that is configured to perform adaptive loudness processing on the decoded audio data using the LPSM (and optionally also using program boundary metadata). Typically, decoder 152 includes a buffer for storing (e.g., in a non-transitory manner) the encoded audio bitstream received from subsystem 150.

Various implementations of the encoder 100 and decoder 152 may be configured to perform different embodiments of the method of the present invention.

The frame buffer 110 is a buffer memory coupled to receive an encoded input audio bitstream. In operation, the buffer 110 stores at least one frame of the encoded audio bitstream (e.g., in a non-transitory manner), and a sequence of frames of the encoded audio bitstream is transmitted (asserted) from the buffer 110 to the parser 111.

The parser 111 is coupled and configured to extract loudness processing state metadata (LPSM) from each frame of the encoded input audio in which such metadata is included, and optionally also extract program boundary metadata (and/or other metadata) from each frame of the encoded input audio in which such metadata is included, to deliver at least the LPSM (and optionally also program boundary metadata and/or other metadata) to the audio state validator 102, the loudness processing stage 103, the stage 106, and the subsystem 108 to extract audio data from the encoded input audio, and to deliver the audio data to the decoder 101. The decoder 101 of the encoder 100 is configured to decode the audio data to generate decoded audio data, and to deliver the decoded audio data to the loudness processing stage 103, the audio stream selection stage 104, the subsystem 108, and typically also to the state validator 102.

The state verifier 102 is configured to authenticate and verify the LPSM (and typically other metadata) that is passed to it. In some embodiments, the LPSM is a data block that has been included in the input bitstream (e.g., according to an embodiment of the present invention) (or is included in a data block that has been included in the input bitstream). The block may include a cryptographic hash (hash-based message authentication code or "HMAC") for processing the LPSM (and optionally other metadata) and/or the underlying audio data (provided to the verifier 102 from the decoder 101). The data block may be digitally signed in these embodiments so that downstream audio processing units can relatively easily authenticate and verify the processing state metadata.

For example, HMAC is used to generate a digest, and the protection value included in the bitstream of the present invention may include the digest. The digest may be generated for an AC-3 frame as follows:

1. After the AC-3 data and LPSM are encoded, the frame data bytes (concatenated frame_data#1 and frame_data#2) and the LPSM data bytes are used as input to the HMAC hash function. Other data that may be present in the auxdata field is not considered when computing the digest. Such other data may be bytes that are neither AC-3 data nor LSPSM data. The protection bits included in the LPSM may not be considered when computing the HMAC digest.

2. After the digest is calculated, it is written to the bitstream in a field reserved for protection bits.

3. The final step in the generation of a complete AC-3 frame is to calculate the CRC checksum, which is written at the very end of the frame and takes into account all the data belonging to the frame, including the LPSM bits.

Other cryptographic algorithms, including but not limited to any of one or more non-HMAC cryptographic methods, may be used for verification of the LPSM (e.g., in the verifier 102) to ensure secure transmission and reception of the LPSM and/or underlying audio data. For example, verification (using such cryptographic methods) may be performed in each audio processing unit of an embodiment of the present invention that receives an audio bitstream to determine whether loudness processing state metadata and corresponding audio data included in the bitstream have undergone (and/or have been derived from) a specific loudness processing (indicated by the metadata) and have not been modified since such specific loudness processing was performed.

The state validator 102 transmits control data to the audio stream selection stage 104, the metadata generator 106, and the session loudness measurement subsystem 108 to indicate the result of the validation operation. In response to the control data, the stage 104 may select (and transmit to the encoder 105) any of the following:

the adaptively processed output of the loudness processing stage 103 (e.g., when the LPSM indicates that the audio data output from the decoder 101 has not been subjected to a particular type of loudness processing, and the control bit from the validator 102 indicates that the LPSM is valid); or

Audio data output from decoder 101 (e.g., when the LPSM indicates that the audio data output from decoder 101 has been subjected to a particular type of loudness processing that may be performed by stage 103, and the control bit from validator 102 indicates that the LPSM is valid).

The stage 103 of the encoder 100 is configured to perform adaptive loudness processing on the decoded audio data output from the decoder 101 based on one or more audio data characteristics indicated by the LPSM extracted by the decoder 101. The stage 103 may be an adaptive transform-domain real-time loudness and dynamic range control processor. The stage 103 may receive user input (e.g., a user target loudness/dynamic range value or a dialnorm value), or other metadata input (e.g., one or more types of third-party data, musical information, identifiers, proprietary or standard information, user annotation data, user preference data, etc.), and/or other input (e.g., other input from fingerprint processing), and use such input to process the decoded audio data output from the decoder 101. Stage 103 may perform adaptive loudness processing on decoded audio data (output from decoder 101) indicating a single audio program (as indicated by program boundary metadata extracted by parser 111), and may reset the loudness processing in response to receiving decoded audio data (output from decoder 101) indicating a different audio program as indicated by program boundary metadata extracted by parser 111.

When the control bit from validator 102 indicates that the LPSM is invalid, session loudness measurement subsystem 108 may operate to determine the loudness of a segment of decoded audio (from decoder 101) indicative of conversation (or other speech) using the LPSM (and/or other metadata) as extracted by decoder 101. When the control bit from validator 102 indicates that the LPSM is valid, operation of session loudness measurement subsystem 108 may be disabled when the LPSM indicates the loudness of a previously determined segment of conversation (or other speech) of decoded audio (from decoder 101). Subsystem 108 may perform loudness measurement on decoded audio data indicative of a single audio program (as indicated by program boundary metadata extracted by parser 111), and may reset the measurement in response to receiving decoded audio data indicative of a different audio program as indicated by such program boundary metadata.

There are useful tools (e.g., the Dolby LM100 loudness meter) for conveniently and easily measuring the level of conversation in audio content. Some embodiments of the APU of the present invention (e.g., stage 108 of encoder 100) are implemented to include such a tool (or perform the functionality of such a tool) to measure the average conversation loudness of the audio content of an audio bitstream (e.g., a decoded AC-3 bitstream passed from decoder 101 of encoder 100 to stage 108).

If stage 108 is implemented to measure the true average conversational loudness of the audio data, the measurement may include the steps of isolating segments of the audio content that primarily contain speech. The primarily speech audio segments are then processed according to a loudness measurement algorithm. For audio data decoded from an AC-3 bitstream, the algorithm may be a standard K-weighted loudness measurement (according to the international standard ITU-R BS.1770). Alternatively, other loudness measurements (e.g., those based on psychoacoustic models of loudness) may be used.

Isolation of speech segments is not crucial for measuring the average conversational loudness of audio data. However, it improves the accuracy of the measurement and generally provides more satisfactory results from the listener's perspective. Since not all audio content contains conversation (speech), loudness measurement of the entire audio content can provide a sufficient approximation of the audio's conversational level, if speech is present.

The metadata generator 106 generates (and/or passes to the stage 107) metadata to be included by the stage 107 in the encoded bitstream for output from the encoder 100. The metadata generator 106 may pass the LPSM (and optionally also program boundary metadata and/or other metadata) extracted by the decoder 101 and/or the parser 111 to the stage 107 (e.g., when the control bit from the validator 102 indicates that the LPSM and/or other metadata are valid), or generate a new LPSM (and optionally also program boundary metadata and/or other metadata) and pass the new metadata to the stage 107 (e.g., when the control bit from the validator 102 indicates that the LPSM and/or other metadata extracted by the decoder 101 are invalid), or it may pass to the stage 107 a combination of the metadata extracted by the decoder 101 and/or the parser 111 and the newly generated metadata. Metadata generator 106 may include the loudness data generated by subsystem 108 and at least one value representing the type of loudness processing performed by subsystem 108 in the LPSM it transmits to stage 107 for inclusion in the encoded bitstream to be output from encoder 100 .

The metadata generator 106 may generate guard bits (which may include or consist of a hash-based message authentication code or "HMAC") that are useful for at least one of decryption, authentication, or verification of the LPSM (and optionally other metadata) and/or underlying audio data to be included in the coded bitstream. The metadata generator 106 may provide such guard bits to the stage 107 for inclusion in the coded bitstream.

In typical operation, the session loudness measurement subsystem 108 processes the audio data output from the decoder 101 to generate loudness values (e.g., gated and ungated session loudness values) and dynamic range values in response thereto. In response to these values, the metadata generator 106 may generate loudness processing state metadata (LPSM) for inclusion (by the stuffer/formatter 107) in the encoded bitstream to be output from the encoder 100.

Additionally, optionally, or alternatively, subsystems 106 and/or 108 of encoder 100 may perform additional analysis of the audio data to generate metadata representative of at least one characteristic of the audio data for inclusion in the encoded bitstream to be output from stage 107 .

Encoder 105 encodes the audio data output from selection stage 104 (eg, by performing compression thereon) and transmits the encoded audio to stage 107 for inclusion in an encoded bitstream to be output from stage 107 .

Stage 107 multiplexes the encoded audio from encoder 105 and metadata (including LPSM) from generator 106 to generate an encoded bitstream to be output from stage 107, preferably such that the encoded bitstream has the format specified by the preferred embodiment of the invention.

The frame buffer 109 is a buffer memory that stores (e.g., in a non-temporary manner) at least one frame of the encoded audio bitstream output from stage 107, and then the sequence of frames of the encoded audio bitstream is transmitted from the buffer 109 to the delivery system 150 as output from the encoder 100.

The LPSM generated by metadata generator 106 and included in the encoded bitstream by stage 107 represents the loudness processing state of the corresponding audio data (e.g., what type of loudness processing has been performed on the audio data) and the loudness of the corresponding audio data (e.g., measured session loudness, gated and/or ungated loudness, and/or dynamic range).

As used herein, "gating" a loudness and/or level measurement performed on audio data refers to a specific level or loudness threshold under which calculated values exceeding the threshold are included in the final measurement (e.g., short-term loudness values below -60 dBFS are ignored in the final measurement). Absolute gating refers to a fixed level or loudness, while relative gating refers to a value that depends on the current "ungated" measurement value.

In some implementations of encoder 100, the encoded bitstream buffered in memory 109 (and output to delivery system 150) is an AC-3 bitstream or an E-AC-3 bitstream and includes audio data segments (e.g., segments AB0 to AB5 of a frame shown in FIG. 4 ) and metadata segments, wherein the audio data segments represent audio data and at least some of the metadata segments each include loudness processing state metadata (LPSM). Stage 107 inserts the LPSM (and optionally also program boundary metadata) into the bitstream in the following format. Each metadata segment including the LPSM (and optionally also program boundary metadata) is included in a wasted bits segment of the bitstream (e.g., the wasted bits segment "W" shown in FIG. 4 or FIG. 7 ), in the "addbsi" field of a bitstream information ("BSI") segment of a frame of the bitstream, or in the auxdata field at the end of a frame of the bitstream (e.g., the AUX segment shown in FIG. 4 or FIG. 7 ). A frame of a bitstream may include one or two metadata segments, each of which includes an LPSM, and if a frame includes two metadata segments, one of the metadata segments may be present in the addbsi field of the frame and the other in the AUX field of the frame. In some embodiments, each metadata segment including an LPSM includes an LPSM payload (or container) segment having the following format:

A header (typically comprising a synchronization word identifying the start of the LPSM payload, followed by at least one identification value, such as the LPSM format version, length, period, count, and substream association value as shown in Table 2 below); and

After the first part,

At least one session indication value (e.g., parameter “session channel” of Table 2), which indicates whether the corresponding audio data indicates a session or does not indicate a session (e.g., which channel of the corresponding audio data indicates a session);

At least one loudness adjustment compliance value (e.g., parameter "Loudness Adjustment Type" of Table 2), which indicates whether the corresponding audio data complies with the indicated set of loudness adjustments;

At least one loudness processing value (e.g., one or more of the parameters "Session-gated Loudness Correction Flag" and "Loudness Correction Type" of Table 2) indicating at least one type of loudness processing that has been performed on the corresponding audio data; and

At least one loudness value (e.g., one or more of the parameters "ITU Relative Gated Loudness," "ITU Speech Gated Loudness," "ITU (EBU 3341) Short-Term 3s Loudness," and "True Peak" in Table 2) representing at least one loudness (e.g., peak or average loudness) characteristic of corresponding audio data.

In some embodiments, each metadata segment containing LPSM and program boundary metadata contains a core header (and optionally additional core elements), and following the core header (or core header and other core elements) an LPSM payload (or container) segment having the following format:

header, which typically includes at least one identification value (e.g., LPSM format version, length, period, count, and substream association value, as shown in Table 2 proposed herein), and

Following the header is the LPSM and program boundary metadata. The program boundary metadata may include a program boundary frame count, an encoded value, and (in some cases) an offset value. An encoded value (e.g., an "offset_exist" value) indicates whether the frame includes only a program boundary frame count or both a program boundary frame count and an offset value.

In some implementations, each of the wasted bits segments or metadata segments in the "addbsi" field or auxdata field inserted into a frame of the bitstream by stage 107 has the following format:

Core header (typically includes a synchronization word identifying the start of the metadata segment, followed by identification values, core element version, length and period, extension element count, and substream association value as shown in Table 1 below); and

At least one protection value following the core header (e.g., the HMAC digest and audio fingerprint value of Table 1, which is useful for at least one of decryption, authentication, or verification of at least one of the loudness processing state metadata or corresponding audio data); and

If the metadata segment includes an LPSM, an LPSM payload identification (ID) and an LPSM payload size value also follow the core header, identifying the following metadata as an LPSM payload and indicating the size of the LPSM payload.

The LPSM Payload (or Container) segment (preferably having the format described above) follows the LPSM Payload ID and LPSM Payload Size values.

In some embodiments, each metadata segment in the auxdata field (or "addbsi" field) of a frame has a three-layer structure:

A high-level structure that includes: a flag indicating whether the auxdata (or addbsi) field contains metadata; at least one ID value indicating what type of metadata is present; and typically also a value indicating how many bits of metadata (e.g., of each type of metadata) are present (if metadata is present). One type of metadata that may be present is LPSM, another type of metadata that may be present is program boundary metadata, and another type of metadata that may be present is media research metadata (e.g., Nielsen Media Research metadata).

a middle-level structure including core elements of metadata for each identified type (e.g., metadata for each identified type such as a core header of the type described above, a protection value, and an LPSM payload ID and payload size value); and

A low-level structure, including each payload for a core element (e.g., an LPSM payload if the core element identifies it as present, and/or another type of metadata payload if the core element identifies it as present).

Data values in such a three-layer structure can be nested. For example, protection values for an LPSM payload and/or another metadata payload identified by a core element can be included after a payload identified by a core element (and thus after a core header of the core element). In one example, the core header can identify the LPSM payload and the other metadata payload, a payload ID and payload size value for a first payload (e.g., an LPSM payload) can follow the core header, the first payload itself can follow the ID and size values, a payload ID and payload size value for a second payload can follow the first payload, the second payload itself can follow these ID and size values, and protection values for both payloads (or the core element value and both payloads) can follow the final payload.

In some embodiments, if decoder 101 receives an audio bitstream generated according to an embodiment of the present invention with a cryptographic hash, the decoder is configured to parse and retrieve the cryptographic hash from a data block determined from the bitstream, the block including loudness processing state metadata (LPSM) and, optionally, program boundary metadata. Validator 102 can use the cryptographic hash to validate the received bitstream and/or associated metadata. For example, if validator 102 finds that the LPSM is valid based on a match between a reference cryptographic hash and the cryptographic hash retrieved from the data block, it can disable processor 103 from operating on the corresponding audio data and cause selection stage 104 to pass the (unchanged) audio data. Additionally, optionally, or alternatively, other types of cryptographic techniques can be used in place of cryptographic hash-based approaches.

The encoder 100 of FIG2 can determine (in response to the LPSM extracted by the decoder 101 and, optionally, program boundary metadata) that a post-processing/pre-processing unit has already performed some type of loudness processing on the audio data to be encoded (in elements 105, 106, and 107), and can therefore generate (in a generator 106) loudness processing state metadata that includes specific parameters for and/or extracted from the previously performed loudness processing. In some implementations, the encoder 100 can generate (and include in the encoded bitstream output therefrom) processing state metadata that represents the processing history of the audio content, as long as the encoder is aware of the type of processing that has been performed on the audio content.

3 is a block diagram of a decoder (200) as an embodiment of an audio processing unit of the present invention, and a block diagram of a post-processor (300) coupled to the encoder 200. The post-processor (300) is also an embodiment of an audio processing unit of the present invention. Any component or element of the decoder 200 and the post-processor 300 can be implemented as one or more processes and/or one or more circuits (e.g., ASICs, FPGAs, or other integrated circuits) using hardware, software, or a combination of hardware and software. The decoder 200 includes a frame buffer 210, a parser 205, an audio decoder 202, an audio state verification stage (verifier) 203, and a control bit generation stage 204, connected as shown. Typically, the decoder 200 also includes other processing elements (not shown).

The frame buffer 201 (buffer memory) stores (in a non-transitory manner) at least one frame of the encoded audio bitstream received by the decoder 200. From the buffer 201 to the parser 205 a sequence of frames of the encoded audio bitstream is transferred.

Parser 205 is coupled and configured to extract loudness processing state metadata (LPSM) and optionally also program boundary metadata and other metadata from each frame of the encoded input audio, pass at least the LPSM (and program boundary metadata if any was extracted) to audio state validator 203 and stage 204, pass the LPSM (and optionally also program boundary metadata) as output (e.g., to post-processor 300), extract audio data from the encoded input audio, and pass the extracted audio data to decoder 202.

The encoded audio bitstream input to the decoder 200 may be one of an AC-3 bitstream, an E-AC-3 bitstream, or a Dolby E bitstream.

The system of FIG3 also includes a post-processor 300. Post-processor 300 includes a frame buffer 301 and other processing elements (not shown), including at least one processing element coupled to buffer 301. Frame buffer 301 stores (e.g., in a non-transitory manner) at least one frame of a decoded audio bitstream received by post-processor 300 from decoder 200. The processing elements of post-processor 300 are coupled and configured to receive and adaptively process a sequence of frames of the decoded audio bitstream output from buffer 301 using metadata (including LPSM values) output from decoder 202 and/or control bits output from stage 204 of decoder 200. Generally, post-processor 300 is configured to perform adaptive loudness processing on the decoded audio data using LPSM values and, optionally, program boundary metadata (e.g., based on a loudness processing state indicated by the LPSM for audio data indicating a single audio program and/or one or more audio data characteristics).

Various implementations of decoder 200 and post-processor 300 are configured to perform different embodiments of the method of the present invention.

The audio decoder 202 of the decoder 200 is configured to decode the audio data extracted by the parser 205 to generate decoded audio data, and is configured to transmit the decoded audio data as output (eg, to the post-processor 300 ).

The state validator 203 is configured to authenticate and verify the LPSM (and typically other metadata) that is passed to it. In some embodiments, the LPSM is a data block that is already included in the input bitstream (e.g., according to an embodiment of the present invention) (or is included in a data block that is already included in the input bitstream). The block may include a cryptographic hash (hash-based message authentication code or "HMAC") for processing the LPSM (and optionally other metadata) and/or the underlying audio data (provided to the validator 203 from the parser 205 and/or decoder 202). The data block may be digitally signed in these embodiments so that downstream audio processing units can relatively easily authenticate and verify the processing state metadata.

Other cryptographic algorithms, including but not limited to any of one or more non-HMAC cryptographic methods, may be used for verification of the LPSM (e.g., in the verifier 203) to ensure secure transmission and reception of the LPSM and/or underlying audio data. For example, verification (using such cryptographic methods) may be performed in each audio processing unit of an embodiment of the present invention that receives an audio bitstream to determine whether loudness processing state metadata and corresponding audio data included in the bitstream have undergone (and/or have been derived from) a specific loudness processing (indicated by the metadata) and have not been modified since such specific loudness processing was performed.

State validator 203 transmits control data to control bit generator 204 and/or transmits the control data as an output (e.g., to post-processor 300) to indicate the result of the validation operation. In response to the control data (and optionally also in response to other metadata extracted from the input bitstream), stage 204 may generate (and transmit to post-processor 300) any of the following:

a control bit indicating that the decoded audio data output from the decoder 202 has been subjected to a specific type of loudness processing (when the LPSM indicates that the audio data output from the decoder 202 has been subjected to a specific type of loudness processing and the control bit from the validator 203 indicates that the LPSM is valid); or

A control bit indicating that the decoded audio data output from the decoder 202 should be subjected to a particular type of loudness processing (e.g., when the LPSM indicates that the audio data output from the decoder 202 has not been subjected to the particular type of loudness processing, or when the LPSM indicates that the audio data output from the decoder 202 has been subjected to the particular type of loudness processing and the control bit from the validator 203 indicates that the LPSM is invalid).

Alternatively, the decoder 200 transmits the metadata extracted from the input bitstream by the decoder 202 and the LPSM (and optionally also the program boundary metadata) extracted from the input bitstream by the parser 205 to the post-processor 300, and the post-processor 300 performs loudness processing on the decoded audio data using the LPSM (and optionally also the program boundary metadata), or performs verification of the LPSM and then, if the verification indicates that the LPSM is valid, performs loudness processing on the decoded audio data using the LPSM (and optionally also the program boundary metadata).

In some embodiments, if decoder 200 receives an audio bitstream generated according to an embodiment of the present invention with a cryptographic hash, the decoder is configured to parse and retrieve the cryptographic hash from a data block determined from the bitstream, the block comprising loudness processing state metadata (LPSM). Verifier 203 can use the cryptographic hash to verify the received bitstream and/or associated metadata. For example, if verifier 203 finds that the LPSM is valid based on a match between a reference cryptographic hash and a cryptographic hash retrieved from the data block, it can signal a downstream audio processing unit (e.g., post-processor 300, which may be or include a volume adjustment unit) to pass the audio data of the bitstream (unchanged). Additionally, optionally, or alternatively, other types of encryption techniques can be used in place of cryptographic hash-based approaches.

In some implementations of decoder 200, the encoded bitstream received (and buffered in memory 201) is an AC-3 bitstream or an E-AC-3 bitstream and includes audio data segments (e.g., segments AB0 to AB5 of a frame shown in FIG4) and metadata segments, wherein the audio data segments represent audio data and at least some of the metadata segments each include loudness processing state metadata (LPSM) and optionally program boundary metadata. Decoder stage 202 (and/or parser 205) is configured to extract the LPSM (and optionally program boundary metadata) from the bitstream in the following format. Each of the metadata segments including the LPSM (and optionally program boundary metadata) is included in a wasted bits segment of a frame of the bitstream, or in an "addbsi" field of a bitstream information ("BSI") segment of a frame of the bitstream, or in an auxdata field at the end of a frame of the bitstream (e.g., an AUX segment shown in FIG4). A frame of a bitstream may include one or two metadata segments, each of which may include an LPSM, and if a frame includes two metadata segments, one of the metadata segments may be present in the addbsi field of the frame and the other in the AUX field of the frame. In some embodiments, each metadata segment including an LPSM includes an LPSM payload (or container) segment having the following format:

header (typically including a synchronization word that identifies the start of the LPSM payload, followed by identification values, such as the LPSM format version, length, period, count, and substream association values shown in Table 2 below); and

After the first part,

At least one session indication value (e.g., parameter “session channel” of Table 2), which indicates whether the corresponding audio data indicates a session or does not indicate a session (e.g., which channel of the corresponding audio data indicates a session);

At least one loudness adjustment compliance value (e.g., parameter "Loudness Adjustment Type" of Table 2), which indicates whether the corresponding audio data complies with the indicated set of loudness adjustments;

At least one loudness processing value (e.g., one or more of the parameters "Session-gated Loudness Correction Flag" and "Loudness Correction Type" of Table 2) indicating at least one type of loudness processing that has been performed on the corresponding audio data; and

At least one loudness value (e.g., one or more of the parameters "ITU Relative Gated Loudness," "ITU Speech Gated Loudness," "ITU (EBU 3341) Short-Term 3s Loudness," and "True Peak" of Table 2) indicating at least one loudness (e.g., peak or average loudness) characteristic of corresponding audio data.

In some embodiments, each metadata segment containing LPSM and program boundary metadata contains a core header (and optionally additional core elements), and following the core header (or core header and other core elements) an LPSM payload (or container) segment having the following format:

header, which typically includes at least one identification value (e.g., LPSM format version, length, period, count, and substream association value, as shown in Table 2 proposed herein), and

Following the header is the LPSM and program boundary metadata. The program boundary metadata may include a program boundary frame count, an encoded value, and (in some cases) an offset value. An encoded value (e.g., an "offset_exist" value) indicates whether the frame includes only a program boundary frame count or both a program boundary frame count and an offset value.

In some implementations, the parser 205 (and/or the decoder stage 202) is configured to extract from the wasted bits segment or the "addbsi" field or the auxdata field of a frame of the bitstream each metadata segment having the following format:

A core header (typically comprising a synchronization word identifying the start of a metadata segment, followed by at least one identification value, such as a core element version, length and period, an extension element count, and a substream association value as shown in Table 1 below); and

At least one protection value following the core header (e.g., the HMAC digest and audio fingerprint value of Table 1) that is useful for at least one of decryption, authentication, or verification of at least one of the loudness processing state metadata or corresponding audio data; and

If the metadata segment includes an LPSM, an LPSM payload identification (ID) and an LPSM payload size value also follow the core header, identifying the following metadata as an LPSM payload and indicating the size of the LPSM payload.

The LPSM Payload (or Container) segment (preferably having the format described above) follows the LPSM Payload ID and LPSM Payload Size values.

More generally, the coded audio bitstream generated by the preferred embodiment of the present invention has a structure that provides a mechanism for tagging metadata elements and sub-elements as core (mandatory) or extensions (optional elements). This enables the data rate of the bitstream (including its metadata) to be scalable across a wide range of applications. The core (mandatory) elements of the preferred bitstream syntax should be able to signal the presence (in-band) and/or remote (out-of-band) of extension (optional) elements associated with the audio content.

Core elements are required to be present in every frame of the bitstream. Some subelements of core elements are optional and can be present in any combination. Extension elements do not need to be present in every frame (to prevent the bit rate from being too high). Therefore, extension elements can be present in some frames and not in others. Some subelements of extension elements are optional and can be present in any combination, while some subelements of extension elements can be mandatory (i.e., if the extension element is present in a frame of the bitstream).

In one class of embodiments, an encoded audio bitstream is generated (e.g., by an audio processing unit implementing the present invention) comprising a sequence of audio data segments and metadata segments. The audio data segments represent audio data, and at least some of the metadata segments each comprise loudness processing state metadata (LPSM) and optionally program boundary metadata, the audio data segments being time-division multiplexed with the metadata segments. In a preferred embodiment of this class, each metadata segment has a preferred format to be described herein.

In a preferred format, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment including the LPSM is included as additional bitstream information (e.g., stage 107 of a preferred implementation of encoder 100) in an "addbsi" field (as shown in FIG. 6 ) of a bitstream information (“BSI”) segment of a frame of the bitstream, or in an auxdata field of a frame of the bitstream, or in a wasted bits segment of a frame of the bitstream.

In the preferred format, each frame includes core elements in the addbsi field (or wasted bits segment) of the frame having the format shown in Table 1 below:

Table 1

In the preferred format, each addbsi (or auxdata) field or wasted bits segment containing an LPSM contains a core header (optionally with additional core elements) and, after the core header (or core header and other core elements), the following LPSM values (parameters):

The payload ID (identifying the metadata as LPSM) follows the core element value (as shown in Table 1);

The payload ID is followed by the payload size (indicating the size of the LPSM payload); and

The LPSM data has the format shown in the following table (Table 2) (followed by the payload ID and payload size values):

Table 2

In another preferred format of an encoded bitstream generated according to the present invention, the bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment (e.g., by stage 107 of a preferred implementation of encoder 100) comprising an LPSM (and optionally program boundary metadata) is included in either: a wasted bits segment of a frame of the bitstream; or an "addbsi" field of a bitstream information ("BSI") segment of a frame of the bitstream (as shown in FIG6); or an auxdata field at the end of a frame of the bitstream (e.g., an AUX segment as shown in FIG4). A frame may include one or two metadata segments, each metadata segment including an LPSM, and if a frame includes two metadata segments, one of the metadata segments may be present in the addbsi field of the frame and the other in the AUX field of the frame. Each metadata segment including LPSM has the format indicated in the above reference Tables 1 and 2 (i.e., it includes: the core elements shown in Table 1, followed by a payload ID (identifying the metadata as LPSM); and the above-mentioned payload size value, followed by a payload (LPSM data having the format shown in Table 2)).

In another preferred format, the encoded bitstream is a Dolby E bitstream, and each metadata segment including an LPSM (and optionally program boundary metadata) is the first N sample positions of a Dolby E guardband interval. A Dolby E bitstream including such metadata segments (including an LPSM) preferably includes a value representing the LPSM payload length indicated in the Pd word of the SMPTE 337M preamble (the SMPTE 337M Pa word repetition rate is preferably kept consistent with the associated video frame rate).

In a preferred format in which the encoded bitstream is an E-AC-3 bitstream, each metadata segment including the LPSM (and optionally program boundary metadata) is included as additional bitstream information (e.g., by stage 107 of a preferred implementation of encoder 100) in a wasted bits segment or "addbsi" field of a bitstream information ("BSI") segment of a frame of the bitstream. Further aspects of the encoding of an E-AC-3 bitstream having the LPSM in this preferred format are described below:

1. During the generation of an E-AC-3 bitstream, when the E-AC-3 encoder (which inserts LPSM values into the bitstream) is "active", for each frame generated (sync frame), the bitstream shall include a metadata block (including LPSM) carried in the addbsi field (or wasted bits segment) of the frame. The bits required to carry the metadata block shall not increase the encoder bit rate (frame length);

2. Each metadata block (including LPSM) should contain the following information:

1. loudness_correction_type_flag: where "1" indicates that the loudness of the corresponding audio data is corrected upstream from the encoder, and "0" indicates that the loudness is corrected by a loudness corrector embedded in the encoder (e.g., the loudness processor 103 of the encoder 100 of FIG. 2 );

2. speech_channel: Indicates which source channel contains speech (in the previous 0.5 seconds). If no speech is detected, this should be indicated as such;

3.speech_loudness: indicates the overall speech loudness (in the previous 0.5 seconds) of each corresponding audio channel containing speech;

4. ITU_loudness: Indicates the overall ITU BS.1770-2 loudness of each corresponding audio channel; and

5. gain: loudness synthesis gain for inversion in the decoder (to show inversion);

3. When the E-AC-3 encoder (which inserts LPSM values into the bitstream) is "active" and receives an AC-3 frame with the "trust" flag, the loudness controller in the encoder (e.g., loudness processor 103 of encoder 100 of FIG. 2 ) should be bypassed. The "trusted" source dialnorm and DRC values should be passed (e.g., via generator 106 of encoder 100 ) to the E-AC-3 encoder components (e.g., stage 107 of encoder 100 ). LPSM block generation continues, and the loudness_correction_type_flag is set to "1." The loudness controller bypass sequence must be synchronized with the start of the decoded AC-3 frame in which the "trust" flag is present. The loudness controller bypass sequence should be implemented by decrementing the leveler_amount control from a value of 9 to a value of 0 over 10 audio block periods (i.e., 53.3 milliseconds) and placing the leveler_back_end_meter control in bypass mode (this operation should result in a seamless transition). The term "trusted" bypass of the calibrator means that the dialnorm value of the source bitstream is also reused at the output of the encoder. (i.e., if the dialnorm value of the "trusted" source bitstream is -30, then the output of the encoder should use -30 for the output dialnorm value);

4. When the E-AC-3 encoder (which inserts LPSM values into the bitstream) is "active" and receives an AC-3 frame without the "trust" flag, the loudness controller embedded in the encoder (e.g., the loudness processor 103 of the encoder 100 of FIG. 2 ) should be active. LPSM block generation continues, and the loudness_correction_type_flag is set to "0". The loudness controller activation sequence should be synchronized with the start of the decoded AC-3 frame where the "trust" flag disappears. The loudness controller activation sequence should be implemented as follows: increment the leveler_amount control from a value of 0 to a value of 9 over 1 audio block period (i.e., 5.3 milliseconds), and put the leveler_back_end_meter control in "active" mode (this operation should result in a seamless transition and include an overall reset of the back_end_meter); and

5. During encoding, the Graphical User Interface (GUI) should indicate to the user the following parameters: "Input Audio Program: [Trusted/Untrusted]" - the state of this parameter is based on the presence of a "Trusted" flag within the input signal; and "Real-time Loudness Correction: [Enabled/Disabled]" - the state of this parameter is based on whether the loudness controller embedded in the encoder is active.

When decoding an AC-3 or E-AC-3 bitstream having an LPSM (preferred format of LPSM) included in the Wasted Bits Segment or the "addbsi" field of the Bitstream Information ("BSI") segment of each frame of the bitstream, the decoder should parse the LPSM block data (wasted bits segment or LPSM block data in the addbsi field) and pass all extracted LPSM values to the graphical user interface (GUI). The set of extracted LPSM values is updated every frame.

In another preferred format for an encoded bitstream generated according to the present invention, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment that includes an LPSM (e.g., by stage 107 of a preferred implementation of encoder 100) is included as additional bitstream information in an "addbsi" field (as shown in FIG6 ) of a bitstream information ("BSI") segment of a frame of the bitstream (or in an AUX segment or a wasted bits segment). In this format (which is a variation of the format described above with reference to Tables 1 and 2), each addbsi (or AUX or wasted bits) field that includes an LPSM contains the following LPSM values:

The core elements shown in Table 1 are followed by a payload ID (identifying the metadata as LPSM) and a payload size value, followed by a payload (LPSM data) having the following format (similar to the mandatory elements shown in Table 2 above):

LPSM payload version: a 2-bit field indicating the version of the LPSM payload;

dialchan: A 3-bit field indicating whether the left channel, right channel, and/or center channel of the corresponding audio data contains spoken conversation. The bit allocation of the dialchan field may be as follows: bit 0 indicating the presence of conversation in the left channel is stored in the most significant bit of the dialchan field; bit 2 indicating the presence of conversation in the center channel is stored in the least significant bit of the dialchan field. Each bit of the dialchan field is set to "1" if the corresponding channel contains spoken conversation during the first 0.5 seconds of the program.

loudregtyp: A 4-bit field that indicates which loudness adjustment standard the program loudness complies with. Setting the "loudregtyp" field to "000" indicates that the LPSM does not indicate loudness adjustment compliance. For example, one value for this field (e.g., 0000) may indicate that compliance with a loudness adjustment standard is not indicated, another value for this field (e.g., 0001) may indicate that the program's audio data complies with the ATSC A/85 standard, and another value for this field (e.g., 0010) may indicate that the program's audio data complies with the EBU R128 standard. In this example, if this field is set to any value other than "0000," the loudcorrdialgat field and the loudcorrtyp field should follow in the payload.

loudcorrdialgat: A 1-bit field indicating whether loudness correction for session gating has been applied. If the program's loudness has been corrected using session gating, the value of the loudcorrdialgat field is set to "1". Otherwise, it is set to "0".

loudcorrtyp: A 1-bit field indicating the type of loudness correction applied to the program. If the program's loudness has been corrected using an infinite look-ahead (file-based) loudness correction process, the value of the loudcorrtyp field is set to "0". If the program's loudness has been corrected using a combination of real-time loudness measurement and dynamic range control, the value of this field is set to "1".

loudrelgate: A 1-bit field indicating whether related gated loudness data (ITU) is present. If the loudrelgate field is set to "1", the 7-bit ituloudrelgat field shall follow in the payload;

loudrelgat: A 7-bit field indicating the relative gated program loudness (ITU). This field indicates the overall loudness of the audio program measured according to ITU-R BS.1770-3 without any gain adjustment due to the applied dialnorm and dynamic range compression. Values from 0 to 127 are interpreted as -58 LKFS to +5.5 LKFS in steps of 0.5 LKFS.

loudspchgate: A 1-bit field indicating whether the loudness data (ITU) of the speech gate is present. If the loudspchgate field is set to "1", the 7-bit loudspchgat field shall follow in the payload;

loudspchgat: A 7-bit field indicating the loudness of the speech-gated program. This field indicates the overall loudness of the entire corresponding audio program, measured according to equation (2) of ITU-R BS.1770-3 without any gain adjustment, due to the applied dialnorm and dynamic range compression. Values from 0 to 127 are interpreted as -58 LKFS to +5.5 LKFS, in steps of 0.5 LKFS;

loudstrm3se: A 1-bit field indicating whether short-term (3-second) loudness data is present. If this field is set to "1", the 7-bit loudstrm3s field shall follow in the payload;

loudstrm3s: A 7-bit field indicating the ungated loudness of the preceding 3 seconds of the corresponding audio programme, measured according to ITU-R BS.1771-1 without any gain adjustment, due to the applied dialnorm and dynamic range compression. Values from 0 to 256 are understood as -116 LKFS to +11.5 LKFS, in steps of 0.5 LKFS;

truepke: A 1-bit field indicating whether true peak loudness data is present. If the truepke field is set to "1", then an 8-bit truepk field shall follow in the payload; and

truepk: An 8-bit field indicating the true peak sample value of the program due to applied dialnorm and dynamic range compression, measured according to Annex 2 of ITU-R BS.1770-3 without any gain adjustment. Values from 0 to 256 are interpreted as -116 LKFS to +11.5 LKFS in steps of 0.5 LKFS.

In some embodiments, a core element of a metadata segment in a wasted bits segment or auxdata field (or "addbsi" field) of a frame of an AC-3 bitstream or E-AC-3 bitstream includes a core header (typically including an identification value, such as a core element version), and, following the core header, includes: a value indicating whether fingerprint data (or other protection value) is included for the metadata of the metadata segment, a value indicating whether external data (related to the audio data corresponding to the metadata of the metadata segment) is present, a payload ID and payload size value for each type of metadata identified by the core element (e.g., LPSM and/or metadata other than LPSM), and a protection value for at least one type of metadata identified by the core element. The metadata payload of the metadata segment follows the core header and (in some cases) is nested within the value of the core element.

Typical embodiments of the present invention include program boundary metadata in an encoded audio bitstream in an efficient manner that enables accurate and robust determination of at least one boundary between consecutive audio programs indicated by the bitstream. Typical embodiments enable accurate and robust determination of program boundaries in scenarios where they allow accurate program boundary determination even in cases where bitstreams indicating different programs are spliced together (to generate the inventive bitstream) in a manner that enables truncating one or both of the spliced bitstreams (and thereby discarding program boundary metadata that had been included in at least one of the pre-spliced bitstreams).

In a typical embodiment, the program boundary metadata in a frame of the inventive bitstream is a program boundary marker indicating a frame count. Typically, the marker indicates the number of frames between the current frame (including the marked frame) and the program boundary (the start or end of the current audio program). In some preferred embodiments, the program boundary marker is inserted at the beginning and end of each bitstream segment indicating a single program (i.e., in a frame that occurs within some predetermined number of frames after the start of the segment, and in a frame that occurs within some predetermined number of frames before the end of the segment) in a symmetrical and efficient manner, so that when two such bitstream segments are concatenated (so as to indicate a sequence of two programs), the program boundary metadata can appear on both sides of the boundary between the two programs (i.e., symmetrically).

Maximum robustness can be achieved by inserting a program boundary marker in every frame of the bitstream indicating a program, but this is generally impractical due to the associated increase in data rate. In a typical embodiment, program boundary markers are inserted only in a subset of the frames of the coded audio bitstream (which may indicate an audio program or sequence of audio programs), and the boundary marker insertion rate is a non-increasing function of the increasing separation of each of the frames of the bitstream in which the marker is inserted from the program boundary closest to said each of those frames, where the "boundary marker insertion rate" represents the average ratio of the number of frames that include program boundary markers (indicating a program) to the number of frames that do not include program boundary markers (indicating a program), where the average is a sliding average over a number (e.g., a relatively small number) of consecutive frames of the coded audio bitstream.

Increasing the rate at which boundary markers are inserted (e.g., at locations in the bitstream closer to program boundaries) increases the data rate required to deliver the bitstream. To compensate for this, the size (number of bits) of each inserted marker is preferably reduced as the rate of boundary marker insertion increases (e.g., such that the size of the program boundary marker in the Nth frame of the bitstream (where N is an integer) is a non-increasing function of the distance (number of frames) between the Nth frame and the nearest program boundary). In one class of embodiments, the rate of boundary marker insertion is a logarithmically decreasing function of increasing distance (at each marker insertion location) from the nearest program boundary, and for each marker-containing frame that includes one of the markers, the size of the marker in the marker-containing frame is equal to or greater than the size of each marker in frames that are closer to the nearest program boundary than the marker-containing frame. Typically, the size of each marker is determined by an increasing function of the number of frames from the marker's insertion location to the nearest program boundary.

For example, consider the embodiments of Figures 8 and 9, in which each column identified by a frame number (in the top row) indicates a frame of an encoded audio bitstream. The bitstream indicates an audio program having a first program boundary (indicating the start of the program) and a second program boundary (indicating the end of the program), the first program boundary occurring immediately to the left of the column identified by frame number "17" on the left side of Figure 9, and the second program boundary occurring immediately to the right of the column identified by frame number "1" on the right side of Figure 8. The program boundary markers included in the frames shown in Figure 8 count down the number of frames between the current frame and the second program boundary. The program boundary markers included in the frames shown in Figure 9 count up the number of frames between the current frame and the first program boundary.

In the embodiments of Figures 8 and 9, program boundary markers are inserted only in each of the ^2Nth frames of the first X frames of the encoded bitstream following the start of the audio program indicated by the bitstream, and in each of ^{the 2Nth frames of} the next X frames of the bitstream that are closest to the end of the program indicated by the bitstream, where a program comprises Y frames, X being an integer less than or equal to Y/2, and N being a positive integer between 1 and log ₂ (X). Thus (as indicated in Figures 8 and 9), program boundary markers are inserted in the 2nd frame (N=1) of the bitstream (the frame containing the marker closest to the start of the program), in the 4th frame (N=2), in the 8th frame (N=3), and so on, as well as in the 8th frame from the end of the bitstream, in the 4th frame from the end of the bitstream, and in the 2nd frame from the end of the bitstream (the frame containing the marker closest to the end of the program). In this example, the program boundary marker in the ^2Nth frame from the start (or end) of the program comprises log2( ^2N+2 ) binary bits, as indicated in Figures 8 and 9. Thus, the program boundary marker in the 2nd frame (N=1) from the start (or end) of the program comprises _log2 (2N ⁺² )= _log2 ( ²³ )=3 binary bits, and the marker in the 4th frame (N=2) from the start (or end) of the program comprises _log2 ( ^2N+2 )= _log2 ( ²⁴ )=4 binary bits, and so on.

In the examples of Figures 8 and 9, the format of each program boundary marker is as follows. Each program boundary marker includes a leading "1" bit, a "0" bit sequence after the leading bit (or no "0" bit or one or more consecutive "0" bits), and a 2-bit tail code. The tail code is "11" for the marker in the last X frames of the bit stream (the frames closest to the end of the program), as indicated in Figure 8. The tail code is "10" for the marker in the first X frames of the bit stream (the frames closest to the start of the program), as indicated in Figure 9. Therefore, in order to read (decode) each marker, the number of 0s between the leading "1" bit and the tail code is counted. If the tail code is identified as "11", the marker indicates that there are (2 ^Z+1 -1) frames between the current frame (including the marked frame) and the end of the program, where Z is the number of 0s between the leading "1" bit and the tail code of the marker. The decoder can be efficiently implemented to ignore the first and last bits of each such marker, thereby determining the inverse of the sequence of the other (middle) bits of the marker (e.g., if the sequence of middle bits is "0001," where the "1" bit is the last bit in the sequence, then the inverse sequence of the middle bits is "1000," where the "1" bit is the first bit in the inverse sequence), and thereby identifying the binary value of the inverse sequence of the middle bits as the index of the current frame (the frame in which the marker is included) relative to the end of the program. For example, if the inverse sequence of the middle bits is "1000," then this inverse sequence has a binary value of 2 ⁴ =16, and the frame is identified as the 16th frame before the end of the program (as indicated by the column describing frame "0" of FIG. 8 ).

If the tail code is identified as "10", the marker indicates that there are (2 ^{Z + 1} -1) frames between the start of the program and the current frame (including the marked frame), where Z is the number of zeros between the leading "1" bit of the marker and the tail code. The decoder can be efficiently implemented to ignore the first and last bits of each such marker, thereby determining the inverse of the sequence of the marker's middle bits (for example, if the sequence of the middle bits is "0001", where the "1" bit is the last bit in the sequence, then the inverse sequence of the middle bits is "1000", where the "1" bit is the first bit in the inverse sequence), and thereby identifying the binary value of the inverse sequence of the middle bits as the index of the current frame (the frame in which the marker is included) relative to the start of the program. For example, if the inverse sequence of the middle bits is "1000", then this inverse sequence has a binary value of 2 ⁴ =16, and the frame is identified as the 16th frame after the start of the program (as indicated by the column describing frame "32" of Figure 9).

In the examples of Figures 8 and 9, program boundary markers appear only in every ^2Nth of the first X frames of the encoded bitstream following the start of an audio program indicated by the bitstream, and appear in every ^2Nth of the ^{2Nth frames} (of the last X frames of the bitstream) nearest the end of the program indicated by the bitstream, where the program comprises Y frames, X is an integer less than or equal to Y/2, and N is a positive integer in the range of 1 to _log2 (X). Including program boundary markers adds only an average bit rate of 1.875 bits/frame to the bit rate required to transmit the bitstream without the markers.

In a typical implementation of the embodiments of Figures 8 and 9 where the bitstream is an AC-3 encoded audio bitstream, each frame contains the audio content and metadata for 1536 samples of digital audio. For a sampling rate of 48kHz, this represents 32 milliseconds of digital audio or a rate of 31.25 frames per second of audio. Thus, in such an embodiment, a program boundary marker in a frame that is separated from a program boundary by some number of frames ("X" frames) indicates that the boundary occurs 32X milliseconds after the end of the frame containing the marker (or 32X milliseconds before the beginning of the frame containing the marker).

In the embodiment of Figure 8 and Figure 9, in which the bitstream is a typical implementation of an E-AC-3 encoded audio bitstream, each frame of the bitstream contains audio content and metadata for 256, 512, 768, or 1536 samples of digital audio, depending on whether the frame contains 1, 2, 3, or 6 audio data blocks. For a sampling rate of 48kHz, this represents a rate of 5.333, 10.667, 16, or 32 milliseconds of digital audio or 189.9, 93.75, 62.5, or 31.25 frames per second of audio, respectively. Therefore, in such an embodiment (assuming that each frame indicates 32 milliseconds of digital audio), the program boundary marker in a frame that is separated from the program boundary by some number of frames ("X" frames) indicates that the boundary occurs 32X milliseconds after the end of the frame containing the marker (or 32X milliseconds before the beginning of the frame containing the marker).

In some embodiments where program boundaries may occur within frames of the audio bitstream (i.e., not aligned with the start or end of a frame), program boundary metadata included in frames of the bitstream includes a program boundary frame count (i.e., metadata indicating the number of full frames between the start or end of the frame containing the frame count and the program boundary) and an offset value. The offset value indicates the offset (typically a number of samples) between the start or end of the frame containing the program boundary and the actual location of the program boundary within the frame containing the program boundary.

The coded audio bitstream can indicate a sequence of programs (audio tracks) within a corresponding video program sequence, and the boundaries of such audio programs tend to occur at the edges of video frames rather than at the edges of audio frames. Furthermore, some audio codecs (e.g., the E-AC-3 codec) use audio frame sizes that are not aligned with video frames. Furthermore, in some cases, an initial coded audio bitstream undergoes transcoding to generate a transcoded bitstream, and the initial coded bitstream has a different frame size than the transcoded bitstream, such that program boundaries (determined by the initial coded bitstream) are not guaranteed to occur at the frame boundaries of the transcoded bitstream. For example, if the initial coded bitstream (e.g., bitstream "IEB" in FIG. 10 ) has a frame size of 1536 samples per frame, and the transcoded bitstream (e.g., bitstream "TB" in FIG. 10 ) has a frame size of 1024 samples per frame, then due to the different frame sizes of the different codecs, the transcoding process may cause the actual program boundaries to occur not at the frame boundaries of the transcoded bitstream, but somewhere within its frames (e.g., 512 samples within a frame of the transcoded bitstream, as shown in FIG. 10 ). The embodiment of the present invention in which the program boundary metadata included in the frames of the encoded audio bitstream includes an offset value as well as a program boundary frame count is useful in the three situations indicated in this paragraph (and in other situations).

The embodiment described above with reference to Figures 8 and 9 does not include an offset value (e.g., an offset field) in any of the frames in the coded bitstream. In a variation of this embodiment, an offset value is included in each frame of the coded audio bitstream that includes a program boundary marker (e.g., frames corresponding to frames numbered 0, 8, 12, and 14 in Figure 8 and frames numbered 18, 20, 24, and 32 in Figure 9).

In one class of embodiments, a data structure (in each frame of the coded bitstream containing the inventive program boundary metadata) includes a code value indicating whether the frame includes only a program boundary frame count or includes both a program boundary frame count and an offset value. For example, the code value may be the value of a single-bit field (hereinafter referred to as an "offset_exist" field), where a value of "offset_exist" = 0 may indicate that the frame does not include any offset value, and a value of "offset_exist" = 1 may indicate that the frame includes both a program boundary frame count and an offset value.

In some embodiments, at least one frame of an AC-3 or E-AC-3 encoded audio bitstream includes a metadata segment containing LPSM and program boundary metadata (and optionally other metadata) for an audio program identified by the bitstream. Each such metadata segment (which may be included in the addbsi field, or the auxdata field, or the wasted bits segment of the bitstream) includes a core header (and optionally additional core elements) and, following the core header (or the core header and other core elements), an LPSM payload (or container) segment having the following format:

header (usually including at least one identification value, such as LPSM format version, length, period, count, and substream association value), and

Following the header: Program Boundary Metadata (which may include a program boundary frame count, an encoded value, and in some cases an offset value, and an LPSM. An encoded value (e.g., an "offset_exist" value) indicates whether the frame includes only a program boundary frame count or both a program boundary frame count and an offset value). The LPSM may include:

At least one session indication value indicating whether the corresponding audio data indicates a session or does not indicate a session (e.g., which channels of the corresponding audio data indicate a session). The session indication value may indicate whether a session exists in any combination of channels or in all channels of the corresponding audio data;

at least one loudness adjustment compliance value indicating whether the corresponding audio data complies with the indicated loudness rule set;

at least one loudness processing value indicating at least one type of loudness processing that has been performed on the corresponding audio data; and

At least one loudness value indicating at least one loudness (eg, peak loudness or average loudness) characteristic of the corresponding audio data.

In some embodiments, the LPSM payload segment includes a code value ("offset_exist" value) indicating whether the frame includes only a program boundary frame count or includes both a program boundary frame count and an offset value. For example, in one such embodiment, when such a code value indicates (e.g., when offset_exist=1) that the frame includes a program boundary frame count and an offset value, the LPSM payload segment may include an offset value that is an 11-bit unsigned integer (i.e., having a value between 0 and 2048) and indicates the number of additional audio samples between an indicated frame boundary (a frame boundary including a program boundary) and an actual program boundary. If the program boundary frame count indicates the number of frames (at the current frame rate) to the frame containing the program boundary, then the exact position of the program boundary (relative to the start or end of the frame including the LPSM payload segment) (in number of samples) can be calculated as:

S＝(frame_counter*frame size)+offset，

Where S is the number of samples from the start or end of the frame containing the LPSM payload segment to the program boundary, "frame_counter" is the frame count indicated by the program boundary frame count, "frame size" is the number of samples per frame, and "offset" is the number of samples indicated by the offset value.

Some embodiments in which the rate of insertion of program boundary markers increases near actual program boundaries implement the following rule: if a frame is less than or equal to a certain number of frames ("Y") from the frame containing the program boundary, then the offset value is not included in the frame. Typically, Y = 32. For E-AC-3 encoders that implement this rule (Y = 32), the encoder does not insert an offset value in the last second of an audio program. In this case, the receiving device is responsible for maintaining a counter and thereby performing its own offset calculation (responsive to program boundary metadata, including the offset value, in frames of the encoded bitstream that are more than Y frames from the frame containing the program boundary).

For programs whose audio programs are known to be "frame aligned" with the video frames of the corresponding video program (such as contribution feeds typically using Dolby E encoded audio), it may not be necessary to include offset values in the encoded bitstream indicating the audio program. Therefore, offset values are typically not included in such encoded bitstreams.

With reference to FIG. 11 , consider below the case where encoded audio bitstreams are joined together to generate an embodiment of the inventive audio bitstream.

The bitstream at the top of Figure 11 (labeled "Scene 1") shows the entire first audio program (P1) including program boundary metadata (program boundary markers, F), followed by the entire second audio program (P2), also including program boundary metadata (program boundary markers, F). The program boundary markers in the end portion of the first program (some of which are shown in Figure 11) are the same or similar to those described with reference to Figure 8 and determine the location of the boundary between the two programs (i.e., the boundary at the beginning of the second program). The program boundary markers in the beginning portion of the second program (some of which are shown in Figure 11) are the same or similar to those described with reference to Figure 9 and also determine the location of the boundary. In a typical embodiment, the encoder or decoder implements a counter (calibrated by the markers in the first program) that counts down to the program boundary, and the same counter (calibrated by the markers in the second program) counts up from the same program boundary. As shown in the boundary counter diagram in Scene 1 of Figure 11, the down count of such a counter (calibrated by the markers in the first program) reaches 0 at the boundary, and the up count of the counter (calibrated by the markers in the second program) indicates the same location of the boundary.

The second bitstream starting at the top of FIG11 (labeled "Scene 2") indicates the entire first audio program (P1) including program boundary metadata (program boundary markers, F), followed by the entire second audio program (P2) that does not include program boundary metadata. The program boundary markers in the end portion of the first program (some of which are shown in FIG11) are the same or similar to those described with reference to FIG8 and determine the location of the boundary between the two programs (i.e., the boundary at the beginning of the second program), just as in Scenario 1. In a typical embodiment, the encoder or decoder implements a counter (calibrated by the markers in the first program) that counts down to the program boundary, and the same counter (without further calibration) continues counting up from the program boundary (as shown in the boundary counter diagram in Scenario 2 of FIG11).

The third bitstream (labeled "Scene 3") starting at the top of FIG. 11 shows a truncated first audio program (P1) including program boundary metadata (Program Boundary Markers, F), which has been spliced with an entire second audio program (P2) also including program boundary metadata (Program Boundary Markers, F). The splicing has removed the last "N" frames of the first program. The program boundary markers in the beginning of the second program (some of which are shown in FIG. 11) are the same or similar to those described with reference to FIG. 9, and they also determine the location of the boundary (the splice) between the truncated first program and the entire second program. In a typical embodiment, the encoder or decoder implements a counter (calibrated by the markers in the first program) that counts down to the end of the truncated first program, and the same counter (calibrated by the markers in the second program) counts up from the beginning of the second program. The beginning of the second program is the program boundary in Scene 3. As shown in the boundary counter diagram in scenario 3 of FIG11 , such a counter's down count (calibrated by the program boundary metadata in the first program) is reset (in response to the program boundary metadata in the second program) before reaching 0 (in response to the program boundary metadata in the first program). Thus, while the truncation of the first program (by splicing) prevents the counter from identifying the program boundary between the truncated first program and the start of the second program solely in response to the program boundary metadata in the first program (i.e., under calibration by the program boundary metadata in the first program), the program metadata in the second program nevertheless resets the counter such that the reset counter correctly indicates the position of the program boundary between the truncated first program and the start of the second program (as the position corresponding to the "0" count of the reset counter).

A fourth bitstream (labeled "Scene 4") indicates a truncated first audio program (P1) that includes program boundary metadata (program boundary markers, F) and a truncated second audio program (P2), which includes program boundary metadata (program boundary markers, F) and has been spliced with a portion (non-truncated portion) of the first audio program. The program boundary markers in the beginning portion of the entire (pre-truncated) second program (some of which are shown in FIG11) are the same as or similar to those described with reference to FIG9, and the program boundary markers in the ending portion of the entire (pre-truncated) first program (some of which are shown in FIG11) are the same as or similar to those described with reference to FIG8. The splicing has removed the last "N" frames of the first program (and thus removed some of the program boundary markers included therein before splicing) and the first "M" frames of the second program (and thus removed some of the program boundary markers included therein before splicing). In a typical embodiment, an encoder or decoder implements a counter (calibrated by a marker in the truncated first program) that counts down to the end of the truncated first program, and the same counter (calibrated by a marker in the truncated second program) that counts up from the beginning of the truncated second program. As shown in the boundary counter diagram in scenario 4 of FIG11 , such a counter (calibrated by program boundary metadata in the first program) that counts down is reset (in response to program boundary metadata in the second program) before reaching zero (in response to program boundary metadata in the first program). Thus, the truncation of the first program (by splicing) prevents the counter from identifying the program boundary between the truncated first program and the beginning of the truncated second program solely in response to program boundary metadata in the first program (i.e., under calibration by program boundary metadata in the first program). However, the reset counter does not correctly indicate the location of the program boundary between the end of the truncated first program and the beginning of the truncated second program. Consequently, the truncation of the two spliced bitstreams can prevent the boundary between them from being accurately determined.

Embodiments of the present invention can be implemented in hardware, firmware, or software, or a combination thereof (e.g., as a programmable logic array). Unless otherwise indicated, the algorithms or processes included as part of the present invention are not inherently related to any specific computer or other device. Specifically, various general-purpose machines can be used with programs written according to the teachings herein, or it may be more convenient to construct more specialized devices (e.g., integrated circuits) to perform the required method steps. Therefore, the present invention can be implemented in one or more computer programs executed on one or more programmable computer systems (e.g., the implementation of any one of the elements of FIG. 1 , or the encoder 100 (or its elements) of FIG. 2 , or the decoder 200 (or its elements) of FIG. 3 , or the post-processor 300 (or its elements) of FIG. 3 ), each programmable computer system including at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices in a known manner.

Each such program may be implemented in any desired computer language (including machine, assembly or high-level procedural, logical or object-oriented programming languages) to communicate with a computer system. In any case, the language may be a compiled or interpreted language.

For example, when implemented using a sequence of computer software instructions, the various functions and steps of the embodiments of the present invention may be implemented using a multi-threaded software instruction sequence running in appropriate digital signal processing hardware. In this case, the various devices, steps, and functions of the embodiments may correspond to a portion of the software instructions.

Each such computer program is preferably stored on or downloaded to a storage medium or device (such as solid-state memory or media, or magnetic or optical media) readable by a general or special purpose programmable computer to configure and operate the computer when the storage medium and device are read by the computer system to perform the processes described herein. The system of the present invention can also be implemented as a computer-readable storage medium configured with (i.e., storing) a computer program, wherein the storage medium so configured causes the computer system to operate in a specific and predetermined manner to perform the functions described herein.

A number of embodiments of the present invention have been described. However, it will be appreciated that various modifications may be made without departing from the spirit and scope of the present invention. In light of the above teachings, numerous modifications and variations of the present invention are possible. It will be appreciated that the present invention may be practiced other than as specifically described herein, within the scope of the appended claims.

Note:

1. An audio processing unit, comprising:

a buffer memory for storing at least one frame of a coded audio bitstream, wherein the coded audio bitstream comprises audio data and a metadata container, wherein the metadata container comprises a header, one or more metadata payloads, and protection data;

an audio decoder coupled to the buffer memory and configured to decode the audio data; and

a parser coupled to or integrated with the audio decoder for parsing the encoded audio bitstream,

The header includes a synchronization word that identifies the beginning of the metadata container, the one or more metadata payloads describe an audio program associated with the audio data, the protection data is located after the one or more metadata payloads, and the protection data can be used to verify the integrity of the metadata container and the one or more payloads within the metadata container.

2. The audio processing unit according to Note 1, wherein the metadata container is stored in an AC-3 or E-AC-3 reserved data space selected from the following: a skip field, an auxdata field, an addbsi field, and a combination thereof.

3. The audio processing unit according to note 1 or 2, wherein the one or more metadata payloads include metadata indicating at least one boundary between consecutive audio programs.

4. The audio processing unit of claim 1 or 2, wherein the one or more metadata payloads comprises a program loudness payload containing data indicating a measured loudness of an audio program.

5. The audio processing unit of claim 4, wherein the program loudness payload includes a field indicating whether the audio channel contains spoken conversation.

6. The audio processing unit according to supplementary note 4, wherein the program loudness payload includes a field indicating a loudness measurement method that has been used to generate the loudness data contained in the program loudness payload.

7. The audio processing unit of claim 4, wherein the program loudness payload includes a field indicating whether the loudness of the audio program has been corrected using session gating.

8. The audio processing unit of claim 4, wherein the program loudness payload includes a field indicating whether the loudness of the audio program has been corrected using an infinite look-ahead or file-based loudness correction process.

9. The audio processing unit of claim 4, wherein the program loudness payload includes a field indicating the overall loudness of the audio program in the absence of any gain adjustment attributable to dynamic range compression.

10. The audio processing unit of claim 4, wherein the program loudness payload includes a field indicating the overall loudness of the audio program in the absence of any gain adjustment attributable to session normalization.

11. The audio processing unit according to Supplement 4, wherein the audio processing unit is configured to perform adaptive loudness processing using the program loudness payload.

12. The audio processing unit according to any one of Notes 1 to 11, wherein the encoded audio bit stream is an AC-3 bit stream or an E-AC-3 bit stream.

13. The audio processing unit according to any one of Notes 4 to 11, wherein the audio processing unit is configured to extract the program loudness payload from the encoded audio bitstream and authenticate or verify the program loudness payload.

14. An audio processing unit according to any one of Notes 1 to 13, wherein the one or more metadata payloads each include a unique payload identifier, and the unique payload identifier is located at the beginning of each metadata payload.

15. The audio processing unit according to any one of Notes 1 to 13, wherein the synchronization word is a 16-bit synchronization word with a value of 0x5838.

16. A method for decoding a coded audio bitstream, the method comprising:

receiving a coded audio bitstream, the coded audio bitstream being segmented into one or more frames;

extracting audio data and a metadata container from the coded audio bitstream, the metadata container comprising a header followed by one or more metadata payloads followed by protection data; and

verifying the integrity of the container and the one or more metadata payloads by using the protection data,

Wherein the one or more metadata payloads include a program loudness payload containing data indicative of a measured loudness of an audio program associated with the audio data.

17. The method according to Note 16, wherein the coded bit stream is an AC-3 bit stream or an E-AC-3 bit stream.

18. The method according to Supplementary Note 16, further comprising:

Adaptive loudness processing is performed on the audio data extracted from the encoded audio bitstream using the program loudness payload.

19. The method according to Note 16, wherein the container is located in or extracted from an AC-3 or E-AC-3 reserved data space, and the AC-3 or E-AC-3 reserved data space is selected from: a skip field, an auxdata field, an addbsi field, and a combination thereof.

20. The method of claim 16, wherein the program loudness payload includes a field indicating whether the audio channel contains spoken conversation.

21. The method of claim 16, wherein the program loudness payload includes a field indicating a loudness measurement method that has been used to generate the loudness data contained in the program loudness payload.

22. The method of claim 16, wherein the program loudness payload includes a field indicating whether the loudness of the audio program has been corrected using session gating.

23. The method of claim 16, wherein the program loudness payload includes a field indicating whether the loudness of the audio program has been corrected using an infinite look-ahead or file-based loudness correction process.

24. The method of claim 16, wherein the program loudness payload includes a field indicating the overall loudness of the audio program in the absence of any gain adjustment attributable to dynamic range compression.

25. The method of claim 16, wherein the program loudness payload includes a field indicating the overall loudness of the audio program in the absence of any gain adjustment attributable to session normalization.

26. The method of claim 16, wherein the metadata container includes metadata indicating at least one boundary between consecutive audio programs.

27. The method according to Note 16, wherein the metadata container is stored in one or more skip fields or wasted bit segments of a frame.

Claims (29)

1. An audio processing unit, comprising: A buffer memory for storing at least one frame of an encoded audio bitstream, wherein the encoded audio bitstream includes audio data and metadata segments, wherein the metadata segments are separate and distinct data from the audio data, and the metadata segments include a header, one or more metadata payloads, and protection data; An audio decoder, coupled to the buffer memory, is used to decode the audio data; and A parser, coupled to or integrated with the audio decoder, is used to parse the encoded audio bitstream. The header includes a synchronization word that identifies the start of the metadata segment, followed by at least one identifier value. The one or more metadata payloads describe an audio program associated with the audio data. The protection data is located after the one or more metadata payloads and can be used to verify the integrity of the metadata segment and the one or more metadata payloads within the metadata segment. 2. The audio processing unit of claim 1, wherein the metadata segments are stored in an AC-3 or E-AC-3 reserved data space selected from the following: a skip field, an auxdata field, an addbsi field, and combinations thereof. 3. The audio processing unit of claim 1, wherein the one or more metadata payloads include metadata indicating at least one boundary between consecutive audio programs. 4. The audio processing unit of claim 1, wherein the one or more metadata payloads include a program loudness payload containing data indicating the measured loudness of an audio program. 5. The audio processing unit of claim 4, wherein the program loudness payload includes a domain indicating whether the audio channel contains spoken conversation. 6. The audio processing unit of claim 4, wherein the program loudness payload includes a domain indicating a loudness measurement method that has been used to generate the loudness data contained in the program loudness payload. 7. The audio processing unit of claim 4, wherein the program loudness payload includes a domain indicating whether the loudness of the audio program has been corrected using session gating. 8. The audio processing unit of claim 4, wherein the program loudness payload includes a domain indicating whether the loudness of the audio program has been corrected using a file-based loudness correction process. 9. The audio processing unit of claim 4, wherein the program loudness payload includes a domain indicating the overall loudness of the audio program without any gain adjustment attributable to dynamic range compression. 10. The audio processing unit of claim 4, wherein the program loudness payload includes a domain indicating the overall loudness of the audio program without any gain adjustment attributable to session normalization. 11. The audio processing unit of claim 4, wherein the audio processing unit is configured to perform adaptive loudness processing using the program loudness payload. 12. The audio processing unit according to any one of claims 1 to 11, wherein the encoded audio bitstream is an AC-3 bitstream or an E-AC-3 bitstream. 13. The audio processing unit according to any one of claims 4 to 11, wherein the audio processing unit is configured to extract the program loudness payload from the encoded audio bitstream and authenticate or verify the program loudness payload. 14. The audio processing unit of claim 1, wherein each of the one or more metadata payloads includes a unique payload identifier, and the unique payload identifier is located at the beginning of each metadata payload. 15. The audio processing unit according to claim 1, wherein the synchronization word is a 16-bit synchronization word with a value of 0x5838. 16. The audio processing unit according to any one of claims 1 to 11, wherein the at least one identifier value includes at least one of core element version, length and period, extended element count, and substream association value. 17. A method for decoding an encoded audio bitstream, the method comprising: Receive an encoded audio bitstream, which is segmented into one or more frames; Audio data and metadata segments are extracted from the encoded audio bitstream. The metadata segments are separate and distinct from the audio data. Each metadata segment includes a header, followed by one or more metadata payloads, followed by protection data. The header includes a synchronization word identifying the start of the metadata segment, followed by at least one identifier value. The integrity of the metadata segments and the one or more metadata payloads is verified by using the protection data. The one or more metadata payloads include a program loudness payload, which contains data indicating the measured loudness of an audio program associated with the audio data. 18. The method according to claim 17, wherein the encoded audio bitstream is an AC-3 bitstream or an E-AC-3 bitstream. 19. The method of claim 17, further comprising: Adaptive loudness processing is performed on the audio data extracted from the encoded audio bitstream using the program loudness payload. 20. The method of claim 17, wherein the metadata segment is located in or extracted from an AC-3 or E-AC-3 reserved data space, the AC-3 or E-AC-3 reserved data space being selected from: a skip field, an auxdata field, an addbsi field, and combinations thereof. 21. The method of claim 17, wherein the program loudness payload includes a domain indicating whether the audio channel contains spoken conversation. 22. The method of claim 17, wherein the program loudness payload includes a domain indicating a loudness measurement method that has been used to generate the loudness data contained in the program loudness payload. 23. The method of claim 17, wherein the program loudness payload includes a domain indicating whether the loudness of the audio program has been corrected using session gating. 24. The method of claim 17, wherein the program loudness payload includes a domain indicating whether the loudness of the audio program has been corrected using a document-based loudness correction process. 25. The method of claim 17, wherein the program loudness payload comprises a domain indicating the overall loudness of the audio program without any gain adjustment attributable to dynamic range compression. 26. The method of claim 17, wherein the program loudness payload comprises a domain indicating the overall loudness of the audio program without any gain adjustment attributable to session normalization. 27. The method of claim 17, wherein the metadata segmentation includes metadata indicating at least one boundary between consecutive audio programs. 28. The method of claim 17, wherein the metadata segment is stored in one or more skipped fields or wasted bit segments of the frame. 29. The method according to any one of claims 17 to 28, wherein the at least one identifier value includes at least one of core element version, length and period, extended element count, and substream association value.