HK1227166B - Post-encoding bitrate reduction of multiple object audio - Google Patents

Description

Post-encoding bit rate reduction for multi-object audio

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Patent Application No. 14/199,706, filed on March 6, 2014, and entitled “Post-Encoding Bit Rate Reduction for Multi-Object Audio,” which is incorporated herein by reference in its entirety.

Background Art

Audio compression techniques minimize the number of digital bits used to create a representation of an input audio signal. Uncompressed, high-quality digital audio signals often contain large amounts of data. The sheer size of these uncompressed signals often makes them undesirable or unsuitable for storage and transmission.

Compression techniques can be used to reduce the file size of digital signals. These compression techniques reduce the digital storage space required to store the audio signal for future playback or transmission. In addition, these techniques can be used to generate a credible representation of the audio signal with a reduced file size. This low-bit-rate version of the audio signal can then be transmitted at a low bit rate over a network channel with limited bandwidth. This compressed version of the audio signal is then decompressed after transmission to reconstruct a sonically acceptable representation of the input audio signal.

As a general rule, the quality of the reconstructed audio signal is inversely proportional to the number of bits used to encode the input audio signal. In other words, the fewer bits used to encode the audio signal, the greater the difference between the reconstructed audio signal and the input audio signal. Conventional audio compression techniques fix the bit rate during compression encoding, and therefore the level of audio quality. The bit rate is the number of bits used to encode the input audio signal per time period. No further reduction in the bit rate can be achieved without re-encoding the input audio signal at a lower bit rate, or decompressing the compressed audio signal and then re-compressing the decompressed signal at a lower bit rate. These conventional techniques are not "scalable" in situations where different applications require bit streams encoded at different bit rates.

One technique used to create scalable bitstreams is differential encoding. Differential encoding encodes the input audio signal into a high-bitrate bitstream composed of a subset of a lower-bitrate bitstream. The lower-bitrate bitstream is then used to construct the higher-bitrate bitstream. Differential encoding requires extensive analysis of the scaled bitstream and is computationally intensive. This computational intensity requires significant processing power to achieve real-time performance.

Another scalable coding technique uses multiple compression methods to create layered, scalable bitstreams. This approach uses a mix of compression techniques to cover a desired range of scalable bitrates. However, the limited scalability range and limited resolution make this layered approach unsuitable for many types of applications. For these reasons, the desired scenario of storing a single compressed audio bitstream and delivering content from this single bitstream at different bitrates is often difficult to implement.

Summary of the Invention

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Embodiments of a post-encoding bitrate reduction system and method generate one or more scaled compressed bitstreams from a single full file. The full file contains multiple audio object files that have been previously encoded individually. Thus, processing of the full file is performed after the audio object files have been encoded using the scalability characteristics of the full file.

The encoding process used for each encoded audio file is scalable, allowing bits to be shaved off from the frames of the encoded audio file to reduce the file size. This scalability allows data to be encoded at a specific bit rate, and then any percentage of the encoded data can be chopped off or discarded while still retaining the ability to correctly decode the encoded data. For example, if the data was encoded at a bit rate of Z, half of the frames can be chopped off or discarded to achieve half the bit rate (Z/2) and still be correctly decoded.

One instance where this fine-grained scalability and working from a single, encoded full file is valuable is when streaming to devices of varying bandwidths. For example, if there are multiple audio object files located on a server, embodiments of the present system and method will encode these audio object files individually at a certain high bit rate that the content provider wants to achieve. However, if this content is streamed to different and lower bandwidth devices, such as cell phones, cars, televisions, etc., the bit rate needs to be reduced. While working from a single, encoded full file, embodiments of the present system and method allow the bit rate to be adjusted for the bit rate of each individual device. Therefore, each delivery is tailored differently. But a single file is used to deliver different bit rate bitstreams. Furthermore, there is no need to re-encode the encoded audio object files.

Rather than re-encoding the audio object files, embodiments of the present system and method process a single version of the encoded full file and then scale down the bit rate. Furthermore, this bit rate scaling is accomplished without first decoding the full file back to its uncompressed form and then re-encoding the resulting uncompressed data at a different bit rate. This can all be accomplished without re-encoding the encoded audio object files.

While encoding and compression are expensive and computationally demanding processes, the post-encoding bitrate scaling of embodiments of the present systems and methods is a very lightweight process. This means that embodiments of the present systems and methods impose much smaller server requirements when compared to existing systems and methods that perform multiple encodings simultaneously to service each different channel bitrate.

Embodiments of the present system and method generate a scaled compressed bitstream from a single full file. A full file at a full bit rate is created by merging multiple individually encoded audio object files. An audio object is a source signal for a specific sound or combination of sounds. In some embodiments, the full file includes layered metadata corresponding to the encoded audio object files. The layered metadata contains priority information for each encoded audio object file relative to other encoded audio object files. For example, a dialogue audio object in a movie soundtrack may have a higher weight than a street noise audio object (during the same time period). In some embodiments, the entire time length of each encoded audio object file is used in the full file. This means that even if the encoded audio object file contains silent periods, they are still contained in the full file.

Each audio object file is segmented into data frames. A time period is selected and the data frame activities of the data frames of each encoded audio file in that specified time period are compared with each other. This provides a data frame activity comparison for all encoded audio files in the selected time period. Then based on the data frame activity comparison and in some cases layered metadata, bits are allocated to each data frame of the encoded audio object file during the selected time period from the available bit pool. This produces a bit allocation for the selected time period. In some embodiments, the layered metadata includes encoded audio object file priorities so that the files are ranked in order of priority or importance to the user. It should be noted that bits from the available bit pool are allocated to all data frames and all encoded audio object files in the selected time period. In other words, in a given time period, each audio object file and the frames therein receive bits, but some files receive more bits than other files based on their frame activity and other factors.

Measuring data frame activity can be based on any number of parameters available in the coded bitstream. For example, audio level, video activity, and other metrics of frame activity can be used to measure data frame activity. In addition, in some embodiments of the present system and method, data frame activity is measured on the encoder side and embedded in the bitstream, such as a number per frame. In other embodiments, decoded frames can be analyzed for frame activity.

In some embodiments, data frame activity is compared between frames. Typically, during a certain time period, some data frames will have more activity, while other data frames will have less activity. The data frame comparison includes selecting a time period and then measuring the data frame activity within the data frames during that time period. Each frame of the encoded audio object is examined during the selected time period. The data frame activity in each data frame is then compared with the other frames to obtain a data frame activity comparison. This comparison is a measure of the activity of a particular data frame relative to the other data frames during that time period.

Embodiments of the present system and method then reduce the full file by reducing the bits of the data frame according to the bit allocation to generate the reduced frames. This bit reduction uses the scalability of the full file and reduces the bits in the data frame in reverse ranking order. This results in multiple bits being allocated to the data frame in the bit allocation so that lower-ranked bits are reduced before higher-ranked bits. In some embodiments, the scalability of the frames within the encoded audio object file includes extracting tones from a frequency domain representation of the audio object file to obtain a time domain residual signal representing the audio object file with at least some tones removed. The extracted tones and the time domain residual signal are formatted into a plurality of data blocks, each data block comprising a plurality of bytes of data. The data blocks and the bits within the data blocks in the data frames of the encoded audio object file are sorted in order of psychoacoustic importance to obtain a ranking order from the most significant bit to the least significant bit.

Bit-reduced encoded audio object files are obtained from the pruned frames. The bit-reduced encoded audio object files are then multiplexed together and packed into a scaled compressed bitstream such that the scaled compressed bitstream has a target bitrate that is less than or equal to the full bitrate, in order to facilitate post-encoding bitrate reduction of a single full file.

The data frame activity measured for each data frame during the selected time period is compared to a silence threshold to determine whether a minimum amount of activity exists in any data frame. If the data frame activity for a particular data frame is less than or equal to the silence threshold, that data frame is designated as a silent data frame. Furthermore, the number of bits used to represent that data frame is maintained without reducing any bits. On the other hand, if the data frame activity for the particular data frame is greater than the silence threshold, the data frame activity is stored in a frame activity buffer. The available bit pool for the selected time period is determined by subtracting the bits used by silent data frames during the selected time period from the number of bits allocated to the selected time period.

In certain embodiments, the compressed bit stream of scaling can be transmitted through a network channel with a bit rate that is less than or equal to the target bit rate. The bit stream is received by a receiving device and then decompressed to obtain the audio object file of decoding. In some cases, the audio object file of decoding is mixed to create an audio object mix. The user can manually or automatically mix the decoded audio objects to create an audio object mix. In addition, the encoded audio object file in the layered metadata can be prioritized based on the spatial positioning in the audio object mix. In addition, two or more decoded audio object files can be interdependent, so that they carry out spatial masking based on their position in the mix.

Embodiments of the present system and method can also be used to obtain multiple scaled compressed bitstreams from a single full file. This is accomplished by utilizing a scalable bitstream encoder with fine-grained scalability to individually encode multiple audio object files at the full bit rate to obtain multiple encoded audio object files. This fine-grained scalability feature ranks the bits in each data frame of the encoded audio object file in order of psychoacoustic importance to human hearing. A full file is generated by merging the multiple independently encoded audio object files and the corresponding layered metadata. Each of the multiple encoded audio object files is persistent and exists for the entire duration of the full file.

A first scaled compressed bitstream at a first target bitrate is constructed from the full file and a second scaled compressed bitstream at a second target bitrate. This generates multiple scaled bitstreams at different target bitrates from a single full file without requiring any re-encoding of the multiple encoded audio object files. Furthermore, the first target bitrate and the second target bitrate are different from each other and both are less than the full bitrate. The first target bitrate is the maximum bitrate at which the first scaled compressed bitstream will be transmitted over a network channel.

As described above, the data frame activities of each of the data frames in the selected time period in the plurality of encoded audio files are compared with each other to obtain a data frame activity comparison. This data frame activity comparison and the first target bit rate are used to allocate bits to each of the data frames of the encoded audio object file based on the selected time period to obtain the bit allocation for the selected time period. The full file is reduced by reducing the bits of the data frames according to the bit allocation to achieve the first target bit rate and obtain encoded audio object files with reduced bits. These encoded audio object files with reduced bits are multiplexed together and packaged into a first scaled compressed bit stream at the first target bit rate. The first scaled compressed bit stream is transmitted to a receiving device at the first target bit rate and is decoded to obtain decoded audio objects. These decoded audio objects are mixed to create an audio object mixture.

Should be noted that, depending on specific embodiment, alternative embodiment is possible, and the steps and elements discussed herein can be changed, added or deleted. Without departing from the scope of the present invention, these alternative embodiments include alternative steps and alternative elements that can be used, and the structural changes that can be made.

BRIEF DESCRIPTION OF THE DRAWINGS

Referring now to the drawings, wherein like reference numerals designate corresponding parts throughout:

1 is a block diagram illustrating a general overview of an embodiment of a post-encoding bit rate reduction system and method.

2 is a block diagram illustrating a general overview of an embodiment of a post-encoding bitrate reduction system that obtains multiple scaled compressed bitstreams from a single full file.

3 is a block diagram showing details of a first embodiment of the post-encoding bit rate reduction system shown in FIGs. 1 and 2. In FIG.

4 is a block diagram showing details of a second embodiment of the post-encoding bit rate reduction system shown in FIGS. 1 and 2 .

FIG. 5 is a block diagram illustrating an exemplary embodiment of the scalable bitstream encoder shown in FIG. 1 and FIG. 4 .

6 is a block diagram illustrating an illustrative example of an embodiment of a post-encoding bit rate reduction system and method implemented in a networked environment.

FIG. 7 is a block diagram showing details of the frame-by-frame hierarchical bit allocation module shown in FIG. 3 .

8 is a flow chart illustrating the general operation of the embodiment of the post-encoding bit rate reduction system and method shown in FIGs. 1-7.

9 is a flow chart showing details of a first embodiment among the embodiments of the post-encoding bit rate reduction system and method shown in FIGs. 1-8.

10 illustrates an audio frame according to some embodiments of the post-encoding bit rate reduction systems and methods shown in FIGs. 1-9.

FIG. 11 shows an exemplary embodiment of a scalable frame of data generated by the scalable bitstream encoder shown in FIG. 1 .

FIG. 12 shows an exemplary embodiment of an example of dividing a full file into multiple frames and time periods.

FIG13 shows the details of the frames of the full file over the time period.

DETAILED DESCRIPTION

In the following description of embodiments of the post-encoding bit rate reduction system and method, reference is made to the accompanying drawings. These drawings show, by way of illustration, specific examples of how the embodiments of the post-encoding bit rate reduction system and method may be practiced. It should be understood that other embodiments may be used and structural changes may be made without departing from the scope of the claimed subject matter.

I. Introduction

An audio object is a source signal for a specific sound or combination of sounds. In some cases, an audio object also includes associated rendering metadata. Rendering metadata is data accompanying an audio object that indicates how the audio object should be rendered in the audio space during playback. This metadata can include multidimensional audio spatial information, positional information within the space, and ambient placement information.

Audio objects can represent various types of sound sources, such as individual instruments and vocals. Additionally, audio objects can include audio stems, which are sometimes called submixes, subgroups, or buses. An audio stem can also be a single track containing a group of audio content, such as a string section, a horn section, or street noise.

In traditional audio content production environments, audio objects are recorded. Professional audio engineers then mix the audio objects into a final master mix. The resulting mix is then delivered to the end user for playback. Generally speaking, this mix of audio objects is final, and the end user has little ability to make changes to it.

In contrast to traditional audio content production, multi-object audio (or its variations) allows the end user to mix audio objects after delivery. One way to control and specify this post-delivery mixing in a specific or suggested manner is by utilizing embedded metadata transmitted with the audio content. Another way is by providing user controls that allow the end user to directly manipulate and mix audio objects. Multi-object audio allows the end user to create unique and highly personalized audio presentations.

Multi-object audio can be stored as a file on a storage device and then transmitted in a bitstream when requested. The audio bitstream can be compressed or encoded to reduce the bit rate required to transmit the bitstream and the storage space required to store the file. Generally speaking, as an explanation and not a limitation, the compression of a bitstream means that less information is used to represent the bitstream. On the other hand, the encoding of a bitstream means that the bitstream is represented in another form, such as using symbols. However, encoding does not always compress the bitstream.

The encoded bitstream is transmitted over a bandwidth-limited network channel. Embodiments of the post-encoding bitrate reduction system and method utilize individually encoded audio objects and combine them with additional data to generate an encoded bitstream. When individually encoded audio objects are transmitted, the bandwidth of the encoded bitstream containing the encoded audio objects often exceeds the capacity of the network channel. In this case, a bitstream with a lower bitrate that is unsuitable for a particular application may be transmitted over the network channel. This can result in a reduction in the quality of the received audio data.

This degradation in quality is particularly problematic when multiple streams of audio data (such as multiple audio objects) are multiplexed for simultaneous or near-simultaneous transmission over a common network channel. This is because, in some cases, the bandwidth of each encoded audio object is proportionally degraded, which does not take into account the relative content of each audio object or group of audio objects. For example, one audio object may contain music, while another may contain street noise. Proportionately degrading the bandwidth of each audio object will likely have a more detrimental effect on music data than on noise data.

There may be times when an encoded bitstream is being transmitted over a network channel at a particular bitrate and the channel conditions will change. For example, the bandwidth of the channel may become constricted and a lower transmission bitrate may be required. In these situations, embodiments of the post-encoding bitrate reduction system and method can react to this change in network conditions by adjusting the scaling of the encoded bitstream bitrate. For example, when the bandwidth of the network channel becomes limited, the bitrate of the encoded bitstream is reduced so that transmission over the network channel can continue. Rather than re-encoding the audio objects, embodiments of the present system and method process a single version of the encoded bitstream and then scale the bitrate down. The resulting scaled bitstream can then be transmitted over the network channel at the reduced bitrate.

Scenarios may arise where it is desirable to transmit a single encoded bitstream at different bitrates over various network channels. This may occur, for example, when each network channel has different capacity and bandwidth, or when the bitstream is received by devices with different capabilities. In such cases, embodiments of the present system and method alleviate the need to encode or compress each channel separately. Instead, a single version of the encoded bitstream is used, and the bitrate is adjusted in response to the capacity of each channel.

The encoded bitstream can be processed in real time or substantially real time. Substantially real time can occur, for example, when the entire audio file or program is not accessible, such as during the broadcast of a live sporting event. In addition, the audio data can be processed offline and played back in real time. This occurs when the entire audio file or program, such as a video on demand application, is accessed. In the case of an encoded audio bitstream, it can include multiple audio objects, some or all of which include sound information and associated metadata. Such metadata can include, but is not limited to, position information, including position in space, velocity, trajectory, etc., sound wave characteristics, including divergence, radiation parameters, etc.

Each audio object or group of audio objects can be individually encoded using the same or different encoding techniques. Encoding can be performed on frames or blocks of the bitstream. A "frame" is a discrete segment of data in time used in the compression and encoding of audio signals. These data frames can be placed one after another in a serial sequence (like a movie film) to create a compressed audio bitstream. Each frame is of fixed size and represents a constant time interval. The frame size depends on the pulse code modulation (PCM) sampling rate and the encoding bit rate.

Each data frame is typically preceded by a header containing information about the following data. The header may be followed by error detection and correction data, while the remainder of the frame contains the audio data. The audio data consists of PCM data and amplitude (volume) information at a specific point in time. To produce a recognizable sound, tens of thousands of frames are played sequentially to generate a frequency.

Depending on the goals of a particular application, different frames (such as frames of the same object but occurring at different times) can be encoded using different bit rates based on, for example, the audio content of the frame. This approach is known as variable bit rate (VBR) encoding because the size of the encoded data changes over time. This approach can provide flexibility and improve the quality-to-bandwidth ratio of the encoded data. Alternatively, the frames can be encoded using the same bit rate. This approach is known as constant bit rate (CBR) encoding because the size of the encoded data is constant over time.

While it is possible to transmit audio objects independently in an unencoded and uncompressed manner to maintain separation, this is generally not feasible due to the large bandwidth requirements typically required to send the typically large files. Therefore, some form of audio compression and encoding is frequently used to facilitate the economical delivery of multi-object audio to end users. It has been found that encoding an audio signal containing audio objects to reduce its bit rate while still maintaining appropriate acoustic separation between the audio objects is difficult.

For example, some existing audio compression techniques for multiple audio objects are based on object dependencies. In particular, joint coding techniques frequently exploit audio object dependencies based on factors such as position, spatial masking, and frequency masking. However, one challenge with these joint coding techniques is that if the placement of the objects is unknown before delivery, it is difficult to predict spatial and frequency masking between objects.

Another type of existing audio compression technology is discrete object-based audio scene coding, which typically requires computationally expensive decoding and rendering systems, as well as high transmission or data storage rates to carry multiple audio objects individually. Another type of coding technology for delivering multi-object audio is multi-channel spatial audio coding. However, unlike discrete object-based audio scene coding, this spatial audio coding approach does not define separable audio objects. Therefore, spatial audio decoders cannot separate the contribution of each audio object in the downmix audio signal.

Yet another technique for encoding multiple audio objects is Spatial Audio Object Coding (SAOC). However, SAOC cannot completely separate concurrent audio objects in the downmix signal in the time and frequency domains. Consequently, extensive amplification or attenuation of objects by an SAOC decoder, as might be required by interactive user controls, can significantly degrade the audio quality of the reproduced scene.

It should be noted that for didactic purposes and ease of illustration, this document primarily refers to the use of audio data. However, the features described herein can also be applied to other forms of data, including video data and data containing time-series signals such as seismic and medical data. Furthermore, the features described herein can also be applied to virtually any type of data manipulation, such as data storage and data transmission.

II. System Overview

Embodiments of the post-encoding bit rate reduction system and method encode multiple audio object files individually and independently at a full bit rate. Embodiments of the system and method then merge these encoded audio object files along with their associated layered metadata to generate a full file. Multiple bitstreams can be obtained from a single full file. These multiple bitstreams are at a target bit rate that is less than or equal to the full bit rate. This bit rate change, known as scaling, ensures that optimal quality is maintained at each scaled bit rate. In addition, bit rate scaling is achieved without first decoding the full file back to its uncompressed form and then re-encoding the resulting uncompressed data at a different bit rate.

As explained in detail below, this scaling is partially implemented as follows. First, the audio object file is individually encoded using a scalable bitstream encoder that sorts the bits in each frame based on psychoacoustic importance. This scalable encoding also provides bit rate changes in a finely scaled manner by removing the bits in the frame. Secondly, at each frame time interval, the corresponding frame activity in each target file is considered. Then, based on the relative relationship between these frame activity measurements, embodiments of this system and method determine which frame payload of each compressed object file is retained. In other words, each frame payload of an audio object file is scaled based on its measured multimedia frame activity and its relationship to all frame activities in all other audio object files that will be multiplexed together.

Fig. 1 is a block diagram showing a general overview of an embodiment of a post-encoding bit rate reduction system 100. System 100 is located on a server computing device 110. An embodiment of system 100 receives an audio signal 120 as input. Audio signal 120 can comprise various types of content in various forms and types. In addition, audio signal 120 can be analog, digital, or other forms. Its type can be a signal that occurs in a repeated discrete amount, in a continuous stream, or some other type. The content of the input signal can be almost any content, including audio data, video data, or both. In some embodiments, audio signal 120 comprises a plurality of audio object files.

An embodiment of the system 100 includes a scalable bitstream encoder 130 that encodes each audio object file contained in the audio signal 120. It should be noted that the scalable bitstream encoder 130 can be a plurality of encoders. As shown in FIG1 , the output from the scalable bitstream encoder 130 is M independently encoded audio object files, including encoded audio object file (1) to encoded audio object file (M), where M is a non-zero positive integer. The encoded audio object files (1) to (M) are merged with the associated layered metadata to obtain a full file 140.

Whenever a bitstream having a specific target bitrate 160 is desired, the full file 140 is processed by the bit reduction module 150 to generate the desired bitstream. The bit reduction module 150 processes the full file 140 to generate a scaled compressed bitstream 170 having a bitrate less than or equal to the target bitrate 160. Once the scaled compressed bitstream 170 is generated, it can then be sent to the receiving device 180. The server computing device 110 communicates with other devices (such as the receiving device 180) via a network 185. The server computing device 110 accesses the network 185 using a first communication link 190 and the receiving device 180 accesses the network 185 using a second communication link 195. In this manner, the scaled compressed bitstream 170 can be requested by the receiving device 180 and sent to the receiving device 180.

1 , the network channel includes a first communication link 190, a network 185, and a second communication link 195. The network channel has a certain maximum bandwidth, which is communicated to the bit reduction module as a target bit rate 160. The scaled compressed bitstream 170 is transmitted over the network channel at or below the target bit rate so as not to exceed the maximum bandwidth of the channel.

As described above, in some cases, it is desirable to transmit a single full file at different bit rates over multiple network channels having multiple capabilities. FIG2 is a block diagram illustrating a general overview of an embodiment of a post-encoding bit rate reduction system 100 that obtains multiple scaled compressed bit streams from a single full file 140. As shown in FIG2 , full file 140 contains M encoded audio object files at a certain full bit rate. Specifically, FIG2 illustrates encoded audio object file (1) at the full bit rate, encoded audio object file (2) at the full bit rate, encoded audio object file (3) at the full bit rate, and any additional encoded audio object files (as indicated by the ellipsis) that include encoded audio object file (M) at the full bit rate.

The encoded audio object files (1) to the encoded audio object files (M) are independently encoded at full bit rate by the scalable bitstream encoder 130. The full bit rate is higher than the target bit rate 160. Typically, the target bit rate 160 is a bit rate used to transmit content over a network channel without exceeding the available bandwidth of the channel.

In some embodiments, full file 140 uses a high bit rate to encode the M independently encoded audio object files, making the size of full file 140 quite large. This can be problematic if the contents of full file 140 are to be transmitted over a network channel having limited bandwidth. As explained in detail below, to alleviate the difficulties associated with sending large files (such as full file 140) over a limited bandwidth channel, the encoded audio object files (1) through (M) are processed by a bit reduction module 150 to create multiple scaled encoded bitstreams from a single full file 140. This is achieved in part by removing blocks of sequential data in the data frames based on bit allocation.

Although a single target bit rate 160 is shown in FIG1 , in some cases, multiple target bit rates may exist. For example, it may be desirable to transmit the full file 140 via various network channels, each having a different bit rate. As shown in FIG2 , there are N target bit rates 200, where N is a positive, non-zero integer. Target bit rates 200 include target bit rate (1), target bit rate (2), and so on, up to target bit rate (N).

The bit reduction module 150 receives a target bit rate 160 in order to scale the bit rate of the full file 140 so that the resulting scaled encoded bitstream will best fit a particular limited bandwidth channel. The target bit rate 200 is typically sent from an Internet service provider (ISP) to inform embodiments of the system 100 and method about the bandwidth requirements and capabilities of the network channel over which the bitstream will be transmitted. The target bit rate 200 is less than or equal to the full bit rate.

In the exemplary embodiment of FIG2 , target bit rates 200 include N different target bit rates, where N is a non-zero positive integer that can be equal to, less than, or greater than M. Target bit rates 200 include target bit rate (1), target bit rate (2), additional target bit rates in some cases (as indicated by ellipses), and target bit rate (N). Typically, target bit rates 200 will be different from each other, but they may be similar in some embodiments. Furthermore, it should be noted that each of target bit rates 200 can be sent together or separately over time.

The scaled compressed bitstreams shown in FIG2 correspond to target bitrates 200. For example, target bitrate (1) is used to create scaled compressed bitstream (1) at target bitrate (1), target bitrate (2) is used to create scaled compressed bitstream (2) at target bitrate (2), in some cases additional scaled compressed bitstreams at target bitrates (as indicated by ellipses), and scaled encoded file (N), where N is the same non-zero positive integer as described above. In some embodiments, the target bitrates may be similar or identical, but typically the target bitrates differ from one another.

It should be noted that for didactic purposes, a specific number of encoded audio object files, target bit rates, and scaled compressed bitstreams are shown in FIG2 . However, there are cases where N=1, M=1, and a single scaled compressed bitstream is obtained from the full file 140. In other embodiments, N can be a larger number, where several scaled compressed bitstreams are obtained from the full file 140. Furthermore, the scaled compressed bitstreams can be created on the fly in response to a request from a client. Alternatively, the scaled compressed bitstreams can be created in advance and stored on a storage device.

III. System Details

The system details of the components of an embodiment of the post-encoding bit rate reduction system 100 will now be discussed. These components include a bit reduction module 150, a scalable bitstream encoder 130, and a frame-by-frame layered bit allocation module. Furthermore, the decoding of the scaled compressed bitstream 170 at the receiving device 180 will be discussed. It should be noted that only a few of the ways in which this system can be implemented are described in detail below. Many variations are possible.

FIG3 is a block diagram illustrating details of a first embodiment of the post-encoding bitrate reduction system 100 shown in FIG1 and FIG2. In this particular embodiment, the audio object files have been independently and individually encoded and are contained within a full file 140. The full file 140 is input to the embodiment of the post-encoding bitrate reduction system 100. The system 100 receives the individually encoded audio object files at the full bitrate 300 for further processing.

The individually encoded audio object files 300 are processed by the bit reduction module 150. As explained in detail below, the bit reduction module 150 reduces the number of bits used to represent the encoded audio object files in order to achieve the target bit rate 200. The bit reduction module 150 receives the individually encoded audio object files 300 and processes them using a frame-by-frame hierarchical bit allocation module 310. The module 310 reduces the number of bits in each frame based on a hierarchical bit allocation scheme. The output of the module 310 is a bit-reduced encoded audio object file 320.

The statistical multiplexer 330 takes the bit-reduced encoded audio object files 320 and merges them. In some embodiments, the statistical multiplexer 330 allocates channel capacity or bandwidth (measured in number of bits) to each of the encoded audio object files 1 through M based at least in part on a hierarchical bit allocation scheme. In some embodiments, the encoded audio object files are variable bit rate (VBR) encoded data and the statistical multiplexer 330 outputs constant bit rate (CBR) encoded data.

In some embodiments, statistical multiplexer 330 also illustrates other features of the encoded audio object files during bit allocation. For example, the audio content (e.g., music, speech, noise, etc.) of the encoded audio object files may be relevant. Encoded audio object files associated with simple collisions (such as noise) may require less bandwidth than objects associated with music tracks. As another example, the volume of an object can be used in bandwidth allocation (so that loud objects can benefit from more bit allocations). As another example, the frequency of the audio data associated with an object can also be used in bit allocation (so that broadband objects can benefit from more bit allocations).

The bitstream packer 340 then processes the multiplexed bit-reduced encoded audio object files 320 and packs them into frames and containers for transmission. The output of the bitstream packer 340 is a scaled compressed bitstream 170 containing variable-sized frame payloads. The scaled compressed bitstream 170 is at a bit rate less than or equal to the target bit rate 160.

In some embodiments, the audio object files have not yet been encoded. FIG4 is a block diagram illustrating details of a second embodiment of the post-encoding bitrate reduction system 100 shown in FIG1 and FIG2. Unencoded audio object files 400 are received by an embodiment of the system 100. The scalable bitstream encoder 130 independently encodes each audio object file 400 to obtain a full file 140.

The full file 140 is input to the bit reduction module 150. The frame-by-frame layered bit allocation module 310 processes the full file 140 to obtain bit-reduced encoded audio object files 320. The statistical multiplexer 330 takes the bit-reduced encoded audio object files 320 and merges them. The bitstream packager 340 then processes the multiplexed bit-reduced encoded audio object files 320 and packages them into frames and containers for transmission. The output of the bitstream packager 340 is a scaled compressed bitstream 170 containing variable-sized frame payloads. The scaled compressed bitstream 170 is at a bit rate less than or equal to the target bit rate 160.

FIG5 is a block diagram illustrating an exemplary embodiment of the scalable bitstream encoder 130 shown in FIG1 and 4. These embodiments of the scalable bitstream encoder 130 include a plurality of scalable bitstream encoders. In the exemplary embodiment shown in FIG5 , the scalable bitstream encoder 500 includes M encoders, namely, scalable bitstream encoder (1) to scalable bitstream encoder (M), where M is a non-zero positive integer. The input to the scalable bitstream encoder 500 is an audio signal 120. In these embodiments, the audio signal 120 includes a plurality of audio object files. In particular, the audio signal 120 includes M audio object files, including audio object file (1) to audio object file (M).

In the exemplary embodiment shown in FIG5 , scalable bitstream encoder 500 includes M encoders for each of M audio object files. Thus, there is an encoder for each audio object. However, in other embodiments, the number of scalable bitstream encoders may be less than the number of audio object files. Regardless of the number of scalable bitstream encoders, each of the plurality of encoders encodes each of the plurality of audio object files to obtain separately encoded object files 300, i.e., separately encoded audio object file (1) to separately encoded audio object file (M).

FIG6 is a block diagram illustrating an illustrative example of an embodiment of the post-encoding bit rate reduction system 100 and method implemented in a networked environment. In FIG6 , an embodiment of the system 100 and method is shown as being implemented on a computing device in the form of a media database server 600. The media database server 600 can be nearly any device that includes a processor, such as a desktop computer, a laptop computer, and an embedded device such as a mobile phone.

In some embodiments, the system 100 and method are stored on a media database server 600 as a cloud-based service for cross-application, cross-device access. The server 600 communicates with other devices via a network 185. In some embodiments, one of the other devices is a receiving device 180. The media database server 600 accesses the network 185 using a first communication link 190, and the receiving device 180 accesses the network 185 using a second communication link 195. In this manner, the media database server 600 and the receiving device 180 can communicate and transfer data between each other.

A full file 140 containing the encoded audio object files (1) to (M) is located on the media database server 600. The full file 140 is processed by the bit reduction module 150 to obtain a bit-reduced encoded audio object file 320. The bit-reduced encoded audio object file 320 is processed by a statistical multiplexer 330 and a bitstream packer 340 to generate a scaled compressed bitstream 170 at or below a target bitrate. The target bitrate is obtained from the target bitrate 200 shown in FIG.

In the embodiment shown in Figure 6, full file 140 is shown as being stored on media database server 600.As mentioned above, full file 140 comprises M coded audio object files that are independently encoded at full bit rate.As used in this document, bit rate is defined as the rate of the stream of binary digits through a communication link or channel.In other words, bit rate describes the rate at which a bit is transferred from one location to another.Bit rate is usually expressed as the number of bits per second.

Bit rate can indicate download speed, so that for a given bit rate, downloading a 3 gigabyte (Gb) file takes less time than downloading a 1 gigabyte file. Bit rate can also indicate the quality of a media file. As an example, an audio file compressed at 192 kilobits per second (Kbps) will generally have better or higher quality (in the form of greater dynamic range and clarity) than the same audio file compressed at 128 Kbps. This is because more bits are used to represent the data for playback per second. Therefore, the quality of a multimedia file is measured and indicated by its associated bit rate.

1-5 , the encoded audio object files are encoded at a full bit rate that is greater than any target bit rate 200. This means that the encoded audio object files of the full file 140 are of higher quality than the encoded audio object files at any target bit rate 200 contained in the scaled compressed bitstream 170.

The full file 140 and each encoded audio object file are input to an embodiment of the post-encoding bit rate reduction system 100 and method. As discussed in detail below, embodiments of the system 100 and method use frame-by-frame bit reduction to reduce the number of bits used to represent the encoded audio object files. This is achieved without having to re-encode the objects. This produces a bit-reduced file (not shown) containing multiple bit-reduced encoded audio object files 320. This means that at least some of the encoded audio object files of the full file 140 are represented as bit-reduced encoded audio object files 320 by a reduced number of bits compared to the full file 140. Each bit-reduced encoded audio object file 320 is then processed into a single signal by a statistical multiplexer 330 and packaged into a scaled compressed bitstream 170 by a bitstream wrapper 340. The scaled compressed bitstream 170 is at a bit rate less than or equal to the target bit rate. In addition, the target bit rate is less than the full bit rate.

The scaled compressed bitstream 170 is transmitted to the receiving device 180 via the network 185. This transmission typically occurs upon request by the receiving device 180, but many other situations may occur, including storing the scaled compressed bitstream 170 as a file on the media database server 600. The receiving device 180 may be any network-enabled computing device capable of storing or playing back the scaled compressed bitstream 170. Although the receiving device 180 is shown in FIG6 as residing on a different computing device than the embodiment of the post-encoding bitrate reduction system 100 and method, it should be noted that in some embodiments, they may reside on the same computing device (such as the media database server 600).

The receiving device 180 processes the received scaled compressed bitstream 170 by utilizing a demultiplexer 610 to separate the encoded audio object files into their individual components. As shown in FIG6 , these individual components include encoded audio object file (1), encoded audio object file (2), encoded audio object file (3), other encoded audio object files that exist (as indicated by ellipses), up to and including encoded audio object file (M). Each of these individual encoded audio object files is sent to a scalable bitstream decoder 620 that is capable of decoding the encoded audio object file. In some embodiments, the scalable bitstream decoder 630 includes a separate decoder for each encoded audio object file.

As shown in FIG6 , in some embodiments, the scalable bitstream decoder 620 includes a scalable decoder (1) (for decoding the encoded audio object file (1)), a scalable decoder (2) (for decoding the encoded audio object file (2)), a scalable decoder (3) (for decoding the encoded audio object file (3)), other scalable decoders as needed (as indicated by ellipses), and a scalable decoder (M) (for decoding the encoded audio object (file M)). It should be noted that in other embodiments, any number of scalable decoders can be used to decode the encoded audio object file.

The output of scalable bitstream decoder 620 is a plurality of decoded audio object files. Specifically, these multiple decoded audio object files include decoded audio object file (1), decoded audio object file (2), decoded audio object file (3), other decoded audio object files that may be needed (as indicated by ellipsis) and decoded audio object file (M). In this regard, the decoded audio object files can be stored for later use or used immediately. No matter which way, at least a portion of the decoded audio object files is input to mixing device 630. Usually, mixing device 630 is controlled by the user who mixes the decoded audio object files to generate personalized audio object mix 640. However, in other embodiments, the mixing of the decoded audio object files can be automatically processed by embodiments of system 100 and method. In other embodiments, audio object mix 640 is created by a third party supplier.

FIG7 is a block diagram illustrating details of the frame-by-frame hierarchical bit allocation module 310 shown in FIG3 . Module 310 receives individually encoded audio object files 300 that have been encoded at the full bit rate. For a particular time period, each frame of each encoded audio object file in that time period is examined across all encoded audio object files for the particular time period 700. Hierarchical information 710 is input to hierarchical module 720. Hierarchical information 710 includes data regarding how frames should be prioritized and, ultimately, how bits should be allocated within the frames.

The bits available in the bit pool 730 are used by an allocation module 740 to determine how many bits are available to allocate between the frames during the time period. Based on the layering information 710, the allocation module 740 allocates bits between the frames in that time period. The bits are allocated across the encoded audio object files, subbands, and frames based on the layering information 710.

The allocation module 740 generates such a bit allocation 750 indicating the number of bits allocated to each frame in a particular time segment. Based on the bit allocation, the reduction module 760 prunes bits from each frame as needed to conform to the bit allocation 750 for that particular frame. This produces pruned frames 770 for the given time segment. These pruned frames are merged to generate the bit-reduced encoded audio object file 320.

IV. Operation Overview

FIG8 is a flow chart illustrating the general operation of the embodiment of the post-encoding bit rate reduction system 100 and method shown in FIG1-7. The operation begins by inputting a plurality of audio object files (block 800). These audio object files may include source signals combined with associated rendering metadata and may represent various sound sources. These sound sources may include individual instruments and vocals, as well as combinations of sound sources, such as an audio object for a drum kit containing multiple tracks of individual components of the drum kit.

Next, embodiments of the system 100 and method independently and individually encode each audio object file (block 810). This encoding employs one or more scalable bitstream encoders with fine-grained scalability features. Examples of scalable bitstream encoders with fine-grained scalability features are described in U.S. Patent No. 7,333,929, entitled "Modular Scalable Compressed Audio Data Stream," filed on February 19, 2008, and U.S. Patent No. 7,548,853, entitled "Scalable Compressed Audio Bit Stream and Codec Using a Hierarchical Filterbank and Multichannel Joint Coding," filed on June 16, 2009.

The system 100 and method merge the multiple individually encoded audio files and any layered metadata 710 to generate the full file 140 (box 820). The full file 140 is encoded at the full bit rate. It should be emphasized that each audio object file is encoded separately in order to maintain the separation and isolation between the multiple audio object files.

Hierarchical metadata can contain at least three types of hierarchies or priorities. One or any combination of these types of priorities can be included in the hierarchical metadata. The first type of priority is bit priority within a frame. In these cases, bits are placed in order of psychoacoustic importance to human hearing. The second type of priority is frame priority within an audio object file. In these cases, the importance or priority of a frame is based on the activity of the frame. If the frame activity is high relative to other frames during the frame time interval, it will be ranked higher in the hierarchy than frames with lower activity.

The third type of priority is the audio object file priority within the full file. This includes both cross-object masking and user-defined priorities. In cross-object masking, a specific audio object file can be masked by another audio object file based on where the audio object is presented in the audio space. In this case, an audio object file will have a priority higher than the masked audio object file. In user-defined priorities, the user can define that an audio object file is more important to them than another audio object file. For example, for the audio soundtrack for a movie, the audio object file that comprises dialogue will have a higher importance to the user than the audio object file that comprises street noise or the audio object file that comprises background music.

Based on the desired target bit rate, the full file 140 is processed by the bit reduction module 150 to generate a scaled compressed bit stream 170. The scaled compressed bit stream is generated without any re-encoding. In addition, the scaled compressed bit stream is designed to be transmitted over a network channel at a bit rate equal to or less than the target bit rate.

The target bit rate is always less than the full bit rate. In addition, it should be noted that each audio object is independently encoded at a full bit rate that exceeds any target bit rate by 200. In cases where the target bit rate is unknown prior to encoding, each audio object is encoded at the maximum available bit rate or at a bit rate that exceeds the highest expected target bit rate that will be used during transmission.

To obtain a scalable compressed bitstream, embodiments of the system 100 and method divide the full file 140 into a series of frames. In some embodiments, each audio object file in the full file 140 exists throughout the entire duration of the file 140. This is true even if the audio object file contains silent periods during playback.

Referring again to FIG8 , embodiments of the system 100 and method select a frame time interval (or time period) and compare frame activity for frames during the selected time period (block 830). This frame time interval includes frames from each audio object. The frame-by-frame comparison of the selected time period generates a data frame activity comparison for that time period. Generally speaking, frame activity is a measure of how difficult it is to encode the audio in a frame. Frame activity can be determined in a variety of ways. In some embodiments, frame activity is based on a number of extracted tones and the resulting frame residual energy. Other embodiments calculate the entropy of the frame to derive the frame activity.

Bits are assigned or allocated from a pool of available bits in the frames for the selected time period (block 840). Bits are allocated based on data frame activity and hierarchical metadata. Once the bit allocation between frames for the selected time period is known, bits are distributed among the frames. Each frame is then brought into conformity with its bit allocation by trimming bits that exceed the bit allocation for that frame to obtain a trimmed frame (block 850). As explained in detail below, this bit reduction is performed in an ordered manner so that bits with the highest priority and importance are trimmed last.

This bit reduction of the plurality of pruned frames in the plurality of encoded audio object files generates bit-reduced encoded audio object files 320 (block 860). The bit-reduced encoded audio object files 320 are then multiplexed together (block 870). The system 100 and method then packages the multiplexed bit-reduced encoded audio object files 320 using the bitstream wrapper 340 to obtain a scaled compressed bitstream 170 at the target bit rate (block 880).

In some cases, the need to transmit an encoded audio object at several different bit rates may arise. For example, if a full file is stored on a media database server 600, it may be requested by several clients, each with different bandwidth requirements. In this case, multiple scaled compressed bit streams can be obtained from a single full file 140. Furthermore, each scaled compressed bit stream can be at a different target bit rate, where each target bit rate is less than the full bit rate. All of this can be achieved without re-encoding the encoded audio object file.

Embodiments of the system 100 and method can then transmit one or more of the scaled compressed bitstreams to a receiving device 180 at a bitrate equal to or less than the target bitrate. The receiving device 180 then demultiplexes the received scaled compressed bitstreams to obtain a plurality of bit-reduced encoded audio objects. The system 100 and method then decodes these bit-reduced encoded audio objects using at least one scalable bitrate decoder to obtain a plurality of decoded audio object files. The decoded audio object files can then be mixed by an end user, a content provider, or automatically to generate an audio object mix 640.

V. Operational Details

Embodiments of the post-encoding bitrate reduction system 100 and method include embodiments for handling silent periods in audio and embodiments for delivering a single, full file to various network channels of varying bandwidths. The silent period embodiment addresses situations where several audio object files may have significant periods of time during which the audio is silent or at a very low level relative to other audio object files. For example, audio content containing music may have long periods during which the vocal track is silent or at a very low level. When encoding these audio object files using a constant bitrate audio codec, a significant amount of data payload is wasted encoding the silent periods.

System 100 and method utilize the fine-grained scalability of each encoded audio object file to alleviate the waste of any data (or frame) payload during the silent period. This has achieved the reduction of the overall compressed data payload when not affecting the quality of the compressed audio of reconstruction. In certain embodiments, the encoded audio object file has a start and stop time. The start time represents the time point where silence begins and the stop time represents the time point where silence ends. In these cases, system 100 and method can mark the frame between the start and stop times as a null frame. This allows bits to be allocated to the frames of other audio object files during the time period.

In other scenarios, an on-the-fly bitrate reduction scheme may be needed in addition to or instead of the silent period embodiment. This may occur, for example, when a single high-quality encoded audio file or bitstream containing multiple audio object files is stored on a server that needs to simultaneously serve clients utilizing different connection bandwidths. The single full file to various bandwidth network channels embodiment uses the fine-grained scalability characteristics of the audio file or bitstream to reduce the overall bitrate of the encoded audio object file while attempting to maintain as much overall quality as possible.

The details of the operation of the embodiment of the system 100 and method will now be discussed. FIG9 is a flow chart illustrating details of a first embodiment of the embodiment of the post-encoding bit rate reduction system 100 and method shown in FIG1-8. The operation begins by inputting a full file containing a plurality of individually encoded audio object files (block 900). Each of the plurality of encoded audio object files is segmented into data frames (block 905).

The system 100 and method then selects a time segment at the beginning of the full file (block 910). This time segment ideally coincides with the temporal length of each frame. The selected time segment begins at the beginning of the full file. The method processes the data frames for the selected time segment and then continuously processes the remaining data frames by sequentially retrieving time segments in chronological order. In other words, the next time segment selected is one that is temporally adjacent to the previous time segment, and the methods described above and below are used to process the data frames during each time segment.

Next, the system 100 and method selects data frames for a plurality of encoded audio object files during a selected time period (block 915). Frame activity is measured for each data frame in the audio object files during the selected time period (block 920). As described above, various techniques can be used to measure frame activity.

For each data frame during the time period, the system 100 and method makes a determination as to whether the measured frame activity is greater than the silence threshold (block 925). If so, the frame activity for the data frame is stored in a frame activity buffer (block 930). If the measured frame activity is less than or equal to the silence threshold, the data frame is designated as a silent data frame (block 935). This designation means that the data frame has been reduced to a minimum payload, and the number of bits in that frame is used to indicate a data frame without further reduction. The silent data frame is then stored in a frame activity buffer (block 940).

The system 100 and method then compares the data frame activity stored in the frame activity buffer for each data frame in the selected time period with the other data frames for the current time period (945). This produces a data frame activity comparison. The system 100 and method then determines the number of available bits used by any silent frames during the time period (block 950). The number of available bits that can be allocated to the remaining data frames during the time period is then determined. This is accomplished by subtracting the bits used by any silent data frames from the number of bits allocated for use during the time period (block 955).

The bit allocation in the remaining data frames is performed by allocating available bits to the data frames from each encoded audio object file in the selected time period (box 960). This bit allocation is performed based on data frame activity comparison and layered metadata. Next, the ordered bits in the data frames are pruned to conform to the bit allocation (box 965). In other words, bits are removed from the data frames in such a way that the most important bits are removed last and the least important bits are removed first. This continues until only the number of bits allocated to that particular frame remains. The result is a pruned data frame.

These reduced data frames are stored (block 970) and a determination is made as to whether more time segments exist (block 975). If so, the next sequential time segment is selected (block 980). The process begins again by selecting data frames for the plurality of encoded audio object files at the new time segment (block 915). Otherwise, the reduced data frames are packaged into a scalable compressed bitstream (block 985).

V.A. Frames and Containers

As discussed above, in some embodiments, full file 140 includes multiple encoded audio object files. Some or all of these encoded audio object files may contain any combination of audio data, sound information, and associated metadata. In addition, in some embodiments, the encoded audio object files can be divided or partitioned into data frames. The use of data frames or frames can be efficient for streaming applications. Generally speaking, a "frame" is a discrete data segment created by a codec and used in encoding and decoding.

FIG10 illustrates an audio frame 1000 according to some embodiments of the post-encoding bitrate reduction system 100 and method shown in FIG1-9. Frame 1000 includes a frame header 1010, which can be configured to indicate the beginning of frame 1000, and a frame trailer 1020, which can be configured to indicate the end of frame 1000. Frame 1000 also includes one or more coded audio data blocks 1030 and corresponding metadata 1040. Metadata 1040 includes one or more segment headers 1050, which can be configured to indicate the beginning of a new metadata segment. The metadata 1040 can include layered metadata 710 used by layering module 720.

Ungrouped audio objects may be included as object segments 1060. Grouped audio objects 1070 may include group start and end blocks. These blocks may be configured to indicate the start and end of a new group. In addition, grouped audio objects 1070 may include one or more object segments. In some embodiments, frame 1000 may then be encapsulated into a container (such as MP4).

Generally, a "container" or package format is a metafile format whose specifications describe how the data elements and metadata that coexist in a computer file differ. A container refers to the way data is organized within a file, regardless of the encoding scheme used. In addition, a container is used to "wrap" multiple bitstreams together and synchronize frames to ensure that they are presented in the correct order. If necessary, the container can also be responsible for adding information to the streaming server so that the streaming server knows when to send which part of the file. As shown in Figure 10, frame 1000 can be packaged into container 1080. Examples of digital container formats that can be used for container 1080 include Transport Stream (TS), Material Exchange Format (MXF), Moving Pictures Expert Group, Part 14 (MP4), and the like.

V.B. Fine-grained bitstream scalability

The structure and order of elements placed in scaled compressed bitstream 170 provide wide bit range and fine grain scalability of bitstream 170. The structure and order allow bitstream 170 to be smoothly scaled by external mechanisms such as bit reduction module 150.

FIG11 illustrates an exemplary embodiment of a scalable frame of data generated by the scalable bitstream encoder 130 shown in FIG1 . It should be noted that one or more other types of audio compression codecs based on other decomposition rules can be used to provide fine-grained scalability to embodiments of the post-encoding bitrate reduction system 100 and method. In these cases, the other code will provide a different set of psychoacoustically relevant elements.

The scalable compressed bitstream 170 used in the example of FIG11 is composed of a plurality of Resource Interchange File Format (RIFF) data structures (referred to as "chunks"). It should be noted that this is an exemplary embodiment and that other types of data structures may be used. This RIFF file format, well known to those skilled in the art, allows identification of the type of data carried by a chunk and the amount of data carried by the chunk. It should be noted that any bitstream format that carries information about the amount and type of data carried in its defined bitstream data structures may be used with embodiments of the system 100 and method.

FIG11 shows the layout of a scalable bitrate frame block 1100, along with subblocks including a trellis 1 block 1105, a tone 1 block 1110, a tone 2 block 1115, a tone 3 block 1120, a tone 4 block 1125, and a tone 5 block 1130. Additionally, subblocks include a high-resolution trellis block 1135, a time sample 1 block 1140, and a time sample 2 block 1145. These blocks comprise the psychoacoustic data carried within the frame block 1100. While FIG11 depicts only the block identification (ID) and block length of the frame block 1100, the subblock ID and subblock length data are included in each subblock.

Figure 11 shows the order of blocks in a frame of a scalable bitstream. These blocks contain psychoacoustic audio elements generated by the scalable bitstream encoder 130 shown in Figure 1. In addition to the blocks being arranged by psychoacoustic importance, the audio elements within the blocks are also arranged by psychoacoustic importance.

The last block in the frame is an empty block 1150. This is used to fill in blocks when a constant or specific frame size is required. Therefore, empty block 1150 has no psychoacoustic relevance. As shown in FIG11 , the least important psychoacoustic block is time sample 2 block 1145. Conversely, the most important psychoacoustic block is grid 1 block 1105. In operation, if a scalable bitrate frame block 1100 needs to be downsized, data is removed starting with the least psychoacoustically relevant block at the end of the bitstream (time sample 2 block 1145) and moving up the psychoacoustic relevance ranking. This would move from right to left in FIG11 . This means that the most psychoacoustically relevant block (grid 1 block 1105), which has the highest possible quality in the scalable bitrate frame block 1100, is most likely not to be removed.

It should be noted that the highest target bit rate that can be supported by the bitstream (along with the highest audio quality) is defined at the time of encoding. However, the lowest bit rate after scaling can be defined by the audio quality level acceptable for the application. Each psychoacoustic element removed does not use the same number of bits. As an example, the scaling resolution for the exemplary embodiment shown in Figure 11 ranges from 1 bit for the elements of lowest psychoacoustic importance to 32 bits for those of highest psychoacoustic importance.

It should also be noted that the mechanism for scaling the bitstream does not require removing entire blocks at once. As previously noted, the audio elements within each block are arranged so that the most psychoacoustically important data is placed at the beginning of the scalable bitrate frame block 1100 (closest to the right side of Figure 11). For this reason, audio elements can be removed from the end of the block by the scaling mechanism, one element at a time, while maintaining the best possible audio quality as each element is removed from the scalable bitrate frame block 1100. This is what is meant by "fine-grained scalability."

The system 100 and method remove audio elements within the block as needed and then update the block length field for the specific block from which the audio element was removed. Furthermore, the system 100 and method also update the frame block length 1155 and frame checksum 1160. Utilizing the updated block length field for each scaled block, along with the updated frame block length 1155 and updated frame checksum information, a decoder can correctly process and decode the scaled bitstream. Furthermore, the system 100 and method can automatically generate a fixed data rate audio output signal even in the presence of blocks with missing audio elements or blocks that are completely missing from the bitstream. Furthermore, a frame block identifier (frame block ID 1165) is included in the scalable bitrate frame block 1100 for identification purposes. Furthermore, the frame block data 1170 includes (moving from right to left) the checksum 1160 to the empty block 1150.

V.C. bit allocation

The example of carrying out bit allocation between frames during time period will now be discussed. It should be noted that this is only one of several methods in which bit allocation can be performed. Figure 12 shows an exemplary embodiment of an example in which a full file 140 is divided into multiple frames and time periods. As shown in Figure 12, full file 140 is shown as being divided into multiple frames for multiple audio objects. The x-axis is the time axis and the y-axis is the encoded audio object file number. In this example, there are M encoded audio objects, where M is a positive non-zero integer. In addition, in this illustrative example, each encoded audio object file exists for the entire duration of full file 140.

Looking from left to right across the timeline, you can see that each encoded audio object (numbered 1 to M) is divided into X frames, where X is a positive, non-zero integer. Each box is represented by the label F _M,X , where F is the frame, M is the audio object file number, and X is the frame number. For example, frame F _1,2 represents the second frame of the encoded audio object file (1).

As shown in FIG12 , a time period 1200 corresponding to the length of a frame is defined for the full file 140. FIG13 shows details of the frames of the full file 140 within the time period 1200. Within each frame, its frequency components are shown in order of their relative importance to the quality of the full file 140. It should be noted that the x-axis is frequency (in kHz) and the y-axis represents the magnitude of a particular frequency (in decibels). For example, in _F1,1 , it can be seen that 7 kHz is the most important frequency component (in this example), followed by 6 kHz and 8 kHz, and so on. Thus, each frame of each audio object contains these ranked frequency components.

The target bit rate is used to determine the number of available bits for time period 1200. In some embodiments, psychoacoustics (such as masking curves) are used to distribute the available bits across the frequency components in a non-uniform manner. For example, the number of available bits for each of the 1, 19, and 20 kHz frequency components may be 64 bits, while 2048 bits may be available for each of the 7, 8, and 9 kHz frequency components. This is because, according to the masking curves, the human ear is most sensitive to the 7, 8, and 9 kHz frequency components, while the human ear is relatively insensitive to very low and very high components, i.e., frequency components at and below 1 kHz, and frequency components at 19 and 20 kHz. Although psychoacoustics are used to determine the distribution of available bits across the frequency range, it should be noted that many other different techniques may be used to distribute the available bits.

The embodiment of the post-encoding bitrate reduction system 100 and method then measures the frame activity of each frame for each encoded audio object file for the corresponding time period 1200. The frame activity of each data frame of each encoded audio object file in the time period 1200 is compared to each other. This is called a data frame activity comparison, which is the frame activity relative to other frames during the time period 1200.

In some embodiments, frames are assigned frame activity numbers. As an example, assume the number of audio object files is 10, such that the frame activity numbers range from 1 to 10. In this example, 10 represents the frame with the greatest frame activity during time period 1200, and 1 represents the frame with the least frame activity. It should be noted that many other techniques can be used to rank the frame activity within each frame during time period 1200. Based on the data frame activity comparison and the available bits from the bit pool, embodiments of the system 100 and method then allocate the available bits among the frames of the encoded audio object files for time period 1200.

The number of available bits and the data frame activity comparison are used by the system 100 and method to reduce bits in the frame as needed to match the allocated bits. The system 100 and method take advantage of the fine-grained scalability feature and the fact that bits are ranked in order of importance based on hierarchical metadata. For example, referring to FIG13 , for F _1,1 , assume that there are only enough allocated bits to represent the first four frequency components. This means that the frequency components at 7, 6, 8, and 3 kHz will be included in the bit-reduced encoded bitstream. The 5 kHz frequency component of F _1,1 and those lower in the ranking are discarded.

In some embodiments, the data frame activity comparison is weighted by the importance of the audio objects. This information is included in the layered metadata 710. As an example, assume that encoded audio object file #2 is important to the audio signal, which may occur if the audio is a movie soundtrack and encoded audio object file #2 is a dialogue track. Even if encoded audio object file #9 may have the highest relative frame activity ranking of 10 and encoded audio object file #2 has a ranking of 7, the ranking of encoded audio object file #2 can be increased to 10 due to the weighting of the audio object's importance. It should be noted that many variations of the above technique and other techniques can be used to allocate bits.

VI. Alternative Embodiments and Exemplary Operating Environments

Many other variations different from those described herein will be apparent from this document. For example, depending on the embodiment, certain actions, events, or functions of any method and algorithm described herein may be performed in a different order, may be added, merged, or completely excluded (such as, not all described actions or events are necessary for the practice of the method and algorithm). Moreover, in certain embodiments, actions or events may be performed simultaneously, such as by multithreading, interrupt processing, or multiple processors or processor cores or on other parallel architectures, rather than continuously. In addition, different tasks or processes may be performed by different machines and computing systems that can work together.

The various illustrative logic blocks, modules, methods, and algorithmic processes and sequences described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, and process actions have been generally described above with respect to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. The described functionality can be implemented in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of this document.

The various illustrative logical blocks and modules described in connection with the embodiments disclosed herein may be implemented or executed by a machine such as a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be a controller, a microcontroller or a state machine, a combination thereof, or the like. A processor may also be implemented as a computing device such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

The embodiments of the post-encoding bit rate reduction system 100 and method described herein operate in a variety of general-purpose or special-purpose computing system environments or configurations. Generally speaking, the computing environment can include any type of computer system, including but not limited to computer systems based on one or more microprocessors, mainframe computers, digital signal processors, portable computing devices, personal organizers, device controllers, computing engines in appliances, mobile phones, desktop computers, mobile computers, tablet computers, smartphones, and appliances with embedded computers, to name a few.

Such computing devices are typically found in devices having at least some minimum computing power, including but not limited to personal computers, server computers, handheld computing devices, laptop or mobile computers, communication devices such as cell phones and PDAs, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, audio or video media players, and the like. In some embodiments, the computing device will include one or more processors. Each processor may be a special-purpose microprocessor, such as a digital signal processor (DSP), a very long instruction word (VLIW), or other microcontroller, or may be a conventional central processing unit (CPU) having one or more processing cores, including cores based on a special-purpose graphics processing unit (GPU) in a multi-core CPU.

The processing actions of the methods, processes, or algorithms described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in any combination of the two. The software module may be contained in a computer-readable medium that can be accessed by a computing device. Computer-readable media includes both volatile and non-volatile media, and may be removable, non-removable, or some combination thereof. Computer-readable media is used to store information, such as computer-readable or computer-executable instructions, data structures, program modules, or other data. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

Computer storage media includes, but is not limited to, computer or machine readable media or storage devices, such as Blu-ray Discs (BDs), Digital Versatile Discs (DVDs), Compact Discs (CDs), floppy disks, tape drives, hard drives, optical drives, solid-state memory devices, RAM memory, ROM memory, EPROM memory, EEPROM memory, flash memory or other memory technology, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other device that can be used to store the desired information and can be accessed by one or more computing devices.

The software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, hard disk, removable disk, CD-ROM, or any other form of non-temporary computer readable storage medium, media, or physical computer storage known in the art. An exemplary storage medium can be coupled to a processor so that the processor can read information from the storage medium and write information thereto. In an alternative, the storage medium can be an integral part of the processor. The processor and the storage medium can reside in an application specific integrated circuit (ASIC). The ASIC can reside in a user terminal. Alternatively, the processor and the storage medium can reside in a user terminal as discrete components.

As used in this document, the phrase "non-transitory" means "persistent or long-lived." The phrase "non-transitory computer-readable medium" includes any and all computer-readable media, with the sole exception of transitory propagating signals. By way of example, and not limitation, this includes non-transitory computer-readable media such as register memory, processor cache, and random access memory (RAM).

The retention of information such as computer-readable or computer-executable instructions, data structures, program modules, etc. can also be achieved by encoding one or more modulated data signals, electromagnetic waves (such as carrier waves) or other transmission mechanisms or communication protocols using a variety of communication media, and includes any wired or wireless information delivery mechanism. Generally speaking, these communication media refer to signals whose one or more characteristics are set or changed in a manner that encodes information or instructions in the signal. For example, communication media include wired media, such as a wired network or a direct wire connection carrying one or more modulated data signals, and wireless media, such as acoustic, radio frequency (RF), infrared, laser, and other wireless media or multiple modulated data signals or electromagnetic waves for sending, receiving, or both. Any combination of the above should also be included within the scope of communication media.

Additionally, one or any combination of software, programs, computer program products embodying some or all or portions of the various embodiments of the post-encoding bit rate reduction system 100 and methods described herein may be stored, received, transmitted, or read from any desired combination of computer or machine-readable media or storage devices and communication media in the form of computer-executable instructions or other data structures.

The embodiments of the post-encoding bit rate reduction system 100 and method described herein may be further described in the general context of computer-executable instructions, such as program modules, executed by a computing device. Generally speaking, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The embodiments described herein may also be practiced in a distributed computing environment where tasks are performed by one or more remote processing devices, or in a cloud of one or more devices linked via one or more communication networks. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including media storage devices. Furthermore, the instructions described above may be implemented in part or in whole as hardware logic circuits, which may or may not include a processor.

Unless otherwise noted or otherwise understood in the context as used, conditional language used herein, such as "can," "might," "could," "for example," and the like, among others, is generally intended to convey that certain embodiments include, while other embodiments do not, certain features, elements, and/or states. Thus, such conditional language is generally not intended to imply that features, elements, and/or states are in any way required by one or more embodiments or that one or more embodiments must include logic for determining, with or without author input or prompting, whether such features, elements, and/or states are included or are to be performed in any particular embodiment. The terms "including," "having," and the like are synonymous and are used inclusively in an open-ended manner and do not exclude additional elements, features, actions, operations, and the like. Moreover, the term "or" is used in its inclusive sense (and not in its exclusive sense) such that when used, for example, to link a list of elements, the term "or" refers to one, some, or all of the elements in the list.

While the foregoing detailed description has shown, described, and pointed out novel features as applied to various embodiments, it will be understood that various omissions, substitutions, and changes in the form and details of the devices or algorithms shown may be made without departing from the spirit of the present disclosure. As will be appreciated, certain embodiments of the invention described herein may be embodied in a form that does not provide all of the features and benefits set forth herein, as some features may be used or practiced separately from other features.

Furthermore, although the subject matter has been described in language specific to structural features and methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims (21)

1. A method performed by one or more processing devices for generating a scaled compressed bitstream from a single full file, comprising: A full file with full bit rate is created by merging multiple audio object files that are encoded separately and independently using a scalable bitstream encoder that sorts the bits in each frame based on psychoacoustic importance, where the audio object is the source signal of a particular sound or combination of sounds. Each encoded audio object file is divided into data frames; The data frame activity of each encoded audio object file in each data frame of the selected time period is compared with all the frame activities of all other audio object files to be reused together to obtain a comparison of the data frame activities of all encoded audio object files in the selected time period. Based on data frame activity comparison, bits from the available bit pool are allocated to each data frame of the encoded audio object file during a selected time period to obtain bit allocation for the selected time period. The entire file is reduced by shrinking the bits of the data frame according to the bit allocation to generate a truncated frame; Obtain bit-reduced encoded audio object files from the cut frames, and reuse the bit-reduced encoded audio object files together; and The reused bit-reduced encoded audio object file is packaged into a scaled compressed bitstream, such that the scaled compressed bitstream has a target bit rate lower than or equal to the full bit rate, in order to facilitate bit rate reduction after encoding of a single full file. 2. The method of claim 1, further comprising: The full file is created by merging multiple separately encoded audio object files and their corresponding hierarchical metadata, wherein the hierarchical metadata contains priority information for each encoded audio object file relative to other encoded audio object files; and Based on data frame activity comparison and hierarchical metadata, bits from the available bit pool are allocated to each data frame to obtain bit allocations for a selected time period. 3. The method of claim 1, wherein the entire duration of each encoded audio object file is used to create the full file. 4. The method of claim 1, further comprising allocating bits from the available bit pool to all data frames and all audio encoded object files for the selected time period. 5. The method of claim 2, further comprising: Measure the data frame activity of each data frame within a selected time period; and The data frame activity of each data frame is compared with a silence threshold to determine whether there is a minimum amount of activity in any data frame. 6. The method of claim 5, further comprising: If the data frame activity of a specific data frame is less than or equal to the mute threshold, then that specific data frame is designated as a mute data frame with the minimum amount of activity, and the number of bits used to represent the mute data frame remains the same without any reduction in bits; and If the data frame activity of a specific data frame is greater than the mute threshold, the data frame activity is stored in the frame activity buffer. 7. The method of claim 6, further comprising determining a pool of available bits for the selected time period by subtracting bits used by silent data frames during the selected time period from a plurality of bits allocated to the selected time period. 8. The method of claim 2, further comprising reducing bits of the data frame in reverse rank order to achieve multiple bits allocated to the data frame in bit allocation such that lower-ranked bits are reduced before higher-ranked bits. 9. The method of claim 8, further comprising: Extract tones from the frequency domain representation of the audio object file to obtain a time domain residual signal representing the audio object file in which at least some tones have been removed; The extracted pitch and time-domain residual signals are formatted into multiple data blocks, each containing multiple bytes of data; and The data blocks and bits within the data blocks of the audio object file are sorted according to psychoacoustic importance to obtain a ranking order from the most important bit to the least important bit. 10. The method of claim 2, further comprising: A scaled compressed bitstream is transmitted via a network channel at a bit rate less than or equal to the target bit rate; and Receive and decode scaled compressed bitstreams to obtain decoded audio object files. 11. The method of claim 10, further comprising mixing decoded audio object files to create an audio object mix, wherein two or more of the decoded audio object files depend on each other to perform spatial masking based on their position in the mix. 12. The method of claim 2, further comprising prioritizing encoded audio object files in hierarchical metadata based on spatial positioning in audio object mixing. 13. The method of claim 2, further comprising prioritizing audio object files encoded based on the importance of each audio object file to the user in the audio object mix. 14. A method for obtaining multiple scaled compressed bitstreams from a single full file, comprising: Multiple encoded audio object files are obtained by individually and independently encoding multiple audio object files at full bit rate using a scalable bitstream encoder with fine-grained scalability. The encoder ranks the bits in each data frame of the encoded audio object files in order of psychoacoustic importance to human hearing. A full file with full bitrate is generated by merging multiple independently encoded audio object files and their corresponding hierarchical metadata. Construct a first-scaled compressed bitstream with a first target bit rate from the entire file; Constructing a second scaled compressed bitstream with a second target bit rate from the full file, such that multiple scaled bitstreams with different target bit rates are obtained from a single full file without any re-encoding of the multiple encoded audio object files; The data frame activity of each encoded audio object file in each data frame of a selected time period is compared with all the frame activities of all other audio object files to be reused together to obtain a data frame activity comparison. Based on the data frame activity comparison and the first target bit rate, bits are allocated to each data frame of the encoded audio object file in a selected time period to obtain the bit allocation for the selected time period. The entire file is reduced in size by decreasing the number of bits in the data frame according to the bit allocation, to achieve a first target bit rate and obtain a bit-reduced encoded audio object file; and The bit-reduced encoded audio object files are reused and packaged together into a first-scaled compressed bitstream with a first target bit rate; The first target bit rate and the second target bit rate are different from each other, and both are smaller than the full bit rate. 15. The method of claim 14, wherein the first target bit rate is the maximum bit rate at which the first scaled compressed bit stream will be transmitted. 16. The method of claim 15, wherein each of the plurality of encoded audio object files is persistent and exists throughout the entire duration of the entire file. 17. The method of claim 14, further comprising: The first scaled compressed bit stream is transmitted to the receiving device at the first target bit rate; and Decode the first scaled compressed bitstream to obtain the decoded audio object. 18. The method of claim 17, further comprising mixing decoded audio objects to create an audio object mix. 19. A bit rate reduction system after encoding, comprising: The full file contains audio object files that have been encoded at full bit rate and combined with the corresponding hierarchical metadata to form the full file, each encoded separately and independently using a scalable bitstream encoder that sorts the bits in each frame based on psychoacoustic importance. The bit reduction module is used to reduce multiple bits allocated to the data frames of the encoded audio object files based on a comparison of the data frame activity of each data frame in a selected time period of each audio object file with all the frame activities of all other audio object files to be reused together, so as to obtain bit-reduced encoded audio objects. Bitstream wrappers are used to arrange data frames of bit-reduced encoded audio objects in a container for transmission over computer networks; and A multiplexer is used to combine containers containing bit-reduced encoded audio to generate a scaled compressed bitstream at a target bit rate, where the target bit rate is less than the full bit rate. 20. An audio signal receiving system, comprising: A scaled compressed bitstream received over a network at a target bit rate, the bitstream comprising multiple bit-reduced encoded audio object files individually and independently encoded using a scalable bitstream encoder residing on a computing device, the scalable bitstream encoder ordering bits in each frame based on psychoacoustic importance, and the bitstream causing bits in data frames of the full file encoded at full bit rate to be reduced based on data frame activity comparison and corresponding hierarchical metadata, wherein the target bit rate is less than or equal to the full bit rate, and wherein the data frame activity comparison comprises: comparing the data frame activity of each encoded audio object file in each data frame over a selected time period with all frame activities in all other audio object files to be multiplexed together to obtain a data frame activity comparison of all encoded audio object files over the selected time period; A demultiplexer is used to separate a scaled, compressed bitstream into multiple encoded audio object files; and A scalable bitstream decoder that decodes encoded audio objects to obtain decoded audio objects. 21. The audio signal receiving system of claim 20, further comprising a mixing device for mixing decoded audio object files and generating an audio object mix.