WO2007077280A1 - System and method for the rapid perceptual quantification and scalable coding of audio signals - Google Patents

Abstract

The invention relates to a system for the perceptual coding of signals, which is intended for the scalable compression/decompression of musical audio. According to the invention, samples of a signal are obtained over time and a representation of the signal in the spectral range is obtained by means of a mathematical transformation. A psychoacoustic model is used to establish the amount of hearing-insensitive quantification noise in each sample. Favour is given to a pre-determined subassembly of quantification values which are easy to compress, such that said values are more probable than the remaining values. The binary words resulting from the quantification can be coded directly using any suitable binary compression method or can be divided into bit-planes which are compressed separately. The inventive system also employs frequency-predictive coding techniques, making use of the correlation that exists between corresponding coefficients in frequencies that are similar over time and the use of a multi-resolution analysis which is adapted to the most important frequencies present in the signal.

Description

 System and method for rapid perceptual quantification and scalable coding of audio signals

a) TECHNICAL SECTOR

The present invention fits within the perceptual encoders, for the storage or transmission of digital audio files or images efficiently, without noticeable losses. It focuses on audio compression, but the principles on which it is based can be extended directly to image compression.

b) STATE OF THE TECHNIQUE

The present invention is based on the same principles that have been used, for example, in the different ISO / MPEG standards (MP3, AAC, etc. see references [1] and [2]) for audio compression, well known. This includes: sampling and digitization of short audio segments over time, transformation of these into the spectral domain using FFT, DCT, MDCT, Wavelet, etc., calculation of auditory masking thresholds and quantification and coding of samples in the spectral domain depending on these thresholds.

In most current encoders, the quantification and coding process is performed in such a way that the resulting file can be decoded at a certain bit rate. This has its advantages in certain areas, for example if the decoding is done by reading the file from an optical medium (eg a CD whose reading speed) or from a network link at a constant transmission rate (for example, a telephone link). However, decoding at a constant bit rate loses its meaning when the data access speed is not constant and / or this is much higher than that required to decode the data in real time (eg a hard disk , or a CD or DVD reading a song encoded in MP3) or if the transmission is through a packet switching network (eg Internet), the latter being precisely the most common cases. The disadvantage of constant rate coding is that it makes it difficult for said coding to be optimal for the signal to be encoded. For example, if a 128 Kbps rate is set, we would be wasting bits in the simplest parts (eg silences or simple low-frequency sounds) and vice versa, in more complex parts where more bits might be needed for an encoding transparent (that is, with inaudible distortion), quality would have to be sacrificed. There are techniques to solve this problem in part, such as bit reserve or coding using a bitrate variable, but inevitably complicate both the encoder and the decoder and are also far from being optimal.

On the other hand, the quantification and coding process of the most common current standards is based on OCF (Optimum Coding in the Frequency Domairi) [3], so that the spectral coefficients of each critical band are quantified and encoded by Huffman coding following an iterative process implemented through two nested loops. The internal loop is about assigning the largest possible quantification step to each critical band so that the quantization noise remains below the masking threshold in said critical band. The external loop verifies that, once Huffman coding of the coefficients in each of the critical bands has been performed, the maximum number of bits allowed for each audio segment is not exceeded (set by the bit rate: a typical value is 128 Kbps for the MP3). Otherwise, the cycle begins again with a larger quantification step.

This process is not only very computationally expensive but it also makes the scalability of decoding very difficult (that is, that a hierarchy can be defined in the coded bits so that by transmitting only "the most important bits" we can decode a signal similar to the original , although more or less degraded depending on the number of bits that are dispensed with) since once the Huffman code words have been assigned, it is no longer possible to establish which are the more or less significant bits of the coefficients.

To avoid this, some scalable coding systems have been recently developed, but they usually do not make such an efficient use of the

2 SUBSTITUTE SHEET (RULE 26) auditory masking as non-scalable algorithms. Moreover, the problem of defining said hierarchy of most important bits is not trivial, since when the degradation of the signal is inevitable, the amount of distortion introduced is a strongly subjective measure.

On the other hand, although the musical signals have an obvious redundancy both temporal (rhythms and harmonies that are repeated) and spectral (the structure of the musical notes itself is basically a fundamental tone and several harmonics at multiple frequencies of the of this), a commercial encoder that exploits these characteristics has not yet been successfully developed, with the possible exception of the MPEG-2 AAC, which includes some predictability based on the samples close to the one to be encoded.

In the present invention, a solution to these described problems is proposed: optimal coding not dependent on a bit reading rate, but exclusively on the characteristics of the signal to be encoded and on a quality criterion; scalability based on the auditory model and a hierarchy based on more or less unpleasant perception models of the distortion introduced; spectral prediction, and even prediction based on physical modeling and / or instrument samples, so that it would be necessary to quantify perceptually and encode only the prediction residue, plus the parameters of said model and / or samples.

c) DETAILED DESCRIPTION OF THE INVENTION

The system proposed in the present invention shares many of the characteristics of the best known audio perceptual compressors. However, the system proposed here presents two fundamental novel characteristics: on the one hand, the system of quantification of the spectral values, attending to the model of auditory masking, and on the other the ability to perform an efficient predictive coding in the spectral domain . The steps to follow for the encoding of an audio signal are described below, with special emphasis on the quantification method:

3 SUBSTITUTE SHEET (RULE 26) Segmentation of the temporal signal and spectrogram generation of each part

100

The time domain signal, typically coded with PCM, is segmented into computationally manageable audio parts, i.e. of a few seconds. This segmentation can be done automatically or manually, according to formal criteria, choosing musically homogeneous parts, since this

105 favors the compression capacity of the algorithm.

Each audio part is further subdivided into segments of suitable length for transformation to a suitable domain to apply a model of auditory masking. The most common transforms are: FFT, DCT, MDCT, 110 Wavelet, etc.

The philosophy of this system is to generate, from the transformed segments, the spectrogram image. This will be a grayscale image of a variable size, depending on the length of the corresponding audio part. For example, a

115 spectrogram generated from the 256-point DCT of the successive segments of about 8 seconds of audio would have a size of 256x1380 points. It is interesting to use DCT, MDCT (or Wavelet transformations with real spectral coefficients), separating the coefficients into module + signs. In this way, we obtain spectral values similar to those of the FFT module, but whose phase (signs) is much more

120 easy to encode that the phase of the FFT (each sign, that is, each phase value, can be encoded with a single bit), so that any binary compressor obtains compression ratios of the order of 1:20 for these signs . With this in mind, and given that the most usual masking models dispense with the phase, the rest of the algorithm will focus on compressing the corresponding transform module.

125 In this regard, it should be noted that the Wavelet transform can also be advantageous, provided that a suitable auditory masking model is available for it.

An image of the same size as spectrogram 130 is elaborated in a similar manner where the corresponding value of the threshold of

4 SUBSTITUTE SHEET (RULE 26) masking For this, one of the psychoacoustic models proposed in the MPEG / Audio standard [1], or any other, can be used. If more than 256 points are used for the model, it would be necessary to tithe properly, although the most reasonable (and experimentally good result) would be 135 to use the same transform (eg MDCT with 50% overlapping ) with the same number of points (eg 256) both for the calculation of the masking threshold and for the spectrogram of the signal itself.

The MDCT has the advantage over the DCT of compacting coefficients

140 is somewhat greater when a poisoning is performed prior to the time domain signal, in addition to favoring the temporal redundancy between the columns of the image

(interesting in predictive coding) thanks to temporary overlapping. However, it has the disadvantage that in extreme cases, even with only 256-point MDCT could be perceived pre-echo (the 256-point MDCT comes from the overlap of two windows

145 of 512 points, which is already a size large enough to notice the effects of the pre-echo). You can use any of the existing techniques {Temporal Noise

Shaping, for example) to avoid pre-echo.

The arrangement of the image of the successive masking thresholds

150 simultaneous for each segment allows to apply temporary masking models directly: for example, choosing a suitable dispersion function

(typically a step function with decreasing slope from this and of the appropriate duration -200 ms would correspond to 20 or 30 columns of the spectrogram, for example-) and convolving it directly with each row of the image. The threshold of

The resulting masking would be the maximum between the original and the result of the convolution. You can even define a different scattering function for each frequency (row).

Once you have the image of the spectrogram of the signal and the threshold of 160 masking, the next step is to quantify the values of the signal so that the quantization noise at each point is below the corresponding threshold.

5 SUBSTITUTE SHEET (RULE 26) Quantification system

165 This is one of the key points of the system. Typically, we would use 16 bits to quantify the spectral values (experimentally it is found to be sufficient, although in some extreme cases more bits could be used without substantially changing the algorithm). Therefore, we would have, in principle 65536 possible quantification values. The key to this algorithm is to define a series of "privileged values" and

170 take advantage of the quantization noise margin allowed by the masking threshold to try to make the final value of the quantized signal at each point one of these privileged values. One way to assign privileged values would be to subdivide the dynamic range using increasingly smaller steps, and in each iteration try to code as many coefficients as possible, provided that the

175 masking threshold allows in each case. If the initial step is the value of the complete dynamic range, (for example, using 16 bits of quantification, 65535) and in subsequent iterations the step is divided by 2, the privileged values would be those corresponding to the multiples of the step in each iteration, that is, the powers of 2 and their multiples.

180

In any case, any other set of privileged values could be used, and even adaptively chosen, after signal analysis. In the case of the selection of the powers of two and their multiples as a set of privileged values, a possible quantification algorithm would be the following:

185 a) Normalize and scale the total spectral values, for example using the dynamic range allowed by 16 quantization bits, so that the maximum corresponds to 65535 and the minimum to 0. b) Perform the same operation with the masking thresholds 190 corresponding to each spectral value. c) Define the total dynamic range as initial quantification step, in this case 65535. d) Divide each spectral value of the signal by said step and separate the quotient and the rest. If the rest is greater than half the step of

195 quantification, a new remainder is defined as: remainder = step-rest, and

6 SUBSTITUTE SHEET (RULE 26) increase the quotient in one unit (with the latter what we do is quantify to the nearest step, above or below), e) That remainder will be the quantification error in case we use said step to quantify. Therefore, if for a given sample this remainder is 200 less than the masking threshold, we quantify that sample to the value that results from multiplying the quotient by the current step. Otherwise, divide the step by 2 and return to step d) until all samples are quantified.

205

In this way, what we achieve is to try to get the samples to take the value of the steps (which will be the powers of two), or multiples thereof, as long as the masking threshold allows it. The privileged values will therefore be the powers of 2 and their multiples, the latter ordered hierarchically from the

210 step larger to smaller. The choice of this set of privileged values has numerous advantages:

1. The expression of the powers of 2 in binary language is of type 0 ... 010 ... 0, that is, only 1 surrounded by zeros. These types of words are ideal for any

215 binary compressor, since the symbol 0 is much more likely than 1, and therefore there will be many zeros in a row, which can be exploited to improve compression.

2. The powers of 2 respond reasonably well to the dynamics of the audio signals 220: the distance between two of them is smaller for small values and greater for large ones, which are less likely.

The multiples of the powers of two also have an interesting structure, for example:

225 256 (00100000000)

256 x 3 = 768 (01100000000) 256 x 5 = 1280 (10100000000) 256 x 6 = 1536 (11000000000) 256 x 7 = 1792 (11100000000)

7 SUBSTITUTE SHEET (RULE 26) 230

In all of them we find that the least significant 8 bits are 0.

3. The divisions that are made in step d) are always by powers of 2, so they can be implemented simply by binary operations of

235 offset: the most significant bits are the quotient, and the least significant are the rest.

4. The expression of the powers of 2 in floating point format, taking base 2, always presents mantissa = l, so in these cases you just have to code the

240 exponent

In practice, for a transparent coding of any audio signal, including complex signals of pop-rock music, about 100 values (of the 65535 initially possible) are enough, of which only about 15 are the ones that are actually

245 take more than 90% of the time, and of these 15 the majority are powers of 2. This can give an idea of how efficient Huffman encoding, arithmetic, or encoding by bit plane separation can result in this case. , or algorithms such as ZIP or RAR, as is the case indeed: for a given quality compression ratios are obtained higher than those obtained using

250 the MP3 compression system proposed by the Fraunhofer institute. It seems logical to think that if a specific binary compressor was designed for this type of data, the compression gain could be even greater.

This algorithm can be used in a general way for other sets of privileged values 255 without more than applying it on the preprocessed coefficients properly so that a unique correspondence between the desired privileged values and the powers of two and their multiples is established.

As interesting as the high compressibility of the values thus quantified is

260 the possibility of scaling these values, that is, selecting a subset of them so that the signal can be decoded, even at the expense of inevitably introducing a certain amount of distortion. As we have the values quantified, this could be done directly by simply eliminating the least bits

8 SUBSTITUTE SHEET (RULE 26) significant. The scaling process thus described could be carried out in several ways, 265 among which the following two stand out for their simplicity:

1. Or starting from compressed samples (with RAR, for example), decompressing them, eliminating the least significant bits, and compressing again. In this case the scaling is not direct, but it will always be a lot

270 faster than "recompressing" the signal from the beginning.

2. Or by previously separating the samples into bit planes (eg 16 planes) and compressing each one separately. For this case, the PNG algorithm is especially efficient (each plane is still an image

275 binary) In this way, we can choose the most significant bits directly, without compressing or decompressing.

It should be noted that, again, thanks to the structure of the quantized binary words, a good part of the bit planes are simply arrays of zeros or

280 with very few ones, so that compression is still very efficient. Fine grain scalability can easily be achieved by performing said separation in bit planes, subdividing each plane into segments of appropriate size and compressing each segment separately. These segments would be sorted in order of perceptual importance: typically, from the segments

285 corresponding to the most significant bits of the lowest frequencies to the least significant bits of the segments of the highest frequencies.

This division into bit planes and sections thereof arranged according to some predetermined criteria endows the scalability system without in the process of

If the number of decoding bits is chosen, no computational cost is added, since the sections are already compressed and sorted. The scalability of fine grain, in practice, means that any decoding bit rate can be selected between, for example, 8 Kbps and 192 Kbps in 1 Kbps jumps. In the system proposed here it would be easy to achieve no only the scalability of fine grain,

295 but decoding at a constant bit rate: it would be enough to transmit the most significant bit segments in order until the required rate is completed. Note that the way in which the desired constant decoding rate is achieved is much more

9 SUBSTITUTE SHEET (RULE 26) simple and much less computationally expensive than in OCF-based encoders (MP3, AAC, etc.) since quantification is performed in a loop that has

A maximum of 15 iterations for each spectral sample (about 4 iterations on average, in each of which only a few simple binary operations are performed) and coding (eg Huffman) is only performed once, when all coefficients are already quantified. Experimentally it is observed that the cost of quantification / coding of the present algorithm is practically insignificant

305 compared to the cost of implementation through OCF.

The scenario where scalability is probably most useful is streaming audio over the Internet. Using the proposed system, the number of transmission bits could be adjusted to the size of the payload of an IP packet, following the 310 TCP protocol, to dynamically adapt to the available bandwidth of each client, for example. The speed of coding through this system would make it especially suitable for real-time applications.

On the other hand, orno can be seen, the compression algorithm described here is 315 perfectly extensible to the case of image compression (it is, in fact, an image compression - spectrograms - in grayscale) as long as there is a adequate visual masking model.

Finally, decompressing the original audio signal is also extremely

320 simple: just unzip the file with the spectral values, multiply them by their corresponding sign, if they have been compressed separately, and perform the corresponding inverse transform. This has the advantage that the decoder would be so simple that it could be included in each of the compressed files, so that they could "self-decompress." For example, if we opt for separation in

325 bit planes, basically it would only take a small program to execute the same decompression routine of a binary image several times (as many as planes) and then do the inverse transformation of the resulting values.

10 10

SUBSTITUTE SHEET (RULE 26) 330 Predictive coding

Taking into account that the musical signals usually have a high temporal and frequency redundancy, as has already been said, it can be thought that using prediction techniques could compress the signal even more.

335

To efficiently exploit the temporal correlation of the spectral samples in the case of musical signals, the use of the Wavelet transform is especially useful. In spectrograms generated by the FFT module, pure tones sustained over time are horizontal lines in the spectrogram at

340 height of the corresponding frequency. However, the FFT phase is a very irregular signal and very difficult to compress. That is why the MDCT is usually used, which is a transformation that generates real spectral coefficients, and whose module is similar to that of the FFT. The problem with the MDCT is that its module is phase sensitive. That is, the MDCT module of a pure tone is somewhat different depending on

345 the phase of said tone. This causes the spectrogram generated by the MDCT module to be more irregular than that of an FFT for tonal signals, since said module is affected by the phase of the signal which in turn depends on the cut-off points of the poisoning of The signal in time.

350 Since most of the audio signals that are compressed are musical, that is, with high harmonic content, it would be very interesting to obtain spectrograms in which the tones sustained over time were regular horizontal lines on the spectrogram (which would facilitate the predictive coding). On the other hand, the musically relevant frequencies are in most cases relatively well.

355 defined: eg on the western scale the note Lεu has its fundamental frequency at 440 Hz, and the fundamental frequency of the rest of the notes is related to it by approximately whole fractions. If the size of the analysis window for spectrogram elaboration is such that the most probable frequencies produce produce approximately an integer number of periods in said windows, the

360 phase difference between two coefficients representing the same frequency in consecutive time windows would be small and the spectrogram would present more or less regular horizontal lines when the tones are sustained over time. For

SUBSTITUTE SHEET (RULE 26) it is necessary to use different analysis windows for each of the musically relevant frequencies. It would also be necessary for spectral transformation 365 to apply a perceptual masking model to the transformed coefficients. Fortunately, all this can be achieved through the Wavelet transform.

The following procedure is proposed for this:

370 a) Starting from the signal in the time domain, choose the sampling frequency so that the size of the largest Wavelet multi-resolution window corresponds to a small number of periods of the lowest musically relevant frequency of the signal.

375 b) Generate the rest of Wavelet multi-resolution windows taking into account the relationships between the most important musical frequencies of the piece to be compressed, according to its scale (tempered, chromatic, oriental, ...) to its hue, etc. Base functions similar to sinusoids will be used, such as cosine wavelets or Morlet wavelets.

380 c) Generate the spectrogram image according to the chosen transformation. The more tonal components of the relevant musical frequencies this signal possesses, the greater the temporal correlation between spectral coefficients will exist. d) Calculate the masking threshold, according to the

385 transformation chosen, analogously as described in the previous section.

In this way, we will have a spectrogram whose rows will be capable of applying some kind of predictive coding on them. There are multiple ways of

390 apply said predictive coding: a) in the time domain, before obtaining the spectrogram image, b) in the frequency domain, before quantifying, c) in the frequency domain, quantifying the residue and readjusting the prediction, d) in the frequency domain from the values already quantified, e) use prediction in each bit plane. Let's analyze each one of them:

395

12 SUBSTITUTE SHEET (RULE 26) a) Time prediction. The idea would be to use some linear prediction method (for example, to estimate each sample based on a linear combination of previous samples) so that only the prediction residue had to be coded perceptually. However, this has a serious drawback: for

400 recovering the original signal in the decoder would require the original values of the signal over time (which serve to make the prediction for the following samples), but we would not have them exactly, since we have encoded the waste with losses, and by therefore, the linear combination of previous values necessary to regenerate the prediction of the current sample does not

405 would be the same one that was used to predict in the encoder, and the errors would be dragged forward. In any case, assuming that the prediction could be made at destination without any error, or that this error could be maintained within tolerable limits, all we would have to do would be to encode the residue with the same masking threshold.

410 of the signal itself (note that, if the prediction could be recovered without error, the only error would be that corresponding to the quantification of the residue), which would be very advantageous, because the energy of the residue would be much lower, the spectral values in general smaller, and therefore, the margin of freedom allowed by the masking threshold would be proportionately greater, and the

415 more efficient coding. That is why it would be interesting to use, instead of

LPC (Linear Predictive Coding), some other way to predict the signal that does not depend on the successful recovery of previous samples, but can be generated independently. For example, a parametric prediction of each instrument could be developed (automatically or manually), for example at

420 from samples of them (bass drum, box, charles, etc. in a battery) or by physical modeling (for example, finding out the vibration modes of a given instrument by solving the corresponding differential equation and coding only the parameters necessary to model its sound from these modes). In the extreme case, a signal model can be developed

425 musical to be encoded using a structured parametric language such as MIDI or SAOL (see MPEG-4 standard [4]) trying to imitate the original natural signal as much as possible. Once this model is achieved, the original signal is subtracted, and the difference (residue) is encoded using the masking threshold of the original signal, and it is also necessary to code the model parameters.

13 SUBSTITUTE SHEET (RULE 26) 430 This has the added advantage that the model sounds quite "natural" by itself, and the residue can be encoded with very few bits without noticeable distortion.

b) Prediction in the frequency domain, before quantifying. Yes

435 we look at the spectrogram image of a musical signal, especially using Wavelets and properly adjusting the sampling frequency, as stated above, you can clearly see a spatial repetition of temporal patterns that match the musical parts that are similar, by example, successive measures or tones that remain in the

440 time generating horizontal lines on the spectrogram. This suggests that its coding simply as images or data without any structure is somewhat inefficient. In any case, the linear prediction in the spectral domain before quantifying would have equivalent performance and the same problems as the prediction in the time domain:

445 the signal could not be recovered from the quantified residue. However, you can take advantage of the prediction at this point to make a lossless encoder. For this, a lossless spectral transformation can be used, such as intMDCT [5], and a predictive coding of the unquantified spectrogram can be performed. If a multiresolution transform is used (adjusting the

450 sampling frequency and window sizes, as explained) the prediction can be more efficient and a compressor "practically without losses", or without losses with respect to spectral coefficients could be developed.

455 c) Prediction in the frequency domain by quantifying the residue and readjusting the prediction. In this case, the problem of not being able to recover exactly the samples of the linear prediction from the previous samples can be solved, since now we can have them definitively quantified from the perceptual model. Taking advantage of the

460 temporal correlation between the spectral coefficients, predictive coding can be performed on the spectrogram rows. This would typically be done through the realization of a current value prediction filter whose coefficients would be the weights of a certain linear combination of the

14 SUBSTITUTE SHEET (RULE 26) previous values. This is, in essence, a linear predictive coding (LPC).

465 However, in this case it is not possible to proceed in this way (using the prediction filter), since by quantifying each column of the spectrogram we are modifying the prediction of the following values. Therefore, it would be necessary to proceed step by step, preparing the prediction not from the values of the original spectrogram, but through the sum of the prediction and the

470 quantified residue (taking into account the masking threshold, as described) of the corresponding previous samples. The advantage of all this is that if the prediction is good enough, the residue will be much smaller than the signal itself and it will be more likely that either its values are below the masking threshold and quantified to

475 zero, or the relative margin between the values to be quantified and the masking threshold is more favorable and privileged values are taken more often. In both cases we would be increasing the overall compression capacity of the system. Note that in this case the prediction can be retrieved exactly in the decoder, and therefore, the only source

480 of error will come from the quantification of the residue, so it is perfectly valid to quantify said residue with the masking threshold calculated for the original signal.

d) Prediction in the frequency domain from the values already 485 quantified. It would be similar to that of section b) but with the advantage that the prediction residue cannot take any value, but only those that are difference between two that are possible quantification values.

e) Prediction in each bit plane. If we look at the binary images that 490 result from separating the quantified spectrogram image in bit planes, a spatial repetition of temporal patterns that match similar musical parts, for example, successive measures, can also be clearly seen. Again, this suggests that the coding of these images simply as binary images without any structure is 495 somewhat inefficient. Making a prediction in this case also has the advantage that the prediction can only be successful or fail, that is, the prediction residue would be, for example, a black image (hits) with dots.

15 15

SUBSTITUTE SHEET (RULE 26) targets where the prediction would have failed. In this way, with a simple "XOR" operation between the prediction and the fault image we could obtain the original image. To obtain a compression gain, a priori, 500 would simply need to hit more points than if the prediction was simply an image of zeros or ones (ie not predicting). The prediction could be elaborated in a similar way to that suggested in a), b) or c), setting a threshold from which the prediction is considered 1 or 0.

505

Note that some of these techniques are not exclusive, and can be cascaded. For example, you can make a prediction over time, quantify the residual and redo predictive coding on the quantized values or on the bit planes, since the residue is usually still quite correlated with the signal, and exhibits similar traits of frequency and temporal redundancy. 510

Spectrum management and noise modeling

If we look at the spectrogram image of a very harmonic signal (for example a violin note held for several seconds) it can be seen that it has a 515 series of periodic maximums in frequency and that they are sustained in turn over time. That is to say, in the spectrogram a series of horizontal white bands appear more or less equally spaced in frequency, corresponding to the harmonic tones that constitute the majority of the sound energy, plus a series of intermediate values that could be considered "inharmonious", that is, noise, somehow. 520 This phenomenon also occurs in the human voice spectrogram, being well known and exploited in predictive voice encoders.

An attempt may be made to improve the compression of the quantified samples as follows: 525

a) An estimate of the spectrum is made over several seconds. For example, simply doing the arithmetic mean of the rows, obtaining

16 SUBSTITUTE SHEET (RULE 26) a column vector of the same length as the spectrogram columns. 530 b) An auxiliary column vector that is monotonously decreasing and of the same size as the columns thereof is added to the spectrogram matrix. c) The estimation vector is used to sort all the spectrogram columns according to it. Note that the auxiliary vector will be in turn untidy, so that if we now order the spectrogram according to said auxiliary vector, we will recover the original spectrogram. Therefore that vector must also be encoded, so that the decoder can use it. d) If the spectrogram has sufficient temporal correlation, we will have that in 540 the new ordered image, the white bands corresponding to the harmonics are placed together at the top of the spectrogram, and the rest is a more or less noisy sequence of spectral values. The ordered spectral estimation vector typically has a gently decreasing structure from a certain frequency. That is, what would be a slightly colored white 545 noise. If it can be assumed that this situation is more or less stationary (which does not happen if, for example, there are percussion instruments), only the upper part of the spectrogram could be encoded and the lower part modeled as noise, coding the parameters of said model instead of the original values. 550 e) If there are percussion instruments in the signal, another method can be used: once the spectrogram has been ordered as described, select one of each N columns and repeat it N times from a certain frequency ( that frequency will be sought after which the values begin to be clearly smaller). In this way we will only have 555 to encode one of each N columns (from a certain frequency) achieving compression ratios of plus or minus 1: N additional, from that frequency. In this way, although there are strong attacks (cash hits, for example) these are maintained for a few milliseconds (typically N would be 2, 3 or 4 for a 256-point MDCT), and the repetition is not excessively perceived 560, especially if we have in mind that we can save the original signs, with which, we would really only be

17 SUBSTITUTE SHEET (RULE 26) repeating the spectrum envelope. Instead of simply repeating, intermediate values could also be interpolated.

565

d) Detailed presentation of a way of carrying out the invention

1. Starting from a digital audio signal encoded using PCM, segment said signal into parts of the order of several seconds according to a criterion of homogeneity in the properties of said signal. 570

2. Each of these parts is further subdivided into short segments of time, for example of about 11 ms. for the application of some mathematical transform that transforms the temporal samples into spectral coefficients, such as the MDCT (or, where appropriate, some 575 multiresolution transform appropriate to the frequency characteristics of the signal to be encoded, as explained).

3. Perform this transformation by separating the module and the signs of the MDCT coefficients. The coefficients module constitutes the image of the 580 spectrogram module. The signs can be saved in a binary image of the same size, where 1 corresponds to the positive numbers and 0 to the negative ones or those that have a null module.

4. Use the MDCT coefficients to calculate the masking threshold 585 at each point of the spectrogram.

5. Use the margin provided by the masking threshold at each point to quantify the spectral coefficients one by one, trying to take some of the privileged values, as explained in section 590 above. Alternatively, a prediction model for the spectrogram rows can be developed so that what is quantified is the prediction residue, taking into account the original masking threshold at that point, and using the sum of this quantified residue plus the prediction for

18 SUBSTITUTE SHEET (RULE 26) elaborate the predictions of the following coefficients. The coefficients of the 595 prediction model will be stored as collateral information.

6. Separate the quantified spectrogram into bit planes, subdivide each of these bit planes into smaller sections and establish a hierarchy that orders those planes from the most important to the least important. For example, 600 taking as more important the sections of the plane of more significant bits located in lower frequencies and as less important the sections of the plane of less significant bits located in higher frequencies.

605

7. Compress each of these sections using a suitable binary image compressor.

8. Likewise compress the image of the spectrogram signs obtained in step 3. 610

9. Store these images as well as how to sort the spectrogram sections and the prediction model information, where appropriate, which will be sufficient for decoding each of the parts of the signal as obtained in step 1 .615

10. In order to decode each part, the decompression of each section of the spectrogram will be decompressed, as well as the image of signs, they will be ordered again according to the criteria chosen in step 6 and the inverse transform will be performed to the one used in step 2 to obtain the corresponding signal in the 620 time domain.

11. Finally all the parts are assembled obtaining the final decoded signal.

625

19 SUBSTITUTE SHEET (RULE 26) e) Industrial applicability of the invention

There are several features that distinguish the present invention from similar ones for the compression of audio signals. On the one hand, the 630 quantification-coding method is less complex and much more computationally efficient (and therefore much faster to perform) than those based on OCF, as is the case with the best-known audio compressors , such as MP3, AAC, etc. On the other hand, the proposed quantification-coding system allows scalable bit chains to be developed, as well as decodable bit chains at a constant bit rate, 635 also very efficiently, reaching quality / file size commitments at the state level of current art. Therefore, it could be interesting to use the present invention in any field, but particularly in the compression / transmission of scalable audio data in real time, development of compression and decompression devices of low cost and / or low energy consumption. 640

On the other hand, it is clear that a very high percentage of the audio signals that are compressed are musical signals. These types of signals have frequency / temporal redundancy characteristics that can also be exploited. The present invention seeks to take advantage of this using multiresolution analysis and adjusting the 645 sampling rate so that the compression of signals with high harmonic content is more efficient, as explained. Optimizing the compression of musical signals also differentiates the proposed system from other more general audio compression systems.

650

f) References

[1] MPEG, "International standard IS 11172-3. Coding of moving pictures and associated audio for digital storage media at up to 1-5 Mbit / s, part 3: Audio," 1991.

[2] M. Bosi, K. Brandenburg, S. Quackenbush, L. Fielder, K. Akagiri, H. Fuchs, M. Dietz, J. Herré, G. Davidson, and Y. Oikawa, "ISO / IEC MPEG- 2 advanced audio coding, "J. Audio Eng. Soc, vol. 45, no. 10, pp. 789-814,

20 SUBSTITUTE SHEET (RULE 26) 1997

[3] K. Brandenburg, "OCF - A new coding algorithm for high quality sound you mean," Proc. ICASSP-87, IEEE Press, pp. 5.1.1-5.1.4, 1987.

[4] B. L. Vercoe, W. G. Gardner, and E. D. Scheirer, "Structured audio: The creation, transmission and rendering of parametric sound representations," Proc. IEEE, vol. 85, no. 5, pp. 922-940, 1998.

[5] R. Geiger, J. Herré, J. Koller, and K. Brandenburg, "IntMDCT - A link between perceptual and lossless audio coding," in Proc of the IEEE International Confon Acoustics, Speech and Signal Processing, 2002, VoI. 2, May 2002, pp. 1813-1816, Orlando, FL.

21 SUBSTITUTE SHEET (RULE 26)

Claims

Claims What is claimed is: 1.- A method to encode / decode audio signals (or images where appropriate) using a perceptual model for the elimination of redundant information, characterized by: using an initial segmentation of the signal to be compressed according to homogeneity criteria of the signal; said segments can last several seconds. Divide and poison the signal into smaller segments, suitable for using some mathematical transformation that matches each time window (domain 1) with a series of spectral coefficients in a domain more suitable for the application of a perceptual irrelevance model (domain 2) , being able to separate the module and the phase of said coefficients (if they are complex numbers) or the module and the signs (if they are real numbers) and code them separately if it is more advantageous for compression. Calculate, for each coefficient, the masking threshold that allows the noise associated with the quantification of said coefficient to be inaudible. Quantify these coefficients one by one trying, provided that the masking threshold allows it, that its final quantified value is one of those that have previously been defined as privileged values: a more or less reduced subset of the total possible quantification values. Perform an entropy coding that exploits on the one hand the fact that the privileged values will be taken with greater probability than the rest, and on the other the structure of the binary words by which said privileged values are expressed to achieve the maximum possible compression . Also take advantage of the structure of the binary words by means of which the quantified values are expressed to separate them in bit planes and provide the system with scalability in decoding. The possibility of using some type of multi-resolution mathematical transform by choosing base functions appropriate to the signal to be encoded (typically approximately sinusoidal for musical signals, with high harmonic content) and adjusting the sampling frequency and the time window size depending on the frequency so that each of these windows comprises approximately an integer number of periods of the most significant frequencies of the signal; in this way, the phase differences between coefficients SUBSTITUTE SHEET (RULE 26) Consecutive over time corresponding to the same frequencies (spectrogram rows) are smaller, and the generated spectrograms are less irregular, particularly for harmonic signals, such as musical signals. Take advantage of this circumstance to apply models of temporal prediction on the spectral coefficients thus generated and thus increase the compression capacity of the system. 2. The method of claim 1 when the initial partition of musical fragments is performed according to a criterion of musical homogeneity of the signal (that is, that the signal within each time fragment does not present abrupt changes in volume, hue, rhythm, etc.) this selection is already made manually or automatically. 3. The method of claim 1 when the DCT, MDCT or Wavelet transformation is used and the module and the signs of the transformed values are separated and coded separately, or the signs are again assigned to the corresponding spectral values. Once its module is quantified and said coefficients are coded with their corresponding sign. 4.- A method of perceptual quantification of spectral values that consists of: ordering the total possible quantification values according to some criteria, so that the occurrence of some values is favored over others, provided that the masking threshold is allow. One way to do this would be to establish successive partitions of the dynamic range of the spectral values by choosing a smaller quantification step each time and trying, whenever the masking threshold allows, that the coefficients take a quantification value when quantified. use the largest possible step. In this way, values that are multiples of the larger steps will be privileged over those that are multiples of the smaller steps. Different sets of privileged values can be defined for different frequency ranges according to the statistical properties of each area of the spectrum. 5. The quantification method of claim 4 when the quantification algorithm is as follows: 2. 3 SUBSTITUTE SHEET (RULE 26) a) Normalize and scale the total spectral values, for example using the dynamic range that allow 16 bits of quantification, so that the maximum corresponds to 65535 and the minimum to 0. b) Perform the same operation with the corresponding masking thresholds at each spectral value. c) Define the total dynamic range as initial quantification step, in this case 65535. d) Divide each spectral value of the signal by said step and separate the quotient and the rest. If the remainder is greater than half of the quantification step, a new remainder is defined as: remainder = step-rest, and the ratio is increased by one unit (with the latter what we do is quantify to the nearest step, above or below). e) That remainder will be the quantification error in case we use this step to quantify. Therefore, if for a given sample this remainder is less than the masking threshold, we quantify that sample to the value that results from multiplying the quotient by the current step. Otherwise, divide the step by 2 and return to step d) until all samples are quantified. 6. The method of claim 4 (or 5 where appropriate) when the values thus quantified are directly compressed with a suitable binary word compressor, or are made to correspond previously with a word dictionary chosen according to the probability of occurrence of quantified symbols, so that further compression is more efficient. 7. The method of claim 4 (or 5 where appropriate) when the values thus quantified are separated into bit planes, and these planes are further subdivided into sections, and these sections are compressed separately, ordered according to some criterion of perceptual importance (typically the most significant bits will be more important) so that only a part of the sections can be selected for decoding, providing the scalability system. The same method can be used to achieve a constant transmission or decoding bit rate: it is sufficient to set said rate and select the sections of the bit planes in the predetermined order until the number of bits available for each time unit is completed. 24 SUBSTITUTE SHEET (RULE 26) 8. The method of claim 7 when to compress the sections (small binary images) the algorithm PNG (Deflate), JBIG or some well known algorithm of compression of binary matrices with few ones (sparse) is used. 105 9. The method of claim 4 (or 5 where appropriate) when to achieve scalability those coefficients and / or less significant bits thereof are eliminated according to some perceptual criteria and the resulting coefficients are compressed again as it is indicated in claim 6. In this case there would be 110 added cost of "recompression" but the compression could be higher, depending on the algorithm used. 10. The method of claim 4 (or 5 where appropriate) and those of claims 6, 7, 8 and 9 depending on it, when used in any coding system of 15 signals, whether audio, images or of any other type. 11. The method of claim 1 when a temporal masking model is applied to the image of the spectrogram corresponding to the masking threshold consisting of convolving each row of said spectrogram with a dispersion function that represents said temporary masking. This temporal dispersion function may be different for each frequency (row). The threshold that will be taken for the subsequent quantification process will be the one resulting from taking the maximum between the original masking threshold and the result of the convolution at each point of the image. 25 12. The method of claim 1 when, as a previous step to the transformation of the samples from the time domain to the spectral domain, a prediction is made by a model of the musical signal to be encoded, using a structured parametric language, as per MIDI or SAOL example (see MPEG-4 standard), using physical instrument modeling (or human voice), or using samples of them, trying to imitate the original natural signal as much as possible. This prediction can be generated automatically or manually. In any case, the input signal to the encoder described in claim 1 is the prediction residue, that is, the difference between the original and the predicted signal. The method will, however, use the masking threshold calculated from the original signal, since the SUBSTITUTE SHEET (RULE 26) Prediction can be retrieved without any error in the decoder, and the only existing quantification error will be that resulting from the coding with waste losses. The parameters (or sound samples, if necessary) necessary for reworking the prediction in the decoder are compressed and sent to the decoder as 140 collateral information. 13.- A method to achieve more regular spectrograms consisting of: using to map temporary samples in spectral samples some transformed multi-resolution whose base functions are suitable for musical signals, such as 145 Cosine Wavelets or Morlet Wavelets; adjust the sampling frequency of the temporal signal and choose the analysis window size for each scale (frequency) of the multiresolution transform so that for the frequencies that are most likely to occur in the signal to be compressed, the number of periods that contain such windows be approximately whole. 150 14.- The method of claim 1 when a prediction system is used on the rows or columns of the unquantified spectrogram, and this has been generated using some multi-resolution transformation as indicated in claim 13. In this way a Virtually lossless compressor. 155 15. The method of claim 1 when a prediction system is used on the rows or columns of the unquantified spectrogram, and this has been generated using a lossless invertible transformation, such as IntMDCT. In this way a lossless compressor is achieved. 160 16.- A method of predicting the spectrogram rows, whether generated by FFT, DCT, MDCT, Wavelet or any other transformed, consisting of: for the coefficients of each spectrogram row, select a set of previous coefficients in the time (of the same row or not) and establish the weights of the 165 linear combination that best approximates said coefficient from the previous ones following some criteria, such as minimizing the prediction error (residue) along the rows; follow an iterative process in which the coefficients of each row are traversed from left to right in the spectrogram making said linear prediction for each of the coefficients, obtaining the residue, and quantifying said residue ⁵ SUBSTITUTE SHEET (RULE 26) 170 according to the original masking threshold (for example, using the method of claim 4) at that point; use not the previous original values for the prediction of the following values but the result of adding the prediction plus the quantified residual at each point. 175 17. The method of claim 1 when some prediction method is used on the spectrogram of the values already quantified according to the method of claim 4. In this case the residue is simply encoded without losses. This residue plus the prediction parameters will be what the decoder uses to reconstruct the original quantified samples. 180 18. The method of claim 1 when when a prediction method is used on the binary images resulting from separating the spectrogram, suitably quantified, into bit planes. In this case, for each plane, the binary image corresponding to the prediction error would be encoded and compressed, instead of the 185 original. A simple "xor" operation between the prediction and error images would serve the decoder to recover the original plane. 19.- A method of ordering the spectrogram rows according to their perceptual importance or energy content of the signal, in which a 190 estimate of the spectrogram columns is used - eg. the column vector of the average of the rows - to order the same according to said estimate, looking for the most important values to be located, for example, at the top of it and considering the bottom as noise. 195 20. The sorting method of claim 19 when used to model the lower part of the ordered spectrogram as noise with certain characteristics, or, where appropriate, encode only some columns of the ordered spectrogram from a given row, so that the decoder can obtain the full spectrogram by replicating said columns several times or interpolating the 200 missing values. 27 SUBSTITUTE SHEET (RULE 26) 21. The ordering method of claim 19 when used to separately code the coefficients belonging to subsets of one or more consecutive rows of the spectrogram according to their particular statistical properties. 205 22.- The quantification method of claim 4 when the arrangement of claim 19 is used to establish a different set of privileged values for different frequency bands (subsets of spectrogram rows). 210 23. The method of claim 1, according to any of claims 2 to 22, when it is implemented as software in a general purpose computer. 24. The method of claim 1, according to any of the 215 claims 2 to 22, when it is implemented in a specific purpose machine, or as part of a more general system, such as a portable music encoder / player , a mobile phone, etc. 25. The method of claim 1 particularized according to any of the 220 claims 2 to 22 when it is used for the compression of signals from two or more channels. In particular, when the sum of the channels is compressed on the one hand and their difference on the other. 225 28 SUBSTITUTE SHEET (RULE 26)