HK40063033B - Methods and apparatus to fingerprint an audio signal via normalization - Google Patents

Description

Method and apparatus for fingerprinting audio signals via normalization

Related applications

This patent claims priority and interest in French patent application serial number 1858041, filed on September 7, 2018. The entire contents of French patent application serial number 1858041 are incorporated herein by reference.

Technical Field

This disclosure generally relates to audio signals, and more specifically, to methods and apparatus for fingerprinting audio signals via normalization.

Background Technology

Audio information (e.g., sound, speech, music, etc.) can be represented as digital data (e.g., electronic, optical, etc.). Audio captured (e.g., via a microphone) can be digitized, stored electronically, processed, and/or classified. One method for classifying audio information is by generating an audio fingerprint. An audio fingerprint is a digital summary of audio information created by sampling a portion of an audio signal. Audio fingerprints have historically been used to identify audio and/or verify its authenticity.

Attached Figure Description

Figure 1 is an example system that can implement the teachings of this disclosure.

Figure 2 shows an example implementation of the audio processor in Figure 1.

Figures 3A and 3B depict example unprocessed spectrum diagrams generated by the example frequency range separator in Figure 2.

Figure 3C depicts an example of a normalized spectrum generated by the signal normalizer of Figure 2 based on the unprocessed spectrum of Figures 3A and 3B.

Figure 4 shows the example unprocessed spectrograms of Figures 3A and 3B, which are divided into fixed audio signal frequency components.

Figure 5 is an example of a normalized spectrum generated by the signal normalizer in Figure 2 based on the fixed audio signal frequency components in Figure 4.

Figure 6 is an example of a normalized and weighted spectrogram generated by the point selector of Figure 2 based on the normalized spectrogram of Figure 5.

Figures 7 and 8 are flowcharts showing machine-readable instructions that can be executed to implement the audio processor of Figure 2.

Figure 9 is a block diagram of an example processing platform configured to execute the instructions of Figures 7 and 8 to implement the audio processor of Figure 2.

The accompanying drawings are not to scale. Generally, the same reference numerals will be used throughout the drawings and accompanying written description to refer to the same or similar parts.

Detailed Implementation

Fingerprint- or signature-based media monitoring techniques typically utilize one or more inherent characteristics of the monitored media during a monitoring time interval to generate a substantially unique proxy for that media. Such a proxy is called a signature or fingerprint and can take any form (e.g., a series of numerical values, waveforms, etc.) representing any aspect of the media signal (e.g., audio and/or video signals forming the presentation of the monitored media). A signature can be a series of signatures collected consecutively within a time interval. The terms “fingerprint” and “signature” are used interchangeably herein and are defined herein as meaning a proxy generated based on one or more inherent characteristics of the media for identification purposes.

Signature-based media surveillance typically involves: determining (e.g., generating and/or collecting) a signature representing a media signal (e.g., audio and/or video signal) output by the monitored media device, and comparing the monitored signature with one or more reference signatures corresponding to known (e.g., reference) media sources. Various comparison criteria (such as cross-correlation values, Hamming distance, etc.) can be evaluated to determine whether the monitored signature matches a specific reference signature.

When a match is found between a monitored signature and one of the reference signatures, the monitored media can be identified as corresponding to a specific reference media represented by the reference signature that matches the monitored signature. Since attributes such as media identifiers, presentation time, broadcast channel, etc., are collected against the reference signature, these attributes can be associated with the monitored media (the monitored media whose monitored signature matches the reference signature). Example systems for identifying media based on codes and/or signatures are already known and were first disclosed in Thomas's U.S. Patent 5,481,294, the entire contents of which are incorporated herein by reference.

Historically, audio fingerprinting technology has used the loudest parts of an audio signal (e.g., the most energetic portions) to create a fingerprint over a period of time. However, this method has several serious limitations in some cases. In some examples, the loudest parts of the audio signal may be associated with noise (e.g., unwanted audio) rather than the audio of interest. For example, if a user is trying to fingerprint a song in a noisy restaurant, the loudest part of the captured audio signal might be conversations between restaurant patrons, rather than the song or media being identified. In this example, many sampled portions of the audio signal will contain background noise instead of music, which reduces the usefulness of the generated fingerprint.

Another potential limitation of previous fingerprinting technologies is that, particularly in music, audio in the low-frequency range tends to be the loudest. In some examples, the dominant bass frequency energy causes the sampling portion of the audio signal to be primarily in the bass frequency range. Therefore, fingerprints generated using existing methods often do not include samples from all parts of the audio spectrum that can be used for signature matching, especially samples from the higher frequency ranges (e.g., the treble range, etc.).

The example methods and apparatuses disclosed herein overcome the aforementioned problems by generating fingerprints from audio signals using mean normalization. One example method involves normalizing one or more time-frequency bins of an audio signal according to the audio characteristics of the surrounding audio region. As used herein, a “time-frequency bin” is a portion of the audio signal corresponding to a specific frequency bin (e.g., an FFT bin) at a particular time (e.g., three seconds into the audio signal). In some examples, the normalization is weighted according to the audio category of the audio signal. In some examples, fingerprints are generated by selecting points from the normalized time-frequency bins.

Another example method disclosed herein involves dividing an audio signal into two or more audio signal frequency components. As used herein, an "audio signal frequency component" is a portion of the audio signal corresponding to a frequency range and time period. In some examples, an audio signal frequency component may consist of multiple time-frequency bins. In some examples, audio characteristics are determined for some audio signal frequency components within the audio signal frequency components. In this example, the individual audio signal frequency components within the audio signal frequency components are normalized according to associated audio characteristics (e.g., audio mean, etc.). In some examples, fingerprints are generated by selecting points from the normalized audio signal frequency components.

Figure 1 shows an example system 100 that can implement the teachings of this disclosure. The example system 100 includes an example audio source 102 and an example microphone 104 that captures sound from the audio source 102 and converts the captured sound into an example audio signal 106. An example audio processor 108 receives the audio signal 106 and generates an example fingerprint 110.

Example audio source 102 emits audible sound. The example audio source can be a speaker (e.g., an electroacoustic transducer, etc.), a live performance, dialogue, and/or any other suitable audio source. Example audio source 102 can include desired audio (e.g., audio for fingerprint recognition, etc.) and can also include undesired audio (e.g., background noise, etc.). In the example shown, audio source 102 is a speaker. In other examples, audio source 102 can be any other suitable audio source (e.g., a person, etc.).

Example microphone 104 is a transducer that converts sound emitted by audio source 102 into audio signal 106. In some examples, microphone 104 may be a component of a computer, mobile device (smartphone, tablet, etc.), navigation device, or wearable device (e.g., smartwatch, etc.). In some examples, the microphone may include an audio-to-digital converter to digitize the audio signal 106. In other examples, audio processor 108 may digitize the audio signal 106.

Example audio signal 106 is a digital representation of sound emitted by audio source 102. In some examples, audio signal 106 may be stored on a computer before being processed by audio processor 108. In some examples, audio signal 106 may be transmitted to example audio processor 108 via a network. Alternatively or alternatively, any other suitable method may be used to generate audio (e.g., digital synthesis, etc.).

Example audio processor 108 converts example audio signal 106 into example fingerprint 110. In some examples, audio processor 108 divides audio signal 106 into frequency bins and/or time periods, and then determines the mean energy of one or more of the frequency components of the created audio signal. In some examples, audio processor 108 can normalize the frequency components of the audio signal using the associated mean energy of the audio regions surrounding each time-frequency bin. In other examples, any other suitable audio characteristics can be determined and used to normalize the respective time-frequency bins. In some examples, fingerprint 110 can be generated by selecting the highest energy among the normalized audio signal frequency components. Alternatively or separately, any suitable means can be used to generate fingerprint 110. An example implementation of audio processor 108 is described below with reference to Figure 2.

Example fingerprint 110 is a concise digital digest of audio signal 106, which can be used to identify and/or verify audio signal 106. For example, fingerprint 110 can be generated by sampling and processing multiple portions of audio signal 106. In some examples, fingerprint 110 may include samples of the highest energy portion of audio signal 106. In some examples, fingerprint 110 may be indexed in a database that can be used for comparison with other fingerprints. In some examples, fingerprint 110 can be used to identify audio signal 106 (e.g., to determine what song is playing). In some examples, fingerprint 110 can be used to verify the authenticity of audio.

Figure 2 is an example implementation of the audio processor 108 of Figure 1. The example audio processor 108 includes an example frequency range separator 202, an example audio characteristic determiner 204, an example signal normalizer 206, an example point selector 208, and an example fingerprint generator 210.

Example frequency range separator 202 divides an audio signal (e.g., the digitized audio signal 106 of Figure 1) into time-frequency bins and/or audio signal frequency components. For example, frequency range separator 202 may perform a Fast Fourier Transform (FFT) on the audio signal 106 to transform the audio signal 106 to the frequency domain. Alternatively, example frequency range separator 202 may divide the transformed audio signal 106 into two or more frequency bins (e.g., using the Hamming function, Hann function, etc.). In this example, each audio signal frequency component is associated with a frequency bin in two or more frequency bins. Alternatively or additionally, frequency range separator 202 may aggregate the audio signal 106 into one or more time periods (e.g., the duration of the audio, a six-second period, a one-second period, etc.). In other examples, frequency range separator 202 may use any suitable technique to transform the audio signal 106 (e.g., Discrete Fourier Transform, Sliding Time Window Fourier Transform, Wavelet Transform, Discrete Hadamard Transform, Discrete Walsh Hadamard Transform, Discrete Cosine Transform, etc.). In some examples, the frequency range splitter 202 can be implemented by one or more bandpass filters (BPFs). In some examples, the output of the example frequency range splitter 202 can be represented by a spectrum. The example output of the frequency range splitter 202 is discussed below with reference to Figures 3A to 3B and Figure 4.

Example audio characteristic determiner 204 determines the audio characteristics of a portion of audio signal 106 (e.g., audio signal frequency components, audio regions around time-frequency bins, etc.). For example, audio characteristic determiner 204 may determine the mean energy (e.g., average power, etc.) of one or more audio signal frequency components. Additionally or alternatively, audio characteristic determiner 204 may determine other characteristics of a portion of the audio signal (e.g., mode energy, median energy, mode power, median energy, mean energy, mean amplitude, etc.).

Example signal normalizer 206 normalizes one or more time-frequency bins according to the associated audio characteristics of the surrounding audio region. For example, signal normalizer 206 may normalize the time-frequency bins according to the mean energy of the surrounding audio region. In other examples, signal normalizer 206 normalizes some audio signal frequency components in the audio signal frequency components according to the associated audio characteristics. For example, signal normalizer 206 may use the mean energy associated with the audio signal frequency component to normalize the individual time-frequency bins of that audio signal frequency component. In some examples, the output of signal normalizer 206 (e.g., normalized time-frequency bins, normalized audio signal frequency components, etc.) may be represented as a spectrogram. An example output of signal normalizer 206 is discussed below with reference to Figures 3C and 5.

Example point selector 208 selects one or more points from the normalized audio signal to be used to generate fingerprint 110. For example, example point selector 208 can select multiple energy maximum values of the normalized audio signal. In other examples, point selector 208 can select any other suitable point of the normalized audio.

Alternatively, the point selector 208 may weight the selection of points based on the category of the audio signal 106. For example, if the category of the audio signal is music, the point selector 208 may focus the selection of points on the common frequency range of music (e.g., bass, treble, etc.). In some examples, the point selector 208 may determine the category of the audio signal (e.g., music, speech, sound effects, advertisements, etc.). The example fingerprint generator 210 uses the points selected by the example point selector 208 to generate a fingerprint (e.g., fingerprint 110). The example fingerprint generator 210 may use any suitable method to generate a fingerprint based on the selected points.

Although Figure 2 illustrates an example implementation of the audio processor 108 of Figure 1, one or more of the elements, processes, and/or devices illustrated in Figure 2 can be combined, divided, rearranged, omitted, eliminated, and/or implemented in any other way. Furthermore, the example frequency range separator 202, the example audio characteristic determiner 204, the example signal normalizer 206, the example point selector 208, and the example fingerprint generator 210, and/or the more general example audio processor 108 of Figures 1 and 2, can be implemented using hardware, software, firmware, and/or any combination of hardware, software, and/or firmware. Therefore, for example, any of the example frequency range splitter 202, example audio characteristic determiner 204, example signal normalizer 206, example point selector 208, and example fingerprint generator 210 and/or more generally, example audio processor 108, can be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and/or field-programmable logic devices (FPLDs). When any of the device or system claims of this patent is understood to cover purely software and/or firmware implementations, at least one of the example frequency range splitter 202, example audio characteristic determiner 204, example signal normalizer 206, example point selector 208, and example fingerprint generator 210 is thus explicitly defined as including a non-transitory computer-readable storage device or storage disk (such as a memory, digital universal disc (DVD), optical disc (CD), Blu-ray disc, etc.) having software and/or firmware. Furthermore, the example audio processor 106 of Figures 1 and 2 may include one or more elements, processes, and/or devices other than, or in place of, those illustrated in Figure 2, and/or may include more than one of any or all of the illustrated elements, processes, and devices. As used herein, the phrase “to communicate” (including variations thereof) encompasses direct communication and/or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and/or continuous communication, but additionally includes selective communication at regular intervals, planned intervals, non-periodic intervals, and/or one-off events.

Figures 3A and 3B depict example unprocessed spectrum plots 300 generated by the example frequency range separator of Figure 2. In the example shown in Figure 3A, the example unprocessed spectrum plot 300 includes an example first time-frequency bin 304A surrounded by an example first audio region 306A. In the example shown in Figure 3B, the example unprocessed spectrum plot includes an example second time-frequency bin 304B surrounded by an example audio region 306B. The example unprocessed spectrum plot 300 and the normalized spectrum plot 302 of Figures 3A and 3B each include an example vertical axis 308 representing a frequency bin and an example horizontal axis 310 representing a time bin. Figures 3A and 3B illustrate example audio regions 306A and 306B from which the audio characteristic determiner 204 obtains normalized audio characteristics, and the signal normalizer 206 uses these normalized audio characteristics to normalize the first time-frequency bin 304A and the second time-frequency bin 304B, respectively. In the example shown, the time-frequency bins of the unprocessed spectrogram 300 are normalized to generate a normalized spectrogram 302. In other examples, any suitable number of time-frequency bins of the unprocessed spectrogram 300 can be normalized to generate the normalized spectrogram 302 of Figure 3C.

Example vertical axis 308 has frequency bins generated by Fast Fourier Transform (FFT) and has a length of 1024 FFT bins. In other examples, example vertical axis 308 can be measured by any other suitable technique for measuring frequency (e.g., Hertz, another transform algorithm, etc.). In some examples, vertical axis 308 covers the entire frequency range of audio signal 106. In other examples, vertical axis 308 may cover a portion of audio signal 106.

In the example shown, the horizontal axis 310 represents a time period of 11.5 seconds with a total length of the unprocessed spectrogram 300. In the example shown, the horizontal axis 310 has a 64-millisecond (ms) interval as the unit. In other examples, the horizontal axis 310 can be measured in any other suitable unit (e.g., 1 second, etc.). For example, the horizontal axis 310 covers the entire duration of the audio. In other examples, the horizontal axis 310 can cover a portion of the duration of the audio signal 106. In the example shown, the size of each time-frequency bin in spectrograms 300 and 302 is 64 ms × 1 FFT bin.

In the example shown in Figure 3A, the first time-frequency bin 304A is associated with the intersection of the frequency bins and time bins of the unprocessed spectrogram 300 and a portion of the audio signal 106 associated with that intersection. The example first audio region 306A includes time-frequency bins within a predefined distance from the example first time-frequency bin 304A. For example, the audio characteristic determiner 204 can determine the vertical length of the first audio region 306A (e.g., the length of the first audio region 306A along the vertical axis 308, etc.) based on a set number of FFT bins (e.g., 5 bins, 11 bins, etc.). Similarly, the audio characteristic determiner 204 can determine the horizontal length of the first audio region 306A (e.g., the length of the first audio region 306A along the horizontal axis 310, etc.). In the example shown, the first audio region 306A is square. Alternatively, the first audio region 306A can be any suitable size and shape and can contain any suitable combination of time-frequency bins within the unprocessed spectrogram 300 (e.g., any suitable group of time-frequency bins, etc.). Example audio characteristic determiner 204 can then determine the audio characteristics (e.g., mean energy, etc.) of the time-frequency bins contained within the first audio region 306A. Using the determined audio characteristics, example signal normalizer 206 of FIG2 can normalize the associated values of the first time-frequency bin 304A (e.g., the energy of the first time-frequency bin 304A can be normalized according to the mean energy of the individual time-frequency bins within the first audio region 306A).

In the example shown in Figure 3B, the second time-frequency bin 304B is associated with the intersection of the frequency bins and time bins of the unprocessed spectrogram 300 and a portion of the audio signal 106 associated with that intersection. The example second audio region 306B includes time-frequency bins within a predefined distance from the example second time-frequency bin 304B. Similarly, the audio characteristic determiner 204 can determine the horizontal length of the second audio region 306B (e.g., the length of the second audio region 306B along the horizontal axis 310, etc.). In the example shown, the second audio region 306B is square. Alternatively, the second audio region 306B can be any suitable size and shape and can contain any suitable combination of time-frequency bins within the unprocessed spectrogram 300 (e.g., any suitable group of time-frequency bins, etc.). In some examples, the second audio region 306B can overlap with the first audio region 306A (e.g., containing some of the same time-frequency bins, shifted on the horizontal axis 310, shifted on the vertical axis 308, etc.). In some examples, the second audio region 306B may have the same size and shape as the first audio region 306A. In other examples, the second audio region 306B may have a different size and shape than the first audio region 306A. The example audio characteristic determiner 204 can then determine the audio characteristics (e.g., mean energy, etc.) of the time-frequency bins contained in the second audio region 306B. Using the determined audio characteristics, the example signal normalizer 206 of FIG2 can normalize the associated values of the second time-frequency bin 304B (e.g., the energy of the second time-frequency bin 304B can be normalized to the mean energy of the bins located within the second audio region 306B).

Figure 3C depicts an example of a normalized spectrogram 302 generated by the signal normalizer of Figure 2 by normalizing multiple time-frequency bins of the unprocessed spectrogram 300 from Figures 3A to 3B. For example, some or all of the time-frequency bins in the unprocessed spectrogram 300 can be normalized in the same manner as time-frequency bins 304A and 304B. An example process 700 for generating the normalized spectrogram is described in conjunction with Figure 7. The resulting frequency bins of Figure 3C have now been normalized according to the local mean energy within the local regions surrounding the regions. As a result, the darker regions are those with the highest energy in their respective local regions. This allows the fingerprint to include relevant audio features in regions with even lower energy than typically louder bass frequency regions.

Figure 4 illustrates an example unprocessed spectrogram 300 of Figure 3, divided into multiple fixed audio signal frequency components. The example unprocessed spectrogram 300 is generated by processing the audio signal 106 using a Fast Fourier Transform (FFT). In other examples, any other suitable method can be used to generate the unprocessed spectrogram 300. In this example, the unprocessed spectrogram 300 is divided into multiple example audio signal frequency components 402. The example unprocessed spectrogram 400 includes an example vertical axis 308 and an example horizontal axis 310 of Figure 3. In the illustrated example, each example audio signal frequency component 402 has an example frequency range 408 and an example time period 410. The example audio signal frequency components 402 include an example first audio signal frequency component 412A and an example second audio signal frequency component 412B. In the illustrated example, the darker portions of the unprocessed spectrogram 300 represent the higher-energy portions of the audio signal 106.

Each of the example audio signal frequency components 402 is associated with a unique combination of a continuous frequency range (e.g., a frequency bin, etc.) and a continuous time period. In the illustrated example, the individual audio signal frequency components 402 have frequency bins of equal size (e.g., frequency range 408). In other examples, some or all of the audio signal frequency components 402 may have frequency bins of different sizes. In the illustrated example, the individual audio signal frequency components 402 have time periods of equal duration (e.g., time period 410). In other examples, some or all of the audio signal frequency components 402 may have time periods of different durations. In the illustrated example, the audio signal frequency components 402 constitute the entire audio signal 106. In other examples, the audio signal frequency components 402 may include a portion of the audio signal 106.

In the example shown, the first audio signal frequency component 412A is located in the high-frequency range of the audio signal 106 and has no visible energy point. Example: The first audio signal frequency component 412A is associated with the frequency band between 768 FFT and 896 FFT bands and the time period between 10024 ms and 11520 ms. In some examples, multiple portions of the audio signal 106 exist within the first audio signal frequency component 412A. In this example, because the audio in the low-frequency spectrum of the audio signal 106 (e.g., the audio in the second audio signal frequency component 412B, etc.) has relatively high energy, the portion of the audio signal 106 located within audio signal frequency component 412A is not visible. The second audio signal frequency component 412B is located in the low-frequency range of the audio signal 106 and has a visible energy point. Example: The second audio signal frequency component 412B is associated with the frequency band between 128 FFT and 256 FFT bands and the time period between 10024 ms and 11520 ms. In some examples, because the portion of the audio signal 106 located in the bass spectrum (e.g., the second audio signal frequency component 412B, etc.) has relatively high energy, the fingerprint generated based on the unprocessed spectrogram 300 will include a disproportionate number of samples from the bass spectrum.

Figure 5 is an example of a normalized spectrum 500 generated by the signal normalizer of Figure 2 based on the fixed audio signal frequency components of Figure 4. The example normalized spectrum 500 includes the example vertical axis 308 and the example horizontal axis 310 of Figure 3. The example normalized spectrum 500 is divided into multiple example audio signal frequency components 502. In the illustrated example, each audio signal frequency component 502 has an example frequency range 408 and an example time period 410. The example audio signal frequency components 502 include an example first audio signal frequency component 504A and an example second audio signal frequency component 504B. In some examples, the first audio signal frequency component 504A and the second audio signal frequency component 504B correspond to the same frequency range and time period as the first audio signal frequency component 412A and the second audio signal frequency component 412B of Figure 3. In the illustrated example, the darker portions of the normalized spectrum 500 represent regions of higher energy in the audio spectrum.

An example normalized spectrum 500 is generated by normalizing the unprocessed spectrum 300 by normalizing the individual audio signal frequency components 402 of FIG4 according to their associated audio characteristics. For example, the audio characteristic determiner 204 can determine the audio characteristics (e.g., mean energy, etc.) of the first audio signal frequency component 412A. In this example, the signal normalizer 206 can then normalize the first audio signal frequency component 412A according to the determined audio characteristics to create the example audio signal frequency component 402A. Similarly, an example second audio signal frequency component 402B can be generated by normalizing the second audio signal frequency component 412B according to the audio characteristics associated with the second audio signal frequency component 412B of FIG4. In other examples, the normalized spectrum 500 can be generated by normalizing a portion of the audio signal component 402. In other examples, any other suitable method can be used to generate the example normalized spectrum 500.

In the example shown in FIG5, the first audio signal frequency component 504A (e.g., the first audio signal frequency component 412A of FIG4 after processing by signal normalizer 206, etc.) has visible energy points on the normalized spectrum 500. For example, because the first audio signal frequency component 504A has been normalized according to the energy of the first audio signal frequency component 412A, previously hidden portions of the audio signal 106 (e.g., compared to the first audio signal frequency component 412A) are visible on the normalized spectrum 500. The second audio signal frequency component 504B (e.g., the second audio signal frequency component 412B of FIG4 after processing by signal normalizer 206, etc.) corresponds to the bass range of the audio signal 106. For example, because the second audio signal frequency component 504B has been normalized according to the energy of the second audio signal frequency component 412B, the number of visible energy points has been reduced (e.g., compared to the second audio signal frequency component 412B). In some examples, a fingerprint generated from a normalized spectrogram 500 (e.g., fingerprint 110 in Figure 1) will include samples that are more evenly distributed across the audio spectrum compared to a fingerprint generated from an unprocessed spectrogram 300 in Figure 4.

Figure 6 is an example of a normalized and weighted spectrum diagram 600 generated by the dot selector 208 of Figure 2 based on the normalized spectrum diagram 500 of Figure 5. The example spectrum diagram 600 includes the example vertical axis 308 and the example horizontal axis 310 of Figure 3. The example normalized and weighted spectrum diagram 600 is divided into multiple example audio signal frequency components 502. In the example shown, each example audio signal frequency component 502 has an example frequency range 408 and an example time period 410. The example audio signal frequency components 502 include an example first audio signal frequency component 604A and an example second audio signal frequency component 604B. In some examples, the first audio signal frequency component 604A and the second audio signal frequency component 604B correspond to the same frequency range and time period as the first audio signal frequency component 412A and the second audio signal frequency component 412B of Figure 3, respectively. In the example shown, the darker portions of the normalized and weighted spectrum diagram 600 represent regions of higher energy in the audio spectrum.

An example normalized and weighted spectrogram 600 is generated by weighting the normalized spectrogram 600 using a value range from zero to one based on the category of the audio signal 106. For example, if the audio signal 106 is music, the point selector 208 of Figure 2 will weight the regions of the audio spectrum associated with music along each column. In other examples, the weighting can be applied to multiple columns and can take different ranges from zero to one.

Figures 7 and 8 illustrate flowcharts representing example hardware logic, machine-readable instructions, hardware implementation state machines, and/or any combination thereof for implementing the audio processor 108 of Figure 2. The machine-readable instructions may be an executable program or part of an executable program that is executed by a computer processor, such as processor 912 shown in the example processor platform 900 discussed below in conjunction with Figure 9. The program may be implemented in software stored on a non-transitory computer-readable storage medium, such as a CD-ROM, floppy disk, hard disk drive, DVD, Blu-ray disc, or memory associated with processor 912, but the entire program and/or parts thereof may alternatively be executed by a device other than processor 912, and/or implemented in firmware or dedicated hardware. Furthermore, although the example program has been described with reference to the flowcharts illustrated in Figures 7 and 8, many other methods of implementing the example audio processor 108 may be used alternatively. For example, the execution order of blocks may be changed, and/or some blocks in the blocks may be changed, eliminated, or combined. Alternatively or concurrently, any or all boxes may be implemented by one or more hardware circuits (e.g., discrete and/or integrated analog and/or digital circuits, FPGAs, ASICs, comparators, operational amplifiers, logic circuits, etc.) configured to perform the corresponding operations without executing software or firmware.

As described above, the example processes of Figures 7 and 8 can be implemented using executable instructions (e.g., computer and/or machine-readable instructions) stored on non-transitory computer and/or machine-readable media (such as hard disk drives, flash memory, read-only memory, optical disks, digital universal disks, caches, random access memory, and/or any other storage devices or disks, where information is stored for any duration (e.g., for extended periods, permanently, for simple instances, for temporary buffering, and/or for caching information)). As used herein, the term non-transitory computer-readable media is explicitly defined to include any type of computer-readable storage device and/or disk, excluding propagated signals and transmission media.

"Comprising" and "including" (and all their forms and tenses) are used herein as open-ended terms. Therefore, whenever a claim uses any form of "comprising" or "including" (e.g., comprising, including, comprising, including, having, etc.) as a preamble or within the recounting of any kind of claim, it will be understood that additional elements, terms, etc., may be present without falling outside the scope of the corresponding claim or recounting. As used herein, when the phrase "at least" is used as a transitional term in, for example, the preamble of a claim, it is open-ended in the same way that the terms "comprising" and "including" are open-ended. When used, for example, in the form of A, B, and/or C, the term "and/or" refers to any combination or subset of A, B, C, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, and (7) A and B and C. As used herein, in the context of describing a structure, component, item, object, and/or thing, the phrase "at least one of A and B" is intended to indicate an implementation including any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein, in the context of describing a structure, component, item, object, and/or thing, the phrase "at least one of A or B" is intended to indicate an implementation including any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. As used herein, in the context of describing the execution of a process, instruction, action, activity, and/or step, the phrase "at least one of A and B" is intended to indicate an implementation including any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein, in the context of describing the execution of processes, instructions, actions, activities and/or steps, the phrase “at least one of A or B” is intended to indicate an implementation including any of the following: (1) at least one A, (2) at least one B and (3) at least one A and at least one B.

The processing in Figure 7 begins at block 702. At block 702, audio processor 108 receives a digitized audio signal 106. For example, audio processor 108 may receive audio captured by microphone 104 (e.g., emitted by audio source 102 of Figure 1, etc.). In this example, the microphone may include an analog-to-digital converter to convert the audio into a digitized audio signal 106. In other examples, audio processor 108 may receive audio stored in a database (e.g., volatile memory 914 of Figure 9, non-volatile memory 916 of Figure 9, mass storage device 928 of Figure 9, etc.). In other examples, the digitized audio signal 106 may be transmitted to audio processor 108 via a network (e.g., the Internet, etc.). Alternatively or additionally, audio processor 108 may receive the audio signal 106 by any other suitable means.

In block 704, frequency range splitter 202 windowes the audio signal 106 and transforms the audio signal 106 to the frequency domain. For example, frequency range splitter 202 may perform a Fast Fourier Transform to transform the audio signal 106 to the frequency domain, and may perform a windowing function (e.g., Hamming function, Hann function, etc.). Alternatively or additionally, frequency range splitter 202 may aggregate the audio signal 106 into two or more time bins. In these examples, the time-frequency bin corresponds to the intersection of the frequency bin and the time bin and contains a portion of the audio signal 106.

In box 706, the audio characteristic determiner 204 selects a time-frequency bin for normalization. For example, the audio characteristic determiner 204 can select the first time-frequency bin 304A of Figure 3A. In some examples, the audio characteristic determiner 204 can select a time-frequency bin adjacent to the previously selected first time-frequency bin.

In box 708, audio characteristic determiner 204 determines the audio characteristics of the surrounding audio region. For example, if audio characteristic determiner 204 selects a first time-frequency bin 304A, then audio characteristic determiner 204 can determine the audio characteristics of a first audio region 306A. In some examples, audio characteristic determiner 204 can determine the mean energy of the audio region. In other examples, audio characteristic determiner 204 can determine any other suitable audio characteristic (e.g., mean amplitude, etc.).

In box 710, the audio characteristic determiner 204 determines whether another time-frequency bin should be selected, and then process 700 returns to box 706. If no other time-frequency bin is selected, process 700 proceeds to box 712. In some examples, boxes 706 through 710 are repeated until every time-frequency bin of the unprocessed spectrogram 300 has been selected. In other examples, boxes 706 through 710 may be repeated for any suitable number of iterations.

In block 712, signal normalizer 206 normalizes each time-frequency bin based on associated audio characteristics. For example, signal normalizer 206 may normalize each time-frequency bin selected in block 706 using the associated audio characteristics determined in block 708. For example, the signal normalizer may normalize the first time-frequency bin 304A and the second time-frequency bin 304B according to the audio characteristics (e.g., mean energy) of the first audio region 306A and the second audio region 306B, respectively. In some examples, signal normalizer 206 generates a normalized spectrogram based on the normalization of the time-frequency bins (e.g., the normalized spectrogram 302 of FIG. 3C).

In box 714, dot selector 208 determines whether fingerprint generation should be weighted based on audio category, and process 700 proceeds to box 716. If fingerprint generation is not weighted based on audio category, process 700 proceeds to box 720. In box 716, dot selector 208 determines the audio category of audio signal 106. For example, dot selector 208 may present a cue to the user indicating the category of the audio (e.g., music, speech, sound effects, advertisements, etc.). In other examples, audio processor 108 may use an audio category determination algorithm to determine the audio category. In some examples, the audio category may be a specific person's voice, general human speech, music, sound effects, and/or advertisements.

In box 718, the dot selector 208 weights the time-frequency components based on the determined audio category. For example, if the audio category is music, the dot selector 208 may weight the audio signal frequency components in association with the high and low frequency ranges typically associated with music. In some examples, if the audio category is a particular person's voice, the dot selector 208 may weight the audio signal frequency components in association with that person's voice. In some examples, the output of the signal normalizer 206 may be represented as a spectrogram.

In block 720, fingerprint generator 210 generates a fingerprint of audio signal 106 (e.g., fingerprint 110 of FIG. 1) by selecting energy extrema of the normalized audio signal. For example, fingerprint generator 210 may use frequency time bins and energy associated with one or more energy extrema (e.g., one extremum, twenty extrema, etc.). In some examples, fingerprint generator 210 may select the maximum energy value of the normalized audio signal 106. In other examples, fingerprint generator 210 may select any other suitable feature of the frequency components of the normalized audio signal. In some examples, fingerprint generator 210 may utilize any suitable means (e.g., algorithms, etc.) to generate fingerprint 110 representing audio signal 106. Once fingerprint 110 is generated, process 700 ends.

The processing 800 in Figure 8 begins at block 802. At block 802, audio processor 108 receives a digitized audio signal. For example, audio processor 108 may receive audio (e.g., emitted by audio source 102 of Figure 1 and captured by microphone 104). In this example, the microphone may include an analog-to-digital converter to convert the audio signal into a digitized audio signal 106. In other examples, audio processor 108 may receive audio stored in a database (e.g., volatile memory 914 of Figure 9, non-volatile memory 916 of Figure 9, mass storage device 928 of Figure 9, etc.). In other examples, the digitized audio signal 106 may be transmitted to audio processor 108 via a network (e.g., the Internet, etc.). Alternatively or additionally, audio processor 108 may receive the audio signal 106 by any suitable means.

In block 804, frequency range splitter 202 divides the audio signal into two or more audio signal frequency components (e.g., audio signal frequency component 402 in Figure 3, etc.). For example, frequency range splitter 202 may perform a Fast Fourier Transform to transform the audio signal 106 to the frequency domain and may perform a windowing function (e.g., Hamming function, Hann function, etc.) to create frequency bins. In these examples, each audio signal frequency component is associated with one or more frequency bins. Alternatively or additionally, frequency range splitter 202 may also divide the audio signal 106 into two or more time periods. In these examples, each audio signal frequency component corresponds to a unique combination of one of the two or more time periods and one of the two or more frequency bins. For example, frequency range splitter 202 may divide the audio signal 106 into a first frequency bin, a second frequency bin, a first time period, and a second time period. In this example, the first audio signal frequency component corresponds to a portion of the audio signal 106 within the first frequency range and the first time period; the second audio signal frequency component corresponds to a portion of the audio signal 106 within the first frequency range and the second time period; the third audio signal frequency component corresponds to a portion of the audio signal 106 within the second frequency range and the first time period; and the fourth audio signal frequency component corresponds to a portion of the audio signal 106 within the second frequency range and the second time period. In some examples, the output of the frequency range splitter 202 may be represented as a spectrum (e.g., the unprocessed spectrum 300 of Figure 3).

In box 806, audio characteristic determiner 204 determines the audio characteristics of each audio signal frequency component. For example, audio characteristic determiner 204 may determine the mean energy of each audio signal frequency component. In other examples, audio characteristic determiner 204 may determine any other suitable audio characteristics (e.g., mean amplitude, etc.).

In block 808, signal normalizer 206 normalizes the audio signal frequency components based on determined audio characteristics associated with each individual audio signal frequency component. For example, signal normalizer 206 may normalize the audio signal frequency components according to the mean energy associated with each individual audio signal frequency component. In other examples, signal normalizer 206 may use any other suitable audio characteristics to normalize the audio signal frequency components. In some examples, the output of signal normalizer 206 may be represented as a spectrogram (e.g., the normalized spectrogram 500 of Figure 5).

In box 810, the audio feature determiner 204 determines whether fingerprint generation should be weighted based on audio category, and process 800 proceeds to box 812. If fingerprint generation is not weighted based on audio category, process 800 proceeds to box 816. In box 812, the audio processor 108 determines the audio category of the audio signal 106. For example, the audio processor 108 may present a cue to the user indicating the audio category (e.g., music, speech, etc.). In other examples, the audio processor 108 may use an audio category determination algorithm to determine the audio category. In some examples, the audio category may be a specific person's voice, general human speech, music, sound effects, and/or advertising.

In box 814, signal normalizer 206 weights the frequency components of the audio signal based on the determined audio category. For example, if the audio category is music, signal normalizer 206 may weight the audio signal frequency components along the columns using different scaling values from zero to one, for each frequency position from high to low frequencies associated with the average spectral envelope of music. In some examples, if the audio category is human voice, signal normalizer 206 may weight the audio signal frequency components in association with the spectral envelope of human voice. In some examples, the output of signal normalizer 206 may be represented as a spectrogram (e.g., spectrogram 600 of Figure 6).

In block 816, fingerprint generator 210 generates a fingerprint of audio signal 106 (e.g., fingerprint 110 of FIG. 1) by selecting energy extrema of normalized audio signal frequency components. For example, fingerprint generator 210 may use frequency time bins and energy associated with one or more energy extrema (e.g., twenty extrema, etc.). In some examples, fingerprint generator 210 may select the maximum energy value of the normalized audio signal. In other examples, fingerprint generator 210 may select any other suitable feature of the normalized audio signal frequency components. In some examples, fingerprint generator 210 may utilize another suitable means (e.g., an algorithm, etc.) to generate fingerprint 110 representing audio signal 106. Once fingerprint 110 is generated, processing 800 ends.

Figure 9 is a block diagram of an example processor platform 900 configured to execute the instructions of Figures 7 and/or 8 to implement the audio processor 108 of Figure 2. For example, the processor platform 900 may be a server, personal computer, workstation, self-learning machine (e.g., neural network), mobile device (e.g., mobile phone, smartphone, tablet computer such as iPad ^™ ), personal digital assistant (PDA), internet device, DVD player, CD player, digital video recorder, Blu-ray player, game console, personal video recorder, set-top box, head-mounted device or other wearable device, or any other type of computing device.

The processor platform 900 shown in the example includes a processor 912. The processor 912 shown in the example is hardware. For example, the processor 912 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In this example, the processor 912 implements an example frequency range splitter 202, an example audio characteristic determiner 204, an example signal normalizer 206, an example point selector 208, and an example fingerprint generator 210.

The processor 912 of the illustrated example includes local memory 913 (e.g., cache). The processor 912 of the illustrated example communicates via bus 918 with main memory, which includes volatile memory 914 and non-volatile memory 916. Volatile memory 914 may be implemented using synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), dynamic random access memory, and/or any other type of random access memory device. Non-volatile memory 916 may be implemented using flash memory and/or any other desired type of memory device. Access to main memory 914, 916 is controlled by a memory controller.

The processor platform 900 shown in the example also includes interface circuitry 920. Interface circuitry 920 can be implemented using any type of interface standard, such as Ethernet interface, Universal Serial Bus (USB), interface, Near Field Communication (NFC) interface, and/or PCI Express interface.

In the example shown, one or more input devices 922 are connected to interface circuitry 920. Input devices 922 allow users to input data and/or commands into processor 912. For example, input devices 922 can be implemented using an audio sensor, microphone, camera (still or video), and/or voice recognition system.

One or more output devices 924 are also connected to the interface circuitry 920 of the illustrated example. The output devices 924 may be implemented, for example, by display devices (e.g., light-emitting diodes (LEDs), organic light-emitting diodes (OLEDs), liquid crystal displays (LCDs), cathode ray tube displays (CRTs), flat panel displays (IPS) displays, touchscreens, etc.), haptic output devices, printers, and/or speakers. Therefore, the interface circuitry 920 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and/or a graphics driver processor.

The interface circuitry 920 of the example shown also includes communication devices (such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and/or network interfaces) to facilitate the exchange of data with external machines (e.g., any kind of computing device) via network 926. For example, the communication may be via Ethernet connections, digital subscriber line (DSL) connections, telephone line connections, coaxial cable systems, satellite systems, line-to-line wireless systems, cellular telephone systems, etc.

The processor platform 900 shown in the example also includes one or more mass storage devices 928 for storing software and/or data. Examples of such mass storage devices 928 include floppy disk drives, hard disk drives, optical disk drives, Blu-ray disc drives, redundant array of independent disks (RAID) systems, and digital universal disc (DVD) drives.

The machine-executable instructions 932 used to implement the method of FIG6 may be stored in a mass storage device 928, volatile memory 914, non-volatile memory 916 and/or on a removable non-transitory computer-readable storage medium (such as a CD or DVD).

Based on the foregoing, it will be understood that example methods and apparatuses for creating fingerprints from audio signals have been disclosed, which reduce the amount of noise captured in the fingerprint. Furthermore, by sampling audio from regions of the audio signal with lower energy, more robust audio fingerprints can be created compared to previously used audio fingerprinting methods.

Although certain example methods, apparatuses, and articles of manufacture are disclosed herein, the scope of this patent is not limited thereto. Rather, this patent covers all methods, apparatuses, and articles of manufacture that fall fully within the scope of the claims of this patent.

Claims (25)

1. An apparatus for audio fingerprint recognition, the apparatus comprising: A frequency range splitter transforms an audio signal into the frequency domain. The transformed audio signal includes multiple time-frequency bins, including a first time-frequency bin. An audio characteristic determiner determines a first characteristic of a first group of time-frequency bins in a plurality of time-frequency bins, the first group of time-frequency bins surrounding the first time-frequency bin; A signal normalizer normalizes the audio signal to generate a normalized energy value, wherein the normalization of the audio signal includes normalizing the first time-frequency bin according to the first characteristic; A point selector, wherein the point selector selects one of the normalized energy values; and A fingerprint generator that uses a selected normalized energy value from the normalized energy values to generate a fingerprint of the audio signal. The point selector also includes: Determine the category of the audio signal; and The selection of one of the normalized energy values is weighted according to the category of the audio signal. Wherein, the black portion of the normalized and weighted spectrogram generated by the point selector represents the high-energy region of the audio spectrum, and Each of the multiple time-frequency bins is a unique combination of the following: (1) the time period of the audio signal and (2) the frequency bin of the transformed audio signal. 2. The apparatus of claim 1, wherein the frequency range separator further performs a fast Fourier transform on the audio signal. 3. The apparatus of claim 1, wherein the category of the audio signal includes at least one of music, human speech, sound effects, and advertisements. 4. The apparatus according to any one of claims 1 to 3, wherein the audio characteristic determiner further determines a second characteristic of a second group of time-frequency bins in the plurality of time-frequency bins, the second group of time-frequency bins surrounding a second time-frequency bin in the plurality of time-frequency bins, and the signal normalizer further normalizes the first time-frequency bin according to the first characteristic. 5. The apparatus according to any one of claims 1 to 3, wherein the point selector selects one of the normalized energy values based on the energy extrema of the normalized audio signal. 6. The apparatus of claim 1, wherein each of the normalized energy values corresponds to a corresponding time-frequency bin in the plurality of time-frequency bins. 7. The apparatus of claim 1, wherein the determined first characteristic includes at least one of the following: (i) mean energy value, (ii) mode energy value, (iii) median energy value, (iv) mode power value, and (v) mean amplitude of the audio signal. 8. A method for audio fingerprint recognition, the method comprising the following steps: The audio signal is transformed to the frequency domain, and the transformed audio signal includes multiple time-frequency bins, including a first time-frequency bin. Determine a first characteristic of a first group of time-frequency bins among the plurality of time-frequency bins, wherein the first group of time-frequency bins surrounds the first time-frequency bin; The audio signal is normalized to generate a normalized energy value. The normalization of the audio signal includes normalizing the first time-frequency bin according to the first characteristic. Select one of the normalized energy values; and The fingerprint of the audio signal is generated using one of the selected normalized energy values. The step of selecting one of the normalized energy values includes: Determine the category of the audio signal; and The selection of one of the normalized energy values is weighted according to the category of the audio signal. In this generated, normalized, and weighted spectrogram, the black portion represents the high-energy region of the audio spectrum, and Each of the multiple time-frequency bins is a unique combination of the following: (1) the time period of the audio signal and (2) the frequency bin of the transformed audio signal. 9. The method of claim 8, wherein the step of transforming the audio signal to the frequency domain includes performing a fast Fourier transform on the audio signal. 10. The method of claim 8, wherein the category of the audio signal includes at least one of music, human speech, sound effects, and advertisements. 11. The method according to any one of claims 8 to 10, further comprising: Determine a second characteristic of a second group of time-frequency bins among the plurality of time-frequency bins, the second group of time-frequency bins surrounding a second time-frequency bin among the plurality of time-frequency bins; and The first time-frequency bin is normalized according to the first characteristic. 12. The method according to any one of claims 8 to 10, wherein the step of selecting one of the normalized energy values is based on the energy extrema of the normalized audio signal. 13. The method of claim 8, wherein each of the normalized energy values corresponds to a corresponding time-frequency bin in the plurality of time-frequency bins. 14. The method of claim 8, wherein the determined first characteristic includes at least one of the following: (i) mean energy value, (ii) mode energy value, (iii) median energy value, (iv) mode power value, and (v) mean amplitude of the audio signal. 15. A non-transitory computer-readable storage medium comprising instructions that, when executed, cause a processor to at least: The audio signal is transformed to the frequency domain, and the transformed audio signal includes multiple time-frequency bins, including a first time-frequency bin. Determine a first characteristic of a first group of time-frequency bins among the plurality of time-frequency bins, wherein the first group of time-frequency bins surrounds the first time-frequency bin; The audio signal is normalized to generate a normalized energy value. The normalization of the audio signal includes normalizing the first time-frequency bin according to the first characteristic. Select one of the normalized energy values; and The fingerprint of the audio signal is generated using one of the selected normalized energy values. When the instruction is executed, it also causes the processor to: Determine the category of the audio signal; and The selection of one of the normalized energy values is weighted according to the category of the audio signal. In this context, the black portion of the normalized and weighted spectrogram represents the high-energy region of the audio spectrum, and... Each of the multiple time-frequency bins is a unique combination of the following: (1) the time period of the audio signal and (2) the frequency bin of the transformed audio signal. 16. The non-transitory computer-readable storage medium of claim 15, wherein the processor further performs a fast Fourier transform of the audio signal. 17. The non-transitory computer-readable storage medium of claim 15, wherein the processor further determines a second characteristic of a second group of time-frequency bins in the plurality of time-frequency bins, the second group of time-frequency bins surrounding a second time-frequency bin in the plurality of time-frequency bins, and the processor further normalizes the first time-frequency bin according to the first characteristic. 18. The non-transitory computer-readable storage medium of claim 15, wherein the processor selects the one of the normalized energy values based on the energy extrema of the normalized audio signal. 19. The non-transitory computer-readable storage medium of claim 15, wherein each of the normalized energy values corresponds to a corresponding time-frequency bin in the plurality of time-frequency bins. 20. The non-transitory computer-readable storage medium of claim 15, wherein the determined first characteristic includes at least one of the following: (i) mean energy value, (ii) mode energy value, (iii) median energy value, (iv) mode power value, and (v) mean amplitude of the audio signal. 21. An apparatus for audio fingerprint recognition, the apparatus comprising: At least one memory; Instructions; and At least one processor, the at least one processor executing the instructions to: The audio signal is transformed to the frequency domain, and the transformed audio signal includes multiple time-frequency bins, including a first time-frequency bin. Determine a first characteristic of a first group of time-frequency bins among the plurality of time-frequency bins, wherein the first group of time-frequency bins surrounds the first time-frequency bin; The audio signal is normalized to generate a normalized energy value. The normalization of the audio signal includes normalizing the first time-frequency bin according to the first characteristic. Each normalized energy value in the normalized energy value corresponds to a corresponding time-frequency bin in the plurality of time-frequency bins. Select one of the normalized energy values; and The fingerprint of the audio signal is generated using one of the selected normalized energy values. Among them, selecting one of the normalized energy values includes: Determine the category of the audio signal; and The selection of one of the normalized energy values is weighted according to the category of the audio signal, and In the generated normalized and weighted spectrogram, the black portion represents the high-energy region of the audio spectrum. Each of the multiple time-frequency bins comprises a unique combination of the following: (1) the time period of the audio signal and (2) the frequency bin of the transformed audio signal. 22. The apparatus of claim 21, wherein transforming the audio signal to the frequency domain comprises performing a fast Fourier transform on the audio signal. 23. The apparatus of claim 21, wherein the category of the audio signal includes at least one of music, human speech, sound effects, and advertisements. 24. The apparatus of claim 21, wherein the at least one processor executes the instructions to: Determine a second characteristic of a second group of time-frequency bins among the plurality of time-frequency bins, the second group of time-frequency bins surrounding a second time-frequency bin among the plurality of time-frequency bins; and The first time-frequency bin is normalized according to the first characteristic. 25. The apparatus of claim 21, wherein the determined first characteristic includes at least one of the following: (i) mean energy value, (ii) mode energy value, (iii) median energy value, (iv) mode power value, and (v) mean amplitude of the audio signal.