WO2015118262A1 - Method for synchronization of a musical score with an audio signal - Google Patents

Abstract

The present invention relates to a method for synchronization of a graphical or audible representation of a musical score with an audio signal acquired in real-time and corresponding to the performance of said musical score. This method consists of producing a spectral signature of a temporal sequence of said acquired audio signal, comparing said spectral signature with the spectral signatures of said musical score for a plurality of tempos, in order to calculate a correlation rating for each of said spectral signatures, and determining an estimated tempo corresponding to the best correlation.

Description

Method of synchronizing a musical score with an audio signal Field of the invention

The present invention relates to the field of synchronization of a graphic or sound representation of a musical score with an audio signal acquired in real time and the display of this synchronization on the graphical display of the score.

State of the art

It is known in the prior art patent EP2387029 describing a system that displays the part of the musical score that will be played by a user, said active part, allowing the repositioning of the part that is already displayed on a support , according to an internal or external time code / metronome. In this system, a software application moves the displayed part to match the active part of the music score. For this purpose, this system uses a data file describing the relative positions of each musical staff and the musical indications such as repetitions or codas, and time data corresponding to the beginning and the end of the respective staves in the score of music.

Also known is US Patent 8431809 describing a method for displaying music files, including an electronic music display system for receiving, storing and displaying music files, and display programming. electronic music in accordance with the order and tempo specified by the user. A starting point indicator is configured to display a position in the score that will be played by a user, and a page change indicator is configured to display a change on a subsequent page.

 [0004] US Patent US 2008196575 describes a method and a system for displaying a digital partition that allows the user to follow the note that he currently plays by the synchronized movement of an indicator on the display. of the digital partition. Each note, chord, or time slice of constant duration in the music score data file is adapted to the note, chord, or time slice in the audio data file. Then a single media file is generated, which contains the synchronized content composed of the audio data file and the data file of the digital music score, so that when the media is played, the indicator is moved to the display of the music file. digital partition corresponding to the transmitted audio signal.

 US Patent 2011230987 discloses a method of synchronizing an audio file and a video file in real time in a multimedia device. The chromatic characteristics are calculated from the audio signal of the audio file and the audio signal signal of the reference video file, in order to determine an optimal alignment path between the audio signal of the audio file and the signal of the audio track of the audio file. video file.

[0006] US Pat. No. 2,010,276,333 describes a method of media synchronization comprising a step of determining a ratio of time differences. A media stream is broadcast by a media source. A client device is used to capture an extract of the media stream broadcast by the media source. A time difference ratio, which is associated with a rate at which the media stream is broadcast by the media source, is determined.

 In paragraph 56 of the US patent

US 20110276333, the calculation of the ratio of relative differences in tempo is illustrated by a concrete example. Firstly, a spectrogram of the captured media extract is presented with a plurality of pairs of spectral peaks of the fingerprint, while a spectrogram of a previously recorded reference medium is also presented with a plurality of pairs of spectral peaks of the fingerprint. In a second step, a ratio is calculated by comparing the time difference between a pair of spectral peaks of the spectrogram of the captured media extract, at the time difference between a pair of spectral peaks of the previously recorded reference media spectrogram. Then the ratios of the other pairs of peaks are calculated in the same way and the series of ratios thus calculated generates a ratio histogram. Finally, the highest ratio of the histogram is referred to as the ratio of time differences for synchronization between the extract and the reference. This ratio of time differences characterizes the ratio of the relative speeds between the two media.

Disadvantages of the Prior Art [0008] The solutions of the prior art present difficulties of interaction with the user and synchronization problems. When the music score is displayed, the indicator is moved from a prerecorded audio data sheet. However, the user of an instrument needs to determine in real time the position of his performance on the display of the musical score, to choose to play the covers autonomously, to repeat certain parts voluntarily, and to evaluate his level of performance. No technique of the prior art gives a satisfactory solution.

 In addition, to satisfy the aforementioned needs of the user, it is necessary to solve problems related to the quality and speed of the synchronization process. However, the existing synchronization algorithms generally have some problems.

 For some algorithms, it is necessary to know the starting point and / or the end point for the audio and visual signals, points where the two audio and visual signals are synchronized. Then it is necessary to establish the correspondence between the two audio and visual signals between the starting and ending points. However, it is mandatory to have reference points in advance. It is thus not possible to synchronize online and in real time.

The chromatographic algorithm does not synchronize accurately in real time. In general, a chromatogram is synthesized by running a sequence of chroma vectors across the time axis with a time window. Synchronization is achieved by connecting the spatial positions in the chromatogram of the audio signal played by the user with the audio signal, or reference video. Due to the approximate character of the chroma vectors, they lack precision. In order to achieve a certain accuracy, synchronization requires matching a larger range of spatial positions in the chromatogram, which leads to a delay. Moreover, it does not detect the error effectively.

Solution provided by the invention

In order to overcome the drawbacks of the prior art, the invention relates, according to its most general meaning, to a new method intended to precisely qualify the acquired audio signal and to perform the transformation of the musical score into different tempos, which allows to synchronize both precisely in real time.

 According to the main claim, the invention relates to a synchronization method of a graphic or sound representation of a musical score with an audio signal acquired in real time and corresponding to the interpretation of said musical score. The method includes generating a digital signature of a time sequence of said acquired audio signal, comparing said digital signature with the digital signatures of said musical score, for a plurality of tempo, for calculating a correlation score for each of said digital signatures, determining an estimated time corresponding to the best correlation, at least periodically determining the position of the time sequence of said audio signal with respect to said representation of said musical score, controlling the displacement of said position with respect to said representation of said score according to said estimated tempo corresponding to the best correlation score, periodically recalculating said correlation score.

[00014] Preferably, the digital signature is the spectral signature and not the chromatic one. According to the characteristics of the invention, the acquired audio signal is processed by the Fourier transform, and then processed by a logarithmic function to better weight the low energy notes (treble, bass).

[00016] Preferably, the standardized cross correlation algorithm JP Lewis is used to calculate a correlation score.

According to the features of the invention, two modes of synchronization exist:

- Global mode to determine the absolute initial position of the user of the instrument with long time windows and a high similarity threshold; - Local mode followed by partition in real time and more tolerant to errors, with short time windows and a similar threshold of similarity.

According to a preferred embodiment of the invention, a sound cancellation procedure is performed by anticipating the synthetic audio signal produced from the estimated tempo, and subtracting it to compare the remaining portion of the signal spectral. audio to that of the score.

According to a particularly advantageous embodiment, the identification and display of the musical score are made by recognizing the attachment of the notes by analyzing the dashed graph; by the connection of the connecting arcs on the result of the computation of the barycenter and the direction of curvature, by the reconstruction in M systems of the partition originally organized in N systems. In one embodiment, a dynamic display is achieved by the immediate replacement of the system, which is detected as already played, to that which will be played by a user. A cursor indicating the position is moved relative to the representation of the musical score according to said estimated tempo corresponding to the best correlation score, and the cursor on the graphical display of the score indicates the displacement of the position, the change of a page, or a recovery.

[00022] Electronic equipment comprising a display, an input for an audio signal and a computer characterized in that said computer is controlled to execute a synchronization program of a graphic or sound representation of a musical score with an audio signal acquired in real time and corresponding to the interpretation of said musical score, this synchronization program consisting in: producing a digital signature of a temporal sequence of said acquired audio signal,

 comparing said digital signature with the digital signatures of said musical score, for a plurality of tempo, for calculating a correlation score for each of said signatures,

determining an estimated time corresponding to the best correlation,

 at least periodically determining the position of the temporal sequence of said audio signal with respect to said representation of said musical score,

to control the displacement of said position with respect to said representation of said partition according to said estimated tempo corresponding to the best correlation score,

 - periodically recalculating said correlation score.

 [00023] Computer program for controlling the synchronization of a graphic or sound representation of a musical score with an audio signal acquired in real time and corresponding to the interpretation of said musical score, according to the steps of: - producing a digital signature of a temporal sequence of said acquired audio signal,

 comparing said digital signature with the digital signatures of said musical score, for a plurality of tempo, for calculating a correlation score for each of said signatures,

determining an estimated time corresponding to the best correlation,

 at least periodically determining the position of the temporal sequence of said audio signal with respect to said representation of said musical score,

controlling the displacement of said position with respect to said representation of said musical score according to said estimated tempo corresponding to the best correlation score,

 - periodically recalculating said correlation score.

 The position of the temporal sequence of said audio signal is determined periodically with respect to the representation of the musical score by periodically recalculating the correlation score. So the tempo is checked regularly.

Advantageously, it proceeds to a logarithmic processing of the acquired signal to better weight the low energy grades (treble, bass). The invention is implemented using a calculation by a fast algorithm that allows for increased reactivity in real time, such as the standardized cross-correlation algorithm.

Detailed description of a non-limiting example of

 production

Other features and advantages of the invention will become apparent on reading the description given below of particular embodiments of the invention, given for information but not limited to.

 The invention will be better understood and other features and advantages will appear more clearly on reading the description which follows, with reference to the accompanying drawings in which:

 Figure 1 is a diagram of the principle of the synchronization method according to the invention;

 FIGS. 2a, 2b and 2c are the representations of the processing steps of the audio signal acquired according to the invention;

 Figures 3a, 3b and 3c are the spectral signatures of a musical score at different tempos according to the invention;

 FIGS. 4a, 4b, 4c and 4d represent the correlation scores as a function of the position and of the different tempos according to the invention;

 Figure 5 illustrates the displacement of a cursor on the graphical display of the score;

 FIG. 6 is a diagram of the principle of the synchronization method with the procedure for canceling the sound emitted by a loudspeaker,

FIGS. 7a, 7b, 7c and 7d represent the method of a dynamic display, Figures 8a and 8b illustrate the attachment of the signs remaining outside the five main lines of the scope. Non-limiting description of exemplary embodiments

FIG. 1 represents the synchronization method, which comprises the following steps: the sampling of the audio signal, the creations of the digital signatures of the audio signal and the musical score, the comparison of the two digital signatures and the display of the result of the synchronization on the screen.

 As shown in Figure 1, on the left, an audio signal is recorded by a microphone. We proceed to the successive analysis of windows of sound duration of 200ms. The processing of each sound window is done with an overlap at 75% from one window to the other. After each time interval of 50 ms, ie ¾ of an entire duration of a window, a new window is analyzed.

 The audio signal is picked up by the microphone and is amplified, digitized and stored in a RAM, for temporary storage. The spectral signature for each note and instrument is also pre-recorded in RAM for quick access.

The spectral analysis of the sampling window of the audio signal is performed by the processor. A digital signature of a temporal sequence of this acquired audio signal is produced. Also the reconstruction of the digital signatures of the musical score is performed with a plurality of different tempos. [00033] A correlation method is performed by comparing the digital signature of the audio signal and that of the musical score of each different tempo respectively. A correlation score for each comparison is calculated.

 If the correlation is sufficient, the location position in the score and the tempo of the user of the instrument (or player) are determined, which makes it possible to move the cursor on the display of the score corresponding to the tempo from the game position in the screen. In addition, the display of the dynamic partition is adjusted by replacing the system played and switching to a new page.

 The hard disk is used to pre-record the information of the score: notes, rhythms, information pages and covers and geometric positions of the notes. This information is processed by the processor to reconstruct the digital signature of the partition.

 The acquired audio signal is processed in several steps. First, it is necessary to apply windowing, which consists in multiplying the window by a predetermined function, in order to reduce the effects of noise.

Then the Fourier transform is applied to analyze the frequency characteristic. Centered on the sampling window, several Fourier transforms (FFTs) are applied, with different window sizes but centered on the same point. A characteristic frequency band of each of these transforms is selected. These strips thus collected are then combined following one of the others to represent together the complete digital signature comprising the low frequencies, the average frequencies and high frequencies. Figure 2a shows the result of the raw energy transform for a given window size. The X axis represents the frequency while the Y axis represents the energy.

 [00038] A fast Fourier transform is a technical operation performed on a value window. It gives a result whose size is half that of the analyzed window. The amplitude of the fast Fourier transform is proportional to the size of the window, while the energy is the amplitude squared of the fast Fourier transform. The total energy of a window is calculated, and then divided by the number of points in that window to determine a value of the average energy. Therefore, the average energy is different for each window size of the fast Fourier transform corresponding respectively to the low frequencies, the medium frequencies and the high frequencies. Then a logarithmic function is applied to each window with their respective average energy. Each point of the window is then passed to the log according to the formula:

 V = log10 (Point Value / Average Energy)

 All negative values are then passed to zero. Figure 2b shows the result of the calculation of the fast energy Fourier transform at the logarithmic scale. The high frequency component is more prominent.

 The final stack represents only a selected part of the calculation of the fast Fourier transform of each window, which is the condensed frequency bands interesting to analyze.

In order to present the final standardized stack at the energy scale, a wider time analysis window - in 2 seconds - is selected. In Figure 2C, the X axis represents the sequence of 2 seconds, the Y axis represents the frequency, while the 2D contours represent the value of the energy. The presentation of the discrete energy contour as a function of time and frequency is called spectral.

 The advantages of this transition to logarithm are, on the one hand, to eliminate the noise of the spectral analysis, on the other hand, to reduce the values of harmonics and frequencies in relevant scales, which make it possible to distribute the weight on the characteristic frequency bands of each of the transforms for the correlation.

 The same method is applied for the reconstruction of the digital signature of the musical score.

 Although the rhythm is fixed in the score, the different users of the instrument play at different tempos according to the interpretation of each. Several tempos are used for the reconstruction of the digital signature of the score, of which three are chosen in the representations of Figures 3a, 3b and 3c.

 The reconstruction of the digital signature of the partition is performed from prerecorded raw spectral signatures for each note and each instrument, then stored in RAM for quick access. The log is applied in the same way to the sums of the spectral representations of the notes according to the rhythm and the tempo.

 The rhythm designates notions of ebony, black, and white type temporality, while the tempo defines the general appearance of a piece.

[00046] FIG. 3a is a spectral of a slow tempo signature. The X axis represents the course of time, the Y axis represents the frequencies, while contours represent logarithmic energy.

 As the tempo accelerates in FIGS. 3b and 3c, the spectrals become more and more dense along the X axis.

 [00048] If the musician played perfectly with a perfectly tuned piano, an "ideal" spectral representation of the score would be very close to or equal to the audio spectral acquired.

 The two spectrals are compared using the standardized cross correlation algorithm. In order to quantify the result of this comparison, a correlation score is calculated. In the ideal situation with two identical spectrals, the score is 1, which means a perfect correlation.

 [00050] A large window is chosen by calculating the scores at different tempos to locate the playing position of the musician and the corresponding tempo. The higher the score, the closer the musician's game approaches what should be heard. If the peak score is high enough, the position is retained and the tempo too. This gives a good evaluation of the tempo and the initial position.

 Figures 4a, 4b and 4c illustrate comparison scores as a function of position for the slow, normal and fast tempo. Figure 4d illustrates the overlap of the comparison scores at the three tempos. As soon as the prior similarity threshold is exceeded, the normal tempo and the corresponding position are retained.

[00052] FIG. 5 illustrates the displacement of the cursor on the display of the score corresponding to the tempo and the position determined by the correlation score which exceeds the similarity threshold. [00053] Two synchronization modes exist for the different situations: firstly, global mode for determining the absolute initial position. The player's performance is qualified with long time windows and a high absolute score to locate what he is playing. Secondly, local mode to qualify the player, the score is followed in real time, with short time windows and a lower absolute score to be less sensitive to errors.

 [00054] FIG. 6 illustrates the method of synchronization with a procedure during which different parts of the score are distinguished (for example the part played by the left hand and the part played by the right hand) and the sound emitted by a top is canceled. - speaker playing one of these parts to track the performance of a user playing that part of the score. Similar to the method presented for FIG. 1, this sound emitted by the loudspeaker is canceled in the spectral analysis.

 With the pre-recorded information (notes, rhythms, page and repeat information, etc.), and the synthetic sound bank, the synthetic creation of the synchronized output sound signal is performed in the processor.

 After temporary storage in the RAM, the speaker gives the sound signal in the output.

 The sound signals emitted by the instrument and the loudspeaker are recorded in the microphone during the sampling process.

In the spectral analysis method, the synthetic spectral is subtracted, the remaining part of the spectral is compared to the spectral of the partition. In addition to the synchronization applications, this invention is also suitable for applications such as the identification and display of the musical score. In this case, it proceeds to a processing of the partition organized in N systems of staves (or single range) on a support such as a page, to reconstruct M scope systems on the display screen.

 In a situation, a score is only written for an instrument with a single staff.

This is the case of the vast majority of classical instruments (flute, violin, etc.). In another situation, several staves are brought together to be played together, for example the solo piano, which has a right-hand reach and a left-handed reach. A "system" includes all staves played together.

 It is possible that a score has a large number of instruments and therefore staves that are played at the same time, such as an orchestral score.

In this case, it usually happens that it is possible to represent only one system of musical range per page, especially if the size of the display screen is reduced compared to a normal paper representation.

 The following example illustrates the display in 3 systems of scope per page on the screen music score, which was organized in 4 systems per page scope according to the traditional edition.

A dynamic display is performed on this new mode of display, by the immediate graphic replacement of the system which is detected as already played and whose display has become useless, by the next system of scope to be read and still not visible in the display order. [00064] FIG. 7a illustrates the traditional two-page musical score with eight litter systems, in which they contain covers at different locations. Figure 7b shows the reading order of the litter systems with times.

 In the new mode of the dynamic display, the number of systems per page is adapted to the size of the screen. Figure 7c shows an example of the display of three scope systems per page. The progress of the system display is in positions 1, 2 and 3 successively. Figure 4 illustrates the dynamic system replacement sequence for each position.

 [00066] In order to conform to the new system of the display, the techniques are developed to recognize the signs remaining outside the five main lines of the scope of the traditional partition.

 Figure 8a illustrates the recognition of very serious notes by the graphical analysis of the dashes. A sixteenth note linked to it is determined by the attachment of the system at the top.

 FIG. 8b illustrates the recognition of connecting arcs, by the calculation of the center of gravity and the direction of curvature. The connecting arc in gray, whose direction of curvature is upward, is attached to the upper system. While the black connecting arc of the direction of curvature downward is attached to the lower system.

The invention relates to a method for synchronizing a graphic representation, in particular of a musical score, with an audio signal acquired in real time, and corresponding to the interpretation of the aforementioned musical score. It is not a question of looking for the identification of a captured media extract in a database of a partition collection, but of providing assistance to an interpreter to facilitate automating the tracking of a known partition, for example to allow an instrumentalist to interpret his part synchronously with the reproduction of an orchestral recording.

 In general, the invention is based on a technical solution ensuring high robustness of the treatment. It consists in constructing, for each partition, a plurality of signatures each corresponding to a given speed of interpretation of this partition. When the partition is executed by a user, the method consists in calculating the signature corresponding to this execution speed by the user, and in the collection of speed signatures, which has the best correlation rate. with the signature of the partition executed by the user.

 The solution given in US patent US 20110276333 is however fundamentally different from the invention. It consists of :

 - Generate a signature corresponding to the identified partition,

 calculate the signature of an audio sequence executed by the user,

 determining the time difference ratio between the signature of the audio sequence executed and the signature of the identified partition,

 use this ratio to control the synchronization tracking

This solution associates a single signature with the same score and starts from a questionable postulate according to which there is a homothetic relationship between the tempo and the signature.

However, the invention in the present application provides a solution for abstaining from this postulate. It provides a plurality of signatures for the same partition at different speeds. And it allows to select the right speed for synchronization, even in cases where the variation of the signature is not directly representative of the variation of the speed.

 It is thus possible to follow in real time with great precision the execution of a musical score by a user even when he changes the speed at which he executes this partition. Therefore, the present invention far surpasses the state of the art.

Claims

claims 1 - A method of synchronizing a graphic or sound representation of a musical score with an audio signal acquired in real time and corresponding to the interpretation of said musical score, consisting of:  to produce a digital signature of a temporal sequence of said acquired audio signal,  comparing said digital signature with the digital signatures of said musical score, for a plurality of tempo, for calculating a correlation score for each of said signatures,  determining an estimated time corresponding to the best correlation,  at least periodically determining the position of the temporal sequence of said audio signal with respect to said representation of said musical score,  controlling the displacement of said position with respect to said representation of said musical score according to said estimated tempo corresponding to the best correlation score,  - periodically recalculating said correlation score. 2 - A method of synchronizing a graphic or sound representation of a musical score with an audio signal according to claim 1 characterized in that the digital signature is the spectral signature and not chromatic. 3 - A method of synchronizing a graphic or sound representation of a musical score with an audio signal according to claim 2 characterized in that the acquired audio signal is processed by Fourier transform, and then processed by a logarithmic function to better weight the low energy notes (treble, bass). 4 - A method of synchronizing a graphic or sound representation of a musical score with an audio signal according to claim 1 characterized in that the standardized cross correlation algorithm is used to calculate a correlation score. 5 - Method for synchronizing a graphic or sound representation of a musical score with an audio signal according to claim 1 characterized in that two synchronization modes exist:  - Global mode to determine the absolute initial position of the user of the instrument with long time windows and a high similarity threshold;  - Local mode followed by partition in real time and more tolerant to errors, with short time windows and a similar threshold of similarity. 6 - A method of synchronizing a graphic or sound representation of a musical score with an audio signal according to claim 1 characterized in that a sound cancellation procedure is performed by the anticipation of the synthetic audio signal produced at from the estimated tempo, and subtracted to compare the remaining part of the spectral of the audio signal to that of the score. 7 - A method of synchronizing a graphic or sound representation of a musical score with an audio signal according to claim 1 characterized in that the identification and display of the musical score are made, by recognizing the attachment of the notes. analyzing the graph of dashes; by attaching connecting arcs on the result of the calculation of the center of gravity and the direction of curvature, by the reconstruction in M systems of the partition originally organized into N systems. 8 - A method of synchronizing a graphic or sound representation of a musical score with an audio signal according to claim 1 characterized in that a dynamic display is achieved by the immediate replacement of the system, which is detected as already played, by the one that will be played as a result by a user. 9 - A method of synchronizing a graphic or sound representation of a musical score with an audio signal according to claim 1, characterized in that a cursor on the graphical display of the score indicates the displacement of the position, the change of position. a page, or a recovery. 10 - Electronic equipment comprising a display, an input for an audio signal and a computer characterized in that said computer is controlled to execute a synchronization program of a graphic or sound representation of a musical score with a time-acquired audio signal real and corresponding to the interpretation of said musical score, this synchronization program consisting of:  to produce a digital signature of a temporal sequence of said acquired audio signal,  comparing said digital signature with the digital signatures of said musical score, for a plurality of tempo, for calculating a correlation score for each of said signatures, determining an estimated time corresponding to the best correlation, at least periodically determining the position of the temporal sequence of said audio signal with respect to said representation of said musical score,  controlling the displacement of said position with respect to said representation of said musical score according to said estimated tempo corresponding to the best correlation score,  - periodically recalculating said correlation score. 11 - Computer program for controlling the synchronization of a graphic or sound representation of a musical score with an audio signal acquired in real time and corresponding to the interpretation of said musical score, according to the steps of:  to produce a digital signature of a temporal sequence of said acquired audio signal,  comparing said digital signature with the digital signatures of said musical score, for a plurality of tempo, for calculating a correlation score for each of said signatures,  determining an estimated time corresponding to the best correlation,  at least periodically determining the position of the temporal sequence of said audio signal with respect to said representation of said musical score,  controlling the displacement of said position with respect to said representation of said musical score according to said estimated tempo corresponding to the best correlation score,  - periodically recalculating said correlation score.