JP2003263170A - Method for analyzing waveform of musical sound and method for analyzing and synthesizing waveform of musical sound - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To highly accurately analyze the waveform of a musical sound. <P>SOLUTION: In analyzing the waveform of a musical sound as shown in the figure, a point of which the amplitude is a maximum value is set as a specific point and spectral analysis is performed by short Fourier transformation while successively shifting a time window in the positive direction of a time base from the specific point. Then the spectral analysis is performed while successively shifting the time window in the negative direction of the time base from the specific point. The locus of peaks detected in each frame is judged in both the directions to follow the locus. The spectral analysis is performed under a different analytical condition in each harmonic. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

TECHNICAL FIELD The present invention relates to a musical tone waveform analysis method and a musical tone waveform analysis / synthesis method.

[0002]

2. Description of the Related Art As one method of synthesizing a musical tone, a musical tone waveform is analyzed, a signal of a frequency component contained in the original musical tone waveform obtained by the analysis is generated, and these are added, Analysis that synthesizes musical sound waveforms
An (Analysis & (Re) Synthesis) method is known. In this tone waveform analysis and synthesis method, first,
The musical tone waveform is spectrally analyzed to extract line spectrum components corresponding to the fundamental frequency and its overtone frequency contained in the musical tone. Usually, this spectrum analysis is performed using short-time spectrum analysis by Fourier transform using a time window. That is, a musical tone to be analyzed is sampled, and the musical tone waveform sample is multiplied by a window function to obtain an FFT (Fast Fourier Transform).
er Transform) to detect all frequency positions forming a peak from the amplitude data of the Fourier transform output. The above processing is performed while moving the time window (short-time fast Fourier transform (SFFT)).
orm)), the peaks in each frame are detected, and the obtained peaks are traced. And
Desired data is selected from the obtained loci, and each of them is sinusoidally synthesized and added to synthesize a deterministically obtained waveform of the original musical tone waveform. Then, a residual waveform is obtained by subtracting the deterministically obtained waveform (Deterministic Wave) from the original musical tone waveform.

The deterministically obtained waveform can be modified by modifying the data of the locus, and the residual waveform is EQ (equalizer) or F.
It can be modified by signal processing such as FT. A desired musical tone waveform can be obtained by adding the thus-determined deterministically obtained waveform and the residual waveform. The analysis of the musical tone waveform is not only for synthesizing the musical tone waveform as described above, but also for clarifying the relationship between the characteristic of the musical instrument sound and the physical property of the musical instrument, or the machine recognition of the musical instrument sound, etc. It is also used for.

FIG. 20 is a diagram showing an example of the peak data obtained as a result of such waveform analysis. In the example shown in this figure, the center C of a piano (C4, the fundamental frequency is about 26
1.63 Hz), the horizontal axis is time (unit is ms), the vertical axis is frequency (unit is Hz), and peak data detected for each frame corresponding to the time window is a point. It is represented by. Further, the horizontal thin line represents the fundamental frequency and the frequency of the overtone thereof. In this figure, all points exist independently, and you can see the line that draws the locus in the figure, but information that they are connected has not yet been obtained. Also,
It is considered that noise is present in the area other than the portion visible as a line, that it is an anharmonic component, or that the side lobe of the window function of the FFT is displayed. In order to extract the data that forms the locus from such peak data,
It is necessary to determine how the peak points at each frame time are connected. For this,
Generally, picking of a component having a harmonic relationship in each frame is performed. That is, it is determined that the point having the maximum amplitude in the vicinity of the frequency of 2 × FB, 3 × FB, ..., which is an integral multiple of the known fundamental frequency FB, is the overtone, and the overtone data is picked up for each frame. The method of doing is taken.

[0005]

FIG. 21 corresponds to FIG.
It is the figure which expanded about 20 msec of the fundamental tone in. In this figure, when the time axis is viewed from right to left, it is correct that the peaks are connected in the direction of the arrow at time A. However, in the conventional method described above,
Since the tracking is performed under the condition that the one closest to the harmonic overtone frequency is selected in each frame, there is a problem that it is not certain whether the peak at time a is connected to the peak at a or the peak at b.

FIG. 22 shows a piano C1 (32.7).
It is a figure which shows the result of having detected the musical tone peak of (03 Hz) by the conventional method mentioned above. This data shows that the width of the time window (window size) is 8 when the basic period of C1 is 30.5 ms.
Double, 244.6 ms, and the window moving time (window hop size) is 1/8 of the basic period (3.82 m).
It was obtained as s). In addition, FIG. 23 corresponds to FIG.
It is the figure which expanded the peak in the higher harmonic part of 2.
In these figures, the horizontal axis represents time, the vertical axis represents frequency, and the horizontal thin line represents the fundamental frequency and its overtone. It can be seen from FIG. 23 that the locus of higher harmonics is considerably deviated from the harmonic frequency. This is, for example, 8
It is considered that this is because the overtone (261.62 Hz) component includes 64 cycles in the above window.
As an uncertainty of the short-time fast Fourier transform (SFFT), a certain window size or more is required to accurately obtain the frequency. In order to accurately detect around 32.703 Hz, a window size equal to or larger than the above is required. However, when the window is wide, there is a problem that the time resolution becomes poor. Generally, SF
As a property of FT, even if the frequency and the amplitude change within one window, the information cannot be obtained. If it is assumed that the harmonic component changes faster, it is considered that the 8th overtone has frequency and amplitude fluctuations within 64 cycles. This situation becomes more remarkable as the number of harmonics increases.

Therefore, in order to prevent such a problem, a method is used in which the signal to be analyzed is band-divided by applying BPF, and the window size and the hop size of the window are determined for each divided band to carry out the SFFT. Has been. This method uses a low-pass filter (LPF) 101, a plurality of band-pass filters (BPF) 102, and a high-pass filter (HPF) 103 as in the tone analysis / synthesis apparatus shown in FIG. , And the signals of the respective frequency bands, which are band-divided by the multiplier 104, are multiplied by the corresponding time windows, and the spectrum analysis of each band is executed by the corresponding SFFT units 105. The peak detection and peak tracking processing unit 106 separately provided for the analysis result tracks the locus of the peak, and the sine wave synthesizing unit 107 generates sine waves corresponding to the loci detected in the respective bands. Then, they are combined and the above-mentioned deterministic waveform (Deterministic Wave) is output.

However, with this method, it is difficult to uniformly design the phase characteristics of the LPF 104, the plurality of BPFs 102, and the HPF 103, and from the trajectories of the overtones obtained as a result of the analysis in this way, Determining by sine wave synthesis
Even if the inistic wave is combined, a phase shift occurs between the signal under analysis and the signal under analysis
There was a problem that Residual Wave could not be obtained by the method of subtracting tic Wave.

That is, in the prior art, when trying to separate a waveform having a low fundamental frequency and abundant high-order overtones, the frequency accuracy of the fundamental frequency is increased due to the uncertainty of the SFFT. If the window size is increased in an attempt to raise the value, it becomes impossible to follow the change in the harmonic. Also, if the window size is reduced in order to follow changes in harmonics, the frequency accuracy of the fundamental frequency will decrease, and Deterministic and Res
There is a problem that the separation accuracy of idual becomes worse.

Therefore, it is an object of the present invention to provide a high-accuracy tone analysis method and apparatus capable of accurately following a peak in each frame. Also,
It is an object of the present invention to provide a musical tone waveform separation method and device that can separate with high accuracy even if the waveform has a low fundamental frequency and is rich in high-order overtones.

[0011]

In order to achieve the above object, the musical tone waveform analysis method of the present invention is characterized in that the time window is obtained by performing a Fourier transform using time windows sequentially shifted with respect to the musical tone waveform sample to be analyzed. In a musical tone waveform analysis method for detecting a peak point in each frame corresponding to the position of the spectrum and performing spectrum analysis, the spectrum is analyzed using analysis conditions set corresponding to each of a plurality of overtone groups of the musical tone to be analyzed. It is designed to be analyzed.

The musical tone waveform analysis and synthesis method of the present invention is
A line spectrum component is extracted from the musical tone waveform, a sine wave signal waveform corresponding to the extracted line spectrum component is generated and synthesized, and a residual component obtained by subtracting the synthesized signal waveform from the musical tone waveform and the synthesized signal. A method of analyzing and synthesizing a musical tone waveform by adding a waveform and a musical tone, wherein when the musical tone waveform is analyzed, analysis conditions set respectively corresponding to a plurality of overtone groups of the musical tone waveform are used. Then, the musical tone waveform is analyzed, and the line spectrum component is extracted based on the analysis result. Further, the residual components corresponding to the overtone groups are extracted, and the residual components are sequentially analyzed for different groups.

[0013]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a block diagram showing an example of the hardware configuration of a musical tone analysis / synthesis apparatus for executing the musical tone analysis method of the present invention. In this figure, 1 is a CPU that controls the entire tone analysis / synthesis apparatus, and 2 is a CPU 1.
The program memory 3 for storing various programs such as various control programs, musical tone analysis programs and musical tone synthesizing programs executed by is used for storing various control information, various data described later, and a temporary storage area (buffer) or work area. Data memory, 4 is a display device, 5 is an input device such as a keyboard and pointing device, 6 is a performance operator such as a keyboard, 7 is a musical sound synthesizing unit (synthesizing unit) for synthesizing musical sounds, and 8 is a musical sound waveform sample. Digital-to-analog converter (DA that converts analog signals and outputs to a sound system not shown)
C). Also, 9 is a telephone line, the Internet, L
A network interface circuit 10 for connecting to a communication network 11 such as an AN is a system bus. In the hardware configuration shown in FIG. 1, the musical tone synthesizer 7 and the performance operator 6 are provided, but they need not be provided. Although not shown, a CD-ROM, DVD, MO,
It goes without saying that a drive device for an external storage medium such as an FD may be connected.

FIG. 2 is a diagram showing an example of the internal configuration of the musical sound synthesizing section 7. FIG. 2 (a) shows a plurality of sine waves that generate the waveforms of the frequency components corresponding to the respective loci described above. It is a figure which shows one structural example in the case of having a waveform generation part. In FIG. 2A, 71 and 73 are the system bus 1
An interface circuit for connecting to the CPU 1 via 0, 72 is a sine wave waveform calculator, and is provided with a plurality of sine wave waveform generators SWG1 to SWGn as shown in the figure. The plurality of sine wave waveform generators SWG1 to SWG
The WGn generates a sine wave waveform corresponding to each locus of each peak point detected by analyzing the musical tone waveform. Reference numeral 74 denotes a residual waveform calculation unit, which generates the residual waveform (Residual Wave) described above. Further, 75 is a mixer, which synthesizes the output of the sine wave waveform computing unit 72 and the output of the residual waveform computing unit 74 and outputs a synthesized musical tone to the DAC 8.

FIG. 2B is a diagram showing another example of the configuration of the musical tone synthesizer 7. In this figure, 76 is the CP
Interface circuit with U1, 77 is waveform memory, 7
Reference numeral 8 is a phase generator that controls the reading of waveform data from the waveform memory 77, and 79 is a waveform processing unit that performs desired processing on the read waveform sample. In this example, a synthesized waveform of a sine wave waveform and a synthesized waveform of the residual waveform corresponding to respective loci of peak points obtained by analyzing the musical tone waveform sample are arithmetically generated by the CPU 1 by software, It is adapted to be stored in the waveform memory 77. Then, in response to the tone generation control signal, the phase generator 78 causes the waveform memory 7 to operate.
The synthesized musical tone signal waveform is read from 7 and output to the DAC 8 via the waveform processing section 79.

The waveform analysis processing in the tone analysis method of the present invention will be described below. FIG. 3 is a diagram showing an example of a time waveform of a C4 keystroke sound of the same piano as in FIG.
A case will be described below as an example where peaks are tracked using this waveform. FIG. 4 is a flowchart for explaining the waveform analysis process. When performing the waveform analysis process, it is naturally possible to shift the frame little by little from the beginning of the waveform to be analyzed, find the peaks, find all the peaks, and then follow the peaks. A huge storage capacity may be required to store all the peak data. Therefore, in the present invention, tracking is checked for each analysis frame, and only peaks that can be tracked are stored.

When the waveform analysis processing is started, first, in step S1, sampling processing of the analysis target waveform is executed, and the sampling data of the musical instrument sound to be analyzed is stored in the data memory 3. That is, for example, the musical tone waveform to be analyzed is sampled at a sampling frequency of 44.1 kHz, and the waveform to be analyzed as shown in FIG. 3 is stored in the data memory 3. Next, in step S2, analysis conditions are input and analysis preprocessing is performed. That is, the step S
Amplitude adjustment, equalization processing, filtering processing, and the like of the analysis target data stored in 1 are performed, and at the same time, the analysis range is set.

Subsequently, in step S3, a time point at which the envelope of the data to be analyzed is maximized (hereinafter referred to as Attack Max Point) is detected. This is because there is a high possibility that all the peaks that need tracking follow at the attack max point where the amplitude level is the maximum, so the time when the amplitude level of this time waveform shows the maximum amplitude is at the center position of the window size. This is because the analysis is started in such a manner as to come out. In the example of FIG. 3, the attack max point is found 42.9 ms after the start of sampling.
In this embodiment, the sampling frequency is 44.1k.
Since it is Hz, 42.9 ms is obtained as the 1892th point. As a result, it becomes possible to detect a peak that needs to be tracked without leaking.

Next, in step S4, the initialization of the analysis window and the like are performed. Generally, it is appropriate that the FFT window size is an integral multiple of the fundamental period of the waveform, for example, 8
To be doubled. The C4 keystroke sound shown in FIG. 3 has a fundamental frequency of 261.63 Hz, and the number of sampling points included in one cycle is 44.1 at the sampling frequency fs.
When it is set to kHz, it becomes 168.6 points. Therefore,
The window size is 8 times this, 1348 points (rounded up to the nearest whole number). Further, the FFT size must be a power of 2, and as described above, the larger the FFT size, the higher the frequency resolution. However, if it is too large, it takes a long time to calculate, so here, the minimum power number of 2 that exceeds the window size is selected. Therefore, in this example, 2048 points are determined.

Succeedingly, in a step S5, a flag n for counting the frame position is initialized to 0. Then, in step S6, the analysis window position is set. As mentioned above, the attack max point is 1892 points,
The first frame is 1 because the window size is 1348 points
From the 218th point to the 2566th point. This is the 0th frame. Succeedingly, in step S7, the analysis range of the analysis target waveform is the maximum value framemax of the analysis range of the analysis target waveform.
It is determined whether or not is exceeded. If the determination result is NO, the process proceeds to the next step S8 and the FFT process is executed. If the determination result is YES, step S
Proceed to 15.

The FFT processing is executed in step S8, and the peak is obtained from the result of the FFT processing in step S9. An example of the result of obtaining this peak is shown in FIG. Each peak of this 0th frame is PK0-
1, PK0-2, ..., PK0-M0. Here, each peak has a frequency (Frequency) and an amplitude (Magnitud
It has three pieces of information, e) and phase. This is PK0
-1.Frequency, PK0-1.Magnitude, PK0-1.Phase, ... Note that M0 is the total number of peaks detected in frame 0. These peaks are the reference (Fr
It is stored in a buffer in the data memory 3.

Succeedingly, in a step S10, it is determined whether or not the frame number n is 0. Immediately after the start of processing, n = 0, so the determination result of step S10 is YES, and step S11 is executed. In this step S11, the data stored in the buffer
The data of Frame 0 corresponds to the corresponding location TFra in the table area for storing the analysis result secured in the data memory 3.
Stored in me 0. When n is not 0, the peak follow-up processing is executed in step S12, and subsequently, in step S13, the follow-up result is stored in the table for storing the analysis result. Details of this tracking process will be described later.

FIG. 6 is a diagram showing an image of a table that stores information on peaks obtained as a result of analysis.
In the table shown in this figure, the rows are the line spectrum components, the columns are all the analyzed frames, and the number of the peak detected in each frame is shown in each cell.
In addition, here, each peak point is numbered in order from the lowest frequency. By the step S11, the information of each peak point obtained in the step S9 is written in the column of Frame 0 of this table.

As described above, each peak point includes three elements of frequency, amplitude and phase. Actually, the data of the number of line spectrum components × total number of frames shown in FIG. 6 is three. One sheet will be created. At the time of the above-described musical tone synthesis, this table is read out, a sine wave waveform generator is assigned to each row, and a sine wave waveform is generated based on each data (frequency, amplitude and phase) as the frame progresses. Become. If the channels of the sine wave waveform generator are flexibly assigned depending on the occurrence and disappearance of the spectrum before and after, it is possible to improve the processing efficiency at the time of synthesizing a musical sound, and to reduce the difference in the occurrence of compound sounds. It is possible to effectively use the sine wave waveform generator. For example, in the example shown in FIG. 6, the 12th peak disappears in the xth frame, and the yth and y + th peaks are eliminated.
Although it reappears in one frame, as shown in FIG. 6, it is possible to use an empty channel by referring to the allocation states before and after the frame as long as the continuity of the peak component is not broken. Good.

Now, the above-mentioned step S11 or S13
After the above, the frame number n is incremented by 1 (step S14), and the process returns to the step S6. In the analysis window position setting process of step S6, the analysis window position corresponding to the (n + 1) th frame is set. At this time, the frame is moved along the time axis in the positive direction of the time axis. In this embodiment, the number of moving sample data is 1/8 of the basic period, that is,
21 points. Therefore, for example, for the first frame, the analysis window position (12
The 1239th to 2587th points, which are obtained by shifting the 18th to 2566th points) by 21 points in the positive direction of the time axis, are set as the analysis window positions of the first frame. And
The steps S7, S8, and S9 are executed, and the peaks PK1-1, PK1-2, ..., PK1-M1 (M1 is the total number of found peaks) in the first frame are detected.

In this case, step S10 described above is performed.
The determination result of No is NO, and the peak tracking process of step S12 is executed. Here, the peak detected in the 0th frame (the peak reference 0 and the peak detected in the first frame are bidirectionally compared by a method described later to obtain a tracking result. The tracking result is obtained. Only the connected peaks are stored in the above-mentioned table of the memory, and at the same time, they are used as a reference for the next frame.In the example shown in Fig. 6, the peak numbered 11 in the 0th frame is added. It is shown that there is no connected peak in the first frame.Hereafter, the processes of steps S6 to S14 are repeated, and the peak tracking process is executed until the end of the analysis range of the analysis waveform. First, while sequentially shifting the frame from the 0th frame in the positive direction of the time axis, peak tracking processing is performed up to the last position in the analysis range. It is executed.

When the end of the analysis range is reached,
The determination result of step S7 is YES, and the process proceeds to step S15. In step S15, the frame number n is set to -1. Thereafter, the peak tracking process is started from the 0th frame along the negative direction of the time axis. That is, in step S16, the analysis window position is set. Unlike the processing in step S6, the processing for setting the analysis window position in step S16 is performed by advancing the time in the reverse order. That is, as described above, the 0th frame analysis window position is shifted by 21 points in the negative direction of the time axis, and the 1197th to 2545th points are set as the -1st frame analysis window position. Then, in step S17, it is determined whether or not the analysis range is exceeded, and if the determination result is NO, step S1
At 8, the FFT process is executed. Then, the peak detection processing is executed (S19) and the peak tracking processing is executed (S20), as in the case described above. Then, the follow-up result is stored in the table (S21). Then, in step S22, n-1 is newly set as n, and the processes in and after step S16 are repeatedly executed. When the analysis range is exceeded, the step S
The determination result of 17 is YES, and this waveform analysis processing ends. At this time, the analysis result is obtained in the table shown in FIG.

Next, the peak tracking processing in steps S12 and S20 will be described. It is assumed that the peak tracking process is performed on the p-th frame and the r-th frame that are adjacent to each other. First, for the peaks PKp-q (PKp-1, PKp-2, ..., PKp-Mp) detected in the p-th frame, all peaks PKr-s (PKr-1, PKK in the r-th frame).
r-2,…, PKr-Mr) is calculated. The value indicating this possibility is named CP (Connection Possibility). CP (pq, rs) is a numerical value indicating the possibility of connection between PKp-q and PKr-s as seen from the frame p side. This CP can be calculated by the following equation (1). CP (pq, rs) ＝ FuncA (| PKp-q.Frequency-PKr-s.Frequency |) * FunkB (| PKp-q.Mag nitude-PKr-s.Magnitude |) * FuncC (PredictionPhase (sign (rp) , PKp-q, PKr-s) -PK rs.Phase) (1). Here, FunkA is a function (frequency comparison function) that searches for similar frequencies. This frequency comparison function is shown in Fig. 7.
An example of FuncA is shown. In this figure, the horizontal axis x is the frequency of the peak, the vertical axis is the probability, F is the frequency of the peak to be compared, and FB is the fundamental frequency of the musical tone.
As shown in this example, it is only necessary to use a function that becomes 1 when the frequencies are exactly the same and the integrated value becomes 1 in all regions. FuncB is an amplitude comparison function that searches for similar amplitudes. FIG. 8 is a diagram showing an example of the amplitude comparison function, which is 1 when there is no difference in amplitude and the integrated value for all regions is 1 as in the frequency comparison function. In the case of a musical sound, it is assumed that the moving distance of the frame is short, so that there is no such a sharp change during this period.

Further, FuncC is a function of phase. In addition, as shown in the following equation (2), PredictionPhas
e is a phase prediction function, which is a function in the moving direction of the frame,
The phase at frame r predicted from the information at frame p is calculated. PredictionPhase (sign (rp), PKp-q, PKr-s) = PKp-q.Phase + sign (rp) 2π * PKp-q .Frequency / SamplingFrequency * HopSize (2). Here, HopSize is the number of moving sample points of the frame, which is 21 points in this example. HopSize / SamplingFrequ
ency indicates the moving sample time of the frame, which is 4.76 ms in this example. This formula (2) calculates how many times the phase of the r frame is, assuming that the frequency is PKp-q.Frequency in the p frame and it does not change until the next frame. Further, sign (rp) is the moving direction of the frame, and indicates that the phase advances when it is positive and returns when it is negative. FuncC is a function that becomes 1 when the distance between the phases is short, and for example, a comparison function in the form of the following expression (3) is used. FuncC (phase1-phase2) = (1 + cos (phase1-phase2)) / 2 (3).

For the qth peak in frame p,
CP (pq, r-1), CP (pq, r- for all peaks on the r side
2), ..., CP (pq, r-Mr) is calculated, and it can be said that the peak with the maximum CP value is the peak that may be most connected when the r side is viewed from the p side. FIG. 9 is a diagram showing an example of calculation for PKp-1. In the example shown in FIG. 9, CP (p-1, r-1) has the maximum value, so P
Kp-1 is thought to be most likely to connect with PKr-1. At this time, it is realistic to determine the minimum value of the CP in advance and, for example, if all CP values are 0.001 or less, there is no peak that may be connected. The peak in this case is considered to have disappeared at frame p.

In this way, it is assumed that the peaks in frame r, which are most likely to be connected, are found for all the peaks in frame p. However, in the case as shown in FIG. 10, two peaks on p may be connected to one peak on r. In the example shown in FIG. 10, PKp-3 and PKp-4 are connected to PKr-3. Therefore, in the present invention, this time from the r side to p
I try to look for the peak that is most likely to be connected. Also in this case, the CP shown in the formula (1) is used. However, in this case, the Predicti
The onPhase calculation formula is different. That is, since the phase prediction function PredictionPhase predicts the opposite direction in time, the sign (pr) becomes negative, and the expression (4) is obtained. PredictionPhase (sign (pr), PKr-s, PKp-q) = PKr-s.Phase-2π * PKr-s.Frequency / SamplingFrequency * Hopsize (4).

CP (rs, p-1), CP thus obtained
(rs, p-2), ..., CP (rs, p-Mp) is found to be the maximum value, and the peak most likely to be connected when looking from the r side to the p side is searched. FIG. 11 is a diagram showing an example of this result. In the example shown, PKr-3 is likely to connect with PKp-4. This allows PK
Judge that p-4 and PKr-3 are connected. Also, PKp-3
Can be considered to have disappeared at frame p. By thus performing bidirectional matching, it is possible to find the optimum peak connection.

In FIG. 10, when looking from the p side to the r side, CP (p-3, r-3) takes the maximum value for PKp-3, and CP (p-3 -4, r-3) takes the maximum value. That is, both PKp-3 and PKp-4 may lead to PKr-3. In this case, the values of CP (r-3, p-3) and CP (r-3, p-4) may be calculated again by looking at the p side from the r side. -3, r-3) and CP (p-4, r-
You may substitute in 3) and select the larger one.

In step S20, the 0th
When the CP value for the -1st frame is calculated from the frame side, since the PredictionPhase predicts the opposite direction in time, the phase prediction function shown in the equation (4) is used. In addition, when viewing the -1st frame to the 0th frame, the phase prediction function of Expression (2) is naturally used.

In this way, it is assumed that the connection of each peak in each frame is known from the beginning to the end of the waveform to be analyzed. FIG. 12 is a diagram showing an example of the result of the determination thus made. In this figure, the first reference was the peak of the 0th frame, as described above, and all the peaks of this frame were followed. Of course, it also includes peaks that disappeared due to extinction conditions on the way. Further, in the peak locus shown in FIG. 12, there is a possibility that unnecessary peaks may be included in the tracking target. For example,
The locus of the peak indicated by a is considered to be unnecessary. Therefore, the obtained results are cleaned as needed. The conditions for this cleaning are 1. Being harmonic 2. It has a certain length or more with respect to the total number of frames. One of these conditions may be used, or the AND conditions of 1 and 2 above may be used. FIG. 13 shows an example of the result of cleaning in this way.

Further, the locus of a certain peak may be partly cut off in the middle. FIG. 14 is a diagram showing an example of such a case. In this figure, the Mth peak locus disappears in N-1 frames and appears to be generated in N + 1 frames. However, one locus corresponds to one sine wave oscillator, and when the locus disappears, the corresponding sine wave oscillator is in the unused state or the output is cut off, that is, the sine wave output is generated between the N-1 frame and the N + 1 frame. There is a risk of inconvenience such as generation of click noise because it will be cut off completely. In such a case, it may be considered as a continuous trajectory by supplementing the N frame as shown in FIG. 15 so that the peak has an amplitude of 0. Alternatively, the peak information of frame N-1 and the peak information of frame N + 1 are interpolated,
It is also possible to generate peak information of the frame. In FIG. 12 described above, the locus indicated by a is an example of the interpolation thus performed. Such interpolation can be realized by creating a reference peak as follows. That is, since the PKN-M was not actually found, the amplitude is 0. However, the frequency and phase can be predicted from the N-1 peak data. Therefore, as a reference of a peak that has once been interrupted, the peak is stored as information having only frequency and phase information. This allows
For peaks that could not be connected to frame N, it is possible to create a reference as if there were peaks in frame N and connect N + 1.

As described above, in the conventional SFFT processing, when the window size is increased, the frequency accuracy of the fundamental frequency is improved, but it becomes impossible to follow the change of harmonics, and when the window size is reduced, the fundamental frequency is decreased. However, there was a problem that the frequency accuracy of was deteriorated. The tone analysis and synthesis processing of the present invention which solves such a problem will be described with reference to the flow chart of the processing of FIG. 16 and the waveform chart of FIG.

In FIG. 16, the input analyzed signal is multiplied by the analysis window (Window 1) set based on the first window function (Window Function 1) and the first hop size (Window Hopsize 1). (Step S32), the first short-time Fourier transform (SFFT) is performed (step S33). In this first short-time Fourier transform,
Similar to the above, the analysis by the window size and the window pop size determined from the fundamental frequency is performed.

Then, in step S34, the first peak detection and peak tracking processing is performed from the result of the first SFFT. In the first peak detection and peak tracking processing, the same peak detection and peak tracking as described above are performed, but here, the processing is performed for the peak of the frequency equal to or lower than the eighth harmonic of the fundamental frequency. The output of this step S34 is called the first trajectory information. Succeedingly, in a step S35, the sine wave synthesis of the first trajectory information is executed, and a first deterministic waveform (Deterministic Wave) is outputted. FIG. 17 shows an example of the analyzed signal (in this example, the waveform of the piano C1), and shows the first deterministic waveform. As described above, the sine wave synthesis may be performed by generating discrete sine wave waveform samples and adding them together, or by using a plurality of sine wave waveform generators. It is also possible to generate sinusoidal waveforms that are generated and combine their outputs.

Next, the first Determin obtained in this way
The polarity of the istic wave is inverted (step S36) and added with the signal to be analyzed (step S37). As a result, the first residual waveform is obtained. FIG. 17
Shows an example of this first residual waveform,
This first residual waveform is a first synthesized from the analyzed signal and a locus of peaks having frequencies below the eighth harmonic.
This is the signal of the difference from the Deterministic Wave. Next, for this first Residual Wave, the second window function (Window Fu
nction 2) and the second window hop size (Window H
opsize 2) and the second window function (Window
2) is multiplied (step S39), and the second short time Fourier transform process is executed (S40). This second SFFT
Uses 8 times the period of the 8th harmonic as the window size, that is, the same size as the fundamental period, and the hop size of the window is 1/8 of the 8th harmonic, that is, 1/64 of the fundamental frequency. . By using such a window size and window hop size, it is possible to detect peaks and follow a locus with high accuracy even for high frequency components. And this second SFF
From the output of T, peak detection and peak tracking processing is executed for frequency components of 8th harmonic or higher (S41), and second orbit information is obtained. Next, the sine wave synthesis is performed from the second trajectory information in the same manner as described above (S42), and the second Determinis
Get a tic Wave. This second Deterministic Wave has 8
It is a composite signal of harmonic components above harmonics.

The second Determinis thus obtained
tic Wave and the first Determ obtained in step S35
A final deterministic waveform (Deterministic Wave) is obtained by synthesizing with the inistic Wave (step S43). FIG. 17
Shows an example of this deterministic waveform. In the synthesis in step S43, the first deterministic waveform Deterministic Wave and the second deterministic waveform are added, but as described above, the first deterministic waveform is basically Based on the first trajectory information obtained at a window hop size interval of 1/8 of the period, the second deterministic waveform is obtained at a second hop size interval of 1/64 of the fundamental period. It is based on the orbit information of. Therefore, in order to add these two pieces of data, it is necessary to match the data with a fine interval. That is, it is necessary to interpolate the data of the first deterministic waveform by 8 times to match the precision of 1/64.

As described above, the data of each peak has three pieces of information of amplitude, frequency and phase. FIG. 18 is a diagram for explaining interpolation of amplitude information and frequency information. As shown in FIG.
For example, linear interpolation is performed between peaks P1-1 and P1-2 of the first deterministic waveform data obtained at intervals of / 8, and P1-1-1 and P1
By obtaining -1-2, ..., P1-1-7, it becomes amplitude information or frequency information with an accuracy of 1/64. The interpolation method is not limited to linear interpolation, and polynomial interpolation may be performed by taking data at several points before and after the matching. Regarding the phase information, the phase and frequency data of the peak P1-1 of the first deterministic waveform data obtained at intervals of 1/8 of the fundamental period,
Using the phase and frequency data of peak P1-2, FIG.
Describe the rotation of the phase as shown in, and each point P1-1-1, P1-1-
2,…, Read the phase of P1-1-7. In this way, the first Deterministic Wave and the second Dete
Data can be added after matching the time resolution with rministic Wave.

Further, by inverting the polarity of the second Deterministic Wave (S44) and adding it to the first Residual Wave (S45), the final Residual Wave as shown in FIG. 17 is obtained. You can also get it. As is apparent from FIG. 17, the residual waveform has a very low level except immediately after the rising. In the above, an example in which the frequency band of the analyzed signal is divided into two has been shown, but the invention is not limited to this.
It is possible to realize by the same method as in the above example by dividing into a number according to each case such as 3 divisions and 4 divisions.

Further, in the above, for the musical tone waveform to be analyzed, first, the SFFT is executed for the low-order overtone group, and the SFFT for the high-order overtone group is executed for the residual waveform with the sine wave composite waveform. But
It is not limited to this. For example, for the musical tone waveform to be analyzed, the SFFT process for the low-order overtone group and the SFFT process for the high-order overtone group are executed respectively, and a sine wave synthesis waveform for the low-order overtone group and a sine wave synthesis for the high-order overtone group are performed. The residual waveform may be obtained by subtracting the waveform and the musical tone waveform to be analyzed.

Further, in the above description, the waveform calculation at the time of analysis is executed by software processing. However, the waveform calculation may be executed by using the sine wave synthesizing function of the musical tone synthesizer 7. . in this case,
The analysis result may be given to the sine wave synthesis unit, and the synthesis result may be received from the output. Furthermore, the execution unit of each processing and the tone synthesis unit described above can be realized by either hardware or software. Furthermore, in the above description, the tone analysis method and the tone analysis / synthesis method of the present invention are executed in the tone analysis / synthesis apparatus having the configuration shown in FIG. The musical tone analysis / synthesis method is not limited to this, and can be executed in various devices such as a general-purpose computer such as a personal computer, an electronic musical instrument, a game machine, a karaoke device, and a measuring instrument such as a spectrum analyzer.

[0046]

As described above, according to the tone analysis method of the present invention, it is possible to accurately follow the peak in each frame. In addition, even if the waveform has a low fundamental frequency and abundant high-order overtones, the spectrum can be accurately analyzed.

[Brief description of drawings]

FIG. 1 is a block diagram showing an example of the configuration of a musical sound analysis / synthesis apparatus in which a musical sound analysis method or a musical sound analysis / synthesis method of the present invention is executed.

FIG. 2 is a block diagram showing a configuration example of a musical sound synthesizing unit.

FIG. 3 is a diagram showing an example of a time waveform of a C4 keystroke sound of a piano.

FIG. 4 is a flowchart for explaining a waveform analysis process.

5 is an example of an analysis result of the time waveform shown in FIG.

FIG. 6 is a diagram showing an example of a table storing analysis results.

FIG. 7 is a diagram showing an example of a frequency comparison function.

FIG. 8 is a diagram showing an example of an amplitude comparison function.

FIG. 9 is a diagram showing an example of a calculation result of a possibility that peaks between adjacent frames are connected.

FIG. 10 is a diagram showing an example of a calculation result of a possibility that peaks are connected.

FIG. 11 is a diagram showing an example of a calculation result of a possibility that peaks are connected.

FIG. 12 is a diagram showing an example of a result of determining a peak connection.

13 is a diagram showing an example of a result of cleaning the determination result of FIG.

FIG. 14 is a diagram for explaining a case where a locus of peaks is interrupted.

FIG. 15 is a diagram for explaining an interpolation process in the case of FIG.

FIG. 16 is a flowchart for explaining the musical sound analysis and synthesis processing in the present invention.

17 is a diagram showing an example of waveforms at each stage of the processing shown in FIG.

FIG. 18 is a diagram for explaining tone synthesis in the processing shown in FIG.

FIG. 19 is a diagram for explaining musical tone synthesis in the processing shown in FIG.

FIG. 20 is a diagram showing an example of analysis results of musical tone waveforms.

FIG. 21 is an enlarged view of part of FIG. 20.

FIG. 22 is a diagram showing an example of an analysis result of a musical tone waveform having a low fundamental frequency and abundantly containing high-order overtones.

FIG. 23 is a diagram showing an example of an analysis result of a musical tone waveform having a low fundamental frequency and abundant high-order overtones.

FIG. 24 is a diagram showing an example of a conventional musical tone waveform analysis and synthesis apparatus.

[Explanation of symbols]

1 CPU, 2 program memory, 3 data memory, 4 display device, 5 input device, 6 performance operator, 7
Tone synthesizer, 8 digital-to-analog converter, 9 network interface, 10 system bus, 11 communication network, 71, 73, 76 CPU interface circuit, 72 sine wave waveform calculator, 74 residual waveform calculator, 75 mixer, 77 waveform Memory, 78 Phase generator, 79 Waveform processor, 101 Low-pass filter, 102 Band-pass filter, 103 High-pass filter, 104 Multiplier, 105 Short-time FFT section, 1
06 peak detection and peak tracking processing unit, 107 sine wave synthesis unit

Claims (3)

[Claims] 1. A musical tone waveform analysis for performing spectrum analysis by performing a Fourier transform on a musical tone waveform sample to be analyzed using a time window that is sequentially shifted to detect a peak point in each frame corresponding to the position of the time window. In the method, the spectrum analysis is performed by using analysis conditions set corresponding to each of a plurality of overtone groups of the analysis target musical sound. 2. A residual component obtained by extracting a line spectrum component from a musical tone waveform, generating and synthesizing a sine wave signal waveform corresponding to the extracted line spectrum component, and subtracting the synthesized signal waveform from the musical tone waveform. And a synthesized signal waveform, which is a musical tone waveform analysis / synthesis method for synthesizing a musical tone, wherein when the musical tone waveform is analyzed, it is set correspondingly to each of a plurality of overtone groups of the musical tone waveform. A method for analyzing and synthesizing a musical tone waveform, characterized in that the musical tone waveform is analyzed under the analysis conditions described above, and the line spectrum component is extracted based on the analysis result. 3. The musical tone waveform analysis / synthesis method according to claim 2, wherein the residual components corresponding to the overtone groups are extracted, and the residual components are sequentially analyzed for different groups.