JP2001117580A - Audio signal processing device and audio signal processing method - Google Patents

Abstract

(57) [Summary] [Problem] It is possible to more accurately detect the position of an input singing voice or a musical instrument sound on a musical score. First, when an audio signal is input, the audio signal is divided into predetermined time units (frames), and characteristic parameters are obtained from the audio signals in frame units. Then, the obtained feature parameters are symbol-quantized by referring to the codebook, and the symbol observation probability is obtained by referring to the note dictionary. Then, a hidden Markov model is formed in accordance with the note sequence stored in advance, and the note corresponding to the frame of the input voice is specified using the Viterbi algorithm. As a result, it is possible to specify the position of the input voice in the note sequence.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an audio signal processing apparatus and an audio signal processing method for associating a previously stored note sequence with an input audio in a time series.

[0002]

2. Description of the Related Art Conventionally, there has been proposed a method for following a current position of an instrumental music score by detecting a pitch or a sounding time of an instrumental sound (for example, "An On-Line"). Algorithm for Real-time Accompanm
ient "(R. Dannenberg. Proceedings of the ICMC 198
4) and "The Synthetic performer in the Context of
Live Musical Performance "(B. Vercoe. Proceedings
of the ICMC 1984.)).

[0003]

However, actual performances and singing voices do not always progress completely according to the musical score, and uncertain factors such as subtle tempo, timing pitch shift, and fluctuations have an adverse effect. However, with the above-described conventional method, it may not be possible to accurately detect where the musical instrument sound or the singing voice is located on the musical score.

[0004] The present invention has been made in view of the above circumstances, and provides an audio signal processing capable of more accurately detecting the position of an input singing voice or instrument sound on a musical score. It is an object to provide an apparatus and an audio signal processing method.

[0005]

According to a first aspect of the present invention, there is provided an audio signal processing apparatus for associating an input voice with one of notes in a note sequence. Note string storage means for storing note string information described in a time sequence, parameter acquisition means for acquiring feature parameters from a speech signal input in frame units, and representative feature parameters of the speech signal as feature vectors. By referring to the codebook clustered into symbols, the number of states for each note, the state transition probability, and the note information storage means for recognition storing the observation probability of the symbol, by referring to the note information storage means for recognition, Observation symbols of the input voice are acquired from the characteristic parameters acquired by the parameter acquisition means, and the observation probability of acquiring the observation probability of the observation symbols is acquired. An acquisition unit, and based on the number of states and a state transition probability stored in the recognition note information storage unit, store each state of the note sequence stored in the note sequence information storage unit on a finite state network using a hidden Markov model. State forming means, the state transition determining means for determining a state transition according to the observation probability obtained by the observation probability obtaining means, and the hidden Markov model formed by the state forming means, and the state transition The image processing apparatus further includes a correspondence unit that associates each frame of the input audio signal with the note string information based on the state transition determined by the determination unit.

According to a second aspect of the present invention, in the audio signal processing apparatus according to the first aspect, based on a result of the association by the associating means, a current input speech is converted to the note string information. The display device further comprises display means for displaying which part of the image is.

According to a third aspect of the present invention, in the audio signal processing apparatus according to the first or second aspect, the parameter acquiring means determines at least energy, delta energy, and zero crossing from an input audio signal. , Pitch, delta pitch, and pitch error as characteristic parameters.

According to a fourth aspect of the present invention, in the audio signal processing apparatus according to the third aspect, the energy, delta energy, zero cross, delta pitch, and energy stored in the recognition note information storage means are stored. The five observation probabilities of the pitch error are calculated using an observation function using a Gaussian distribution, and the observation probabilities of the pitches stored in the recognition note information storage means are the same as the observation function using a Gaussian distribution. It is calculated using a step observation probability function, and when calculating the observation probability of this pitch, the observation function using the Gaussian distribution and the step observation function are selectively used depending on the presence or absence of the pitch. It is characterized by:

According to a fifth aspect of the present invention, in the audio signal processing apparatus according to any one of the third and fourth aspects, the state forming means includes a sound having a pitch,
3 types of left-to-right depending on the pitchless sound and silence
Forming a hidden Markov model of the pattern, forming the pitched sound and the pitchless sound as a three-state model,
It is characterized in that the pitchless sound is formed as a one-state model.

According to a sixth aspect of the present invention, in the audio signal processing apparatus according to the fifth aspect, when the state forming means forms the hidden Markov model of the sound having the pitch, It is characterized in that a note and a note connected by a slur and a single note are formed as different models.

According to a seventh aspect of the present invention, there is provided the audio signal processing apparatus according to any one of the first to sixth aspects, wherein the learning musical tone waveform data and the learning musical note obtained by converting the musical tone waveform into a musical note. Input means for inputting column data;
A learning model forming means for forming a hidden Markov model on a finite network for each note of the learning note sequence data input from the input means, and a likelihood of the model formed by the learning model forming means during learning. Parameter estimating means for estimating a parameter having a maximum value by a k-means algorithm, wherein the recognition note information storing means includes a state transition of a feature vector in each note obtained by the parameter estimated by the parameter estimating means. It is characterized by storing probabilities and observation probabilities.

According to an eighth aspect of the present invention, in the audio signal processing apparatus according to any one of the first to seventh aspects, the state transition determining means determines a state transition by a Viterbi algorithm. It is characterized by:

According to a ninth aspect of the present invention, in the audio signal processing apparatus according to the eighth aspect, the note string storing means stores duration data corresponding to the note string. The state transition determining means includes the duration data stored in the note string storage means in the Viterbi algorithm.

According to a tenth aspect of the present invention, there is provided an audio signal processing method for associating an input voice with any one of notes stored in advance in a note sequence. A parameter acquisition step of acquiring characteristic parameters, a codebook in which symbols representative of pre-stored representative characteristic parameters of the audio signal are clustered into symbols, a state number, a state transition probability, and observation of the symbol for each note An observation probability acquisition step of acquiring an observation symbol of the input voice from the feature parameter acquired in the parameter acquisition step by referring to the probability, and an observation probability acquisition step of acquiring an observation probability of the observation symbol; and a state number stored in advance. Based on the state transition probability and each state of the note sequence stored in advance, State formation step formed by the Markov model, the observation probability acquired by the observation probability acquisition step, and a state transition determination step of determining a state transition according to the hidden Markov model formed by the state formation step, Based on the state transition determined by the state transition determining step,
A step of associating each frame of the input audio signal with the note sequence information.

The audio signal processing method according to claim 11 is the audio signal processing method according to claim 10,
Based on the result of the association by the association step,
It is characterized by further comprising a display step of displaying which part of the note string information the current input voice is.

According to a twelfth aspect of the present invention, in the audio signal processing method according to the tenth or eleventh aspect, in the parameter acquiring step, at least energy, delta energy, and zero-crossing from an input audio signal are obtained. , Pitch, delta pitch, and pitch error as characteristic parameters.

According to a thirteenth aspect of the present invention, in the audio signal processing method according to the twelfth aspect,
First, the five observation probabilities of the energy, delta energy, zero cross, delta pitch, and pitch error are calculated and stored using an observation function using a Gaussian distribution.
Observation probability calculation step, the observation probability of the pitch,
A second observation probability calculated and stored by using an observation function using the Gaussian distribution and the step observation function in accordance with the presence or absence of the pitch, using an observation function using a Gaussian distribution and a step observation probability function. And a calculation step, wherein the observation probability is obtained by referring to the observation probabilities stored in the first and second observation probability calculation steps.

According to a fourteenth aspect of the present invention, in the audio signal processing method according to any one of the twelfth and thirteenth aspects, in the state forming step, a sound having a pitch, a sound having no pitch, 3 types of le according to silence
forming a hidden Markov model of the ft-to-right type, forming the pitched sound and the pitchless sound as a three-state model, and forming the pitchless sound as a one-state model. I have.

According to a fifteenth aspect of the present invention, in the audio signal processing method according to the fourteenth aspect,
In the state forming step, when forming the hidden Markov model of the sound having the pitch, a note connected to the preceding note by a slur and a single note are formed as different models.

A sound signal processing method according to a sixteenth aspect of the present invention is the audio signal processing method according to any one of the tenth to fifteenth aspects, wherein the learning musical tone waveform data and the learning musical note obtained by converting the musical tone waveform into a musical note. An input step of inputting column data; a learning model forming step of forming a hidden Markov model on a finite network for each note of the learning note string data input in the input step; A parameter estimating step of estimating a parameter with the maximum likelihood of the model formed by the forming means by a k-means algorithm; and a state transition probability of a feature vector in each note obtained by the parameter estimated by the parameter estimating step; A probability storage step of storing an observation probability. In the step, the observation probability is obtained by referring to the observation probability stored in the probability storage step, and in the state forming step, a note stored in advance based on the state transition probability stored in the probability storage step is used. It is characterized in that each state of the sequence is formed by a hidden Markov model on a finite state network.

According to a seventeenth aspect of the present invention, in the audio signal processing method according to any one of the tenth to sixteenth aspects, the state transition determining step determines a state transition by a Viterbi algorithm. It is characterized by:

The audio signal processing method according to the eighteenth aspect is the audio signal processing method according to the seventeenth aspect,
In the state transition determining step, duration data corresponding to a note sequence stored in advance is included in the Viterbi algorithm.

[0023]

Embodiments of the present invention will be described below with reference to the drawings. A. Configuration of Embodiment A-1. 1. Overall Configuration First, FIG. 1 is a diagram showing a configuration of an audio signal processing device according to an embodiment of the present invention. In FIG. 1, reference numeral 1 denotes a microphone, which collects sounds such as a singer's singing voice and a musical instrument sound, and outputs the collected sounds to an input audio signal cutout unit 3 as an input audio signal Sv. Reference numeral 2 denotes an analysis window generation unit. The analysis window generation unit 2 generates an analysis window (for example, a hamming window) AW having a period that is a fixed multiple of the period of the pitch detected in the previous frame, and cuts out an input audio signal. Output to section 3. When the initial state or the previous frame is silent, the analysis window AW is used as an analysis window of a fixed period set in advance and the input audio signal extracting unit 3.
Output to

The input audio signal extracting section 3 multiplies the input analysis window AW by the input audio signal Sv, cuts out the input audio signal Sv in frame units, and outputs the frame audio signal Fv.
It is output to the fast Fourier transform unit 4 as Sv. The fast Fourier transform unit 4 obtains a frequency spectrum from the frame audio signal FSv, and outputs the frequency spectrum to the feature parameter analysis unit 5.

The characteristic parameter analysis unit 5 extracts a characteristic parameter characterizing the spectral characteristics of the input speech, and outputs the characteristic parameter to the symbol quantization unit 7. In the present embodiment, six types of feature vectors (energy, delta energy, zero-cross rate, pitch frequency, delta pitch, and total error) described later are used as the feature parameters. As will be described later in detail, the recognition note information storage unit 6 stores a codebook 6a and a note dictionary 6b storing probability data indicating the number of states of the feature vector and the state transition probability and the symbol observation probability in each note. Have.

The symbol quantization unit 7 selects a characteristic symbol having the maximum likelihood in the frame with reference to the codebook 6a stored in the recognition note information storage unit 6, and determines the observation probability of the selected characteristic symbol. Note row state forming unit 8
Output to

The note string state forming unit 8 refers to the recognition note information storage unit 6 and a note string information storage unit 11 described later, and describes the note string information storage unit 11 using a hidden Markov model (HMM). To form a note sequence state. The state transition determination unit 9 determines a state transition according to a Viterbi algorithm described later, using the observation probability of a feature symbol in frame units obtained from the input speech output from the symbol quantization unit 7. Thereby, the note of the input voice can be specified at each time in frame units.

The matching unit 10 determines the time of each frame of the input speech and the position of the note sequence stored in the note sequence information storage unit 11 based on the specified note of the input speech frame. Identify and associate the input voice with the note sequence. The display device 12 includes the matching unit 10
The result of associating the input voice with the note sequence is displayed.

A-2. Recognition note information storage unit Next, the codebook 6a and the note dictionary 6b stored in the recognition note information storage unit 6 will be described. Codebook 6a
Are clustered into a predetermined number of symbols using a representative feature parameter of the audio signal as a feature vector.

A-2-1. Feature Vectors First, before describing the codebook 6a, six types of feature vectors used in the present embodiment will be described.

Energy Energy is a coefficient representing the intensity of sound, and is parameterized into a feature vector by the following equation.

(Equation 1) Delta energy Delta energy is a coefficient that represents the sound intensity as a difference, and is parameterized into a feature vector by the following equation.

(Equation 2) Zero-cross rate The zero-cross rate has a feature that the zero-cross rate decreases as the voiced sound becomes, and is parameterized into a feature vector by the following equation.

(Equation 3) Pitch frequency The pitch frequency is defined as "Fundamental Frequency Estimation
in the SMS Analysis "(P. Cano. DAFX Proceedings 199
It can be determined by the Two-Way Mismatch method described in 8.).

Delta Pitch Delta pitch is parameterized into a feature vector by the following equation:

(Equation 4) Total Error The total error indicates a voiced soundness by calculating a mismatch between the error of the predicted pitch with the measured pitch and the error of the predicted pitch of the measured pitch from two directions. First, the measured pitch (m) of the predicted pitch (p)
Is expressed by the following equation.

(Equation 5) In the above equation, f _k is the k-th predicted peak frequency, Δ
f _k is the frequency difference between the k-th predicted peak frequency and the measured pitch, a _k is the k-th predicted amplitude, and A _max is the maximum value of the amplitude. On the other hand, measurement pitch (m)
The pitch error from to the predicted pitch (p) is represented by the following equation.

(Equation 6) In the above equation, f _k is the k-th predicted peak frequency, Δ
f _k is the frequency difference between the k-th predicted peak frequency and the measured pitch, a _k is the k-th predicted amplitude, and A _max is the maximum value of the amplitude.

Therefore, the total error is as follows.

(Equation 7) A-2-2. Note dictionary The note dictionary 6b stores a left-to-right type hidden Markov model for each note. Here, note dictionary 6
For b, three types of models are prepared according to the physical characteristics of the input voice. Specifically, models are prepared for each of three types of sounds, pitched sound, pitchless sound, and silence.
As pitched sounds, each pitch (C0, C # 0,
D0, D # 0, E0, F0, F # 0, G0, G # 0 ...
...), The model, that is, the number of states and the state transition probabilities are stored (see FIGS. 2 and 3). In addition, as described later, an observation probability value for each feature vector symbol calculated in advance according to a Gaussian distribution or the like is stored. Here, in the case of a sound with a pitch, the number of states is three, and three states, namely, a rising state (attack), a steady state (steady state), and a release state of the sound are respectively simulated.

In the present embodiment, plosive sounds (/ p /, / b /...) And fricative sounds (/ s /, / sh /...) Are used as pitchless sounds.
One model is prepared for each. In the present embodiment, the number of states is set to three states of attack, steady, and release even in the model of the pitchless sound, so that fine nuances such as an expiration sound, a popping sound, and a fricative sound can be reproduced. . In the case of silence (SILENCE),
The number of states is set to one. In the case of a pitched sound, for a note connected to a whole note by a slur, if a model different from a single note is prepared, it is possible to more accurately associate an input voice with a note. A-2-3. Codebook The codebook 6a is generated based on five types of energy, Δ energy, zero cross rate, delta pitch, and total error among the above six types of feature vectors, and is divided into clusters. Each cluster contains a typical set of vectors representing each symbol. (FIG. 4
reference). In the present embodiment, a known LBG algorithm is used when creating the codebook 6a. The codebook 6a is created according to a continuous density function of a Gaussian distribution represented by the following equation.

(Equation 8) Here, C _jm indicates the mixing weight coefficient of the component m in the state j. N (j; μ, Σ _j ) indicates a multidimensional Gaussian distribution of the mean vector μ and the covariance matrix Σ,
Since the number of learning parameters becomes enormous due to being multidimensional in this state, N (j; μ,
Σ _j ) is a constant function.

In order to create the codebook 6a according to such a Gaussian distribution, the observation sequence y _t to _l of the quantization vector N _l in the frame becomes a subset M _l . Where M
_l is the number of mixed elements in the codebook to be created. When the codebook 6a is created, parameters are estimated by the following calculation.

(Equation 9)

(Equation 10) The mixing weight coefficient C _jm is obtained by _calculating the quantization vector N _lm
It is used at the time of the second mixing, and is represented by the following equation.

[Equation 11] Next, in the present embodiment, for the pitch frequency, which is a feature vector other than the above five types, the function used to calculate the observation probability has two types: a pitched sound, a pitchless sound, and a silent sound. Divided. Here, FIG. 5A shows a step function for calculating an observation probability b (y) in the case of a pitched sound, and FIG. 5B shows an observation probability b (y) in the case of a pitchless sound or a silent sound. ) Is shown below. As shown in the figure, in the case of a pitched sound, when the pitch F ₀ = 0, that is, when no pitch is detected, the calculated observation probability is a constant. On the other hand, when the pitch is detected, that is, when F ₀ = 0, a function for calculating the observation probability b ′ (y) (FIG. 5)
(A) Refer to the graph shown in the upper right of FIG. 2) as a continuous density function according to a Gaussian distribution. Then, the pitch F ₀ Dev used in this calculation is calculated by the following equation.

(Equation 12) In the above equation, F _Perfect0 indicates the pitch frequency that should be played. For example, when the average rate A4 = 440 Hz, F _Perfect0 used for calculating the observation probability at each interval is the following numerical value. C3... 261.626 Hz D3... 293.66 Hz E3... 329.628 Hz As described above, 6 according to the continuous density function of the Gaussian distribution, etc.
The observation probabilities for the types of feature vectors are calculated and stored in the note dictionary 6b, and the codebook 6a is also created in accordance with the Gaussian distribution.

A-3. Note string information storage unit Next, the note string information storage unit 11 will be described. As shown in FIG. 6, the note string information storage unit 11 describes note strings such as music pieces in chronological order. In the present embodiment,
The duration information is stored for each note in the note sequence. Therefore, when storing a note sequence of a musical piece represented by a musical score as shown in FIG. 6A, the note sequence information and the duration information as shown in FIG. 6B are stored. Here, the duration information is expressed as follows.

In the case of a pitched sound or a pitchless sound, it is expressed as follows: S1: whole note S2: half note S3: quarter note S4: eighth note.

On the other hand, in the case of silence (rest), it is expressed as follows: U1: full rest U2: two-minute rest U3: four-minute rest U4: eight-minute rest

Accordingly, the actual time of the above-mentioned duration is determined by the speed notation of the musical score and the set tempo.

B. Next, an operation of the audio signal processing device having the above configuration will be described. B-1. Outline operation First, the outline operation of this audio signal processing device is shown in FIG.
This will be described with reference to the flowchart shown in FIG. First,
When an input audio signal is generated by the microphone 1, the audio signal is subjected to fast Fourier transform on a frame basis to obtain a frequency spectrum. Then, feature parameter analysis is performed from the obtained frequency spectrum to obtain the above-described six types of feature parameters (step S1).

Next, the symbol quantization of the six types of characteristic parameters obtained by the symbol quantization unit 7 is performed by referring to the recognition note information storage unit 6 (step S2). Then, the symbol quantization unit 7 outputs the note dictionary 6
By referring to b, the observation probability of the symbol quantized symbol is obtained.

Thereafter, by referring to the note string information stored in the note string information storage section 11 and the note dictionary 6b, the note string state forming section 8 changes the state of the note string to HMM.
It is composed of a model (step S3). Then, based on the symbol observation probability acquired by the symbol quantization unit 7 and the HMM model formed by the note sequence state forming unit 8 as described above, the state transition determining unit 9 performs state transition using the Viterbi algorithm. Is determined (step S4). The HMM model and the Viterbi algorithm will be described later. Then, based on the state transition determined by the state transition determining unit 9, the matching unit 10 performs temporal association between the input voice and the note sequence (step S5), and the association result is displayed on the display device 12. It is displayed (step S6).

B-2. Details of Operation Next, each process described in the outline operation will be described in detail.

B-2-1. Characteristic Parameter Analysis and Symbol Quantization FIG. 8 is a diagram for describing a process of acquiring characteristic parameters from an input audio signal generated by the microphone 1 and performing symbol quantization. As shown in the figure, the input audio signal is converted into a frequency spectrum by fast Fourier transform on a frame basis. A characteristic parameter analysis is performed on this frequency spectrum. Then, for each feature vector, the symbol with the maximum likelihood is found from the codebook 6a of the note information storage unit 6 for recognition, and the observation probability of the found symbol is acquired by referring to the note dictionary 6b.

B-2-2. Hidden Markov Model Next, referring to FIG. 9, a hidden Markov model (HM
M) will be described. Since the state of the voice transits in one direction, in the present embodiment, as described above, the left-to-
The right type model is used.

At time t, the probability of transition from state i to j (state transition probability) is represented as a _ij . In the example shown in FIG. 9 represents the probability of staying in the state a _11, it represents the probability of transition from state to state and a _12. Such a state transition probability is stored for each note in the note dictionary 6b as described above (see FIG. 3). In the present embodiment, for the pitched sound and the pitchless sound, the state indicates attack, the state indicates steady, and the state indicates release.

Each state has a feature vector, and each state has a different observation symbol. This is Y =
{Y ₁ , y ₂ ,..., Y _t }. The probability (symbol observation probability) of generating the symbol y _{t of the} feature vector when the state is j at the time _t is represented as b _j (y _t ). Here, when the model shown in the drawing is M, and the probability that the observed symbol sequence y transitions to the state is represented by Y, the probability that X and Y occur simultaneously can be expressed by the following equation.

(Equation 13) In the present embodiment, based on the note string information stored in the note string information storage unit 11, an FSN (finite state network) as shown in FIG. 9 is formed for each note. For example, FIG.
If the information as shown in FIG. 3B is stored, the hidden Markov is based on the number of states and the state transition probabilities of the intervals “E3”, “G3”, “silence”... Stored in the note dictionary 6b. Form a model.

B-2-3. In this embodiment, the hidden Markov model formed as described above based on the note sequence information stored in the note sequence information storage unit 11, the characteristic symbol acquired by the symbol quantization unit 7, and the symbol From the observation probability of, the state transition of the input voice is determined according to the Viterbi algorithm.
The outline is briefly described. The Viterbi algorithm is
This is to derive the most likely observation state sequence when the model M outputs the observation symbol sequence y. Φ
Observation vector y ₁ when _j (t) is in the state at time t
Assuming that the probability is the highest when transitioning to y _t , the partial likelihood at this time is expressed by the following induction formula.

[Equation 14] In the above equation, a _ij is the state transition rate from state i to j (that is, determined by the note sequence stored in the note sequence information storage unit 11), and b _j (y _t ) is the time of each feature vector. The symbol observation probability at t, which is determined based on the feature vector of the input voice and the above-described recognition note information storage unit 6 and the like.

Φ ₁ (1) = 1, Φ _j (1) = a _1j b
Assuming that _j (y ₁ ), for 1 <j <N, the maximum likelihood probability P ′ (Y | M) is represented by the following equation.

(Equation 15) By recursively obtaining the maximum probability for the state j in this manner, an optimal path, that is, an observation state sequence having the highest probability is derived, and a state transition is determined.

Further, in the present embodiment, the note string information storage unit 11 stores the duration information corresponding to each note in the note string, and the duration information is stored in the Viterbi algorithm for calculating the optimum path. And the following algorithm including

In this algorithm, the duration of each note n and time t is kept, and a penalty function P for transitioning from the state at time t to the state j at time t + 1 is introduced.

(Equation 16) For details on the introduction of such a penalty function P, see "Robus
t Parametric Modeling of Durations in HMMs "(D. Bu
rshtein. ICASSP Proceedings 1995).

In the penalty function P, ΔD (t) represents the difference between the duration of the target sound to be played and the duration of the actual sound. That is, ΔD (t) =
D (t) -Dn. Here, D (t) is the actual duration of the sound, and Dn is the duration of the sound to be originally played, and is represented by a symbol indicating the duration in the note sequence (S1, S2, etc. 6 (b)). As described above, the actual time length of the duration is determined by the speed notation of the musical score and the set tempo. For example, in the music score shown in FIG.
If the duration information is "S3", that is, a quarter note,
Dn = 60/120 = 500 msec. In the penalty function, l (u) = logp (u) is shown. Here, p (u) is the probability density of ΔD, and is modeled by a Gaussian mixture density.

When the penalty function P is included in the Viterbi algorithm and the logarithm of the model parameter is taken, the inductive expression is expressed as follows.

[Equation 17] With this formula, it is possible to determine the optimal path in consideration of the duration of the phoneme.

In the example shown in FIG. 10, the probability calculated by the above equation is indicated by ○ or （(○> △). For example, based on observation from time tm1 to time tm3 (time tm1 and the like indicate time in frame units), the state “Sile
The probability that a path from the “nce” to the state “C ₁ ” is formed is higher than the probability that a path from the state “Silence” to the state “Silence” is formed, and is the best probability at the time tm3. The state transition is determined as shown. Such an operation is performed at a time (t) corresponding to each frame of the input voice.
m1, tm2, tm3,...)
In the example shown in FIG. 10, the transition is determined as a transition as shown by the bold line in the figure. Thereby, the note of the input voice can be specified at each time of each frame unit. FIG.
In the case shown in, the times tm1 and tm2 are "Silence",
tm3 to tm10 are "C1", and tm11 to tm10 are "D
0 ".

In this way, it is possible to correlate the notes of the input voice specified at the time of the frame unit with each note constituting the note string stored in the note string information storage unit 11. As a result, the result of associating the input voice with each note in the note sequence can be displayed on the display device 12. Here, as a display method of the display device 12, as shown in FIG. 10, a musical score may be displayed, and an arrow or the like may indicate a position on the musical score where the input voice at the present time is. Alternatively, the note corresponding to the current input voice may be displayed in a different color from the other notes, and the position of the current input voice in the note sequence may be displayed, or may be arbitrary.

C. Modified Example In the above-described embodiment, the recognition note information storage unit 6 is used.
In the note dictionary 6b, the transition probability, the symbol observation probability, and the like calculated in advance have been described. However, learning data (a set of musical tone and singing voice waveform data and data indicating a note sequence corresponding thereto) is used as needed. May be input to estimate and rewrite these parameters. in this case,
Using data representing the note sequence of the learning data, a hidden Markov model extended to FSN is generated for each note. Then, in order to maximize the likelihood of the input learning data, it is obtained by estimating the parameters of each note model. Here, a method of estimating parameters using a known K-means method will be briefly described.

Initialization First, the waveform data of the learning data is divided for each note in a note string notation.

Estimation Next, the transition time is calculated by counting the time required for the transition and dividing the time by the transition time of the state. That is, the transition probability is calculated by the following equation.

(Equation 18) In this process, it is necessary to manage the counter in order to track each transition state and output symbol during the learning time.

The mixing weight coefficient in the continuous density function of the Gaussian distribution used for the observation probability of the five types of feature vectors other than the pitch frequency is estimated for each state i by the following equation.

[Equation 19] The following probability function is used for the remaining feature vector, that is, the pitch frequency.

(Equation 20) Also, as described above, when there is a pitch sound for the pitch frequency, a continuous density function of a Gaussian distribution is used as in the other five types of feature vectors. In this case,
It is estimated in the same way as the above five types of feature vectors.

Segmentation In the estimation process described above, segmentation is performed again using the estimated parameters.

Repetition The above is repeated until convergence.

By performing the learning as described above, more accurate parameters can be estimated and described in the note dictionary 6b. That is, it is possible to improve the accuracy of the state transition determination performed with reference to the note dictionary 6b, and it is possible to more accurately detect the position of the note sequence in which the input voice is located.

[0063]

As described above, according to the present invention,
It is possible to more accurately detect the position of the input singing voice or instrument sound on the music score.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an audio signal processing device according to an embodiment of the present invention.

FIG. 2 is a diagram for explaining a note dictionary which is a component of the audio signal processing device.

FIG. 3 is a diagram for explaining a note dictionary which is a component of the audio signal processing device.

FIG. 4 is a diagram for explaining a codebook which is a component of the audio signal processing device.

FIG. 5 is a diagram for explaining a function for calculating an observation probability of a symbol described in the note dictionary.

FIG. 6 is a diagram for explaining a note sequence information storage unit that is a component of the audio signal processing device.

FIG. 7 is a flowchart for explaining the operation of the audio signal processing device.

FIG. 8 is a diagram illustrating a process of acquiring a feature vector from an input voice.

FIG. 9 is a diagram for explaining a hidden Markov model used in the audio signal processing device.

FIG. 10 is a diagram for explaining correspondence between an input voice and a note sequence.

FIG. 11 is a diagram illustrating a display example of a result of associating an input voice with a note sequence.

[Explanation of symbols]

1 microphone, 2 analysis window generation unit, 4 fast Fourier transform unit, 5 feature parameter analysis unit, 6 note information storage unit for recognition, 6a codebook, 6b note dictionary ,
8... Note sequence state forming unit 9... State transition determining unit 11
... Note string information storage unit, 12... Display device

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] December 15, 1999 (1999.12.
15)

[Procedure amendment 1]

[Document name to be amended] Drawing

[Correction target item name] Fig. 8

[Correction method] Change

[Correction contents]

FIG. 8

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G10L 15/00 G10L 3/00 551G (72) Inventor Pedro Keino Spain Barcelona 08002 Merce 12 (72) Inventor Alex Roscos Spain Barcelona 08002 Merce 12 F term (reference) 5D015 AA06 CC03 CC06 CC13 CC14 CC18 HH23 KK02 5D082 BB01 BB14 BB15 5D378 KK02 KK03 KK05 MM02 MM14

Claims (18)

[Claims] 1. An audio signal processing apparatus for associating an input voice with any one of notes in a note sequence, a note sequence storing means for storing note sequence information described in a time sequence, and a sound input in frame units. Parameter acquisition means for acquiring a characteristic parameter from a signal, a codebook in which a representative characteristic parameter of a voice signal is clustered into a symbol as a characteristic vector, a number of states for each note, a state transition probability, and an observation probability of the symbol. By referring to the note information storing means for recognition storing the observation symbol of the input voice from the characteristic parameter obtained by the parameter obtaining means, by referring to the note information storing means for recognition. Observation probability acquisition means for acquiring a probability, based on the number of states and the state transition probability stored in the note information storage means for recognition. There are a state forming means for each state of the sequence of notes stored in the sequence of notes data storing means formed by Hidden Markov Model on finite state networks, and observation probability acquired by the observation probability acquiring means,
A state transition determining unit that determines a state transition according to the hidden Markov model formed by the state forming unit; and, based on the state transition determined by the state transition determining unit, each frame of the input voice signal and the An audio signal processing apparatus comprising: a correspondence unit that associates the note string information with the note string information. 2. The apparatus according to claim 1, further comprising a display unit for displaying which part of the note string information the current input voice is based on a result of the association by the association unit. An audio signal processing device as described in the above. 3. The parameter acquisition unit according to claim 1, wherein the parameter acquisition unit acquires at least energy, delta energy, zero cross, pitch, delta pitch, and pitch error from the input audio signal as characteristic parameters. Audio signal processing device. 4. The five observation probabilities of energy, delta energy, zero cross, delta pitch, and pitch error stored in the recognition note information storage means are calculated using an observation function using a Gaussian distribution. The observation probability of the pitch stored in the recognition note information storage means for recognition is calculated using an observation function using a Gaussian distribution and a step observation probability function.When calculating the observation probability of this pitch, The audio signal processing device according to claim 3, wherein an observation function using the Gaussian distribution and the step observation function are selectively used depending on the presence or absence of the pitch. 5. The state forming means includes a sound having a pitch,
3 types of left-to-right depending on the pitchless sound and silence
A hidden Markov model of a pattern is formed, the sound with pitch and the sound without pitch are formed as a three-state model, and the sound without pitch is formed as a one-state model. 5. The audio signal processing device according to any one of 4. 6. The state forming means, when forming the hidden Markov model of the pitched sound, forms a note connected with a preceding note by a slur and a single note as different models. The audio signal processing device according to claim 5. 7. An input means for inputting learning musical sound waveform data and learning musical note sequence data obtained by converting the musical sound waveform into a musical note, and a finite network for each musical note of the learning musical note sequence data input from the input means. And a parameter estimating means for estimating, by a k-means algorithm, a parameter that maximizes the likelihood of the model formed by the learning model forming means during learning. 7. The method according to claim 1, wherein the recognition note information storage means stores a state transition probability and an observation probability of a feature vector in each note obtained by a parameter estimated by the parameter estimation means. An audio signal processing device according to any one of the above. 8. The audio signal processing device according to claim 1, wherein said state transition determining means determines a state transition by a Viterbi algorithm. 9. The note string storage means stores duration data corresponding to the note string, and the state transition determining means stores the duration data stored in the note string storage means into the Viterbi data. The audio signal processing device according to claim 8, wherein the audio signal processing device is included in an algorithm. 10. A voice signal processing method for associating an input voice with any one of a pre-stored note sequence, comprising: a parameter obtaining step of obtaining feature parameters from a voice signal input in frame units; A codebook obtained by clustering representative characteristic parameters of the obtained speech signal into symbols as a feature vector,
By referring to the number of states, the state transition probability, and the observation probability of the symbol for each note, the observation symbol of the input voice is acquired from the characteristic parameter acquired in the parameter acquisition step, and the observation probability of the observation symbol is acquired. Observation probability acquisition step of acquiring, based on the number of states and state transition probabilities stored in advance,
A state forming step of forming each state of a note string stored in advance on a finite state network by a hidden Markov model; an observation probability obtained by the observation probability obtaining step; and the hidden Markov formed by the state forming step. A state transition determining step of determining a state transition in accordance with the model, and, based on the state transition determined by the state transition determining step, an associating step of associating each frame of the input voice signal with the note string information. An audio signal processing method, comprising: 11. The apparatus according to claim 10, further comprising a display step of displaying which part of the note string information the current input voice is based on a result of the association by the association step. The audio signal processing method according to the above. 12. The parameter acquiring step according to claim 10, wherein at least energy, delta energy, zero cross, pitch, delta pitch and pitch error are acquired from the input audio signal as characteristic parameters. Audio signal processing method. 13. The energy, delta energy,
A first observation probability calculation step of calculating and storing five types of observation probabilities of zero cross, delta pitch and pitch error using an observation function using a Gaussian distribution, and using the Gaussian distribution for the observation probability of the pitch. A second observation probability calculation step of separately calculating and storing the observation function using the Gaussian distribution and the step observation function according to the presence or absence of the pitch using the observation function and the step observation probability function The voice according to claim 12, further comprising: in the observation probability acquiring step, the observation probability is acquired by referring to the observation probability stored in the first and second observation probability calculation steps. Signal processing method. 14. In the state forming step, three types of left-t are selected according to a pitched sound, a pitchless sound, and a silence.
An o-right type hidden Markov model is formed, the pitched sound and the pitchless sound are formed as a three-state model, and the pitchless sound is formed as a one-state model. Item 12 or 1
3. The audio signal processing method according to any one of 3. 15. In the state forming step, when forming the hidden Markov model of the pitched sound, a note connected with a previous note, a note connected by a slur, and a single note are formed as different models. The audio signal processing method according to claim 14. 16. An inputting step of inputting learning musical tone waveform data and learning musical note sequence data obtained by converting the musical musical tone into notes, and a finite network for each musical note of the learning musical note sequence data input in the inputting step. A learning model forming step of forming a hidden Markov model by: a parameter estimating step of estimating, by a k-means algorithm, a parameter that maximizes a likelihood of a model formed by the learning model forming means during learning; A probability storage step of storing a state transition probability and an observation probability of a feature vector in each note obtained by the parameter estimated by the estimation step, and the observation probability acquisition step stores the observation probability stored by the probability storage step. Obtain the observation probability by referring to In step, on the basis of the probability memory stored state transition probabilities by step, according to claim 10 in which each state of pre-stored sequence of notes, and forming by Hidden Markov Model on a finite state network
16. The audio signal processing method according to any one of claims 15 to 15. 17. The audio signal processing method according to claim 10, wherein said state transition determining step determines a state transition by a Viterbi algorithm. 18. The method according to claim 17, wherein in the state transition determining step, duration data corresponding to a note string stored in advance is included in the Viterbi algorithm.
3. The audio signal processing method according to 1.