JP2018141899A - Musical instrument sound recognition apparatus and musical instrument sound recognition program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the recognition accuracy of multiple intervals.SOLUTION: The musical instrument sound recognition device 10 according to the present invention is a musical instrument sound recognition device for recognizing musical instrument sound which is a sound of a musical instrument, which comprises a frequency analyzing unit 13 analyzing frequency for record data of recorded musical instrument sound for each frequency of each musical interval included in twelve scales, a F0 estimator 14 which processes the record data by a low pass filter which reduces frequency components higher than a predetermined frequency and estimates the interval which is the fundamental frequency of the reduced signal which is the processed signal, a multi-tone analysis unit 15 for identifying a plurality of musical intervals relating to record data on the basis of frequency analysis results for each frequency of each interval and estimation results of interval as a fundamental frequency, and an output unit 16 for outputting an identification result indicating a plurality of intervals identified by the multi-tone analysis unit 15.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a musical instrument sound recognition apparatus and a musical instrument sound recognition program.

  In recent years, with the development of machine learning, the development of music recognition technology in the field of music and the like has been remarkable. In pitch recognition such as singing voice or performance sound, a single tone can be easily recognized by a technique for extracting a fundamental frequency (F0) such as an autocorrelation method. On the other hand, for example, a performance sound mixed with a plurality of pitches played by an instrument capable of playing multiple sounds such as a piano or a guitar, for example, a part of the plurality of pitches when the F0 estimation described above is performed. It can happen that it can only be recognized, or it can recognize an incorrect pitch.

  Conventionally, frequency analysis such as Fourier transform has been used as a method for recognizing a performance sound in which a plurality of pitches are mixed. For example, Patent Document 1 describes a technique for recognizing chords by picking up a musical sound from a piano, converting it into an acoustic signal (recorded data), and applying a fast Fourier transform to the acoustic signal.

JP 2004-163767 A

  In frequency analysis based on Fourier transform, frequency analysis is performed using orthogonal frequencies, so that it is necessary to secure a certain time frame length and increase frequency resolution. In particular, in the low sound range, the pitch changes due to a minute change in frequency, so that a Fourier transform with a frequency resolution finer than the difference in frequency between adjacent pitches is necessary, and it is necessary to ensure a sufficient time frame length. For this reason, in frequency analysis by Fourier transform, it may be difficult to recognize a plurality of pitches by analyzing a fast melody. Further, for example, for each pitch constituting a chord, it is necessary to consider the inversion form. However, it is difficult to recognize each pitch in consideration of the inversion form only by performing the conventional frequency analysis described above. . That is, depending on only the conventional frequency analysis, it is difficult to distinguish and recognize so-called turning forms in which the combination of intervals is the same and the bass that is the lowest tone is different. In this way, the conventional method can fully recognize multiple pitches because it cannot recognize the pitch of a fast melody and cannot recognize each pitch in consideration of the turning form. Not be able to.

  The present invention has been made in view of the above circumstances, and an object thereof is to improve the recognition accuracy of a plurality of pitches.

  A musical instrument sound recognition apparatus according to an aspect of the present invention is a musical instrument sound recognition apparatus that recognizes a musical instrument sound that is a sound of a musical instrument, and for recording data obtained by recording the musical instrument sound, the frequency of each pitch included in a predetermined scale. Analyzing unit that performs frequency analysis every time, and processing the recorded data with a low-pass filter that reduces frequency components higher than a predetermined frequency, and estimating the pitch that is the fundamental frequency of the reduced signal that is the processed signal An identification unit for identifying a plurality of intervals related to the recording data based on a frequency analysis result for each frequency of each pitch by the estimation unit and an estimation result of a pitch that is a fundamental frequency by the estimation unit; An output unit that outputs an identification result indicating a plurality of pitches identified by the identification unit.

  The musical instrument sound recognition program according to one aspect of the present invention includes a computer, an analysis unit that performs frequency analysis for each frequency of each pitch included in a predetermined scale, with respect to recording data obtained by recording a musical instrument sound that is a sound of a musical instrument, The recording data is processed by a low-pass filter that reduces frequency components higher than a predetermined frequency, and the estimation unit that estimates the pitch that is the fundamental frequency of the reduced signal that is the processed signal, and the analysis unit, Based on the frequency analysis result for each frequency of the pitch and the estimation result of the pitch that is the fundamental frequency by the estimation unit, an identification unit that identifies a plurality of intervals related to the recording data, and a plurality of pitches identified by the identification unit And functioning as an output unit that outputs the identification result shown.

  In the musical instrument sound recognition apparatus and the musical instrument sound recognition program according to the present invention, a frequency analysis is performed for each frequency corresponding to each pitch included in a predetermined scale with respect to recording data obtained by recording a musical instrument sound, and a plurality of pitches are identified. The When performing frequency analysis by Fourier transform as in the prior art, it is necessary to ensure a certain time frame length in order to increase the frequency resolution. On the other hand, since the musical instrument sound recognition apparatus according to the present invention performs frequency analysis for each frequency corresponding to each pitch, there is no time frame length limitation and the pitch can be analyzed with an arbitrary time frame length. . Thereby, for example, a fast melody can be analyzed in a short time, and a fast melody can be appropriately identified. Further, in the musical instrument sound recognition apparatus according to the present invention, the recorded data is processed by the low-pass filter, the pitch that becomes the fundamental frequency of the reduced signal after processing is estimated, and the result of the estimation is taken into consideration to generate a plurality of pitches. Identified. Considering the estimation result of the pitch that becomes the fundamental frequency, the lowest pitch among the plurality of pitches related to the recording data is determined, so the so-called inversion type in which the combination of pitches is the same and the bass that is the lowest tone is different A plurality of intervals can be recognized by distinguishing them. For example, when the basic frequency is simply estimated for recording data in which a plurality of pitches are mixed, there may be a problem that only a part of the plurality of pitches can be recognized. In this regard, in the musical instrument sound recognition apparatus according to the present invention, the pitch that becomes the fundamental frequency is estimated for the reduced signal after the frequency component higher than the predetermined frequency is reduced by the low-pass filter. It is possible to estimate the pitch that becomes the fundamental frequency in a state in which the pitch other than the bass among the plurality of pitches included is reduced, that is, in a state in which mixing of a plurality of pitches is suppressed, and to improve the estimation accuracy of the fundamental frequency. be able to. Thereby, a plurality of pitches can be appropriately recognized by distinguishing the inversion form. As described above, according to the musical instrument sound recognition apparatus according to the present invention, it is possible to recognize the pitch of a fast melody and to recognize each pitch in consideration of the turning form, thereby improving the recognition accuracy of a plurality of pitches. Can be made.

  According to the present invention, it is possible to improve the recognition accuracy of a plurality of intervals.

It is a block diagram which shows the function structure of a musical instrument sound recognition apparatus. It is a figure which shows the hardware constitutions of the musical instrument sound recognition apparatus shown by FIG. It is a figure for demonstrating the frequency analysis by the frequency analysis part shown by FIG. 1, and is a figure which shows the amplitude intensity | strength of each pitch contained in 12 scales. It is a figure for demonstrating the frequency analysis by the frequency analysis part shown by FIG. 1, and is a figure for demonstrating scale vector derivation | leading-out of each pitch contained in 12 scales. It is a figure which shows the structural example of the neural network with which the multiple sound analysis part shown by FIG. 1 is provided. It is a flowchart which shows a series of processes of the musical instrument sound recognition method which the musical instrument sound recognition apparatus shown by FIG. 1 performs. It is a figure for demonstrating the conventional frequency analysis. It is a block diagram which shows the module structure of a musical instrument sound recognition program.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same reference numerals are used for the same or equivalent elements, and redundant descriptions are omitted.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 is a block diagram showing a functional configuration of a musical instrument sound recognition apparatus. The instrument sound recognition apparatus 10 shown in FIG. 1 is an apparatus that recognizes an instrument sound that is a sound played by an instrument. The instrument sound recognition device 10 recognizes a plurality of pitches included in the instrument sound. In the present embodiment, the musical instrument sound recognition apparatus 10 is described as recognizing a chord configuration as an example of a plurality of pitches included in a musical instrument sound. However, a plurality of pitches included in a musical instrument sound are a plurality of pitches that do not constitute a chord. A combination of these may be used. The instrument sound is a performance sound played by an instrument that can play multiple sounds, such as a guitar or a piano. A chord is a sound obtained by synthesizing a plurality of sounds having different pitches (frequency), and is a combination of specific pitches. Chords are also called chords.

  The instrument sound recognition apparatus 10 functionally includes a recording unit 11, a recording data storage unit 12, a frequency analysis unit 13 (analysis unit), an F0 estimation unit 14 (estimation unit), and a multiple sound analysis unit 15 ( An identification unit) and an output unit 16. The musical instrument sound recognition apparatus 10 is configured by, for example, hardware shown in FIG.

  FIG. 2 is a diagram illustrating a hardware configuration of the musical instrument sound recognition apparatus 10. As shown in FIG. 2, the instrument sound recognition apparatus 10 physically includes one or a plurality of processors 1001, a memory 1002 that is a main storage device, a storage 1003 such as a hard disk or a semiconductor memory, and data transmission / reception such as a network card. The computer system includes a communication device 1004, an input device 1005, and an output device 1006 such as a display, which are devices. Each function shown in FIG. 1 has an input device 1005, an output device 1006, and communication under the control of the processor 1001 by loading predetermined computer software on hardware such as the memory 1002 shown in FIG. This is realized by operating the apparatus 1004 and reading and writing data in the memory 1002 and the storage 1003.

  With reference to FIG. 1 again, details of each function of the musical instrument sound recognition apparatus 10 will be described.

  The recording unit 11 records instrument sounds in predetermined time units and obtains them as recorded data. The recording unit 11 samples instrument sounds in predetermined time units, and sequentially records the sampled instrument sounds as recording data. The sampling frequency is, for example, 16000 Hz or 44100 Hz. The sampled instrument sounds are referred to as samples (acoustic signals), and a predetermined number (n; n is an integer equal to or greater than 1) of samples arranged in time series are collectively referred to as recording data (frame). Each sample is the amplitude value (volume) of the instrument sound at the time when the sample is acquired, and is represented by 16 bits, for example.

  The recording unit 11 arranges each sample in time series (in the order of sampling), and sets the recording data for each predetermined number of samples. The number n of samples included in one recording data is, for example, 256. When the sampling frequency is 16000 Hz, the recording data corresponds to an instrument sound of about 0.016 seconds. The recording unit 11 continuously samples the instrument sound and continues to acquire recording data. The recording unit 11 sequentially outputs each recording data to the recording data storage unit 12.

  The recording data storage unit 12 sequentially receives the recording data from the recording unit 11 and stores the recording data acquired by the recording unit 11. The recorded data storage unit 12 is configured by, for example, a FIFO (First In First Out) buffer. In this case, when the number of recordable data items stored in the record data storage unit 12 is stored, the record data storage unit 12 is the oldest of the record data stored in the record data storage unit 12 (stored first. A) Discard the recorded data and store new recorded data. That is, the recording data storage unit 12 temporarily stores (buffers) a plurality of recording data. The recording data storage unit 12 outputs the stored recording data to the frequency analysis unit 13 and the F0 estimation unit 14.

  The frequency analysis unit 13 performs frequency analysis on the recorded data for each frequency of each pitch included in a predetermined scale. The predetermined scale is, for example, 12 scales. In the present embodiment, description will be made assuming that the predetermined scale is the 12th scale. When the predetermined scale is 12 scales, the pitches included in one octave are, for example, “A”, “B ♭”, “B”, “C”, “C #”, “D”, “E ♭”, “E” in order from the lowest tone. “F”, “F #”, “G”, and “A ♭”, each of which is an individual frequency. A value obtained by multiplying the frequency of a certain pitch by about 1.0592 is the frequency of the pitch one level higher. For example, assuming that the lowest “A” frequency is 55.00 Hz, the frequency of the pitch one higher (“B ♭”) is 55.00 × 1.0592≈58.25 Hz. A value obtained by doubling the frequency of a certain pitch is a frequency of the same pitch that is one octave higher. For example, if the frequency of “A” in an octave is 55.00 Hz, the frequency of “A” that is one octave higher is 55.00 × 2 = 110.00 Hz. Which frequency band (octave) is to be used for frequency analysis is determined in accordance with the frequency that the instrument plays, which is known in advance.

上記（１）式において、ｆは周波数（１２音階に含まれるいずれかの音程の周波数）、nは録音データに含まれるサンプリング数、x(k)はk番目のサンプルの振幅値、Sampling Rateはサンプリング周波数、Power(f)は周波数がfである音程（１２音階に含まれるいずれかの音程）の振幅強度を示している。 The frequency analysis unit 13 derives an amplitude intensity for each frequency of each pitch by integrating the signal of the recording data with a sine wave and a cosine wave corresponding to the frequency of each pitch included in the 12th scale, and the amplitude strength is calculated. The feature value of each pitch. More specifically, the frequency analysis unit 13 integrates the waveform of the recording data with a sine wave and a cosine wave corresponding to the frequency of each pitch, and adds the square of the integration result to the amplitude intensity of each pitch. To do. Specifically, the frequency analysis unit 13 derives the amplitude intensity for each frequency of each pitch based on the following equation (1).

In the above equation (1), f is the frequency (frequency of any pitch included in the 12th scale), n is the number of samplings included in the recording data, x (k) is the amplitude value of the kth sample, and Sampling Rate is The sampling frequency, Power (f), indicates the amplitude intensity of a pitch (any pitch included in the 12th scale) whose frequency is f.

Based on the above equation (1), the frequency analysis unit 13 derives the amplitude intensity for each frequency of 1.0592 times, for example, in order from 55 Hz which is the lowest “A” frequency. That is, as shown in FIG. 3, the frequency analysis unit 13 derives the amplitude intensity for m + 1 frequencies from 55 Hz, which is the lowest “A” frequency, to 55 × 1.0592 ^m Hz. Note that m + 1 is larger than the number of pitches (12) constituting at least 12 scales. That is, m is an integer of 11 or more. For example, the frequency analysis unit 13 derives the amplitude intensity for the frequency of the pitch of 3 octaves (12 × 3 = 36).

  The frequency analysis unit 13 performs a process of adding together the amplitude intensities of the same pitch with different octaves, and finally converts the amplitude intensities of twelve different pitches into a scale vector (chroma vector) that is a characteristic amount of each pitch. Derived as In the example shown in FIG. 4, the amplitude intensity of the same pitch of three octaves of an octave of 55 Hz to 104 Hz, an octave of 110 Hz to 208 Hz, and an octave of 220 Hz to 416 Hz is added up, and a scale that is a characteristic amount of each pitch is obtained. A vector (chroma vector) is derived. The frequency analysis unit 13 supplies a value (first feature value based on the feature value of each pitch) obtained by normalizing the scale vector value, which is the feature value of each pitch, to a value of 0 to 1 to the multiple sound analysis unit 15. Output. Normalization of each pitch is derived, for example, by dividing the scale vector of each pitch by the scale vector of the pitch having the largest scale vector.

  The F0 estimation unit 14 estimates a pitch that is a fundamental frequency (F0) related to the recording data. The fundamental frequency is the frequency of the lowest frequency component included in the signal. The F0 estimation unit 14 processes the recorded data with a low-pass filter that reduces frequency components higher than a predetermined frequency, and estimates the pitch that is the fundamental frequency of the reduced signal that is the signal after the processing. For example, for the sound up to the lowest octave frequency band (55 Hz to 110 Hz), the pitch may be analyzed by the F0 estimation unit. In this case, a low-pass filter that passes only sound of 110 Hz or less is used.

  The purpose of performing the processing by the F0 estimation unit 14 will be described. As described above, the frequency analysis unit 13 derives the amplitude intensity for every 12 types of pitches. The multiple sound analysis unit 15 described later estimates a chord (a chord) based on a combination of pitches having a large amplitude intensity (details will be described later). Here, a chord may not be uniquely identified from a combination of pitches. That is, even if the combination of pitches is the same, for example, C (the root is the base that is the lowest note) and C / E that is the first inversion (3 degrees is the base that is the lowest note) And C / G that is the second rotation type (5 degrees is the base that is the lowest note), and there is a case where the chord cannot be accurately estimated simply by deriving the amplitude intensity for each pitch. is there. On the other hand, by extracting the strongest scale in the F0 extraction unit only in the lowest octave frequency band, it becomes easy to determine the bass sound. The bass sound is often a single sound in the lowest octave frequency band, and is characterized by being easily determined by the autocorrelation method. By using a reduced signal in which the frequency components higher than the predetermined frequency are reduced by the low-pass filter and the frequency components of the pitch other than the bass included in the chord are reduced, the base frequency (basic frequency) ) Can be easily estimated, and the chord can be estimated by the multiple sound analysis unit 15 to be described later even when the turning form is taken into consideration.

  The F0 estimator 14 uses the autocorrelation method, for example, for the reduced signal that has been processed by the above-described low-pass filter to indicate the likelihood that each pitch included in the 12th scale is a fundamental frequency. One likelihood is derived for each pitch. The autocorrelation method is a method generally used for deriving a fundamental frequency, and prepares two identical data including a plurality of samples, and the correlation value (autocorrelation) between data in a state shifted by k samples. Value) and a fundamental frequency is derived from the autocorrelation value. The autocorrelation value is derived by adding the values obtained by multiplying the corresponding samples between the two pieces of data in the state where the samples are shifted, and adding them for all the samples. Correlation between data (autocorrelation value) is determined according to the added value. For example, the autocorrelation value when k samples are shifted is the likelihood that “sampling frequency” / “k” (Hz) is the fundamental frequency. In this embodiment, since the frequency is determined for each pitch constituting the 12th scale, an autocorrelation value shifted by “sampling frequency” / “frequency of each pitch constituting the 12th scale” (sample) is derived. The first likelihood indicating the likelihood that each pitch included in the 12th scale is the fundamental frequency is derived. The F0 estimation unit 14 increases the first likelihood as the pitch of the autocorrelation value increases. In this way, the F0 estimation unit 14 estimates the pitch that is the fundamental frequency by deriving the first likelihood of each pitch. The F0 estimation unit 14 outputs a value (second feature value based on the first likelihood of each interval) obtained by normalizing the first likelihood of each interval to a value of 0 to 1 to the multiple sound analysis unit 15. . Normalization of each pitch is derived, for example, by dividing the first likelihood of each pitch by the first likelihood of the pitch having the largest first likelihood.

  The multiple sound analysis unit 15 identifies a chord related to the recording data based on the frequency analysis result for each frequency of each pitch by the frequency analysis unit 13 and the estimation result of the pitch that is the fundamental frequency by the F0 estimation unit 14. . As described above, the multiple feature analysis unit 15 receives a first feature value that is a normalized value of a scale vector (chroma vector) that is a feature value of each pitch, as a frequency analysis result for each frequency of each pitch. In addition, as a result of estimating the pitch that is the fundamental frequency, a second feature value that is a value obtained by normalizing the first likelihood indicating the likelihood that each pitch is the fundamental frequency is input.

  The multiple sound analyzer 15 has a neural network N1 shown in FIG. The multiple sound analyzer 15 uses the neural network N1 to identify and learn chords related to the recorded data. The neural network N1 has a first feature quantity based on a scale vector (chroma vector) that is a feature quantity of each pitch, and a second feature quantity based on a first likelihood that indicates the likelihood that each pitch is a fundamental frequency. Is input, and the second likelihood indicating the likelihood of the chord related to the recording data is output. The second likelihood is, for example, a sigmoid function value and can take a value of 0 to 1. The higher the second likelihood, the higher the possibility that the chord is related to (included in) the recorded data. For example, the multiple sound analysis unit 15 identifies a chord having the largest second likelihood as a chord related to the recording data, and outputs information specifying the chord to the output unit 16.

  The neural network N1 includes an input layer 31 including a plurality of input nodes 311 corresponding to the first feature value of each pitch, and an input layer 32 including a plurality of input nodes 321 corresponding to the second feature value of each pitch. The intermediate layer 33 including a plurality of intermediate nodes 331 and the output layer 34 including an output node 341 are provided. The input node 311 is provided corresponding to each of the first feature values of each of the twelve pitches received from the frequency analysis unit 13, and each of the twelve input nodes 311 has a first of any pitch. Is input. The input nodes 321 are provided corresponding to the second feature values of the twelve pitches received from the F0 estimating unit 14, and each of the twelve input nodes 321 includes the second feature value of any pitch. Is input. The intermediate node 331 performs a predetermined calculation using the first feature amount input to the one or more input nodes 311 and the second feature amount input to the one or more input nodes 321, and calculates the calculation result. Output to the output node 341. There are as many output nodes 341 as the number of chords to be estimated, and each output node 341 outputs the second likelihood of one of the chords. The output node 341 calculates the second likelihood of the chord using the calculation result received from the intermediate node 331, and outputs the second likelihood.

  The output unit 16 outputs an identification result indicating the chord identified by the multiple sound analysis unit 15. The output unit 16 outputs information specifying the chord input from the multiple sound analysis unit 15 to the outside of the instrument sound recognition apparatus 10 as an identification result.

  Next, a series of processes of the instrument sound recognition method in the instrument sound recognition apparatus 10 will be described with reference to FIG. FIG. 6 is a flowchart showing a series of processing of the instrument sound recognition method performed by the instrument sound recognition apparatus 10.

  First, the recording unit 11 samples instrument sounds in predetermined time units, and sequentially acquires the sampled instrument sounds as recording data (step S1). Then, the recording unit 11 arranges the samples in time series and sequentially outputs the recorded data to the recording data storage unit 12 as recording data for each predetermined number of samples.

  Subsequently, the recording data storage unit 12 sequentially receives the recording data from the recording unit 11 and stores the recording data acquired by the recording unit 11 (step S2). The recording data storage unit 12 outputs the stored recording data to the frequency analysis unit 13 and the F0 estimation unit 14.

  Subsequently, the frequency analysis unit 13 performs frequency analysis on the recorded data for each frequency of pitches included in the 12th scale (step S3). The frequency analysis unit 13 derives an amplitude intensity for each frequency of each pitch by integrating the signal of the recording data with a sine wave and a cosine wave corresponding to the frequency of each pitch included in the 12th scale, and the amplitude strength is calculated. The feature value of each pitch. The frequency analysis unit 13 supplies a value (first feature value based on the feature value of each pitch) obtained by normalizing the scale vector value, which is the feature value of each pitch, to a value of 0 to 1 to the multiple sound analysis unit 15. Output.

  Subsequently, the F0 estimation unit 14 estimates a pitch that is a fundamental frequency (F0) related to the recording data (step S4). The F0 estimation unit 14 processes the recorded data with a low-pass filter that reduces frequency components higher than a predetermined frequency, and estimates the pitch that is the fundamental frequency of the reduced signal that is the signal after the processing. The F0 estimator 14 uses the autocorrelation method, for example, for the reduced signal that has been processed by the above-described low-pass filter to indicate the likelihood that each pitch included in the 12th scale is a fundamental frequency. One likelihood is derived for each pitch. The F0 estimation unit 14 outputs a value (second feature value based on the first likelihood of each interval) obtained by normalizing the first likelihood of each interval to a value of 0 to 1 to the multiple sound analysis unit 15. .

  Subsequently, the multiple tone analysis unit 15 performs chords related to the recording data based on the frequency analysis result for each frequency of each pitch by the frequency analysis unit 13 and the pitch estimation result that is the fundamental frequency by the F0 estimation unit 14. Is identified (step S5). The multiple sound analysis unit 15 identifies a chord related to the recording data using the neural network N1. The neural network N1 has a first feature quantity based on a scale vector (chroma vector) that is a feature quantity of each pitch, and a second feature quantity based on a first likelihood that indicates the likelihood that each pitch is a fundamental frequency. Is input, and the second likelihood indicating the likelihood of the chord related to the recording data is output. For example, the multiple sound analysis unit 15 identifies a chord having the largest second likelihood as a chord related to the recording data, and outputs information specifying the chord to the output unit 16.

  And the output part 16 outputs the identification result which shows the chord identified by the multiple sound analysis part 15 to the exterior of the musical instrument sound recognition apparatus 10 (step S6). The above is an example of a series of processes of the instrument sound recognition method performed by the instrument sound recognition apparatus 10.

  Next, an instrument sound recognition program P for causing a computer to function as the instrument sound recognition apparatus 10 will be described with reference to FIG.

  The instrument sound recognition program P includes a main module P10, a recording module P11, a recording data storage module P12, a frequency analysis module P13, an F0 estimation module P14, a multiple sound analysis module P15, and an output module P16. The main module P10 is a part that comprehensively controls processing as the musical instrument sound recognition apparatus 10. The functions expressed by executing the recording module P11, the recording data storage module P12, the frequency analysis module P13, the F0 estimation module P14, the multiple sound analysis module P15, and the output module P16 are the recording unit 11 and the recording data storage, respectively. The functions of the unit 12, the frequency analysis unit 13, the F0 estimation unit 14, the multiple sound analysis unit 15, and the output unit 16 are the same.

  The musical instrument sound recognition program P is provided by, for example, a recording medium such as a CD-ROM, a DVD, or a ROM, or a semiconductor memory. The musical instrument sound recognition program P may be provided via a network as a computer data signal superimposed on a carrier wave.

  Next, the effect of the musical instrument sound recognition apparatus 10 according to the present embodiment will be described in comparison with the conventional musical instrument sound recognition technology.

  In conventional instrument sound recognition technology, it is common to perform frequency analysis by Fourier transform when recognizing chords. In the frequency analysis by Fourier transform, frequency analysis by orthogonal frequency is performed. As shown in FIG. 7, when the time frame length is T, the base frequency is 1 / T, 2 / T,... N / T ( When the number of samples is n), the frequency interval is 1 / T. For this reason, for example, in order to be able to analyze up to a low-range sound (55 Hz), the difference between the upper pitch and the sound (58 Hz) is 3 Hz, so 1 / T is 3 Hz or higher. A small frequency resolution is required. Therefore, the time frame length T is required to be 333 milliseconds or more, and it is difficult to analyze a melody whose pitch changes in a time faster than 333 milliseconds (hereinafter, sometimes referred to as “first problem”).

  In addition, even if the combination of pitches is the same for a chord, for example, C (the root is the base that is the lowest note) and C / E that is the first turn form (3 degrees is the lowest note) 2) and C / G (the 5th degree is the base that is the lowest note), so the chord cannot be accurately estimated by simply recognizing the combination of pitches. In some cases (hereinafter, referred to as “second problem”). As described above, the conventional musical instrument sound recognition technology has the first and second problems described above, and it is difficult to say that the recognition accuracy of chords is sufficiently secured.

  On the other hand, in the musical instrument sound recognition apparatus 10 according to the present embodiment, in order to solve the first problem, the recorded data obtained by recording the musical instrument sound has a frequency for each frequency corresponding to each pitch included in the 12th scale. Analyze and identify multiple pitches. That is, the musical instrument sound recognition apparatus 10 changes the base frequency of the discrete Fourier transform in the conventional musical instrument sound recognition technology to a frequency corresponding to each pitch included in the 12 scales, and performs frequency analysis without using the orthogonal frequency. . In such a configuration, since the frequency interval is determined independently of the time frame length, the time frame length is not limited in the frequency analysis, and the pitch can be analyzed with an arbitrary time frame length. Thereby, for example, a fast melody can be analyzed in a short time, and a fast melody can be appropriately identified. Moreover, it becomes possible to analyze a slow melody in a long time. As shown in FIG. 3, since the frequency of each pitch of the 12th scale becomes higher as the pitch becomes higher, the pitch can be recognized without fine frequency analysis such as Fourier transform. Become. Therefore, there is an advantage that even an inexpensive device can be implemented in terms of calculation amount.

  In the musical instrument sound recognition apparatus 10 according to the present embodiment, in order to solve the second problem, the recording data is processed by the low-pass filter, and the pitch that becomes the fundamental frequency of the reduced signal after processing is estimated, and the estimation is performed. A plurality of pitches are identified in consideration of the above result. Considering the estimation result of the pitch that becomes the fundamental frequency, the lowest pitch among the plurality of pitches related to the recording data is determined, so the so-called inversion type in which the combination of pitches is the same and the bass that is the lowest tone is different Differentiating complex chords can be recognized. For example, when the basic frequency is simply estimated for recording data in which a plurality of pitches are mixed, there may be a problem that only a part of the plurality of pitches can be recognized. In this regard, the musical instrument sound recognition apparatus 10 according to the present embodiment estimates the pitch that becomes the fundamental frequency for the reduced signal after the frequency component higher than the predetermined frequency is reduced by the low-pass filter. It is possible to estimate the pitch that becomes the fundamental frequency in a state where the pitch other than the bass is reduced among the multiple pitches included in the data, that is, in a state where mixing of multiple pitches is suppressed, and the estimation accuracy of the fundamental frequency is improved. Can be improved. Thereby, a plurality of pitches can be appropriately recognized by distinguishing the inversion form.

  As described above, according to the musical instrument sound recognition apparatus 10 according to the present embodiment, it is possible to recognize a pitch of a fast melody and to recognize each pitch in consideration of the turning form, and to recognize a plurality of pitches. Can be improved.

  The frequency analysis unit 13 derives an amplitude intensity for each frequency of each pitch by integrating a signal of the recording data with a sine wave and a cosine wave corresponding to the frequency of each pitch included in the 12 scales. And the multiple sound analysis unit 15 identifies chords related to the recording data by using the feature value of each pitch as a frequency analysis result for each frequency of each pitch. Thereby, frequency analysis can be appropriately performed for each frequency of the pitches included in the 12th scale, and the recognition accuracy of each pitch included in the 12th scale can be improved.

  The F0 estimation unit derives, for each pitch, a first likelihood indicating the likelihood that each pitch included in the 12th scale is a fundamental frequency using the autocorrelation method for the low frequency signal described above. A pitch that is a fundamental frequency is estimated based on the first likelihood. Thereby, the pitch which is a fundamental frequency can be estimated with high precision.

  The multiple sound analysis unit 15 receives the first feature value based on the feature value of each pitch and the second feature value based on the first likelihood of each pitch, and the likelihood of a plurality of pitches related to the recording data. The chords related to the recording data are identified using the neural network N1 that outputs the second likelihood indicative of. Thus, both the first feature value corresponding to the frequency analysis result for each frequency of each pitch and the second feature value based on the first likelihood indicating the likelihood that each pitch is a fundamental frequency are considered. Thus, the chord can be estimated appropriately.

  In addition, the block diagram used for description of the said embodiment has shown the block of the functional unit. These functional blocks (components) are realized by any combination of hardware and / or software. Further, the means for realizing each functional block is not particularly limited. That is, each functional block may be realized by one device physically and / or logically coupled, and two or more devices physically and / or logically separated may be directly and / or indirectly. (For example, wired and / or wirelessly) and may be realized by these plural devices.

  For example, the instrument sound recognition apparatus 10 in the above embodiment may function as a computer that performs processing of the instrument sound recognition apparatus 10 in the above embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the musical instrument sound recognition apparatus 10 according to the present embodiment. The musical instrument sound recognition apparatus 10 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

  In the following description, the term “apparatus” can be read as a circuit, a device, a unit, or the like. The hardware configuration of the musical instrument sound recognition device 10 may be configured to include one or a plurality of the devices illustrated in FIG. 2 or may be configured not to include some devices.

  Each function in the musical instrument sound recognition apparatus 10 reads predetermined software (program) on hardware such as the processor 1001 and the memory 1002, so that the processor 1001 performs calculation, and communication by the communication apparatus 1004, memory 1002, and storage This is realized by controlling reading and / or writing of data in 1003.

  For example, the processor 1001 controls the entire computer by operating an operating system. The processor 1001 may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic device, a register, and the like.

  Further, the processor 1001 reads a program (program code), a software module, and / or data from the storage 1003 and / or the communication device 1004 to the memory 1002, and executes various processes according to these. As the program, a program that causes a computer to execute at least a part of the operations described in the above embodiments is used. For example, the frequency analysis unit 13 of the musical instrument sound recognition apparatus 10 may be realized by a control program stored in the memory 1002 and operated by the processor 1001, and may be realized similarly for other functional blocks. Although the above-described various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. Note that the program may be transmitted from a network via a telecommunication line.

  The memory 1002 is a computer-readable recording medium and includes, for example, at least one of ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), and the like. May be. The memory 1002 may be called a register, a cache, a main memory (main storage device), or the like. The memory 1002 can store programs (program codes), software modules, and the like that can be executed to implement the instrument sound recognition method according to the above-described embodiment.

  The storage 1003 is a computer-readable recording medium such as an optical disc such as a CD-ROM (Compact Disc ROM), a hard disc drive, a flexible disc, a magneto-optical disc (eg, a compact disc, a digital versatile disc, a Blu-ray). (Registered trademark) disk, smart card, flash memory (for example, card, stick, key drive), floppy (registered trademark) disk, magnetic strip, and the like. The storage 1003 may be referred to as an auxiliary storage device. The storage medium described above may be, for example, a database, a server, or other suitable medium including the memory 1002 and / or the storage 1003.

  The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via a wired and / or wireless network, and is also referred to as a network device, a network controller, a network card, a communication module, or the like.

  The input device 1005 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, or the like) that accepts an external input. The output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may have an integrated configuration (for example, a touch panel).

  Each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured with a single bus or may be configured with different buses between apparatuses.

  The musical instrument sound recognition apparatus 10 includes hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). A part or all of each functional block may be realized by the hardware. For example, the processor 1001 may be implemented by at least one of these hardware.

  Although the present invention has been described in detail above, it will be apparent to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented as a modified and changed mode without departing from the spirit and scope of the present invention defined by the description of the scope of claims. Therefore, the description of the present specification is for illustrative purposes and does not have any limiting meaning to the present invention.

  As long as there is no contradiction, the order of the processing procedures, sequences, flowcharts, and the like of each aspect / embodiment described in this specification may be changed. For example, the methods described herein present the elements of the various steps in an exemplary order and are not limited to the specific order presented.

  Input / output information or the like may be stored in a specific location (for example, a memory) or may be managed by a management table. Input / output information and the like can be overwritten, updated, or additionally written. The output information or the like may be deleted. The input information or the like may be transmitted to another device.

  The determination may be performed by a value represented by 1 bit (0 or 1), may be performed by a true / false value (Boolean: true or false), or may be compared with a numerical value (for example, a predetermined value) Comparison with the value).

  Each aspect / embodiment described in the present specification may be used alone, may be used in combination, or may be used by switching according to execution. In addition, notification of predetermined information (for example, notification of “being X”) is not limited to explicitly performed, and may be performed implicitly (for example, notification of the predetermined information is not performed). Good.

  Software, whether it is called software, firmware, middleware, microcode, hardware description language, or other names, instructions, instruction sets, codes, code segments, program codes, programs, subprograms, software modules , Applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc. should be interpreted broadly.

  Also, software, instructions, etc. may be transmitted / received via a transmission medium. For example, software may use websites, servers, or other devices using wired technology such as coaxial cable, fiber optic cable, twisted pair and digital subscriber line (DSL) and / or wireless technology such as infrared, wireless and microwave. When transmitted from a remote source, these wired and / or wireless technologies are included within the definition of transmission media.

  Information, signals, etc. described herein may be represented using any of a variety of different technologies. For example, data, commands, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description are voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these May be represented by a combination of

  Note that the terms described in this specification and / or terms necessary for understanding this specification may be replaced with terms having the same or similar meaning.

  As used herein, the terms “system” and “network” are used interchangeably.

  In addition, information, parameters, and the like described in this specification may be represented by absolute values, may be represented by relative values from a predetermined value, or may be represented by other corresponding information. .

  The names used for the above parameters are not limiting in any way. Further, mathematical formulas and the like that use these parameters may differ from those explicitly disclosed herein.

  The terms “connected”, “coupled”, or any variation thereof, means any direct or indirect connection or coupling between two or more elements and It can include the presence of one or more intermediate elements between two “connected” or “coupled” elements. The coupling or connection between the elements may be physical, logical, or a combination thereof. As used herein, the two elements are radio frequency by using one or more wires, cables and / or printed electrical connections, and as some non-limiting and non-inclusive examples By using electromagnetic energy, such as electromagnetic energy having a wavelength in the region, microwave region, and light (both visible and invisible) region, it can be considered to be “connected” or “coupled” to each other.

  As used herein, the phrase “based on” does not mean “based only on,” unless expressly specified otherwise. In other words, the phrase “based on” means both “based only on” and “based at least on.”

  Any reference to elements using the designations “first”, “second”, etc. as used herein does not generally limit the amount or order of those elements. These designations can be used herein as a convenient way to distinguish between two or more elements. Thus, a reference to the first and second elements does not mean that only two elements can be employed there, or that in some way the first element must precede the second element.

As long as “including”, “comprising” and variations thereof are used in the specification or claims, these terms are inclusive of the term “comprising”. Intended to be Further, the term “or” as used herein or in the claims is not intended to be an exclusive OR.

  In this specification, a device is intended to include a plurality of devices unless the context or technology clearly indicates that there is only one device.

  DESCRIPTION OF SYMBOLS 10 ... Musical instrument sound recognition apparatus, 13 ... Frequency analysis part (analysis part), 14 ... F0 estimation part (estimation part), 15 ... Multiple sound analysis part (identification part), 16 ... Output part, N1 ... Neural network, P ... Musical instrument sound recognition program.

Claims (5)

An instrument sound recognition device for recognizing an instrument sound that is a sound of an instrument,
For the recording data recording the instrument sound, an analysis unit for performing frequency analysis for each frequency of each pitch included in a predetermined scale,
The recording data is processed by a low-pass filter that reduces a frequency component higher than a predetermined frequency, and an estimation unit that estimates a pitch that is a fundamental frequency of a reduced signal that is a signal after the processing;
An identification unit for identifying a plurality of pitches related to the recording data based on a frequency analysis result for each frequency of each pitch by the analysis unit and a pitch estimation result that is the fundamental frequency by the estimation unit;
An instrument sound recognition apparatus comprising: an output unit that outputs an identification result indicating the plurality of pitches identified by the identification unit. The analysis unit derives an amplitude intensity for each frequency of each pitch by integrating the signal of the recording data with a sine wave and a cosine wave corresponding to the frequency of each pitch included in the 12th scale, The intensity is the characteristic amount of each pitch,
The musical instrument sound recognition apparatus according to claim 1, wherein the identification unit identifies a plurality of pitches related to the recording data by using the characteristic amount of each pitch as a frequency analysis result for each frequency of the pitch.   The estimation unit derives, for each pitch, a first likelihood indicating the likelihood that each pitch included in the 12 scales is the fundamental frequency using the autocorrelation method for the reduced signal, The musical instrument sound recognition apparatus according to claim 2, wherein a pitch that is the fundamental frequency is estimated based on the first likelihood of the pitch.   The identification unit receives a first feature value based on the feature value of each pitch and a second feature value based on the first likelihood of each pitch, and inputs a plurality of pitches related to the recorded data. The musical instrument sound recognition apparatus according to claim 3, wherein a plurality of pitches related to the recorded data are identified using a neural network that outputs a second likelihood indicating the likelihood of the recorded sound. Computer
An analysis unit that performs frequency analysis for each frequency of each pitch included in a predetermined scale with respect to recording data obtained by recording an instrument sound that is a sound of an instrument;
The recording data is processed by a low-pass filter that reduces a frequency component higher than a predetermined frequency, and an estimation unit that estimates a pitch that is a fundamental frequency of a reduced signal that is a signal after the processing;
An identification unit for identifying a plurality of pitches related to the recording data based on a frequency analysis result for each frequency of each pitch by the analysis unit and a pitch estimation result that is the fundamental frequency by the estimation unit;
An output unit for outputting an identification result indicating the plurality of pitches identified by the identification unit;
Musical instrument sound recognition program to function as.

Images