JP2017090848A - Music analysis device and music analysis method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To specify a structural section of music with high precision.SOLUTION: A music analysis device 100 is configured to analyze a structural section as a section of a musical structure in music, and comprises: a feature extraction part 22 which extracts, for each unit section of an acoustic signal X representing sound of the music, a timbre feature quantity FT representing a feature of a timbre of the sound and a chord feature quantity FC representing a feature of a chord of the sound; a timbre observation model which represents, for each structural section including a plurality of unit sections, a generation process of the timbre feature quantity FT probabilistically; an estimation processing part 24 which estimates, for each chord observation model representing the generation process of the chord feature quantity FC probabilistically for each unit section within each structural section, a posterior distribution through estimation processing to which the timbre feature quantity FT and chord feature quantity FC that the feature extraction part 22 extracts are applied; and a structure analysis part 26 which specifies a plurality of structural sections of the music from a result of the estimation processing.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for analyzing music.

  Play a specific section such as “Chibi” or “A melody” in the music selectively, or connect the sections extracted from multiple songs to each other (for example, remix for DJ use) To do this, it is necessary to analyze the musical structure of the music. Patent Document 1 discloses a technique for detecting and integrating repeated sections by analyzing the similarity of each feature amount sequentially extracted from an acoustic signal, and selecting a chorus section of music from a plurality of combined repeated sections. Is disclosed.

JP 2004-233965 A

  However, it is actually difficult to analyze the structure of music with high accuracy using existing analysis techniques including the technique of Patent Document 1. In view of the above circumstances, an object of the present invention is to specify a structure section of a music piece with high accuracy.

  In order to solve the above-described problems, a music analysis device according to a preferred aspect of the present invention provides a timbre feature amount representing a timbre feature of a sound and a chord of the sound for each unit section of a sound signal representing the sound of the music. A feature extraction unit that extracts chord feature values representing the features of the melody and a musical structure division within the musical composition, and a stochastic feature value generation process is stochastically expressed for each structure section including at least one unit section For each of the timbre observation model and the chord observation model that probabilistically represents the chord feature generation process for each unit section in each structural section, the timbre feature amount and chord feature amount extracted by the feature extraction unit were applied. An estimation processing unit that estimates the posterior distribution by the estimation process, and a structure analysis unit that specifies a plurality of structural sections of the music from the result of the estimation process. In the above aspect, the posterior distributions of the timbre observation model for each structural section and the chord observation model for each unit section in each structural section are estimated by the estimation process. That is, a timbre observation model that reflects the tendency that the timbres are roughly unified within one structural section of the music, and a chord that reflects the tendency that the chords sequentially transition for each unit section within the structural section The observation model is used for the estimation process. Therefore, it is possible to specify the structure section of the music with high accuracy.

  In a preferred aspect of the present invention, the unit section is a measure of music. In the above aspect, since the timbre feature value and the chord feature value are extracted for each unit section with each of the plurality of measures of the music as a unit section, the structure section tends to transition at the boundary of adjacent measures. Therefore, there is an advantage that the structure section of the music can be estimated with high accuracy.

  In a preferred aspect of the present invention, the feature extraction unit extracts a basic timbre feature amount including a plurality of elements according to an acoustic timbre from an acoustic signal for each unit section, and for each of the plurality of unit sections, the unit section A first process for selecting a minimum value for each element of the basic timbre feature quantity between each of a plurality of peripheral unit sections before and after the first and a maximum value of a minimum value over a plurality of peripheral unit sections for each element. The timbre feature amount is generated by executing the two processes. In the above aspect, for each of the plurality of unit sections of the acoustic signal, the first process for selecting the minimum value for each element of the basic timbre feature quantity between each of the peripheral unit sections and the minimum over the plurality of peripheral unit sections A timbre feature amount is generated by executing the second process of selecting the maximum value for each element. That is, a common component between the unit section and each peripheral unit section is extracted in the first process, and a timbre feature amount is generated by integrating elements over a plurality of peripheral unit sections. Therefore, it is possible to easily reduce unsteady noise components that change over time and generate appropriate timbre feature quantities.

  In a preferred aspect of the present invention, the estimation processing unit executes the estimation process using an infinite mixture distribution with an infinite number of probability distributions as a timbre observation model. In the above aspect, since the infinite mixture distribution is used as a timbre observation model, the number of mixtures in the probability distribution varies according to the timbre characteristics of the acoustic signal. Therefore, there is an advantage that the posterior distribution of the timbre observation model can be appropriately estimated according to the characteristics of the acoustic signal.

  In a preferred aspect of the present invention, the estimation processing unit includes a plurality of state sequences corresponding to each unit section in the structure section for the plurality of structure sections, and from the last state of each structure section to another structure section. A state transition model capable of transition to the first state is used for the estimation process. In the above aspect, a state transition that includes a series of a plurality of states corresponding to a unit section with respect to a plurality of structural sections and that can transition from the last state of each structural section to the leading state of another structural section The model is applied to the estimation process. That is, a tendency that a structural section composed of a plurality of unit sections transitions sequentially in music is appropriately expressed in the state transition model. Therefore, it is possible to estimate the structure section of the music with high accuracy.

  In a preferred aspect of the present invention, the estimation processing unit repeats the estimation process multiple times with different initial values, thereby estimating a structure estimation sequence in which identification codes indicating the structure sections are arranged for each unit section of the music piece. The process is specified for each process, and the structure analysis unit specifies the structure section of the music from the plurality of structure estimation sequences specified by the estimation process performed a plurality of times. In the above aspect, the structure section of the music is specified from the plurality of structure estimation sequences generated by the estimation process using different initial values. Therefore, there is an advantage that the structure section of the music can be estimated with high accuracy robustly against the fluctuation of the initial value applied to each estimation process (difference in the structure estimation sequence for each estimation process).

  In the music analysis method according to a preferred aspect of the present invention, the computer system uses a timbre feature amount representing the timbre feature of the sound and a chord representing the chord feature of the sound for each unit section of the acoustic signal representing the sound of the tune. A timbre observation model that extracts features and extracts the structure of musical structure in the music and includes at least one unit section. The posterior distribution is estimated by the estimation process using the extracted timbre features and chord features for each of the chord observation models that probabilistically express the generation process of the chord features for each unit interval. Identify multiple structural sections of the song.

It is a lineblock diagram of a music analysis device concerning a 1st embodiment of the present invention. It is a flowchart of a feature extraction process. It is explanatory drawing of the unit area (measure) of object music. It is a flowchart of a noise compression process. It is explanatory drawing of a noise suppression process. It is explanatory drawing of a structure estimation series. It is explanatory drawing of a music structure model. It is explanatory drawing of the process by a structure analysis part. It is a schematic diagram of an estimation result image. It is a flowchart of operation | movement of a music analysis apparatus. It is explanatory drawing of operation | movement of the structure analysis part in 2nd Embodiment.

<First Embodiment>
FIG. 1 is a configuration diagram of a music analysis apparatus 100 according to the first embodiment of the present invention. The music analysis apparatus 100 according to the first embodiment is an information processing apparatus that analyzes the musical structure of an arbitrary piece of music (hereinafter referred to as “target music”). Specifically, the music analysis device 100 includes a plurality of sections (hereinafter referred to as “structure sections”) in which the target music is divided on the time axis according to a musical structure (musical meaning and auditory impression). To analyze. Each structural section is, for example, a section such as “Intro”, “A melody”, “B melody”, “C melody”, or “rust” (elements constituting the target music).

  As illustrated in FIG. 1, the music analysis device 100 according to the first embodiment is realized by a computer system including an arithmetic processing device 12, a storage device 14, and a display device 16. For example, a portable information processing device such as a mobile phone or a smartphone, or a portable or stationary information processing device such as a personal computer can be used as the music analysis device 100. The display device 16 (for example, a liquid crystal display panel) displays an image instructed from the arithmetic processing device 12. For example, the analysis result of the target music is displayed on the display device 16.

  The storage device 14 stores a program executed by the arithmetic processing device 12 and various data used by the arithmetic processing device 12. A known recording medium such as a semiconductor recording medium or a magnetic recording medium or a combination of a plurality of types of recording media is arbitrarily employed as the storage device 14. The memory | storage device 14 of 1st Embodiment memorize | stores the acoustic signal X showing the sound (for example, performance sound and singing sound) of object music. It is also possible to supply the acoustic signal X to the music analysis device 100 from an external device such as a playback device that reproduces the acoustic signal X recorded on a recording medium (for example, an optical disc).

  The arithmetic processing unit 12 functions as a plurality of elements (a feature extraction unit 22, an estimation processing unit 24, and a structure analysis unit 26) for analyzing the structure of the target musical piece by executing a program stored in the storage device 14. To do. A configuration in which the function of the arithmetic processing device 12 is realized by a set of a plurality of devices (that is, a system), or a configuration in which a dedicated electronic circuit shares a part of the function of the arithmetic processing device 12 may be employed. Each function realized by the arithmetic processing unit 12 will be described in detail below.

<Feature extraction unit 22>
The feature extraction unit 22 extracts the feature amount of the acoustic signal X. The feature extraction unit 22 of the first embodiment extracts a timbre feature value FT (T: timbre) and a chord feature value FC (C: chord) for each unit section obtained by dividing the acoustic signal X on the time axis. The timbre feature amount FT is a feature amount that represents the timbre feature of the sound represented by the acoustic signal X. In the first embodiment, a timbre feature amount FT corresponding to the MFCC (Mel-Frequency Cpestrum Coefficient) of the acoustic signal X is exemplified. On the other hand, the chord feature value FC is a feature value representing the feature of the acoustic chord (code) represented by the acoustic signal X. In the first embodiment, the feature value FC corresponding to the chroma vector is exemplified. In the chroma vector, a numerical value obtained by adding the intensities of frequency components corresponding to a scale tone (for example, each of twelve semitones of the equal temperament) over the plurality of octaves in the acoustic signal X is arranged for each of a plurality of different scale sounds. It is a dimension vector.

  FIG. 2 is a flowchart of a process SA1 in which the feature extraction unit 22 of the first embodiment extracts the timbre feature quantity FT and the chord feature quantity FC (hereinafter referred to as “feature extraction process”). When the analysis of the target music is instructed by the user, the feature extraction process SA1 in FIG.

  When the feature extraction process SA1 is started, the feature extraction unit 22 divides the acoustic signal X into a plurality of unit sections on the time axis as illustrated in FIG. 3 (SA11). Specifically, the feature extraction unit 22 detects a beat point of the acoustic signal X, and demarcates a section corresponding to one bar corresponding to a predetermined number of beat points as a unit section. For detection of beat points and measures (unit intervals), for example, a technique described in JP-A-2015-114361 is preferably used. However, the method of dividing the acoustic signal X into a plurality of unit sections is arbitrary, and is not limited to the above examples. As described above, in the first embodiment, the timbre feature value FT and the chord feature value FC are extracted for each unit section (that is, for each measure) with each of the plurality of measures of the target music as a unit section. There is an advantage that the structural section of the target music can be estimated with high accuracy on the musical premise that the structural section transitions at the boundaries of the bars to be played.

  When the acoustic signal X is divided into a plurality of unit sections, the feature extraction unit 22 extracts the basic tone color feature quantity FT0 and the basic chord feature quantity FC0 from the acoustic signal X for each unit section (SA12). The basic timbre feature quantity FT0 is a timbre feature quantity that is the basis of the timbre feature quantity FT, and the basic chord feature quantity FC0 is a chord feature quantity that is the basis of the chord feature quantity FC.

  Specifically, as illustrated in FIG. 3, the feature extraction unit 22 calculates the MFCC of the acoustic signal X for each of a plurality of sections σ obtained by dividing any one unit section on the time axis. A vector in which MFCCs are arranged over a plurality of sections σ within a unit section is generated as a basic timbre feature quantity FT0 of the unit section. The section σ is a section corresponding to, for example, a sixteenth note of the target music. Therefore, the basic timbre feature value FT0 of the first embodiment is I pieces corresponding to the product of the number of dimensions of any one MFCC (for example, 12 dimensions) and the total number of sections σ in the unit section (for example, 16). Is an I-dimensional vector in which the elements are arranged (I is a natural number of 2 or more). That is, the basic timbre feature amount FT0 expresses the temporal transition of the timbre within a unit interval (within one measure). In addition, the feature extraction unit 22 calculates a chroma vector of the acoustic signal X for each section σ in the unit section, and a vector in which the chroma vectors are arranged over a plurality of sections σ in the unit section is a basic chord feature of the unit section. Generated as quantity FC0. That is, the basic chord feature value FC represents a temporal transition (that is, chord progression) of a chord within a unit section.

  By the way, the basic tone color feature value FT0 may contain an unsteady additive noise component that varies with time. The feature extraction unit 22 of the first embodiment suppresses noise components by extracting components (common components) that are commonly contained over a plurality of unit sections close to each other on the time axis (SA13). FIG. 4 is a flowchart of a process SA13 in which the feature extraction unit 22 suppresses a noise component from the basic timbre feature quantity FT0 (hereinafter referred to as “noise suppression process”) SA13, and FIG. 5 is an explanatory diagram of the noise suppression process SA13.

  When the noise suppression process SA13 is started, the feature extraction unit 22 selects one unit section (hereinafter referred to as “target unit section”) from the plurality of unit sections of the acoustic signal X (SA130). Specifically, the feature extraction unit 22 sequentially selects each of the plurality of unit sections as the target unit section from the beginning to the end of the acoustic signal X.

  As illustrated in FIG. 5, J (J is a natural number greater than or equal to 2) unit sections located around one target unit section is hereinafter referred to as “peripheral unit section”. The number J of peripheral unit sections is arbitrary. Further, in FIG. 5, a plurality of unit sections positioned in front and rear of the target unit section are set as peripheral unit sections, but a plurality of unit sections positioned only in one of the front and rear of the target unit section are peripheral. It can also be a unit interval.

  When the target unit section is selected, the feature extraction unit 22 sequentially executes the first process SA131 and the second process SA132 for the target unit section. Specific contents of the first process SA131 and the second process SA132 will be described in detail below. In the following description, as illustrated in FIG. 5, the basic timbre feature quantity FT0 of the target unit section is an I-dimensional vector including I elements fA (1) to fA (I). Assume that the basic timbre feature value FT0 of the j-th (j = 1 to J) peripheral unit section is an I-dimensional vector including I elements fB (1, j) to fB (I, j). To do.

  The first process SA131 is a process of selecting the minimum value for each element of the basic timbre feature quantity FT0 between the target unit section and each of the J peripheral unit sections. Specifically, as illustrated in FIG. 5, in the first process SA131, I elements fC (1, j) to fC (I, j) for each of the J peripheral unit sections around the target unit section. ) Series is generated. The i-th element fC (i, j) generated for the j-th peripheral unit section is the same as the i-th element fA (i) in the basic timbre feature FT0 of the target unit section and the peripheral unit section. Is the minimum value (fC (i, j) = min {fA (i), fB (i, j)}) of the i-th element fB (i, j) in the basic timbre feature quantity FT0. . As understood from the above description, the noise component that appears only in one of the target unit section and the peripheral unit section is removed in the first process SA131. That is, the first process SA131 corresponds to a process of extracting a common component of the basic timbre feature quantity FT0 between the target unit section and the peripheral unit section.

  The second process SA132 calculates the maximum value of the elements fC (i, 1) to fC (i, J) (that is, the minimum value of the elements fA (i) and fB (i, j)) over J peripheral unit sections. This is a process for selecting each of the I elements. Specifically, as illustrated in FIG. 5, in the second process SA132, an I-dimensional vector including I elements fD (1) to fD (I) is generated as the basic timbre feature value FT1. The i-th element fD (i) of the basic timbre feature FT1 is the maximum value (fD (i)) of the i-th elements fC (i, 1) to fC (i, J) over J peripheral unit sections. = Max {fC (i, 1) to fC (i, J)}). As understood from the above description, the second process SA132 is a process for generating the basic timbre feature quantity FT1 by integrating elements fC (i, j) over J peripheral unit sections.

  When the first process SA131 and the second process SA132 are executed for one target unit section, the feature extraction unit 22 completes the generation of the basic timbre feature value FT1 (SA130 to SA132) for all unit sections of the acoustic signal X. Whether or not (SA133). If the determination result is negative (SA133: NO), the feature extraction unit 22 proceeds to the above-described step SA130, and the unit section in which the basic timbre feature quantity FT1 has not been generated (for example, immediately after the current target unit section). The first process SA131 and the second process SA132 are sequentially executed after selecting (unit section) as the target unit section (SA130). When the basic timbre feature value FT1 is generated for all unit sections of the acoustic signal X (SA133: YES), the noise suppression process SA13 of FIG. 4 ends.

  When the basic timbre feature quantity FT1 and the basic chord feature quantity FC0 are generated for each unit section of the acoustic signal X by the processing illustrated above (SA11 to SA13), the feature extraction unit 22 performs the basic timbre feature quantity FT1 and the basic chord feature. Dimensional compression is executed for the quantity FC0 (SA14). Specifically, the feature extraction unit 22 generates a low-dimensional (for example, five-dimensional) timbre feature amount FT by dimensional compression on the basic timbre feature amount FT1, and generates a low-dimensional (for example, 10) dimensional compression on the basic chord feature amount FC0. (Dimensional) chord feature value FC is generated (SA14). For the dimensional compression of the basic tone color feature value FT1 and the basic chord feature value FC0, a known technique such as principal component analysis is arbitrarily employed. The timbre feature value FT and the chord feature value FC are generated for each unit section of the acoustic signal X by the feature extraction processing SA1 (SA11 to SA14) exemplified above.

  In the first embodiment, for each of the plurality of unit sections of the acoustic signal X (target unit section), a first process SA131 for selecting a minimum value for each element of the basic timbre feature quantity FT0 between each peripheral unit section and The basic timbre feature value FT1 is generated by executing the second process SA132 for selecting the maximum value of the minimum values over J peripheral unit sections for each element. According to the above configuration, since the common component between the target unit section and each peripheral unit section is extracted by the first process SA131, the non-stationary noise component that varies with time can be easily reduced and appropriately It is possible to generate a timbre feature quantity FT.

  By the way, only from the viewpoint of suppressing the noise component of the basic timbre feature FT0, the result of executing the first process SA131 between any one target unit section and the immediately preceding or immediately following peripheral unit section (for example, FIG. 5). Are also assumed to be extracted as the timbre feature quantity FT (hereinafter referred to as “proportional”). However, the tone color of the acoustic signal X (basic tone color feature value FT0) can vary greatly before and after the boundary of the structural section. Therefore, in the comparative configuration in which only the first process SA131 is executed, the timbre variation at the boundary of the structural section is suppressed as a noise component, and an appropriate timbre feature quantity FT cannot be generated in the unit sections before and after the boundary. there is a possibility. In the first embodiment, since the basic timbre feature quantity FT1 is generated in the second process SA132 that integrates the elements fC (i, j) over J peripheral unit sections, the unit sections before and after the boundary of the structure section are also generated. There is an advantage that the timbre feature amount FT can be appropriately generated. That is, in the first embodiment, by executing both the first process SA131 and the second process SA132, it is possible to appropriately generate the timbre feature quantity FT before and after the boundary of the structural section while reducing the noise component. There is.

<Estimation processing unit 24>
The estimation processing unit 24 in FIG. 1 estimates the music structure (each structural section) of the target music using the timbre feature quantity FT and chord feature quantity FC extracted by the feature extraction unit 22 for each unit section. Specifically, the estimation processing unit 24 of the first embodiment uses the timbre feature extracted by the feature extraction unit 22 for a probability model that describes the probability that a time series of feature values is observed under a specific music structure. Estimate the probability distribution of the posterior probability when the time series of the amount FT and the chord feature amount FC are observed (hereinafter referred to as “posterior distribution”), and estimate the music structure of the target music that maximizes the posterior distribution (MAP ( maximum a posteriori). As illustrated in FIG. 6, in the music structure of the target music, codes (hereinafter referred to as “section codes”) s serving as labels for identifying each structural section are arranged in time series for each unit section of the acoustic signal X. It is expressed by a structure estimation sequence Y. In FIG. 6, the section code s is represented by one alphabet (A, B, C,...). As illustrated in FIG. 6, a common section code s is added to a plurality of unit sections included in any one structure section, and a plurality of unit sections included in different structure sections are separately provided. An interval code s is added. That is, the time when the content of the section code s changes is a candidate for the boundary of each structural section of the target music.

  The music structure model MS, the timbre observation model MTk, and the chord observation model MCk are used for the process in which the estimation processing unit 24 of the first embodiment estimates the music structure of the target music (hereinafter referred to as “estimation process”). The music structure model MS is a probability model in which the music structure is described stochastically. The timbre observation model MTk is a probabilistic model that stochastically describes the generation process of the timbre feature quantity FT, and the chord observation model MCk is a probabilistic model that probabilistically describes the generation process of the chord feature quantity FC. The estimation processing unit 24 uses the timbre feature quantity FT and the chord feature quantity FC extracted for each unit section by the feature extraction unit 22 for each of the music structure model MS, the timbre observation model MTk, and the chord observation model MCk. To estimate the posterior distribution. A time series of section codes s that maximizes the posterior distribution of the music structure model MS is estimated as the structure estimation series Y. Specifically, the structure estimation sequence Y is generated so that the section code s changes when it is estimated that the structure section of the target music transitions.

  FIG. 7 is an explanatory diagram of the music structure model MS. The music structure model MS of the first embodiment is a state transition model (specifically, a hidden Markov model) in which a plurality of states linked to each other are arranged in a state space. Specifically, as illustrated in FIG. 7, a plurality of state sequences (for example, one column in FIG. 7) corresponding to different unit sections in any one structural section are parallel in the plurality of structural sections. Placed in. An arbitrary state expressed by the music structure model MS includes a section code s (s = A, B, C,...) Of each structure section, and a length of time spent in the state in the structure section (hereinafter referred to as the structure code). It is specified in combination with u). The residence time u is expressed by the number of unit sections (number of bars) from the beginning in the structure section. That is, the arbitrary state (s, u) stays in the structural section from the start point of the structural section to the uth unit section from the starting point of the structural section specified by the section code s. Meaning).

  The music structure model MS of the first embodiment can transition from the state at the end of each structure section to the state at the beginning of another structure section. Specifically, in the music structure model MS of the first embodiment, when the residence time u is less than the threshold D, the state transitions to the state immediately after the current construction interval (that is, the residence time u is one unit interval). When the dwell time u reaches the threshold value D, the state transitions to the head state (u = 1) of another structural section with the transition probability τ. The threshold value D is set to the number of unit sections that are highly likely to cause structural section transitions. For example, in a normal music piece, there are many cases where a transition of a structural section occurs in units of 4 bars or 8 bars. Considering the above tendency, a configuration in which the threshold value D is set to 4 or 8, for example, is suitable. The transition probability τ is a probability of transition from the last state of an arbitrary structural section to the first state of another structural section, and is set for any one combination of structural sections. The prior distribution of the transition probability τ is preferably a Dirichlet distribution.

数式(1)の記号Normal(μ_sk,η ^(T),Λ_sk,η ^-1,(T))は、平均μ_sk,η ^(T)および精度行列（共分散行列の逆行列）Λ_sk,η ^-1,(T)で規定される正規分布を意味する。平均μ_sk,η ^(T)および精度行列Λ_sk,η ^-1,(T)の事前分布には、例えば正規-ウィシャート（Normal-Wishart）分布が好適に採用される。また、数式(1)の記号ω_ηは、第η番目の正規分布Normal(μ_sk,η ^(T),Λ_sk,η ^-1,(T))の加重値である。加重値ω_ηの事前分布には例えばＧＥＭ（Griffith-Engen-McCloskey）分布が好適に採用される。数式(1)から理解される通り、第１実施形態では、確率分布の混合数が無限である無限混合分布（具体的には確率分布に正規分布を採用した無限混合ガウス分布）を音色観測モデルＭTkとして利用する。すなわち、音色観測モデルＭTkは、加重値ω_ηを適用した無限個の正規分布Normal(μ_sk,η ^(T),Λ_sk,η ^-1,(T))の加重和として表現される。 The state of the kth unit section among the plurality of unit sections of the target section is expressed as (sk, uk). In the first embodiment, a probability model expressed by the following equation (1) is adopted as a timbre observation model MTk that probabilistically represents the occurrence of the timbre feature quantity FT in the kth unit section of the target music.

The symbol Normal (μ _{sk, η} ^(T) , Λ _{sk, η} ^{-1, (T)} ) in equation (1) is the mean μ _{sk, η} ^(T) and accuracy matrix (inverse matrix of covariance matrix) Λ _{sk , η-} ^{1, (T)} means a normal distribution. As the prior distribution of the mean μ _{sk, η} ^(T) and the accuracy matrix Λ _{sk, η} ^{-1, (T)} , for example, a Normal-Wishart distribution is preferably employed. Further, the symbol omega _eta equation (1), the eta-th normal distribution ^{_{Normal (μ sk, η (T}} ), Λ sk, η -1, (T)) is a weighted value of. The prior distribution of weight omega _eta for example GEM (Griffith-Engen-McCloskey) distribution is preferably employed. As understood from Equation (1), in the first embodiment, an infinite mixture distribution with an infinite number of probability distributions (specifically, an infinite mixture Gaussian distribution in which a normal distribution is adopted as the probability distribution) is used as a timbre observation model. Use as MTk. That is, the timbre observation model MTk is expressed as a weighted sum of an infinite number of normal distributions Normal (μ _{sk, η} ^(T) , Λ _{sk, η} ^{-1, (T)} ) to which the weight value ω _η is applied.

  The timbre of the music is different between the different structural sections, but is roughly unified within any one structural section (for example, the performance part and musical impression of the music change for each structural section. Tend to). Considering the above tendency, the timbre observation model MTk of the first embodiment is a probability model that expresses the generation process of the timbre feature quantity FT for each structural section, as understood from Equation (1). It does not depend on the residence time u in the section. That is, the timbre observation model MTk is common over a plurality of states corresponding to one arbitrary structural section.

数式(2)から理解される通り、第１実施形態では、平均μ_sk,uk ^(C)および精度行列Λ_sk,,uk ^-1,(C)で規定される正規分布Normal(μ_sk,uk ^(C),Λ_sk,,uk ^-1,(C)）が和音観測モデルＭCkとして利用される。音色観測モデルＭTkに適用される平均μ_sk,η ^(T)および精度行列Λ_sk,η ^-1,(T)と同様に、数式(2)の平均μ_sk,uk ^(C)および精度行列Λ_sk,,uk ^-1,(C)の事前分布には、例えば正規-ウィシャート分布が好適に採用される。 In the first embodiment, the probability model expressed by the following formula (2) is adopted as the chord observation model MCk that stochastically represents the occurrence of the chord feature value FC in the k-th unit section of the target song. To do.

As understood from the equation (2), in the first embodiment, the normal distribution Normal (μ _{sk, uk} defined by the mean μ _{sk, uk} ^(C) and the accuracy matrix Λ _{sk ,, uk} ^{-1, (C)} is used. ^(C) , Λ _{sk, uk} ^{-1, (C)} ) are used as the chord observation model MCk. Similar to the average μ _{sk, η} ^(T) and accuracy matrix Λ _{sk, η} ^{-1, (T)} applied to the timbre observation model MTk, the average μ _{sk, uk} ^(C) and the accuracy matrix Λ in equation (2) For example, a normal-Wishart distribution is suitably used as the prior distribution of _{sk ,, uk-} ^{1, (C)} .

  As described above, the timbre of the music tends to be roughly unified within one structural section, whereas the chord of the music is not only different between the different structural sections, There is a tendency that it fluctuates every unit section (every bar). Considering the above tendency, the chord observation model MCk of the first embodiment is chorded for each combination of the structural section and the unit section (for each state (sk, uk)) as understood from the equation (2). This is a probabilistic model expressing the generation process of the feature value FC, and depends on both the section code sk and the residence time uk.

In addition, the prior distribution of the probability model demonstrated above is not limited to the above-mentioned illustration. For example, the prior distribution of the transition probabilities τ configuration and that the Logistic Normal distribution, configuration of applying the Pitman-Yor process prior distribution of the weights omega _eta tone observation model MTk also be employed. The total number of structural sections in the music structure model MS is also arbitrary, and can be extended to a probability model that assumes, for example, an infinite number of states, such as a nonparametric Bayesian HMM (Hidden Markov Model).

  The estimation processing unit 24 of the first embodiment uses the timbre feature amount FT and chord feature extracted by the feature extraction unit 22 for each of the probability models (music structure model MS, timbre observation model MTk, chord observation model MCk) described above. The posterior distribution when the time series of the quantity FC is observed is estimated by an iterative estimation algorithm such as the variational Bayes method, and the structure estimation series Y that maximizes the posterior distribution is specified. The estimation process described above is repeated N times using the time series of the timbre feature value FT and the chord feature value FC extracted by the feature extraction unit 22.

The initial value extracted (sampled) from each prior distribution in the estimation process is different for each estimation process. For example, normal - Wishart average tone observation model MTk extracted distributed as prior distribution _{μ sk,} η ^(T) and precision matrix ^{_{Λ sk, η -1, (T}} ) and, likewise normal - Wishart distribution priors The mean μ _{sk, uk} ^(C) and the accuracy matrix Λ _{sk ,, uk} ^{−1, (C)} of the chord observation model MCk output as can be different for each estimation process (that is, for each trial). Therefore, N structural estimation sequences Y that are different from each other are generated at the stage where the estimation process has been repeated N times.

As described above, in the first embodiment, the posterior distributions of the timbre observation model MTk for each structural section and the chord observation model MCk for each unit section in each structural section are estimated by the estimation process. That is, the timbre observation model MTk reflecting the tendency that the timbre is roughly unified within one structure section of the target music, and the tendency that the chords sequentially transition for each unit section in the structure section are reflected. The chord observation model MCk thus used is used for the estimation process. Therefore, it is possible to specify the structural section of the target music with high accuracy compared to a configuration that does not take into account the difference between the timbre transition tendency and the chord transition tendency in the music. Further, in the first embodiment, since the infinite mixture distribution exemplified in Equation (1) is used as the timbre observation model MTk, the timbre characteristics of the acoustic signal X (specifically, the timbre complexity of the target music) are used. Accordingly, the number of mixtures of the normal distribution Normal (μ _{sk, η} ^(T) , Λ _{sk, η} ^{-1, (T)} ) varies. Therefore, there is an advantage that the posterior distribution of the timbre observation model MTk can be appropriately estimated according to the characteristics of the acoustic signal X.

  In the first embodiment, a sequence of a plurality of states corresponding to a unit section is included for each of a plurality of structural sections, and transition from the last state of each structural section to the leading state of another structural section is possible A state transition model is applied to the estimation process as the music structure model MS. That is, the tendency that the structure section composed of a plurality of unit sections transitions sequentially in the target music is appropriately expressed in the music structure model MS. Therefore, it is possible to estimate the structure section of the target music with high accuracy.

<Structural analysis unit 26>
The structure analysis unit 26 in FIG. 1 specifies a plurality of structural sections of the target song from the result of the estimation process by the estimation processing unit 24. FIG. 8 is an explanatory diagram of a process in which the structure analysis unit 26 specifies a structure section. In the first embodiment, as illustrated in FIG. 8, sections are obtained by N estimation processes in which initial values of variables of each probability model (music structure model MS, timbre observation model MTk, chord observation model MCk) are different. N (N = 5 in FIG. 8) structure estimation sequences Y having different arrangements of codes s are generated. The structure analysis unit 26 specifies the structure section of the music from the N structure estimation sequences Y generated by the estimation processing unit 24.

  In FIG. 8, the time points (hereinafter referred to as “candidate boundaries”) when the section code s changes in each structure estimation series Y are illustrated by broken lines. Each candidate boundary is a point that is a candidate for the boundary of the structure section of the target music piece. As understood from FIG. 8, the position of each candidate boundary may be different among the N structure estimation sequences Y due to the difference in the initial value of the probability model. The candidate boundary tends to be common. The greater the number of structure estimation sequences Y with common candidate boundaries, the higher the probability that the candidate boundary is the actual boundary of the structure section of the target song. Considering the above tendency, as illustrated in FIG. 8, the structure analysis unit 26 of the first embodiment selects a candidate boundary that is common over the number of structure estimation sequences Y that exceeds a predetermined threshold NTH as an estimation boundary. Then, a structure estimation sequence Z representing the music structure of the target music is generated with each estimated boundary as the boundary of the structure section. The estimated boundary is a candidate boundary having a high probability of being the actual boundary of the structure section of the target music piece. The threshold value NTH is set to a predetermined positive number (for example, N / 2) lower than the total number N of the structure estimation series Y.

  Similar to the structure estimation sequence Y, the structure estimation sequence Z is a time series of a plurality of section codes (labels) sZ corresponding to each unit section of the target music piece. As illustrated in FIG. 8, a section code sZ is set for each unit section in each structure section partitioned by the estimated boundary. Specifically, the structure analysis unit 26 represents a representative value (for example, a median value) of the section code s over the N structure estimation sequences Y for a plurality of unit sections in one structure section partitioned by the estimation boundary. A common section code sZ is set for each unit section in each structure section having the same representative value in the structure estimation series Z. Each structural section having a common representative value of the section code s is a section in which timbre and chord transitions are similar or common to each other (that is, a section in which musical meanings such as “A melody” and “rust” are common). It is estimated to be.

  The structure analysis unit 26 of the first embodiment causes the display device 16 to display the structure estimation sequence Z generated by the above processing. Specifically, as illustrated in FIG. 9, the structure analysis unit 26 causes the display device 16 to display an estimation result image 42 representing the structure estimation sequence Z superimposed on the signal waveform 44 of the acoustic signal X. The estimation result image 42 is a rectangular figure extending in the direction of the time axis, and is divided into a plurality of regions 46 at the position of the estimation boundary designated by the structure estimation sequence Z. That is, each area 46 of the estimation result image 42 represents a structural section of the target music. Of the plurality of regions 46 of the estimation result image 42, each region 46 corresponding to a separate section code sZ in the structure estimation series Z is displayed in a different display mode (color or gradation), and the section code sZ is common. Are displayed in a common display mode. Therefore, the user can grasp the temporal relationship between each structural section of the target music and the acoustic signal X by visually recognizing the estimation result image 42.

  As described above, in the first embodiment, each structural section (structure estimation sequence Z) of the target music is specified from N structure estimation sequences Y generated by estimation processing using different initial values. Therefore, there is an advantage that the music structure of the target music can be estimated with high accuracy robustly against fluctuations in the initial value used for each estimation process.

  FIG. 10 is a flowchart of the overall operation of the music analysis device 100 according to the first embodiment. For example, when the process of FIG. 10 is started in response to an instruction from the user, the feature extraction unit 22 performs the feature extraction process SA1 described with reference to FIGS. 2 and 4 on the acoustic signal X. The timbre feature value FT and the chord feature value FC are extracted for each unit section. The estimation processing unit 24 estimates the structure estimation sequence Y indicating the music structure of the target music in the estimation process using the timbre feature value FT and the chord feature value FC extracted for each unit section by the feature extraction unit 22 (SA2). . The estimation processing by the estimation processing unit 24 is repeated N times (SA3: NO). When the generation of N structure estimation sequences Y is completed by repeating the estimation process (SA3: YES), the structure analysis unit 26 compares the N structure estimation sequences Y with each other to perform a plurality of estimations on the time axis. A boundary is specified, and a structure estimation sequence Z representing the music structure of the target music is generated using the estimated boundary as the boundary of each structural section (SA4). Further, the structure analysis unit 26 causes the display device 16 to display the estimation result image 42 representing the structure estimation series Z (SA5).

Second Embodiment
A second embodiment of the present invention will be described. Regarding the elements whose functions and functions are the same as those of the first embodiment in each aspect exemplified below, the detailed description of each is appropriately omitted by using the reference numerals used in the description of the first embodiment.

  FIG. 11 is an explanatory diagram of the operation of the structure analysis unit 26 in the second embodiment. FIG. 11 illustrates the structure estimation sequence Z identified by the structure analysis unit 26. In FIG. 11, a plurality of unit sections having a common section code sZ are represented by one section. That is, one section shown in FIG. 11 corresponds to one structure section of the target music piece.

  As understood from FIG. 11, in the structure estimation sequence Z, a specific arrangement of a plurality of section codes sZ may be observed a plurality of times. For example, in the example of the first row in FIG. 11, the sequence “A-B-A-B” and the sequence “C-D-C-D” are repeated a plurality of times in the target music. The structure analysis unit 26 according to the second embodiment integrates a plurality of structure sections corresponding to the arrangement of section codes sZ repeated a plurality of times in the target music piece into one structure section. Specifically, in the structure estimation sequence Z of FIG. 11, the section code sZ is set to “G” in the four structure sections corresponding to the array of section codes sZ “A-B-A-B”. The four structural sections integrated into one structural section and corresponding to the array of section codes sZ “C-D-C-D” are one structural section in which the section code sZ is set to “H”. Integrated into. By cumulatively executing the integration of the plurality of structural sections described above, the structure analysis unit 26 of the second embodiment generates a plurality of hierarchical structure estimation sequences Z having different section lengths and total numbers of the structural sections. Generate. For integration of a plurality of structural sections, statistical processing such as 2-gram is preferably used. The structure analysis unit 26 selects any one of the plurality of structure estimation sequences Z generated by the above procedure as a definitive analysis result, and causes the display device 16 to display an estimation result image 42 representing the structure estimation sequence Z. In addition, it is also possible to display the estimation result image 42 corresponding to each of the plurality of structure estimation sequences Z on the display device 16 in parallel, and allow the user to select a desired estimation result image 42. In the second embodiment, the same effect as in the first embodiment is realized.

<Modification>
Each aspect illustrated above can be variously modified. Specific modifications are exemplified below. Two or more modes arbitrarily selected from the following examples can be appropriately combined within a range that does not contradict each other.

(1) The dimension compression (SA14) for the basic tone color feature value FT1 and the basic chord feature value FC0 can be omitted. That is, a configuration in which the basic timbre feature amount FT1 in the first embodiment is used in the estimation process as the timbre feature amount FT, and a configuration in which the basic chord feature amount FC0 is used as the chord feature amount FC in the estimation process may be employed. It is also possible to omit the noise suppression process SA13 and use the basic timbre feature quantity FT0 as the timbre feature quantity FT for the estimation process.

(2) In each of the above-described forms, the music structure model MS that transitions to another structural section with the transition probability τ when the residence time u reaches the threshold D is exemplified, but the probability π of the continuation length of the structural section is specified. It is also possible to use the hidden semi-Markov model (HSMM: hidden semi-Markov model) as the music structure model MS. For example, it is a numerical value with a large probability of π with a continuation length (for example, 4 bars or 8 bars) in which the transition of the structural section is likely to occur. A Dirichlet distribution is suitable as the prior distribution of the probability π.

(3) In each of the above forms,
Configuration 1: A configuration in which a timbre observation model MTk for each structural section and a chord observation model MCk for each unit section in each structural section are used for estimation processing;
Configuration 2: A configuration in which a feature amount is extracted for each unit section using a measure of the target music as a unit section;
Configuration 3: Configuration for generating timbre feature value FT by noise suppression processing SA13 including first processing SA131 and second processing SA132;
Configuration 4: Configuration using infinite mixture distribution as timbre observation model MTk;
Configuration 5: A series of a plurality of states (s, u) corresponding to each unit section in a structure section is included for a plurality of structure sections, and from the last state of each structure section to the first state of another structure section A configuration using a state transition model (music structure model MS) capable of transition for estimation processing;
Configuration 6: The configuration of generating the structure estimation sequence Z from the N structure estimation sequences Y generated by the estimation process with different initial values is exemplified. Each can be established independently of each other. That is, the other configurations are not essential for each of the configurations 1 to 6. For example, in the configuration in which the configuration 6 is omitted, the estimation processing unit 24 executes the estimation processing only once, and the structure estimation sequence Y generated by the estimation processing is determined as the structure estimation sequence Z.

(4) The music analysis device 100 can be realized by a server device that communicates with a terminal device (for example, a mobile phone or a smartphone) via a communication network such as a mobile communication network or the Internet. Specifically, the music analysis device 100 generates a structure estimation sequence Z by analyzing the acoustic signal X received from the terminal device via the communication network, and transmits the structure estimation sequence Z to the terminal device.

(5) The music analysis device 100 exemplified in each of the above-described embodiments is realized by the cooperation of the arithmetic processing device 12 and a program, as illustrated in each of the above-described embodiments. The program according to a preferred aspect of the present invention includes, for each unit section of the acoustic signal X representing the sound of the target musical piece, a timbre feature amount FT representing the sound timbre feature and a chord feature amount FC representing the chord feature of the sound. A timbre observation model MTk that probabilistically represents the generation process of the timbre feature quantity FT for each structural section including at least one unit section, which is a musical structure division in the target music The timbre feature quantity FT and the chord feature quantity FC extracted by the feature extraction unit 22 are applied to each of the chord observation models MCk that probabilistically express the generation process of the chord feature quantity FC for each unit section in each structural section. The computer is caused to function as an estimation processing unit 24 that estimates the posterior distribution by the estimation process and a structure analysis unit 26 that specifies a plurality of structural sections of the target music from the result of the estimation process. The programs exemplified above can be provided in a form stored in a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, and an optical recording medium (optical disk) such as a CD-ROM is a good example, but a known arbitrary one such as a semiconductor recording medium or a magnetic recording medium This type of recording medium can be included. It is also possible to distribute the program to a computer in the form of distribution via a communication network.

(6) The present invention is also specified as an operation method (music analysis method) of the music analysis device 100 according to each of the above-described embodiments. For example, in the music analysis method according to one aspect of the present invention, the computer system (a single computer or a system constituted by a plurality of computers) that realizes the music analysis apparatus 100 is a unit of the acoustic signal X that represents the sound of the target music. For each section, a timbre feature quantity FT representing the characteristics of the acoustic timbre and a chord feature quantity FC representing the characteristics of the acoustic chord are extracted (feature extraction process SA1), and the musical structure in the target music is classified. There is a timbre observation model MTk that stochastically represents the generation process of the timbre feature quantity FT for each structural section including at least one unit section, and the generation process of the chord feature quantity FC is probabilistic for each unit section in each structural section. For each of the chord observation models MCk expressed in (2), the posterior distribution is estimated by the estimation process using the timbre feature quantity FT and the chord feature quantity FC (estimation process SA2). Identifying a plurality of structural sections elephant song (music structure analysis SA4).

DESCRIPTION OF SYMBOLS 100 ... Music analysis apparatus, 12 ... Operation processing apparatus, 14 ... Memory | storage device, 16 ... Display apparatus, 22 ... Feature extraction part, 24 ... Estimation processing part, 26 ... Structural analysis part.

Claims (6)

A feature extraction unit that extracts a timbre feature amount representing a timbre feature of the sound and a chord feature amount representing a chord feature of the sound for each unit section of the sound signal representing the sound of the music;
A timbre observation model that stochastically represents the generation process of the timbre feature amount for each structural section that is a musical structural division in the music and includes at least one unit section, and for each unit section in each structural section An estimation processing unit that estimates a posteriori distribution by an estimation process that applies the timbre feature amount and the chord feature amount extracted by the feature extraction unit for each of the chord observation models that stochastically express the generation process of the chord feature amount;
A music analysis device comprising: a structure analysis unit that identifies a plurality of structural sections of the music from the result of the estimation process. The feature extraction unit extracts a basic timbre feature amount including a plurality of elements according to the acoustic timbre from the acoustic signal for each unit section, and a plurality of unit sections before and after the unit section for each of the plurality of unit sections. A first process for selecting a minimum value for each element of the basic timbre feature quantity between each of the peripheral unit sections, and selecting a maximum value of the minimum values over the plurality of peripheral unit sections for each element. The music analysis device according to claim 1, wherein the timbre feature amount is generated by executing a second process. The music estimation device according to claim 1, wherein the estimation processing unit executes the estimation processing using an infinite mixture distribution in which the number of mixing probability distributions is infinite as the timbre observation model. The estimation processing unit includes a plurality of state series corresponding to each unit section in the structure section for a plurality of structure sections, and transition from the last state of each structure section to the first state of another structure section The music analysis device according to any one of claims 1 to 3, wherein a state transition model capable of being used is used for the estimation process. The estimation processing unit repeats the estimation process multiple times with different initial values, thereby identifying a structure estimation sequence in which identification codes indicating structure sections are arranged for each unit section of the music for each estimation process. And
The music analysis device according to any one of claims 1 to 4, wherein the structure analysis unit specifies a structure section of the music from a plurality of structure estimation sequences specified by the estimation process performed a plurality of times. Computer system
For each unit section of the acoustic signal representing the sound of the music, extract a timbre feature amount representing the timbre feature of the sound and a chord feature amount representing a chord feature of the sound,
A timbre observation model that stochastically represents the generation process of the timbre feature amount for each structural section that is a musical structural division in the music and includes at least one unit section, and for each unit section in each structural section For each of the chord observation models that probabilistically represent the chord feature generation process, the posterior distribution is estimated by the estimation process using the extracted timbre feature amount and the chord feature amount,
A music analysis method for identifying a plurality of structural sections of the music from the result of the estimation process.

Images