TWI768589B - Deep learning rhythm practice system - Google Patents

Abstract

A deep learning rhythm practice system mainly includes a processing device and a comparison device. Through music played by the processing device, practice of finding beats can be executed. Meanwhile, hand-clapping sounds generated during the practice will be recorded with the music and then transformed into an audio signal. The audio signal is transmitted to the comparison device by the processing device. The comparison device in which a deep learning algorithm is stored then calculates the audio signal via the deep learning algorithm to thereby effectively and automatically detect and determine whether the hand-clapping sounds are included in the audio signal, and further determine whether the hand-clapping sounds are tapped on the beats correctly. Hence, the accuracy of rhythm practice and comparison is increased effectively.

Description

Deep learning rhythm practice system

The present invention relates to a rhythm practice system, in particular to a rhythm practice system for deep learning.

In the process of learning music, it is very important to have a correct sense of rhythm. The rhythm practice method in the beginner period is usually to beat the beat with the sound of the metronome. After the practice has a certain foundation, you can Play music, listen to the music and beat the beat with the beat of the music, and the most convenient way to beat the beat is to clap your hands, while in traditional rhythm practice, you have to rely on the teacher to guide the students, follow the music and use the sound of clapping to beat the beat on the rhythm. On the correct beat, if there is no teacher's guidance on the scene, students will not be able to effectively determine their rhythm practice with the music to beat the beat, and whether the beat is on the correct beat, because the combination of clapping sounds In the sound of music, it is not easy to correctly find the clapping point in the sound of music, and it is not easy to correctly detect the sound of clapping in the sound of music for the recorded audio after practice, resulting in some judgments. There are errors and omissions, so there is a real need for improvement in rhythm practice and comparison.

Therefore, the purpose of the present invention is to provide a deep learning rhythm practice system, which can perform operations on the recorded audio signals through the algorithm of audio recognition and in-depth practice, so as to effectively detect and compare whether the clapping sound is in the correct position. On the beat rhythm, the accuracy of rhythm practice is greatly improved.

Therefore, a deep learning rhythm practice system of the present invention includes a processing device and a detection and comparison device; wherein, the detection and comparison device includes a storage module and a deep learning module connected to the storage module. , and the aforementioned storage module at least stores a training data set, a verification data set and a music set with beats, so that the music set with beats provided by the storage module of the detection and comparison device is used for The processing device is connected and downloaded to perform clapping rhythm practice, and the processing device further records and converts the aforementioned clapping rhythm practice into audio signals and outputs them. The audio signal is calculated to further automatically detect and compare the input audio signal to determine whether there is a clapping sound, and at the same time, it can better compare whether the clapping sound is on the correct beat rhythm, which greatly improves the accuracy of rhythm practice comparison. .

FIG. 1 is a schematic diagram of a preferred embodiment of the present invention.

FIG. 2 is a schematic diagram of an audio signal (the red arrow indicates the position where the clapping sound appears) with music sound + clapping sound with a length of 10 seconds at the same time in the preferred embodiment of the present invention.

3 is a schematic diagram of an audio signal with a length of 0.1 seconds marked as no clapping sound according to the preferred embodiment of the present invention.

FIG. 4 is a schematic diagram of an audio signal with a length of 0.1 seconds marked as having a clapping sound according to the preferred embodiment of the present invention.

FIG. 5 is a schematic diagram of extracting audio features by deep learning according to the preferred embodiment of the present invention.

FIG. 6 is a schematic block diagram of a flow chart of the preferred embodiment of the present invention.

The foregoing and other technical contents, features and effects of the present invention will be clearly understood in the following detailed description of the preferred embodiments with reference to the drawings.

Referring to FIG. 1, a preferred embodiment of the present invention, the rhythm training of the deep learning The learning system includes a processing device and a detection and comparison device connected with the processing device; wherein, the processing device is an application program that can be used on mobile devices with data communication such as smart phones and tablet computers, and at the same time The detection and comparison device is set in a terminal, and can be used in connection with the processing device.

Continuing the above, the detection and comparison device includes a storage module and a deep learning module connected to the storage module, wherein the storage module can be a flash memory, or other storage media, etc., and The storage module stores at least a training data set, a verification data set and a music set with beat points, and the training data set is recorded with at least 10 kinds of music sounds of clapping by a plurality of different recorders, And each recorder records 10 audio signals, corresponding to 10 kinds of music sounds, each audio signal is 60 seconds, and at the same time, in the audio signal with multiple clapping sounds, the audio signals without clapping sound are processed for 0.1 second. Mark, and record and store a plurality of audio signals with clapping sound for 0.1 seconds to provide for use as a training model, and in the verification data set, a number of different bits that are different from the training data set are recorded. The recorder's clapping sound in at least 10 kinds of music sounds, and each recorder records 10 audio signals corresponding to 10 kinds of music sounds, each audio signal is 60 seconds, and in the audio signal with multiple clapping sounds, In addition, a plurality of audio signals without clapping sound are marked for 0.1 second, and a plurality of audio signals with clapping sound are marked for 0.1 second, which is used to verify the correctness of the model.

Furthermore, for the sampling rate of the audio signal stored in the training data set and the verification data set, 16KHz is taken as an example in this embodiment, and the method is to pass the recorded audio signal through a 4th-order Butterworthe high-pass filter, The 3dB pass frequency is 0.5Hz, which is used to filter out the DC offset, and then the average normalization method is used to scale the filtered audio signal to the [-1,1] interval, and the formula is as follows:

Where X _max and X _min are the minimum and maximum values in the data, respectively; μ is the average value of the data, then the data will be scaled to the [-1,1] interval and the average value is =0

Continuing the above, after the aforementioned operations, the normalized audio signals can be sequentially cut into audio signals with a length of 0.1 seconds, and each 0.1 second audio signal is marked as having a clapping sound or no audio signal by manual discrimination. The clapping sound is as shown in Figures 2 to 4. As for the music set with beat points, a plurality of music data with beat points corresponding to the audio signals of the training data set and the verification data set are stored and recorded, so as to It is used for connection and download with the processing device.

Continuing from the above, as for the audio signal input by the deep learning module to the processing device, the technology of the deep learning module is used to improve the accuracy of recognition. Please refer to FIG. 5 again. In the deep learning module There are at least five feature extraction layers, one flat layer, two classification layers and one fully connected layer, and the aforementioned deep learning module takes the input signal as a 0.1-second audio signal and the sampling frequency is 16K, so the length of the input signal is 1600 sampling points, and each feature extraction layer further includes a convolutional neural network layer, a batch normalization layer, an excitation layer, a maximum pooling layer and a discarding layer, and the convolutional neural network The layer has thirty-two feature maps, and the convolution kernel length of the convolutional neural network layer is 16, and the pooling length of the maximum pooling layer is designed to be 2, so after the five feature extraction layers are The flat layer is used to convert a two-dimensional feature matrix into a one-dimensional feature vector to provide the following classification layers as input signals, and each classification layer further includes a fully connected neural network layer , a batch of regularization layers, an excitation layer and a rejection layer, and the fully connected neural network layer has one hundred and twenty-eight neurons, and after the two classification layers is the fully connected layer, which has Two neurons are used to calculate the probability corresponding to the two output signals of the fully connected layer, and the two output signals correspond to no clapping sound and clapping sound respectively, and the classification result is is a category corresponding to a greater probability, so the function of the batch normalization layer after the convolutional neural network layer and the fully connected neural network layer in the feature extraction layer and the classification layer is to normalize the signal. , so that when the normalized signal enters the excitation layer, the speed, performance and stability of the neural network can be improved, and the purpose of setting the maximum pooling layer in the feature extraction layer is to reduce the speed of the network. the complexity and the possibility of overfitting, and the max pooling layer with a pooling length of 2 can reduce the number of each feature map by half, and then through the cooperation of the discarding layer can be used to Reduce the effect of overfitting, thereby forming an optimized deep learning module; at the same time, the deep learning module uses all the 0.1-second audio signals in the training data set and the corresponding marks with or without clapping, and match the steepest The descent method is used to train the model to find the optimal model parameters, and then use all the 0.1-second audio signals in the verification data set and the corresponding marks with or without clapping to obtain the classification results and matching probability, and calculate the verification results. Correctness, the verification accuracy of the present invention can reach more than 95%.

Finally, the processing device includes a playback module, an input module, and a transceiver module respectively connected to the playback module and the input module, and the playback module can load the playback module through the transceiver module. The music set with beat points in the storage module is played, and the input module has a recording function, which can record the music and clapping during playback, and convert the recording file into an audio signal. As for the transceiver module The audio signal can be output and connected to the detection and comparison device for training and verification of the audio signal.

Referring to FIG. 1 to FIG. 6 , when the user wants to practice clapping rhythm, he can turn on the mobile device to click on the playback module, so that the playback module is connected to the storage module through the transceiver module, so as to Select the music stored in the music collection with beats and load it to play, and the user can beat the beat at the rhythm point of the desired clapping sound when the playback module is playing the music. Practice the sense of rhythm, so after practicing several times with a variety of different music, if the user wants to know clearly whether the rhythm of his clapping is in the correct position When the tap is clicked, the next time you click on the playback module to load the music for playback, you can also click to activate the input device for synchronous recording, and then the input device can play correspondingly to the current user. The clapping of music and the playing music are recorded every 0.1 seconds, and the recorded recording file is converted into an audio file for output to the transceiver module, and the transceiver module uses the audio file. The output is connected to the detection and comparison device. At this time, the detection and comparison device can detect and identify whether the audio signal of every 0.1 second in the audio file has a clap sound by optimizing the deep learning model. , if there is a clapping sound, then compare the time of the 0.1-second audio signal to see if it is on the beat of the music rhythm. If so, it is judged as a clapping sound at the correct time. If not, it is judged as a wrong time. The sound of clapping hands, and so on, at the same time, the detection and comparison device will also connect the verification and comparison result to the processing device, and receive it by the transceiver module, so that the processing device outputs the result status for display For users to understand, so in use, it is not limited by the venue, you can practice rhythm training at any time, and compare and verify the clapping sound of rhythm beats in real time, so the clapping sound of music beats can be compared. It is quite efficient and greatly improves the accuracy of rhythm practice.

To sum up the above, the deep learning rhythm practice system of the present invention can provide a music collection with beat points stored in the storage module of the detection and comparison device, which can be downloaded and played by the processing device through connection. , so that the user can practice the clapping sound of rhythm and beats with the music set with beat points, and at the same time, the processing device can simultaneously record the clapping sound and the music and convert it into an audio signal, so as to be connected to the The detection and comparison device uses a deep-practice algorithm to perform operations on the recorded audio signals, and performs detection and comparison with the stored data in the storage module, which can not only effectively and automatically determine whether the input audio signals appear or not. The clapping sound can further verify and compare whether the clapping sound is in the correct rhythm, so in use, it can be used for rhythm training at any time without being restricted by the venue. It can effectively improve the accuracy of verification rhythm practice comparison.

However, the above descriptions are only to illustrate the preferred embodiments of the present invention, and should not limit the scope of implementation of the present invention. , shall still fall within the scope covered by the patent of the present invention.

Claims (9)

A rhythm practice system for deep learning, which includes a processing device, and a detection and comparison device connected with the processing device; wherein, the processing device includes a playback module, an input module, and a playback module respectively. group, the transceiver module connected to the input module, and the playback module can be connected to the detection and comparison device through the transceiver module to load music with beats for playback, and the input module can The clapping sound and music are recorded, and the recording file is converted into an audio signal and output to the transceiver module, so that the transceiver module can output the audio signal; in addition, the detection and comparison device includes a storage module , and a deep learning module connected to the storage module, and the aforementioned storage module stores at least a training data set, a verification data set and a music set with beat points, and the deep learning module has at least five Feature extraction layer, a flat layer, two classification layers and a fully connected layer, these feature extraction feature layers can extract a plurality of feature maps from the audio signal, and the flat layer converts the two-dimensional feature matrix into one-dimensional features vector, the fully connected layer then calculates the corresponding probability, and generates a classification result corresponding to a category with a higher probability, so that the deep learning module uses the training data set and cooperates with the steepest descent method for model training to find the best Then use the verification data set to obtain the classification results and matching probability, and calculate the correctness of verification. The deep learning rhythm practice system according to claim 1, wherein the training data set stores audio signals recorded with clapping sounds in a plurality of music sounds, and each audio signal is 60 seconds and is sequentially cut into lengths 0.1 second of audio signal, and by manual discrimination, each audio signal without clapping sound is marked for 0.1 second, and each audio signal with clapping sound is marked for 0.1 second to provide as a training model use. The deep learning rhythm practice system according to claim 1, wherein the verification data set is stored and recorded with a plurality of music sounds different from the training data set. The audio signal of the clapping sound, and each segment of the audio signal is 60 seconds, is sequentially cut into audio signals with a length of 0.1 seconds, and each of the audio signals without the clapping sound is marked for 0.1 seconds by manual discrimination. Each audio signal with a clapping sound is marked for 0.1 seconds to verify the correctness of the model. The deep learning rhythm practice system according to claim 1, wherein the music set with beats stores and records a plurality of music data with beats corresponding to the audio signals of the training data set and the verification data set. The rhythm training system for deep learning according to claim 1, wherein each feature extraction layer further includes a convolutional neural network layer, a batch normalization layer, an excitation layer, and a maximum pooling layer and a discard layer. The rhythm practice system for deep learning according to claim 5, wherein the convolutional neural network layer has thirty-two feature maps, and the length of the convolution kernel of the convolutional neural network layer is 16. The rhythm training system for deep learning according to claim 5, wherein the pooling length of the maximum pooling layer is 2. The rhythm practice system for deep learning according to claim 1, wherein each classification layer further includes a fully connected neural network layer, a batch normalization layer, an excitation layer and a rejection layer. The rhythm practice system for deep learning according to claim 1, wherein the fully connected layer has two neurons.

Images