TWI902105B - Speech enhancement system - Google Patents

Abstract

The present invention discloses a speech enhancement system. The processing unit within the mobile device executes a preprocessing procedure according to the first reference audio received by the wireless audio device and the second reference audio received by the mobile device so as to obtain various feature masks by a feature audio model. According to the requirements, the actual audio corresponding to the reference audio undergoes multiplication processing by the feature mask. When the user communicates with someone by the wireless audio device, the voice of the man communicated with the user is served as the actual audio. The processing unit of the mobile device enhances the voice of the man communicated with the user, so that the user hears the voice clearly, and the clarity of audio reception by the wireless audio device is improved.

Description

Voice enhancement system

This case concerns a speech enhancement system, particularly a speech enhancement system with higher clarity.

Typically, the microphone in a hearing aid is located on the main body. When a user wearing a hearing aid communicates, both the voice of the person they are communicating with and the user's own voice are picked up, processed, and played back by the microphone on the hearing aid. Because the distance between the person and the user is relatively far, the volume transmitted to the hearing aid is attenuated, and it is easily disturbed by ambient noise, such as from noisy places like roadsides, shopping malls, restaurants, and train stations. In places like these, because the user's own voice is closer to the hearing aid's receiver, and due to the effect of bone conduction, the volume and quality of the user's own voice transmitted to the hearing aid's receiver are usually greater than the volume and quality of the voice of the person communicating with the user. This results in the volume of the person communicating with the user being lower and the clarity being lower.

Therefore, developing a speech enhancement system that overcomes the above-mentioned shortcomings is an urgent need at present.

The purpose of this invention is to provide a voice enhancement system in which the processing unit of the mobile device performs a preprocessing procedure on the first reference audio signal and the second reference audio signal received by the wireless audio device to obtain various feature masks using a feature sound model. The feature masks can be used to multiply the features in the actual audio signal corresponding to the reference audio signal as needed. When a user wears the wireless audio device to communicate, the voice of the other party communicating with the user is the actual audio signal. The processing unit of the mobile device can enhance the voice of the other party communicating with the user, so that the user can hear the voice clearly, thereby enhancing the clarity of the wireless audio device's sound reception.

To achieve the above objectives, a broader embodiment of this application provides a voice enhancement system comprising a wireless audio device and a mobile device. The wireless audio device includes a first receiving unit for receiving a first reference audio message. Data transmission occurs between the mobile device and the wireless audio device to receive the first reference audio message, and the mobile device includes a second receiving unit and a processing unit. The second receiving unit receives the second reference audio message and the actual audio message. The processing unit executes preprocessing procedures. The processing unit has a memory unit that stores the feature audio model. The preprocessing procedure includes the following steps: converting the first reference audio and the second reference audio into first audio embedding features and second audio embedding features, and loading the first audio embedding features, the second audio embedding features, and the actual audio into the feature audio model to obtain a first feature mask and a second feature mask. The processing unit converts the actual audio from time-domain to frequency-domain format and uses the second feature mask to multiply the features in the actual audio corresponding to the second reference audio to provide the first output audio.

Some typical embodiments that embody the characteristics and advantages of this case will be described in detail in the following section. It should be understood that this case can have various variations in different forms, all of which do not depart from the scope of this case, and the descriptions and diagrams therein are essentially for illustrative purposes, not for limiting this case.

Please refer to Figures 1 and 2, where Figure 1 is a system architecture diagram of the voice enhancement system of this invention, and Figure 2 is a system architecture diagram of some components of the voice enhancement system shown in Figure 1. As shown in the figures, the voice enhancement system 1 of this invention includes a first wireless audio device 2, a second wireless audio device 3, and a mobile device 4. The first wireless audio device 2 may be, but is not limited to, a hearing aid, wireless headphones, VR/AR/MR glasses, a smartwatch, or a smartphone, etc., and includes a first receiving unit 21 and a first playback unit 22. The first receiving unit 21 is used to receive a first reference audio message, wherein the first reference audio message is in time-domain format and is provided by an audio source adjacent to the first receiving unit 21. The first playback unit 22 is used to play a first output audio message and/or a second output audio message.

The second wireless audio device 3 can be, but is not limited to, a laptop, tablet, or smartphone. In one embodiment, the mobile device 4 may be located within the second wireless audio device 3, while in another embodiment, the mobile device 4 may be independent of the second wireless audio device 3. In this case, the mobile device 4 is described as being independent of the second wireless audio device 3, as shown in Figure 1. The mobile device 4 transmits data wirelessly or wiredly to both the first wireless audio device 2 and the second wireless audio device 3, for example, via Bluetooth, Wi-Fi, 2.4G, or USB 2.0/3.0, to receive the first reference audio signal received by the first wireless audio device 2. Mobile device 4 includes a second radio unit 41 and a processing unit 42. The second radio unit 41 is used to receive a second reference audio message and an actual audio message, wherein the second reference audio message and the actual audio message are in time domain form, and both the second reference audio message and the actual audio message are provided by the same sound source (e.g., human voice) adjacent to the second radio unit 41.

Processing unit 42 is used to execute a preprocessing procedure, which includes the following steps: First, the first reference audio and the second reference audio are converted into a first audio embedding feature and a second audio embedding feature, respectively. Next, the first audio embedding feature, the second audio embedding feature, and the actual audio are loaded into the feature sound model in the memory unit within processing unit 42 to obtain a first feature mask and a second feature mask. The internal circuit structure and detailed functions of processing unit 42 will be further explained below. In addition, processing unit 42 can also determine whether the sound originates from the same person by comparing the first or second audio embedding feature obtained in the preprocessing procedure with the newly received sound embedding feature through Euclidean distance.

As shown in Figure 2, the processing unit 42 includes a time-domain/frequency-domain converter 421, a frequency discriminator 422, a matrix reshaping unit 423, an encoder 424, a memory unit 425, a mixer 426, and a frequency-domain/time-domain converter 427. The time-domain/frequency-domain converter 421 includes a short-time Fourier transform (STFT) function to receive the first reference audio signal from the first receiver unit 21 and the second reference audio signal from the second receiver unit 41, and convert the first reference audio signal and the second reference audio signal from time-domain form to frequency-domain form. The frequency discriminator 422 includes Mel-scale Frequency Cepstral Coefficients (MFCCs) to identify a first audio feature in the first reference audio and a second audio feature in the second reference audio based on the first and second reference audio in frequency domain form. The first audio feature is a quantizable attribute of the information in the first reference audio, and the second audio feature is a quantizable attribute of the information in the second reference audio. The matrix reshaper 423 converts the first audio feature of the first reference audio and the second audio feature of the second reference audio into a first matrix and a second matrix, respectively, that is, quantizes the first and second audio features in matrix form. Encoder 424 encodes the first matrix and the second matrix to output the first audio embedding feature and the second audio embedding feature, respectively, wherein the dimensions of the first audio embedding feature and the second audio embedding feature are 64. Memory unit 425 stores the feature sound model. When the first audio embedding feature, the second audio embedding feature, and the actual audio are loaded into the feature sound model, the first feature mask and the second feature mask are obtained.

Furthermore, the feature audio model within memory unit 425 can also be continuously trained using data from the first and second audio embedding features to enhance the effect. In this embodiment, the feature audio model of memory unit 425 includes one layer of gated recurrent unit (GRU) and two layers of long short-term memory (LSTM).

Please refer to Figure 3 in conjunction with Figures 1 and 2. Figure 3 is a partial detailed architecture diagram of the processing unit within the mobile device of the speech enhancement system shown in Figure 1 when performing multiplication processing on the second reference audio message. Figure 3 only shows the control architecture for further receiving actual audio messages after the mobile device 4 establishes a characteristic sound model, which is part of the control architecture shown in Figure 2. As shown in the figure, the second receiving unit 41 of the mobile device 4 further receives the actual audio messages, which are in time-domain format. The time-domain/frequency-domain converter 421 converts the actual audio messages from time-domain format to frequency-domain format. Mixer 426 receives a second feature mask and actual audio in frequency domain form from memory unit 425, and uses the second feature mask in the feature sound model to perform spectrogram multiplication on the features in the actual audio corresponding to the second reference audio, such as Hadamard matrix processing, to provide a first output audio message. Frequency domain/time domain converter 427 converts the first output audio message from frequency domain form to time domain form. Mobile device 4 further outputs the time domain form of the first output audio message to the first playback unit 22 of the first wireless audio device 2. In other words, the mobile device 4 weakens the reference audio information corresponding to the first wireless audio device 2 in the actual audio it receives, so that when the user plays the first output audio using the first playback unit 22 of the first wireless audio device 2, the audio source received by the first wireless audio device 2 in the first output audio is attenuated, that is, the user can clearly hear the audio source received by the mobile device 4.

Please refer to Figure 4 in conjunction with Figures 1 and 2. Figure 4 is a partial detailed architecture diagram of the processing unit within the mobile device of the speech enhancement system shown in Figure 1 during the product processing of the first reference audio message. Figure 4 only shows the control architecture for further receiving actual audio messages after the mobile device 4 establishes a characteristic sound model, which is part of the control architecture shown in Figure 2. As shown in the figure, the second receiving unit 41 of the mobile device 4 further receives the actual audio messages, which are in time-domain format. The time-domain/frequency-domain converter 421 converts the actual audio messages from time-domain format to frequency-domain format. Mixer 426 receives a first feature mask and actual audio in frequency domain form from memory unit 425, and uses the first feature mask in the feature sound model to perform spectrogram multiplication on the features in the actual audio corresponding to the first reference audio, such as Hadamard matrix processing, to provide a second output audio. Frequency domain/time domain converter 427 converts the second output audio from frequency domain form to time domain form. Mobile device 4 further outputs the second output audio in time domain form to the first playback unit 22 of the first wireless audio device 2. In other words, the mobile device 4 weakens the reference audio information corresponding to the actual audio received by the mobile device 4, so that when the user plays the second output audio using the first playback unit 22 of the first wireless audio device 2, the audio source received by the mobile device 4 in the second output audio is attenuated, that is, the user can clearly hear the audio source received by the first wireless audio device 2.

Of course, in some embodiments, the mobile device 4 may also include a second playback unit 43 to play the first output audio message and/or the second output audio message. Alternatively, in some embodiments, the second wireless audio device 3 may also have a playback unit to play the first output audio message and/or the second output audio message.

In some embodiments, as shown in Figures 5 and 6, the processing unit 42 of the mobile device 4 includes a storage unit 428 for storing a first output audio message and/or a second output audio message, so as to convert the first output audio message and/or the second output audio message into a first voice memo and/or a second voice memo, respectively, so that the voice enhancement system 1 has a voice memo function.

In some embodiments, in addition to multiplying one of the two reference audio messages, the other reference audio message can be multiplied simultaneously to obtain a clearer sound source. Please refer to Figure 7 in conjunction with Figures 1 and 2, where Figure 7 is a partial detailed architecture diagram of the processing unit within the mobile device of the speech enhancement system shown in Figure 1 when the first and second reference audio messages are multiplied simultaneously. Figure 7 only shows the control architecture for further receiving actual audio messages after the mobile device 4 establishes a characteristic sound model, i.e., part of the control architecture in Figure 2. As shown in the figure, the second receiving unit 41 of the mobile device 4 further receives the actual audio messages, where the actual audio messages are in time-domain format. The time-domain to frequency-domain converter 421 converts actual audio signals from time-domain format to frequency-domain format. In this embodiment, mixer 426 includes a first sub-mixer 426a and a second sub-mixer 426b. The first sub-mixer 426a receives a second feature mask and actual audio in frequency domain form from memory unit 425, and uses the second feature mask in the feature sound model to perform spectrogram multiplication on the features in the actual audio corresponding to the second reference audio, for example, Hadamard matrix multiplication, to provide a first output audio. The second sub-mixer 426b receives the feature sound model from memory unit 425 and actual audio in frequency domain form, and uses the first feature mask in the feature sound model to perform spectrogram multiplication on the features in the actual audio corresponding to the first reference audio. Multiplication, such as Hadamard matrix processing, is used to provide a second output audio signal. The frequency-to-time converter 427 converts the first and second output audio signals from frequency-domain to time-domain formats, respectively. Furthermore, in this embodiment, the processing unit 42 of the mobile device 4 includes an adder/subtractor 429 to amplify or attenuate the converted time-domain first and second output audio signals to output a complete output audio signal. The mobile device 4 then outputs the complete time-domain output audio signal to the first playback unit 22 of the first wireless audio device 2. In other words, the mobile device 4 simultaneously enhances the reference audio information corresponding to both the actual audio received by the mobile device 4 and the reference audio information corresponding to the first wireless audio device 2. This allows the user to clearly hear both the audio source received by the mobile device 4 and the audio source received by the first wireless audio device 2 when playing the complete output audio using the first playback unit 22 of the first wireless audio device 2. Of course, in this embodiment, the processing unit 42 of the mobile device 4 may further include a storage unit 428 to store the first output audio information and/or the second output audio information, converting the first output audio information and/or the second output audio information into a first voice memo and/or a second voice memo, respectively, thus enabling the voice enhancement system 1 to have a voice memo function.

In some embodiments, the mobile device 4 can use a feature masking method to add background sound sources received by the mobile device 4 and the first wireless audio device 2 to the background sound source, thereby making the output audio signal softer. In this embodiment, the first wireless audio device 2 further receives background sound sources and uses the background sound sources to generate a background feature mask in the memory unit 425. The method of generating the feature mask is similar to the method of generating the feature mask in Figure 2, so it will not be described in detail here.

Please refer to Figure 8 in conjunction with Figures 1 and 2. Figure 8 is a partial detailed architecture diagram of the processing unit within the mobile device of the speech enhancement system shown in Figure 1, where the second reference audio message and background audio source are multiplied. Figure 8 only shows the control architecture for further receiving actual audio messages after the mobile device 4 establishes a characteristic sound model, which is part of the control architecture shown in Figure 2. As shown in the figure, the second receiving unit 41 of the mobile device 4 further receives the actual audio messages, which are in time-domain format. The time-domain/frequency-domain converter 421 converts the actual audio messages from time-domain format to frequency-domain format. In this embodiment, mixer 426 includes a first sub-mixer 426a and a second sub-mixer 426b. The first sub-mixer 426a receives a second feature mask and actual audio in frequency domain form from memory unit 425, and performs spectrogram multiplication (e.g., Hadamard matrix multiplication) on the features in the actual audio corresponding to the second reference audio using the second feature mask in the feature sound model to provide a first output audio. The second sub-mixer 426b receives the feature sound model from memory unit 425 and actual audio in frequency domain form, and performs spectrogram multiplication on the features in the actual audio corresponding to the background audio using a background feature mask in the feature sound model. Multiplication, such as Hadamard matrix processing, is performed to provide a second output audio signal. The frequency-to-time converter 427 converts both the first and second output audio signals from frequency-domain to time-domain formats. Furthermore, in this embodiment, the processing unit 42 of the mobile device 4 includes an adder/subtractor 429 to compare the converted time-domain first and second output audio signals to output a complete output audio signal. The mobile device 4 then outputs the complete time-domain output audio signal to the first playback unit 22 of the first wireless audio device 2. In other words, the mobile device 4 simultaneously enhances the received actual audio signal corresponding to the reference audio signal provided by the mobile device 4 and the background audio signal corresponding to the first wireless audio device 2. This allows the user to clearly hear the audio source received by the mobile device 4 when playing the complete output audio signal using the first playback unit 22 of the first wireless audio device 2. Furthermore, the audio source is softened, making the sound less rigid. Of course, in this embodiment, the processing unit 42 of the mobile device 4 includes a storage unit 428 for storing the first output audio signal and/or the second output audio signal, respectively converting the first output audio signal and/or the second output audio signal into a first voice memo and/or a second voice memo, thus enabling the voice enhancement system 1 to have a voice memo function.

In summary, this case discloses a voice enhancement system in which the processing unit of the mobile device performs a preprocessing procedure on the first reference audio signal received by the wireless audio device and the second reference audio signal received by the mobile device. This preprocessing process utilizes a feature sound model to obtain various feature masks. The feature masks can be used to multiply the features in the actual audio signal corresponding to the reference audio signal as needed. For example, when a user wears a wireless audio device to communicate, the voice of the other party is the actual audio signal. The processing unit of the mobile device can enhance the voice of the other party, so that the user can hear the voice clearly, thereby enhancing the clarity of the wireless audio device's reception.

1: Voice Enhancement System 2: First Wireless Audio Device 21: First Receiver Unit 22: First Playback Unit 3: Second Wireless Audio Device 4: Mobile Device 41: Second Receiver Unit 42: Processing Unit 43: Second Playback Unit 421: Time Domain/Frequency Domain Converter 422: Frequency Discriminator 423: Matrix Reshaping Unit 424: Encoder 425: Memory Unit 426: Mixer 426a: First Sub-Mixer 426b: Second Sub-Mixer 427: Frequency Domain/Time Domain Converter 428: Storage Unit 429: Adder/Subtractor

Figure 1 is a system architecture diagram of the voice enhancement system of this case; Figure 2 is a system architecture diagram of some components of the voice enhancement system shown in Figure 1; Figure 3 is a partial detailed architecture diagram of the processing unit within the mobile device of the voice enhancement system shown in Figure 1 when performing product processing on the second reference audio; Figure 4 is a partial detailed architecture diagram of the processing unit within the mobile device of the voice enhancement system shown in Figure 1 when performing product processing on the first reference audio; Figure 5 is a partial detailed architecture diagram of another embodiment of the processing unit within the mobile device of the voice enhancement system shown in Figure 1 when performing product processing on the second reference audio; Figure 6 is a partial detailed architecture diagram of another embodiment of the processing unit within the mobile device of the voice enhancement system shown in Figure 1 when performing product processing on the first reference audio message; Figure 7 is a partial detailed architecture diagram of the processing unit within the mobile device of the voice enhancement system shown in Figure 1 when performing product processing on the first reference audio message and the second reference audio message simultaneously; and Figure 8 is a partial detailed architecture diagram of the processing unit within the mobile device of the voice enhancement system shown in Figure 1 when performing product processing on the second reference audio message and the background sound source.

2: First Wireless Audio Device

21: First radio unit

22: First Playback Unit

4: Mobile Devices

41: Second radio unit

42: Processing Unit

421: Time-Domain/Frequency-Domain Converter

422: Frequency Discriminator

423: Matrix Reshaper

424: Encoder

425: Memory Unit

426: Mixer

427: Frequency Domain/Time Domain Converter

Claims (10)

A speech enhancement system includes: a wireless audio device including a first receiving unit for receiving a first reference audio message; and a mobile device for data transmission with the wireless audio device to receive the first reference audio message, wherein the mobile device includes: a second receiving unit for receiving a second reference audio message and an actual audio message; and a processing unit for performing preprocessing. The program, the processing unit, includes: a memory unit storing a feature sound model; a time-domain/frequency-domain converter converting the first reference audio and the second reference audio from time-domain to frequency-domain form; a frequency discriminator identifying a first audio feature in the first reference audio and a second audio feature in the second reference audio; and a matrix reshaping unit converting the first audio feature and the second reference audio into frequency-domain form. The second audio feature is converted into a first matrix and a second matrix respectively; and an encoder encodes the first matrix and the second matrix to output a first audio embedding feature and a second audio embedding feature respectively; wherein the preprocessing procedure includes the following steps: converting the first reference audio and the second reference audio into the first audio embedding feature and the second audio embedding feature; to The processing unit loads the first audio embedding feature, the second audio embedding feature, and the actual audio into the feature sound model to obtain a first feature mask and a second feature mask; wherein, the processing unit converts the actual audio from a time-domain form to a frequency-domain form, and performs product processing on the features in the actual audio corresponding to the second reference audio using the second feature mask to provide a first output audio. The speech enhancement system as described in claim 1, wherein the processing unit further includes a mixer that receives the second feature mask and the actual audio message, and uses the second feature mask to multiply the features in the actual audio message corresponding to the second reference audio message. The voice enhancement system as described in claim 1, wherein when the second receiver unit of the mobile device receives the actual audio message, the time-domain/frequency-domain converter converts the actual audio message from time-domain form to frequency-domain form, and uses the first feature mask to multiply the features in the actual audio message corresponding to the first reference audio message to provide a second output audio message. The speech enhancement system as described in claim 3, wherein the speech enhancement system includes an adder/subtractor for amplifying or attenuating the first output audio message and the second output audio message to output a complete output audio message. The speech enhancement system as described in claim 3 further includes a frequency domain to time domain converter to convert the first output audio message and/or the second output audio message from frequency domain form to time domain form. The voice enhancement system as described in claim 5, wherein the wireless audio device includes a playback unit for playing the first output audio message and/or the second output audio message in time-domain form. The voice enhancement system as described in claim 3 further includes a storage unit for storing the first output audio message and/or the second output audio message, and converting the first output audio message and/or the second output audio message into a first voice memo and/or a second voice memo, respectively. The speech enhancement system as described in claim 1, wherein the characteristic sound model further includes a background feature mask corresponding to a background sound source of the wireless audio device, the time-domain/frequency-domain converter converts the actual audio from time-domain to frequency-domain form, and uses the background feature mask to multiply the features in the actual audio corresponding to the background sound source to provide a second output audio. The speech enhancement system as described in claim 1, wherein the dimensions of the first audio embedding feature and/or the second audio embedding feature are 64 respectively. The speech enhancement system as described in claim 1, wherein the second reference audio and the actual audio are provided by the same sound source.