WO2026018464A1 - Voice processing device, voice processing method, and voice processing program - Google Patents

Abstract

Provided is a section detection unit (102) that detects, from time-series voice data, a mixed voice section that is a time range in which a mixed voice in which a plurality of voices is mixed is present. Also provided is a voice separation unit (103) that separates the mixed voice in the mixed voice section into the plurality of voices.

Description

Audio processing device, audio processing method, and audio processing program

This disclosure relates to audio processing.

Technology related to this disclosure is that of Patent Document 1. The technology in Patent Document 1 separates an audio signal containing a mixture of the voices of multiple speakers into audio signals for each speaker.

Patent No. 6789455

Processing distortion occurs in a time-series signal output by a speech separation technique such as that described in Patent Document 1. When another process (hereinafter referred to as post-processing) is performed after speech separation, this processing distortion may adversely affect the post-processing.
For example, when speech recognition is performed as a post-processing step after speech separation, processing distortion can degrade speech recognition accuracy.

The primary objective of this disclosure is to reduce processing distortion caused by audio separation.

The speech processing device according to the present disclosure includes:
a section detection unit that detects a mixed voice section, which is a time range in which a mixed voice containing a plurality of voices exists, from time-series voice data;
and a sound separating unit that separates the mixed sound in the mixed sound section into the plurality of sounds.

This disclosure makes it possible to reduce processing distortion caused by audio separation.

FIG. 1 is a diagram showing an example of a functional configuration of a voice processing device according to a first embodiment. FIG. 1 is a diagram showing an example of a hardware configuration of a voice processing device according to a first embodiment. FIG. 2 is a diagram showing an example of operation of the audio processing device according to the first embodiment. FIG. 2 is a diagram showing an example of operation of the audio processing device according to the first embodiment. FIG. 2 is a diagram showing an example of operation of the audio processing device according to the first embodiment. FIG. 2 is a diagram showing an example of operation of the audio processing device according to the first embodiment. FIG. 2 is a diagram showing an example of operation of the audio processing device according to the first embodiment. FIG. 10 is a diagram showing an example of the functional configuration of a voice processing device according to a second embodiment. FIG. 10 is a diagram showing an example of the functional configuration of a voice processing device according to a third embodiment. FIG. 10 is a diagram showing an example of the functional configuration of a voice processing device according to a fourth embodiment. FIG. 13 is a diagram showing an example of the functional configuration of a voice processing device according to a fifth embodiment. FIG. 13 is a diagram showing an example of the operation of the audio processing device according to the fifth embodiment. FIG. 13 is a diagram showing an example of the functional configuration of a voice processing device according to a sixth embodiment. FIG. 20 is a diagram showing an example of the configuration of a model generation unit according to the sixth embodiment.

The following describes the embodiments using the drawings. In the following description of the embodiments and in the drawings, the same reference numerals indicate the same or corresponding parts.

Embodiment 1.
***Configuration Description***
FIG. 1 shows an example of the functional configuration of a voice processing device 100 according to this embodiment.
FIG. 2 shows an example of the hardware configuration of the audio processing device 100 according to this embodiment.
The audio processing device 100 is a computer.
The audio processing device 100 reduces processing distortion due to audio separation by limiting the target range of audio separation.
More specifically, the audio processing device 100 reduces processing distortion due to audio separation by limiting the target range of audio separation to a time range in which a mixed audio in which multiple audio components are mixed exists.
The operation procedure of the voice processing device 100 corresponds to a voice processing method, and the program that realizes the operation of the voice processing device 100 corresponds to a voice processing program.

First, we will explain an example of the hardware configuration of the audio processing device 100 shown in Figure 2.

The audio processing device 100 includes, as hardware, a processor 901, a main memory device 902, an auxiliary memory device 903, and a communication device 904.
1 , the speech processing device 100 includes, as functional components, a feature extraction unit 101, a section detection unit 102, a speech separation unit 103, a speech restoration unit 104, and a speech connection unit 105. The functional components in FIG. 1 are realized, for example, by a program.
The auxiliary storage device 903 stores programs that realize these functions.
These programs are loaded from the auxiliary storage device 903 into the main storage device 902. The processor 901 then executes these programs to perform the operations of the functional components in FIG.
FIG. 2 shows a schematic diagram of a state in which a processor 901 is executing a program that implements the functions of the functional components of FIG.
Although not shown in FIG. 2, the audio processing device 100 may also have an input/output device.
The input/output device is, for example, a mouse, a keyboard, a recording medium reading device, a recording medium writing device, a display, and the like.

Next, an example of the functional configuration of the audio processing device 100 shown in Figure 1 will be described.

The feature extraction unit 101 acquires the mixed audio signal sequence 200. For example, the feature extraction unit 101 acquires the mixed audio signal sequence 200 from a communication network via the communication device 904. Alternatively, the feature extraction unit 101 may acquire the mixed audio signal sequence 200 from a recording medium reading device of an input/output device.
The mixed audio signal sequence 200 is a signal sequence of time-series audio signals.
The mixed audio signal sequence 200 includes a time range in which a single audio signal exists (hereinafter referred to as a single audio section) and a time range in which a mixed audio signal exists (hereinafter referred to as a mixed audio section).
A single voice is a voice from a single speaker. A mixed voice is a voice that is a mixture of multiple voices from multiple speakers.

The feature extraction unit 101 extracts features of the audio signals included in the mixed audio signal sequence 200. For example, the feature extraction unit 101 extracts an amplitude spectrum obtained by a short-time Fourier transform as a feature. Alternatively, the feature extraction unit 101 may extract features obtained by a neural network such as a multilayer perceptron or Conv1-D.
The feature extraction unit 101 then outputs feature data 110, which indicates the extracted features in time series, to the section detection unit 102. The feature data 110 indicates the feature of a single voice in a single voice section, and indicates the feature of a mixed voice in a mixed voice section.
The feature extraction unit 101 also outputs the mixed speech signal sequence 200 to the segment detection unit 102 .

The section detection unit 102 acquires the feature data 110 from the feature extraction unit 101 as time-series audio data.
Then, the section detection unit 102 analyzes the feature amount and detects an application section from the feature amount data 110. The application section is made up of a mixed voice section, an immediately preceding section, and an immediately following section.
The mixed voice section is a time range in the feature data 110 in which the mixed voice exists.
The immediately preceding section is a time range in which a single voice exists and is located immediately before the mixed voice section in the time series of the feature data 110 .
The immediately following section is a time range in which a single voice exists and which is located immediately after the mixed voice section in the time series of the feature data 110 .
The section detection unit 102 outputs the application section feature amount 120 to the audio separation unit 103. The application section feature amount 120 is a feature amount of the application section.
Furthermore, the section detection unit 102 outputs the mixed speech signal sequence 200 and the unmixed speech section definition information 1200 to the speech connection unit 105 .
The non-mixed voice section definition information 1200 is information that defines a non-mixed voice section in the mixed voice signal sequence 200. A non-mixed voice section is a section other than a mixed voice section in the mixed voice signal sequence 200. Specifically, a non-mixed voice section includes a forward section and a backward section. The forward section is a time range located before the mixed voice section in the time series of the mixed voice signal sequence 200. The backward section is a time range located after the mixed voice section in the time series of the mixed voice signal sequence 200.
The processing performed by the section detection unit 102 corresponds to section detection processing.

The audio separation unit 103 acquires the applied section feature 120 from the section detection unit 102 .
The audio separation unit 103 separates the feature of the mixed audio in the mixed audio section into feature of multiple audios included in the mixed audio using the application section feature 120. In this way, strictly speaking, the audio separation unit 103 handles the feature of the audio, not the audio itself, but for simplicity of explanation, the audio separation unit 103 may be described as handling the audio itself. In other words, separating the feature of the mixed audio into feature of multiple audios by the audio separation unit 103 may be described as the audio separation unit 103 separating the mixed audio into multiple audios.
Hereinafter, the multiple sounds obtained by separation by the sound separation unit 103 will be referred to as separated sounds. Strictly speaking, the expression "separated sounds" also means the feature quantities of the separated sounds in the processing by the sound separation unit 103. Furthermore, the expressions "forward connected separated sounds,""backward connected separated sounds,""previous single sounds," and "next single sounds," which will be described later, also strictly mean the feature quantities of the respective sounds in the processing by the sound separation unit 103.

The speech separation unit 103 extracts a forward connected separated speech from the plurality of separated speeches using the applied section feature 120. The forward connected separated speech is a separated speech that is estimated to be the same speech as the immediately preceding single speech. The immediately preceding single speech is a single speech that exists in the immediately preceding section.
Furthermore, the speech separation unit 103 extracts a backward-connected separated speech from the plurality of separated speeches using the applied section feature 120. The backward-connected separated speech is a separated speech that is estimated to be the same speech as the immediately following single speech. The immediately following single speech is a single speech that exists in the immediately following section.
The audio separation unit 103 outputs the separated audio information 130 to the audio restoration unit 104. The separated audio information 130 includes features of each separated audio, an identifier of the separated audio corresponding to the forward connected separated audio, an identifier of the separated audio corresponding to the backward connected separated audio, a feature of the immediately preceding single audio, and a feature of the immediately following single audio.
The processing performed by the audio separation unit 103 corresponds to audio separation processing.

The audio restoration unit 104 acquires the separated audio information 130 from the audio separation unit 103 .
The audio restoration unit 104 restores the audio signal of each separated audio from the feature of each separated audio included in the separated audio information 130. The audio restoration unit 104 also restores the audio signal of the immediately preceding single audio from the feature of the immediately preceding single audio. The audio restoration unit 104 also restores the audio signal of the immediately following single audio from the feature of the immediately following single audio.
Then, the audio restoration unit 104 outputs restored audio information 140 to the audio connection unit 105. The restored audio information 140 includes audio signals of each restored separated audio, an audio signal of the restored previous single audio, an audio signal of the restored next single audio, an identifier of the separated audio corresponding to the forward connected separated audio, and an identifier of the separated audio corresponding to the backward connected separated audio.

The audio connection unit 105 acquires the mixed audio signal sequence 200 and the unmixed audio segment definition information 1200 from the segment detection unit 102 .
Furthermore, the audio connection unit 105 acquires the restored audio information 140 from the audio restoration unit 104 .
The audio connection unit 105 uses the unmixed audio section definition information 1200 and the restored audio information 140 to connect the audio signal of the separated audio to the audio signal of the single audio in the mixed audio signal sequence 200 .
More specifically, the audio connection unit 105 connects the audio signal of the forward connected separated audio to the audio signal of the forward single audio. The forward single audio is a single audio present in the forward section. The audio connection unit 105 can identify the forward single audio by referring to the unmixed audio section definition information 1200.
The audio connection unit 105 also connects the audio signal of the backward connected separated audio to the audio signal of the backward single audio. The backward single audio is a single audio present in the backward section. The audio connection unit 105 can identify the backward single audio by referring to the unmixed audio section definition information 1200.
As a result of the concatenation of the audio signals, the audio concatenation unit 105 generates a separated audio signal sequence 300 that is composed of only audio signals of a single audio. The audio concatenation unit 105 then outputs the separated audio signal sequence 300.
For example, the audio connection unit 105 transmits the separated audio signal sequence 300 to the sender of the mixed audio signal sequence 200 via the communication device 904. The audio connection unit 105 may also write the separated audio signal sequence 300 to a recording medium via a recording medium writing device of the input/output device.

***Explanation of Operation***
Next, an example of the operation of the audio processing device 100 according to the present embodiment will be described with reference to FIGS.

Here, it is assumed that the feature data 110 shown in Fig. 3A is generated by the feature extraction unit 101. The section detection unit 102 acquires the feature data 110 shown in Fig. 3A.
The waveform in FIG. 3 shows the transition of the feature quantity over time.
The waveforms indicated by the dashed lines and the solid lines in FIG. 3A represent the features of the speech of different speakers.
In the example of Figure 3(a), the speaker represented by the dashed waveform starts speaking while the speaker represented by the solid waveform starts speaking, and the two voices are temporarily mixed together, as indicated by reference numeral 1101. After that, the speaker represented by the dashed waveform stops speaking, and the speaker represented by the solid waveform continues speaking.

The section detection unit 102 analyzes the feature amounts indicated in the feature amount data 110 in FIG. 3A, and detects a time range in which speech exists from the feature amount data 110 as a speech section 121.
Here, it is assumed that the section detection unit 102 detects three voice sections 121 as shown in FIG. 3(b).
The section detection unit 102 can detect the speech section 121 by using, for example, the method described in the following reference 1. The section detection unit 102 can detect the speech section 121 by changing the configuration of the training data described in reference 1. Reference 1 uses a classifier that distinguishes between noise and speech. The section detection unit 102 can detect the speech section 121 by using a classifier that distinguishes between a non-speech section and a speech section instead of the classifier in reference 1.
Reference 1: International Publication WO2019162990A1

Next, the section detection unit 102 detects an application section 125 from the voice section 121. The application section 125 is made up of a preceding section 122, a mixed voice section 123, and a following section .
The mixed voice section 123 is a time range in which the mixed voice exists.
The immediately preceding section 122 is a time range located immediately before the mixed voice section 123 and in which a single voice exists.
The immediately following section 124 is a time range located immediately after the mixed voice section 123 and in which a single voice exists.
The section detection unit 102 detects the mixed voice section 123 by the following method. Then, the section detection unit 102 detects a predetermined time range immediately before the detected mixed voice section 123 as the immediately preceding section 122. Furthermore, the section detection unit 102 detects a predetermined time range immediately after the detected mixed voice section 123 as the immediately following section 124.
This predetermined time can also be dynamically set depending on the positional relationship of the section detected by the section detection unit 102 with other sections.
Specifically, when mixed audio sections 123 are adjacent to each other at a short time, the section detection unit 102 sets the immediately preceding section 122 and the immediately following section 124 based on the section information so that they do not overlap with other mixed audio sections.
By dynamically setting the default time based on the positional relationship with other sections, mixed audio from another mixed audio section will not be mistakenly mixed into the immediately preceding section 122 or the immediately following section 124, thereby improving the detection accuracy of the section detection unit 102.
The section detection unit 102 may notify the user of the position and time section length of the detected mixed voice section 123 using a display (not shown) or the like. Furthermore, when the default time is changed, the section detection unit 102 may notify the user of the position of the section whose default time has been changed and the default time after the change using a display (not shown) or the like. The user is, for example, the speaker, an operator who operates the voice processing device 100, or the like.

Method (1) of detecting the mixed speech section 123—rule-based The section detection unit 102 detects, for example, a point where the power of the speech signal suddenly increases in the speech section 121 as the start point of the mixed speech section 123. The section detection unit 102 also detects a point where the power of the speech signal suddenly decreases as the end point of the mixed speech section 123.

Method (2) for Detecting the Mixed Speech Section 123—Rule-Based The section detection unit 102 can also detect the mixed speech section 123 using the zero crossing point density.
The zero-crossing density of the speech signal is high in the speech section 121. Using this property, the section detection unit 102 first detects a time range in which the zero-crossing density is equal to or higher than a threshold as the speech section 121. It is also considered that the zero-crossing density will be even higher in the mixed speech section 123. For this reason, the section detection unit 102 detects a time range in the speech section 121 in which the zero-crossing density is even higher as the mixed speech section 123.

Method (3) for Detecting the Mixed Voice Section 123—Statistical Base The section detection unit 102 may detect the mixed voice section 123 by learning a GMM (Gaussian Mixture Model).
For example, the section detection unit 102 learns a noise GMM and a speech GMM, compares the likelihood of each learning result, and detects the speech section 121 .
Furthermore, the section detection unit 102 learns the noise GMM, the single-voice GMM, and the multiple-voice GMM, and compares the likelihood of each learning result to detect the mixed voice section 123 .
In addition to using the GMM, the section detection unit 102 can detect the voice section 121 and the mixed voice section 123 using high-dimensional features such as skewness.
The section detection unit 102 may also configure a Hidden Markov Model (HMM).

Method (4) for Detecting the Mixed Voice Section 123—Neural Network Based The section detection unit 102 may also detect the mixed voice section 123 using a neural network based method.
For example, the segment detection unit 102 detects the speech segment 121 and the mixed speech segment 123 by learning using a multi-layer perceptron, a recurrent neural network (RNN), a long short term memory (LSTM), or the like.

The section detection unit 102 can use a combination of the above detection methods.
For example, the section detection unit 102 can combine detection method (3) and detection method (4). In this case, the section detection unit 102 detects the voice section 121 and the mixed voice section 123 by adding together the statistical-based likelihood and the likelihood obtained from the neural network.
Furthermore, the section detection unit 102 may exclude a detected section from subsequent processing if the length of the detected section is shorter than a set threshold. The threshold is a preset time section length, and is set to a value shorter than a normal audio waveform. If the section detection unit 102 detects a section shorter than this threshold, there is a possibility that a signal other than audio, such as noise, has been mistakenly detected. By excluding sections shorter than the threshold from subsequent processing, unnecessary audio separation processing is prevented. This reduces the amount of processing and waveform distortion due to unnecessary audio separation processing.

The section detection unit 102 outputs the application section feature 120, which is the feature of the application section 125 detected as described above, to the audio separation unit 103.

The speech separation unit 103 separates the mixed speech in the mixed speech section 123 into a plurality of separated speeches using the application section feature 120 .
The audio separation unit 103 can separate the mixed audio into a plurality of separated audios by using, for example, the technique described in Patent Document 1. Alternatively, the audio separation unit 103 may separate the mixed audio into a plurality of separated audios by learning the audio separation model described in Patent Document 1.
Furthermore, the audio separation unit 103 uses the applied section feature 120 to extract forward connected separated audio and backward connected separated audio from the plurality of separated audios.

FIG. 4B shows a state in which the separated audio information 130 separates the mixed audio (a mixture of the audio indicated by the dashed lines and the audio indicated by the solid lines) in the mixed audio section 123 shown in FIG. 4A into separated audio.
Separated speech 126 is speech indicated by a dashed line that has been separated from the mixed speech in the mixed speech section 123. Separated speech 127 is speech indicated by a solid line that has been separated from the mixed speech.
Furthermore, the separated speech 126 is estimated to be the same speech as the immediately preceding single speech 128, and corresponds to forward connected separated speech. The speech separation unit 103 estimates that the separated speech 126 and the immediately preceding single speech 128 are the same speech because the separated speech 126 is more similar to the immediately preceding single speech 128 in terms of features than the immediately following single speech 129.
Furthermore, the separated speech 127 is estimated to be the same speech as the immediately following single speech 129, and corresponds to backward-connected separated speech. The speech separation unit 103 estimates that the separated speech 127 and the immediately following single speech 129 are the same speech because the separated speech 127 is more similar to the immediately following single speech 129 than to the immediately preceding single speech 128 in terms of features.
The audio separation unit 103 outputs the separated audio information 130 to the audio restoration unit 104 .
As described above, the separated audio information 130 includes the features of the separated audio 126 and the separated audio 127, an identifier of the separated audio 126 corresponding to the forward connected separated audio, an identifier of the separated audio 127 corresponding to the backward connected separated audio, the features of the immediately preceding single audio 128, and the features of the immediately following single audio 129.

The audio restoration unit 104 restores the audio signals of the separated audio 126 and the separated audio 127 based on the separated audio information 130. The audio restoration unit 104 also restores the audio signal of the immediately preceding single audio 128 from the feature of the immediately preceding single audio 128. The audio restoration unit 104 also restores the audio signal of the immediately following single audio 129 from the feature of the immediately following single audio 129.
The audio restoration unit 104 outputs the restored audio information 140 to the audio connection unit 105 .
As described above, the restored audio information 140 includes audio signals of the restored separated audio 126 and the separated audio 127, an audio signal of the restored previous single audio 128, an audio signal of the restored next single audio 129, an identifier of the separated audio 126 corresponding to the forward connected separated audio, and an identifier of the separated audio 127 corresponding to the backward connected separated audio.

4A shows a front section 1201 and a rear section 1202. The front section 1201 is a section in which a single voice exists and is located before the mixed voice section 123. The rear section 1202 is a section in which a single voice exists and is located after the mixed voice section 123. In the non-mixed voice section definition information 1200, the front section 1201 and the rear section 1202 are defined as non-mixed voice sections.
4C, the audio connection unit 105 refers to the non-mixed audio section definition information 1200 and identifies a forward single audio signal 201, which is the audio signal of a forward section 1201, and a backward single audio signal 202, which is the audio signal of a backward section 1202, in the mixed audio signal sequence 200. Then, the audio connection unit 105 connects the audio signal of the forward connected separated audio 126 to the end of the forward single audio signal 201 (the part corresponding to the immediately preceding single audio 128). In addition, the audio connection unit 105 connects the audio signal of the backward connected separated audio 127 to the beginning of the backward single audio signal 202 (the part corresponding to the immediately following single audio 129).

The audio connection unit 105 may determine whether the audio signal of the previous single audio 128 is similar to the tail portion of the forward single audio signal 201. In this case, if the audio signal of the previous single audio 128 is similar to the tail portion of the forward single audio signal 201, the audio connection unit 105 connects the audio signal of the forward connected and separated audio 126 to the tail of the previous single audio signal 201.
Similarly, the audio connection unit 105 may determine whether the audio signal of the immediately following single audio 129 is similar to the beginning portion of the following single audio signal 202. In this case, if the audio signal of the immediately following single audio 129 is similar to the beginning portion of the following single audio signal 202, the audio connection unit 105 connects the audio signal of the following connected separated audio 127 to the beginning of the following single audio signal 202.
The audio concatenation unit 105 determines the similarity using a method such as a correlation function or a mean square error.
If the number of speakers is three or more, the audio connection unit 105 identifies the separated audio having the immediately preceding single audio 128 that is most similar to the tail portion of the immediately preceding single audio signal 201 as the forward connected separated audio 126. Then, the audio connection unit 105 connects the audio signal of the forward connected separated audio 126 to the tail of the immediately preceding single audio signal 201.
Similarly, the audio splicing unit 105 identifies the separated audio having the immediately following single audio 129 that is most similar to the beginning portion of the backward single audio signal 202 as the backward connected separated audio 127. Then, the audio splicing unit 105 splices the audio signal of the backward connected separated audio 127 to the beginning of the backward single audio signal 202.

In the example of (c) of Figure 4, the audio connection unit 105 connects the audio signal of the forward connection separated audio 126 to the forward single audio signal 201 and the audio signal of the backward connection separated audio 127 to the backward single audio signal 202 using different audio output channels.
In this embodiment, it is not determined which audio signal is the audio signal of which speaker. Therefore, as shown in the lower part of (c) of Fig. 4, audio signals of different speakers, the audio signal indicated by the solid line and the audio signal 2020 indicated by the dashed line, may be output on the same audio output channel.

Furthermore, as shown in FIG. 5, there are cases where the continuously occurring voice of one speaker is temporarily overlapped with the voice of another speaker.
In the example of Figure 5, the speaker of the solid waveform begins speaking while the speaker of the dashed waveform is still speaking, and the two voices are temporarily mixed together. However, the speaker of the solid waveform finishes speaking in a short time, and thereafter only the speaker of the dashed waveform continues speaking.

5A, the section detection unit 102 detects the time range in which the solid line waveform and the dashed line waveform overlap as a mixed voice section 123. The section detection unit 102 also detects the time range of the single voice immediately before and after the mixed voice section 123 as a immediately before section 122 and an immediately after section 124.
Furthermore, the audio separation unit 103 separates the mixed audio into separated audio 126 and separated audio 1270, as shown in Fig. 5B. In Fig. 5B, the separated audio 126 corresponds to the forward connected separated audio and the backward connected separated audio. In other words, the separated audio 126 is estimated to be the same audio as the immediately preceding single audio 128 and the immediately following single audio 129.
On the other hand, the separated audio 1270 corresponds to neither the forward connected separated audio nor the backward connected separated audio.

In this case, as shown in (c) of Fig. 5, the audio connection unit 105 connects the audio signal of the separated audio 126 to the end of the forward single audio signal 203 (the part corresponding to the immediately preceding single audio 128). The audio connection unit 105 also connects the audio signal of the separated audio 126 to the beginning of the backward single audio signal 204 (the part corresponding to the immediately following single audio 129). The forward single audio signal 203 is the audio signal of the forward section 1203. The backward single audio signal 204 is the audio signal of the backward section 1204.
On the other hand, since the separated audio 1270 does not correspond to either the forward connected separated audio or the backward connected separated audio, the audio connection unit 105 does not connect the separated audio 1270 to any single audio in any time range. Then, the audio connection unit 105 sets the separated audio 1270 to an audio output channel (back channel) different from the audio output channel that connects the audio signal of the separated audio 126 to the forward single audio signal 203 and the backward single audio signal 204.

As described above, the audio connection unit 105 may determine the similarity between the audio signal of the immediately preceding single audio 128 and the end portion of the preceding single audio signal 203, and the similarity between the audio signal of the immediately following single audio 129 and the beginning portion of the following single audio signal 204.

In addition, as shown in Figure 6, the audio connection unit 105 may connect the audio signal of the forward connection separated audio 126 to the forward single audio signal 201 and connect the audio signal of the backward connection separated audio 127 to the backward single audio signal 202 using the same audio output channel.
(a) and (b) of FIG. 6 are the same as (a) and (b) of FIG.
6(c), the audio connection unit 105 connects the audio signal of the forward connected separated audio 126 to the forward single audio signal 201 and the audio signal of the backward connected separated audio 127 to the backward single audio signal 202 on the same audio output channel. More specifically, on the same audio output channel, the audio connection unit 105 sets the single audio signal obtained by connecting the audio signal of the backward connected separated audio 127 to the backward single audio signal 202 behind the single audio signal obtained by connecting the audio signal of the forward connected separated audio 126 to the forward single audio signal 201. Here, the audio connection unit 105 shifts the single audio obtained by connecting the audio signal of the backward connected separated audio 127 to the backward single audio signal 202 backward in time.

Also, even when the separated audio 1270 shown in FIG. 5 exists, the audio connection unit 105 may set the audio signal of the separated audio 1270 to an audio output channel where the audio signal of the separated audio 126 is connected to the front single audio signal 203 and the rear single audio signal 204, as shown in FIG. 7.
(a) and (b) of FIG. 7 are the same as (a) and (b) of FIG.
In (c) of Figure 7, the audio connection unit 105 sets the audio signal of separated audio 1270 to the audio output channel where the audio signal of separated audio 126 is connected to the front single audio signal 203 and the rear single audio signal 204-1. More specifically, the audio connection unit 105 does not connect the audio signal of separated audio 1270 to any single audio in any time range, but sets the audio signal of separated audio 1270 behind the audio signal of the single audio obtained by connecting the front single audio signal 203, the audio signal of separated audio 126, and the audio signal of the rear single audio signal 204-1. Note that in the example of (c) of Figure 7, the rear single audio signal 204 is separated into the rear single audio signal 204-1 and the rear single audio signal 204-2. Also, as in the case of Figure 6, the audio connection unit 105 shifts the audio signal of separated audio 1270 and the rear single audio signal 204-2 later in time.

***Description of Effects of the Embodiment***
In this embodiment, only mixed voice segments that require voice separation are detected as targets for voice separation.
In the prior art, speech separation is performed on the entire input speech, which causes processing distortion to occur in the entire speech, and this processing distortion adversely affects post-processing.
In this embodiment, by limiting the target range of audio separation, it is possible to reduce adverse effects (processing distortion) caused by audio separation.

Furthermore, even if there is a processing delay in audio separation, this embodiment limits the range of audio separation, thereby reducing the processing delay.

Furthermore, even if the computational cost of audio separation is high, this embodiment limits the range of audio separation, thereby reducing computational costs.

Embodiment 2.
In this embodiment, the adverse effects of processing distortion in audio restoration are mitigated.
In this embodiment, differences from the first embodiment will be mainly described.
It should be noted that matters not explained below are the same as those in the first embodiment.

FIG. 8 shows an example of the functional configuration of the audio processing device 100 according to this embodiment.
Compared to FIG. 1, in FIG. 8, the likelihood 1231 of the mixed voice section 123 is output from the section detection section 102 to the voice restoration section 104.

In this embodiment, the section detection unit 102 calculates likelihood 1231 of the mixed voice section 123 in addition to the operations described in the first embodiment.
Then, the segment detection unit 102 outputs the calculated likelihood 1231 to the speech restoration unit 104 .

The speech restoration unit 104 restores the speech signal of each separated speech using the likelihood 1231 .
Specifically, the speech restoration unit 104 first calculates a weighted feature W using the following equation 1, and then restores the speech signal of each separated speech using this weighted feature W as an input. In the following equation 1, alpha is the likelihood 1231. Furthermore, W_before is a feature of the mixed speech before separation. W_after is a feature of the speech after separation (in the example of FIG. 4, separated speech 126 and separated speech 127).
W = alpha × W_after + (1 - alpha) × W_before Formula 1
However, in this embodiment, the above W, W_after, and W_before may be the audio signals themselves. In this case, the separated audio restored by the audio restoration unit 104 is defined as W_after, the audio before separation is defined as W_before, and the audio W obtained by taking the weighted sum of Equation 1 is defined as the output of the audio restoration unit 104.
For example, Reference 2 below discloses that a weighted sum of speech before and after separation is calculated as a process for mitigating adverse effects of processing distortion.
Reference document 2: Patent No. 7345702

***Description of Effects of the Embodiment***
In this embodiment, the likelihood of the mixed speech section is used to calculate a weighted sum of the speech before separation and the speech after separation, thereby reducing the processing distortion. As a result, this embodiment can reduce the adverse effects of processing distortion on post-processing.

Embodiment 3.
In this embodiment, an example will be described in which the feature amount data 110 is divided into pieces of a predetermined size.
In this embodiment, differences from the first embodiment will be mainly described.
It should be noted that matters not explained below are the same as those in the first embodiment.

FIG. 9 shows an example of the functional configuration of a voice processing device 100 according to this embodiment.
Compared to FIG. 1, a first dividing unit 131 and a second dividing unit 132 are added in FIG.
The feature extraction unit 101 also outputs the feature data 110 and the mixed speech signal sequence 200 to the first division unit 131 , and outputs the feature data 110 to the second division unit 132 .

The functions of the first division unit 131 and the second division unit 132 are also realized by a program, similar to the feature extraction unit 101, etc. The program that realizes the functions of the first division unit 131 and the second division unit 132 is executed by the processor 901.

The first dividing unit 131 divides the feature data 110 into pieces of a predetermined size that are suitable for the section detection unit 102 to detect the mixed voice section 123.
The first dividing unit 131 then outputs divided feature data 1131 obtained by dividing the feature data 110 and the mixed speech signal sequence 200 to the section detection unit 102 .
The section detection unit 102 acquires division feature data 1131 instead of the feature data 110, and detects the mixed voice section 123 using the division feature data 1131. The detection process for the mixed voice section 123 in the section detection unit 102 is the same as that described in the first embodiment, and therefore a description thereof will be omitted.
In this embodiment, the section detection unit 102 outputs application section definition information 1230 to the audio separation unit 103 instead of the application section feature amount 120. The application section definition information 1230 is information that defines the application section 125.

The second division unit 132 divides the feature data 110 into pieces of a predetermined size. The second division unit 132 divides the feature data 110 into pieces of a size suitable for audio separation by the audio separation unit 103. The second division unit 132 may divide the feature data 110 into pieces of the same size as the division size in the first division unit 131, or may divide the feature data 110 into pieces of a different size.
Then, the second dividing unit 132 outputs divided feature data 1132 obtained by dividing the feature data 110 to the audio separation unit 103 .
The audio separation unit 103 acquires segmentation feature data 1132 from the second segmentation unit 132. The audio separation unit 103 also acquires application section definition information 1230 from the application section feature 120. The audio separation unit 103 identifies an application section 125 using the application section definition information 1230, and performs audio separation for the identified application section 125 using the segmentation feature data 1132. The audio separation process itself in the audio separation unit 103 is the same as that described in embodiment 1, and therefore will not be described again.

In this embodiment, the section detection unit 102 operates using division feature data 1131, and the audio separation unit 103 operates using division feature data 1132. Therefore, the section detection unit 102 and the audio separation unit 103 can operate in parallel.
The first dividing unit 131 may divide the feature data 110 by providing overlapping portions between adjacent pieces of divided feature data 1131. Similarly, the second dividing unit 132 may divide the feature data 110 by providing overlapping portions between adjacent pieces of divided feature data 1132.

***Description of Effects of the Embodiment***
In this embodiment, the mixed speech section is detected using the division feature data, so that the processing delay in the detection of the mixed speech section can be reduced to the size of the division feature data.
Similarly, in this embodiment, speech separation is performed using the division feature data, so that the processing delay in speech separation can be reduced to the size of the division feature data.

In this embodiment, the feature data is divided into sizes suitable for detecting mixed speech segments, which reduces the deterioration in accuracy of detecting mixed speech segments due to the division of feature data.
Similarly, in this embodiment, feature data is divided into sizes suitable for speech separation, which reduces the deterioration in speech separation accuracy caused by dividing feature data.

Embodiment 4.
In this embodiment, an example will be described in which a feature suitable for detecting a mixed voice section and a feature suitable for voice separation are extracted.
In this embodiment, differences from the first embodiment will be mainly described.
It should be noted that matters not explained below are the same as those in the first embodiment.

FIG. 10 shows an example of the functional configuration of a voice processing device 100 according to this embodiment.
Compared to FIG. 1, in FIG. 10, a first feature extraction unit 141 and a second feature extraction unit 142 are provided instead of the feature extraction unit 101.

The functions of the first feature extraction unit 141 and the second feature extraction unit 142 are also realized by a program, similar to the feature extraction unit 101, etc. The program that realizes the functions of the first feature extraction unit 141 and the second feature extraction unit 142 is executed by the processor 901.

The first feature extraction unit 141 acquires the mixed audio signal sequence 200. For example, the first feature extraction unit 141 acquires the mixed audio signal sequence 200 from a communication network via the communication device 904. Alternatively, the first feature extraction unit 141 may acquire the mixed audio signal sequence 200 from a recording medium reading device of an input/output device.
The first feature extracting unit 141 then extracts features from the mixed speech signal sequence 200 that are suitable for the section detecting unit 102 to detect the mixed speech section 123 .
Then, the first feature extraction unit 141 outputs first feature data 1141 indicating the extracted features and the mixed audio signal sequence 200 to the section detection unit 102. The first feature extraction unit 141 also outputs the mixed audio signal sequence 200 to the second feature extraction unit 142.

The section detection unit 102 acquires first feature data 1141 instead of the feature data 110, and detects the mixed voice section 123 using the first feature data 1141. The detection process for the mixed voice section 123 in the section detection unit 102 is the same as that described in the first embodiment, and therefore a description thereof will be omitted.
In this embodiment, the section detection unit 102 outputs application section definition information 1240 to the audio separation unit 103 instead of the application section feature amount 120. The application section definition information 1240 is information that defines the application section 125.

The second feature extracting unit 142 extracts features from the mixed audio signal sequence 200 that are suitable for the audio separating unit 103 to perform audio separation.
The second feature amount extraction unit 142 may extract the same type of feature amount as the feature amount extracted by the first feature amount extraction unit 141, or may extract a different type of feature amount.
Then, the second feature extraction unit 142 outputs second feature data 1142 indicating the extracted feature to the audio separation unit 103 .

The audio separation unit 103 acquires second feature data 1142 from the second feature extraction unit 142. The audio separation unit 103 also acquires application section definition information 1240 from the application section feature 120. The audio separation unit 103 identifies the application section 125 using the application section definition information 1240, and performs audio separation for the identified application section 125 using the second feature data 1142. The audio separation process itself in the audio separation unit 103 is the same as that described in embodiment 1, so a description thereof will be omitted.

In this embodiment, the section detection unit 102 operates using the first feature data 1141, and the audio separation unit 103 operates using the second feature data 1142. Therefore, the section detection unit 102 and the audio separation unit 103 can operate in parallel.

***Description of Effects of the Embodiment***
In this embodiment, features suitable for detecting mixed speech segments are extracted. Therefore, according to this embodiment, it is possible to improve the accuracy of detecting mixed speech segments. Furthermore, it is possible to use a model suitable for detecting mixed speech segments.
Similarly, in this embodiment, features suitable for speech separation are extracted. Therefore, according to this embodiment, it is possible to improve the accuracy of speech separation. Furthermore, it is possible to use a model suitable for speech separation.

Embodiment 5.
In this embodiment, an example of performing voice emphasis will be described.
In this embodiment, differences from the first embodiment will be mainly described.
It should be noted that matters not explained below are the same as those in the first embodiment.

FIG. 11 shows an example of the functional configuration of a voice processing device 100 according to this embodiment.
Compared to FIG. 1, a voice enhancement unit 106 is added in FIG.

The functions of the speech enhancement unit 106 are also realized by a program, similar to the feature extraction unit 101, etc. The program that realizes the functions of the speech enhancement unit 106 is executed by the processor 901.

The speech enhancement unit 106 performs speech enhancement to separate noise from speech.
The speech enhancement unit 106 uses, for example, the speech enhancement technique disclosed in Reference 3 below.
Reference 3: https://www. merl. com/publications/docs/TR2016-113. pdf

In this embodiment, in addition to the processing described in the first embodiment, the section detection unit 102 determines the noise level in the feature data 110. Then, the section detection unit 102 determines a time range (hereinafter referred to as a speech enhancement section) for which the speech enhancement unit 106 performs speech enhancement, depending on the noise level.
For example, as shown in FIG. 12, the section detection unit 102 designates a time range that is shorter than the speech section 121 and longer than the application section 125 as a speech emphasis section 1255 .
The section detection unit 102 detects the emphasized speech section 1255 by the following method.
Then, the section detection unit 102 outputs the speech emphasis section feature amount 1251 and the application section definition information 1252 to the speech emphasis unit 106. The speech emphasis section feature amount 1251 is a feature amount of the speech emphasis section 1255. The application section definition information 1252 is information that defines the application section 125.

Method (1) of detecting the speech emphasis section 1255—rule-based When the power of the speech signal at a point in time slightly before the start point of the speech section 121 (hereinafter referred to as the candidate start point) is higher than the power of the speech signal at a point in time a fixed time before the candidate start point (in other words, when the power of the speech signal at the candidate start point is higher than that of the non-speech section before the candidate start point), the section detection unit 102 detects the candidate start point as the start point of the speech emphasis section 1255. Also, when the power of the speech signal at a point in time slightly after the end point of the speech section 121 (hereinafter referred to as the candidate end point) is higher than the power of the speech signal at a point in time a fixed time after the candidate end point, the section detection unit 102 detects the candidate end point as the end point of the speech emphasis section 1255.

Method (2) for Detecting the Emphasized Speech Section 1255—Statistical Base The section detection unit 102 may detect the emphasized speech section 1255 by learning a GMM.
For example, the section detection unit 102 learns a noise GMM and a speech GMM, compares the likelihood of each learning result, and detects the speech section 121 .
In detecting the speech emphasis section 1255, the section detection unit 102 performs learning of the noise GMM using a noise with high power or a special noise to which speech emphasis should be applied.
The section detection unit 102 may also configure an HMM.

Method (3) for Detecting the Emphasized Speech Section 1255—Neural Network Based The section detection unit 102 detects the speech section 121 and the emphasized speech section 1255 by learning using a multilayer perceptron, RNN, LSTM, or the like.

The speech emphasizing unit 106 performs speech emphasis on the speech emphasis section feature 1251 .
Then, the speech emphasizing unit 106 outputs the speech emphasis application section feature amount 160 to the speech separating unit 103. The speech emphasis application section feature amount 160 is a feature amount of the application section 125 after the speech emphasis.
The speech emphasizing unit 106 outputs, as a speech enhancement application section feature 160, a feature corresponding to the application section 125 defined in the application section definition information 1252 out of the speech enhancement section feature 1251 after speech enhancement.

The speech separation unit 103 performs speech separation using speech enhancement application section features 160 instead of application section features 120. The speech separation process itself in the speech separation unit 103 is the same as that described in embodiment 1, so a description thereof will be omitted.

In addition, instead of the feature extraction unit 101 as shown in FIG. 10, a configuration in which a first feature extraction unit 141 and a second feature extraction unit 142 are provided may be added to the speech enhancement unit 106. In this case, a third feature extraction unit that extracts a third feature suitable for speech enhancement may be provided. In this case, the third feature is input from the third feature extraction unit to the speech enhancement unit 106.

***Description of Effects of the Embodiment***
According to this embodiment, noise can be removed by speech enhancement. Furthermore, in this embodiment, the target range of speech enhancement is limited according to the noise level, so that processing distortion due to speech enhancement can be limited.

Furthermore, even if there is a processing delay in the speech enhancement, the processing delay can be reduced because the range of speech enhancement is limited in this embodiment.
Furthermore, even if the calculation cost of speech enhancement is high, the calculation cost can be reduced because the range of speech enhancement is limited in this embodiment.

Embodiment 6.
In this embodiment, an example of generating a model for deriving the likelihood 1231 of the mixed voice section 123 described in the second embodiment will be described.
In this embodiment, differences from the second embodiment will be mainly described.
The matters not explained below are the same as those in the second embodiment.

FIG. 13 shows an example of the functional configuration of a voice processing device 100 according to this embodiment.
13, a model generation unit 107 and a model storage unit 108 are added compared to FIG. 8.

The function of the model generation unit 107 is also realized by a program, similar to the feature extraction unit 101 etc. The program that realizes the function of the model generation unit 107 is executed by the processor 901.
The model storage unit 108 is realized by, for example, the auxiliary storage device 903 .

In the learning phase, the model generation unit 107 generates a likelihood derivation model 170 through learning. The likelihood derivation model 170 is a model for deriving a likelihood 1231 of the mixed voice section 123.
The model generation unit 107 stores the generated likelihood derivation model 170 in the model storage unit 108 .
The model storage unit 108 stores the likelihood derivation model 170 .

In this embodiment, the section detection unit 102 derives the likelihood 1231 using the likelihood derivation model 170 in the inference phase.
Then, the segment detection unit 102 outputs the derived likelihood 1231 to the speech restoration unit 104 .
As described in the second embodiment, the speech restoration unit 104 restores the speech signal of each separated speech using the likelihood 1231 .

Specifically, the model generating unit 107 generates the likelihood derivation model 170 as follows.
Here, the speech separation unit 103 is assumed to learn the speech separation model described in Patent Document 1 and separate the mixed speech into a plurality of separated speeches.
The loss function of the speech separation model used by the speech separation unit 103 to train the speech separation model is defined as L_sep(W_true, W_after), where W_true is the true speech signal, and W_after is the speech signal after separation.
The likelihood 1231 output by the section detection unit 102 is set as E_osd.
In this case, the loss function of the speech separation model when training the speech separation model in this embodiment is set to L_sep(W_true', W_after).
W_true' can be expressed by the following equation 2.
W_true'=E_osd×W_true+(1-E_osd)×W_before
Formula 2

As shown in FIG. 14, the model generation unit 107 is composed of, for example, training data 171, an audio mixing unit 172, a feature extraction unit 173, a teacher generation unit 174, and a model training unit 175.

***Description of Effects of the Embodiment***
In this embodiment, a model for deriving the likelihood of a mixed speech segment is generated, which makes it possible to derive the likelihood more accurately and effectively reduce processing distortion.

Although the first to sixth embodiments have been described above, two or more of these embodiments may be combined and implemented.
Alternatively, one of these embodiments may be partially implemented.
Alternatively, two or more of these embodiments may be partially combined and implemented.
Furthermore, the configurations and procedures described in these embodiments may be modified as necessary.

***About Use Cases***
The following use cases are assumed for the speech processing device 100 described in the first to sixth embodiments.
(1) When multiple speakers' voices overlap during a conference, lecture, etc., the voices of each speaker are separated and speech recognition is performed for recording, transcription, etc.
(2) In face-to-face sales, when the voices of the salesperson and the customer overlap, the voices of each speaker are recorded separately and kept as evidence.
(3) When a smartphone, smart speaker, etc. recognizes a user's instructions, questions, etc. to a voice assistant, other people's voices are removed. This allows the smartphone, smart speaker, etc. to accurately understand the user's instructions, questions, etc.
(4) Develop hearing aids, headsets, etc. that can separate the voices of multiple speakers and amplify the voice of the person you are speaking with, making it easier for people with hearing difficulties to hear.
(5) When evaluating the ability to communicate through spoken language in education, training, etc., the voices of multiple speakers are separated and the content and fluency of their speech are measured.
(6) When listening to the voices of patients and users in medical and nursing care settings, other voices can be removed, improving the quality of diagnosis and nursing care services.

***Additional information about hardware configuration***
Here, a supplementary explanation of the hardware configuration of the audio processing device 100 will be given.
The processor 901 shown in FIG. 2 is an integrated circuit (IC) that performs processing.
The processor 901 is a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or the like.
The main storage device 902 shown in FIG. 2 is a RAM (Random Access Memory).
The auxiliary storage device 903 shown in FIG. 2 is a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or the like.
The communication device 904 shown in FIG. 2 is an electronic circuit that performs data communication processing.
The communication device 904 is, for example, a communication chip or a NIC (Network Interface Card).

The auxiliary storage device 903 also stores an OS (Operating System).
At least a part of the OS is executed by the processor 901 .
The processor 901 executes at least a part of the OS, and also executes programs that implement the functions of the functional components shown in FIGS. 1, 8 to 11, and 13 (hereinafter referred to as FIG. 1, etc.).
The processor 901 executes the OS, which performs task management, memory management, file management, communication control, and the like.
In addition, at least one of information, data, signal values, and variable values indicating the results of processing of the functional components shown in Figure 1, etc. is stored in at least one of the main memory device 902, the auxiliary memory device 903, and the register and cache memory within the processor 901.
1 and the like may be stored on a portable recording medium such as a magnetic disk, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, a DVD, etc. The portable recording medium on which the program for realizing the functions of the functional components shown in FIG. 1 and the like is stored may be distributed.

Furthermore, at least one "part" of the functional components shown in FIG. 1 etc. may be read as a "circuit", a "step", a "procedure", a "process", or a "circuitry".
The audio processing device 100 may also be realized by a processing circuit, such as a logic IC (Integrated Circuit), a GA (Gate Array), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array).
In this case, the functional components shown in FIG. 1 and the like are realized as parts of the processing circuit.
In this specification, the term "processing circuitry" refers to a generic concept that encompasses a processor and a processing circuit.
That is, a processor and a processing circuit are each specific examples of "processing circuitry."

Various aspects of the present disclosure are summarized below as appendices.
(Appendix 1)
a section detection unit that detects a mixed voice section, which is a time range in which a mixed voice containing a plurality of voices exists, from time-series voice data;
a sound separation unit that separates the mixed sound in the mixed sound section into the plurality of sounds.
(Appendix 2)
The section detection unit
2. The audio processing device according to claim 1, further comprising: a processor for detecting, from the audio data, the mixed audio section; a preceding section which is a time range in which a single audio section is present and which is located immediately before the mixed audio section in the time series of the audio data; and a succeeding section which is a time range in which a single audio section is present and which is located immediately after the mixed audio section in the time series of the audio data.
(Appendix 3)
The audio processing device further comprises:
An audio processing device as described in Appendix 1 or 2, having an audio connection unit that connects at least one of the multiple separated audios obtained by separation by the audio separation unit to any single audio.
(Appendix 4)
The audio connection unit
connecting any one of the plurality of separated voices to a forward single voice that is a single voice existing in a forward section that is a time range located before the mixed voice section in the time series of the voice data;
4. The audio processing device according to claim 3, wherein one of the plurality of separated audio segments is connected to a subsequent single audio segment, which is a single audio segment present in a subsequent section, which is a time range located after the mixed audio segment in the time series of the audio data.
(Appendix 5)
The section detection unit
detecting from the audio data the mixed audio section, a preceding section which is a time range in which a single audio section is present and which is located immediately before the mixed audio section in the time series of the audio data, and a succeeding section which is a time range in which a single audio section is present and which is located immediately after the mixed audio section in the time series of the audio data;
The audio separation unit
extracting, from the plurality of separated sounds, a separated sound that is estimated to be the same sound as the immediately preceding single sound that is the single sound present in the immediately preceding section, as a forward connected separated sound;
extracting, from the plurality of separated sounds, a separated sound that is estimated to be the same as the immediately following single sound that is the single sound present in the immediately following section, as a backward connected separated sound;
The audio connection unit
Connecting the forward connected separated audio to the forward single audio;
5. The audio processing device of claim 4, wherein the rear connected separated audio is connected to the rear single audio.
(Appendix 6)
The audio connection unit
The audio processing device according to claim 5, wherein the connection of the forward connected separated audio to the forward single audio and the connection of the backward connected separated audio to the backward single audio are performed using different audio output channels. (Supplementary Note 7)
The audio connection unit
An audio processing device as described in Appendix 5, wherein when there is a separated audio among the multiple separated audios that does not correspond to either the forward connected separated audio or the backward connected separated audio, the separated audio is not connected to a single audio of any time range, and the separated audio is set to an audio output channel that is different from the audio output channel to which the forward connected separated audio is connected to the forward single audio and the audio output channel to which the backward connected separated audio is connected to the backward single audio.
(Appendix 8)
The audio connection unit
The audio processing device according to Supplementary Note 5, wherein the connection of the forward connected separated audio to the forward single audio and the connection of the backward connected separated audio to the backward single audio are performed on the same audio output channel. (Supplementary Note 9)
The audio connection unit
9. The audio processing device according to claim 8, wherein, in the same audio output channel, a single audio obtained by connecting the backward connected separated audio, the immediately following single audio, and the backward single audio is set behind a single audio obtained by connecting the forward single audio, the immediately preceding single audio, and the forward connected separated audio.
(Appendix 10)
The audio connection unit
An audio processing device as described in Appendix 8, in which, when there is a separated audio among the multiple separated audios that does not correspond to either the forward-connected separated audio or the backward-connected separated audio, the separated audio is set to an audio output channel where the forward-connected separated audio is connected to the forward single audio and the backward-connected separated audio is connected to the backward single audio, without connecting the separated audio to a single audio of any time range.
(Appendix 11)
The section detection unit
acquiring, as the speech data, time-series feature data in which a feature of a single speech is indicated in a single speech section, which is a time range in which a single speech exists, and a feature of the mixed speech is indicated in the mixed speech section; analyzing the feature data to detect the mixed speech section from the feature data;
The audio separation unit
Separating the feature of the mixed voice in the mixed voice section into feature of the plurality of voices;
The audio processing device further comprises:
a sound restoration unit that restores a plurality of separated sounds that are the plurality of sounds from feature quantities of the plurality of sounds obtained by separation by the separation unit,
The audio connection unit
4. The audio processing device according to claim 3, wherein at least one of the separated audio signals obtained by the restoration of the audio restoration unit is connected to one single audio signal.
(Appendix 12)
The audio restoration unit
12. The speech processing device according to claim 11, wherein the plurality of separated speeches are reconstructed using likelihoods of the mixed speech segments.
(Appendix 13)
The audio processing device further comprises:
a first division unit that divides the audio data into sizes suitable for detecting the mixed audio segment by the segment detection unit;
a second division unit that divides the audio data into sizes suitable for audio division by the audio separation unit,
The section detection unit
detecting the mixed voice section from the voice data divided by the first dividing unit;
The audio separation unit
2. The audio processing device according to claim 1, wherein the mixed audio in the mixed audio section is separated into the plurality of audio segments using the audio data divided by the second dividing unit.
(Appendix 14)
The audio processing device further comprises:
a first feature extraction unit that generates first feature data indicating features suitable for detecting the mixed voice segment by the segment detection unit;
a second feature extraction unit that generates second feature data indicating features suitable for audio segmentation by the audio separation unit;
The section detection unit
acquiring the first feature data as the audio data, and detecting the mixed audio section from the first feature data;
The audio separation unit
2. The speech processing device according to claim 1, wherein the mixed speech in the mixed speech section is separated into the plurality of speeches using the second feature data.
(Appendix 15)
The audio processing device further comprises:
a speech enhancement unit that performs speech enhancement to separate noise from speech;
The section detection unit
2. The speech processing device according to claim 1, wherein a time range for which the speech enhancement unit performs speech enhancement is determined according to a noise level in the speech data.
(Appendix 16)
The audio processing device further comprises:
a model generation unit that generates a likelihood derivation model that is a model for deriving the likelihood of the mixed voice section;
The audio restoration unit
13. The speech processing device according to claim 12, wherein the plurality of separated speeches are reconstructed using likelihoods of the mixed speech segments derived using the likelihood derivation model.
(Appendix 17)
A computer detects a mixed voice section, which is a time range in which a mixed voice containing a plurality of voices exists, from the time-series voice data;
The audio processing method, in which the computer separates the mixed audio in the mixed audio section into the plurality of audios.
(Appendix 18)
A section detection process for detecting a mixed voice section, which is a time range in which a mixed voice containing a plurality of voices exists, from the time-series voice data;
and a sound separation process for separating the mixed sound in the mixed sound section into the plurality of sounds.

100 Speech processing device, 101 Feature extraction unit, 102 Section detection unit, 103 Speech separation unit, 104 Speech restoration unit, 105 Speech connection unit, 106 Speech enhancement unit, 107 Model generation unit, 108 Model storage unit, 110 Feature data, 120 Application section feature, 121 Speech section, 122 Previous section, 123 Mixed speech section, 124 Next section, 125 Application section, 126 Separation Speech, 127 separated speech, 128 immediately preceding single speech, 129 immediately following single speech, 130 separated speech information, 131 first segmentation unit, 132 second segmentation unit, 140 restored speech information, 141 first feature extraction unit, 142 second feature extraction unit, 160 speech enhancement application section features, 170 likelihood derivation model, 171 training data, 172 speech mixing unit, 173 feature extraction unit, 174 teacher Generation unit, 175 model learning unit, 200 mixed audio signal sequence, 201 front single audio signal, 202 rear single audio signal, 203 front single audio signal, 204 rear single audio signal, 300 separated audio signal sequence, 901 processor, 902 main memory device, 903 auxiliary memory device, 904 communication device, 1131 split feature data, 1132 split feature data, 1141 first feature data data, 1142 second feature data, 1200 unmixed speech section definition information, 1201 forward section, 1202 backward section, 1203 forward section, 1204 backward section, 1230 application section definition information, 1231 likelihood, 1240 application section definition information, 1251 speech enhancement section feature, 1252 application section definition information, 1255 speech enhancement section, 1270 separated speech, 2020 speech signal.

Claims (18)

a section detection unit that detects a mixed voice section, which is a time range in which a mixed voice containing a plurality of voices exists, from time-series voice data;
a sound separation unit that separates the mixed sound in the mixed sound section into the plurality of sounds. The section detection unit
2. The audio processing device according to claim 1, further comprising: a step of detecting, from the audio data, the mixed audio section; a preceding section which is a time range in which a single audio section is present and which is located immediately before the mixed audio section in the time series of the audio data; and a succeeding section which is a time range in which a single audio section is present and which is located immediately after the mixed audio section in the time series of the audio data. The audio processing device further comprises:
The audio processing device according to claim 1 , further comprising an audio connection unit that connects at least one of a plurality of separated audios, which are the plurality of audios obtained by separation by the audio separation unit, to one single audio. The audio connection unit
connecting any one of the plurality of separated voices to a forward single voice that is a single voice existing in a forward section that is a time range located before the mixed voice section in the time series of the voice data;
The audio processing device according to claim 3 , wherein any one of the plurality of separated audio segments is connected to a subsequent single audio segment that is a single audio segment present in a subsequent section that is a time range that is located after the mixed audio segment in the time series of the audio data. The section detection unit
detecting, from the audio data, the mixed audio section, a preceding section which is a time range in which a single audio section is present and which is located immediately before the mixed audio section in the time series of the audio data, and a succeeding section which is a time range in which a single audio section is present and which is located immediately after the mixed audio section in the time series of the audio data;
The audio separation unit
extracting, from the plurality of separated sounds, a separated sound that is estimated to be the same sound as the immediately preceding single sound that is the single sound present in the immediately preceding section, as a forward connected separated sound;
extracting, from the plurality of separated sounds, a separated sound that is estimated to be the same as the immediately following single sound that is the single sound present in the immediately following section, as a backward connected separated sound;
The audio connection unit
Connecting the forward connected separated audio to the forward single audio;
The audio processing device according to claim 4 , wherein the rear connected separated audio is connected to the rear single audio. The audio connection unit
6. The audio processing device according to claim 5, wherein the connection of the forward connected separated audio to the forward single audio and the connection of the backward connected separated audio to the backward single audio are performed in different audio output channels. The audio connection unit
6. The audio processing device according to claim 5, wherein when there is a separated audio among the plurality of separated audios that does not correspond to either the forward-connected separated audio or the backward-connected separated audio, the separated audio is not connected to a single audio of any time range, and the separated audio is set to an audio output channel that is different from the audio output channel to which the forward-connected separated audio is connected to the forward single audio and the audio output channel to which the backward-connected separated audio is connected to the backward single audio. The audio connection unit
6. The audio processing device according to claim 5, wherein the connection of the forward connected separated audio to the forward single audio and the connection of the backward connected separated audio to the backward single audio are performed on the same audio output channel. The audio connection unit
9. The audio processing device according to claim 8, wherein, in the same audio output channel, a single audio obtained by connecting the backward connected separated audio, the immediately following single audio, and the backward single audio is set behind a single audio obtained by connecting the forward single audio, the immediately preceding single audio, and the forward connected separated audio. The audio connection unit
10. The audio processing device according to claim 8 or 9, wherein when there is a separated audio among the plurality of separated audios that does not correspond to either the forward-connected separated audio or the backward-connected separated audio, the separated audio is set to an audio output channel in which the forward-connected separated audio is connected to the forward single audio and the backward-connected separated audio is connected to the backward single audio, without connecting the separated audio to a single audio of either time range. The section detection unit
acquiring, as the speech data, time-series feature data in which a feature of a single speech is indicated in a single speech section, which is a time range in which a single speech exists, and a feature of the mixed speech is indicated in the mixed speech section; analyzing the feature data to detect the mixed speech section from the feature data;
The audio separation unit
Separating the feature of the mixed voice in the mixed voice section into feature of the plurality of voices;
The audio processing device further comprises:
a sound restoration unit that restores a plurality of separated sounds that are the plurality of sounds from feature quantities of the plurality of sounds obtained by separation by the separation unit,
The audio connection unit
The audio processing device according to any one of claims 3 to 10, wherein at least one of the separated audio signals obtained by the restoration of the audio restoration unit is connected to one single audio signal. The audio restoration unit
The speech processing device according to claim 11 , wherein the plurality of separated speeches are reconstructed using likelihoods of the mixed speech segments. The audio processing device further comprises:
a first division unit that divides the audio data into sizes suitable for detecting the mixed audio segment by the segment detection unit;
a second division unit that divides the audio data into sizes suitable for audio division by the audio separation unit,
The section detection unit
detecting the mixed voice section from the voice data divided by the first dividing unit;
The audio separation unit
The audio processing device according to any one of claims 1 to 12, wherein the mixed audio in the mixed audio section is separated into the plurality of audio signals using the audio data divided by the second dividing unit. The audio processing device further comprises:
a first feature extraction unit that generates first feature data indicating features suitable for detecting the mixed voice segment by the segment detection unit;
a second feature extraction unit that generates second feature data indicating features suitable for audio segmentation by the audio separation unit;
The section detection unit
acquiring the first feature data as the audio data, and detecting the mixed audio section from the first feature data;
The audio separation unit
The voice processing device according to any one of claims 1 to 13, wherein the mixed voice in the mixed voice section is separated into the plurality of voices using the second feature data. The audio processing device further comprises:
a speech enhancement unit that performs speech enhancement to separate noise from speech;
The section detection unit
15. The speech processing device according to claim 1, wherein a time range for which the speech enhancement unit performs speech enhancement is determined in accordance with a noise level in the speech data. The audio processing device further comprises:
a model generation unit that generates a likelihood derivation model that is a model for deriving the likelihood of the mixed voice section;
The audio restoration unit
The speech processing device according to claim 12 , wherein the plurality of separated speeches are reconstructed using the likelihood of the mixed speech segments derived using the likelihood derivation model. A computer detects a mixed voice section, which is a time range in which a mixed voice containing a plurality of voices exists, from the time-series voice data;
The audio processing method, in which the computer separates the mixed audio in the mixed audio section into the plurality of audios. a section detection process for detecting a mixed voice section, which is a time range in which a mixed voice containing a plurality of voices exists, from the time-series voice data;
and a sound separation process for separating the mixed sound in the mixed sound section into the plurality of sounds.