CN108538283B - A conversion method from lip image features to speech coding parameters - Google Patents

Abstract

The invention relates to a method for converting lip image characteristics into voice coding parameters, which comprises the following steps: 1) constructing a voice coding parameter converter, which comprises an input cache and a trained predictor, sequentially receiving lip feature vectors according to time sequence, and storing the lip feature vectors in the input cache of the converter; 2) sending k latest lip feature vectors cached at the current moment into a predictor as a short-time vector sequence at regular intervals, and obtaining a prediction result which is a coding parameter vector of a voice frame; 3) the speech coding parameter converter outputs a prediction result. Compared with the prior art, the method has the advantages of direct conversion, no need of character conversion, convenience in construction training and the like.

Description

A conversion method from lip image features to speech coding parameters

technical field

The invention relates to the technical fields of computer vision, digital image processing and microelectronics, in particular to a conversion method from lip image features to speech coding parameters

Background technique

Lip recognition is to generate corresponding text expressions based on lip videos. The following are the existing related technical solutions:

(1) CN107122646A, title of invention: a method for realizing lip language unlocking. The principle is to compare the lip features collected in real time with the pre-stored lip features to determine the identity, but only the lip features can be obtained.

(2) CN107437019A, title of invention: identity verification method and device for lip language recognition. The principle is similar to (1), the difference lies in the use of 3D images.

(3) CN106504751A, Title of Invention: Adaptive lip language interaction method and interaction device. The principle is still to recognize the lips as text, and then perform command interaction based on the text, and the conversion steps are complicated.

(4) LipNet is a deep learning lip recognition algorithm released by Oxford University and DeepMind. Its purpose is to recognize lips as text. Compared with the previous technology, the recognition rate is higher, but the conversion process is also complicated.

(5) CN107610703A, title of invention: a multilingual translator based on lip language acquisition and voice pickup. It uses the existing speech recognition module to recognize text, and then uses the existing speech synthesis module to convert the text into speech.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a conversion method from lip image features to speech coding parameters in order to overcome the above-mentioned defects in the prior art

The object of the present invention can be realized through the following technical solutions:

A method for converting from lip image features to speech coding parameters, comprising the following steps:

1) construct a speech coding parameter converter, including input buffer and trained predictor, receive lip feature vector in chronological order, and store it in the input buffer of the converter;

2) every certain time, the k latest lip feature vectors cached at the current moment are sent into the predictor as a short-term vector sequence, and obtain a prediction result, and the prediction result is a coding parameter vector of a speech frame;

3) The speech coding parameter converter outputs the prediction result.

The training method of the predictor specifically includes the following steps:

21) Simultaneous acquisition of video and voice: through video and audio acquisition equipment, synchronously collect video and corresponding voice data, and extract lip images from the video, including the entire mouth and a rectangular area centered on the mouth, and obtain a A lip video composed of a series of lip images I ₁ , I ₂ ,...,In, the speech data _is a sequence of speech samples S ₁ , S ₂ ,..., S _M , and the lip images are Keep the time corresponding relationship with the voice data;

22) Obtain the short-term sequence FIS _t of lip feature vectors at any time t, calculate its image feature vector FI for each frame of lip image I in the lip video, and obtain a series of lip feature vectors FI ₁ , FI ₂ , ...,FI _n , for a given arbitrary time t, extract k continuous lip feature vectors as the short-term sequence of lip feature vectors at time t FIS _t =(FI _t-k+1 ,..., FI _t-2 , F _t-1 , FI _t ), where FI _t is a lip feature vector closest to t in time, and k is a specified parameter;

23) Obtain the speech frame coding parameter vector FA _t at any time t, and extract L continuous speech sample values as a speech frame A _t =(S _t-L+1 ,...,S _t-2 for any time t , S _t-1 , S _t ), where S _t is a speech sample closest to t in time, and the speech analysis algorithm is used to calculate the encoding parameters of the speech frame, which is the speech frame encoding parameter vector FA _{t at time t} , where , L is a fixed parameter;

24) Use samples to train the predictor: take any time t, and use the training sample pair {FIS _t , A _t } obtained in steps 22) and 23) as the input and expected output of the predictor, and randomly select multiple samples within the effective range. A t-value, depending on the type of predictor, to train the predictor.

In the step 22), the frame rate of the lip feature vector is doubled by performing time interpolation on the lip feature vector, or the frame rate of the lip feature vector is increased by using a high-speed image acquisition device for acquisition.

The predictor adopts an artificial neural network, and the artificial neural network is composed of 3 LSTM layers and 2 fully connected layers connected in sequence.

In the described step 22), obtaining the lip feature vector specifically includes the following steps:

For each frame of lip image, extract a total of 20 feature points around the inner and outer edges of the lips, obtain the center coordinates of these 20 feature points, and subtract the center coordinates from the coordinates of each point to obtain 40 coordinate data , normalize the 40 coordinate values, and finally obtain a lip feature vector.

In described step 23), described speech analysis algorithm is LPC10e algorithm, and described encoding parameter vector is LPC parameter, comprises 1 first half frame unvoiced sound mark, 1 second half frame unvoiced sound mark, 1 pitch period. , 1 gain and 10 reflection coefficients.

Compared with the prior art, the present invention has the following characteristics:

1. Direct conversion: The present invention uses machine learning technology to construct a special converter, which realizes the conversion from the lip image feature vector to the speech frame coding parameter vector. The predictor may be implemented by an artificial neural network, but is not limited to an artificial neural network.

2. No text conversion required: The converter takes the lip image feature vector sequence as input, and the speech frame encoding parameter vector as output. The output speech frame encoding parameter vector can be directly synthesized into speech sample frames by speech synthesis technology without going through the intermediate link of "text".

3. Easy to construct training: The present invention also provides a training method for the designed predictor and a method for constructing training samples.

Description of drawings

Figure 1 is the composition and interface structure diagram of the converter.

Figure 2 shows the training flow chart of the predictor.

Figure 3 shows the artificial neural network structure of the predictor.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

The invention designs a converter from lip image features to speech coding parameters, which can convert the received lip image feature vector sequences into speech frame coding parameter vector sequences and output them.

The converter mainly includes input buffers, predictors, and configuration parameters. At its core is a predictor, which is a machine learning model that can be trained using training samples. After the training is completed, the predictor can output a short-term sequence prediction of the lip feature vector as a corresponding speech coding parameter vector.

As shown in Figure 1, the converter mainly includes input buffers, predictors, and configuration parameters, as well as input and output interfaces. The transformer takes a lip feature vector and stores it in the input buffer. At a certain time interval Δt, the k latest lip feature vectors in the cache are sent to the predictor, and the tester obtains a prediction result and outputs the result from the output port. The predicted result is the encoding parameter of a speech frame. Configuration parameters mainly store the configuration parameters of the predictor.

The working process of the converter is described as follows:

(1) The converter receives a series of lip feature vectors FI ₁ , FI ₂ , . . . , FI _n and stores them in the input buffer. These lip feature vectors are sequentially input in chronological order.

(2) Every certain time Δt, the converter takes the k latest lip feature vectors cached at the current moment as a short-term vector sequence FIS _t =(FI _t-k+1 ,...,FI _{t- 2} , FI _t-1 , FI _t ) into the predictor, and get a prediction result FA _t . The predicted result is a vector of encoded parameters for a speech frame. Among them, Δt is equal to the duration occupied by a speech frame, and k is a fixed parameter.

(3) After a prediction result FA _t is obtained, it is output from the output interface immediately.

_The above steps continue to run in a loop, thereby converting the sequence _of lip image feature vectors _{FI 1} , _{FI 2} , . Since the frequency of the speech frame and the frequency of the video frame are not necessarily equal, the number n of the input image feature vector FI and the number m of the output speech frame parameter vector FA are not necessarily equal either.

In the converter described in this patent, a predictor is involved, and the predictor is implemented by a machine learning model with data prediction capability, such as implemented by an artificial neural network, but is not limited to an artificial neural network. It needs to be trained (i.e. let the predictor learn) before it can be applied. The following is the training method. The principle is shown in Figure 2. The lip image feature vector is extracted from the lip video, and the speech coding parameter vector is extracted from the corresponding speech. A short-term sequence of lip image feature vectors FIS _t = (FI _t-k+1 , . . . , FI _t-2 , FI _t-1 , FI _t ) is used as an input sample for training; corresponding to FIS _t The encoded parameter vector FA _t of the speech frame serves as the desired output, the label. Thereby, a large number of training samples and label pairs {FIS _t , FA _t } are obtained to realize the training of the predictor, where t is a random and any valid time.

Training the predictor specifically includes the following steps:

(1) Simultaneously capture video and voice. Through video and audio capture equipment, video and corresponding voice data are collected synchronously. The lip portion needs to be included in the video. Extract the lip part from the video, that is, a rectangular area containing the entire mouth and centered on the mouth. The final lip video is composed of a series of _lip images I ₁ ,I ₂ ,...,In. The speech data is represented as a sequence of speech samples S ₁ , S ₂ ,..., S _M (here, M is an uppercase, representing the number of samples. The number of speech frames is represented by a lowercase m). Image and voice maintain a time-correspondence relationship.

(2) The short-term sequence FIS _{t of the lip feature vector at any time t} . Calculate its image feature vector FI for each frame of lip image I in the lip video, and then obtain a series of lip feature vectors FI ₁ , FI ₂ , . . . , FI _n . For a given arbitrary time t, extract k continuous lip feature vectors as the short-term sequence of lip feature vectors at time t FIS _t =(FI _t-k+1 ,...,FI _t-2 ,F _{t -1} , FI _t ), where FI _t is a lip feature vector closest to t in time, and k is a specified parameter. In order to improve the frame rate of the lip feature vector, temporal interpolation can be performed on the lip feature vector to double the frame rate, or a high-speed image acquisition device can be directly used.

(3) Speech frame encoding parameter vector FA _{t at any time t} . For any time t, extract L consecutive speech samples as a speech frame A _t =(S _t-L+1 ,...,S _t-2 ,S _t-1 ,S _t ), where S _t is time The one speech sample that is closest to t on . Using the speech analysis algorithm, the encoding parameters of the speech frame are calculated, that is, the speech frame encoding parameter vector FA t at time _t . where L is a fixed parameter.

(4) Train the predictor with samples. At any time _t , a training sample pair {FIS _t , A _t } is obtained according to (2) and (3), where FIS _t is the input of the predictor, and At is the expected output of the predictor, that is, the label. A large number of samples can be obtained by randomly selecting a large number of t values within the valid range. Using these samples, the predictor is trained according to the type of predictor.

(5) The trained predictor is used as a component to construct a lip sound converter.

Example 1:

The following is a specific implementation method, but the methods and principles described in the present invention are not limited to the specific numbers given therein.

(1) Predictor, which can be realized by artificial neural network. Predictors can also be built using other machine learning techniques. In the following process, the predictor uses an artificial neural network, that is, the predictor is equivalent to an artificial neural network.

In this embodiment, the neural network consists of 3 LSTM layers + 2 fully connected layers Dense connected in sequence. Dropout layers are added between every two layers and between the internal feedback layers of the LSTM, which are not shown in the figure for the sake of clarity. As shown in Figure 3:

Among them, each of the three layers of LSTM has 80 neurons, and the first two layers use the "return_sequences" mode. The two Dense layers have 100 and 14 neurons, respectively.

The first LSTM layer receives the input of the lip feature sequence in the format of a 3-dimensional array (BATCHES, STEPS, LIP_DIM). The last fully connected layer is the output layer of the neural network, and the output format is a 2-dimensional array (BATCHES, LPC_DIM). In these formats, BATCHES specifies the number of samples (habitually called the number of batches) fed into the neural network each time. BATCHES is usually a value greater than 1 during training, and BATCHES=1 during application; the shape of an input sample is determined by STEPS, LIP_DIM Specify, STEPS specifies the length of a short-term sequence of lip features (habitually called the number of steps), that is, FIS _t = (FI _t-k+1 ,...,FI _t-2 ,F _t-1 ,FI The value of k in _t ), that is, STEPS=k; LIP_DIM specifies a dimension of a lip feature vector FI, and for a lip feature vector composed of 40 coordinate data, LIP_DIM=40. In the output format, LPC_DIM is the dimension of a speech coding parameter vector. For LPC10e, LPC_DIM=14.

The number of neurons and the number of layers can be adjusted appropriately according to different application scenarios. In application scenarios with large vocabulary, the number of neurons and layers can be set to be larger.

(2) Determination of the value of k. The value of k needs to be determined according to the application scenario. For simple application scenarios, it may only be necessary to recognize Chinese characters one by one. Since the pronunciation of a Chinese character is about 0.5 seconds, if the video is 50 frames per second, then k is the number of video frames contained in 0.5 seconds, that is, k=50x0.5 = 25. For situations that use more words, it is necessary to identify the vocabulary or even short sentences as a whole, and the value of k is doubled accordingly. For example, in the two words "size" and "truck", due to the similar mouth shapes of "big" and "ka", it is difficult to distinguish single words, then it is necessary to perform whole word recognition on "size" and "truck", and k needs to be at least equal to 2x25 = around 50

(3) Calculation of lip feature vector. For each frame of image, a total of 20 feature points are extracted around the inner and outer edges of the lips to describe the current shape of the lips. Calculate the center coordinates of these 20 points, and subtract the center coordinates from the coordinates of each point. Each point has two coordinate values of x and y, and 20 points have a total of 40 coordinate data. After normalizing the 40 coordinate values, it is used as a lip feature vector FI. A series of lip feature vectors FI ₁ , FI ₂ , . . . , FI _n can be obtained from continuous video images. Given that the image frame rate is usually not high, the lip feature vector can be interpolated to increase the frame rate. Consecutive k lip feature vectors form a short-term sequence FIS _t =(FI _t-k+1 ,...,FI _t-2 ,FI _t-1 ,FI _t ) as the input sample of the predictor, where FI _t is a lip feature vector closest to t in time.

(4) Calculation of speech frame coding parameter vector. The speech frame at time t is A _t =(S _t-L+1 ,...,S _t-2 ,S _t-1 ,S _t ), where S _t is a speech sample closest to t in time, t for any valid time. Here, a sampling rate of 8000 Hz can be used for speech, and L is set to 180, that is, every 180 samples is taken as an audio frame, which takes 22.5 ms. Speech coding can use LPC10e algorithm. Using this algorithm to analyze a speech frame A _t , the encoding parameter vector FA _t of the frame can be obtained, that is, 14 numerical LPC parameters, including 1 unvoiced and voiced sound mark in the first half of the frame, 1 unvoiced sound mark in the second half of the frame, 1 pitch period, 1 gain, and 10 reflection coefficients. In this way, the speech frame coding parameter vector FA _t at any valid moment can be calculated. Different speech frames can overlap. This audio frame encoding parameter vector is used as the expected output format when the predictor is trained and the actual output format of the predictor when applied.

(5) Training of the predictor: a short-term sequence FIS _t of the above lip feature vector is used as the input sample, and the speech frame encoding parameter vector FA _t at the corresponding moment is used as its label (ie, the prediction target), forming a sample pair {FIS _t , FA _t }, since t can take a value in any valid time, a large number of training samples can be obtained for the training of the predictor. During training, the mean square error (MSE) is used to calculate the prediction error, and the method of error back propagation is used to gradually adjust the network weights. Finally provide a usable predictor.

(6) After the predictor is trained, it is used as a module in the converter. The predictor structure description data and weight data, as well as other parameters, are stored in "configuration parameters", and the configuration parameters are read out when the converter is started, and the predictor is reconstructed according to these parameters.

(7) The method described in this paper can be implemented by means of software, and can also be implemented by means of hardware in part or in whole.

Claims (2)

1. A method for converting lip image features into speech coding parameters, comprising the steps of:

1) constructing a voice coding parameter converter, which comprises an input cache and a trained predictor, sequentially receiving lip feature vectors according to time sequence, and storing the lip feature vectors in the input cache of the converter;

2) sending k latest lip feature vectors cached at the current moment into a predictor as a short-time vector sequence at regular intervals, and obtaining a prediction result, wherein the prediction result is a coding parameter vector of a speech frame, the predictor adopts an artificial neural network, the artificial neural network is formed by sequentially connecting 3 LSTM layers and 2 full-connection layers, and the training method of the predictor specifically comprises the following steps:

21) synchronously acquiring video and voice: synchronously acquiring videos and corresponding voice data through video and audio acquisition equipment, extracting lip images from the videos, wherein the lip images comprise the whole mouth and a rectangular area with the mouth as the center, and acquiring a series of lip images I₁,I₂,...,I_nForming lip video, the voice data being a voice sample value sequence S₁,S₂,...,S_MKeeping the corresponding relation between the lip image and the voice data;

22) lip feature vector short-time sequence FIS (fuzzy inference System) for acquiring any time t_tCalculating image characteristic vector FI of each frame of lip image I in the lip video to obtain a series of lip characteristic vectors FI₁,FI₂,...,FI_nFor a given arbitrary time t, k continuous lip feature vectors are extracted as lip feature vector short-time sequence FIS at the time t_t＝(FI_t-k+1,...,FI_t-2,F_t-1,FI_t) Wherein, FI_tThe method is characterized in that the lip feature vector closest to t in time is obtained, k is a designated parameter, and the method specifically comprises the following steps:

for each frame of lip image, extracting 20 feature points which surround the inner edge and the outer edge of the lip, acquiring the central coordinates of the 20 feature points, subtracting the central coordinates from the coordinates of each point to obtain 40 coordinate data, and performing normalization processing on the 40 coordinate values to finally acquire a lip feature vector;

23) obtaining the coding parameter vector FA of the speech frame at any time t_tFor any time t, extracting L continuous voice sampling values as a voice frame A_t＝(S_t-L+1,...,S_t-2,S_t-1,S_t) In which S is_tIs a speech sample closest to t in time, and adopts speech analysis algorithm to calculate the coding parameter of the speech frame, i.e. the coding parameter vector FA of the speech frame at t moment_tWherein, L is a fixed parameter;

24) training the predictor with the samples: at any given time t), the training sample pairs { FIS) obtained according to steps 22) and 23) are obtained_t，A_tThe speech analysis algorithm is an LPC10e algorithm, the coding parameter vector is an LPC parameter and comprises 1 first half frame voiced and unvoiced flag, 1 second half frame voiced and unvoiced flag, 1 pitch period, 1 gain and 10 reflection coefficients;

3) the speech coding parameter converter outputs a prediction result.

2. The method as claimed in claim 1, wherein in step 22), the lip feature vector is temporally interpolated to double its frame rate, or the frame rate of the lip feature vector is increased by using a high-speed image capturing device.

Images