CN1246793C - Method of hand language translation through a intermediate mode language - Google Patents

Abstract

A method for sign language translation through an intermediate pattern language, comprising: collecting sign language word data, extracting feature information in the sign language word data, performing sign language continuous sentence recognition according to the feature information, and then recording the recognition results of the intermediate pattern language data and converting the intermediate Convert the pattern language data into the non-sign language words and output them; and collect the non-sign language word data, and convert the non-sign language words into the intermediate pattern language data according to the corresponding relationship between the intermediate pattern language data and the non-sign language language and record , according to the intermediate mode language data, find the corresponding sign language word data in the sign language vocabulary database, and then synthesize the sign language word data into sign language image information output; the present invention corresponds both sign language and non-sign language mode languages to the intermediate mode language , which is conducive to the expansion of the sign language translation system, and facilitates the mutual conversion between non-sign language languages and sign languages.

Description

Method for sign language interpretation via intermediate mode language

Technical field :

The present invention relates to a method for translating sign language through intermediate pattern language, in particular to a method for translating sign language into non-sign language through an intermediate pattern language data form and translating non-sign language language into sign language through an intermediate pattern language data form method.

Background technology :

Language is an indispensable tool for people to communicate with each other. However, there are hundreds of languages in use in the world today. If local dialects and sign languages of the deaf are added, the number is even more difficult to count; there are so many The variety of languages makes it very difficult to communicate between people who use different languages, not only between physically healthy people, but also among some physically disabled people; therefore, various languages The translation between them has always been a major problem plaguing all human beings.

With the continuous progress of human science and technology, especially in the past 20 years, the rapid development of computer technology has made it possible to use computers to translate a perceptual language into other languages. But, utilize computer to carry out language translation at present and usually only directly translate the language of a kind of mode into the language of another mode; The method of this translation and its system have following shortcoming:

In the field of automatic computer translation, there is usually only a fixed vocabulary correspondence between a translated language and a translated language. Therefore, the existing translation system cannot realize the translation of multiple languages to one translated language; A corresponding fixed vocabulary correspondence relationship is established between each translation language; this way, on the one hand, it will make the design and implementation of the corresponding translation system a huge workload, and on the other hand, the language extension of the translation system is not easy to realize.

Invention content :

The main purpose of the present invention is to provide a method for sign language translation through intermediate mode language, sign language is translated into an intermediate mode language, and then the language of the intermediate mode is further translated into the required non-sign language language form; or non-sign language language Convert to an intermediate mode language, and then translate that intermediate mode language into sign language.

Another object of the present invention is to provide a method for sign language translation through the intermediate mode language. Both sign language and non-sign language mode languages correspond to the intermediate mode language, which is conducive to the expansion of the sign language translation system, so as to realize non-sign language language and sign language. conversion between each other.

The purpose of the present invention is achieved like this:

The present invention provides a method for interpreting sign language through an intermediate pattern language, which is to translate sign language into a non-sign language language through an intermediate pattern language data form, and the specific steps include:

Step 101: collecting sign language word data;

Step 102: extracting feature information in the sign language word data;

Step 103: Perform sign language continuous sentence recognition according to the feature information, and then record the recognition result of the intermediate mode language data;

Step 104: According to the corresponding relationship between the intermediate pattern language data and the corresponding non-sign language language, convert the intermediate pattern language data into the non-sign language words and output them.

The present invention also provides a method for translating sign language through intermediate pattern language, which is to translate non-sign language language into sign language through an intermediate pattern language data form, and the specific steps include:

Step 201: Collect non-sign language language word data;

Step 202: According to the corresponding relationship between the intermediate mode language data and the non-sign language language, the non-sign language words are converted into intermediate mode language data and recorded;

Step 203: According to the intermediate mode language data, find the corresponding sign language word data in the sign language word database, and then synthesize the sign language word data into sign language image information for output.

The method for translating sign language into non-sign language or non-sign language into sign language also includes: collecting corresponding face information while collecting sign language word data or non-sign language data; and then extracting feature data in the face information , and finally use the feature data to synthesize the output face image during the translation output.

The specific method of collecting sign language word data is: using data gloves to collect sensory data of each joint of the hand; using a position tracker to input the position and direction data of sign language gestures; wherein, the data glove is set on the left and right hands of the human body; the position tracker It includes a transmitter and more than one receiver; the transmitter emits electromagnetic waves, and the receiver is installed on the left and right wrists of the human body. The receiver receives the electromagnetic waves and calculates the position and direction data of the receiver relative to the transmitter.

The specific method of extracting the feature information in the sign language word data is: calculate the position and direction of the left and right hands relative to the reference, perform normalization processing on each component of the sensory data of each joint of the hand, and use the processed data as an implicit Markov model (Hidden Markov Mode1, referred to as HMM) training samples, the establishment of sign language sample model library.

As mentioned above, an HMM can be expressed with parameters: λ=(A, B, π),

Among them, A={a _ij } is the state transition probability matrix,

And satisfy the formula: a _ij =P[q _t+1 =S _j |q _t =S _i ], 1≤i, j≤N;

And satisfy the constraints: a _ij ≥ 0, 1 ≤ i, j ≤ N, $\underset{j=1}{\overset{N}{\mathrm{\Σ}}}{a}_{\mathrm{ij}}=1,$ 1≤i≤N;

In the above formula, N is the state number of the model;

π={π _i }, π _i represents the probability of starting from the i-th state node,

And satisfy the formula: π _i =P[q ₁ =S _i ], 1≤i≤N;

And satisfy the constraints: π _i ≥ 0, $\underset{i=1}{\overset{N}{\mathrm{\Σ}}}{\mathrm{\π}}_i=1,$ 1≤i≤N;

B={b _j (k)} is the probability density of the observed signal, when the observed symbol is a continuous vector, b _j (k) is a continuous probability density function, and: $b_j\left(k\right)=\underset{m=1}{\overset{m}{\mathrm{\Σ}}}{c}_{\mathrm{jm}}G\[{\mathrm{\μ}}_{\mathrm{jm}},{\mathrm{\Σ}}_{\mathrm{jm}},o_k\],$ 1≤j≤N;

Among them, N is the state number of the model, M is the number of mixing items, O _k is the observation vector at time k; c _jm is the mixing ratio (Mixing proportion), and it satisfies:

$\underset{m=1}{\overset{m}{\mathrm{\Σ}}}{c}_{\mathrm{jm}}=1,$ 1≤j≤N, c _jm ≥0, 1≤j≤N, 1≤m≤M

Wherein: G is taken as the Gaussian probability density function, and μ _jm and ∑ _jm are respectively the mean vector and the covariance matrix of the mth component in the Gaussian mixture probability density;

$\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; jm</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; jm</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; K</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\sqrt{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}}{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; jm</span>}}<span\; class="notranslate">\; |</span>}^{\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}}}\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; [</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; jm</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; jm</span>}}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; jm</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ]</span>$

K groups of training data O=[O ⁽¹⁾ corresponding to the same gesture word, O ⁽²⁾ ,..., O ^(k) ], wherein $o^{\left(k\right)}=[o_1^{\left(k\right)}{o}_2^{\left(K\right)}\&hellip;o_{T_k}^{\left(k\right)}]$ is the kth group of training data, T _k is the total number of frames of the kth group of training data;

The revaluation formulas of π _i , a _ij and c _jm , μ _m , ∑ _m are:

${\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; \&pi;</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; K</span>}}\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; K</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&gamma;</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>$

That is, at time t=1, the expected probability of state node i appearing;

${\stackrel{\&OverBar;}{a}}_{\mathrm{ij}}=\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k-1}{\mathrm{\&Sigma;}}}{\mathrm{\&xi;}}_t^{\left(k\right)}(i,j)/\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k-1}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}_t^{\left(k\right)}\left(i\right),$ in,

The numerator part is the expected probability of transferring from state node i to state node j;

The denominator part is the expected probability of transferring from state node i;

${\stackrel{\&OverBar;}{c}}_{\mathrm{jm}}=\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}_t^{\left(k\right)}(j,m)/\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k}{\mathrm{\&Sigma;}}}\underset{m=1}{\overset{m}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}_t^{\left(k\right)}(j,m),$ in,

The numerator part is the expected probability of appearing in the mth branch of the state node j;

The denominator part is the expected probability of appearing at state node j;

${\stackrel{\&OverBar;}{\mathrm{\&mu;}}}_m=\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}_t^{\left(k\right)}(j,m)\&Center\; Dot;o_t^{\left(k\right)}/\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}_t^{\left(k\right)}(j,m),$ in,

The molecular part is the expected probability of the observation sequence 0 appearing in the mth branch of the state node j;

The denominator part is the expected probability that the mth branch of the state node j appears;

${\stackrel{\&OverBar;}{\mathrm{\&Sigma;}}}_m=\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}_t^{\left(k\right)}(j,m)\&Center\; Dot;(o_t^{\left(k\right)}-{\mathrm{\&mu;}}_m){(o_t^{\left(k\right)}-{\mathrm{\&mu;}}_m)}^{\&prime;}/\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}_t^{\left(k\right)}(j,m),$ in,

The molecular part is the expected probability of the mean square error of the observation sequence 0 appearing in the mth branch of the state node j;

The denominator part is the expected probability that the mth branch of state node j appears.

The specific methods for continuous sentence recognition in sign language are as follows:

After the model library is built, use the Viterbi decoding method to calculate the likelihood probability of the test sample and various possible model sequences, and the word sequence corresponding to the model sequence with the largest probability value is the recognition result.

Suppose the capacity of the vocabulary is v, the model numbers of words are k=1,...,V, the corresponding model parameters are (π ^k , A ^k , c ^k , μ ^k , U ^k ), the state of each word The number is L, and the input sign language frame sequence number is i=1, 2,..., N:

When vocabulary is transferred within the same model (i > 1):

$\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; \}</span>$

T(i,j,k)=k

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; j</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; \}</span>$

When the vocabulary is transferred at the boundary of the model (j=1):

$\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; \&le;</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; \}</span>$

$\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; \&le;</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; \}</span>$

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; j</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; \&le;</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; \}</span>$

At the beginning (i=1), Pr(1,k)=p(1,1,k), Pr(j,k)=0, j>1

T(1,j,k)=-1

F(1,j,k)＝-1

The above formula can be used to recursively calculate each Pr(L, k), so that the global maximum probability can be obtained:

${\mathrm{<span\; class="notranslate">\; PR</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \}</span>$

The optimal path (T i , F j ) is backtracked from T(i, j, k) and F( _i , j, _k ) (in reverse order):

${\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; N</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \}</span>$

F _N =L

T _i =T(i+1, T _i+1 , F _i+1 ), N-1≥i≥1

F _i =F(i+1, T _i+1 , F _i+1 ), N-1≥i≥1

Get the recognition result; among them,

p(i, j, k): the probability of the i-th frame appearing in the j-th state of word k;

Pr(j, k): Up to the current input frame i, the approximate maximum probability of the jth state of word k is transferred from the start state to the word k;

T(i, j, k): record the serial number of the model where the previous frame is located;

F(i, j, k): Record the state of the previous frame in the model T(i, j, k).

In order to improve the recognition accuracy, a second-order Markov chain (Bigram) is embedded in the Viterbi search process, that is, the prior probability of the sentence can be calculated by the following formula:

$\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; W</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{<span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span>\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Pi;</span>}}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>$

in,

W is the recognized sentence;

w ₁ , w ₂ ,...w _n are each word in the recognized sentence W;

P(w ₁ |wi _-1 ) is the occurrence frequency of word pairs.

The specific method of synthesizing sign language word data into sign language image information is as follows:

Using the human body representation model of VRML to establish a virtual human;

Determine the angle value of each degree of freedom of the virtual person;

Calculate the position and direction of each limb of the virtual human, and determine a posture of the virtual human;

Ignore the non-upper limb joint angles of the pose;

Continuously display each gesture in a sign language movement according to the specified time interval, and generate corresponding sign language movement images.

When generating the sign language moving image, smooth interpolation is further performed between adjacent frames of the sign language moving image: the specific interpolation is calculated according to the following formula:

${\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\frac{{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}}{{\mathrm{<span\; class="notranslate">\; t</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>$

in,

f ₁ and f ₂ are two adjacent image frames in a sign language movement;

t _f1 and t _f2 are the time values of f ₁ and f ₂ from the starting point respectively;

t ₁ is the time value of the inserted frame from the starting point;

t ₁ is the time value of the starting point;

G _i (t _f ) is the value of the interpolated degree of freedom curve function

G _i (f ₁ ) is the function value of the degree of freedom curve at the starting point;

G _i (t _f1 ) is the degree of freedom curve function of f ₁ ;

G _i (t _f2 ) is the function value of the degree of freedom curve of f ₂ .

When generating sign language motion images, a quadruple-based motion interpolation method is further used to perform smooth transition processing on complex joints in discontinuous sign language frames. The specific smoothing processing is calculated according to the following formula:

${\mathrm{<span\; class="notranslate">\; q</span>}}_{{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; sin</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; \&theta;</span>}}{\mathrm{<span\; class="notranslate">\; sin</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}}{\mathrm{<span\; class="notranslate">\; q</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; sin</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}\mathrm{<span\; class="notranslate">\; \&theta;</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; sin</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}}{\mathrm{<span\; class="notranslate">\; q</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

in,

f ₁ and f ₂ are image frames of two adjacent gestures in a sign language movement;

t _f1 and t _f2 are the time values of f ₁ and f ₂ from the starting point respectively;

q _f1 and q _f2 are the directions of joints at t _f1 and t _f2 respectively;

t _f is the time value of the insertion frame from the starting point;

θ is ... and is determined by q _f1 ·q _f1 = cos θ.

The feature data extracted in the face information at least includes: the detection of frontal face features and the detection of side face feature points; wherein,

The detection of front face features at least includes: rough positioning of facial features, key feature point detection and feature shape detection based on deformed templates;

The feature point detection of the side face includes at least: the extraction of the profile line of the face and the detection of the feature points of the face.

The rough positioning of facial features is as follows: first locate the position of the iris, and then obtain the position data of other organs of the face according to the position data of the central point of the iris, the statistical prior data of the facial organ structure and the facial gray distribution characteristics.

The detection of the key feature points of the face is: obtain the main feature points on the corners of the eyes, the corners of the mouth, and the chin curve, as the initial value of the corresponding organ template parameters; specifically include: the detection of the key points of the eyes, the detection of the key points of the mouth And the detection of the key points of the chin; among them: the key points of the eyes include the boundary points of the left and right corners of the eyes and the upper and lower eyelids; the key points of the mouth include the two corners of the mouth, the highest point of the upper lip and the lowest point of the lower lip; the key points of the chin include the left and right corners of the mouth The intersection point of the extension line of the chin and the chin, the intersection point of the vertical line passing through the middle lip point and the chin point, the intersection point of the vertical line passing through the left and right corners of the mouth and the chin point, the intersection point of a straight line passing through the left corner point of 45 degrees to the lower left and the chin, and the point passing through the right mouth corner point Go down to the right 45 degrees to the intersection of the straight line and the chin.

The feature shape detection based on the deformed template includes: detecting the feature shape of the eye area to obtain the eye template parameters; detecting the mouth shape to obtain the mouth template parameters; detecting the chin shape to obtain the chin template parameters.

The extraction of the profile of the human face is as follows: using the skin color feature of the human face to segment the human face area; then adopting edge detection, and locating the contour line according to the prior data of the human face profile.

The detection of the feature points on the side of the human face is as follows: with the tip of the nose as the boundary, the contour line of the human face is divided into upper and lower sections; by curve fitting, the approximate function expression of the contour line is obtained, and the first derivative of the function is calculated The point that is zero is used as the feature point of the face side.

The method of using feature data to synthesize and output face images is: define multiple feature points on the face model, the feature points can be extracted from the front and side images of a specific person, and the automatic extraction of feature points belongs to the face image Detection and analysis category; it is assumed that analysis and recognition technology has been applied to extract the required features or deformation curves from specific face images, and then use them as deformation parameters for general face models. The general face-neutral model is a 3D mesh, and the 3D coordinates of each feature point are known. During the modification process from the general face-neutral model to the specific face-neutral model, two transformations are required: first : It is necessary to perform an overall transformation on the neutral model of the general face, and complete the modification of the overall contour of the face so that it matches the shape of the face and the approximate position of the facial features of a specific person. Suppose the point coordinates on the face model before transformation are (x, y, z), and after transformation are (x′, y′, z′), and the coordinates of the center point of the face before and after transformation are o(x ₀ , y ₀ , z ₀ ) and o(x′ ₀ , y′ ₀ , z′ ₀ ), where the center point o of the face is defined as the intersection between the line connecting the corners of the two eyes and the longitudinal axis of the face. Parameters p, q1, and q2 are respectively defined as the distance from the center point to the sideburns, from the center point to the midpoint of the forehead, and from the center point to the chin; parameter u is defined as the distance from the center point of the mouth to the lower edge of the ear; parameters r1, r2, r3 On the side, it is defined as the distance from the highest point of the forehead to the hairline, the distance from the center point to the upper edge of the ear, and the distance from the corner of the mouth to the lower edge of the ear.

For the upper half of the face (the part above the eye level), use the following modified formula:

x′=x ₀ ′+ ^p′ / _p (xx ₀ )

y'=y ₀ '+ ^q1' / _q1 (yy ₀ )

z' = z ₀ '+ ^{r1 '} / _r1 (zz ₀ ) can be similarly modified for the middle and lower parts of the face.

The modification formula of the point on the eye area in the local transformation is as follows:

Let (x, y, z) be the coordinates of the eye region points before transformation, and (x′, y′, z′) be the coordinates of the eye region points after transformation, then:

x'=ax+by+cz

        y′=dx+ey+fz

Among them, a, b, c, d, e, f can be obtained by bringing in three sets of feature points before and after transformation, and solving six-element linear equations; after this transformation, the movement of the tiny position of the eye and the change of its shape can be achieved . It can be seen from the above formula that the modification only occurs in the x and y directions, without modification in depth, so the side information of the feature face cannot be well reflected. In the same way, eyebrows and mouth can be modified.

Let the coordinates of the nose area points before and after transformation be (x, y, z) and (x′, y′, z′), respectively, and the coordinates of the center point of the nose are (x ₀ , y ₀ , z ₀ ) and (x′ ₀ , y′ ₀ , z′ ₀ ), the transformation of the nose part is calculated according to the following formula:

x′=x ₀ ′+ ^p′ / _p (xx ₀ )

y'=y ₀ '+ ^q' / _q (yy ₀ )

z′=z ₀ ′+ ^r′ / _r (zz ₀ )

After the overall transformation and local transformation are completed, a neutral 3D face mesh with specific facial features is obtained.

The method for synthesizing and outputting the face image by using the feature data also includes: the lip model adopts two parabolas to fit the upper lip line, and one parabola to fit the lower lip line. Select two mouth corner points, the highest point of the two upper lip parabolas, the lowest point of the lower lip parabola and the intersection of the two upper lip parabolas; in addition, add several points on the lower lip parabola, upper lip parabola, and the connection line between the two mouth corner points The points become the points on the upper and lower inner lip parabolas respectively. The parabola of the outer contour of the lip satisfies the following formula:

ya(xb) ² +c

Among them, the coefficients a, b, c can be solved by bringing the coordinates of known points into the above equation.

The dynamic lip movement model uses the above-mentioned mouth opening model and is described as five interrelated parabolas, including two for the upper lip, one for the lower lip, and one for the upper and lower inner lips. In the parameter-driven lip motion synthesis, the corresponding lip opening state can be determined by giving the distance of the opening in the vertical and horizontal directions, or giving the corner points of the mouth and the highest points of the upper and lower lips.

The present invention provides a method for sign language translation through intermediate mode language, which translates sign language and non-sign language into an intermediate mode language, and then further translates the language of the intermediate mode into the required language form; sign language and non-sign language mode language Both correspond to the intermediate mode language, which is beneficial to the expansion of the sign language translation system and facilitates the mutual conversion between non-sign language languages and sign languages.

Below in conjunction with accompanying drawing and specific embodiment the present invention is described in detail further:

Description of drawings :

Fig. 1 is a schematic diagram of the basic principle of the present invention.

Fig. 2 is a schematic flow chart of translating sign language into intermediate mode language in the present invention.

Fig. 3 is a schematic flow chart of translating intermediate pattern language into sign language in the present invention.

Fig. 4 is a schematic diagram of the overall process structure of an embodiment of the present invention.

Specific implementation methods :

Referring to Fig. 1, the basic principle of the present invention is: use the intermediate model language _M as _the necessary way of conversion between sign language _MA and non-sign language language _MB ; Language M _B is translated into sign language M _A through an intermediate model language M.

Embodiment 1: Sign language is converted into voice output

Referring to Fig. 2 and Fig. 4, the specific method of translating sign language into non-sign language language through an intermediate mode language data form is:

First, collect sign language word data; In one embodiment of the present invention, adopt two data gloves with 18 sensors and its supporting equipment-position tracker as gesture input device, this position tracker is made up of a transmitter and several receivers The transmitter is composed of electromagnetic waves, and each receiver receives the electromagnetic waves, and then calculates the position and direction data of the receiver relative to the transmitter according to the received electromagnetic waves. A receiver is equipped on the left and right wrists of the human body performing sign language movements. Because the position of the transmitter is not fixed, the coordinates of the sign language data collected during the test often change, so a third receiver should be placed on the human body , using the position of the receiver as a reference point and a reference coordinate system, by referring to the position and direction data of the third receiver, the position and direction of the receivers on the left and right hands relative to the reference coordinate system are obtained as invariant features.

The specific method of extracting the feature information in the sign language word data is: calculate the position and direction of the left and right hands relative to the reference, normalize each component of the sensory data of each joint of the hand, and use the processed data as HMM Training samples and building a sign language sample model library.

As mentioned above, an HMM can be expressed with parameters: λ=(A, B, π),

Among them, A={a _ij } is the state transition probability matrix,

And satisfy the formula: a _ij =P[q _i+j =S _j |q _l =S _i ], 1≤i, j≤N;

And satisfy the constraints: a _ij ≥ 0, 1 ≤ i, j ≤ N, $\underset{j=1}{\overset{N}{\mathrm{\&Sigma;}}}{a}_{\mathrm{ij}}=1,$ 1≤i≤N;

In the above formula, N is the state number of the model;

π={π _i }, π _i represents the probability of starting from the i-th state node,

And satisfy the formula: π _i =P[q ₁ =S _i ], 1≤i≤N;

And satisfy the constraints: π _i ≥ 0, $\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{\mathrm{\&pi;}}_i=1,$ 1≤i≤N;

B={b _j (k)} is the probability density of the observed signal, when the observed symbol is a continuous vector, b _j (k) is a continuous probability density function, and: $b_j\left(k\right)=\underset{m=1}{\overset{m}{\mathrm{\&Sigma;}}}{c}_{\mathrm{jm}}G[{\mathrm{\&mu;}}_{\mathrm{jm}},{\mathrm{\&Sigma;}}_{\mathrm{jm}},o_k],1\&le;j\&le;N;$

Among them, N is the state number of the model, M is the number of mixing items, O _k is the observation vector at time k; c _jm is the mixing ratio (Mixing proportion), and it satisfies:

$\underset{m=1}{\overset{m}{\mathrm{\&Sigma;}}}{c}_{\mathrm{jm}}=1,$ 1≤j≤N, c _jm ≥0, 1≤j≤N, 1≤m≤M

Wherein: G is taken as the Gaussian probability density function, and μ _jm and ∑ _jm are respectively the mean vector and the covariance matrix of the mth component in the Gaussian mixture probability density;

$\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; jm</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; jm</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\sqrt{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}}{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; jm</span>}}<span\; class="notranslate">\; |</span>}^{\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}}}\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; [</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; jm</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; jm</span>}}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; jm</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ]</span>$

K groups of training data O=[O ⁽¹⁾ corresponding to the same gesture word, O ⁽²⁾ ,..., O ^(K) ], wherein $o^{\left(k\right)}--[o_1^{\left(k\right)}{o}_2^{\left(k\right)}...o_{T_k}^{\left(k\right)}]$ is the kth group of training data, T _k is the total number of frames of the kth group of training data;

The revaluation formulas of π _i , a _ij and c _jm , μ _m , ∑ _m are:

${\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; \&pi;</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; K</span>}}\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&gamma;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>$

That is, at time t=1, the expected probability of state node i appearing;

${\stackrel{\&OverBar;}{a}}_{\mathrm{ij}}=\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k-1}{\mathrm{\&Sigma;}}}{\mathrm{\&xi;}}_t^{\left(k\right)}(i,j)/\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k-1}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}_t^{\left(k\right)}\left(i\right),$ in,

The numerator part is the expected probability of transferring from state node i to state node j;

The denominator part is the expected probability of transferring from state node i;

${\stackrel{\&OverBar;}{c}}_{\mathrm{jm}}=\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}_t^{\left(k\right)}(j,m)/\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k}{\mathrm{\&Sigma;}}}\underset{m=1}{\overset{m}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}_t^{\left(k\right)}(j,m),$ in,

The numerator part is the expected probability of appearing in the mth branch of the state node j;

The denominator part is the expected probability of appearing at state node j;

${\stackrel{\&OverBar;}{\mathrm{\&mu;}}}_m=\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}_t^{\left(k\right)}(j,m)\&Center\; Dot;o_t^{\left(k\right)}/\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}_t^{\left(k\right)}(j,m),$ in,

The molecular part is the expected probability of the observation sequence 0 appearing in the mth branch of the state node j;

The denominator part is the expected probability that the mth branch of the state node j appears;

${\stackrel{\&OverBar;}{\mathrm{\&Sigma;}}}_m=\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}_t^{\left(k\right)}(j,m)\&Center\; Dot;(o_t^{\left(k\right)}-{\mathrm{\&mu;}}_m){(o_t^{\left(k\right)}-{\mathrm{\&mu;}}_m)}^{\&prime;}/\underset{k=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{t=1}{\overset{T_k}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}_t^{\left(k\right)}(j,m),$ in,

The molecular part is the expected probability of the mean square error of the observation sequence 0 appearing in the mth branch of the state node j;

The denominator part is the expected probability that the mth branch of state node j appears.

After extracting the feature information of the sign language word data, the sign language continuous sentence recognition is performed according to the feature information, and then the recognition result of the intermediate mode language data is recorded; the specific method is:

The semi-continuous hidden Markov model (Semi-Continuous Hidden Markov Model, referred to as SCHMM) is used to process the hand shape, position and direction separately, so as to reduce the number of codebooks, and then describe sign language by establishing a multi-dimensional letter string of position, direction and hand shape.

First, use a single data stream to build a continuous model for all words, and use six parts of data, including the left and right hand shape of the mean vector on all state nodes, the position of the right hand relative to the left hand, the direction of the right hand relative to the left hand, and the distance between the three receivers. Clustering, the steps of clustering are as follows:

Initialization: Randomly select multiple vectors from the training vector set as the initial codebook;

Find the closest codeword: For each training vector, find the closest codeword vector in the current codebook, and assign this vector to the set corresponding to the codeword;

Correction of the codeword: Correct the codeword to the mean value of all training vectors in the corresponding set;

The above two steps until the mean square error of the vectors in each category is lower than a given threshold.

After the cluster centers are obtained, the mean vectors of each state node of all models are quantized. Each state node only records the sequence number of the codebook closest to it. During identification, for each frame of identification data, it is only necessary to calculate the distance between it and each codebook, and use the distance to the codebook recorded by the node to replace the distance to the mean vector of the node.

After the above model library is built, use the Viterbi decoding method to calculate the likelihood probability of the test sample and various possible model sequences, and the word sequence corresponding to the model sequence with the largest probability value is the recognition result.

Suppose the capacity of the vocabulary is V, the model numbers of words are k=1,...,V, the corresponding model parameters are (π ^k , A ^k , c ^k , μ ^k , U ^k ), the state of each word The number is L, and the input sign language frame sequence number is i=1, 2,..., N;

When vocabulary is transferred within the same model (j > 1):

$\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; \}</span>$

T(i,j,k)=k

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; j</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; \}</span>$

When the vocabulary is transferred at the boundary of the model (j=1):

$\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; \&le;</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; \}</span>$

$\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; \&le;</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; \}</span>$

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; j</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; \&le;</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; \}</span>$

At the beginning (i=1), Pr(1,k)=p(1,1,k), Pr(j,k)=0, j>1

T(1,j,k)=-1

F(1,i,k)＝-1

The above formula can be used to recursively calculate each Pr(L, k), so that the global maximum probability can be obtained:

${\mathrm{<span\; class="notranslate">\; PR</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \}</span>$

The optimal path (T i , F i ) is backtracked from T(i, j, k) and F( _i , j, _k ) (in reverse order):

${\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; PR</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \}</span>$

F _X =L

T _i =T(i+1, T _j+1 , F _j+1 ), N-1≥i≥1

F _i =F(i+1, T _i+1 , F _i+1 ), N-1≥i≥1

Get the recognition result; among them,

p(i, j, k): the probability of the i-th frame appearing in the j-th state of word k;

Pr(j, k): until the current input frame i, the maximum probability of transitioning from the start state to the jth state of the word k:

T(i, j, k): record the serial number of the model where the previous frame is located;

F(i, j, k): Record the state of the previous frame in the model T(i, j, k).

In order to improve the recognition accuracy, the second-order Marko chain (Bigram) is embedded in the Viterbi search process, that is, the prior probability of the sentence can be calculated by the following formula:

$\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; W</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Pi;</span>}}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>$

in,

W is the recognized statement;

w ₁ , w ₂ ,...w _n are each word in the recognized sentence W;

P(w _i |wi _-1 ) is the occurrence frequency of word pairs.

Thus obtained can realize the conversion of the action information of the sign language into the sentence of the intermediate mode language.

When it is necessary to translate the input sign language into a non-sign language sentence (for example: voice sentence), then according to the corresponding relationship between the intermediate pattern language data and the corresponding non-sign language language, the intermediate pattern language data is converted into the non-sign language language word and output. In this embodiment, text is used as an intermediate mode language between sign language and non-sign language; take speech output as an example: when speech output is required, text in the text can be expressed by speech synthesis. Among them, the speech synthesis can be connected and broadcasted after the audio data of the isolated words are simply smoothed. The concrete speech synthesis method then adopts the existing system of synthesizing speech with text files.

Embodiment 2: Speech is converted into sign language output

Referring to Fig. 3 and Fig. 4, the specific method of translating non-sign language language into sign language through an intermediate mode language data form is:

First, collect non-sign language word data, and use the existing speech recognition technology to convert spoken language into intermediate mode language data; in this embodiment, the speech recognition can be realized using the speech data development tool developed by IBM.

Then, according to the corresponding relationship between the intermediate pattern language data and the non-sign language language, the non-sign language words are converted into intermediate pattern language data and recorded; in this embodiment, the adopted intermediate pattern language is stored as text.

Finally, according to the text, find the corresponding sign language word data in the sign language vocabulary database, and then synthesize the sign language word data into sign language image information output; the specific implementation includes: text input, analysis, segmentation, text representation The conversion of natural language to sign language code and the synthesis of sign language word data into sign language image information and other steps.

Among them, the input, analysis, and segmentation of the text can be realized by using the existing natural language recognition method, and then the segmented natural language words correspond to the corresponding sign language words in the sign language lexicon, and the sign language feature data of the sign language words are obtained . The sign language feature data is used to finally synthesize corresponding sign language images.

The concrete method of above-mentioned sign language image synthesis is:

First, a virtual human is established using the human body representation model of Virtual Reality Modeling Language (VRML); the angle values of each degree of freedom of the virtual human are determined, and the position and direction of each limb of the virtual human are calculated to determine the A gesture of the virtual human;

Since sign language is the movement of the upper limbs of the human body, and sign language movement is the projection of the movement of the human body on the joints of the upper limbs of the human body, when displaying sign language (that is, mapping sign language to a virtual human posture), a sign language movement representation can be expanded into a complete human movement representation; That is to say, the display of sign language can be performed using the general method of human body motion display, therefore, by ignoring the non-upper limb joint angles of the virtual human posture obtained in the above steps, a sign language posture of the virtual human can be obtained; or by ignoring the above-mentioned The non-upper limb joint angles of the virtual human posture obtained in the step can express a sign language posture of the virtual human as a complete human body movement posture.

After obtaining all the gesture data of the sign language movement, each gesture in a sign language movement is continuously displayed according to the specified time interval, and a corresponding sign language movement image is generated.

When generating the sign language moving image, smooth interpolation is further performed between adjacent frames of the sign language moving image; the specific interpolation is calculated according to the following formula:

${\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\frac{{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}}{{\mathrm{<span\; class="notranslate">\; t</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>$

in,

f ₁ and f ₂ are two adjacent image frames in a sign language movement;

t _f1 and t _f2 are the time values of f ₁ and f ₂ from the starting point respectively;

t _f is the time value of the inserted frame from the starting point;

t ₁ is the time value of the starting point;

G _i (t _f ) is the function value of the inserted degree of freedom curve;

G _i (t ₁ ) is the function value of the degree of freedom curve at the starting point;

G _i (t _f1 ) is the degree of freedom curve function of f ₁ ;

G _i (t _f2 ) is the function value of the degree of freedom curve of f ₂ .

When generating sign language motion images, a quadruple-based motion interpolation method is further used to perform smooth transition processing on complex joints in discontinuous sign language frames. The specific smoothing processing is calculated according to the following formula:

${\mathrm{<span\; class="notranslate">\; q</span>}}_{{\mathrm{<span\; class="notranslate">\; 1</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; sin</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; \&theta;</span>}}{\mathrm{<span\; class="notranslate">\; sin</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}}{\mathrm{<span\; class="notranslate">\; q</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; sin</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}\mathrm{<span\; class="notranslate">\; \&theta;</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; sin</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}}{\mathrm{<span\; class="notranslate">\; q</span>}}_{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

in,

f ₁ and f ₂ are image frames of two adjacent gestures in a sign language movement;

t _f1 and t _f2 are the time values of f ₁ and f ₂ from the starting point respectively;

q _f1 and q _f2 are the directions of joints at t _f1 and t _f2 respectively;

t _f is the time value of the insertion frame from the starting point;

θ is ..... and is determined by q _f1 , q _f2 = cos θ.

In the above two embodiments, only the recognition and synthesis methods of sign language movements are given. In fact, sign language usually also contains human facial expression information and lip movement information; moreover, when recognizing or synthesizing sign language, it is often necessary to describe the facial expression characteristics and lip movement information of the specific expression party.

Referring to Fig. 4, in the embodiment of the present invention, the specific face of the expressing party is further detected to obtain the corresponding facial features and lip movement information, and synthesized at the output end, and output synchronously with the sign language sentence. The specific detection and synthesis methods of face and lip movement information are as follows:

First, extract feature data in the face information, which at least includes: detection of frontal face features and detection of side face feature points; wherein,

The detection of front face features at least includes: rough positioning of facial features, key feature point detection and feature shape detection based on deformed templates;

The feature point detection of the side face includes at least: the extraction of the profile line of the face and the detection of the feature points of the face.

The rough localization of facial features is as follows: first locate the position of the iris of the human eye, and then obtain the positions of other organs of the human face according to the position data of the two iris center points, the statistical prior data of the facial organ structure and the facial gray distribution characteristics data.

The detection of the key feature points of the face is: obtain the main feature points on the corners of the eyes, the corners of the mouth, and the chin curve, as the initial value of the corresponding organ template parameters; specifically include: the detection of the key points of the eyes, the detection of the key points of the mouth and The detection of the key points of the chin; among them: the key points of the eyes include the left and right corners of the eyes and the boundary points of the upper and lower eyelids; the key points of the mouth include the two corner points of the mouth, the highest point of the upper lip and the lowest point of the lower lip; the key points of the chin include the extension of the left and right corners of the mouth The intersection point of the line and the chin, the intersection point of the vertical line passing through the middle lip point and the chin point, the intersection point of the vertical line passing through the left and right corners of the mouth and the chin point, the intersection point of the chin and the 45-degree line passing through the left corner point of the left mouth point, and the right side point passing through the right mouth corner point The intersection of the lower 45-degree line and the chin.

The feature shape detection based on the deformed template includes: detecting the feature shape of the eye area to obtain the eye template parameters; detecting the mouth shape to obtain the mouth template parameters; detecting the chin shape to obtain the chin template parameters.

The extraction of the profile of the human face is as follows: using the skin color feature of the human face to segment the human face area; then adopting edge detection, and locating the contour line according to the prior data of the human face profile.

The detection of the feature points on the side of the human face is as follows: with the tip of the nose as the boundary, the contour line of the human face is divided into upper and lower sections; by curve fitting, the approximate function expression of the contour line is obtained, and the first derivative of the function is calculated The point that is zero is used as the feature point of the face side.

In order to extract features from a specific face, 41 feature points can be defined on the face model. Here the feature points can be extracted from the front and side images of a specific person. The automatic extraction of feature points belongs to the category of face image detection and analysis. It is assumed that the analysis and recognition technology has been applied to extract the required features or deformation curves, which are then treated as deformation parameters for general face models. Since the general face-neutral model is a 3D mesh body, and the 3D coordinates of each feature point are known, two transformations are required during the modification process from the general face-neutral model to the specific face-neutral model. Firstly, an overall transformation should be performed on the neutral model of the general face. The purpose of the overall transformation is to complete the modification of the overall contour of the face so that it matches the shape of the face and the approximate position of the facial features of a specific person. Suppose the point coordinates on the face model before transformation are (x, y, z), and after transformation are (x′, y′, z′), and the coordinates of the center point of the face before and after transformation are o(x ₀ , y ₀ , z ₀ ) and o(x′ ₀ , y′ ₀ , z′ ₀ ). Among them, the center point o of the face is defined as the intersection point between the line connecting the corners of the two eyes and the longitudinal central axis of the face. Parameters p, q1, and q2 are respectively defined as the distance from the center point to the sideburns, from the center point to the midpoint of the forehead, and from the center point to the chin; parameter u is defined as the distance from the center point of the mouth to the lower edge of the ear; parameters r1, r2, r3 On the side, it is defined as the distance from the highest point of the forehead to the hairline, the distance from the center point to the upper edge of the ear, and the distance from the corner of the mouth to the lower edge of the ear.

For the upper part of the face (the part above the eye level), the following modified formula is used:

x′=x ₀ ′+ ^p′ / _p (xx ₀ )

y'=y ₀ '+ ^q1' / _q1 (yy ₀ )

z'=z ₀ '+ ^r1' / _r1 (zz ₀ )

The middle and lower parts of the face can be modified similarly.

The modification formula of the point on the eye area in the local transformation is as follows:

Let: (x, y, z) be the coordinates of the eye area points before transformation, (x′, y′, z′) be the coordinates of the eye area points after transformation, then we have

x'=ax+by+cz

y′=dx+ey+fz

Among them, the variables a, b, c, d, e, f can be obtained by bringing in the three sets of feature points before and after the transformation, and solving the six-element linear equations. After this transformation, the movement of the tiny position of the eye and its shape can be achieved. Change.

It can be seen from the above formula that the modification only occurs in the x and y directions, without modification in depth, so the side information of the feature face cannot be well reflected. In the same way, eyebrows and mouth can be modified.

The transformation formula for the nose part is as follows: Let the coordinates of the nose area points before and after the transformation be (x, y, z) and (x′, y′, z′), respectively, and the coordinates of the center point of the nose are (x ₀ , y ₀ , z ₀ ) and (x′ ₀ , y′ ₀ , z′ ₀ ),

x′=x ₀ ′+ ^p′ / _p (xx ₀ )

y'=y ₀ '+ ^q' / _q (yy ₀ )

z′=z ₀ ′+ ^r′ / _r (zz ₀ )

After the overall transformation and local transformation are completed, a neutral 3D face mesh with specific facial features is obtained.

The lip model in the face image uses two parabolas to fit the upper lip line, one parabola to fit the lower lip line, select two corner points of the mouth, the highest point of the two upper lip parabolas, the lowest point of the lower lip parabola, and the intersection point of the two upper lip parabolas . In addition, add two points on the parabola of the lower lip, add two points on the parabola of the upper lip, and add three groups of two overlapping points on the connection line between the two corner points of the mouth. For the mouth-opening form, each pair of coincident points separates and becomes each point on the upper and lower inner lip parabolas respectively. The parabolic equation for the outer contour of the lip satisfies:

y=a(xb) ² +c

Among them, the coefficients a, b, c can be solved by bringing the coordinates of known points into the above equation.

The dynamic lip movement model uses the above-mentioned mouth opening model and is described as five interrelated parabolas, including two for the upper lip, one for the lower lip, and one for the upper and lower inner lips. In the parameter-driven lip motion synthesis, the corresponding lip opening state can be determined by giving the distance of the opening in the vertical and horizontal directions, or giving the corner points of the mouth and the highest points of the upper and lower lips.

In Chinese, each phonetic unit that can be distinguished naturally in the language is a syllable. Usually a Chinese character is a syllable. Generally, a syllable is composed of an initial consonant and a final consonant. Mouth shape, in Chinese Pinyin, there are 19 initials and 39 finals, and the finals are divided into single finals, compound finals and nasal finals. When the single vowel is pronounced, the tongue position and lip shape remain unchanged throughout the pronunciation process, so it can be regarded as a mouth shape. Define several basic mouth shapes on the basis of the common mouth shapes in Chinese pronunciation, change the mouth shape parameters interactively, adjust the grid point position on the mouth area on the skin mesh body, form a mesh body representing the basic mouth shape and pre-set Store it down. According to the basic mouth shapes described above, a library of vowel mouth shapes can be derived, and the derivation rules are as follows:

(1) There are basic mouth shapes corresponding to the single final pronunciation

(2) For the pronouncing mouth shapes of compound finals and nasal finals, it can be split into a linear combination of multiple basic mouth shapes.

For all compound finals and nasal finals, the corresponding mouth shape parameters can be obtained, thus forming a finals mouth shape library. When synthesizing, find out the mouth shapes corresponding to the initials and finals according to the Chinese pinyin, and then synthesize them. If necessary, you can interpolate between the mouth shapes to smooth the changes of the lips.

Finally, it should be noted that the above embodiments are only used to illustrate and not limit the technical solutions of the present invention, although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the present invention can still be modified Or an equivalent replacement, any modification or partial replacement that does not depart from the spirit and scope of the present invention shall fall within the scope of the technical solutions claimed in the present invention.

Claims (27)

1. A method for translating sign language by intermediate mode language is characterized in that the sign language is translated into non-sign language by an intermediate mode language data form, and the method comprises the following specific steps:

step 101: collecting sign language word data;

step 102: extracting characteristic information in the sign language word data;

step 103: carrying out sign language continuous statement identification according to the characteristic information, and then recording the identification result of the intermediate mode language data;

step 104: and converting the intermediate mode language data into non-sign language words and outputting the non-sign language words according to the corresponding relation between the intermediate mode language data and the corresponding non-sign language.

2. The method of claim 1 for sign language translation using an intermediate mode language, wherein: the method for translating sign language into non-sign language further comprises: collecting corresponding face information while collecting sign language word data; then, extracting the characteristic data in the face information, and finally synthesizing the output face image when the characteristic data is used for translation output.

3. The method of claim 1 for sign language translation using an intermediate mode language, wherein: the specific method for collecting sign language word data comprises the following steps: collecting sensing data of each joint of the hand by using a data glove; inputting position and direction data of sign language gestures by adopting a position tracker; wherein, the data glove is arranged on the left hand and the right hand of the human body; the position tracker comprises a transmitter and more than one receiver; the transmitter sends out electromagnetic waves, the receiver is arranged on the left wrist and the right wrist of a human body, and the receiver receives the electromagnetic waves and calculates the position and the direction data of the receiver relative to the transmitter.

4. The method of claim 1 for sign language translation using an intermediate mode language, wherein: the specific method for extracting the characteristic information in the sign language word data comprises the following steps:

and calculating the positions and the directions of the left hand and the right hand relative to the reference, carrying out normalization processing on each component of the sensing data of each joint of the hands, taking the processed data as a training sample of the HMM, and establishing a sign language sample model library.

5. The method of claim 1 for sign language translation using an intermediate mode language, wherein: the specific method for recognizing the sign language continuous sentences comprises the following steps: and calculating the likelihood probability of the test sample and various possible model sequences by using a Viterbi decoding method, and taking the word sequence corresponding to the model sequence with the maximum probability value as an identification result.

6. The method of claim 5 for sign language translation using an intermediate mode language, wherein: the specific method for sign language continuous sentence recognition also comprises embedding a second-order Markov chain in the Viterbi search process based on a statistical language model, namely, the sentence W ═ W₁，w₂，…，w_nCan be calculated using the following equation:

$P\left(W\right)=P(w_1,w_2,\mathrm{\&Lambda;}{w}_n)=\underset{i=1}{\overset{n}{\mathrm{\&Pi;}}}P\left(w_i\right|w_{i-1});$

wherein,

w is the recognized sentence;

w₁，w₂，…w_nfor each word in the sentence W being recognized;

P(w_i|w_i-1) The frequency of occurrence of word pairs.

7. The method of claim 2 for sign language translation using an intermediate mode language, wherein: the extracting the feature data in the face information at least comprises the following steps: detecting front face features and side face feature points; wherein,

the detection of the front face features at least comprises the following steps: coarse positioning of facial features, detection of key feature points and detection of feature shapes based on a deformation template;

the detection of the characteristic points of the face at the side at least comprises the following steps: extracting a face side contour line and detecting face side characteristic points.

8. The method of claim 7 for sign language translation using an intermediate mode language, wherein: the coarse positioning of the facial features is as follows: the method comprises the steps of firstly positioning the position of an iris, and then obtaining position data of other organs of a human face according to position data of the center point of the iris, statistical prior data of the structure of the facial organs and the gray level distribution characteristics of the facial organs.

9. The method of claim 7 for sign language translation using an intermediate mode language, wherein: the detection of the key feature points of the human face is as follows: acquiring main characteristic points on an eye corner point, a mouth corner point and a chin curve as initial values of corresponding organ template parameters; the method specifically comprises the following steps: detecting eye key points, mouth key points and chin key points; wherein: the eye key points comprise left and right eye corner points and boundary points of upper and lower eyelids; the key points of the mouth part comprise two mouth corner points, an upper lip highest point and a lower lip lowest point; the key points of the chin comprise intersection points of extension lines of the left and right mouth corners and the chin, intersection points of a perpendicular line passing through the middle lip point and the chin, intersection points of a perpendicular line passing through the left and right mouth corner points and the chin, intersection points of a straight line passing through the left mouth corner point and 45 degrees left and lower and the chin, and intersection points of a straight line passing through the right mouth corner point and 45 degrees right and lower and the chin.

10. The method of claim 7 for sign language translation using an intermediate mode language, wherein: the characteristic shape detection based on the deformed template comprises the following steps: detecting the characteristic shape of the eye area to obtain eye template parameters; detecting the shape of the mouth to obtain mouth template parameters; and detecting the shape of the chin to obtain parameters of the chin template.

11. The method of claim 7 for sign language translation using an intermediate mode language, wherein: the extraction of the side contour line of the human face is as follows: segmenting a face region by using the skin color characteristics of the face; then edge detection is adopted, and contour lines are positioned according to prior data of the face contour.

12. The method of claim 7 for sign language translation using an intermediate mode language, wherein: the detection of the face side characteristic points is as follows: dividing the contour line of the human face into an upper section and a lower section by taking the nose tip point as a boundary; and obtaining an approximate function expression of the contour line through curve fitting, calculating a point with a first derivative of the function being zero, and taking the point as a face side characteristic point.

13. The method of claim 2 for sign language translation using an intermediate mode language, wherein: the method for synthesizing and outputting the face image by using the feature data comprises the following steps: defining more than one characteristic point on the face model, and taking the characteristic points as deformation parameters of a general face model; integrally transforming a general face neutral model to modify the whole outline of the face so as to match the whole outline with the positions of the specific face shape and five sense organs; and obtaining a neutral three-dimensional face mesh body with specific face features.

14. A method for sign language translation using an intermediate mode language according to claim 2 or 13, wherein: the method for synthesizing and outputting the face image by using the feature data further comprises the following steps: establishing a parameterized lip movement model according to the lip characteristic data, and finally synthesizing a corresponding mouth shape according to the lip movement model and the corresponding mouth shape of the language pronunciation; the method specifically comprises the following steps:

the lip model adopts two parabolas to fit an upper lip line, and one parabola is used to fit a lower lip line; selecting two mouth corner points, the highest points of two upper lip parabolas, the lowest point of a lower lip parabola and the intersection point of the two upper lip parabolas, adding two points on the lower lip parabola, adding two points on the upper lip parabola, and adding three groups of two coincident points on the connecting line of the two mouth corner points; when opening the mouth, the coincident points are separated and become points on the parabola of the upper and lower inner lips; and, the parabolic equation of the outer contour of the lip satisfies:

y＝a(x-b)²+c

and a, b and c can be obtained by substituting the coordinate values of each point of the lip line into the equation to solve.

15. A method for sign language translation through an intermediate mode language is characterized in that a non-sign language is translated into a sign language through an intermediate mode language data form, and the method specifically comprises the following steps:

step 201: collecting non-sign language word data;

step 202: converting the non-sign language words into intermediate mode language data and recording according to the corresponding relation between the intermediate mode language data and the non-sign language;

step 203: and finding corresponding sign language word data in the sign language word library according to the intermediate mode language data, synthesizing the sign language word data into sign language image information and outputting the sign language image information.

16. The method of sign language translation using an intermediate mode language according to claim 15, wherein: the method for translating the non-sign language into the sign language further comprises the following steps: collecting corresponding face information while collecting non-sign language word data; then, extracting the characteristic data in the face information, and finally synthesizing the output face image when the characteristic data is used for translation output.

17. The method of sign language translation using an intermediate mode language according to claim 15, wherein: the specific method for synthesizing the sign language word data into the sign language image information comprises the following steps:

establishing a virtual human by adopting a VRML human body representation model;

determining the angle value of each degree of freedom of the virtual human;

calculating the position and the direction of each limb of the virtual human, and determining a gesture of the virtual human;

ignoring the non-upper extremity joint angle for that pose;

and continuously displaying each gesture in one sign language motion according to a specified time interval to generate a corresponding sign language motion image.

18. The method of claim 17, wherein the method comprises: when generating the sign language moving image, further performing smooth interpolation between adjacent frames of the sign language moving image; the specific interpolation is calculated according to the following formula:

$G_i\left(t_{f^{\text{'}}}\right)=G_i\left(t_1\right)+\frac{t_{f^{\text{'}}}-t_{f_1}}{t_{f_2}-t_{f_1}}(G_i\left(t_{f_2}\right)-G_i\left(t_{f_1}\right))$

wherein,

f₁and f₂Two adjacent image frames in a sign language movement are respectively;

t_f1and t_f2Are respectively f₁And f₂A time value from a starting point;

t_f’is the time value from the starting point of the inserted frame;

t₁a time value as a starting point;

G_i(t_f’) Function values for the interpolated degree of freedom curves;

G_i(t₁) A degree of freedom curve function value as a starting point;

G_i(t_f1) Is f₁A degree of freedom curve function of;

G_i(t_f2) Is f₂Function value of the degree of freedom curve of (1).

19. The method of claim 17, wherein the method comprises: when generating the sign language moving image, the method further adopts a four-tuple-based motion interpolation method to carry out smooth transition processing on the complex joints in the discontinuous sign language frames, and the specific smooth processing is calculated according to the following formula:

$q_{t_{f^{\text{'}}}}=\frac{\sin (1-t_{f^{\text{'}}})\mathrm{\&theta;}}{\sin \mathrm{\&theta;}}{q}_{f_1}+\frac{\sin \left(t_{f^{\text{'}}}\mathrm{\&theta;}\right)}{\sin \mathrm{\&theta;}}{q}_{f_2}$

wherein,

f₁and f₂Respectively being image frames of two adjacent gestures in a sign language movement;

t_f1and t_f2Are respectively f₁And f₂A time value from a starting point;

q_f1and q is_f2Are respectively a jointt_f1，t_f2The direction of the time;

t_f’time value of inserting frame distance from starting point;

theta is $q_{f_1}\&CenterDot;q_{f_2}=\cos \mathrm{\&theta;}$ And (4) determining.

20. The method of claim 16, wherein the method comprises: the extracting the feature data in the face information at least comprises the following steps: detecting front face features and side face feature points; wherein,

the detection of the front face features at least comprises the following steps: coarse positioning of facial features, detection of key feature points and detection of feature shapes based on a deformation template;

the detection of the characteristic points of the face at the side at least comprises the following steps: extracting a face side contour line and detecting face side characteristic points.

21. The method of claim 20, wherein the method comprises: the coarse positioning of the facial features is as follows: the method comprises the steps of firstly positioning the position of an iris, and then obtaining position data of other organs of a human face according to position data of the center point of the iris, statistical prior data of the structure of the facial organs and the gray level distribution characteristics of the facial organs.

22. The method of claim 20, wherein the method comprises: the detection of the key feature points of the human face is as follows: acquiring main characteristic points on an eye corner point, a mouth corner point and a chin curve as initial values of corresponding organ template parameters; the method specifically comprises the following steps: detecting eye key points, mouth key points and chin key points; wherein: the eye key points comprise left and right eye corner points and boundary points of upper and lower eyelids; the key points of the mouth part comprise two mouth corner points, an upper lip highest point and a lower lip lowest point; the key points of the chin comprise intersection points of extension lines of the left and right mouth corners and the chin, intersection points of a perpendicular line passing through the middle lip point and the chin, intersection points of a perpendicular line passing through the left and right mouth corner points and the chin, intersection points of a straight line passing through the left mouth corner point and 45 degrees left and lower and the chin, and intersection points of a straight line passing through the right mouth corner point and 45 degrees right and lower and the chin.

23. The method of claim 20, wherein the method comprises: the characteristic shape detection based on the deformed template comprises the following steps: detecting the characteristic shape of the eye area to obtain eye template parameters; detecting the shape of the mouth to obtain mouth template parameters; and detecting the shape of the chin to obtain parameters of the chin template.

24. The method of claim 20, wherein the method comprises: the extraction of the side contour line of the human face is as follows: segmenting a face region by using the skin color characteristics of the face; then edge detection is adopted, and contour lines are positioned according to prior data of the face contour.

25. The method of claim 20, wherein the method comprises: the detection of the face side characteristic points is as follows: dividing the contour line of the human face into an upper section and a lower section by taking the nose tip point as a boundary; and obtaining an approximate function expression of the contour line through curve fitting, calculating a point with a first derivative of the function being zero, and taking the point as a face side characteristic point.

26. The method of claim 16, wherein the method comprises: the method for synthesizing and outputting the face image by using the feature data comprises the following steps: defining more than one characteristic point on the face model, and taking the characteristic points as deformation parameters of a general face model; integrally transforming a general face neutral model to modify the whole outline of the face so as to match the whole outline with the positions of the specific face shape and five sense organs; and obtaining a neutral three-dimensional face mesh body with specific face features.

27. A method for sign language translation using an intermediate mode language according to claim 16 or 26, wherein: the method for synthesizing and outputting the face image by using the feature data further comprises the following steps: establishing a parameterized lip movement model according to the lip characteristic data, and finally synthesizing a corresponding mouth shape according to the lip movement model and the corresponding mouth shape of the language pronunciation; the method specifically comprises the following steps:

the lip model adopts two parabolas to fit an upper lip line, and one parabola is used to fit a lower lip line; selecting two mouth corner points, the highest points of two upper lip parabolas, the lowest point of a lower lip parabola and the intersection point of the two upper lip parabolas, adding two points on the lower lip parabola, adding two points on the upper lip parabola, and adding three groups of two coincident points on the connecting line of the two mouth corner points; when opening the mouth, the coincident points are separated and become points on the parabola of the upper and lower inner lips; and, the parabolic equation of the outer contour of the lip satisfies:

y＝a(x-b)²+c

and a, b and c can be obtained by substituting the coordinate values of each point of the lip line into the equation to solve.

Images