CN107515674B - It is a kind of that implementation method is interacted based on virtual reality more with the mining processes of augmented reality - Google Patents

Abstract

The invention discloses a mining operation multi-interaction realization method based on virtual reality and augmented reality, which belongs to the technical field of virtual reality and augmented reality, and includes two modes of virtual reality and augmented reality. Selection and replacement of models and materials, scene roaming, arbitrary movement and placement of models, video embedding, generation of QR codes, triggers to achieve natural interaction, voice interaction, etc.; in augmented reality scenarios, you can select models, play voices, and demonstrate model operation Dynamic and control model rotation stop, screenshot and function expansion; in the two modes, various interactive modes of voice control, gesture control and keyboard and mouse control are realized. The present invention is applied to the virtual simulation application scene of mining operation, can be used to train mining workers in mining areas and students majoring in mining engineering, reduces training capital, improves workers' skills, and provides advanced and fast methods for guiding production construction and scientific and technological research means.

Description

A method for implementing multi-interaction in mining operations based on virtual reality and augmented reality

technical field

The invention belongs to the technical field of virtual reality and augmented reality, and in particular relates to a multi-interaction realization method for mining operations based on virtual reality and augmented reality.

Background technique

2016 is called "the first year of virtual reality" by the industry. Some people may mistakenly think that this technology is a new technology developed in recent years. In fact, virtual reality (Virtual Reality, referred to as VR) technology emerged in the 1990s. After 2000, virtual reality technology introduced advanced technologies such as XML and JAVA in the integrated development, and applied powerful 3D computing capabilities and interactive technologies. , improve rendering quality and transmission speed, and enter a new era of development. Virtual reality technology is the product of the development of economic and social productivity, and has broad application prospects. Research on virtual reality technology in my country started in the early 1990s. With the rapid development of computer graphics and computer system engineering, virtual reality technology has been given considerable attention. The "Research Report on VR User Behavior in China in the First Half of 2016" jointly released by the National Advertising Research Institute and other institutions shows that in the first half of 2016, the potential users of virtual reality in China reached 450 million, with about 27 million shallow users and about 2.37 million heavy users. , It is expected that the domestic virtual reality market will usher in explosive growth. The augmented reality (Augmented Reality, referred to as AR) technology is a new technology developed on the basis of virtual reality. Its application fields are also very wide, and it has shown good application prospects in fields such as industry, medical treatment, military affairs, municipal administration, television, games, and exhibitions.

At present, VR and AR technologies continue to develop, and the scope of application is becoming wider and wider. However, these two technologies are more used in military, entertainment and other fields. For applications in education, industry, engineering and other fields, because the field itself involves many Multidisciplinary factors such as physics and geography still need more research and development. For the field of mining industry, the geological conditions of mines in my country are relatively complex, and most of them are mined underground. During the mining process, because the mining environment is located underground and the process is quite complicated, disasters such as gas and water damage occur from time to time. At the same time, mining is an industry with long construction period, large investment and high safety hazards, and safety accidents are prone to occur. Therefore, the safety training of mining employees has always been the top priority of mining work. However, the existing traditional training and teaching system is basically theoretical introduction plus mold display or two-dimensional image display, mainly classroom explanation, supplemented by simple animation, audio and video introduction, lack of practice and lack of real scenes. Even watching the mold is not a good grasp of the actual operation process of the tool. With the continuous development of technology, various training systems for coal mining have also been developed accordingly, but there are also problems such as poor authenticity of the system scene, poor immersion effect, and few interactive functions, which can only be demonstrated simply.

Contents of the invention

Aiming at the above-mentioned technical problems existing in the prior art, the present invention proposes a multi-interaction realization method for mining operations based on virtual reality and augmented reality, which has a reasonable design, overcomes the deficiencies of the prior art, and has good effects.

In order to achieve the above object, the present invention adopts the following technical solutions:

A multi-interaction realization method for mining operations based on virtual reality and augmented reality, using a multi-interaction simulation system for underground mining operations, the system includes two modes: virtual reality mode and augmented reality mode; virtual reality mode includes modeling, roaming , model and material replacement, video embedding into virtual scene, model movement, application scene interaction, QR code generation and voice interaction; augmented reality mode includes model selection, model explanation, dynamic model demonstration, gesture control model interaction, screenshot generation Icons, 360-degree rotation and stop, function mode switching and function expansion; the system has designed two hidden menus, that is, model replacement in virtual reality mode, material selection menu and model selection menu in augmented reality mode; the first one The menu will only be displayed when the user enters a specific area, and will be hidden when the user leaves; the second click can display the secondary menu somewhere, and click the menu again to hide;

The multi-interactive implementation method of the mining operation specifically includes the following steps:

Step 1: Construction of the entire environment scene for mining operations

According to the real environment of underground mining operations, use the modeling tool 3DMax to carry out 1:1 scale modeling to realize the environmental simulation of the entire underground mining operation; use the UE4 engine to edit the model including creating, editing textures and materials, adding Physical collision, adding lighting, effect lighting and special effects to the overall environment, and baking and rendering;

Step 2: Roaming of virtual reality application scenarios

In the UE4 engine, set up, down, left, and right keys on the keyboard, bind the Up, Down, Right, and Left direction control functions, and bind the Turnaround control function for the mouse to realize the roaming of the virtual reality scene of the entire underground mining operation;

Step 3: Replacement of tool models for underground mining operations and simulated materials for mine geology

Add a hidden menu in the virtual underground mining scene. When roaming to the mining site, a model or material selection menu will automatically appear. Users can select a model or material from the menu to replace according to their needs;

Step 4: Embed the video material into the 3D application scene and control the playback and stop

Embed the video material into the virtual reality scene, play it in three-dimensional space, simulate the monitoring and display device of the mining environment, set the keyboard X key, bind the MediaPlayer media class of the UE4 platform, and control the playback and stop of the video through the OpenSource and Close functions;

Step 5: Select the model and move it anywhere

Select the model with the mouse and move the model to any position that needs to be simulated to achieve mechanical movement simulation in the real scene;

Step 6: Realize the intentional interaction of the application scenario

When the user roams to a specific location in the virtual reality application scene, the system detects that the user intends to enter, and automatically turns on the ambient light to realize natural interaction in the virtual scene;

Step 7: QR code generation

Bind the F key of the keyboard, add a QR code generation function, set the keyboard key to control the function of generating a QR code, and the user presses the F key on the keyboard, and the system generates a QR code with a virtual scene panorama with the set sampling points;

Step 8: Implement Voice Interaction

The user controls the coal shearer in the virtual reality scene through keywords including forward rotation, reverse rotation, raising arm, lowering arm, and stop, and simulates its operation effect;

Step 9: AR Dynamic Demonstration Function Mode Switching

The user clicks the AR mode button in the upper right corner of the system to switch to the AR demonstration mode.

Preferably, in step 3, instantiate the model as a specific Actor, add the SetMesh function and SetMaterial function to replace the model and model material, set the Widget Blueprint user interface and Boxcollision collision detection, and realize the hidden menu function of the three-dimensional space.

Preferably, in step 5, add a mouse event for the model to be operated, select the model through the GetHitResult function, and then change the coordinate value of the SetActorLocation function of the model according to the coordinates of the mouse in the three-dimensional space. When the mouse clicks again, the The coordinates of the mouse in the x, y, and z directions are assigned to the model, and the GetHitResult function sets the model to the unselected mode.

Preferably, in step 6, a TriggerBox trigger is set. When the first-person character triggers the TriggerBox and the system detects that the user intends to enter a certain area, a certain device in this area will be automatically enabled.

Preferably, in step 7, the user presses the F key on the keyboard, and the system generates a two-dimensional code of the virtual scene panorama with the set sampling points. On the mobile phone, the user activates the gyroscope, switches to the VR split-screen mode, sets the mobile phone parameters, and then uses the VR glasses to experience the virtual underground mining operation environment scene, realizing a 720-degree viewing angle display, and also realizing multi-scene and multi-angle viewing on the mobile phone. roaming experience.

Preferably, in step 8, speech recognition is implemented based on the Pocket-sphinx library, by improving the Chinese keyword dictionary, through preprocessing, feature extraction, acoustic model training, language model training, speech decoding and search to realize the recognition function, and finally through UE4 The writing function of the engine controls the function to realize the voice control of the model in the three-dimensional space; the specific implementation steps of the voice recognition are as follows:

Step 8.1: Preprocessing

Process the input original voice signal, filter out the unimportant information and background noise, and process the endpoint detection, voice framing and pre-emphasis of the voice signal;

Pre-emphasis is realized by a first-order FIR high-pass digital filter, and the transfer function of the first-order FIR high-pass digital filter is:

H(z)=1-az ^-1 ;

Among them, a is the coefficient of the pre-emphasis filter, and the value range is 0.9 to 1.0. If the speech sampling value at time n is x(n), the signal after pre-emphasis is

y(n)=x(n)-a*x(n-1);

Step 8.2: Feature Extraction

Feature extraction is performed by the method of Mel Frequency Cepstral Coefficient (MFCC); the specific steps are as follows:

Step 8.2.1: Using the critical band effect of human hearing, MEL cepstrum analysis technology is used to process the speech signal to obtain the vector sequence of MEL cepstrum coefficients;

Step 8.2.2: represent the frequency spectrum of input speech with MEL cepstral coefficient vector sequence, set several band-pass filters with triangular or sinusoidal filtering characteristics in the range of speech spectrum;

Step 8.2.3: Find the output data of each band-pass filter through the band-pass filter bank;

Step 8.2.4: take the logarithm of the output data of each bandpass filter, and do discrete cosine transform (DCT);

Step 8.2.5: Obtain the MFCC coefficient; the solution formula is as follows:

Wherein, C _i is a feature parameter, k is a variable, 1≤k≤P; P is the number of triangular filters, F(k) is the output data of each filter, and i is the data length;

Step 8.3: Acoustic Model Training

Acoustic model parameters are trained according to the characteristic parameters of the training speech library;

During recognition, the characteristic parameters of the speech to be recognized are matched with the acoustic model to obtain the recognition result; the acoustic model is realized by using a Gaussian mixture model-hidden Markov model (GMM-HMM), which specifically includes the following steps:

Step 8.3.1: Find the joint probability density function of the mixed Gaussian model:

Among them, M represents the number of Gaussians in the mixed Gaussian model, C _m represents the weight, u _m represents the mean value, ∑ _m represents the covariance matrix, D is the dimension of the observation vector; Θ={C _m , u _m , ∑ _m } to estimate, use the following formula to solve:

Among them, j is the number of current iteration rounds, N represents the number of elements in the training data set, x ^(t) is the feature vector at time t, h _m (t) represents the posterior probability of C _m at time t; GMM parameters are passed through the EM algorithm make an estimate that maximizes its probability of generating speech observation features on the training data;

Step 8.3.2: Solve the three main components of the HMM

Let the state sequence be q ₁ , q ₂ ,…,q _N , and let the transition probability matrix A=[a _ij ]i,j∈[1,N], then the calculated transition probability between the Markov chain states is : a _ij ＝P(q _t ＝j|q _t-1 ＝i); the initial probability of the Markov chain π＝[π _i ]i∈[1,N], where π _i ＝P(q ₁ ＝ i); make the observation probability distribution of each state b _i (o _t )=P(o _t |q _t =i), and use the GMM model to describe the observation probability distribution of the state; according to step 8.3.1, the solution formula is:

Among them, N is the number of states, i and j represent states, a _ij represents the probability of jumping from state i to state j at time t-1 at time t-1, o _t is the observed value at time t, C _i,m is the mixing coefficient , represents the weight between different Gaussians, u _{i, m} represents the mean value between different Gaussians, ∑ _{i, m} represents the covariance matrix between different Gaussians; the parameters of HMM are estimated by the Baum-Welch algorithm, and finally generated Acoustic model files;

Step 8.4: Language Model Training

The N-Gram model is used to train the language model; the probability of the i-th word appearing in a sentence depends on the N-1 words before it, that is, the context of a word is defined as the N-1 words that appear before the word words whose expression is:

Use the conditional probability formula S to replace the above expression with the following formula:

P(sentence)＝P(w ₁ )*P(w ₂ |w ₁ )*P(w ₃ |w ₂ )…*P(w _n |w ₁ ,w ₂ ,…,w _n-1 )

Among them, P(w ₁ ) is the probability of w ₁ appearing in the article, P(w ₁ ,w ₂ ) is the probability of w ₁ and w ₂ appearing continuously, P(w ₂ |w ₁ ) is the known w ₁ has The probability of w ₂ appears in the case of occurrence, assuming that the probability of recognizing sentence is represented by P(s), P(s)=P(w ₁ ,w ₂ ,…,w _n ) means the word set w ₁ ,w ₂ ,… , the probability that w _n appears continuously and generates S;

Simplified by the Markov assumption into the following formula:

P(sentence)＝P(w ₁ )*P(w ₂ |w ₁ )*P(w ₃ |w ₂ )…*P(w _n |w _n-1 )

Among them, P(w _i |wi _-1 )=P(wi _-1 , _wi )/P( _wi ), P(wi _-1 , _wi ) and P(W _i ) are calculated from the corpus , and finally P(sentence) can be obtained, the language model stores the probability statistics of P(wi _-1 , _wi ), and the entire recognition process is realized by finding the maximum value of P(sentence);

Step 8.5: Speech decoding and search algorithm

For the input speech signal, build a recognition network according to the trained acoustic model, language model and dictionary mapping file created by the g2p tool, and find the best path in the network according to the search algorithm. This path is able to Output the word string of the speech signal with the maximum probability, so that the text contained in the speech sample is determined, and the speech decoding is realized by the Viterbi algorithm. The specific process is as follows:

Step 8.5.1: Input the parameters of the HMM model and the observation sequence O={o ₁ ,o ₂ ,…,o _T }, then all state probabilities at t=1 are:

δ ₁ (i) = π _i b _i (o ₁ )

ψ ₁ (i) = 0

Step 8.5.2: Gradually deduce to t=2,3,...,T, then:

Step 8.5.3: Terminate the traversal:

Step 8.5.4: backtrack the optimal path, t=T-1, T-2,...,1;

Step 8.5.5: Output the optimal hidden state path

Among them, δ _t (i) is the joint probability of all nodes passed by the optimal path at time t, ψ _t (i) is the hidden state at time t, T is time, and P ^* is the probability of the optimal path , is the endpoint of the optimal path.

Preferably, a is 0.97.

Preferably, in step 9, the following steps are specifically included:

Step 9.1: Model Selection

Select the shearer model, roadheader model, wind coal drill model and fully mechanized mining support model, each type of model is a 1:1 modeling simulation of real coal mining tools;

Step 9.2: Model Explanation

After selecting the model, the user selects the tool model option to be learned through the model selection menu in the augmented reality mode, and the system will play the corresponding voice explanation, and click the button again to stop the voice;

Step 9.3: Model presentation

Import the tool simulation running animation made in the 3DMax modeling process into the Unreal Engine engine, set the corresponding selection menu, and click to demonstrate the operating status of the corresponding coal mining tool in AR mode;

Step 9.4: Take screenshot to generate icon

In the main menu of AR mode, add a button to bind the screenshot function of the camera, and add a scrolling menu bar on the right side of the menu. When the screenshot function is successfully triggered, the screenshot will be displayed on the right scrolling menu bar through the set dynamic material conversion function. , during the demonstration, if the user clicks the screenshot button, the system will generate an icon on the side of the interface;

Step 9.5: Rotate

Instantiate the set model as an Actor, add the Rotation function, and realize the clockwise rotation of the model;

Step 9.6: Function Extension

Add a secondary UI to control the map switch, and realize the operation demonstration function including the earth, Saturn, Mercury, planets with atmospheres, and galaxies; add WidgetBlueprint code to realize the display or hide of the knowledge introduction panel; design the return key to return to AR editing main module;

Step 9.7: Control the model with dynamic gestures, superimpose the real environment and the virtual model, and interact with the model by gestures, including the following steps:

Step 9.7.1: Initialize the video capture, read the logo file and camera parameters;

Step 9.7.2: Grab the video frame image;

Step 9.7.3: Execute the detection mark and identify the mark template in the video frame, and use the OpenCV library function to perform motion detection on the acquired video frame image, and judge whether a motion track is detected;

If: the judgment result is that the motion track is detected, then perform step 9.7.4:

Or if the judgment result is that no motion track is detected, then continue to perform the detection mark and identify the mark template in the video frame, and then perform step 9.7.12;

Motion detection is performed based on the color histogram and the background difference, and the background update is performed on the collected frames and the pixels other than the motion gesture area obtained after the motion detection of each frame. The formula is as follows;

Among them, u _t is the corresponding pixel of the background image, u _t+1 is the updated background image pixel; I _t is the pixel of the current frame image, If _is the mask value of the current frame image pixel, that is, whether Do background update; a∈[0,1] is the update speed of the background image model;

Step 9.7.4: Preprocessing the image including denoising;

Through the motion detection step, if motion information is detected, the video frame image containing motion gestures is preprocessed: the image is median-filtered through the medianBlur function of OpenCV to remove the salt and pepper noise;

Step 9.7.5: Convert to HSV space;

Use the cvtColor function to convert the color space of the image to obtain its HSV space data, and reset the brightness v value in the HSV space as shown in the following formula:

Among them, r and g are the red and green pixels in the skin color area, and r>g;

Step 9.7.6: Segment the hand region;

Step 9.7.7: Carry out morphological processing to remove impurities;

Comparing the obtained motion binary image with the binary image obtained by back-projection, and performing image morphology closing operation to obtain a relatively complete binary image of motion skin color gesture; and removing noise points in the image;

Step 9.7.8: Obtain hand contour;

After preliminary morphological operations, noise is removed, and the boundary of the hand is made clearer, the gesture contour is obtained through OpenCV's findContours function, and then the false contour is removed;

Step 9.7.9: Draw the outline of the hand and calibrate the information;

Step 9.7.10: Comparing the contour information and setting the direction vector;

Compare the contours obtained in each frame, set the comparison condition, and assign a value to the direction flag variable by comparison;

Step 9.7.11: Simulate the force on the model according to the vector coordinates to realize the interaction between dynamic gestures and the virtual model;

After the dynamic gesture is judged by the outline, the virtual model is subjected to a force simulation operation according to different judgment results. According to the value of the direction mark in the outline judgment process, the coordinate values of the model in the three-dimensional space will be carried out on the three coordinate axes of x, y, and z. The multiplication calculation on the above, through the change of the coordinate value, the change of the model position is realized to achieve the simulation of the force;

Step 9.7.12: Calculate the transformation matrix of the camera relative to the detected marker;

Step 9.7.13: superimpose the virtual object on the detected mark, and return to step 9.7.2 to realize the superimposed display of the real environment and the virtual model.

Beneficial technical effects brought by the present invention:

(1) The 3D model of the present invention is built in equal proportions, the texture map is close to the real through the editing of the UE4 engine platform, and the ambient lighting of the application scene is rendered by simulated baking of real lighting. The entire virtual reality scene is more real and has a strong sense of immersion.

(2) The present invention realizes a variety of functional interactions through technical solutions, such as changing the tool model through the hidden menu during the roaming process of the virtual underground mining scene, changing the mine material to simulate different mining geology, freely moving the position of the mining tool, and The video information is embedded in the display screen of the machine to show the real scene, and the voice function is used to control the forward rotation, reverse rotation, lifting arm, lowering arm, stop, etc. of the shearer.

(3) The present invention also connects the display on the PC to the display on the mobile phone through the function of generating a two-dimensional code. The function of the mobile phone can use the built-in gyroscope of the mobile phone to generate gravity induction. If it is set to the VR glasses mode, it can be used easily. VR glasses to experience real-time scene immersion.

(4) The present invention also utilizes the AR development SDK—ARToolKit to realize the AR dynamic demonstration function. Through the AR editing demonstration function, the user can select the mining tool model in real time, perform 360 rotation display, voice explanation, dynamic operation display, screenshot saving, etc., and more The most important thing is that it displays the tool model in AR mode, and the display effect of combining the virtual model with the real environment, which can not only show the intuitive three-dimensionality of the model, but also show its authenticity, so that it has better learning and educational functions. Effect.

(5) The AR module of the present invention, in addition to its dynamic demonstration function, adds processing to the video stream. When the dynamic gesture enters the camera's perspective, it will interact with the model, and the dynamics of the hand from far to near will be passed to the model. A simulated force forward in a three-dimensional space, the dynamic from top to bottom will give the model an upward simulated force, and the dynamics of turning the hand forward will give the model a downward simulation. Similarly, if the hand twists or left and right Tilting gives the model a simulated force with a vector direction.

(6) In addition to the function realization of the AR module in the coal mine application scene, the present invention also expands the display function of AR in the field of astronomy. Add the AR display function of the earth, Saturn, Mercury, planets with dynamic atmospheres, and galaxies. At the same time, add the knowledge briefing panel display function to this AR display module, which enriches the application of AR in the field of educational display.

Description of drawings

Fig. 1 is an overall functional structure diagram realized by the present invention.

Fig. 2 is a schematic diagram of the function of generating a two-dimensional code in the present invention.

Fig. 3 is a schematic diagram of the realization of the interaction function of the speech recognition of the present invention.

FIG. 4 is a schematic diagram of the realization of the AR mode of the present invention.

Fig. 5 is a flowchart of the realization of the dynamic gesture interaction function of the present invention.

Detailed ways

Below in conjunction with accompanying drawing and specific embodiment the present invention is described in further detail:

The invention provides a mining operation multi-interaction realization method based on virtual reality and augmented reality. The entire technical function included in the present invention can be understood in conjunction with accompanying drawing 1. Its specific implementation steps are as follows:

Step 1: Construction of the entire environment scene for underground mining operations. Utilize 3DMax modeling tools to create relevant models based on real mining operating environments. Import the models into the UE4 engine by category, and use the UE4 platform to write materials for the models, simulate natural light and ambient light, add physical collision detection, adjust the parameters of the system, and bake and render.

Step 2: Add a first-person character in the virtual application scene, and add mouse and keyboard control events to the character. Bind the up, down, left, and right keys of the keyboard to the Up, Down, Right, and Left functions to control the coordinate change of the first-person character in the virtual three-dimensional space to realize roaming. Add the Turnaround function to the mouse to control the 720-degree rotation of the first-person perspective in the virtual three-dimensional space.

Step 3: Set up the interactive menu to realize functional interaction such as changing tool models and mine geological materials for underground mining operations. First create a Widget Blueprint user interface, set the menu options, and add click events for the options. Then add a Box collision collision detection area to the model, when the character enters the Box collision collision detection area. The created Widget Blueprint UI is displayed. Leaving the Box collision collision detection area, the WidgetBlueprint user interface is hidden. Instantiate the shearer model as an Actor, and add the SetMesh function to replace other tool models. In the same way, add the SetMaterial function to the mine geological model in the three-dimensional space to realize the replacement of materials. The present invention sets four types of mining tool models for users to choose, and sets the mine geology into a material selectable mode, and replaces the models and materials through the displayed style menu. After the replacement, leave the detection area, the menu is automatically hidden, without affecting the overall roaming visual effect, and can achieve the function of real-time interaction.

Step 4: Video embedding, playing in three-dimensional space, simulating the monitoring and display equipment of the mining environment. The invention sets the keyboard X key to be bound to the MediaPlayer media class of the UE4 platform, and controls the playing and stopping of the video stream through the Open-Source and Close functions. This operation can simulate the screen display of underground mine mining control equipment and the screen display of real-time environmental monitoring, highlighting the authenticity and dynamics of the 3D scene and making the simulated virtual scene closer to reality.

Step 5: The selected model can be dragged to any position that the user wants to place, and the intentional interaction function that the device is automatically turned on can be realized. Add a mouse event for the model to be operated, select the model through the GetHitResult function, and then change the coordinate value of the SetActorLocation function of the model according to the coordinates of the mouse in the three-dimensional space. When the mouse is clicked again, the coordinates of the mouse in the x, y, and z directions are assigned to the model, and the GetHitResult function sets the model to the unselected mode. In this embodiment, the user can click on the shearer model in the scene and place it in other mining positions in the mining operation scene.

The system adds a TriggerBox trigger in a specific area, the first-person character enters this area, triggers the TriggerBox trigger, and correspondingly triggers the ambient light control function SetVisible in the next area, and the light is turned on, thereby realizing the automatic sensor light function set by the present invention. This is also the function of detecting human intention designed by the present invention, so as to realize more natural system interaction.

Step 6: QR code generation function. A single PC-side display cannot satisfy the experience of multiple users. This invention generates by adding a QR code, and scanning the QR code can realize the display of multi-user mobile phones. Through the connection of the QR code, the mobile phone jumps to the panoramic display of coal mining operations page. On the mobile phone, the user can enable the gyroscope, switch to the VR split-screen mode, set the mobile phone parameters, and use the VR glasses to experience the virtual underground coal mining environment, realizing a 720-degree perspective display. At the same time, a multi-scenario and multi-angle roaming experience on the mobile phone can be realized. This function is mainly to add QR code generation and hiding functions by binding the F and V keys of the keyboard. Add 6 point collection points of the scene in the UE4 engine, generate a panorama through the location of the collection points, and then generate a network connection in the form of a QR code to generate information and related mobile terminal settings to achieve end-to-end conversion. The process of realizing this function is shown in Figure 2.

Step 7: Realize the voice control function. The invention utilizes Pocket-sphinx to realize Chinese keyword recognition. The specific voice control implementation principle flow is shown in Figure 3. The present invention adds a voice recognition function to the Actor created with the shearer model, and enables the voice recognition class after system initialization, and saves references to this class. Then create and bind a method to the speech recognition function OnWordSpoken. Whenever the user speaks the set control words, this method will be triggered, and the forward rotation, reverse rotation, lifting arm, and lowering of the coal shearer will be realized through keyword matching. Arm and stop and other related controls. The speech recognition realized by this method is based on the improvement of the English speech recognition system Sphinx developed by Carnegie Mellon University in the United States. The voice recognition method of the present invention is an isolated word recognition method for a large number of vocabulary, non-specific people, and continuous Chinese syllables. Able to recognize set words from different people very well. Finally, through the encoding technology of UE4, the triggering of the action control function corresponding to the matching word after speech vocabulary recognition is realized, and the corresponding action control of the model is realized. This recognition system includes five parts: speech preprocessing, feature extraction, acoustic model training, language model training and speech decoding. The following is the specific process of speech recognition:

Step 7.1: Preprocessing.

Process the input original voice signal, filter out the unimportant information and background noise, and perform endpoint detection, voice framing, pre-emphasis and other processing of the voice signal. The purpose of the pre-emphasis of the voice signal is to emphasize the high-frequency part of the voice, remove the influence of lip radiation, and increase the high-frequency resolution of the voice. Generally, pre-emphasis is realized through a first-order FIR high-pass digital filter with a transfer function of H(z)=1-az ^-1 . a is the coefficient of the pre-emphasis filter, and its value ranges generally from 0.9 to 1.0. This paper takes 0.97. Let the speech sampling value at time n be x(n), and the signal after pre-emphasis is

y(n)=x(n)-a*x(n-1)

Step 7.2: Feature Extraction.

This paper uses the method of Mel frequency cepstral coefficient (MFCC) to extract. MFCC parameters are based on human auditory characteristics. He utilizes the critical band effect of human hearing and uses MEL cepstrum analysis technology to process speech signals to obtain MEL cepstrum coefficient vector sequences, and use MEL cepstrum coefficients to represent the spectrum of input speech. Several band-pass filters with triangular or sine-shaped filter characteristics are set within the voice spectrum range, then the voice energy spectrum is passed through the filter bank, each filter output is sought, logarithm is taken to it, and discrete cosine transform ( DCT), the MFCC coefficients can be obtained. The solution formula is as follows:

Among them, C _i is a characteristic parameter, k is a variable, 1≤k≤P; P is the number of triangular filters, F(k) is the output data of each filter, and is the data length.

Step 7.3: Acoustic model training.

Acoustic model parameters are trained according to the characteristic parameters of the training speech library. During recognition, the characteristic parameters of the speech to be recognized can be matched with the acoustic model to obtain the recognition result. In this paper, Gaussian Hybrid Model-Hidden Markov Model (GMM-HMM) is used as the acoustic model.

Step 7.3.1: Find the joint probability density function of the mixture Gaussian model:

Among them, M represents the number of Gaussians in the mixed Gaussian model, C _m represents the weight, u _m represents the mean value, ∑ _m represents the covariance matrix, and D is the dimension of the observation vector. Use the maximum expectation algorithm (EM) to estimate the parameter variables of the mixed Gaussian model: Θ={C _m , u _m , ∑ _m }, and use the following formula to solve it:

Among them, j is the current iteration number, N represents the number of elements in the training data set, x ^(t) is the feature vector at time t, h _m (t) represents the posterior probability of C _m at time t. The GMM parameters are estimated by the EM algorithm, which maximizes its probability of generating speech observation features on the training data.

Step 7.3.2: Solve the three main components of the HMM.

Let the state sequence be q ₁ , q ₂ ,…,q _N , and let the transition probability matrix A=[a _ij ]i,j∈[1,N], then the calculated transition probability between the Markov chain states is : a _ij ＝P(q _t ＝j|q _t-1 ＝i); the initial probability of the Markov chain π＝[π _i ]i∈[1,N], where π _i =P(q ₁ = i); let the observation probability distribution of each state b _i (o _t )=P(o _t |q _t =i), use the GMM model to describe the observation probability distribution of the state; according to step 7.3.1, the solution formula is:

Among them, N is the number of states, i and j represent states, a _ij represents the probability of jumping from state i to state j at time t-1 at time t-1, o _t is the observed value at time t, C _i, m is the mixing coefficient , represents the weight between different Gaussians, u _i, m represents the mean value between different Gaussians, ∑ _i, m represents the covariance matrix between different Gaussians; the parameters of HMM are estimated by the Baum-Welch algorithm, and finally generated Acoustic model files;

Step 7.4: Language model training.

The language model is used to constrain the word search. Language modeling can effectively combine the knowledge of Chinese grammar and semantics to describe the internal relationship between words, thereby improving the recognition rate and reducing the search scope. In this paper, the N-Gram model is used to train the language model. The probability that the i-th word appears in a sentence depends on the N-1 words before it, that is, the context of a word is defined as the N-1 words that appear before the word, and its expression formula is:

In this paper, N=2 and N=3 are used, that is, the probability of occurrence of the current word is determined by the previous one or two words P(w ₂ |w ₁ ), P(w ₃ |w ₂ ,w ₁ ).

To put it simply, the language model is the model obtained by statistical corpus, the corpus is the text library used for training, and the dictionary file stores the corpus for training and the corresponding speech. The language model is the combined probability of the expressed corpus. For example, if P(w ₁ ) is the probability of w ₁ appearing in the article, P(w ₁ ,w ₂ ) is the probability of w ₁ and w ₂ appearing continuously, and P(w ₂ |w ₁ ) is the known w ₁ has The probability of w ₂ appears in the case of occurrence, assuming that the probability of recognizing sentence is represented by P(s), P(s)=P(w ₁ ,w ₂ ,…,w _n ) means the word set w ₁ ,w ₂ ,… , the probability that w _n appears continuously and generates S, use the conditional probability formula S to replace the entire formula with:

P(sentence)＝P(w ₁ )*P(w ₂ |w ₁ )*P(w ₃ |w ₂ )…*P(w _n |w ₁ ,w ₂ ,…,w _n-1 )

Then use the Markov assumption to simplify it into:

P(sentence)＝P(w ₁ )*P(w ₂ |w ₁ )*P(w ₃ |w ₂ )…*P(w _n |w _n-1 )

We know that P( _wi |wi _-1 )＝P(wi _-1 , _wi )/P( _wi ), both P(wi _-1 , _wi ) and P( _wi ) can be obtained from After the corpus is counted, P (sentence) can be obtained in the end. The language model stores the probability statistics of P(w _i-1 , w _i ), and realizes the whole recognition process by finding the maximum value of P(sentence).

Step 7.5: Speech decoding and search algorithm.

For the input speech signal, build a recognition network according to the trained acoustic model, language model and dictionary, and find the best path in the network according to the search algorithm, which is the path that can output the speech signal with the greatest probability Word string, so as to determine the text contained in this speech sample. This text adopts Viterbi algorithm to realize the decoding of speech. The specific process is as follows:

(1) Input the parameters of the HMM model and the observation sequence O={o ₁ ,o ₂ ,...,o _T }, then all state probabilities at t=1 are:

δ ₁ (i) = π _i b _i (o ₁ )

ψ ₁ (i) = 0

(2) Gradually deduce to t=2,3,...,T, then:

(3) Terminate traversal:

(4) Backtracking to the optimal path, t=T-1, T-2,...,1;

output optimal hidden state path Among them, δ _t (i) is the joint probability of all nodes passed by the optimal path at time t, ψ _t (i) is the hidden state at time t, T is time, and P ^* is the probability of the optimal path , is the endpoint of the optimal path. Finally, speech recognition is realized through the optimal path.

After the user speaks out the lifting arm, lowering arm, forward rotation, reverse rotation and stop voice, the simulation system realizes the corresponding operation of the coal shearer, and the system recognizes the keywords spoken by the user and displays them in the upper left corner of the interface.

Step 8: AR dynamic demonstration function mode switching.

Set a widget blueprint on the interface, add the openLevel function, and switch to the new Map, that is, the AR mode. Enter the AR demonstration mode, which specifically realizes the demonstration and explanation of tool models in the coal mining process, so as to realize the learning and educational application functions of AR technology.

Step 9: Model selection, model explanation and dynamic demonstration in AR mode.

In the AR dynamic demonstration module of the present invention, in order to make the user interface more concise and convenient for AR display, a second-level hidden menu is designed. In this embodiment, the additional sub-function selection of model selection, model explanation, model demonstration and function expansion is designed to be hidden. In the second level menu, the model selection is divided into models such as coal shearer, roadheader, wind coal drill, fully mechanized mining support, etc. After the user selects, the submenu can be hidden, and the model explanation, model dynamic demonstration and function expansion menu are also implemented in the same way. Refer to Figure 1 for specific implementation contents. This article takes NFT (Natural Feature Tracking) as an example to implement AR technology. The principle is shown in Figure 4. The specific process is as follows:

Step 9.1: Through camera calibration, the distortion parameters caused by the deviation of the camera manufacturing process, that is, the intrinsic matrix of the camera, are obtained to restore the one-to-one correspondence between the 3D space and the 2D space of the camera model.

Step 9.2: According to the hardware parameters of the camera itself, we can calculate the corresponding projection matrix (Projection Matrix).

Step 9.3: Perform feature extraction on the natural picture to be recognized, and obtain a set of feature points {p}.

Step 9.4: Perform feature extraction on the image captured by the camera in real time, which is also a set of feature points {q}.

Step 9.5: Use the ICP (Iterative Closest Point) algorithm to iteratively solve the R, T matrix (Rotation&Translation) of these two groups of feature points, that is, the Pose matrix, which is often called the Model View Matrix in graphics. Suppose two points in three-dimensional space are: Their Euclidean distance is:

To find the matrices R and T of p and q changes, for Where i, j=1, 2,..., N, use the least square method to find the optimal solution. Make:

R and T at the minimum time, R and T at this time are the MVP matrix. Among them, E is the distance sum of the corresponding points in the two point sets after transformation, and N is the number of points in the point set.

Step 9.6: Obtain the MVP matrix (Model View Projection), and perform three-dimensional graphics drawing.

Step 10: Take a screenshot to generate an icon.

In the main menu of AR mode, add a button to bind the screenshot function of the camera, and add a scrolling menu bar on the right side of the menu. When the screenshot function is successfully triggered, the screenshot will be displayed on the right scrolling menu bar through the set dynamic material conversion function. . During the demonstration process, when the user clicks the screenshot button, the system will generate an icon on the left side of the interface, which is convenient for the user to record and observe the difficulties and doubts in the learning process in detail, which can strengthen the learning effect.

Step 11: The model rotates to stop displaying.

In AR mode, what the user sees is the superposition of the real scene and the virtual model. Instantiate the set model as an Actor, add the Rotation function, and realize the clockwise rotation of the model. With this design, the model rotation is set, and the user has a 360-degree observation and learning of the tool model, which can better achieve visual effects. This demonstration learning mode is more authentic and immersive.

Step 12: AR function expansion module.

The present invention adds an AR education display extension function, controls Map switching by adding a secondary UI, and realizes the demonstration of different objects. It includes the display function of the earth, Saturn, Mercury, planets with atmospheres and galaxies. The planets rotate. Through the AR mode, the moving planets are displayed in front of the user's eyes, and the knowledge introduction function is added to improve the expanded education display function of this system. .

Step 13: Dynamic gestures to interact with the model.

AR mode adds OpenCV video information processing. After initializing the video stream, first perform motion detection. If dynamic hand motion is detected, image processing is performed, and the gesture is subjected to graphic processing to denoise, convert to HSV mode, morphological processing, and outline drawing. , calibration information and contour information are compared, and finally the force simulation of the model is carried out to realize the interaction between the dynamic gesture and the virtual model. The specific implementation principle flow is shown in Figure 5. In particular, this dynamic gesture interaction realizes the recognition and control of simulating three-dimensional gestures. The dynamic hand acquired by the video stream is two-dimensional information. Here, through matrix operations, it will be compared with the calculated conversion matrix of the camera relative to the detected logo. Get a 3D motion gesture motion information, so as to realize the force simulation of the model in different directions in the 3D space; specifically include the following steps:

Step 13.1: Motion Detection

This method is based on the motion detection of the difference between the color histogram and the background. When the program starts the camera, it takes a certain amount of time. During this time, 20 frames of images can be collected, and the cyclic background update of these 20 frames is as follows, and each frame After the motion detection, the pixels other than the motion gesture area are also updated for the background.

Among them, u _t is the corresponding pixel of the background image, u _t+1 is the updated background image pixel; I _t is the pixel of the current frame image, If _is the mask value of the current frame image pixel, that is, whether Do background update; a ∈ [0,1] is the update speed of the background image model, generally 0.8 to 1, this method takes 0.8.

Step 13.2: Image Preprocessing

Through the simple motion detection step in step 13.1, if motion information is detected, start to preprocess the video frame images containing motion gestures: use OpenCV's medianBlur function to perform median filtering on the image to remove salt and pepper noise:

Step 13.3: Convert to HSV space

Through the cvtColor function, the color space conversion of the image is carried out to obtain the data of its HSV space, and the brightness v value in the HSV space is reset to a relatively small brightness value (reducing the interference of static skin color); for the brightness v in the HSV space The value reset is as follows:

Among them, r and g are the red and green pixels of the skin color area of interest, and r>g;

Step 13.4: Segment the hand region and perform morphological processing

Comparing the obtained motion binary image with the binary image obtained by back projection, performing some image morphological closing operations to obtain a relatively complete binary image of motion skin color gesture; remove the noise in the image;

Step 13.5: Get Gesture Outlines

After preliminary morphological operations, noise is removed, and the boundary of the hand is made clearer, the gesture contour is obtained through OpenCV's findContours function, and then the false contour is removed;

Step 13.6: Draw contours, calibration information

Step 13.7: Comparing contour information, setting direction vector

Since the hand is constantly moving, the outline we get is constantly changing. Compare the contours obtained in each frame and set the comparison conditions. Assign a value to the direction flag variable by comparison. The status comparison and analysis are shown in Table 1:

Table 1: State Analysis

Step 13.8: Act on the virtual model through the direction vector to generate force simulation

After the dynamic gesture is judged by the outline, the virtual model is subjected to force simulation operation according to different judgment results. According to the value of the direction mark in the contour judgment process, the coordinate value of the model in the three-dimensional space will be multiplied on the three coordinate axes of x, y, and z. Through the change of the coordinate value, the change of the position of the model is realized to achieve the force simulation.

In this embodiment, a group of palm movements from far to near, from bottom to top, and palm twisting in various directions are selected to simulate and display different force effects on the model. The gesture movement models move forward, upward and according to The different twisting directions of the hand have a running effect of force in all directions. This function demonstrates the interaction between dynamic gestures and the virtual model. This interaction can help users observe the model from multiple angles, and realize the interaction between teaching and users, increasing the fun.

Of course, the above descriptions are not intended to limit the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or replacements made by those skilled in the art within the scope of the present invention shall also belong to the present invention. protection scope of the invention.

Claims (8)

1. A method for realizing multi-interaction of mining operations based on virtual reality and augmented reality, characterized in that: an underground mining operation multi-interaction simulation system is adopted, the system includes two modes of virtual reality mode and augmented reality mode; virtual reality mode includes specific Scene modeling, roaming, replacement of models and their materials, video embedding into virtual scenes, model movement, application scene intention interaction, QR code generation and voice interaction; the augmented reality mode includes model selection, model explanation, dynamic model demonstration, gestures Control model interaction, screenshot generation icon, 360-degree rotation and stop, function mode switching and function expansion; the system has designed two hidden menus, that is, model replacement in virtual reality mode, material selection menu and model selection in augmented reality mode menu; the first type of menu will be displayed when the user enters a specific area, and will be hidden when the user leaves; the second type of click can display a secondary menu somewhere, and click the menu again to hide; The multi-interactive implementation method of the mining operation specifically includes the following steps: Step 1: Construction of the entire environment scene for mining operations According to the real environment of underground mining operations, use the modeling tool 3DMax to carry out 1:1 scale modeling to realize the environment simulation of the entire underground mining operation; use the UE4 engine to edit the model including creating, editing textures and materials, adding Physical collision, adding lighting, effect lighting and special effects to the overall environment, and baking and rendering; Step 2: Roaming of virtual reality application scenarios In the UE4 engine, set up, down, left, and right keys on the keyboard, bind the Up, Down, Right, and Left direction control functions, and bind the Turnaround control function for the mouse to realize the roaming of the virtual reality scene of the entire underground mining operation; Step 3: Replacement of tool models for underground mining operations and simulated materials for mine geology Add a hidden menu in the virtual underground mine mining scene. When roaming to the ore mining place, the model or material selection menu will automatically appear. Users can select the model or material from the menu to replace according to their needs; Step 4: Embed the video material into the 3D application scene and control the playback and stop Embed the video material into the virtual reality scene, play it in three-dimensional space, simulate the monitoring and display device of the mining environment, set the keyboard X key, bind the MediaPlayer media class of the UE4 platform, and control the playback and stop of the video through the OpenSource and Close functions; Step 5: Select the model and move it anywhere Select the model with the mouse and move the model to any position that needs to be simulated to achieve mechanical movement simulation in the real scene; Step 6: Realize the intentional interaction of the application scenario When the user roams to a specific location in the virtual reality application scene, the system detects that the user intends to enter, and automatically turns on the ambient light to realize natural interaction in the virtual scene; Step 7: QR code generation Bind the F key of the keyboard, add a QR code generation function, set the keyboard key to control the function of generating a QR code, and the user presses the F key on the keyboard, and the system generates a QR code with a virtual scene panorama with the set sampling points; Step 8: Implement Voice Interaction The user controls the coal shearer in the virtual reality scene through keywords including forward rotation, reverse rotation, raising arm, lowering arm, and stop, and simulates its operation effect; Step 9: AR Dynamic Demonstration Function Mode Switching The user clicks the AR mode button in the upper right corner of the system to switch to the AR demonstration mode. 2. The multi-interactive implementation method for mining operations based on virtual reality and augmented reality according to claim 1, characterized in that: in step 3, the model is instantiated into a specific Actor, and the SetMesh function and the SetMaterial function are added to replace the model And model material, set Widget Blueprint user interface and Box collision collision detection, realize the hidden menu function of three-dimensional space. 3. the multi-interactive realization method of mining operation based on virtual reality and augmented reality according to claim 1, is characterized in that: in step 5, add mouse event for the model to be operated, select model by GetHitResult function, then according to The coordinates of the mouse in the three-dimensional space, change the coordinates of the SetActorLocation function of the model, when the mouse clicks again, assign the coordinates of the mouse in the x, y, and z directions to the model, and the GetHitResult function will set the model to unselected model. 4. The multi-interaction method for mining operations based on virtual reality and augmented reality according to claim 1, characterized in that: in step 6, a TriggerBox trigger is set, when the first-person character triggers the TriggerBox, the system detects that the user intends to Entering a certain area will automatically activate a certain device in this area. 5. The multi-interactive implementation method of mining operations based on virtual reality and augmented reality according to claim 1, characterized in that: in step 7, the user presses the F key of the keyboard, and the system generates a virtual scene panorama that contains the sampling points set Two-dimensional code, the user scans the two-dimensional code with the mobile phone, and jumps to the virtual application scene display page on the mobile phone. The scene of the underground mining operation environment can be displayed in a 720-degree viewing angle, and it can also realize the multi-scenario and multi-angle roaming experience on the mobile phone. 6. The multi-interactive implementation method of mining operations based on virtual reality and augmented reality according to claim 1 is characterized in that: in step 8, voice recognition is realized based on the Pocket-sphinx library, by improving the Chinese keyword dictionary, after pre-processing Processing, feature extraction, acoustic model training, language model training, voice decoding and search to realize the recognition function, and finally through the writing function control function of the UE4 engine to realize the control of the voice on the model in the three-dimensional space; the specific implementation steps of the voice recognition are as follows: Step 8.1: Preprocessing Process the input original voice signal, filter out the unimportant information and background noise, and process the endpoint detection, voice framing and pre-emphasis of the voice signal; Pre-emphasis is realized by a first-order FIR high-pass digital filter, and the transfer function of the first-order FIR high-pass digital filter is: H(z)=1-az ^-1 ; Among them, a is the coefficient of the pre-emphasis filter, and the value range is 0.9 to 1.0. If the speech sampling value at time n is x(n), the signal after pre-emphasis is y(n)=x(n)-a*x(n-1); Step 8.2: Feature Extraction Feature extraction is performed by the method of Mel Frequency Cepstral Coefficient (MFCC); the specific steps are as follows: Step 8.2.1: Using the critical band effect of human hearing, MEL cepstrum analysis technology is used to process the speech signal to obtain the vector sequence of MEL cepstrum coefficients; Step 8.2.2: represent the frequency spectrum of input speech with MEL cepstral coefficient vector sequence, set several band-pass filters with triangular or sinusoidal filtering characteristics in the range of speech spectrum; Step 8.2.3: Find the output data of each band-pass filter through the band-pass filter bank; Step 8.2.4: take the logarithm of the output data of each bandpass filter, and do discrete cosine transform (DCT); Step 8.2.5: Obtain the MFCC coefficient; the solution formula is as follows: Wherein, C _i is a feature parameter, k is a variable, 1≤k≤P; P is the number of triangular filters, F(k) is the output data of each filter, and i is the data length; Step 8.3: Acoustic Model Training Acoustic model parameters are trained according to the characteristic parameters of the training speech library; During recognition, the characteristic parameters of the speech to be recognized are matched with the acoustic model to obtain the recognition result; the acoustic model is realized by using a Gaussian mixture model-hidden Markov model (GMM-HMM), which specifically includes the following steps: Step 8.3.1: Find the joint probability density function of the mixed Gaussian model as follows: Among them, M represents the number of Gaussians in the mixed Gaussian model, C _m represents the weight, u _m represents the mean value, ∑ _m represents the covariance matrix, D is the dimension of the observation vector; Θ={C _m , u _m , ∑ _m } to estimate, use the following formula to solve: Among them, j is the number of current iteration rounds, N represents the number of elements in the training data set, x ^(t) is the feature vector at time t, h _m (t) represents the posterior probability of C _m at time t; GMM parameters are passed through the EM algorithm make an estimate that maximizes its probability of generating speech observation features on the training data; Step 8.3.2: Solve the three components of the HMM Let the state sequence be q ₁ , q ₂ ,..., q _N , and let the transition probability matrix A=[a _ij ]i, j∈[1, N], then the obtained Markov chain state jump The probability is: a _ij =P(q _t =j|q _t-1 =i); the initial probability of the Markov chain π=[π _i ]i∈[1,N], where π _i =P(q ₁ =i); let the observation probability distribution of each state b _i (o _t )=P(o _t |q _t =i), use the GMM model to describe the observation probability distribution of the state; according to step 8.3.1, solve the formula for: Among them, N is the number of states, i and j represent states, a _ij represents the probability of jumping from state i to state j at time t-1 at time t-1, o _t is the observed value at time t, C _{i, m} are mixing coefficients , represents the weight between different Gaussians, u _{i, m} represents the mean value between different Gaussians, ∑ _{i, m} represents the covariance matrix between different Gaussians; the parameters of HMM are estimated by the Baum-Welch algorithm, and finally generated Acoustic model files; Step 8.4: Language Model Training The N-Gram model is used to train the language model; the probability of the i-th word appearing in a sentence depends on the N-1 words before it, that is, the context of a word is defined as the N-1 words that appear before the word words whose expression is: Use the conditional probability formula S to replace the above expression with the following formula: P(sentence)＝P(w ₁ )*P(w ₂ |w ₁ )*P(w ₃ |w ₂ )...*P(w _n |w ₁ ,w ₂ ,...,w _{n- 1} ) Among them, P(w ₁ ) is the probability that w ₁ appears in the article, P(w ₁ , w ₂ ) is the probability that w ₁ and w ₂ appear consecutively, P(w ₂ |w ₁ ) is the known w ₁ has The probability of w ₂ appears in the case of occurrence, assuming that the probability of recognizing the sentence is represented by P(s), P(s)=P(w ₁ , w ₂ ,...,w _n ) represents the word set w ₁ , w ₂ ,..., w _n appear continuously and generate the probability of S; Simplified by the Markov assumption into the following formula: P(sentence)＝P(w ₁ )*P(w ₂ |w ₁ )*P(w ₃ |w ₂ )...*P(w _n |w _n-1 ) Among them, P(w _i |wi _-1 )＝P(wi _-1 ， _wi )/P( _wi ), P(wi _-1 ， _wi ) and P( _wi ) are calculated from the corpus , and finally P(sentence) can be obtained. The language model stores the probability statistics of P(w _i-1 , w _i ), and realizes the entire recognition process by finding the maximum value of P(sentence); Step 8.5: Speech decoding and search algorithm For the input speech signal, build a recognition network according to the trained acoustic model, language model and dictionary mapping file created by the g2p tool, and find the best path in the network according to the search algorithm. The word string of the voice signal is output with the greatest probability, so that the text contained in the voice sample is determined, and the voice decoding is realized by the Viterbi algorithm. The specific process is as follows: Step 8.5.1: Input the parameters of the HMM model and the observation sequence O={o ₁ , o ₂ ,..., o _T }, then all state probabilities at t=1 are: δ ₁ (i) = π _i b _i (o ₁ ) ψ ₁ (i) = 0 Step 8.5.2: Gradually deduce to t=2, 3, ..., T, then: Step 8.5.3: Terminate the traversal: Step 8.5.4: Backtracking the optimal path, t=T-1, T-2, ..., 1; Step 8.5.5: Output the optimal hidden state path Among them, δ _t (i) is the joint probability of all nodes passed by the optimal path at time t, ψ _t (i) is the hidden state at time t, T is time, and P ^* is the probability of the optimal path , is the endpoint of the optimal path. 7. The multi-interactive realization method of mining operations based on virtual reality and augmented reality according to claim 6, characterized in that: a is 0.97. 8. The method for realizing multi-interaction of mining operations based on virtual reality and augmented reality according to claim 1, characterized in that: in step 9, specifically comprising the following steps: Step 9.1: Model Selection Select the shearer model, roadheader model, air coal drill model and fully mechanized mining support model, each type of model is a 1:1 modeling simulation of real coal mining tools; Step 9.2: Model Explanation After selecting the model, the user selects the tool model option to be learned through the model selection menu in the augmented reality mode, and the system will play the corresponding voice explanation, and click the button again to stop the voice; Step 9.3: Model presentation Import the tool simulation running animation created in the 3DMax modeling process into the UE4 engine, set the corresponding selection menu, and click to demonstrate the operating status of the corresponding coal mining tool in AR mode; Step 9.4: Take screenshot to generate icon In the main menu of AR mode, add a button to bind the screenshot function of the camera, and add a scrolling menu bar on the right side of the menu. When the screenshot function is successfully triggered, the screenshot will be displayed on the right scrolling menu bar through the set dynamic material conversion function. , during the demonstration, if the user clicks the screenshot button, the system will generate an icon on the side of the interface; Step 9.5: Rotate Instantiate the set model as an Actor, add the Rotation function, and realize the clockwise rotation of the model; Step 9.6: Function Extension Add a secondary UI to control the map switch, and realize the operation demonstration function including the earth, Saturn, Mercury, planets with atmospheres, and galaxies; add WidgetBlueprint code to realize the display or hide of the knowledge introduction panel; design the return button to return to the AR editor main module; Step 9.7: Control the model with dynamic gestures, superimpose the real environment and the virtual model, and interact with the model by gestures, including the following steps: Step 9.7.1: Initialize the video capture, read the logo file and camera parameters; Step 9.7.2: Grab the video frame image; Step 9.7.3: Execute the detection mark and identify the mark template in the video frame, and use the OpenCV library function to perform motion detection on the acquired video frame image, and judge whether a motion track is detected; If: the judgment result is that the motion track of the gesture is detected, then perform step 9.7.4; Or if the judgment result is that no motion track is detected, then continue to perform the detection mark and identify the mark template in the video frame, and then perform step 9.7.12; Motion detection is performed based on the color histogram and the background difference, and the background update is performed on the collected frames and the pixels other than the motion gesture area obtained after the motion detection of each frame. The formula is as follows; Among them, u _t is the corresponding pixel of the background image, u _t+1 is the updated background image pixel; I _t is the pixel of the current frame image, If _is the mask value of the current frame image pixel, that is, whether Do background update; a ∈ [0, 1] is the update speed of the background image model; Step 9.7.4: Preprocessing the image including denoising; Through the motion detection step, if motion information is detected, the video frame image containing motion gestures is preprocessed: the image is median-filtered through the medianBlur function of OpenCV to remove the salt and pepper noise; Step 9.7.5: Convert to HSV space; Use the cvtColor function to convert the color space of the image to obtain its HSV space data, and reset the brightness v value in the HSV space as shown in the following formula: Among them, r and g are the red and green pixels in the skin color area, and r>g; Step 9.7.6: Segment the hand region; Step 9.7.7: Carry out morphological processing to remove impurities; Comparing the obtained motion binary image with the binary image obtained by back-projection, and performing image morphology closing operation to obtain a relatively complete binary image of motion skin color gesture; and removing noise points in the image; Step 9.7.8: Get hand contour; Through the preliminary morphological operation, remove the noise and make the boundary of the hand clearer, get the gesture contour through the findContours function of OpenCV, and then perform the pseudo contour removal operation; Step 9.7.9: Draw the outline of the hand and calibrate the information; Step 9.7.10: Comparing the contour information and setting the direction vector; Compare the contours obtained in each frame, set the comparison condition, and assign a value to the direction flag variable by comparison; Step 9.7.11: Simulate the force on the model according to the vector coordinates to realize the interaction between dynamic gestures and the virtual model; After the dynamic gesture is judged by the contour, the virtual model is subjected to a force simulation operation according to different judgment results. According to the value of the direction mark in the contour judgment process, the coordinate value of the model in the three-dimensional space will be carried out on the three coordinate axes of x, y, and z. The multiplication calculation on the above, through the change of the coordinate value, the change of the model position is realized to achieve the simulation of the force; Step 9.7.12: Calculate the transformation matrix of the camera relative to the detected marker; Step 9.7.13: superimpose the virtual object on the detected mark, and return to step 9.7.2 to realize the superimposed display of the real environment and the virtual model; Step 9.7.14: When the VR mode is clicked, the system switches the display mode, the camera is turned off, and the above steps stop.