CN111796926A - Instruction execution method, device, storage medium and electronic device - Google Patents

Abstract

The embodiment of the application discloses an instruction execution method, an instruction execution device, a storage medium and electronic equipment, wherein when a user instruction is received, a first feature vector is generated according to the user instruction; acquiring current panoramic data and generating a second eigenvector according to the panoramic data; fusing the first feature vector and the second feature vector to generate a user feature matrix; acquiring an intention label matched with a user instruction according to the user characteristic matrix and a pre-trained intention prediction model; the user instruction is executed according to the intention tag. According to the scheme, the context information and the contextual information of the user instruction are supplemented by collecting panoramic data, so that the implicit intention of the user can be more accurately understood, and the user instruction can be better executed.

Description

Instruction execution method, device, storage medium and electronic device

technical field

The present application relates to the field of terminal technologies, and in particular, to an instruction execution method, apparatus, storage medium, and electronic device.

Background technique

The user generally controls the terminal to perform corresponding operations through control commands, such as voice commands, touch operation commands, and the like. With the development of terminal technology, the terminal needs to understand the implicit intention of the control command issued by the user, so as to better execute the user command. However, this method of user intent recognition is only based on the user's voice command, which makes it difficult to accurately understand and describe the user's actual needs.

SUMMARY OF THE INVENTION

The embodiments of the present application provide an instruction execution method, apparatus, storage medium, and electronic device, which can improve the accuracy of identifying the implicit intention of the user by the terminal, so as to execute the user instruction more accurately.

In a first aspect, an embodiment of the present application provides an instruction execution method, including:

When receiving a user instruction, generate a first feature vector according to the user instruction;

Obtain current panoramic data, and generate a second feature vector according to the panoramic data;

Fusion processing is performed on the first feature vector and the second feature vector to generate a user feature matrix;

According to the user feature matrix and the pre-trained intent prediction model, obtain an intent label matching the user instruction;

The user instruction is executed according to the intent tag.

In a second aspect, an embodiment of the present application provides an instruction execution device, including:

a signal feature extraction module, configured to generate a first feature vector according to the user instruction when a user instruction is received;

a scene feature extraction module, used to obtain current panoramic data, and generate a second feature vector according to the panoramic data;

a feature fusion module, configured to perform fusion processing on the first feature vector and the second feature vector to generate a user feature matrix;

an intent prediction module, configured to obtain an intent label matching the user instruction according to the user feature matrix and the pre-trained intent prediction model;

An instruction execution module, configured to execute the user instruction according to the intent tag.

In a third aspect, a storage medium provided by an embodiment of the present application stores a computer program thereon, and when the computer program runs on a computer, the computer executes the instruction execution method provided by any embodiment of the present application.

In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, the memory having a computer program, and the processor is configured to execute the method provided by any embodiment of the present application by invoking the computer program. instruction execution method.

According to the technical solution provided by the embodiment of the present application, when a user instruction is received, a first feature vector is generated according to the user instruction, current panoramic data is acquired, a second feature vector is generated according to the panoramic data, and the first feature vector and the second feature vector are compared between the first feature vector and the second feature vector. Fusion processing is performed to generate a user feature matrix. Next, the user feature matrix is used as the input data of the pre-trained intent prediction model, and the intent label matching the user instruction is obtained, and the user instruction is executed according to the intent tag. The data supplements the contextual information and contextual information of the user's instruction, which can more accurately understand the user's implicit intention, so as to better execute the user's instruction.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 is a schematic diagram of a panoramic perception architecture of an instruction execution method provided by an embodiment of the present application.

FIG. 2 is a schematic diagram of an application scenario of an instruction execution method provided by an embodiment of the present application.

FIG. 3 is a first schematic flowchart of an instruction execution method provided by an embodiment of the present application.

FIG. 4 is a schematic flowchart of a second method of an instruction execution method provided by an embodiment of the present application.

FIG. 5 is a third schematic flowchart of an instruction execution method provided by an embodiment of the present application.

FIG. 6 is a schematic structural diagram of an instruction execution apparatus provided by an embodiment of the present application.

FIG. 7 is a schematic diagram of a first structure of an electronic device provided by an embodiment of the present application.

FIG. 8 is a schematic diagram of a second structure of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of this application.

The terms "first," "second," and "third," etc. in this application are used to distinguish different objects, rather than to describe a specific order. Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or modules is not limited to the listed steps or modules, but some embodiments also include unlisted steps or modules, or some embodiments Other steps or modules inherent to these processes, methods, products or devices are also included.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

Referring to FIG. 1 , FIG. 1 is a schematic diagram of a panoramic perception architecture of an instruction execution method provided by an embodiment of the present application. The instruction execution method is applied to an electronic device. The electronic device is provided with a panoramic perception architecture. The panoramic perception architecture is an integration of hardware and software in an electronic device for implementing the instruction execution method.

Among them, the panoramic perception architecture includes an information perception layer, a data processing layer, a feature extraction layer, a scenario modeling layer, and an intelligent service layer.

The information perception layer is used to obtain the information of the electronic device itself or the information in the external environment. The information perception layer may include a plurality of sensors. For example, the information perception layer includes a distance sensor, a magnetic field sensor, a light sensor, an acceleration sensor, a fingerprint sensor, a Hall sensor, a position sensor, a gyroscope, an inertial sensor, an attitude sensor, a barometer, a heart rate sensor, and other sensors.

Among them, the distance sensor can be used to detect the distance between the electronic device and the external object. The magnetic field sensor can be used to detect the magnetic field information of the environment in which the electronic device is located. The light sensor can be used to detect the light information of the environment where the electronic device is located. Acceleration sensors can be used to detect acceleration data of electronic devices. The fingerprint sensor can be used to collect the user's fingerprint information. Hall sensor is a magnetic field sensor made according to the Hall effect, which can be used to realize automatic control of electronic equipment. The location sensor can be used to detect the current geographic location of the electronic device. Gyroscopes can be used to detect the angular velocity of electronic devices in various directions. Inertial sensors can be used to detect motion data of electronic devices. The attitude sensor can be used to sense the attitude information of the electronic device. A barometer can be used to detect the air pressure in the environment in which the electronic device is located. The heart rate sensor may be used to detect the user's heart rate information.

The data processing layer is used to process the data obtained by the information perception layer. For example, the data processing layer can perform data cleaning, data integration, data transformation, data reduction and other processing on the data obtained by the information perception layer.

Among them, data cleaning refers to cleaning a large amount of data obtained by the information perception layer to eliminate invalid data and duplicate data. Data integration refers to integrating multiple single-dimensional data obtained by the information perception layer into a higher or more abstract dimension to comprehensively process multiple single-dimensional data. Data transformation refers to converting the data type or format of the data obtained by the information perception layer, so that the transformed data can meet the processing requirements. Data reduction refers to reducing the amount of data to the greatest extent possible on the premise of keeping the original data as much as possible.

The feature extraction layer is used to perform feature extraction on the data processed by the data processing layer to extract features included in the data. The extracted features may reflect the state of the electronic device itself, the state of the user, or the environmental state of the environment in which the electronic device is located.

Among them, the feature extraction layer can extract features or process the extracted features by filtering method, packaging method, integration method and other methods.

The filtering method refers to filtering the extracted features to remove redundant feature data. The packing method is used to filter the extracted features. The integration method refers to the integration of multiple feature extraction methods to construct a more efficient and accurate feature extraction method for feature extraction.

The scenario modeling layer is used to construct a model according to the features extracted by the feature extraction layer, and the obtained model can be used to represent the state of the electronic device, the state of the user, or the environment state, etc. For example, the scenario modeling layer can construct a key value model, a pattern identification model, a graph model, an entity relationship model, an object-oriented model, etc. according to the features extracted by the feature extraction layer.

The intelligent service layer is used to provide users with intelligent services according to the model constructed by the scenario modeling layer. For example, the intelligent service layer can provide users with basic application services, can perform system intelligent optimization for electronic devices, and can also provide users with personalized intelligent services.

In addition, the panoramic perception architecture may also include multiple algorithms, each of which may be used to analyze and process data, and the multiple algorithms may constitute an algorithm library. For example, the algorithm library may include Markov algorithm, invisible Dirichlet distribution algorithm, Bayesian classification algorithm, support vector machine, K-means clustering algorithm, K-nearest neighbor algorithm, conditional random field, residual network, long-term Algorithms such as short-term memory networks, convolutional neural networks, and recurrent neural networks.

Based on the above panoramic perception architecture, the electronic device collects the user's panoramic data through the information perception layer and/or other methods, and the data processing layer processes the panoramic data, for example, performs data cleaning and data integration on the obtained panoramic data. Next, the intelligent service layer responds to the user instruction according to the instruction execution method proposed in this application. For example, when receiving the user instruction, it generates the first feature vector according to the user instruction, obtains the current panoramic data, and generates the second feature vector according to the panoramic data, The first feature vector and the second feature vector are fused to generate a user feature matrix. Next, the user feature matrix is used as the input data of the pre-trained intent prediction model, and the intent label matching the user instruction is obtained. According to the intent The tag executes the user's instruction, and the solution supplements the context information and contextual information of the user's instruction by collecting panoramic data, which can more accurately understand the user's implicit intention, and thus better execute the user's instruction.

The embodiment of the present application provides an instruction execution method, and the execution body of the instruction execution method may be the instruction execution apparatus provided by the embodiment of the present application, or an electronic device integrated with the instruction execution apparatus, wherein the instruction execution apparatus may adopt hardware or implemented in software. The electronic device may be a smart phone, a tablet computer, a palmtop computer, a notebook computer, or a desktop computer and other devices.

Please refer to FIG. 2. FIG. 2 is a schematic diagram of an application scenario of the instruction execution method provided by an embodiment of the present application. Taking the instruction execution device integrated in an electronic device as an example, the electronic device can receive user instructions, such as voice instructions, touch instructions or grip instructions. When receiving the user's instruction, the first feature vector is generated according to the user's instruction; then the current panoramic data of the electronic device is collected, such as user information, sensor status data, application program usage information in the electronic device, etc. The collected panoramic data generates a second feature vector, and then the first feature vector and the second feature vector are fused to generate a user feature matrix. According to the user feature matrix and the pre-trained intent prediction model, the user feature The intent tag corresponding to the matrix, and execute the user instruction according to the intent tag. In this way, based on the panoramic data collected when the instruction is received, the user instruction is supplemented with complete context information and information of the contextual environment, so that the electronic device It is able to more accurately understand the user's implicit intention, so as to better execute the user's instructions.

Please refer to FIG. 3 , which is a schematic flowchart of a first type of instruction execution method provided by an embodiment of the present application. The specific process of the instruction execution method provided by the embodiment of the present application may be as follows:

Step 101: When a user instruction is received, generate a first feature vector according to the user instruction.

In this embodiment, the user command may be of various types, such as voice command, touch command, and hold command. When receiving a user instruction, the electronic device obtains signal data corresponding to the user instruction, normalizes the signal data, generates a corresponding first feature vector, and uses the first feature vector to represent the information contained in the user instruction. Among them, the first feature vector can be expressed as follows:

s ₁ ={yi ₁ ,yi ₂ ,...,yi _n }

For example, in one embodiment, the electronic device is provided with a voice component, such as a microphone, and the electronic device can continuously collect the user's voice data through the microphone. After the electronic device turns on the voice recognition function, the user can control the electronic device through voice commands, such as "play music", "make a call" and other commands. For example, when a user instruction is received, the step of generating the first feature vector according to the user instruction includes: when a voice instruction is received, acquiring the voice data collected by the voice component; , generating a semantic feature vector of the speech data, and using the semantic feature vector as the first feature vector.

Among them, the self-encoding neural network model consists of an encoder encoder and a decoder decoder. The output of the network is equal to the input. The network also includes an intermediate hidden layer, which can extract the semantic feature vector of the speech data. In this scheme, the self-encoding cyclic neural network is used to extract the semantic feature vector from the speech data, and the input data and the output data of the self-encoding cyclic neural network are the above-mentioned speech data. During the training of the network, there is no need to label the speech data, and a large amount of speech data is collected in advance as the input and output of the network, and the network determines the parameters through self-learning.

Or, in another embodiment, other feature extraction methods may also be used to obtain speech feature vectors. Step 101. When a user instruction is received, the step of generating a first feature vector according to the user instruction may include:

When a voice command is received, obtain the voice data collected by the voice component;

Convert the voice data into a spectrogram according to an audio feature extraction algorithm;

According to the pre-trained self-encoding convolutional neural network and the spectrogram, a semantic feature vector of the speech data is generated, and the semantic feature vector is used as the first feature vector.

The audio feature extraction algorithm may be an MFCC (Mel Frequency Cepstrum Coefficient, Mel Frequency Cepstrum Coefficient) algorithm or an FFT (Fast Fourier Transformation, Fast Fourier Transform) algorithm, and the audio feature extraction algorithm converts the speech data into a spectrogram , using the spectrogram as the input and output data of the self-encoding convolutional neural network, and extracting semantic feature vectors from the network. Similar to the above-mentioned self-encoding cyclic neural network, the self-encoding convolutional neural network is also a kind of self-encoder, wherein the self-encoding convolutional neural network uses a convolutional layer to build an auto-encoder, and the output of the self-encoder is trained by training the output of the self-encoder. The data is aligned with the input data to obtain valuable information in its intermediate hidden layers.

The electronic device acquires the semantic vector feature through the above solution, and uses the semantic vector feature as the first feature vector.

For another example, in one embodiment, the user instruction is a touch instruction; when the user instruction is received, the step of generating the first feature vector according to the user instruction may include: when the touch instruction is received, obtaining the touch touch data collected by a control sensor; convert the touch data into the first feature vector.

The electronic device is provided with a touch sensor for detecting touch operations, such as a touch sensor provided under the touch screen, which can detect various touch operations, touch gestures, etc. input by the user. When a touch command is received, the touch data collected by the touch sensor is acquired, and the touch data includes a touch position and a touch track.

For another example, in one embodiment, the user instruction is a hold instruction; when the user instruction is received, the step of generating the first feature vector according to the user instruction may include: when the hold instruction is received, obtaining the hold instruction holding data collected by the holding sensor; converting the holding data into the first feature vector.

Wherein, a holding sensor is provided at positions such as the frame and the back cover of the electronic device, which can detect the holding instruction triggered by the holding operation of the user. The grip data may be location information, such as location coordinates. In order to facilitate the subsequent fusion of multiple feature vectors into one feature matrix, a preset length of the feature vectors needs to be pre-defined. The electronic device converts the obtained position coordinates into a vector representation with a preset length. For example, if the preset vector length is 10, the position coordinates with a length of 2 are converted into a first feature vector with a length of 10 by means of repeated superposition.

Step 102: Acquire current panoramic data, and generate a second feature vector according to the panoramic data.

In this embodiment of the present application, the panoramic data of the electronic device includes, but is not limited to, the following types of data: terminal state data, user state data, and sensor state data.

The user state data includes the user's face image indirectly captured by the camera of the electronic device, and user information such as the user's age and gender obtained from the user database.

The terminal operation data includes the operation mode of the user terminal in each time interval, and the operation mode includes game mode, entertainment mode, audio-visual mode, etc. The operation mode of the terminal can be determined according to the type of the currently running application program. The type of the application can be obtained directly from the classification information of the application installation package; or, the terminal running data can also include the terminal's remaining power, display mode, network status, screen-off/locking status, and so on.

Sensor status data includes signals collected by various sensors on the electronic device. For example, the electronic device includes the following sensors: distance sensor, magnetic field sensor, light sensor, acceleration sensor, fingerprint sensor, Hall sensor, position sensor, gyroscope, inertial sensor Sensors, attitude sensors, barometers, heart rate sensors, etc. Acquire sensor state data of the electronic device when it receives a user instruction, or acquire sensor state data of the electronic device for a period of time before receiving the user instruction. In some embodiments, the state data of some sensors may be acquired in a targeted manner.

Please refer to FIG. 4. FIG. 4 is a schematic flowchart of a second instruction execution method provided by an embodiment of the present application. Step 102, obtaining current panoramic data, and generating a second feature vector according to the panoramic data may include:

Step 1021: Acquire current terminal status data, user status data and sensor status data;

Step 1022: Generate a terminal status feature according to the terminal status data, generate a user status feature according to the user status data, and generate a terminal context feature according to the sensor status data;

Step 1023 , fuse the terminal state feature, the user state feature and the terminal context feature to generate the second feature vector.

Wherein, in some embodiments, the step of acquiring the user status data may include: invoking a camera component to capture the user's face image; recognizing the user's face image according to a preset convolutional neural network model, and generating a user emotion label; Obtain user information, and use the user emotion tag and the user information as the user state data. User information may be obtained from a user database, and the user information may include user gender, user age, user preferences, and the like.

Wherein, in some embodiments, the step of acquiring the current terminal state data includes: determining the program category described by the currently running application program; determining the current running mode of the terminal according to the program category, wherein the running mode includes game mode, Entertainment mode, video and audio mode; take the running mode as terminal state data.

To sum up, the terminal state feature ys ₁ is generated according to the terminal state data, and the four-dimensional terminal attitude feature ys ₂ -ys ₅ is obtained from the magnetometer, accelerometer, and gyroscope through the Kalman filtering algorithm according to the state data of the sensor, for example, Obtain the air pressure characteristics ys ₆ through the data collected by the barometer, determine the WIFI connection state ys ₇ through the network module, locate the data collected through the location sensor, and obtain the user's current location attributes (such as shopping malls, homes, companies, parks, etc.), Generate the feature ys ₈ ; further combine the 10-axis information of the magnetometer, the acceleration sensor, the gyroscope, and the barometer to obtain new multi-dimensional data by using the filtering algorithm or the principal component analysis algorithm, and generate the corresponding feature ys ₉ . The feature ys ₁₀ is generated according to the user's emotional tag, and the features ys ₁₁ to ys ₁₃ are generated according to the user's gender, age, and hobby. For the features in the non-numeric form in the above features, the index number can be established to convert it into a digital representation. For example, for the feature of the running mode of the current system terminal state, the index number is used to represent the current state mode, such as 1 is the game mode, 2 is the entertainment mode, and 3 is the video mode. If the current running mode is the game mode, it is determined that the current system state ys1=1. After acquiring the features represented by all numbers, a long vector is obtained by fusing the above feature data, and the long vector is normalized to obtain the second feature vector s ₂ :

s ₂ ={ys ₁ ,ys ₂ ,…,ys _m }

Step 103: Perform fusion processing on the first feature vector and the second feature vector to generate a user feature matrix.

Among them, the first eigenvector s ₁ and the second eigenvector s ₂ can be superimposed in matrix to generate the following user feature matrix:

If the lengths of the first eigenvector and the second eigenvector are not equal, the short-length vector can be adjusted by means of zero-padding. If n<m, the length of the first eigenvector s ₁ can be extended by means of zero-padding. is m. If n>m, the length of the second feature vector s ₂ is extended to n by means of zero padding.

Or, in an optional embodiment, after the first eigenvector and the second eigenvector are adjusted to have the same length, the first eigenvector s ₁ and the second eigenvector s ₂ are subjected to matrix superposition, and then, in order to generate More abundant eigenvectors provide subsequent operations, and the superimposed matrix is flipped to generate the following matrix:

Combine the matrices before and after the flip operation to obtain the following matrix as the user feature matrix.

Step 104: Acquire an intent label matching the user instruction according to the user feature matrix and the pre-trained intent prediction model.

In the embodiment of the present application, the intent prediction model is a classification model, which represents the relationship between the user feature matrix and the intent label. For example, the intent prediction model can be obtained by training a classification algorithm such as a convolutional neural network, a BP neural network (Back Propagation, back propagation), or an SVM (Support Vector Machine, support vector machine) algorithm. The following takes the convolutional neural network as an example to collect a large number of sample data of test users, extract features according to steps 101 to 103, and then label the extracted features. The user instruction is recorded, and the first feature vector is generated; the panoramic data when the user instruction is received is collected, and the second feature vector is generated according to the panoramic data. Then record the response of the electronic device to the user's instruction and the operation performed by the user based on the response. In an optional embodiment, the response of the electronic device to the user's instruction and the operation performed by the user based on the response may be recorded. The intent label data is automatically generated, or, in other embodiments, the user feature matrix can be labeled by manual labeling.

Input the above-mentioned user feature matrix with intent labels into the convolutional neural network for training, wherein the structure and hyperparameters of the convolutional neural network can be preset by the user as needed, and after training, the weight parameters of the network are determined to generate an intent prediction model .

In addition, it should be noted that, in order to fully characterize the user's intent, an intent label corresponding to a user feature matrix is an intent label set, and the intent label set has multiple sets of intent labels that can represent user instructions, location, time, environment, state, and habits. and other information labels, which describe the current panoramic portrait of the user.

Inputting the user feature matrix obtained in step 103 into the intent prediction model, an intent label matching the user instruction can be generated.

Step 105: Execute the user instruction according to the intent tag.

The intent tag output by the intent prediction model is a tag set composed of multiple tags. These tags form a panoramic image corresponding to the user's instruction. The electronic device responds to the user's instruction and starts the corresponding target application, and pushes the tag set to the target application. In this way, when the electronic device responds to the user's instruction, it is not only a simple operation such as starting the software, but also further, after the target application is opened, it can perform more specific operations according to the received tag set.

For example, referring to FIG. 5 , FIG. 5 is a third schematic flowchart of an instruction execution method provided by an embodiment of the present application. Step 105: Executing the user instruction according to the intent tag includes:

Step 1051: Determine the target application corresponding to the user instruction according to the intent tag;

Step 1052: Start the target application, and send the intent tag to the target application, where the target application performs a corresponding operation based on the intent tag.

For example, the user triggers the voice command "Xiaoou Xiaoou, order me a meal". After the system's voice recognition module detects the command, it obtains more data that can reflect the user's intention through panoramic perception technology, such as terminal status data, User status data and sensor status data, etc., use these data to supplement the context information of the instruction to obtain more comprehensive intent information, such as: current time, current location, scene, user eating habits, user hunger, etc. When the electronic device opens the third-party ordering application according to the instruction, it pushes the information to the third-party ordering application, and the third-party ordering application can recommend the specific ordering place and ordering content to the user according to the above information.

As can be seen from the above, the instruction execution method proposed in the embodiment of the present application, when receiving a user instruction, generates a first feature vector according to the user instruction, obtains the current panoramic data, generates a second feature vector according to the panoramic data, and analyzes the first feature vector. Perform fusion processing with the second feature vector to generate a user feature matrix, and then use the user feature matrix as the input data of the pre-trained intent prediction model, obtain the intent label matching the user instruction, and execute the user instruction according to the intent tag, This solution supplements the context information and context information of user instructions by collecting panoramic data, and can more accurately understand the user's implicit intention, thereby better executing user instructions.

In an embodiment, an instruction execution apparatus is also provided. Please refer to FIG. 6 , which is a schematic structural diagram of an instruction execution apparatus 400 according to an embodiment of the present application. The instruction execution apparatus 400 is applied to electronic equipment, and the instruction execution apparatus 400 includes a first feature extraction module 401, a second feature extraction module 402, a feature fusion module 403, an intention prediction module 404 and an instruction execution module 405, as follows:

The first feature extraction module 401 is configured to generate a first feature vector according to the user instruction when a user instruction is received.

In this embodiment, the user command may be of various types, such as voice command, touch command, and hold command. When receiving a user instruction, the first feature extraction module 401 obtains the signal data corresponding to the user instruction, normalizes the signal data, generates a corresponding first feature vector, and uses the first feature vector to represent that the user instruction contains Information. Among them, the first feature vector can be expressed as follows:

s ₁ ={yi ₁ ,yi ₂ ,...,yi _n }

For example, in one embodiment, the electronic device is provided with a voice component, such as a microphone, and the electronic device can continuously collect the user's voice data through the microphone. After the electronic device turns on the voice recognition function, the user can control the electronic device through voice commands, such as "play music", "make a call" and other commands. For example, the first feature extraction module 401 is further configured to: acquire the voice data collected by the voice component when a voice command is received; The semantic feature vector is used as the first feature vector.

Among them, the self-encoding neural network model consists of an encoder encoder and a decoder decoder. The output of the network is equal to the input. The network also includes an intermediate hidden layer, which can extract the semantic feature vector of the speech data. In this scheme, the self-encoding cyclic neural network is used to extract the semantic feature vector from the speech data, and the input data and the output data of the self-encoding cyclic neural network are the above-mentioned speech data. During the training of the network, there is no need to label the speech data, and a large amount of speech data is collected in advance as the input and output of the network, and the network determines the parameters through self-learning.

Or, in another embodiment, other feature extraction methods may also be used to obtain speech feature vectors. The first feature extraction module 401 is also used for: when receiving a voice command, acquiring the voice data collected by the voice component; converting the voice data into a spectrogram according to an audio feature extraction algorithm; according to the pre-trained self-encoding convolution A neural network and the spectrogram generate a semantic feature vector of the speech data, and use the semantic feature vector as the first feature vector.

The audio feature extraction algorithm may be an MFCC (Mel Frequency Cepstrum Coefficient, Mel Frequency Cepstrum Coefficient) algorithm or an FFT (Fast Fourier Transformation, Fast Fourier Transform) algorithm, and the audio feature extraction algorithm converts the speech data into a spectrogram , using the spectrogram as the input and output data of the self-encoding convolutional neural network, and extracting semantic feature vectors from the network. Similar to the above-mentioned self-encoding cyclic neural network, the self-encoding convolutional neural network is also a kind of self-encoder, wherein the self-encoding convolutional neural network uses a convolutional layer to build an auto-encoder, and the output of the self-encoder is trained by training the output of the self-encoder. The data is aligned with the input data to obtain valuable information in its intermediate hidden layers.

The first feature extraction module 401 obtains the semantic vector feature through the above solution, and uses the semantic vector feature as the first feature vector.

For another example, in one embodiment, the user command is a touch command; the first feature extraction module 401 is further configured to: when receiving a touch command, obtain touch data collected by a touch sensor; The touch data is converted into the first feature vector.

The electronic device is provided with a touch sensor for detecting touch operations, such as a touch sensor provided under the touch screen, which can detect various touch operations, touch gestures, etc. input by the user. When a touch command is received, the touch data collected by the touch sensor is acquired, and the touch data includes a touch position and a touch track.

For another example, in one embodiment, the user instruction is a holding instruction; the first feature extraction module 401 is further configured to: when receiving the holding instruction, acquire the holding data collected by the holding sensor; The holding data is converted into the first feature vector.

Wherein, a holding sensor is provided at positions such as the frame and the back cover of the electronic device, which can detect the holding instruction triggered by the holding operation of the user. The grip data may be location information, such as location coordinates. In order to facilitate the subsequent fusion of multiple feature vectors into one feature matrix, a preset length of the feature vectors needs to be pre-defined. The electronic device converts the obtained position coordinates into a vector representation with a preset length. For example, if the preset vector length is 10, the position coordinates with a length of 2 are converted into a first feature vector with a length of 10 by means of repeated superposition.

The second feature extraction module 402 is configured to acquire current panoramic data, and generate a second feature vector according to the panoramic data.

In this embodiment of the present application, the panoramic data of the electronic device includes, but is not limited to, the following types of data: terminal state data, user state data, and sensor state data. The user state data includes the user's face image indirectly captured by the camera of the electronic device, and user information such as the user's age and gender obtained from the user database.

The terminal operation data includes the operation mode of the user terminal in each time interval, and the operation mode includes game mode, entertainment mode, audio-visual mode, etc. The operation mode of the terminal can be determined according to the type of the currently running application program. The type of the application can be obtained directly from the classification information of the application installation package; or, the terminal running data can also include the terminal's remaining power, display mode, network status, screen-off/locking status, and so on.

Sensor status data includes signals collected by various sensors on the electronic device. For example, the electronic device includes the following sensors: distance sensor, magnetic field sensor, light sensor, acceleration sensor, fingerprint sensor, Hall sensor, position sensor, gyroscope, inertial sensor Sensors, attitude sensors, barometers, heart rate sensors, etc. Acquire sensor state data of the electronic device when it receives a user instruction, or acquire sensor state data of the electronic device for a period of time before receiving the user instruction. In some embodiments, the state data of some sensors may be acquired in a targeted manner.

Optionally, in an embodiment, the second feature extraction module 402 is further configured to: acquire current terminal status data, user status data and sensor status data; generate terminal status features according to the terminal status data, The state data generates a user state feature, and generates a terminal context feature according to the sensor state data; the second feature vector is generated by fusing the terminal state feature, the user state feature and the terminal context feature.

Wherein, in some embodiments, the second feature extraction module 402 is further configured to: call the camera component to capture the user's face image; identify the user's face image according to a preset convolutional neural network model, and generate a user emotion label ; Acquire user information, and use the user emotion tag and the user information as the user state data. User information may be obtained from a user database, and the user information may include user gender, user age, user preferences, and the like.

Wherein, in some embodiments, the second feature extraction module 402 is further configured to: determine the program category described by the currently running application program; determine the current running mode of the terminal according to the program category, wherein the running mode includes game mode, entertainment mode, video and audio mode; take the running mode as terminal state data.

In summary, the terminal state feature ys ₁ is generated according to the terminal state data, and the four-dimensional terminal attitude feature ys ₂ -ys ₅ is obtained from the magnetometer, accelerometer, and gyroscope through the Kalman filtering algorithm according to the state data of the sensor, and is obtained by The data collected by the barometer obtains the air pressure characteristics ys ₆ , determines the WIFI connection state ys ₇ through the network module, locates the data collected by the position sensor, obtains the user's current location attributes (such as shopping malls, homes, companies, parks, etc.), and generates The feature ys ₈ ; the 10-axis information of the magnetometer, the acceleration sensor, the gyroscope, and the barometer can be further combined with the filtering algorithm or the principal component analysis algorithm to obtain new multi-dimensional data, and the corresponding feature ys ₉ can be generated. The feature ys ₁₀ is generated according to the user's emotional tag, and the features ys ₁₁ to ys ₁₃ are generated according to the user's gender, age, and hobby. For the features in the non-numeric form in the above features, the index number can be established to convert it into a digital representation. For example, for the feature of the running mode of the current system terminal state, the index number is used to represent the current state mode, such as 1 is the game mode, 2 is the entertainment mode, and 3 is the video mode. If the current running mode is the game mode, it is determined that the current system state ys1=1. After acquiring the features represented by all numbers, a long vector is obtained by fusing the above feature data, and the long vector is normalized to obtain the second feature vector s ₂ :

s ₂ ={ys ₁ ,ys ₂ ,…,ys _m }

The feature fusion module 403 is configured to perform fusion processing on the first feature vector and the second feature vector to generate a user feature matrix.

The feature fusion module 403 can perform matrix superposition of the first feature vector s ₁ and the second feature vector s ₂ to generate the following user feature matrix:

If the lengths of the first eigenvector and the second eigenvector are not equal, the short-length vector can be adjusted by means of zero-padding. If n<m, the length of the first eigenvector s ₁ can be extended by means of zero-padding. is m. If n>m, the length of the second feature vector s ₂ is extended to n by means of zero padding.

Or, in an optional implementation manner, after the first feature vector and the second feature vector are adjusted to the same length, the feature fusion module 403 performs matrix stacking on the first feature vector s ₁ and the second feature vector s ₂ , Then, in order to generate more abundant feature vectors and provide subsequent operations, the superimposed matrix is flipped to generate the following matrix:

Combine the matrices before and after the flip operation to obtain the following matrix as the user feature matrix.

The intent prediction module 404 is configured to acquire an intent label matching the user instruction according to the user feature matrix and the pre-trained intent prediction model.

In the embodiment of the present application, the intent prediction model is a classification model, which represents the relationship between the user feature matrix and the intent label. For example, the intent prediction model can be obtained by training a classification algorithm such as a convolutional neural network, a BP neural network (Back Propagation, back propagation), or an SVM (Support Vector Machine, support vector machine) algorithm. The following takes the convolutional neural network as an example to collect a large number of sample data of test users, extract features according to steps 101 to 103, and then label the extracted features. The user instruction is recorded, and the first feature vector is generated; the panoramic data when the user instruction is received is collected, and the second feature vector is generated according to the panoramic data. Then record the response of the electronic device to the user's instruction and the operation performed by the user based on the response. In an optional embodiment, the response of the electronic device to the user's instruction and the operation performed by the user based on the response may be recorded. The intent label data is automatically generated, or, in other embodiments, the user feature matrix can be labeled by manual labeling.

Input the above-mentioned user feature matrix with intent labels into the convolutional neural network for training, wherein the structure and hyperparameters of the convolutional neural network can be preset by the user as needed, and after training, the weight parameters of the network are determined to generate an intent prediction model .

In addition, it should be noted that, in order to fully characterize the user's intent, an intent label corresponding to a user feature matrix is an intent label set, and the intent label set has multiple sets of intent labels that can represent user instructions, location, time, environment, state, and habits. and other information labels, which describe the current panoramic portrait of the user.

By inputting the user feature matrix obtained by the feature fusion module 403 into the intent prediction model, an intent label matching the user instruction can be generated.

The instruction execution module 405 is configured to execute the user instruction according to the intention tag.

The intent tag acquired by the intent prediction module 404 is a tag set composed of multiple tags, and these tags constitute a panoramic portrait corresponding to the user instruction. The instruction execution module 405 responds to the user instruction and starts the corresponding target application, while pushing the tag set. For the target application, in this way, when the electronic device responds to the user's instruction, it is not only a simple operation such as starting the software, but further, after the target application is opened, it can perform more specific operations according to the received tag set.

For example, the instruction execution module 405 is further configured to: determine the target application corresponding to the user instruction according to the intent tag; start the target application, and send the intent tag to the target application, where the target application is based on The intent tag executes the corresponding operation.

For example, the user triggers the voice command "Xiaoou Xiaoou, order me a meal". After the system's voice recognition module detects the command, it obtains more data that can reflect the user's intention through panoramic perception technology, such as terminal status data, User status data and sensor status data, etc., use these data to supplement the context information of the instruction to obtain more comprehensive intent information, such as: current time, current location, scene, user eating habits, user hunger, etc. When the electronic device opens the third-party ordering application according to the instruction, it pushes the information to the third-party ordering application, and the third-party ordering application can recommend the specific ordering place and ordering content to the user according to the above information.

It can be seen from the above that, in the instruction execution device proposed in the embodiment of the present application, when receiving a user instruction, the first feature extraction module 401 generates a first feature vector according to the user instruction, and the second feature extraction module 402 obtains the current panoramic data, according to the panoramic view. The data generates a second feature vector, and the feature fusion module 403 fuses the first feature vector and the second feature vector to generate a user feature matrix, and then the intention prediction module 404 uses the user feature matrix as the pre-trained intention prediction model. Input data, obtain the intent tag matching the user instruction, and the instruction execution module 405 executes the user instruction according to the intent tag. This solution supplements the context information and context information of the user instruction by collecting panoramic data, and can more accurately understand the user's hidden information. Intentions are included to better execute user instructions.

The embodiments of the present application also provide an electronic device. The electronic device may be a smart phone, a tablet computer or the like. As shown in FIG. 7 , FIG. 7 is a schematic diagram of a first structure of an electronic device provided by an embodiment of the present application. Electronic device 300 includes processor 301 and memory 302 . The processor 301 is electrically connected to the memory 302 .

The processor 301 is the control center of the electronic device 300, uses various interfaces and lines to connect various parts of the entire electronic device, executes the electronic device by running or calling the computer program stored in the memory 302, and calling the data stored in the memory 302. Various functions of the device and processing data, so as to carry out the overall monitoring of the electronic device.

In this embodiment, the processor 301 in the electronic device 300 loads the instructions corresponding to the processes of one or more computer programs into the memory 302 according to the following steps, and the processor 301 executes the instructions stored in the memory 302 . A computer program in , which implements various functions:

When receiving a user instruction, generate a first feature vector according to the user instruction;

Obtain current panoramic data, and generate a second feature vector according to the panoramic data;

Fusion processing is performed on the first feature vector and the second feature vector to generate a user feature matrix;

According to the user feature matrix and the pre-trained intent prediction model, obtain an intent label matching the user instruction;

The user instruction is executed according to the intent tag.

In some embodiments, the user instruction is a voice instruction; when receiving the user instruction, when generating the first feature vector according to the user instruction, the processor 301 performs the following steps:

When a voice command is received, obtain the voice data collected by the voice component;

According to the pre-trained self-encoding recurrent neural network, a semantic feature vector of the speech data is generated, and the semantic feature vector is used as the first feature vector.

In some embodiments, the user instruction is a voice instruction; when receiving the user instruction, when generating the first feature vector according to the user instruction, the processor 301 performs the following steps:

When a voice command is received, obtain the voice data collected by the voice component;

Convert the voice data into a spectrogram according to an audio feature extraction algorithm;

According to the pre-trained self-encoding convolutional neural network and the spectrogram, a semantic feature vector of the speech data is generated, and the semantic feature vector is used as the first feature vector.

In some embodiments, the user instruction is a touch instruction; when receiving the user instruction, when generating the first feature vector according to the user instruction, the processor 301 performs the following steps:

When receiving the touch command, obtain the touch data collected by the touch sensor;

Convert the touch data into the first feature vector.

In some embodiments, the panoramic data includes terminal state data, user state data and sensor state data; when acquiring the current panoramic data and generating the second feature vector according to the panoramic data, the processor 301 performs the following steps:

Obtain current terminal status data, user status data and sensor status data;

generating terminal status features according to the terminal status data, generating user status features according to the user status data, and generating terminal scene features according to the sensor status data;

The second feature vector is generated by fusing the terminal state feature, the user state feature and the terminal context feature.

In some embodiments, when acquiring the current user state data, the processor 301 performs the following steps:

Call the camera component to capture the user's face image;

Identify the user's face image according to a preset convolutional neural network model, and generate a user emotional label;

Obtain user information, and use the user emotion tag and the user information as the user state data.

In some embodiments, the first feature vector and the second feature vector are fused to generate a user feature matrix, the processor 301 performs the following steps:

Perform matrix superposition processing on the first feature vector and the second feature vector to generate a user feature matrix.

In some embodiments, when executing the user instruction according to the intent tag, the processor 301 performs the following steps:

Determine the target application corresponding to the user instruction according to the intent tag;

The target application is started, and the intent tag is sent to the target application, wherein the target application performs a corresponding operation based on the intent tag.

Memory 302 may be used to store computer programs and data. The computer program stored in the memory 302 contains instructions executable in the processor. A computer program can be composed of various functional modules. The processor 301 executes various functional applications and data processing by calling the computer program stored in the memory 302 .

In some embodiments, as shown in FIG. 8 , FIG. 8 is a schematic diagram of a second structure of an electronic device provided by an embodiment of the present application. The electronic device 300 further includes: a radio frequency circuit 303 , a display screen 304 , a control circuit 305 , an input unit 306 , an audio circuit 307 , a sensor 308 and a power supply 309 . The processor 301 is electrically connected to the radio frequency circuit 303 , the display screen 304 , the control circuit 305 , the input unit 306 , the audio circuit 307 , the sensor 308 and the power supply 309 respectively.

The radio frequency circuit 303 is used to send and receive radio frequency signals to communicate with network equipment or other electronic equipment through wireless communication.

The display screen 304 may be used to display information entered by or provided to the user and various graphical user interfaces of the electronic device, which may consist of images, text, icons, video, and any combination thereof.

The control circuit 305 is electrically connected to the display screen 304 for controlling the display screen 304 to display information.

Input unit 306 may be used to receive input numbers, character information, or user characteristic information (eg, fingerprints), and generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control. Wherein, the input unit 306 may include a fingerprint identification module.

The audio circuit 307 can provide an audio interface between the user and the electronic device through speakers and microphones. Among them, the audio circuit 307 includes a microphone. The microphone is electrically connected to the processor 301 . The microphone is used for receiving voice information input by the user.

The sensor 308 is used to collect external environment information. The sensor 308 may include one or more of an ambient brightness sensor, an acceleration sensor, a gyroscope, and the like.

Power supply 309 is used to power various components of electronic device 300 . In some embodiments, the power supply 309 may be logically connected to the processor 301 through a power management system, so as to implement functions such as managing charging, discharging, and power consumption through the power management system.

Although not shown in FIG. 8 , the electronic device 300 may further include a camera, a Bluetooth module, and the like, which will not be repeated here.

As can be seen from the above, the embodiment of the present application provides an electronic device, the electronic device can receive user instructions, such as voice instructions, touch instructions, or holding instructions, etc., and when receiving the user instruction, generates a first feature vector; then collect the current panoramic data of the electronic device, such as user information, sensor status data, application usage information in the electronic device, etc., generate a second feature vector according to the collected panoramic data, and then use the first feature vector The vector and the second feature vector are fused to generate a user feature matrix. According to the user feature matrix and the pre-trained intent prediction model, an intent label corresponding to the user feature matrix is generated, and user instructions are executed according to the intent label. In this way, according to the panoramic data collected when receiving the command, the user's command is supplemented with complete contextual information and the information of the contextual environment, so that the electronic device can more accurately understand the user's implicit intention, so as to better execute the user's instruction.

An embodiment of the present application further provides a storage medium, where a computer program is stored in the storage medium, and when the computer program runs on a computer, the computer executes the instruction execution method described in any of the foregoing embodiments.

It should be noted that those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium , the storage medium may include, but is not limited to, a read only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and the like.

The instruction execution method, apparatus, storage medium, and electronic device provided by the embodiments of the present application have been described in detail above. The principles and implementations of the present application are described herein using specific examples, and the descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application; meanwhile, for those skilled in the art, according to the Thoughts, there will be changes in specific embodiments and application scopes. To sum up, the contents of this specification should not be construed as limitations on the present application.

Claims (12)

1. An instruction execution method, comprising:

when a user instruction is received, generating a first feature vector according to the user instruction;

acquiring current panoramic data and generating a second eigenvector according to the panoramic data;

fusing the first feature vector and the second feature vector to generate a user feature matrix;

acquiring an intention label matched with the user instruction according to the user characteristic matrix and a pre-trained intention prediction model;

executing the user instruction according to the intention tag.

2. The instruction execution method of claim 1 wherein the user instruction is a voice instruction; when a user instruction is received, generating a first feature vector according to the user instruction, wherein the step comprises the following steps:

when a voice instruction is received, voice data acquired by a voice component is acquired;

and generating a semantic feature vector of the voice data according to a pre-trained self-coding recurrent neural network, and taking the semantic feature vector as the first feature vector.

3. The instruction execution method of claim 1 wherein the user instruction is a voice instruction; when a user instruction is received, generating a first feature vector according to the user instruction, wherein the step comprises the following steps:

when a voice instruction is received, voice data acquired by a voice component is acquired;

converting the voice data into a spectrogram according to an audio feature extraction algorithm;

and generating a semantic feature vector of the voice data according to a pre-trained self-coding convolutional neural network and the spectrogram, and taking the semantic feature vector as the first feature vector.

4. The instruction execution method of claim 1, wherein the user instruction is a touch instruction; when a user instruction is received, generating a first feature vector according to the user instruction, wherein the step comprises the following steps:

when a touch instruction is received, acquiring touch data acquired by a touch sensor;

converting the touch data into the first feature vector.

5. The instruction execution method of any of claims 1 to 4, wherein the panoramic data comprises terminal state data, user state data, and sensor state data; the method comprises the steps of obtaining current panoramic data and generating a second eigenvector according to the panoramic data, wherein the steps comprise:

acquiring current terminal state data, user state data and sensor state data;

generating terminal state characteristics according to the terminal state data, generating user state characteristics according to the user state data, and generating terminal scene characteristics according to the sensor state data;

and fusing the terminal state feature, the user state feature and the terminal scene feature to generate the second feature vector.

6. The instruction execution method of claim 5 wherein the step of obtaining current user state data comprises:

calling a camera assembly to capture a face image of a user;

identifying the user face image according to a preset convolutional neural network model to generate a user emotion label;

and acquiring user information, and taking the user emotion label and the user information as the user state data.

7. The instruction execution method of any one of claims 1 to 4, wherein the step of performing fusion processing on the first feature vector and the second feature vector to generate a user feature matrix comprises:

and performing matrix superposition processing on the first eigenvector and the second eigenvector to generate a user characteristic matrix.

8. The instruction execution method of any of claims 1-4, wherein executing the user instruction according to the intent tag comprises:

determining a target application corresponding to the user instruction according to the intention label;

and starting the target application and sending the intention label to the target application, wherein the target application executes corresponding operation based on the intention label.

9. The instruction execution method of any one of claims 1 to 4, wherein the step of obtaining the intent tag matching the user instruction according to the user feature matrix and a pre-trained intent prediction model comprises:

and acquiring an intention label matched with the user instruction according to the user characteristic matrix and an intention prediction model obtained based on neural network training.

10. An instruction execution apparatus, comprising:

the first feature extraction module is used for generating a first feature vector according to a user instruction when the user instruction is received;

the second feature extraction module is used for acquiring current panoramic data and generating a second feature vector according to the panoramic data;

the feature fusion module is used for carrying out fusion processing on the first feature vector and the second feature vector to generate a user feature matrix;

the intention prediction module is used for acquiring an intention label matched with the user instruction according to the user characteristic matrix and a pre-trained intention prediction model;

and the instruction execution module is used for executing the user instruction according to the intention label.

11. A storage medium having stored thereon a computer program, characterized in that, when the computer program is run on a computer, it causes the computer to execute the instruction execution method according to any one of claims 1 to 9.

12. An electronic device comprising a processor and a memory, the memory storing a computer program, wherein the processor is configured to perform the instruction execution method of any of claims 1 to 9 by invoking the computer program.

Images