CN106815264B - Information processing method and system - Google Patents

Abstract

The embodiment of the invention discloses an information processing method and system. The method comprises the following steps: obtaining operation information and image data; the operation information is obtained by detecting a trigger operation aiming at a display interface of a first application when the first application in the mobile terminal is in an activated state; the image data is acquired by an image acquisition unit of the mobile terminal; the image acquisition unit and the display unit are on the same plane; analyzing the image data, identifying facial feature data in the image data, and obtaining facial expression information based on the facial feature data; first feedback information is generated based on the facial expression information and the operation information.

Description

An information processing method and system

technical field

The present invention relates to information processing technology, in particular to an information processing method and system.

Background technique

With the rapid development of Internet technology, a large number of mobile terminal applications (APPs) have emerged. These applications attach great importance to the user experience, and are very concerned about whether the user's interactive experience and application functions can effectively meet the user's needs. The user's feedback and evaluation of the application is an important basis for data operations, product managers, designers, and software engineers to optimize programs.

In the prior art, the methods for collecting application feedback mainly include the following: 1. Users actively rate or comment through the application store; 2. Users actively comment or rate feedback through websites or forums; 3. Specified parameter information collection. Among them, the above-mentioned methods 1 and 2 occupy most of the proportions. In the process of using the app, the app will occasionally prompt the user to go to the app store to rate or comment. This feedback collection method is not convenient and flexible enough, and the user experience is poor. The parameter information specified in Mode 3 cannot intuitively correspond to the specific feedback experience of the user, and cannot be directed to the specific use process of the user.

SUMMARY OF THE INVENTION

In order to solve the existing technical problems, the embodiments of the present invention provide an information processing method and system, which can directly obtain a user's usage feedback.

In order to achieve the above-mentioned purpose, the technical scheme of the embodiment of the present invention is realized as follows:

An embodiment of the present invention provides an information processing method, and the method includes:

Obtaining operation information and image data; wherein, the operation information is operation information obtained by detecting a trigger operation on the display interface of the first application when the first application in the mobile terminal is activated; and the image data is the image data acquired by the image acquisition unit of the mobile terminal; the image acquisition unit and the display unit are on the same plane;

Identifying facial feature data in the image data, and obtaining facial expression information based on the facial feature data;

First feedback information is generated based on the facial expression information and the operation information.

In the above solution, generating the first feedback information based on the facial expression information and the operation information includes:

A corresponding first user experience parameter is obtained based on the facial expression information and/or the operation information, and first feedback information of the operation position corresponding to the operation information is generated based on the first user experience parameter.

In the above solution, after analyzing the image data and identifying the facial feature data in the image data, the method further includes:

Obtain the relative position information and direction information of the mobile terminal itself;

The eye feature data, the relative position information and the direction information in the facial feature data are associated to obtain focus point information; the focus point information represents the focus position information of the eyes on the mobile terminal.

In the above solution, the method further includes: generating second feedback information based on the attention point information and the operation information.

In the above solution, the generating the second feedback information based on the focus point information and the operation information includes:

A corresponding second user experience parameter is obtained based on the point of interest information within the preset time period and the operation information within the preset time period, and a second feedback of the operation position corresponding to the operation information is generated based on the second user experience parameter information.

In the above solution, generating the first feedback information based on the facial expression information and the operation information includes:

First feedback information is generated based on the facial expression information, the point of interest information, and the operation information.

In the above solution, generating the first feedback information based on the facial expression information, the point of interest information and the operation information includes:

A corresponding third user experience parameter is obtained based on the facial expression information and the point of interest information within a preset time period and operation information within the preset time period, and the third user experience parameter is generated based on the third user experience parameter. The operation information corresponds to the first feedback information of the operation position.

An embodiment of the present invention also provides an information processing system, the system includes: an acquisition unit, an image processing unit, and an information generation unit; wherein,

The obtaining unit is configured to obtain operation information and image data; wherein, the operation information is an operation obtained by detecting a trigger operation on the display interface of the first application when the first application in the mobile terminal is activated information; the image data is the image data acquired by the image acquisition unit of the mobile terminal; the image acquisition unit and the display unit are on the same plane;

The image processing unit is used to identify facial feature data in the image data, and obtain facial expression information based on the facial feature data;

The information generating unit is configured to generate first feedback information based on the facial expression information obtained by the image processing unit and the operation information obtained by the obtaining unit.

In the above solution, the information generating unit is configured to obtain a corresponding first user experience parameter based on the facial expression information and/or the operation information, and generate an operation position corresponding to the operation information based on the first user experience parameter the first feedback information.

In the above solution, the obtaining unit is also used to obtain the relative position information and direction information of the mobile terminal itself;

The image processing unit is further configured to associate the eye feature data in the facial feature data, the relative position information of the mobile terminal itself obtained by the obtaining unit, and the direction information to obtain focus information; The attention point information represents the focus position information of the eyes on the mobile terminal.

In the above solution, the information generating unit is further configured to generate second feedback information based on the attention point information and the operation information.

In the above solution, the information generating unit is configured to obtain the corresponding second user experience parameter based on the point of interest information within the preset time period and the operation information within the preset time period, and based on the second user experience parameter Second feedback information corresponding to the operation position of the operation information is generated.

In the above solution, the information generating unit is configured to generate the first feedback information based on the facial expression information, the attention point information and the operation information.

In the above solution, the information generating unit is configured to obtain a corresponding third user experience parameter based on the facial expression information and the point of interest information within a preset time period and in combination with operation information within the preset time period , generating first feedback information of the operation position corresponding to the operation information based on the third user experience parameter.

In the information processing method and system according to the embodiments of the present invention, the method includes: obtaining operation information and image data; wherein, the operation information is that a first application in the mobile terminal is in an activated state, and detection of a target for the first application is detected. The operation information obtained by the trigger operation of the display interface of the mobile terminal; the image data is the image data collected by the image acquisition unit of the mobile terminal; the image acquisition unit and the display unit are on the same plane; identify the image data in the image data Facial feature data, obtaining facial expression information based on the facial feature data; generating first feedback information based on the facial expression information and the operation information. By adopting the technical solutions of the embodiments of the present invention, during the process of the user using the first application, the facial expression information representing the user is identified, the operation information is obtained, and the first feedback information is generated based on the facial expression information and the operation information, thereby As a basis for further optimizing and modifying the first application, on the one hand, the direct and active acquisition of user feedback information is realized, and the user does not need to log in to the application store or website to comment or score, which greatly improves the user's operating experience; On the other hand, the technical solution of the embodiment of the present invention collects and identifies information during the use of the user, so as to know the specific location of the poor user experience in the first application, so as to facilitate the subsequent operation and maintenance. To improve or optimize the problem, it provides a detailed basis for the operation and maintenance of the application.

Description of drawings

1 is a schematic diagram of an application scenario of an information processing method according to an embodiment of the present invention;

2 is a schematic flowchart of an information processing method according to Embodiment 1 of the present invention;

3 is a schematic flowchart of an information processing method according to Embodiment 2 of the present invention;

4 is a schematic flowchart of an information processing method according to Embodiment 3 of the present invention;

FIG. 5 is a schematic diagram of the composition and structure of an information processing system according to an embodiment of the present invention.

Detailed ways

The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

1 is a schematic diagram of an application scenario of an information processing method according to an embodiment of the present invention; as shown in FIG. 1 , including a server 11 and a mobile terminal 12; the mobile terminal 12 and the server 11 can be connected through a network (such as a wired network) and/or wireless network) connection. At least one application (APP) is pre-installed in the mobile terminal 12 . The server 11 may be the server or server cluster of the at least one application; the server 11 may also be a server or server cluster to which a third-party user experience information statistics platform belongs.

The technical solutions of the various embodiments of the present invention are applied to the above-mentioned server 11 and mobile terminal 12; when an application is run by the mobile terminal 12, the operation information of the user on the display interface of the application is obtained, and the facial feature data of the user is obtained at the same time , send the operation information and the facial feature data to the server 11 for analysis and processing, so that the server 11 recognizes the user's facial expression information according to the facial feature data, and then based on the facial expression information and the The operation information obtains first feedback information, and the first feedback information indicates that the user's facial expression is happy, calm, or dissatisfied at the location included in the operation information, so as to know whether the location included in the operation information causes the user to operate Poor experience, that is, whether optimization improvements are needed.

The above example in FIG. 1 is only an example of an application architecture for implementing the embodiment of the present invention, and the embodiment of the present invention is not limited to the application structure described in the above FIG. 1 . Based on the application architecture, various embodiments of the present invention are proposed.

Example 1

Embodiments of the present invention provide an information processing method. FIG. 2 is a schematic flowchart of an information processing method according to Embodiment 1 of the present invention; as shown in FIG. 2 , the information processing method includes:

Step 201: Obtain operation information and image data; wherein, the operation information is operation information obtained by detecting a trigger operation on a display interface of the first application when a first application in the mobile terminal is activated; the The image data is image data acquired by the image acquisition unit of the mobile terminal; the image acquisition unit and the display unit are on the same plane.

The information processing method described in this embodiment is applied to an information processing system; the information processing system may be implemented by a server in this embodiment, and of course, in other embodiments, it may be implemented by a mobile terminal and a server. In this step, the obtaining of the operation information and the image data includes: the server obtains the operation information and the image data of the mobile terminal.

Specifically, when the mobile terminal activates the first application, it outputs a display interface representing the first application, detects a trigger operation on the display interface, and obtains operation information; the operation information includes operation gesture information and operation position information; Wherein, the operation gesture information includes: click gesture, double-click gesture, sliding gesture, drag gesture, zoom gesture, rotation gesture, parameter (such as volume parameter, brightness parameter, etc.) adjustment gesture, etc.; the operation position information is: Operation position information of the operation gesture; the operation position information may be for a function button. Further, the operation information may also be continuous operation information within a period of time.

When detecting a trigger operation for the display interface, the mobile terminal generates a first instruction, and enables an image acquisition unit of the mobile terminal based on the first instruction; in other implementation manners, the mobile terminal also The image acquisition unit of the mobile terminal may be enabled when the first application is activated, or the mobile terminal may also enable the image acquisition unit of the mobile terminal based on the detected trigger instruction. Make specific restrictions. In this embodiment, the image acquisition unit and the display unit of the mobile terminal are on the same plane, and it can be understood that the image acquisition unit may be implemented by the front camera of the mobile terminal.

In this embodiment, the mobile terminal sends the obtained operation information and corresponding image data to the information processing system.

Step 202: Identify facial feature data in the image data, and obtain facial expression information based on the facial feature data.

In this step, the information processing system analyzes the image data, and first performs preprocessing on the image data (eg, denoising, normalization of pixel positions or illumination variables), and segmentation, localization, or tracking of faces, and so on. Further, the extraction of facial feature data is performed on the image data, including converting the pixel data into representations of the shape, motion, color, muscle and spatial structure of the face and its components, and the extracted facial feature data is used for subsequent follow-up. expression classification. Further, the pre-expression classifier in the information processing system, the expression classifier includes the correspondence between multiple groups of facial feature data and expression information, or the expression classifier includes an expression classification model, and the facial expression The feature data is input into the expression classifier, and the expression information corresponding to the facial feature data is output, that is, the facial expression information is obtained based on the facial feature data. For the facial analysis method and the recognition of expression information described in this step, reference may be made to any analysis and recognition method in the prior art, which will not be described in detail in this embodiment.

Of course, in other embodiments, this step can also be implemented by a mobile terminal, that is, the mobile terminal analyzes and recognizes the obtained image data, obtains the facial expression information, and then sends the facial expression information to the information processing system. For a specific implementation manner, reference may be made to the above description, which will not be repeated here.

Step 203: Generate first feedback information based on the facial expression information and the operation information.

Here, the generating the first feedback information based on the facial expression information and the operation information includes: obtaining a corresponding first user experience parameter based on the facial expression information and/or the operation information, and based on the first The user experience parameter generates the first feedback information of the operation position corresponding to the operation information.

Specifically, the information processing system pre-stores a plurality of sets of correspondence between facial expression information and the first user experience parameter. For example, when the facial expression information is pleasant, the corresponding first user experience parameter is 5; when the facial expression information is calm, the corresponding first user experience parameter is 3; when the facial expression information is dissatisfaction , the corresponding first user experience parameter is 0. Of course, in other implementation manners, the facial expression information and the first user experience parameter may also be preset in other corresponding manners, which are not described in detail in this embodiment. That is to say, the facial expression information can represent the user's experience degree, so when the user experience degree represented by the facial expression information reaches the first preset threshold, it means that the user experience degree is good, according to the facial expression information corresponding The first feedback information corresponding to the operation information (including the operation position information) is generated from the operation information (including the operation position information), and at this time, the first feedback information indicates that the first application is at the operation position to the user Bringing a better experience, the operation function at the operation position or the provided content is worthy of recommendation; correspondingly, when the user experience degree represented by the facial expression information does not reach the second preset threshold, the second preset The threshold value is less than the first preset threshold value, which means that the user experience is poor, and the first feedback information corresponding to the operation information (including the operation position information) is generated according to the operation information (including the operation position information) corresponding to the facial expression information. , at this time, the first feedback information indicates that the first application brings a poor experience to the user at the operation position, and the operation function or provided content at the operation position needs to be further optimized or improved.

As another implementation manner, the information processing system obtains the corresponding first user experience parameter based on a combination of the facial expression information and the operation information. For example, when the facial expression information is pleasant, and the number of operations included in the operation information is less than a first threshold, the corresponding first user experience parameter is 5; when the facial expression information is calm, and the When the number of operations included in the operation information is greater than the first threshold and smaller than the second threshold, the corresponding first user experience parameter is 3; when the facial expression information is dissatisfied and the number of operations included in the operation information is greater than the second threshold, The corresponding first user experience parameter is 0. Of course, in other implementation manners, the facial expression information and the first user experience parameter may also be preset in other corresponding manners, which are not described in detail in this embodiment. That is to say, the combination of the facial expression information and the operation information can represent the user's experience. For example, in a scene, the user wants to find a function entry through an input operation, and can only see it through multiple sliding operations. The user will show an unhappy expression; then in this scenario, the information processing system obtains the facial expression information representing the unhappy expression and the operation information containing multiple sliding operations, and combines the above two information to obtain the corresponding The first user experience parameter is 0, which means that in the operation position corresponding to the operation information, the user experience is poor and needs to be optimized or improved.

By adopting the technical solutions of the embodiments of the present invention, during the process of using the first application, the user's facial expression information is identified, operation information is obtained, and first feedback information is generated based on the facial expression information and the operation information, thereby serving as a The basis for further optimizing and modifying the first application. In this way, on the one hand, the direct and active acquisition of user feedback information is realized, and the user does not need to log in to the application store or website to comment or score, which greatly improves the user's operating experience; On the one hand, the technical solutions of the embodiments of the present invention collect and identify information during the use of the user, so as to know the specific location of the poor user experience in the first application, so as to be able to solve specific problems in the subsequent operation and maintenance process. Improve or optimize to provide a detailed basis for the operation and maintenance of the application.

Embodiment 2

Embodiments of the present invention provide an information processing method. FIG. 3 is a schematic flowchart of an information processing method according to Embodiment 2 of the present invention; as shown in FIG. 3 , the information processing method includes:

Step 301: Obtain operation information and image data; wherein, the operation information is operation information obtained by detecting a trigger operation on a display interface of the first application when a first application in the mobile terminal is activated; the The image data is image data acquired by the image acquisition unit of the mobile terminal; the image acquisition unit and the display unit are on the same plane.

The information processing method described in this embodiment is applied to an information processing system; the information processing system may be implemented by a server in this embodiment, and of course, in other embodiments, it may be implemented by a mobile terminal and a server. In this step, the obtaining operation information and image data includes: the information processing system obtains the operation information and image data of the mobile terminal.

Specifically, when the mobile terminal activates the first application, it outputs a display interface representing the first application, detects a trigger operation on the display interface, and obtains operation information; the operation information includes operation gesture information and operation position information; Wherein, the operation gesture information includes: click gesture, double-click gesture, sliding gesture, drag gesture, zoom gesture, rotation gesture, parameter (such as volume parameter, brightness parameter, etc.) adjustment gesture, etc.; the operation position information is: Operation position information of the operation gesture; the operation position information may be for a function button. Further, the operation information may also be continuous operation information within a period of time.

When detecting a trigger operation for the display interface, the mobile terminal generates a first instruction, and enables an image acquisition unit of the mobile terminal based on the first instruction; in other implementation manners, the mobile terminal also The image acquisition unit of the mobile terminal may be enabled when the first application is activated, or the mobile terminal may also enable the image acquisition unit of the mobile terminal based on the detected trigger instruction. Make specific restrictions. In this embodiment, the image acquisition unit and the display unit of the mobile terminal are on the same plane, and it can be understood that the image acquisition unit may be implemented by the front camera of the mobile terminal.

In this embodiment, the mobile terminal sends the obtained operation information and corresponding image data to the information processing system.

Step 302: Identify facial feature data in the image data; and obtain relative position information and orientation information of the mobile terminal itself.

In this step, the information processing system analyzes the image data, and first performs preprocessing on the image data (eg, denoising, normalization of pixel positions or illumination variables), and segmentation, localization, or tracking of faces, and so on. Further, the extraction of facial feature data is performed on the image data, including converting the pixel data into the representation of the shape, motion, color, muscle and spatial structure of the face and its components; the extraction method of the specific facial feature data can refer to Any facial recognition method in the prior art will not be specifically described in this embodiment.

In this embodiment, the information processing system obtains relative position information and direction information of the mobile terminal; the relative position information is the relative position relationship between the mobile terminal and the holder. Specifically, the mobile terminal is provided with at least one of the following sensing units: a gravity sensing unit, an acceleration sensing unit, a distance sensing unit, an iris recognition unit, etc. The gravity sensing unit or the acceleration sensing unit obtains direction information, and the direction information may be the angle between the direction of the center of gravity of the mobile terminal relative to the long-side direction or the short-side direction of the mobile terminal, and the direction information is also That is, the posture change information of the mobile terminal. The mobile terminal may also obtain relative position information between the mobile terminal and the holder through the distance sensing unit or the iris identification unit, wherein the distance sensing unit and the iris identification unit usually Set on the same surface of the display unit of the mobile terminal, when the user holds the mobile terminal, the distance to the user can be detected by the distance sensing unit, or the iris recognition unit can be used to identify The relative orientation between the mobile terminal and the user's eyes.

Step 303 : Associate the eye feature data, the relative position information and the direction information in the facial feature data to obtain focus information; the focus information represents the focus position information of the eyes on the mobile terminal.

In this embodiment, the information processing system obtains the point of interest information based on the eye feature data included in the facial feature data and the relative position information of the mobile terminal itself and the direction information, and obtains the point of interest information. The information represents the focus position information of the eyes of the user holding the mobile terminal on the mobile terminal, and can also be understood as the position information where the content browsed by the eyes of the holding user is located. Specifically, the information processing system may obtain the line-of-sight direction information of the eyes of the holding user based on the eye feature data, and further, based on the relative position information of the mobile terminal itself and the direction information, determine the The relative positional relationship between the mobile terminal and the holding user, the focusing range of the eye of the holding user focusing on the mobile terminal is obtained based on the gaze direction information and the relative positional relationship, and the focusing range is generated based on the focusing range Focus information.

Step 304: Generate second feedback information based on the attention point information and the operation information.

Here, the generating the second feedback information based on the attention point information and the operation information includes: obtaining a corresponding second user based on the attention point information within a preset time period and the operation information within the preset time period an experience parameter, generating second feedback information of the operation position corresponding to the operation information based on the second user experience parameter.

Specifically, the information processing system obtains the corresponding second user experience parameter based on a combination of the focus point information and the operation information within a preset time period. For example, within a preset time period t, when the range ratio of the point of interest information (that is, the focus position of the holding user's eyes on the mobile terminal) changes is greater than the first threshold, and the number of operations included in the operation information is greater than When the second threshold is the second threshold, the corresponding second user experience parameter is 0; when the range ratio of the point of interest information (that is, the focus position of the holding user's eyes on the mobile terminal) is greater than the third threshold and less than the first threshold (the When the first threshold is less than the third threshold), and the number of operations included in the operation information is less than the second threshold and greater than the fourth threshold (the fourth threshold is less than the second threshold), the corresponding second user experience The parameter is 3; when the point of interest information (that is, the focus position of the user's eyes on the mobile terminal) does not change, or the range ratio of the change is less than the third threshold, and the number of operations included in the operation information is less than the third threshold. When there are four thresholds, the corresponding second user experience parameter is 5. Of course, in other implementation manners, the information processing system may also refer to other implementation manners for the manner in which the information processing system obtains the point of interest information by associating the eye feature data, the relative position information and the direction information, that is, the information processing system The focus position information of the user's eyes on the mobile terminal can be obtained according to the eye feature data, the relative position information and the direction information. Reference can be made to any image recognition combined modeling technology in the prior art. This embodiment is not described in detail. For example, in a scenario, if the user wants to find a function entry through an input operation, the user will look around for the location of the function entry; then the proportion of the range of changes in the user's focus information obtained by the information processing system will be greater than the first threshold. ; Correspondingly, in the process of the user searching for the location of the function entry, with multiple trigger operations, that is, the number of operations included in the information processing system's obtained operation information is greater than the second threshold; in this scenario, the user It takes a long time to find the function entry, indicating that the application brings a poor operation experience to the user at the current operation position, and needs to be optimized or improved.

By adopting the technical solutions of the embodiments of the present invention, in the process of the user using the first application, by identifying the user's point of interest information and obtaining the operation information, the second feedback information is generated based on the point of interest information and the operation information, thereby As a basis for further optimizing and modifying the first application, on the one hand, the direct and active acquisition of user feedback information is realized, and the user does not need to log in to the application store or website to comment or score, which greatly improves the user's operating experience; On the other hand, the technical solution of the embodiment of the present invention collects and identifies information during the use of the user, so as to know the specific location of the poor user experience in the first application, so as to facilitate the subsequent operation and maintenance. To improve or optimize the problem, it provides a detailed basis for the operation and maintenance of the application.

Embodiment 3

Embodiments of the present invention provide an information processing method. FIG. 4 is a schematic flowchart of an information processing method according to Embodiment 3 of the present invention; as shown in FIG. 4 , the information processing method includes:

Step 401: Obtain operation information and image data; wherein, the operation information is operation information obtained by detecting a trigger operation on a display interface of the first application when a first application in the mobile terminal is activated; the The image data is image data acquired by the image acquisition unit of the mobile terminal; the image acquisition unit and the display unit are on the same plane.

The information processing method described in this embodiment is applied to an information processing system; the information processing system may be implemented by a server in this embodiment, and of course, in other embodiments, it may be implemented by a mobile terminal and a server. In this step, the obtaining operation information and image data includes: the information processing system obtains the operation information and image data of the mobile terminal.

Specifically, when the mobile terminal activates the first application, it outputs a display interface representing the first application, detects a trigger operation on the display interface, and obtains operation information; the operation information includes operation gesture information and operation position information; Wherein, the operation gesture information includes: click gesture, double-click gesture, sliding gesture, drag gesture, zoom gesture, rotation gesture, parameter (such as volume parameter, brightness parameter, etc.) adjustment gesture, etc.; the operation position information is: Operation position information of the operation gesture; the operation position information may be for a function button. Further, the operation information may also be continuous operation information within a period of time.

When detecting a trigger operation for the display interface, the mobile terminal generates a first instruction, and enables an image acquisition unit of the mobile terminal based on the first instruction; in other implementation manners, the mobile terminal also The image acquisition unit of the mobile terminal may be enabled when the first application is activated, or the mobile terminal may also enable the image acquisition unit of the mobile terminal based on the detected trigger instruction. Make specific restrictions. In this embodiment, the image acquisition unit and the display unit of the mobile terminal are on the same plane, and it can be understood that the image acquisition unit may be implemented by the front camera of the mobile terminal.

In this embodiment, the mobile terminal sends the obtained operation information and corresponding image data to the information processing system.

Step 402: Identify facial feature data in the image data, and obtain facial expression information based on the facial feature data.

In this step, the information processing system analyzes the image data, and first performs preprocessing on the image data (eg, denoising, normalization of pixel positions or illumination variables), and segmentation, localization, or tracking of faces, and so on. Further, the extraction of facial feature data is performed on the image data, including converting the pixel data into representations of the shape, motion, color, muscle and spatial structure of the face and its components, and the extracted facial feature data is used for subsequent follow-up. expression classification. Further, the pre-expression classifier in the information processing system, the expression classifier includes the correspondence between multiple groups of facial feature data and expression information, or the expression classifier includes an expression classification model, and the facial expression The feature data is input into the expression classifier, and the expression information corresponding to the facial feature data is output, that is, the facial expression information is obtained based on the facial feature data. For the facial analysis method and the recognition of expression information described in this step, reference may be made to any analysis and recognition method in the prior art, which will not be described in detail in this embodiment.

Of course, in other embodiments, this step can also be implemented by a mobile terminal, that is, the mobile terminal analyzes and recognizes the obtained image data, obtains the facial expression information, and then sends the facial expression information to the information processing system. For a specific implementation manner, reference may be made to the above description, which will not be repeated here.

Step 403: Obtain the relative position information and direction information of the mobile terminal itself; associate the eye feature data, the relative position information and the direction information in the facial feature data to obtain the point of interest information; the point of interest information represents The focus position information of the eyes on the mobile terminal.

In this embodiment, the information processing system obtains relative position information and direction information of the mobile terminal; the relative position information is the relative position relationship between the mobile terminal and the holder. Specifically, the mobile terminal is provided with at least one of the following sensing units: a gravity sensing unit, an acceleration sensing unit, a distance sensing unit, an iris recognition unit, etc. The gravity sensing unit or the acceleration sensing unit obtains direction information, and the direction information may be the angle between the direction of the center of gravity of the mobile terminal relative to the long-side direction or the short-side direction of the mobile terminal, and the direction information is also That is, the posture change information of the mobile terminal. The mobile terminal may also obtain relative position information between the mobile terminal and the holder through the distance sensing unit or the iris identification unit, wherein the distance sensing unit and the iris identification unit usually Set on the same surface of the display unit of the mobile terminal, when the user holds the mobile terminal, the distance to the user can be detected by the distance sensing unit, or the iris recognition unit can be used to identify The relative orientation between the mobile terminal and the user's eyes.

Further, the information processing system associates the eye feature data contained in the facial feature data, the relative position information of the mobile terminal itself, and the direction information, and obtains point of interest information, and the point of interest information represents The focus position information of the eyes of the holding user on the mobile terminal of the mobile terminal may also be understood as the position information of the content browsed by the eyes of the holding user. Specifically, the information processing system may obtain the line-of-sight direction information of the eyes of the holding user based on the eye feature data, and further, based on the relative position information of the mobile terminal itself and the direction information, determine the The relative positional relationship between the mobile terminal and the holding user, the focusing range of the eye of the holding user focusing on the mobile terminal is obtained based on the gaze direction information and the relative positional relationship, and the focusing range is generated based on the focusing range Focus information.

Step 404: Generate first feedback information based on the facial expression information, the attention point information, and the operation information.

Here, the generating the first feedback information based on the facial expression information, the attention point information and the operation information includes:

A corresponding third user experience parameter is obtained based on the facial expression information and the point of interest information within a preset time period and operation information within the preset time period, and the third user experience parameter is generated based on the third user experience parameter. The operation information corresponds to the first feedback information of the operation position.

Specifically, the information processing system obtains the corresponding third user experience parameter based on the facial expression information within the preset time period and the point of interest information within the preset time period, in combination with the operation information. For example, within the preset time period t, the obtained facial expression information is pleasant, and the point of interest information (that is, the focus position of the holding user's eyes on the mobile terminal) does not change, or the range ratio of the change is smaller than the third threshold, and when the number of operations included in the operation information is less than the fourth threshold, the corresponding second user experience parameter is 5; The ratio of the range of changes in the focus position on the mobile terminal) is greater than a third threshold and less than a first threshold (the first threshold is less than the third threshold), and the number of operations included in the operation information is less than the second threshold and greater than the fourth threshold (the fourth threshold is less than the second threshold), the corresponding second user experience parameter is 3; or, the obtained facial expression information is dissatisfaction, and the point of interest information (that is, holding the user's eyes on the mobile terminal) When the ratio of the range of changes in the focus position) is greater than the first threshold, and the number of operations included in the operation information is greater than the second threshold, the corresponding second user experience parameter is 0. Certainly, the manner in which the information processing system obtains the third user experience parameter may also refer to other manners, which will not be repeated in this embodiment. For example, in a scenario, if the user wants to find a function entry through an input operation, the user will look around for the location of the function entry; then the proportion of the range of changes in the user's focus information obtained by the information processing system will be greater than the first threshold. ; Correspondingly, in the process of the user finding the location of the function entry, with multiple trigger operations, that is, the number of operations included in the information processing system to obtain the operation information is greater than the second threshold; Correspondingly, after the user spends In the scenario of searching for the location of the function entry for a long time, the user will show an unhappy expression; in this scenario, the information processing system The analysis of the operation information determines that the current user experience is poor, that is, it takes a long time for the user to find the function entry, indicating that the application brings a poor operation experience to the user at the current operation position, and needs to be optimized or Improve.

By adopting the technical solutions of the embodiments of the present invention, during the process of the user using the first application, by identifying the user's facial expression information, the user's point of interest information, and obtaining operation information, based on the facial expression information and the point of interest information and the operation information to generate the first feedback information, so as to be used as the basis for further optimization and modification of the first application. In this way, on the one hand, the direct and active acquisition of the user feedback information is realized, and the user does not need to log in to the application store or website. comments or ratings, which greatly improves the user's operating experience; on the other hand, the technical solution of the embodiment of the present invention collects and identifies information during the user's use process, so as to know the poor user experience in the first application. The specific location is convenient for the improvement or optimization of specific problems in the subsequent operation and maintenance process, and provides a detailed basis for the operation and maintenance of the application.

Embodiment 4

The embodiment of the present invention also provides an information processing system. FIG. 5 is a schematic diagram of the composition and structure of an information processing system according to an embodiment of the present invention; as shown in FIG. 5 , the system includes: an acquisition unit 51, an image processing unit 52 and an information generation unit 53; wherein,

The obtaining unit 51 is configured to obtain operation information and image data; wherein, the operation information is obtained by detecting a triggering operation on the display interface of the first application when the first application in the mobile terminal is activated. operation information; the image data is the image data acquired by the image acquisition unit of the mobile terminal; the image acquisition unit and the display unit are on the same plane;

The image processing unit 52 is used to identify facial feature data in the image data obtained by the acquisition unit 51, and obtain facial expression information based on the facial feature data;

The information generating unit 53 is configured to generate first feedback information based on the facial expression information obtained by the image processing unit 52 and the operation information obtained by the obtaining unit 51 .

The information generating unit 53 is configured to obtain the corresponding first user experience parameter based on the facial expression information, and generate first feedback information of the operation position corresponding to the operation information based on the first user experience parameter.

In this embodiment, when the mobile terminal activates the first application, it outputs a display interface representing the first application, detects a trigger operation on the display interface, and obtains operation information; the operation information includes operation gesture information and operation position information; wherein, the operation gesture information includes: click gesture, double-click gesture, slide gesture, drag gesture, zoom gesture, rotation gesture, parameter (such as volume parameter, brightness parameter, etc.) adjustment gesture, etc.; the operation position The information is the operation position information of the operation gesture; the operation position information may be for a function button. Further, the operation information may also be continuous operation information within a period of time.

When detecting a trigger operation for the display interface, the mobile terminal generates a first instruction, and enables an image acquisition unit of the mobile terminal based on the first instruction; in other implementation manners, the mobile terminal also The image acquisition unit of the mobile terminal may be enabled when the first application is activated, or the mobile terminal may also enable the image acquisition unit of the mobile terminal based on the detected trigger instruction. Make specific restrictions. In this embodiment, the image acquisition unit and the display unit of the mobile terminal are on the same plane, and it can be understood that the image acquisition unit may be implemented by the front camera of the mobile terminal. Further, the mobile terminal sends the obtained operation information and corresponding image data to the information processing system.

In this embodiment, the image processing unit 52 analyzes the image data, and first performs preprocessing on the image data (such as denoising, normalization of pixel positions or illumination variables), and segmentation, positioning or tracking of faces, etc. . Further, the extraction of facial feature data is performed on the image data, including converting the pixel data into representations of the shape, motion, color, muscle and spatial structure of the face and its components, and the extracted facial feature data is used for subsequent follow-up. expression classification. Further, the pre-expression classifier in the information processing system, the expression classifier includes the correspondence between multiple groups of facial feature data and expression information, or the expression classifier includes an expression classification model, and the facial expression The feature data is input into the expression classifier, and the expression information corresponding to the facial feature data is output, that is, the facial expression information is obtained based on the facial feature data. For the facial analysis method and the recognition of expression information described in this embodiment, reference may be made to any analysis and recognition method in the prior art, which is not described in detail in this embodiment.

In this embodiment, the information generating unit 53 pre-stores a plurality of sets of correspondences between the facial expression information and the first user experience parameter. For example, when the facial expression information is pleasant, the corresponding first user experience parameter is 5; when the facial expression information is calm, the corresponding first user experience parameter is 3; when the facial expression information is dissatisfaction , the corresponding first user experience parameter is 0. Of course, in other implementation manners, the facial expression information and the first user experience parameter may also be preset in other corresponding manners, which are not described in detail in this embodiment. That is to say, the facial expression information can represent the user's experience degree, so when the user experience degree represented by the facial expression information reaches the first preset threshold, it means that the user experience degree is good, according to the facial expression information corresponding The first feedback information corresponding to the operation information (including the operation position information) is generated from the operation information (including the operation position information), and at this time, the first feedback information indicates that the first application is at the operation position to the user Bringing a better experience, the operation function at the operation position or the provided content is worthy of recommendation; correspondingly, when the user experience degree represented by the facial expression information does not reach the second preset threshold, the second preset The threshold value is less than the first preset threshold value, which means that the user experience is poor, and the first feedback information corresponding to the operation information (including the operation position information) is generated according to the operation information (including the operation position information) corresponding to the facial expression information. , at this time, the first feedback information indicates that the first application brings a poor experience to the user at the operation position, and the operation function or provided content at the operation position needs to be further optimized or improved.

As another implementation manner, the information generating unit 53 obtains the corresponding first user experience parameter based on the combination of the facial expression information and the operation information. For example, when the facial expression information is pleasant, and the number of operations included in the operation information is less than a first threshold, the corresponding first user experience parameter is 5; when the facial expression information is calm, and the When the number of operations included in the operation information is greater than the first threshold and smaller than the second threshold, the corresponding first user experience parameter is 3; when the facial expression information is dissatisfied and the number of operations included in the operation information is greater than the second threshold, The corresponding first user experience parameter is 0. Of course, in other implementation manners, the facial expression information and the first user experience parameter may also be preset in other corresponding manners, which are not described in detail in this embodiment. That is to say, the combination of the facial expression information and the operation information can represent the user's experience. For example, in a scene, the user wants to find a function entry through an input operation, and can only see it through multiple sliding operations. The user will show an unhappy expression; then in this scenario, the information processing system obtains the facial expression information representing the unhappy expression and the operation information containing multiple sliding operations, and combines the above two information to obtain the corresponding The first user experience parameter is 0, which means that in the operation position corresponding to the operation information, the user experience is poor and needs to be optimized or improved.

Those skilled in the art should understand that the functions of each processing module in the information processing system of the embodiment of the present invention can be understood by referring to the relevant description of the foregoing information processing method, and each processing module in the information processing system of the embodiment of the present invention can be realized by The functions described in the embodiments of the present invention are implemented by analog circuits, and may also be implemented by running software that executes the functions described in the embodiments of the present invention on an intelligent terminal.

Embodiment 5

An embodiment of the present invention further provides an information processing system, as shown in FIG. 5 , the system includes: an acquisition unit 51, an image processing unit 52, and an information generation unit 53; wherein,

The obtaining unit 51 is configured to obtain operation information and image data; wherein, the operation information is obtained by detecting a triggering operation on the display interface of the first application when the first application in the mobile terminal is activated. operation information; the image data is the image data acquired by the image acquisition unit of the mobile terminal; the image acquisition unit and the display unit are on the same plane; also used to obtain the position information and orientation information of the mobile terminal itself;

The image processing unit 52 is used to identify the facial feature data in the image data obtained by the acquiring unit 51, and combine the eye feature data in the facial feature data with the mobile terminal itself obtained by the acquiring unit 51. The position information and direction information are associated to obtain the point of interest information; the point of interest information represents the focus position information of the eyes on the mobile terminal;

The information generating unit 53 is configured to generate second feedback information based on the point of interest information obtained by the image processing unit 52 and the operation information obtained by the obtaining unit 51 .

The information generating unit 53 is configured to obtain a corresponding second user experience parameter based on the point of interest information within a preset time period and the operation information within the preset time period, and generate a corresponding second user experience parameter based on the second user experience parameter The operation information corresponds to the second feedback information of the operation position.

In this embodiment, when the mobile terminal activates the first application, it outputs a display interface representing the first application, detects a trigger operation on the display interface, and obtains operation information; the operation information includes operation gesture information and operation position information; wherein, the operation gesture information includes: click gesture, double-click gesture, slide gesture, drag gesture, zoom gesture, rotation gesture, parameter (such as volume parameter, brightness parameter, etc.) adjustment gesture, etc.; the operation position The information is the operation position information of the operation gesture; the operation position information may be for a function button. Further, the operation information may also be continuous operation information within a period of time. When detecting a trigger operation for the display interface, the mobile terminal generates a first instruction, and enables an image acquisition unit of the mobile terminal based on the first instruction; in other implementation manners, the mobile terminal also The image acquisition unit of the mobile terminal may be enabled when the first application is activated, or the mobile terminal may also enable the image acquisition unit of the mobile terminal based on the detected trigger instruction. Make specific restrictions. In this embodiment, the image acquisition unit and the display unit of the mobile terminal are on the same plane, and it can be understood that the image acquisition unit may be implemented by the front camera of the mobile terminal.

The image processing unit 52 analyzes the image data, first performing preprocessing on the image data (eg, denoising, normalization of pixel positions or illumination variables), and segmentation, localization or tracking of faces, and the like. Further, the extraction of facial feature data is performed on the image data, including converting the pixel data into the representation of the shape, motion, color, muscle and spatial structure of the face and its components; the extraction method of the specific facial feature data can refer to Any facial recognition method in the prior art will not be specifically described in this embodiment.

In this embodiment, the obtaining unit 51 obtains relative position information and direction information of the mobile terminal; the relative position information is the relative position relationship between the mobile terminal and the holder. Specifically, the mobile terminal is provided with at least one of the following sensing units: a gravity sensing unit, an acceleration sensing unit, a distance sensing unit, an iris recognition unit, etc. The gravity sensing unit or the acceleration sensing unit obtains direction information, and the direction information may be the angle between the direction of the center of gravity of the mobile terminal relative to the long-side direction or the short-side direction of the mobile terminal, and the direction information is also That is, the posture change information of the mobile terminal. The mobile terminal may also obtain relative position information between the mobile terminal and the holder through the distance sensing unit or the iris identification unit, wherein the distance sensing unit and the iris identification unit usually Set on the same surface of the display unit of the mobile terminal, when the user holds the mobile terminal, the distance to the user can be detected by the distance sensing unit, or the iris recognition unit can be used to identify The relative orientation between the mobile terminal and the user's eyes.

In this embodiment, the image processing unit 52 associates the eye feature data included in the facial feature data, the relative position information of the mobile terminal itself, and the direction information to obtain focus point information, and the focus point information is obtained. The point information represents the focus position information of the eyes of the user holding the mobile terminal on the mobile terminal, and can also be understood as the position information where the content browsed by the eyes of the holding user is located. Specifically, the information processing system may obtain the line-of-sight direction information of the eyes of the holding user based on the eye feature data, and further, based on the relative position information of the mobile terminal itself and the direction information, determine the The relative positional relationship between the mobile terminal and the holding user, the focusing range of the eye of the holding user focusing on the mobile terminal is obtained based on the gaze direction information and the relative positional relationship, and the focusing range is generated based on the focusing range Focus information.

In this embodiment, the information generating unit 53 obtains the corresponding second user experience parameter based on a combination of the point of interest information and the operation information within a preset time period. For example, within a preset time period t, when the range ratio of the point of interest information (that is, the focus position of the holding user's eyes on the mobile terminal) changes is greater than the first threshold, and the number of operations included in the operation information is greater than When the second threshold is the second threshold, the corresponding second user experience parameter is 0; when the range ratio of the point of interest information (that is, the focus position of the holding user's eyes on the mobile terminal) is greater than the third threshold and less than the first threshold (the When the first threshold is less than the third threshold), and the number of operations included in the operation information is less than the second threshold and greater than the fourth threshold (the fourth threshold is less than the second threshold), the corresponding second user experience The parameter is 3; when the point of interest information (that is, the focus position of the user's eyes on the mobile terminal) does not change, or the range ratio of the change is less than the third threshold, and the number of operations included in the operation information is less than the third threshold. When there are four thresholds, the corresponding second user experience parameter is 5. Of course, in other implementation manners, the information processing system may also refer to other implementation manners for the manner in which the information processing system obtains the point of interest information based on the eye feature data, the relative position information of the mobile terminal itself, and the direction information. The information processing system can obtain the focal position information of the user's eyes on the mobile terminal according to the eye feature data, the relative position information of the mobile terminal itself and the direction information, and can refer to any image recognition method in the prior art. Combining with the modeling technology, no detailed description is given in this embodiment. For example, in a scenario, if the user wants to find a function entry through an input operation, the user will look around for the location of the function entry; then the proportion of the range of changes in the user's focus information obtained by the information processing system will be greater than the first threshold. ; Correspondingly, in the process of the user searching for the location of the function entry, with multiple trigger operations, that is, the number of operations included in the information processing system's obtained operation information is greater than the second threshold; in this scenario, the user It takes a long time to find the function entry, indicating that the application brings a poor operation experience to the user at the current operation position, and needs to be optimized or improved.

Those skilled in the art should understand that the functions of each processing module in the information processing system of the embodiment of the present invention can be understood by referring to the relevant description of the foregoing information processing method, and each processing module in the information processing system of the embodiment of the present invention can be realized by The functions described in the embodiments of the present invention are implemented by analog circuits, and may also be implemented by running software that executes the functions described in the embodiments of the present invention on an intelligent terminal.

Embodiment 6

An embodiment of the present invention further provides an information processing system, as shown in FIG. 5 , the system includes: an acquisition unit 51, an image processing unit 52, and an information generation unit 53; wherein,

The obtaining unit 51 is configured to obtain operation information and image data; wherein, the operation information is obtained by detecting a triggering operation on the display interface of the first application when the first application in the mobile terminal is activated. operation information; the image data is the image data acquired by the image acquisition unit of the mobile terminal; the image acquisition unit and the display unit are on the same plane; also used to obtain the position information and orientation information of the mobile terminal itself;

The image processing unit 52 is used to identify the facial feature data in the image data obtained by the acquisition unit 51, and obtain facial expression information based on the facial feature data; The data is associated with the position information and direction information of the mobile terminal itself obtained by the obtaining unit 51 to obtain focus point information; the focus point information represents the focus position information of the eyes on the mobile terminal;

The information generating unit 53 is configured to generate first feedback information based on the facial expression information obtained by the image processing unit 52 , the point of interest information and the operation information obtained by the obtaining unit 51 .

Wherein, the information generating unit 53 is configured to obtain the corresponding third user experience parameter based on the facial expression information and the point of interest information in the preset time period and in combination with the operation information in the preset time period, The first feedback information of the operation position corresponding to the operation information is generated based on the third user experience parameter.

In this embodiment, when the mobile terminal activates the first application, it outputs a display interface representing the first application, detects a trigger operation on the display interface, and obtains operation information; the operation information includes operation gesture information and operation position information; wherein, the operation gesture information includes: click gesture, double-click gesture, slide gesture, drag gesture, zoom gesture, rotation gesture, parameter (such as volume parameter, brightness parameter, etc.) adjustment gesture, etc.; the operation position The information is the operation position information of the operation gesture; the operation position information may be for a function button. Further, the operation information may also be continuous operation information within a period of time. When detecting a trigger operation for the display interface, the mobile terminal generates a first instruction, and enables an image acquisition unit of the mobile terminal based on the first instruction; in other implementation manners, the mobile terminal also The image acquisition unit of the mobile terminal may be enabled when the first application is activated, or the mobile terminal may also enable the image acquisition unit of the mobile terminal based on the detected trigger instruction. Make specific restrictions. In this embodiment, the image acquisition unit and the display unit of the mobile terminal are on the same plane, and it can be understood that the image acquisition unit may be implemented by the front camera of the mobile terminal.

In this embodiment, the image processing unit 52 analyzes the image data, and first performs preprocessing on the image data (such as denoising, normalization of pixel positions or illumination variables), and segmentation, positioning or tracking of faces, etc. . Further, the extraction of facial feature data is performed on the image data, including converting the pixel data into representations of the shape, motion, color, muscle and spatial structure of the face and its components, and the extracted facial feature data is used for subsequent follow-up. expression classification. Further, the pre-expression classifier in the information processing system, the expression classifier includes the correspondence between multiple groups of facial feature data and expression information, or the expression classifier includes an expression classification model, and the facial expression The feature data is input into the expression classifier, and the expression information corresponding to the facial feature data is output, that is, the facial expression information is obtained based on the facial feature data. For the facial analysis method and the recognition of expression information described in this embodiment, reference may be made to any analysis and recognition method in the prior art, which is not described in detail in this embodiment.

In this embodiment, the obtaining unit 51 obtains relative position information and direction information of the mobile terminal; the relative position information is the relative position relationship between the mobile terminal and the holder. Specifically, the mobile terminal is provided with at least one of the following sensing units: a gravity sensing unit, an acceleration sensing unit, a distance sensing unit, an iris recognition unit, etc. The gravity sensing unit or the acceleration sensing unit obtains direction information, and the direction information may be the angle between the direction of the center of gravity of the mobile terminal relative to the long-side direction or the short-side direction of the mobile terminal, and the direction information is also That is, the posture change information of the mobile terminal. The mobile terminal may also obtain relative position information between the mobile terminal and the holder through the distance sensing unit or the iris identification unit, wherein the distance sensing unit and the iris identification unit usually Set on the same surface of the display unit of the mobile terminal, when the user holds the mobile terminal, the distance to the user can be detected by the distance sensing unit, or the iris recognition unit can be used to identify The relative orientation between the mobile terminal and the user's eyes.

Further, the image processing unit 52 associates the eye feature data included in the facial feature data and the relative position information of the mobile terminal and the direction information to obtain the point of interest information, and the point of interest information represents The focus position information of the eyes of the holding user on the mobile terminal of the mobile terminal may also be understood as the position information of the content browsed by the eyes of the holding user. Specifically, the information processing system may obtain the gaze direction information of the eyes of the holding user based on the eye feature data, and further, determine the movement based on the relative position information of the mobile terminal and the direction information The relative positional relationship between the terminal and the holding user, obtaining the focus range of the holding user's eyes on the mobile terminal based on the gaze direction information and the relative positional relationship, and generating the attention based on the focus range point information.

In this embodiment, the information generating unit 53 obtains the corresponding third user experience parameter based on the facial expression information within the preset time period and the point of interest information within the preset time period, in combination with the operation information. For example, within the preset time period t, the obtained facial expression information is pleasant, and the point of interest information (that is, the focus position of the holding user's eyes on the mobile terminal) does not change, or the range ratio of the change is smaller than the third threshold, and when the number of operations included in the operation information is less than the fourth threshold, the corresponding second user experience parameter is 5; The ratio of the range of changes in the focus position on the mobile terminal) is greater than a third threshold and less than a first threshold (the first threshold is less than the third threshold), and the number of operations included in the operation information is less than the second threshold and greater than the fourth threshold (the fourth threshold is less than the second threshold), the corresponding second user experience parameter is 3; or, the obtained facial expression information is dissatisfaction, and the point of interest information (that is, holding the user's eyes on the mobile terminal) When the ratio of the range of changes in the focus position) is greater than the first threshold, and the number of operations included in the operation information is greater than the second threshold, the corresponding second user experience parameter is 0. Of course, the manner in which the information processing system obtains the third user experience parameter may also refer to other manners, which will not be repeated in this embodiment. For example, in a scenario, if the user wants to find a function entry through an input operation, the user will look around for the location of the function entry; then the proportion of the range of changes in the user's focus information obtained by the information processing system will be greater than the first threshold. ; Correspondingly, in the process of the user finding the location of the function entry, with multiple trigger operations, that is, the number of operations included in the information processing system to obtain the operation information is greater than the second threshold; Correspondingly, after the user spends In the scenario of searching for the location of the function entry for a long time, the user will show an unhappy expression; in this scenario, the information processing system The analysis of the operation information determines that the current user experience is poor, that is, it takes a long time for the user to find the function entry, indicating that the application brings a poor operation experience to the user at the current operation position, and needs to be optimized or Improve.

Those skilled in the art should understand that the functions of each processing module in the information processing system of the embodiment of the present invention can be understood by referring to the relevant description of the foregoing information processing method, and each processing module in the information processing system of the embodiment of the present invention can be realized by The functions described in the embodiments of the present invention are implemented by analog circuits, and may also be implemented by running software that executes the functions described in the embodiments of the present invention on an intelligent terminal.

In Embodiments 4 to 6 of the present invention, the image processing unit 52 and the information generating unit 53 in the information processing system can be implemented by a central processing unit (CPU, Central Processing Unit), a digital signal in the system in practical applications The acquisition unit 51 in the information processing system can be implemented by the transceiver antenna or Transceiver implementation.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored, or not implemented. In addition, the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms. of.

The unit described above as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may all be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integration The unit can be implemented either in the form of hardware or in the form of hardware plus software functional units.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by program instructions related to hardware, the aforementioned program can be stored in a computer-readable storage medium, and when the program is executed, execute Including the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk and other various A medium on which program code can be stored.

Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention may be embodied in the form of software products in essence or the parts that make contributions to the prior art. The computer software products are stored in a storage medium and include several instructions for A computer device (which may be a personal computer, a server, or a network device, etc.) is caused to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic disk or an optical disk and other mediums that can store program codes.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention. should be included within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (8)

1. An information processing method, characterized in that the method comprises:

obtaining operation information and image data; the operation information is obtained by detecting a trigger operation aiming at a display interface of a first application when the first application in the mobile terminal is in an activated state; the image data is acquired by an image acquisition unit of the mobile terminal; the image acquisition unit and the display unit are on the same plane;

identifying facial feature data in the image data, and obtaining facial expression information based on the facial feature data;

obtaining relative position information and direction information of the mobile terminal;

associating the eye feature data, the relative position information and the direction information in the face feature data to obtain the information of the attention point; specifically, gaze direction information of eyes is obtained based on the eye feature data, a relative position relationship between the mobile terminal and the eyes is determined based on the relative position information and the direction information, a focusing range of the eyes focused on the mobile terminal is obtained based on the gaze direction information and the relative position relationship, and the point-of-interest information is generated based on the focusing range; the focus point information represents focusing position information of eyes on the mobile terminal;

generating first feedback information based on the facial expression information, the range of change in the point of interest information, and the operation information.

2. The method of claim 1, further comprising:

and generating second feedback information based on the point of interest information and the operation information.

3. The method of claim 2, wherein generating second feedback information based on the point of interest information and the operation information comprises:

and obtaining a corresponding second user experience parameter based on the attention point information in a preset time period and the operation information in the preset time period, and generating second feedback information of an operation position corresponding to the operation information based on the second user experience parameter.

4. The method of claim 1, wherein the generating first feedback information based on the facial expression information, the point of interest information, and the operation information comprises:

and obtaining a corresponding third user experience parameter by combining operation information in a preset time period based on the facial expression information and the attention point information in the preset time period, and generating first feedback information of an operation position corresponding to the operation information based on the third user experience parameter.

5. An information processing system, the system comprising: an acquisition unit, an image processing unit and an information generation unit; wherein,

the acquisition unit is used for acquiring operation information and image data; the operation information is obtained by detecting a trigger operation aiming at a display interface of a first application when the first application in the mobile terminal is in an activated state; the image data is acquired by an image acquisition unit of the mobile terminal; the image acquisition unit and the display unit are on the same plane; the system is also used for obtaining the relative position information and the direction information of the mobile terminal;

the image processing unit is used for identifying facial feature data in the image data and obtaining facial expression information based on the facial feature data; the face feature data acquisition unit is further configured to associate eye feature data in the face feature data, the relative position information of the mobile terminal obtained by the acquisition unit and the direction information to obtain point-of-interest information; specifically, gaze direction information of eyes is obtained based on the eye feature data, a relative position relationship between the mobile terminal and the eyes is determined based on the relative position information and the direction information, a focusing range of the eyes focused on the mobile terminal is obtained based on the gaze direction information and the relative position relationship, and the point-of-interest information is generated based on the focusing range; the focus point information represents focusing position information of eyes on the mobile terminal;

the information generating unit is used for generating first feedback information based on the facial expression information obtained by the image processing unit, the change range of the focus point information and the operation information obtained by the obtaining unit.

6. The system of claim 5, wherein the information generating unit is further configured to generate second feedback information based on the point of interest information and the operation information.

7. The system according to claim 6, wherein the information generating unit is configured to obtain a corresponding second user experience parameter based on the point of interest information in a preset time period and the operation information in the preset time period, and generate second feedback information of an operation position corresponding to the operation information based on the second user experience parameter.

8. The system according to claim 5, wherein the information generating unit is configured to obtain a corresponding third user experience parameter based on the facial expression information and the point of interest information in a preset time period in combination with the operation information in the preset time period, and generate the first feedback information of the operation position corresponding to the operation information based on the third user experience parameter.

Images