WO2023024871A1 - Interface interaction method and device - Google Patents

Abstract

An interface interaction method and device. The interface interaction method comprises: presenting, by means of a display apparatus of a head-mounted device (100), a current application interface of a current application being used by a user (S101); obtaining an interface interaction operation of the user about the current application interface (S102); and presenting a corresponding application operation interface on the basis of the interface interaction operation, wherein the application operation interface comprises a plurality of interfaces (S103), the plurality of interfaces comprise the current application interface and a plurality of side interfaces of the current application interface, the plurality of side interfaces comprise a corresponding application identifier interface and a corresponding function interface, the application identifier interface is used for presenting a plurality of application identifiers, the function interface comprises a shortcut function interface, the shortcut function interface comprises at least one shortcut instruction identifier, and each shortcut instruction identifier is used for triggering and generating a corresponding shortcut instruction. According to the method, extremely simplified interaction is ensured, and user operation experience is improved.

Description

A method and device for interface interaction

This application is based on the application of CN application number 202110977113.7 and application date of 2021.08.24, and claims its priority. The disclosure content of this CN application is hereby incorporated into this application as a whole.

technical field

The present application relates to the communication field, and in particular to an interface interaction technology.

Background technique

The existing head-mounted display device (augmented reality or virtual reality device) has an unfriendly interface, and the interaction steps are too cumbersome, which is inconvenient for the user to operate, thus affecting the user experience.

Contents of the invention

An object of the present application is to provide an interface interaction method and device.

According to one aspect of the present application, an interface interaction method is provided, the method comprising:

Presenting the current application interface of the current application being used by the user through the display device of the head-mounted device;

Obtaining the user's interface interaction operations on the current application interface;

Presenting a corresponding application operation interface based on the interface interaction operation, where the application operation interface includes multiple interfaces, and the multiple interfaces include the current application interface and multiple side interfaces of the current application interface, so The multiple side interfaces include a corresponding application identification interface and a function interface, the application identification interface is used to present multiple application identifications, the function interface includes a shortcut function interface, and the shortcut function interface includes at least one shortcut instruction identification, Each shortcut instruction identifier is used to trigger and generate a corresponding shortcut instruction.

According to one aspect of the present application, an interface interaction device is provided, and the device includes:

A module, configured to present the current application interface of the current application being used by the user through the display device of the head-mounted device;

A one-two module, configured to obtain the interface interaction operation of the user on the current application interface;

A third module, configured to present a corresponding application operation interface based on the interface interaction operation, wherein the application operation interface includes multiple interfaces, and the multiple interfaces include the current application interface and multiples of the current application interface a plurality of side interfaces, the multiple side interfaces include corresponding application identification interfaces and function interfaces, the application identification interfaces are used to present multiple application identifications, the function interfaces include a shortcut function interface, and the shortcut function interface includes At least one shortcut instruction identifier, each shortcut instruction identifier is used to trigger and generate a corresponding shortcut instruction.

According to one aspect of the present application, a computer device is provided, wherein the device includes:

processor; and

A memory arranged to store computer-executable instructions which, when executed, cause the processor to perform the steps of any one of the methods described above.

According to one aspect of the present application, there is provided a computer-readable storage medium on which computer programs/instructions are stored, wherein the computer program/instructions, when executed, cause the system to perform any of the methods described above. step.

According to one aspect of the present application, a computer program product is provided, including computer programs/instructions, which is characterized in that, when the computer program/instructions are executed by a processor, the steps of any one of the methods described above are implemented.

Compared with the prior art, this application presents multiple interfaces in the application operation interface through the user's interactive operation on the interface of the current application interface, so as to facilitate the user to perform corresponding operations on the screen without returning from the current application first. The main interface, and then find the corresponding operation position to click and other cumbersome actions to perform corresponding operations, which ensures that the interaction is extremely simplified and the user operation experience is improved.

Description of drawings

Other characteristics, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG. 1 shows a flowchart of an interface interaction method according to an embodiment of the present application;

Fig. 2 shows an example diagram of an application operation interface according to another embodiment of the present application;

FIG. 3 shows functional modules of a head-mounted device according to an embodiment of the present application;

FIG. 4 illustrates an exemplary system that may be used to implement various embodiments described in this application.

The same or similar reference numerals in the drawings represent the same or similar components.

Detailed ways

The application will be described in further detail below in conjunction with the accompanying drawings.

In a typical configuration of this application, the terminal, the device serving the network, and the trusted party all include one or more processors (for example, a central processing unit (Central Processing Unit, CPU)), an input/output interface, a network interface and Memory.

Memory may include non-permanent memory in computer-readable media, random access memory (Random Access Memory, RAM) and/or non-volatile memory, such as read-only memory (Read Only Memory, ROM) or flash memory ( Flash Memory). Memory is an example of computer readable media.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. The example of the storage medium of computer includes, but not limited to Phase-Change Memory (Phase-Change Memory, PCM), Programmable Random Access Memory (Programmable Random Access Memory, PRAM), Static Random-Access Memory (Static Random-Access Memory, SRAM), dynamic random access memory (Dynamic Random Access Memory, DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (Electrically-Erasable Programmable Read -Only Memory, EEPROM), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory, CD-ROM), Digital Versatile Disc (Digital Versatile Disc, DVD) or other optical storage , magnetic cassette, magnetic tape disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

The equipment referred to in this application includes, but is not limited to, user equipment, network equipment, or equipment formed by integrating user equipment and network equipment through a network. The user equipment includes but is not limited to any mobile electronic product that can interact with the user, such as a smart phone, a tablet computer, a head-mounted device, etc., and the mobile electronic product can use any operating system, such as the Android operating system , iOS operating system, etc. Wherein, the network device includes an electronic device that can automatically perform numerical calculation and information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC) ), Programmable Logic Device (PLD), Field Programmable Gate Array (Field Programmable Gate Array, FPGA), Digital Signal Processor (Digital Signal Processor, DSP), embedded devices, etc. The network equipment includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud formed by multiple servers; here, the cloud is composed of a large number of computers or network servers based on Cloud Computing, Among them, cloud computing is a kind of distributed computing, a virtual supercomputer composed of a group of loosely coupled computer sets. The network includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless self-organizing network (Ad Hoc network) and the like. Preferably, the device may also be a program running on the user device, network device, or a device formed by integrating user device and network device, network device, touch terminal or network device and touch terminal through a network.

Of course, those skilled in the art should understand that the above-mentioned equipment is only an example, and other existing or future equipment that may be applicable to this application should also be included in the scope of protection of this application, and it is included in this application by reference. this.

In the description of the present application, "plurality" means two or more, unless otherwise specifically defined.

Fig. 1 shows an interface interaction method according to one aspect of the present application, where the method is applied to a head-mounted device, and the method specifically includes step S101, step S102 and step S103. In step S101, the current application interface of the current application being used by the user is presented through the display device of the head-mounted device; in step S102, the interface interaction operation of the user on the current application interface is obtained; in step S103, based on The interface interaction operation presents a corresponding application operation interface, wherein the application operation interface includes multiple interfaces, and the multiple interfaces include the current application interface and multiple side interfaces of the current application interface, the The multiple side interfaces include a corresponding application identification interface and a function interface, the application identification interface is used to present multiple application identifications, the function interface includes a shortcut function interface, and the shortcut function interface includes at least one shortcut instruction identification, each A shortcut instruction identifier is used to trigger and generate a corresponding shortcut instruction. Here, the head-mounted device includes but is not limited to augmented reality glasses, virtual reality glasses, mixed reality glasses, augmented reality helmets, virtual reality helmets, mixed reality helmets, and the like. The head-mounted device includes an acquisition device for obtaining the user's head movement information, including but not limited to a three-axis sensor, inertial measurement unit, gyroscope, 6DOF, etc.; the head-mounted device also includes a data processing device for processing the data , storage, transmission, retrieval and data processing, etc.; the head-mounted device also includes a display device for presenting application interface information of the application, such as a liquid crystal display, an optical display, and the like.

Specifically, in step S101, a current application interface of a current application being used by a user is presented through a display device of a head-mounted device. For example, the user holds and wears the headset, the headset is in use, and multiple applications are installed on the headset, such as functional applications or third-party applications built into the headset. The head-mounted device starts the corresponding application based on the user's application starting operation (such as directly selecting the operation, or selecting and confirming the operation, etc.), and uses the application currently being used by the user as the current application. The head-mounted device presents the application interface of the current application through the display device, and the application interface includes the current application interface being used after the current application is started. Here, when the head-mounted device is started, it is determined that the display position information of the interface of the head-mounted device is directly in front of the user, such as taking the directly in front of the user's field of view as the center of the screen, and presenting the current application interface on the screen. Here, the current There is no limitation on the presentation space, location area, etc. of the application interface. In some embodiments, a gaze point is set at a fixed position in the screen, and the position of the gaze point in the screen does not change, such as a gaze point is set in the center of the screen. Controls in the interface obtain the focus through the gaze point, such as dynamically moving the position of the interface through head movement. When the control in the interface moves to the gaze point, the control obtains the focus.

In step S102, interface interaction operations of the user on the current application interface are acquired. For example, the interface interaction operation includes a preset operation for starting the corresponding application operation interface, and the specific preset operation includes but is not limited to voice information (such as "start interactive mode" and other voice information), touch information (such as double-clicking or pressing Predetermined trajectory sliding, etc.), button information (such as inputting specific button information or inputting button information in a preset order, etc.), gesture information (such as gesture information of user gestures, etc.), head movement information (such as user nodding or shaking head quickly, etc.) Or import data information (such as importing corresponding instructions through a preset interface), wherein the head motion information includes any item such as head motion angle, distance, speed, and orientation. The corresponding head-mounted device includes a collection device, such as a voice input device (such as a microphone, etc.), a camera device or an attitude sensor (such as a three-axis gyroscope), and the like. The head-mounted device can collect interface interaction operations on the current application interface through the collection device.

In step S103, a corresponding application operation interface is presented based on the interface interaction operation, wherein the application operation interface includes a plurality of interfaces, and the plurality of interfaces include the current application interface and a plurality of the current application interface. Side interface, the multiple side interfaces include corresponding application identification interfaces and function interfaces, the application identification interface is used to present multiple application identifications, the function interface includes a shortcut function interface, and the shortcut function interface includes at least A shortcut instruction identifier, each shortcut instruction identifier is used to trigger and generate a corresponding shortcut instruction.

For example, after acquiring the corresponding interface interaction operation, the head-mounted device matches the interface interaction operation with the preset operation corresponding to the interaction instruction, and if they match, determines the interaction instruction corresponding to the interface interaction operation. For example, the collected head movement information, voice information, touch information, button information, gesture information or imported data information are similarly matched with preset operations to determine the corresponding similarity, if the similarity is greater than or equal to the similarity threshold, generate a corresponding interaction instruction, wherein the interaction instruction is used to start and present a corresponding application operation interface, and the application operation interface is used to perform corresponding operations, including switching between applications, invoking, or setting functions. The head-mounted device starts the corresponding interface interaction mode based on the interaction command corresponding to the interface interaction operation. In this mode, the head-mounted device presents the corresponding application operation interface. The application operation interface includes multiple interfaces including the current application interface. In this interface, other interfaces except the current application interface are distributed on the sides of the current application interface (such as upper, lower, left, and right sides). In some cases, we can directly present other interfaces on the side of the current application interface, or we can make adaptive adjustments to the current application interface (such as shrinking or enlarging according to a certain ratio), and then present other interfaces on the current application The side of the interface allows the user of the headset to intuitively observe other interfaces. In some implementations, the interface elements in the current application interface in this mode are in a non-interactive state. For example, each interactive element in the current application interface cannot be interacted with through touch, voice or gestures. Yes, in this mode, some or all of the interactive elements in other interfaces on the side of the current application interface are in an interactive state. Furthermore, the current application interface can be blocked or shielded and displayed in a certain way, such as by superimposing and presenting the application identification of the current application within the interface range of the current application interface to identify the current application.

In addition to the current application interface, the multiple interfaces also include side interfaces such as an application identification interface and a function interface; the side interfaces refer to interfaces distributed on the side of the current application interface, and each side interface is connected to the current application interface. For example, the boundary of the side interface overlaps with the boundary of the current application interface or is separated by a certain distance threshold, and for another example, the interface side of the side interface overlaps with the current application interface by some pixels. The interface ranges of each side interface may be the same, or may be set according to different ratios. Each side interface may be distributed on different sides of the current application interface, or multiple side interfaces may be distributed on the same side of the current application interface. The application identification interface is used to present multiple application identifications, and the multiple application identifications may be part or all of all applications installed on the head-mounted device, or the multiple application identifications may be all applications installed on the head-mounted device The applications associated with the current application determined in . If the head-mounted device obtains the user's trigger operation related to an application ID among the multiple application IDs, the head-mounted device determines to execute the corresponding trigger command, such as starting and jumping to the application corresponding to the application ID and exiting the current interface interaction mode, Another example is to start the application corresponding to the application identification, and superimpose display (for example, in the form of picture-in-picture) on the application operation interface including the current application interface, and another example is to call the application corresponding to the application identification in the background. The side interface also includes a function interface, the function interface includes a shortcut function interface, and the shortcut function interface includes at least one shortcut instruction identifier, and each shortcut instruction identifier is used to trigger a corresponding shortcut instruction, and the triggering methods include but are not limited to head movements, Interaction methods such as voice, button, touch or gesture. The shortcut commands include device-related shortcut commands, such as operation command information related to devices frequently used by users, such as opening settings, taking pictures, recording videos, scanning, calling xx, viewing call records, and opening photo albums. The instruction information may also include application instructions related to the current application, such as the operation instruction information corresponding to the application that is frequently used by the user in the current application, the default provided by the application, or the application associated with the current application, such as logging out, switching accounts, and entering the next step. , return to the previous step, save, etc., so that the corresponding shortcut instructions can be configured according to different current applications. In some embodiments, the shortcut instruction may also include switching instructions of other applications, such as an operation instruction for switching the current interface to other applications, such as triggering the "photograph" shortcut instruction to jump into the corresponding camera interface of the camera application , and close the application operation interface including the current application interface; for another example, by triggering the "photograph" shortcut command, start the corresponding camera interface of the camera application, and superimpose display (for example, in the form of picture-in-picture) on the application operation including the current application interface interface. In some cases, the shortcut instruction also includes operation instructions for regulating the application operations of other background applications without switching the application interface in the current application interface. If the current shortcut instruction is to send a greeting email to someone, then Sending the greeting email can be completed in the current operation interface without switching to the mail application interface. Here, the above-mentioned shortcut instruction information is only an example, and is not limited.

In some cases, the side interface can be explicitly presented around the current application interface of the current application; in other implementations, the side interface can be hidden around the current application interface of the current application, such as according to different requirements Present the side interface around the current application interface of the current application with different transparency (such as 0-100%). When the user's gaze point is within the interface range of a certain side interface or the interface overlap area, the The range or distance of the landing point determines the side interface that is closer, so that the corresponding side interface is visibly presented on the current screen. In some implementations, the side interface can be directly and simultaneously presented around the current application interface of the current application, and all the side interfaces and the current application are distributed in different spatial positions of the current screen, so that the user can simultaneously view the presented The side interface and the current application interface; in other cases, the side interface can also be partially presented around the current application interface of the current application, such as presenting a certain proportion of the interface information near the border of the side interface according to the screen size. The current screen, or present the overlapping part of the side interface and the current application on the current screen, so that the user can view the presented current application interface and part of the side interface at the same time, and then view the side interface in turn according to the user's operation, such as the user turning the head , the screen sequentially presents the corresponding side interfaces; in other embodiments, the side interfaces are not directly presented on the current screen, but the side interfaces can be presented on the screen sequentially according to user operations, for example, the current application interface is currently presented on the screen , the user turns the head, and the screen presents the corresponding side interfaces in sequence.

In some embodiments, the function interface further includes a parameter setting interface, and the parameter setting interface includes at least one parameter setting identifier. For example, the parameter setting application includes the function setting application of the device's own parameters of the head-mounted device, such as network connection application, screen brightness adjustment application, flashlight operation application, sound adjustment application, head control mode switching application, voice mode switching application or external device connection application etc. The parameter setting application of the head-mounted device has a corresponding parameter setting identifier, and at least one parameter setting identifier is integrated into an interface to form a corresponding parameter setting interface. In some embodiments, at least one parameter setting identifier contained in the parameter setting interface may be an identifier corresponding to a parameter setting application related to the current application. For example, the current application is mainly used for audio and video output, and the corresponding parameter setting identifier includes a corresponding Volume adjustment, video brightness adjustment, video window ratio adjustment, etc. In some cases, at least one parameter setting identifier in the parameter setting interface has nothing to do with the application type of the current application, and corresponds to the parameter identifier corresponding to the parameter setting of the head-mounted device itself. For example, different current applications, the parameters in the parameter setting interface The setting flag is unchanged. The parameter setting logo in the parameter setting interface can be a functional application that only displays parameter-related information, such as displaying the current power, displaying the current time, and other functional display applications; of course, in some cases, these functional display applications can be non-interactive. The pure function display application can also be a function display application with certain interactive functions, for example, a volume adjustment application mainly displaying the volume can also include a volume adjustment function, etc. In other cases, the parameter setting logo in the parameter setting interface can realize the switching function of the parameter setting page corresponding to the parameter setting application, such as switching from the application operation interface containing the current application interface to the corresponding one by touching the corresponding parameter setting logo. For the parameter setting page, etc., for example, jump to the corresponding parameter setting page and close the application operation interface including the current application interface, or start the corresponding parameter setting page and superimpose it on the application operation interface including the current application interface.

In some implementation manners, the parameter setting interface and the shortcut function interface are distributed on different sides of the current application interface. For example, in order to better distinguish between different function categories of each interface, and to present the functions of each interface element in a simple and clear manner, we can set different interfaces on different sides of the current application interface. Referring to FIG. 2 for a specific example, an application identification interface is displayed on the right side of the current application interface, a parameter setting interface is displayed on the upper side of the current application interface, and a shortcut command interface is displayed on the left side of the current application interface.

In some implementations, the method further includes step S104 (not shown). In step S104, the user's application usage record information about the application installed in the head-mounted device is acquired, and a corresponding shortcut instruction identifiers, wherein each shortcut instruction identifier includes indication information indicating an application corresponding to a piece of application usage record information. For example, the application usage record information includes the user's historical usage records about the applications installed on the head-mounted device, and the application indication information corresponding to each application usage record information corresponds to the application identification information of a historically used application or the application corresponding to the application An operation instruction of an operation function (such as an operation-related function inside an application, etc.). The head-mounted device can generate a corresponding shortcut instruction identifier according to the application usage record information, and the shortcut instruction identifier is used to point to the application that the user has used in history or the operation instruction of the application operation function of the application that has been used in history, etc., by touching the shortcut instruction ID, the headset can start the application, or start the application and execute the operation instructions corresponding to the operation function of the application.

In some implementations, the method further includes step S105 (not shown). In step S105, the head movement information of the user is obtained, and the user's gaze point is determined to be on the interface according to the head movement information. The gaze position information in; if the gaze position information is within the identification range of a trigger identification, execute the trigger instruction corresponding to the trigger identification, wherein the trigger identification includes the at least one application identification, the at least one parameter setting an identifier and the at least one shortcut instruction identifier.

For example, the application operation interface of the head-mounted device includes a plurality of triggering identifiers, and the multiple triggering identifiers include but are not limited to shortcut instruction identifiers in the shortcut instruction interface, parameter setting identifiers in the parameter setting interface, and application identifiers in the application identifier interface. wait. When the gaze position information of the user's gaze point in the interface is within the identification range of a trigger identification, the trigger instruction corresponding to the trigger identification may be triggered and executed. The user can align the gaze point with the corresponding trigger mark or the mark range of the trigger mark by controlling the motion of moving the head, so as to determine and execute the trigger instruction of the trigger mark. Specifically, the head-mounted device obtains the user's head movement information, dynamically moves the interface presented on the screen according to the head movement information, and sets a gaze point at a fixed position on the screen, such as setting a gaze point at the center of the screen. The position of the gaze point in the screen remains unchanged. When the interface presented on the screen is dynamically moved, the gaze position information of the gaze point in the interface changes. If the gaze position information is within the identification range corresponding to a certain trigger identification, then determine The user selects the trigger logo to determine and execute corresponding trigger instructions, such as displaying a parameter setting page corresponding to setting functions such as volume up, down, or mute, or launching and jumping to other applications. Specifically, in some embodiments, the rotation vector sensor collects the user's head movement information through the inertial measurement unit of the head-mounted device, calculates the Euler angle according to the output result of the rotation vector sensor, and then calculates the Euler angle according to the X and Y direction angles. Changes to dynamically move the position of the user interface.

In some implementations, if the gaze position information is within the identification range of a certain trigger mark, executing the trigger instruction corresponding to the trigger mark includes: if the gaze position information is within the trigger range of a certain trigger mark, and After obtaining the user's trigger confirmation operation, execute the trigger instruction corresponding to the trigger identifier, wherein the trigger identifier includes the at least one application identifier, the at least one parameter setting identifier, and the at least one shortcut instruction identifier. For example, there may be large errors in determining and executing trigger instructions only through gaze position information, such as false triggers during head movements, etc., and the headset can determine the corresponding trigger instructions based on the user's further confirmation operations. The gaze position information of the user determines the trigger logo selected by the user. If the selection confirmation operation (such as head nodding, voice input confirmation or touchpad click input confirmation, key input confirmation, etc.) of the selected trigger logo is obtained, the head Only when the device is worn can the trigger instruction corresponding to the selected trigger logo be executed.

In some implementations, part or all of the trigger signs in the application operation interface include corresponding voice prompt signs, wherein the voice prompt signs are used to indicate that the corresponding trigger signs can be triggered by voice, and the trigger signs include the At least one application identifier, the at least one parameter setting identifier, and the at least one shortcut instruction identifier.

For example, in addition to executing the corresponding triggering instruction based on the aforementioned gaze position information, the corresponding triggering sign can also confirm and execute the corresponding triggering instruction through voice input. In some implementations, the method further includes step S106 (not shown). In step S106, the voice information of the user is obtained; if the voice information is the same or similar to the preset voice text of a certain trigger identifier, Then execute the trigger instruction corresponding to the trigger identifier, where the trigger identifier includes the at least one application identifier, the at least one parameter setting identifier, and the at least one shortcut instruction identifier. In some cases, each trigger identifier is set with a corresponding preset voice text, and the head-mounted device obtains the collected voice information, and matches the text corresponding to the voice information with multiple preset voice texts. If the voice and text match, then confirm and execute the trigger instruction corresponding to the preset voice and text. In other cases, in order to better remind the user to trigger the corresponding instruction through voice information, the corresponding voice prompt logo can be presented in the trigger logo, as shown in Figure 2, some or all of the trigger logos include the corresponding voice logo for prompting The user's trigger mark can be triggered by voice, etc., thereby prompting the user to realize interface interaction conveniently and quickly through voice information.

In some implementations, the triggering instruction includes starting other applications and jumping to the other application interface; wherein, the method further includes step S107 (not shown), in step S107, closing the other application The interface beyond the interface. For example, the other applications include applications containing page display content, that is, trigger instructions corresponding to other applications include waking up (such as starting or changing from running in the background to running in the foreground, etc.) and jumping to other application interfaces of the application. Specifically, other applications include but are not limited to the application corresponding to the application identification in the application identification interface, the application corresponding to the shortcut instruction identification in the shortcut instruction interface, or the parameter setting application corresponding to the parameter setting identification in the parameter setting interface that needs to jump to the page, etc. After the user selects and confirms other applications, the headset closes interfaces other than the other applications and presents other application interfaces of the other applications. Further, when presenting other application interfaces, if the user's interface interaction operation on other application interfaces is acquired again, the application operation interface corresponding to the other application interface is presented, for example, while the application interface of the other application is presented, in the Sides of other application interfaces present functional interfaces and application identification interfaces. In some cases, the presentation position of the other application interface is the same or similar to the position of the aforementioned current application interface, so as to maintain the same or similar user pose for interface interaction and the like.

In some implementations, the multiple side interfaces further include a corresponding voice command interface, wherein the voice command interface includes at least one voice command identifier. For example, multiple side interfaces also include a voice command interface. As shown in FIG. Voice input information of the user, then carry out voice recognition to the voice input information, determine the voice text information corresponding to the voice input information, determine the corresponding voice command based on the voice text information; or, identify the voice text information and the voice command of the voice command (such as text information, etc.), and if matched, the head-mounted device executes the voice instruction corresponding to the voice text prompt information.

In some implementations, the method further includes step S108 (not shown), in step S108, while presenting the voice instruction interface, present a voice prompt identifier corresponding to the trigger identifier in other side interfaces, wherein The voice prompt identifier is used to indicate that the corresponding trigger identifier can be triggered by voice, and the trigger identifier includes the at least one application identifier, the at least one parameter setting identifier, and the at least one shortcut instruction identifier. For example, while the head-mounted device presents the voice command interface, it can also give the corresponding voice interaction prompt function to the interactive elements in the current interface, such as presenting the voice prompt logo of the trigger mark in other side interfaces, etc., for prompting The user's trigger mark can be triggered by voice, etc., thereby prompting the user to realize interface interaction conveniently and quickly through voice information.

The above mainly introduces various embodiments of an interface interaction method of the present application by way of examples. In addition, the present application also provides specific devices capable of implementing the above embodiments, which will be introduced below with reference to FIG. 3 .

FIG. 3 shows a head-mounted device for interface interaction according to an aspect of the present application, which specifically includes a one-module 101 , a one-two module 102 and a one-three module 103 . — module 101, used to present the current application interface of the current application that the user is using through the display device of the head-mounted device; module 102, used to obtain the user’s interface interaction operation on the current application interface; Module 103, configured to present a corresponding application operation interface based on the interface interaction operation, where the application operation interface includes a plurality of interfaces, and the plurality of interfaces include the current application interface and a plurality of the current application interface. Side interface, the multiple side interfaces include corresponding application identification interfaces and function interfaces, the application identification interface is used to present multiple application identifications, the function interface includes a shortcut function interface, and the shortcut function interface includes at least A shortcut instruction identifier, each shortcut instruction identifier is used to trigger and generate a corresponding shortcut instruction.

In some embodiments, the function interface further includes a parameter setting interface, and the parameter setting interface includes at least one parameter setting identifier. In some implementation manners, the parameter setting interface and the shortcut function interface are distributed on different sides of the current application interface.

Here, the specific implementations corresponding to the one-one module 101, the one-two module 102 and the one-three module 103 shown in FIG. 3 are the same or similar to the embodiments of step S101, step S102 and step S103 shown in FIG. No more details are included here by reference.

In some implementations, the device further includes a module (not shown), configured to obtain the user's application usage record information about the applications installed in the head-mounted device, and generate a corresponding shortcut according to the application usage record information Instruction identifiers, wherein each shortcut instruction identifier includes indication information indicating an application corresponding to a piece of application usage record information.

In some implementations, the device further includes a module (not shown), configured to acquire head movement information of the user, and determine the position of the user's gaze point in the interface according to the head movement information. Gaze position information; if the gaze position information is within the identification range of a certain trigger identification, execute the trigger instruction corresponding to the trigger identification, wherein the trigger identification includes the at least one application identification, the at least one parameter setting identification and The at least one shortcut instruction identifier.

In some implementations, if the gaze position information is within the identification range of a certain trigger mark, executing the trigger instruction corresponding to the trigger mark includes: if the gaze position information is within the trigger range of a certain trigger mark, and After obtaining the user's trigger confirmation operation, execute the trigger instruction corresponding to the trigger identifier, wherein the trigger identifier includes the at least one application identifier, the at least one parameter setting identifier, and the at least one shortcut instruction identifier.

In some implementations, part or all of the trigger signs in the application operation interface include corresponding voice prompt signs, wherein the voice prompt signs are used to indicate that the corresponding trigger signs can be triggered by voice, and the trigger signs include the At least one application identifier, the at least one parameter setting identifier, and the at least one shortcut instruction identifier.

In some implementations, the device further includes a module (not shown), configured to acquire the voice information of the user; if the voice information is the same as or similar to the preset voice text of a certain trigger flag, execute The trigger instruction corresponds to the trigger identifier, and the trigger identifier includes the at least one application identifier, the at least one parameter setting identifier, and the at least one shortcut instruction identifier.

In some implementations, the triggering instruction includes starting other applications and jumping to the other application interfaces; wherein, the method further includes a module (not shown) configured to close the other application interfaces external interface.

In some implementations, the multiple side interfaces further include a corresponding voice command interface, wherein the voice command interface includes at least one voice command identifier. In some implementations, the device further includes a module (not shown), configured to present a voice prompt sign corresponding to the trigger sign in other side interfaces while presenting the voice interface, wherein the The voice prompt identifier is used to indicate that the corresponding trigger identifier can be triggered by voice, and the trigger identifier includes the at least one application identifier, the at least one parameter setting identifier, and the at least one shortcut instruction identifier.

Here, the specific implementations corresponding to the fourteenth to eighth modules are the same as or similar to the embodiment of step S104 to step S108 shown in FIG. 1 , so they are not repeated here, and are included here by reference.

In addition to the methods and devices introduced in the above embodiments, the present application also provides a computer-readable storage medium, the computer-readable storage medium stores computer codes, and when the computer codes are executed, as described in any one of the preceding items The described method is carried out.

The present application also provides a computer program product, when the computer program product is executed by a computer device, the method described in any one of the preceding items is executed.

The present application also provides a kind of computer equipment, and described computer equipment comprises:

one or more processors;

memory for storing one or more computer programs;

When the one or more computer programs are executed by the one or more processors, the one or more processors are made to implement the method as described in any one of the preceding items.

FIG. 4 illustrates an exemplary system that may be used to implement various embodiments described in this application;

As shown in FIG. 4 , in some embodiments, the system 300 can be used as any one of the above-mentioned devices in each of the above-mentioned embodiments. In some embodiments, system 300 may include one or more computer-readable media (e.g., system memory or NVM/storage device 320 ) having instructions and be coupled to and configured to execute The instructions are one or more processors (eg, processor(s) 305 ) that implement a module to perform the actions described in this application.

For one embodiment, system control module 310 may include any suitable interface controller to provide at least one of processor(s) 305 and/or any suitable device or component in communication with system control module 310 Any suitable interface.

The system control module 310 may include a memory controller module 330 to provide an interface to the system memory 315 . The memory controller module 330 may be a hardware module, a software module and/or a firmware module.

System memory 315 may be used, for example, to load and store data and/or instructions for system 300 . For one embodiment, system memory 315 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, system memory 315 may include Double Data Rate Type Quad Synchronous Dynamic Random Access Memory (DDR4 SDRAM).

For one embodiment, system control module 310 may include one or more input/output (I/O) controllers to provide interfaces to NVM/storage devices 320 and communication interface(s) 325 .

For example, NVM/storage 320 may be used to store data and/or instructions. NVM/storage 320 may include any suitable non-volatile memory (e.g., flash memory) and/or may include any suitable non-volatile storage device(s) (e.g., one or more hard drives ( HDD), one or more compact disc (CD) drives, and/or one or more digital versatile disc (DVD) drives).

NVM/storage device 320 may include a storage resource that is physically part of the device on which system 300 is installed, or it may be accessible by the device without necessarily being part of the device. For example, NVM/storage 320 may be accessed over a network via communication interface(s) 325 .

Communication interface(s) 325 may provide an interface for system 300 to communicate over one or more networks and/or with any other suitable device. System 300 may communicate wirelessly with one or more components of a wireless network according to any of one or more wireless network standards and/or protocols.

For one embodiment, at least one of processor(s) 305 may be packaged with logic of one or more controllers of system control module 310 (eg, memory controller module 330 ). For one embodiment, at least one of the processor(s) 305 may be packaged with the logic of one or more controllers of the system control module 310 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 305 may be integrated on the same die as the logic of the one or more controllers of the system control module 310 . For one embodiment, at least one of the processor(s) 305 may be integrated on the same die with the logic of the one or more controllers of the system control module 310 to form a system on chip (SoC).

In various embodiments, system 300 may be, but is not limited to, a server, workstation, desktop computing device, or mobile computing device (eg, laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, system 300 may have more or fewer components and/or a different architecture. For example, in some embodiments, system 300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), non-volatile memory ports, multiple antennas, graphics chips, application-specific integrated circuits ( ASIC) and speakers.

It should be noted that the present application can be implemented in software and/or a combination of software and hardware, for example, it can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware devices. In one embodiment, the software program of the present application can be executed by a processor to realize the steps or functions described above. Likewise, the software program (including associated data structures) of the present application can be stored in a computer-readable recording medium such as RAM memory, magnetic or optical drive or floppy disk and the like. In addition, some steps or functions of the present application may be implemented by hardware, for example, as a circuit that cooperates with a processor to execute each step or function.

In addition, a part of the present application can be applied as a computer program product, such as a computer program instruction. When it is executed by a computer, the method and/or technical solution according to the present application can be invoked or provided through the operation of the computer. Those skilled in the art should understand that computer program instructions exist in computer-readable media in forms including but not limited to source files, executable files, installation package files, etc. Limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding post-installation program program. Here, a computer readable medium may be any available computer readable storage medium or communication medium that can be accessed by a computer.

Communication media includes the media whereby communication signals embodying, for example, computer readable instructions, data structures, program modules or other data are transmitted from one system to another. Communication media can include guided transmission media such as cables and wires (e.g., fiber optics, coaxial, etc.) and wireless (unguided transmission) media capable of propagating waves of energy, such as acoustic, electromagnetic, RF, microwave, and infrared . Computer readable instructions, data structures, program modules or other data may be embodied, for example, as a modulated data signal in a wireless medium such as a carrier wave or similar mechanism such as embodied as part of spread spectrum technology. The term "modulated data signal" means a signal that has one or more of its characteristics changed or set in such a manner as to encode information in the signal. Modulation can be analog, digital or mixed modulation techniques.

By way of example, and not limitation, computer-readable storage media may include volatile and nonvolatile, volatile, volatile, or Removable and non-removable media. For example, computer-readable storage media include, but are not limited to, volatile memories such as random access memories (RAM, DRAM, SRAM); and nonvolatile memories such as flash memory, various read-only memories (ROM, PROM, EPROM) , EEPROM), magnetic and ferromagnetic/ferroelectric memory (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, tapes, CDs, DVDs); or other media known now or developed in the future capable of storing data for computer systems Computer readable information/data used.

Here, an embodiment according to the present application includes an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, triggering The operation of the device is based on the foregoing methods and/or technical solutions according to multiple embodiments of the present application.

It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, but that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be regarded as exemplary and not restrictive in all points of view, and the scope of the application is defined by the appended claims rather than the foregoing description, and it is intended that the scope of the present application be defined by the appended claims rather than by the foregoing description. All changes within the meaning and range of equivalents of the elements are embraced in this application. Any reference sign in a claim should not be construed as limiting the claim concerned. In addition, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means stated in the device claims may also be realized by one unit or device through software or hardware. The words first, second, etc. are used to denote names and do not imply any particular order.

Claims (15)

An interface interaction method, wherein, applied to a head-mounted device, the method includes: Presenting the current application interface of the current application being used by the user through the display device of the head-mounted device; Obtaining the user's interface interaction operations on the current application interface; Presenting a corresponding application operation interface based on the interface interaction operation, where the application operation interface includes multiple interfaces, and the multiple interfaces include the current application interface and multiple side interfaces of the current application interface, so The multiple side interfaces include a corresponding application identification interface and a function interface, the application identification interface is used to present multiple application identifications, the function interface includes a shortcut function interface, and the shortcut function interface includes at least one shortcut instruction identification, Each shortcut instruction identifier is used to trigger and generate a corresponding shortcut instruction. The method according to claim 1, wherein the function interface further includes a parameter setting interface, and the parameter setting interface includes at least one parameter setting identifier. The method according to claim 2, wherein the parameter setting interface and the shortcut function interface are distributed on different sides of the current application interface. The method according to claim 2, wherein the method further comprises: Obtaining the user's application usage record information about the application installed in the head-mounted device, and generating a corresponding shortcut instruction identifier according to the application usage record information, wherein each shortcut instruction identifier includes an indication indicating an application corresponding to the application usage record information information. The method according to claim 2, wherein the method further comprises: Obtain the head movement information of the user, and determine the gaze position information of the user's gaze point in the interface according to the head movement information; If the gaze position information is within the identification range of a trigger identification, execute the trigger instruction corresponding to the trigger identification, wherein the trigger identification includes the at least one application identification, the at least one parameter setting identification and the at least one Shortcut ID. The method according to claim 5, wherein, if the gaze position information is within the range of a certain trigger mark, executing the trigger instruction corresponding to the trigger mark includes: If the gaze position information is within the trigger range of a certain trigger mark, and the trigger confirmation operation of the user is acquired, then execute the trigger instruction corresponding to the trigger mark, wherein the trigger mark includes the at least one application mark, The at least one parameter setting identifier and the at least one shortcut instruction identifier. The method according to claim 2, wherein, part or all of the trigger signs in the application operation interface include corresponding voice prompt signs, wherein the voice prompt signs are used to indicate that the corresponding trigger signs can be triggered by voice, and the The triggering identifier includes the at least one application identifier, the at least one parameter setting identifier, and the at least one shortcut instruction identifier. The method according to claim 2, wherein the method further comprises: Obtain voice information of the user; If the voice information is the same as or similar to the preset voice text of a certain trigger identifier, execute the trigger instruction corresponding to the trigger identifier, the trigger identifier includes the at least one application identifier, the at least one parameter setting identifier and the At least one shortcut ID. The method according to any one of claims 5 to 8, wherein the trigger instruction includes starting other applications and jumping to the other application interface; wherein the method further includes: Close interfaces other than the other application interfaces described. The method according to claim 1, wherein the multiple side interfaces further include a corresponding voice command interface, wherein the voice command interface includes at least one voice command identifier. The method according to claim 10, wherein the method further comprises: While presenting the voice instruction interface, a voice prompt mark corresponding to the trigger mark is presented in other side interfaces, wherein the voice prompt mark is used to indicate that the corresponding trigger mark can be triggered by voice, and the trigger mark includes the At least one application identifier, the at least one parameter setting identifier, and the at least one shortcut instruction identifier. A head-mounted device for interface interaction, the device comprising: A module, configured to present the current application interface of the current application being used by the user through the display device of the head-mounted device; A one-two module, configured to obtain the interface interaction operation of the user on the current application interface; A third module, configured to present a corresponding application operation interface based on the interface interaction operation, wherein the application operation interface includes multiple interfaces, and the multiple interfaces include the current application interface and multiples of the current application interface a plurality of side interfaces, the multiple side interfaces include corresponding application identification interfaces and function interfaces, the application identification interfaces are used to present multiple application identifications, the function interfaces include a shortcut function interface, and the shortcut function interface includes At least one shortcut instruction identifier, each shortcut instruction identifier is used to trigger and generate a corresponding shortcut instruction. A computer device, wherein the device includes: processor; and A memory arranged to store computer-executable instructions which, when executed, cause the processor to perform the steps of the method as claimed in any one of claims 1 to 11. A computer-readable storage medium on which computer programs/instructions are stored, wherein the computer program/instructions cause a system to execute the steps of the method as claimed in any one of claims 1 to 11 when executed. A computer program product, comprising computer programs/instructions, characterized in that, when the computer program/instructions are executed by a processor, the steps of the method described in any one of claims 1 to 11 are implemented.

Images