CN117130471A - Man-machine interaction method, electronic equipment and system - Google Patents

Abstract

The application discloses a man-machine interaction method, electronic equipment and a system, and relates to the technical field of terminals. A first application for managing the virtual space of the head-mounted display device is installed on the electronic device; after the first application is started, the electronic equipment receives a screen capturing gesture, a screen recording gesture or a gesture returning to the previous stage of the display interface of the first application. Because the first application runs in the foreground, system gestures such as screen capturing gestures, screen recording gestures or returning to a previous stage gesture act on the virtual space of the head-mounted display device. Therefore, when a user watches the picture in the virtual space, the user can realize operations such as screen capturing, screen recording, returning to the upper level and the like on the display interface of the virtual space of the head-mounted display device only by making gestures on the touch screen of the electronic device. The user's line of sight does not need to leave the virtual space, avoiding the line of sight jumping between the virtual space and the electronic device screen. Moreover, the gesture operation is convenient and quick, the operation cost of the user is low, and the experience is good.

Description

Human-computer interaction methods, electronic devices and systems

Technical field

The present application relates to the field of terminal technology, and in particular to a human-computer interaction method, electronic device and system.

Background technique

With the development of computer graphics technology, extended reality (XR) technologies such as augmented reality (AR), virtual reality (VR), mixed reality (mediated reality, MR) are gradually applied to people's lives. . The human-computer interaction method of existing XR equipment is usually operated through electronic devices such as mobile phones as remote controls. How to improve the convenience of operating XR equipment through electronic equipment is a problem that needs to be solved.

Contents of the invention

Embodiments of the present application provide a human-computer interaction method, electronic device and system, which can provide a convenient human-computer interaction method, facilitate users to operate the virtual space of XR devices through mobile phones and other electronic devices, and improve the user experience.

In order to achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

In a first aspect, a human-computer interaction method is provided, applied to an electronic device, and the electronic device is connected to a head-mounted display device. The method includes: after the electronic device starts the first application, it receives the user's input on the first application display interface. The first gesture; the first application is used to manage the virtual space of the head-mounted display device; the first gesture includes a screenshot gesture, a screen recording gesture, or a return to the previous level gesture. Since the first application is running in the foreground, system gestures such as screenshot gestures, screen recording gestures, or return to the previous level gestures act on the head-mounted display device. In response to the first gesture, the electronic device performs an action corresponding to the first gesture with respect to the virtual space of the head-mounted display device.

Among them, the first gesture is a screenshot gesture, and the action corresponding to the first gesture performed on the display interface of the virtual space includes: taking a screenshot of the display interface of the virtual space; the first gesture is a screen recording gesture, and the first gesture is performed on the display interface of the virtual space. The actions corresponding to the gesture include: recording the screen of the display interface of the virtual space; the first gesture is a gesture to return to the previous level, and the action corresponding to the first gesture performed on the display interface of the virtual space includes: causing the virtual space to display the upper part of the current display interface. Level 1 display interface.

In this method, the user is supported to perform system gestures (screen capture gestures, screen recording gestures or return to the previous level gestures) on the touch screen of the electronic device to realize screenshots, screen recordings, and Return to the previous level and other operations. In this way, when the user is watching the screen in the virtual space, he only needs to make system gestures on the touch screen of the electronic device to take screenshots, record the screen, and return to the previous level on the display interface of the virtual space of the head-mounted display device. . The user's sight does not need to leave the virtual space, avoiding the jump between the virtual space and the electronic device screen. Moreover, gesture operation is convenient and fast, and the user operation cost is low and the user experience is good.

With reference to the first aspect, in one implementation, if the first application is closed or retreats to the background, the electronic device exits and displays the first application display interface, and the electronic device displays the application interface of the second application; for example, the second application Applications can be desktop applications, system applications, third-party applications, etc. The electronic device receives the user's first gesture on the second application display interface; in response to the first gesture on the second application display interface, the electronic device performs an action corresponding to the first gesture on the second application display interface. Wherein, the first gesture is a screenshot gesture, and the action corresponding to the first gesture performed on the second application display interface includes: taking a screenshot of the second application display interface; the first gesture is a screen recording gesture, and the first gesture is performed on the second application display interface. Actions corresponding to the gesture include: recording the screen of the second application display interface; the first gesture is a gesture to return to the previous level, and actions corresponding to the first gesture performed on the second application display interface include: the electronic device displays the second application display interface Previous level display interface.

In this method, after receiving the system gesture, the electronic device distinguishes between different situations and triggers the electronic device itself to perform the action corresponding to the system gesture, or triggers the head-mounted display device to perform the action corresponding to the system gesture. If the first application is not running in the foreground, the electronic device itself performs the action corresponding to the system gesture (performs the action corresponding to the first gesture for the second application display interface). Supports operations such as taking screenshots, recording screens, and returning to the previous level on electronic devices through gestures.

With reference to the first aspect, in one embodiment, the screenshot gesture includes any one of sliding down with multiple fingers on the screen, sliding up with multiple fingers, double-clicking the screen with one knuckle, or double-clicking the screen with two knuckles; the screen recording gesture includes Swipe down with multiple fingers on the screen, slide up with multiple fingers, double-tap the screen with one knuckle, or double-tap the screen with two knuckles; among them, the screenshot gesture is different from the screen recording gesture; the gesture to return to the previous level includes single-finger tapping from the left side of the screen Either swipe side to right, swipe one finger from the right side of the screen to the left, or swipe your finger inward from both sides of the screen.

With reference to the first aspect, in one implementation, the electronic device generates a display interface of the virtual space and sends the display interface to the head-mounted display device. That is to say, the electronic device generates the display data and display interface of the head-mounted display device, the electronic device provides display data for the head-mounted display device, and the head-mounted display device serves as a display device for the electronic device.

In this implementation, the electronic device locally stores information about the display interface of the head-mounted display device, and the electronic device takes screenshots, records, or generates the display interface of the virtual space based on the locally obtained display interface of the head-mounted display device. Level 1 display interface. In this way, when taking a screenshot or recording a screen, the electronic device does not need to obtain a large amount of data on the virtual space display interface from the head-mounted display device, reducing the data interaction between the electronic device and the head-mounted display device, and avoiding the interference between the electronic device and the head-mounted display device. Displays the restrictions on device communication rate.

With reference to the first aspect, in one implementation, the electronic device triggers the head-mounted display device to take a screenshot, record a screen, or generate a higher-level display interface of the display interface of the virtual space. Further, the head-mounted display device can send the screenshot image generated by the screenshot or the screen recording file generated by the screen recording to the electronic device.

In a second aspect, a human-computer interaction method is provided, applied to an electronic device, and the electronic device is connected to a head-mounted display device. The method includes: the electronic device starts a first step for managing the virtual space of the head-mounted display device. After application, the user's first operation on the first application display interface is received; in response to the first operation, the electronic device triggers the virtual space of the head-mounted display device to display a screenshot virtual button, a screen recording virtual button, and a return to the previous level virtual button. and at least one of the virtual keys to return to the homepage. Among them, the screenshot virtual button is used to take a screenshot of the display interface of the virtual space; the screen recording virtual button is used to record the screen of the virtual space display interface; the return to the previous level virtual button is used to trigger the virtual space to display the previous level of the current display interface. level display interface; the return to homepage virtual button is used to trigger the virtual space to display the desktop interface.

In this method, in response to the first operation input by the user on the application interface of the first application on the electronic device, virtual buttons such as screenshots, screen recordings, return to the previous level, and return to the home page pop up in the virtual space of the head-mounted display device. These virtual buttons do not depend on the current display interface of the virtual space and can pop up on any display interface of the virtual space. In this way, when the user needs to take screenshots, record the screen, return to the previous level, return to the homepage, etc., the user does not need to leave the virtual space, and does not need to exit the current application displayed in the virtual space. He only needs to make simple gestures, and the operation is simple and fast. , the user experience is better.

Combined with the second aspect, in one embodiment, the electronic device triggers the virtual space of the head-mounted display device to display at least one of a screenshot virtual button, a screen recording virtual button, a return to the previous level virtual button, and a return to homepage virtual button, including : The electronic device triggers the virtual space of the head-mounted display device to display a shortcut operation panel. The shortcut operation panel is provided with at least one of a screenshot virtual button, a screen recording virtual button, a return to the previous level virtual button, and a return to homepage virtual button.

Among them, the shape, size, and position of the quick operation panel can be set according to specific conditions.

In one implementation, the shortcut operation panel is superimposed on the application interface of the second application displayed in the virtual space.

In another implementation, the shortcut operation panel is displayed in a blank area of the virtual space display interface.

In connection with the second aspect, in one embodiment, at least one of a screenshot virtual button, a screen recording virtual button, a return to the previous level virtual button, and a return to the homepage virtual button is superimposed on the application interface of the second application displayed in the virtual space. .

Combined with the second aspect, in one embodiment, the first operation includes: any one of single-finger tapping, multi-finger tapping, knuckle tapping, or single-finger circling.

Combined with the second aspect, in one implementation, in response to the first operation, the electronic device triggers the virtual space of the head-mounted display device to display a screenshot virtual button, a screen recording virtual button, a return to the previous level virtual button, and a return to home page virtual button. At least one of the above, including: in response to the first operation, the electronic device generates first display data and sends the first display data to the head-mounted display device; the first display data is used for the head-mounted display device to display screenshots in the virtual space At least one of the virtual buttons, the screen recording virtual button, the return to the previous level virtual button, and the return to the homepage virtual button.

In this implementation, the electronic device generates display data and a display interface of the head-mounted display device, the electronic device provides display data for the head-mounted display device, and the head-mounted display device serves as a display device of the electronic device.

Combined with the second aspect, in one implementation, in response to the first operation, the electronic device triggers the virtual space of the head-mounted display device to display a screenshot virtual button, a screen recording virtual button, a return to the previous level virtual button, and a return to home page virtual button. At least one of the following, including: in response to the first operation, the electronic device sends a first event corresponding to the first operation to the head-mounted display device; the first event is used to trigger the head-mounted display device to generate first display data, and the first The display data is used by the head-mounted display device to display at least one of a screenshot virtual button, a screen recording virtual button, a return to the previous level virtual button, and a return to homepage virtual button in the virtual space.

A third aspect provides an electronic device having the function of implementing the method described in the first or second aspect. This function can be implemented by hardware, or it can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

In a fourth aspect, an electronic device is provided, including: a processor, a touch screen and a memory; the memory is used to store computer execution instructions, and when the electronic device is running, the processor executes the computer execution instructions stored in the memory to The electronic device is caused to perform the method described in any one of the first aspect or the second aspect.

In a fifth aspect, an electronic device is provided, including: a processor; the processor is configured to be coupled to a memory, and after reading instructions in the memory, execute any of the above first or second aspects according to the instructions. method described in one item.

In a sixth aspect, a computer-readable storage medium is provided. Instructions are stored in the computer-readable storage medium. When the computer-readable storage medium is run on a computer, the computer can execute any one of the above-mentioned first or second aspects. method described.

A seventh aspect provides a computer program product containing instructions that, when run on a computer, enable the computer to execute the method described in any one of the above first or second aspects.

An eighth aspect provides a device (for example, the device may be a chip system). The device includes a processor and is used to support an electronic device to implement the functions involved in the first or second aspect. In a possible design, the device further includes a memory, which is used to store necessary program instructions and data of the electronic device. When the device is a chip system, it may be composed of a chip or may include a chip and other discrete components.

Among them, the technical effects brought by any one of the design methods in the third to eighth aspects can be referred to the technical effects brought by different design methods in the first aspect or the second aspect, and will not be described again here.

Description of the drawings

Figure 1 is a schematic diagram of the system architecture applicable to the human-computer interaction method provided by the embodiment of the present application;

Figure 2 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application;

Figure 3A is a schematic diagram of the optical structure of a head-mounted display device provided by an embodiment of the present application;

Figure 3B is a schematic diagram of the hardware structure of a head-mounted display device provided by an embodiment of the present application;

Figure 4A is a schematic diagram of an example scenario of human-computer interaction between the user and the head-mounted display device;

Figure 4B is a schematic diagram of another scenario example of human-computer interaction between the user and the head-mounted display device;

Figure 5 is a schematic diagram of a scenario example in which a screenshot gesture is received when a mobile phone displays the application interface of an AR application provided by an embodiment of the present application;

Figure 6 is a schematic diagram of a scenario example in which a screenshot gesture is received when the mobile phone does not display the application interface of the AR application provided by the embodiment of the present application;

Figure 7 is a schematic diagram of a scene example in which a screen recording gesture is received when a mobile phone displays the application interface of an AR application provided by an embodiment of the present application;

Figure 8 is a schematic diagram of a scene example in which a screen recording gesture is received when the mobile phone does not display the application interface of the AR application provided by the embodiment of the present application;

Figure 9 is a schematic diagram of a scenario example in which the mobile phone receives a return gesture when displaying the application interface of an AR application provided by an embodiment of the present application;

Figure 10 is a schematic diagram of a scenario example in which a return gesture is received when the mobile phone does not display the application interface of the AR application provided by the embodiment of the present application;

Figure 11 is a schematic flowchart of a human-computer interaction method provided by an embodiment of the present application;

Figures 12-13 are a detailed flow chart of the human-computer interaction method provided by the embodiment of the present application;

Figure 14 is a schematic diagram of a scenario example in which the mobile phone receives the first operation provided by the embodiment of the present application;

Figures 15A-15D are schematic diagrams of scene examples in which users perform human-computer interaction in virtual space provided by embodiments of the present application;

Figures 16A-16C are schematic diagrams of examples of virtual space display interfaces provided by embodiments of the present application;

Figure 17 is another schematic flow chart of the human-computer interaction method provided by the embodiment of the present application;

Figure 18 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

Figure 19 is a schematic structural diagram of a chip system provided by an embodiment of the present application.

Detailed ways

In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "above," "the" and "the" are intended to also include, for example, "a "or more" unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more than two (including two). The term "and/or" is used to describe the relationship between associated objects, indicating that there can be three relationships; for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and B exists alone, Where A and B can be singular or plural. The character "/" generally indicates that the related objects are in an "or" relationship.

Reference in this specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Therefore, the phrases "in one embodiment", "in some embodiments", "in other embodiments", "in other embodiments", etc. appearing in different places in this specification are not necessarily References are made to the same embodiment, but rather to "one or more but not all embodiments" unless specifically stated otherwise. The terms “including,” “includes,” “having,” and variations thereof all mean “including but not limited to,” unless otherwise specifically emphasized. The term "connected" includes both direct and indirect connections unless otherwise stated. “First” and “second” are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features.

In the embodiments of this application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "such as" in the embodiments of the present application is not to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the words "exemplary" or "such as" is intended to present the concept in a concrete manner.

The human-computer interaction method provided by the embodiment of the present application can be applied to the system shown in Figure 1. The system 10 may include an electronic device 100 and a head-mounted display device 200 .

The electronic device 100 and the head-mounted display device 200 may be connected in a wired or wireless manner. Wired connections may include wired connections that communicate through interfaces such as a universal serial bus (USB) interface and a high definition multimedia interface (HDMI) interface. Wireless connections may include one or more wireless connections that communicate through Bluetooth, wireless fidelity (Wi-Fi) direct connection (such as Wi-Fi p2p), Wi-Fi softAP, Wi-Fi LAN, radio frequency and other technologies. Multiple items. The embodiment of the present application does not limit the connection method between the two.

The electronic device 100 may be a mobile phone, a tablet computer, or a laptop computer (Laptop) with a touch-sensitive surface or a touch panel, a desktop computer with a touch-sensitive surface or a touch panel, or other non-portable terminal equipment. The electronic device 100 may run a specific application program to provide content transmitted to the head-mounted display device 200 for display. The application program may be, for example, a video application, a game application, a music application, a desktop application, a mirroring screen application, etc.

Possible implementation forms of the head-mounted display device 200 include helmets, glasses, headphones and other electronic devices that can be worn on the user's head. The head-mounted display device 200 uses AR, VR, MR and other technologies to display images in the virtual space, allowing users to experience 3D scenes and providing users with AR/VR/MR experiences. The 3D scene may include 3D images, 3D videos, audio, etc. It can be understood that the virtual space shown in Figure 1 is a plane. In actual use, the virtual space may be a curved space with curvature.

The head-mounted display device 200 can be worn on the user's head and is equivalent to an epitaxial display of the electronic device 100 . The electronic device 100 provides display data to the head-mounted display device 200 .

The electronic device 100 can also serve as an input device to receive user operations such as clicks and slides, and can deliver rays to the AR/VR/MR virtual space (AR space, VR space or MR space) to simulate mouse actions to facilitate user interaction. Control operations performed on the content displayed on the head-mounted display device 200 .

When the electronic device 100 is used as an input device, user input can be received through various sensors configured therein, such as touch-sensitive sensors, acceleration sensors, gyroscope sensors, magnetic sensors, pressure sensors, etc. Among them, acceleration sensors and gyroscope sensors can be used to detect the user's operation of the mobile electronic device 100, which can be used to change the direction of the ray; touch-sensitive sensors, pressure sensors, etc. can be used to detect the user's touch on a touch panel such as a touch screen. Operations, such as sliding operations, click operations, short press operations, long press operations, etc.

The head-mounted display device 200 may be configured with some physical buttons to receive some user inputs, such as buttons for turning the screen on and off, buttons for adjusting screen brightness, buttons for switching space mode and mirror mode, etc. These user inputs may be transmitted to the electronic device 100 through a wired or wireless communication connection between the head-mounted display device 200 and the electronic device 100, thereby triggering the electronic device 100 to respond thereto. For example, in response to a user input switching from the spatial mode to the mirror mode, the electronic device 100 may stop transmitting the display data of the spatial mode to the head-mounted display device 200 and start transmitting the display data of the mirror mode. The display data in the mirror mode is mainly the screen stream of the electronic device 100 , which can be provided by the mirror projection application on the electronic device 100 . The display data of the spatial mode may be provided by a specific application program on the electronic device 100, which may be a video application, a game application, a music application, a desktop application, etc.

After the user sees the image displayed by the head-mounted display device 200, he or she can control the display content in the virtual space and the working status of the head-mounted display device 200 by inputting user operations on the electronic device 100 or the head-mounted display device 200. , such as switch status, screen brightness, etc.

FIG. 2 schematically shows a hardware architecture diagram of the electronic device 100 provided by the embodiment of the present application. As shown in Figure 2, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, and a battery. 142. Antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, sensor module 180, camera 191, and display screen 192. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (GPU), an image signal processing unit. (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processing unit (NPU), etc. . Among them, different processing units can be independent devices or integrated in one or more processors.

The controller can generate operation control signals based on the instruction operation code and timing signals to complete the control of fetching and executing instructions.

The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have been recently used or recycled by processor 110 . If the processor 110 needs to use the instructions or data again, it can be called directly from the memory. Repeated access is avoided and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuitsound, I2S) interface, a pulse code modulation (PCM) interface, and a universal asynchronous receiver (universal asynchronous receiver) /transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and/or Universal serial bus (USB) interface, etc.

The charging management module 140 is used to receive charging input from the charger. Among them, the charger can be a wireless charger or a wired charger.

The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110.

The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example: Antenna 1 can be reused as a diversity antenna for a wireless LAN. In other embodiments, antennas may be used in conjunction with tuning switches.

The mobile communication module 150 can provide solutions for wireless communication including 2G/3G/4G/5G applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, perform filtering, amplification and other processing on the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be disposed in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.

A modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 192 . In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be independent of the processor 110 and may be provided in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide applications on the electronic device 100 including wireless local area network (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), Near field communication technology (NFC), infrared technology (IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (codedivision multiple access, CDMA), broadband code Wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM , and/or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou satellite navigation system (beidounavigation satellite system, BDS), quasi-zenith satellite system (quasi- zenith satellitesystem (QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 100 implements display functions through a GPU, a display screen 192, an application processor, and the like. The GPU is an image processing microprocessor and is connected to the display screen 192 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 192 is used to display images, videos, etc. Display 192 includes a display panel. The display panel can use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode). (AMOLED), flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 192, where N is a positive integer greater than 1.

Pressure sensors are used to sense pressure signals and convert pressure signals into electrical signals. In some embodiments, a pressure sensor may be provided on display screen 192 . There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor may include at least two parallel plates of conductive material. When a force acts on a pressure sensor, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is performed on the display screen 192, the electronic device 100 detects the strength of the touch operation according to the pressure sensor. The electronic device 100 may also calculate the touched position based on the detection signal of the pressure sensor.

Touch sensor, also called "touch panel". The touch sensor can be disposed on the display screen 192, and the touch sensor and the display screen 192 form a touch screen, which is also called a "touch screen". Touch sensors are used to detect touches on or near them. The touch sensor can pass the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation may be provided through display screen 192 .

The electronic device 100 may detect gestures made by the user on the display screen 192 through a pressure sensor or a touch sensor. For example, swipe up gestures, swipe down gestures, swipe left gestures, swipe right gestures, click gestures, long press gestures, tap gestures, etc.

The electronic device 100 can implement the shooting function through an ISP, a camera 191, a video codec, a GPU, a display screen 192, an application processor, and the like.

The ISP is used to process the data fed back by the camera 191. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera sensor through the lens, the optical signal is converted into an electrical signal, and the camera sensor passes the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye.

Camera 191 is used to capture still images or video. The object passes through the lens to produce an optical image that is projected onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to convert it into a digital image signal. In some embodiments, the electronic device 100 may include 1 or N cameras 191, where N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can continuously learn by itself.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100 .

Internal memory 121 may be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the stored program area can store an operating system, at least one application program required for a function (such as a sound playback function, an image playback function, etc.). The storage data area may store data created during use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.

The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, and the application processor. Such as music playback, recording, etc. Among them, the audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert analog audio input into a digital audio signal. Audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110 , or some functional modules of the audio module 170 may be provided in the processor 110 .

Internal memory 121 may be used to store an application program for one or more applications, the application program including instructions. When executed by the processor 110, the application program causes the electronic device 100 to generate content for presentation to the user. For example, the application may include an application for managing the head-mounted display device 200, a game application, a conferencing application, a video application, a desktop application or other applications, etc.

The GPU can be used to perform mathematical and geometric operations based on data obtained from the processor 110 (such as data provided by an application program), render images using computer graphics technology, computer simulation technology, etc., and determine the conditions for use in the head-mounted display device 200 image shown on. In some embodiments, the GPU may add corrections or pre-distortion to the rendering process of the image to compensate for or correct for distortion caused by the optical components of the head mounted display device 200 .

In this embodiment of the present application, the electronic device 100 can send the image processed by the GPU to the head-mounted display device 200 through the mobile communication module 150, the wireless communication module 160, or a wired interface.

It should be noted that the structure illustrated in FIG. 2 does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figures, or some components may be combined, some components may be separated, or some components may be arranged differently. The components illustrated may be implemented in hardware, software, or a combination of software and hardware.

FIG. 3A schematically shows the optical structure of the head-mounted display device 200 provided by the embodiment of the present application. As shown in FIG. 3A , the head-mounted display device 200 may include: a display screen 201 , an optical component 202 , a display screen 203 , and an optical component 204 . The display screen 201 and the display screen 203 may be a whole, that is, the left and right parts of a whole screen. The optical component 202 and the optical component 204 have the same material and structure. Optical assembly 202 and optical assembly 204 may be composed of one or more lenses, which may include one or more of convex lenses, Fresnel lenses, or other types of lenses.

Display 201 and optical assembly 202 correspond to the user's left eye. When the user wears the head-mounted display device 200, the image a1 may be displayed on the display screen 201. The light emitted when the display screen 201 displays the image a1 will form a virtual image a1' of the image a1 in front of the user's left eye after being transmitted by the optical component 202.

Display 203 and optical assembly 204 correspond to the user's right eye. When the user wears the head-mounted display device, the display screen 203 may display image a2. The light emitted when the display screen 203 displays the image a2 is transmitted by the optical component 204 and will form a virtual image a2' of the image a2 in front of the user's right eye.

Image a1 and image a2 are two images with parallax for the same object, such as object a. Parallax refers to the difference in the position of the object in the field of view when the same object is viewed from two points at a certain distance. The virtual image a1' and the virtual image a2' are located on the same plane, and this plane can be called a virtual image plane.

When wearing the head-mounted display device 200, the user's left eye will focus on the virtual image a1', and the user's right eye will focus on the virtual image a2'. Then, the virtual image a1’ and the virtual image a2’ will be superimposed in the user’s brain to form a complete and three-dimensional image. This process is called vergence. During the convergence process, the intersection point of the two eyes will be considered by the user to be the actual location of the object described in image a1 and image a2. Due to the convergence process, the user can feel the 3D scene provided by the head-mounted display device 200 .

FIG. 3B schematically shows the hardware architecture of the head-mounted display device 200 provided by the embodiment of the present application. As shown in FIG. 3B , the head-mounted display device 200 may include: a processor 210, a memory 211, a communication module 212, a sensor system 213, a camera 214, a display device 215, and an audio device 216. Each of the above components can be coupled and communicate with each other.

It can be understood that the structure shown in FIG. 3B does not constitute a specific limitation on the head-mounted display device 200. In other embodiments of the present application, the head-mounted display device 200 may include more or fewer components than shown in the figures, or combine some components, or split some components, or arrange different components. For example, the head-mounted display device 200 may also include physical buttons such as on/off keys, volume keys, screen brightness adjustment keys, and various interfaces, such as USB interfaces. The components illustrated may be implemented in hardware, software, or a combination of software and hardware.

The processor 210 may include one or more processing units. For example, the processor may include an AP, a modem processor, a GPU, an ISP, a controller, a video codec, a DSP, a baseband processor, and/or an NPU, etc. Among them, different processing units can be independent devices or integrated in one or more processors. The controller can generate operation control signals based on the instruction operation code and timing signals to complete the control of fetching and executing instructions, allowing each component to perform corresponding functions, such as human-computer interaction, motion tracking/prediction, rendering display, audio processing, etc.

Memory 211 may store some executable instructions. The memory 211 may include a program storage area and a data storage area. Among them, the stored program area can store an operating system, at least one application program required for a function (such as a sound playback function, an image playback function, etc.). The storage data area may store data created during use of the head-mounted display device 200 (such as audio data, etc.). In addition, the memory 211 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 210 executes various functional applications and data processing of the head-mounted display device 200 by executing instructions stored in the memory 211 and/or instructions stored in a memory provided in the processor.

The communication module 212 may include a mobile communication module and a wireless communication module. Among them, the mobile communication module can provide wireless communication solutions including 2G/3G/4G/5G applied on the head-mounted display device 200. The wireless communication module can provide information that is applied on the head-mounted display device 200, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (bluetooth, BT), global Wireless communication solutions such as global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (infrared, IR), etc. The wireless communication module may be one or more devices integrating at least one communication processing module.

Sensor system 213 may include an accelerometer, compass, gyroscope, magnetometer, or other sensor for detecting motion, or the like. The sensor system 213 is used to collect corresponding data. For example, the acceleration sensor collects the acceleration of the head-mounted display device 200 , the gyroscope sensor collects the movement speed of the head-mounted display device 200 , etc. The data collected by the sensor system 213 can reflect the movement of the head of the user wearing the head-mounted display device 200 . In some embodiments, the sensor system 213 may be an inertial measurement unit (IMU) disposed within the head-mounted display device 200 . In some embodiments, the head mounted display device 200 may send data acquired by the sensor system to the processor 210 for analysis. The user can trigger the head-mounted display device 200 to perform corresponding functions by inputting head movement operations on the head-mounted display device 200 . The movement of the user's head may include: whether to rotate, the direction of rotation, etc.

Sensor system 213 may also include optical sensors for tracking the user's eye position and capturing eye movement data in conjunction with camera 214 . This eye movement data may be used, for example, to determine the distance between the user's eyes, the 3D position of each eye relative to the head-mounted display device 200 , the magnitude and gaze direction of each eye's twist and rotation (i.e., roll, pitch, and shake), etc. . In one example, infrared light is emitted within the head mounted display device 200 and reflected from each eye. The reflected light is detected by the camera 214 or an optical sensor, and the detected data is transmitted to the processor 210 such that the processor 210 The position, pupil diameter, movement status, etc. of the user's eyes are analyzed from changes in the infrared light reflected by each eye.

Camera 214 may be used to capture still images or video. This static image or video may be an externally facing image or video of the user's surroundings or an internally facing image or video. The camera 214 can track the movement of one or both eyes of the user. The camera 214 includes but is not limited to a traditional color camera (RGB camera), a depth camera (RGB depth camera), a dynamic vision sensor (dynamic vision sensor, DVS) camera, etc. The depth camera can obtain the depth information of the subject. In some embodiments, camera 214 may be used to capture images of the user's eyes and send the images to processor 210 for analysis. The processor 210 can determine the state of the user's eyes based on the images collected by the camera 214, and perform corresponding functions according to the state of the user's eyes. That is to say, the user can trigger the head-mounted display device 200 to perform corresponding functions by inputting eye movement operations on the head-mounted display device 200 . The status of the user's eyes may include: whether it is turned, the direction of the turn, whether it has not been turned for a long time, the angle at which it looks to the outside world, etc.

The audio device 216 is used to implement audio collection and output. Audio devices 216 may include, but are not limited to, microphones, speakers, headphones, etc.

The head-mounted display device 200 presents or displays images through a GPU, a display device 215, an application processor, and the like.

The GPU is an image processing microprocessor and is connected to the display device 215 and the application processor. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information. The GPU is used to perform mathematical and geometric calculations based on data obtained from the processor 210, and render images using computer graphics technology, computer simulation technology, etc., to provide content for display on the display device 215. The GPU is also used to add corrections or pre-distortion to the rendering process of the image to compensate or correct for distortion caused by optical components in the display device 215 . The GPU may also adjust content provided to display device 215 based on data from sensor system 213 . For example, the GPU may add depth information to the content provided to the display device 215 based on the 3D position of the user's eyes, interpupillary distance, etc.

The display device 215 may include one or more display screens and one or more optical components. Here, the structure of the display screen and optical components and the positional relationship between them may refer to the relevant description in FIG. 3A. The display screen may include a display panel, and the display panel may be used to display images to present a three-dimensional virtual scene to the user. The display panel can adopt LCD, OLED, AMOLED, FLED, Miniled, MicroLed, Micro-oLed, QLED, etc. Optical components can be used to direct light from the display to the exit pupil for perception by the user. In some embodiments, one or more optical elements (eg, lenses) in an optical assembly may have one or more coatings, such as anti-reflective coatings. The amplification of image light by optical components allows displays to be physically smaller, lighter and consume less power. In addition, the amplification of the image light can increase the field of view of the content displayed on the display screen. For example, the optical component can make the field of view of the content displayed on the display screen the entire field of view of the user.

In this embodiment of the present application, the display screen in the head-mounted display device 200 can be used to display display data transmitted from the electronic device 100 to provide the user with an XR experience.

The head-mounted display device 200 can display images in the virtual space, allowing the user to experience the 3D scene, and providing the user with an AR/VR/MR experience. Exemplarily, as shown in FIG. 4A , the virtual space of the head-mounted display device 200 displays a desktop interface, which can display desktop icons of various applications, such as "Contacts", "Calendar", and "Weather" , "Video", "Time", "Shopping", "Game", "Short Video", "Music" and other desktop icons for applications. Not limited to applications, the desktop interface can also include objects such as files and folders. In some virtual spaces, users can operate display objects in the virtual space by moving rays in the virtual space.

The electronic device 100 can be used as an input device for the head-mounted display device 200 , such as using a mobile phone as an input device for AR glasses. The user can implement human-computer interaction with the head-mounted display device 200 by operating on the electronic device 100 . As an example, the electronic device 100 is a mobile phone and the head-mounted display device 200 is AR glasses. Referring to FIG. 4A , an AR application is installed on the mobile phone 100 , and the AR application is used to manage the virtual space of the AR glasses 200 . In response to the user's click operation on the application icon of the AR application, the mobile phone 100 starts the AR application and displays the touchpad interface 101 . After the AR glasses 200 display the virtual space, the user can use an AR application to implement human-computer interaction with the virtual space of the AR glasses 200 . Optionally, the user can control the pointing position of the rays in the virtual space by moving the position and orientation of the mobile phone 100 to achieve human-computer interaction with objects displayed in the virtual space. Optionally, the touch panel interface 101 is provided with various virtual keys, and the user can realize human-computer interaction with objects displayed in the virtual space by operating the virtual keys. Exemplarily, the touchpad interface 101 includes a touch area 102. The user can slide up, down, left, or right (perform a sliding gesture) in the touch area 102, and the user can slide up, down, left, or right accordingly. Move the ray in the virtual space to the right; you can also perform clicks, long presses and other operations (perform click gestures, long press gestures) in the touch area 102 to achieve human-computer interaction with objects displayed in the virtual space.

For example, referring to FIG. 4A , the user can slide up, down, left or right in the touch area 102 to move the ray in the virtual space so that the ray moves to the desktop icon of the short video application on the desktop interface; then The user can perform a click operation in the touch area 102 to select the desktop icon of the short video application. In response to the click operation, the head-mounted display device 200 displays the application interface of the short video application in the virtual space.

Generally speaking, by inputting gestures on the electronic device 100, human-computer interaction with the virtual space of the head-mounted display device 200 is realized, which is convenient for the user and provides a better user experience. For example, in the above implementation, sliding gestures, click gestures, long press gestures, etc. are performed in the touch area 102. During the human-computer interaction process, in addition to the above operations, the user also needs to perform operations such as screenshots, screen recordings, and returning to the previous level in the virtual space of the head-mounted display device 200 .

Electronic devices with touch screens, such as mobile phones, support a wide range of gesture operations. For example, a screenshot of the mobile phone display interface can be achieved by performing a multi-finger sliding gesture on the touch screen; for example, a screenshot of the mobile phone display interface can be achieved by performing a double-click gesture on the touch screen; for example, a left sliding gesture can be performed on the touch screen to capture the display of the mobile phone. The interface returns to the previous level and so on.

In the prior art, in order to avoid conflicts with the system gestures of the electronic device 100 (for example, the gesture of sliding down with multiple fingers to take a screenshot, the gesture of double-clicking the knuckle to take a screenshot, the gesture of sliding left to return to the previous level, etc.), when the user When operations such as taking screenshots, recording screens, and returning to the previous level need to be performed in the virtual space of the head-mounted display device 200 , gestures are not supported on the electronic device 100 , but can be implemented through virtual buttons or other methods.

For example, in one implementation, the user clicks virtual buttons on the electronic device 100 to take screenshots, record screens, return to the previous level, and other operations in the virtual space of the head-mounted display device 200 . For example, referring to FIG. 4B, the touch panel interface 101 of the electronic device 100 includes a "Home" button 103. The user can click the "Home" button 103 to switch the virtual space display interface to a desktop interface. The touch panel interface 101 includes a "return" button 104, and the user can click the "return" button 104 to return the display interface of the virtual space to the previous level. The touch panel interface 101 includes a "Screenshot" button 105, and the user can click the "Screenshot" button 105 to take a screenshot of the display interface of the virtual space. The touch panel interface 101 includes a "record screen" button 106, and the user can click the "record screen" button 106 to record the screen of the display interface of the virtual space. Use the virtual buttons on the electronic device 100 to perform operations such as taking screenshots, recording screens, and returning to the previous level in the virtual space. The user needs to move his eyes away from the head-mounted display device 200 and move to the touch panel interface of the electronic device 100 , to find the corresponding virtual key. During the line of sight jump, since the eyes need to adjust the focus to see the two devices clearly, a strong sense of discomfort will occur and the user experience will be poor.

Embodiments of the present application provide a human-computer interaction method that, when an electronic device is used as an input device for a head-mounted display device, supports the user to perform system gestures on the touch screen of the electronic device to realize the display of the virtual space of the head-mounted display device. Use the interface to take screenshots, record screens, return to the previous level and other operations. It simplifies the user's operation process and improves the user's experience.

Below, taking the electronic device as a mobile phone and the head-mounted display device as AR glasses as an example, the human-computer interaction method provided by the embodiment of the present application will be introduced in detail with reference to the accompanying drawings.

There is an AR application installed on the mobile phone, and the user can start the AR application on the mobile phone to manage the virtual space of the AR glasses. When the mobile phone starts an AR application and displays the application interface of the AR application, if it receives the user's system gestures on the mobile phone's touch screen, such as screenshot gestures, screen recording gestures, return to the previous level gestures, etc., these system gestures will act on the AR glasses. Virtual space, not mobile phones. Specifically, after the mobile phone starts the AR application, the application interface of the AR application is displayed. If the user's screenshot gesture on the touch screen of the mobile phone is received, the display interface of the mobile phone is not taken, but the display interface of the virtual space of the AR glasses is taken. If the user's screen recording gesture on the touch screen of the mobile phone is received, the display interface of the mobile phone will not be recorded, but the display interface of the virtual space of the AR glasses will be recorded. If the user's return to the previous level gesture on the touch screen of the mobile phone is received, the display interface of the mobile phone is not returned to the previous level, but the display interface of the virtual space of the AR glasses is returned to the previous level. In this way, when users watch the screen in the virtual space, they only need to make system gestures on the touch screen of the mobile phone to take screenshots, record the screen, return to the previous level and other operations on the display interface of the virtual space of the AR glasses. The user's eyes do not need to leave the virtual space, avoiding the jump between the virtual space and the mobile phone screen. Moreover, gesture operation is convenient and fast, and the user operation cost is low and the user experience is good.

When the phone exits the AR application, or the AR application retreats to the background, the phone will no longer display the application interface of the AR application. For example, the mobile phone can display the desktop interface, or the mobile phone can display the application interface of other applications (such as the application interface of a video application, the application interface of a chat application, etc.). If the phone receives the user's system gestures on the touch screen of the phone, such as screenshot gestures, screen recording gestures, return to the previous level gestures, etc., these system gestures will act on the phone. Specifically, when the mobile phone does not display the application interface of the AR application, if a screenshot gesture of the user on the touch screen of the mobile phone is received, the display interface of the mobile phone is screenshotted. If the user's screen recording gesture on the touch screen of the mobile phone is received, the display interface of the mobile phone is recorded. If the user's return to the previous level gesture on the touch screen of the mobile phone is received, the display interface of the mobile phone is returned to the previous level. In this way, the user can make system gestures on the touch screen of the mobile phone to easily take screenshots, record the screen, return to the previous level and other operations on the mobile phone display interface.

Figures 5-10 show schematic diagrams of some scenario examples in which the mobile phone receives system gestures. It should be noted that the specific gestures shown in Figures 5 to 10 are only examples in corresponding scenarios and are used to schematically illustrate system gestures. In other implementations, system gestures such as screenshot gestures, screen recording gestures, and return to the previous level gestures may be in other specific forms. For example, the screenshot gesture can be sliding down with multiple fingers on the screen, sliding up with multiple fingers, or double-clicking the screen with one knuckle, double-clicking the screen with two knuckles, etc. For example, screen recording gestures can be swiping up and down on the screen with multiple fingers, double-clicking the screen with one knuckle, double-clicking the screen with two knuckles, etc. Understandably, screenshot gestures are different from screen recording gestures. For example, the gesture to return to the previous level can be a single finger sliding from the left side of the screen to the right, a single finger sliding from the right side of the screen to the left, or a finger sliding inward from both sides of the screen, etc.

Taking the screenshot gesture of sliding down with multiple fingers on the screen as an example, Figure 5 shows a schematic diagram of a scene example in which a screenshot gesture is received when the mobile phone displays the application interface of an AR application.

As shown in Figure 5, after starting the AR application, the mobile phone 100 displays the touch panel interface 101; the virtual space of the AR glasses 200 displays the application interface of the short video application. The user makes a screenshot gesture on the touch screen of the mobile phone 100; for example, as shown in Figure 5, the user uses three fingers to slide down on the screen of the mobile phone 100. After receiving the screenshot gesture, the mobile phone 100 triggers the generation of a screenshot of the display interface of the virtual space of the AR glasses 200 . For example, a screenshot is taken of the application interface of the short video application and a screenshot image is generated. Furthermore, the mobile phone 100 can also save the generated screenshot image.

In one implementation, after receiving the screenshot gesture, the mobile phone 100 determines that the current display interface of the mobile phone 100 is an application interface of an AR application (such as the touchpad interface 101), and the mobile phone 100 obtains an image of the display interface of the virtual space of the AR glasses 200, And take a screenshot of the image to generate a screenshot of the display interface of the virtual space.

In another implementation manner, after receiving the screenshot gesture, the mobile phone 100 determines that the current display interface of the mobile phone 100 is an application interface of an AR application (such as the touchpad interface 101). The mobile phone 100 notifies the AR glasses 200 to take a screenshot of the image of the display interface of the virtual space. In one example, the mobile phone 100 generates a screenshot event according to the screenshot gesture, and distributes the screenshot event to the AR glasses 200 . In another example, the mobile phone 100 sends a first instruction message to the AR glasses 200 , and the first instruction message is used to instruct the AR glasses 200 to take a screenshot of the display interface of the virtual space. After receiving the screenshot event or receiving the first instruction message, the AR glasses 200 take a screenshot of the image of the display interface of the virtual space and generate a screenshot of the display interface of the virtual space. The AR glasses 200 can save the generated screenshot image; further, the AR glasses 200 can also send the generated screenshot image to the mobile phone 100, and the mobile phone 100 can also save the generated screenshot image.

In this scenario, the AR application on the mobile phone is running in the foreground, and the mobile phone displays the application interface of the AR application. The mobile phone receives the user's screenshot gesture and takes a screenshot of the display interface of the virtual space of the AR glasses.

Taking the screenshot gesture of sliding down with multiple fingers on the screen as an example, Figure 6 shows a schematic diagram of a scene example in which the screenshot gesture is received when the mobile phone does not display the application interface of the AR application.

As shown in Figure 6, the mobile phone 100 does not start the AR application, and the mobile phone 100 displays the desktop interface. It is understandable that the mobile phone 100 can also display application interfaces of other applications, such as application interfaces of video applications, application interfaces of chat applications, etc. The virtual space of the AR glasses 200 displays the application interface of the video application. The user makes a screenshot gesture on the touch screen of the mobile phone 100; for example, as shown in Figure 6, the user slides down using three fingers. After receiving the screenshot gesture, the mobile phone 100 generates a screenshot of the mobile phone's display interface, for example, generates a screenshot of the mobile phone's desktop interface. Further, the mobile phone 100 can save the generated screenshot image.

In this scenario, the AR application is not running on the phone or the AR application is running in the background, and the application interface of the AR application is not displayed on the phone. The mobile phone receives the user's screenshot gesture and takes a screenshot of the display interface of the mobile phone.

Taking the screen recording gesture of sliding upwards with multiple fingers on the screen as an example, Figure 7 shows a schematic diagram of a scene example in which the screen recording gesture is received when the mobile phone displays the application interface of the AR application.

As shown in Figure 7, after starting the AR application, the mobile phone 100 displays the touch panel interface 101; the virtual space of the AR glasses 200 displays the application interface of the short video application. The user makes a screen recording gesture on the touch screen of the mobile phone 100; for example, as shown in Figure 7, the user uses three fingers to slide upward on the screen of the mobile phone 100. After receiving the screen recording gesture, the mobile phone 100 triggers screen recording on the display interface of the virtual space of the AR glasses 200 and generates a screen recording file. For example, the application interface of the short video application is screen-recorded and a screen-recording file is generated. Furthermore, the mobile phone 100 can also save the generated screen recording file.

In one implementation, after receiving the screen recording gesture, the mobile phone 100 determines that the current display interface of the mobile phone 100 is an application interface of an AR application (such as the touchpad interface 101), and the mobile phone 100 records the display interface of the virtual space of the AR glasses 200. screen and generate a screen recording file.

In another implementation manner, after receiving the screen recording gesture, the mobile phone 100 determines that the current display interface of the mobile phone 100 is an application interface of an AR application (such as the touchpad interface 101). The mobile phone 100 notifies the AR glasses 200 to record the screen of the display interface of the virtual space. In one example, the mobile phone 100 generates a screen recording event according to the screen recording gesture, and distributes the screen recording event to the AR glasses 200 . In another example, the mobile phone 100 sends a second instruction message to the AR glasses 200 , and the second instruction message is used to instruct the AR glasses 200 to record the screen of the display interface of the virtual space. After receiving the screen recording event or receiving the second instruction message, the AR glasses 200 record the screen of the display interface of the virtual space and generate a screen recording file of the display interface of the virtual space. The AR glasses 200 can save the generated screen recording file; further, the AR glasses 200 can also send the generated screen recording file to the mobile phone 100, and the mobile phone 100 can also save the screen recording file.

In this scenario, the AR application on the mobile phone is running in the foreground, and the mobile phone displays the application interface of the AR application. The mobile phone receives the user's screen recording gesture and performs screen recording on the display interface of the virtual space of the AR glasses.

Taking the screen recording gesture of sliding upwards with multiple fingers on the screen as an example, Figure 8 shows a schematic diagram of a scene example in which the screen recording gesture is received when the mobile phone does not display the application interface of the AR application.

As shown in Figure 8, the mobile phone 100 does not start the AR application, but the mobile phone 100 starts the video call application and displays the application interface of the video call application. It is understandable that the mobile phone 100 can also display application interfaces of other applications. The virtual space of the AR glasses 200 displays the application interface of the video application. The user makes a screen recording gesture on the touch screen of the mobile phone 100; for example, as shown in Figure 8, the user uses three fingers to slide upward on the screen of the mobile phone 100. After receiving the screen recording gesture, the mobile phone 100 starts recording the screen of the mobile phone display interface (the application interface of the video call application), and generates a screen recording file of the mobile phone display interface. For example, a screen recording file of the application interface of the video call application is generated. Further, the mobile phone 100 can save the generated screen recording file.

In this scenario, the AR application is not running on the phone or the AR application is running in the background, and the application interface of the AR application is not displayed on the phone. The mobile phone receives the user's screen recording gesture and performs screen recording on the display interface of the mobile phone.

Taking the gesture of returning to the previous level as an example of sliding one finger from the left side of the screen to the right, Figure 9 shows a schematic diagram of a scenario in which the gesture of returning to the previous level is received when the mobile phone displays the application interface of an AR application.

As shown in Figure 9, after starting the AR application, the mobile phone 100 displays the touch panel interface 101; the virtual space of the AR glasses 200 displays the application interface of the short video application. The user makes a gesture of returning to the previous level on the touch screen of the mobile phone 100; for example, as shown in Figure 9, the user uses a single finger to slide from the left edge to the right on the screen of the mobile phone 100. After receiving the gesture of returning to the previous level, the mobile phone 100 triggers the virtual space of the AR glasses 200 to return to the previous level display interface. Exemplarily, in response to the mobile phone 100 receiving the return to the previous level gesture, the virtual space of the AR glasses 200 displays the desktop interface.

In one implementation, after receiving the gesture of returning to the previous level, the mobile phone 100 determines that the current display interface of the mobile phone 100 is the application interface of the AR application (such as the touchpad interface 101), and the mobile phone 100 determines the current display interface of the virtual space according to the AR glasses 200 The display data of the upper-level display interface of the display interface is generated, and the display data of the upper-level display interface is sent to the AR glasses 200 . The AR glasses 200 display in the virtual space according to the display data of the upper-level display interface.

In another implementation manner, after receiving the return to the previous level gesture, the mobile phone 100 determines that the current display interface of the mobile phone 100 is an application interface of an AR application (such as the touchpad interface 101). The mobile phone 100 notifies the AR glasses 200 to return to the previous level display interface. In one example, the mobile phone 100 generates a return to the previous level event according to the return to the previous level gesture, and distributes the return to the previous level event to the AR glasses 200 . In another example, the mobile phone 100 sends a third instruction message to the AR glasses 200 , and the third instruction message is used to instruct the AR glasses 200 to return to the previous level display interface. After receiving the return to the upper level event or receiving the third instruction message, the AR glasses 200 display the upper level display interface of the current display interface.

In this scenario, the AR application on the mobile phone is running in the foreground, and the mobile phone displays the application interface of the AR application. The mobile phone receives the user's return gesture to the previous level, and the display interface of the virtual space of the AR glasses returns to the previous level.

Taking the gesture of returning to the previous level as an example of sliding one finger from the left side of the screen to the right, Figure 10 shows a schematic diagram of a scenario in which the gesture of returning to the previous level is received when the mobile phone does not display the application interface of the AR application.

As shown in Figure 10, the mobile phone 100 does not start the AR application, but the mobile phone 100 starts the chat application and displays the application interface of the chat application. The virtual space of the AR glasses 200 displays the application interface of the video application. The user makes a gesture of returning to the previous level on the touch screen of the mobile phone 100; for example, as shown in Figure 10, the user uses a single finger to slide from the left edge to the right on the screen of the mobile phone 100. After the mobile phone 100 receives the gesture of returning to the previous level, the mobile phone 100 displays the previous level display interface of the current interface. For example, the current interface of the mobile phone is the application interface of the chat application, and the upper-level display interface of the application interface of the chat application is the desktop interface. In response to receiving the return to previous level gesture, the mobile phone 100 displays the desktop interface.

In this scenario, the AR application is not running on the phone or the AR application is running in the background, and the application interface of the AR application is not displayed on the phone. The mobile phone receives the user's return to the previous level gesture and displays the previous level display interface of the current display interface.

The embodiment of the present application provides a human-computer interaction method that supports making system gestures on electronic devices (such as mobile phones). The electronic device determines whether the electronic device should perform the action corresponding to the system gesture or the head-mounted display device (such as AR glasses) should perform the action corresponding to the system gesture by determining whether the first application is running in the foreground.

For example, FIG. 11 shows a schematic flowchart of the human-computer interaction method provided by the embodiment of the present application. As shown in Figure 11, the electronic device and the head-mounted display device are communicated and connected. The electronic device receives the user's first gesture, such as a screenshot gesture, a screen recording gesture, or a return gesture to the previous level. The electronic device determines whether a first application (such as an AR application) is running in the foreground. The first application is used to manage the virtual space of the head-mounted display device. In one implementation, if the electronic device displays the application interface of the first application, it is determined that the first application is running in the foreground; if the electronic device does not display the application interface of the first application, it is determined that the first application is not running in the foreground.

If the electronic device determines that the first application is running in the foreground, the head-mounted display device (such as AR glasses) performs an action corresponding to the system gesture. For example, the scenarios are shown in Figure 5, Figure 7 and Figure 9.

If the electronic device determines that the first application is not running in the foreground, the electronic device (such as a mobile phone) performs an action corresponding to the system gesture. For example, the scenarios are shown in Figure 6, Figure 8 and Figure 10.

In this method, after receiving the system gesture, the electronic device distinguishes between different situations and triggers the electronic device itself to perform the action corresponding to the system gesture, or triggers the head-mounted display device to perform the action corresponding to the system gesture. Supports users to perform system gestures on the touch screen of electronic devices to enable operations such as taking screenshots, recording screens, and returning to the previous level on the display interface of the virtual space of the head-mounted display device. In this way, when the user is watching the screen in the virtual space, he only needs to make system gestures on the touch screen of the mobile phone to take screenshots, record the screen, return to the previous level and other operations on the virtual space display interface of the head-mounted display device. The user's eyes do not need to leave the virtual space, avoiding the jump between the virtual space and the mobile phone screen. Moreover, gesture operation is convenient and fast, and the user operation cost is low and the user experience is good.

For example, FIG. 12 and FIG. 13 show a detailed flowchart of the human-computer interaction method provided by the embodiment of the present application.

As shown in FIG. 12 , the operating system of the mobile phone 100 is installed with application 1 and application 2. The application interface of application 1 is displayed on the mobile phone 100 , and the application interface of application 2 is displayed in the virtual space of the AR glasses 200 . For example, application 1 is a mobile phone desktop application, and application 2 is a video application. After application 1 starts running, the activity (Activity) corresponding to application 1 is generated. After application 2 starts running, the activity (Activity) corresponding to application 2 is generated. It can be understood that an application can include one or more interfaces, and each interface corresponds to an Activity; an Activity includes one or more layers. Each layer contains one or more elements (controls). The framework layer of the operating system includes window management service (windowmanager service, WMS), display management unit, surface synthesis (SurfaceFlinger) module, activity management service (Activity manager service, AMS), etc. WMS is used for window management (for example, adding windows, deleting windows, modifying windows, etc.). When application 1 is started, WMS creates window 1 corresponding to application 1; when application 2 is started, WMS creates window 2 corresponding to application 2. The display management unit creates a corresponding display (Display) for each window and establishes a one-to-one correspondence between the window and the Display. The surface synthesis (SurfaceFlinger) module is used to obtain the display data of each application interface from WMS (for example, the number of layers included in the interface, the display elements of each layer). SurfaceFlinger determines the synthesis instructions corresponding to each display data (for example, OPENGL ES instructions, HWComposer instructions) based on the display data of the application interface. The graphics rendering synthesis component uses synthesis instructions to perform graphics synthesis on the display data to generate images of each layer of the application interface; it also synthesizes and renders the images of each layer to generate an interface corresponding to the Activity. Further, the graphics rendering synthesis component generates the interfaces of different applications to corresponding displays, and each display corresponds to one screen.

In one example, SurfaceFlinger obtains the display data 1 of the Activity of application 1 from WMS. The graphics rendering synthesis component uses synthesis instructions to perform graphic synthesis on the display data 1 to generate images of each layer; and synthesizes and renders the images of each layer. , generate the interface of application 1 to display 1 (display1). The interface saved on the display 1 is used for display on the screen of the mobile phone 100 . In this way, the screen of the mobile phone 100 displays the application interface of application 1; for example, as shown in FIG. 4A, the mobile phone 100 displays the desktop interface. SurfaceFlinger obtains the display data 2 of the Activity of application 2 from WMS. The graphics rendering synthesis component uses the synthesis instructions to perform graphic synthesis on the display data 2 to generate images of each layer; it synthesizes and renders the images of each layer to generate the interface of application 2. Go to display 2 (display2). The interface saved on display 2 is used for display in the virtual space of AR glasses 200 . Further, the mobile phone 100 sends the interface information of Application 2 to the AR glasses 200, so that the AR glasses 200 can display the application interface of Application 2 in the virtual space.

The mobile phone 100 receives a system gesture input by the user (such as a screenshot gesture, a screen recording gesture, or a return to the previous level gesture, etc.). The mobile phone determines that the desktop application is running in the foreground and the AR application is not running in the foreground. It is determined that the mobile phone performs the action corresponding to the system gesture. Referring to Figure 12, in response to receiving the system gesture, the mobile phone takes a screenshot or recording of the application interface of Application 1 or returns to the previous level operation.

In other embodiments, the operating system of the mobile phone 100 is also installed with application 3. Application 3 is used to manage the virtual space of the AR glasses 200, and the application interface of application 3 is displayed on the mobile phone 100. For example, application 3 is an AR application. As shown in Figure 13, after application 3 starts running, the activity (Activity) corresponding to application 3 is generated. WMS creates window 3 corresponding to application 3. SurfaceFlinger obtains the display data 3 of the Activity of application 3 from WMS. The graphics rendering synthesis component uses synthesis instructions to perform graphic synthesis on the display data 3 to generate images of each layer; it synthesizes and renders the images of each layer to generate the interface of application 3. to display1(display1). The interface saved on the display 1 is used for display on the screen of the mobile phone 100 . In this way, the screen of the mobile phone 100 displays the application interface of application 3; for example, as shown in FIG. 4A , after the mobile phone 100 starts the AR application, the touch panel interface 101 is displayed.

The mobile phone 100 receives a system gesture input by the user (such as a screenshot gesture, a screen recording gesture, or a return to the previous level gesture, etc.). The mobile phone determines that the AR application is running in the foreground and determines that the AR glasses perform actions corresponding to the system gestures. Referring to Figure 13, in response to receiving the system gesture, screenshot or record the application interface of Application 2 or return to the previous level operation.

In one implementation of the prior art, a virtual button is provided on the desktop interface of the virtual space. The user can move the ray position and click the virtual button on the desktop interface to perform screenshots, screen recordings and other operations in the virtual space. However, the operation steps of this operation method are relatively cumbersome. For example, when the user is viewing the display interface of other applications, he needs to exit the currently displayed application interface, enter the desktop interface, and then click the virtual button on the desktop interface to take a screenshot. , screen recording and other operations. The user experience is poor.

Embodiments of the present application also provide a human-computer interaction method. When a user inputs a first operation (such as knuckle tapping) on the application interface of an AR application on an electronic device, in response to the first operation, the head-mounted display device A shortcut operation panel pops up in the virtual space. The shortcut operation panel is equipped with virtual buttons such as screenshots, screen recordings, return to the previous level, and return to the home page. Users can move the ray position and click the virtual buttons on the shortcut operation panel to take screenshots of the virtual space display interface, record the screen, return to the previous level, return to the home page, and other operations. When the user is watching the screen in the virtual space, he only needs to input the first operation (such as knuckle tapping) on the touch screen of the mobile phone, and then he can use the virtual buttons in the virtual space to take screenshots, record the screen, etc. of the display interface of the virtual space. Return to the previous level and other operations. The user's line of sight does not need to leave the virtual space, avoiding the poor experience caused by the line of sight jumping between the virtual space and the mobile phone screen. Moreover, the virtual keys are set on the pop-up shortcut operation panel and do not depend on the display interface of the virtual space. When the user is using a specific application, there is no need to exit the application and enter the desktop interface for operation. The operation is fast, the user's operation cost is low and the user experience is good.

Taking the electronic device as a mobile phone and the head-mounted display device as AR glasses as an example, the human-computer interaction method provided by the embodiment of the present application will be introduced in detail below.

For example, as shown in Figure 14, the mobile phone 100 and the AR glasses 200 are connected through communication. The mobile phone 100 starts the AR application and displays the touch panel interface 101; the virtual space of the AR glasses 200 displays the application interface 301 of the short video application. The user can wear the AR glasses 200 to view the display interface in the virtual space.

The user can input a preset first operation on the touch panel interface 101 of the mobile phone 100 . In one example, the touch panel interface 101 includes a touch area 102, and the user can input a first operation in the touch area 102. For example, the first operation may include single-finger tapping, multi-finger tapping, knuckle tapping, single-finger circling, etc. It can be understood that the first operation may also include other forms, which are not limited by the embodiments of the present application.

Referring to Figure 14, take the first operation as knuckle tapping as an example. The mobile phone 100 receives the user's knuckle tapping action in the touch area 102. In response to the knuckle tapping action, the shortcut operation panel 302 pops up in the virtual space of the AR glasses 200. The shortcut operation panel 302 is provided with virtual buttons. For example, as shown in Figure 14, the shortcut operation panel 302 includes a "Home" button 303, a "Return to the previous level" button 304, a "Screenshot" button 305, and a "Screen Recording" button 306. Among them, the "Home" button 303 is used to return to the desktop interface of the virtual space, the "Return to the previous level" button 304 is used to return to the previous level display interface of the current display interface, and the "Screenshot" button 305 is used to return to the display interface of the virtual space. To take a screenshot, the "record screen" button 306 is used to record the screen of the display interface of the virtual space. Optionally, the shortcut operation panel 302 also includes a "Close" button 307 for closing the shortcut operation panel 302.

For example, as shown in Figure 15A, the user can select the "Home" button 303; for example, the user slides up, down, left or right in the touch area 102 displayed on the mobile phone 100 to move the ray in the virtual space. Go to the "Home" button 303, and then click once in the touch area 102 to select the "Home" button 303. In response to the user's selection operation of the "Home" button 303, the virtual space displays the desktop interface 310 (home page).

In one example, the user can select the "Return to Previous Level" button 304; in response to the user's selection operation of the "Return to Previous Level" button 304, the virtual space displays the upper level display interface of the application interface 301 of the short video application. . For example, the upper-level display interface of the application interface 301 of the short video application is the desktop interface 310. In response to the user's selection operation of the "return to previous level" button 304, the virtual space displays the desktop interface 310.

For example, as shown in Figure 15B, the user can select the "Screenshot" button 305; for example, the user slides up, down, left or right in the touch area 102 displayed on the mobile phone 100 to move the ray in the virtual space. Go to the "Screenshot" button 305, and then click once in the touch area 102 to select the "Screenshot" button 305. In response to the user's selection operation of the "Screenshot" button 305, a screenshot is taken of the display interface of the virtual space (the application interface 301 of the short video application) to generate a screenshot image.

For example, as shown in Figure 15C, the user can select the "record screen" button 306; for example, the user slides up, down, left or right in the touch area 102 displayed on the mobile phone 100 to move the virtual space. Ray to the "Record Screen" button 306, and then click once in the touch area 102 to select the "Record Screen" button 306. In response to the user's selection operation on the "record screen" button 306, the screen is recorded on the display interface of the virtual space and a screen recording file is generated.

For example, as shown in Figure 15D, the user can select the "Close" button 307; in response to the user's selection operation of the "Close" button 307, the virtual space stops displaying the shortcut operation panel 302.

It should be noted that the shortcut operation panel 302 shown in FIGS. 15A to 15D is only an example. In specific implementation, the shortcut operation panel can be in other forms. For example, the shape, size, and position of the shortcut operation panel can be set according to specific circumstances.

In one example, as shown in FIG. 16A , the shortcut operation panel 302 can be superimposed on the current display interface of the virtual space (the application interface 301 of the short video application). Optionally, the shortcut operation panel 302 is provided on one side of the current display interface of the virtual space.

In another example, as shown in FIG. 16B , the shortcut operation panel 302 can be displayed in a blank area in the virtual space, and is located on the same layer as the application interface 301 of the short video application.

In another example, as shown in Figure 16C, the frame of the shortcut operation panel 302 may not be displayed in the virtual space, but the "Home" button 303, the "Return to the previous level" button 304, and the "Screenshot" button 305 may be directly displayed. , "record screen" button 306, etc.

In the human-computer interaction method provided by the embodiment of the present application, in response to the first operation input by the user on the application interface of the AR application on the electronic device, the virtual space of the head-mounted display device pops up to take a screenshot, record the screen, and return to the previous level. , return to homepage and other virtual keys. These virtual buttons do not depend on the current display interface of the virtual space and can pop up on any display interface. In this way, when users need to take screenshots, record screens, return to the previous level, return to the homepage, etc., they do not need to exit the current application, and the operation is simple and fast.

For example, FIG. 17 shows a schematic flowchart of the human-computer interaction method provided by the embodiment of the present application. As shown in Figure 17, the method includes:

S401. Communication connection between the electronic device and the head-mounted display device.

The electronic device can serve as an input device for the head-mounted display device.

S402. The electronic device starts the first application and displays the application interface of the first application.

A first application (such as an AR application) is installed on the electronic device for managing the virtual space of the head-mounted display device. For example, the user can click the application icon of the first application on the desktop interface of the electronic device to start the first application. For example, as shown in Figure 14, the mobile phone 100 displays the touchpad interface 101 of the AR application.

S403. The electronic device receives the user's first operation on the application interface of the first application.

In one example, the touch panel interface 101 includes a touch area 102, and the user can input a first operation in the touch area 102. For example, the first operation may include single-finger tapping, multi-finger tapping, knuckle tapping, single-finger circling, etc. It can be understood that the first operation may also include other forms, which are not limited by the embodiments of the present application.

S404. In response to the first operation, the virtual space of the head-mounted display device displays at least one of a screenshot virtual button, a screen recording virtual button, a return to the previous level virtual button, and a return to homepage virtual button.

In one implementation, after receiving the first operation, the electronic device generates display data of the virtual keys and sends the display data of the virtual keys to the head-mounted display device; the head-mounted display device responds to the received data of the virtual keys. Display data, display screenshot virtual buttons, screen recording virtual buttons, return to the previous level virtual button, return to homepage virtual button, etc. in the virtual space.

In another implementation manner, after receiving the first operation, the electronic device generates a corresponding first event. The electronic device distributes the first event to the head mounted display device. The head-mounted display device receives the first event, generates display data of the virtual buttons, and displays in the virtual space virtual buttons for taking screenshots, virtual buttons for recording screens, virtual buttons for returning to the previous level, and virtual buttons for returning to the home page based on the display data of the virtual buttons. wait.

In one example, the virtual space of the head-mounted display device displays a shortcut operation panel. The shortcut operation panel includes virtual buttons for taking screenshots, virtual buttons for recording screens, virtual buttons for returning to the previous level, virtual buttons for returning to the homepage, etc. Illustratively, as shown in Figure 14, Figure 16A and Figure 16B.

In another example, virtual keys for taking screenshots, virtual keys for recording screens, virtual keys for returning to the previous level, virtual keys for returning to homepage, etc. are directly displayed on the application interface in the virtual space. For example, as shown in Figure 16C, the application interface 301 of the short video application displays a "Home" button 303, a "Return to the previous level" button 304, a "Screenshot" button 305 and a "Screen Recording" button 306 in a superimposed manner.

It can be understood that, in order to implement the above functions, each of the above devices includes corresponding hardware structures and/or software modules for performing each function. Persons skilled in the art should easily realize that, in conjunction with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is performed by hardware or computer software driving the hardware depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of the present application.

Embodiments of the present application can divide the above-mentioned device into functional modules according to the above-mentioned method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or software function modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. In actual implementation, there may be other division methods.

In the case of using an integrated unit, FIG. 18 shows a possible structural diagram of the electronic device involved in the above embodiment. The electronic device 1800 includes: a processing unit 1801, a communication unit 1802, and a storage unit 1803. Among them, the processing unit 1801 is used to control and manage the actions of the electronic device 1800; the communication unit 1802 is used to support the communication between the electronic device 1800 and other network entities; the storage unit 1803 is used to save the instructions and data of the electronic device 1800. The above instructions It can be used to perform various steps in the corresponding embodiments of this application.

Of course, the unit modules in the above-mentioned electronic device 1800 include but are not limited to the above-mentioned processing unit 1801, communication unit 1802 and storage unit 1803. For example, the electronic device 1800 may further include a display unit and the like, and the display unit is used to display a user interface of the electronic device 1800 .

The processing unit 1801 may be a processor or a controller, such as a central processing unit (CPU), a digital signal processor (DSP), or an application-specific integrated circuit (ASIC). , field programmable gate array (fieldprogrammable gate array, FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The communication unit 1802 may be a transceiver, a transceiver circuit, or the like. The storage unit 1803 may be a memory.

The electronic device 1800 provided in the embodiment of the present application may be the electronic device 100 shown in FIG. 2 . The above-mentioned processors, memories, communication interfaces, etc. may be connected together, for example, through a bus. The processor calls the program code stored in the memory to execute each step in the above method embodiment.

An embodiment of the present application also provides a chip system. As shown in Figure 19, the chip system includes at least one processor 1901 and at least one interface circuit 1902. The processor 1901 and the interface circuit 1902 may be interconnected via wires. For example, interface circuitry 1902 may be used to receive signals from other devices, such as memory of a routing device. As another example, interface circuit 1902 may be used to send signals to other devices (eg, processor 1901). For example, the interface circuit 1902 can read instructions stored in the memory and send the instructions to the processor 1901. When the instructions are executed by the processor 1901, the electronic device can be caused to perform various steps in the above embodiments. Of course, the chip system may also include other discrete devices, which are not specifically limited in the embodiments of this application.

Embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores computer program code. When the above-mentioned processor executes the computer program code, the electronic device executes the method in the above-mentioned embodiment.

Embodiments of the present application also provide a computer program product. When the computer program product is run on a computer, it causes the computer to execute the method in the above embodiment.

Among them, the electronic device 1800, chip system, computer-readable storage medium or computer program product provided by the embodiment of the present application are all used to execute the corresponding method provided above. Therefore, the beneficial effects it can achieve can be referred to the above. The beneficial effects of the corresponding methods provided will not be described again here.

Through the above description of the embodiments, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be allocated as needed. It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, for example, multiple units or components may be The combination can either be integrated into another device, or some features can be omitted, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.

In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application are essentially or contribute to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium , including several instructions to cause a device (which can be a microcontroller, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage media include: U disk, mobile hard disk, ROM, magnetic disk or optical disk and other media that can store program codes.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be covered by the protection scope of the present application. . Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims (16)

1. A human-machine interaction method, characterized by being applied to an electronic device, the electronic device being connected to a head-mounted display device, the method comprising:

the electronic device starts a first application, wherein the first application is used for managing a virtual space of the head-mounted display device;

the electronic equipment receives a first gesture of a user on the first application display interface; the first gesture comprises a screen capturing gesture, a screen recording gesture or a gesture returning to the previous stage;

and responding to the first gesture, and executing an action corresponding to the first gesture by the electronic equipment aiming at a display interface of the virtual space.

2. The method according to claim 1, wherein the method further comprises:

the electronic equipment exits to display the first application display interface;

the electronic equipment receives a first gesture of a user on a second application display interface;

and responding to the first gesture on the second application display interface, and executing an action corresponding to the first gesture by the electronic equipment aiming at the second application display interface.

3. A method according to claim 1 or 2, characterized in that,

the screen capturing gesture comprises any one of multi-directional downward sliding, multi-directional upward sliding, single-finger joint double-click screen or double-finger joint double-click screen on the screen;

The screen recording gesture comprises any one of multi-directional downward sliding, multi-directional upward sliding, single-finger joint double-click screen or double-finger joint double-click screen on the screen; wherein the screen capture gesture is different from the screen capture gesture;

the return-to-previous gesture includes any one of a single finger sliding from the left side of the screen to the right, a single finger sliding from the right side of the screen to the left, or a finger sliding inward from both sides of the screen.

4. The method of any of claims 1-3, wherein the electronic device receives a first gesture of a user prior to the first application display interface, the method further comprising:

and the electronic equipment generates a display interface of the virtual space and sends the display interface to the head-mounted display equipment.

5. The method of claim 4, wherein the electronic device performing the action corresponding to the first gesture with respect to the display interface of the virtual space comprises:

and the electronic equipment performs screen capturing, screen recording or generates a higher-level display interface of the virtual space.

6. A method according to any one of claims 1-3, wherein the electronic device performing, for a display interface of the virtual space, an action corresponding to the first gesture, comprises:

And triggering the head-mounted display device to screen capture, record or generate a superior display interface of the virtual space by the electronic device.

7. A human-machine interaction method, characterized by being applied to an electronic device, the electronic device being connected to a head-mounted display device, the method comprising:

the electronic device starts a first application, wherein the first application is used for managing a virtual space of the head-mounted display device;

the electronic equipment receives a first operation of a user on the first application display interface;

and responding to the first operation, the electronic equipment triggers at least one of a virtual space display screen capturing virtual key, a screen recording virtual key, a last-stage virtual key and a homepage returning virtual key of the head-mounted display equipment.

8. The method of claim 7, wherein the electronic device triggering at least one of a virtual space display screen capture virtual key, a return to superior virtual key, and a return to home virtual key of the head mounted display device comprises:

the electronic equipment triggers the virtual space of the head-mounted display equipment to display a shortcut operation panel, and at least one of a screen capturing virtual key, a screen recording virtual key, a return-to-last-stage virtual key and a return-to-homepage virtual key is arranged on the shortcut operation panel.

9. The method of claim 8, wherein the shortcut panel is superimposed over an application interface of a second application displayed in the virtual space.

10. The method of claim 8, wherein the shortcut panel is displayed in a blank area of the virtual space display interface.

11. The method of claim 7, wherein at least one of the screen capture virtual key, the return to superior virtual key, and the return to home virtual key is superimposed over an application interface of a second application displayed in the virtual space.

12. The method of any of claims 7-11, wherein the first operation comprises: any one of a single finger tap, a multi-finger tap, a knuckle tap, or a single finger stroke.

13. The method of any of claims 7-12, wherein the electronic device, in response to the first operation, triggers at least one of a virtual space display screen capture virtual key, a return to superior virtual key, and a return to home virtual key of the head mounted display device, comprising:

In response to the first operation, the electronic device generates first display data, and sends the first display data to the head-mounted display device; the first display data is used for displaying at least one of a screen capturing virtual key, a screen recording virtual key, a return-to-last-stage virtual key and a return-to-homepage virtual key in the virtual space by the head-mounted display device.

14. The method of any of claims 7-12, wherein the electronic device, in response to the first operation, triggers at least one of a virtual space display screen capture virtual key, a return to superior virtual key, and a return to home virtual key of the head mounted display device, comprising:

responding to the first operation, and sending a first event corresponding to the first operation to the head-mounted display device by the electronic device; the first event is used for triggering the head-mounted display device to generate first display data, and the first display data is used for displaying at least one of a screen capturing virtual key, a screen recording virtual key, a return-to-last-stage virtual key and a return-to-home virtual key in a virtual space by the head-mounted display device.

15. An electronic device, the electronic device comprising: a processor, a touch screen, and a memory, the processor, touch screen being coupled to the memory; the memory is used for storing computer program codes; the computer program code comprising computer instructions which, when executed by the processor, cause the electronic device to perform the method of any of claims 1-14.

16. A computer readable storage medium comprising computer instructions which, when run on an electronic device, cause the electronic device to perform the method of any of claims 1-14.