WO2018082162A1 - Function triggering method and device for virtual reality apparatus, and virtual reality apparatus - Google Patents

Abstract

A function triggering method and device for a virtual reality apparatus, and a virtual reality apparatus. The method comprises: controlling a first infrared camera to acquire a left-eye image, and controlling a second infrared camera to acquire a right-eye image (S110); identifying a left-eye action according to the left-eye image, and identifying a right-eye action according to the right-eye image (S120); determining whether the left-eye action and/or the right-eye action is a specified action completed within a set time period and corresponding to a current application (S130); and if yes, triggering the current application to implement a function corresponding to the specified action (S140). The present invention can reduce hardware costs and improve user experience.

Description

Function triggering method, device and virtual reality device for virtual reality device Technical field

The present invention relates to the field of virtual reality device technologies, and in particular, to a function triggering method, apparatus, and virtual reality device for a virtual reality device.

Background technique

The general mobile phone product video recording is achieved by clicking the photo or video button on the touch screen to achieve the corresponding function, or there are special shortcut keys to achieve the photo. In contrast, virtual reality devices generally do not have a touch screen. If you want to take a photo or video, one method is to add an additional touch pad or touch screen, and the other is to add a button specifically for taking pictures.

Summary of the invention

According to a first aspect of the present invention, a function triggering method for a virtual reality device is provided. The virtual reality device includes at least two infrared cameras, and the function triggering method includes:

Controlling the first infrared camera to collect the left eye image, and controlling the second infrared camera to collect the right eye image;

Identifying an action of the left eye according to the left eye image, and recognizing an action of the right eye according to the right eye image;

Determining whether the action of the left eye and/or the action of the right eye is a specified action corresponding to the current application completed within a set time, and if so, triggering the current application to perform a function corresponding to the specified action.

According to a second aspect of the present invention, a function triggering apparatus for a virtual reality device is provided, the virtual reality device comprising at least two infrared cameras, the function triggering device comprising:

a control module, configured to control the first infrared camera to collect the left eye image, and control the second infrared camera to collect the right eye image;

An identification module, configured to identify an action of the left eye according to the left eye image, according to the right eye The image identifies the action of the right eye;

a determining module, configured to determine whether the action of the left eye and/or the action of the right eye is a specified action corresponding to the current application completed within a set time; and

a triggering module, wherein the action of the left eye and/or the action of the right eye is a specified action corresponding to a current application completed within a set time, triggering the current application execution to correspond to the specified action The function.

According to a third aspect of the invention, there is provided a virtual reality device comprising the function triggering device according to the second aspect of the invention.

According to a fourth aspect of the present invention, a virtual reality device is provided, comprising at least two infrared cameras, a processor and a memory, the two infrared cameras respectively for collecting a left eye image and a right eye image, wherein the memory is used for And storing instructions for controlling the processor to perform the function triggering method according to the first aspect of the present invention.

According to a fifth aspect of the invention, there is provided a computer readable storage medium storing program code for performing the function triggering method according to the first aspect of the invention.

Other features and advantages of the present invention will become apparent from the Detailed Description of the <RTIgt;

DRAWINGS

The accompanying drawings, which are incorporated in FIG

1 is a flow chart of an embodiment of a function triggering method for a virtual reality device according to the present invention;

2 is a block schematic diagram of an implementation structure of a function triggering apparatus for a virtual reality device according to the present invention;

3 is a block schematic diagram of an implementation structure of a virtual reality device in accordance with the present invention.

detailed description

Various exemplary embodiments of the present invention will now be described in detail with reference to the drawings. It should be noted that the relative arrangements of the components and steps set forth in these embodiments, unless specifically stated otherwise, Numerical expressions and numerical values do not limit the scope of the invention.

The following description of the at least one exemplary embodiment is merely illustrative and is in no way

Techniques, methods and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but the techniques, methods and apparatus should be considered as part of the specification, where appropriate.

In all of the examples shown and discussed herein, any specific values are to be construed as illustrative only and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

It should be noted that similar reference numerals and letters indicate similar items in the following figures, and therefore, once an item is defined in one figure, it is not required to be further discussed in the subsequent figures.

In order to solve the problem that the hardware device triggers the corresponding function of the virtual reality device by increasing the hardware circuit, the hardware circuit occupies space, thereby increasing the hardware cost of processing the virtual reality device, and providing a virtual reality device. The function trigger method.

Wherein, since the virtual reality device is, for example, a head-mounted virtual reality device, the occlusion of the eye is equivalent to a nighttime state during the wearing of the user, so it is necessary to use a camera with strong nighttime image capturing capability to track the movement and position of the eye, and The infrared camera has strong night image capturing capability. Therefore, the virtual reality device includes at least two infrared cameras, and the infrared camera has low power supply, and the infrared rays used are respectively two kinds: near infrared rays or short-wave infrared rays, and the wavelength is 0.76 to 1.5 micrometers. The human body is deep, about 5 to 10 mm; far infrared or long-wave infrared, with a wavelength of 1.5 to 400 μm, is mostly absorbed by the superficial skin, and the penetration depth is less than 2 mm.

1 is a flow chart of an embodiment of a function triggering method for a virtual reality device in accordance with the present invention.

According to FIG. 1, the function triggering method comprises the following steps:

Step S110, controlling the first infrared camera to collect the left eye image, and controlling the second infrared camera to collect the right eye image.

Specifically, the first infrared camera is controlled to record the left eye, and the second infrared camera is controlled to record the right eye; or the first infrared camera is controlled to take a picture on the left eye, and the second infrared camera is controlled to take a picture on the right eye. Shooting of the first infrared camera and the second infrared camera The frequencies can be, for example, the same, all taken every 100 ms. Since both infrared cameras are used to track the user's eyes, the pixel resolution requirements of the two infrared cameras are relatively low, and the resolution can be, for example, 320*240.

In step S120, the motion of the left eye is recognized based on the left eye image, and the motion of the right eye is recognized based on the right eye image.

In the case where the resolution of the infrared camera is 320*240, the left eye image and the right eye image are all square. Since the infrared camera is only sensitive to black and white or shade color, the output data is also two-dimensional, and the human The eye structure is generally dark in the middle, and the surrounding is light-colored. Therefore, the infrared camera is used to track the eye. When the user looks at the front, if the two-dimensional data output by the infrared camera is mapped to the corresponding square image, respectively. On the above, it should be the dark infrared camera data in the middle, surrounded by infrared camera data representing the light color, when the eye rotates, the dark data on the image will also move in the same direction or in the opposite direction; When the eyes are closed, the eyes are blocked at this time, and there will be no more depths. The corresponding two-dimensional data of the infrared camera maps the image of a single color, thereby judging the state of the left and right eyes of the user and the left eye. And the position of the right eye, and then according to the position and state of the left and right eyes of the user at different times, the motion of the left eye of the user and the action of the right eye can be judged.

Therefore, step S120 may specifically include:

Identifying the position of the dark data in the left eye image to determine the state and position of the left eye, and then determining the motion of the left eye according to the state and position of the left eye at different times; identifying the position of the dark data in the image of the right eye to determine the right The state and position of the eye, and then determine the action of the right eye according to the state and position of the right eye at different times.

For example, at time t0, the state of the left eye is detected as blinking, and the state of the right eye is blinking; at time t1, the state of the left eye is detected as closed eyes, and the state of the right eye is closed eyes; at time t2, detection is performed. The state to the left eye is blinking, and the state of the right eye is blinking; at time t3, the state of the left eye is closed and the state of the right eye is closed; at time t4, the state of the left eye is detected as When the blinking or right eye state is blinking, it can be considered that the user's left eye and right eye motions are blinking.

For example, at time t0, it is detected that the position of the left eye map on the left eye image is the left side of the square area, and the position of the right eye maps the position on the right eye image as the left side of the square area; at time t1, the left eye is detected. The position of the map on the left eye image is above the square area, and the position map of the right eye The position on the image of the right eye is above the square area; at time t2, the position of the left eye map on the left eye image is detected as the right side of the square area, and the position of the right eye is mapped to the square area. Right side; at time t3, it is detected that the position of the left eye map on the left eye image is below the square area, the position of the right eye maps the position on the right eye image below the square area; at time t4, the left eye is detected The position mapped on the left eye image is the left side of the square area, and the position of the right eye maps the position on the right eye image to the left side of the square area, and it can be considered that the actions of the left eye and the right eye of the user are blinking eyes.

For example, when it is detected that the state of the left eye is closed eyes at times t0, t1, t2, t3, and t4, it is considered that the operation of the left eye of the user is closed.

The interval between the times t0, t1, t2, t3, and t4 is short, and the interval between the times t0 and t4 may be less than the set time.

In step S130, it is determined whether the action of the left eye and/or the action of the right eye is the designated action corresponding to the current application completed within the set time. If yes, step S140 is performed, and if no, step S110 is continued.

The specified action includes, for example, at least one of blinking, closed eyes, and blinking, or any combination. For example, in the current application, the specified action corresponding to triggering a certain function may be blinking the left eye and closing the eye of the right eye, or the left eye and the right eye may simultaneously turn the eye, or may be the left eye closed eye.

Further, since blinking is a normal physiological need of the human eye in a normal state, in order to avoid interference of the user's normal blinking on the detection result of the human eye motion, for example, in the case where the closed eye time in the normal blinking is 100 ms, the setting is specified. The blinking time of the blink of the action is 500ms, so that the interference of the normal blinking motion can be eliminated.

On this basis, the function triggering method further includes: presetting a specified action corresponding to different functions for different applications. Specifically, the application may be, for example, a video playing application, a shooting application, or other APP installed in a virtual reality device. For example, a specified action of blinking in the video playing application for the corresponding playing or pause function may be preset, and the shooting action is a designated action corresponding to the photographing function.

Step S140, triggering the current application to perform a function corresponding to the specified action.

In a specific embodiment of the present invention, the virtual reality device further includes a color camera, and the current application is a shooting application, for example, the specified action corresponding to the camera function in the shooting application is left. When both the eye and the right eye are closed and the shooting application is turned on, if both the left eye motion and the right eye motion are detected to be closed for 2 seconds, for example, the shooting application is triggered to perform the photographing function. Specifically, the color camera is controlled to capture the current preview screen.

Further, in order to enable the user to clearly know that the function corresponding to the specified action is performed, in a specific embodiment of the present invention, after performing step S140, the method further includes:

The alarm prompts the user.

The specific alarm prompting the user may include, for example, any one of vibration, sound generation, and music playing, or any combination. In this way, the user can be reminded that the function corresponding to the specified action is completed, thereby further improving the user experience.

In this way, the action of the left eye and/or the action of the right eye is tracked by the infrared camera, and the current application is triggered when the action of the left eye and/or the action of the right eye is the specified action corresponding to the current application completed within the set time. Performing functions corresponding to specified actions can reduce hardware costs and improve user experience.

Corresponding to the above function triggering method, the present invention also provides a function triggering device for a virtual reality device, the virtual reality device comprising at least two infrared cameras.

2 is a block schematic diagram of an implementation structure of a function triggering apparatus for a virtual reality device according to the present invention.

According to FIG. 2, the function triggering device 200 includes a control module 210, an identification module 220, a determining module 230, and a triggering module 240.

The control module 210 is configured to control the first infrared camera to collect the left eye image, and the second infrared camera to collect the right eye image.

The identification module 220 is configured to recognize the motion of the left eye according to the left eye image, and recognize the motion of the right eye according to the right eye image.

The determining module 230 is configured to determine whether the action of the left eye and/or the action of the right eye is a specified action corresponding to the current application that is completed within the set time.

When the triggering module 240 is used for the action of the left eye and/or the action of the right eye is a specified action corresponding to the current application that is completed within the set time, the current application is triggered to perform a function corresponding to the specified action.

In a specific embodiment of the present invention, the foregoing identification module is specifically configured to: identify a left eye The position of the dark data in the image is used to determine the state and position of the left eye, and then the motion of the left eye is determined according to the state and position of the left eye at different times; the position of the dark data in the image of the right eye is recognized to determine the state of the right eye and Position, and then determine the action of the right eye according to the state and position of the right eye at different times.

Wherein, the specified action includes at least one of blinking, closed eyes and blinking eyes or any combination.

Further, the function triggering device further includes an alarm module, configured to issue an alarm to prompt the user after the triggering module 240 triggers the current application to perform the function corresponding to the specified motion.

On this basis, the function triggering device further includes a setting module for presetting the specified actions corresponding to different functions for different applications.

The present invention also provides a virtual reality device. In one aspect, the virtual reality device includes the aforementioned function triggering device 200 for a virtual reality device.

3 is a block schematic diagram of an implementation structure of the virtual reality device in accordance with another aspect of the present invention.

According to FIG. 3, the virtual reality device 300 includes at least two infrared cameras, a memory 301 and a processor 302. The two infrared cameras are respectively used for acquiring a left eye image and a right eye image. The memory 301 is configured to store an instruction. The instructions are for controlling the processor 302 to operate to perform the above-described function triggering method for the virtual reality device.

The processor can be, for example, a central processing unit CPU, a microprocessor MCU, or the like. The memory includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a nonvolatile memory such as a hard disk, and the like.

In addition, as shown in FIG. 3, the virtual reality device 300 further includes an interface device 303, an input device 304, a display device 305, a communication device 306, a speaker 307, a microphone 308, and the like. Although a plurality of devices are illustrated in FIG. 3, the virtual reality device of the present invention may relate only to some of the devices, such as the processor 301, the memory 302, the display device 305, and the like.

The communication device 306 can be wired or wirelessly communicated, for example.

The interface device 303 includes, for example, a headphone jack, a USB interface, etc., such that the USB interface of the interface device 303 establishes a connection with other electronic devices such as a tracking light ball, a handle, etc. via a USB data line.

The input device 304 described above may include, for example, a touch screen, a button, and the like.

The display device 305 described above is, for example, a liquid crystal display, a touch display, or the like.

The virtual reality device of the present invention may be, for example, a head mounted virtual reality product.

The above embodiments mainly focus on the differences from the other embodiments, but it should be apparent to those skilled in the art that the above embodiments may be used alone or in combination with each other as needed.

The various embodiments in the present specification are described in a progressive manner, and the same or similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments, but the field It should be clear to the skilled person that the above embodiments can be used individually or in combination with each other as needed. In addition, for the device embodiment, since it corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the corresponding part of the method embodiment. The system embodiments described above are merely illustrative, and the modules illustrated as separate components may or may not be physically separate.

The invention can be an apparatus, method and/or computer program product. The computer program product can comprise a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement various aspects of the present invention.

The computer readable storage medium can be a tangible device that can hold and store the instructions used by the instruction execution device. The computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) Or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, for example, with instructions stored thereon A raised structure in the hole card or groove, and any suitable combination of the above. A computer readable storage medium as used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (eg, a light pulse through a fiber optic cable), or through a wire The electrical signal transmitted.

The computer readable program instructions described herein can be downloaded from a computer readable storage medium to various computing/processing devices, or via a network, such as the Internet, a local area network, a wide area network, and/or Or the wireless network is downloaded to an external computer or an external storage device. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in each computing/processing device .

Computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine related instructions, microcode, firmware instructions, state setting data, or in one or more programming languages. Source code or object code written in any combination, including object oriented programming languages such as Smalltalk, C++, etc., as well as conventional procedural programming languages such as the "C" language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer, partly on the remote computer, or entirely on the remote computer or server. carried out. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or wide area network (WAN), or can be connected to an external computer (eg, using an Internet service provider to access the Internet) connection). In some embodiments, the customized electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be customized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams can be implemented by computer readable program instructions.

The computer readable program instructions can be provided to a general purpose computer, a special purpose computer, or a processor of other programmable data processing apparatus to produce a machine such that when executed by a processor of a computer or other programmable data processing apparatus Means for implementing the functions/acts specified in one or more of the blocks of the flowcharts and/or block diagrams. The computer readable program instructions can also be stored in a computer readable storage medium that causes the computer, programmable data processing device, and/or other device to operate in a particular manner, thereby storing instructions The computer readable medium then includes an article of manufacture that includes instructions for implementing various aspects of the functions/acts recited in one or more of the flowcharts.

The computer readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device to perform a series of operational steps on a computer, other programmable data processing device or other device to produce a computer-implemented process. Thus, instructions executed on a computer, other programmable data processing apparatus, or other device implement the functions/acts recited in one or more of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the invention. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a portion of an instruction that includes one or more components for implementing the specified logical functions. Executable instructions. In some alternative implementations, the functions noted in the blocks may also occur in a different order than those illustrated in the drawings. For example, two consecutive blocks may be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, can be implemented in a dedicated hardware-based system that performs the specified function or function. Or it can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.

The embodiments of the present invention have been described above, and the foregoing description is illustrative, not limiting, and not limited to the disclosed embodiments. Numerous modifications and changes will be apparent to those skilled in the art without departing from the scope of the invention. The choice of terms used herein is intended to best explain the principles, practical applications, or technical improvements of the techniques in the <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; The scope of the invention is defined by the appended claims.

Claims (11)

A function triggering method for a virtual reality device, the virtual reality device comprising at least two infrared cameras, wherein the function triggering method comprises: Controlling the first infrared camera to collect the left eye image, and controlling the second infrared camera to collect the right eye image; Identifying an action of the left eye according to the left eye image, and recognizing an action of the right eye according to the right eye image; Determining whether the action of the left eye and/or the action of the right eye is a specified action corresponding to the current application completed within a set time, and if so, triggering the current application to perform a function corresponding to the specified action. The function triggering method according to claim 1, wherein the triggering the function of the current application to perform the function corresponding to the specified action further comprises: An alert is issued to alert the user. The function triggering method according to claim 1 or 2, wherein the action of recognizing the left eye according to the left eye image, and the action of recognizing the right eye according to the right eye image is specifically: Identifying a position of the dark data in the left eye image to determine a state and a position of the left eye, and determining an action of the left eye according to a state and a position of the left eye at different times; identifying the right eye The position of the dark data in the image is used to determine the state and position of the right eye, and the action of the right eye is determined according to the state and position of the right eye at different times. The function triggering method according to any one of claims 1 to 3, wherein the specified action includes at least any one or any combination of blinking, closed eyes, and blinking. A function triggering device for a virtual reality device, the virtual reality device comprising at least two infrared cameras, wherein the function triggering device comprises: a control module, configured to control the first infrared camera to collect the left eye image, and control the second infrared camera to collect the right eye image; An identification module, configured to identify an action of the left eye according to the left eye image, and identify an action of the right eye according to the right eye image; a determining module, configured to determine whether the action of the left eye and/or the action of the right eye is The specified action corresponding to the current application completed within the set time; and, a triggering module, wherein the action of the left eye and/or the action of the right eye is a specified action corresponding to a current application completed within a set time, triggering the current application execution to correspond to the specified action The function. The function triggering device according to claim 5, wherein the function triggering device further comprises: And an alarm module, configured to issue an alarm to prompt the user after the triggering module triggers the current application to perform a function corresponding to the specified movement. The function triggering device according to claim 5 or 6, wherein the identification module is specifically configured to: Identifying a position of the dark data in the left eye image to determine a state and a position of the left eye, and determining an action of the left eye according to a state and a position of the left eye at different times; identifying the right eye The position of the dark data in the image is used to determine the state and position of the right eye, and the action of the right eye is determined according to the state and position of the right eye at different times. The function triggering device according to any one of claims 5-7, wherein the specified action comprises at least any one or any combination of blinking, closing eyes and blinking eyes. A virtual reality device, comprising the function triggering device according to any one of claims 5-8. A virtual reality device, comprising: at least two infrared cameras, a processor and a memory, wherein the two infrared cameras are respectively configured to collect a left eye image and a right eye image, the memory is configured to store an instruction, The instructions are for controlling the processor to perform the function triggering method of any one of claims 1-4. A computer readable storage medium characterized by storing program code for performing the function triggering method according to any one of claims 1-4.

Images