CN106201284A - user interface synchronization system and method - Google Patents

Abstract

A user interface synchronization system includes a graphics output module and a command conversion module coupled to an electronic device, as well as an imaging module and an eye movement command analysis module coupled to a wearable device. The graphics output module accesses image data of the electronic device and transmits the image data to the wearable device via a wireless network. The imaging module displays the image data on the output unit through the wearable device for the user to watch and operate. The eye movement command analysis module analyzes the acquired eye movement command and transmits the eye movement command to the electronic device via the wireless network. The instruction conversion module converts the eye movement instruction when receiving the eye movement instruction to output the eye movement instruction into an action instruction that can be executed by the electronic device.

Description

User interface synchronization system and method

technical field

The invention relates to a user interface synchronization system and method thereof, in particular to a user interface synchronization system and method used between an electronic device and a wearable device.

Background technique

Eye tracking refers to the tracking of eye movement by analyzing the user's eye image or the movement pattern of the eye relative to the head. Eye tracker is a device that can track and measure eyeball position and eye movement information. It is widely used in the research of visual system, psychology, and cognitive linguistics. At present, there are many methods of eye movement tracking. The more common eye movement tracking techniques include Purkinje image tracking method (Dual-Purkinje-Image, DPI), infrared image system method (Infra-Red Video System, IRVS), infrared eye movement mapping method ( Infra-Red Oculography, IROG), etc., can determine the user's gaze direction by obtaining the user's eye image.

Recently, some brands of mobile terminals have introduced technology that integrates eye-tracking technology into the mobile terminal. By tracking the user's gaze direction, the mobile terminal can generate corresponding operation instructions to issue corresponding instructions to the mobile terminal. This type of technology can facilitate the user to operate the display screen. When one of the user's hands cannot be free to operate the mobile terminal, the user can still input corresponding instructions by tracking the user's eyeballs. Another scenario can be applied to video playback. According to the direction of the user's gaze, it is judged whether the user is looking at the display screen. When it is judged that the user is not looking at the display screen, the pause function is temporarily activated to ensure that the user will not miss any highlights in the movie. .

However, according to the current existing technology, the user's eyes are photographed through the front lens of the mobile terminal, and the obtained images are easily affected by environmental factors (such as high-light and low-light environments), and it is easy to encounter difficulties in tracking eye movements. When it is desired to perform accurate eye movement tracking (such as gaze direction detection), since the distance between the front lens of the mobile terminal, the display screen and the user's eyes is not fixed, it is impossible to obtain the user's gaze direction through triangular distance measurement. The user's image briefly determines whether the user is looking left or right.

Contents of the invention

The main purpose of the present invention is to solve the problem that the electronic devices in the prior art cannot accurately operate through the eye-tracking technology.

In order to solve the above problems, the present invention provides a user interface synchronization system for pairing an electronic device with a wearable device. The user interface synchronization system includes:

a graphic output module, coupled to the electronic device, to access the image data of the electronic device, and transmit the image data of the electronic device to the wearable device via the wireless network;

An image module, coupled to the wearable device, displays the image data on the output unit of the wearable device for the user to watch and operate;

An eye movement instruction analysis module, coupled to the wearable device, analyzes the eye movement instructions acquired by the wearable device; and

An instruction conversion module, coupled to the electronic device or the wearable device, converts the eye movement instruction when receiving the eye movement instruction, so as to output the eye movement instruction as an executable instruction for the electronic device action command.

Further, the imaging module establishes a user interface window displaying the image data on the output unit of the wearable device, and enlarges or reduces the user interface window in proportion to the length and width of the display screen of the electronic device.

Further, the eye movement command analysis module analyzes the gaze direction of the user according to the acquired eye image, and forms a cursor on the user interface window that can move according to the gaze direction.

Further, when the gaze direction of the user stays on the user interface window, the eye movement instruction analysis module records the coordinate position on the display screen of the electronic device corresponding to the gaze direction, and when the gaze direction stays approximately When the set threshold time is exceeded on the same graphical interface, a trigger command is sent to the instruction conversion module via the wireless network to start one or more programs corresponding to the graphical interface.

Further, when the eye movement instruction analysis module detects that the user's gaze direction is moving rapidly from left to right, it transmits the eye movement instruction of turning the page on the left to the instruction conversion module, so that the electronic device executes the The action instruction of turning the page from the left to the right, the eye movement instruction analysis module transmits the eye movement instruction of turning the page on the right to the instruction conversion module when detecting that the user's gaze direction is moving rapidly from right to left , so that the electronic device executes an action instruction of turning pages from right to left.

Further, the eye movement instruction analysis module sets the reference coordinates when detecting the trigger action of the user, continuously detects the gaze direction of the user, and records the X-axis movement distance and the Y-axis movement distance of the gaze direction relative to the reference coordinates , when the X-axis moving distance or the Y-axis moving distance is greater than a threshold value, transmitting an eye movement instruction corresponding to the eye moving direction and moving distance to the instruction conversion module, so that the electronic device executes the corresponding eye movement instruction. An action command to scroll in the moving direction.

Further, the wireless network adopts a wireless fidelity direct (WiFi Direct) protocol, a Bluetooth wireless transmission (Bluetooth), or a virtual wireless AP (Wi-Fi soft AP).

Another object of the present invention is to provide a controlled electronic device, which includes a display screen, a wireless transmission unit, and a processor connected to the display screen and the wireless transmission unit. The processor has a graphics processing unit for transmitting image data to the display screen to provide a user interface for user operation. The processor includes an arithmetic unit for loading and executing the following programs:

The graphic output module is used for accessing the image data of the electronic device, and transmitting the image data displayed by the electronic device to the wearable device through the wireless network, so that the wearable device can output for visual operation by the user.

The instruction conversion module receives the eye movement instruction provided by the wearable device via the wireless network, and outputs the eye movement instruction as an action instruction that can be executed by the electronic device.

Further, the wireless network adopts a wireless fidelity direct (WiFi Direct) protocol, a Bluetooth wireless transmission (Bluetooth), or a virtual wireless AP (Wi-Fi soft AP).

Another object of the present invention is to provide a wearable device at the main control end, which includes an output unit, a wireless transmission unit, a camera unit, and a processor connected to the output unit, the wireless transmission unit, and the camera unit. The camera unit is used for shooting and obtaining the user's eye image. The processor is provided with a graphics processing unit for transmitting image data to the output unit for a user interface operated by a user. The processor includes an arithmetic unit for loading and executing the following programs:

The imaging module obtains image data of the electronic device through the wireless network, and transmits the image data to the graphic processing unit of the wearable device to be displayed on the output unit for the user to watch and operate.

The eye movement command analysis module obtains the eye image captured by the camera unit, obtains the eye movement command from the eye image, and transmits the eye movement command to the electronic device through the wireless network to activate the electronic device. One or more programs for a device.

Further, the imaging module establishes a user interface window displaying the image data on the output unit of the wearable device, and enlarges or reduces the user interface window in proportion to the length and width of the display screen of the electronic device.

Further, the eye movement instruction analysis module analyzes the user's gaze direction according to the acquired eye image, and forms a cursor on the user interface window that can move according to the gaze direction.

Further, when the gaze direction of the user stays on the user interface window, the eye movement command analysis module records the coordinate position corresponding to the gaze direction on the display screen of the electronic device, and detects the gaze direction When generally staying on the same graphical interface for more than the set threshold time, a trigger command is sent to the electronic device via the wireless network to start one or more programs corresponding to the graphical interface.

Further, when the eye movement instruction analysis module detects that the user's gaze direction is moving rapidly from left to right, it transmits the eye movement instruction of turning the page from the left to the electronic device, so that the electronic device performs For the program of turning the page to the right, the eye movement instruction analysis module transmits the eye movement instruction of turning the page on the right to the electronic device when it detects that the user's gaze direction is moving rapidly from right to left, so that the electronic device The device executes the program of turning pages from right to left.

Further, the eye movement command analysis module sets the reference coordinates when detecting the trigger action of the user, continuously detects the gaze direction of the user, and records the X-axis movement distance and the Y-axis movement of the gaze direction relative to the reference coordinates distance, when the X-axis moving distance or the Y-axis moving distance is greater than the threshold value, the eye movement command corresponding to the moving direction and moving distance of the eye is transmitted to the electronic device, so that the electronic device executes the action corresponding to the eye A program that scrolls in the mobile direction.

Further, the wireless network adopts a wireless fidelity direct (WiFi Direct) protocol, a Bluetooth wireless transmission (Bluetooth), or a virtual wireless AP (Wi-Fi soft AP).

Another object of the present invention is to provide an interface synchronization method for a wearable device and an electronic device, including: accessing the image data of the electronic device, and transmitting the image data of the electronic device to the wearable device via a wireless network ;display the image data on the output unit of the wearable device through the wearable device, so that the user can watch and operate; analyze the eye movement instructions acquired by the wearable device, and send the eye movement instructions through the wireless network transmitting to the electronic device; and converting the eye movement command when receiving the eye movement command, so as to output the eye movement command as an action command executable by the electronic device.

Further, after the wearable device receives the image data, it enlarges or shrinks proportionally according to the length and width of the display screen of the electronic device to create a user interface window for displaying the image data.

Further, the wearable device analyzes the user's gaze direction according to the acquired eye image, and forms a cursor on the user interface window that can move according to the gaze direction.

Further, when the user's gaze direction stays on the user interface window, the coordinate position on the display screen of the electronic device corresponding to the gaze direction is recorded, and the gaze direction roughly stays on the same graphical interface for more than When the threshold time is set, a trigger command is sent to the electronic device via the wireless network to start one or more programs corresponding to the graphical interface.

Further, when it is detected that the gaze direction of the user is moving rapidly from left to right, the eye movement command of turning the page on the left is transmitted to the electronic device, so that the electronic device performs the turning of the page from left to right. Action command, when detecting that the user’s gaze direction is moving rapidly from right to left, transmit the eye movement command of turning the page on the right to the electronic device, so that the electronic device performs the action of turning the page from the right to the left action command.

Further, set the reference coordinates when the trigger action of the user is detected, continuously detect the gaze direction of the user, and record the X-axis movement distance and the Y-axis movement distance of the gaze direction relative to the reference coordinates, when the X-axis movement When the distance or the Y-axis movement distance is greater than the threshold, the eye movement command corresponding to the eye movement direction and movement distance is transmitted to the electronic device, so that the electronic device executes the scrolling action command corresponding to the eye movement direction .

Another object of the present invention is to provide a computer-readable recording medium, on which a program is recorded. After the program is loaded and executed in electronic devices and wearable devices, the above method can be completed.

Another object of the present invention is to provide a computer program product, which can implement the above-mentioned method when the computer program product is loaded into an electronic device and a wearable device for execution.

Therefore, compared with the aforementioned prior art, the present invention has the following excellent effects:

1. The user interface synchronization system of the present invention can transmit the image data of the electronic device to the output unit of the wearable device, so as to operate the electronic device by tracking the user's eye movements.

2. The distance between the front lens and the user's eyes can be maintained at a fixed distance, which makes it easier to detect the user's eye movements.

Description of drawings

FIG. 1: A schematic block diagram showing the user interface synchronization system of the present invention.

Fig. 2: A schematic view showing the usage status of the user interface synchronization system of the present invention.

Figure 3: Schematic representing the user interface window (a).

Figure 4: Schematic diagram showing the trajectories generated by eye movements on the user interface window (1).

Figure 5: Schematic diagram showing the trajectory generated by eye movements on the user interface window (2).

Figure 6: Schematic representation of the user interface window (2).

Figure 7: Schematic representation of another user interface window.

FIG. 8 : shows a schematic flow chart (1) of the user interface synchronization method of the present invention.

FIG. 9 : Schematic flow diagram (2) showing the user interface synchronization method of the present invention.

FIG. 10 : Schematic flow chart (3) showing the user interface synchronization method of the present invention.

FIG. 11 : Schematic flow diagram (4) showing the user interface synchronization method of the present invention.

Symbol Description

100 User Interface Synchronization System

10 electronics

11 display screen

12 processing units

13 graphics processing unit

14 storage units

16 wireless transmission unit

17 Graphic output module

18 instruction conversion module

CU1 processor

20 wearable devices

21 output unit

22 processing units

23 graphics processing unit

24 storage units

25 camera unit

26 wireless transmission unit

27 image module

28 Eye Movement Command Analysis Module

CU2 processor

W User interface window

W1 cursor

W2 timer

W3, W4, W5, W6 Arrows (up, down, left, right)

Steps S201-S205

Steps S2051A～S2054A

Steps S2051B-S2053B

Steps S2051C-S2055C

detailed description

Relevant detailed description and technical contents of the present invention are as follows with regard to coordinating illustrations now. Furthermore, for the convenience of illustration, the proportions of the diagrams in the present invention are not necessarily drawn according to the actual scale, but may be exaggerated. These diagrams and their proportions are not intended to limit the scope of the present invention.

The present invention is a user interface synchronization system 100 , which is used to transmit the image of the electronic device 10 to the wearable device 20 , and the wearable device 20 tracks the user's eye movements to operate the electronic device 10 .

The electronic device 10 includes at least a display screen 11 , a processing unit 12 (Central Processing Unit, CPU), and a graphics processing unit 13 (Graphics Processing Unit, GPU) capable of inputting and outputting images. Specifically, the electronic device 10 can be, for example, a cellular phone, a smart phone, a tablet computer, a handheld mobile communication device, a personal digital assistant (Personal Digital Assistant, PDA) or similar portable electronic devices. In addition, The electronic device 10 can also be an electronic device with a display and control interface, such as a calculator, a desktop computer, a notebook computer, or a vehicle-mounted computer.

The wearable device 20 specifically refers to a wearable device worn on the user's head, which can provide a user-operable user interface through the output unit 21, and capture the user's eye image through the camera unit 25, The user interface described above is operated by the gaze direction of the user. The wearable device 20 includes at least an output unit 21 that provides images to the user's eyes, a camera unit 25 that takes pictures of the user's eyes to obtain images of the user's eyes, a processing unit 22 (Central Processing Unit, CPU), and can input A graphics processing unit 23 (Graphics Processing Unit, GPU) for outputting images. Specifically, the wearable device 20 can be smart glasses, an eye tracker, an augmented reality device, a virtual reality device, or similar smart wearable devices.

As shown in Figure 1, it is a schematic block diagram of the user interface synchronization system of the present invention, as shown in the figure:

The following describes the hardware architectures of the electronic device 10 and the wearable device 20 respectively. After the hardware architectures are explained, the software architectures will be further described later.

Electronics:

As mentioned above, the electronic device 10 is used as a controlled terminal to transmit image data to the wearable device 20 , and operate the electronic device 10 through the wireless network through the eye movement tracking function of the wearable device 20 . The electronic device 10 includes a display screen 11 , a processing unit 12 (Processing Unit), a Graphics Processing Unit 13 (Graphics Processing Unit, GPU) capable of inputting and outputting images, a storage unit 14 , and a wireless transmission unit 16 .

The processing unit 12 and the graphics processing unit 13 can jointly constitute the processor CU1, so that the processing unit 12 and the graphics processing unit 13 can be integrated on a single chip, thereby reducing the required volume of components. For example, the processor CU1 may be developed by, for example, ARM Holdings, Ltd. series of processors, and Loongson processors developed by the Institute of Computing Technology (ICT) of the Chinese Academy of Sciences, etc., are not limited in the present invention.

In another preferred embodiment, the processing unit 12 and the graphics processing unit 13 can individually constitute processors, which respectively process logic operations and image processing, and jointly or cooperatively process some program instructions.

In another preferred embodiment, the processing unit 12 and the storage unit 14 together constitute a processor, and the processing unit 12 can load a program pre-stored in the storage unit 14 and execute a corresponding algorithm.

In this implementation, the processing unit 12 and the graphics processing unit 13 jointly constitute a processor CU1 , and the processor CU1 is coupled to the storage unit 14 . The processor CU1 can be a central processing unit (Central Processing Unit, CPU), or other programmable microprocessor (Microprocessor), digital signal processor (Digital Signal Processor, DSP) with general purpose or special purpose, Programmable controller, application specific integrated circuit (Application Specific Integrated Circuits, ASIC), programmable logic device (Programmable Logic Device, PLD) or other similar devices or a combination of these devices.

The display screen 11 is used to display graphic data, such as user operation interface, graphical interface, or multimedia images, etc., by displaying the image or operation interface on the display screen 11 for users to read. The display screen 11 can be an active matrix organic light emitting device (AMOLED) display, a thin film transistor (Thin Film Transistor, TFT) display, or other similar display devices, which are not limited in the present invention. The display screen 11 is driven by the control circuit, by inputting corresponding signals to the data line driving circuit and the scanning line driving circuit, to drive the light-emitting units (pixel elements) of the panel on the corresponding coordinates. The display screen 11 publishes the corresponding multimedia data on the display screen 11 after the graphics processing unit 13 accesses the data in the storage unit 14 for visual viewing by the user.

The wireless transmission unit 16 is capable of data transmission through a wireless network. Specifically, the wireless network adopts a wireless fidelity direct (WiFi Direct) protocol, a Bluetooth wireless transmission (Bluetooth), or a virtual wireless AP (Wi-Fi soft AP). In another preferred embodiment, the wireless transmission unit 16 can be paired with the wearable device 20 through radio frequency identification (Radio Frequency Identification, RFID) technology, so as to carry out medium and short distance communication with the wearable device 20 Wireless data transmission.

Wearable device:

The wearable device 20 can be used as a master control terminal, by receiving the image data of the electronic device 10 and outputting the image data to the eyes of the user for the user to watch and operate. As mentioned above, the wearable device 20 includes an output unit 21, a processing unit 22 (Central Processing Unit, CPU), a graphics processing unit 23 (Graphics Processing Unit, GPU) capable of inputting and outputting images, a camera unit 25, and Wireless transmission unit 26.

The processing unit 22 is substantially the same as the processing unit 12 of the electronic device 10 , and the description of the processing unit 22 will not be repeated here. Same as the processing unit 12 of the electronic device 10, in a preferred embodiment, the processing unit 22 and the graphics processing unit 23 together constitute the processor CU2, so that the processing unit 22 and the graphics processing unit 23 can be integrated in a single chip superior. In another preferred embodiment, the processing unit 22 and the graphics processing unit 23 can individually constitute processors, which respectively process logic operations and image processing, and jointly or cooperatively process some program instructions. In another preferred embodiment, the processing unit 22 and the storage unit 24 together constitute a processor, and the processing unit 22 can load the program pre-stored in the storage unit 24 and execute the corresponding algorithm.

In this implementation, the processing unit 22 and the graphics processing unit 23 jointly constitute a processor CU2 , and the processor CU2 is coupled to the storage unit 24 . The processing unit 22 can be a central processing unit (Central Processing Unit, CPU), or other programmable microprocessors (Microprocessor), digital signal processors (Digital Signal Processor, DSP) with general purpose or special purpose, and can Programmable controller, application specific integrated circuit (Application Specific Integrated Circuits, ASIC), programmable logic device (Programmable Logic Device, PLD) or other similar devices or a combination of these devices.

The output unit 21 is used for displaying graphic data, and sending the graphic data to the user's eyes for visual operation by the user. The output unit 21 can be a display screen, such as an active matrix organic light emitting device (AMOLED) display, a thin film transistor (thin film transistor, TFT) display, or other similar display devices. In another preferred embodiment, the output unit 21 may be a retinal display, which directly projects images on the retina for the user to watch through retinal image projection (Retinal Imaging Display, RID) technology. Through the reflection of the glass on the retinal display, the light beam can be directly imaged on the retina, so that the image beam can be integrated with the real scene seen by the naked eye. The output unit 21 can issue the corresponding multimedia data to the eyes of the user after accessing the data in the storage unit 24 through the graphics processing unit 23 (GPU) for the user to view.

The camera unit 25 is a camera equipped with a Charge Coupled Device (CCD) or a Complementary Metal-Oxide Semiconductor (CMOS), which is not limited in the present invention. The camera unit 25 is used to take images of the user's eyes, and the obtained eye images will be sent to the eye movement command analysis module 28 for further analysis, so as to track the user's eye movements and convert the eye movements into corresponding eye movement commands.

The wireless transmission unit 26 is capable of data transmission through a wireless network. Specifically, the wireless network adopts a wireless fidelity direct (WiFi Direct) protocol, a Bluetooth wireless transmission (Bluetooth), or a virtual wireless AP (Wi-Fi soft AP).

The hardware architectures of the electronic device 10 and the wearable device 20 have been described above in detail. Here, based on the above hardware architectures, the architecture of the user interface synchronization system of the present invention is further described in detail:

In conjunction with Fig. 2, it is a schematic diagram of the use state of the user interface synchronization system of the present invention, as shown in the figure:

The present invention can pair the electronic device 10 with the wearable device 20 . After the pairing is successful, the electronic device 10 can transmit the image data on the display screen 11 to the wearable device 20 through the wireless network, so that the user can read the display screen 11 through the wearable device 20 . The wearable device 20 can capture the user's eye image through the camera unit 25 , so as to wirelessly operate the electronic device 10 through eye movements, so as to activate one or more programs of the electronic device 10 .

Referring to FIG. 1 , the user interface synchronization system 100 includes a graphics output module 17 coupled to the electronic device 10, an instruction conversion module 18, an image module 27 coupled to the wearable device 20, and an eye movement instruction analysis module. Module 28.

In this embodiment, the graphics output module 17 and the command conversion module 18 are pre-stored in the storage unit 14 of the electronic device 10, so that the processing unit 12 of the electronic device 10 is mounted on the graphics output module 17 And the program of the instruction conversion module 18 executes its algorithm. The image module 27 and the eye movement instruction analysis module 28 are pre-stored in the storage unit 24 of the wearable device 20, so that the processing unit 22 of the wearable device 20 can mount the image module 27 and the eye movement instruction analysis module 28. The program of the instruction analysis module 28 then executes its algorithm. In another preferred embodiment, the command conversion module 18 can also be loaded on the wearable device 20, and the wearable device 20 converts the eye movement commands into the electronic device 10 through the command conversion module 18. After the action command is executed, the action command is sent to the electronic device 10 via the wireless network to start one or more programs. In the above steps, the wearable device 20 encrypts the action command, and the electronic device 10 only needs to decrypt the action command and execute it. The above method can be understood as another similar method of the present invention. implement.

The graphic output module 17 is coupled to the electronic device 10 to access the image data of the electronic device 10 and transmit the image data of the electronic device 10 to the wearable device 20 via the wireless network. The graphics output module 17 is used to access the image data (user interface) provided in the graphics processing unit 13, and the obtained image data is synchronized with the image displayed on the display screen 11, or according to preset Or the user sets to forcibly turn off the display screen 11, so that the graphic output module 17 directly transmits the image data to the wearable device 20 through the wireless network, so as to reduce the extra burden of the graphic output module 17 and reduce the power consumption of the electronic device.

The image module 27 is paired with the graphic output module 17 via the wireless transmission unit 26, and the image module 27 is used to display image data on the output unit 21 of the wearable device 20 for the user to watch and operate . As shown in FIG. 3 , the image module 27 can establish a user interface window W for displaying the image data, and adjust the user interface window W according to the length and width of the display screen 11 of the electronic device 10 in proportion to enlarge or reduce, so that the user can easily Operate the user interface window W within the best visual perception range. The image module 27 displays a movable cursor W1 on the user interface window W, and the cursor W1 moves along the gaze direction of the user, and the gaze direction is calculated by the eye movement command analysis module 28 .

In a preferred embodiment, the graphics output module 17 can compress a series of image data through streaming technology (Streaming media), and then transmit the data in segments through the wireless network, and the image coupled to the wearable device 20 The module 27 decompresses the streaming packet to display the image data on the output unit 21 of the wearable device 20 in real time.

After the image data is displayed on the user interface window W provided by the output unit 21, the user can operate the cursor W1 on the user interface window W through eye movements, or generate corresponding eye movements through continuous eye movements Instructions operate the user interface window W. After the eye movement instruction analysis module 28 analyzes the user's eye movement, it converts the eye movement into an eye movement instruction, and transmits the eye movement instruction to the instruction coupled to the electronic device 10 through the wireless network. The conversion module 18 is used for the analysis of the command conversion module 18, and further finds that the eye movement command corresponds to the motion command of the electronic device 10, so as to start one or more programs corresponding to the motion command through the processing unit 12. In a preferred embodiment, the command conversion module 18 includes a look-up table, through which the command conversion module 18 can convert the eye movement command into an action command corresponding to the electronic device 10 . Therefore, when the wearable device 20 is successfully paired with the electronic device 10, the user can operate the user interface window W through the wearable device 20. At this time, the camera unit 25 will continue to capture the user's eye image, and through the captured The user can operate the electronic device 10 through the wearable device 20 through eye movements.

Specifically, the eye movement command analysis module 28 includes two functions, one is to analyze the corresponding eye movements through the user's eye images, and judge the user's gaze direction through the eye movements; The gaze direction judges the eye movement command that the user wants to input.

For the technology of obtaining eye movement (gaze direction) through eye images, Purkinje image tracking method (Dual-Purkinje-Image, DPI), infrared image system method (Infra-Red Video System, IRVS), infrared eye movement diagram can be used method (Infra-Red Oculography, IROG), which is not limited in the present invention. Through the above method, multiple eye images of the user can be used to collect multiple sample points (as shown in FIG. 4 ) where the eye image is mapped on the output unit 21. The obtained sample points are obtained by In order to further analyze the eye movement command that the user intends to input.

The following is a description of some major eye movement commands. For example, the user can input a page turning command to the electronic device 10 through the eye movement command, so that the electronic device 10 can flip the screen on the user interface window W to achieve page turning. Effect. Referring to Figure 4, in order to reveal a series of sample points when the user scrolls to the right, the system defaults to set the middle position as the start point and end point of the eye track command. In the figure, the start position is displayed as is ○, the end position is displayed as ╳, and the detected coordinates are arranged in dotted order. When the eye movement command analysis module 28 detects that the user's gaze direction is moving rapidly from the left (ie, the middle position) to the right (the right position), the eye movement command for turning the page on the left side is transmitted to the electronic device 10. Instruction conversion module 18. At this time, when the instruction conversion module 18 receives the eye movement instruction for turning the page on the left side, it calls the corresponding action instruction through the lookup table, so that the processing unit 12 performs the operation of turning the page from the left side to the right side. event.

In the part of turning the page to the right, please refer to Figure 5. In order to reveal a series of sample points when the user operates the right page turning, in the system default, in the illustration, the starting position is displayed as ○, and the ending position is Displayed as ╳, the detected coordinates are arranged in dotted order. When the eye movement command analysis module 28 detects that the user's gaze direction is moving rapidly from the right (ie, the middle position) to the left (the left side position), it transmits the eye movement command of turning the page on the right to the electronic device 10. The instruction conversion module 18, at this time, upon receiving the right page turning, the instruction conversion module 18 finds the corresponding action instruction through the lookup table, so that the processing unit 12 executes the event of turning the page from the right to the left.

When the page-turning command input is completed, the electronic device 10 enters a short period of non-response (for example, one second), which is to prevent the user from being caught when the eyes return to the middle position after turning the page through eye movements. It is judged as a page turning command.

In another preferred embodiment, the user can input an action command of scrolling pages to the electronic device 10 through an eye movement command, so that the pages on the user interface W are scrolled in the direction of the user's gaze. Specifically, the eye movement instruction analysis module 28 detects the user's eye movement, and sets a reference point when detecting the user's trigger action (the described trigger action can be blinking, closing eyes, drawing a circle) or other predefined eye movements), and continuously record the X-axis movement distance and the Y-axis movement distance of the gaze direction relative to the reference coordinates. When the X-axis moving distance or the Y-axis moving distance is greater than the threshold value, the eye movement command analysis module 28 transmits the instruction corresponding to the moving direction of the eye (that is, the positive and negative values of the X-axis or Y-axis) and the moving distance. The eye movement command is sent to the command conversion module 18, and the command conversion module 18 transmits the command to the processing unit 12, so that the processing unit 12 executes a scrolling motion command corresponding to the eye movement direction. Wherein, the obtained eye moving direction will be used as a reference value for determining the scrolling direction, and the obtained moving distance will be used as a reference value for determining the scrolling speed.

Referring to FIG. 6 , the user can manipulate the cursor W1 on the user interface window W by staring at the user interface window W. Referring to FIG. The eye movement instruction analysis module 28 analyzes the gaze direction of the user according to the acquired eye images, and forms a cursor W1 on the user interface window W that can move according to the gaze direction. When the user's gaze direction stays on the user interface window W, the eye movement instruction analysis module 28 records the coordinate position corresponding to the gaze direction on the display screen 11 of the electronic device 10, and when the gaze direction is approximately When staying on the same graphical interface for more than the set threshold time, the trigger command is sent to the command conversion module 18 via the wireless network, and the command conversion module 18 converts the trigger command into an action command of light touch start, and starts Corresponding to the software program or instruction corresponding to the graphical interface at the coordinate position.

As shown in Figure 6, in a preferred embodiment, when the image module 27 receives the image data, it can define a range and objectify the corresponding graphical interface on the output unit 21 (the defined range can be obtained by the graphics processing unit 13 of the electronic device 10), when the user's gaze direction stays within the range defined by one of the graphical interfaces, the cursor W1 will be converted into a timer W2, and mark the time needed to start the graphical interface. time. As shown in the figure, the timer W2 includes a percentage number and a timing bar. When the user's gaze stays on the graphical interface, the cursor W1 will be converted into a timer W2. At this time, the percentage number and the timing bar will display the remaining Time, when the percentage figure reaches 100%, the timing bar is also displayed as a full grid at this time, and the eye movement command analysis module 28 is to transmit the trigger command to the command conversion module 18 via the wireless network to start the corresponding graphic interface. software programs or instructions.

Please also refer to FIG. 7, which discloses another preferred embodiment. The user interface window W can be provided with multiple operating areas, and the user can watch the graphical interface corresponding to each operating area to generate a corresponding Eye movement commands in the operating area. For example, when the user places the cursor W1 (gazing direction) on the arrow graphic W3 in the right direction, the eye movement instruction analysis module 28 will transmit the eye movement instruction of turning the page on the right to the instruction conversion module of the electronic device 10 18. When the user keeps the cursor W1 (gazing direction) on the arrow graphic W4 in the left direction, the eye movement instruction analysis module 28 converts the instruction of the eye movement instruction for turning the page on the right to the electronic device 10 Module 18; when the user keeps the cursor W1 (gazing direction) on the arrow graphic W5 in the upper direction, the eye movement instruction analysis module 28 will transmit the eye movement instruction of turning the page up to the electronic device 10 Conversion module 18; when the user stays the cursor W1 (gazing direction) on the arrow graphic W6 in the downward direction, the eye movement instruction analysis module 28 will transmit the eye movement instruction of turning the page down to the instruction of the electronic device 10 Conversion module 18; after receiving the above-mentioned eye movement instructions, the instruction conversion module 18 can find the corresponding movement instructions through a lookup table or an object-oriented method, and send the movement instructions to the processing unit 12 for Start the corresponding one or more programs.

The following is a detailed description of the user interface synchronization method of the present invention with reference to the diagram. Referring to FIG. 8, it is a schematic flow diagram of the user interface synchronization method of the present invention, as shown in the figure:

The user interface synchronization method of the present invention is applied to the electronic device 10 and the wearable device 20, and the screen on the electronic device 10 can be transferred to the wearable device 20 to operate wirelessly through the eye control function of the wearable device 20 The user interface of the electronic device 10 . The specific process of interface synchronization is as follows:

Initially, the electronic device 10 is first paired with the wearable device 20, and the pairing can be performed through encryption, key establishment or other mutually verifiable methods, so that the electronic device 10 and the wearable device 20 can establish online. (step S201)

When the pairing is completed, access the image data of the electronic device 10 (such as the image on the display screen 11 ), and transmit the image data of the electronic device 10 to the wearable device 20 via the wireless network. (step S202)

After receiving the image data, the wearable device 20 displays the image data on the output unit 21 of the wearable device 20 for the user to watch and operate. The wearable device 20 is based on the length and width of the display screen 11 of the electronic device 10 according to the same ratio of zooming in or out to create a user interface window W for displaying the image data. (step S203)

The wearable device 20 analyzes the gaze direction of the user according to the acquired eye image, and forms a cursor W1 on the user interface window W that can move according to the gaze direction. (step S204)

The eye movement command acquired by the wearable device 20 is analyzed, and the eye movement command is transmitted to the electronic device 10 via the wireless network. The electronic device 10 converts the eye movement command when receiving the eye movement command, so as to output the eye movement command as an action command executable by the electronic device 10 . (step S205)

The following are three different implementation modes for step S205. It is understandable that the following three implementation modes can be performed simultaneously in step S205:

For the first implementation mode, refer to the contents of FIG. 9 below. Firstly, obtain the range corresponding to each graphical interface on the display screen array, which can be obtained from the graphics processing unit 13 of the electronic device 10, and the range of each graphical interface includes the range corresponding to the graphical interface One or more programs that interface. (Step S2051A) When the user's gaze direction stays on the user interface window, record the coordinate position corresponding to the gaze direction on the display screen 11 of the electronic device 10 to confirm the user's gaze direction (step S2052A). When the user's gaze direction stays on one of the graphical interfaces, start a timer to record the dwell time of the user's gaze direction, and judge whether the gaze direction exceeds the set threshold time, such as 1 to 2 seconds (step S2053A), When the user's gaze direction stays on the same graphical interface for more than the set threshold time (eye movement instruction), the trigger instruction is transmitted to the electronic device 10 via the wireless network to activate one or more corresponding to the graphical interface. Multiple programs (action instructions). (step S2054A). On the contrary, if the gaze direction is out of the scope of the graphical interface, return to step S2052A and continue to detect the gaze direction of the user.

For the second implementation mode, refer to the contents of FIG. 10 below. First, start the decision-making program, in the decision-making program, continuously detect the user's gaze direction (step S2051B), when detecting that the user's gaze direction is moving rapidly from left to right, transfer the eye movement instruction of turning the page on the left to the user The electronic device 10, so that the electronic device 10 executes an action instruction of turning pages from left to right (step S2052B). When it is detected that the gaze direction of the user is moving rapidly from right to left, the eye movement command of turning the page on the right is transmitted to the electronic device 10, so that the electronic device 10 performs the action of turning the page from the right to the left instruction. (step S2053B)

For the third implementation mode, refer to the content of FIG. 11 below. First, the wearable device 20 detects whether the user performs a trigger action (step S2051C). When the trigger action of the user is detected, a reference coordinate is set, the gaze direction of the user is continuously detected, and the X-axis movement distance and the Y-axis movement distance of the gaze direction relative to the reference coordinate are recorded (step S2052C). Next, determine whether the X-axis moving distance or the Y-axis moving distance is greater than a threshold (step S2053C), if yes, go to the next step, if not, go back to step S2052C. When the X-axis moving distance or the Y-axis moving distance is greater than the threshold, the eye moving direction is determined (step S2054C). Send the eye motion command including the eye moving direction and moving distance to the electronic device 10, so that the electronic device 10 executes the scrolling motion command corresponding to the eye moving direction (step S2055C). In the above steps, if the distance of the user's gaze direction returns to a state smaller than the threshold, return to the procedure before step 2051C, and continue to detect whether the user generates a trigger action.

The method steps described in the present invention can also be implemented as a computer-readable recording medium for storing in computer-readable recording media such as optical discs, hard disks, and semiconductor memory devices, and through the computer-readable recording medium Loaded on an electronic device for access and use by the electronic device or electronic equipment.

The method steps described in the present invention can also be implemented as a computer program product, which is used to store in the hard disk of the network server, memory device, such as app store, google play, windows market, or other similar online distribution platforms of application programs, It can be implemented by uploading the computer program product to the server and then providing the user with a fee to download it.

To sum up, the user interface synchronization system of the present invention can transmit the image data of the electronic device to the output unit of the wearable device, so as to operate the electronic device by tracking the user's eye movements. The distance between the front lens and the user's eyes can be maintained at a fixed distance, which makes it easier to detect the user's eye movements.

The present invention has been described in detail above, but the above is only a preferred embodiment of the present invention, and should not limit the scope of the present invention with this, that is, all equivalent changes made according to the patent scope of the present invention and modification, all should still belong to the scope of patent coverage of the present invention.

Claims (24)

1. A user interface synchronization system for pairing an electronic device with a wearable device, wherein the user interface synchronization system comprises: a graphic output module, coupled to the electronic device, to access the image data of the electronic device, and transmit the image data of the electronic device to the wearable device via the wireless network; An image module, coupled to the wearable device, displays the image data on the output unit of the wearable device for the user to watch and operate; An eye movement instruction analysis module, coupled to the wearable device, analyzes the eye movement instructions acquired by the wearable device; and An instruction conversion module, coupled to the electronic device or the wearable device, converts the eye movement instruction when receiving the eye movement instruction, so as to output the eye movement instruction as an executable instruction for the electronic device action command. 2. The user interface synchronization system according to claim 1, wherein the image module establishes a user interface window displaying the image data on the output unit of the wearable device, and according to the display screen of the electronic device The length and width are enlarged or reduced in proportion to adjust the user interface window. 3. The user interface synchronization system according to claim 2, wherein the eye movement command analysis module analyzes the user's gaze direction according to the obtained eye image, and forms a gazing direction on the user interface window. A cursor that can be moved according to this gaze direction. 4. The user interface synchronization system as claimed in claim 3, wherein, when the user's gaze direction stays on the user interface window, the eye movement command analysis module is to record the gaze direction corresponding to the electronic window. The coordinate position on the display screen of the device, and when the gaze direction roughly stays on the same graphical interface for more than the set threshold time, the trigger instruction is transmitted to the instruction conversion module via the wireless network to start the graphical interface corresponding to one or more programs. 5. The user interface synchronization system as claimed in claim 3, wherein the eye movement instruction analysis module transmits the eye that turns the page on the left side when it detects that the user's gaze direction is moving rapidly from left to right. part of the action command to the command conversion module, so that the electronic device executes the action command of turning the page from left to right, and the eye movement command analysis module detects that the user's gaze direction is moving quickly from right to left and transmitting the eye movement instruction of turning the page from the right to the instruction conversion module, so that the electronic device executes the action instruction of turning the page from the right to the left. 6. The user interface synchronization system as claimed in claim 5, wherein the eye movement instruction analysis module sets a reference coordinate when detecting the trigger action of the user, continuously detects the gaze direction of the user, and records The X-axis movement distance and the Y-axis movement distance of the gaze direction relative to the reference coordinates, when the X-axis movement distance or the Y-axis movement distance is greater than a threshold value, an eye movement command corresponding to the eye movement direction and movement distance is transmitted to the instruction conversion module, so that the electronic device executes the action instruction of scrolling corresponding to the eye movement direction. 7. The user interface synchronization system according to claim 1, wherein the wireless network adopts Wi-Fi Direct protocol, Bluetooth wireless transmission, or virtual wireless access point. 8. A controlled electronic device, comprising: a display screen, a wireless transmission unit, and a processor connected to the display screen and the wireless transmission unit, the processor comprising: a graphic processing unit, configured to transmit image data to the display screen to provide a user interface for user operation; The computing unit is used to mount and execute the following programs: A graphics output module, used to access the image data of the electronic device, and transmit the image data displayed by the electronic device to the wearable device via the wireless network, and the wearable device outputs it for visual operation by the user; and instruction conversion The module receives the eye movement instruction provided by the wearable device via the wireless network, and outputs the eye movement instruction as an action instruction that can be executed by the electronic device. 9. The electronic device according to claim 8, wherein the wireless network adopts Wi-Fi Direct, Bluetooth wireless transmission, or virtual wireless access point. 10. A wearable device at the master control end, comprising: an output unit, a wireless transmission unit, a camera unit, and a processor connected to the output unit, the wireless transmission unit, and the camera unit, the The camera unit is used to shoot and obtain the user's eye image, and the processor includes: a graphic processing unit, configured to transmit the image data to the output unit to provide a user interface for user operation; The computing unit is used to mount and execute the following programs: The imaging module obtains the image data of the electronic device through the wireless network, and transmits the image data to the graphics processing unit of the wearable device to be displayed on the output unit for the user to watch and operate; The eye movement instruction analysis module obtains the eye image captured by the camera unit, obtains the eye movement instruction from the eye image, and transmits the eye movement instruction to the electronic device through the wireless network to activate the One or more programs of an electronic device. 11. The wearable device according to claim 10, wherein the imaging module establishes a user interface window displaying the image data on the output unit of the wearable device, and according to the display screen of the electronic device The length and width of the user interface window are enlarged or reduced in proportion to adjust the user interface. 12. The wearable device according to claim 11, wherein the eye movement instruction analysis module analyzes the gaze direction of the user based on the acquired eye image, and forms a A cursor that can be moved according to this gaze direction. 13. The wearable device according to claim 12, wherein when the user's gaze direction stays on the user interface window, the eye movement instruction analysis module will record the gaze direction corresponding to the electronic device. The coordinate position on the display screen of the computer, and when it is detected that the gaze direction stays on the same graphical interface for more than the set threshold time, the trigger instruction is sent to the electronic device via the wireless network to start the corresponding graphical interface. one or more programs. 14. The wearable device according to claim 12, wherein when the eye movement instruction analysis module detects that the user's gaze direction is moving rapidly from left to right, the eye movement instruction to turn the page on the left is transmitted to the electronic device, so that the electronic device executes the program of turning the page from left to right, and when the eye movement command analysis module detects that the user’s gaze direction is moving rapidly from right to left, it transmits the program of turning the page on the right eye movement instructions to the electronic device, so that the electronic device executes a program of turning pages from right to left. 15. The wearable device according to claim 12, wherein the eye movement command analysis module detects the user's trigger action, sets the reference coordinates, continuously detects the user's gaze direction, and records the gaze direction. The X-axis movement distance and the Y-axis movement distance of the direction relative to the reference coordinates. When the X-axis movement distance or the Y-axis movement distance is greater than the threshold, the eye movement command corresponding to the eye movement direction and movement distance is transmitted to the an electronic device, so that the electronic device executes a scrolling program corresponding to the moving direction of the eyes. 16. The wearable device according to claim 10, wherein the wireless network adopts Wi-Fi Direct protocol, Bluetooth wireless transmission, or virtual wireless access point. 17. A method for interface synchronization of a wearable device and an electronic device, comprising: accessing the image data of the electronic device, and transmitting the image data of the electronic device to the wearable device via the wireless network; Displaying the image data on the output unit of the wearable device for the user to watch and operate; Analyzing the eye movement command acquired by the wearable device, and transmitting the eye movement command to the electronic device via the wireless network; converting the eye movement command when receiving the eye movement command, so as to The eye movement instruction is output as an action instruction that can be executed by the electronic device. 18. The interface synchronization method according to claim 17, characterized in that, after receiving the image data, the wearable device establishes and displays the image according to the length and width of the display screen of the electronic device according to proportional enlargement or reduction. User interface window for image data. 19. The interface synchronization method according to claim 18, wherein the wearable device analyzes the gaze direction of the user based on the acquired eye image, and forms an image on the user interface window that can be based on the gaze direction. direction to move the cursor. 20. The interface synchronization method according to claim 19, wherein, when the user's gaze direction stays on the user interface window, the coordinate position on the display screen corresponding to the gaze direction of the electronic device is recorded, and When the gaze direction stays on the same graphical interface for more than a set threshold time, a trigger command is sent to the electronic device via the wireless network to start one or more programs corresponding to the graphical interface. 21. The interface synchronization method according to claim 19, wherein when it is detected that the user's gaze direction is moving rapidly from left to right, the eye movement instruction for turning the page on the left is transmitted to the electronic device to Make the electronic device execute the action command of turning the page from left to right, and when it is detected that the user's gaze direction is moving rapidly from right to left, transmit the eye action command of turning the page on the right to the electronic device, so that The electronic device is made to execute the action instruction of turning pages from right to left. 22. The interface synchronization method according to claim 19, characterized in that, when the trigger action of the user is detected, the reference coordinates are set, the user's gaze direction is continuously detected, and the X of the gaze direction relative to the reference coordinates is recorded. Axis moving distance and Y-axis moving distance, when the X-axis moving distance or the Y-axis moving distance is greater than the threshold value, transmit the eye movement command corresponding to the eye moving direction and moving distance to the electronic device, so that the electronic The device executes an action instruction of scrolling corresponding to the eye movement direction. 23. A computer-readable recording medium, characterized in that a program is recorded on the readable recording medium, and when the electronic device and the wearable device are loaded with the program and executed, it can complete the tasks as claimed in claims 17 to 1. The method described in any one of 22. 24. A computer program product, characterized in that when the computer program product is loaded into an electronic device or a wearable device for execution, the method according to any one of claims 17 to 22 can be completed.