TWI522892B - Electronic device with virtual input function - Google Patents

Description

Electronic device with virtual input function

The present invention relates to an electronic device, and more particularly to an electronic device having a virtual input function.

With the development of technology, more complex and more humane electronic products have been introduced. For example, laptop computers, tablet computers, and smart phones have the same functions as a desktop computer, plus small size and light weight. Such characteristics, so that users can carry around, has become an indispensable tool for the lives and work of ordinary people.

However, when a user uses a physical keyboard such as a smart phone or a tablet computer or a mouse electronic device, the user can handwrite or operate the virtual keyboard displayed on the screen to perform word processing, which is relatively inconvenient. If you use a physical keyboard and a mouse, increasing the carrying capacity and weight will destroy the portability.

In view of this, the present invention provides an electronic device that provides a more convenient and intuitive operation of the user without a physical input device.

The present invention provides an electronic device, which includes a screen, an image capturing unit, a storage unit, and a processing unit. The image capturing unit is disposed on the back of the electronic device, and the processing unit is coupled to the screen, the image capturing unit, and the storage unit. The image capturing unit is configured to capture a continuous plurality of hand images of the user. The storage unit is used to record a plurality of modules. The processing unit is configured to access and execute the module recorded in the storage unit. The module includes: a display module, an integration module, an identification module, a mapping module, and an input module. The display module is used to display the user interface on the screen. The integrated module is configured to integrate the user interface and each of the hand images to generate a plurality of integrated images, wherein the display module displays each of the integrated image screens on the screen. The identification module is configured to recognize the gesture of the user according to each of the hand images. The mapping module is configured to obtain a position of the gesture in the hand image relative to the user interface to generate a plurality of detection locations. The input module is configured to generate an input signal corresponding to the gesture on the detection location of the user interface, thereby operating the user interface.

The present invention further provides an electronic device, including a screen, an image capturing unit, a transceiver unit, a storage unit, and a processing unit, wherein the image capturing unit and the transceiver unit are disposed on the back of the electronic device, and the processing unit is coupled to the screen and the image capturing device. Unit, transceiver unit and storage unit. The image capturing unit is configured to capture a continuous plurality of hand images of the user. The transceiver unit has a transmitter and a receiver. The storage unit is used to record a plurality of modules. The processing unit is configured to access and execute the module recorded in the storage unit. The module includes: a display module, an integrated module, a detection module, and a display Shooting module and input module. The display module is used to display the user interface on the screen. The integrated module is configured to integrate the user interface and each of the hand images to generate a plurality of integrated images, wherein the display module displays each of the integrated image screens on the screen. The detecting module detects the user's gesture by using the transceiver unit, wherein the gesture corresponds to each of the hand images captured by the image capturing unit. The mapping module is configured to obtain a position of the gesture relative to the user interface to generate a plurality of detection locations. The input module is configured to generate an input signal corresponding to the gesture on the detection location of the user interface, thereby operating the user interface.

Based on the above, the electronic device of the present invention can provide a virtual input operation of the user interface at a position other than the electronic device, and the gesture position is mapped to the user interface by recognizing the user's gesture and its position. The relative position, and thus the corresponding input signal, is generated at the relative position. Therefore, the gesture operation of the user at a position other than the electronic device can correspondingly generate an input signal on the user interface, thereby operating the user interface. In addition, the electronic device of the present invention further integrates the user interface and the hand image, so that the user can see the relative relationship between the hand and the object on the user interface on the screen. Accordingly, the user can achieve the effect of operating the electronic device close to using the physical input device without carrying an input device such as a physical keyboard or a mouse, and provide a more intuitive operation of the user.

The above described features and advantages of the invention will be apparent from the following description.

100, 400‧‧‧ electronic devices

110, 410‧‧‧ screen

120, 420‧‧‧ image capture unit

425‧‧‧ transceiver unit

130, 430‧‧‧ storage unit

131, 431‧‧‧ display module

132, 432‧‧‧ integrated modules

133‧‧‧Intelligence module

433‧‧‧Detection module

134, 434‧‧‧ mapping module

135, 435‧‧‧Switch Module

140, 440‧‧ ‧ processing unit

S201~S211, S501~S511‧‧‧Virtual input operation method

30‧‧‧Integrated imagery

32‧‧‧Virtual keyboard

34‧‧‧User interface

A‧‧‧Hand

A’‧‧‧Hand image

FIG. 1 is a block diagram of an electronic device according to an embodiment of the invention.

2 is a flow chart of a method for virtual input operation according to an embodiment of the invention.

FIG. 3 is a schematic diagram of a usage scenario of an electronic device according to an embodiment of the invention.

4 is a block diagram of an electronic device in accordance with another embodiment of the present invention.

FIG. 5 is a flow chart of a method for virtual input operation according to another embodiment of the invention.

At present, when a user uses a physical keyboard such as a smart phone or a tablet computer or a mouse electronic device, the user can handwrite or operate the virtual keyboard displayed on the screen to perform word processing, which is relatively inconvenient. The electronic device provided by the present invention can provide a virtual input operation to the user interface at a position other than the electronic device, and the image capturing device of the electronic device can capture the image of the user's hand to display the hand image. Integrated into the user interface, the user can see the relative positional relationship between the hand and the object on the user interface on the screen to provide a more intuitive operation of the user. In order to make the content of the present invention more comprehensible, the following specific embodiments are exemplary of the invention.

FIG. 1 is a block diagram of an electronic device according to an embodiment of the invention. Referring to FIG. 1 , the electronic device 100 of the embodiment has a display function, such as a tablet computer, a personal digital assistant, a smart phone, an integrated computer, an e-book, and the like. The present invention is not limited to the type of electronic device 100. The electronic device 100 includes a screen 110, an image capturing unit 120, a storage unit 130, and one or more processing units 140, the functions of which are described below.

The screen 110 is used to display a screen output by the electronic device 100 and is provided for viewing by the user. In this embodiment, the screen 110 is, for example, a liquid crystal display (LCD), a Light-Emitting Diode (LED) display, a Field Emission Display (FED), or other kinds of displays. However, although the electronic device 100 of the present invention has a virtual input function, the screen 110 can be integrated with a liquid crystal display, for example, by a resistive, capacitive or other kind of touch sensing element, and the present invention is not limited thereto.

The image capture module 120 includes at least one lens disposed on the back of the electronic device 100 and includes a photosensitive element for sensing the intensity of light entering each of the lenses to generate an image. The photosensitive element is, for example, a Charge Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS) element, or other components, and the present invention is not limited thereto.

The storage unit 130 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory (Flash memory), hard A disc or other similar device or combination of these devices is used to record a plurality of modules executable by processing unit 140 that can be loaded into processing unit 140 to perform the functions of a virtual input operation.

The processing unit 140 is, for example, a central processing unit (CPU), or other programmable general purpose or special purpose microprocessor (Microprocessor), digital signal processor (DSP), programmable Controllers, Application Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), or other similar devices or combinations of these devices. The processing unit 140 is coupled to the screen 110, the image capturing unit 120, and the storage unit 130, and can access and execute the modules recorded in the storage unit 130.

The module includes a display module 131, an integration module 132, a recognition module 133, a mapping module 134, and a switching module 135. The modules are, for example, computer programs, which can be loaded into the processing unit 140 to perform virtual input. The function of the operation.

2 is a flow chart of a method for virtual input operation according to an embodiment of the invention. Referring to FIG. 2, the method of the present embodiment is applied to the electronic device 100 of FIG. 1, and the detailed steps of the method for virtual input operation of the present invention are described below with the components in the electronic device 100.

Referring to FIG. 1 and FIG. 2, first, the processing unit 140 displays the user interface on the screen 110 by using the display module 131 (step S201). The user interface may be, for example, a dynamic tile interface provided by a Windows operating system, a web browsing interface, an email interface, a desktop background, or a display interface provided by other software programs. The present invention is not limited thereto.

Next, the processing unit 140 uses the image capturing unit 120 to capture a plurality of consecutive hand images of the user (step S203). In detail, in this embodiment, use The user interface can be operated on the desktop. The electronic device 100 can be slanted on the desktop, wherein the image capturing unit 120 faces the desktop to continuously capture the user's hand image. However, the invention is not limited thereto. In other embodiments, the user can operate the user interface at a position that can be captured by the image capturing unit 120, such as any plane or even floating.

Then, the processing unit 140 integrates the user interface and each of the hand images with the integration module 132 to generate a plurality of integrated images, and then displays the integrated images on the screen 110 by using the display module 131 (step S205). In detail, the integration module 132 can nest each of the hand images on the user interface to generate the integrated image. In an embodiment, each of the hand images that are nested on the user interface may be rendered in a translucent visual effect to prevent objects on the user interface from being obscured by the hand image. However, the present invention does not limit. In other embodiments, the integration module 132 can determine the visual effect of each of the integrated images according to the type of the object on the user interface.

Next, the processing unit 140 recognizes the gesture of the user based on each of the hand images by the recognition module 133 (step S207). In detail, the identification module 133 can first detect the position of the hand in each of the hand images. In an embodiment, the recognition module 133 can detect the hand in each of the hand images by using a feature extraction algorithm combined with a motion detection algorithm for detecting a moving object. The position of the part and the movement track are used to recognize the gesture operation of the user on the desktop. In addition, in an embodiment, the storage unit 130 may store a plurality of gesture template images that are trained in advance. (template image). The recognition module 133 can compare the gesture image of the hand image according to the movement trajectory of the hand image to identify the gesture operation of the user on the desktop.

Thereafter, the processing unit 140 acquires the position of the gesture in the hand image with respect to the user interface by using the mapping module 134 to generate a plurality of detection positions (step S209). In detail, there is a mapping relation between each of the hand images and the user interface. After the processing unit 140 obtains the position of the gesture in each of the hand images in step S207, the processing unit 140 obtains the position relative to the user interface, that is, the aforementioned “detection position”, by using the mapping relationship.

In an embodiment, the mapping relationship between each of the hand images and the user interface may be pre-stored in the form of a look-up table in the storage unit 130 of the electronic device 100. The lookup table is defined as the "first lookup table". The input index of the lookup table is the coordinates of each of the hand images, and the output of the lookup table is the coordinates of the user interface. In this embodiment, the aforementioned "coordinates" may be hand images and pixel coordinates in the user interface. Accordingly, the mapping module 134 can input the coordinates of the gestures on the hand images to the first lookup table to obtain the detection position on the user interface.

After the processing unit 140 obtains the detection position on the user interface by using the mapping module 134, the input module 135 generates an input signal corresponding to the gesture at the detection position, thereby operating the user interface (step S211). . In detail, since the processing unit 140 has utilized the detection position obtained by the mapping module 134, that is, The user may use the input module 135 to sequentially generate an input signal at the detection position according to the movement trajectory of the gesture, so that the gesture operation of the user on the desktop can be correspondingly performed. The user interface is generated by generating an input signal at the user interface.

It should be noted that the foregoing electronic device 100 provides an operation interface of the user on an interface other than the electronic device 100 such as a desktop to operate the object on the user interface. In an embodiment, the object on the user interface may be a virtual keyboard or a virtual mouse. An application example of the electronic device 100 will be described below using a virtual keyboard.

FIG. 3 is a schematic diagram of a usage scenario of an electronic device 100 according to an embodiment of the invention.

In this embodiment, the user interface is an email interface. In addition, the user interface includes a virtual keyboard. Referring to FIG. 1 and FIG. 3 simultaneously, when the processing unit 140 integrates the continuous image of the user interface 34 and the hand A (ie, the hand image A') by using the integration module 132 in step S205, an integrated image is generated. 30, wherein the user interface 34 further includes a virtual keyboard 32. In the present embodiment, the hand image A' is a translucent image such that the keys of the virtual keyboard 32 are not obscured by the hand image A'. In other embodiments, the virtual keyboard 32 can also be selectively rendered in a translucent manner.

Thereafter, the recognition module 133 recognizes the user's gesture as a keystroke based on the hand image A'. The processing unit 140 can use the mapping module 134 to obtain the pressing of the keyboard tap gesture in the hand image A' relative to the virtual keyboard 32. Key position. The input module 135 can generate an input signal at each of the button positions to operate the virtual keyboard. For example, when the keyboard tap gesture in the hand image A′ is the button G relative to the button position of the virtual keyboard 32, the input module 135 will generate an input signal on the button G of the virtual keyboard 32, and the user interface 34 The letter "G" will be displayed at the typing cursor.

The above embodiments mainly utilize a technique of pattern recognition to recognize a user's operation gesture. In another embodiment, the technique of frequency signal detection can be utilized to recognize the user's operation gesture.

4 is a block diagram of an electronic device in accordance with another embodiment of the present invention. Referring to FIG. 1 , the electronic device 400 of the present embodiment is, for example, an electronic device having a display function such as a tablet computer, a personal digital assistant, a smart phone, or an electronic book. The present invention is not limited to the type of the electronic device 400 . The electronic device 400 includes a screen 410, an image capturing unit 420, a transceiver unit 425, a storage unit 430, and one or more processing units 440. The image capturing unit 420 and the transceiver unit 425 are disposed on the back of the electronic device 400, and the processing unit 440 is coupled to the screen 410, the image capturing unit 420, the transceiver unit 425, and the storage unit 430.

The transceiver unit 425 includes a transmitter and a receiver. In this embodiment, the transceiver unit is an infrared transmitter/receiver. In other embodiments, the transceiver unit 430 can be a transceiver of other frequency signals, and the present invention is not limited thereto.

In this embodiment, the structure and functions of the screen 410, the image capturing unit 420, the storage unit 430, and the processing unit 440 are similar to the screen 110 and image capturing. The unit 120, the storage unit 130, and the processing unit 140 are the same as those described herein. The module stored in the storage unit 440 in this embodiment includes a display module 431, an integration module 432, a detection module 433, a mapping module 434, and a switching module 435. Similarly, these modules are, for example, computer programs that can be loaded into processing unit 440 to perform the functions of virtual input operations.

FIG. 5 is a flow chart of a method for virtual input operation according to another embodiment of the invention. Referring to FIG. 5, the method of the present embodiment is applied to the electronic device 400 of FIG. 4, and the detailed steps of the method for virtual input operation of the present invention are described below with the components in the electronic device 400.

Referring to FIG. 4 and FIG. 5, first, the processing unit 440 displays the user interface on the screen 410 by using the display module 431 (step S501). Next, the processing unit 440 uses the image capturing unit 420 to capture a plurality of consecutive hand images of the user (step S503). Then, the processing unit 440 integrates the user interface and each of the hand images with the integration module 432 to generate a plurality of integrated images, and then displays the integrated images on the screen 410 by using the display module 431 (step S505). Step S501, step S503, and step S505 are similar to step S201, step S03, and step S205. Please refer to the related description of the foregoing paragraphs, and details are not described herein again.

The processing unit 440 detects the user's gesture by the transceiver unit 425 by using the detecting module 433 (step S507). In this embodiment, the detection module 443 can detect and locate the gesture by the infrared light transmitted and reflected by the transceiver unit 425. In other words, the detecting module 443 can use the transceiver unit 425 to detect the moving trajectory of the gesture and the position of the gesture in the real space.

It should be noted that, in other embodiments, the processing unit 440 may first perform step S507 to detect the user's gesture by using the detecting module 433, and then perform step S503 and step S505 to integrate the generated. The image is displayed on screen 410, and the present invention is not limited thereto.

Thereafter, the processing unit 450 uses the mapping module 434 to obtain the position of the gesture in each of the hand images relative to the user interface to generate a plurality of detection positions (step S509). In detail, there is a positional mapping relationship between the position of the above gesture in the real space and the screen of the user interface. After the processing unit 440 obtains the position of the gesture in the real space in step S507, the processing unit 440 obtains the position relative to the user interface, that is, the aforementioned "detection position" by using the above mapping relationship.

In an embodiment, the mapping relationship between the gesture and the user interface may be pre-stored in the storage unit 430 of the electronic device 400 in the form of a lookup table, where the lookup table is defined as a “second lookup table”. . The input index of the lookup table is the actual coordinates of each of the gestures, and the output of the lookup table is the coordinates of the user interface (eg, pixel coordinates). Accordingly, the mapping module 434 can input the real coordinates of the gesture to the second lookup table to obtain the detected position on the user interface.

Next, the processing unit 450 will generate an input signal corresponding to the gesture by using the input module 435 at the detection position, thereby operating the user interface (step S511). Step S511 is similar to step S211. Please refer to the related description of the foregoing paragraphs, and details are not described herein again.

Similar to the usage scenario of FIG. 3, in an embodiment, the electronic device 400 The display module 410 also displays a virtual keyboard or a virtual mouse. The detection module 433 can detect the user's gesture as a keyboard tap gesture and directly locate the real coordinates of the keyboard tap gesture. The processing unit 440 can utilize the mapping module 434 to obtain the true position of the keyboard tap gesture relative to the key position of the virtual keyboard 32. The input module 435 can generate an input signal at each of the button positions to operate the virtual keyboard on the user interface.

In summary, the electronic device of the present invention can provide a virtual input operation to the user interface at a position other than the electronic device, and the gesture position is mapped to the user by recognizing the gesture and the position of the user. The relative position of the interface, which in turn produces a corresponding input signal at the relative position. Therefore, the gesture operation of the user at a position other than the electronic device can correspondingly generate an input signal on the user interface, thereby operating the user interface. In addition, the electronic device of the present invention further integrates the user interface and the hand image, so that the user can see the relative relationship between the hand and the object on the user interface on the screen. Accordingly, the user can achieve the effect of operating the electronic device close to using the physical input device without carrying an input device such as a physical keyboard or a mouse, and provide a more intuitive operation of the user.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

S201~S211‧‧‧Method of virtual input operation

Claims (6)

An electronic device comprising: a screen; an image capturing unit disposed on the back of the electronic device to capture a continuous plurality of hand images of a user; a storage unit for recording a plurality of modules; and one or a plurality of processing units coupled to the screen, the image capturing unit, and the storage unit to access and execute the module recorded in the storage unit, the module comprising: a display module, displaying a use The interface is integrated with the user interface and each of the hand images to generate a plurality of integrated images, wherein the display module displays each of the integrated images on the screen; an identification module Identifying a gesture of the user according to each of the hand images; and mapping a module to obtain a position of the gesture relative to the user interface in each of the hand images to generate a plurality of detection locations; An input module that generates an input signal corresponding to the gesture on the user interface to operate the user interface, wherein the storage unit includes a first lookup table, wherein the A lookup table index coordinate input image of each of the hand, the output of the first look-up table for the UI coordinates, the mapping module wherein the input to each of the hand gesture Movies The coordinates of the image are plotted on the first lookup table to obtain the detected position on the user interface. The electronic device of claim 1, wherein the storage unit comprises a plurality of gesture template images, and the recognition module compares the hand images according to the position and movement trajectory of the hand in the hand image. And the gesture template image is used to identify the gesture. The electronic device of claim 1, wherein the user interface comprises a virtual keyboard, and the recognition module recognizes the gesture of the user as a keyboard tap gesture according to each of the hand images. The mapping module obtains a button position of the keyboard tap gesture relative to the virtual keyboard in each of the hand images, and the input module generates the input signal to the button position of the virtual keyboard, thereby operating the virtual keyboard . An electronic device includes: a screen; an image capturing unit disposed on the back of the electronic device to capture a continuous plurality of hand images of a user; a transceiver unit disposed on the back of the electronic device, and Having a transmitter and a receiver; a storage unit for recording a plurality of modules; and one or more processing units coupled to the screen, the image capturing unit, the transceiver unit, and the storage unit for accessing Executing the module recorded in the storage unit, the module includes: a display module for displaying a user interface on the screen; an integrated module integrating the user interface and each of the hand images to generate a plurality of integrated images, wherein the display module displays each of the integrated images The detection module uses a transceiver unit to detect a gesture of the user, wherein the gesture corresponds to each of the hand images captured by the image capturing unit; a mapping module Obtaining a position of the gesture relative to the user interface to generate a plurality of detection locations; and an input module generating an input signal corresponding to the gesture to the detection location on the user interface, The user interface is configured to operate, wherein the storage unit includes a second lookup table, wherein an input index of the second lookup table is a coordinate of the gesture in the real space, and an output of the second lookup table is the use a coordinate of the user interface, wherein the mapping module inputs a coordinate of the gesture in the real space to the second lookup table to obtain the detected position of the user interface. The electronic device of claim 4, wherein the detecting module uses the transceiver unit to detect a movement trajectory of the gesture and a position of the gesture in a real space. The electronic device of claim 4, wherein the user interface comprises a virtual keyboard, and the detecting module identifies the gesture of the user as a keyboard tap gesture according to each of the hand images. The mapping module obtains a key position of the keyboard tap gesture relative to the virtual keyboard, and the input module generates the input signal The key position of the virtual keyboard is operated to operate the virtual keyboard.