CN101699387B - Non-touch interactive system and method - Google Patents

Abstract

The invention relates to a non-contact interaction system and a non-contact interaction method. A contactless display system enables a user to interact with a displayed image by moving a finger or pointer towards a selected portion of the image. The image may be dynamically magnified or translated in response to the detected movement. The method of operation can be manually switched between contact and non-contact operation to increase flexibility.

Description

Non-touch interactive system and method

technical field

The present invention relates to a system and method for navigating a small screen display. More specifically, the present invention relates to such systems and methods with non-contact sensors for tracking the trajectory of a user's fingers pointing at a virtual keyboard.

Background technique

Various types of small screen displays can be found on products such as cell phones, personal digital assistants (PDAs), mobile computers, and imagers. Increasingly, users must navigate small-screen displays to browse the web with a mobile phone or view photos with a PDA.

Small touch screens are common to support interaction with programs running on portable devices such as PDAs and mobile phones. Small touch screens are also finding their way into household products, such as Honeywell's TH8321U1006 thermostat, Honeywell's 6271V security panel, and various personal health monitoring devices. They have been used for many years in package delivery, retail warehousing and refinery field operations.

Known navigation systems use zoom and pan controls or via a fisheye viewer. However, using these controls is inconvenient and inefficient in some situations. For example, it is difficult to use fisheye navigation with a touchscreen. It's also difficult to zoom and pan graphics using a mouse or touchscreen. Navigation using a stylus can be more difficult for users on small screens.

There is therefore a continuing need for another natural and easier-to-use means of controlling the size and ratio of navigation graphics. There is therefore a continuing need for natural and easy-to-use alternative methods of controlling the navigation of large-scale and small-scale graphics.

It would also be desirable to be able to use this approach in navigating large scale graphics such as maps or building floor plans.

Contents of the invention

A display system comprising: a multi-dimensional display device on a surface on which an image may be presented; a plurality of non-contact sensors positioned adjacent the surface; and a control circuit coupled to the sensors and the display device, the circuit responsive to signals from the sensors signal determining the trajectory of the pointing member moving towards the region of the surface.

Description of drawings

Accompanying drawing 1 is the illustration of non-touch interactive system according to the present invention;

Accompanying drawing 2 is the block diagram of some software elements of the system among accompanying drawing 1;

Accompanying drawing 3 is the flowchart of illustration interaction method;

Accompanying drawing 4A, 4B have shown two kinds of different applications of the system in accompanying drawing 1;

Accompanying drawing 5 is non-touch mode input screen, comprises building name, street number and address, city, zip code, first alarm, floor number, detector;

Accompanying drawing 6 is the middle non-touch mode screen, including building name, street number and address, city, zip code, first alarm, floor number, detector;

Figure 7 is a non-touch mode area display with exit button, including building name, street number and address, city, zip code, first alarm, floor number, detector;

Figures 8A, 8B, and 8C illustrate various aspects of the touchless sensor of the system of Figure 1;

Accompanying drawing 9A, 9B, 9C have shown a form of virtual keyboard non-touch navigation, and in Fig. 9A, when finger is close enough to touch screen, will select and enlarge a box of visual keyboard according to the position of finger; And

Accompanying drawings 10A, 10B and 10C show another form of non-touch navigation of the virtual keyboard. In FIG. 10A, the virtual keyboard will be enlarged continuously according to the position of the fingers.

Detailed ways

Although embodiments of the invention may take many different forms, specific embodiments thereof are shown in the drawings and are described in detail herein with the understanding that the disclosure of the invention is by way of illustration of the principles of the invention, and that certain These are some of the best modes shown, and are not intended to limit the invention to the specific examples shown.

Embodiments of the invention include a touchless or contactless interface that senses the position of a user's finger or hand in three dimensions. In disclosed embodiments, multiple capacitive sensors may be arranged at the edge of the display device. The trajectory of a finger or hand pointing to a point on a virtual keyboard displayed on the device can be traced. This allows the system to predict the point on the display device screen that the user is trying to select before the finger actually touches the screen.

Z-axis finger position data can be used in accordance with the invention to control zooming or zooming out on eg a map display. Alternatively, the fisheye on the map display can be controlled by utilizing this non-touch pointing method. Several parameters of the fisheye, such as zoom ratio, zoom range, zoom shape (rectangle, rounded rectangle, ellipse, etc.) and the scale of the distortion edge around the fisheye can be modified in the process.

As the user moves his/her finger, the graphic content in the display is updated accordingly. Finger movements can also control zoom/span operations or fisheye effects on the map display. These processes are efficient and intuitive to the user.

The same method can also be used to control and interact with the virtual keyboard on the small screen display. It overcomes the longstanding problem associated with small virtual keyboards that the keys are always much smaller than a human fingertip.

Precise interaction requires magnification of only the target area of the keyboard (eg a small subset of some keys). The non-touch interface uses the z-axis data related to the position of the hands to infer the desired target area on the keyboard and automatically scales or magnifies the desired area on the virtual keyboard.

In some applications, input signals from non-touch devices may disrupt interactions that do not need to be non-touch. In one aspect of the invention, a number of different methods can be used to stop/start non-touch interactions intuitively and quickly. In one embodiment, the user's right hand may be used to point and control the zoom control or fisheye control, and the left hand may be used to operate buttons that start or stop touchless navigation. In this processing, the left hand can also be used to quickly change fisheye or zoom parameters on the fly, while the right hand makes pointing and dragging actions to provide a very effective two-handed interaction.

FIG. 1 shows a touchless or contactless interaction system 100 . The system 100 includes a programmable processing unit 151 connected by a data bus 152 to a touch screen input buffer 153 , a non-touch input buffer 154 , a display buffer 155 , a buffer 156 , and a storage unit or register 157 . The touch screen 158 is coupled with the processor through the touch screen buffer 153 . A non-touch sensing device 159 , such as a plurality of capacitive-based non-contact sensors, is coupled to the processor through a non-touch input buffer 154 .

Graphics display 160 is coupled to the processor through display buffer 155 . The display device 160 includes a display screen on which various images are presented. The non-touch sensor 159 is located around the periphery of the display screen of the display device 160 which will be discussed in more detail subsequently.

Input/output device 161 is coupled to the processor through input/output buffer 156 . Input/output devices may include any combination of devices that allow the system to interact with external information.

The storage unit 157 includes information and/or programs or executable software necessary for the processor 151 to implement a touchless interaction system. For example, display control software 157a may be stored in unit 157 in a computer readable form. Other system control software may also be stored in unit 157 .

FIG. 2 shows various software modules 200 of the system 100 executed by the processor 151 . Module 200 is stored in unit 157 in magnetic computer or optical computer readable form. The software 200 includes a command execution module 202, a command recognizer module 204, a data receiver 206, a graphics system display module 208, and a domain model 210, which provides information about displayed regions. The operation of the various modules will be discussed with respect to process 250 of FIG. 3 .

As shown in the flowchart of FIG. 3, at 252 data from a sensor such as sensor 158 or 159 is loaded from buffer 153, 154 into a respective receiver such as 206a, 206b. At 254, the data is loaded into input buffer 204b.

At 256, gesture analyzer 204a analyzes the data. At 258 , gesture analyzer 204 a transmits the system command to command execution 202 . At 260 , the command execution changes the state of the domain objects of the model 210 .

At 262 , command execution 202 notifies graphics system module 208 to change the state of the visual image on display 160 . At 264 , the graphics system module 208 updates the graphics on the display unit 160 . At 266, the command execution module 202 then updates the system status.

Figure 4A illustrates one embodiment of the present invention, a non-contact, navigational area display, such as may be used to estimate a building's alarm condition. In a second embodiment, Figure 4B shows a small display touchless navigation according to the present invention.

Figure 5 shows the display of the embodiment type or application of Figure 4a in an initial display state. In step 1, as shown, the user can tap the button to enter the touchless navigation state. FIG. 6 is a confirmation screen presented to the user on the display unit 160 . The user can enter the touchless navigation state as shown in step 3. FIG. 7 shows the screen presentation on the display unit 160 in the non-touch navigation state. Provides a button to exit the non-touch state.

FIGS. 8A , 8B, and 8C show various parts of features of the proximity sensor 159 arranged with respect to the periphery of the screen 160 a of the display unit 160 . As shown herein, sensor 159 defines an outer cross-sectional frusto-conical sensing region 160b and an inner region 160c.

When the user's finger or pointing device is located in the outer area 160b, the area display or map can be navigated or scrolled and zoomed. As the user's finger approaches the screen, in regions 160b and 160c, for example, the rendered image zooms from one level to a more detailed level. When the user's finger enters the inner area 160c, in one embodiment, the user can only zoom in and out on the map or display. Region 160c can help the end user zoom in/out the map or display smoothly and without jitter or jerk.

Figures 9A, 9B, 9C show the steps of a solution using the system 100 for navigating the virtual keyboard. Areas can be selected and enlarged by user deactivating keys. It is then possible for the user to activate different keys to select and expand the second area, and so on, until the desired entry is made.

Figures 10A, 10B, 10C illustrate the steps of another solution using the system 100 for navigating a virtual keyboard. In the embodiments of Figures 10A, 10B, 10C, any portion of the keyboard that the user's finger moves can be enlarged or magnified to enable the user to seamlessly activate consecutive groups of keys. It should be understood that embodiments of the present invention may be included in electronic devices such as wireless telephones, mobile computers, or imaging devices, all of which may have relatively small keypads.

From the foregoing, it should be noted that various changes and modifications are available without departing from the spirit and scope of the invention. It is to be understood that no limitation with respect to the particular devices described herein is intended or should be inferred. It is, of course, intended by the appended claims to cover all such changes which fall within the scope of such claims.

Claims (9)

1. A display system comprising: Multi-dimensional display devices capable of presenting images on their surfaces; a plurality of non-contact sensors located near the surface; and a control circuit coupled to the sensor and the display device, the circuit establishing a first multi-dimensional sensing region near a surface of the display device and a second multi-dimensional sensing region within the first multi-dimensional sensing region, the first multi-dimensional sensing region the multidimensional sensing region has a predetermined shape, and the second multidimensional sensing region has a different shape than the first multidimensional sensing region, and the circuitry determines an indication of movement toward a region of the surface in response to a signal from the sensor the trajectory of the part and, in response thereto, changing the image on the surface; and Wherein the image can be scrolled and zoomed when the pointing member is in the first area and can only be zoomed when in the second area. 2. The system of claim 1, comprising a display control circuit coupled to the control circuit, the display control circuit dynamically adjusting a magnification parameter of the image presented on the surface in response to the determined trajectory. 3. The system of claim 1, wherein the sensor comprises a capacitive sensor. 4. The system of claim 1, wherein the sensor is configured to define a cross-sectional conical region within which the indicating member can be sensed. 5. The system of claim 1, including a manual operating element coupled to a control circuit that switches between a non-contact mode and a contact-type mode of interacting with the image on the surface. 6. The system of claim 1, wherein the control circuit is responsive to a selected one of a contact-type mode or a non-contact mode of interaction with the image on the surface. 7. The system of claim 1, comprising a plurality of contact-type sensors associated with a screen of a display device. 8. The system of claim 1 including display management software stored on a computer readable medium coupled to the control circuit. 9. The system of claim 8, wherein the display management software, when executed by the control circuit, presents dynamically changing images on the surface of the display unit in response to the determined trajectory of the pointing member.

Images