CN102033656B - Gesture recognition method and interactive system using the method - Google Patents

Abstract

The invention discloses a gesture recognition method and an interactive system using the same, wherein the method comprises the following steps: continuously capturing the image window by using an image sensor; capturing shading shadow information of at least one indicator in the image window; when a single indicator is judged to be included according to the shading shadow information, calculating the position coordinate of the indicator relative to the interactive system according to the position of the shading shadow in the image window; and when a plurality of indicators are judged to be included according to the shading shadow information, performing gesture recognition according to the interrelation of shading shadows in the image window. The invention also provides an interactive system. In the gesture recognition method and the interactive system using the same, because the contact point coordinates of a plurality of indicators do not need to be calculated respectively, even if the indicators are mutually shielded relative to the image sensor, the gesture recognition can be correctly carried out.

Description

Gesture recognition method and interactive system using the method

technical field

The present invention relates to an interactive system, and in particular to a gesture recognition method and an interactive system using the method.

Background technique

Please refer to FIG. 1 , which shows a conventional touch control system 9 . The touch control system 9 includes a touch surface 90 and at least two cameras 91 and 92 , and the field of view of the cameras 91 and 92 includes the entire touch surface 90 . When the user touches the touch surface 90 with a finger, the cameras 91 and 92 can capture an image window including the light and shadow of the tip of the finger. The processing unit can calculate the two-dimensional position coordinates of the finger touching the touch surface 90 according to the light-shading position of the fingertip in the image window, and perform corresponding operations on the relative control display according to the change of the two-dimensional position coordinates. action.

However, the operating principle of the touch control system 9 is to calculate the two-dimensional position coordinates of the finger touching the touch surface 90 according to the position of the light shadow of the fingertip in each image window. When the user uses multiple fingers to touch the touch surface 90 , compared to the camera 92 , since the fingers will occlude each other, the image window captured by the camera 92 may not necessarily show the shading light and shadow of all fingertips.

For example, in FIG. 1 , the user uses fingers 81 and 82 to touch the touch surface 90, at this time camera 91 captures image window W ₉₁ , which includes shadows I 81 and I 82 of fingers ₈₁ and ₈₂ ; however, due to finger 81 and 82 occlude each other with respect to the camera 92 , so the image window W ₉₂ captured by the camera 92 only includes one occluded light and shadow. When the processing unit calculates the two-dimensional position coordinates of the finger touching the touch surface 90 according to the image windows W ₉₁ and W ₉₂ , the two-dimensional position coordinates may not be calculated correctly, resulting in malfunctions.

To solve this problem, two cameras 93 and 94 can be set at the other two corners of the touch surface 90 to capture two image windows W ₉₃ and W ₉₄ , and the processing unit can use the image windows W 91 and W ₉₄ The W ₉₃ respectively calculates the two-dimensional position coordinates of the fingers 81 and 82 touching the touch surface 90 . However, such a solution increases system cost.

Contents of the invention

In view of this, the purpose of the present invention is to propose a gesture recognition method and an interactive system using the method to solve the problems existing in the above-mentioned existing touch system, which is based on the continuous image window captured by the image sensor Gesture recognition based on the interrelationship between shading light and shadow can solve the problem that the contact point coordinates cannot be calculated correctly due to mutual occlusion of pointers.

The present invention proposes a gesture recognition method for an interactive system. The interactive system includes an image sensor, a reflective element, and at least one light source. The resulting image window that blocks light and shadow. The gesture recognition method includes the following steps: using the image sensor to capture an image window; capturing shading light and shadow information in the image window; determining whether multiple pointers are included according to the shading light and shadow information; and when determining When multiple pointers are included, gesture recognition is performed based on the interrelationship of occluded light and shadow in consecutive image windows.

In an embodiment of the gesture recognition method of the present invention, the occlusion light and shadow information includes an average number of light shadows, an average distance between light and shadows and/or a maximum distance between light and shadows.

In an embodiment of the gesture recognition method of the present invention, the step of performing gesture recognition according to the mutual relationship of shading light and shadow in the continuous image window further includes the following steps: comparing one of the average number of light shadows and the average light shadow distance with a preset threshold value comparison; when the average light and shadow number or the average light and shadow distance is greater than the preset threshold value, perform up, down, left, right, zoom in or zoom out gesture recognition; when the average light and shadow number or the average light and shadow distance is less than the When the threshold value is preset, the rotating gesture is recognized; and the display screen of the image display is updated according to the recognized gesture.

The present invention further provides an interactive system, which includes a light emitting unit, an image sensor and a processing unit. The image sensor is used to continuously capture at least one image window in which the light and shadow formed by the indicator shielding the light-emitting unit are blocked. The processing unit performs gesture recognition according to the interrelationship of occluded light and shadow in continuous image windows captured by the image sensor.

In an embodiment of the interactive system of the present invention, the light emitting unit is an active light source or a passive light source. When the light emitting unit is a passive light source, the light emitting unit includes a reflective mirror and the interactive system further includes at least one active light source.

The present invention further proposes a gesture recognition method for an interactive system, the interactive system includes a light emitting unit and an image sensor, and the image sensor is used to capture the shading light and shadow formed by covering the light emitting unit with a plurality of pointers image window. The gesture recognition method includes the following steps: using the image sensor to continuously capture image windows; and performing gesture recognition according to the interrelationships of multiple shading lights and shadows in the continuous image windows.

In an embodiment of the gesture recognition method of the present invention, the interrelationship between the shading lights and shadows includes the average light-shadow distance change, the maximum light-shadow distance change, and the displacement direction of the shading lights and shadows.

According to the gesture recognition method and the interactive system using the method of the present invention, in the first mode, the interactive system controls the action of the cursor according to the change of the two-dimensional coordinates of the pointer; in the second mode, the interactive system controls the movement of the cursor according to the multiple The interrelationship between the shading light and shadow of each indicator updates the display screen of the display, such as scrolling the display screen, zooming in/out of objects, rotation of objects, switching screens or menus, etc. In the gesture recognition method of the present invention and the interactive system using the method, since there is no need to calculate the contact point coordinates of multiple pointers, gestures can be performed correctly even if the pointers are mutually occluded with respect to the image sensor identify.

Description of drawings

FIG. 1 shows a schematic diagram of an existing touch control system;

Figure 2a shows a perspective view of an interactive system according to an embodiment of the present invention;

Fig. 2b shows a schematic diagram of the operation of the interactive system of the first embodiment of the present invention;

Fig. 3a shows a schematic diagram of cursor manipulation using the interactive system of the first embodiment of the present invention;

FIG. 3b shows a schematic diagram of an image window captured by the image sensor of FIG. 3a;

Fig. 4a shows the flowchart of the gesture recognition method of the interactive system according to the embodiment of the present invention;

Figure 4b shows a flowchart of the second mode in Figure 4a;

5a-5d respectively show schematic diagrams of recognizing right/left/down/up gestures in the gesture recognition method of the interactive system according to the first embodiment of the present invention;

5e to 5f are schematic diagrams of recognizing zoom-in/zoom-out gestures in the gesture recognition method of the interactive system according to the first embodiment of the present invention;

5g to 5h respectively show schematic diagrams of recognizing rotation gestures in the gesture recognition method of the interactive system according to the first embodiment of the present invention;

Figure 6a shows a schematic diagram of the operation of the interactive system of the second embodiment of the present invention;

6b-6c show schematic diagrams of image windows captured by the image sensor of FIG. 6a;

7a-7b respectively show schematic diagrams of recognizing right/left gestures in the gesture recognition method of the interactive system according to the second embodiment of the present invention;

7c to 7d respectively show schematic diagrams of recognizing zoom-in/zoom-out gestures in the gesture recognition method of the interactive system according to the second embodiment of the present invention; and

7e to 7f respectively show schematic diagrams of recognizing rotation gestures in the gesture recognition method of the interactive system according to the second embodiment of the present invention.

Description of main component symbols

10, 10′ interactive system 100 panels

100a First side of panel 100b Second side of panel

100c 3rd side of panel 100d 4th side of panel

100d 4th mirror image 100s surface of panel

11 luminous unit 11a reflective mirror

121 First light source 121′ Second mirror image

122 Second light source 122′ Third mirror image

13. 13′ image sensor 14 processing unit

15 Graphic Display 150 Display

151 Cursor 20, 20′, 20″ image window

RS real image space IS virtual image space

T ₈₁ , the point of contact of the T indicator T ₈₁ ′, the point of contact of the first mirror image of T′

A ₈₁ The angle between the contact point and the third side A ₈₁ ′ The angle between the first mirror image and the third side

R ₈₁ first sensing path R ₈₁ ' second sensing path

I ₈₁ , I ₈₂ first shading light and shadow I ₈₁ ′ second shading light and shadow

I ₁ , I ₂ the first shading light and shadow I ₁ ′, I ₂ ′ the second shading light and shadow

I ₈₁ ″, I ₈₂ ″ Shading light and shadow G ₁ first light and shadow group

G ₂ Second light and shadow group C Center line

Sav Average distance between light and shadow 8 User

81, 82 Finger Touch System 9 Touch System

91～94 Camera 90 Touch Surface

W ₉₁ ～W ₉₄ image window S ₁ ～S ₅ steps

Detailed ways

In order to make the above and other objects, features and advantages of the present invention more apparent, a detailed description will be given below with reference to the accompanying drawings. In addition, it should be noted that in the description of the present invention, the same members are denoted by the same symbols.

Please refer to FIG. 2a and FIG. 2b at the same time. FIG. 2a shows a perspective view of the interactive system 10 according to the embodiment of the present invention, and FIG. 2b shows a schematic diagram of the operation of the interactive system 10 according to the first embodiment of the present invention. The interactive system 10 includes a panel 100 , a light emitting unit 11 , a first light source 121 , a second light source 122 , an image sensor 13 , a processing unit 14 and an image display 15 .

The panel 100 includes a first side 100a, a second side 100b, a third side 100c, a fourth side 100d and a surface 100s. Examples of the panel 100 include a whiteboard or a touch screen.

The light emitting unit 11 is disposed on the surface 100s of the first side 100a of the panel 100 . The light emitting unit 11 can be an active light source or a passive light source. When the light emitting unit 11 is an active light source, it can emit light by itself and the light emitting unit 11 is preferably a line light source. When the light emitting unit 11 is a passive light source, it is used to reflect the light emitted by other light sources (such as the first light source 121 and the second light source 122); at this time, the light emitting unit 11 includes a reflective mirror surface 11a facing the third side 100c of the panel , wherein the reflective mirror surface 11a can be formed with appropriate materials. The first light source 121 is disposed on the surface 100s of the second side 100b of the panel, and preferably emits light toward the fourth side 100d of the panel. The second light source 122 is disposed on the surface 100s of the third side 100c of the panel, and preferably emits light toward the first side 100a of the panel; wherein the first light source 121 and the second light source 122 are preferably active light sources, such as wires light source, but not limited thereto.

Please refer to FIG. 2b again. When the light emitting unit 11 is a passive light source (such as a reflective element), the first light source 121 can map a second mirror image 121' relative to the reflective mirror surface 11a, and the second light source 122 can reflect the second mirror image 121' relative to the reflective mirror surface 11a. A third mirror image 122' is mapped out, and the fourth side 100d of the panel can be mapped to a fourth mirror image 100d' relative to the reflective mirror surface 11a; wherein the light emitting unit 11, the first light source 121, the second light source 122 and the fourth side of the panel 100d jointly define a real image space RS; the light emitting unit 11, the second mirror image 121', the third mirror image 122' and the fourth mirror image 100d' jointly define a virtual image space IS.

The image sensor 13 is disposed at a corner of the panel 100 , in this embodiment, the image sensor 13 is disposed at a corner where the third side 100c and the fourth side 100d of the panel meet. The field of view VA of the image sensor 13 includes at least the real image space RS and the virtual image space IS, for capturing images including the real image space RS, the virtual image space IS, and a pointer (pointer) located in the real image space RS, such as a finger 81. Image window with shaded shadows. In one embodiment, the image sensor 13 includes a lens (or lens group) for adjusting the field of view VA of the image sensor 13, so that the image sensor 13 can capture the real image space RS and Complete image of virtual image space IS. Examples of the image sensor 13 include, but are not limited to, CCD image sensors and CMOS image sensors.

The processing unit 14 is coupled to the image sensor 13 and is used for processing images captured by the image sensor 13 to identify one or more pointers. When only one pointer is recognized, the two-dimensional position coordinates of the pointer touching the panel surface 100s are relatively calculated according to the position of the shaded light and shadow of the pointer in the image window. When multiple pointers are identified, the processing unit 14 performs gesture recognition according to the relationship between the shading light and shadow of the pointers in the image window, and controls the image display to update its display screen according to the recognized gestures. The detailed calculation method Will be described in detail later.

The image display 15 is coupled to the processing unit 14, and a cursor 151 may be displayed on a display screen 150 of the image display 15, as shown in FIG. 2b. The processing unit 14 relatively controls the movement of the cursor 151 on the display screen 150 according to the calculated change in the two-dimensional position coordinates of the pointer touching the panel surface 100s, or according to the multiple shaded lights and shadows in the image window captured by the image sensor 13 The interrelationships between the two elements update the display screen of the display screen 150, such as display screen scrolling, object zooming, object rotation, screen switching or menus, and the like.

In order to clearly show the interactive system of the present invention, in Fig. 2a and Fig. 2b, the panel 100 is independent from the image display 15, but it is not used to limit the present invention, in other embodiments, the panel 100 can also be combined on the display screen 150 of the image display 15 . In addition, when the panel 100 is a touch screen, the display screen 150 of the image display 15 can also be used as the panel 100, and the light emitting unit 11, the first light source 121, the second light source 122 and the image sensor 13 are It is arranged on the surface of the display screen 150 .

It can be understood that although in FIG. 2a and FIG. 2b, the panel 100 is shown as a rectangle and the light emitting unit 11, the first light source 121 and the second light source 122 are shown as three vertically arranged on the panel 100. side, but it is only an embodiment of the present invention and is not intended to limit the present invention. In other embodiments, the panel 100 can be made into other shapes; the light emitting unit 11 , the first light source 121 , the second light source 122 and the image sensor 13 can also be disposed on the panel 100 in other spatial relationships. The spirit of the present invention is to use the image sensor 13 to capture the image window, perform gesture recognition according to the displacement of the shading light and shadow in the image window and the relationship between the shading light and shadow, and relatively update the image display according to the recognized gesture display screen.

first embodiment

Please refer to FIG. 3a and FIG. 3b. FIG. 3a shows a schematic diagram of cursor manipulation using the interactive system 10 according to the first embodiment of the present invention; FIG. 3b shows the image window 20 captured by the image sensor 13 in FIG. 3a schematic diagram. As shown in the figure, when a pointing object, such as the tip of a finger 81, touches the panel surface 100s in the real image space RS, represented by a contact point _T81 here, the pointing object is relative to the light emitting unit 11 (reflective in this embodiment). The reflective mirror surface 11a of the element) maps the first mirror image in the virtual image space IS, which is represented by the contact point T ₈₁ ′ here. The image sensor 13 captures the tip image of the pointer according to the first sensing route R ₈₁ to form a first shadow I ₈₁ in the image window 20; and captures the second sensing route R ₈₁ ' according to the second A mirror image of the tip to form a second shadow I ₈₁ ′ in the image window 20, as shown in FIG. 3b. In this embodiment, the processing unit 14 prestores the relative relationship between the one-dimensional position of the shading light and shadow in the image window 20 and the angle between the sensing line and the third side 100c of the panel. Therefore, when the image sensor 13 captures the tip image of the pointer and its first mirror image to form the image window 20, the processing unit 14 can respectively calculate the first included angle according to the one-dimensional position of the shaded light and shadow of the image window 20. A ₈₁ and the second included angle A ₈₁ ′. Next, according to the trigonometric function relationship, the processing unit 14 can obtain the two-dimensional position coordinates of the touch point T ₈₁ where the pointer touches the panel surface 100s.

For example, in one embodiment, the panel surface 100s forms a rectangular coordinate system, the third side 100c serves as the X-axis of the rectangular coordinate system, the fourth side 100d serves as the Y-axis of the rectangular coordinate system, and the image sensor 13 position as the origin. Therefore, the coordinates of the contact point T ₈₁ in the Cartesian coordinate system can be expressed as (distance from the fourth side 100d, distance from the third side 100c). In addition, the distance D ₁ between the first side 100 a and the third side 100 c of the panel is pre-stored in the processing unit 14 . Thereby, the processing unit 14 can obtain the two-dimensional position coordinates of the touch point T ₈₁ where the indicator 81 touches the panel surface 100s according to the following steps: (a) The processing unit 14 obtains the first sensing route R ₈₁ and the panel The first included angle A ₈₁ between the third side 100c and the second included angle A ₈₁ ′ between the second sensing route R ₈₁ ′ and the third side 100c of the panel; (b) according to the equation D ₂ =2D ₁ /(tanA ₈₁ +tanA ₈₁ ′) Calculate the distance D ₂ between the touch point T ₈₁ of the pointer 81 and the fourth side 100d of the panel; (c) Calculate the touch point T ₈₁ according to D ₂ ×tanA ₈₁ the y-coordinate. Therefore, the two-dimensional position coordinates of the contact point T ₈₁ are (D ₂ , D ₂ ×tanA ₈₁ ).

Please refer to FIG. 3a and FIG. 3b again, the operation of the interactive system 10 according to the first embodiment of the present invention includes two modes. When the processing unit 14 determines according to the image window 20 captured by the image sensor 13 that only one pointer touches the panel surface 100s, the interactive system 10 is controlled to work in the first mode. In the first mode, the image sensor 13 continuously captures images at a sampling frequency, and the processing unit 14 calculates the one-dimensional position of the indicator 81 in the image window 20 according to the occluded light and shadow of the indicator 81, and calculates the 100 s when the indicator 81 touches the surface of the panel. The two-dimensional position coordinates of the touch point _T81 , and according to the change of the two-dimensional position coordinates of the touch point _T81 , the movement of the cursor 151 on the image display 15 is relatively controlled. For example, when the indicator 81 moves toward the fourth side 100d of the panel, the contact point T ₈₁ ′ of the first mirror image also moves toward the fourth mirror image 100d ′ at the same time. At this time, the shading light and shadow I ₈₁ corresponding to the indicator and the shading light and shadow I ₈₁ ′ corresponding to the first mirror image in the image window 20 also move toward the left side of the image window 20 . In this way, the processing unit 14 calculates the two-dimensional position coordinates of the touch point T ₈₁ according to the positions of the shading light and shadow I ₈₁ and I ₈₁ ′ in each image window 20, and according to the change of the two-dimensional position coordinates, relatively The cursor 151 on the image display 15 is controlled to move toward the left of the display screen 150 . It can be understood that the relationship between the moving direction of the pointer and the moving directions of the shadows I ₈₁ and I ₈₁ ′ in the image window 20 and the moving direction of the cursor 151 is not limited to the content disclosed in the above-mentioned embodiments, The moving direction of the shading light and shadow I ₈₁ and I ₈₁ ′ and the cursor 151 may be opposite to the moving direction of the pointer according to different processing methods of the software.

When the processing unit 14 determines according to the image window 20 captured by the image sensor 13 that there are multiple pointers touching the panel surface 100s, the interactive system 10 is controlled to work in the second mode. In the second mode, the processing unit 14 no longer calculates the two-dimensional position coordinates of each touch point T ₈₁ according to each image window 20 one by one, but only determines the gesture (gesture) according to the mutual relationship of multiple shading lights and shadows in the image window 20 , and update the display screen of the display screen 150 of the image display 15 according to the determined gesture, such as scrolling the screen, zooming in and out of objects, rotating objects, switching screens, or displaying menus.

Please refer to FIG. 4a, which shows a flowchart of the gesture recognition method of the present invention. The method includes the following steps: using an image sensor to capture an image window (step S ₁ ); capturing shading light and shadow information in the image window (step S ₂ ); judging whether the shading light and shadow information includes multiple indicator (step S ₃ ); if not, enter the first mode (step S ₄ ); if yes, enter the second mode (step S ₅ ).

Please refer to FIG. 4 b , which shows the implementation of the second mode in step S ₅ in FIG. 4 a , the shading shadow information includes the average number of shadows, the average distance between shadows and the maximum distance between shadows. The second mode includes the following steps: determine whether one of the average light and shadow number and the average light and shadow distance is greater than a preset threshold value (step _S51 ); if so, perform rotation gesture recognition according to the relationship between shading light and shadow in the continuous image window (step S ₅₂ ); if not, perform up/down/left/right/zoom in/zoom out gesture recognition according to the interrelationships of shading light and shadow in the continuous image window (step _S53 ); and update the display screen of the image display according to the recognized gesture ( Step _S54 ). It can be understood that, in Fig. 4b, it can be set to perform rotation gesture recognition when the average light and shadow number is less than the preset threshold value, and to perform translation gesture recognition when the average light and shadow number is greater than the preset threshold value; or set to when the average light and shadow distance The rotation gesture recognition is performed when the threshold is smaller than the preset threshold value, and the translation gesture recognition is performed when the average light-shadow distance is greater than the preset threshold value.

In another embodiment, the second mode may only include one step: performing rotation gesture recognition according to the relationship between occluded light and shadow in consecutive image windows. In another embodiment, the second mode may only include one step: performing up/down/left/right/zoom in/zoom out gesture recognition according to the interrelationship of occluded light and shadow in consecutive image windows. That is to say, the second mode of the interactive system can only perform one of the recognition of the rotation gesture or the recognition of the up/down/left/right/zoom in/zoom out gesture.

Please also refer to FIGS. 3a-4b. When using the interactive system 10 of the first embodiment of the present invention for gesture recognition, firstly use the image sensor 13 to capture an image to form an image window 20, which includes at least one corresponding to the indication. The shading light shadow I ₈₁ of the object contact point T ₈₁ and at least one shading light shadow I ₈₁ ′ corresponding to the first mirror image contact point T ₈₁ ′ (step S ₁ ). Next, the processing unit 14 retrieves shading information in the image window 20, such as the average number of shading shadows, the average distance between shadows and the maximum distance between shadows, etc., for use in subsequent steps (step S ₂ ). Next, the processing unit 14 determines whether there are multiple indicators in the image window 20 according to the captured shading light and shadow information (step S ₃ ), wherein each indicator will generate a maximum of 2 shading lights and shadows on the image window 20, so when When more than two shadows appear on the image window 20, it means that there are multiple pointers.

When it is determined that only one pointer is included, as shown in FIG. 3a and FIG. 3b , the processing unit 14 controls the interactive system 10 to enter the first mode (step S ₄ ). In the _first mode, the processing unit 14 calculates the point of contact of the pointer touching the panel surface _100s (eg T ₈₁ ), and relatively control the movement of the cursor 151 on the image display 15 according to the change of the two-dimensional position coordinates.

When the processing unit 14 determines that there are multiple indicators touching the panel surface 100s according to the shading light and shadow information, as shown in FIGS. 5a to 5h , the interactive system 10 is controlled to enter the second mode (step S ₅ ). In the second mode, the processing unit 14 performs gesture recognition according to the mutual relationship between shading lights and shadows in the image window 20, and controls the screen update of the screen displayed on the display screen 150 of the image display 15 according to the recognized gesture, for example Perform screen scrolling, zoom in and out of objects or windows, rotate objects, switch screens or display menus, etc.

Referring to FIG. 5 a to FIG. 5 h , the implementation of the second mode will be described next, wherein the light emitting unit 11 is described by taking a passive light source as an example in this description. In addition, it can be understood that what is shown in Fig. 5a to Fig. 5h is only illustrative and not intended to limit the present invention.

Screen scrolling gesture: Please refer to FIGS. 5a-5d, when the processing unit 14 determines that there are multiple touch points (such as T ₁ and T ₂ ) according to the shading light and shadow information in the image window 20 captured by the image sensing unit 13 , enter the second mode. Next, the processing unit 14 determines whether the average number of shading lights and shadows in the image window 20 is greater than a preset threshold value, such as 6, or whether the average light-shadow distance Sav is greater than a preset threshold value (step S ₅₁ ). When the average number of shading lights and shadows is not greater than the preset threshold value or the average light-shadow distance Sav is not greater than the preset threshold value, pan gesture recognition is performed (step S ₅₃ ).

When recognizing the translation gesture, firstly group the occluded light and shadow into groups, for example, according to the center line C of the image window 20 as the basis for grouping, so as to distinguish the first light and shadow group G ₁ from the second light and shadow group G ₂ , wherein the first light and shadow group G ₁ may be a real image light and shadow group or a virtual image light and shadow group, and the second light and shadow group G ₂ may be a virtual image light and shadow group or a real image light and shadow group.

For example, in FIGS. 5 a to 5 d, when the average light and shadow number in the image window 20 is not greater than the preset threshold value or the average light and shadow distance Sav is not greater than the preset threshold value, the processing unit 14 performs up/down/left/right gesture recognition (step _S53 ). For example, in FIG. 5a, the processing unit 14 recognizes that both the first light and shadow group _G1 and the second light and shadow group _G2 in the image window 20 are moving to the right, so it is determined that the user is performing a gesture of scrolling the display screen to the right/left , so that the corresponding display screen is updated relative to the display screen 150 of the control image display 15 (step S ₅₄ ).

Similarly, in FIG. 5b, the processing unit 14 recognizes that both the first light and shadow group _G1 and the second light and shadow group _G2 in the image window 20 are moving to the left, so it is determined that the user is performing scrolling of the display screen to the left/right Therefore, the corresponding display screen is updated relative to the display screen 150 of the control image display 15 (step S ₅₄ ).

In FIG. 5c, the processing unit 14 recognizes that the average light-shadow distance between the first light-shadow group _G1 and the second light-shadow group _G2 in the image window 20 is gradually increasing, so it is determined that the user is performing the scrolling down/up of the display screen. Gestures, so the corresponding display screen update is performed relative to the control display screen 150 (step S ₅₄ ).

In FIG. 5d , the processing unit 14 recognizes that the average light-shadow distance between the first light-shadow group _G1 and the second light-shadow group _G2 in the image window 20 is gradually decreasing, so it is determined that the user is performing the gesture of scrolling the display screen up/down , so the corresponding display screen is updated relative to the control display screen 150 (step S ₅₄ ).

In another embodiment, when the processing unit 14 determines that there are multiple contact points according to the shading light and shadow information in the image window 20 captured by the image sensing unit 13, it directly performs translation gesture recognition (step S ₅₃ ) instead of Execute step _S51 .

Object zoom-in and zoom-out gestures: Before performing the steps of zooming in and out of objects, the user first forms a single contact point on the panel surface 100s to enter the first mode, and controls the cursor 151 to move to the object O in the first mode, as shown in the figure 3a shown. Next, the user forms a plurality of contact points on the panel surface 100s, as shown in FIGS. 5e-5f. When the processing unit 14 determines that there are multiple contact points (such as T ₁ , T ₂ ) according to the shading light and shadow information in the image window 20 captured by the image sensing unit 13 , it enters into the second mode.

Next, the processing unit 14 determines whether one of the average light shadow number and the average light shadow distance Sav in the image window 20 is greater than a preset threshold (step S ₅₁ ). When the average number of lights and shadows is not greater than the preset threshold value or the average light and shadow distance Sav is not greater than the preset threshold value, first group the shaded lights and shadows, for example, according to the centerline C of the image window 20 as the basis for grouping, to distinguish the first light and shadow group G ₁ and the second light and shadow group G ₂ .

For example, in FIGS. 5e-5f, if the average number of light shadows in the image window 20 is not greater than the preset threshold value or the average light shadow distance Sav is not greater than the preset threshold value, the processing unit 14 performs zoom-in/zoom-out gesture recognition (step S ₅₃ ). For example, in FIG. 5e, the processing unit 14 recognizes that the average distance between the first light-shadow group _G1 and the second light-shadow group _G2 in the image window 20 does not change substantially, but the maximum light-shadow distance increases. Therefore, it is determined that the user is performing the operation of moving the object. The gesture of zooming in/zooming out, thus correspondingly updates the display screen of the display screen 150 of the control image display 15 (step S ₅₄ ).

For example, in FIG. 5f, the processing unit 14 recognizes that the average distance between the first light-shadow group _G1 and the second light-shadow group _G2 in the image window 20 has substantially no change and the maximum light-shadow distance has decreased. Therefore, it is determined that the user is performing the operation of moving the object. The gesture of zooming out/zooming in, thus correspondingly updating the display screen relative to the display screen 150 of the control image display 15 (step S ₅₄ ).

In another embodiment, when the processing unit 14 determines that there are multiple contact points according to the shading light and shadow information in the image window 20 captured by the image sensing unit 13, the zoom-in/zoom-out gesture recognition is performed directly (step _S53 ) Step S ₅₁ is not executed.

In addition, before performing zoom-in/zoom-out gesture recognition, it is also possible to directly perform the second mode without entering the first mode. When identifying, you can directly enter the second mode.

Object rotation gesture: before performing the step of object rotation, the user first forms a single contact point on the panel surface 100s to enter the first mode, and controls the cursor 151 to move to the object O in the first mode, as shown in FIG. 3a Show. Next, the user forms multiple contact points T on the panel surface 100s, as shown in FIGS. When a contact point T is reached, the second mode is entered.

Next, the processing unit 14 determines whether one of the average light shadow number and the average light shadow distance Sav in the image window 20 is greater than a preset threshold (step S ₅₁ ). When the average number of shadows is larger than the preset threshold value or the average distance Sav between shadows is larger than the preset threshold value, the grouping of the shadows is not performed, and the rotation direction is directly determined according to the number of shadows displaced toward the two sides of the image window 20 .

For example, in FIG. 5g, the processing unit 14 recognizes that the number of shadows moving to the right in the image window 20 is greater than the number of shadows moving to the left, so it is determined that the user is performing a gesture of rotating the object to the right/left, and thus relatively controls the image display. 15. Update the corresponding display screen (step S ₅₄ ).

For example, in FIG. 5h, the processing unit 14 recognizes that the number of shadows moving to the left in the image window 20 is greater than the number of shadows moving to the right, so it is determined that the user is performing a gesture of rotating the object to the left/right, and thus relatively controls the image display. 15. Update the corresponding display screen (step S ₅₄ ).

In another embodiment, when the processing unit 14 determines that there are multiple contact points according to the shading light and shadow information in the image window 20 captured by the image sensing unit 13, it directly performs rotation gesture recognition (step S ₅₂ ) instead of Execute step _S51 .

In addition, it is also possible to directly enter the second mode without entering the first mode before performing the rotation gesture recognition. For example, when the panel 100 is a touch panel, since the user can directly point to the object, it can directly perform the rotation gesture recognition. Enter the second mode.

Screen switching gesture: the user directly forms multiple contact points T on the panel surface 100s, as shown in Figures 5g-5h, when the processing unit 14 judges the When multiple contact points T are included, enter the second mode directly.

The processing unit 14 directly determines whether to perform screen switching according to the number of shading lights and shadows displaced toward the two sides of the image window 20 . For example, in FIG. 5g and FIG. 5h, the processing unit 14 recognizes that the number of shadows moving to the right or to the left in the image window 20 is greater than the number of shadows moving to the left or to the right, so it is determined that the user is performing a screen switching gesture, Therefore, the corresponding screen switching function is performed relative to the control image display 15 .

Display menu gesture: the user directly forms multiple contact points T on the panel surface 100s, as shown in Figures 5g-5h, when the processing unit 14 determines the When multiple contact points T are included, enter the second mode directly.

The processing unit 14 directly determines whether to display the menu according to the number of shading lights and shadows displaced toward the two sides of the image window 20 . For example, in FIG. 5g, the processing unit 14 recognizes that the number of occlusion light shadows moving to the right in the image window 20 is greater than the number of occlusion light shadows moving to the left; in FIG. 5h, the processing unit 14 recognizes the number of occlusion light shadows moving to the left in the image window 20 If it is greater than the number of shadows moving to the right, it is determined that the user is performing a gesture to display a menu, and thus the corresponding menu is displayed on the control image display 15 .

second embodiment

Please refer to FIGS. 6a to 6c. FIG. 6a shows a schematic diagram of the operation of the interactive system 10' according to the second embodiment of the present invention. The schematic diagram of the image windows 20' and 20" taken. In this embodiment, the interactive system 10' includes a light emitting unit 11, a first light source 121, a second light source 122, and image sensors 13 and 13'. The light emitting unit 11 is active light source, and preferably emit light towards the third side 100c of the panel. The light emitting unit 11, the first light source 121 and the second light source 122 are respectively arranged on the first side 100a, the second side 100b and the fourth side 100d of the panel. Therefore, the image The image window 20 ′ captured by the sensor 13 includes only the first shadows I ₈₁ and I ₈₂ of the tip of the pointer; the image window 20 ″ captured by the image sensor 13 ′ only includes the tip of the pointer. Shade light and shadow I ₈₁ ″ and I ₈₂ ″.

In this embodiment, the processing unit 14 performs gesture recognition according to the interrelationships of multiple shading lights and shadows in the image windows 20 ′ and 20 ″ captured by the image sensors 13 and 13 ′.

Screen scrolling gesture: please refer to FIGS. For example T ₁ and T ₂ ), then enter the second mode. Then, the processing unit 14 determines whether the average light and shadow number in the image windows 20' and 20 "is greater than a preset threshold value or determines whether the number of light shadows in the image windows 20' and 20 " Whether the average distance between light and shadow is greater than a preset threshold (step S ₅₁ ).

When the average light and shadow number in the image windows 20' and 20" is not greater than the preset threshold or the average light and shadow distance is not greater than the preset threshold, the processing unit 14 performs left/right gesture recognition (step _S53 ). For example, FIG. 7a 7b, the processing unit 14 respectively recognizes that the shading lights and shadows in the image windows 20' and 20" are moving to the right or to the left, so it is determined that the user is performing the gesture of scrolling the display screen down/up, so the relative Control the display screen 150 of the image display 15 to update the corresponding display screen (step S ₅₄ ).

In another embodiment, when the processing unit 14 determines that there are multiple contact points according to the shading light and shadow information in the image windows 20 and 20 ″ captured by the image sensing units 13 and 13 ′, it directly performs translation gesture recognition ( Step S ₅₃ ) Step S ₅₁ is not executed.

Object zoom-in and zoom-out gestures: before performing the steps of zooming in and out, the user controls the cursor to move to the object to be zoomed in and out. Next, the user forms a plurality of contact points (such as T ₁ and T ₂ ) on the panel surface 100s, as shown in FIGS. 7c-7d. When the processing unit 14 determines that there are multiple contact points according to the shading light and shadow information in the image windows 20 ′ and 20 ″ captured by the image sensing units 13 and 13 ′, the second mode is entered.

Next, the processing unit 14 determines whether the average number of shadows in the image windows 20 and 20" is greater than a preset threshold or whether the average distance between shadows in the image windows 20 and 20" is greater than a preset threshold (step _S51 ). When the average number of light and shadows in the image window 20 is not greater than the preset threshold value or the average distance between light and shadows is not greater than the preset threshold value, the processing unit 14 performs zoom-in/zoom-out gesture recognition (step S ₅₃ ). For example, in FIGS. 7c and 7d, the processing unit 14 respectively recognizes that the average distance between light and shadow in the image window 20' and 20" increases or decreases, and therefore determines that the user is performing a gesture of zooming in/out the object, so the relative control The image display 15 updates the corresponding display screen (step S ₅₄ ).

In another embodiment, when the processing unit 14 determines that there are multiple contact points according to the shading light and shadow information in the image windows 20 ′ and 20 ″ captured by the image sensing units 13 and 13 ′, it directly zooms in/out Gesture recognition (step S ₅₃ ) without executing step S ₅₁ .

In addition, before performing zoom-in/zoom-out gesture recognition, it is also possible to directly perform the second mode without entering the first mode. When identifying, you can directly enter the second mode.

Object rotation gesture: before performing the step of object rotation, the user controls the cursor to move to the object to be rotated. Next, the user forms a plurality of contact points T on the panel surface 100s, as shown in FIGS. When it is determined that the shading light and shadow information includes multiple contact points T, the second mode is entered.

Next, the processing unit 14 determines whether the average light and shadow number in the image windows 20' and 20" is greater than a preset threshold or whether the processing unit 14 determines whether the average light and shadow distance in the image windows 20' and 20" is greater than a preset threshold (step S ₅₁ ). When the average number of light shadows is greater than the preset threshold value or the average light shadow distance is greater than the preset threshold value, the rotation direction is determined according to the number of shading light shadows moving toward the two sides of the image windows 20 ′ and 20 ″.

For example, in FIG. 7e, the processing unit 14 recognizes that the number of occlusion light shadows moving to the right in the image windows 20' and 20" is greater than the number of occlusion light shadows moving to the left; in FIG. In 20″, the number of shading lights and shadows moving to the left is greater than the number of shading lights and shadows moving to the right, so it is determined that the user is performing a gesture of rotating the object to the right/left, and accordingly the corresponding display screen update is performed on the control image display 15 (step _S54 ).

In another embodiment, when the processing unit 14 determines that there are multiple contact points according to the shading light and shadow information in the image windows 20 ′ and 20 ″ captured by the image sensing units 13 and 13 ′, the rotation gesture recognition is performed directly. (Step S ₅₂ ) Step S ₅₁ is not executed.

In addition, it is also possible to directly perform the second mode without entering the first mode before performing the rotation gesture recognition.

Screen switching gesture or menu display gesture: the user directly forms multiple contact points T on the panel surface 100s, as shown in Fig. When it is determined that the shading light and shadow information in 20' and 20" includes multiple contact points T, it will directly enter the second mode.

The processing unit 14 directly determines whether to perform screen switching or display menu gestures based on the number of shading lights and shadows displaced toward the two sides of the image windows 20' and 20". For example, in Fig. 7e and Fig. 7f, the processing unit 14 recognizes the 20′ and 20″, the number of shading light shadows moving to the right or left is greater than the number of shading light shadows moving to the left or right, so it is determined that the user is performing a gesture of switching screens or displaying menu gestures, and thus proceeding with respect to the control image display 15. Corresponding screen switching or display menu.

It can be understood that the control functions corresponding to the interrelationships of shading light and shadow in the above second mode are not limited to the content disclosed in FIGS. 5a-5h and FIGS. 7a-7f. The spirit of the present invention is to perform gesture recognition based on the interrelationship of shading light and shadow in the image window without calculating the position coordinates of the contact points one by one to avoid the situation that the coordinates of the contact points cannot be calculated due to mutual occlusion of pointers.

As mentioned above, since the existing touch control system performs gesture recognition through the change of the two-dimensional coordinates of the contact points, it is easy to fail to calculate the coordinates of the contact points correctly when the pointers cover each other. The present invention recognizes the interrelationship of shaded light and shadow in the image window as the basis for gesture recognition, so only one image sensor can be used for correct gesture recognition, which has the effect of reducing system cost.

Although the present invention has been disclosed by the above-mentioned embodiments, the above-mentioned embodiments are not intended to limit the present invention, and any skilled person in the technical field to which the present invention belongs should be able to make various changes without departing from the spirit and scope of the present invention with modification. Therefore, the protection scope of the present invention should be determined by the scope defined in the appended claims.

Claims (9)

1. A gesture recognition method for an interactive system, the interactive system comprising an image sensor, a reflective element, and at least one light source, the image sensor is used to capture images including at least one indicator that blocks the light source and/or the An image window that shields light and shadow formed by the reflective element, and the gesture recognition method includes the following steps: using the image sensor to capture an image window; Retrieving shading light and shadow information in the image window, wherein the shading light and shadow information includes an average number of light shadows, an average light shadow distance, and a maximum light shadow distance; determining whether multiple pointers are included according to the occlusion light and shadow information; and When it is determined that multiple pointers are included, gesture recognition is performed only based on the interrelationship of occluded light and shadow in the continuous image windows without calculating the two-dimensional position coordinates of each of the pointers one by one according to the image windows, wherein, according to the continuous image windows The step of performing gesture recognition by shielding the relationship between light and shadow also includes the following steps: comparing one of the average number of light shadows and the average light shadow distance with a preset threshold; When the average light and shadow number or the average light and shadow distance is less than the preset threshold value, perform gesture recognition of up, down, left, right, zoom in or zoom out; When the average light-shadow number or the average light-shadow distance is greater than the preset threshold value, perform rotation gesture recognition; and updating a display screen of the image display according to the recognized gesture; Wherein, when the gesture is recognized as up, down, left, and right, the display screen is updated to scroll the display screen; when the gesture is recognized as zooming in and out, the display screen is updated to zoom in and out of the object; When the gesture is recognized as a rotation gesture, the display screen is updated as object rotation; when the gesture is recognized as moving left and right with different numbers of shading lights and shadows, the display screen is updated as screen switching or displaying a menu. 2. The gesture recognition method according to claim 1, the method further comprising the following steps: When it is determined that a single indicator is included, calculate the position coordinates of the indicator relative to the interactive system according to the position of the shaded light in the image window; and Relatively control the movement of the cursor on the image display according to the change of the position coordinates of the pointer in the continuous image window. 3. The gesture recognition method according to claim 1, wherein, when the average light-shadow number or the average light-shadow distance is less than a preset threshold value, the gesture recognition method further comprises the following steps: According to the center line of the image window, the shaded light and shadow are divided into a real image light and shadow group and a virtual image light and shadow group; Wherein, when the displacement of the real image light and shadow group and the virtual image light and shadow group in the image window are in the same direction, the left or right gesture recognition is performed; when the real image light and shadow group and the virtual image light and shadow group in the image window When the average light-shadow distance between the groups changes, perform up or down gesture recognition; when the maximum light-shadow distance between the real image light-shadow group and the virtual image light-shadow group in the image window changes, perform zoom-in or zoom-out gesture recognition . 4 . The gesture recognition method according to claim 1 , wherein, in the step of recognizing the rotating gesture, the rotation direction is recognized according to the direction of the shading light and shadow with a higher number of shading lights and shadows displaced in the same direction in the consecutive image windows. 5 . The gesture recognition method according to claim 1 , wherein the step of performing gesture recognition according to the interrelationship of occluded light and shadow in the continuous image window is performing up, down, left, right, zoom in or zoom out gesture recognition. 6 . The gesture recognition method according to claim 1 , wherein the step of performing gesture recognition according to the mutual relationship of occluded light and shadow in continuous image windows is to perform rotation gesture recognition. 7 . 7. An interactive system, the interactive system comprising: A light emitting unit, the light emitting unit is a passive light source, and the passive light source includes a reflective mirror; at least one active light source; The image sensor is used to continuously capture at least one image window of the shading light and shadow formed by the indicator covering the light-emitting unit, and the image window includes a real image shading light and shadow and a virtual image shading light and shadow; image display; and A processing unit, coupled to the image display, when it is determined that there is only one indicator according to the image window captured by the image sensor, according to the occluded light and shadow of the indicator located in the image window One-dimensional position, calculating the two-dimensional position coordinates of the pointer; when it is determined that there are multiple pointers according to the image window captured by the image sensor, only according to the image captured by the image sensor gesture recognition without calculating the two-dimensional position coordinates of each indicator according to the image windows one by one, wherein the processing unit is based on the real image shading light and shadow and virtual image shading in the continuous image window Gesture recognition is performed on the relationship between light and shadow, and the display screen of the image display is updated according to the recognized gesture. 8. A gesture recognition method for an interactive system, the interactive system comprising a light-emitting unit and an image sensor, the light-emitting unit is a passive light source with a reflective mirror surface, and the image sensor is used to capture a number of shelters including a plurality of indicators The image window for shading light and shadow formed by the light-emitting unit, the image window includes real image shading light and shadow and virtual image shading light and shadow, and the gesture recognition method includes the following steps: using the image sensor to continuously capture image windows; dividing the shading light and shadow into a real image light and shadow group and a virtual image light and shadow group according to the center line of the image window; and Gesture recognition is performed only according to the mutual relationship of multiple shading lights and shadows in the continuous image window without calculating the two-dimensional position coordinates of each of the pointers one by one according to the image windows, wherein the mutual relationship of the shading lights and shadows includes the real image shading The average light-shadow distance change, the maximum light-shadow distance change and the displacement direction between the light-shadow and the virtual image shading light-shadow. 9. The gesture recognition method according to claim 8, wherein the interrelationship of the shading light and shadow includes the average light-shadow distance change, the maximum light-shadow distance change and the displacement direction of the shading light and shadow, and the gesture recognition method further comprises the following steps : Update the display screen of the graphics display according to the recognized gesture.

Images