TWI866828B - Hand tracking system and method - Google Patents

Abstract

This disclosure provides a hand tracking system. A camera is configured to obtain a hand image of a hand of a user. A tracker is adapted to be attached to an object and configured to obtain tracker data. A processor is configured to determine the hand being touching the object based on the hand image or the tracker data. The processor is configured to determine a contact portion of the object based on the object. The contact portion is adapted to be in contact with the hand of the user. The processor is configured to determine a hand gesture of the user based on the hand image. The processor is configured to determine a hand pose of the hand and an object pose of the object based on the hand gesture and the contact portion.

Description

Hand tracking system and method

The present invention relates to a hand tracking system, and more particularly to a hand tracking system and method.

In order to provide users with an immersive experience, technologies related to extended reality (XR), such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), are constantly being developed. AR technology allows users to bring virtual elements into the real world. VR technology allows users to enter a whole new virtual world to experience a different life. MR technology merges the real world with the virtual world. In addition, in order to provide users with a fully immersive experience, visual content, audio content, or other sensory content can be provided through one or more devices.

The present disclosure provides a hand tracking system and a hand tracking method to improve hand tracking when the hand is holding an object.

The hand tracking system disclosed herein includes a camera, a tracker, a memory, and a processor. The camera is configured to obtain a hand image of a user's hand. The tracker is suitable for being attached to an object and is configured to obtain tracker data. The memory is configured to store program code. The processor is configured to access the program code to execute: based on the hand image or the tracker data, determine that the hand is touching the object; based on the object, determine a contact portion of the object, wherein the contact portion is suitable for contacting the user's hand; based on the hand image, determine the user's hand gesture; and based on the hand gesture and the contact portion, determine the hand pose of the hand and the object pose of the object.

The hand tracking method disclosed herein includes: obtaining a hand image of a user's hand through a camera; obtaining tracker data through a tracker, wherein the tracker is suitable for being attached to an object; determining through a processor that the hand is touching the object based on the hand image or the tracker data; determining a contact portion of the object based on the object, wherein the contact portion is suitable for contacting the user's hand; determining the user's gesture based on the hand image; and determining the hand posture of the hand and the object posture of the object based on the gesture and the contact portion.

Based on the above, according to the hand tracking system and the hand tracking method, the hand and the object can be correctly tracked when the hand holds the object.

Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Whenever possible, the same drawing reference numerals are used in the drawings and description to refer to the same or like components.

Throughout this specification and the appended patent applications disclosed herein, specific terms are used to refer to specific components. It should be understood by those skilled in the art that electronic device manufacturers may refer to the same component by different names. This document does not intend to distinguish between components that have the same function but different names. In the following description and claims, words such as "including" and "comprising" are open terms and should be interpreted as "including but not limited to...".

To improve the immersive experience, physical objects in the real world can be used to represent virtual objects in the virtual world. For example, when a user can physically touch and interact with virtual objects, it can make the virtual experience feel more real and more immersive. This is especially true for tasks that require precision or dexterity, such as painting, assembling a piece of furniture, or playing a musical instrument. In addition, using physical objects to grab as virtual objects can also reduce the cognitive load on the user. This is because they do not have to learn new and complex controls. Instead, they can rely on their existing knowledge of how to interact with physical objects in the real world.

However, when a user is holding a physical object, part of the user's hand may be obscured by the physical object, thereby reducing the accuracy of hand tracking. In addition, when the position of the physical object or hand is not clearly identified, an offset may appear between the virtual object and the virtual hand in the virtual world. Due to the offset, the virtual object may appear not to be attached to the virtual hand. In other words, the virtual hand or the virtual gesture of the virtual hand may look strange. In other words, the unrealistic phenomenon may have a negative impact on the user experience, thereby reducing the immersive experience. Therefore, technicians in this field seek to provide a more realistic way to convert the relationship between a physical object and a hand into a relationship between a virtual object and a virtual hand.

FIG. 1 is a schematic diagram of a hand tracking system according to an embodiment of the present disclosure. Referring to FIG. 1 , the hand tracking system 100 may include a camera 110, a tracker 120, a memory 130, and a processor 140. In one embodiment, the camera 110, the tracker 120, the memory 130, and/or the processor 140 may be integrated into a single electronic device or separated from each other, but is not limited thereto. In one embodiment, the camera 110, the memory 130, and the processor 140 may be included in a head-mounted display (HMD) device, and the tracker 120 may be included in an object. In another embodiment, the camera 110 may be placed somewhere other than the user's head to capture an image of the user. However, the present disclosure is not limited thereto.

In one embodiment, the camera 110 may include, for example, a complementary metal oxide semiconductor (CMOS) camera or a charge coupled device (CCD) camera. In one embodiment, the camera 110 may be disposed on an HMD device, wearable glasses (e.g., AR/VR goggles), an electronic device, other similar devices, or a combination of these devices. However, the present disclosure is not limited thereto.

In one embodiment, the tracker 120 may include, for example, a gyroscope, an accelerometer, an inertial measurement unit (IMU) sensor, other similar devices, or a combination of these devices. However, the present disclosure is not limited thereto.

In one embodiment, the memory 130 may include, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk, other similar devices, or a combination of these devices, and may be used to record multiple program codes or modules. However, the present disclosure is not limited thereto.

In one embodiment, each component in the hand tracking system 100 may include a communication circuit, and the communication circuit may include, for example, a wired network module, a wireless network module, a Bluetooth module, an infrared module, a radio frequency identification (RFID) module, a Zigbee network module, or a near field communication (NFC) network module, but the present disclosure is not limited thereto. In other words, each component in the hand tracking system 100 may communicate with each other through wired or wireless communication.

In one embodiment, the processor 140 may be coupled to the camera 110, the tracker 120, and the memory 130. In embodiments, the processor 140 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), any other type of integrated circuit, a state machine, an advanced RISC machine (ARM) based processor, and the like.

In one embodiment, the processor 140 may transmit data to and receive data from the camera 110, the tracker 120, and the memory 130. In addition, the memory 130 may be configured to store program code. The processor 140 may access the program code from the memory 130 to execute the hand tracking method of the present disclosure, and the details thereof are described below.

FIG2 is a schematic flow chart of a hand tracking method according to an embodiment of the present disclosure. Referring to FIG1 and FIG2, the hand tracking method 200 can be executed by the hand tracking system 100 in FIG1, and the details of each step in FIG2 will be described below using the components shown in FIG1.

First, in step S210 , the camera 110 may be configured to obtain a hand image of a user's hand. Then, in step S220 , the tracker 120 may be adapted to be attached to an object and configured to obtain tracker data.

Then, in step S230, the processor 140 may be configured to determine that the hand is touching (e.g., touching) the object based on the hand image or the tracker data. Next, in step S240, the processor 140 may be configured to determine the contact portion of the object based on the object. In one embodiment, the contact portion may be applicable to contact with the user's hand. For example, the contact portion may be a handle or grip of the object, but the present disclosure is not limited thereto. In addition, in step S250, the processor 140 may be configured to determine the user's gesture based on the hand image. In addition, in step S260, the processor 140 may be configured to determine a hand posture (e.g., position and orientation) of the hand and an object posture of the object based on the hand gesture and the contact portion. In addition, the processor 140 may be configured to display a virtual hand and a virtual object based on the hand posture and the object posture, respectively.

In this way, the hand and the object can be tracked correctly. Therefore, the virtual hand and the virtual object in the virtual world can be displayed in a realistic manner, thereby bringing a fully immersive experience to the user. In addition, the implementation details of the hand tracking method 200 can refer to the following description to obtain sufficient teachings, suggestions and implementation examples.

FIG. 3A is a schematic diagram of an interaction scenario between a hand and an object according to an embodiment of the present disclosure. FIG. 3B is a schematic diagram of an interaction scenario between a hand and an object according to an embodiment of the present disclosure. Referring to FIG. 3A and FIG. 3B , interaction scenario 300A and interaction scenario 300B depict exemplary embodiments of how a hand H grasps an object in various ways.

First, refer to FIG. 3A. The interactive scenario 300A may include a hand H, an object O1, an object O2, and an object O3. In one embodiment, the object O1 may be a gun, the object O2 may be a racket, and the object O3 may be a bat. However, the present disclosure is not limited thereto.

In one embodiment, the user may intend to hold one of the objects O1, O2, and O3 on the hand H. However, since the shapes and functions of the objects O1, O2, and O3 may be different, the best way to grasp the objects O1, O2, and O3 may be different. That is, the best grasping position of the object O1, O2, or O3 may depend on the intended use. In one embodiment, the best grasping position may also be referred to as a contact portion. In other words, the contact portion may be adapted to come into contact with the user's hand H.

For example, referring to object O1, the contact portion of object O1 may be a grip close to the trigger of a gun, so that the user can easily pull the trigger to shoot a target. Subsequently, referring to object O2, the contact portion of object O2 may be a grip away from the frame of a racket, so that the user can easily hit a target with the racket. In addition, referring to object O3, the contact portion of object O3 may be a handle away from the barrel of a bat, so that the user can easily hit a target with the bat.

Now refer to FIG. 3B. Interaction scenario 300B may include holding scenario 301, holding scenario 302, and holding scenario 303. Referring to holding scenario 301, hand H is in contact with the grip of a gun (e.g., holding). Referring to holding scenario 302, hand H is in contact with the grip of a racket. Referring to holding scenario 303, hand H is in contact with the handle of a bat.

It is worth noting that the holding scenario 301, holding scenario 302, or holding scenario 303 depicts an ideal situation of how the user's hand H contacts the object O1, the object O2, or the object O3. In one embodiment, in order to perform hand tracking of the hand, all 21 (joint) nodes of the hand H need to be captured by the camera 110. However, since some nodes of the hand H may be blocked by the object O1, the object O2, or the object O3, the hand tracking of the hand H may not be performed properly. For example, the virtual object (corresponding to the object O1, the object O2, or the object O3) may appear not to be attached to the virtual hand (corresponding to the hand H). That is, a virtual hand or a virtual hand's virtual gestures may look weird. In other words, unrealistic phenomena may have a negative impact on the user experience, thereby reducing the immersive experience.

FIG. 4A is a schematic diagram of a hand tracking context of a single hand according to an embodiment of the present disclosure. FIG. 4B is a schematic diagram of a hand tracking context of a single hand according to an embodiment of the present disclosure. Referring to FIG. 4A and FIG. 4B , hand tracking context 400A and hand tracking context 400B depict exemplary embodiments of a tracking process of a hand H.

First, refer to Figure 4A. The hand tracking scenario 400A may include a hand H, an object OBJ, and a tracker TRK. As shown in Figure 4A, the object type of the object OBJ may be a table tennis racket. In one embodiment, the tracker TRK may be adapted to be attached to the object OBJ or embedded in the object OBJ, but is not limited thereto. That is, the tracker TRK may be associated with the object OBJ to provide information of the object OBJ. For example, the tracker TRK may be configured to provide three linear acceleration values and/or three angular velocities about three axes. However, the present disclosure is not limited thereto. Furthermore, for different objects OBJ, the tracker TRK may be installed at a suitable position. In other words, the (relative) position of the tracker TRK to the object OBJ may be known. However, the present disclosure is not limited thereto.

First, in the tracking process 401A, such as step S230 of Figure 2, in response to satisfying or triggering a specific condition, the processor 140 may be configured to determine whether the hand H is in contact with the object OBJ. In one embodiment, based on the tracker data of the tracker TRK and/or the hand image, the processor 140 may be configured to determine whether the hand H is in contact with the object OBJ. In another embodiment, a distance sensor or a contact sensor may be included in the object OBJ. Based on the sensor data of the distance sensor or the contact sensor, the processor 140 may be configured to determine whether the hand H is in contact with the object OBJ. However, the present disclosure is not limited to this.

Subsequently, in the tracking process 402A, the processor 140 may be configured to determine the object type of the object OBJ. The object type may also be referred to as the role of the object OBJ. In one embodiment, the object type may include sports equipment, shooting equipment, or weapons. However, the present disclosure is not limited to this. In one embodiment, the object information of the object OBJ or the tracker number of the tracker TRK may be stored in the tracker TRK. Based on the object information or the tracker number, the object type of the object OBJ may be determined. That is, the object information may include the object type, and the object type or the tracker number may indicate the object OBJ associated with the tracker TRK. In another embodiment, the object database may be stored in the memory 130 and may be configured to store a plurality of object types and object information for identifying the plurality of object types. For example, the object database may be a neural network model pre-trained with each object type. That is, the object database and the object recognition algorithm may be used to determine the object type of the object OBJ based on the image (e.g., hand image) obtained by the camera 110. However, the present disclosure is not limited thereto.

In addition, as shown in step S240 of FIG. 2 , in response to the determined object type, the processor 140 may be configured to determine the contact portion of the object OBJ based on the object type. For example, the contact portion may be the optimal gripping position of the object OBJ. That is, the position of the contact portion of the object OBJ may be different for different object types. Then, after determining the contact portion, since the object type is determined and the installation position of the tracker TRK may be known, the distance between the tracker TRK and the contact portion may also be determined (based on the object type), and the distance may also be referred to as an offset distance.

On the other hand, in addition to determining the contact portion based on the object type, the contact portion may also be determined based on the hand image (e.g., the contact portion where the user's hand H in the hand image contacts the object OBJ). In addition, for different object types, the distance between the tracker TRK (the installation position) and the contact portion may be pre-stored in the memory 130. That is, for each object type, the tracker TRK may be installed at a specific position of the object OBJ. Therefore, in addition to being based on the object information, the processor 140 may also be configured to determine the object type of the object OBJ based on the distance between the tracker TRK and the contact portion.

In addition, based on the tracker data, the coordinates of the tracker TRK in space can be determined. Therefore, based on the coordinates and offset distance of the tracker TRK, the coordinates or position of the contact portion can be determined. Since the contact portion is configured to be suitable for contact with the user's hand H. Therefore, based on the coordinates or position of the contact portion, the tracking frame TB can be determined. The tracking frame TB can be an area for performing hand tracking of the hand H. That is, after determining the object type and the contact portion, the tracking frame TB used for hand tracking can be determined.

Then, in the tracking process 403A, based on the tracking frame TB, the hand tracking of the hand H can be preliminarily performed. It is worth noting that, as mentioned above, since some nodes of the hand H may be blocked by the object OBJ at this time, the hand tracking of the hand H may not be properly performed. Therefore, based on the tracking frame TB, a plurality of tracking nodes ND among all the nodes of the hand H can be preliminarily determined. These plurality of tracking nodes ND can be key nodes or feature nodes of the hand H, and can be used to determine the preliminary hand posture of the hand H. That is, based on these plurality of tracking nodes ND, the hand tracking of the hand H can be preliminarily performed. Similarly, since the coordinates of the tracker TRK on the object OBJ are known, object tracking of the object OBJ can be preliminarily performed to obtain a preliminary object pose of the object OBJ.

Next, in the tracking process 404A, such as step S250 of FIG. 2 , the gesture of the hand H can be determined based on the tracking result of the preliminary hand tracking. For example, for different object types, the user may use different gestures to grasp the object OBJ. That is, when the object type is known, the gesture used by the user to grasp the object OBJ is predictable. Moreover, the gesture model MD can be further pre-trained to perform gesture recognition based on the hand image and/or the object type. However, the present disclosure is not limited to this. That is to say, the gesture model MD can be used to determine the gesture of the hand H based on the hand image and/or the object type. After the hand gesture is determined, as shown in step S260 of FIG. 2 , the hand posture of the optimized hand H and the object posture of the optimized object OBJ may be determined to ensure that the hand H is placed on the contact portion. That is, the hand posture of the optimized hand H and the object posture of the optimized object OBJ may be determined to ensure that the virtual object may appear to be attached to the virtual hand in the virtual world. In other words, the surface of the virtual hand may be fitted to the surface of the virtual object (e.g., the contact portion). Therefore, the virtual hand and the virtual object may be displayed in a realistic manner, thereby providing the user with a fully immersive experience.

It is worth mentioning that in the tracking scenario 400A, it is assumed that the hand H holds the object OBJ properly. That is, the hand H may be placed on the contact portion of the object OBJ. However, in some cases, the hand H may hold the object OBJ incorrectly. That is, the hand H may be placed somewhere other than the contact portion. Among them, the area where the hand H is placed on the object OBJ can be referred to as the touch area. In other words, the touch area is different from the contact portion. Alternatively, the hand H may be placed on the contact portion, but only a part of the hand H may be in contact with the contact portion, or the hand H may be placed on the contact portion in the wrong direction. In other words, the touch area may be partially different from the contact portion. Therefore, the processor 140 may be configured to determine whether the hand H is correctly placed on the contact portion based on the (optimized) hand posture. In addition, in response to the hand H not being correctly placed on the contact portion, the processor 140 may be configured to generate an instruction message to instruct the user to correctly place the hand H on the contact portion. For example, the instruction message may be a text message or an image displayed in the virtual world, so that the user can follow the instruction message to correctly place the hand H on the contact portion. However, the present disclosure is not limited thereto.

Now refer to FIG. 4B. Similar to the hand tracking scenario 400A, the hand tracking scenario 400B may include a hand H, an object OBJ, and a tracker TRK. As shown in FIG. 4B, the object type of the object OBJ may be a baseball. In addition, the hand tracking scenario 400B may include a tracking process 401B, a tracking process 402B, a tracking process 403B, and a tracking process 404B.

It is worth noting that, since the difference between the hand tracking context 400A and the hand tracking context 400B is the object type of the object OBJ, the implementation details of the hand tracking context 400B can refer to the hand tracking context 400A to obtain sufficient teachings, suggestions and implementation examples, and the details are not redundantly described herein. That is, the tracking process 401B, the tracking process 402B, the tracking process 403B and the tracking process 404B can refer to the tracking process 401A, the tracking process 402A, the tracking process 403A and the tracking process 404A. However, the present disclosure is not limited thereto.

FIG5A is a schematic diagram of a hand tracking scenario for two hands according to an embodiment of the present disclosure. FIG5B is a schematic diagram of a hand tracking scenario for two hands according to an embodiment of the present disclosure. Referring to FIG5A and FIG5B , hand tracking scenario 500A and hand tracking scenario 500B depict an exemplary embodiment of a tracking process for two hands.

First, refer to Figure 5A. The hand tracking scenario 500A may include a hand H1 (also referred to as hand H), a hand H2 (also referred to as an additional hand), an object OBJ, and a tracker TRK. As shown in Figure 5A, the object type of object OBJ may be a bowling ball. In addition, since the bowling ball is held by both hands, two parts of object OBJ may be in contact with hand H1 and hand H2, respectively. For example, the additional contact portion of object OBJ may be a gripping area including a thumb hole and two finger holes, and the gripping area may be suitable for contact with hand H2. In addition, the contact portion of object OBJ may be a gripping area for holding the weight of object OBJ, and the gripping area may be suitable for contact with hand H1. However, the present disclosure is not limited thereto.

First, in the tracking process 501A, such as step S230 of FIG. 2 , in response to a specific condition being met or triggered, the processor 140 may be configured to determine whether the hand H1 and the hand H2 are touching (e.g., in contact with) the object OBJ. In one embodiment, based on the tracker data of the tracker TRK and/or the hand image of the hand H1 and the additional hand image of the hand H2 obtained by the camera 110, the processor 140 may be configured to determine whether the hand H1 and the hand H2 are in contact with the object OBJ. In another embodiment, a distance sensor or a contact sensor may be included in the object OBJ. Based on the sensor data of the distance sensor or the contact sensor, the processor 140 may be configured to determine whether the hand H1 and the hand H2 are in contact with the object OBJ. However, the present disclosure is not limited thereto.

Subsequently, in the tracking process 502A, the processor 140 may be configured to determine the object type of the object OBJ. In addition, as shown in step S240 of FIG. 2 , the processor 140 may be configured to determine the contact portion and the additional contact portion of the object OBJ based on the object type and/or the hand image and the additional hand image. The above process may refer to the description of the tracking process 402A. That is, in one embodiment, the object type, the contact portion, and the additional contact portion of the object OBJ may be determined based on the object information of the object OBJ stored in the tracker TRK or the tracker number of the tracker TRK. That is, for different object types, the positions of the contact portion and the additional contact portion of the object OBJ may be different. In another embodiment, the object type, the contact portion, and the additional contact portion of the object OBJ may be determined based on the object database and/or the hand image and the additional hand image in the memory 130. However, the present disclosure is not limited thereto.

For example, the contact portion plus the additional contact portion may be the optimal gripping position for the object OBJ. After determining the contact portion and the additional contact portion, since the object type is determined, the offset distance between the tracker TRK and the contact portion and the additional offset distance between the tracker TRK and the additional contact portion may also be determined. In addition, based on the tracker data, the coordinates of the tracker TRK in space may be determined. Therefore, based on the coordinates, the offset distance, and the additional offset distance of the tracker TRK, the coordinates or position of the hand H1 and the coordinates or position of the hand H2 may be determined to determine the tracking frame TB1 and the tracking frame TB2. That is, after determining the object type and the contact portion and/or the additional contact portion, the tracking frame TB1 and/or the tracking frame TB2 for hand tracking may be determined.

Then, in the tracking process 503A, based on the tracking frame TB1 and the tracking frame TB2, multiple tracking nodes ND1 and multiple tracking nodes ND2 can be determined to preliminarily perform hand tracking of the hand H1 and hand tracking of the hand H2. And, the tracking results of the hand tracking of the hand H1 and/or the hand H2 can be used to determine the preliminary hand pose of the hand H1, the preliminary hand pose of the hand H2 (also called the additional hand pose), or the preliminary object pose of the object OBJ.

Next, although not depicted in FIG. 5A , similar to the tracking process 404A, such as step S250 of FIG. 2 , based on the tracking results of the hand tracking of the hand H1 and the hand H2, the gesture of the hand H1 and the additional gesture of the hand H2 can be determined. For example, for different object types, the user may use different gestures and additional gestures to grasp the object OBJ. In other words, when the object type is known, the gesture and additional gesture used by the user to grasp the object OBJ can be predicted. After determining the hand gesture and the additional hand gesture, as shown in step S260 of FIG. 2 , the hand posture of the optimized hand H1, the additional hand posture of the optimized hand H2, and the object posture of the optimized object OBJ may be determined to ensure that the hand H1 and the hand H2 are placed on the contact portion and the additional contact portion, respectively. That is, the hand posture of the optimized hand H1, the hand posture of the optimized hand H2, and the object posture of the optimized object OBJ may be determined to ensure that the virtual object in the virtual world may appear to be attached to the virtual hand and the virtual additional hand. In other words, the surface of the virtual hand and the surface of the virtual additional hand may fit to the surface of the virtual object. Therefore, the virtual hand, virtual additional hand, and virtual objects can be displayed in a realistic manner, thus bringing a fully immersive experience to the user.

It is worth mentioning that in the tracking scenario 500A, it is assumed that the object type of the object OBJ can be determined based on the object information, the tracker number or the object database. However, in some cases, the tracker TRK has just been attached to the object, and the object information has not yet been stored in the tracker TRK. In one embodiment, the processor 140 may be configured to determine the object type of the object OBJ based on the contact portion and the additional contact portion. More specifically, for different object types, the user may adopt different ways to hold the object OBJ. Therefore, based on how the user is holding the object OBJ, the shape, size, direction, etc. of the object OBJ can be determined. For example, the two positions of the hand H1 and the hand H2 can form the boundary of the object OBJ. Based on the boundary of object OBJ and/or the gesture of hand H1 and the additional gesture of hand H2, the object type of object OBJ can be determined. Information for identifying the object type can be pre-stored in an object recognition database, but is not limited thereto. In other words, although the object type is initially unknown, the object type can be determined based on the contact portion and the additional contact portion. After the object type is known, hand tracking of hand H1 and hand H2 can be performed more accurately.

Reference is now made to FIG. 5B. Similar to hand tracking scenario 500A, hand tracking scenario 500B may include hand H1, hand H2, object OBJ, and tracker TRK. As shown in FIG. 5B, the object type of object OBJ may be a baseball bat. In addition, hand tracking scenario 500B may include tracking process 501B and tracking process 502B. For the sake of clarity, FIG. 5B has been simplified for better understanding. For example, tracker TRK and node ND are not depicted in FIG. 5B.

It is worth noting that, since the difference between the hand tracking scenario 500A and the hand tracking scenario 500B is the object type of the object OBJ, the implementation details of the hand tracking scenario 500B can refer to the hand tracking scenario 500A to obtain sufficient teachings, suggestions, and implementation examples, and the details are not redundantly described herein. That is, the tracking process 501B and the tracking process 502B can refer to the tracking process 501A and the tracking process 502A. In addition, although it is not depicted in FIG. 5B, processes similar to the tracking process 503A and the tracking process 404A can be included in the hand tracking scenario 500B. However, the present disclosure is not limited thereto.

FIG. 5C is a schematic diagram of a hand tracking scenario for two hands according to an embodiment of the present disclosure. FIG. 5D is a schematic diagram of a hand tracking scenario for two hands according to an embodiment of the present disclosure. Referring to FIG. 5C and FIG. 5D , hand tracking scenario 500C and hand tracking scenario 500D depict exemplary embodiments of a tracking process for two hands. It is noteworthy that portions of hand H1 or portions of hand H2 may be obscured. In addition, for the sake of clarity and for better understanding, FIG. 5C and FIG. 5D have been simplified. For example, tracker TRK and node ND are not depicted in FIG. 5C and FIG. 5D .

First, refer to FIG. 5C. Hand tracking scenario 500C may include hand H1, hand H2, and object OBJ. As shown in FIG. 5C, the object type of object OBJ may be a rifle. In addition, hand tracking scenario 500C may include a side view 501C and a rear view 502C. It is worth noting that since the difference between hand tracking scenario 500A and hand tracking scenario 500C is the object type of object OBJ, the implementation details of hand tracking scenario 500C may refer to hand tracking scenario 500A for sufficient teachings, suggestions, and implementation examples, and the details are not redundantly described herein.

As shown in the side view 501C, the object OBJ may be held by the hand H1 and the hand H2. In addition, the tracking frame TB1 and the tracking frame TB2 may have been determined. However, as shown in the rear view 502C, part of the hand H1 is blocked by the hand H2. That is, since the hand H1 may not be completely captured by the camera 110, the tracking frame TB1 cannot be properly determined. In addition, even if the tracking frame TB1 is properly determined, some of the tracking nodes ND1 of the hand H1 may be blocked by the hand H2, which may negatively affect the hand tracking of the hand H1.

It is worth mentioning that by determining the object type of object OBJ and the gesture of hand H1, the remaining portion of tracking node ND1 placed on object OBJ can be determined. That is, even if part of hand H1 is blocked by hand H2, hand tracking of hand H1 can still be performed correctly, thereby increasing the user experience. And vice versa. On the other hand, although the object type of object OBJ is unknown, the object type of object OBJ can be determined based on tracking frame TB1 and tracking frame TB2. More specifically, the object type of OBJ can be determined based on the contact portion of hand H1 and the additional contact portion of hand H2. For example, as shown in FIG5C, according to side view 501C, hand H1 is separated from hand H2. In addition, according to the rear view 502C, the hand H1 is close to the hand H2. Therefore, the object OBJ can be determined to be a rifle. However, the present disclosure is not limited thereto.

Now refer to Figure 5D. Hand tracking context 500D may include hand H1, hand H2, and object OBJ. As shown in Figure 5D, the object type of object OBJ may be a pistol. In addition, hand tracking context 500D may include side view 501D. It is worth noting that since the difference between hand tracking context 500A and hand tracking context 500D is the object type of object OBJ, the implementation details of hand tracking context 500D can refer to hand tracking context 500A for sufficient teachings, suggestions, and implementation examples, and the details are not redundantly described herein.

As shown in the side view 501D, the object OBJ may be held by the hand H1 and the hand H2. Similar to the rear view 502C, part of the hand H1 may be blocked by the hand H2. Therefore, by determining the object type of the object OBJ and the gesture of the hand H1, the hand tracking of the hand H1 can still be correctly performed, thereby increasing the user experience. On the other hand, although the object type of the object OBJ is unknown, the object type of the object OBJ can be determined based on the tracking frame TB1 and the tracking frame TB2, more precisely, based on the contact portion of the hand H1 and the additional contact portion of the hand H2. For example, as shown in FIG. 5D, according to the side view 501D, the hand H1 is close to the hand H2. Therefore, the object OBJ can be determined to be a pistol. However, the present disclosure is not limited to this.

In summary, according to the hand tracking system 100 and the hand tracking method 200, the hand H (hand H1 and/or hand H2) and the object OBJ can be correctly tracked. Therefore, the virtual hand and the virtual object in the virtual world can be displayed in a real way, thereby bringing a fully immersive experience to the user.

It should be understood by those skilled in the art that various modifications and variations may be made to the disclosed embodiments without departing from the scope or spirit of the present disclosure. In view of the foregoing, it is intended that the present disclosure covers modifications and variations of the present disclosure that fall within the scope of the appended patent applications and their equivalents.

100: Hand tracking system 110: Camera 120, TRK: Tracker 130: Memory 140: Processor 200: Hand tracking method 300A, 300B: Interaction scenario 301, 302, 303: Holding scenario 400A, 400B, 500A, 500B, 500C, 500D: Hand tracking scenario 401A, 401B, 402A, 402B, 403A, 403B, 404A, 404B, 501A, 501B, 502A, 502B, 503A: Tracking process 501C, 501D: Side view 502C: Rear view H, H1, H2: Hand MD: Gesture model ND, ND1, ND2: Tracking node OBJ: Object S210, S220, S230, S240, S250, S260: Step TB, TB1, TB2: Tracking box

FIG. 1 is a schematic diagram of a hand tracking system according to an embodiment of the present disclosure. FIG. 2 is a schematic flow chart of a hand tracking method according to an embodiment of the present disclosure. FIG. 3A is a schematic diagram of an interaction scenario between a hand and an object according to an embodiment of the present disclosure. FIG. 3B is a schematic diagram of an interaction scenario between a hand and an object according to an embodiment of the present disclosure. FIG. 4A is a schematic diagram of a hand tracking scenario of a single hand according to an embodiment of the present disclosure. FIG. 4B is a schematic diagram of a hand tracking scenario of a single hand according to an embodiment of the present disclosure. FIG. 5A is a schematic diagram of a hand tracking scenario of both hands according to an embodiment of the present disclosure. FIG. 5B is a schematic diagram of a hand tracking scenario of both hands according to an embodiment of the present disclosure. FIG. 5C is a schematic diagram of a hand tracking scenario of both hands according to an embodiment of the present disclosure. FIG. 5D is a schematic diagram of a hand tracking scenario of both hands according to an embodiment of the present disclosure.

200:Hand tracking method

S210, S220, S230, S240, S250, S260: Steps

Claims (11)

A hand tracking system includes: a camera configured to obtain a hand image of a user's hand; a tracker adapted to be attached to an object and configured to obtain tracker data; a memory configured to store program code; and a processor configured to access the program code to execute: based on the hand image or the tracker data, determining that the hand is touching the object; based on the object, determining a contact portion of the object, wherein the contact portion is adapted to contact the user's hand; based on the hand image, determining a gesture of the user; and based on the gesture and the contact portion, determining a hand posture of the hand and an object posture of the object. The hand tracking system of claim 1, wherein the processor is further configured to access the program code to execute: Determine the object type of the object based on object information stored in the tracker or in the object database stored in the memory. The hand tracking system of claim 1, wherein the processor is further configured to access the program code to execute: Determine the object type of the object based on the distance between the contact portion and the tracker. A hand tracking system as described in claim 1, wherein the processor is further configured to access the program code to execute: Determine the contact portion of the object based on the object type of the object. The hand tracking system of claim 1, wherein the processor is further configured to access the program code to execute: Based on the object type of the object, determine the offset distance between the tracker and the contact portion; Based on the offset distance, determine the tracking frame of the hand; Based on the tracking frame, perform hand tracking of the hand; and Based on the tracking result of the hand tracking, determine the hand posture and the object posture. A hand tracking system as described in claim 5, wherein the processor is further configured to access the program code to execute: Based on the tracking frame, determine multiple tracking nodes; and Based on the multiple tracking nodes, perform the hand tracking. A hand tracking system as described in claim 1, wherein the processor is further configured to access the program code to execute: Based on the hand posture, determine whether the hand is correctly placed on the contact portion; and In response to the hand not being correctly placed on the contact portion, generate an operation instruction message to instruct the user to correctly place the hand on the contact portion. The hand tracking system of claim 1, wherein the processor is further configured to access the program code to execute: based on the additional hand image or the tracker data, determining that an additional hand is touching the object; and based on the additional hand image, determining an additional contact portion of the object, wherein the additional contact portion is applicable to contact with the additional hand. A hand tracking system as described in claim 8, wherein the processor is further configured to access the program code to execute: Based on the additional hand image, determine an additional gesture of the additional hand; and Based on the additional gesture and the additional contact portion, determine an additional hand posture of the additional hand. A hand tracking system as described in claim 8, wherein the processor is further configured to access the program code to execute: Determine the object type of the object based on the contact portion and the additional contact portion. A hand tracking method, comprising: Obtaining a hand image of a user's hand through a camera; Obtaining tracker data through a tracker, wherein the tracker is suitable for being attached to an object; Determining that the hand is touching the object through a processor based on the hand image or the tracker data; Determining a contact portion of the object based on the object, wherein the contact portion is suitable for contacting the user's hand; Determining a gesture of the user based on the hand image; and Determining a hand posture of the hand and an object posture of the object based on the gesture and the contact portion.

Images