KR102810896B1 - Create virtual reality hand gestures - Google Patents

Abstract

The method comprises the steps of receiving at least one of touch data or force data representing a touch input received from a controller, determining one or more model(s), generating image data using the one or more models, wherein the image data represents at least a hand gesture corresponding to the touch input received from the controller, and transmitting the image data to a virtual reality (VR) environment for display.

Description

Create virtual reality hand gestures

Claims of priority and cross-references to related applications

This application claims priority to and is a PCT application of U.S. Patent Application No. 16/195,718, filed November 19, 2018, entitled “VIRTUAL REALITY HAND GESTURE GENERATION,” which claims priority under 35 U.S.C. §120 to U.S. Provisional Patent Application No. 62/687,780, filed June 20, 2018, entitled “VIRTUAL REALITY HAND GESTURE GENERATION.”

Also, application Ser. No. 16/195,718, filed November 19, 2018, is a continuation-in-part of pending U.S. patent application Ser. No. 15/834,374, filed December 7, 2017, entitled “ELECTRONIC CONTROLLER WITH FINGER SENSING AND AN ADJUSTABLE HAND RETAINER,” under 35 U.S.C. §120, which claims priority as a continuation-in-part of U.S. patent application Ser. No. 15/679,521, filed Aug. 17, 2017, entitled “Electronic controller with hand retainer and finger motion sensing,” which itself claims priority as a continuation-in-part of U.S. patent application Ser. No. 29/580,635, filed Oct. 11, 2016, which claims priority to U.S. Provisional Patent Application No. 62/520,958, filed June 16, 2017.

The video game industry has produced many innovations in both hardware and software. For example, a variety of handheld video game controllers have been designed, manufactured, and marketed for a variety of gaming applications. Some of the innovations have applicability outside the video game industry, such as controllers for industrial machinery, defense systems, robotics, etc.

Additionally, virtual reality (VR) systems are an application of great contemporary interest and rapid technological advancement both within and outside the video game industry. Controllers for VR systems must perform a number of different functions and meet strict (and sometimes competitive) design constraints, while often optimizing specific desired characteristics. In some cases, these controllers include sensors for measuring the user's grip force, which is then used to perform predefined gameplay functions. To achieve these goals, various types of sensors have been used, including, among others, force sensing resistors (FSRs), which use a variable resistance to measure the amount of force applied to the FSR. However, existing controllers with FSRs tend to exhibit rather poor response times. Additionally, the controllers may not accurately depict and detect hand positions, gestures, and/or movements throughout the gameplay experience.

FIG. 1 illustrates an environment of a user interacting with a virtual reality (VR) system according to an exemplary embodiment of the present disclosure.
FIG. 2 illustrates an exemplary controller held in a user's hand according to an exemplary embodiment of the present disclosure.
FIG. 3 illustrates an exemplary controller according to an exemplary embodiment of the present disclosure.
FIG. 4 illustrates the exemplary controller of FIG. 3 in a user's hand according to an exemplary embodiment of the present disclosure.
FIG. 5 illustrates the exemplary controller of FIG. 3 in a user's hand according to an exemplary embodiment of the present disclosure.
FIG. 6 illustrates the exemplary controller of FIG. 3 in a user's hand according to an exemplary embodiment of the present disclosure.
FIG. 7 illustrates a pair of exemplary controllers according to an exemplary embodiment of the present disclosure.
FIG. 8a illustrates a front view of an exemplary right controller according to another exemplary embodiment of the present disclosure.
Figure 8b illustrates a rear view of the exemplary right controller of Figure 8a.
FIG. 9a illustrates an exemplary force sensing resistor (FSR) according to an exemplary embodiment of the present disclosure.
Figure 9b illustrates a front view of the exemplary FSR of Figure 9a.
Figure 9c illustrates a cross-section of the exemplary FSR of Figure 9b taken along the section (AA) of Figure 9b.
FIG. 10A illustrates a first hand gesture of a user holding an exemplary controller according to an exemplary embodiment of the present disclosure.
FIG. 10b illustrates a second hand gesture of a user holding an exemplary controller according to an exemplary embodiment of the present disclosure.
FIG. 10c illustrates a third hand gesture of a user holding an exemplary controller according to an exemplary embodiment of the present disclosure.
FIG. 10d illustrates a fourth hand gesture of a user holding an exemplary controller according to an exemplary embodiment of the present disclosure.
FIG. 10e illustrates a fifth hand gesture of a user holding an exemplary controller according to an exemplary embodiment of the present disclosure.
FIG. 10f illustrates a sixth hand gesture of a user holding an exemplary controller according to an exemplary embodiment of the present disclosure.
FIG. 11 illustrates an exemplary process according to an exemplary embodiment of the present disclosure.
FIG. 12 illustrates an exemplary process for training model(s) according to an exemplary embodiment of the present disclosure.
FIG. 13 illustrates an exemplary process for using touch input to generate gestures according to an exemplary embodiment of the present disclosure.

Disclosed herein are motion capture systems and controllers for use in a virtual reality (VR) environment. An exemplary motion capture system may include cameras, projectors, and/or other sensors positioned around the environment to track the movements of a user manipulating a controller, as well as the movements of the controller. For example, multiple cameras may be mounted within the environment and capture images of the controller and the user. In some cases, the multiple cameras may capture some or all angles and positions within the environment. Alternatively, the multiple cameras may focus on or capture images within a predefined range or region of the environment. When the controller is manipulated relative to the environment and when the user manipulates his or her hands, the cameras may detect the positions and orientations of the user and/or the controller(s), respectively.

In some cases, to detect the location of the controller (or portions thereof) and the user, the controller(s) and/or the user may each include markers. For example, the markers may be coupled to the controller and/or the user. The markers may include digital watermarks, infrared reflectors, etc. The motion capture system(s) may project light into the environment, which is then reflected by the markers. The cameras may capture the incident light reflected by the markers, and the motion capture system(s) may track and plot the locations of the markers within the environment to determine movements, locations and/or orientations of the controller and/or the user.

An exemplary controller may be held by a user and may include one or more force sensitive resistors (FSRs) or other types of sensors that detect touch input from the user. In some cases, the FSR may be coupled to a surface of the controller, such as a structure mounted within a handle of the controller and/or a structure mounted beneath a control that is actuated by at least one thumb of the controller. In some cases, the FSR may measure a resistance value corresponding to an amount of force applied by the user. The FSR may also associate force(s) with specific locations, regions, and/or portions of the controller. For example, the FSR may determine an amount of force applied to an outer surface of the handle and/or may determine a location(s) on the controller that corresponds to a touch input from the user. In some embodiments, the controller may determine, through force data generated by the FSR, an amount of force applied by the user to the handle of the controller and/or an amount of force applied by the user to buttons on the controller. The controller may convert various pressures or squeezes of force into digitized numerical values that may be used in video game controls and/or game mechanisms.

In some cases, the FSR may act as a switch that detects when an applied force exceeds a threshold, and in some cases may dynamically update and/or adjust it. For example, the threshold may be adjusted to a lower value to reduce hand fatigue during gameplay (e.g., when a user frequently presses a control associated with the FSR to fire a weapon during gameplay). Conversely, the threshold may be adjusted to a higher value to reduce instances of accidental control actions.

The controller may also include an array of proximity sensors spatially distributed along the length of the handle and responsive to the proximity of the user's fingers. The proximity sensors may include any suitable technology, such as capacitive sensors for detecting touch input and/or proximity of the user's hand to the controller. The proximity sensor array may generate touch data indicative of the position of the finger(s) holding the controller, or, when the user is not holding the controller, the distance (e.g., via capacitance measurements) between the handle and the user's fingers. In some cases, the proximity sensors may also detect the size of the user's hand holding the controller, which may configure the controller based on other settings. For example, based on hand size, the controller may adjust to make force-based input easier for users with smaller hands.

Implementing a motion capture system and controller for use in a VR environment can expand the spectrum of natural interactions beyond what is currently possible using conventional controllers. For example, in combination, the motion capture system(s) can capture motion data of a hand and/or a controller, while the controller can capture touch data corresponding to touch inputs on the controller and force data associated with the user's touch inputs. The motion data, touch data, and/or force data can be correlated to generate models representing the user's hand gestures.

For example, a user may include markers placed on their knuckles, fingertips, wrists, joints, etc. The controller may also include markers (e.g., top, bottom, sides, etc.). As noted above, the marker(s) may reflect incident light. The motion capture system may detect and record the movements of the user's hand(s) and the positions of the controller(s) via cameras that detect the positions of the markers. For example, projectors of the motion capture system(s) may project infrared light, which is then reflected by markers on the hand and/or controller. Cameras of the motion capture system(s) may capture images of the environment. The images are used to represent the positions of the markers in the environment. The positions of the markers are tracked over time and animated in a three-dimensional (3D) virtual space. This tracking may generate animated 3D skeletal data (or models). For example, the user may make a fist or grasp the controller with two fingers (e.g., the little finger and the fourth finger). The cameras can capture the positions of the user's fingertips, knuckles, and/or other parts of the hand, wrist, and/or arm via markers. In some cases, the positions are relative to the controller.

At the same time or at different times, the proximity sensor array can detect touch input or lack of touch input on the controller. The touch data can represent the positions of the user's fingers relative to the controller, for example, via capacitance measurements. The capacitance can vary with the distance the fingers are placed between the fingers and the controller. By doing so, the controller can detect when the user is holding the controller with one finger, two fingers, three fingers, etc. With capacitance, the controller can also detect the relative placement of the fingers with respect to the controller, such as when the user's fingers are not touching the controller.

Additionally, the FSR can capture force data representing force values received by the controller(s), such as the force with which the user grips the controller. For example, when the user makes a fist or grips the controller body with two fingers, the FSR can capture force values corresponding to each of these grips. For example, the FSR can detect an increase in force values when the user makes a fist and grips the controller compared to when the user holds the controller with two fingers.

The touch data and force data can be correlated with each other. For example, when a user grasps a controller with four fingers, force values detected on the controller can be correlated with specific locations on the controller. In doing so, the touch data and force data can be correlated with each other to determine which fingers of the user are grasping the controller, as well as the relative force of each finger with which the user is grasping the controller. The same can be true when the user is grasping the controller with two fingers, where force values are detected and associated with specific parts of the controller body. Knowing where the touch input is received from the proximity sensor array, as well as the amount of force with which the user is grasping the controller as detected by the FSR, the controller and/or other communicatively coupled remote system can associate the touch input with specific fingers of the user. In some cases, the controller (or other communicatively coupled remote system) can associate the touch data with the force data by correlating the timestamps associated with the touch data with the timestamps of the force data.

The amount of force with which a user grips a controller (i.e., force data), the location of touch input or lack of touch input on the controller (i.e., touch data), as well as motion captured by a camera of a motion capture system (i.e., motion data) can be used to train models representing the user's hand gestures. For example, the motion capture system can associate (e.g., using motion data) a clenched fist with touch data and/or force data received from the controller. As another example, if a user grasps a controller with two fingers, the motion data can represent a hand gesture (e.g., a two-finger grip), while the touch data can represent the proximity of the hand (or fingers) to the controller and the force data can represent how firmly the user is holding the controller. Using these associations, models can be created and trained to represent the user's gestures. The models can be continually trained to become more accurate over time.

The models can generate animations of hand gestures on the display by characterizing touch inputs and/or force values associated with touch inputs on the controller, and the VR environment can utilize the models for use in gameplay. More specifically, the models can input touch data and/or force data to generate hand gestures within the VR environment. As examples, the gestures can include various video game controls, such as breaking rocks or squeezing balloons (e.g., making a fist gesture), toggling through available weapons usable by the game character (e.g., scrolling or sliding fingers along the controller), dropping objects (e.g., making an open hand gesture), firing a weapon (e.g., the pinky, ring, and middle fingers touching the controller, but the index finger and thumb pointing outward), etc. That is, the location of the touch input on the controller, as well as the force with which the user is holding the controller, can be known. This information can be used with previously trained models to generate hand gestures (e.g., making a fist) within the VR environment and/or on the VR display. Additionally, the model(s) may utilize previously generated animations and/or image data when rendering and/or generating hand gestures for display.

An exemplary virtual reality (VR) environment

FIG. 1 illustrates an exemplary environment (100) in which motion capture system(s) (102) and a user (104) reside. The motion capture system(s) (102) are illustrated as being mounted on the walls of the environment (100), although in some cases the motion capture system(s) (102) may be mounted elsewhere within the environment (100) (e.g., on the ceiling, floor, etc.). Additionally, while FIG. 1 illustrates four motion capture system(s) (102), the environment (100) may include more or fewer than four motion capture system(s) (102).

The motion capture system(s) (102) may include projector(s) configured to generate and project light and/or images (106) within/into the environment (100). The images (106) may include visible light images that are perceptible to a user (104), visible light images that are not perceptible to a user (104), images having non-visible light, and/or combinations thereof. The projector(s) may include any number of technologies capable of generating the images (106) and projecting the images (106) onto surfaces or objects within the environment (100). In some examples, suitable technologies may include digital micromirror devices (DMDs), liquid crystal on silicon displays (LCOSs), liquid crystal displays, 3LCDs, and the like. The projector(s) may have a field of view that describes a particular solid angle, and the field of view may vary with changes in the configuration of the projector(s). For example, the field of view may narrow when zooming is applied.

The motion capture system(s) (102) may include high resolution cameras, infrared (IR) detectors, sensors, and the like. The camera(s) may image the environment (100) in visible wavelengths, invisible wavelengths, or both. The camera(s) also have a field of view that describes a particular stereoscopic angle, and the field of view of the camera(s) may vary with changes in the configuration of the camera(s). For example, an optical zoom of the camera(s) may narrow the camera field of view.

In some cases, the environment (100) may include multiple cameras and/or different types of cameras. For example, the cameras may include three-dimensional (3D), infrared (IR) cameras and/or red-green-blue (RGB) cameras. In some cases, the 3D camera and IR camera may capture information to detect depths of objects within the environment (e.g., markers), while the RGB camera may detect edges of objects by identifying changes in color within the environment (100). In some examples, the motion capture system(s) (102) may include a single camera configured to perform all of the functions described above.

One or more components of the motion capture system(s) (102) may be mounted to the chassis in a fixed orientation or may be mounted to the chassis via actuators to allow the chassis and/or one or more components to move. By way of example, the actuators may include piezoelectric actuators, motors, linear actuators, and other devices configured to displace or move the chassis and/or one or more components mounted thereon, such as projector(s) and/or camera(s). For example, the actuators may include a pan motor, a tilt motor, and the like. The pan motor may rotate the chassis in a yaw motion while the tilt motor changes the pitch of the chassis. In some examples, the chassis may additionally or alternatively include a roll motor, which may cause the chassis to move in a rolling motion. By panning, tilting, and/or rolling the chassis, the motion capture system(s) (102) may capture different views of the environment (100).

The motion capture system(s) (102) may also include a ranging system. The ranging system may provide distance information from the motion capture system(s) (102) to scanned entities, objects (e.g., the user (104) and/or the controller (110)) and/or sets of objects. The ranging system may include and/or utilize radar, light detection and ranging (LIDAR), ultrasonic ranging, stereo ranging, structured light analysis, time-of-flight observations (e.g., round-trip time-of-flight measurements for pixels detected by a camera), and the like. In structured light analysis, and as noted above, the projector(s) may project a structured light pattern within the environment (100) and the camera(s) may capture images of the reflected light pattern. The motion capture system(s) (102) can analyze the deformation of the reflected pattern due to lateral displacement between the projector and the camera to determine depths or distances corresponding to different points, areas or pixels within the environment (100).

The motion capture system(s) (102) can determine or know the distance(s) between each of the components of the motion capture system(s) (102) that can aid in the recovery of the structured light pattern and/or other light data from the environment (100). The motion capture system(s) (102) can also use the distances to calculate other distances, dimensions, and/or otherwise aid in the characterization of entities or objects within the environment (100). In implementations where the relative angles and sizes of the projector field of view and the camera field of view can vary, the motion capture system(s) (102) can determine and/or know these dimensions.

Within the environment (100), a user (104) may wear a VR headset (108) and hold controllers (110). The VR headset (108) may include an internal display (not shown) that presents a simulated view of the virtual environment, gameplay, or displays objects within the virtual space. The VR headset (108) may include a headband with additional sensors. In some embodiments, the VR headset (108) may include a helmet or cap and may include sensors positioned at various locations on the top of the helmet or cap to receive optical signals.

As discussed in detail herein, the user (104) and/or the controllers (110) may include markers. The motion capture system (102) may detect the location of the user (104) and/or the controllers (110) within the environment (100) via projector(s) that project light and camera(s) that capture images of reflections of the markers. The markers may be used to determine the orientation and/or location of the user (104) within the environment (100), or portions of the user (104) within the environment (100) (e.g., a hand or fingers), as well as the orientation and/or location of the controllers (110) within the environment (100). A ranging system may also assist in determining the locations of the user (104) (or portions thereof) and the controllers (110) by determining distances between the motion capture system(s) (102) and the markers.

The motion capture system(s) (102), the VR headset (108), and/or the controllers (110) may be communicatively coupled to one or more remote computing resource(s) (112). The remote computing resource(s) (112) may be remote from the environment (100) and the motion capture system(s) (102), the VR headset (108), and/or the controllers (110). For example, the motion capture system(s) (102), the VR headset (108), and/or the controllers (110) may be communicatively coupled to the remote computing resource(s) (112) via a network (114). In some cases, the motion capture system(s) (102), the VR headset (108), and/or the controllers (110) may be communicatively coupled to a network (114) via wired technologies (e.g., wired, USB, fiber optic cable, etc.), wireless technologies (e.g., RF, cellular, satellite, Bluetooth, etc.) and/or other connection technologies. The network (114) represents any type of communications network, including a data and/or voice network, and may be implemented using wired infrastructure (e.g., cable, CAT5, fiber optic cable, etc.), wireless infrastructure (e.g., RF, cellular, microwave, satellite, Bluetooth, etc.) and/or other connection technologies.

The remote computing resource(s) (112) may be implemented as one or more servers, and in some cases may form part of a network-accessible computing platform implemented as computing infrastructure, such as processors, storage, software, data access, etc., that is maintained and accessible over a network, such as the Internet. The remote computing resource(s) (112) do not require end-user knowledge of the physical location and configuration of the system delivering the services. Common expressions associated with such remote computing resource(s) (112) may include “on-demand computing,” “software as a service (SaaS),” “platform computing,” “network-accessible platform,” “cloud services,” “data centers,” and the like.

The motion capture system(s) (102), the VR headset (108), and/or the controllers (110) may include one or more communication interfaces to facilitate wireless connection to a network (114) and/or one or more remote computing resource(s) (112). Additionally, the one or more communication interfaces may also allow for data transfer (e.g., communication between) the motion capture system(s) (102), the VR headset (108), and/or the controllers (110). However, in some cases, the one or more communication interfaces may also include wired connections.

The remote computing resource(s) (112) includes a processor(s) (116) and a memory (118) that may store or otherwise have access to one or more model(s) (120). The remote computing resource(s) (112) may receive motion data (122) and touch data (124) from the motion capture system(s) (102) and/or force data (126) from the controllers (110). The touch data (124) may include a touch profile indicating a location (or locations) on the controller(s) (110) corresponding to a user's touch input. The touch data (124) may also indicate a lack of touch input on the controller(s) (110). In doing so, the touch data (124) may indicate which finger(s) is touching the controller and/or which parts of the finger(s) are touching the controller(s) (110). In some cases, an array of proximity sensors (e.g., capacitive sensors) spatially distributed along the handle of the controller (110) may detect touch input and generate and/or transmit touch data to remote computing resource(s) (112). Additionally, the FSR may generate force data (126) representing force values of the touch input to the controller (110). As described herein, the touch data (124) and/or force data (126) may represent a hand position, grip, or gesture within the VR environment. In turn, the remote computing resource(s) (112) may transmit animation(s) (128) or other image data to the VR headset (108) for display.

As discussed in detail herein, the remote computing resource(s) (112) may generate animations (128) displayed on the VR headset (108) using the model(s) (120). In some cases, the remote computing resource(s) (112) may generate and/or train the model(s) (120) using motion data (122), touch data (124), and/or force data (126). The remote computing resource(s) (112) may generate and/or train the model(s) (120) through interactions with users and through receiving the motion data (122), touch data (124), and/or force data (126). The processor(s) (116) may analyze the motion data (122) and correlate the motion data (122) with the touch data (124) and/or force data (126). Additionally, the processor(s) (116) can analyze the touch data (124) and associate the touch data (124) with force data (126).

The processor(s) (116) can correlate the time associated with the capture of the motion data (122) to learn characteristics of the users. For example, the processor(s) (116) can learn characteristics of the touch data (124) (e.g., position on the controller (110)) and/or the force data (126) and associate these characteristics with specific hand gestures. After performing the data analysis, the processor(s) (116) can generate a model(s) (120) to correlate the motion data (122), the touch data (124), and/or the force data (126). In other words, the processor(s) (116) can analyze the touch data (124) and/or the force data (126) to correlate or otherwise associate the touch data (124) and/or the force data (126) with hand gestures as represented by the motion data (122). Training the model(s) (120) based on motion data (122), touch data (124), and/or force data (126) allows the model(s) (120) to use the touch data (124) and/or force data (126) received from subsequent interactions of the users (i.e., during gameplay) to determine hand gestures. That is, the model(s) (120) can receive the touch data (124) and/or force data (126) as inputs and use the touch data (124) and/or force data (126) to determine hand gestures of the user (104). For example, when a user holds a controller (110), the controller (110) can receive touch data (124) generated by a proximity sensor array, where the touch data (124) represents a location of a touch input on the controller (110). Touch data (124) may also indicate proximity of a user's hand to the controller (110) by measuring capacitance values between the user's fingers and the controller (110). For example, the user may hover his or her fingers over the controller (110). The controller (110) may transmit the touch data (124) to the remote computing resource(s) (112), where the touch data (124) is input into the model(s) (120). Additionally, the FSR of the controller (110) may generate force data (126) indicating an amount of force associated with the touch input. The controller (110) may transmit the force data (126) to the remote computing resource(s) (112).

Upon receiving touch data (124) and/or force data (126) from the controller(s) (110), the processor(s) (116) may select one or more of the model(s) (120) based on characteristics of the touch data (124) and/or force data (126). For example, the processor(s) (116) may select a particular model(s) (120) for generating hand gestures based on the amount of force with which the user (104) grips the controller (110) (using the force data (126)) and/or the location of the user's (104) grip on the controller (110) (using the touch data (124)).

Additionally, in some cases, the processor(s) (116) may select the model(s) (120) based in part on other user characteristics, such as user interests, gender, age, etc. For example, based on how the user (104) holds the controller (110) and/or where the controller (110) receives touch input, the processor(s) (116) may identify the age and/or hand size of the user (104). This information may be used to select different model(s) (120) and/or generate animation(s) (128) representing the user's (104) hands.

For example, the processor(s) (116) can input touch data into the model(s) (120). The processor(s) (116) using the model(s) (120) can generate animation(s) (128) corresponding to the touch data (124) and/or force data (126). For example, using the touch data (124) and/or force data (126), and by inputting the touch data (124) and/or force data (126) into the model(s) (120), the processor(s) (116) can determine that the user is making a fist and holding the controller (110). The processor(s) (116) can generate an animation (128) depicting the user (104) making a fist and transmit the animation (128) to the VR headset (108) for display.

In some cases, the processor(s) (116) may use rankings to determine the most probable hand gesture represented by the touch data (124) and/or the force data (126) using the profiles stored in association with the model(s) (120). For example, the processor(s) (116) may compare the touch data (124) and/or the force data (126) to some or all of the model(s) (120) to determine the probability that certain hand gestures correspond to a user's touch input. In such cases, the model(s) (120) may be stored in association with the touch data (124) representing the location of a touch input received at the controller (110) and/or the force data (126) representing the relative force of the touch input at the controller (110). In such cases, the touch data (124) and/or the force data (126) may characterize the model(s) (120). Thus, during the game play, when the remote computing resource(s) (112) receives touch data (124) and/or force data (126), the remote computing resource(s) (112) may select one or more model(s) (120) to generate the animation (128) by comparing the received touch data (124) and/or force data (126) to the stored touch data and/or force data associated with the model(s) (120).

In some cases, the remote computing resource(s) (112) may also perform predictive modeling of future events. Predictive modeling may determine the probability that an outcome may or may not occur. For example, the processor(s) (116) may use the motion data (122), the touch data (124), and/or the force data (126) available from the memory (118) to determine the probability of future hand gestures. For example, after receiving the first touch data (124) and/or the first force data (126), and inputting the touch data (124) and/or the force data (126) into the model(s) (120) to determine a first hand gesture, the processor(s) (116) may predict a subsequent second hand gesture and generate the second hand gesture for display on the VR headset (108). That is, the processor(s) (116) can use previous motion data (122), touch data (124), and/or force data (126) to predict future hand gestures of the user (104) and generate corresponding animation(s) (128). In some examples, the prediction can reduce latency time between gestures generated by the remote computing resource(s) (112) that are displayed on the VR headset (108).

Additionally, the processor(s) (116) may determine a particular probability and/or confidence associated with the predicted gesture. For example, if the predicted second hand gesture is within a particular confidence level or threshold, the processor(s) (116) may generate animation(s) (128) corresponding to the second hand gesture and provide the gesture to the VR headset (108) for display.

In some examples, validation operations, such as statistical analysis techniques, may verify the accuracy of the model(s) (120). That is, by repeatedly capturing the motion data (122), the touch data (124), and/or the force data (126), as noted above, the processor(s) (116) may train the model(s) (120) to better correlate the touch data (124) and/or the force data (126) with the hand gestures (e.g., machine learning algorithms or techniques) represented in the motion data (122). Training the model(s) (120) may increase the accuracy with which the displayed animation(s) (128) represent the touch data (124) and/or the force data (126) received by the controller (110).

The processor(s) (116) may also include components that learn the model(s) (120) based on interactions with different types of users. For example, the processor(s) (116) may build and/or improve the model(s) (120), or learn combinations and/or blends of existing model(s) (120). The model generation techniques described herein may also include at least one of gradient boosting techniques and/or hyperparameter tuning for training the model(s). Gradient boosting may include generating the predictive model in the form of an ensemble of weak predictive models, which may be decision trees, for example. The predictive model may be built in a stepwise manner and may allow for optimization of any differential loss function. Hyperparameter tuning may include optimization of hyperparameters during the training process. For example, the model (120) may receive a training data set. When evaluating the overall accuracy of the model (120), hyperparameters can be tuned.

Additionally, or alternatively, training the model(s) (120) may include identifying input features that increase the accuracy of the model(s) (120) and/or other input features that decrease the accuracy of the model(s) (120) or have little or no effect on the accuracy of the model(s) (120). The model(s) (120) may be retrained to utilize features that increase accuracy while refraining from utilizing features that decrease accuracy or have little or no effect on accuracy.

As used herein, a processor, such as processor(s) (116), may include multiple processors and/or processors having multiple cores. Additionally, the processors may include one or more cores of different types. For example, the processors may include application processor units, graphics processing units, etc. In one implementation, the processor may include a microcontroller and/or a microprocessor. The processor(s) (116) may include a graphics processing unit (GPU), a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, what is functionally described herein may be performed at least in part by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOC), complex programmable logic devices (CPLDs), and the like. Additionally, each of the processor(s) (116) may have its own local memory that may store program components, program data, and/or one or more operating systems.

Memory may include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program components or other data. Such memory (118) may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems or any other medium that may be used to store the desired information and that may be accessed by the computing device. Memory (118) may be implemented as a computer-readable storage medium (“CRSM”), which may be any available physical medium that is accessible by the processor(s) (116) to execute instructions stored in the memory (118). In one basic implementation, CRSM may include random access memory (“RAM”) and flash memory. In other implementations, the CRSM may include, but is not limited to, read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), or any other type of medium that can be used to store desired information and that can be processed by the processor(s).

Example controller

FIG. 2 illustrates a user (104) holding a controller (110) (which may represent and/or be similar to the controller (110) of FIG. 1 ). The controller (110) may include markers (200) that may be coupled and/or attached to any portion of the controller (110), such as a handle, a strap, a grip, and the like. Similarly, portions of the user (104) may include markers (202) that are attached to and/or along the hand of the user (104), such as a fingertip, a knuckle, a knuckle, a wrist, and the like. In some cases, the markers (200, 202) may be attached to the user (104) and/or the controller (110), respectively, using adhesives.

The markers (200, 202) may include infrared elements, reflectors and/or images that respond to electromagnetic radiation (e.g., infrared light) emitted by the projector(s) of the motion capture system(s) (102). Additionally or alternatively, the markers (200, 202) may include tracking beacons that emit electromagnetic radiation (e.g., infrared light) that is captured by the cameras of the motion capture system(s) (102).

As noted above, the motion capture system(s) (102) can scan at least a portion of the environment (100) and objects contained therein to detect markers (200, 202). For example, the projector(s) can project infrared light toward the user (104) and the controller(s) (110), the markers (200, 202) can reflect the light, and the cameras(s) and/or sensors of the motion capture system(s) (102) can capture the reflected light. By analyzing the images therein, the position and/or orientation of the hands of the controller(s) (110) and/or the user (104) can be determined. For example, the remote computing resource(s) (112) (or other computing devices) can analyze and parse images captured by the cameras and identify locations of markers (200, 202) within the environment (100). The remote computing resource(s) (112) can use the locations of the markers (200, 202) to determine gestures performed by the user (104), hand positions, finger positions (i.e., which fingers are open, closed, etc.). Additionally, the motion capture system(s) (102) (or other computing systems) can utilize information about the locations/patterns of the markers (200, 202) to generate a skeletal model representing the hand (e.g., an animated 3D skeletal model) and hand gestures (e.g., a fist).

FIGS. 3 through 7 illustrate exemplary controllers (300) (which may represent and/or be similar to the controller (110) of FIGS. 1 and 2 ) according to exemplary embodiments of the present disclosure. In some cases, electronic systems, such as VR video game systems, robots, weapons, or medical devices, may utilize the controller (300). The controller (300) may include a controller body (310) having a handle (312) and a hand retainer (320) for retaining the controller (300) in a user's hand (e.g., the user's left hand). In some cases, the handle (312) may include a substantially cylindrical tubular housing. In this context, the substantially cylindrical shape need not have a constant diameter or a perfectly circular cross-section.

The controller body (310) may include a head (between the handle (312) and the distal end (311)) that may optionally include one or more thumb-operated controls (314, 315, 316). For example, the head may include a tilt button, or any other button, knob, wheel, joystick or trackball that is considered a thumb-operated control that is conveniently operated by the user's thumb during normal operation and while the controller (300) is held in the user's hand.

In some cases, the controller (300) may include a tracking member (330) affixed to the controller body (310) and may include two noses (332, 334) each protruding from a corresponding one of two opposing distal ends of the tracking member (330). In some cases, the tracking member (330) may include a tracking arc having an arched shape. In some cases, the tracking member (330) may include a plurality of tracking transducers (which may represent and/or may be similar to markers (200, 202) of FIG. 2 ) disposed therein/on. In some cases, each protruding nose (332, 334) may include at least one tracking transducer. Additionally or alternatively, the controller body (310) may include at least one distal tracking transducer disposed adjacent the distal end (311). Tracking transducers, which may include tracking sensors, may be responsive to electromagnetic radiation (e.g., infrared light) emitted by the motion capture system(s) (102). Additionally or alternatively, the tracking transducers may include tracking beacons that emit electromagnetic radiation (e.g., infrared light) that is received by cameras of the motion capture system(s) (102). For example, projectors of the motion capture system(s) (102) may broadcast pulsed infrared light broadly toward the controller (300). Here, a plurality of tracking transducers of the tracking member (330) may include infrared light sensors that receive or cast shadows on the broadcasted pulsed infrared light. In some cases, the tracking transducers at each nose (332, 334) (e.g., three sensors at each nose) may be positioned to overhang the user's hand over each distal end of the tracking member (330) for increased exposure (i.e., around the user's hand) and to receive or transmit electromagnetic radiation emitted by the projectors to the cameras at more angles without unacceptable amounts of shadowing.

The material of the tracking member (330) and/or the controller body (310) may comprise a substantially rigid material, such as a hard plastic, that is rigidly secured together so as not to significantly translate or rotate relative to one another. For example, as illustrated in FIGS. 3 through 7, the tracking member (330) may be coupled to the controller body (310) at two locations. A hand retainer (320) may be attached to the controller (300) (e.g., the controller body (310) and/or the tracking member (330)) adjacent these two locations to bias the user's palm relative to the outer surface of the handle (312) between these two locations.

In certain embodiments, the tracking member (330) and the controller body (310) may comprise integral monolithic components having material continuity rather than being assembled together. For example, a single injection molding process may mold the tracking member (330) and the controller body (310) together to create a single integral hard plastic component that includes both the tracking member (330) and the controller body (310). Alternatively, the tracking member (330) and the controller body (310) may comprise separately manufactured parts that are later assembled together. In either case, the tracking member (330) may be attached to the controller body (310).

The hand retainer (320) is shown in the open position in FIG. 3. The hand retainer (320) may be selectively deflected in the open position by the curved elastic member (322) to facilitate insertion of the user's left hand between the hand retainer (320) and the controller body (310) when the user is holding the controller (300) whose field of view is blocked by the VR headset (e.g., the VR headset (108)). For example, the curved elastic member (322) may comprise a flexible metal strip that is resiliently bendable, or may comprise an alternative plastic material, such as nylon, that is substantially resiliently bendable. A cushion or fabric material (324) (e.g., a neoprene outer shell) may partially or completely cover the curved elastic member (322) to provide user comfort. Alternatively, the fabric material (324) may be attached only to a side of the curved elastic member (322), such as the side facing the user's hand.

The hand retainer (320) may be length-adjustable, for example, by including a draw cord (326) that is tightened by a spring biased choke (328). The draw cord (326) may have excess length that is used as a strap. A cushion or fabric material (324) may be coupled to the draw cord (326). In certain embodiments, the tension of the tightened draw cord (326) may preload the curved elastic member (322). In such embodiments, the tension imparted to the hand retainer (320) by the curved elastic member (322) (to bias it in the open position) may cause the hand retainer (320) to automatically open when the draw cord (326) is released. Additionally or alternatively, the length of the hand retainer (320) may be adjusted in other ways, such as by attaching a cleat, an elastic band (which temporarily stretches when the hand is inserted, thus providing elastic tension to press against the back of the hand), a hook and loop strap to allow for length adjustment, etc.

The hand retainer (320) is positioned between the handle (312) and the tracking member (330) and can come into contact with the back of the user's hand. For example, FIG. 4 illustrates the controller (300) while the user's left hand is inserted therein but is not holding the controller body (310). In FIG. 4, the hand retainer (320) is closed and tightened over the hand to physically bias the user's palm against the outer surface of the handle (312). When closed, the hand retainer (320) can maintain the controller (300) in the user's hand even when the user is not holding the controller body (310).

The handle (312) of the controller body (310) includes an array of proximity sensors that are spatially distributed partially or completely around the outer surface. In some cases, the proximity sensors of the proximity sensor array need not necessarily be of the same size and need not have the same spacing between them. In some cases, the proximity sensor array may include a grid spatially distributed around the controller body (310). The proximity sensor array responds to the proximity of a user's finger(s) to the outer surface of the handle (312). For example, the proximity sensor array may include an array of capacitive sensors embedded beneath the outer surface of the handle (312), wherein the outer surface includes an electrically insulating material for detecting touch from the user. The capacitance between the capacitive sensor array and a portion of the user's hand may be inversely proportional to the distance therebetween. To sense the capacitance, an RC oscillator circuit may be connected to elements of the capacitive sensor array, and it may be noted that the time constant of the RC oscillator circuit, and thus the oscillation period and frequency, will vary depending on the capacitance. In this manner, the circuit can detect the release of the finger(s) from the outer surface of the handle (312). As mentioned above, the proximity sensor array can generate touch data (e.g., touch data (124)) in response to touch input from a user, where the touch data indicates the proximity of the user's finger(s) to the outer surface of the handle (312).

The hand retainer (320) can be configured to close tightly around the user's hand to prevent the controller (300) from falling out of the user's hand and to prevent excessive movement of the fingers relative to the proximity sensor array on the handle (312), thereby reliably detecting finger motions and positions. Additionally, the motion capture system(s) (102) and/or remote computing resource(s) (112) can include algorithms that implement anatomically possible finger motions to better utilize touch data (124) from the proximity sensor array to render hand unfolding, finger pointing, or other motions of the fingers relative to the controller (300) or relative to each other (e.g., hand gestures) of the controlled character. In this manner, the user's movements of the controller (300) and/or fingers can aid in controlling a VR gaming system, a defense system, a medical system, an industrial robot or machine, or other device. In VR system applications (e.g., for gaming, training, etc.), the system can render a throwing motion based on the movement of the tracking transducers, and can render the release of the thrown object by detecting the release of the user's fingers (e.g., using touch data (124)) from the outer surface of the handle (312) of the controller (300).

Thus, the hand retainer (320) allows the user to "let go" of the controller (300) without the controller (300) actually being separated from the hand, placed on the floor, thrown, and/or dropped, which may enable additional functionality of the controlled electronics system. For example, detecting the user's release and/or restoration of their grip on the handle (312) of the controller body (310) may indicate corresponding throws and/or grabs of objects within gameplay. Thus, the hand retainer (320) may securely secure and maintain the user's hand during such animations. In some cases, the position of the hand retainer (320) in the embodiments of FIGS. 3-7 may assist the tracking member (330) in protecting the back of the user's hand from impact in the real world, for example, when the user moves in response to a sensed prompt in the VR environment (e.g., while substantially blinded by the VR headset (108).

As discussed herein, the controller (300) may include an FSR to detect force values associated with touches from a user (e.g., force data (126)). The force data (126) may be used in conjunction with touch data (124) to indicate movements and/or grips of the user in a VR environment.

In certain embodiments, the controller (300) may include a rechargeable battery disposed within the controller body (310) and/or the hand retainer (320) (e.g., a hand retention strap) may include an electrically conductive charging wire electrically coupled to the rechargeable battery. The controller (300) may also include a radio frequency (RF) transmitter for communicating with the remaining motion capture system(s) (102). The rechargeable battery may power the RF transmitter, which may be responsive to thumb-operated controls (314, 315, 316), a proximity sensor array in the handle (312) of the controller body (310), and/or tracking sensors in the tracking member (330).

Figures 5 and 6 illustrate the controller (300) during operation when the hand retainer (320) is closed and when the hand is holding the controller body (310). Figures 5 and 6 also illustrate that the thumb can actuate one or more of the thumb-operated controls (e.g., the track pad (316)).

FIG. 7 illustrates that in certain embodiments, the controller (300) may include a left controller of a pair of controllers that may include a similar right controller (700). In certain embodiments, the controllers (300, 700) may individually generate touch data (124) and/or force data (126) from the proximity sensors and FSR arrays, respectively, from the user's two hands simultaneously. Collectively, the remote computing resource(s) (112) may receive motion data (122) (from the camera(s) of the motion capture system(s) (102)) as well as touch data (124) and/or force data (126) (from the controllers (300, 700)) to enhance the VR experience.

FIGS. 8A and 8B illustrate front and rear views of a right controller (800), respectively, according to other exemplary embodiments of the present disclosure. In some cases, the right controller (800) may include components discussed above with respect to the controller(s) (110) of FIG. 1 and the controller (300) of FIGS. 3-7.

The controller (800) may include a controller body including a head (810) and a handle (812). In the embodiments of FIGS. 8A and 8B, the head (810) may include at least one thumb-operable control (A, B, 808), and may also include an index finger-operable control (e.g., a trigger (809)). In some cases, the handle (812) may include a tubular housing partially enclosed by an outer shell (840).

The inner surface of the outer shell (840) may include a spatially distributed array of proximity sensors. The proximity sensor array may respond to the proximity of a user's fingers to the outer shell (840). The proximity sensors of the proximity sensor array need not necessarily be of the same size, and they need not necessarily be regularly or evenly spaced from one another. In certain embodiments, the proximity sensor array may be a plurality of capacitive sensors that may be connected to a flex circuit bonded to the inner surface of the outer shell (840).

The tracking member (830) may be attached to the controller body at the ends of the head (810) and the handle (812). The hand retainer (820) is configured to physically bias the user's palm against the outer shell (840) between the ends of the head (810) and the handle (812). The hand retainer (820) is preferably positioned between the handle (812) and the tracking member (830) and may include a hand retainer strap that is adjustable in length and contacts the back of the user's hand. In the embodiments of FIGS. 8A and 8B , the hand retainer (820) may include a draw cord (828) that is adjustable in length by a cord lock device (826) (adjacent to the distal end of the handle (812)) that selectively prevents sliding motion by the draw cord (828) at the location of the cord lock device (826).

In the embodiments of FIGS. 8A and 8B, tracking transducers (832, 833) are positioned on the tracking member (830). In some cases, protruding protrusions at opposite distal ends of the tracking member (830) may include tracking transducers (822, 833). In some cases, the distal region of the head (810) may include additional tracking transducers (834). The tracking transducers (832, 833, and 834) may include tracking sensors that respond to electromagnetic radiation (e.g., infrared light) emitted by the motion capture system(s) (102), or may include tracking beacons that emit electromagnetic radiation (e.g., infrared light) received by the motion capture system(s) (102). For example, the motion capture system(s) (102) may include projector(s) that widely broadcasts pulsed infrared light toward the controller (800). Here, the plurality of tracking transducers (832, 833, 834) may include infrared light sensors that receive the broadcasted pulsed infrared light. The motion capture system(s) (102) may receive responses from the tracking sensors, and the motion capture system(s) (102) and/or remote computing resource(s) (112) may interpret these responses to effectively track the position and orientation of the controller (800).

A printed circuit board (PCB) may be mounted within the handle (812) and may electrically connect components within the controller (800), such as buttons, a battery, etc. The PCB may include a force sensing resistor (FSR), and the controller (800) may include a plunger that transfers a compressive force applied through the outer shell (840) inwardly toward the exterior of the tubular housing of the handle. In certain embodiments, the FSR in conjunction with a proximity sensor array may facilitate detection of both the initiation of a grip by a user and the relative strength of such grip by the user, which may facilitate certain gameplay features.

Example force sensing resistor (FSR)

FIGS. 9A through 9C illustrate different views of a force sensing resistor (FSR) (900) according to an exemplary embodiment of the present disclosure. As shown in the cross-section of the FSR (900) in FIG. 9C , the FSR (900) may include a first substrate (902), which in some cases may include polyimide. The FSR (900) may further include a second substrate (904) disposed on (or over) the first substrate (902). The first substrate (902) and the second substrate (904) may include two primary substrates (or layers) of the FSR (900) (i.e., a two-layer FSR (900)). However, it should be understood that the FSR (900) may include additional layers. In some cases, the first substrate (902) may represent the “bottom” or “base” substrate for the two primary substrates of the FSR (900), although in some cases there may be layers of material behind (or beneath) the first substrate (902) (i.e., in the negative Z direction, as illustrated in FIG. 9C ).

The first substrate (902) includes a conductive material disposed on a front surface (i.e., a surface facing the positive Z direction) of the first substrate (902). The conductive material may include a plurality of interlocked metal fingers. Meanwhile, the second substrate (904) (sometimes referred to as a resistive “film”) may include a resistive material disposed on a back surface (i.e., a surface facing the negative Z direction) of the second substrate (904). The resistive material may include a semiconductor material, such as an ink composition (e.g., silver ink, carbon ink, mixtures thereof, etc.) that exhibits a certain level of electrical resistance (e.g., a relatively high sheet resistance in the range of 300 kiloohms (kOhm) per square (kOhm) to 400 kOhm per square (kOhm/sq). In some cases, the sheet resistance of the second substrate (904) is 350 kOhm/sq. However, the second substrate (904) may include other sheet resistance values, including values outside the sheet resistance ranges specified herein, such as when the FSR (900) is used in other applications (e.g., non-controller based applications). In some embodiments, the material of the second substrate (904) may include Mylar, with the resistive material disposed on the backside of the second substrate (904). In some embodiments, the second substrate (904) may be made of polyimide having a resistive material (e.g., a conductive ink composition) on the backside. The use of polyimide for the second substrate (904) allows for mass production of the FSR (900) using a reflow oven, whereas Mylar may not be able to withstand such high temperatures.

The FSR (900) may include one or more spacer layers interposed between the first substrate (902) and the second substrate (904) such that a central portion of the second substrate (904) is suspended above and spaced apart from the first substrate (902). FIG. 9C illustrates two spacer layers, including, without limitation, a coverlay (906) disposed on the first substrate (902) around the periphery of the first substrate (902) and an adhesive layer (908) disposed on the coverlay (906). The material of the coverlay (906) may include polyimide, and thus may include the same material as the first substrate (902). The thickness of the coverlay (906) (measured in the Z direction) may range from 10 microns to 15 microns. The thickness of the adhesive layer (908) (measured in the Z direction) may range from 50 microns to 130 microns. Thus, the total distance that the second substrate (904) is spaced from the first substrate (902) may be the sum of the thicknesses of one or more spacer layers (e.g., the thickness of the coverlay (906) plus the thickness of the adhesive layer (908). These layers may include thicknesses outside the thickness ranges specified herein, such as when the FSR (900) is used in other applications, such as non-controller based applications. As such, it is to be understood that these thickness ranges are non-limiting. Additionally, in some cases, the thickness of the adhesive layer (908) is made as thin as possible (e.g., at the low end of the specified thickness range) to allow for an initial response (e.g., the FSR (900) begins to sense the input) under very lightly applied forces (F). The adhesive may be varied in both material and thickness to increase or decrease the stiffness of the FSR (900).

The substrate (904) may include an actuator (910) (e.g., a disk-shaped, compliant plunger) configured to transmit a force (F) across the front surface of the second substrate (904). The material of the actuator (910) may include Poron, a compliant material that deforms to some extent when force is applied to the actuator (910). The actuator (910) may be concentric with the center of the active area of the FSR (900) to center the applied force (F). The actuator (910) may also span a portion of the active area of the FSR (900) to evenly distribute the applied force (F) across that portion of the active area of the FSR (900).

The thickness (measured in the Z direction) of the second substrate (904) can range from 50 microns to 130 microns. At this exemplary thickness, the second substrate (904) is flexible. For example, the material of the second substrate (904) can include flexible Mylar with a thickness within the range described above. The functional operation of the FSR (900) relies on the flexibility of the second substrate (904) to allow the resistive material on the back surface of the second substrate (904) to contact the conductive material on the front surface of the first substrate (902) under a compressive force (F) applied to the actuator (910). The thickness (measured in the Z direction) of the first substrate (902) can range from 20 microns to 30 microns. At this thickness, the polyimide is also flexible. Thus, the first substrate (902) is also flexible. Meanwhile, the thickness (measured in the Z direction) of the actuator (910) may range from 780 microns to 810 microns. These layers may include thicknesses outside the thickness ranges specified herein, such as when the FSR (900) is used in other applications (e.g., non-controller based applications). As such, these thickness ranges should be understood to be non-limiting.

The FSR (900) can exhibit a varying resistance in response to a variable force (F) applied to the actuator (910). For example, as the force (F) applied to the actuator (910) increases, the resistance can decrease. In this manner, the FSR (900) can be represented as a variable resistor whose value is controlled by the applied force (F). The FSR (900) can include a “shunt mode” FSR (900) or a “through mode” FSR (900). In a shunt mode FSR (900), the conductive material disposed on the front surface of the first substrate (902) can include a plurality of interlocked metal fingers. When an applied force (F) is applied to the front (or top) of the actuator (910), the resistive material on the back surface of the second substrate (904) may contact some of the interdigitated metal fingers, which shunts the interdigitated metal fingers, thereby changing the resistance across the output terminals of the FSR (900). An example of a conductivity for the interdigitated metal fingers may include copper, such as HA copper or RA copper. The interdigitated metal fingers may also include metal plating.

The subtractive manufacturing process can form a plurality of interdigitated metal fingers. The finger width and spacing between the interdigitated metal fingers can provide an optimal balance between maximum sensitivity of the FSR (900) and minimal manufacturing etch tolerances. In some cases, the interdigitated metal fingers can include a uniform pattern or non-uniform patterns (e.g., denser fingers toward the center and less dense fingers toward the outside). Additionally, in some cases, no additional copper plating can be provided over the underlying copper prior to gold plating, as adding additional copper plating over the underlying copper prior to gold plating can result in an undesirable increase in sensed resistance. Thus, in some cases, omitting any additional copper plating on the interdigitated metal fingers prior to gold plating can achieve optimal sensitivity in the FSR (900).

In a through-mode implementation, the conductive material on the first substrate (902) can include a solid region of conductive material having a semiconducting (or resistive) material disposed thereon. The second substrate (904) can have a similar structure (e.g., a solid region of conductive material having a semiconducting (or resistive) material disposed thereon). The solid regions of conductive material on each of the substrates (902 and 904) can be coupled to individual output terminals, such that when the two substrates (902 and 904) are in contact under an applied force (F), current can pass through one layer to the other.

Through these implementations, the FSR (900) can have less hysteresis and more repeatability (from one FSR (900) to another FSR (900)) than conventional FSRs, such as those that use Mylar as the material of the lower substrate. The loading hysteresis describes the effect of a previously applied force on the current FSR (900) resistance. The response curves also model a true analog input that can be utilized in many game mechanics of a VR game system, such as breaking virtual rocks or squeezing virtual balloons. However, while the examples herein describe an applied force (F), the FSR (900) is actually sensitive to the applied pressure (force x area), since the same amount of force applied to a small spot on the second substrate (904) versus a large area on the first surface can result in different resistance responses of the FSR (900). Therefore, the actuator (910) can serve to maintain repeatability across the FSR (900) in terms of the response curve under the applied force (F).

The FSR (900) may include an open circuit in the absence of an external force (or load). In some embodiments, to account for any contact of the first substrate (902) and the second substrate (904) under zero or negligible applied force, the threshold circuit may set a threshold resistance value above which the first substrate (902) and the second substrate (904) are considered to be "in contact," meaning that even if the two primary substrates (i.e., 902 and 904) are actually in contact, the FSR (900) may indicate an open circuit until the threshold resistance value is met.

The FSR (900) may be mounted on a planar surface of a structure within a handheld controller, such as the controllers (110, 300, 800) disclosed herein. The FSR (900) may be mounted at any suitable location within the controller body to measure a resistance value corresponding to an amount of force associated with user touch inputs applied to an external surface of the controller body (e.g., force applied by a finger pressing when controlling, force applied by hand gripping a handle of the control). The FSR (900) may be mounted on a planar surface of the PCB, and may itself be mounted within the tubular housing of the handle. In this configuration, the plunger may interface with the actuator (910) of the FSR (900), which may allow a compressive force to be transferred from the plunger to the actuator (910). However, other configurations are possible in which the plunger is omitted and the actuator (910) interfaces with a portion of the tubular housing of the handle.

Additionally, or alternatively, the FSR (900) may be mounted on a planar surface of the structure within the head (between the handle and the distal end). The structure may be mounted within the head under one or more of the thumb-actuated controls. For example, the FSR (900) may be mounted under a thumb-actuated control (e.g., a track pad). Thus, when the user's thumb depresses the thumb-actuated control while the controller is in operation, the FSR (900) located under the thumb-actuated control may measure a resistance value corresponding to the amount of force applied by the user's thumb to the thumb-actuated control. In some cases, the controller may include multiple FSRs (900) disposed within the controller body, such as one or more FSRs (900) mounted within the handle and/or one or more FSRs (900) mounted under one or more corresponding controls on the head of the controller body.

The FSR (900) may activate variable analog inputs when implemented in a controller. For example, the resistance of the FSR (900) may be varied based on the applied force (F) by gripping the handle with varying amounts of force or pressing the thumb-operated control(s). The resistance may be converted to various digital values representing FSR inputs for controlling game mechanics (e.g., picking up and throwing objects).

The FSR (900) may utilize a variety of touches or touch styles. For example, a "simple threshold" style may mean that an FSR input event occurs when a digitized FSR input value meets or exceeds a threshold value. Since the digitized FSR input value corresponds to a particular resistance value measured by the FSR (900), which in turn corresponds to a particular amount of force applied to the FSR (900), a "soft press" of this style may be thought of as registering an FSR input event when the resistance value measured by the FSR (900) meets the threshold resistance value and/or when the applied force (F) meets a threshold amount of force. For example, if a handle of a controller (e.g., controllers (110, 300, and/or 800)) includes an FSR (900), the handle may be squeezed until a force threshold is reached, in response to which an FSR (900) input event is registered as a "soft press." The force required to "unpress" may be part of a threshold for debounce purposes and/or to mimic a tact switch with a physical snap rate. The "hair trigger" style may set a baseline threshold, and when a digitized FSR input value associated with the FSR (900) meets or exceeds the baseline threshold, the binding is activated (i.e., the FSR input event is registered, similar to a press-and-hold button actuation). Any subsequent decrease in force will then deactivate the binding (i.e., the FSR input event is "unregistered", similar to the user releasing the button), and an increase in force after deactivating the binding will reactivate the binding. The "hip fire" style may be similar to the "simple threshold" style of the soft press, except that in configurations with multiple binding levels, the "hip fire" style may be used to utilize a time delay to override lower FSR input values if the time delay reaches the higher threshold quickly enough. The amount of time delay varies between different substyles (e.g., aggressive, normal, and relaxed).

In some cases, the additional soft press thresholds may include multi-level thresholds, such as a threshold for a "hip fire" style of soft press. Different styles of soft presses for FSR-based inputs may allow the user to activate a number of different game-related analog inputs by gripping or pressing the FSR-based input mechanism with different forces. For example, a VR game may allow the user to squeeze the handle of the controller body with increasing force to break rocks or squeeze balloons. As another example, a shooter-based game may allow the user to toggle between different types of weapons by pressing a thumb-operated control with different levels of authorization.

In some cases, the user may adjust the threshold to reduce hand fatigue associated with operating the FSR-based input mechanism. In some cases, the threshold may include a default threshold for a particular game (e.g., a lower default threshold for a shooter game, a higher default threshold for an exploration game, etc.).

Example hand gestures

FIGS. 10A-10F illustrate different variations of a user (1000) holding a controller (1002) (which may represent and/or be similar to the controller (110) of FIGS. 1 and 2 , the controller (300) of FIGS. 3-7 , and/or the controller (800) of FIG. 8 ). In general, and in accordance with the disclosure, by detecting the location and force of a touch input of the user (1000) at the controller (1002) (e.g., using a proximity sensor and/or an array of FSRs (900)), the remote computing resource(s) (112) may generate an animation (e.g., the animation (128)) for display on the VR headset (108). The animation may be similar to the hand gestures depicted in FIGS. 10A-10F , respectively. That is, using previously trained model(s) (120), the remote computing resource(s) (112) can generate images of the hand based on touch data (124) and/or force data (126) received from the controller(s) (1002).

Beginning with FIG. 10A, a user (1000) is depicted holding a controller (1002) in an unfolded position. The user's (1000) fingers and thumb do not contact the controller (1002), but instead the controller (1002) may contact the palm of the user's (1000) hand. The controller (1002) may detect this contact, generate touch data (124) and/or force data (126), and transmit the touch data (124) and/or force data (126) to the remote computing resource(s) (112). Here, the touch data (124) may represent or indicate that the palm of the user's (1000) is touching the controller (1002). Since the user (1000) is not gripping the controller (1002) of FIG. 110a with his or her fingers, the force data (126) may represent the level of force with which the palm of the user (1000) is biased against the controller (1002). In some cases, since the user (1000) is not gripping the controller (1002), the controller (1002) may only generate touch data (124) that represents the proximity of the user's finger(s) to the outer surface of the handle (312).

The remote computing resource(s) may input touch data (124) and/or force data (126) to a model(s) (120) that may generate hand image data (e.g., animation (128)) corresponding to the hand gesture. In some cases, the remote computing resource(s) (112) may select a designated model(s) (120) to input touch data (124) and/or force data (126). In some cases, in gameplay, an extended hand gesture may indicate picking up an object, dropping an object, etc.

FIG. 10b illustrates a user (1000) holding a controller (1002) with all four fingers and a thumb. Here, touch data (124) generated by a proximity sensor array of the controller (1002) may represent the grip of the user (1000). Force data (126) generated by an FSR (e.g., FSR (900)) may represent the force with which the user (1000) holds the controller (1002). The controller (1002) may transmit the touch data (124) and/or the force data (126) to a remote computing resource(s) (112), and the remote computing resource(s) (112) may select a model(s) (120) corresponding to the touch data (124) and/or the force data (126). The animation (128) corresponding to the model(s) (120) can generate hand gestures representing a clenched fist gesture, a grasping gesture, etc.

FIG. 10c illustrates a user (1000) holding a controller (1002) with all four fingers excluding the thumb. In this example, the remote computing resource(s) (112) can utilize the touch data (124) and/or force data (126) to determine associated model(s) (120), wherein the model(s) (120) represents that the user (1000) is holding an object with all four fingers excluding the thumb. The model(s) (120) can generate an animation (128) for display on the VR headset (108) that represents this configuration of touch (e.g., thumb, trigger actuator, etc.) on the controller (1002).

FIG. 10d illustrates a user (1000) holding a controller (1002) with a middle finger and a fourth finger. Here, the touch data (124) may represent the touch of the middle finger and the fourth finger. The touch data (124) may also represent the proximity of the index finger and the little finger (which are not in contact with the controller (1002)) to an outer surface of the handle of the controller (1002). The force data (126) may represent force values associated with the grip of the middle finger and/or the fourth finger of the user (1000). The model(s) (120) may generate associated animations (128) based on the touches of the middle finger and the fourth finger and their associated force values.

FIG. 10e illustrates a user (1000) holding a controller (1002) with a fourth finger and a little finger. The remote computing resource(s) (112) can utilize touch data (124) associated with the touch of the fourth finger and little finger and/or the lack of touch of the index finger and/or middle finger to select associated model(s) (120), corresponding animation (128), and generate a hand gesture for display on the VR headset (108). The remote computing resource(s) (112) can also utilize force data (126) generated from the FSR when selecting model(s) (120) and generating the hand gesture.

FIG. 10f illustrates a user holding a controller (1002) with an index finger, a middle finger, and a little finger. The remote computing resource(s) (112) uses touch data (124) and/or force data (126) to generate an associated hand gesture on the VR headset (108), such as the user (1000) firing a weapon.

FIGS. 10A-10F illustrate specific combinations of a user's (1000) fingers and thumbs touching the controller (1002) to generate associated hand gestures, although other combinations are possible. In these situations, the controller (1002) may use the FSR (900) to detect a force associated with the user's (1000) touch input, as well as a proximity sensor array to detect a position associated with the touch input. The controller (1002) may transmit the touch data (124) and/or the force data (126) to the remote computing resource(s) (112), which may select the model(s) (120) corresponding to the touch data (124) and/or the force data (126). As noted above, the model(s) (120) may be previously trained and/or generated using previous motion data (122), touch data (124), and/or force data (126). Thus, later, by receiving the touch data (124) and/or force data (126), the remote computing resource(s) (112) can associate the touch data (124) and/or force data (126) with one or more model(s) (120). As the model(s) (120) are associated with an animation (128), the remote computing resource(s) (112) can select the one or more model(s) (120), generate a corresponding animation (128), and transmit the animation (128) to the VR headset (108) for display.

Example processes

Figures 11-13 illustrate various processes according to embodiments of the present application. The processes described herein are illustrated as a collection of blocks in a logical flow diagram representing a series of operations, some or all of which may be implemented in hardware, software, or a combination thereof. In the context of software, the blocks may represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, cause the processors to perform the operations noted. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the blocks are described should not be construed as limiting, unless specifically stated otherwise. Any number of the described blocks may be combined in any order and/or in parallel to implement a process or alternative processes, and not all of the blocks need be executed. For purposes of discussion, the processes are described with reference to the environments, architectures and systems described in the examples herein, such as those described in connection with FIGS. 1 through 10, although the processes may be implemented in various other environments, architectures and systems.

Beginning with FIG. 11, at block (1102), the process (1100) may receive touch data corresponding to a touch input at the controller. The touch data may indicate the location(s) on the controller at which the touch input was received and/or the proximity of the user's finger(s) to the controller (e.g., a capacitance value from an array of proximity sensors or capacitive sensors).

At block (1104), the process (1100) may receive force data corresponding to a touch input at the controller. The force data may represent an amount of force associated with the touch input at the controller. In some cases, the force data may be received when force values exceed a particular force threshold.

At block (1106), the process (1100) may receive motion data corresponding to movements of a user manipulating a controller. The motion data may represent movements of the user, such as movements of the user's finger(s) and wrist(s). The motion data may also represent motions of the controller.

At block (1108), the process (1100) can train the model(s) using the touch data, the force data, and/or the motion data. For example, to train the model(s), the process (1100) can associate the touch data, the force data, and/or the motion data to correspond to movements of the user, as represented by the motion data. That is, using the touch data, the force data, and/or the motion data, the process (1100) can train the model(s) to learn characteristics of the user's touch and associate those characteristics with specific hand gestures determined from the motion data. In some cases, the characteristics can include the location and force of the touch input(s) on the controller. In some cases, the touch data, the force data, and/or the motion data can be associated using a timestamp corresponding to when the touch data, the force data, and/or the motion data were each captured. By overlaying touch data, force data, and/or motion data on a time scale, the process (1100) can associate the touch data and/or force data with the motion data and identify the user's hand gesture(s). When training the model(s), in later examples, the process (1100) can receive the touch data and/or force data and determine the associated gesture (without receiving the motion data).

At block (1108), the process (1100) may loop back to block (1102) to receive additional touch data, additional force data (e.g., block (1104)), and/or additional motion data (e.g., block (1106)). This additional data may be used to further train the model(s), which may allow for more accurate hand gesture determinations based on the touch data and/or force data received in later instances (e.g., during gameplay). That is, the process (1100) may continue to correlate the touch data, force data, and/or motion data such that as the process (1100) receives subsequent touch data and/or force data, the process (1100) can accurately determine associated hand gestures corresponding to the touch data and/or force data (via the model(s)). Here, correlating the touch data, the force data, and/or the motion data, as discussed above, may include matching a timestamp of the touch data, a timestamp of the force data, and/or a timestamp of the motion data.

At block (1110), the process (1100) may receive touch data. In some cases, the touch data received at block (1110) may correspond to touch data received during gameplay.

At block (1112), the process (1100) may receive force data. In some cases, the force data received at block (1112) may correspond to touch data received during gameplay.

At block (1114), the process (1100) may select model(s). In some cases, to select model(s), the touch data received at block (1100) may be compared to touch data or a touch profile corresponding to previously generated model(s). Additionally or alternatively, selecting model(s) may include comparing force data received at block (1112) to force data or a touch profile corresponding to previously generated model(s). The touch profile of the model(s) may include force data indicative of a hand gesture of the model(s) and/or force values associated with a location associated with the touch data indicative of a hand gesture of the model(s). As an example, the touch data may represent a touch input at the center of the controller, such as a middle finger and/or an index finger touching the controller (e.g., FIG. 10d ). In some cases, the touch data may associate touch inputs with particular fingers of the user and/or may represent fingers that are not touching the controller. Using touch data and/or force data, the model can be selected.

At block (1116), the process (1100) can input touch data and/or force data into the model(s). More specifically, since the model(s) have previously been trained to associate touch data and/or force data with motion data and corresponding hand gestures, once trained, the model(s) can receive the touch data and/or force data and determine hand gestures. In other words, the touch data can indicate which fingers are gripping the controller or which fingers are not gripping the controller, as well as the location on the controller corresponding to the touch, or lack thereof. Thus, after the model(s) are trained, the model(s) can accept touch data and/or force data representing touches from a user received during gameplay.

At block (1118), the process (1100) can generate image data corresponding to the touch data and/or force data. For example, after inputting the touch data and/or force data into the model(s), the process (1100) can generate a hand gesture using the touch data and/or force.

At block (1120), the process (1100) may present image data on the display. At process (1100), the presentation of a hand gesture on the display may correspond to a hand gesture of a user interacting with the controller. Additionally, to reduce latency between the reception of touch data and the presentation of image data, the process may perform blocks (1110-1120) in real time and/or substantially concurrently with one another.

From block (1120), process (1100) can loop to block (1110). Therein, process (1100) can loop between blocks (1110, 1120) to continuously receive touch data and force data and generate animations corresponding to touch inputs from a user. By doing so, when a user plays a game, touch data and/or force data received from the controller can be changed according to levels, scenes, frames, etc. in the game. By continuously inputting touch data and/or force data into the model(s), process (1100) can continuously generate hand gestures to select and display the corresponding model(s).

As previously mentioned, in some cases, the process (1100) between blocks (1102) and (1108) may occur during a first time instance in which the model(s) are trained while the user is not playing in gameplay mode. For example, the training (or generation) of the model(s) may occur in a facility where motion data (captured from motion capture system(s) (102)), touch data, and/or force data are captured and correlated with each other to associate the touch data and/or force data with specific hand gestures. At a later time instance, after the model(s) are trained, the process (1100) between blocks (1110) and (1120) may occur while the user is in gameplay mode.

As illustrated in FIG. 12, at block (1202), the process (1200) may receive touch data corresponding to touch inputs at the controller. For example, the remote computing resource(s) (112) may receive touch data (124) from the controller (e.g., controller (110, 300, and/or 800)). The touch data (124) may indicate location(s) on the controller corresponding to the user's touch input(s). For example, the touch data (124) may indicate that all four of the user's fingers are touching the controller, indicating the locations of the touch(s), or in some cases, indicating which fingers are not touching the controller and/or which areas of the controller are not receiving touch inputs.

At block (1204), the process (1200) may receive force data corresponding to a touch input at the controller. For example, the remote computing resource(s) (112) may receive force data (126) from a controller (e.g., controller (110, 300, and/or 800)). The force data (126) may represent an amount of force associated with a touch input at the controller or a relative strength associated with a user's grip on the controller. In an example, where the user is not gripping the controller (e.g., as illustrated in FIG. 10A), the remote computing resource(s) (112) may not receive force data (126) from the controller.

At block (1206), the process (1200) can receive motion data corresponding to movements of the user manipulating the controller. For example, the remote computing resource(s) (112) can receive the motion data (122) from the motion capture system(s) (102). The motion data (122) can represent movements of the user and/or movements of the controller using markers (200, 202). As noted above, the projector(s) of the motion capture system(s) (102) can project light onto the markers (200, 202) placed on the user and/or the controller. The markers (200, 202) can reflect this light, which is then captured by the camera(s) of the motion capture system(s) (102).

At block (1208), the process (1200) can train the model(s) using the touch data, the force data, and/or the motion data. For example, the remote computing resource(s) (112) can train (or generate) the model(s) (120) using the motion data (122), the touch data (124), and/or the force data (126). In some examples, training the model(s) (120) can include associating the touch data (124), the force data (126), and/or the motion data (122) to determine characteristics of the touch data (124) and/or the force data (126) that correspond to the movements of the user. In doing so, the remote computing resource(s) (112) can generate image data or animation(s) corresponding to the touch data (124) received from the controller. That is, by correlating the touch data (124), the force data (126), and/or the motion data (122), the remote computing resource(s) (112) can, at a later time, upon receiving the touch data (124) and/or the force data (126), correlate the touch data (124) and/or the force data (126) with a user's gesture using the previous motion data (122). In some cases, correlating the touch data (124), the force data (126), and/or the motion data (122) may include matching a timestamp of the touch data (124), a timestamp of the force data (126), and/or a timestamp of the motion data (122). By doing so, the remote computing resource(s) (112) can learn (e.g., using a machine learning algorithm) how the touch data (124) and/or the force data (126) are associated with the user's hand gestures.

From block (1208), the process (1200) can loop to block (1202) to receive additional touch data (124), additional force data (126), and/or additional motion data (122). For example, the remote computing resource(s) (112) can receive additional touch data (124) (e.g., block (1202)), additional force data (126) (e.g., block (1204)), and/or additional motion data (122) (e.g., block (1206)) to train the model(s) (120). Training the model(s) (120) may allow for more accurate determination of hand gestures performed by the user.

As illustrated in FIG. 13, at block (1302), the process (1300) may receive touch data. For example, the remote computing resource(s) (112) may receive touch data (124) from a controller (e.g., controller (110, 300, and/or 800)). In some cases, a proximity sensor array of the controller may generate the touch data (124). The touch data (124) may indicate the placement of a user's finger or hand on the controller (110).

At block (1304), the process (1300) may receive force data. For example, the remote computing resource(s) (112) may receive force data (126) from a controller (e.g., controller (110, 300, and/or 800)). In some cases, an FSR of the controller (e.g., FSR (900)) may generate force data (126) that may represent an amount of force associated with a user's touches on the controller.

At block (1306), the process (1300) may input touch data and/or force data into the model(s). For example, the processor(s) (116) of the remote computing resource(s) (112) may input touch data (124) and/or force data (126) into the model(s) (120). More specifically, since the model(s) (102) has previously been trained to associate touch data (124) and/or force data (126) with motion data (122) and hand gestures, once trained, the model(s) (120) may receive the touch data (124) and/or force data (126) to determine hand gestures. In some cases, the remote computing resource(s) (112) may optionally input the touch data (124) and/or the force data (126) into model(s) (120) that closely matches or is associated with the touch data (124) and/or the force data (126). For example, if the touch data (124) indicates that the user is gripping the controller (110) with four fingers, the processor(s) (116) may select a model (120) corresponding to a four-finger grip.

At block (1318), the process (1300) can generate image data corresponding to the touch data and/or the force data. For example, the processor(s) (116) of the remote computing resource(s) (112) can determine a hand gesture corresponding to the touch data (124) and/or the force data (126) using the model(s) (120). The remote computing resource(s) (112) can generate image data, such as an animation (128) corresponding to the hand gesture. For example, the model(s) (120) can generate an animation (128) of a hand (e.g., breaking a rock or dropping an object) using the touch data (124) and/or the force data (126).

At block (1320), the process (1300) may present image data on a display. For example, the remote computing resource(s) (112) may transmit the image data to the VR headset (108) (or another computing device), whereby the VR headset (108) may display the image data. The VR headset (108) may display a hand gesture on the display based on touch data (124) and/or force data (126) received from the controller(s) (110). In this way, the presentation of the hand gesture on the display of the VR headset (108) may be correlated with the hand gesture with which the user interacts with the controller(s) (110). Additionally, to reduce latency between the receipt of the touch data (124) and the presentation of the image data on the VR headset (108), the process (1300) may perform blocks (1302-1310) in real time and/or substantially concurrently with one another. Additionally, pre-creation of the model(s) (120) may allow for faster computing when receiving touch data (124) and/or force data (126) to generate associated hand gestures.

From block (1310), the process (1300) can be looped to block (1302), where the process (1300) can be repeated between blocks (1302 and 1310) to continuously generate image data. As a result, touch data (124) and/or force data (126) corresponding to the manner in which the user grasps and grips the controller (110) can be updated, and by inputting the touch data (124) and/or force data (126) into the model(s) (120), the process (1300) can continuously generate hand gestures for display on the VR headset (108).

conclusion

While the invention has been described with reference to specific examples, it is to be understood that the scope of the invention is not limited to these specific examples. Since other modifications and variations to suit particular operating requirements and environments will be apparent to those skilled in the art, the invention is not to be considered limited to the examples selected for the purpose of disclosure, but includes all changes and modifications that do not constitute a departure from the true spirit and scope of the invention.

Although this application describes embodiments having particular structural features and/or methodological acts, it should be understood that the scope of the claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are merely some embodiments that fall within the scope of the claims of this application.

Exemplary provisions

Embodiments of the present disclosure can be described in terms of the following provisions:

1. In the system,

one or more processors; and

One or more non-transitory computer-readable media, which, when executed by said one or more processors, cause said one or more processors to:

A step of receiving first motion data corresponding to a first movement of a user's hand that operates one or more controllers through one or more sensors;

A step of receiving first touch data corresponding to a first touch of the user's hand that operates the one or more controllers through the one or more controllers;

A step of receiving first force data corresponding to the first touch of the user's hand that operates the one or more controllers through the one or more controllers;

A step of associating the first motion data with the first touch data and the first force data;

A step of generating a model corresponding to a gesture of the hand based at least in part on the first motion data, the first touch data and the first force data;

A step of receiving second motion data corresponding to a second movement of the user's hand that operates the one or more controllers through the one or more sensors;

A step of receiving second touch data corresponding to a second touch of the user's hand that operates the one or more controllers through the one or more controllers;

A step of receiving second force data corresponding to the second touch of the user's hand that operates the one or more controllers through the one or more controllers;

A step of associating the second motion data with the second touch data and the second force data;

A step of training the model corresponding to the gesture of the hand, based at least in part on the second motion data, the second touch data and the second force data;

A step of receiving third touch data corresponding to a third touch of the user's hand that operates the one or more controllers through the one or more controllers;

A step of receiving third force data corresponding to the third touch of the user's hand that operates the one or more controllers; and

A system comprising one or more non-transitory computer-readable media storing computer-executable instructions that cause the system to perform operations including generating image data corresponding to the representation of the hand based at least in part on the model corresponding to the gesture and the third force data.

2. A system according to claim 1, wherein the one or more non-transitory computer-readable media store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to further perform an operation including transmitting the image data corresponding to the surface of the hand to a virtual reality display.

3. A system according to claim 1, wherein one or more of the sensors include a depth sensor.

4. In paragraph 1,

The step of associating the first motion data with the first touch data and the first force data includes the step of matching a first time stamp of the first motion data with a first time stamp of the first touch data and a first time stamp of the first force data,

The system of claim 1, wherein the step of associating the second motion data with the second touch data and the second force data comprises the step of matching a second time stamp of the second motion data with a second time stamp of the second touch data and a second time stamp of the second force data.

5. In the system,

one or more processors; and

One or more non-transitory computer-readable media, which, when executed by said one or more processors, cause said one or more processors to:

A step of receiving motion data corresponding to hand movements that operate the controller;

A step of receiving touch data corresponding to a touch input of the hand that operates the above controller;

a step of associating the above motion data with the above touch data; and

A system comprising one or more non-transitory computer-readable media storing computer-executable instructions that cause the system to perform operations including generating a training model corresponding to the hand gesture based at least in part on associating the motion data with the touch data.

6. In the fifth paragraph, the motion data includes first motion data, the touch data includes first touch data, the movement of the hand includes the first movement, the touch input of the hand includes the first touch input, and the one or more non-transitory computer-readable media, when executed by the one or more processors, cause the one or more processors to:

A step of receiving second motion data corresponding to a second movement of the user's hand that operates the controller;

A step of receiving second touch data corresponding to a second touch input of the hand that operates the controller; and

A system storing computer-executable instructions that cause the computer to further perform operations including generating an updated training model corresponding to the gesture of the hand, based at least in part on the second motion data and the second touch data.

7. In the sixth paragraph, when executed by one or more processors, the one or more processors,

A step of generating image data corresponding to the surface of the hand using the updated training model; and

A system storing computer executable instructions that cause the computer to further perform operations including the step of transmitting the image data corresponding to the surface of the hand.

8. A system in accordance with claim 7, wherein transmitting the image causes a remote device to display the representation of the hand.

9. In the fifth paragraph, the one or more non-transitory computer-readable media stores computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to further perform an operation including receiving force data corresponding to an amount of force associated with the touch input.

A system wherein generating said training model corresponding to said gesture of said hand is further based at least in part on said force data.

10. A system in accordance with claim 5, wherein the touch data represents one or more locations on the controller receiving the touch input.

11. A system in accordance with claim 5, wherein the step of associating the motion data and the touch data includes the step of associating a time stamp of the motion data with a time stamp of the touch data.

12. In paragraph 5,

comprising a step of receiving said motion data from a camera communicatively coupled to said system;

A system comprising the step of receiving said touch data from a controller communicatively coupled to said system.

13. In the system,

one or more processors; and

One or more non-transitory computer-readable media, which, when executed by said one or more processors, cause said one or more processors to:

From the controller,

Touch data representing touch input received from the above controller; or

A step of receiving at least one of force data representing an amount of force associated with said touch input;

A step of analyzing at least one of the touch data or the force for a trained model associated with a hand gesture;

determining, at least in part based on said analysis, that at least one of said touch data or said force data corresponds to said hand gesture;

A step of generating image data representing the above hand gesture; and

A system comprising one or more non-transitory computer-readable media storing computer-executable instructions that cause the system to perform operations including transmitting the image data for display.

14. A system in accordance with claim 13, wherein the touch data indicates a position on the controller corresponding to the touch input.

15. In the 13th paragraph, the touch data includes first touch data, the force data includes first force data, the trained model includes the first trained model, the touch input includes the first touch input, the image data includes first image data, the hand gesture includes the first hand gesture, and the one or more non-transitory computer-readable media, when executed by the one or more processors, cause the one or more processors to:

From the above controller,

Second touch data representing a second touch input received from the controller; or

A step of receiving at least one of second force data representing an amount of force associated with said second touch input;

A step of analyzing at least one of the second touch data or the second force data for a second trained model associated with a second hand gesture;

A step of determining that at least one of the second touch data or the second force data corresponds to a second hand gesture;

A step of generating second image data representing the second hand gesture; and

A system storing computer-executable instructions that cause the computer to perform operations including the step of transmitting said second image data for display.

16. In the 13th paragraph, the image data includes first image data, the hand gesture includes a first hand gesture, and the one or more non-transitory computer-readable media, when executed by the one or more processors, cause the one or more processors to:

A step of using at least one predictive modeling technique and determining a second hand gesture based at least in part on the first hand gesture;

A step of generating second image data representing the second hand gesture; and

A system storing computer-executable instructions that cause the computer to further perform operations including transmitting said second image data for display.

17. In the 13th paragraph, the trained model,

Previous touch data received from one or more controllers during the first time period;

Previous force data received from said one or more controllers during said first time period; or

A system comprising a previously trained model using at least one of the previous image data received from one or more cameras during said first time period.

18. In the 13th paragraph, the controller comprises a first controller, the touch data comprises first touch data, the force data comprises first force data, the image data comprises first image data, and the one or more non-transitory computer-readable media, when executed by the one or more processes,

From the second controller,

Second touch data representing a touch input received from the second controller; or

A step of receiving at least one of second force data representing an amount of force associated with the touch input received from the second controller;

analyzing at least one of the second touch data or the second force data for one or more trained models associated with one or more hand gestures;

A step of generating second image data representing a hand gesture received from the second controller; and

A system storing computer-executable instructions that cause the computer to further perform operations including transmitting said second image data for display.

19. A system according to claim 13, wherein the image data includes a three-dimensional (3D) representation of the hand gesture.

20. In the 13th paragraph, the one or more non-transitory computer-readable media, when executed by the one or more processors, cause the one or more processors to:

A step of comparing at least one of the touch data or the force data with one or more trained models, wherein the one or more trained models include the trained model, and individual trained models among the trained models are associated with one or more hand gestures, further performing computer-executable instructions that cause the comparing step to be performed,

The system wherein analyzing at least one of the touch data or the force data for the trained model is based at least in part on the comparison.

Claims (15)

In the system,
one or more processors; and
One or more non-transitory computer-readable media, which, when executed by said one or more processors, cause said one or more processors to:
A step of receiving images captured by one or more cameras, said images depicting light reflected or emitted by markers coupled to a hand operating the controller;
A step of analyzing said images to generate motion data representing tracked positions of said markers, said tracked positions corresponding to movements of said hand operating said controller;
A step of receiving touch data representing a distance disposed between a handle of said controller and fingers of said hand, said touch data being detected by sensors spatially distributed along the length of said handle;
A step of receiving force data corresponding to the amount of force associated with the touch input of the hand that operates the controller;
a step of associating the motion data, the touch data, and the force data; and
A system comprising one or more non-transitory computer-readable media storing computer-executable instructions that cause the system to perform operations including: training a model based at least in part on the associated motion data, the touch data, and the force data, such that a trained model is generated, wherein the trained model is configured to enable a gesture of the hand to be determined without subsequent touch data and subsequent motion data associated with the force data, based on subsequent touch data and subsequent force data associated with the subsequent touch data. In the first aspect, the motion data includes first motion data, the touch data includes first touch data, the movement of the hand includes a first movement, the distance is a first distance, and the one or more non-transitory computer-readable media, when executed by the one or more processors, cause the one or more processors to:
receiving second images captured by said one or more cameras, said second images depicting light reflected or emitted by said markers coupled to said hand operating said controller;
A step of analyzing said second images to generate second motion data representing second tracked positions of said markers, said second tracked positions corresponding to second movements of said hand operating said controller;
A step of receiving second touch data representing a second distance disposed between the handle and fingers of the hand, the second touch data being detected by the sensors;
a step of associating the second motion data with the second touch data; and
A system storing computer-executable instructions that cause the computer to further perform operations comprising generating an updated trained model configured to determine the gesture of the hand based at least in part on the second motion data and the second touch data. In the first paragraph, the one or more non-transitory computer-readable media, when executed by the one or more processors, cause the one or more processors to:
A step of generating image data corresponding to the expression of the hand using the trained model; and
A system storing computer executable instructions that cause the computer to further perform operations including transmitting the image data corresponding to the expression of the hand. In the third aspect, the system wherein transmitting the image data causes the remote device to display the representation of the hand. A system in accordance with claim 1, wherein the touch data represents one or more locations on the controller where at least a portion of the hand touched the controller. A system in accordance with claim 1, wherein the step of associating the motion data and the touch data includes the step of associating a time stamp of the motion data with a time stamp of the touch data. In the first paragraph,
A system wherein the step of receiving the touch data comprises the step of receiving the touch data from the controller communicatively coupled to the system. In the system,
one or more processors; and
One or more non-transitory computer-readable media, which, when executed by said one or more processors, cause said one or more processors to:
From a handheld controller,
Touch data indicating proximity of at least a portion of a hand to said handheld controller, said touch data being detected by sensors; and
A step of receiving force data representing the amount of force applied to the handheld controller;
A step of selecting a trained model from among a plurality of trained models trained at least partially based on characteristics of the touch data and the force data, the associated motion data, the touch data, and the force data;
A step of inputting the above touch data and the above force data into the trained model;
Using the trained model, determining that the touch data and the force data correspond to a hand gesture without subsequent motion data associated with the subsequent touch data and the force data, based on subsequent touch data and subsequent force data associated with the subsequent touch data;
A step of generating image data representing the above hand gesture; and
A system comprising one or more non-transitory computer-readable media storing computer-executable instructions that cause the system to perform operations including causing an image of the hand gesture to be presented on a display based at least in part on the image data. In the 8th paragraph, the characteristic is,
a location on said handheld controller where at least a portion of said hand touches said handheld controller; or
The above amount of the above force of the above pressing
A system comprising at least one of: In the eighth paragraph, the touch data includes first touch data, the force data includes first force data, the trained model includes a first trained model, the image data includes first image data, the hand gesture includes a first hand gesture, and the one or more non-transitory computer-readable media, when executed by the one or more processors, cause the one or more processors to:
From the above handheld controller,
second touch data indicating a second proximity of at least a portion of said hand to said handheld controller; or
A step of receiving at least one of second force data representing a second amount of force of a second press on said handheld controller;
selecting a second trained model from among the plurality of trained models based at least in part on at least one characteristic of the second touch data or the second force data;
A step of inputting at least one of the second touch data or the second force data into the second trained model;
A step of using the second trained model to determine that at least one of the second touch data or the second force data corresponds to a second hand gesture;
A step of generating second image data representing the second hand gesture; and
A system storing computer-executable instructions that cause the computer to further perform actions including causing a second image of the second hand gesture to be presented on the display based at least in part on the second image data. In the eighth paragraph, the hand gesture comprises a first hand gesture, and the one or more non-transitory computer-readable media, when executed by the one or more processors, causes the one or more processors to:
A step of predicting a second hand gesture based at least in part on the first hand gesture;
A step of generating second image data representing the second hand gesture;
After the above prediction, from the handheld controller,
second touch data indicating a second proximity of at least a portion of said hand to said handheld controller; or
receiving at least one of second force data representing a second amount of force of a second press on said handheld controller; and
A system storing computer-executable instructions that cause the computer to further perform actions including causing a second image of the second hand gesture to be presented on the display based at least in part on the second image data. In the 8th paragraph, the trained model,
Previous touch data received from one or more controllers during the first time period;
Previous force data received from said one or more controllers during said first time period; or
A system comprising a previously trained model using at least one of the previous image data received from one or more cameras during said first time period. In the 8th paragraph, the step of selecting the trained model from among the plurality of trained models comprises:
comparing said touch data with stored touch data associated with said plurality of trained models; or
Comparing the above force data with stored force data associated with the plurality of trained models.
A system based on at least one of the following. In the 8th paragraph, the step of selecting the trained model from among the plurality of trained models is further based on additional characteristics of the user holding the handheld controller. delete

Images