WO2016179654A1 - Wearable garments, and wearable garment components, configured to enable delivery of interactive skills training content - Google Patents
Wearable garments, and wearable garment components, configured to enable delivery of interactive skills training content Download PDFInfo
- Publication number
- WO2016179654A1 WO2016179654A1 PCT/AU2016/050349 AU2016050349W WO2016179654A1 WO 2016179654 A1 WO2016179654 A1 WO 2016179654A1 AU 2016050349 W AU2016050349 W AU 2016050349W WO 2016179654 A1 WO2016179654 A1 WO 2016179654A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- sensor
- data
- garment according
- wearable garment
- central processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Classifications
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B19/00—Teaching not covered by other main groups of this subclass
- G09B19/003—Repetitive work cycles; Sequence of movements
- G09B19/0038—Sports
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B15/00—Teaching music
Definitions
- the present invention relates to devices, frameworks and methodologies configured to enable delivery of interactive skills training content.
- Embodiments of the invention have been particularly developed to provide wearable garments, and wearable garment components, configured to enable delivery of interactive skills training content. While some embodiments will be described herein with particular reference to that application, it will be appreciated that the invention is not limited to such a field of use, and is applicable in broader contexts.
- a wearable garment including: a plurality of sensor strands, wherein each sensor strand includes one or more sensor units, wherein each sensor unit includes: (i) a microprocessor; (ii) a memory module; and (iii) a set of one or more motion sensor components; a sensor strand connection port, wherein the sensor strand connection port is configured to couple a plurality of sensor strands to a central processing device; and a central processing device, the central processing device including (i) a power supply; (ii) a microprocessor; and (iii) a memory module; wherein the memory module is configured to store software instructions executable by the microprocessor that enable the processing device to identify one or more sets of predefined observable data conditions in sensor data, including sensor data received by the central processing device from the plurality of connected sensor units.
- a wearable garment including: a plurality of sensor strands, wherein each sensor strand includes one or more mounting locations, each mounting location configured to enable connection and mounting of a respective sensor unit, wherein each sensor unit includes: (i) a microprocessor; (ii) a memory module; and (iii) a set of one or more motion sensor components; a sensor strand connection port, wherein the sensor strand connection port is configured to couple a plurality of sensor strands to a central processing device; and a mounting location configured to enable connection and mountain of a central processing device, the central processing device including (i) a power supply; (ii) a microprocessor; and (iii) a memory module; wherein the memory module is configured to store software instructions executable by the microprocessor that enable the processing device to identify one or more sets of predefined observable data conditions in sensor data, including sensor data received by the central processing device from the plurality of sensor units.
- a wearable garment including: a plurality of sensor strands, wherein each sensor strand includes one or more sensor unit mounting locations, wherein each sensor unit mounting location is configured to enable removable mounting of a sensor unit which includes: (i) a microprocessor; (ii) a memory module; and (iii) a set of one or more motion sensor components; a sensor strand connection port, wherein the sensor strand connection port is configured to couple a plurality of sensor strands to a central processing device; and a central processing device, the central processing device including (i) a power supply; (ii) a microprocessor; and (iii) a memory module; wherein the memory module is configured to store software instructions executable by the microprocessor that enable the processing device to identify one or more sets of predefined observable data conditions in sensor data, including sensor data received by the central processing device from one or more connected sensor units; wherein the garment is configured to operate when the sensor unit mounting locations include: one or more sensor unit mounting locations with sensor units mounted; and
- One embodiment provides a computer program product for performing a method as described herein.
- One embodiment provides a non-transitory carrier medium for carrying computer executable code that, when executed on a processor, causes the processor to perform a method as described herein.
- One embodiment provides a system configured for performing a method as described herein.
- any one of the terms comprising, comprised of or which comprises is an open term that means including at least the elements/features that follow, but not excluding others.
- the term comprising, when used in the claims should not be interpreted as being limitative to the means or elements or steps listed thereafter.
- the scope of the expression a device comprising A and B should not be limited to devices consisting only of elements A and B.
- Any one of the terms including or which includes or that includes as used herein is also an open term that also means including at least the elements/features that follow the term, but not excluding others. Thus, including is synonymous with and means comprising.
- exemplary is used in the sense of providing examples, as opposed to indicating quality. That is, an "exemplary embodiment” is an embodiment provided as an example, as opposed to necessarily being an embodiment of exemplary quality.
- FIG. 1 A schematically illustrates a framework configured to enable generation and delivery of content according to one embodiment.
- FIG. 1 B schematically illustrates a framework configured to enable generation and delivery of content according to a further embodiment.
- FIG. 2A illustrates an example framework including server-side and client-side components.
- FIG. 2B illustrates a further example framework including server-side and client-side components.
- FIG. 2C illustrates a further example framework including server-side and client-side components.
- FIG. 2D illustrates a further example framework including server-side and client-side components.
- FIG. 3A illustrates operation of an example framework.
- FIG. 3B illustrates operation of a further example framework.
- FIG. 3C illustrates operation of a further example framework.
- FIG. 4A illustrates performance analysis equipment according to one embodiment.
- FIG. 4B illustrates performance analysis equipment according to one embodiment.
- FIG. 4C illustrates performance analysis equipment according to one embodiment.
- FIG. 4D illustrates performance analysis equipment according to one embodiment.
- FIG. 4E illustrates a MSU-enabled garment arrangement according to one embodiment.
- FIG. 4F illustrates a MSU-enabled garment arrangement according to one embodiment, with example connected equipment.
- FIG. 4G illustrates a MSU-enabled garment arrangement according to one embodiment, with example connected equipment.
- FIG. 4H illustrates MSUs according to one embodiment.
- FIG. 4I illustrates a MSU and housing according to one embodiment.
- Embodiments described herein relate to technological frameworks whereby user skill performances are monitored using Performance Sensor Units (PSUs), and data derived from those PSUs is processed thereby to determine attributes of the user skill performances.
- attributes of performances are used to drive computer programs, such as computer programs configured to provide skills training.
- attributes of performances are determined for alternate purposes, such as providing multi-user competitive activities and the like.
- Embodiments are directed, in particular, to wearable garments, and wearable garment components, configured to enable delivery of interactive skills training content.
- these include garments, such as garments formed of resilient materials (e.g. compression shirts and pants) which provide mounting locations for motion sensor units.
- the garments preferably also carry a processing unit, which is configured to enable downloading of data which configures the sensor units to provide data in a predetermined form, thereby to enable monitoring for particular motion based skills and skill symptoms.
- the frameworks described herein make use of PSUs to collect data representative of performance attributes, and provide feedback and/or instruction to a user thereby to assist in that user improving his/her performance. For instance, this may include providing coaching advice, directing the user to perform particular exercises to develop particular required underlying sub-skills, and the like.
- a training program is able to adapt based on observation of whether a user's performance attributes improve based on feedback/instruction provided. For example, observation of changes in performance attributes between successive performance attempt iterations are indicative of whether the provided feedback/instruction has been successful or unsuccessful. This enables the generation and delivery of a wide range of automated adaptive skills training programs.
- Human motion-based skill performances These are performances where human motion attributes are representative of defining characteristics of a skill.
- motion- based performances include substantially any physical skill which involves movement of the performer's body.
- a significant class of motion-based performances are performances of skills that are used in sporting activities.
- Audio-based skill performances are performances where audibly-perceptible attributes are representative of defining characteristics of a skill.
- audio-based skill performances include musical and/or linguistic performances.
- a significant class of audio-based performances are performances of skills associated with playing musical instruments.
- Some embodiments relate to computer-implemented frameworks that enable the defining, distribution and implementation of content that is experienced by end-users in the context of performance monitoring. This includes content that is configured to provide interactive skills training to a user, whereby a user's skill performance is analysed by processing of Performance Sensor Data (PSD) derived from one or more PSUs that are configured to monitor a skill performance by the user.
- PSD Performance Sensor Data
- inventive subject matter is embodied across aspects of the technologies and methodologies described herein, including but not limited to: (i) analysis of a skill thereby to understand its defining characteristics; (ii) defining of protocols thereby to enable automated analysis of a skill using one or more PSUs; (iii) defining and delivery of content that makes use of the automated analysis thereby to provide interactive end-use content, such as skills training; (iv) adaptive implementation of skills training programs; (v) hardware and software that facilitates the delivery of content to end users; (vi) hardware and software that facilitates the experiencing of content by end users; and (vii) technology and methodologies developed to facilitate the configuration and implementation of multiple motion sensor units for the purpose of human activity monitoring.
- Performance Sensor Unit is a hardware device that is configured to generate data in response to monitoring of a physical performance. Examples of sensor units configured to process motion data and audio data are primarily considered herein, although it should be appreciated that those are by no means limiting examples.
- Performance Sensor Data Data delivered by a PSU is referred to as Performance Sensor Data. This data may comprise full raw data from a PSU, or a subset of that data (for example based on compression, reduced monitoring, sampling rates, and so on).
- An audio sensor unit is a category of PSU, being a hardware device that is configured to generate and transmit data in response to monitoring of sound.
- an ASU is configured to monitor sound and/or vibration effects, and translate those into a digital signal (for example a MIDI signal).
- a digital signal for example a MIDI signal.
- an ASU is a pickup device including a transducer configured to capture mechanical vibrations in a stringed instrument and concert those into electrical signals.
- Audio Sensor Data This is data delivered by one or more ASUs.
- a motion sensor unit is a category of PSU, being a hardware device that is configured to generate and transmit data in response to motion. This data is in most cases defined relative to a local frame of reference.
- a given MSU may include one or more accelerometers; data derived from one or more magnetometers; and data derived from one or more gyroscopes.
- a preferred embodiment makes use of one or more 3-axis accelerometers, one 3-axis magnetometer, and one 3-axis gyroscope.
- a motion sensor unit may be "worn” or “wearable”, which means that it is configured to be mounted to a human body in a fixed position (for example via a garment).
- Motion Sensor Data Data delivered by a MSU is referred to as Motion Sensor Data (MSD).
- This data may comprise full raw data from a MSU, or a subset of that data (for example based on compression, reduced monitoring, sampling rates, and so on).
- a MSU-enabled garment is a garment (such as a shirt or pants) that is configured to carry a plurality of MSUs.
- the MSUs are mountable in defined mountain zones formed in the garment (preferably in a removable manner, such that individual MSUs are able to be removed and replaced), and coupled to communication lines.
- a POD device is a processing device that receives PSD (for example MSD from MSUs). In some embodiments it is carried by a MSU-enabled garment, and in other embodiments it is a separate device (for example in one embodiment the POD device is a processing device that couples to a smartphone, and in some embodiments POD device functionality is provided by a smartphone or mobile device).
- the MSD is received in some cases via wired connections, in some cases via wireless connections, and in some cases via a combination of wireless and wired connections.
- a POD device is responsible for processing the MSD thereby to identify data conditions in the MSD (for example to enable identification of the presence of one or more symptoms).
- the role of a POD device is performed in whole or in part by a multi-purpose end-user hardware device, such as a smartphone.
- at least a portion of PSD processing is performed by a cloud-based service.
- Motion capture data is data derived from using any available motion capture technique.
- motion capture refers to a technique whereby capture devices are used to capture data representative of motion, for example using visual markers mounted to a subject at known locations.
- An example is motion capture technology provided by Vicon (although no affiliation between the inventors/applicant and Vicon is to be inferred).
- MCD is preferably used to provide a link between visual observation and MSD observation.
- a skill is an individual motion (or set of linked motions) that is to be observed (visually and/or via MSD), for example in the context of coaching.
- a skill may be, for example, a rowing motion, a particular category of soccer kick, a particular category of golf swing, a particular acrobatic manoeuvre, and so on.
- sub-skills This is primarily to differentiate between a skill being trained, and lesser skills that form part of that skill, or are building blocks for that skill. For example, in the context of a skill in the form of juggling, a sub-skill is a skill that involves throwing a ball and catching it in the same hand.
- a symptom is an attribute of a skill that is able to be observed (for example observed visually in the context of initial skill analysis, and observed via processing of MSD in the context of an end-user environment).
- a symptom is an observable motion attribute of a skill, which is associated with a meaning.
- identification of a symptom may trigger action in delivery of an automated coaching process.
- a symptom may be observable visually (relevant in the context of traditional coaching) or via PSD (relevant in the context of delivery of automated adaptive skills training as discussed herein).
- Symptoms are, at least in some cases, associated with one causes (for example a given symptom may be associated with one or more causes).
- a cause is also in some cases able to be observed in MSD, however that is not necessarily essential.
- one approach is to first identify a symptom, and then determine/predict a cause for that symptom (for example determination may be via analysis of MSD, and prediction may be by means other than analysis of MSD). Then, the determined/predicted cause may be addressed by coaching feedback, followed by subsequent performance assessment thereby to determine whether the coaching feedback was successful in addressing the symptom.
- Observable Data Condition is used to describe conditions that are able to be observed in PSD, such as MSD (typically based on monitoring for the presence of an ODC, or set of anticipated ODCs) thereby to trigger downstream functionalities.
- MSD typically based on monitoring for the presence of an ODC, or set of anticipated ODCs
- an ODC may be defined for a given symptom (or cause); if that ODC is identified in MSD for a given performance, then a determination is made that the relevant symptom (or cause) is present in that performance. This then triggers events in a training program.
- Training Program is used to describe an interactive process delivered via the execution of software instructions, which provides an end user with instructions of how to perform, and feedback in relation to how to modify, improve, or otherwise adjust their performance.
- the training program is an "adaptive training program", being a training program that executes on the basis of rules/logic that enable the ordering of processes, selection of feedback, and/or other attributes of training to adapt based on analysis of the relevant end user (for example analysis of their performance and/or analysis of personal attributes such as mental and/or physical attributes).
- some embodiments employ a technique whereby a POD device is configured to analyse a user's PSD (such as MSD) in respect of a given performance thereby to determine presence of one or more symptoms, being symptoms belonging to a set defined based on attributes of the user (for example the user's ability level, and symptoms that the user is known to display from analysis of previous iterations).
- PSD such as MSD
- a process is performed thereby to determine/predict a cause.
- feedback is selected thereby to seek to address that cause.
- complex selection processes are defined thereby to select specific feedback for the user, for example based on (i) user history, for example prioritising untried or previously successful feedback over previously unsuccessful feedback; (ii) user learning style; (iii) user attributes, for example mental and/or physical state at a given point in time, and/or (iv) a coaching style, which is in some cases based on the style of a particular real-world coach.
- FIG. 1 A provides a high-level overview of an end-to-end framework which is leveraged by a range of embodiments described herein.
- an example skill analysis environment 101 is utilised thereby to analyse one or more skills, and provide data that enables the generation of end user content in relation to those skills. For instance, this in some embodiments includes analysing a skill thereby to determine ODCs that are able to be identified by PSUs (preferably ODCs that are associated with particular symptoms, causes, and the like). These ODCs are able to be utilised within content generation logic implemented by an example content generation platform 102 (such as a training program).
- generating content preferably includes defining a protocol whereby prescribed actions are taken in response to identification of specific ODCs.
- a plurality of skill analysis environments and content generation platforms are preferably utilised thereby to provide content to an example content management and delivery platform 103.
- This platform is in some embodiments defined by a plurality of networked server devices.
- the purpose of platform 103 is to make available content generated by content generation platforms to end users.
- the downloading in some embodiments includes an initial download of content, and subsequently further downloads of additional required content. The nature of the further downloads is in some cases affected by user interactions (for instance based on an adaptive progression between components of a skills training program and/or user selections).
- Example equipment 104 is illustrated in the form of a MSU-enabled garment that carries a plurality of MSUs and a POD device, in conjunction with user interface devices (such as a smartphone, a headset, HUD eyewear, retinal projection devices, and so on).
- user interface devices such as a smartphone, a headset, HUD eyewear, retinal projection devices, and so on.
- a user downloads content from platform 103, and causes that content to be executed via equipment 104.
- this may include content that provides an adaptive skills training program for a particular physical activity, such as golf or tennis.
- equipment 104 is configured to interact with an example content interaction platform 105, being an external (e.g. web-based) platform that provides additional functionality relevant to the delivery of the downloaded content.
- content interaction platform 105 being an external (e.g. web-based) platform that provides additional functionality relevant to the delivery of the downloaded content.
- various aspects of an adaptive training program and/or its user interface may be controlled by server-side processing.
- platform 105 is omitted, enabling equipment 104 to deliver previously downloaded content in an offline mode.
- a guitar training program A user downloads a guitar training program that is configured to provide training in respect of a given piece of music.
- a PSU in the form of a pickup is used, thereby to enable analysis of PSD representative of the user's playing of a guitar.
- the training program is driven based on analysis of that PSD, thereby to provide the user with coaching.
- the coaching may include tips for finger positioning, remedial exercises to practice progression between certain finger positions, and/or suggestion of other content (e.g. alternate pieces of music) that may be of interest and/or assistance to the user.
- a golf training program A user downloads a golf training program, which is configured to operate with a MSU-enabled garment. This includes downloading of sensor configuration data and state engine data to a POD device provided by the MSU-enabled garment. The user is instructed to perform a performance defined certain form of swing (for example with a certain intensity, club, or the like) and plurality of MSUs carried by the MSU-enabled garment provide MSD representative of the performance. The MSD is processed thereby to identify symptoms and/or causes, and training feedback is provided. This is repeated for one or more further performance iterations, based on training program logic designed to assist the user in improving his/her form. Instructions and/or feedback are provided by way of a retinal display projector which delivers user interface data directly into the user's field of vision.
- FIG. 1 B provides a more detailed overview of a further example end-to-end technological framework that is present in the context of some embodiments.
- This example is particularly relevant to motion-based skills training, and is illustrated by reference to a skill analysis phase 100, a curriculum construction phase 1 10, and an end user delivery phase 120. It will be appreciated that this is not intended to be a limiting example, and is provide to demonstrate a particular end-to-end approach for defining and delivering content.
- FIG. 1 A illustrates a selection of hardware used at that stage in some embodiments, being embodiments where MCD is used to assist in analysis of skills, and subsequently to assist and/or validate determination of ODCs for MSD.
- the illustrated hardware is a wearable sensor garment 106 which carries a plurality of motion sensor units and a plurality of motion capture (mocap) markers (these are optionally located at similar positions on the garment), and a set of capture devices 106a-106c.
- a set of example processes are also illustrated.
- Block 107 represents a process including capturing of video data, motion capture data (MCD), and motion sensor data (MSD) for a plurality of sample performances. This data is used by processes represented in block 108, which include breaking down a skill into symptoms and causes based on expert analysis (for example including: analysis of a given skill, thereby to determine aspects of motion that make up that skill and affect performance, preferably at multiple ability levels; and determination of symptoms and causes for a given skill, including ability level specific determination of symptoms and causes for a given skill).
- Block 109 represents a process including defining of ODCs to enable detection of symptoms/causes from motion sensor data. These ODCs are then available for use in subsequent phases (for example they are used in a given curriculum, applied in state engine data, and the like).
- phase 100 is described here by reference to an approach that makes use of MCD, that is not intended to be a limiting example.
- approaches that make use of MSD from the outset e.g. there is no need to make use of MCD to assist and/or validate determination of ODCs for MSD
- approaches that make use of machine learning of skills e.g. there is no need to make use of MCD to assist and/or validate determination of ODCs for MSD
- approaches that make use of machine learning of skills and so on.
- Phase 1 10 is illustrated by reference to a repository of expert knowledge data 1 1 1 . For example, one or more databases are maintained, these containing information defined subject to aspects of phase 101 and/or other research and analysis techniques.
- Examples of information include: (i) consensus data representative of symptoms/causes; (ii) expert-specific data representative of symptoms/causes; (iii) consensus data representative of feedback relating to symptoms/causes; (iv) expert-specific data representative of feedback relating to symptoms/causes; and (v) coaching style data (which may include objective coaching style data, and personalised coaching style data). This is a selection only.
- Block 1 12 represents a process including configuring an adaptive training framework.
- a plurality of skills training programs relating to respective skills and aspects thereof, are delivered via a common adaptive training framework.
- This is preferably a technological framework that is configured enable the generation of skill-specific adaptive training content that leverages underlying skill-nonspecific logic.
- such logic relates to methodologies for: predicting learning styles; tailoring content delivery based on available time; automatically generating a lesson plan based on previous interactions (including refresher teaching of previously learned skills); functionally to recommend additional content to download; and other functionalities.
- Block 1 13 represents a process including defining of a curriculum for a skill. This may include defining a framework of rules for delivering feedback in response to identification of particular symptoms/causes.
- the framework is preferably an adaptive framework, which provides intelligent feedback based on acquired knowledge specific to an individual user (for example knowledge of the user's learning style, knowledge of feedback that has been successful/unsuccessful in the past, and the like).
- Block 1 14 represents a process including making a curriculum available for download by end users, for example making it available via an online store.
- a given skill may have a basic curriculum offering, and/or one or more premium curriculum offerings (preferably at different price points).
- a basic offering is in some embodiments based upon consensus expert knowledge, and a premium offering based on expert-specific expert knowledge.
- example end-user equipment is illustrated.
- This includes a MSU- enabled garment arrangement 121 , comprising a shirt and pants carrying a plurality of MSUs, with a POD device provided on the shirt.
- the MSUs and POD device are configured to be removable from the garments, for example to enable cleaning and the like.
- a headset 122 is connected by Bluetooth (or other means) to the POD device, and configured to deliver feedback and instructions audibly to the user.
- a handheld device 123 (such as an iOS or Android smartphone) is configured to provide further user interface content, for example instructional videos/animations and the like.
- Other user interface devices may be used, for example devices configured to provide augmented reality information (such as displays viewable via wearable eyewear and the like).
- a user of the illustrated end-user equipment downloads content for execution (for example from platform 103), thereby to engage in training programs and/or experience other forms of content that leverage processing of MSD. For example, this may include browsing an online store or interacting with a software application thereby to identify desired content, and subsequently downloading that content.
- content is downloaded to the POD device, the content including state engine data and curriculum data.
- the former includes data that enables the POD device to process MSD, thereby to identify symptoms (and/or perform other forms of motion analysis).
- the latter includes data required to enable provision of a training program, including content that is delivered by the user interface (for example instructions, feedback, and the like) and instructions for the delivery of that content (such as rules for the delivery of an adaptive learning process).
- engine data and/or curriculum data is obtained from a remote server on an ongoing basis.
- Functional block 125 represents a process whereby the POD device performs a monitoring function, whereby a user performance is monitored for ODCs as defined in state engine data. For example, a user is instructed via device 123 and/or headset 122 to "perform activity X", and the POD device then processes the MSD from the user's MSUs thereby to identify ODCs associated with activity X (for example to enable identification of symptoms an/or causes). Based on the identification of ODCs and the curriculum data (and in some cases based on additional inputs), feedback is provided to the user via device 123 and/or headset 122 (block 126). For example, whilst repeatedly performing "activity X", the user is provided audible feedback with guidance on how to modify their technique.
- the curriculum data in some embodiments is configured to adapt the feedback and/or stages of a training program based on a combination of (i) success/failure of feedback to achieve desired results in terms of activity improvement; and (ii) attributes of the user, such as mental and/or physical performance attributes.
- a skill analysis phase is implemented thereby to analyse a skill that is to be observed in the end-user delivery phase. More specifically, the skill analysis phase preferably includes analysis to: (i) determine attributes of a skill, for example attributes that are representative of the skill being performed (which is particularly relevant where the end user functionality includes skill identification), and attributes that are representative of the manner in which a skill is performed, such as symptoms and causes (which are particularly relevant where end user functionality includes skill performance analysis, for instance in the context of delivery of skills training); and (ii) define ODCs that enable automated identification of skill attributes (such as the skill being performed, and attributes of the performance of that skill such as symptoms and/or causes) such that end user hardware (PSUs, such as MSUs) is able to be configured for automated skill performance analysis.
- PSUs end user hardware
- the nature of the skill analysis phase varies significantly depending on the nature of a given skill (for example between the categories of motion-based skills and audio-based skills).
- exemplary embodiments are now described in relation to a skill analysis phase in the context of a motion-based skill. That is, embodiments are described by reference to analysing a physical activity, thereby to determine ODCs that are used to configure a POD device that monitors data from body-mounted MSUs.
- This example is selected to be representative of a skill analysis phased in a relatively challenging and complex context, where various novel and inventive technological approaches have been developed to facilitate the task of generating effective ODCs for motion-based skills.
- MCD is used primarily due to the established nature of motion capture technology (for example using powerful high speed cameras); motion sensor technology on the other hand is currently continually advancing in efficacy.
- MCD analysis technology assists in understanding and/or validating MSD and observations made in respect of MSD.
- MSD is utilised in a similar manner to MCD, in the sense of capturing data thereby to generate three dimensional body models similar to those conventionally generated from MCD (for example based on a body avatar with skeletal joints) It will be appreciated that this assumes a threshold degree of accuracy and reliability in MCD. However, in some embodiments this is able to be achieved, hence rendering MCD assistance unnecessary.
- Machine learning methods for example where MSD and/or MCD is collected for a plurality of sample performances, along with objectively defined performance outcome data (for example, in the case or rowing: power output; and in the case of golf: ball direction and trajectory).
- Machine learning method are implemented thereby to enable automated defining of relationships between ODCs and effects on skill performance.
- Such an approach when implemented with a sufficient sample size, enables computer identification of ODCs to drive prediction of skill performance outcome.
- ODCs that affect swing performance are automatically identified using analysis of objectively defined outcomes, thereby to enable reliable automated prediction of an outcome in relation to an end-user swing using end- user hardware (for example a MSU-enabled garment).
- end user devices are equipped with a "record" function, which enables recording of MSD representative of a particular skill as respectively performed by the end users (optionally along with information regarding symptoms and the like identified by the users themselves).
- the recorded data is transmitted to a central processing location to compare the MSD for a given skill (or a particular skill having a particular symptom) for a plurality of users, and hence identify ODCs for the skill (and/or symptom). For example, this is achieved by identifying commonalities in the data.
- curriculum construction includes defining logical processes whereby ODCs are used as input to influence the delivery of training content.
- training program logic is configured to perform functions including but not limited to:
- this may include coaching feedback relevant to a symptom and/or cause of which the ODCs are representative.
- this may include: (i) determining that a given skill (or sub-skill) has been sufficiently mastered, and progressing to a new skill (or sub-skill); or (ii) determining that a user has a particular difficulty, and providing the user with training in respect of a different skill (or sub-skill) that is intended to provide remedial training to address the particular difficulty.
- ODCs i.e. data attributes that are able to be identified in MSD, or PSD more generally
- this enables a wide range of training to be provided, ranging from the likes of assisting a user to improve a gold swing motion, to the likes of assisting a user in mastering a progression of notes when playing a piece of music on a guitar.
- ODCs are used for purposes including skill identification and skill attribute measurement.
- feedback provided by the user interface includes suggestions on how to modify movement so as to improve performance, or more particularly (in the context of motion sensors) suggestions to more closely so as to replicate motion attributes that are predefined as representing optimal performance.
- a user downloads a training package to learn a particular skill, such as a sporting skill (in some embodiments a training package includes content for a plurality of skills).
- training packages may relate a wide range of skills, including the likes of soccer (e.g. specific styles of kick), cricket (e.g. specific bowling techniques), skiing/snowboarding (e.g. specific aerial manoeuvres), and so on.
- a common operational process performed by embodiments of the technology disclosed herein is (i) the user interface provides an instruction to perform an action defining or associated with a skill being trained; (ii) the POD device monitor input data from sensors determine symptom model values associated with the user's performance of the action; (iii) the user's performance is analysed; and (iv) a user interface action is performed (for example providing feedback and/or an instruction to try again concentrating on particular aspects of motion). Additional examples of operation methods and content generation methods are disclosed in PCT/AU2016/000020.
- content is made available for download to end user devices. This is preferably made available via one or more online content marketplaces, which enable users of web-enabled devices to browse available content, and cause downloading of content to their respective devices.
- downloadable content includes the following three data types:
- sensor configuration data Data representative of sensor configuration instructions, also referred to as "sensor configuration data”. This is data configured to cause configuration of a set of one or more PSUs to provide sensor data having specified attributes.
- sensor configuration data includes instruction that cause a given PSU to: adopt an active/inactive state (and/or progress between those states in response to defined prompts); deliver sensor data from one or more of its constituent sensor components based on a defined protocol (for example a sampling rate and/or resolution).
- a given training program may include multiple sets of sensor configuration data, which are applied for respective exercises (or in response to in-program events which prompt particular forms of ODC monitoring).
- multiple sets of sensor configuration data are defined to be respectively optimised for identifying particular ODCs in different arrangements of end-user hardware.
- sensor configuration data is defined thereby to optimise the data delivered by PSUs to increase efficiency in data processing when monitoring for ODCs. That is, where a particular element of content monitors for n particular ODCs, the sensor configuration data is defined to remove aspects of sensor data that is superfluous to identification of those ODCs.
- State engine data which configures a performance analysis device for example a POD device) to process input data received from one or more of the set of connected sensors thereby to analyse a physical performance that is sensed by the one or more of the set of connected sensors.
- this includes monitoring for a set of one or more ODCs that are relevant to the content being delivered. For example, content is driven by logic that is based upon observation of particular ODCs in data delivered by PSUs.
- User interface data which configures the performance analysis device to provide feedback and instructions to a user in response to the analysis of the physical performance (for example delivering of a curriculum including training program data).
- the user interface data is at least in part downloaded periodically from a web server.
- the content data includes computer readable code that enables the POD device (or another device) to configure a set of PSUs to provide data in a defined manner which is optimised for that specific skill (or set of skills). This is relevant in the context of reducing the amount of processing that is performed at the POD device; the amount of data provided by sensors is reduced based on what is actually required to identify symptoms of a specific skill or skills that are being trained. For example, this may include:
- the POD device provides configuration instructions to the sensors based on a skill that is to be trained, and subsequently receives data from the sensor or sensors based on the applied configurations, so as to allow delivery of a PSU-driven training program.
- the sensor configuration data in some cases includes various portions that loaded onto the POD device at different times.
- the POD device may include a first set of such code (for example in its firmware) which is generic across all sensor configurations, which is supplemented by one or more additional sets of code (which may be downloaded concurrently or at different times) which in a graduated manner increase the specificity by which sensor configuration is implemented.
- one approach is to have base-level instructions, instructions specific to a particular set of MSUs, and instructions specific to configuration of those MSUs for a specific skill that is being trained.
- Sensors are preferably configured based on specific monitoring requirements for a skill in respect of which training content is delivered. This is in some cases specific to a specific motion- based skill that is being trained, or even to a specific attribute of a motion-based skill that is being trained.
- state engine data configures the POD device in respect of how to process data obtained from connected sensors (i.e. PSD) based on a given skill that is being trained.
- each skill is associated with a set of ODCs (which are optionally each representative of symptoms), and the state engine data configures the POD device to process sensor data thereby to make objective determinations of a user's performance based on observation of particular ODCs. In some embodiments this includes identifying the presence of a particular ODC, and then determining that an associated symptom is present. In some cases this subsequently triggers secondary analysis to identify an ODC that is representative of one of a set of causes associated with that symptom.
- the analysis includes determinations based on variations between (i) symptom model data determined from sensor data based on the user's performance; and (ii) predefined baseline symptom model data values. This is used, for example, to enable comparison of the user's performance in respect of each symptom with predefined characteristics.
- User interface data in some embodiments includes data that is rendered thereby to provide graphical content that is rendered via a user interface.
- data is maintained on the POD device (for example video data is streamed from the POD device to a user interface device, such as a smartphone or other display).
- data defining graphical content for rendering via the user interface is stored elsewhere, including (i) on a smartphone; or (ii) at a cloud-hosted location.
- User interface data additionally includes data configured to cause execution of an adaptive training program. This includes logic/rules that are responsive to input including PSD (for example ODCs derived from MSD) and other factors (for example user attributes such as ability levels, learning style, and mental/physical state).
- PSD for example ODCs derived from MSD
- other factors for example user attributes such as ability levels, learning style, and mental/physical state.
- download of such data enables operation in an offline mode, whereby no active Internet connection is required in order for a user to participate in a training program.
- skills training content is structured (at least in respect of some skills) to enable user selection of both (i) a desired skill; and (ii) a desired set of "expert knowledge” in relation to that skill.
- "expert knowledge” allows a user to engage in training to learn a particular skill based on a specific expert's interpretation of that skill.
- an individual skill may have multiple different expert knowledge variations.
- a soccer chip kick might have a first expert knowledge variation based on Player X's interpretation of an optimal form of chip kick, and a second expert knowledge variation based on Player Y's interpretation of an optimal form of chip kick.
- content is made available to users via an online marketplace (for example an online marketplace delivered by a cloud hosted platform.
- an online marketplace for example an online marketplace delivered by a cloud hosted platform.
- a user accesses that marketplace (for example via a web browser application executing on a personal computer or mobile device), and obtains desired training content.
- the user configures a POD device to perform functionalities including functionalities relating to provision of training in respect of a desired activity and/or skill (for example by causing a server to download code directly to the POD device via the POD device's Internet connection, which may be to a local WiFi network).
- a set of training program rules are able to be executed on the POD device (or in further embodiments a secondary device coupled to the POD device) to provide an interactive training process.
- the interactive training process provides, to a user, feedback/instructions responsive to input representative of user performance. This input is derived from the PSUs, and processed by the POD device.
- the interactive training process is in some embodiments operated based on a set of complex rules, which take into consideration: (i) observed user performance attributes relative to predefined performance attributes; (ii) user attribute data, including historical performance data; (iii) a skill training progression pathway (which may be dynamic variable); and (iv) other factors.
- the present disclosure focuses primarily on the example of a POD device that receives user performance data derived from a set of motion sensors (for example including wearable motion sensors coupled to garments; the motion sensors being configured to enable analysis of user body position variations in three dimensions). For example, this is particularly applicable to training in respect of physical activities, such as sports and other activities involving human movements. However, the technology is equally applicable in respect of data derived from other forms of sensor. Examples include sensors that monitor audio, video, position, humidity, temperature, pressure, and others. It will be appreciated that data from such sensors may be useful for skills training across a wide range of activity types. For example, audio sensors are particularly useful for training activities such as language skills, singing, and the playing of musical instruments.
- Skills training content is rendered via a user interface (for example in a graphical and/or audible form).
- a user interface for example in a graphical and/or audible form.
- a preferred approach is for training content to be downloaded directly to a POD device , and rendered via a separate device that includes video and/or audio outputs which allow a user to experience rendered content.
- the separate device may include one or more of a mobile device such as a smartphone (which in some embodiments executes an application configured to render content provided by the POD device), a headset, a set of glasses having an integrated display, a retinal display device, and other such user interface devices.
- the POD device provides a local web server configured to deliver content to the mobile device.
- the mobile device executes a web browser application (or in some cases a proprietary app), which navigates to a web address in respect of which code is obtained from the POD device as a local web server.
- Skills training content is in preferred embodiments obtained from an online marketplace.
- This marketplace preferably enables a user to select and procure various different skills training packages, and manage the downloading of those to the user's POD device (or POD devices).
- the term "skills training package” describes an obtainable set of skills training content. This may relate to a single skill, a variety of skills relating to a common activity, or various other arrangements.
- the present disclosure should not be limited by reference to any specific implementation option for structuring how skills training data is organised, made available for procurement, monetised, or the like.
- Browsing and selection of downloadable content via a first web-enabled device with content download subsequently being effected to a second web-enabled device.
- content is browsed via a smartphone, and then caused to be downloaded directly from a web source to a POD device.
- a POD device that is separate from a user interface device.
- a mobile device is used to provide a user interface
- a POD device is a processing unit mounted in a MSU-enabled garment.
- a POD device that is integrated with a user interface device.
- a smartphone takes the role of a POD device.
- a POD device is defined as a processing unit which couples to a smartphone, for example via a cradle type mount.
- FIG. 2A shows an exemplary computer implemented framework according to one embodiment.
- Various alternate embodiments are illustrated in FIG. 2B to FIG. 2D, where similar features have been designated corresponding reference numerals.
- Each illustrated framework includes multiple computing devices (also referred to as “machines” or “terminals”), which are each configured to provide functionality (for example performance of "computer implemented methods") by executing computer-executable code (which may be stored on a computer-readable carrier medium) via one or more microprocessors (also referred to simply as “processors"). It will be appreciated that the various computing devices include a range of other hardware components, which are not specifically illustrated.
- FIG. 2A illustrates a central administration and content management platform 200. This platform is able to be defined by a single computing device (for example a server device), or more preferably by a plurality of networked computing devices.
- Platform 200 is configured to provide functionalities that are accessed by a plurality of users (such as the subjects referred to above) via computing devices operated by those users.
- FIG. 2A illustrates a set of user-side equipment 220 operated in relation to an exemplary user. In practice, each of a plurality of users operates respective sets of similar equipment 220 (not shown).
- Equipment 220 includes a mobile device 230.
- mobile device 230 takes the form of a Smartphone.
- different mobile devices are used such as a tablet, a PDA, a portable gaming device, or the like.
- mobile device 230 is defined by purpose-configured hardware, specifically intended to provide functionalities relevant to the described overall framework.
- a primary function of mobile device 230 is to deliver, via a user interface, content that is obtained from platform 200. This content is able to be downloaded on an "as required" basis (in an online mode), downloaded in advance (thereby to enable operation in an offline mode), or both.
- Mobile device 230 is able to be coupled to one or more pieces of external user interaction hardware, such as external headphones, microphones, a wearable device that provides a graphical display (for example glasses configured to provide augmented reality displays, retina projection displays), and so on.
- external user interaction hardware such as external headphones, microphones, a wearable device that provides a graphical display (for example glasses configured to provide augmented reality displays, retina projection displays), and so on.
- mobile device 230 is configured to interact with platform 200 via a mobile app (for example an iOS or Android app), which is downloaded from an app download server 271 .
- server 271 is a third party operated server, although other embodiments make use of first party servers).
- Such a mobile app is stored on a memory device 234 and executed via a processor 233.
- the mobile app configures mobile device 230 to communicate with an app interaction server 272 via an available Internet connection, with app interaction server 272 in turn providing a gateway to data available via platform 200.
- mobile device 230 is configured to interact with platform 200 via a web browser application, which upon navigation to a predefined web address configures mobile device 230 to communicate with a mobile device web server 274 via an available Internet connection.
- Web server 274 provides a gateway to data available via platform 200.
- the web browser application is executed based on code stored in memory 234 of mobile device 230, and provides a user interface specific to platform 200 via browser-renderable user interface code that is downloaded to device 230 via server 274.
- Equipment 220 additionally includes a personal computer (PC) 240.
- PC personal computer
- This is able to be substantially any computing device that is correctly and adequately configured to enable a further hardware device, in the form of a POD device 250, to communicate with platform 200.
- the POD device connects to PC 240 via a wired connection (such as a USB connection) or a wireless connection (such as a WiFi or a Bluetooth connection). Functionally, this allows downloading of data from platform 200 to POD device 250.
- a wired connection such as a USB connection
- a wireless connection such as a WiFi or a Bluetooth connection
- POD device 250 accessing platform 200 via mobile service 230, and a web server 273 (see FIG. 2C). This involves accessing specific functionalities of device 230 relevant to operation of POD device 250 or, in some embodiments, merely accessing an Internet connection provided through mobile device 230.
- POD device 250 accessing platform 200 via web server 273 (see FIG. 2D).
- a given user operates mobile device 230 (or another suitable configured computing device) to access a user interface (for example via a mobile app or a web page), thereby to instruct platform 200 to deliver particular data to a POD device 250 associated with that user.
- the data is directly downloaded to POD device 250 via an available Internet connection.
- skills training content to be rendered on mobile device 230 is first downloaded to POD device 250. This is implemented such that mobile device 230 is able to provide skills training data in an offline mode (with no Internet connection), with necessary content being provided by POD device 250. This is particularly relevant in examples where there is no mobile device 230, and the user interface is provided via a user interface delivery device 220 which communicates only with POD device 250 (for example a headset, set of glasses having an inbuilt display, retinal projection device, or the like).
- FIG. 17 schematically illustrates a further framework, with example process flows relevant to that framework.
- POD device 250 is configured to perform processing of data collected from one or more PSUs 260. These PSUs are connected to POD 250 via wired and/or wireless connections. For example, in one embodiment a POD device is connected to a first set of PSUs via a direct wired coupling, and to a second set of PSUs via a RF-link to a bridging component, the bridging component in turn being connected to the second set of PSUs via a direct wired coupling.
- a range of PSUs are used across various embodiments depending upon the nature of the data being collected. In turn, the nature of the data being collected is dependent upon the skill or activity being undertaken by the user. For instance, the following user cases are relevant to a number of examples and embodiments considered herein:
- MSUs are integrated into clothing articles (MSU-enabled garments) that are configured to be worn by a subject.
- clothing articles include compression- type clothing (such as a shirt or pants) which each includes a plurality of spaced apart MSUs at known positions.
- the clothing includes preformed mounting locations for releasably receiving respective MSUs to enable movement of MSUs between the available mounting locations.
- a compression shirt supports a plurality of motion MSUs and has a mounting to complementarily releasably receive a POD device, such that the mounting couples the POD device to the MSUs via wired connections that extend through and which are enveloped by the shirt.
- the shirt is able to be coupled with a complementary set of compression pants that include further plurality of motion MSUs which are wired to a common RF communication module. That RF communication module communicates MSD to a further RF module provided on the shirt, or by the POD device, thereby to enable the POD device to receive data from all MSUs on the shirt and pants.
- ASUs • ASUs.
- different audio sensors are used. Examples of available sensors include microphone-based sensors, sensors that plug into audio input ports (for example via 2.5mm or 3.5mm jack connectors), thereby to receive audio signals, pickups that generate MIDI signals, and so on.
- POD device 250 is able to be configured via software to process data from substantially any form of PSU that provides an output signal (for example a digital output signal) that is received by the POD device.
- an output signal for example a digital output signal
- exemplary POD devices may include:
- a POD device configured to be carried by a garment, which physically couples to a plurality of MSUs also carried by that garment (and in some cases wirelessly couples, directly or indirectly, to one or more further MSUs).
- a POD device that includes a microphone.
- a POD device that includes an audio input port (such as a 3.5mm headphone jack).
- PSU personal area network
- a POD device coupled to one or more ASUs is in some cases used to provide training in various musical skills (for example singing, playing of instruments, and the like).
- the manner by which the user interface provides feedback and/or instructions varies based on hardware configurations.
- the user interface is audio-only (for example using headphones), in which case instructions and feedback are audio-based.
- the user interface includes visual information, which requires a display screen (for example a display screen provided by a smartphone device, appropriate glasses and/or retinal display devices, and so on).
- the arrangement of user-side equipment in FIG. 2A is able to be configured to function as shown in FIG. 3A.
- a marketplace platform is technically configured for delivering POD/engine data to a POD device to, in turn, allow configuration of the POD device to deliver training content in respect of a specific skill (or set of skills).
- the POD device is configured to process received data from the sensors based on POD/engine data that was previously downloaded from the marketplace. Based on this processing, the POD device provides instructions to a mobile device to display platform content via its user interface (for example thereby to provide feedback, instruct the user to perform a specific task, and so on).
- the mobile device downloads platform content, where relevant, from the platform.
- a further feedback device is used in other embodiments (for example an audio device, glasses with digital displays, and so on), and in FIG. 3A this is illustrated as being directly coupled to the POD device.
- FIG. 3B illustrates an alternate arrangement whereby the mobile device operates in an offline mode.
- user interface data is downloaded to the POD device, and provided to the mobile device via the POD device.
- FIG. 3C A further alternate arrangement is illustrated in FIG. 3C, where there is no mobile device, and the POD device provides feedback/instructions directly via a feedback device (such as headphones, glasses with a screen, a retina projection device, or another feedback device).
- a feedback device such as headphones, glasses with a screen, a retina projection device, or another feedback device.
- Described below are various hardware configurations implemented in embodiments there to enable monitoring of an end-user's attempted performance of a given skill, which includes identification of predefined observable data conditions (for example observable data conditions defined by way of methodologies described above) in sensor data collected during that attempted performance.
- predefined observable data conditions for example observable data conditions defined by way of methodologies described above
- a wearable garment may include any one or more of: bodysuits, shirts (short or long sleeve), pants (short or long), gloves, footwear, hats, and so on.
- a wearable garment is defined by multiple separable garment items (for example a shirt and pants) which are configured to communicate with one another (for example via wired couplings or wireless communication).
- the garments are preferably manufactured from resilient materials, for example as compression garments. This assists in maintaining sensor components stationary relative to a wearer's body.
- the garments are preferably manufactured to enable removal of electrical components (such as sensor units and a POD device), for example to enable maintenance or the like.
- the garments include a plurality of sensor strands, each sensor strand including one or more sensor units.
- the sensor strands each commence from sensor strand connection port 1208, wherein the sensor strand connection port is configured to couple a plurality of sensor strands to a central processing device, which is referred to as a POD device in a manner consistent to disclosure further above.
- the sensor strands may include a single sensor unit, or multiple sensor units.
- a sensor strand includes multiple sensor units, they are preferably connected in-line. That is, where a strand includes n sensor units SUi ... SU quarter, a communication addressed to a sensor unit Su / is received by and re-transmitted by each of SUi ... SU,. 7 .
- Various addressing protocols may be used, however these are configured such that communications are addressed based on sensor unit mounting locations. This allows sensor units to be installed without a need to ensure a given specific sensor unit is installed at a specific mounting location (which is particularly useful if sensor units are removed for garment washing), and also allows swapping out of sensor units (for example in the case of a fault).
- addressing protocols are in part based on identifiers associated with individual sensor units, in which case the POD device performs an auto-configuration step upon recognising a sensor unit thereby to identify the mounting location at which that sensor unit is installed and associate the sensor's identifier with that mounting location.
- addressing is achieved by techniques that do not require knowledge of sensor identifiers, such as including a retransmission count in messages (for example a message includes a retransmission integer set by the POD device, which is decremented upon each transmission, and the message received and processed by a sensor unit in the case that the decrementing count reaches zero).
- a message includes a retransmission integer set by the POD device, which is decremented upon each transmission, and the message received and processed by a sensor unit in the case that the decrementing count reaches zero.
- each sensor unit includes a circuit board component mounted within a sealed container.
- the sealed container includes two connection ports; one for upstream communication along the sensor strand, one for downstream communication along the sensor strand.
- the sensor unit is able to identify an installed orientation, such that which of the ports is the upstream and downstream port determined based on installation orientation. In other embodiments there is a predefined installation orientation such that the sensor unit is not able to be installed in reverse.
- the connection ports are preferably configured for a snap-locking mounting to complementary connection ports on the sensor strands, such that a physically observable coupling correspondingly provides electronic/communicative coupling.
- the sensor strands include connecting lines, including one or more lines for communication, and one or more lines for power supply (with power for sensor units being provided by the POD device).
- the connecting lines are sealed, such that submersion of the garment in water (for example during cleaning) does not cause damage to the lines.
- connector modules that provide connection of the POD device and sensor units to the connecting lines provide watertight seals.
- all electrical components are provided in a waterproof or water resistant configuration (for example snap-locking engagement of POD device and sensor unit connection ports to sensor strand connection ports provides watertight or water resistant sealing).
- the proximal sensor unit is configured to (i) relay, in a downstream direction, sensor instructions provided by the central processing unit and addressed to one or more of the downstream sensor units; and (ii) relay, in an upstream direction, sensor data provided by a given one of the downstream sensor units to the central processing unit.
- This may include an activation/deactivation instruction.
- the sensor instruction also include sensor configuration data, wherein the sensor configuration data configures the sensor unit to provide sensor data in a defined manner.
- the sensor configuration data is in some cases defined by reference to sampling rates, monitoring a reduced selection of information observable by the sensor components, and other configuration attributes defined specifically for a skill that is being observed by the POD device.
- Each sensor unit includes (i) a microprocessor; (ii) a memory module; and (iii) a set of one or more motion sensor components. More detailed disclosure of exemplary sensor hardware is provided further below. However, these basic components enable a sensor component to receive communications from a POD device, and provide observed data from the sensor components in a predefined manner (for example defined by reference to resolution, sample rates, and so on). In some embodiments each sensor unit includes a local power supply, however it is preferably that power is supplied along the sensor strands from the POD device (or another central power supply) rather than requiring individualised charging of sensor unit batteries or the like.
- the set of one or more sensor components includes one or more of: (i) a gyroscope; (ii) a magnetometer; and (iii) an accelerometer.
- a gyroscope In preferred embodiments described below there is one of each of these components, and each is configured to provide three- axis sensitivity.
- the central processing device includes (i) a power supply; (ii) a microprocessor; and (iii) a memory module.
- the memory module is configured to store software instructions executable by the microprocessor that enable the processing device to perform various functionalities, including configuration of sensor units to transmit sensor data in a predefined manner and to identify one or more sets of predefined observable data conditions in sensor data, including sensor data received by the central processing device from the plurality of connected sensor units.
- the POD device also includes sensor components (for example the same sensor components as a sensor unit) thereby to enable motion observation at the position of the POD device.
- the POD device is mounted to the garment in a pouch provided in a location that, in use, is proximal the upper centre of a user's back (for example between shoulder blades).
- FIG. 4A illustrates a selection of hardware components of a wearable garment according to one embodiment. It will be appreciated that these are illustrated without reference to geometric/spatial configurations resulting from configuration of the garment itself.
- the POD device 400 of FIG. 4A includes a processor 401 coupled to a memory module 402, the memory module being configured to store software instructions thereby to provide functionalities described herein. These include:
- Each skill is defined by data including sensor configuration instructions, rules for identifying observable data conditions in sensor data, and rules relating to feedback (and/or other actions) that are when particular observable data conditions are identified.
- a rechargeable power supply 403 provides power to POD device 400, and to one or more connected devices (including sensor units and, where provided, one or more control units).
- Local sensor components 405 for example three-axis magnetometer, three-axis accelerometer, and three- axis gyroscope) enable the POD device to function as a sensor unit.
- Inputs/outputs 406 are also provided, and these may include the likes of: power/reset buttons; lights configured to display operational characteristics; and in some embodiments a display screen.
- the primary modes of communications between the POD device and a user are by external (and self-powered) user interface devices.
- POD device 400 includes one or more wireless communications modules 404, thereby to enable communications/interactions with one or more remote devices.
- the communications modules may include any one or more of the following:
- WiFi is in some embodiments used to deliver user interface content (including image, text, audio and video data) for rendering at a Ul display device 431 .
- This may include a smartphone, tablet, device with heads-up-display (such as an augmented reality headset or eyewear), and other such devices.
- the Ul display device may be used to select and/or navigate training content available to be delivered via the POD device.
- Bluetooth is in some embodiments used to deliver renderable audio data to a Bluetooth headset or the like, thereby to provide audible instructions/feedback to a user.
- ⁇ ANT+ (or other such communications modules) configured to enable interaction with monitoring devices, such as heart rate monitors and the like.
- ⁇ RF communications modules are provided thereby to enable communication with wireless sensor units, for example sensor units that are configured to be attached to equipment (such as a skateboard, gold club, and so on). In some cases this includes a wireless sensor strand, defined by a plurality of wired sensor units connected to a common hub that wirelessly communicates with the POD device.
- wireless sensor units for example sensor units that are configured to be attached to equipment (such as a skateboard, gold club, and so on).
- equipment such as a skateboard, gold club, and so on.
- this includes a wireless sensor strand, defined by a plurality of wired sensor units connected to a common hub that wirelessly communicates with the POD device.
- the POD device includes a circuit board, and optionally additional hardware components, provided in a sealed or sealable container (water proof or water resistant).
- a sealed or sealable container water proof or water resistant
- This container is able to be mounted to the garment (or example in a specifically configured pouch), and that mounting includes connection of one or more couplings.
- a single coupling connects the POD device to all available sensor strands. Again, this may be a snap-lock coupling (water proof or water resistant), which provides both physical and electronic coupling substantially simultaneously.
- FIG. 4A illustrates multiple sensor strands (Strand 1 .... Strand h) coupled to a sensor connection port 408.
- Each sensor strand includes a plurality of sensor units (Sensor Unit 1 ... Sensor Unit n), however it should be appreciated that in some embodiments a given strand includes only a single sensor unit.
- FIG. 4B illustrates an alternate arrangement of sensor strands.
- some embodiments provide garments configured with one or more "partial" sensor strands.
- Each partial sensor strand includes (i) none or more sensors units; and (ii) a connector module that is configured to couple to a complementary connector module provided by a secondary garment.
- the phrase "none or more” indicates that in some cases a partial sensor strand is defined by a sensor strand line connecting the POD device to a connector module without any intervening sensor units, and on other cases partial sensor strand is defined by a sensor strand line on which one or more sensor units are provided, the strand terminating at a connector module.
- Coupling of the connector module to the complementary connector module provided by a secondary garment functionally connects one or more of the partial sensor strands to a corresponding one or more secondary garment partial sensor strands, thereby to enable communication between (i) one or more sensor units provided on the one or more secondary garment partial sensor strands; and (ii) the central processing device.
- a garment in the example of FIG. 4B, includes a shirt and pants. There are four shirt sensor strands, and two pants sensor strands.
- a connector arrangement 409 couples partial pants strands thereby to enable communication between the sensor units provided on the pants, and the pod device (and powering of those sensor units by the POD device).
- this sort of arrangement is used to enable connection to sensor units provided on footwear, handwear, headwear, and so on.
- connector ports are provided proximal arm, neck and foot apertures thereby to enable elongation of a provided sensor strand by one or more further sensor units carried by a further garment item or device.
- sensors carried by secondary garments include specialist sensor components that measure attributes other than motion.
- pressure sensor components may be used (for example thereby to measure grip strength on a gold club, to measure force being applied to the ground or another object, and so on).
- the POD device is configured to know, for a given training program, the sensor arrangement that is to be provided. For example, a user is provided instructions in terms of the sensor units that should be connected, and the POD device performs a check to ensure that sensors are responding, and expected sensor data is being provided.
- FIG. 4B also illustrates an equipment mountable sensor unit 440.
- This unit includes a processor 441 , memory 442 and sensor components 445 substantially in the same manner as does a sensor unit 420. However, it additionally includes a wireless communications module 446, thereby to enable wireless communications (for example RF communication) with POD device 400, and a local power supply 443. Inputs/outputs (such as lights, power/reset buttons, and the like) are also provided.
- FIG. 4C expands on FIG. 4B by providing a control unit 430.
- This control unit is physically coupled to the distal end of one of the shirt strands, for example as a wrist-mounted control unit.
- the control unit is integrated with a sensor unit.
- Control unit 430 includes input devices 431 , such as one or more buttons, and output devices 432, such as one or more lights and/or a display screen (preferably a low-power screen).
- Control unit 430 is provided to assist a user in providing basic commands to control the provision of training content via the POD device.
- commands may include "previous" and "next", for example to repeat a previous audible instruction, or skip forward to a next stage in a training curriculum.
- audible content is provided to assist a user in operating the input devices, for example by audibly providing selectable menu items.
- control unit 430 additionally includes a wireless communications module (for example RF) configured to receive wireless signals provided by equipment mountable sensor unit 440.
- a wireless communications module for example RF
- wireless sensor unit data is able to be received both at the POD device directly (via modules 404) and indirectly (via module 433, via control unit 430 and along a sensor strand, in this case being shirt sensor strand 4).
- This provides redundancy for the wireless communications; it should be appreciated that there can be challenges in reliably receiving wireless communications where signals pass through a human body (which is predominately water). Having two spaced apart locations (either as shown in FIG. 4D, or via an alternate arrangement), there is a significantly increased chance that all sensor data from unit 440 will be received and made available for analysis.
- the POD device implements a data integrity protocol thereby to determine how to combine/select data provided by each of the two pathways.
- a data integrity protocol thereby to determine how to combine/select data provided by each of the two pathways.
- unit 430 is provided on its own strand, rather than on a sensor strand which might otherwise include a terminal connector for attachment of a sensor-enabled handwear component.
- FIG. 4E provides a schematic representation (not to scale) of a two-piece garment according to one embodiment. This is labelled with reference numerals corresponding to previous figures.
- the illustrated garment is a two-piece garment, being defined by three sensor strands on the shirt component, and two sensor strands which provide sensor units on the pants component (with a connector 409 coupling sensor strands between the garment components).
- sensor units are by no means intended to be limiting, and instead provides a rough guide as to potential sensor unit locations for a garment having this number of sensor units.
- a general principle illustrated in FIG. 4E is to provide sensors away from joints. Data collected from the respective sensor units' gyroscopes, accelerometers and magnetometers enables processing thereby to determine relative sensor locations, angles, movements and so on across multiple axis (noting that providing three 3-axis sensors in effect provides nine degrees of sensitivity for each sensor unit). Rich data relating to body movement is hence able to be determined.
- the sensitivity/operation of each sensor is able to be selectively tuned for particular skills, for example to set levels for each individual sensor component, report only on particular motion artefacts, and so on.
- This is of utility from a range of perspectives, including reducing power consumption at the sensor units, reducing processing overheads at the POD device, and increasing sensitivity to particular crucial motion artefacts (for example by applying a kinematic model which monitors only motions having particular defined characteristics, for example high resolution monitoring of motion in a rowing action, as opposed to motion of a person walking towards a rowing machine).
- FIG. 4F expands on FIG. 4E by way of illustrating a piece of remote equipment, in this case being a skateboard, which carries a wireless sensor unit 440.
- sensor unit 440 communicate wirelessly with POD device 400 via multiple communication pathways, thereby to manage limitations associated with wireless communications.
- signals transmitted by sensor unit 440 are configured to be received by a wireless communications module provided by POD device 400, and by a wireless communications module provided by wrist control unit 430 (which transmits the received sensor data via the sensor strand to which it is connected).
- FIG. 4G expands on FIG. 4F by illustrating a mobile device 481 , and a wireless headset 482.
- POD device 400 communicates with mobile device 481 (for example a smartphone or tablet, which may operate any of a range of operating systems including iOS, Android, Windows, and so on) thereby to provide to mobile device data configured to enable rendering of content in a user interface display, that content assisting in guiding a user through a skills training program.
- the content may include video data, text data, images, and so on.
- POD device 400 operates as a local web server for the delivery of such content (that is, the mobile device connects to a wireless network advertised by the POD device).
- Headset 482 (which need not be a headset of the design configuration illustrated) enables the user to receive audible feedback and/or instructions from the POD device without a need to carry or refer to a mobile device 481 .
- This is relevant, for example, in the context of skills where it would be unfeasible or otherwise generally inconvenient to refer to a mobile device, for example whilst rowing, jogging, swimming, snowboarding, and so on.
- a wired headset may be used, for example via a 3.5mm headphone jack provided by the garment, which is wire-connected to the POD device.
- FIG. 4H illustrates a sensor strand according to one embodiment.
- This includes a plurality of sensor units 420.
- Each sensor unit includes a processor 421 coupled to memory 422.
- Upstream and downstream data connections 423 and 424 are provided (these may in some embodiments be functionally distinguished based on install orientation).
- Inputs/outputs 425 may be provided, such as lights and/or a power/reset button.
- the illustrated embodiment includes a haptic feedback unit 426, which may be used to assist in providing feedback to a user (for example activating haptic feedback on a right arm sensor unit corresponding with an instruction to do something with the user's right arm).
- the illustrated sensor components 427 are a 3-axis magnetometer 427a, a 3-axis accelerometer 427b, and a 3-axis gyroscope 427c.
- FIG. 4I illustrates an exemplary sensor unit 420, showing a housing 496 according to one embodiment.
- This housing is formed of plastic material, and encloses, in a watertight manner, a circuit board 497 which provides components illustrated in FIG. 4H.
- Connectors 498 enable connection to a sensor strand provided by a garment.
- Various embodiments described above make use of data derived from a set of sensor units thereby to enable analysis of a physical performance. These sensor units are mounted to a user's body, for example by way of wearable garments that are configured to carry the multiple sensor units.
- This section, and those which follow, describe exemplary methodologies that are in some embodiments for configuration of sensor units thereby to enable analysis of movements, such as human body movements, based on data derived from the sensors.
- a known and popular approach for collecting data representative of a physical performance is to use optical motion capture techniques.
- optical motion capture techniques position optically markers observable at various locations on a user's body, and using video capture techniques to derive data representative of location and movement of the markers.
- the analysis typically uses a virtually constructed body model (for example a complete skeleton, a facial representation, or the like), and translates location and movement of the markers to the virtually constructed body model.
- a computer system is able to recreate, substantially in real time, the precise movements of a physical human user via a virtual body model defined in a computer system.
- such technology is provided by motion capture technology organisation Vicon.
- Motion capture techniques are limited in their utility given that they generally require both: (i) a user to have markers positioned at various locations on their body; and (ii) capture of user performance using one or more camera devices. Although some technologies (for example those making use of depth sensing cameras) are able to reduce reliance on the need for visual markers, motion capture techniques are nevertheless inherently limited by a need for a performance to occur in a location where it is able to be captured by one or more camera devices.
- Embodiments described herein make use of motion sensor units thereby to overcome limitations associated with motion capture techniques.
- Motion sensor units also referred to as Inertial Measurement Units, or IMUs
- IMUs Inertial Measurement Units
- Such sensor units measure and report parameters including velocity, orientation, and gravitational forces.
- Each sensor unit provides data based on its own local frame of reference.
- each sensor inherently provides data as though it defines in essence the centre of its own universe. This differs from motion capture, where a capture device is inherently able to analysis each marker relative to a common frame of reference.
- Each sensor unit cannot know precisely where on a limb it is located. Although a sensor garment may define approximate locations, individual users will have different body attributes, which will affect precise positioning. This differs from motion capture techniques where markers are typically positioned with high accuracy.
- processing of sensor data leads to defining data representative of a virtual skeletal body model. This, in effect, enables data collected from a motion sensor suit arrangement to provide for similar forms of analysis as are available via conventional motion capture (which also provides data representative of a virtual skeletal body model).
- both motion capture data and sensor-derived data may be collected during an analysis phase, thereby to validate whether a skeletal model data, derived from processing of motion sensor data, matches a corresponding skeletal model derived from motion capture technology. This is applicable in the context of a process for objectively defining skills (as described above), or more generally in the context of testing and validating data sensor data processing methods.
- processor may refer to any device or portion of a device that processes electronic data, e.g., from registers and/or memory to transform that electronic data into other electronic data that, e.g., may be stored in registers and/or memory.
- a "computer” or a “computing machine” or a “computing platform” may include one or more processors.
- the methodologies described herein are, in one embodiment, performable by one or more processors that accept computer-readable (also called machine-readable) code containing a set of instructions that when executed by one or more of the processors carry out at least one of the methods described herein.
- Any processor capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken are included.
- a typical processing system that includes one or more processors.
- Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit.
- the processing system further may include a memory subsystem including main RAM and/or a static RAM, and/or ROM.
- a bus subsystem may be included for communicating between the components.
- the processing system further may be a distributed processing system with processors coupled by a network. If the processing system requires a display, such a display may be included, e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data entry is required, the processing system also includes an input device such as one or more of an alphanumeric input unit such as a keyboard, a pointing control device such as a mouse, and so forth.
- the processing system in some configurations may include a sound output device, and a network interface device.
- the memory subsystem thus includes a computer-readable carrier medium that carries computer-readable code (e.g., software) including a set of instructions to cause performing, when executed by one or more processors, one of more of the methods described herein.
- computer-readable code e.g., software
- the software may reside in the hard disk, or may also reside, completely or at least partially, within the RAM and/or within the processor during execution thereof by the computer system.
- the memory and the processor also constitute computer-readable carrier medium carrying computer-readable code.
- a computer-readable carrier medium may form, or be included in a computer program product.
- the one or more processors operate as a standalone device or may be connected, e.g., networked to other processor(s), in a networked deployment, the one or more processors may operate in the capacity of a server or a user machine in server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment.
- the one or more processors may form a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.
- PC personal computer
- PDA Personal Digital Assistant
- each of the methods described herein is in the form of a computer-readable carrier medium carrying a set of instructions, e.g., a computer program that is for execution on one or more processors, e.g., one or more processors that are part of web server arrangement.
- a computer-readable carrier medium carrying computer readable code including a set of instructions that when executed on one or more processors cause the processor or processors to implement a method.
- aspects of the present invention may take the form of a method, an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects.
- the present invention may take the form of carrier medium (e.g., a computer program product on a computer-readable storage medium) carrying computer-readable program code embodied in the medium.
- the software may further be transmitted or received over a network via a network interface device.
- the carrier medium is shown in an exemplary embodiment to be a single medium, the term “carrier medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions.
- the term “carrier medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by one or more of the processors and that cause the one or more processors to perform any one or more of the methodologies of the present invention.
- a carrier medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media.
- Non-volatile media includes, for example, optical, magnetic disks, and magneto-optical disks.
- Volatile media includes dynamic memory, such as main memory.
- Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise a bus subsystem. Transmission media also may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.
- carrier medium shall accordingly be taken to included, but not be limited to, solid-state memories, a computer product embodied in optical and magnetic media; a medium bearing a propagated signal detectable by at least one processor of one or more processors and representing a set of instructions that, when executed, implement a method; and a transmission medium in a network bearing a propagated signal detectable by at least one processor of the one or more processors and representing the set of instructions.
- Coupled when used in the claims, should not be interpreted as being limited to direct connections only.
- the terms “coupled” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other.
- the scope of the expression a device A coupled to a device B should not be limited to devices or systems wherein an output of device A is directly connected to an input of device B. It means that there exists a path between an output of A and an input of B which may be a path including other devices or means.
- Coupled may mean that two or more elements are either in direct physical or electrical contact, or that two or more elements are not in direct contact with each other but yet still co-operate or interact with each other.
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- Educational Administration (AREA)
- Educational Technology (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Entrepreneurship & Innovation (AREA)
- Multimedia (AREA)
- Electrically Operated Instructional Devices (AREA)
Abstract
Description
Claims
Applications Claiming Priority (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| AU2015901666A AU2015901666A0 (en) | 2015-05-08 | Wearable garments, and wearable garment components, configured to enable delivery of interactive skills training content | |
| AU2015901666 | 2015-05-08 | ||
| AUPCT/AU2016/000020 | 2016-02-02 | ||
| PCT/AU2016/000020 WO2016123648A1 (en) | 2015-02-02 | 2016-02-02 | Frameworks, devices and methodologies configured to enable delivery of interactive skills training content, including content with multiple selectable expert knowledge variations |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2016179654A1 true WO2016179654A1 (en) | 2016-11-17 |
Family
ID=57247634
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/AU2016/050349 Ceased WO2016179654A1 (en) | 2015-05-08 | 2016-05-09 | Wearable garments, and wearable garment components, configured to enable delivery of interactive skills training content |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2016179654A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US12482208B2 (en) | 2023-05-30 | 2025-11-25 | Snap Inc. | Mirroring 3D assets for virtual experiences |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AU2009202962A1 (en) * | 2006-10-26 | 2009-08-13 | Baker, Richard John Mr | Method and apparatus for providing personalised audio-visual instruction |
| US20120246795A1 (en) * | 2011-03-31 | 2012-10-04 | Adidas Ag | Sensor Garment |
| US20140114453A1 (en) * | 2005-01-26 | 2014-04-24 | K-Motion Interactive, Inc. | Method and system for athletic motion analysis and instruction |
-
2016
- 2016-05-09 WO PCT/AU2016/050349 patent/WO2016179654A1/en not_active Ceased
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140114453A1 (en) * | 2005-01-26 | 2014-04-24 | K-Motion Interactive, Inc. | Method and system for athletic motion analysis and instruction |
| AU2009202962A1 (en) * | 2006-10-26 | 2009-08-13 | Baker, Richard John Mr | Method and apparatus for providing personalised audio-visual instruction |
| US20120246795A1 (en) * | 2011-03-31 | 2012-10-04 | Adidas Ag | Sensor Garment |
Non-Patent Citations (1)
| Title |
|---|
| CHO, G. ET AL.: "Smart Clothing: Technology and Applications", 2010, ISBN: 978-1-4200-8852-6, XP055145274 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US12482208B2 (en) | 2023-05-30 | 2025-11-25 | Snap Inc. | Mirroring 3D assets for virtual experiences |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10918924B2 (en) | Frameworks, devices and methodologies configured to enable delivery of interactive skills training content, including content with multiple selectable expert knowledge variations | |
| CN107533806B (en) | Framework, apparatus and method configured to enable delivery of interactive skills training content including content with multiple selectable expert knowledge variations | |
| US10441847B2 (en) | Framework, devices, and methodologies configured to enable gamification via sensor-based monitoring of physically performed skills, including location-specific gamification | |
| US10755466B2 (en) | Method and apparatus for comparing two motions | |
| US20200126284A1 (en) | Motion control based on artificial intelligence | |
| US11210834B1 (en) | Article of clothing facilitating capture of motions | |
| US10942968B2 (en) | Frameworks, devices and methodologies configured to enable automated categorisation and/or searching of media data based on user performance attributes derived from performance sensor units | |
| US11551396B2 (en) | Techniques for establishing biomechanical model through motion capture | |
| JP6999543B2 (en) | Interactive Skills Frameworks and methods configured to enable analysis of physically performed skills, including application to distribution of training content. | |
| US20210307652A1 (en) | Systems and devices for measuring, capturing, and modifying partial and full body kinematics | |
| JP2019534062A (en) | Fitness monitoring system | |
| WO2016179654A1 (en) | Wearable garments, and wearable garment components, configured to enable delivery of interactive skills training content | |
| JPWO2018179664A1 (en) | Information processing apparatus, information processing method and program | |
| KR20210002425A (en) | Method of providing auto-coaching information and system thereof | |
| KR20200071457A (en) | Method of providing auto-coaching information and system thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16791827 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 16791827 Country of ref document: EP Kind code of ref document: A1 |
|
| 32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 21/02/2018) |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 16791827 Country of ref document: EP Kind code of ref document: A1 |