AU2012241079B2

AU2012241079B2 - A system and method that generates outputs

Info

Publication number: AU2012241079B2
Application number: AU2012241079A
Authority: AU
Inventors: Richard James Neil Helmer
Original assignee: Commonwealth Scientific and Industrial Research Organization CSIRO
Current assignee: Commonwealth Scientific and Industrial Research Organization CSIRO
Priority date: 2006-06-29
Filing date: 2012-10-12
Publication date: 2015-04-23
Anticipated expiration: 2027-06-29
Also published as: AU2012241079A1

Abstract

The present invention relates to a system and method that generates outputs based on the operating position of a sensor which is determined by the biomechanical positions or gestures of individual operators. The system including a garment on which one or more than one sensor is removably attached and the sensors provide a signal based on the biomechanical position, movement, action or gestures of the person wearing the garment, a transmitter receiving signals from the sensors and sends signals to a computer that is calibrated to recognise the signals as representing particular positions that are assigned selected outputs. Suitably the outputs are audio outputs of an instrument, such as a guitar, and the outputs simulate the sound of a guitar that would be played when the biomechanical motion, action, gesture or position of the operator resembles those that would occur when an actual instrument is played. strum /Play select Visual feedback -- - -------- -------- Display '' Calibration, configuration performance monitoring User I Sensor Hardware PC logic wInterface . interface P (e.g. shirt) ' .---- Audio feedback Audio '-------------------- -Output Performance

Description

AUSTRALIA Patents Act 1990 COMPLETE SPECIFICATION Standard Patent Applicant (s) COMMONWEALTH SCIENTIFIC AND INDUSTRIAL RESEARCH ORGANISATION Invention Title: A SYSTEM AND METHOD THAT GENERATES OUTPUTS The following statement is a full description of this invention, including the best method for performing it known to me/us: - 2 A SYSTEM AND METHOD THAT GENERATES OUTPUTS FIELD AND BACKGROUND OF THE INVENTION The present invention relates to a system and 5 method that generates outputs based on the position of a sensor. The position of the sensor can be related to many variables but is preferably based on the movement or gestures of a person. 10 According to one embodiment of the present invention the outputs are preferably in the form of sounds which can be played based on the gestures of a person to produce music. 15 An example of an electronic musical instrument that plays sound based on movement and gestures of a person is the "Virtual Air Guitar" system that is currently featuring at the Heureka Science Centre in Finland. The Virtual Air Guitar system utilizes visual 20 recognition technology whereby the hand movements and gestures of a user wearing particular gloves are observed by a webcam and analysed by gesture recognition software. The software monitors the relative spatial position between the user's hands and assumes that the right hand 25 is located on the body of a virtual guitar and that the left hand is located in the fret of the virtual guitar. Movement of the right hand in an upward and downward direction is interpreted as simulating strumming of the guitar while movement of the left hand toward and away 30 from the right hand is interpreted as simulating movement of the left hand along the fret arm of a guitar. The system enables the user to make gestures or actions as if they were playing an actual guitar to 35 produce sound within the mathematical constraints of the model. However, the optical model has inherent complications arising from a single camera source, e.g. 5280173_1 (GHMatter) P61130.AU.2 - 3 it would be very difficult to play the Virtual Air Guitar behind one's head or if any other object blocks the operator's hands from the field of view of the webcam. Furthermore, in our view it would be difficult to usefully 5 employ the sound produced by the system in a musical performance that is able to be reliably reproduced or played to accompany other musical instruments. SUMMARY OF INVENTION 10 System of the invention According to the present invention there is provided a system that is operated by a user to produce 15 outputs, the system including: a garment that is worn by the user having at least two sensors attached to the garment for sensing the biomechanical position of two different joints of a user's body, each sensor being operated over two or more 20 operating positions that each represent a different biomechanical position of the user; an electronic device that receives output signals from the sensors and has a display that displays graphically representations of the operating positions, 25 and the electronic device also producing the outputs of the system; and an input device that can be operated to calibrate the system by assigning each graphical representation of each operating position to at least one of the outputs of 30 the system, and assigning ranges of the output signals of each sensor to each of the graphical representations of the operating positions; wherein a set of outputs can be reproduced by the user by repeating biomechanical movements of the joints 35 for which the garment has sensors. 5280173_1 (GHMatters) P61130.AU.2 Throughout this specification, the notion of the electronic device that produces an output embraces electronic devices that transmits an output, plays an output or creates an output that is stored. 5 It is preferred that the system be in the form of a musical system where the outputs are audible outputs. For example, the audible output may be any sound, a single note or a group of notes played together as a chord, riff, 10 bar of music or other section of music. In any event, it is preferred that the audible output be a pre-recorded or stored sample of sound. The output may also re-produce the sound of any musical instrument including stringed instruments, percussion instruments, brass, woodwind, 15 vocal or any synthesised sound. The audible output may also be any noise or sound created by a single instrument or group of instruments playing together. 20 Although the electronic device of the present invention may have pre-recorded or stored audible outputs, it is preferred that the electronic device have the capacity to receive additional outputs as desired. For 25 example, the additional outputs may be downloaded from the internet, or supplied on optical computer readable media including disks, CDs, DVDs and alike. It is also preferred that the electronic device 30 may produce or play accompanying outputs simultaneously with the outputs based on the operating position of the sensor. For example, the electronic device may automatically play a pre-recorded musical track that lacks the sound of one or more than one musical instrument that 35 can be provided using the system of the present invention. In other words, the system can be used to provide a 5280173_1 (GHMOtters) P6130.AU.2 - 5 complete musical sound with the operator providing their own interpretation to one part of the overall music heard. It is preferred that the operating positions of the 5 sensor be a function of the relative angular positions of one or more body joint of an operator such as finger, wrist, elbow, shoulder, neck, back, hip, knee or ankle joints. It will be appreciated that the angular positions of the above mentioned body joints is dependent on a 10 number of factors including flexibility, shape, size, weight and height of individual people. An advantage of the system of the present invention is that once an individual has calibrated the system to their particular biomechanical movements, the individual through repeating 15 their own biomechanical movements can quickly learn to produce sounds and indeed, produce music that can be repeated or even played with other musical instruments including multiple devices of the instrument disclosed here. 20 It is preferred that the system includes a garment or clothing that is worn by a person and that the sensor is fitted to, or incorporated in the garment. Although it is possible that the sensor may be any 25 dial, switch, key or control knob, it is preferred that the sensor includes an electrical resistor that provides variable electrical resistance depending on the various operating positions of the sensor. 30 It is preferred that the resistor provides variable electrical resistance that is at least in part responsible for the signals received by the electronic device. 35 In the situation where the system includes a garment and the sensor fitted thereto is in the form a variable resistor, it is preferred that the variable 5280173_1 (GHMatters) P61130,AU.2 - 6 resistor be in the form of the arrangement described in our earlier Australian provisional application 2005905666 dated 13 October 2005 and entitled a SYSTEM FOR DETECTING MOVEMENT which is the priority document of International 5 application PCT/AU2006/001521 (WO07/041806). The patent specifications of the provisional application and International application are hereby incorporated into this specification by direct reference. 10 It is preferred that the electronic device is calibrated such that a range of values of the signals from the sensor are associated with a particular output. In other words, the electronic device preferably associates a range of values of signals from the sensor with one or 15 more operating positions. In the situation where the signals from the sensor and thus the operating positions are a function of the relative angular orientation of the body joints, it is preferred that the range of values of the signal selected for the operating positions are based 20 or derived from the angular range over which the body joints are orientated. It is preferred that the range of values of the signals from the sensor that are associated with selected 25 operating position be adjustable. In other words, the range of signals associated with each operating position can be recalibrated on each occasion when a different person uses the system. 30 It is preferred that the electronic device store the configuration in which the system is calibrated or operated for each individual. In other words, once the operator has calibrated the system to their particular biomechanical movements and desired outputs, the 35 personalized calibration can be recalled after use of the system by subsequent operators. Moreover, it is possible for the electronic device to receive the signals of 5280173_1 (GHMatters) P61130.AU.2 - 7 sensors operated by two or more operators and for the electronic device to produce outputs based on the signals from the sensors from each operator. The outputs based on the signals and, preferably the biomechanical movement of 5 multiple operators may be produce or played consecutively, concurrently or simultaneously. It is preferred that the system include two or more sensors, each having two or more than two operating 10 positions. When the system includes two or more sensors each having multiple operating positions, it is preferred that two of the sensors be paired such that at least one of the 15 operating positions of one of the sensors be associated with a number of outputs that equals or is less than the number of operating positions of the other sensor. In the situation when the first sensor has two or more operating positions and the second sensor has two or more operating 20 positions, at least one operating position of the first sensor is associated with two or more outputs and at least one of the operating positions of the second sensor is associated with an output that identifies or matches with one of the two outputs associated with each operating 25 position of the first sensor such that during use, the electronic device will produce or play one of the particular outputs associated with the operating positions of the first sensor that is either: i) identified by the operating position of the second sensor or ii) matches the 30 output associated with the operating position of the second sensor. In other words, the total number of possible outputs is the number of calibrated operating position multiplied together. For example, two sensors each calibrated with two operating positions will enable 35 the system to a total of four outputs. Similarly, when the system contains two sensors, one calibrated with three 52801731 (GHMatters) P61130.AU.2 -8 operating positions and another calibrated with two, a total of six outputs is possible. Preferably, the number of the outputs associated 5 with each operating position of one of the sensors equals the number of operating position of the other sensor. It is even more preferred that the system includes two sensors for monitoring the angular position of the 10 elbows of the operator. The angular orientation of the elbow, and indeed, other bodily joints may range from approximately 45 to 180 degrees but is ultimately depending on the bodily joint 15 involved and the flexibility of the individual. In the situation where the sensor monitors the angular orientation of an elbow, it is preferred that a range of values of the signal and thus the operating positions have an angular orientation ranging over 5 to 45 degrees. The 20 range of angular orientation being a range anywhere between the elbow of the operator being completely curled and completely extended. It is even more preferred that each operating 25 position be defined over an angle ranging from 5 to 20 degrees. It is preferred that the system includes a feedback means that communicates to the operator which 30 operating position the sensor(s) is/are located. It is preferred that the feedback means includes a means for visually communicating with the operator. Although the visual means may be of any form including a 35 sequence of lights, dialogue or pictures, it is preferred that the system includes a monitor for displaying information on the operating positions to the operator and 5280173_1 (GHMatters) P61130.AU.2 - 9 which operating position the sensor is located relative to other operating positions. The feedback means may provide this information both during calibration of the system and/or during general use of the system. 5 It is preferred that the visual means includes a picture or schematic illustration of a musical instrument or a part thereof corresponding to the output sounds and each operating position of the sensor simulates the 10 operating position of part of the operator playing the instrument. For example, when the outputs are sample sounds of the guitar, it is preferred that the feedback means provides visual information that simulates the position of a operator's hand on the fret board for the 15 guitar. In another example, the outputs of the system may be sounds of the piano and in this situation, it is preferred that the feedback means provides visual information that simulates the keys of the key board. 20 It is also preferred that the feedback means also communicates to the operator information identifying which particular output has been associated with the selected operating positions. Although this may be achieved audibly by the system producing or playing the various 25 outputs associated with each operating position, it is also possible that the visual means may display information of the output associated with each operating position. For example, it is preferred that the visual means also identify that the riff, chord, key, notes, or 30 sounds that are associated with each operating position. It is preferred that a period of delay between the sensor being located in an operating position and the electronic device producing or playing a particular output 35 associated with the operating position be minimised. This is to ensure that an operator of the system is at least provided with the perception that the output of the 5280173_1 (GHMatters) P61130.AU.2 - 10 electronic device is substantially spontaneous or occurs shortly after the sensor is located into an operating position. 5 It is preferred that the delay period be equal to or less than 50 milliseconds, preferably equal to or less than 20 milliseconds. It is preferred that the delay period be less than 10 or equal to 10 milliseconds. It is preferred that the electronic device has a memory for storing data on any one or a combination of: e the operating positions, preferably ranges of 15 values of the signals of the sensor for the operating positions during calibration of the system; * the outputs, preferably audible outputs; and e the allocation of particular outputs to selected 20 operating positions. It is preferred that the electronic device has a data process unit for process data on any one or a combination of the following: 25 e carrying out algorithms analysing the signal from the sensor; e carrying out algorithms that assessing whether the signals equals or falls within the ranges of the values of the signals that identify particular 30 operation positions; and * carrying out algorithms that calculate the speed at which the value of a signal of a sensor changes, and preferably determines the volume at which an audible output should be played based on 35 the rate of change of the signal. 5280173_1 (GHMatters) P61130.AU.2 - 11 According to the present invention there is provided a system including: a sensor that a person operates over two or more than two operating positions; and 5 an electronic device that produces outputs, wherein the electronic device receives signals from the sensor based on the operating positions of the sensor and the electronic device is calibrated so that selected operating positions of the sensor and thus the signals received by 10 the electronic device are associated with particular outputs such that, once calibrated, a desired sequence of outputs can be produced by locating the sensor in the respective operating positions. 15 According to an embodiment of the present invention there is provided a system that generates outputs based on the operating position of a sensor which is determined by the biomechanical positions or gestures of individual operators. The system including a garment 20 on which one or more than one sensor is removably attached and the sensors provide a signal based on the biomechanical position, action, motion, movement or gestures of the person wearing the garment, a transmitter receiving signals from the sensors and sends signals to a 25 computer that is calibrated to recognise the signals as representing particular positions that are assigned selected outputs. Suitably the outputs are audio outputs of an instrument, such as a guitar, and the outputs simulate the sound of a guitar that would be played when 30 the biomechanical position, action, motion, movement or gesture of the operator resembles those that would occur when an actual instrument is played. According to the present invention there is also 35 provided a garment that is worn by a person that is operating a system that produces or plays outputs, the garment including: 5280173_1 (GHMatters) P61130.AU.2 - 12 - one or more than one sensor that is removably attached to the garment and at least one of the sensors is operable over two or more than two operating position; and 5 - a transmitter that is removably attached to the garment and receives signals from the sensors and wirelessly transmits signals to an electronic device, wherein the sensors are adapted to transmit signals base 10 on biomechanical movement and positions of the person wearing the garment. According to the present invention there is provided an electronic musical instrument including: 15 a sensor that a person operates over two or more than two operating positions; an electronic device that produces and/or plays audible outputs, wherein the electronic device receives signals from the sensor identifying the operating 20 positions of the sensor and the electronic device is operated/calibrated so that selected operating positions of the sensor and thus the signals received by the electronic device are associated with particular outputs such that, once calibrated, a desired sequence of the 25 audible outputs can be produced and/or played by locating the sensor in the respective operating positions. Although the system of the present invention may be utilised to produce or play sample sounds of any 30 musical instrument, two embodiments of the present invention, each relating to particular musical instruments will now be described in detail. Guitar Embodiment 35 According to one embodiment of the invention, the system can be configured to simulate an acoustic or electrically amplified guitar. In particular, according 5280173_1 (GHMatters) P61130.AU.2 - 13 to this embodiment, it is preferred that one of the sensors monitors the angular orientation of an elbow such that the signals therefrom are, for example, calibrated into at least six ranges, each representing operating 5 positions. Preferably, the six operating positions are each associated with a pairs of outputs, each pair representing a chord that produces the sound of a chord played by an upstrum or a downstrum. Monitoring the angular orientation and, thus, the operating position of 10 the sensor in this manner enables movement of the elbow to be used to simulate a hand being slid back and forth along a fret board of a guitar as if selecting for play six different chords. For convenience, this sensor is hereinafter referred to as the fret board sensor. 15 According to this embodiment, it is also preferred that another sensor, hereinafter referred to as the strumming sensor, monitors the angular orientation of the other elbow such that the signals from the strumming 20 sensor are calibrated into at least two different operating positions. In particular, the electronic device is calibrated such that two ranges of values of the signal from the strumming sensor are each selected as operating positions. Preferably, one of the operating positions 25 represents when the elbow is oriented in a curled position and the other when the elbow is oriented in an extended position and the two operating positions are separated by a further range of angular orientations that is not assigned an output. Moreover, the electronic device 30 receives a signal indication that the elbow has moved'from the curled position to an extended position, the electronic device can produce or play an output that corresponds to a down strumming of the strings of the guitar for a particular chord determined by the signal 35 from the fret sensor. Similarly, when the electronic device receives a signal indicating that the strumming sensor has moved from an extended position to the curled 5280173_1 (GHMatters) P61130.AU.2 - 14 position, the electronic device can produce or play a sound that corresponds to an upward strumming of the guitar of a particular chord or note determined by the signal of the fret broad sensor. 5 According to this embodiment, the operating positions of the fret board sensor are associated with a chord or note in the form of either the chord heard as a down strum or an up strum and the system outputs the sound 10 of either version of the chord based on whether the signal from the strumming sensor identifies an up strum or down strum. In addition, changes in the signal from the strumming sensor, ie a change in angular orientation of the strumming arm is used as the trigger for either 15 playing or not playing the particular sound and the rate of change of the signal determines the volume at sound played. Preferably, volume increases with increasing rates of change of the signal. 20 It is also possible that the signal of the strumming sensor may also be calibrated into 3, 4, 5 or more operating positions. In this situation, each operating position is associated with an output that reflects a single string of a guitar being plucked or a 25 different aspect of producing sound from a guitar. Tambourine Embodiment According to another embodiment of the present 30 invention the system is configured to simulate a tambourine. In particular, according to this embodiment it is preferred that a sensor monitors the angular orientation of an elbow such that the signals from the elbow sensor are calibrated into at least two different 35 positions. One position represents when the elbow is oriented in a curled position and the other when the elbow is oriented in an extended position. Moreover, when the 5280173_1 (GHMatters) P61130.AU.2 - 15 electronic device receives a signal indication that the elbow has moved from the curled position to an extended position, the electronic device can produce or play and output that corresponds to a sound that replicates or 5 simulates cymbals of the tambourine crashing. In addition, when the electronic device receives a signal indicating that the elbow has moved from an extended position to the curled position, the electronic device can produce or play a sound that replicates or simulates sound 10 of a tambourine that hits an objection such as a hand or leg. Method of the invention 15 According to the present invention there is provided a method of operating a system including: a garment that is worn by a user having at least two sensors attached to the garment for sensing the biomechanical position of two different joints of a user's body, each 20 sensor being operated over two or more operating positions that each represent a different biomechanical position of the user; an electronic device that has a display that graphically displays representations of the two or more operating positions, the electronic device also producing 25 the outputs of the system; and an input device that can be operated to calibrate the system, wherein the method includes the steps of: a) transmitting output signals from the sensors to the electronic device, wherein the sensors are adapted to 30 transmit signals based on biomechanical movement and positions of the operator; and b) assigning each graphical representation of the operating positions to at least one of the outputs of the system, 35 c) assigning ranges of the output signals of each sensor to each of the graphical representations of the operating positions; and 5280173_1 (GHMatters) P61130.AU.2 - 16 d) reproducing a set of outputs by the user repeating biomechanical movements of the joints for which the garment has sensors 5 It is preferred that the method be a method of operating a musical system and that the outputs be audible outputs. Preferably, the audible outputs are a sample of notes, chords, riffs, bar or a section of actual music. 10 It is preferred that step a) includes manipulating the sensor between operating positions so as produce a sequence of outputs. It is preferred that step a) involves moving the 15 sensor between known operating positions having known outputs assigned thereto so as produce a known sequence of outputs. The method of the present invention therefore enables novice and experienced musicians alike to learn to play music including music written on a score or even a 20 simplified method of performance annotation. In the situation when only one sensor is present, assigning particular outputs to the operating positions of the senor is performed on single output per operating 25 position basis. In the situation when two or more sensors having multiple operating positions are present, it is preferred that the two sensors are paired such that step b) involves 30 assigning to at least one of the operating positions of the one of the sensors a number of outputs that equals or is less than the number of operating positions of the other sensor. For example, in the situation when the first sensor has two or more operating positions and the 35 second sensor has two or more operating positions, preferably, step b) involves assigning to at least one operating position of the first sensor two or more outputs 5280173_1 (GHMatters) P61130.AU.2 - 17 and the operating positions of the second sensor identifies one of the outputs associated with each operating position of the first sensor such that, when step a) is carried out, the electronic device will produce 5 or play the particular output of the operating position of the first sensor that is identified by the operating position of the second sensor. It is preferred that the step b) involves 10 assigning a number of the outputs to each operating position of one of the sensors the number of operating positions of the other sensor. It is preferred that the method includes the step 15 of: c) defining particular operating positions based on the signal from the sensor received by the electronic device. Preferably, step c) involves defining particular operating positions as a range of values of the signals of the sensor received by the electronic device. These can 20 be initially provided by a supplier and later customised by an individual user. Although the operating positions of the sensor may be defined in any frame of reference using keys, control 25 knobs, switches and alike, in order for the invention to be used in situations where gestures are made as if playing a musical instrument, it is preferred that step c) involves the operating positions be defined by the angular orientation of one or more body parts. Preferably, the 30 operating positions of the sensor be a function of the relative angular positions of one or more body joints such as finger, wrist, elbow, shoulder, neck, back, hips, knees or ankles. 35 In situations where the operating positions are defined by a range of values of signals and are a function of angular orientation, it is preferred that the signals 5280173_1 (GHMalters) P61 130.AU.2 - 18 from the sensor vary with the angular orientations of the bodily joint. In the situation where the operating positions are 5 a function of angular positions of the a body joint, it is preferred that step b) involves assigning particular outputs to the operating position based on the operating position resembling the angular position or gesture that would be required when operating an actual instrument. 10 For example, when the invention is used to produce the sound of a guitar, the operating positions and thus the angular orientations of the operator's arms and suitably their elbows, are co-ordinate with sound of the output. By co-ordinating the operating positions and the outputs 15 in this manner, playing music is intuitive, particular to operators who have prior experience in playing the actual instrument being simulated and represents a method of learning for those of limited experience. 20 It is preferred that any one or a combination of steps a), b) and c) are carried out while the electronic device communicates to the operator information relating to either one or both of: e the operating positions in which the sensor is 25 located; and/or * the output assigned to the particular operating positions. Preferably the information is communicated to the 30 operator by visual and/or audio sources. It is preferred that the visual source visually communicates all of the possible operating positions and that the operating position in which the sensor is located 35 at a particular time is indicated. Preferably, the information communicated to the operator includes a picture or schematic illustration of an instrument or a 5280173_1 (GHMatters) P61130.AU.2 - 19 part thereof and each operating position of the sensor simulates the operating position of part of the operator playing the instrument. 5 It is preferred that the visual source communicate information identifying the particular output assigned to the operating positions. It is preferred that the audio source audibly 10 communicate to the operator information identifying which particular output assigned to the operating positions. It is preferred that that the method includes the step of: d) storing data on the signals or the values of 15 ranges from the sensor that have been selected as defining the operating positions. It is preferred that step d) also involve storing data on the outputs assigned to the operating positions. 20 Preferably the data stored by step d) is stored with reference to the individual operator and, therefore, the electronic device can be readily switched between stored calibrations/configurations when used by one than one 25 operator. It is preferred that the method includes analysing the signal from the sensor using an algorithm such that the visual source communicates to the operator the 30 operating position in which the sensor is located relative to the other operating positions. It is preferred that the method includes analysing the signal from the sensor using an algorithm that 35 calculates whether the signals equal or falls within the ranges of the values of the signals that have been 52801731 (GHMatters) P61130.AU.2 - 20 selected as defining the operating positions in accordance with step c). It is preferred that the method includes analysing 5 the signal from the sensor using an algorithm that calculates the speed at which the value of a signal of a sensor changes, which in turn, preferably determines the volume at which an audible output should be played. 10 It is preferred that the method includes loading samples of output onto the electronic device. The samples many be obtained from any source including the internet and/or from optical computer readable mediums suitably CD's and DVD and alike. 15 It is preferred that the method includes playing a partially complete music at the same time as carrying out step a) such that the outputs played by step a) accompanies the music being played. 20 According to the present invention there is also provided a method of operating a system including one or more sensors having a plurality of operating positions and an electronic device for producing or playing outputs 25 based on biomechanical positions or movement of an individual operator, the method including the steps of: a) transmitting a signal from the sensors to the electronic device, wherein the sensors are adapted to transmit signals based on biomechanical movement and 30 positions of the operator; and b) assigning at least one output to the operating positions of the or each sensor such that when adjusting the sensors between different operating positions, the electronic device produces or plays the desired output. 35 It will be appreciated that the method of the present invention may also include any one or a number of 5280173.1 (GHMatters) P61130.AU.2 - 21 the preferred features described under the heading "System of the invention", such as: * period of delay for the output to be produced or played once the sensor has been located in 5 selected operating position; e the signal from the sensor being at least in part formed by variable resistor and, by of example, has the structure of the arrangement shown in our earlier provisional application 2005905666; and 10 e the outputs being in the form of guitar sounds and two sensors, each providing signals based on the angular orientation of the elbows of the operator, and thereafter carrying out steps a) and b), and optionally steps c) and d) so as to output an 15 actual guitar being played. BRIEF DESCRIPTION OF THE DRAWINGS The present invention will now be described with reference to the accompanying drawings; of which: 20 Figure 1 is a schematic illustration of the components of a system according to a preferred embodiment of the present invention whereby the sound of a guitar is played by a computer based on the angular orientation of the elbows of a musician; 25 Figure 2 is a flow diagram illustrating the communication channels in the system illustrated in the Figure 1; Figure 3 is a screen shot of a monitor of the computer shown in Figure 1 that provides visual feedback to the 30 musician; Figure 4 is a screen shot of a window provided by pressing the "Calibrate Fret Arm Sensor" button shown in Figure 3; Figure 5 is a screen shot of a window provided by 35 pressing the "Assign Samples" button shown in Figure 3; 5280173_1 (GHMatters) P81130.AU.2 - 22 Figure 6 is a screen shot of a window provided by pressing the "Calibrate Strum Arm Sensor" button shown in Figure 3; Figure 7 is a graph showing signals based on the 5 angular orientation of the elbows of the musician that are received by the computer when the system shown in figure 1 is configured as a guitar; Figure 8 is a graph showing the signal of the fret arm shown in Figure 5, wherein the graph identifies four 10 operating positions or zones; Figure 9 is a graph showing the signal of the strum arm shown in Figure 5, wherein the graph identifies two operating positions or zones; Figure 10 is a graph showing the signal for a period 15 of 2 seconds, i.e. from 10 to 12 seconds shown in Figure 9, wherein the value of the signal is taken every 10 milliseconds; and Figure 11 is a screen shot of the computer monitor similar to that shown in Figure 3 with the screen shot 20 providing visual feedback on the status of the system to the musician. DETAILED DESCRIPTION A preferred embodiment of the present invention 25 will now be described with reference to a system and method that a musician can utilize to produce the sound of a guitar. In particular, as will be explained in greater detail below, operation of the preferred embodiment is based on the angular orientation of the elbows of the 30 musician that are monitored using an especially adapted garment. However, it will be appreciated that the present invention can be used to produce the sound of essentially any musical instrument or any other audible output. It is even possible that the present invention 35 may also be used to control visual outputs such as pyrotechnics or light displays. It is also within the scope of the present invention that any bodily joint, 5280173_1 (GHMatter) P61130.AU.2 - 23 switch, keys or control knob may be used as the bases on which to control the operation of the invention. With reference to Figure 1, the preferred 5 embodiment of the invention comprises a garment in the form of a shirt 20, suitably a long sleeved short made from a textile having elastic properties and is equipped with sensors 21 located in the elbow regions. The shirt 20 also includes a transmitter 22 that wirelessly transmits 10 signals to a receiver 23 that is coupled to a computer 24 via a USB port or similar connection. The computer 24 may be any suitable PC or McIntosh machine that has a monitor and a processing unit. The computer 24 is connected to speakers 25 and amplifier if needed. 15 The sensors 24 located at the elbow region of the shirt 20 are preferably in the form of a variable resistor comprising a stiffened filament that is fixed to the shirt 20 and held in particular shapes while connected to 20 contact points that apply a potential difference to the filament. The filament is configured between the contact points in shapes such as I, J or hook shapes, C or curved shapes and X cross shapes that change as the textile of the garment is deformed through being stretched and/or 25 compressed. In particular, when a person wearing the shirt 20 bends their elbow the length of the filament between the contact points changes and, therefore, changes the electrical resistance. Further detail on the structure of a suitable sensor 21 is provided in the 30 patent specification of our earlier Australian provisional application 2005905666 which is now proceeding as International application PCT/AU2006/001521 (WO07/041806). The patent specifications of the provisional application and International application are hereby incorporated into 35 this specification. 5280173_1 (GHMatters) P61130.AU 2 - 24 When the shirt 20 is worn by a musician, changes in angular orientation of the elbows, as indicated by the direction of arrows A, and thus changes in the signals from the sensor 21, preferably measured as either 5 electrical current or voltage are conveyed from the transmitter 22 to the receiver 23 and then directly to the computer 24. The signals from the sensors 21 are converted from an analogue signal to a digital signal either by the transmitter 22 or receiver 23 as desired. 10 The operational status of the system is continuously communicated to the musician by way of sound from the speakers 25 and by information presented on the monitor of the computer 24. 15 Communication between the components of the system is shown by the arrows in the flow diagram in Figure 2. Initially, signals from the sensor 21 are conveyed via the transmitter 22 and receiver 22 to the computer 24 which is shown in the flow diagram as the hardware interface. The 20 processing unit of the computer 24 using suitable algorithms embedded in coding or suitable software, identified in the flow diagram as the PC logic, analyses the signals received and based on the analysis, the computer plays audio outputs via the speaker 25. The 25 audio output can be any sound sample that has been pre stored but suitably, is in the form of sound that correlate at least in part of the gestures or actions of the operator wearing the garment. The sounded produce providing both the desired audio stimulus and feedback on 30 the operating status of the system. In addition, the operating status and the analysis of the signal performed by the computer are also fed back to the musician via the computer monitor. Both the audio 35 and visual feedback to the musician is depicted in Figures 1 and 2 by dashed arrows B and C respectively. 5280173_1 (GHMatters) P61130.AU.2 - 25 The shirt 20 or garment may be any garment including loose or tight fitting garments to which the sensors 21 are fitted to, or incorporated in, the fabric of the garment. Ideally, the sensors 21 and transmitter 5 22 are removably attached to the garment by suitable couplings or interconnecting means including hook and loop fasteners, press studs or co-operating clasps. In the case of the embodiment shown on Figure 1, the sensors 21 are removably attached to the elbows of the garment and 10 signals from the sensors 21 at each elbow are conducted to a single transmitter 22 that is removably located to another part of the garment, suitably remotely located from the sensors 21 so as not to interfere with biomechanical movement of the operator. The sensors 21 15 and transmitters 22 may be directly or indirectly attached to the garment and in the situation where they are indirectly attached preferably the sensors 21 and transmitter 22 are mounted to a separate textile substrate which, when in use, is attached to the garment. The 20 textile substrate and garment may be made of any suitable fabric including elasticized or non-elasticized fabrics. Guitar Configuration 25 Figure 3 is a screen shot of the computer monitor we have developed as a template to aid in the production of guitar sounds. The top section of the screen comprises 6 fret 30 boxes, numbered 1 to 6, which represent 6 operating positions on the fret arm of the guitar and, therefore, 6 different chord positions. Fret box 1 represents a chord position closest to the guitar body and fret box 2 represents the next chord away from guitar board and so 35 forth until fret box 6 represents the chord position furthest from the guitar body. Although Figure 3 shows that the fret arm is separated into 6 different positions, 5280173_1 (GHMatters) P61130.AU.2 - 26 the template is entirely configurable in the sense that the number of the fret boxes could be increased up to 12 or more positions if desired. 5 Located immediately below the 6 fret boxes is a fret line which is broken into six sections by triangular or arrow symbols. The fret line represents the length of the fret arm of a guitar and the length of each section between the arrows represents an operating position which 10 in practice represents a range of angular orientations of the musician's elbows and which correspond with fret boxes 1 to 6. The lines running downwardly from the fret line provide a pictorial representation of the optimal orientation within each section or operating position. 15 By pressing the "Calibrate Fret Arm Sensor" button located at the right hand end of the fret lines, the window shown in Figure 4 will appear. This enables the boundaries of the operating positions and thus the range 20 of angular orientations of the musician's elbow defining each chord or operating position to be adjusted or calibrated individually. The calibration process can be implemented a number of ways, however, the most efficient process we have devised is to first calibrate the system 25 to the fully curled and fully extended positions for the fret sensor which is achieved by activating the MIN and MAX buttons shown in Figure 4, and thereafter segregating or dividing the full range into the required number of operating positions. The operating positions may be 30 evenly or unevenly distributed between the fully curled and fully extended positions. If desired, the arrows defining the boundaries of each operating position or section of the fret line can be calibrated on an individual based using the buttons I and VI based on the 35 operators range of motion between the fully curled an fully extended positions. 5280173_1 (GHMatters) P61130.AU.2 - 27 By pressing the "Assign Samples" button in Figure 3, the window provided in the Figure 5 will appear. By activating the drop boxes for each fret box which are labelled Fret 1, Fret 2......... Fret 6, a range of pre-stored 5 sample sounds can be allocated to the particular fret box. Figure 5 shows that the sample sounds allocated to fret boxes 1, 2, 3 and 4 are: 0 Chuck down which is a percussion sound; e E major; 10 e A Major; and * D Major. Referring back to Figure 3, located below the fret lines are a strumming arm box that represents the status 15 of the strumming arm of the guitar. By pressing the "Calibrate Strum Arm Sensor" the window shown in Figure 6 will appear. The window shown in Figure 6 has buttons "from..." and "...to" associated with both up strumming and down strumming actions. We have found that in order to 20 simulate the sound produced by the strumming action, the strumming arm sensor is best calibrated into two ranges of elbow angular orientation that are separated by a range that is not allocated or interpreted as an action or assigned an output. 25 Typical signals from the fret and strumming arm sensors are provided in Figure 7. The voltage signal from the strumming arm sensor is shown as a thin line and is substantially sinusoidal and represents a strumming arm 30 motion of about 120 beats per minute. The bold line represents the signal from the fret arm sensor and is typical of the step wise motion of the hand moving along the fret arm of an actual guitar. 35 Figure 8 illustrates the signal of the fret arm sensor shown in Figure 7. In addition, Figure 8 has four voltage ranges that equate to the ranges of angular 52801731 (GHMaters) P81130.AU.2 - 28 orientation and, therefore, ranges of signals of the fret elbow sensor. By following the calibration process described above in relation to Figures 3 to 5, operating positions of the angular orientation of the fret elbow 5 sensor are calibrated as follows: Voltage range for fret arm Operating Sample sound sensor based on angular position/Part allocated orientation 3.90 to 4.15 1 Chuck down 3.80 to 3.90 2 E maj 3.65 to 3.80 3 A maj 3.35 to 3.65 4 D maj Figure 9 illustrates the signal from the strumming arm sensor shown in Figure 7. In particular, the signal 10 is a solid line having a sinusoidal wave form. We have found that a realistic guitar strumming sound is produced by defining two discrete operating positions or zones within the typical arm motion. Figure 7 indicates the location of two operating zones, each being defined by a 15 range of angular orientations and, therefore, a range of signals of the strumming arm sensor. Zone 1 ranges from the 3.75 to 3.80 volts and zone 2 ranges from 3.95 to 4.00 volts. The algorithms carried out by computer analysis enable the rate of change of the signal within each zone 20 to be determined. In the situation where the computer determines that the operating zone 1 is played in the upward direction, i.e. an up strum and the fret arm is orientated such that when the signal from the fret arm sensor falls within the boundaries set for fret box 3, the 25 output played is that allocated in Figure 5, i.e. A major as an up strum. In the situation where the computer determines that the operating zone is played in the downward direction, i.e. a down strum and the fret arm is orientated such that when the signal from the fret arm 30 sensor falls within the boundaries set for fret box 3, the 5280173.1 (GHMatters) P61130.AU.2 - 29 output played is that allocated in Figure 5, i.e. A major as a down strum. Similarly, fret operating positions 2 and 4 i.e will be played as E major and D major as up strums when the strum arm sensor detects an up strum or as 5 down strums when the strum sensor detects a down strum. Although the drop boxes for each fret position shown in Figure 5 show outputs as either "up" or "down" which implies that the output is an up strumming sound or and down strumming sound, in the situation where a separate 10 strumming sensor is used, two outputs are allocated to each fret position, namely the chord play as an up strum or a down strum. The differentiation between "up" and "down" chords as shown in the drop box of Fret 4 in Figure 5 is only applicable if a strumming sensor was not in use 15 and, therefore, the number of output for each fret position is limit to one. However, when a strumming sensor is used in conjunction with the fret sensor, two outputs, namely sounds representing up and down strumming of each chord can be assigned to each fret position. 20 The algorithms used to determine whether the signal falls within the Fret boxes 1 to 6 or the strumming zones 1 or 2 may be carried out using any suitable computer language that enables analysis of analogue and/or 25 digital data. We have found that this sort of analysis is more easily carried out on digital data. Figure 10 is a graph showing the strumming signal in Figure 9 for a period from 10 to 12 seconds that has been digitised by sampling the signal from the strumming arm sensor every 10 30 milliseconds (100 Hz) . Each sampling event is indicated by a "+" symbol. The algorithms analyse the rate of the change of the voltage within each strumming zone by calculating the spaces between the "+" symbols and thereby determines whether an up strumming is occurring or whether 35 down strumming is occurring. In addition, the rate of change of the voltage is calculated and is the basis on which the volume of the output is played. We have found 5280173_1 (GHMatters) P61130.AU.2 - 30 that the frequency at which the signal is sampled is important to give the musician responsive audio feedback. Generally speaking, a signal sampling rate in the order of 1 to 50 milliseconds provide the musician with an ability 5 to control volume and expression of the sound in a realistic manner that simulates the sound of an actual guitar. Figure 11 is a screen shot of the computer monitor 10 that is viewed by a musician during a performance and after the calibration of the system. Although not shown in Figures 3 and 11, it is envisaged that the screens be colour so as to aid in the visibility of the information displayed. In the case of Figure 11, fret box 3 is shaded 15 relative to the other fret boxes indicating that the orientation of the fret arm is such that the signal from the fret sensor falls within fret box 3. Similarly, light coloured bars continuously extend and retract from the left to right along the fret arm lines and the strumming 20 arm box which provide visual feedback on the operating position of the fret and strumming arm sensor at any particular stage. We have found that it is important to minimise the 25 period between the system's output and a change in the orientation of the musician elbow, ie. "motion to sound". Ideally the system latency for "motion to sound" is less than 50 milliseconds and preferably less than 10 milliseconds to give a good dynamic performance. A 30 latency of 35 milliseconds is generally acceptable. The latency period is an accumulation of: e the time taken for the signal of the sensor be conveyed from the transmitter; which according to 35 the preferred embodiment is no more that 1 or 2 milliseconds; 5280173_1 (GHMatters) P61130.AU.2 - 31 * the time taken for the transmitter to transmit the signal to a receiver thereafter to through a USB port of the computer; * the time taken of the central processing unit of 5 the computer carry out the algorisms to the status of the signals; and e the time taken to send an signal to the speakers to emit sound. 10 One of the benefits of the present invention over other systems presently available is that depending on the particular outputs allocated to the operating positions, the boundaries of angular orientation of each operating position can be adjusted so that angular orientations 15 resemble or co-inside with the angular orientations for the same output of a actual instrument. Those skilled in the art of the present invention will appreciate that many variations and modifications may 20 be made to the preferred embodiment without departing from the spirit and scope of the present invention. For example, rather than the sensor being in the form of variable electrical sensors as described above, it 25 is possible that the sensors may be in the form of optical fibres and that the amount of light conducted by the fibres and, ultimately, the signal received the computer is a function of the bending of the fibres which resembles body movement. 30 As a further example, the output from any bodily motion may be processed in additional conditioning units to create additional effects, such as reverb, distortion and delay, as is customary with existing instrument 35 recording and/or effect units and the like. 5280173_1 (GHMatter) P61130,AU.2

Claims

1. A system that is operated by a user to produce outputs, the system including: 5 a garment that is worn by the user having at least two sensors attached to the garment for sensing the biomechanical position of two different joints of a user's body, each sensor being operated over two or more operating positions that each represent a different 10 biomechanical position of the user; an electronic device that receives output signals from the sensors and has a display that displays graphically representations of the operating positions, and the electronic device also producing the outputs of 15 the system; and an input device that can be operated to calibrate the system by assigning each graphical representation of each operating position to at least one of the outputs of the system, and assigning ranges of the output signals of 20 each sensor to each of the graphical representations of the operating positions; wherein a set of outputs can be reproduced by the user by repeating biomechanical movements of the joints for which the garment has sensors. 25

2. The system according to claim 1, wherein the input device includes a plurality of selectable buttons corresponding to the graphical representations of each operating position of the sensors, and operating the 30 buttons allows the user to assign ranges of the output signals of each sensor to the graphical representations of the operating positions of the sensors, and allows the outputs of the system to be assigned to the graphical representations of the operating positions of at least one 35 of the sensors. 6265333_1 (GHMatters) P61130.AU.2 BYRON - 33

3. The system according to claim 1 or 2, wherein the sensors include first and second sensors, and the input device can be operated to assign each graphical representation of the operating positions of the first 5 sensor to at least two outputs of the system, and the input device can be operated to associate the graphical representation of the operating positions of the second sensor to each of the two outputs of the graphical operating positions of the first sensor, such that the 10 output of the system is determined by the operating positions of the first and second sensors.

4. The system according to any one of claims 1 to 3, wherein the outputs of the system are in the form of 15 either one or a combination of visual outputs and audible outputs, wherein the audible outputs are a pre-recorded sound or a stored sample of sound, and the audible outputs are produced or played with accompanying outputs that are unrelated to the operating position of the sensors. 20

5. The system according to any one of claims 1 to 4, wherein the sensors includes an electrical resistor that provides variable electrical resistance depending on the various operating positions of the sensors and thereby 25 provide the output signals of the resistors.

6. The system according to any one of claims 1 to 5, wherein the range of output signals that are assigned to graphical representations of the operating positions are 30 adjustable by operating the input device or are adjustable so as to recalibrate the system to suit different individual users.

7. The system according to any one of claims 1 to 6, 35 wherein the system includes a feedback means that communicates to the user the operating positions of the sensors by indicating on a display the graphical 6265333_1 (GHMatters) P61130.AU.2 BYRON - 34 representation of the operating position for the output signal being received by the electronic at any one time and wherein the display includes a picture or schematic illustration of a musical instrument or a part thereof, 5 and each operating position represents or simulates biomechanical positions of the operator playing the instrument which is shown on the display, and wherein the feedback means also communicates to the user information identifying which particular output of the system that has 10 been assigned to each graphical representation of the operating positions.

8. The system according to any one of claims 1 to 7, wherein the delay between the sensors being located in an 15 operative position and the electronic device producing or playing the desired output is equal to or less than 50 milliseconds, and preferably less than or equal to 10 milliseconds. 20

9. The system according to any one of claims 1 to 8, wherein the electronic device has a memory that stores data on any one or a combination of: ethe operating positions, including ranges of values of the output signals of the sensors that define 25 operating positions; ethe outputs of the system, including audible outputs; and ethe assignment or allocation of particular outputs to selected operating positions; and the 30 electronic device has a data process unit that processes data on any one or a combination of the following: 0 carrying out algorithms analysing the signal from the sensor; e carrying out algorithms that assess whether the 35 signals equals or falls within the ranges of the values of the signals that defines calibrated operation positions; and 6265333_1 (GHMatters) P61130.AU.2 BYRON - 35 e carrying out algorithms that calculate the speed at which the value of a signal of a sensor changes, and thereby determines the volume at which an audible output should be played based on the rate of change of the 5 signal.

10. The system according to any one of claims 1 to 9, wherein the system includes a transmitter that receives output signals from the sensors and wirelessly transmits 10 the output signals to the electronic device.

11. The system according to any one of claims 1 to 10, wherein the sensors are removably attached to the garment, thereby allowing the garment to be refreshed by washing or 15 placed entirely as desired.

12. A method of operating a system including: a garment that is worn by a user having at least two sensors attached to the garment for sensing the biomechanical 20 position of two different joints of a user's body, each sensor being operated over two or more operating positions that each represent a different biomechanical position of the user; an electronic device that has a display that graphically displays representations of the two or more 25 operating positions, the electronic device also producing the outputs of the system; and an input device that can be operated to calibrate the system, wherein the method includes the steps of: a) transmitting output signals from the sensors to 30 the electronic device, wherein the sensors are adapted to transmit signals based on biomechanical movement and positions of the operator; b) assigning each graphical representation of the operating positions to at least one of the outputs of the 35 system, 6265333_1 (GHMatters) P61130.AU.2 BYRON - 36 c) assigning ranges of the output signals of each sensor to each of the graphical representations of the operating positions; and d) reproducing a set of outputs by the user 5 repeating biomechanical movements of the joints for which the garment has sensors.

13. The method according to claim 12, wherein the method includes operating buttons of the input device so 10 as to i) assign ranges of the output signals of each sensor to the graphical representations of the operating positions of the sensors, and ii) assign outputs of the system to the 15 graphical representations of the operating positions of at least one of the sensors.

14. The method according to claim 12 or 13, wherein the sensors of the system include first and second 20 sensors, and the input device comprises a plurality of selectable buttons for each sensor, wherein the method includes assigning each graphical representation of the operating positions of the first sensor to at least two outputs of the system, and associating the graphical 25 representation of the operating positions of the second sensor to each of the two outputs of the graphical operating positions of the first sensor, and determining the output of the system based on the operating positions of the first and second sensors. 30

15. The method according to any one of claims 12 to 14, wherein the method includes moving the sensors between known operating positions that have been graphically represented and to which known outputs have been assigned 35 so as produce a known sequence of outputs. 6265333_1 (GHMatters) P61130.AU.2 BYRON - 37

16. The method according to any one of claims 12 to 15, wherein step c) involves defining particular operating positions over a range of selectable biomechanical positions and thus to a range of values of the signals of 5 the sensors received by the electronic device.

17. The method according to any one of claims 12 to 16, wherein any one or a combination of steps a), b) and c) are carried out while the electronic device 10 communicates or presents to the user information relating to either one or both of: e the outputs signals of the sensors and whether this falls within the graphical representations of the operating positions; and/or 15 e the type of output assigned to the calibrated operating positions, and wherein the information is presented or communicated to the operator by visual and/or audio sources. 20

18. The method according to any one of claims 12 to 17, wherein the method further includes step e) that involves storing data on the signals or the range of values of the output signals that have been assigned to 25 the graphical representation of the operating positions for each user.

19. The method according to claim 18, wherein step e) stores data profiles relating to either one or a 30 combination of operating positions and outputs assigned to each operating position with reference to individual operators, and the electronic device can be switched between different stored profiles to facilitate use by more than one operator without requiring recalibration. 35

20. The method according to any one of claims 12 to 19, wherein the method includes analysing the signals from 6265333_1 (GHMatters) P61130.AU.2 BYRON - 38 the sensors using an algorithm that calculates whether the output signals equal or falls within the ranges of the values of the signals that have been selected as defining or calibrated with the operating positions in accordance 5 with step c), and wherein the method includes analysing the output signals from the sensors using an algorithm that calculates the rate of change of the value of the output signals of the sensors, and in turn, determines the volume at which an audible output of the system is played. 6265333_1 (GHMatters) P61130.AU.2 BYRON