DE20121798U1 - Method for real-time control of a jointed object in which an object is controlled via use of a virtual object in a computer program representing a real object - Google Patents

Abstract

Method has the following steps: definition of a moving pattern and its parameters (106) for the object, definition of a similar object surface pattern (108), input of parameter control signals (116) using a 3-D input device, calculation of skeleton alteration data based on the control signal and the moving pattern, calculation of the surface of a the object based on the skeleton alterations and the surface pattern and repetition of the preceding steps as required. Independent claims are made for a computer software product and a system for real time movement control of real and virtual hinged or articulated objects.

Description

System for freely specifiable real-time control

BACKGROUND AND FIELD OF APPLICATION OF THE INVENTION

The invention relates to systems, i.e. products, for graphic visualization and real-time control of virtual and/or real objects for generating and/or influencing image or sound sequences, with the aid of which objects displayed on a screen or real objects such as robots or the like can be manipulated or controlled in their properties and/or actions in a comfortable and reliable manner (quasi) in real time by means of interactive control commands from a user. In particular, the present invention relates to systems for animating and controlling the motion of a three-dimensional (real or virtual) articulated object in real time, to systems for freely specifying control of animated graphics, video or audio data using optical or acoustic parameters, and to a system for real-time motion control of virtual and/or real objects.

It should be noted that "objects" in the sense of the invention can be virtual objects, but also real objects such as robots or other remote-controlled objects with multiple degrees of freedom.

In the context of interactive communication between humans and machines, problems often arise that are due to the insufficient adaptation of the machine to the characteristics of the reception, processing and output of information by the human.

P 24392 EN

t ·

·

user. On the one hand, this mismatch can lead to a flood of information that the user can no longer handle, particularly when several tasks have to be completed. On the other hand, if the user is not challenged enough, for example in a highly automated system where humans only have a control function, the monotony of the work situation can lead to a drop in performance and malfunctions can no longer be controlled due to a lack of practice in how to proceed in such situations. The lack of consideration of the knowledge and level of training of the machine user should also be mentioned here. In many cases, human behaviour, for example in the selection, evaluation and linking of information, in decision-making, problem-solving and in the planning and execution of actions, is still not sufficiently taken into account and supported when it comes to the design of technical systems.

The systems currently used to display and control objects in virtual environments are increasingly taking into account people's ability to absorb and process information, but they have one major disadvantage: when entering control commands to directly influence the scene displayed, the user is still dependent on conventional methods for manually entering information, such as a mouse, trackball, joystick, graphics tablet with stylus or touchscreen. The input mechanisms required for this must first be learned by the user in order to be able to carry out the work at an appropriate reaction speed. In contrast, the innate or existing learned abilities of people to communicate using acoustic signals (e.g. speech) or optical signals (e.g. facial expressions, gestures, signs and movements) are used when entering

P 24392 EN

of information for the control of objects is only inadequately taken into account.

In order to adapt technical systems to people, prior knowledge about their characteristics, behavioral patterns, skills and level of knowledge is necessary. When exchanging information between a user and an information system, the sensory, cognitive and motor characteristics of people are of particular interest.

On the side of the sensory properties of humans determined by the sensory channels, conventional machines and devices for outputting information essentially address the following perception channels:

- the visual channel (eyes) through optical signals,

- the auditory channel (ears) through acoustic signals and

- the tactile channel (sense of touch) through haptic signals.

After the signals have been processed in the brain (cognition), the following channels are essentially available on the side of the human motor properties determined by the output channels:

- arm, hand and finger or leg and foot motor skills as well as body, head, eye or mouth movements, i.e. physical movements, signs, gestures and facial expressions to generate mechanical or optical signals,

- speech motor skills for generating acoustic signals.

Signals can be entered into an information system via these channels in order to trigger a desired action of the system.

·

·

• *

P 24392 EN

&diams;· <

An ideal medium for communication between a user and an information system should be tailored to the sensory and perceptual abilities as well as the motor skills and the specific characteristics of the human user. The information should be structured in such a way that an optimal match is achieved between the representation of the information provided and the user's mental model: If the information to be displayed to the user is presented in a way that, for example, appeals to his spatial perception, the user can handle astonishingly complex amounts of information per unit of time. The information system should also be able to receive, understand and process as many types of information provided by a user as possible and convert it into appropriate actions. This has the advantage that the user can react more efficiently and quickly to new events and situations. User-friendliness and task suitability are therefore typical characteristics of such an ideal communication medium. These characteristics can be expressed as follows:

- Matching the type, extent and speed of output and presentation of the information provided with the sensory characteristics of the human user,

- Taking into account all information channels of the user when recording, recognizing and interpreting received control signals from the user,

- easy to learn and intuitive to use,

P 24392 EN

- high bandwidth of information transmission to the brain and high throughput of information,

- dynamic adaptation of the application to the individual characteristics, skills, tasks, work and organizational techniques of the user,

- Use of a natural interaction language with high semantic content,

- Reliability, robustness and maintainability of the medium,

- social acceptance of the medium among the population,

- Consideration of health, ergonomic and safety-related aspects etc.

The aim of developing suitable interfaces between humans and machines is to build on the properties of human communication channels and skills in order to provide devices, interaction techniques and interfaces that ensure effective two-way communication via these channels. To achieve this goal, so-called "virtual realities" (VR) are particularly suitable. The term "virtual reality" (VR) refers to the computer-based generation of an intuitively perceptible or perceptible scene, consisting of its graphical representation and the interaction options for the user. A virtual environment gives a user access to information that would otherwise not be available at the given place or time. It relies on natural aspects of human perception by using visual information in three spatial dimensions. This information can, for example, be used in a targeted manner.

P 24392 EN

6
changed or enriched with further sensory stimuli. The essential prerequisites are the control of the perspective in real time and the possibility for the user of the system to actively influence the scene displayed.

When navigating through virtual environments, the user can use the type of control that is natural for him. This can be, for example, appropriate arm or leg movements, movements to position the head or eyes, rotational movements of the body or movements directed at an object. By using the user's existing skills for control, the cognitive load during the interaction between man and machine can be reduced. This can increase the range of communication between man and machine and improve the usability of the machine. While in conventional forms of human-machine communication the control of the machine is command-oriented, when controlling objects in virtual environments no specific commands have to be newly learned and used: the computer "observes" the user passively and reacts appropriately to the user's eye, head and/or hand movements under real-time conditions.

The manipulation of the properties and influencing of the actions of objects in a displayed scene requires a complicated interplay of sensory, cognitive processing and motor skills, which is influenced by many factors (individual behavioral patterns and abilities, experiences, environmental influences, etc.). Additional difficulties arise when interacting in a virtual world. In order to control, manipulate or influence objects, a reflex-like or cognitive sensory-motor feedback is particularly important, which is provided by receptors, for example.

P 24392 EN

in the skin, kinesthetic sensations, the sense of balance and visual and/or acoustic sensations. In many cases, this results in a necessary redundancy that is not always present in VR applications. Due to the often inadequate sensory feedback in VR applications, learning motor skills is also made more difficult.

In commercially available VR applications, a distinction is made between systems in which the user is fully integrated into the virtual environment ("immersion") and systems that only provide a "window" to virtual reality. In addition to the well-known forms of human-machine communication such as

- Direct manipulation of objects through manual fine motor operations (pointing, touching, grasping, moving, holding, etc.),

- formal interaction languages (programming languages, command languages and formal query languages), - natural language interaction,

- gestural interaction using non-verbal symbolic commands (facial expressions, gestures, signs, movements) and

- hybrid task-oriented forms of interaction

Virtual realities can also be seen as a new form of human-machine communication. As the name "virtual reality" suggests, a certain level of realism in the representation is necessary: the user should be presented with the sensory information that is required to complete a task or achieve a goal.

Visual perception not only provides information about the position, movement, shape, structure, contour, texture, color

P 24392 EN

8th
or patterning of objects, etc., but also information about the relative body position of the viewer and their movements as well as the nature of the three-dimensional environment. Synthetically generated environments can be made more realistic if as much of the information that occurs in natural environments (motion parallax, vanishing points of the perspective representation, spatial depth and plasticity, lighting and shadows, occlusion, gloss effect, mirror effects and diffuse reflection, etc.) is simulated. How much and what information should be presented depends on the respective task. The differences between the real and virtual world determine how realistic the simulation is perceived to be.

The visual information must be simulated by a computer to create virtual realities. Similar aspects are relevant here as in painting. In the computer-aided simulation of three-dimensional worlds, the projection of individual light rays is usually simulated. The starting point of such a simulation is the specification of the environment to be simulated. To do this, the individual objects must be defined with their properties and their position. For visualization, the intensities of individual pixels are then calculated and projected onto the output medium.

These simulations enable completely new types of learning and practice (examples: vehicle or aircraft simulators), but at the same time they also abstract from certain aspects of the real world. VR applications therefore simultaneously enrich and limit the user's experience options.

P 24392 DE _? . .. . » .... ·* *«·;

9
Basically, VR systems consist of sensors and actuators and their coupling. Important hardware components include the following:

- "Displays" for presenting the virtual environment: Today, visual presentations mainly use monitors, "Head Mounted Displays" (HMD), "Binocular Omni-Oriented Monitors" (BOOM) and projection systems; however, auditory or tactile displays that respond to acoustic or manual user inputs are also used.

- Positioning and orientation systems for recording the user's location and perspective: A distinction is made between determining the absolute position ("position tracking") and measuring the flexion of joints ("angle measurement"). Electromagnetic, kinematic, acoustic, optical and image processing procedures are used.

- Interaction and manipulation systems for the user to act and react in the virtual environment: Pointing devices (2D or 3D mice, trackballs, joysticks, etc.) or tactile devices (touch screens, electromagnetic graphics tablets with styluses, etc.) are used for this purpose; so-called "data gloves" with diffraction and pressure sensors are also being used to an increasing extent. Voice control should also be mentioned in this context.

- Calculation systems and software for generating the virtual environment under real-time requirements.

- Networks for the integration of different users, through which new forms of collaboration can develop.

P 24392 EN

10
The various technical variants of helmet- or head-based systems for visualizing virtual reality are collectively referred to as "Visually Coupled Systems" (VCS). They consist of the following important components:

1.a display attached to the head or helmet,

2.a device for determining the head and/or eye movements of the user,

3.a source of visual information that depends on the head and/or gaze direction of the user.

When using such a system for VR applications, information from both the real and the virtual environment can be presented simultaneously. This is referred to as "see-through displays" for the representation of enriched realities.

Tracking head movements is an important part of VR applications. Usually, the position and orientation of the head in space are determined, and advanced systems can also track the direction of gaze. Most systems use either ultrasound, magnetic or light energy to communicate between the transmitters attached to the head and the receivers. Important technical data that play a role in selecting these systems are:

- the number of degrees of freedom for the directions of movement that can be registered and tracked,

- the detectable angle range,

- the static accuracy (shock sensitivity),

P 24392 EN

- the resolution,

- the reliability,

- the data throughput and the screen scanning frequency,

- the interface to the computer and - other performance aspects.

VR applications can be used successfully in a number of different areas in practice. Some possible applications are outlined below as examples.

- Use in the field of education: By learning how to handle (virtual) objects, interactive demonstrations, visualization of abstract concepts, virtual training of behavior in dangerous situations, virtual exploration of distant places or eras, knowledge can be imparted, creative skills can be trained and behavioral patterns can be practiced.

- Use in driving and flight training in appropriate simulators: By using simulators, behavior can be trained, especially in emergency situations.

- Use in the field of computer games: The possibility of navigating through a virtual scene and the possibility of specifically controlling and influencing virtual objects creates a realistic impression, which can significantly increase the attractiveness of a computer game for the user.

The technologies available today for entering information into a data processing system can be divided into four groups according to the sensors used:

P 24392 EN

!.mechanical input systems (e.g. keyboards, mice, trackballs and joysticks),

2.electrical input systems (e.g. tactile displays and graphics tablets),
3.optical input systems (e.g. light pen) and

4. acoustic input systems (e.g. speech input and speech interpretation systems).

In the following, we will briefly discuss the tools commonly used according to the current state of the art for entering information that are used to control objects in the field of VR applications.

Conventional input systems such as keyboards, mice, trackballs and joysticks are widely used today. They are used to control cursors, mouse pointers, etc., for example to navigate through a virtual scene or move virtual objects on the screen. The disadvantage of these input systems is that they require a surface (i.e. a fixed location) in order to be operated efficiently.

With a touchscreen, however, it is possible to point directly at objects shown on the screen with your finger, without the need for additional space-consuming devices on the desk. Low-resolution touchscreens have 10 to 50 positions horizontally and vertically and use a horizontal and vertical row of infrared LEDs and photo sensors to create a grid of invisible light rays directly in front of the screen. When the screen is touched, both vertical and horizontal light rays are interrupted. From this information, the current finger position can be determined.

·

·

P 24392 EN

Another well-known embodiment of touch-sensitive information input devices is the capacitively coupled touch panel. This provides a resolution of around 100 positions in each direction. When a user touches the conductively coated glass plate of the touchscreen with a finger, the current finger position can be determined based on the change in impedance. Other high-resolution panels use two transparent layers that are minimally spaced apart. One of these is coated with a conductive material, the other with a resistive material. When the finger is pressed against the glass, these two layers touch each other, and the current finger position can then be determined by measuring the resulting voltage drop. A lower-resolution and cheaper variant of this technology uses a grid of fine wires instead of these layers.

According to the state of the art, there are currently several solutions available to the problem of real-time motion control of virtual objects, each of which is optimized for a specific application. Each of these solutions therefore has certain limitations. In order to begin to explain some of the most important of these solutions, it is necessary to briefly discuss their most important aspects.

A possibility for real-time motion control of virtual or real objects has recently emerged as a result of the introduction of input devices for computers that allow the simultaneous input of control signals for several, independent degrees of freedom. The possibilities created by this far exceed those that exist, for example, when using a mouse that can only be controlled two-dimensionally (e.g. on the surface of a desk). It is also known to equip a mouse with additional switches, for example.

P 24392 EN ..

·

14
However, these switches have the disadvantage that they do not allow the input of analog data, but are rather limited to binary data (on/off).

Various input devices are also known from the prior art that can generate analog control signals with different, independent degrees of freedom, whereby each of these analog signals can thus be used as a parameter value of a control. For example, from the patent specification US-A-5,757,360, an egg-shaped input device for computers is known that can be moved freely in space by the user's hand, determines its current positions, directions of movement, speeds and accelerations and transmits this kinematic data wirelessly to the computer. An analog movement sequence is identified in the form of a movement pattern, from which movement commands are derived and implemented in an animated graphic representation. The movement patterns are automatically recognized using a pattern recognition algorithm. Control commands are also generated. The disadvantage of this method is that it cannot be freely specified, since the user's movements, which are recorded analogously by the input device, are assigned to corresponding movements of stored movement sequences of an animated graphic representation and can only be displayed as such.

Input devices that have manually operated force-moment sensors are known, for example, from patent documents DE 36 11 336 C2, DE 37 64 287 and EP 0 979 990 A2. From the last-mentioned patent document, it is also known to use such a force-moment sensor to control a real or virtual mixing or control console, for example to create and design new color, light and/or sound compositions. Here-

P 24392 EN .. .,

·

In turn, the intuitive spatial control in three translational and three rotational degrees of freedom can be advantageously transferred to a continuous spatial mixing or control of a large number of optical and/or acoustic parameters.

Manually controllable input systems that allow navigation in three dimensions are now used successfully in a number of different technical fields of application. One such field of application is, for example, control devices for controlling the functions of electronic musical instruments (especially synthesizers and master keyboards) that have a so-called MIDI interface (Musical Instrument Digital Interface).

One such three-dimensional control device for electronic musical instruments is the D-Beam Controller integrated in the EM-30 and EM-50 keyboards from Roland. By using an extremely sensitive infrared light beam, the hand and/or body movements of a user can be detected without contact above a surface coated with infrared sensor elements. Thanks to the D-Beam technology, these hand and/or body movements of the user can be used to modify or control individual acoustic parameters of recorded improvisations or compositions in real time. The D-Beam Controller converts the user's movements into MIDI signals and analog control signals.

Since one of the preferred embodiments of the present invention is also based on a control device for controlling stored parameterized audio data with the aid of parameterized control signals, which are transmitted with the aid of controllable virtual objects via a MIDI interface,

P 24392 EN

16
to at least one electronic musical instrument

In the following, the aspects of the MIDI standard that are important for understanding the invention will be briefly presented.
5

The MIDI interface is a digital data transfer format between electronic musical instruments, computers and peripheral devices. The MIDI standard is now widely used and has been used by many musicians and composers since its introduction in 1983.

MIDI offers a very efficient way of representing audio data, and this makes MIDI a very attractive data transfer protocol not only for composers or artists, but also for a wide range of computer applications that are able to generate sound patterns (sounds), such as multimedia applications or computer games. Thanks to the publication of the General MIDI System Specification, the most common PC/MIDI interfaces are now widely accepted by users.

In addition, MIDI is supported by the Microsoft Windows operating system and other operating systems. Due to the development and marketing of inexpensive synthesizers, the MIDI standard is becoming increasingly popular with a continuously growing number of applications.

MIDI was originally developed to link two or more keyboards from different manufacturers. However, no one predicted at the time that the MIDI data format would be used to create complete music productions using sequencer systems. Today, MIDI is mainly used as a transmission medium to replace or supplement digitized audio data in computer games or multimedia applications.

P 24392 EN

· t <

MIDI was standardized by the MIDI Manufacturers Association (MMA), to which all manufacturers of digital musical instruments worldwide belong. This committee defines the standard that is binding for all members, including the command structure of the MIDI protocol specified in the MIDI standard. Without this standard, incompatibilities between devices from different manufacturers would be the result.

In contrast to the transmission of analog audio data, when sound samples are transmitted via a MIDI interface from one or more keyboards to a computer (or vice versa), only bit sequences (so-called "MIDI events") are transmitted, which contain the significant acoustic parameters of the pieces of music played or to be played on the individual keyboards in an electronically readable form. These programming commands contain MIDI sequences that instruct the synthesizer, for example, which sound tracks should be played, which solo or accompanying instruments should be used for an arrangement and which musical parameters should be transmitted. Acoustic parameters include, for example, pitches, note or rest values, volumes, tempos, articulation instructions, timbres, pedal, vibrato, chorus, reverb, overtone and/or other special effects. In the following, these acoustic parameters will be referred to as "MIDI playing information". When audio data is played back on a keyboard, this does not involve analog recordings of pieces of music previously played on a keyboard, but rather an exact reproduction of the recording process itself. The polyphonic voices of a playback synthesizer are at least partially occupied.

Compared to the use of sampled audio data stored on a floppy disk or CD-ROM, the generation of sound patterns (sounds) using MIDI

P 24392 DE .. ., ··.

Synthesizers have many advantages. One of these advantages is the amount of space needed to store the parameterized audio data. Files that typically store digitally sampled audio data in a PCM format (such as ".WAV" files) tend to be quite large. This is especially true for long pieces of music recorded in stereo quality at a high sampling rate. In contrast, MIDI files are extremely small. For example, files that store high-quality sampled audio data in stereo quality contain about 10 MBytes per minute of music played, while a typical MIDI sequence is less than 10 kBytes per minute of music played. This is because, as mentioned above, MIDI files do not contain the sampled audio data, but only the programming commands required by a synth to generate the desired sound.

Since the MIDI performance information does not convey any direct information about the type of audio data represented, individual acoustic parameters of this MIDI performance information can be subsequently exchanged at will. This also has great advantages:

- A composer can subsequently instrument or rearrange his work in a variable way.

- Any errors that may have occurred during the installation

(e.g. "wrong notes") can be corrected later.

- Several synthesizers can, for example, reproduce one and the same voice (unison) to achieve more fullness of sound and much more.

P 24392 EN

The additional possibility of editing pieces of music recorded on one or more synthesizers or other electronic musical instruments on the screen, i.e. being able to change, add to, delete, move or transpose individual notes or rests, groups of notes or entire staves, considerably simplifies the work of a composer.

A complete MIDI word usually consists of three bytes. The so-called status byte is sent first. This is a message about the type of message. The status byte is followed by two data bytes that contain information about the content of the message. The following example refers to the MIDI representation for "switching on" a medium pitch tone (c ¹ ) that should sound at a medium volume (Italian: "mezzoforte", mf):

20

Type of
News
(Status-
Byte) Pitch
information
(1. Data
Byte) Volume-
information
(2. Data
Byte) MIDI word
(binary) _1001000O2 0011110O ₂ 0100011O ₂ MIDI word
(decimal) 144io 6Oio 7O ₁₀ Musical
information "Note On" Coated
"C" ( ^c1 ) "Mezzoforte"
(mf)

The first bit (most significant bit, MSB) in the binary representation of the status byte is always assigned the value "1", for data bytes the MSB always has the value "0". In this way, status and data bytes can be clearly distinguished. For the decimal values Z _s for a status byte, the following applies due to the "1" in the MSB: Z ₈ e [128 ₁₀ ; 255i ₀ ] . Since the MSB is set to "0" for the data bytes, i.e. can no longer be used as a value indicator, there are only 1 bits left for the data bytes.

P 24392 EN

&diams; ·

·

20
Bytes only have seven bits each, so that the following applies to the decimal values Z ₀₁ and Z _D 2 of the two data bytes: &Zgr;_&Oacgr;1 e [Oio;127io] and Z ₀₂ e [0i ₀ ; 127i ₀ ] . These 128 different "note on" and "note off" combinations are completely sufficient, since 12 8 different pitches - arranged in an equal-tempered, chromatic scale (i.e. at intervals of semitones) - far exceed the range (ambitus) of a modern concert grand piano with 88 keys.

In order to be able to specifically address individual devices within a MIDI system, there are a total of 16 MIDI channels. The MIDI channel of the transmitter must be identical to that of the respective receiver. Basically, a MIDI data line transports all playing information on all 16 MIDI channels, with the connected sound generators selecting the messages that are intended for them. The address A _k of a selected MIDI channel k (where ke {&Ogr;&iacgr;&ogr;, . . . , 15io} or A _k 6 {000O ₂ , . . . , 111I ₂ )) is transmitted using the last four bits of the status byte. This means that only the first four bits of the status byte contain the status information of a MIDI sequence (e.g. "Note On", "Note Off", etc.).

1. Quartet of 2. Quartet of Status bytes Status bytes (Status- (Channel- Information) address) Type of message 100I ₂ 000O ₂ (Status byte, binary) Type of message 9io &Ogr;&iacgr;&ogr; (Status byte, decimal)

For the example of the "Note On" command described above, this means that the first channel (k = 0) with the address A ₀ = 000O ₂ was selected.

P 24392 EN

21
If a byte of such a MIDI word is received by a synthesizer, the MSB is first used to check whether it is a status byte or a data byte. The receiver must also check all status bytes for their channel addressing, which it must receive based on the respective MIDI channel setting. If a MIDI event with the address assigned to it is discovered, the receiver decodes the data bytes following the status byte and generates the corresponding tones. This principle can be compared to the reading of a choir singer or instrumentalist who, when singing or playing a piece of music, only reads out the voice assigned to him from a polyphonic composition, an arrangement or a score.

The following briefly describes the functions of two of the most common control devices found on synthesizers and master keyboards: the "Pitch Bend Wheel" for continuously detuning the pitch of tones of struck keys on a synthesizer and the "Modulation Wheel" for modulating the timbre characteristics of the pitch of tones of struck keys on a synthesizer.

The MIDI data format provides the smallest possible interval between different pitches as equal-tempered semitone steps or "chromas" (i.e. enharmonically equivalent intervals such as augmented primes, minor seconds or doubly diminished thirds). In order to achieve a continuous frequency variation ("pitch bending"), so-called pitch bend wheel data is required, which can be generated with a corresponding control instrument of the synthesizer (the "pitch bend wheel"). This is usually a wheel or (less often) a joystick that can be moved in four directions. In synthesizers that are equipped with these pitch bend wheels, the pitches of the

P 24392 DE ···· .··,··-:

Keys that are pressed down on the synthesizer keyboard can be detuned towards higher or lower frequencies by rotating the pitch bend wheel. This can normally be used to create detunings of up to one equal-tempered whole tone towards higher or lower frequencies. The pitch bend wheel is usually equipped with a return mechanism that springs back to the middle or normal position when the wheel is released. This position corresponds to the pitch of the pressed keys based on equal-tempered tuning.

Using a modulation function, which can usually be carried out using another control instrument on the synthesizer (the "modulation wheel"), the notes of the keys played on the synthesizer keyboard can optionally be given a vibrato effect. Alternatively, this control instrument can also be used to achieve other effects, such as modifying the brightness or resonance of the notes played by changing their overtone spectrum. If the modulation wheel is turned down to a first stop point, the effect depth is minimal; if it is turned up to a second stop point opposite, the effect depth is maximum.

In order to be able to explain the control of individual MIDI functions using parameterized control signals, the most important MIDI controllers required for this purpose will be briefly discussed below.

Using the first data byte of a MIDI word, a maximum of 128 different controller addresses and thus up to 128 different playing aids or other MIDI functions can be addressed. The second data byte is responsible for the control range. For example, using the data byte for

P 24392 EN

Controller No. 1, which is responsible for frequency modulation, can be used to add a vibrato or tremolo effect to the sounds produced by the synthesizer. The following table lists the most common controllers with their numbers (addresses) and names:

Controller number
(decimal) Controller designation 1 modulation 2 Breath Controller 4 Foot Controller 5 Portamento Time 6 Data Entry Slider 7 Volume 8th balance 10 panorama 11 Expression Pedal 12 to 31 User-Defined 91 External Effects Depth 92 Tremolo Depth 93 Chorus Depth 94 Detune Depth 95 Phaser Depth

Addresses 12 to 31 are not used and offer the user the opportunity to freely assign MIDI functions. Depending on the nature of the synthesizer, these addresses can sometimes be assigned very unusual physical parameters, e.g. oscillator frequency or pulse width of the generated oscillations.

Controllers 32 to 38 are used to more finely resolve the value range of controller addresses 1 to 6. The same is achieved by controller numbers 39 to S3 for addresses 7 to 31.

The game aids described so far have one thing in common: they can be used (in 128 or more

P 24392 EN

&bull;&diams;

·

24
The generic term for these controllers is "continuous controllers". In contrast, there are other controllers that perform switch functions and are therefore generally called "switch controllers". The following table provides an overview of the most important of these controllers:

Controller number
(decimal) Controller designation 64 Sustain Pedal 65 Portamento Switch 66 Sustenuto 67 Soft Pedal 69 Hold pedal 70 to 90 undefined 96 Data Entry (-/No) 97 Data Entry (+/Yes)

A closer look at the second data byte of controller 64 reveals only two values, namely

0000000O ₂ (= O ₁₀ ) for "Pedal Off" and 1111111I ₂ (= 127io) for "Pedal On".

In principle, however, the MIDI data format allows a more differentiated interpretation of the value range I ₁₀ to 126i _0. For example, a multi-stage sustain effect can be provided in this way. In synthesizers that allow this multi-stage sustain effect, the decay phase of sound events when the pedal is "half" depressed is correspondingly shortened compared to the decay phase of sound events of the same pitch and volume when the pedal is depressed all the way down.

Normally, the controller addresses, e.g. for the modulation wheel, the sustain pedal or foot-operated swell pedals to influence the dynamics, are already defined.

P 24392 EN ···· ."."-J

When one of these controllers is used, the controller address sent is fixed. However, modern synthesizers and master keyboards also have freely definable controllers, i.e. the controllers provided for them (pedals, swells, wheels and sliders) can be assigned any controller number. Some sound generators allow the controller functions to be redefined in the device itself. The received controller number can be freely assigned to an internal function.

A variety of different technologies and processes are used today to generate sound using electronic musical instruments, computers or sound cards. Two widely used techniques for generating sound using electronic musical instruments are frequency modulation (FM synthesis) and the use of wavetables (WAV synthesis).

FM synthesis is a process used by sound cards and computers to imitate the timbres of acoustic musical instruments by electronic means. Sounds generated using FM synthesis are easily recognizable as such - in contrast to sound patterns generated by WAV synthesis using wavetables. Master keyboards and synthesizers that have wavetables are relatively expensive, but are often preferred by many professional and amateur musicians because of their high playback quality. The sound patterns are generated by wavetables that contain stored, digitally sampled original sounds (engl.:

Wavetables combine and/or reproduce sound samples (also known as "samples") of acoustic musical instruments. Compared to FM synthesis, in which electronic sounds are generated using a computer, wavetable sound patterns appear much more realistic.

P 24392 EN .. .·

Sound cards can be used to expand the capabilities of conventional computers for generating audio signals and sound patterns. Sound cards are essential for any application that uses sound effects. In order to be able to translate analog audio data into the digital computer language, sound cards have the appropriate devices for digitizing analog sounds. The sound generation by a sound card is based on either FM synthesis or WAV synthesis.
10

OBJECT OF THE PRESENT INVENTION

Based on the above-mentioned prior art, the present invention is dedicated to the task of providing products that enable the user to actively control virtual and/or real objects, using the user's existing skills to send out information. In particular, the products are intended to simplify the movement control of objects in real time or the real-time control of electronically generated audio signals.

This object is achieved according to the invention by systems, i.e. products, having the features of the independent patent claims. Advantageous embodiments of systems which further develop the idea of the invention are defined in the dependent patent claims.

SUMMARY OF THE PRESENT INVENTION

To solve the problem defined in the previous section, the invention proposes an efficient technique for graphic visualization and real-time control of virtual or real objects for generating and/or influencing image or sound sequences, with the help of which

P 24392 EN

e *

Objects displayed on a screen or real objects can be manipulated, controlled or influenced in their properties and/or actions in a comfortable and reliable manner in real time by interactive control commands from a user. In particular, the present invention includes systems for animation and motion control of an articulated object in (quasi) real time, systems for freely specifying control of optical or acoustic parameters and a system for real-time motion control of virtual or real objects.

The present invention relates in particular to input devices that can generate control signals for different, independent degrees of freedom, each using analog signals. Each of these analog signals can thus be used as a parameter value when controlling virtual objects.

In contrast to conventional manually operated, mechanical or touch-sensitive input mechanisms such as keyboards, mice, trackballs, joysticks, graphics tablets with styluses, tactile displays, etc., the technologies used to enter the control commands issued by the user are devices for recording, recognizing, interpreting and processing three-dimensional movements of a user. The user is therefore no longer dependent on the presence of additional hardware devices for manually entering control commands. The evaluation of the entered information can be carried out instead or in addition using signal or pattern recognition methods. Furthermore, the type of input method can be tailored specifically to the user's individual abilities.

P 24392 EN

One aspect of the present invention is that reference patterns for the animation and movement control of objects, for the design of the shape, color and structure of the surfaces of virtual objects and for the generation and/or influencing of image or sound sequences are stored in advance. These reference patterns can then - depending on the control signals of the user - be called up in real time as parameter values for the generation and/or influencing of animation effects, for the surface design of the objects, for the control or targeted manipulation of video or audio data, etc. The storage of the respective reference patterns can be carried out in a standardized manner, so that, based on such a real-time control, a change in the above-mentioned parameter values can be carried out with relatively little effort by a modular exchange of the respective reference patterns at a corresponding interface of the real-time control.

1.System for animation and motion control of an articulated object in real time by controlling it using parameterized motion patterns

According to the underlying invention, it is possible to dispense with a complete reprogramming of a real-time motion sequence; rather, known motion patterns can be made available to a real-time motion controller in a parameterized form via a standardized interface.

A preferred embodiment of the underlying invention is based on a technique for real-time motion control of an articulated object. In this process, at least one parameterized motion pattern is first defined for at least one surface pattern for the object. If control signals are then converted into real-time motion using an input device,

P 24392 DE .. .. e. .... »« &diams;««&diams;

time motion control, calculates these skeleton change data (i.e. the position changes of the contours of a virtual object or a framework consisting of the joint points of a virtual object connected by edges) as parameter values based on the control signals and at least one parameterized movement pattern. The skeleton change data reflects translational and/or rotational changes in the position or orientation of the joints of the object. Finally, the surface of the virtual joint object is calculated based on the skeleton change data and at least one surface pattern. By repeating the last-mentioned steps in close succession, one obtains (depending on the user's control signals) a motion control of the object that an observer can no longer distinguish from a motion control in real time.

By exchanging or redefining a parameterized movement pattern and/or surface pattern, motion control of a new type of joint object can be achieved with very little effort.

According to the embodiment of the underlying invention, control signals for at least six independent degrees of freedom can be input simultaneously and processed in real time. These are, for example, control signals for three translational and three rotational degrees of freedom in a virtual three-dimensional vector space.

Based on the determined instantaneous position vectors of the input device, according to the invention, in particular the first and second time derivatives of the control signals (i.e. the speed, acceleration, angular velocity and angular acceleration vectors) can be determined.

P 24392 EN

30
and processed. The control system therefore determines the change in the control signals over time, so that, for example, in contrast to the aforementioned patent specification US-A-5,757,360, no separate acceleration sensor is required. Instead, the acceleration can be determined by evaluating the development of the control signals over time using the motion control system itself. The assignment of the control signals or their temporal derivatives as parameter values for the stored movement patterns can be freely specified.

2.System for real-time control of acoustic parameters of recorded audio data using parameterized sound patterns

In addition or alternatively, in a further embodiment of the underlying invention, audio data can also be specifically influenced in real time using parameterized control signals from the user. Digitally stored, parameterized sound patterns, such as the acoustic parameters of digitally recorded pieces of music (according to the so-called MIDI standard), which can be controlled via the parameterized control signals from the user, can be specifically modified. These sound patterns can then be called up as parameter values in real time, depending on the respective control signals from the user.

3. System for freely specifiable control of animated graphics, video and/or audio data

According to a further embodiment of the present invention, a technique for freely specifying control of animated graphics, video and/or audio data using optical or acoustic parameters is provided.

P 24392 EN

31
To generate and/or influence these image or sound sequences, at least one predefined, parameterized movement, surface, sound and/or lighting pattern is stored as a reference pattern. For this purpose, control signals with different degrees of freedom are evaluated by an input device and used as parameter values to generate or influence animated graphics, video and/or audio data with the help of at least one stored reference pattern. The assignment of the control signals of each degree of freedom to a specific reference pattern can be freely specified by the user.

In addition, time derivatives of the control signals can be assigned to a specific reference pattern if necessary. The coupling parameters of the assignment, such as the attenuation and the gain, can also be freely set by the user.

4. System for real-time motion control of virtual and/or real objects

According to the invention, a system for real-time motion control of objects can also be provided, which has a memory in which at least one parameterized motion pattern and at least one surface pattern of the object are stored. This system can also have an input device for inputting control signals for real-time motion control. Furthermore, a motion generator can be provided to calculate skeleton change data based on the control signals as parameter values and at least one parameterized motion pattern. The skeleton change data reflects translational and/or rotational changes in the position or orientation of the joints of the object. In addition, a surface generator can be provided, which calculates the surface of virtual joint objects based on their

P 24392 EN

Skeleton change data and a selected surface pattern are calculated.

The input device can be designed for the simultaneous input of control signals for at least six independent degrees of freedom. In particular, it can be designed for the input of control signals for three translational and three rotational degrees of freedom in a three-dimensional vector space. In addition, a unit for determining and processing the time derivatives of these control signals can also be provided.

In order to enable the user to control stored audio data in real time, a sound generator can also be provided for generating sound signals by linking digitally parameterized sound patterns with digital control signals which are present in parameterized form.

In addition, a unit can also be provided for the optional assignment of the control signals or their derivatives as parameter values for the parameterized movement, surface, sound and/or lighting patterns.

5. System for freely specifiable real-time control of animated graphics, video and/or audio sequences

In a further embodiment of the present invention, a system for freely specifiable real-time control of animated graphics, video and/or audio sequences is provided. The system has a memory in which at least one predefined parameterized movement, surface, sound and/or lighting pattern is stored as a reference pattern for generating and/or influencing animated graphics, video and/or audio sequences in real time. The system also has a computing unit for

P 24392 EN

33
Evaluation of control signals of different degrees of freedom from an input device as parameter values for generating and/or influencing animated graphics, video and/or audio sequences with the aid of at least one stored reference pattern. In addition, a setting unit is also provided to enable a freely selectable assignment of control signals of one degree of freedom to a specific reference pattern. This setting unit can optionally enable the assignment of temporal derivatives of the control signals to a specific reference pattern. Furthermore, it can manage a freely selectable setting of coupling parameters of the assignment, such as the temporal attenuation and the gain.

BRIEF DESCRIPTION OF THE DRAWINGS

Further properties, features, advantages and advantages of the underlying invention result from the subordinate dependent patent claims and from the following description of the preferred embodiments of the invention, which are shown in the following drawings.

FIG. 1 is a simplified block diagram illustrating the input and output signals of the freely specifiable real-time controller 102 for animated graphics, video and/or audio sequences according to the preferred embodiment of the underlying invention,

FIG. 2 is a detailed block diagram illustrating the components and the input and output signals of the freely specifiable real-time controller 102 for animated graphics, video and/or audio sequences,

FIG. 3 is a flow chart illustrating the

·

·

·

P 24392 EN

34
the freely specifiable real-time control 102

for animated graphics, video and/or audio sequences,

FIG. 4 is a block diagram illustrating the control of individual or multiple acoustic parameters of digitized sound patterns by a system comprising a computer whose control unit 401 is conductively connected to at least one electronic musical instrument 402 via a MIDI interface and a 3D input device 104 for inputting parameterized control signals for the real-time motion control of virtual objects that can be displayed on a screen, and

FIG. 5 is a block diagram illustrating the control of individual or multiple acoustic parameters of digitized sound patterns by a system comprising a computer having an electronic tone generator 502 and a sound card 510, and a 3D input device 104 for inputting parameterized control signals for real-time motion control of virtual objects that can be displayed on a screen.

DETAILED DESCRIPTION OF THE INVENTION

In the following, the functions of the modules contained in a first embodiment of the present invention, as shown in Figures 1 to 5, are described in more detail.

Referring to Figure 1, the present invention will first be explained schematically. The human-machine interface is represented by the input device 104. This is in particular a so-called 3D input

P 24392 EN

35
output device that can, for example, generate control signals 116 for six independent degrees of freedom when it is manipulated accordingly by a user. These degrees of freedom include, for example, three translational degrees of freedom, which are referred to below as x, y and &zgr;, as well as three rotational degrees of freedom, which are referred to below as φ _&khgr; , <py and φ _&zgr;. The variables x, y and &zgr; denote the orthogonal axes of a three-dimensional Cartesian coordinate system. This can be described mathematically as a three-dimensional vector space V, which is represented by an orthonormal basis consisting of the orthonormal unit vectors e _x , e _y and e ₂ . If these variables are summarized in vector notation, the result for each discrete point in time &eegr; (after omitting the units) is:

the position vector &khgr;(&eegr;) := [&khgr;(&eegr;) , &ggr; (&eegr;) , &zgr;(&eegr;) ] e

and

vj

the direction of rotation vector φ (η) := [φ* (η) , φ^ (η) , _{φ ζ (} η) ] e

The transformed position vector x _T (x,Ax), which results after the translation of a point object P with the position vector &khgr; by the path differences Ax, Ay and Az in the direction of the respective axes x, y and &zgr;, respectively, can be calculated in this vector space V in a simple manner by adding the position vector &khgr; and a linear combination of the orthonormal unit vectors e _v , e _v and

e _z can be expressed as:

tu . tu . in. vj . vj

x _T (x ,Ax) : = &khgr; + Ax

. j_ . VJ A US . VJ A^WA A A

with Ax := Ax-e _x + Ay-e + Az-e _z V Ax,Ay,Az e

The variables φ*, φ^ and #? _z indicate the direction of rotation of virtual objects around the x-, y- and z-axis of this coordinate system.

P 24392 EN

36
systems. The transformed position vector &khgr;_&kgr;(&khgr;,&Dgr;&phgr;), which

after the rotation of a point object P with the position vector &khgr;
by the angles of rotation Ag^ ₁ �Agr;ψγ and �Agr; _{φζ ζ} , can be found in this
Vector space V can be expressed as follows:

xjj (&khgr;,&Agr;&phgr;_&khgr;&igr; A(Py ₁ &Agr;&phgr;_&zgr; ) := R _z (Acp _z ) ■ R _y (A9 _y ) ·
with &Dgr;&phgr; := [ACp _x , α(rgr;γ, _{αφ ζ} ] τ ^V

where the rotation matrices R _x (ACp _x ), R _y (Acp _y ) and R _z (AcpJ are as follows
are defined:

&Xgr;&iacgr;&eegr;(&Dgr;&phgr;_&khgr; )

0 sin(Acp _y )

and

Of course, further degrees of freedom can be added relatively easily, for example with the help of key controls or switches. It should be noted that
Switches or buttons usually use binary control signals
(On/Off), whereas the above-mentioned three translational degrees of freedom x, y or &zgr; or the three rotational degrees of freedom φ*, <Py or φ _&zgr; can each produce analog control signals, which can then be used, for example, in byte-

&bull; *

P 24392 EN

37
wise coding in 2 ⁸ = 256 levels as digital signals for

available for further processing.

Since the three translational and the three rotational degrees of freedom x, y, ζ, Φ*, Ψγ and _{Φ ζ} are to be understood as "analog signals" or digital signals quantized in 256 steps, according to the present invention, the temporal change of these control signals 116 can also be evaluated by the freely specifiable real-time control 102 for animated graphics, video and/or audio sequences, which is described in more detail below. In particular, it is therefore possible to determine the three-dimensional vectors for

the speed &ngr; (&eegr;) := [j&(n) ,

the acceleration a\n) := [Ä(n) , #(n) , Jfe(n) ] ^T e «* ³ , the angular velocity ω (η) : = [ißt (η) , φ^ (η) , (ßc _z (η) ] ^τ e

as well as

the angular acceleration α (η) := [ϕε (η) , <β£ (η) , ϕ| (η) ] ^δe

the degrees of freedom &khgr;, y, &zgr;, &phgr;_&khgr; , &psgr;&ggr; or &phgr;_&zgr; and, if necessary, to use them as further degrees of freedom independently of the absolute value of the respective control signals 116. No further sensors (e.g. speed or acceleration sensors) are required on the part of the input device 104.

While on the input side the control signals 116 from the 3D input device 104 are fed to the freely specifiable real-time controller 102 for animated graphics, video and/or audio sequences according to the present invention, the latter outputs animations depending on these control signals 116.

P 24392 EN

38
Output signals 120, with the aid of which a 3D animation can be generated, for example, on the screen of a computer. This 3D animation 114 can comprise the following:

- controllable movement patterns 106 of at least one movable joint body,

- controllable surface patterns 108 of the joint body and/or its surroundings,

- controllable sound patterns 110 and/or

- controllable lighting patterns 112 for the joint body and/or its surroundings.

It should be noted that the generation of this SD animation 114 is carried out in real time with respect to the control signals 116. In contrast to some known examples according to the state of the art, it is therefore normally not necessary to store sound or image sequences in advance in order to then trigger them at the push of a button.

A great advantage of the present invention is that a new 3D animation 114 does not have to be programmed from scratch, as is the case in the prior art. Instead, stored movement patterns 106, surface patterns 108, sound patterns 110 and/or lighting patterns 112 can be connected in an interface-like manner to the freely specifiable real-time control 102 for animated graphics, video and/or audio sequences. By exchanging or re-specifying at least one of the above-mentioned patterns, a completely new 3D animation 114 can be created in a simple manner. The programming effort required to create a new 3D animation 114 is thus significantly reduced.

P 24392 EN

Since the brain of various animal species and humans can process extremely complex amounts of information per unit of time in the course of evolution with regard to spatial perception and processing of movement patterns, a preferred embodiment of the underlying invention makes simple use of these learned and inherited abilities by making known movement patterns available in a modular manner via a standardized interface of the freely specifiable real-time controller 102 for animated graphics, video and/or audio sequences.

It should be noted that the 3D input device 104 is only one example of an input device for generating control signals 116. In principle, any other type of input device is also suitable, although the 3D input device 104 has the great advantage that the input of control signals 116 of different degrees of freedom can be carried out in a particularly intuitive manner. With regard to the specific embodiments of such 3D input devices 104, reference is made to the prior art cited in the introduction to the description. Such products are commercially available, for example, from the company LogiCad3d.

With reference to Figure 2, the structure of the freely specifiable real-time controller 102 according to the invention for animated graphics, video and/or audio sequences will now be explained in detail. The gray arrows 116 and 120 show the input and output of signals from and to the freely specifiable real-time controller 102, namely the control signals 116 (x, y, ζ, Φ^, <py or _{Φ ζ} ) from the 3D input device 104 and the animation output data 120.

&bull; * 9

·

P 24392 EN

40
As can be seen from Figure 2, the control signals 116 are fed to a free specification module 2 02. One function of the free specification module 2 02 is to determine the first time derivatives (velocities or angular velocities) or the second time derivatives (accelerations or angular accelerations) from the absolute values of the entered "analog data" x, y, z, Ip _x , cpy or _{φ ζ} . Thus, without the aid of an additional sensor (acceleration sensor), for example, the speed and acceleration with respect to the aforementioned "analog data" can be determined from the control signals 116.

A second important function of the free specification module 202 is to assign the control signals 116 of the various degrees of freedom to the components for generating and/or influencing the graphics, image or sound sequences, i.e. the motion generator 204, the surface generator 206, the acoustic generator 208 or the lighting generator 210. For example, a mask can be provided for the user when initializing the real-time control 102 according to the invention for animated graphics, video and/or audio sequences, which allows the user to access the free specification module 202 in such a way that the user can assign one degree of freedom of the control signals 116 to one of the above-mentioned components for generating and/or influencing the graphics, image or sound sequences.

In addition, the user can use this free specification module 202 to freely set the coupling parameters for the assignment of control signals to stored reference patterns, such as for carrying out spectral filtering to attenuate or amplify the signal components of audio signals in predeterminable frequency ranges. The adjustable coupling parameters determine the temporal coupling as well as the attenuation or amplification of the change in the

P 24392 EN

41
corresponding reference pattern depending on the assigned control signal and thus the signal response behavior of the system. Depending on the specification by the user, the free specification module 202 ensures that the coupling parameters of the assignment - such as the damping or amplification - can be freely set. For example, if the coupling parameters are changed with regard to the movement pattern of a walking movement of the joint object, it is possible to switch continuously from a dynamic to a limping walking movement. According to the invention, different moods or emotions, expressed by the facial expressions of a displayed face, can also be set in this way. The movement patterns can be coupled with the mood patterns as follows:

Movement Shown Mood Shown pattern Character- pattern Facial (Parameters #1) figure (Parameters #2) facial expressions dynamic younger funny, Smile, gear, fast athlete cheerful wink, Movements Etc. limping age sad, hanging mouth Gang, 1angs ame Man melancholy angle , and cautious frown Movements Etc.

The user can therefore choose seamlessly between the walking movement of a young athlete and that of an old man and, coupled with this, easily select different characters and/or moods.

P 24392 DE , ..·&diams;

42
In the sense of a higher-order assignment, it is also possible, instead of setting individual coupling parameters, to simply select the character (young/old etc.) and/or the mood (funny/sad etc.) during the specification, with the free specification module 2 02 then automatically setting the individual coupling parameters (for example for the gait, the face etc.). Instead of setting individual coupling parameters, this higher-order (linked) assignment enables the selection of a character and/or a mood. The choice of mood can influence various patterns, so that, for example, the sound, the scene lighting and the movement are changed at the same time. This means that one action by the user can automatically change several coupling parameters of one pattern as well as several coupling parameters of different patterns (movement, surface, image and/or sound). This makes it possible to create choreographies that use real-time control of a character figure and the background.

Depending on the user specification, the free specification module 202 assigns the corresponding control signals 116 as follows:

- as parameterized motion control signals 212 to a motion generator 2 04,

- as parameterized surface control signals 214 to a surface generator 206,

- as parameterized acoustic control signals 216 to an acoustic generator 2 08 and/or

- as parameterized lighting control signals 218 to a lighting generator 210.

P 24392 DE ··..·*» .".'"J .**. "I

43
The motion generator 2 04 links the motion control signals 212 assigned to it with at least one predefined parameterized motion pattern 106 to generate motion animation output signals 220. Thus, different motion patterns of the virtual joint object (e.g. walking in different directions, bending or jumping, etc.) can be controlled in real time.

The surface generator 206 links the surface control signals 214 assigned to it, the motion animation output signals 220 generated by the motion generator 204 and data from at least one predetermined parameterized surface pattern 108 to generate surface output signals 222. Thus, for example, a change in structure, color, color saturation or brightness of certain surfaces of a moving object and/or its surroundings can be initiated.

The acoustic generator 208 links the acoustic control signals 216 assigned to it with at least one parameterized predefined sound pattern 110. The product of this link is the sound output signals 224, which represent part of the animation output signals 120. The stored sound patterns can be, for example, parameterized digital audio files.

In a corresponding manner, the lighting generator 210 links the lighting control signals 218 assigned to it with at least one predefined parameterized lighting pattern 112 to generate lighting output signals 226. The lighting pattern 112 can relate to a virtual object and/or its surroundings.

As already mentioned, the predefined movement patterns 106, surface patterns 108, sound patterns 110 and lighting

P 24392 EN

The control patterns 112 are each parameterized so that the corresponding associated control signals can be used as parameter values by the motion generator 204, the surface generator 206, the acoustic generator 208 or the lighting generator 210 in order to carry out real-time control of the corresponding patterns depending on the control signals 116, wherein the real-time control of the corresponding patterns is reflected in the corresponding change in the generated animation data 220, 222, 224 and 226, which are combined to form the animation output signals 120.

The user can therefore control the movement patterns 106, the surface patterns (geometries, textures, etc.) 108, the sound patterns 110 and/or the lighting patterns 112 in real time by manipulating the input device 104 accordingly and the corresponding assignment by the free specification module 202 (which was set by the user himself or in advance by the manufacturer). On the one hand, he can trigger predefined patterns, for which a switch on the input device 104 can be used, for example. On the other hand, he can carry out real-time control of the aforementioned patterns using the "analog control signals" x, y, z, φ _χ , (p _{y or φ χ )} .

The function of the generators 204, 206, 208 and 210 will be explained in detail using the motion generator 204:

It is assumed that a joint body is to be controlled in real time during the animation. In this case, the motion generator 204 continuously calculates the new position and orientation of the joints of the joint body depending on the motion control signals 212.

P 24392 EN .. » ·--..

A movement pattern 106 is defined, for example, as follows: First, skeleton data is defined that describes the type and position of the joints of the joint body. In addition to the position of the joints, the type of joints (described by the type and number of degrees of freedom) is also defined. For each joint j of the J joints of a virtual joint object under consideration, the following string Sj can be defined, for example:
10

Sj := {Namej, Positionj, Degrees of Freedomj,

Maximum_Amplitudej, Dampingj, . . . }

for 1 < j < J

A motion sequence therefore consists of a list or a function that determines which of the defined joints moves when and how, depending on the parameterized control signals. A motion sequence can therefore consist, for example, of a list of the joints "to be processed". For each cycle, by processing the list or a function, it is calculated which new position and/or orientation of the joints results from the individual contributions of the various joints depending on the control signals.

At a time η, the position of a joint, defined by the coordinates x, y, z, as well as its orientation, defined by the angular coordinates φ*, <Py or <p _z , are calculated as follows:

■ · · ■

P 24392 EN

&eegr; '&khgr;^&Ngr; 46 &eegr;-1 &iacgr;^&Dgr;&khgr; 0 y y &zgr;
φ&khgr; Δφ _&khgr; ·
αφ &zgr;
φ&khgr; ,<&Rgr;&zgr;, j
+ &Sgr; _{Δϕ ζ} .

(D

This equation (1) is therefore calculated for each joint defined in the skeleton data. The new position and orientation of a joint at time η is thus calculated from the position in the previous cycle n-1 plus the new,
contributions dependent on the control signals

Ax _jf Ay.,, Az _j( A<p _x>j , A(p _yj and Aq> _I(j .

Each joint j thus provides a change contribution for the position and orientation of every joint, including itself.

The contributions of each joint j to the displacement of the position or orientation of every other joint and of the same joint are calculated using the following formula:

j, Äy-j, &Dgr;&zgr; j, &Agr;&phgr;_&KHgr;) j, &Dgr;<^ _# j, &Agr;&phgr;_&zgr; , j]

with f _j: \?»

¹⁸

V j e {1,2,3, ...,J}.

The vector-valued function fj represents how a certain movement sequence, e.g. the gait of a person, can be reproduced in a standardized manner.

Referring to the flow chart of Figure 3,
Finally, the process according to the present invention will be explained again. Steps 301a, 301b and 301c are

P 24392 EN

refer to a setting and assignment process in the free specification module 202. This process can either be set by the user or in advance. After starting in step 301a, the user and/or a programmer can use the free specification module 202 to assign the individual control signals 116 of the various degrees of freedom to different patterns (movement patterns 106, surface patterns 108, sound patterns 110 and/or lighting patterns 112). In a step 301c, the coupling parameters between the control signals and the changes to the corresponding patterns actually carried out in real time can then be set. These coupling parameters can, for example, relate to the attenuation and amplification of sound patterns.

When a real-time animation is started in step 3 02, the control signals are then passed on from the 3D input device 104 to the real-time controller 102 for animated graphics, video and/or audio sequences in a step 3 04. The control signals are then evaluated in a step 3 06, whereby, for example, the first and second time derivatives of the "analog control signals" are determined. Finally, in a step 308, the control signals of the various degrees of freedom are assigned to the movement patterns 106, surface patterns 108, sound patterns 110 and/or lighting patterns 112 already mentioned above.

With regard to the lighting pattern 112, it should be noted that both the lighting situation for an animated articulated object itself and the lighting of the background or various scenes can be changed in real time by appropriate assignment and input of corresponding control signals 116.

P 24392 EN

Referring to the embodiment of the underlying invention shown schematically in Figure 4, a method or a system can be provided for controlling individual or multiple acoustic parameters of digitized sound patterns 410, which uses the following devices:

- a computer whose control unit 401 is conductively connected to at least one electronic musical instrument 402 via a MIDI interface 404a+b and

- an input device 104 for inputting parameterized control signals 116 for the real-time motion control of virtual objects that can be displayed on a screen.

The digitized sound patterns 410 can be digitally stored in the memory of at least one electronic musical instrument 402 according to the MIDI standard and generated or played on at least one electronic musical instrument 402 according to the MIDI standard. According to the invention, one or more of these acoustic parameters can be generated or influenced in real time by controlling the virtual objects by moving the input device 104.

As shown in Figure 5, alternatively for controlling individual or multiple acoustic parameters of digitized sound patterns 512 according to a variation of this embodiment of the underlying invention, a method or a system can be provided that uses the following devices:

- a computer whose control unit 501 is connected to at least one sound card 510 and to at least one sound generator 502 and

P 24392 EN

- an input device 104 for inputting parameterized control signals 116 for the real-time motion control of virtual objects that can be displayed on a screen.

The digitized sound patterns 512 can be digitally stored in the memory of at least one sound card 510 of the computer and generated and played on at least one sound generator 502 of the same computer. According to the invention, individual or multiple acoustic parameters of the digitally stored sound patterns 512 can be generated or influenced in real time by controlling the virtual objects by moving the input device 104.

The controllable virtual objects, which are controlled to generate or influence individual or multiple acoustic parameters of the digitally stored sound patterns 410 or 512 by moving the input device 104, can be, for example, the control elements, buttons, switches, knobs and/or sliders of a virtual mixer, with the help of which, for example, individual frequency ranges of the overtone spectrum of a stored, digitized sound pattern 410 or 512 can be specifically attenuated or amplified.

The input device 104 may, for example, be a wireless control medium that can be moved in three dimensions ("3D mouse") and that is capable of simultaneously processing parameterized control signals 116 of movements in at least six independent degrees of freedom in real time.

The controlled acoustic parameters of a specific digitized sound pattern 410 or 512 may be

P 24392 EN

for example, pitches, note or rest values, volumes, tempos, articulation instructions, timbres, pedal, vibrato, chorus, reverb, overtone and/or other special effects.
5

In this case, parameterized control signals 116 of movements of the input device 104 can be input in three translational and three rotational degrees of freedom. In addition, it can be provided according to the invention that temporal derivatives of the parameterized control signals 116 can also be determined and processed. The assignment of parameterized control signals 116, each with a degree of freedom, to one or more parameters of a specific digitized sound pattern 410 or 512 can be freely specified according to the invention. Optionally, the temporal derivatives of the parameterized control signals 116 can also be assigned to one or more parameters of a specific digitized sound pattern 410 or 512. The coupling parameters of the assignment of parameterized control signals 116 can also be freely specified.

The meaning of the symbols provided with reference numerals in Figures 1 to 5 can be found in the attached list of reference numerals.
25

P 24392 EN

List of reference symbols

symbol

simplified block diagram to illustrate the input and output signals of the freely specifiable real-time controller 102 for animated graphics, video and/or audio sequences according to the preferred embodiment of the underlying invention

Freely specifiable real-time control of animated graphics, video and/or audio sequences

3D input device for entering the control signals

parameterized movement patterns for joint objects

parameterized surface patterns (geometries, textures) for joint objects

parameterized sound patterns

parameterized lighting patterns for articulated objects and the background of virtual scenes

3D animation displayed on computer screen

Control signals from the 3D input device 104 for the freely specifiable real-time control 102 for animated graphics, video and/or audio sequences

Transfer of the parameterized movement patterns 106, surface patterns 108, sound patterns 110 and/or lighting patterns 112 as reference patterns to the corresponding generators 204, 206, 208 and 210 in the freely specifiable real-time control 102 for animated graphics, video and/or audio sequences

Animation output signals from the real-time control for animated graphics, video and/or audio sequences for 3D animation

Animation output signals from the control unit 401 of a computer for 3D animation

Status signals from the 3D animation 114 for the control unit 4 01 of the computer

Detailed block diagram to illustrate the components as well as the input and output signals of the freely specifiable real-time control 102 for animated graphics, video and/or audio sequences

Free specification module for assigning the control signals or their temporal derivatives as parameter values for the parameterized movement patterns 106, surface patterns 108, sound patterns 110 or lighting patterns

Motion generator for calculating the skeleton change data of joint objects on the basis of the control signals 116 as parameter values and at least one parameterized movement pattern 106, wherein the skeleton change data reflect translational and/or rotational changes in the position or orientation of the joint objects

Surface generator for calculating the surface virtual

P 24392 EN

No. Symbol

eller joint objects based on their skeleton change data and at least one surface pattern each Acoustic generator for generating sound sequences based on parameterized sound patterns

Lighting generator for generating lighting output signals based on parameterized lighting patterns

Motion control signals from the free specification module to motion generator 2

Surface control signals from the free specification module 202 to the surface generator 206

Sound control signals from the free specification module 202 for the acoustic generator 208

Lighting control signals from the free specification module 2 02 to the lighting generator 210

Motion animation output signals from motion generator 2 04 to surface generator 2 06

Surface output signals from the surface generator

Sound output signals from the acoustic generator 208 as part of the animation output signals 120

Illumination output signals from the illumination generator

Flowchart illustrating the actions performed by the freely specifiable real-time controller 102 for animated graphics, video and/or audio sequences

Start of the setup phase (= preparation of a real-time animation)

Defaults, initializations and assignments for the free specification module 202

Parameter presetting (e.g. attenuation or gain) End of the setup phase

Start of the control phase (= start of a real-time animation)

Input of control signals Evaluation of the control signals

Assignment of control signals

Generation of motion data or influencing motion patterns 106 using the animation output signals 120

Generation of surface data or influencing surface patterns 108 using the animation output signals

Generation of sound data or influencing of sound

star 110 using the animation output signals 120

Generation of lighting data or influencing lighting patterns 112 using the animation output signals 12

Block diagram illustrating the control of individual or multiple acoustic parameters of digitized sound patterns by a system consisting of a computer

P 24392 EN

No. symbol ter, whose control unit 401 is connected to at least one electronic
tronic musical instrument 402 via a MIDI interface
point is conductively connected and a 3D input device 104
for entering parameterized control signals for the
Real-time motion control of virtual objects based on
can be displayed on a screen 401 Control unit of a computer 402 Master keyboard or synthesizer according to the MIDI standard
(here: keyboard with 73 keys, ie with an ambitus of
six octaves) 404a MIDI/IN connector (input jack) of the master keyboard
or Synthesizers 402 404b MIDI/OUT connector (output jack) of the master keyboard
or Synthesizers 402 406 Speaker connection socket of the master keyboard or synth
thesizers 402 408 speaker 410 Memory of the master keyboard or synthesizer 402, in which
a database with digitized sound patterns stored
chert is 412a MIDI/IN signal from the computer’s control unit 401 to
Master keyboard or synthesizer 402 412b MIDI/OUT signal from master keyboard or synthesizer 402
to the control unit 401 of the computer 414 Handover of the digitized sound samples 410 as reference
pattern to the master keyboard or the synthesizer 402 500 Block diagram to illustrate the control of individual
one or more acoustic parameters digitized
Sound patterns through a system consisting of a computer
ter, which has an electronic tone generator 502 and
a sound card 510 and a 3D input device
104 for entering parameterized control signals for
the real-time motion control of virtual objects, the
can be displayed on a screen 501 Control unit of the computer, which has an electronic
Tone generator 502 and a sound card 510 502 electronic sound generator of the computer according to the MIDI
default 504a MIDI/IN signal input of the electronic tone generator 504b MIDI/OUT signal output of the electronic tone generator 506 Computer speaker jack 508 speaker 510 Computer sound card 512 Memory of the sound card 510, in which a database with
digitized sound patterns are stored 514a parameterized MIDI control signals from the control unit
501 of the computer to the sound card 510 514b MIDI/IN signal from the sound card 510 to the electronic
Tone generator 5 02

P 24392 EN

symbol 54 Sound generator 502 to No. 514c MIDI/OUT signal from electronic
Computer control unit 501

·

&bull; t

Claims (27)

1. A system for real-time motion control of an articulated object, comprising: a) means for defining at least one parameterized movement pattern ( 106 ) for the joint object, b) means for defining at least one parameterized surface pattern ( 108 ) for the joint object, c) means for inputting parameterized control signals ( 116 ) by means of a 3D input device ( 104 ), d) means for calculating skeleton change data on the basis of the parameterized control signals ( 116 ) and on the basis of at least one parameterized movement pattern ( 106 ), wherein the skeleton change data reflect translational and/or rotational changes in the position or orientation of the joints of the joint object, e) means for calculating the surface of the joint object on the basis of the skeleton change data and on the basis of at least one surface pattern ( 108 ), wherein the means c), d) and e) can be activated cyclically in order to obtain a real-time motion control of the articulated object depending on the control signals ( 116 ). 2. System according to claim 1, characterized in that means are further provided for generating sound signals ( 224 ) by linking stored digital, parameterized sound patterns ( 110 ) with parameterized control signals ( 116 ). 3. System according to one of the preceding claims, characterized in that time derivatives of the parameterized control signals ( 116 ) can be determined and processed. 4. System according to one of the preceding claims, characterized in that the assignment of the parameterized control signals ( 116 ) or their derivatives to the movement patterns ( 106 ), surface patterns ( 108 ) or sound patterns ( 110 ) can be freely specified. 5. System according to one of the preceding claims, characterized in that parameterized lighting patterns ( 112 ) can also be stored, which can be used to generate lighting effects by means of corresponding links with parameterized control signals ( 116 ). 6. System for freely specifiable real-time control of animated graphics, video and/or audio sequences, comprising - means for generating movement, surface and/or sound data based on at least one predefined parameterized movement pattern ( 106 ), surface pattern ( 108 ) and/or sound pattern ( 110 ), - means for evaluating parameterized control signals ( 116 ) of different degrees of freedom from a 3D input device ( 104 ) and for generating movement data ( 310 ), surface data ( 312 ) and/or sound data ( 314 ) with the aid of the stored movement patterns ( 106 ), surface patterns ( 108 ) and/or sound patterns ( 110 ), and - Means for the freely specifiable assignment of parameterized control signals ( 116 ) of one degree of freedom each to a specific movement pattern ( 106 ), surface pattern ( 108 ) and/or sound pattern ( 110 ). 7. System according to claim 6, further comprising means for assigning temporal derivatives of the parameterized control signals ( 116 ) to a specific movement pattern ( 106 ), surface pattern ( 108 ) and/or sound pattern ( 110 ). 8. System according to claim 6 or 7, characterized in that coupling parameters of the assignment of parameterized control signals ( 116 ), such as attenuation and amplification, are freely specifiable. 9. System for real-time motion control of virtual or real articulated objects, wherein the system has a memory in which at least one parameterized motion pattern ( 106 ) and at least one parameterized surface pattern ( 108 ) of an articulated object are stored, and the system further comprises: - a 3D input device ( 104 ) for inputting parameterized control signals ( 116 ) for real-time motion control, - a motion generator ( 204 ) for calculating skeleton change data on the basis of the parameterized control signals ( 116 ) and on the basis of at least one parameterized movement pattern ( 106 ), wherein the skeleton change data reflect translational and/or rotational changes in the position or orientation of the joints of the joint object, and - a surface generator ( 206 ) for calculating the surface of the joint object based on the skeleton change data and the parameterized surface pattern ( 108 ). 10. System according to claim 9, characterized by a sound generator ( 210 ) for generating sound signals by linking at least one digital parameterized sound pattern ( 110 ) with parameterized control signals ( 116 ). 11. System according to one of claims 9 or 10, characterized by a free specification module ( 202 ) for determining and processing time derivatives of the parameterized control signals ( 116 ). 12. System according to one of claims 9 to 11, characterized by a free specification module ( 202 ) for the optional assignment of the parameterized control signals ( 116 ) or their derivatives for the movement data ( 310 ), surface data ( 312 ) and/or sound data ( 314 ). 13. System for freely specifiable real-time control of animated graphics, video and/or audio sequences, comprising - a memory in which at least one predefined parameterized movement pattern ( 106 ), surface pattern ( 108 ), sound pattern ( 110 ) and/or lighting pattern ( 112 ) is stored for the generation of movement data ( 310 ), surface data ( 312 ), sound data ( 314 ) and/or lighting data ( 316 ), and - a computing unit for evaluating parameterized control signals ( 116 ) of different degrees of freedom from a 3D input device ( 104 ) for generating movement data ( 310 ), surface data ( 312 ), sound data ( 314 ) and/or lighting data ( 316 ) with the aid of the stored movement patterns ( 106 ), surface patterns ( 108 ), sound patterns ( 110 ) and/or lighting patterns ( 112 ), further comprising a free specification module ( 202 ) for the freely specifiable assignment of parameterized control signals ( 116 ) each of a degree of freedom to a specific movement pattern ( 106 ), surface pattern ( 108 ), sound pattern ( 110 ) and/or lighting pattern ( 112 ). 14. System according to claim 13, characterized in that the free specification module ( 202 ) further enables the assignment of temporal derivatives of the parameterized control signals ( 116 ) to a specific movement pattern ( 106 ), surface pattern ( 108 ), sound pattern ( 110 ) and/or lighting pattern ( 112 ). 15. System according to claim 13 or 14, characterized in that the free specification module ( 202 ) enables a freely specifiable setting of coupling parameters of the assignment of parameterized control signals ( 116 ), such as attenuation and amplification. 16. System for real-time control of acoustic parameters of digitized sound patterns ( 410 ), comprising: - a computer whose control unit ( 401 ) is conductively connected to at least one electronic musical instrument ( 402 ) via a MIDI interface (404a + b) and - a 3D input device ( 104 ) for inputting parameterized control signals ( 116 ) where the digitized sound patterns ( 410 ) - are digitally stored in the memory of at least one electronic musical instrument ( 402 ) and - can be generated and played on at least one electronic musical instrument ( 402 ), characterized in that individual or multiple acoustic parameters of the digitally stored sound patterns ( 410 ) can be generated or influenced in real time by moving the 3D input device ( 104 ). 17. System according to claim 16, characterized in that control elements, buttons, switches, rotary knobs and/or sliders of a virtual mixing console are provided for generating or influencing individual or multiple acoustic parameters of the digitally stored sound patterns ( 410 ). 18. System according to claim 17, characterized in that the controlled acoustic parameters of a particular digitized sound pattern ( 410 ) are pitches, note or rest values, volumes, tempos, articulation instructions, timbres, pedal, vibrato, chorus, reverb, overtone and/or other special effects. 19. System according to one of claims 16 to 18, characterized in that time derivatives of the parameterized control signals ( 116 ) can be determined and processed. 20. System according to one of claims 16 to 19, characterized in that the assignment of parameterized control signals ( 116 ) each having a degree of freedom to one or more parameters of a specific digitized sound pattern ( 410 ) can be freely specified. 21. System for real-time control of acoustic parameters of digitized sound patterns ( 512 ), comprising: - a computer whose control unit ( 501 ) is connected to at least one sound card ( 510 ) and to at least one sound generator ( 502 ) and - a 3D input device ( 104 ) for inputting parameterized control signals ( 116 ) for real-time motion control of objects, where the digitized sound patterns ( 512 ) - are digitally stored in the memory of at least one sound card ( 510 ) of the computer and - can be generated and played on at least one sound generator ( 502 ) of the same computer, characterized in that individual or multiple acoustic parameters of the digitally stored sound patterns ( 512 ) can be generated or influenced in real time by moving the 3D input device ( 104 ). 22. System according to claim 21, characterized in that temporal derivatives of the parameterized control signals ( 116 ) can be assigned to one or more parameters of a specific digitized sound pattern ( 512 ). 23. System according to one of claims 21 or 22, characterized in that coupling parameters of the assignment of parameterized control signals ( 116 ) can be freely specified. 24. System for real-time control of acoustic parameters of digitally recorded sound patterns ( 410 or 512 ), which - are digitally stored in the memory of at least one sound card ( 510 ) of a computer and/or at least one electronic musical instrument ( 402 ) according to the MIDI standard and wherein the control unit ( 401 or 501 ) of the computer - is conductively connected to at least one sound card ( 510 ) and at least one tone generator ( 502 ) or to at least one electronic musical instrument ( 402 ) via a MIDI interface (404a + b or 504a + b) and - is conductively connected to a 3D input device ( 104 ) for inputting parameterized control signals ( 116 ) for real-time motion control of virtual objects, characterized in that individual or multiple acoustic parameters of the digitally stored sound patterns ( 410 or 512 ) can be generated or changed in real time by controlling the virtual objects by moving the 3D input device ( 104 ). 25. System according to claim 24, characterized in that the virtual objects which are moved to generate or influence individual or multiple acoustic parameters of the digitally stored sound patterns ( 410 or 512 ) are the control elements, buttons, switches, knobs and/or sliders of a virtual mixing console. 26. System according to one of claims 24 or 25, characterized in that the controlled acoustic parameters of a specific digitized sound pattern ( 410 or 512 ) are pitches, note or rest values, volumes, tempos, articulation instructions, timbres, pedal, vibrato, chorus, reverb, overtone and/or other special effects. 27. System according to one of claims 24 to 26, characterized in that coupling parameters of the assignment of parameterized control signals ( 116 ) can be freely specified.