EP1652405B1 - Device and method for the generation, storage or processing of an audio representation of an audio scene - Google Patents

Abstract

The audio signal processing circuit (12) has a large number of input channels (16) carrying separate signals (Ek1-Ekm). The circuit may be a WFS (Wave Field Synthesis) rendering unit. It has a large number of output channels (14) carrying loudspeaker signals (LS1- LSn). A display device (18) provides the inputs for the audio signal processing circuit. Temporally non- overlapping audio objects are allocated to the same input channel. The input for the display device comes from the device generating the audio scene (10) with separate audio objects.

Description

The present invention is in the field of wave field synthesis and more particularly relates to apparatus and methods for generating, storing or manipulating an audio representation of an audio scene.

There is an increasing demand for new technologies and innovative products in the field of consumer electronics. It is an important prerequisite for the success of new multimedia systems to offer optimal functionalities and capabilities. This is achieved through the use of digital technologies and especially computer technology. Examples of these are the applications that offer an improved, realistic audiovisual impression. In previous audio systems, a significant weakness lies in the quality of the spatial sound reproduction of natural, but also of virtual environments.

Methods for multi-channel speaker reproduction of audio signals have been known and standardized for many years. All the usual techniques have the disadvantage that both the installation site of the loudspeakers and the position of the listener are already impressed on the transmission format. If the speakers are arranged incorrectly with respect to the listener, the audio quality suffers significantly. An optimal sound is only possible in a small area of the playback room, the so-called sweet spot.

A better natural spatial impression as well as a stronger envelope in the audio reproduction can be achieved with the help of a new technology. The basics of this Technology, the so-called Wave Field Synthesis (WFS), was researched at the TU Delft and first presented in the late 1980s ( Berkhout, AJ; de Vries, D .; Vogel, P .: Acoustic control by wave-field synthesis. JASA 93, 1993 ).

Due to the enormous demands of this method on computer performance and transmission rates, wave field synthesis has rarely been used in practice. Only the advances in the areas of microprocessor technology and audio coding allow today the use of this technology in concrete applications. The first professional products are expected next year. In a few years, the first wave field synthesis applications for the consumer sector will be launched.

• Jeder Punkt, der von einer Welle erfasst wird, ist Ausgangspunkt einer Elementarwelle, die sich kugelförmig bzw. kreisförmig ausbreitet.

The basic idea of WFS is based on the application of Huygens' principle of wave theory:

• Every point, which is detected by a wave, is the starting point of an elementary wave, which spreads in a spherical or circular manner.

Applied to the acoustics can be simulated by a large number of speakers, which are arranged side by side (a so-called speaker array), any shape of an incoming wavefront. In the simplest case, a single point source to be reproduced and a linear arrangement of the speakers, the audio signals of each speaker must be fed with a time delay and amplitude scaling so that the radiated sound fields of each speaker properly overlap. With multiple sound sources, the contribution to each speaker is calculated separately for each source and the resulting signals added together. If the sources to be reproduced are in a room with reflective walls, reflections must also be reproduced as additional sources via the loudspeaker array. The cost of the calculation therefore depends heavily on the number of sound sources, the reflection characteristics of the recording room and the number of speakers.

The advantage of this technique is in particular that a natural spatial sound impression over a large area of the playback room is possible. In contrast to the known techniques, the direction and distance of sound sources are reproduced very accurately. To a limited extent, virtual sound sources can even be positioned between the real speaker array and the listener.

Although wavefield synthesis works well for environments whose characteristics are known, irregularities occur when the texture changes, or when wave field synthesis is performed based on environmental conditions that do not match the actual nature of the environment.

However, the technique of wave field synthesis can also be used advantageously to supplement a visual perception with a corresponding spatial audio perception. Until now, production in virtual studios focused on providing an authentic visual impression of the virtual scene. The matching to the image acoustic impression is usually impressed by manual operations in the so-called post-production subsequently the audio signal or classified as too complex and time-consuming in the realization and therefore neglected. This usually leads to a contradiction of the individual sense sensations, which leads to the space being designed, i. H. the designed scene is perceived as less authentic.

Generally speaking, for example, the audio material is a movie of a plurality of audio objects. An audio object is a sound source in the film setting. For example, when thinking of a movie scene in which two people and are in a dialogue, while z. For example, when approaching a rider and a train, there are a total of four sound sources in this scene over a period of time, namely the two persons, the approaching rider, and the approaching train. If it is assumed that the two persons in dialogue are not talking at the same time, then at least two audio objects are likely to be active at any one time, namely the rider and the train, if at that time both persons are silent. If, however, another person speaks at a different time, three audio objects are active, namely the rider, the train and the one person. If, in fact, the two persons speak at the same time, then four audio objects are active at this time, namely the rider, the train, the first person and the second person.

Generally speaking, an audio object is such that the audio object describes a sound source in a film setting that is active or "alive" at a particular time. This means that an audio object is still characterized by a start time and an end time. For example, in the previous example, the tab and the train are active throughout the setting. As the two approach, the listener will perceive this by the louder sounds of the rider and the train becoming louder, and possibly - in an optimal wave field synthesis setting - also changing the positions of those sound sources accordingly. On the other hand, the two speakers in dialogue are constantly generating new audio objects, because whenever a speaker stops speaking the current audio object is over and when the other speaker starts to speak, a new audio object is started, which in turn ends is when the other speaker stops talking, and then when the first speaker starts speaking again, a new audio object is started again.

There are existing wave field synthesis rendering devices capable of producing a certain number of loudspeaker signals from a certain number of input channels, knowing the individual positions of the loudspeakers in a wave field synthesis loudspeaker array.

The wave-field synthesis renderer is effectively the "heart" of a wave-field synthesis system that computes the loudspeaker signals for the many loudspeakers of the loudspeaker array in amplitude and phase so that the user not only has an optimal visual impression but also an optimal acoustic impression.

Since the introduction of multi-channel audio in movies in the late 1960's, the goal of the sound engineer has always been to give the listener the impression that he is properly involved in the scene. Adding a surround channel to the reproduction system was another milestone. New digital systems followed in the 90s, which resulted in an increase in the number of audio channels. Today, 5.1 or 7.1 systems are standard systems for movie playback.

These systems, in many cases, have proven to be a great potential for creatively supporting the perception of movies, and provide good opportunities for sound effects, atmospheres, or surround mixed music. On the other hand, the wave field synthesis technique is so flexible that it provides maximum freedom in this regard.

However, the use of 5.1 or 7.1 systems has resulted in several "standardized" ways to handle the mix of movie soundtracks.

Rendering systems usually have fixed speaker positions, such as the left channel in the case of 5.1 ("left"), the center channel, the right channel, the surround left channel ("surround left") and the surround right channel ("surround right"). As a result of these fixed (few) positions, the ideal sound image sought by the sound engineer is limited to a small number of seats, the so-called sweet spot. Although the use of phantom sources between the above-mentioned 5.1 positions leads in certain cases to improvements, but not always to satisfactory results.

The sound of a movie usually consists of dialogues, effects, atmospheres and music. Each of these elements is mixed in consideration of the limitations of 5.1 and 7.1 systems. Typically, the dialog is merged into the center channel (in 7.1 systems also in a half-left and a half-right position). This implies that when the actor moves across the screen, the sound does not follow. Motion sound effects can only be realized if they move quickly, so that the listener is unable to detect when the sound is passing from one speaker to another.

Lateral sources also can not be positioned due to the large audible gap between the front and surround speakers, so that objects can not move slowly from back to front and vice versa.

Furthermore, surround speakers are placed in a diffuse array of loudspeakers, thus creating a sound image that is a kind of envelope for the listener. Therefore, accurately positioned sound sources behind the listeners are avoided to avoid the unpleasant sonic interference field associated with such accurately positioned sources.

Wave field synthesis as a completely new way of building the sound field perceived by the listener overcomes these essential shortcomings. The consequence for cinema applications is that an accurate sound image can be achieved without limitations with regard to a two-dimensional positioning of objects. This opens up a wide variety of possibilities in the design and mixing of sound for cinema use. Due to the complete sound image reproduction, which is achieved by the technique of wave field synthesis, now sound sources can be freely positioned. Further, sound sources may be placed as focused sources within the audience room as well as outside the audience room.

In addition, stable sound source directions and stable sound source positions can be generated using point-shaped radiating sources or plane waves. Finally, sound sources can be moved freely within, outside, or through the audience room.

This leads to an enormous potential of creative possibilities and also to the possibility to place sound sources exactly according to the picture on the screen, for example for the entire dialogue. This actually makes it possible to embed the listener not only visually, but also acoustically in the film.

Due to historical circumstances, the sound design, ie the activity of the sound engineer, is based on the channel or track or "track" paradigm. This means that the coding format or the number of speakers, ie 5.1 systems or 7.1 systems, determine the reproduction setup. In particular, a special sound system also requires a special encoding format. As a consequence, it is impossible to make any changes to the master file without the complete mix again perform. For example, it is not possible to selectively change, ie change, a dialog track in the final master file without changing all the other notes in that scene as well.

On the other hand, the channels do not matter to a viewer / listener. He does not care which sound system produces a sound, whether an original sound description was object-oriented, channel-oriented, etc. The listener also does not care if and how an audio setting has been mixed. All that counts for the listener is the sound impression, so whether he likes a sound setting for a film or a sound setting without a film or not.

On the other hand, it is essential that new concepts be adopted by the persons who are to work with the new concepts. Responsible for the sound mixing are the sound engineers. Sound engineers are "calibrated" to work channel-oriented due to the channel-oriented paradigm. For them, it is actually the goal, for. For example, for a cinema with 5.1 sound system, mix the six channels. Here they use z. B. recorded in a virtual studio audio signals and mix the ultimate z. B. 5.1 or 7.1 speaker signals. This is not about audio objects, but channel orientation. Thus, in this case, an audio object typically has no start time or end time. Instead, a signal will be active for a speaker from the first second of the movie to the last second of the movie. This is because any one of the (few) speakers of the typical cinema sound system will always produce any sound, since there should always be a sound source that is broadcast over the particular speaker, even if it is just background music.

For this reason, existing wave-field synthesis rendering units are used to channel-orientate work so that they have a certain number of input channels, from which, when in the input channels the audio signals and associated information are entered, the speaker signals for each speaker or groups of speakers of a wave field synthesis speaker array are generated.

On the other hand, the technique of wave field synthesis makes an audio scene much more "transparent" in that, in principle, an unlimited number of audio objects viewed over a movie, that is, viewed over an audio scene, may be present. With regard to channel-oriented wave-field synthesis rendering devices, this can become problematic if the number of audio objects in an audio scene exceeds the typically always predetermined maximum number of input channels of the audio processing device. In addition, for a user, ie for a sound engineer, for example, who creates an audio presentation of an audio scene, the plurality of audio objects that also exist at certain times and at other times again do not exist, that is a defined start and a defined end time have confusing, which in turn could lead to a psychological threshold between the sound masters and the wave field synthesis, the Tonmeister is just bring a significant creative potential, is built.

The object of the present invention is to provide a concept for generating, storing or editing an audio presentation of an audio scene that has a high acceptance on the part of the users for whom corresponding tools are intended.

This object is achieved by an apparatus for generating, storing or editing an audio presentation of an audio scene according to claim 1, a method for generating, storing or editing an audio presentation of an audio scene solved according to claim 15 or a computer program according to claim 16.

The present invention is based on the finding that for audio objects, as they occur in a typical film setting, only an object-oriented description can be processed clearly and efficiently. The object-oriented description of the audio scene with objects that have an audio signal and to which a defined start and a defined end time are assigned, correspond to the typical conditions in the real world, in which it is rare that a noise the whole There is time. Instead, it is customary, for example in a dialogue, for a dialogue partner to start talking and ceasing to speak or for sounds to typically have a beginning and an end. In this respect, the object-oriented audio scene description, which assigns each sound source in real life its own object, adapted to the natural conditions and therefore optimal in terms of transparency, clarity, efficiency and clarity.

On the other hand, z. Tonmeister, for example, who want to create an audio presentation of an audio scene, who want to incorporate their creative potential to "synthesize" an audio presentation of a audio scene in a movie theater possibly taking into account special audio effects, because of the channel paradigm accustomed typically to work with either hardware or software-implemented mixers, which are a consistent implementation of the channel-oriented operation. In hardware- or software-implemented mixers, each channel has knobs, knobs, etc., which manipulate the audio signal in that channel.

According to the invention, a balance is made between the object-oriented audio presentation that is life-affirming, and the channel-oriented presentation provided to the sound engineer is achieved in that an imaging device is used to image the object-oriented description of the audio scene to a plurality of input channels of an audio processing device, such as a wave field synthesis rendering unit. According to the invention, the mapping means is adapted to assign a first audio object to an input channel and to assign a second audio object whose start time is after an end time of the first audio object to the same input channel and a third audio object whose start time is after the start time of the first audio object and before the end time of the first audio object is to assign to another of the plurality of input channels.

This timing, which assigns concurrent audio objects to different input channels of the wave field synthesis rendering unit, but which assigns sequentially occurring audio objects to the same input channel, has proven to be extremely channel efficient. This means that a relatively small number of input channels of the wave field synthesis rendering unit is occupied on average, which serves for clarity, and on the other hand, the computing efficiency of the already very compute-consuming wave field synthesis rendering unit accommodates. Due to the relatively small number of concurrently occupied channels on average, the user, for example the sound engineer, can get a quick overview of the complexity of an audio scene at a specific time without having to search laboriously from a plurality of input channels, which object is currently active or which object is currently not active. On the other hand, the user can easily perform a manipulation of the audio objects as in object-oriented representation by his usual channel controller.

This is expected to increase the acceptance of the inventive concept to the effect that the users with the concept according to the invention a familiar working environment is delivered, which nevertheless contains a much higher innovative potential. The inventive concept, which is based on the mapping of the object-oriented audio approach into a channel-oriented rendering approach, thus meets all requirements. First, the object-oriented description of an audio scene, as it has been done, is best adapted to nature and therefore efficient and clear. On the other hand, the habits and needs of the users are taken into account, in that the technique is directed to the users and not vice versa.

Fig. 1

ein Blockschaltbild der erfindungsgemäßen Vorrichtung zum Erzeugen einer Audiodarstellung;

Fig. 2

eine schematische Darstellung einer Benutzerschnittstelle für das in Fig. 1 gezeigte Konzept;

Fig. 3a

eine schematische Darstellung der Benutzerschnittstelle von Fig. 2 gemäß einem Ausführungsbeispiel der vorliegenden Erfindung;

Fig. 3b

eine schematische Darstellung der Benutzerschnittstelle von Fig. 2 gemäß einem anderen Ausführungsbeispiel der vorliegenden Erfindung;

Fig. 4

ein Blockschaltbild einer erfindungsgemäßen Vorrichtung gemäß einem bevorzugten Ausführungsbeispiel;

Fig. 5

eine zeitliche Darstellung der Audioszene mit verschiedenen Audioobjekten; und

Fig. 6

eine Gegenüberstellung einer 1:1-Umsetzung zwischen Objekt und Kanal und einer Objekt-Kanal-Zuweisung gemäß der vorliegenden Erfindung für die in Fig. 5 dargestellte Audioszene.

Preferred embodiments of the present invention will be explained below in detail with reference to the accompanying drawings. Show it:

Fig. 1

a block diagram of the inventive device for generating an audio presentation;

Fig. 2

a schematic representation of a user interface for in Fig. 1 concept shown;

Fig. 3a

a schematic representation of the user interface of Fig. 2 according to an embodiment of the present invention;

Fig. 3b

a schematic representation of the user interface of Fig. 2 according to another embodiment of the present invention;

Fig. 4

a block diagram of a device according to the invention according to a preferred embodiment;

Fig. 5

a temporal representation of the audio scene with various audio objects; and

Fig. 6

a comparison of a 1: 1 conversion between object and channel and an object-channel assignment according to the present invention for the in Fig. 5 illustrated audio scene.

Fig. 1 shows a block diagram of a device according to the invention for generating an audio presentation of an audio scene. The inventive apparatus comprises means 10 for providing an object-oriented description of the audio scene, wherein the object-oriented description of the audio scene comprises a plurality of audio objects, wherein an audio object is assigned at least an audio signal, a start time and an end time. The device according to the invention further comprises an audio processing device 12 for generating a plurality of loudspeaker signals LSi 14 which is channel-oriented and which generates the plurality of loudspeaker signals 14 from a plurality of input channels EKi. An imaging device 18 for mapping the object-oriented description of the audio scene onto the plurality of input channels 16 of the channel-oriented audio signal processing device 12 is located between the providing device 10 and the channel-oriented audio signal processing device, which is embodied, for example, as a WFS rendering unit the mapping means 18 is adapted to assign a first audio object to an input channel, such as EK1, and to assign a second audio object whose start time is after an end time of the first audio object to the same input channel, such as the input channel EK1, and a third one Audio object whose start time is after the start time of the first audio object and before the end time of the first audio object to assign another input channel of the plurality of input channels, such as the input channel EK2. The mapping device 18 is thus designed to assign temporally non-overlapping audio objects to the same input channel, and to assign overlapping audio objects to different parallel input channels.

In a preferred embodiment in which the channel oriented audio signal processing device 12 comprises a wave field synthesis rendering unit, the audio objects are further specified to be associated with a virtual position. This virtual position of an object may change during the lifetime of the object, which would correspond to the case where, for example, a rider approaches a scene center, such that the rider's gallop gets louder and louder and closer to the auditorium. In this case, an audio object includes not only the audio signal associated with that audio object and a start time and an end time, but additionally a position of the virtual source which may change over time and, optionally, other properties of the audio object such as ob It should have point source characteristics or whether it should emit a plane wave, which would correspond to a virtual position with infinite distance to the viewer. In the art, further properties for sound sources, ie for audio objects are known, depending on the features of the channel-oriented audio signal processing device 12 of Fig. 1 can be considered.

According to the invention, the structure of the device is hierarchical in that the channel-oriented audio signal processing means for receiving audio objects is not directly combined with the means for providing, but combined with the same via the mapping means. As a result, the entire audio scene is to be known and stored only in the means for providing, but the mapping device, and still less the channel-oriented audio signal processing device, must already have knowledge of the overall audio setting. Instead Both the mapping device 18 and the audio signal processing device 12 operate under the direction of the audio scene provided by the providing device 10.

In a preferred embodiment of the present invention, the in Fig. 1 shown device further provided with a user interface, as shown in Fig. 2 at 20 is shown. The user interface 20 is configured to have one user interface channel per input channel, and preferably one manipulator for each user interface channel. The user interface 20 is coupled via its user interface input 22 to the mapping device 18 to obtain the mapping information from the mapping device, since the assignment of the input channels EK1 to EKm is to be displayed by the user interface 20. On the output side, if the user interface 20 has the manipulator feature for each user interface channel, it is coupled to the device 10 for providing. In particular, the user interface 20 is configured to provide, via its user interface output 24 with regard to the original version, manipulated audio objects of the device 10 for providing, thus receiving a changed audio scene which is then returned to the mapping device 18 and, correspondingly to the input channels Channel-oriented audio signal processing device 12 is provided.

Depending on the implementation, the user interface 20 is designed as a user interface, as shown in FIG Fig. 3a that is, as a user interface that always represents only the current objects. Alternatively, the user interface 20 is configured to operate as in FIG Fig. 3b to be constructed, so that always all objects are displayed in an input channel. As well in Fig. 3a as well as in Fig. 3b a time line 30 is shown, which comprises the objects A, B, C in chronological order, wherein the object A includes a start time 31a and an end time 31b. Coincidentally, it falls in Fig. 3a the end time 31b of the first object A coincides with a start time of the second object B, which in turn has an end time 32b, which in turn coincides coincidentally with a start time of the third object C, which in turn has an end time 33b. The start times 32a and 33b correspond to the end times 31b and 32b and are in the Fig. 3a, 3b not shown for clarity.

At the in Fig. 3a shown mode, in which only current objects are displayed as a user interface channel is right in Fig. 3a a mixer channel icon 34 is shown, which includes a slider 35 and stylized buttons 36, over the properties of the audio signal of the object B or virtual positions, etc. can be changed. Once the timestamp in Fig. 3a 37, which reaches the end time 32b of the object B, the stylized channel representation 34 would not display the object B but the object C. The user interface in FIG Fig. 3a would then, if z. B. an object D would take place simultaneously to the object B, another channel, such as the input channel i + 1, represent. In the Fig. 3a The illustration shown provides the sound engineer with a simple overview of the number of parallel audio objects at a time, ie the number of active channels that are even displayed. Non-active input channels are used at the in Fig. 3a shown embodiment of the user interface 20 of Fig. 2 not displayed at all.

At the in Fig. 3b In the embodiment shown in which all objects in an input channel are displayed side by side, there is likewise no display of unoccupied input channels. Nevertheless, the input channel i, to which the channels assigned temporally in chronological order belong, is shown in triplicate, once as an object channel A, on another occasion as an object channel B and again as an object channel C according to the invention. It is preferred to use the channel, such as the input channel i for the object B (reference numeral 38 in FIG Fig. 3b ) z. B. highlight color or brightness, on the one hand to give the sound engineer a clear overview of which object is currently being fed to the respective channel i, and which objects z. B. sooner or later run on this channel, so that the sound engineer can already predictively in the future via the appropriate software or hardware controller manipulate the audio signal of an object through this channel controller or channel switch. The user interface 20 of Fig. 2 and in particular the expressions thereof in Fig. 3a and Fig. 3b are thus designed to provide a visual representation as desired for the "occupancy" of the input channels of the channel-oriented audio signal processing device generated by the imaging device 18.

Subsequently, reference will be made to Fig. 5 a simple example of the functionality of the imaging device 18 of Fig. 1 given. Fig. 5 shows an audio scene with different audio objects A, B, C, D, E, F and G. Thus it can be seen that the objects A, B, C and D overlap in time. In other words, these objects A, B, C and D are all active at a particular time 50. In contrast, the object E does not overlap with the_objects A, B. The object E overlaps only with the objects D and C, as can be seen at a time 52. Again overlapping the object F and the object D, as it is at a time 54 z. B. can be seen. The same applies to the objects F and G, the z. B. overlap at a time 56, while the object G does not overlap with the objects A, B, C, D and E.

A simple and in many ways detrimental channel assignment would be where in Fig. 5 shown example assign each audio object to an input channel so that the 1: 1 conversion left in the table in Fig. 6 would be obtained. A disadvantage of this concept is that many input channels are needed or that, when there are many audio objects, which is the case very quickly in a movie, the number of input channels of the wave field synthesis rendering unit is the number of processable virtual sources in one limited in the real film setting, which of course is not desirable because technology limits should not affect the creative potential. On the other hand, this 1: 1 conversion is very confusing, in that at some point each input channel typically receives an audio object, but when a particular audio scene is viewed, typically relatively few input channels are active, but the user can not easily determine this because he always has all the audio channels at a glance.

Moreover, this concept of 1: 1 assignment of audio objects to input channels of the audio processor means that in order to minimize or limit the number of audio objects, audio processing equipment having a very high number of input channels must be provided Immediately increase the computational complexity, the required computing power and the required storage capacity of the audio processing device to calculate the individual loudspeaker signals, which directly results in a higher price of such a system.

The inventive assignment object channel of in Fig. 5 as shown by the imaging device 18 according to the present invention is shown in FIG Fig. 6 shown in the right section of the table. Thus, the parallel audio objects A, B, C and D are sequentially assigned to the input channels EK1, EK2, EK3 and EK4. However, the object E no longer has to, as in the left half of Fig. 6 assigned to the input channel EK5, but can be assigned to a free channel, such as the input channel EK1 or, as indicated by the bracket, the input channel EK2. The same applies to the object F, which in principle can be assigned to all channels except the input channel EK4. The same applies to the object G, which can also be assigned to all channels except the channel to which the object F was previously allocated (in the example the input channel EK1).

In a preferred embodiment of the present invention, the imaging device 18 is designed to always occupy channels with the lowest possible atomic number, and to occupy always adjacent input channels EKi and EKi + 1, so that no holes. On the other hand, this "neighborhood feature" is not essential, as it does not matter to a user of the audio authoring system according to the present invention whether he is currently serving the first, seventh, or any other input channel of the audio processing device, as long as he is through the inventive user interface is enabled to manipulate precisely this channel, for example by a controller 35 or by buttons 36 a mixer channel display 34 of the current channel. Thus, the user interface channel i need not necessarily discuss the input channel i, but it can also be so far a channel assignment done, such that the user interface channel i z. B. the input channel EKm corresponds, while the user interface channel i + 1 corresponds to the input channel k etc.

This is avoided by the user interface channel re-mapping that there are channel holes, so that the sound engineer can always see immediately and clearly displayed the current user interface channels side by side.

Of course, the user interface concept of the present invention may also be applied to an existing hardware mixing console that includes actual hardware controls and hardware buttons that a master will manually operate to achieve optimal audio mixing. An advantage of the present invention is that even such a sound mixer typically very familiar and heart-growing hardware mixing console can also be used by z. B. by typically present on the mixing console indicators, such as LEDs are always the current channels for the sound engineer clearly marked.

The present invention is further flexible in that cases can also be dealt with where the wave field synthesis loudspeaker setup used for production is covered by the reproduction setup e.g. B. differs in a movie theater. Therefore, according to the invention, the audio content is encoded in a format that can be processed by various systems. This format is the audio scene, i. H. the object-oriented audio presentation and not the speaker signal presentation. In this respect, the treatment process is understood as an adaptation of the content to the reproduction system. According to the invention, not only a few master channels but an entire object-oriented scene description are processed in the wave field synthesis reproduction process. The scenes are processed for each reproduction. This is typically done in real time to adapt to the current situation. Typically, this adaptation takes into account the number of speakers and their positions, the characteristics of the reproduction system, such as the frequency response, the sound pressure level, etc., the room acoustics conditions or other image reproduction conditions.

A major difference in the wave field synthesis mix compared to the channel-based approach of current systems consists in the freely available positioning of the sound objects. In conventional reproduction systems based on stereophonic principles, the position of the sound sources is relatively coded. This is important for blending concepts that pertain to visual content, such as movies, because positioning of the sound sources relative to the image is attempted by a proper system setup.

In contrast, the wave field synthesis system requires absolute positions for the sound objects, which is given as additional information about the audio signal of an audio object to this audio object in addition to the start time and the end time of this audio object.

In the traditional channel-oriented approach, the basic idea was to reduce the number of tracks in multiple pre-mix runs. These pre-mix runs are organized into categories such as dialogue, music, sound, effects, etc. During the mixing process, all the required audio signals are fed into the mixing console and mixed simultaneously by different sound engineers. Each pre-mix reduces the number of tracks until only one track per replica speaker exists. These final tracks form the final master file (Final Master).

All relevant mixing tasks, such as Equalization, Dynamics, Positioning, etc., are performed on the mixing console or using special equipment.

The goal of re-engineering the post-production process is to minimize user training and integrate the integration of the new system of the invention into existing users' knowledge. In the wave field synthesis application of the present invention, all tracks or objects to be rendered at different positions will exist within the master file / distribution format, unlike conventional production facilities that are optimized to reduce the number of tracks during the production process. On the other hand, for practical reasons, it is necessary to give the re-recording engineer the opportunity to use the existing mixing consoles for wave field synthesis productions.

Thus, according to the invention, current mixing consoles are used for the conventional mixing tasks, the output of these mixing consoles then being introduced into the inventive system for producing an audio representation of an audio scene where the spatial mixing is performed. This means that the wave field synthesis authoring tool according to the present invention is implemented as a workstation having the ability to record the audio signals of the final mix and to convert them to the distribution format in another step. For this purpose, two aspects are considered according to the invention. The first is that all audio objects or tracks still exist in the final master. The second aspect is that the positioning is not performed in the mixing console. This means that the so-called authoring, so the Tonmeister post-processing is one of the last steps in the production chain. According to the invention, the wave field synthesis authoring system according to the present invention, ie the inventive device for generating an audio presentation, is implemented as a standalone workstation that can be integrated into different production environments by feeding audio outputs from the mixer into the system. As such, the mixer represents the user interface coupled to the device for generating the audio presentation of an audio scene.

The system according to the invention according to a preferred embodiment of the present invention is disclosed in Fig. 4 shown. Same reference numerals as in Fig. 1 or 2 indicate the same elements. The basic system design is based on the goal of modularity and the ability to integrate existing mixing consoles into the inventive wave field synthesis authoring system as user interfaces.

For this reason, in the audio processing device 12, a central controller 120 is formed, which communicates with other modules. This allows the use of alternatives for certain modules as long as they all use the same communication protocol. If that is in Fig. 4 When the system shown is considered a black box, one generally sees a number of inputs (from the provisioning means 10) and a number of outputs (loudspeaker signals 14) and the user interface 20. Integrated in this black box next to the user interface is the actual WFS Renderer 122, which performs the actual wave field synthesis calculation of the loudspeaker signals using various input information. Also provided is a space simulation module 124 configured to perform certain room simulations that are used to create room characteristics of a recording room or to manipulate room characteristics of a recording room.

Furthermore, an audio recording device 126 and a recording reproduction device (also 126) are provided. The device 126 is preferably provided with an external input. In this case, the entire audio signal is either already object-oriented or channel-oriented provided and fed. Then the audio signals do not come from the scene protocol, which then only perceives control tasks. The input audio data is then converted from the device 126, if necessary, into an object-oriented representation and then supplied internally to the imaging device 18, which then performs the object / channel mapping.

All audio connections between the modules are switchable by a matrix module 128 to connect corresponding channels to corresponding channels as required by the central controller 120. In a preferred embodiment, the user has the ability to feed 64 input channels of virtual source signals to the audio processor 12, thus having 64 input channels EK1-EKm in this embodiment. This allows existing consoles to be used as user interfaces for premixing the source virtual signals. Spatial mixing is then performed by the wave-field synthesis authoring system and, in particular, by the heart, WFS renderer 122.

The complete scene description is stored in the provisioning facility 10, also referred to as a scene log. By contrast, the main communication or the required data traffic is performed by the central controller 120. Changes in the scene description, such as may be achieved by the user interface 20, and in particular by a hardware mixing console 200 or a software GUI, that is, a graphical software user interface 202, are provided via a user interface controller 204 of the providing device 10 as a modified scene protocol fed. By providing a modified scene log, the entire logical structure of a scene is clearly represented.

For the realization of the object-oriented approach, the mapping device 18 assigns each switch object to a render channel (input channel) in which the object exists for a certain time. Usually, a number of objects exist in chronological order on a particular channel, as determined by the Fig. 3a, 3b and 6 has been shown. Although the authoring system of the present invention supports this object orientation, the wavefield synthesis renderer must retrieve the objects do not know yourself. It simply receives signals in the audio channels and a description of how these channels need to be processed. The scene protocol providing means, that is, the knowledge of the objects and the associated channels, may perform a transformation of the object-related metadata (eg, the source position) to channel-related metadata and transmit them to the WFS renderer 122. The communication between other modules is performed by special protocols in such a way that the other modules contain only necessary information, as indicated schematically by the block function protocols 129 in FIG Fig. 4 is shown.

The control module according to the invention further supports the hard disk storage of the scene description. It preferably distinguishes between two file formats. A file format is an author format where the audio data is stored as uncompressed PCM data. Further, session-related information, such as a grouping of audio objects, that is, sources, layer information, etc., is also used to be stored in a special file format based on XML.

The other type is the distribution file format. In this format, audio data can be stored in a compressed manner and there is no need to additionally store the session-related data. It should be noted that the audio objects still exist in this format and that the MPEG-4 standard can be used for distribution. According to the invention, it is preferable to always do the wave field synthesis processing in real time. This makes it possible that no pre-rendered audio information, so already finished speaker signals must be stored in any file format. This is a great advantage, since the loudspeaker signals can take up very considerable amounts of data, which not least due to the large number of speakers used in a wave field synthesis environment.

The one or more wave-field synthesis renderer modules 122 are typically supplied with source virtual signals and a channel-oriented scene description. A wave field synthesis renderer calculates according to the wave field synthesis theory, the driver signal for each speaker, so a speaker signal of the speaker signals 14 of Fig. 4 , The wave field synthesis renderer will also compute signals for Sobwoofer speakers, which are also needed to support the wave field synthesis system at low frequencies. Room simulation signals from the room simulation module 124 are rendered using a number (typically 8 to 12) of static plane waves. Based on this concept, it is possible to integrate different solutions for room simulation. Without using the space simulation module 124, the wave-field synthesis system already produces acceptable sound images with stable perception of the source direction for the listening area. However, there are certain shortcomings in the perception of the depth of the sources, since usually no early spatial reflections or reverberation phenomena are added to the source signals. According to the invention, it is preferred that a space simulation model is used that reproduces wall reflections modeled, for example, such that a mirror source model is used to generate the early reflections. These mirror sources, in turn, may be treated as audio objects of the scene protocol, or may actually be added by the audio processor itself. The record / playback tools 126 are a useful complement. Scaling objects that are ready to be blended in a conventional manner during premixing, so that only spatial mixing needs to be performed, can be converted from the conventional mixer to an audio object reproducing apparatus be fed. Further, it is preferred to also have an audio recording module which records the output channels of the mixer in a time code controlled manner and stores the audio data on the playback module. The rendering module will receive a start time code to play a particular audio object in conjunction with a respective output channel supplied to the player 126 from the imaging device 18. The recording / playback device can independently start and stop the playback of individual audio objects, depending on the description of the start time and the stop time point associated with an audio object. Once the merge procedure is completed, the audio content may be taken from the player module and exported to the distribution file format. The distribution file format thus contains a finished scene log of a finished mixed scene. The goal of the user interface concept according to the invention is to implement a hierarchical structure which is adapted to the tasks of the cinema mixing process. Here, an audio object is considered as a source that exists as a representation of the individual audio object for a given time. A start time and a stop / end time are typical for a source, ie for an audio object. The source or audio object needs resources of the system during the time that the object or source "lives".

Preferably, each sound source includes metadata in addition to the start time and the stop time. These metadata are "type" (a level wave or point source at a given time), "direction," "volume," "mute," and "flags" for directional loudness and directional delay. All of this metadata can be used automatically.

Furthermore, it is preferred that despite the object-oriented approach, the authoring system according to the invention also serves the conventional channel concept in that z. B. objects, who are "alive" throughout the film or generally over the entire scene, also get their own channel. This means that these objects are in principle simple channels in 1: 1 implementation as they are based on Fig. 6 is presented.

In a preferred embodiment of the present invention, at least two objects may be grouped. For each group it is possible to choose which parameters should be grouped and how they should be calculated using the master of the group. Groups of sound sources exist for a given time, which is defined by the start time and the end time of the members.

An example of using groups is to use them for standard virtual surround setups. These could be used for fading out of a scene or zooming in on a scene. Alternatively, the grouping can also be used to integrate surround reverb effects and record into a WFS mix.

Further, it is preferable to form another logical entity, namely the layer or layer. In order to structure a mixture or a scene, in a preferred embodiment of the present invention groups and sources are arranged in different layers. Using layers, pre-dubs can be simulated in the audio workstation. Layers can also be used to change display attributes during the authoring process, for example to show or hide different parts of the current mix item.

A scene consists of all previously discussed components for a given period of time. This period could be a film spool or z. B. be the entire movie, or else only z. B. a movie section of certain duration, such as five minutes. The scene again consists of a number of layers, groups, and sources that belong to the scene.

Preferably, the complete user interface 20 should include both a graphics software part and a hardware part to allow haptic control. However, although this is preferred, the user interface could also be fully implemented as a software module for cost reasons.

A design concept for the graphical system is used, which is based on so-called "spaces". There are a small number of different spaces in the user interface. Each space is a special editing environment that presents the project from a different approach, with all the tools needed for a space. Therefore, you no longer have to pay attention to different windows. All tools needed for an environment are in the appropriate space.

To give the sound engineer an overview of all audio signals at a given time, the already on the basis of Fig. 3a and 3b described adaptive mixing space used. It can be compared to a conventional mixer that only displays the active channels. The adaptive mixing space also presents audio object information instead of pure channel information. As has been shown, these objects are represented by the imaging device 18 of FIG Fig. 1 Input channels assigned to the WFS rendering unit. In addition to the adaptive mixing space there is also the so-called timeline space, which provides an overview of all input channels. Each channel is displayed with its corresponding objects. The user has the option of object-to-channel assignment although for simplicity, automatic channel assignment is preferred.

Another space is the positioning and editing space, which shows the scene in a three-dimensional view. This space is to enable the user to record or edit movements of the source objects. Movements can be generated using, for example, a joystick or other input / display devices known for graphical user interfaces.

Finally, there is a room space that contains the room simulation module 124 of FIG Fig. 4 supports to also provide a room editing option. Each room is described by a particular set of parameters stored in a Room Presets library. Depending on the room model, different types of parameter sets as well as different graphical user interfaces can be used.

Depending on the circumstances, the inventive method for generating an audio representation in hardware or in software can be implemented. The implementation may be on a digital storage medium, in particular a floppy disk or CD with electronically readable control signals, which may interact with a programmed computer system so as to carry out the inventive method. The invention thus also consists in a computer program product with a program code stored on a machine-readable carrier for carrying out the method according to the invention, when the computer program product runs on a computer. In other words, the invention is thus also a computer program with a program code for carrying out the method when the computer program runs on a computer.

Claims (16)

1. Apparatus for generating, storing, or editing an audio representation of an audio scene, comprising:

   audio processing means (12) for generating a plurality of speaker signals from a plurality of input channels (EK1, EK2, ..., Ekm) (16);

   means (10) for providing an object-oriented description of the audio scene, wherein the object-oriented description of the audio scene includes a plurality of audio objects, wherein an audio object is associated with an audio signal, a starting time instant, and an end time instant; and

   mapping means (18) for mapping the object-oriented description of the audio scene to the plurality of input channels of the audio processor, wherein the mapping means is configured to assign a first audio object to an input channel, and to assign a second audio object whose starting time instant lies after the end time instant of the first audio object to the same input channel, and to assign a third audio object whose starting time instant lies after the starting time instant of the first audio object and before the end time instant of the first audio object to another of the plurality of input channels.
2. Apparatus of claim 1, wherein the audio processing means (12) includes wave-field synthesizing means (122) configured to calculate the plurality of speaker signals for the speakers, knowing positions of a plurality of speakers.
3. Apparatus of claim 1 or 2, wherein the audio object is further associated with a virtual position, and wherein the audio processing means (12) is configured to take the virtual positions of the audio objects into account in generating the plurality of speaker signals.
4. Apparatus of any one of the preceding claims, wherein the audio processing means (12) is coupled to the means (10) for providing exclusively via the mapping means (18), to receive audio object data to be processed.
5. Apparatus of one of the preceding claims, wherein a number of input channels of the audio processing means is predetermined and is smaller than an allowed number of audio objects in the audio scene, wherein at least two audio objects are present that do not overlap temporally.
6. Apparatus of one of the preceding claims, further comprising a user interface (20), wherein the user interface comprises a number of separate user interface channels, wherein a user interface channel is associated with an input channel of the audio processor, and wherein the user interface (20) is coupled to the mapping means (80) to identify the audio object just assigned to the user interface channel at a time instant.
7. Apparatus of claim 6, wherein the user interface (20) is configured to identify user interface channels associated with input channels of the audio processing means to which currently one audio object is assigned.
8. Apparatus of claim 7, wherein the user interface is configured as hardware mixing console having hardware manipulating means for each user interface channel, and wherein each hardware manipulating means is associated with an indicator to identify a currently active user interface channel.
9. Apparatus of claim 7, wherein the user interface comprises a graphical user interface configured to display, on an electrical display device, only the user interface channels associated with an input channel of the audio processing means to which currently one audio object is assigned.
10. Apparatus of one of claims 6 to 9, wherein the user interface (20) further comprises a manipulating means for a user interface channel, which is configured to manipulate an audio object assigned to the input channel of the audio processing means (12), which corresponds to the user interface channel, wherein the user interface is coupled to the means (10) for providing to replace an audio object by a manipulated version thereof, and wherein the mapping means (18) is configured to assign, instead of the audio object, the manipulated version thereof to an input channel of the audio processing means (12).
11. Apparatus of claim 10, wherein the manipulating means is configured to change position, type, or audio signal of an audio object.
12. Apparatus of one of claims 6 to 9, wherein the user interface is configured to illustrate a temporal occupation for a user interface channel, wherein the temporal occupation represents a temporal sequence of the audio objects assigned to a user interface channel, and wherein the user interface is further configured to mark a current time instant (37) in the temporal occupation.
13. The apparatus of claim 12, wherein the user interface (20) is configured to illustrate the temporal occupation as time axis, which comprises the assigned audio objects proportional to their length as well as an indicator (37) moving with time proceeding.
14. Apparatus of one of the preceding claims,
    wherein the means (10) for providing is configured to permit grouping of audio objects such that the audio objects that are grouped are marked by group information with regard to their group membership, and
    wherein the mapping means (18) is configured to preserve the group information so that manipulation of a group property affects all members of the group, independently of the fact which input channel of the audio processing means the audio objects of the group are associated with.
15. Method of generating, storing, or editing an audio representation of an audio scene, comprising:

    generating (12) a plurality of speaker signals from a plurality of input channels (EK1, EK2, ..., Ekm) (16);

    providing (10) an object-oriented description of the audio scene, wherein the object-oriented description of the audio scene includes a plurality of audio objects, wherein an audio object is associated with an audio signal, a starting time instant, and an end time instant; and

    mapping (18) the object-oriented description of the audio scene to the plurality of input channels of the audio processing means by assigning a first audio object to an input channel, and by assigning a second audio object whose starting time instant lies after the end time instant of the first audio object to the same input channel, and by assigning a third audio object whose starting time instant lies after the starting time instant of the first audio object and before the end time instant of the first audio object to another of the plurality of input channels.
16. Computer program with a program code for performing the method of claim 15, when the program is executed on a computer.

Images