WO2026018859A1 - Information processing method, information processing system, and program - Google Patents

Abstract

Provided is an information processing method comprising: a step for acquiring, in a first terminal, first sound information; a step (S102) for converting, in the first terminal, the first sound information into second sound information for generating a representative sound arriving at a reference position from a representative point which is set in a three-dimensional sound field by using an acoustic signal; a step (S105) for detecting, in a second terminal, the position of a user or the head direction of the user in the three-dimensional sound field; a step for calculating, in the second terminal, a position of a playback representative point corresponding to the position of the representative point on the basis of the detected position of the user or the head direction of the user and the reference position; and a step (S106) for generating, in the second terminal, an output sound signal by using the received second sound information and a head transfer function corresponding to the calculated position of the playback representative point.

Description

Information processing method, information processing system, and program

This disclosure relates to an information processing method, an information processing system, and a program.

Conventionally, there is known technology related to sound reproduction that allows a user to perceive three-dimensional sound in a virtual three-dimensional space (see, for example, Patent Document 1). Furthermore, in order to make a user perceive sound as if it is coming from a sound source object to the user in such a three-dimensional space, processing is required to generate output sound information from the original sound information. In particular, reproducing three-dimensional sound in response to the user's body movements in a virtual space requires a huge amount of processing. Advances in computer graphics (CG) have made it relatively easy to create visually complex virtual environments, and technology to realize corresponding auditory information has become important. In addition, when processing from sound information to generate output sound information is performed in advance, a large memory area is required to store the pre-calculated processing results. Furthermore, transmitting such large amounts of processed data may require a wide communication bandwidth.

In order to create a more realistic sound environment, the number of objects that emit sound in the virtual three-dimensional space increases, and secondary sounds based on acoustic effects such as reflected sound, diffracted sound, and reverberation also increase. Furthermore, these secondary sounds must be appropriately modified in response to the user's movements, requiring a large amount of processing. Therefore, with the aim of reducing this large amount of processing, a conversion technique known as panning processing (or simply panning) is known, which represents sounds in a three-dimensional space using sounds from several representative points that are set in advance within the three-dimensional space.

Japanese Patent Application Laid-Open No. 2020-18620

However, conversion processes such as panning can result in a degradation of sound quality as a trade-off for reducing the amount of processing. Therefore, the purpose of this disclosure is to provide an information processing method for appropriately performing conversion processes.

An information processing method according to one aspect of the present disclosure is an information processing method executed by a plurality of information processing terminals, comprising the steps of: acquiring, at a first terminal, one of the plurality of information processing terminals, first sound information including an acoustic signal and information about the position of a sound source object within a three-dimensional sound field; and converting, at the first terminal, the first sound information into second sound information for generating, using the acoustic signal, a representative sound arriving at a reference position from a representative point set within the three-dimensional sound field. the first terminal transmitting the second sound information to a second terminal that is another information processing terminal among the plurality of information processing terminals; the second terminal detecting the user's position or head direction in the three-dimensional sound field; the second terminal calculating the position of a reproduction representative point corresponding to the position of the representative point based on the detected user's position or head direction and the reference position; and the second terminal generating an output sound signal using a head-related transfer function corresponding to the calculated position of the reproduction representative point and the received second sound information.

Furthermore, an information processing system according to one aspect of the present disclosure is an information processing system including a first terminal and a second terminal, wherein the first terminal comprises an acquisition unit that acquires sound information including an acoustic signal and information about the position of a sound source object within a three-dimensional sound field, a conversion unit that converts the first sound information into second sound information using the acoustic signal to generate a representative sound that arrives at a reference position from a representative point set within the three-dimensional sound field, and a transmission unit that transmits the second sound information to the second terminal, and the second terminal comprises a detector that detects the position or head direction of a user within the three-dimensional sound field, a calculation unit that calculates the position of a reproduction representative point corresponding to the position of the representative point based on the detected position or head direction of the user and the reference position, and an output unit that outputs an output sound signal using a head-related transfer function corresponding to the calculated position of the reproduction representative point and the received second sound information.

Furthermore, one aspect of the present disclosure can also be realized as a program for causing a computer to execute the information processing method described above.

Note that these comprehensive or specific aspects may be realized as a system, device, method, integrated circuit, computer program, or non-transitory recording medium such as a computer-readable CD-ROM, or as any combination of a system, device, method, integrated circuit, computer program, and recording medium.

This disclosure makes it possible to perform conversion processing appropriately.

FIG. 1 is a schematic diagram showing a use example of a sound reproduction system according to an embodiment. FIG. 2 is a block diagram showing the functional configuration of the sound reproduction system according to the embodiment. FIG. 3 is a diagram illustrating an example of an audio signal according to the embodiment. FIG. 4 is a block diagram illustrating a functional configuration of an acquisition unit according to the embodiment. FIG. 5 is a block diagram illustrating a functional configuration of the output sound generating unit according to the embodiment. FIG. 6 is a diagram for explaining another example of the sound reproducing system according to the embodiment. FIG. 7 is a diagram for explaining another example of the sound reproducing system according to the embodiment. FIG. 8 is a diagram for explaining another example of the sound reproducing system according to the embodiment. FIG. 9 is a diagram for explaining another example of the sound reproduction system according to the embodiment. FIG. 10 is a diagram for explaining another example of the sound reproducing system according to the embodiment. FIG. 11 is a diagram illustrating another example of the sound reproduction system according to the embodiment. FIG. 12 is a diagram illustrating another example of the sound reproduction system according to the embodiment. FIG. 13 is a diagram illustrating another example of the sound reproduction system according to the embodiment. FIG. 14 is a diagram illustrating another example of the sound reproduction system according to the embodiment. FIG. 15 is a diagram illustrating another example of the sound reproduction system according to the embodiment. FIG. 16 is a diagram illustrating another example of the sound reproduction system according to the embodiment. FIG. 17A is a flowchart illustrating an example of the operation of the information processing device according to the embodiment. FIG. 17B is a flowchart illustrating an example of the operation of the information processing device according to the embodiment. FIG. 18 is a flowchart of the audio reproduction process according to the embodiment. FIG. 19 is a diagram for explaining synthesis of head-related transfer functions in the audio reproduction process according to the embodiment. FIG. 20 is a diagram for explaining the arrangement of the representative direction according to the embodiment. FIG. 21 is a diagram illustrating an example of time shift value calculation according to the embodiment. FIG. 22 is a diagram illustrating an example of time shift value calculation according to the embodiment. FIG. 23 is a diagram illustrating an example of gain value calculation according to the embodiment. FIG. 24 is a diagram showing the results of gain value calculation according to the embodiment. FIG. 25 is a diagram showing the results of gain value calculation according to the embodiment. FIG. 26 is a diagram for verifying the results of the time shift value calculation according to the embodiment. FIG. 27 is a diagram for verifying the results of gain value calculation according to the embodiment. FIG. 28 is a diagram showing the results of a localization experiment for verifying the results of the gain value calculation according to the embodiment. FIG. 29 is a diagram showing the results of a localization experiment for verifying the results of gain value calculation according to the embodiment. FIG. 30 is a diagram showing the results of a localization experiment for verifying the results of gain value calculation according to the embodiment. FIG. 31 is a diagram showing the results of a localization experiment for verifying the results of gain value calculation according to the embodiment. FIG. 32 is a diagram showing set fixed gain values according to the embodiment. FIG. 33 is a diagram showing the results of a localization experiment for verifying the results of gain value calculation according to the embodiment. FIG. 34 is a diagram showing the results of a localization experiment for verifying the results of gain value calculation according to the embodiment.

(Knowledge that formed the basis of disclosure)
Conventionally, a technology related to sound reproduction that allows a user to perceive stereoscopic sound in a virtual three-dimensional space (hereinafter sometimes referred to as a three-dimensional sound field) has been known (see, for example, Patent Document 1). Using this technology, a user can perceive sound as if a sound source object exists at a predetermined position in the virtual space and the sound is coming from that direction. Localizing a sound image at a predetermined position in the virtual three-dimensional space in this way requires, for example, computational processing to generate a binaural sound arrival time difference and a binaural sound level difference (or sound pressure difference) that are perceived as stereoscopic sound for a sound signal emitted by a sound source object (also referred to as a sound emitted from the sound source object or a reproduced sound). This computational processing is performed by applying a stereophonic filter. A stereophonic filter is an information processing filter that, when an output sound signal obtained by applying the filter to original sound information is reproduced, causes the position (such as the direction and distance of the sound), the size of the sound source, the size of the space, and the like to be perceived with a three-dimensional effect.

One example of the computational process for applying such a stereophonic filter is the process of convolving a head-related transfer function (HRTF) with the target sound signal to make the sound appear to be coming from a specific direction. By performing this HRTF convolution process at a sufficiently fine angle relative to the direction from which the reproduced sound is coming from the sound source object's position to the user's position, the sense of realism experienced by the user is improved.

In addition, in recent years, there has been active development of technology related to virtual reality (VR). Virtual reality focuses on appropriately changing the position of a sound source object in a virtual three-dimensional space in response to the user's movements, allowing the user to experience the sensation of moving within the virtual space. To achieve this, it is necessary to move the localized position of the sound image within the virtual space relative to the user's movements. This processing has traditionally been performed by applying a stereophonic filter, such as the head-related transfer function described above, to the original sound information. However, when a user moves within a three-dimensional space, the sound transmission path changes every time the positional relationship between the sound source object and the user changes due to factors such as sound reverberation and interference. This requires determining the sound transmission path from the sound source object based on the positional relationship between the sound source object and the user each time, and convolving the transfer function to take into account sound reverberation and interference. However, this type of information processing requires an enormous amount of processing power, and without a large-scale processing device, it may not be possible to improve the sense of realism.

In order to reduce this ever-increasing amount of processing, attempts are being made to apply panning to the reproduced sound and reduce the amount of head-related transfer function convolution. Specifically, rather than convolving the head-related transfer function into the reproduced sound for each of the many sound source objects in three-dimensional space, the reproduced sound from the sound source objects is re-expressed using sounds from several representative points (representative sounds) set in advance in three-dimensional space. Then, simply by convolving the head-related transfer function from the representative points to the user's position into the representative sounds, it is possible to allow the user to perceive three-dimensional sound that is comparable to that of the original. If there are fewer representative points than the number of original sound source objects, naturally there will be fewer targets for head-related transfer function convolution, which is advantageous in terms of processing volume.

A more specific outline of this disclosure is as follows:

An information processing method according to a first aspect of the present disclosure is an information processing method executed by a plurality of information processing terminals, and includes the steps of: acquiring, at a first terminal which is one of the plurality of information processing terminals, first sound information including an acoustic signal and information on the position of a sound source object within a three-dimensional sound field; the first sound information is information for causing the sound source object within the three-dimensional sound field to emit a playback sound using the acoustic signal; converting, at the first terminal, the first sound information into second sound information for generating a representative sound that arrives at a reference position from a representative point set within the three-dimensional sound field using the acoustic signal; transmitting, at the first terminal, the second sound information to a second terminal which is another of the plurality of information processing terminals; detecting, at the second terminal, the position of a user or the direction of the head within the three-dimensional sound field; calculating, at the second terminal, the position of a playback representative point corresponding to the position of the representative point based on the detected position or direction of the user's head and the reference position; and generating, at the second terminal, an output sound signal using a head-related transfer function corresponding to the calculated position of the playback representative point and the received second sound information.

With this information processing method, when the second terminal is worn by the user, the first terminal can be provided separately from the second terminal worn by the user. Therefore, the first terminal, which is not restricted by the need for a user to wear it and can have relatively high processing performance compared to the second terminal, can perform panning processing on the audio signal, including the movement of the sound source object specified in the content, which requires relatively high processing resources. Furthermore, panning processing can summarize sounds from fewer representative directions, compressing the amount of information and reducing communication bandwidth restrictions when transmitting and receiving data between the first and second terminals. Furthermore, when the second terminal is worn by the user, the second terminal worn by the user can easily detect the position and orientation of the user's head and use this information to output an output sound signal from the panned audio signal, thereby instantly updating the direction of sound arrival in response to head movement. Furthermore, when outputting the output sound signal, it is only necessary to convolve head-related transfer functions for a relatively small number of representative directions (i.e., a few directions after panning processing), thereby reducing the processing resources required for the second terminal. In this way, according to the present disclosure, conversion processing can be performed appropriately to obtain multiple benefits.

Furthermore, the information processing method according to the second aspect is the information processing method according to the first aspect, in which the converting step applies time shift adjustment and gain adjustment to the reproduced sound to convert it into a representative sound.

This allows time shift adjustment and gain adjustment to be applied to the playback sound, converting it into a representative sound that is less likely to cause discomfort.

Furthermore, the information processing method according to the third aspect is the information processing method according to the second aspect, in which the adjustment amount of the time shift adjustment is set to 0 at the representative point.

This allows the time shift adjustment amount to be set so that it becomes 0 at the representative point, which is essentially the correct value.

Furthermore, the information processing method according to the fourth aspect is the information processing method according to the second or third aspect, in which the adjustment amount for the time shift adjustment at the predetermined position is set using the adjustment amount for a position closer to the representative point than the predetermined position.

By doing this, when setting the amount of time shift adjustment at a predetermined position, the adjustment amount for a position close to the representative point as viewed from the predetermined position is used, making it possible to set an adjustment amount that is closer to the adjustment amount for a position close to the representative point, for example. As a result, it is less likely that jumps will occur, as the amount of time shift adjustment changes suddenly.

Furthermore, the information processing method according to the fifth aspect is an information processing method according to any one of the second to fourth aspects, in which the amount of gain adjustment at a predetermined position is calculated by using two of three adjacent representative points surrounding the predetermined position to calculate the amount of adjustment for each of the two representative points that minimizes the error, and then fixing the ratio of the calculated adjustment amounts and calculating the amount of adjustment for the remaining representative point.

This allows two representative points to be used to calculate the adjustment amount for each of the two representative points that minimizes the error, and then the ratio of the calculated adjustment amounts can be fixed and the adjustment amount for the remaining representative point can be calculated sequentially. This makes it possible to adjust the influence of the two representative points initially used to calculate the adjustment amount and the remaining representative point on the calculation of the adjustment amount for gain adjustment at a specified position.

Furthermore, the information processing method according to the sixth aspect is an information processing method according to any one of the second to fifth aspects, in which the amount of gain adjustment at a predetermined position is calculated by calculating the amount of adjustment for each of two representative points located horizontally that minimize the error for a horizontal position obtained by removing the vertical component from the predetermined position, out of three adjacent representative points surrounding the predetermined position, and then fixing the ratio of the calculated adjustment amounts and calculating the amount of adjustment for the remaining representative point.

With this, after calculating the amount of gain adjustment for a horizontal position at two representative points located in the horizontal direction, the ratio of the calculated adjustment amounts can be fixed and the adjustment amount for the remaining representative point can be calculated. The sense of sound localization in the horizontal direction is relatively accurate. Therefore, by first calculating the adjustment amount for the horizontal position and then fixing the ratio of the adjustment amounts, the influence of the two horizontal representative points on the calculation of the amount of gain adjustment for a specified position is made greater than the influence of the remaining representative point on the calculation of the amount of gain adjustment for a specified position, thereby increasing the accuracy of the sense of localization in the horizontal direction and making it possible to output an output sound signal in which the sound image is more appropriately localized.

Furthermore, the information processing method according to the seventh aspect is an information processing method according to any one of the second to sixth aspects, in which the amount of time shift adjustment is corrected based on the expected value of the error so as to reduce the error from the calculated correct value.

This allows the amount of time shift adjustment to be set so as to reduce the error from the calculated correct value.

Furthermore, the information processing method according to the eighth aspect is an information processing method according to any one of the second to seventh aspects, in which the amount of gain adjustment at a predetermined position is corrected based on the expected value of the error so as to reduce the error from the calculated correct value.

This allows the gain adjustment amount at a specified position to be set so as to reduce the error from the calculated correct value.

Furthermore, the information processing method according to the ninth aspect is an information processing method according to any one of the second to sixth aspects, in which the adjustment amount for gain adjustment at a predetermined position is set using two or more of the adjustment amounts at multiple virtual representative points corresponding to multiple directions within a predetermined angle range in the horizontal direction centered on the direction of the representative point of the starting point, with the representative point for the predetermined position as the starting point.

As a result, two or more adjustment amounts can be used as the amount of gain adjustment at a predetermined position at multiple virtual representative points corresponding to multiple directions within a predetermined angle range in the horizontal direction centered on the direction of the representative point of origin. This makes it possible to set the amount of gain adjustment at a predetermined position so that two or more adjustment amounts all have an effect.

Furthermore, an information processing method according to a tenth aspect is the information processing method according to the ninth aspect, wherein the adjustment amount for gain adjustment at a predetermined position is set using all of the adjustment amounts at multiple virtual representative points corresponding to multiple directions within a predetermined angle range in the horizontal direction centered on the direction of the representative point of the starting point, with the representative point for the predetermined position as the starting point.

This allows the adjustment amounts for gain adjustment at a predetermined position to be used as the adjustment amounts for gain adjustment at multiple virtual representative points corresponding to multiple directions within a predetermined angle range in the horizontal direction centered on the direction of the representative point of origin. This makes it possible to set the adjustment amount for gain adjustment at a predetermined position so that all adjustment amounts have an effect.

Furthermore, an information processing method according to an eleventh aspect is the information processing method according to the ninth aspect, in which the amount of gain adjustment at a predetermined position is calculated by averaging the adjustment amounts at multiple virtual representative points.

This allows the gain adjustment amount for a given position to be calculated by averaging two or more adjustment amounts.

Furthermore, the information processing method according to the twelfth aspect is the information processing method according to the ninth aspect, in which the amount of gain adjustment at a predetermined position is calculated by taking a weighted average of the adjustment amounts at multiple virtual representative points based on the angle difference between the direction of each virtual representative point and the original direction corresponding to that virtual representative point.

This allows the gain adjustment amount for a specified position to be calculated by taking a weighted average of two or more adjustment amounts.

Furthermore, the information processing method according to the thirteenth aspect is an information processing method according to any one of the second to twelfth aspects, in which the amount of gain adjustment at a predetermined position is set to a predetermined value that is independent of the direction of the predetermined position in the horizontal plane, by adjusting the amount of adjustment at an elevation angle representative point that includes a vertical component, among three adjacent representative points that surround the predetermined position.

This allows the amount of gain adjustment for a given position to be set by setting the amount of adjustment for the elevation angle representative point to a given value that is independent of the direction of the given position in the horizontal plane.

Furthermore, the information processing method according to the fourteenth aspect is the information processing method according to the thirteenth aspect, in which the predetermined value is a value that changes depending on the direction of the predetermined position in a vertical plane.

This allows the amount of adjustment for the elevation angle representative point to be set to a predetermined value that does not depend on the direction of the specified position in the horizontal plane, but changes depending on the direction of the specified position in the vertical plane, making it possible to set the amount of gain adjustment for the specified position.

Furthermore, the information processing method according to the fifteenth aspect is the information processing method according to the fourteenth aspect, in which the predetermined value is a value that gradually increases between 0 and 1 when the direction of the predetermined position in the vertical plane is from 0° to 90°.

This allows the amount of adjustment for the elevation angle representative point to be set to a predetermined value that gradually increases between 0 and 1 when the direction of the predetermined position in the vertical plane is between 0° and 90°, regardless of the direction of the predetermined position in the horizontal plane, thereby making it possible to set the amount of gain adjustment for the predetermined position.

Furthermore, an information processing system according to a sixteenth aspect is an information processing system including a first terminal and a second terminal, wherein the first terminal comprises an acquisition unit that acquires sound information including an acoustic signal and information about the position of a sound source object within a three-dimensional sound field, a conversion unit that converts the first sound information into second sound information using the acoustic signal to generate a representative sound that arrives at a reference position from a representative point set within the three-dimensional sound field, and a transmission unit that transmits the second sound information to the second terminal, and the second terminal comprises a detector that detects the position of a user's position or head direction within the three-dimensional sound field, a calculation unit that calculates the position of a reproduction representative point corresponding to the position of the representative point based on the detected user's position or head direction and the reference position, and an output unit in the second terminal that outputs an output sound signal using a head-related transfer function corresponding to the calculated position of the reproduction representative point and the received second sound information.

This can achieve the same effects as the information processing method described above.

Furthermore, a program according to a seventeenth aspect is a program for causing a computer to execute the information processing method described above.

This allows the same effects as the information processing method described above to be achieved using a computer.

Furthermore, these comprehensive or specific aspects may be realized as a system, device, method, integrated circuit, computer program, or non-transitory recording medium such as a computer-readable CD-ROM, or as any combination of a system, device, method, integrated circuit, computer program, and recording medium.

The following describes the embodiments in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, of the components in the following embodiments, components that are not recited in independent claims are described as optional components. Note that each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical components are assigned the same reference numerals, and duplicate explanations may be omitted or simplified.

Furthermore, in the following description, ordinal numbers such as first, second, and third may be attached to elements. These ordinal numbers are attached to elements in order to identify them, and do not necessarily correspond to any meaningful order. These ordinal numbers may be rearranged, newly added, or removed as appropriate.

Furthermore, in the following description, we may refer to acoustic signals contained in sound information, but these may also be expressed as voice signals or sound signals. In other words, in this disclosure, the term "acoustic signal" has the same meaning as the term "voice signal" or "sound signal."

(Embodiment)
[overview]
First, an overview of the sound reproduction system according to the embodiment will be described. Fig. 1 is a schematic diagram showing a use example of the sound reproduction system according to the embodiment. Fig. 1 shows a user 99 using the sound reproduction system 100.

The sound reproduction system 100 shown in Figure 1 is used simultaneously with a 3D video reproduction device 300, for example. By simultaneously viewing 3D images and 3D sound, the images enhance the auditory realism, and the sound enhances the visual realism, allowing the viewer to experience the feeling of being at the scene where the images and sounds were captured. For example, when an image (moving image) of people having a conversation is displayed, it is known that even if the position of the sound image (sound source object) of the conversation sound is not aligned with the person's mouth, the user 99 will perceive it as the conversation sound coming from the person's mouth. In this way, the position of the sound image can be corrected using visual information, and the sense of realism can be enhanced by combining the image and sound.

The 3D video reproduction device 300 is an image display device worn on the head of the user 99. Therefore, the 3D video reproduction device 300 moves integrally with the head of the user 99. For example, as shown in the figure, the 3D video reproduction device 300 is a glasses-type device that is supported by the ears and nose of the user 99.

The 3D video playback device 300 changes the image displayed in response to the movement of the user 99's head, making the user 99 perceive the movement of their head within the three-dimensional image space. In other words, when an object in the three-dimensional image space is located directly in front of the user 99, if the user 99 turns to the right, the object moves to the user 99's left, and if the user 99 turns to the left, the object moves to the user 99's right. In this way, the 3D video playback device 300 moves the three-dimensional image space in the opposite direction to the user 99's movement.

The 3D video playback device 300 displays two images with a parallax difference to each of the user's 99 eyes. Based on the parallax difference between the displayed images, the user 99 can perceive the three-dimensional position of objects on the images. Note that when the user 99 uses the audio playback system 100 with their eyes closed, for example when using it to play healing sounds for sleep induction, the 3D video playback device 300 does not need to be used at the same time. In other words, the 3D video playback device 300 is not an essential component of the present disclosure. In addition to dedicated video display devices, the 3D video playback device 300 may also be a general-purpose mobile device owned by the user 99, such as a smartphone or tablet device.

Such general-purpose mobile terminals are equipped with a display for displaying images, as well as various sensors for detecting the terminal's position and movement. They also have a processor for information processing, and can be connected to a network to send and receive information to and from server devices such as cloud servers. In other words, the 3D video playback device 300 and sound playback system 100 can be realized by combining a smartphone with general-purpose headphones or other devices without information processing capabilities.

As in this example, the 3D video playback device 300 and audio playback system 100 may be realized by appropriately arranging the head movement detection function, video presentation function, video information processing function for presentation, sound presentation function, and audio information processing function for presentation in one or more devices. If the 3D video playback device 300 is not required, it is sufficient to appropriately arrange the head movement detection function, audio presentation function, and audio information processing function for presentation in one or more devices. For example, the audio playback system 100 can be realized by a processing device such as a computer or smartphone that has the audio information processing function for presentation, and headphones or the like that have the head movement detection function and the audio presentation function.

The sound reproduction system 100 is a sound presentation device worn on the head of the user 99. Therefore, the sound reproduction system 100 moves integrally with the head of the user 99. For example, the sound reproduction system 100 in this embodiment is a so-called over-ear headphone type device. Note that there are no particular limitations on the form of the sound reproduction system 100, and it may be, for example, two earplug-type devices worn independently on the left and right ears of the user 99.

The sound reproduction system 100 changes the sound presented in accordance with the movement of the user's 99's head, allowing the user 99 to perceive the user as if they were moving their head within a three-dimensional sound field. For this reason, as described above, the sound reproduction system 100 moves the three-dimensional sound field in the opposite direction to the movement of the user 99.

Here, when the user 99 moves within the three-dimensional sound field, the position of the sound source object changes relative to the user 99's position within the three-dimensional sound field. As a result, each time the user 99 moves, it is necessary to perform calculations based on the positions of the sound source object and the user 99 to generate an output sound signal for playback. Since such processing typically requires a huge amount of processing, in this disclosure, a panning process is applied as one type of conversion processing to reduce the amount of processing, and the reproduced sound is represented by a representative sound from a representative point. As a result, it is possible for the user 99 to perceive the reproduced sound from the sound source object simply by convolving a head-related transfer function with the representative sound. In the following, in this embodiment, a case will be described in which panning processing is used as an example of conversion processing, but the conversion processing is not limited to panning processing, and any conversion processing can be applied as long as it is expected to reduce the amount of processing, depending on the conditions. Additionally, panning processing will be explained using specific examples, but panning processing is not limited to the specific example methods explained below, and existing panning processing methods such as VBAP (Vector Based Amplitude Panning), DBAP (Distance Based Amplitude Panning), and Ambisonics can also be applied.

[composition]
Next, the configuration of the sound reproduction system 100 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the sound reproduction system according to this embodiment.

As shown in FIG. 2, the sound reproduction system 100 according to this embodiment includes an information processing device 101, a communication module 102, a detector 103, a driver 104, and a database 105.

The information processing device 101 is an arithmetic device for performing various signal processing in the sound reproduction system 100. The information processing device 101 is equipped with a processor and memory, such as a computer, and is realized by the processor executing a program stored in the memory. Execution of this program provides the functions related to each functional unit described below.

The information processing device 101 has an acquisition unit 111, a path calculation unit 121, an output sound generation unit 131, and a signal output unit 141. Details of each functional unit of the information processing device 101 will be described below, along with details of the configuration other than the information processing device 101.

The communication module 102 is an interface device for accepting input of sound information to the sound reproduction system 100. The communication module 102 includes, for example, an antenna and a signal converter, and receives sound information from an external device via wireless communication. More specifically, the communication module 102 uses the antenna to receive a wireless signal representing sound information converted into a format for wireless communication, and then uses the signal converter to reconvert the wireless signal into sound information. In this way, the sound reproduction system 100 acquires sound information from an external device via wireless communication. The sound information acquired by the communication module 102 is acquired by the acquisition unit 111. In this way, the acquisition unit 111 is an example of a sound acquisition unit. The sound information is input to the information processing device 101 in the above manner. Note that communication between the sound reproduction system 100 and the external device may also be performed via wired communication.

The sound information acquired by the sound reproduction system 100 is encoded in a predetermined format, such as MPEG-H 3D Audio (ISO/IEC 23008-3). As an example, the encoded sound information includes information about the sound reproduced by the sound reproduction system 100 and information about the localization position when the sound image of that sound is localized at a predetermined position within a three-dimensional sound field (i.e., perceived as sound coming from a predetermined direction). Sound information can also be interpreted as information about a sound source object. In other words, sound information includes the position of the sound source object within a three-dimensional sound field and the sound produced by the sound source object. In addition, sound information may include a flag that determines whether or not to apply panning processing. This flag will be described later.

Sound information is obtained as input data as described above, and includes audio signals (acoustic signals), which are information about the reproduced sound, and other information, such as information about the position of sound source objects within a three-dimensional sound field. This other information may also include information for defining the three-dimensional sound field. For this reason, other information is sometimes collectively referred to as spatial information, which includes information about the position of sound source objects and information for defining the three-dimensional sound field. When viewing the input data primarily as audio signals, it can be said that the input data is sound information in which other information (metadata) is attached to the audio signal. When viewing the input data primarily as spatial information, it can be said that the input data is information in which the audio signal is attached to spatial information. Alternatively, since the input data has both aspects, it can also be thought of as sound spatial information.

As a specific example, the sound information includes information about multiple sounds including a first reproduced sound and a second reproduced sound, and the sound images when each sound is reproduced are localized so that they are perceived as coming from different positions in a three-dimensional sound field. Therefore, the sound source object for the first reproduced sound is localized at a first position in the three-dimensional sound field, and the sound source object for the second reproduced sound is localized at a second position in the three-dimensional sound field. In this way, the sound information may include multiple sounds. In other words, the sound information may include multiple audio signals corresponding to the first reproduced sound and the second reproduced sound, respectively, and the positions of multiple sound source objects at first and second positions that correspond one-to-one to the multiple audio signals.

3 is a diagram illustrating an example of an audio signal according to an embodiment. For example, as shown in FIG. 3(a), the audio information may include an audio signal of a first direct sound arriving from a first position (from a first direction) to the position of the user 99, and an audio signal of a second direct sound arriving from a second position (from a second direction) to the position of the user 99. The audio information immediately after acquisition may include only information about the reproduced sound. In this case, information about a predetermined position may be acquired separately, and subsequent processing may be performed once this information is collected. Alternatively, the audio information may include multiple audio signals and the position of a sound source object that corresponds to the multiple audio signals in a many-to-one relationship. For example, this type of audio information is used in a situation where multiple reproduced sounds are produced from a certain sound source object. For example, each of the multiple audio signals corresponds to a direct sound that arrives directly from the position of the sound source object to the position of the user 99, and a secondary sound (sound resulting from indirect propagation) that accompanies the direct sound and arrives via a path different from that of the direct sound.

3(b), the sound information immediately after acquisition includes an audio signal related to the direct sound, which is converted into sound information including audio signals for reverberation, primary reflections, diffraction, etc., by a conversion process that calculates the secondary sounds. This conversion process that calculates the secondary sounds uses information on the spatial environment conditions of the three-dimensional sound field (e.g., the position, reflection, diffraction characteristics, etc. of objects in the three-dimensional sound field). In this way, secondary sounds are computationally generated from sound information related to a single reproduced sound based on the spatial environment conditions of the three-dimensional sound field, and therefore are not included in the sound information immediately after acquisition, and sound information including these secondary sounds is generated by the conversion process that calculates the secondary sounds. Further secondary sounds may be generated from one secondary sound as a result of its propagation. Note that information on the spatial environment conditions is part of the spatial information and may be acquired together with the audio signal from the input sound information. Alternatively, the audio signal and spatial information may be acquired separately. In other words, the sound information may be acquired from a single file or bitstream, or may be acquired separately in multiple files or bitstreams. For example, the audio signal and spatial information may be obtained from separate files or bitstreams, or the audio signal and spatial information may each be obtained from multiple files or bitstreams.

As such, there are no particular limitations on the form of the input sound information, and the sound reproduction system 100 only needs to be equipped with an acquisition unit 111 that can handle various forms of sound information.

Here, an example of the acquisition unit 111 will be described using Figure 4. Figure 4 is a block diagram showing the functional configuration of the acquisition unit according to the embodiment. As shown in Figure 4, the acquisition unit 111 in this embodiment includes, for example, an encoded sound information input unit 112, a decoding processing unit 113, and a sensing information input unit 114.

The encoded sound information input unit 112 is a processing unit that receives the coded (in other words, encoded) sound information acquired by the acquisition unit 111. The encoded sound information input unit 112 outputs the input sound information to the decoding processing unit 113. The decoding processing unit 113 is a processing unit that decodes (in other words, decodes) the sound information output from the encoded sound information input unit 112 to generate the playback sound and the position of the sound source object contained in the sound information in a format that can be used for subsequent processing. The sensing information input unit 114 will be described below, along with the function of the detector 103.

The detector 103 is a device for detecting the speed of movement of the user 99's head. The detector 103 is configured by combining various sensors used for detecting movement, such as a gyro sensor and an acceleration sensor. In this embodiment, the detector 103 is built into the sound reproduction system 100, but it may also be built into an external device, such as a 3D image reproduction device 300 that operates in response to the movement of the user 99's head, similar to the sound reproduction system 100. In this case, the detector 103 does not need to be included in the sound reproduction system 100. Alternatively, the detector 103 may be an external imaging device that captures the movement of the user 99's head and detects the movement of the user 99 by processing the captured image.

The detector 103 is, for example, fixed integrally to the housing of the sound reproduction system 100 and detects the speed of movement of the housing. After the sound reproduction system 100 including the housing is worn by the user 99, it moves integrally with the user 99's head, and as a result, the detector 103 can detect the speed of movement of the user 99's head.

The detector 103 may, for example, detect the amount of rotation about at least one of three mutually orthogonal axes in three-dimensional space as the rotation axis, or may detect the amount of displacement about at least one of the three axes as the displacement direction, as the amount of movement of the user 99's head. The detector 103 may also detect both the amount of rotation and the amount of displacement as the amount of movement of the user 99's head.

The sensing information input unit 114 acquires the speed of head movement of the user 99 from the detector 103. More specifically, the sensing information input unit 114 acquires the amount of head movement of the user 99 detected by the detector 103 per unit time as the speed of movement. In this way, the sensing information input unit 114 acquires at least one of the rotation speed and the displacement speed from the detector 103. The amount of head movement of the user 99 acquired here is used to determine the position and posture (in other words, the coordinates and orientation) of the user 99 in the three-dimensional sound field. Therefore, the acquisition unit 111 also functions as a position acquisition unit via the sensing information input unit 114. In the sound reproduction system 100, the position of the sound image object relative to the user 99 is determined based on the determined coordinates and orientation of the user 99, and sound is reproduced. Specifically, the above functions are realized by the path calculation unit 121 and the output sound generation unit 131.

The path calculation unit 121 includes an arrival direction calculation function that calculates the relative arrival direction of the reproduced sound from the position of the sound source object to the position of the user 99 based on the determined coordinates and orientation of the user 99, and a conversion process that calculates the secondary sound described above. Therefore, the path calculation unit 121 includes a function that calculates the propagation path from the sound source object and calculates the secondary sound and the arrival direction of the secondary sound that arrives at the position of the user 99 through indirect propagation of the reproduced sound according to the calculated propagation path of the reproduced sound. Note that the arrival direction of the secondary sound includes additional information such as what object the sound will be reflected from in the case of a reflected sound, and the attenuation rate at the time of reflection. The additional information is included in the arrival direction of the secondary sound calculated from the input sound information. In other words, the additional information is computationally generated and obtained from the sound information.

To summarize the spatial information, it includes the spatial position of the sound source object in the space (three-dimensional sound field) (information about the position of the sound source object), sound reflection and diffraction characteristics at the sound source object (also information about the conditions of the spatial environment), and further information such as the size of the three-dimensional sound field. Based on the spatial information, the path calculation unit 121 generates secondary sounds depending on which sound source object the reproduced sound is reflected or diffracted from, and calculates additional information such as the direction from which the secondary sound arrives and the volume of the secondary sound after it has attenuated due to reflection or diffraction. The sound information (input data) includes spatial information in the form of audio signals and accompanying metadata, and as described above, the spatial information includes, as information other than the audio signal, information necessary to turn the sound into stereophonic sound and position the sound source object within the three-dimensional sound field, and/or information used to calculate information necessary to turn the sound into stereophonic sound and position the sound source object within the three-dimensional sound field.

The path calculation unit 121 may be realized by any processing as long as it can calculate the direction of arrival of the reproduced sound when it reaches the user as direct sound, and can calculate the direction of arrival of secondary sounds that arrive at the position of the user 99 due to secondary propagation of the reproduced sound. The path calculation unit 121 determines from which direction in the three-dimensional sound field the reproduced sound and secondary sounds will be perceived by the user 99 as coming from, based on the coordinates and orientation of the user 99, and processes the sound information so that when the output sound signal is reproduced, it is perceived as such a sound.

The output sound generation unit 131 is a processing unit that generates an output sound signal by processing information about the reproduced sound contained in the sound information.

Here, an example of the output sound generation unit 131 will be described using FIG. 5. FIG. 5 is a block diagram showing the functional configuration of the output sound generation unit according to the embodiment. As shown in FIG. 5, the output sound generation unit 131 in this embodiment includes, for example, a generation unit 134 and a synthesis unit 135.

The generation unit 134 is a processing unit used when applying panning processing to perform conversion processing to convert the reproduced sound into a representative sound, and then convolving a head-related transfer function with the converted representative sound. The generation unit 134 acquires the reproduced sound and the position of the representative point, and performs conversion processing to convert the reproduced sound into a representative sound so that the reproduced sound is reproduced using sound from the representative point. The generation unit 134 has the same function as the panning unit described later in Figure 15.

For example, if a sound source object is located halfway between two representative points, a sound is generated so that the same sound as the playback sound is played from each of the two representative points. In other words, the playback sound is distributed to the two representative points. Then, a representative sound can be generated by adjusting the gain of the generated sound to match the position of the sound source object. The conversion of the playback sound to a representative sound is not limited to this example. For example, the playback sound may be converted to a representative sound by performing time shift adjustment and gain adjustment, as described below, or any other existing conversion may be used as long as the playback sound can be converted to a representative sound that can be reproduced as sound from a representative point. Furthermore, in this specification, the conversion process of the playback sound to a representative sound may be interpreted as the process of distributing the playback sound to representative points (representative directions). Specifically, the sound signals of the playback sound associated with the position of each sound source object are distributed to the positions of the representative points, and a representative sound that arrives at the listener from the representative points (representative directions) is generated. Here, the representative direction refers to the direction of the representative point position as seen from the listener, or the direction of the listener as seen from the representative point position. An example of the conversion that performs time shift adjustment and gain adjustment will be described later. The generation unit 134 acquires the same number of representative sounds as the number of representative points obtained by the conversion, and head-related transfer functions corresponding to the representative directions from each representative point to the position of the user 99, and performs convolution processing of the acquired head-related transfer functions on the representative sounds to generate a sound signal.

In other words, the panning unit performs panning to represent the sound source by panning sounds from a specific representative direction based on the sound source directions of multiple sound sources (target signals) acquired by the path calculation unit 121, by time shifting the sound sources and adjusting the gain. Specifically, the panning unit synthesizes the sound source (target signal) by panning in a representative direction that approximates the sound source direction of the sound source. As a result, the panning unit generates an HRIR for the sound source direction equivalently. Here, in this embodiment, "equivalent" and "equivalently" refer to signals with an error below a specific level and that are nearly similar, as shown in the examples described below. Specifically, the panning unit generates an HRIR for the sound source direction equivalently by panning the sound source by synthesizing HRIRs for several directions that are closest to the sound source direction of the sound source or that are most similar to the HRIR for the sound source direction. In this embodiment, this direction is described as a "specific representative direction" (hereinafter simply referred to as a "representative direction") described below. This reduces the amount of calculation required to generate the signal at the ear.

In other words, the panning unit synthesizes a sound image from multiple sound sources using sounds from multiple representative directions. For example, two or three representative directions can be used, but the number of representative directions is not limited to this. Specifically, the panning unit can group together the sound sources into fewer representative points than the number of sound sources, and synthesize a sound image using only the HRIRs of the representative directions for these representative points.

At this time, the panning unit calculates the time shift (delay) that maximizes the cross-correlation between the HRIR in the sound source direction and the HRIR in the representative direction. The time shift obtained here, or a time shift with a negative sign added to this time shift, is applied to the sound source, and subsequent processing is performed assuming that the signal after the time shift is in the representative direction.

This time shift may also be permitted to be a time shift shorter than the sampling frequency (a shift in which the sample position is expressed as a decimal; hereafter referred to as a "decimal shift"). This decimal shift can be performed by oversampling.

Here, the panning unit applies a gain to the signal of the representative direction obtained by time-shifting the sound source, and then calculates the sum of the values calculated for each representative point convolved with the HRIR at each representative point, thereby synthesizing a signal equivalent to the sound source convolved with the HRIR of the sound source direction.

On the other hand, when synthesizing the HRIR (vector) of the sound source direction by the sum of the HRIR (vector) of the representative direction, the panning unit may calculate the gain by orthogonalizing the error signal vector between the synthesized HRIR (vector) and the HRIR (vector) of the sound source direction to the HRIR (vector) of the representative direction. Note that the HRIR (vector) is the time waveform of the HRIR as if it were a vector. Hereinafter, this HRIR (vector) will also be referred to as the "HRIR vector."

The panning unit corrects this gain so that the energy balance of the HRIRs for the left and right ears from the sound source position is maintained in an HRIR substantially synthesized by panning from HRIRs from multiple representative points. In other words, the panning unit may correct the gain so that the energy balance of the HRIRs for the left and right ears of the listener from the sound source is maintained in an HRIR substantially synthesized by panning.

In this embodiment, the panning unit calculates the gain value of the HRIR gain in the representative direction for each sound source direction and a time shift value corresponding to the time shift of the HRIR, and stores these values in the HRIR table 200, which will be described later.

The panning unit then time-shifts each sound source using the time shift and gain values corresponding to the sound source direction of each sound source, multiplies the gain, and sums these to create a sum signal. The panning unit treats this sum signal as existing at the position of the representative point. The panning unit can convolve the HRIR at the position of the representative point with this sum signal to generate a signal at the listener's ear.

The synthesis unit 135 generates an output sound signal. The synthesis unit 135 may perform EQ adjustment on the sound signal. Specifically, in the panning process, EQ adjustment may be performed to increase the gain of the high-frequency domain, which is likely to be attenuated, and to emphasize this high-frequency domain. Therefore, the synthesis unit 135 functions as an EQ adjustment unit. Note that, if there are multiple sound signals, the EQ adjustment performed by the synthesis unit 135 may be performed on only some or all of the multiple sound signals.

Referring again to Figure 2, the output sound generation unit 131 obtains the head-related transfer function used to generate the output sound signal from the database 105. The database 105 is an information storage device that functions both as a storage device for storing information and as a storage controller that reads out the stored information and outputs it to an external component. The database 105 stores a head-related transfer function for each direction of arrival for the user 99. The head-related transfer functions included in the database 105 are a set of general-purpose head-related transfer functions that can be used by everyone, a set of head-related transfer functions optimized for each individual user 99, or a set of head-related transfer functions that are publicly available. The database 105 receives an inquiry from the output sound generation unit 131 using the direction of arrival as a query, and outputs the head-related transfer function corresponding to that direction of arrival to the output sound generation unit 131. The output sound generation unit 131 may also output the entire set of head-related transfer functions, or may output the characteristics of the set of head-related transfer functions itself.

The signal output unit 141 is a functional unit that outputs the generated output sound signal to the driver 104. The signal output unit 141 generates a waveform signal by performing signal conversion from digital to analog based on the output sound signal, and then generates sound waves in the driver 104 based on the waveform signal, presenting the sound to the user 99. The driver 104 has, for example, a diaphragm and a drive mechanism such as a magnet and voice coil. The driver 104 operates the drive mechanism in accordance with the waveform signal, causing the diaphragm to vibrate. In this way, the driver 104 generates sound waves by vibrating the diaphragm in accordance with the output sound signal (this means "reproducing" the output sound signal; in other words, "reproducing" does not include perception by the user 99), and the sound waves propagate through the air to the ears of the user 99, who then perceives the sound.

[Another configuration example]
In the above example, the sound reproduction system 100 according to the present embodiment is a sound presentation device, and has been described as including an information processing device 101, a communication module 102, a detector 103, a database 105, and a driver 104. However, the functions of the sound reproduction system 100 may be realized by a plurality of devices or by a single device. Specific examples will be described using Figures 6 to 15. Figures 6 to 15 are diagrams for explaining other examples of the sound reproduction system according to the embodiment.

For example, the information processing device 601 may be included in the audio presentation device 602, and the audio presentation device 602 may perform both acoustic processing and sound presentation. Furthermore, the information processing device 601 and the audio presentation device 602 may share the acoustic processing described in this disclosure, or a server connected to the information processing device 601 or the audio presentation device 602 via a network may perform some or all of the acoustic processing described in this disclosure.

Note that although the above description refers to the information processing device 601, if the information processing device 601 performs acoustic processing by decoding a bitstream generated by encoding at least a portion of the data of the audio signal or spatial information used in acoustic processing, the information processing device 601 may be called a decoding device, and the acoustic reproduction system 100 (i.e., the stereophonic sound reproduction system 600 in the figure) may be called a decoding processing system.

Here, we will explain an example in which the sound reproduction system 100 functions as a decoding processing system.

<Example of encoding device>
FIG. 7 is a functional block diagram showing the configuration of an encoding device 700, which is an example of an encoding device according to the present disclosure.

Input data 701 is data to be encoded, including spatial information and/or audio signals, input to encoder 702. Details of spatial information will be explained later.

The encoder 702 encodes the input data 701 to generate encoded data 703. The encoded data 703 is, for example, a bit stream generated by the encoding process.

Memory 704 stores encoded data 703. Memory 704 may be, for example, a hard disk or SSD (Solid-State Drive), or may be some other storage device.

In the above explanation, a bitstream generated by the encoding process was given as an example of encoded data 703 stored in memory 704, but data other than a bitstream may also be used. For example, encoding device 700 may convert a bitstream into a predetermined data format and store the converted data in memory 704. The converted data may be, for example, a file or multiplexed stream containing one or more bitstreams. Here, the file may have a file format such as ISOBMFF (ISO Base Media File Format). Encoded data 703 may also be in the form of multiple packets generated by dividing the bitstream or file. When converting the bitstream generated by encoder 702 into data other than the bitstream, encoding device 700 may be equipped with a conversion unit (not shown), or the conversion process may be performed by a CPU (Central Processing Unit).

<Example of a decoding device>
FIG. 8 is a functional block diagram showing the configuration of a decoding device 800, which is an example of a decoding device according to the present disclosure.

Memory 804 stores, for example, the same data as the encoded data 703 generated by encoding device 700. Memory 804 reads the stored data and inputs it as input data 803 to decoder 802. Input data 803 is, for example, a bitstream to be decoded. Memory 804 may be, for example, a hard disk or SSD, or may be some other storage device.

Note that decoding device 800 may not use the data stored in memory 804 as input data 803 as is, but may convert the read data and generate converted data as input data 803. The data before conversion may be, for example, multiplexed data containing one or more bitstreams. Here, the multiplexed data may be a file having a file format such as ISOBMFF. The data before conversion may also be in the form of multiple packets generated by dividing the bitstream or file. When converting data different from the bitstream read from memory 804 into a bitstream, decoding device 800 may be equipped with a conversion unit (not shown), or the conversion process may be performed by a CPU.

The decoder 802 decodes the input data 803 to generate an audio signal 801 that is presented to the listener.

<Another example of an encoding device>
Fig. 9 is a functional block diagram showing the configuration of an encoding device 900, which is another example of an encoding device according to the present disclosure. In Fig. 9, components having the same functions as those in Fig. 7 are assigned the same reference numerals, and descriptions of these components will be omitted.

The coding device 700 differs from the coding device 700 in that the coding device 700 is equipped with a memory 704 that stores coded data 703, whereas the coding device 900 is equipped with a transmission unit 901 that transmits coded data 703 to the outside.

The transmitter 901 transmits a transmission signal 902 to another device or server based on the encoded data 703 or data in a different data format generated by converting the encoded data 703. The data used to generate the transmission signal 902 is, for example, a bit stream, multiplexed data, a file, or a packet, as described in the encoding device 700.

<Another example of a composite device>
Fig. 10 is a functional block diagram showing the configuration of a decoding device 1000, which is another example of a decoding device according to the present disclosure. In Fig. 10, components having the same functions as those in Fig. 8 are assigned the same reference numerals, and descriptions of these components will be omitted.

Decoding device 800 differs from decoding device 800 in that it is equipped with memory 804 that reads input data 803, whereas decoding device 1000 is equipped with a receiving unit 1001 that receives input data 803 from outside.

The receiving unit 1001 receives the received signal 1002, acquires the received data, and outputs the input data 803 to be input to the decoder 802. The received data may be the same as the input data 803 to be input to the decoder 802, or may be data in a different data format from the input data 803. If the received data is in a different data format from the input data 803, the receiving unit 1001 may convert the received data into the input data 803, or a conversion unit or CPU (not shown) provided in the decoding device 1000 may convert the received data into the input data 803. The received data is, for example, a bit stream, multiplexed data, a file, or a packet as described for the encoding device 900.

<Decoder function explanation>
FIG. 11 is a functional block diagram showing the configuration of a decoder 1100, which is an example of the decoder 802 in FIG. 8 or 10.

The input data 803 is an encoded bitstream and includes encoded audio data, which is an encoded audio signal, and metadata used for acoustic processing.

The spatial information management unit 1101 acquires metadata contained in the input data 803 and analyzes the metadata. The metadata includes information describing elements that act on sounds arranged in a sound space. The spatial information management unit 1101 manages the spatial information necessary for acoustic processing obtained by analyzing the metadata and provides the spatial information to the rendering unit 1103. Note that although the information used for acoustic processing is referred to as spatial information in this disclosure, it may be called something else. The information used for acoustic processing may be called, for example, sound space information or scene information. Furthermore, if the information used for acoustic processing changes over time, the spatial information input to the rendering unit 1103 may be called a space state, sound space state, scene state, etc.

Furthermore, spatial information may be managed for each sound space or for each scene. For example, when different rooms are represented as virtual spaces, each room may be managed as a different sound space scene, or even for the same space, spatial information may be managed as different scenes depending on the situation being represented. When managing spatial information, an identifier that identifies each piece of spatial information may be assigned. The spatial information data may be included in a bitstream, which is one form of input data 803, or the bitstream may include an identifier for the spatial information and the spatial information data may be obtained from somewhere other than the bitstream. If the bitstream only includes an identifier for the spatial information, the identifier for the spatial information may be used during rendering to obtain spatial information data stored in the memory of the acoustic signal processing device or an external server as input data.

Note that the information managed by the spatial information management unit 1101 is not limited to information contained in the bitstream. For example, the input data 803 may include data indicating the characteristics or structure of a space obtained from a software application or server that provides VR or AR, as data not included in the bitstream. Furthermore, for example, the input data 803 may include data indicating the characteristics or position of a listener or object, as data not included in the bitstream. Furthermore, the input data 803 may include, as information indicating the position of the listener, information obtained by a sensor provided in a terminal including a decoding device, or information indicating the position of the terminal estimated based on information obtained by the sensor. In other words, the spatial information management unit 1101 may communicate with an external system or server to obtain spatial information and the position of the listener. Furthermore, the spatial information management unit 1101 may obtain clock synchronization information from an external system and perform processing to synchronize with the clock of the rendering unit 1103. Note that the space in the above description may be a virtually created space, i.e., a VR space, or a real space or a virtual space corresponding to a real space, i.e., an AR space or an MR (Mixed Reality) space. Virtual spaces may also be called sound fields or sound spaces. Furthermore, the information indicating a position in the above description may be information such as coordinate values indicating a position within a space, information indicating a relative position with respect to a predetermined reference position, or information indicating the movement or acceleration of a position within a space.

The audio data decoder 1102 decodes the encoded audio data contained in the input data 803 to obtain an audio signal.

The encoded audio data acquired by the stereophonic sound reproduction system 600 is a bitstream encoded in a predetermined format, such as MPEG-H 3D Audio (ISO/IEC 23008-3). Note that MPEG-H 3D Audio is merely one example of an encoding method that can be used to generate the encoded audio data contained in the bitstream, and bitstreams encoded using other encoding methods may also be included as encoded audio data. For example, the encoding method used may be a lossy codec such as MP3 (MPEG-1 Audio Layer-3), AAC (Advanced Audio Coding), WMA (Windows Media Audio), AC3 (Audio Codec-3), or Vorbis, or a lossless codec such as ALAC (Apple Lossless Audio Codec) or FLAC (Free Lossless Audio Codec), or any other encoding method may be used. For example, PCM (Pulse Code Modulation) data may be a type of encoded audio data. In this case, the decoding process may be, for example, a process of converting an N-bit binary number into a number format (e.g., floating-point format) that can be processed by the rendering unit 1103, where the number of quantization bits of the PCM data is N.

The rendering unit 1103 receives an audio signal and spatial information as input, performs acoustic processing on the audio signal using the spatial information, and outputs the processed audio signal 801.

Before rendering begins, the spatial information management unit 1101 reads the metadata of the input signal, detects rendering items such as objects or sounds defined in the spatial information, and sends them to the rendering unit 1103. After rendering begins, the spatial information management unit 1101 tracks changes over time in the spatial information and the listener's position, and updates and manages the spatial information. The spatial information management unit 1101 then sends the updated spatial information to the rendering unit 1103. The rendering unit 1103 generates and outputs an audio signal to which acoustic processing has been applied based on the audio signal included in the input data and the spatial information received from the spatial information management unit 1101.

The spatial information update process and the audio signal output process with added acoustic processing may be executed in the same thread, or the spatial information management unit 1101 and rendering unit 1103 may each be assigned to an independent thread. If the spatial information update process and the audio signal output process with added acoustic processing are executed in different threads, the thread startup frequency may be set individually, or the processes may be executed in parallel.

By having the spatial information management unit 1101 and the rendering unit 1103 execute their processing in different, independent threads, it is possible to allocate computational resources preferentially to the rendering unit 1103. This means that sound output processing that cannot tolerate even the slightest delay, such as sound output processing where a delay of even one sample (0.02 msec) would cause a popping noise, can be safely carried out. In this case, the allocation of computational resources to the spatial information management unit 1101 is limited. However, compared to audio signal output processing, updating spatial information is a less frequent process (for example, processing such as updating the direction of the listener's face). For this reason, it does not necessarily require an instantaneous response like audio signal output processing, so limiting the allocation of computational resources does not have a significant impact on the acoustic quality provided to the listener.

Updating of spatial information may be performed periodically at preset times or intervals, or when preset conditions are met. Furthermore, updating of spatial information may be performed manually by the listener or sound space administrator, or may be triggered by a change in an external system. For example, if a listener operates a controller to instantly warp the position of their avatar, or instantly advance or reverse the time, or if the administrator of the virtual space suddenly changes the environment of the venue, the thread in which the spatial information management unit 1101 is located may be started as a one-off interrupt process in addition to being started periodically.

The role of the information update thread that executes spatial information update processing is, for example, to update the position or orientation of the listener's avatar placed in the virtual space based on the position or orientation of the VR goggles worn by the listener, and to update the position of objects moving within the virtual space. These tasks are handled within a processing thread that runs relatively infrequently, on the order of a few tens of Hz. Processing to reflect the properties of direct sound may be performed in such an infrequent processing thread. This is because the properties of direct sound change less frequently than the frequency with which audio processing frames for audio output occur. Doing so can actually reduce the computational load of the processing relatively, and it can also avoid the risk of pulsive noise occurring when information is updated at an unnecessarily fast frequency.

Figure 12 is a functional block diagram showing the configuration of a decoder 1200, which is another example of the decoder 802 in Figure 8 or Figure 10.

FIG. 12 differs from FIG. 11 in that the input data 803 includes an unencoded audio signal rather than encoded audio data. The input data 803 includes a bitstream including metadata and an audio signal.

The spatial information management unit 1201 is the same as the spatial information management unit 1101 in Figure 11, so a description thereof will be omitted.

The rendering unit 1202 is the same as the rendering unit 1103 in Figure 11, so a description will be omitted.

Note that in the above explanation, the configuration in Figure 12 is called a decoder, but it may also be called an audio processing unit that performs audio processing. Furthermore, a device that includes an audio processing unit may also be called an audio processing device rather than a decoding device. Furthermore, an audio signal processing device (information processing device 601) may also be called an audio processing device.

<Physical configuration of the encoding device>
13 is a diagram showing an example of the physical configuration of an encoding device, and the encoding device shown in FIG. 13 is an example of the encoding devices 700 and 900 described above.

The encoding device in Figure 13 includes a processor, memory, and a communication interface.

The processor may be, for example, a CPU (Central Processing Unit), DSP (Digital Signal Processor), or GPU (Graphics Processing Unit), and the encoding process of the present disclosure may be performed by the CPU, DSP, or GPU executing a program stored in memory. The processor may also be a dedicated circuit that performs signal processing on audio signals, including the encoding process of the present disclosure.

Memory may be composed of, for example, RAM (Random Access Memory) or ROM (Read Only Memory). Memory may also include magnetic storage media such as hard disks or semiconductor memory such as SSDs (Solid State Drives). Internal memory built into the CPU or GPU may also be referred to as memory.

The communication IF (Interface) is a communication module compatible with communication methods such as Bluetooth (registered trademark) or WIGIG (registered trademark). The encoding device has the function of communicating with other communication devices via the communication IF and transmits the encoded bitstream.

The communication module is composed of, for example, a signal processing circuit and an antenna that supports the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) was used as the communication method, but communication methods such as LTE (Long Term Evolution), NR (New Radio), or Wi-Fi (registered trademark) may also be supported. Furthermore, instead of the wireless communication methods mentioned above, the communication IF may also be a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark) (High-Definition Multimedia Interface).

<Physical configuration of the acoustic signal processing device>
Fig. 14 is a diagram showing an example of the physical configuration of an audio signal processing device. Note that the audio signal processing device in Fig. 14 may be a decoding device. Also, part of the configuration described here may be provided in an audio presentation device 602. Also, the audio signal processing device shown in Fig. 14 is an example of the audio signal processing device 601 described above.

The acoustic signal processing device in Figure 14 includes a processor, memory, a communication IF, a sensor, and a speaker.

The processor may be, for example, a CPU (Central Processing Unit), DSP (Digital Signal Processor), or GPU (Graphics Processing Unit), and the CPU, DSP, or GPU may execute a program stored in memory to perform the acoustic processing or decoding processing of the present disclosure. The processor may also be a dedicated circuit that performs signal processing on audio signals, including the acoustic processing of the present disclosure.

Memory may be composed of, for example, RAM (Random Access Memory) or ROM (Read Only Memory). Memory may also include magnetic storage media such as hard disks or semiconductor memory such as SSDs (Solid State Drives). Internal memory built into the CPU or GPU may also be referred to as memory.

The communication IF (Interface) is a communication module compatible with communication methods such as Bluetooth (registered trademark) or WIGIG (registered trademark). The acoustic signal processing device shown in FIG. 14 has the function of communicating with other communication devices via the communication IF, and acquires the bitstream to be decoded. The acquired bitstream is stored in memory, for example.

The communication module is composed of, for example, a signal processing circuit and an antenna that supports the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) was used as the communication method, but communication methods such as LTE (Long Term Evolution), NR (New Radio), or Wi-Fi (registered trademark) may also be supported. Furthermore, instead of the wireless communication methods mentioned above, the communication IF may also be a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark) (High-Definition Multimedia Interface).

The sensor performs sensing to estimate the position or orientation of the listener. Specifically, the sensor estimates the position and/or orientation of the listener based on one or more detection results of the position, orientation, movement, velocity, angular velocity, or acceleration of the entire or part of the listener's body, such as the head, and generates position information indicating the position and/or orientation of the listener. Note that the position information may be information indicating the position and/or orientation of the listener in real space, or information indicating the displacement of the position and/or orientation of the listener based on the position and/or orientation of the listener at a specified point in time. The position information may also be information indicating the position and/or orientation relative to the stereophonic reproduction system or an external device equipped with the sensor.

The sensor may be, for example, an imaging device such as a camera or a ranging device such as LiDAR (Light Detection and Ranging), and may capture the movement of the listener's head and detect the movement of the listener's head by processing the captured image. Alternatively, the sensor may be a device that performs position estimation using wireless signals of any frequency band, such as millimeter waves.

The audio signal processing device shown in FIG. 14 may acquire position information from an external device equipped with a sensor via a communication IF. In this case, the audio signal processing device does not need to include a sensor. Here, the external device is, for example, the audio presentation device 602 described in FIG. 6 or a 3D video playback device worn on the listener's head. In this case, the sensor is configured by combining various sensors such as a gyro sensor and an acceleration sensor.

The sensor may, for example, detect the angular velocity of rotation about at least one of three mutually orthogonal axes in the sound space as the axis of rotation, or may detect the acceleration of displacement with at least one of the three axes as the direction of displacement, as the speed of movement of the listener's head.

For example, the sensor may detect the amount of movement of the listener's head by detecting the amount of rotation around at least one of three mutually orthogonal axes in the sound space, or by detecting the amount of displacement along at least one of the three axes. Specifically, the sensor detects 6 DoF (position (x, y, z) and angle (yaw, pitch, roll)) as the listener's position. The sensor is configured by combining various sensors used for detecting movement, such as gyro sensors and acceleration sensors.

The sensor may be any device capable of detecting the position of the listener, such as a camera or a GPS (Global Positioning System) receiver. It may also use location information obtained by performing self-position estimation using LiDAR (Laser Imaging Detection and Ranging). For example, if the audio signal playback system is implemented using a smartphone, the sensor may be built into the smartphone.

The sensors may also include a temperature sensor such as a thermocouple that detects the temperature of the acoustic signal processing device shown in FIG. 14, and a sensor that detects the remaining charge of a battery included in or connected to the acoustic signal processing device.

A speaker, for example, has a diaphragm, a drive mechanism such as a magnet or voice coil, and an amplifier, and presents the processed audio signal as sound to the listener. The speaker operates the drive mechanism in response to the audio signal (more specifically, a waveform signal indicating the waveform of the sound) amplified via the amplifier, causing the diaphragm to vibrate. In this way, the diaphragm vibrates in response to the audio signal, generating sound waves that propagate through the air and reach the listener's ears, allowing the listener to perceive the sound.

Note that, while the example described here is one in which the acoustic signal processing device shown in FIG. 14 is equipped with a speaker and presents an audio signal after acoustic processing via the speaker, the means for presenting the audio signal is not limited to the above configuration. For example, the audio signal after acoustic processing may be output to an external audio presentation device 602 connected via a communication module. Communication via the communication module may be wired or wireless. As another example, the acoustic signal processing device shown in FIG. 14 may be equipped with a terminal that outputs an analog audio signal, and an audio signal may be presented from the earphone or other device by connecting a cable to the terminal. In the above case, the audio signal is reproduced by headphones, earphones, a head-mounted display, a neck speaker, a wearable speaker, or a surround speaker consisting of multiple fixed speakers that is worn on the head or part of the body of the listener, which is the audio presentation device 602.

<Functional explanation of the rendering section>
FIG. 15 is a functional block diagram showing an example of a detailed configuration of the rendering units 1103 and 1202 shown in FIGS.

The rendering unit is composed of an analysis unit, a panning unit, and a synthesis unit (different from the synthesis unit 135 described above), and applies acoustic processing to the sound data contained in the input signal and outputs it.

The information contained in the input signal is explained below.

The input signal may be composed of, for example, spatial information, sensor information, and sound data. The input signal may also include a bitstream composed of sound data and metadata (control information), in which case the metadata may include spatial information.

Spatial information is information about the sound space (three-dimensional sound field) created by the stereophonic playback system, and is composed of information about the objects contained in the sound space and information about the listener. Objects include sound source objects that emit sound and act as sound sources, and non-sound-emitting objects that do not emit sound. Non-sound-emitting objects function as obstacle objects that reflect sounds emitted by sound source objects, but sound source objects can also function as obstacle objects that reflect sounds emitted by other sound source objects.

Information commonly assigned to sound-source objects and non-sound-producing objects includes position information, shape information, and the rate at which the volume attenuates when the object reflects sound.

Position information is expressed as coordinate values on three axes in Euclidean space, for example the X, Y, and Z axes, but it does not necessarily have to be three-dimensional information. For example, it may be two-dimensional information expressed as coordinate values on two axes, the X and Y axes. The position information of an object is determined by the representative position of a shape expressed in mesh or voxels.

The shape information may also include information about the surface material.

It may also include information indicating whether the object belongs to a living organism or whether the object is a moving object. If the object is a moving object, the position information may change over time, and the changed position information or the amount of change is transmitted to the rendering unit.

Information about sound source objects includes the information commonly assigned to sound source objects and non-sound generating objects described above, as well as sound data and information necessary to radiate the sound data into the sound space.

Sound data is data that represents the sound perceived by a listener, including information about the frequency and intensity of the sound. Sound data is typically a PCM signal, but it may also be data compressed using an encoding method such as MP3. In this case, the signal must be decoded at least before it reaches the synthesis unit, so the rendering unit may include a decoding unit (not shown). Alternatively, the signal may be decoded by the audio data decoder 1102.

At least one piece of sound data needs to be set for one sound source object, but multiple pieces of sound data may also be set. Furthermore, identification information for identifying each piece of sound data may be assigned, and the sound data identification information may be held as information related to the sound source object.

Information necessary for radiating sound data into a sound space may include, for example, information on the reference volume that serves as a reference when playing back the sound data, information indicating the properties (also called characteristics) of the sound data, information on the position of the sound source object, information on the orientation of the sound source object, and information on the directionality of the sound emitted by the sound source object. The reference volume information is, for example, the effective value of the amplitude value of the sound data at the sound source position when radiating the sound data into a sound space, and may be expressed as a floating-point decibel (dB) value.

For example, if the reference volume is 0 dB, it may indicate that sound is emitted into the sound space from the position indicated by the information regarding the position at the same volume as the signal level indicated by the sound data, without increasing or decreasing the volume. If the reference volume is -6 dB, it may indicate that sound is emitted into the sound space from the position indicated by the information regarding the position with the volume of the signal level indicated by the sound data reduced to approximately half. This information is assigned to one piece of sound data or to multiple pieces of sound data collectively.

Information indicating the properties of the sound data may be, for example, information regarding the volume of the sound source, and may be information indicating time-series fluctuations. For example, if the sound space is a virtual conference room and the sound source is a speaker, the volume will transition intermittently over short periods of time. Expressed more simply, this can be said to be alternating between sound and silence.

Furthermore, if the sound space is a concert hall and the sound source is a performer, the volume will be maintained for a certain period of time. Furthermore, if the sound space is a battlefield and the sound source is an explosive, the volume of the explosion will increase for a moment and then remain silent. In this way, information about the volume of the sound source includes not only information about the volume of the sound, but also information about the transition in volume of the sound, and such information may be used as information indicating the properties of the sound data.

Here, the information on transitions in sound volume may be data showing frequency characteristics in a time series. It may be data showing the duration of sections where sound is present. It may be data showing a time series of the duration of sections where sound is present and the duration of sections where sound is absent. It may be data listing multiple sets of data on durations where the amplitude of a sound signal can be considered steady (considered to be roughly constant) and the amplitude values of the signal during that time in a time series. It may be data on durations where the frequency characteristics of a sound signal can be considered steady. It may be data listing multiple sets of data on durations where the frequency characteristics of a sound signal can be considered steady and the frequency characteristics during that time in a time series.

The data format may be, for example, data indicating the outline of a spectrogram. Furthermore, the volume that serves as the basis for the frequency characteristics may be used as the reference volume. Information on the reference volume and information indicating the properties of the sound data may be used to calculate the volume of direct or reflected sound perceived by the listener, as well as in a selection process to select whether or not to perceive it. Other examples of information indicating the properties of sound data and specific ways in which it is used in the selection process will be described later.

Orientation information is typically expressed using yaw, pitch, and roll. Alternatively, the roll rotation may be omitted and the information may be expressed using azimuth (yaw) and elevation (pitch). Orientation information may change over time, and if it does change, it is transmitted to the rendering unit.

Information about the listener is information about the listener's position and orientation in sound space. Position information is expressed as a position on the XYZ axes in Euclidean space, but it does not necessarily have to be three-dimensional information; it can also be two-dimensional information. Orientation information is typically expressed using yaw, pitch, and roll. Alternatively, the roll rotation can be omitted and it can be expressed using azimuth (yaw) and elevation (pitch). Position information and orientation information may change over time, and if they do change, they are transmitted to the rendering unit.

Sensor information includes the amount of rotation or displacement detected by a sensor worn by the listener, as well as the position and orientation of the listener. The sensor information is transmitted to the rendering unit, which updates the position and orientation information of the listener based on the sensor information. The sensor information may be, for example, location information obtained by a mobile device performing self-location estimation using GPS, a camera, or LiDAR (Laser Imaging Detection and Ranging). Information obtained externally from a source other than a sensor via a communications module may also be detected as sensor information. Information indicating the temperature of the audio signal processing device and information indicating the remaining battery capacity may be obtained from the sensor. The computing resources (CPU capacity, memory resources, PC performance) of the audio signal processing device and audio signal presentation device may also be obtained in real time.

The analysis unit performs the same function as the acquisition unit 111 in the above example. In other words, it analyzes the input signal and acquires the information required by the path calculation unit 121 and output sound generation unit 131.

The synthesis unit performs functions equivalent to those of the output sound generation unit 131 and signal output unit 141 in the above example. It processes the input audio signal to generate direct sound based on the audio signal of the direct sound and information on the time of arrival of the direct sound and the volume at the time of arrival of the direct sound calculated by the analysis unit. It also processes the input audio signal to generate reflected sound based on information on the time of arrival of the reflected sound and the volume at the time of arrival of the reflected sound calculated by the analysis unit. The synthesis unit synthesizes the generated direct sound and reflected sound and outputs the result.

The panning unit performs the same functions as the generation unit 134 in the above example. In other words, based on the sound source directions of multiple sound sources (target signals) acquired by the analysis unit, panning is performed to represent the sound sources by panning sounds from a specific representative direction through time shifting and gain adjustment of the sound sources. The processing performed by the panning unit described above may be executed as part of pipeline processing such as that described in WO 2021/180938, for example.

Figure 16 is a block diagram showing an example configuration for the rendering unit 1300 to perform pipeline processing.

The rendering unit 1300 in FIG. 16 includes a reverberation processing unit 1311, an early reflection processing unit 1312, a distance attenuation processing unit 1313, a selection unit 1314, a generation unit 1315, and a binaural processing unit 1316. These multiple components may be composed of multiple components of the rendering unit shown in FIG. 15, or may be composed of at least some of the multiple components of the acoustic signal processing device shown in FIG. 14.

Pipeline processing refers to dividing the processing for adding sound effects into multiple processes, and executing each process sequentially. Each of the multiple processes, for example, performs signal processing on an audio signal, or generates parameters used in signal processing.

The rendering unit 1300 may perform pipeline processing such as reverberation processing, early reflection processing, distance attenuation processing, and binaural processing. However, these processes are merely examples, and the pipeline processing may include other processes or may not include some of the processes. For example, the pipeline processing may include diffraction processing and occlusion processing. Furthermore, for example, reverberation processing may be omitted if it is not necessary. Furthermore, not all sounds may be processed in the binaural processing stage.

Furthermore, each process may be expressed as a stage. Furthermore, audio signals such as reflected sounds generated as a result of each process may be expressed as rendering items. The multiple stages in the pipeline processing and their order are not limited to the example shown in Figure 16. For example, the processing of the panning unit may be performed in a binaural processing stage, which is one of the multiple stages included in the pipeline processing. The binaural processing unit performs functions equivalent to those of the synthesis unit described above.

In the stereophonic sound reproduction system 600 described above, in order to make the user 99 perceive as if they are moving their head within a three-dimensional sound field by changing the sound presented in accordance with the movement of the user 99's head, as explained in the example of the sound reproduction system 100 above, it is necessary to detect the position and orientation of the user 99's head (orientation relative to the position of the sound source object).

At this time, it is necessary to obtain the detection results of the position and orientation of the user's 99's head and perform information processing accordingly. Ideally, as shown in Figures 1 and 2, all components would be built into the device related to the final output portion, such as headphones, i.e., the audio presentation device 602 in Figure 6. However, due to constraints such as power supply availability, information processing performance, housing size and weight, it is necessary to divide the processing so that the information processing portion is performed in the information processing device 601 and the audio output is performed in the audio presentation device 602. Furthermore, in recent years, it has become desirable to connect the audio presentation device 602 to the information processing device 601 via wireless communication, and when sending and receiving information between the audio presentation device 602 and the information processing device 601 via such wireless communication, the limit on the amount of information that can be sent and received simultaneously (i.e., the limit on the communication bandwidth) becomes a bottleneck.

For example, if the information processing device 601 were to output an output sound signal, it would have to transmit that output sound signal to the audio presentation device 602 via wireless communication. This would result in delays due to encoding and decoding processes conforming to wireless communication standards, as well as transmission delays due to the large amount of information in the output sound signal itself, which contains a three-dimensional audio signal, and would likely impair the user 99's experience. Furthermore, since the information processing device 601 must first obtain the detection results of the user 99's head position and orientation before generating the output sound signal, it would be difficult to instantaneously have the audio follow the movement of the user 99's head. Therefore, the movement of the sound source object specified in the content is processed in the information processing device 601, and a three-dimensional audio signal is generated, which is then panned to compress the amount of information. The compressed, panned audio signal is then transmitted to the audio presentation device 602, thereby avoiding limitations on the bandwidth of the transmission path. Since the processing up to this point is predetermined for the content, the information can be transmitted to the audio presentation device 602 in advance and buffered.

Then, the audio presentation device 602 detects the movement of the user 99's head, and the detection result is used to convolve the head-related transfer function corresponding to the representative direction (after the head has moved) according to the detection result into the audio signal that has been panned on the audio presentation device 602, thereby making it possible to instantaneously make the direction of sound arrival follow the movement of the user 99's head. By performing panning processing in advance in the information processing device 601, the audio presentation device 602 only executes the process of convolving the head-related transfer functions for a small number of representative directions. This makes it difficult to increase the processing resources required on the audio presentation device 602 side, and significantly reduces the delay due to information processing on the audio presentation device 602. Because the representative direction associated with head movement is updated on the audio presentation device 602 side, no communication is required between the information processing device 601 and the audio presentation device 602 after head movement is detected and before the head-related transfer function is updated, making it possible to minimize the time required to update the direction of sound arrival.

In this way, in a stereophonic reproduction system 600 divided into an information processing device 601 and an audio presentation device 602, by converting the audio signals sent and received between the information processing device 601 and the audio presentation device 602 into audio signals that have undergone panning processing, it is possible to split the information processing between the two devices while still allowing the direction from which the audio is coming to instantaneously follow the movement of the user's 99's head.

When the information processing device 601 is the first terminal and the audio presentation device 602 is the second terminal, by doing as described above, (1) the first terminal, which is not restricted by the need for the user 99 to wear it and is therefore easier to increase processing performance than the second terminal, can perform panning processing on audio signals, including the movement of sound source objects specified in the content, which requires relatively high processing resources. Furthermore, (2) the panning processing compresses the amount of information by consolidating sounds from, for example, up to 10 representative directions, making it less susceptible to communication bandwidth restrictions when transmitting and receiving between the first and second terminals. Furthermore, (3) the second terminal worn by the user 99 (on the head where the ears are located) can easily detect the position and orientation of the user 99's head and use this information to output an output sound signal from the panned audio signal, thereby instantly updating the direction of sound arrival in response to head movement. Furthermore, (4) the output sound signal can be output by simply convolving head-related transfer functions for a small number of representative directions when panning is performed, thereby reducing the processing resources required for the second terminal. You can get the above benefits.

2 to 5, the first terminal is equipped with functions other than the communication module 102 and some of the functions of the acquisition unit 111 in FIG. 2, and the convolution processing of the path calculation unit 121 and output sound generation unit 131. The path calculation unit 121 calculates the relative arrival direction from the position of the sound source object to the reference position using the reference position, i.e., the coordinate position or orientation of the user 99 already acquired by the first terminal before transmitting sound information to the second terminal, the coordinate position or orientation of the user 99 at the time of initialization of the system (first terminal and second terminal), or a predetermined coordinate position or direction determined in advance by the system. The reference position may be determined in any way as long as it is the same coordinate position or orientation shared by the first terminal and the second terminal. Therefore, the orientation as the reference position may be determined as a specific direction such as "north" in absolute coordinates, or as an average direction calculated from the direction that was in front of the user 99 over a predetermined period of time in the past (e.g., one minute). In the latter case, the orientation as the reference position is updated every predetermined period. In this way, the first terminal and second terminal share the same coordinate position or orientation using the reference position, and then perform a series of rendering processes, as long as the shared reference position does not change during that series of rendering processes. When updating the reference position, the update may be performed by including it in the information update thread in the spatial information update process mentioned in <Description of Decoder Function>, or by including it in a thread for updating other information, or by using a dedicated thread for updating the reference position.

The second terminal is equipped with the detector 103 in Figure 2, other functions of the acquisition unit 111, a function for convolution processing of head-related transfer functions in the output sound generation unit 131, a signal output unit 141, a database 105, and a driver 104. The signal output unit 141 converts the direction to align the above-mentioned reference position with the determined coordinates and orientation of the user 99, and outputs an output sound signal according to the detection result by the detector 103.

When using the configurations shown in Figures 6 to 15, the first terminal is equipped with the analysis unit and panning unit shown in Figure 15. The analysis unit uses a reference position to calculate the relative arrival direction from the position of the sound source object to the reference position based on the first sound information, which is an input signal. The first sound information includes sound source object position information and an audio signal. The analysis unit has functionality equivalent to the path calculation unit 121 described above. The panning unit determines a representative direction from the representative point to the reference position based on the arrival direction calculated by the analysis unit, and performs panning processing to distribute the audio signal to the representative point (representative direction). In other words, by performing panning processing, the first sound information is converted into second sound information including position information of the representative point and a panned audio signal. The second terminal is equipped with a synthesis unit. The synthesis unit converts the direction to match the reference position with the coordinates and orientation of the user 99 detected by the second terminal, and performs convolution processing of the head-related transfer function based on the coordinates and orientation of the user 99.

[Operation]
Here, an example of operation of the stereophonic sound reproduction system 600 when processing is divided between the information processing device 601 and the audio presentation device 602 will be described.

The operation of the information processing device 601 will be described with reference to Figure 17A. Figure 17A is a flowchart showing an example of the operation of the information processing device 601 (first terminal) according to an embodiment. In the example of operation shown in the figure, an acquisition unit (not shown in Figure 15) acquires first sound information including information about the playback sound and information about the position of the sound source object via a communication module (step S101). Based on the first sound information, which is an input signal, the analysis unit uses a reference position to calculate the relative arrival direction from the position of the sound source object to the reference position. The analysis unit may, for example, identify the direct sound and one or more secondary sounds and calculate the propagation path from the sound source object position to the reference position for each sound. Based on the arrival direction calculated by the analysis unit, the panning unit determines a representative direction from the representative point to the reference position and performs panning processing to distribute the audio signal to the representative point (representative direction) (step S102). In other words, by performing panning processing, the first sound information is converted into second sound information including position information of the representative point and the panned audio signal. A transmitting unit (not shown in FIG. 15) transmits the second sound information to the audio presentation device 602 (second terminal) (step S103).

Next, the operation of the audio presentation device 602 (second terminal) will be described with reference to FIG. 17B. A receiving unit, not shown in FIG. 15, receives second sound information from the first terminal (step S104). Next, a sensing information input unit, not shown in FIG. 15, acquires information related to the user's position (at least one of position information and facial orientation information) (step S105). Next, a synthesis unit converts the direction to match the reference position with the determined user coordinates and orientation, performs convolution processing of the head-related transfer function based on the user coordinates and orientation (step S106), and synthesizes and outputs a sound signal (step S107).

[Specific example of panning processing]
To reiterate, in panning processing, reproduced sounds from multiple sound source objects are represented by representative sounds from multiple representative directions. For example, two or three directions can be used as these representative directions. Specifically, in panning processing, the number of representative points is reduced to a number less than the number of sound source objects, and the reproduced sounds can be perceived as sounds coming from the direction of arrival using only the head-related transfer functions of the representative directions for these representative points.

In this case, the panning process calculates the time shift (delay) that maximizes the cross-correlation between the head-related transfer function in the direction of arrival from the sound source object and the head-related transfer function in the representative direction. The time shift obtained here, or a time shift obtained by adding a negative sign to this time shift, is applied to the sound played back from the sound source object, and subsequent processing is performed assuming that the signal after the time shift is in the representative direction.

This time shift may also be permitted to be a time shift shorter than the sampling frequency (a shift in which the sample position is expressed as a decimal; hereafter referred to as a "decimal shift"). This decimal shift can be performed by oversampling.

Here, the panning process applies a gain to the signal of the representative direction obtained by time-shifting the sound played back from the sound source object, and then calculates the sum of these values calculated for each representative point, convolving the head-related transfer function at each representative point, thereby synthesizing a signal equivalent to the sound played back from the sound source object convolved with the head-related transfer function of the direction of arrival.

On the other hand, in panning processing, when synthesizing the head-related transfer function (vector) of the direction of arrival by adding up the head-related transfer functions (vector) of the representative direction, the gain can be calculated so that the error signal vector between the synthesized head-related transfer function (vector) and the head-related transfer function (vector) of the direction of arrival is orthogonal to the head-related transfer function (vector) of the representative direction. Note that a head-related transfer function (vector) is a time waveform of a head-related transfer function, which is an expression of the head-related transfer function in the time domain, considered as a vector. Hereinafter, this head-related transfer function (vector) will also be simply referred to as a "head-related transfer function vector".

In the panning process, this gain is corrected so that the energy balance of the head-related transfer functions from the position of the sound source object to the left and right ears of the user 99 is maintained even in the head-related transfer functions substantially synthesized by the panning process from head-related transfer functions from multiple representative points. In other words, in the panning process, the gain may be corrected so that the energy balance of the head-related transfer functions of the left and right ears of the user 99 due to the sound source object is maintained even in the head-related transfer functions substantially synthesized by the panning process.

In this embodiment, the panning process calculates, for each direction of arrival of the sound source object, a gain value to be multiplied by the head-related transfer function of the representative direction and a time shift value to be applied to the head-related transfer function of the representative direction, and stores these in a head-related transfer function table, which will be described later.

Then, in the panning process, each sound source object is time shifted using the time shift value and gain value corresponding to the direction from which each sound source object is coming, and then a gain is applied, and these are summed to create a sum signal. In the panning process, this sum signal is treated as being present at the position of the representative point. In the panning process, the head-related transfer function at the position of the representative point is convolved with this sum signal to generate a signal at the ear of the user 99.

Below, with reference to the flowchart in Figure 18, the panning process and the associated audio playback process will be explained in detail step by step. First, sound source and direction acquisition processing is performed (step S201). For example, the path calculation unit 121 acquires the direction of the sound source object as seen by the user 99.

Specifically, the acquisition unit acquires the audio signal (target signal) of the sound source object. This audio signal can have any sampling frequency and any quantization bit rate. In this embodiment, for example, an example will be described in which an audio signal with a sampling frequency of 48 kHz and a quantization bit rate of 16 bits is used. Furthermore, the path calculation unit 121 acquires directional information of the sound source object that is added to the audio signal of the content or the audio signals of the participants in the remote call.

The path calculation unit 121 then determines the spatial arrangement of the sound source object and the user 99. As described above, this arrangement may be within a space including a virtual space set in content, etc. The path calculation unit 121 then calculates the direction of the sound source object as seen by the user 99, i.e., the arrival direction, based on the determined arrangement within the space. Similarly, the path calculation unit 121 can also calculate the arrival direction for audio signals of content based on the arrangement of the user 99 by referring to directional information of the audio signal of the sound source object.

The path calculation unit 121 may also calculate the direction of the user 99 from the sound source object.

Next, the panning unit (generation unit 134 in the functional block diagram of Figure 5), which is a processing unit that executes the panning process, performs the panning process (step S202). Here, the panning unit uses directional information to perform panning on the sound source object. In this embodiment, the panning unit performs the panning process from the perspective of how closely the sound synthesized at the ear by the panning process can be made to resemble the sound that should be heard at the ear.

Figure 19 explains the calculations performed by the panning unit when panning the sound source object (sound source S-1) using representative points R-1 and R-2. In Figure 19, the signal to be panned is sound source S-1, but below, to calculate the optimal shift amount and optimal gain for this, calculations are performed using head-related transfer functions from sound source S-1, representative point R-1, and representative point R-2 to the ears.

In the example of Figure 19, the head-related transfer function with P sampling points (number of taps) from sound source S-1 to the ear is a P-dimensional vector. This is designated v{x} (in the following embodiments, vectors are represented as "v{ }").

Here, the panning unit defines the head-related transfer function from representative point R-1 to the ear of user 99 as v{x ₀₁ }, and the head-related transfer function from representative point R-2 to the ear as v{x ₀₂ }. The cross-correlation between v{x} and v{x ₀₁ } is calculated, and v{x ₁ } is calculated by time-shifting v{x ₀₁ } so that this is maximized. Similarly, the cross-correlation between v{x} and v{x ₀₂ } is calculated, and v{x ₂ } is calculated by time-shifting v{x ₀₂ } so that this is maximized.

This v{x ₁ } is multiplied by gain A, and v{x ₂ } is multiplied by gain B, and v{x} is approximated by the sum of these. In other words, v{x} is approximated as the approximate value of v{x} = A × v{x ₁ } + B × v{x ₂ }. This makes it possible to achieve panning processing with reduced error.

The details of this gain calculation and time shift will be explained. First, the gain calculation will be explained. The error vector resulting from the approximation of v{x} is shown in equation (1) below.

In the above formula (1), the arrows on the variables indicate that they are vectors. Here, when A and B are of optimal magnitude, that is, when the magnitude of the error vector is minimized, the error vector v{e} is orthogonal to the plane spanned by the original vectors v{x ₁ } and v{x ₂ }. Therefore, the relationship in the following formula (2) holds.

This results in the following equation (3):

Transforming this equation (3) gives us the following equation (4).

By performing the operation |v{x ₂ }| ² on the upper equation of equation (4) and v{x ₁ }·v{x ₂ } on the lower equation, the following equation (5) is obtained.

A can be calculated by subtracting the lower equation from the upper equation of equation (5) and eliminating B. This is shown in equation (6).

Therefore, gain A is expressed as follows:

Similarly, by eliminating gain A, gain B can be calculated as shown in equation (8) below.

In this way, gains A and B are determined so that the error vector between the composite signal and the target signal is orthogonal to the representative direction vector used.

The gains A and B obtained by this calculation are multiplied by the waveform of the head-related transfer function of v{x ₁ } after the time shift due to cross-correlation and the waveform of the head-related transfer function of v{x ₂ }, thereby enabling synthesis of the head-related transfer function to be output. In other words, these time shift amounts (time shift values) and gains A and B are applied to the sound source S-1 to perform panning processing.

Next, the specific calculation process of the time shift that maximizes cross-correlation will be described.In this embodiment, v{x} and v{x ₀₁ } are treated as vectors of the head related transfer functions of which the number of samples is P points.Therefore, the subscript of the time of head related transfer functions (position of sample points) can be explicitly written as shown in the following formula (9).

Then, we define the cross-correlation between the two vectors in equation (9) as a function of "k" as shown in equation (10) below.

Here, the k that gives the maximum value of φ _xx01 (k) is referred to as k _max01 . The panning unit calculates this k _max01 , for example, by substituting each value for k. Similarly, the k that gives the maximum value of φ _xx02 (k) is referred to as k _max02 . The panning unit calculates this k _max02 in the same way as k _max01 . Either k _max01 or k _max02 will be referred to simply as "k _max " below.

The panning unit stores the gains A and B and k _max01 and k _max02 calculated for the arrival direction of each sound source object that differs every 2° around 360° as gain values and time shift values in a head-related transfer function table, and uses these values in the output process described below. Note that it is also possible to perform only the audio output process described below using a head-related transfer function table in which the gains A and B and the time shifts k _max01 and k _max02 have already been calculated and stored.

Next, the panning unit and output unit perform audio output processing (step S203). First, the panning unit obtains a gain value and time shift value corresponding to the obtained direction of arrival from the head-related transfer function table for each sound source object. Then, the panning unit multiplies each sampling point (sample) of the waveform of that sound source object by this gain value.

In this case, the panning unit may correct the gain so that the energy balance of the left and right ear objects caused by the sound source object is maintained in the head-related transfer functions synthesized by the panning process. In other words, each gain value may be multiplied by an adjustment coefficient that matches the energy balance between the left and right head-related transfer functions with the original head-related transfer functions.

The panning unit then performs a time shift on the signal multiplied by this gain value.

The details of this time shift will be explained below: A vector v{x _{1 } obtained by shifting the elements of the vector v{x 01 }} by k _max samples is generated by the following procedure.

First, when the phase is advanced, that is, when k _max ≧0, zeros are set to the end of the vector for k _max samples, and the length of the vector is maintained. On the other hand, when the phase is delayed, that is, when k _max <0, zeros are set to the beginning of the vector for k _max samples, and the length of the vector is maintained. That is, it is set as shown in the following equation (11).

In this way, a time-shifted vector v{x ₁ } is generated. The positive or negative polarity of the time shift amount value is reversed depending on which is used as the reference when calculating the cross-correlation. Also, when convolving the head-related transfer function with the sound source signal, attention must be paid to the polarity of the time shift amount.

In addition, the panning unit can perform this time shift by a decimal multiple of the number of taps, rather than an integer multiple. It is also possible to perform a time shift before multiplying by a gain value.

The panning unit treats the signal that has been calculated in this way and that has undergone gain and time shifting as a representative point signal located at the position of representative point R. The panning unit then takes the sum of the representative point signals of the sound source objects that are grouped together at representative point R to generate a sum signal. The panning unit then convolves this sum signal with the head-related transfer function at the position of representative point R (head-related transfer function in the direction of the representative point) to generate a signal at the ear of the user 99.

The signal generated by the panning unit at the ear is then output and played back. This output may be, for example, a two-channel analog audio signal corresponding to the left and right ears of the user 99.

This makes it possible to play back audio signals corresponding to the virtual sound field as two-channel audio signals through headphones. This completes the audio playback process.

By configuring it as described above, the following effects can be achieved:

In recent years, when playing content such as movies, AR, VR, MR, and games using VR headphones or HMDs, there has been a demand for rendering technology (binauralization technology) that can properly describe and reproduce the entire 3D sound field. Conventional 3D stereophonic sound (binaural signals) is generated by individually convolving multiple sound source signals with the head-related transfer functions for the corresponding directions of arrival. In this way, convolving head-related transfer functions with individual sound source objects requires a huge amount of computation to follow human movement (6DoF: 6 Degrees of Freedom) with a high level of realism, which has been a problem.

On the other hand, in speaker panning processing, sound images were created between speakers by controlling the volume balance between speakers using the sine law, tangent law, etc. (sound source objects were localized). However, simply controlling the volume balance was not enough to properly reproduce a 3D sound image through headphones.

In contrast, the above audio playback process is characterized by the use of a path calculation unit 121 that acquires the direction of arrival of a sound source object, and a panning unit that expresses the sound source object by performing panning processing of sound from a specific representative direction based on the direction of arrival acquired by the path calculation unit 121 through time shifting and gain adjustment of the sound source object.

With this configuration, sound source objects are synthesized by panning the representative direction, reducing the number of arrival directions, enabling more efficient and effective rendering. This reduces the amount of calculation compared to conventional methods that individually convolve the head-related transfer function into the signal of each sound source object. In other words, the panning unit uses panning processing to equivalently synthesize head-related transfer functions of representative directions that approximate the arrival direction obtained by the path calculation unit 121, and can generate a head-related transfer function for the arrival direction. By reducing the amount of calculation in this way, the system can be applied to VR/AR apps for games, movies, etc. as a 3D sound field playback system. Furthermore, by applying it to smartphones and home appliances, the amount of calculation required to generate stereophonic sound can be reduced, resulting in cost savings. Furthermore, as a method with even lower calculation volume, it can be applied to international standardization, etc.

In the above-described embodiment, an example was described in which the panning unit represents the sound source signal by panning using representative points in two directions, left and right, i.e., an example in which a vector of a head-related transfer function in the left and right direction is used to equivalently synthesize a vector of a head-related transfer function in the direction of arrival. In other words, the above-described embodiment described an example in which the angular directions of the user 99 to the left and right are taken into account as directional information.

However, it is also possible to consider the vertical direction as the direction of arrival. Specifically, it is also possible to equivalently synthesize the vector of the head-related transfer function for the direction of arrival by interpolating the vectors of the head-related transfer functions for three directions. In other words, the panning unit can also perform panning processing using representative points in three directions, including the elevation angle direction (or depression angle direction).

In this case, similar to the interpolation from two directions, the head related transfer functions in the representative directions are time-shifted so as to maximize the cross-correlation with v{x}, and are expressed as vectors v{x ₁ }, v{x ₂ }, and v{x ₃ }. In this case, the error vector v{e} is expressed by the following equation (12).

Substitute this into equation (13) below and solve.

Specifically, the optimal gains A, B, and C can be calculated using the following formula (14):

In the above equation (14), the "-1" at the right shoulder of the matrix means the inverse matrix. The time shift amounts k _max01 , k _max02 , and k _max03 of the HRIR in the representative direction determined to maximize the cross-correlation are also calculated prior to the above gain value, similar to the values in the case of two directions.

Furthermore, in the above-described embodiment, an example was described in which two to four representative points R were used.

However, it is of course possible to use two or more representative points R. For example, as shown in the examples described below, it is possible to use four to six representative points R corresponding to range angles such as 90° and 60°. Furthermore, even in the case of four representative points, it is possible to set them at different positions, such as diagonally (45°, 135°, 225°, and 315°) or vertically and horizontally (0°, 90°, 180°, and 270°) relative to the user 99. It is also possible to select two or three of the four to six representative points R that are closest to the direction of arrival and use them as representative points R for synthesizing the sound source.

In other words, in the audio reproduction process, the panning process may use a gain calculated to minimize the energy or L2 norm of the error signal vector between the synthesized HRIR vector and the HRIR vector in the sound source direction.

[Weighting filter for time shift and gain calculation]
In addition, in the above example, the time shift and the gain that maximize the cross-correlation are calculated by using the head-related transfer function itself. On the other hand, the time shift and/or the gain may be calculated by applying a weighting filter on the frequency axis.

In other words, when calculating the time shift and gain that maximize the cross-correlation, it is possible to use a frequency-axis weighting filter (hereinafter also referred to as a "frequency weighting filter").

This frequency weighting filter preferably has a cutoff frequency near or slightly higher than the frequency band where human hearing sensitivity is high, and attenuates higher bands, i.e., bands where human hearing sensitivity decreases. For example, it is preferable to use a low-pass filter (LPF) with a cutoff frequency of 3000 Hz to 6000 Hz and a bandwidth of approximately 6 dB/oct (octave) to 12 dB/oct.

Specifically, since v{x} and v{x ₀₁ } treat the head-related transfer functions of P points as vectors, the time subscripts of the head-related transfer functions can be explicitly written and expressed as in the above formula (9). Here, the impulse response w _c (n) of the frequency weighting filter is convoluted with the two vectors of the above formula (9) and the length is truncated at P, as shown in the following formula (15).

Here, the operation "*" indicates convolution. Then, the cross-correlation between the two vectors in equation (15) is defined as a function of "k" as shown in equation (16) below.

Here, k that gives the maximum value of φ _xx01 (k) in equation (16) is denoted as k _max . The panning unit generates, for example, a vector v{x ₁ } obtained by shifting the elements of vector v{x ₀₁ } by k _max samples, in the same manner as in equation (11) above, by the following procedure.

Specifically, when the phase is advanced, that is, when k _max ≧0, the length of the vector is maintained by padding zeros to the end of the vector so that it contains k _max samples. In other words, when k _max ≧0, the vector v{x ₁ } becomes v{x ₁ }=(x ₀₁ (0+k _max ), x ₀₁ (1+k _max ), x ₀₁ (2+k _max ), ... x ₀₁ (P−1), ... 0, 0, 0).

Also, when the phase is delayed, that is, when k _max < 0, the vector is padded with zeros at the beginning to maintain the length of the vector so that it contains k _max samples. In other words, when k _max < 0, the vector v{x ₁ } becomes v{x ₁ } = (0, 0, 0, ..., x ₀₁ (0), x ₀₁ (1), x ₀₁ (2), ..., x ₀₁ (P-1+k _max )).

In the above, the vector v{x _01w } may be used as the vector v{x ₀₁ }. In this way, the vector v{x ₁ } can be generated. That is, similar to what has been described above, the cross-correlation can be calculated and used to calculate the time shift.

In addition, in the above-mentioned system, when calculating the error (similarity) between the synthesized head-related transfer function and the original head-related transfer function, A, B, and C that minimize |v{e}| ² of the error signal vector (error vector) v{e} are calculated as in the above-mentioned equation (12).

In this regard, v{e} may be obtained by applying a frequency weighting filter. Specifically, when v{e} is waveform data on the time axis, if v{e} is obtained by convolving an impulse response w(n) of a weighting filter with v{ _e }, then v{e _w } is expressed by the following equation (17).

The operator "*" indicates convolution. Here, the operator "*" is used on vectors, but this is the vector representation of the sequence obtained by convolving the sequence representations of the vectors on the left and right of the operator. In other words, v{x} * v{y} is the vector representation of the result of x(n) * y(n). Hereinafter, unless otherwise specified, the operator "*" on vectors will be treated in the same way.

Then, gains A, B, and C can be calculated by applying v{e _w } to the following equation (18) and solving it.

Alternatively, v{e} _w can be calculated equivalently by the following equation (19).

Using the time shift and gain calculated in this way, it is possible to distribute the target signal to a representative direction (panning process).

Note that the target signal to be panned and the head-related transfer function to be convolved may be the same as those described above. In other words, the target signal and the head-related transfer function to be convolved do not need to be convolved with a weighting filter.

By introducing this type of frequency weighting, it is possible to set the frequency band for approximation with smaller errors (higher accuracy). In particular, since the majority of energy in music and voice signals is concentrated in the low-frequency range, good performance can be achieved by using a weighting filter that weights the low-frequency side.

Furthermore, if the convolution of a weighting filter with impulse response w(n) and a vector is expressed as a convolution matrix W, where each row contains the weighting filter's impulse response w(n) time-shifted by one sample, then equation (17) can be transformed into equation (20) below.

Then, |v{e}| ² can be calculated using the following equation (21).

Here, W ^T represents the transposed matrix of W.

Furthermore, the weighting filter used when calculating the cross-correlation and when calculating the gain may have the same characteristics, or may have different characteristics. If the same filter is used, the weighting filter w may be convolved with the entire set of original head-related transfer functions, and then the time shift amount and gain may be calculated using the same processing as described above.

Furthermore, when using an LPF as a weighting filter to weight the low frequencies and calculate the cross-correlation and optimal gain as described above, if the effective band is limited to around 3000 Hz, the decimal shift described above does not need to be performed. In this case, oversampling is also not required.

In the above-described embodiment, the audio signal is panned and distributed to a plurality of representative directions, and the head-related transfer functions of each representative direction are convolved and expressed. Specifically, the head-related transfer function of the target direction is simulated by the sum of the head-related transfer functions of the representative directions, with the approximate value of v{x} in three directions = A × v{x ₁ } + B × v{x ₂ } + C × v{x ₃ }.

In such cases, the amplitude characteristics of the high frequencies of the HRTF tend to be lower in level than the original HRTF compared to the low frequencies. This is because even a slight time error caused by a slight misalignment of the listening point can cause a large phase rotation of the high frequency components of the HRTF, which tends to be canceled out by the addition caused by the panning process.

In contrast, in the audio playback process according to this embodiment, the tendency for high frequencies to attenuate may be compensated for using a playback high-frequency emphasis filter.

Specifically, it is possible to compensate for the tendency for high frequencies to attenuate by applying a high-frequency emphasis filter to the signal obtained by panning and convolving the head-related transfer function of the representative direction. Alternatively, equivalently, the head-related transfer function of the representative direction itself can be subjected to high-frequency emphasis filter processing in advance to emphasize the high frequencies. This high-frequency emphasis filter may be, for example, an impulse response weighting filter that emphasizes high frequencies by approximately +1 to +1.5 dB, with a turnover frequency of 5,000 to 15,000 Hz or higher.

In this way, by using a panning process to perform filtering that emphasizes the high frequencies of the synthesized audio, the perceived three-dimensionality can be further enhanced.

Even if a decimal shift similar to that described above is performed, mismatches in the high-frequency components of head-transmitted signals remain with the usual 8- to 16-fold oversampling, so a high-frequency emphasis filter may be applied.

In addition, in the panning process, the adjustment amounts for time shift adjustment and gain adjustment may be determined according to the head-related transfer functions contained in database 105, and the time shift adjustment and gain adjustment may be applied to the reproduced sound using the determined adjustment amounts to convert it into a representative sound. Since the optimal values for the adjustment amounts for time shift adjustment and gain adjustment used in the panning process change depending on the head-related transfer function, by first reading out the head-related transfer function contained in database 105 and determining the adjustment amounts for time shift adjustment and gain adjustment that match it, the same adjustment amounts can be reused thereafter as long as this head-related transfer function is used, which is advantageous in terms of processing volume.

The head-related transfer function table is an example of table data containing head-related transfer functions stored in the database 105. The head-related transfer function table stores the head-related transfer functions together with the adjustment amounts for time shift adjustment and gain adjustment determined according to the head-related transfer functions, with the amounts linked to each other. In other words, a head-related transfer function table may be constructed by calculating the adjustment amounts for time shift adjustment and gain adjustment in advance for each head-related transfer function included in the database 105. In this way, table data of the head-related transfer function table linking each head-related transfer function with the adjustment amount may be stored in the database 105. In this way, the database 105 is an example of a storage unit. The calculation of the adjustment amount for each head-related transfer function may be performed by the generation unit 134 or the decoding processing unit 113. Alternatively, the adjustment amount may be calculated by an external device and stored in the memory of the external device. In this case, the memory of the external device corresponds to an example of a storage unit.

Furthermore, the adjustment amounts for time shift adjustment and gain adjustment may be calculated in advance, and an adjustment amount table linked to each of multiple representative directions may be constructed and stored in database 105. The adjustment amount table may include table data linking the head-related transfer functions for each of multiple representative directions with the adjustment amounts for time shift adjustment and gain adjustment, or the head-related transfer functions for each of multiple representative directions may be extracted at the time of rendering or system initialization from head-related transfer functions for the entire celestial sphere (multiple directions) that have been acquired in advance and stored in database 105.

The adjustment amount table may also be a table that includes information on which of a plurality of representative directions to distribute a sound signal that arrives at the listener's position from the direction of each head-related transfer function in the spherical head-related transfer function database, and information on the time shift adjustment amount and gain adjustment amount to be multiplied by the sound signal for each representative direction when distributing the signal.

When performing the convolution process of the head-related transfer function, the adjustment amount table stored in the database 105 is referenced, and the adjustment amounts for the time shift adjustment and gain adjustment associated with the head-related transfer function of the direction to be applied are used. This eliminates the need to calculate the adjustment amount for each convolution process, contributing to a reduction in the amount of processing.

Note that embodiments of the present invention can also be applied to new head-related transfer functions that are not included in the database 105. When decoding a sound signal, when powering on the sound reproduction system 100, or when initializing the sound reproduction system 100, the head-related transfer functions of the entire three-dimensional sound field may be newly read, and the adjustment amount for each head-related transfer function may be calculated using the method disclosed in this embodiment or another method. In this case, table data linking the head-related transfer functions to the adjustment amounts may be stored in the database 105. Alternatively, the adjustment amounts may be calculated by an external device and stored in the memory of the external device. When performing convolution processing of head-related transfer functions, by referencing the adjustment amounts for time shift adjustment and gain adjustment linked to the applied head-related transfer function, it is not necessary to calculate the adjustment amount for each convolution processing, which can contribute to reducing the amount of processing.

In this way, when a new head-related transfer function not stored in database 105 is loaded, the adjustment amounts for the time shift adjustment and gain adjustment to be used in the panning process may be determined for the new head-related transfer function before storing it in database 105, and a head-related transfer function table may be constructed by linking the new head-related transfer function with the determined adjustment amounts, and the head-related transfer function table may be stored in database 105. Then, when performing panning process, these adjustment amounts are read from database 105, and shift adjustment and gain adjustment are applied based on these adjustment amounts. Note that the new head-related transfer function may be one that was previously stored in database 105, but was temporarily removed from database 105 when decoding a sound signal, when powering on sound reproduction system 100, or when initializing sound reproduction system 100, and then re-stored in database 105. Instead of the generation unit 134, a second generation unit may be provided that applies time shift adjustment and gain adjustment to the playback sound using adjustment amounts linked to a new head-related transfer function stored in the database to convert it into a representative sound, and generates an output sound signal by convolving the representative sound with a head-related transfer function corresponding to a representative direction from each representative point toward the user's position.

Below, when using a stereophonic playback system 600 divided into an information processing device 601 and an audio presentation device 602, the above-mentioned panning process may result in degradation of sound quality. Specifically, the panning process described above involves performing time shift adjustment and gain adjustment. When the user 99 is facing a fixed direction, the time shift adjustment and gain adjustment are performed to allow the user 99 facing a fixed direction to perceive a sound source object located at a position other than the representative direction by convolving a head-related transfer function with sounds from multiple preset representative directions. More specifically, when the user 99 is facing a 0° direction and a stationary sound source object located at a 60° direction is generated by outputting an output sound signal in which a head-related transfer function is convolved with sounds from representative directions set at 30° and 90°, an output sound signal in which a sound image of the sound source object is formed at 60° can be generated by multiplying the sound from the 30° representative direction and the sound from the 90° representative direction by the time shift value and gain value calculated for 30° and 90°, and convolving the 30° and 90° head-related transfer functions.

However, when the user 99 rotates their head by -30 degrees, the positional relationship is determined by convolving a 60° head-related transfer function with the time shift value and gain value calculated for 30 degrees, and convolving a 120° head-related transfer function with the time shift value and gain value calculated for 90 degrees. In this case, the user 99 may not perceive the sound source object as being exactly at a 90° position, with 0° being the front of the user 99 after head rotation. In other words, the time shift value and gain value calculated for 30 degrees are calculated assuming the convolution of a 30° head-related transfer function, and therefore are not appropriate for the convolution of a 60° head-related transfer function, and the time shift value and gain value calculated for 90 degrees are calculated assuming the convolution of a 90° head-related transfer function, and therefore are not appropriate for the convolution of a 120° head-related transfer function. As a result, the user 99 may misperceive the localized position of the sound source object.

This is particularly noticeable when perceiving the position of a sound source object when the sound contains an elevation component.

For example, a case will be described in which six representative points (representative directions) arranged horizontally as shown in Figure 20 are used. Figure 20 is a diagram for explaining the arrangement of representative directions in this embodiment. Figure 20 shows that representative directions R1, R2, R3, R4, R5, and R6 are arranged at positions of 30°, 90°, 150°, 210°, 270°, and 330°, respectively, horizontally surrounding the user 99. When a sound source object is in area A1, an output sound signal is output using representative directions R1 and R2. In other words, the reproduced sound emitted by a sound source object in area A1 is distributed to representative directions R1 and R2, and a 30° head-related transfer function is convolved with the sound distributed to representative direction R1, and a 90° head-related transfer function is convolved with the sound distributed to representative direction R2. Conversely, the reproduced sound of a sound source object within a 120° range between 330° and 90° is distributed to representative direction R1.

21 and 22 (a) show the case where, as in the above example, time shift values are calculated so that the cross-correlation between head-related transfer functions is maximized. Figures 21 and 22 are diagrams for explaining an example of time shift value calculation according to an embodiment. Here, the time shift values (vertical axis) are shown for each direction (horizontal axis) of the sound source object after head rotation when the user 99 rotates their head and the angle shown at the right shoulder in each figure is used as the representative direction. Note that Figure 21 shows an example of the left ear, and Figure 22 shows an example of the right ear. As shown in (a) of each figure, there are discontinuities in the time shift values at some point within the 360° range, which is presumed to be the cause of the loss of sound localization. Therefore, the results of examining the calculation direction of the time shift value to reduce the occurrence of such discontinuities are shown in (b) of each of Figures 21 and 22.

Here, since the head-related transfer function of the representative direction itself is convolved in the representative direction, the time shift value can be set to 0. Based on this premise, when calculating the time shift value for a predetermined position to prevent value jumps, the time shift value of an adjacent position closer to the representative direction than the predetermined position is used. More specifically, the time shift for the representative direction is set to 0, and the time shift value for the position closest (adjacent) to the representative direction is calculated. The calculation is performed so that this calculated value is closer to the time shift value for the representative direction, 0, than the time shift value when the above cross-correlation is at its maximum.

More specifically, when calculating the time shift value for a predetermined position, the time shift value of the peak closest to the time shift value of a position adjacent to the predetermined position is calculated as the time shift value for the predetermined position, among the several peaks indicated by the cross-correlation. By doing so, it is less likely that the time shift value will form a jump, as shown in Figures 21 and 22. When calculating the time shift value for a predetermined position, the time shift value of the peak closest to the time shift value of a position adjacent to the predetermined position, among the several peaks indicated by the cross-correlation, within a threshold range based on the maximum peak, is calculated as the time shift value for the predetermined position. The threshold range may be, for example, within a coefficient multiple of the correlation value at the maximum peak (the coefficient is less than 1). If the coefficient is set to, for example, 0.8, a time shift value that is relatively close, within 0.8 times the correlation value at which the cross-correlation is maximum, and close to the time shift value of the adjacent position, is obtained.

Next, we will explain how to calculate gain values when the time shift value is determined as described above. Figure 23 is a diagram for explaining an example of gain value calculation according to an embodiment. As shown in Figure 23, in reality, the arrival direction of a sound source object is surrounded by three representative directions in three-dimensional space. In other words, by multiplying each of the three representative directions by a certain gain and distributing the sound, the sound source object can be localized in the arrival direction using sounds from the three representative directions. Here, the three gains may be calculated as shown in equations (12) to (15). However, for example, it is also possible to calculate gain values for localizing the sound source object in the arrival direction in two of the three representative directions as follows, and then calculate the gain value for the remaining representative direction.

Here, the representative direction closest to the zenith is excluded from the two representative directions selected initially. In this way, the gain values of the two representative directions close to the horizontal direction are determined first, and the gain values of the two representative directions close to the horizontal direction are emphasized, and the gain value of the representative direction closest to the zenith is determined accordingly. As already explained, the sense of localization of the user 99 in the horizontal direction is relatively more stable than in the vertical direction, so if the gain values of the two representative directions close to the horizontal direction are emphasized and the gain value of the representative direction closest to the zenith is determined accordingly, it is easy to suppress the impact of the gain value of the representative direction closest to the zenith on deteriorating the sense of localization.

Specifically, the calculation is performed as follows: First, to determine the gain values for the two representative directions, solve the following equation (22).

_G1 is the gain value for one of the two representative directions, and _G2 is the gain value for the other of the two representative directions.

Then, while maintaining the relationship (i.e., ratio) between _G1 and _G2 , the gain value of the representative direction closest to the zenith is determined by solving the following equation (23).

α is the gain value for a resultant vector obtained by multiplying two representative directions by their respective gain values and then summing the results, and _Gt is the gain value for the representative direction closest to the zenith.

By doing the above, the effects shown in Figures 24 and 25, for example, were confirmed. Figures 24 and 25 are diagrams showing the results of gain value calculations according to the embodiment. Figure 24 shows the results of plotting the SNR at an elevation angle of 0°, and Figure 25 shows the results of plotting the SNR at an elevation angle of 46°. In Figures 24 and 25, Comparative Example 1 shows the SNR calculated using the calculation method already described, Comparative Example 2 shows the SNR using a different calculation method, and Example 1 shows the SNR calculated using a method in which gain values for localizing a sound source object in the arrival direction in the two representative directions are calculated, and then a gain value for the remaining representative direction is calculated; in each case, the larger the value, the better the result.

As shown in the figure, at an elevation angle of 0°, there was no significant difference between Comparative Example 1 and Example 1, but both Comparative Example 1 and Example 1 showed better results than Comparative Example 2. At an elevation angle of 46°, Example 1 showed better results than Comparative Examples 1 and 2, especially as the rotation angle approached 90°.

Next, another example of the calculation method of time shift value and gain value will be described.Hereinafter, set the head related transfer function vector of horizontal position, which is a virtual position in the horizontal direction, and calculate the cross-correlation with the head related transfer function vector of two representative directions, excluding the representative direction closest to the zenith, that is, close to the horizontal direction, in the same way as above, and obtain the time shift values _S1 and _S2 that maximize correlation value.Then, use this head related transfer function vector that applies time shift to solve the following formula (24) to calculate the gain values _G1 and _G2 of two representative directions.

Furthermore, a new vector is defined as shown in the following equation (25), and the cross-correlation between the arrival direction and the representative direction closest to the zenith and the defined new vector is calculated to determine the time shift values S _t and S _x that maximize the correlation values, respectively.

Then, using the head-related transfer function vector to which this time shift has been applied, the representative direction closest to the zenith and the gain value of the newly defined vector are calculated by solving the following equation (26).

As a result, the time shift values and gain values in the three representative directions can be obtained using the following equation (27).

By doing this, it is possible to obtain time shift values and gain values for the horizontal representative direction, in which the user 99's sense of localization is relatively stable, so that the sense of localization is less likely to be impaired even if a head-related transfer function different from the head-related transfer function assumed by the information processing device 601 is convolved into the representative direction signal by the audio presentation device 602 as the user's head rotates.

It can be seen that the time shift and gain values calculated in the manner described above have a slight numerical discrepancy when compared to when the angle of the user's 99 head after rotation is replaced with the unrotated angle.

The following example (Example 2) generates a sound in a 75° direction after a 60° rotation by convolving the head-related transfer functions for the representative directions of 30° and 90° with the time shift and gain values calculated using equations (24) to (27) assuming an arrival direction of 15° and representative directions of 30° and 330°, and assuming a 60° rotation of the head, onto the sound distributed in these representative directions. This example is compared with the example (Comparative Example 3) in which the time shift and gain values calculated assuming an arrival direction of 75° and representative directions of 30° and 90° are used to simply convolve the head-related transfer functions for the representative directions of 30° and 90° onto the sound distributed in these representative directions. The results are shown in Figures 26 and 27. Figure 26 is a diagram for verifying the results of the time shift value calculation according to the embodiment. Figure 27 is a diagram for verifying the results of the gain value calculation according to the embodiment.

26 and 27, the dashed lines indicate the calculation results for Comparative Example 3, and the solid lines indicate the calculation results for Example 2. As shown in FIGS. 26 and 27, there is a slight error between the time shift values and gain values in Example 2 and Comparative Example 3. In other words, it is clear that there is an error between the actual correct values. Therefore, for example, in order to reduce such errors, the time shift values and gain values calculated in Example 2 may be corrected so as to reduce the expected value of the error. Specifically, for example, the time shift values and gain values may be corrected by reducing the average value of the errors at each representative point. Alternatively, when calculating the expected value, the expected value may be calculated so as to weight the direction with the larger gain value, and the above correction may be performed. In other words, the direction in which the signal is concentrated within a 120° range distributed in a certain representative direction can be estimated from the distribution of gain values, and the correction amounts for the time shift values and gain values may be determined assuming that the signal is in that direction.

Incidentally, the following method can be considered as a way to suppress the deterioration in sound quality caused by the above-mentioned panning process when using a stereophonic reproduction system 600 that is divided into an information processing device 601 and an audio presentation device 602.

As mentioned above, the degradation in sound quality occurs when gain and shift values that differ from the gain and shift values that should have been used. Therefore, if the gain and shift values used are corrected so that they are closer to the gain and shift values that should have been used, it can be said that this degradation in sound quality can be suppressed. The following explanation focuses specifically on gain values, but the same can be applied to shift values as well.

For example, consider a sound source object whose direction in the horizontal plane (i.e., azimuth direction or bearing) is 15°. In the horizontal plane, the representative directions are set to -150°, -90°, -30°, 30°, 90°, and 150°.

When this sound source object is expressed as a representative sound from a representative direction, for example, a representative sound panned to representative directions of -30° and 30° is generated. The gain values at this time are set to a gain value that takes into account the representative direction of -30° and a gain value that takes into account the representative direction of 30°. However, if the user 99 rotates their head by 60°, the sound source object will change to a position of -45°, and the representative directions will be -90° and -30°. However, at that time, the gain values that take into account the representative direction of -30° and a gain value that takes into account the representative direction of 30° are set, resulting in a degradation in sound quality.

Therefore, as the gain value to be set for the sound source object, it is advisable to select multiple directions (virtual representative points) within a range of, for example, ±60°, starting from -30° and 30°, i.e., -90° to 30° and -30° to 90°, and calculate gain values for each. A single gain value (hereinafter referred to as a global gain value) can then be set using all or several of these gain values. For example, if multiple directions are selected in 5° increments within the range of -90° to 30°, gain values for 25 directions, i.e., -90°, -85°, -80°, ..., 20°, 25°, and 30°, can be calculated. If a single gain value using all 25 gain values is then set as the gain value for the sound source object, even if the head is rotated 60° as described above, the global gain value will already include the gain value for -90°, which reduces degradation in sound quality compared to simply using a gain value for -30°. Even in the range of -30° to 90° starting from 30°, the gain value for -30° is used in the calculation of the global gain value, so deterioration in sound quality can be suppressed compared to simply using the gain value for 30°.

Also, consider the case where the head rotation amount is 120°. The sound source object changes to a position of -105°, and -150° and -90° are used as the representative directions. The global gain value takes into account gain values outside the ranges of -150° and -90°, from -90° to 30° and -30° to 90°. Even in this case, because gain values up to -90° and -30° are taken into account, deterioration in sound quality can be suppressed compared to simply using gain values for -30° and 30°. Alternatively, the range considered from the starting direction can be set to ±120°, so that even when the head rotation amount is 120°, the gain value for the representative direction used at that time is included in the global gain value. In other words, the range to be considered from the direction of the starting point needs to be less than ±180°, and can be arbitrarily set to ±170°, ±160°, ±150°, ±140°, ±130°, ±120°, ±110°, ±100°, ±90°, ±80°, ±70°, ±60°, ±50°, ±40°, ±30°, ±20°, ±10°, and ±5°, etc.

Furthermore, within these ranges, the density of the directions selected is not limited to 5° increments. For example, the density of the directions selected within the ranges can be set arbitrarily, such as 10° increments, 9° increments, 8° increments, 7° increments, 6° increments, 5° increments, 4° increments, 3° increments, 2° increments, 1° increments, or increments of less than 1°. As an example, the density of the directions selected within the ranges only needs to match the density of the selectable directions in the set of head-related transfer functions that has been loaded. However, since the denser the density of the directions selected within the ranges, the greater the amount of calculation processing required, the density of the directions selected within the ranges may be set to a density of directions that has been thinned out from the density of the selectable directions in the set of head-related transfer functions that has been loaded at a rate that corresponds to the processing power.

The following describes a method for calculating a global gain value and its effects based on an example. In the following example, an example is described in which the density of directions selected within a range of ±60° from the starting point is in 5° increments. The global gain value is calculated, for example, as the average of the gain values for each of the multiple directions selected within the range. This average may be a simple average, or a weighted average in which smaller rotation angles, which are thought to occur more frequently, approach 1 and larger rotation angles, which are thought to occur less frequently, approach 0. Therefore, this weighted average can be considered a weighted average based on the angular difference between each direction of the virtual representative points and their corresponding original directions. More specifically, the global gain value may be a weighted average calculated so that the weight assigned to the adjustment amount at virtual representative points with small angular differences is greater than the weight assigned to the adjustment amount at virtual representative points with large angular differences. Alternatively, instead of averaging, a single gain value obtained by performing a calculation to minimize the error vector as described above for all selected virtual representative points may be used as the global gain value.

Below is an example in which the global gain value is calculated by averaging the two representative points that horizontally sandwich the position of the sound source object. However, calculating the global gain value in this manner is not limited to the two points that sandwich the sound source object, and can also be performed for three or more representative points that surround the sound source object.

Two global gain values corresponding to two representative directions horizontally sandwiching a sound source object can be calculated using the following equation (28).

Note that φ indicates one of the two representative directions, and α indicates the difference between the representative direction and the direction of the sound source object, with the direction of the sound source object (target direction) being φ-α. Note that the other of the two representative directions is expressed as φ-60°. Equation (28) can be made into the average value of gain values in 5° increments by replacing M with M = 5k (-12≦k≦12). Equation (29) below is obtained by replacing equation (28) with M = 5k (-12≦k≦12).

The results of a localization experiment using the global gain values calculated based on equation (29) are shown in Figures 28 to 31. Figures 28 to 31 show the results of a localization experiment conducted to verify the results of the gain value calculations according to the embodiment. Note that below, an example using global gain values is shown as an example, and a comparative example simply using gain values before head rotation is shown.

Figure 28 shows the relationship between the front-to-back error rate between the direction (azimuth) perceived by the subject and the actual direction (azimuth) for a sound source object at an elevation angle of 0°, i.e., in the horizontal plane, and the rotation angle of the subject's head. As shown in Figure 28, the example showed a smaller error rate than the comparative example, especially when the head was rotated.

Furthermore, Figure 29 shows the relationship between the average localization error between the direction (azimuth) perceived by the subject and the actual direction (azimuth) for a sound source object at an elevation angle of 0°, and the rotation angle of the subject's head. As shown in Figure 29, the example showed a smaller error than the comparative example, especially when the head was rotated.

Furthermore, Figure 30 shows the relationship between the front-to-back error rate between the direction (azimuth) perceived by the subject and the actual direction (azimuth) for a sound source object at an elevation angle of 40°, and the rotation angle of the subject's head. As shown in Figure 30, the example showed a smaller improvement in error rate than the comparative example.

Furthermore, Figure 31 shows the relationship between the average localization error between the direction (azimuth) perceived by the subject and the actual direction (azimuth) for a sound source object at an elevation angle of 40°, and the rotation angle of the subject's head. As shown in Figure 31, the example showed a smaller effect of reducing the error than the comparative example.

In particular, based on the results of the localization experiment on the sound source object at an elevation angle of 40° shown in Figures 30 and 31, we predicted that the gain value of the representative direction in the elevation angle direction (for example, the representative direction of the zenith), in other words, the representative point in relation to the elevation angle (elevation angle representative point), reduces the effect of using the global gain value, and devised the following improvement.

In other words, if the gain value of the elevation representative point (the representative point containing a vertical component among the three or more representative points surrounding the sound source object) is set to a constant value (fixed gain value) regardless of the direction of the sound source object, it is thought that the effect of the elevation representative point on the sound localization in the horizontal plane relative to the sound source object can be suppressed. The same is thought to be true for the depression representative point on the nadir side opposite the zenith, so here we will only focus on the elevation representative point.

The gain and shift values for the elevation representative points contain components that affect horizontal sound localization during the calculation process. Therefore, even if the global gain value is used to adjust the gain value to be used in the horizontal direction as described above, the effect of using the global gain value is thought to be reduced because the gain and shift values for the elevation representative points maintain the gain and shift values corresponding to the direction before head rotation. Therefore, it is thought to be effective to divide the calculation into stages: in the first stage, the gain value for the elevation representative points is set to maintain a predetermined value regardless of the direction of the sound source object (target direction), and then in the second stage, the global gain value is set corresponding to that predetermined gain value. In other words, by calculating the gain value for the elevation representative points separately from the gain values for representative points in the horizontal plane, the effect of the gain value for the elevation representative points on horizontal sound localization is reduced. This is effective not only when global gain values are used, but also in applications that involve head rotation, especially when the elevation component is included. In other words, it is also effective to use a fixed gain value as the gain value for the elevation angle representative point in the separate gain value adjustments mentioned above.

Below, we will explain in detail how to set the fixed gain value and show the results of a localization experiment using the set fixed gain value as an example.

If the fix gain value is set so that it does not depend on the direction of the sound source object in the horizontal plane, but is also set so that it does not depend on the direction of the sound source object in the vertical plane, it will be difficult to position the sound source object in the elevation direction. In other words, the fix gain value needs to be set so that it depends on the direction of the sound source object in the vertical plane. Therefore, we set appropriate fix gain values for each of the sound source objects in several directions in the vertical plane. If the sound source object is in the horizontal plane (elevation angle 0°), the fix gain value needs to be 0, and if the sound source object is in the zenith direction (elevation angle 90°), the fix gain value needs to be 1. In other words, the fix gain value needs to be a value that gradually increases between 0 and 1 as the direction of the sound source object in the vertical plane changes from 0° to 90°.

Below, we consider the range where the elevation angle of the sound source object is greater than 0° and less than 90°. When the sound source object has an elevation angle of 20° and the fixed gain values are set to 0.05, 0.10, 0.15, and 0.30, the phase shift was calculated by taking the azimuth shift value that maximizes the cross-correlation between the ILD (Interaural Level Difference) in the head related transfer function at the original (i.e., the appropriate gain value that matches the angle after rotation) and the ILD in the head related transfer function with the fixed gain value. As a result, the gain value that minimizes the phase shift was adopted as the fixed gain value when the sound source object has an elevation angle of 20°. As a result, the fixed gain value when the sound source object has an elevation angle of 20° was set to 0.10.

Similarly, experiments were conducted by setting several fixed gain values when the sound source object had an elevation angle of 40°, 60°, and 80°, and fixed gain values of 0.15, 0.30, and 0.60 were set. Figure 32 is a diagram showing the fixed gain values set in this embodiment. As shown in the figure, it is appropriate for the fixed gain value to be set to a relatively small value when the elevation angle of the sound source object is small (for example, an elevation angle of 40° or less), while it is appropriate for the value to increase rapidly when the elevation angle is large (for example, an elevation angle of 60° or more). In other words, it can be said that it is more appropriate for the fixed gain value to curve exponentially rather than being simply proportional to the elevation angle of the sound source object.

Figures 33 and 34 show the results of localization experiments conducted to verify the results of gain value calculations according to the embodiment. Note that the following examples show a case in which an appropriate fixed gain value is used in combination with a global gain value, and a comparative example shows a case in which only the global gain value is used.

Figure 33 shows the relationship between the front-to-back error rate between the direction (azimuth) perceived by the subject and the actual direction (azimuth) for a sound source object at an elevation angle of 40°, and the angle of rotation of the subject's head. As shown in Figure 33, the example showed a smaller error rate than the comparative example, especially when the head was rotated.

Furthermore, Figure 34 shows the relationship between the average localization error between the direction (azimuth) perceived by the subject and the actual direction (azimuth) for a sound source object at an elevation angle of 40°, and the rotation angle of the subject's head. As shown in Figure 34, the example shows that the error is smaller than the comparative example, especially when the head is rotated. In this way, by using a combination of fixed gain values, it is possible to localize the sound source object more appropriately than when simply using global gain values.

When using head-related transfer functions of elevation and depression representative points, instead of using fixed gain values as described above, it is also possible to correct the initial delay according to the head rotation angle. The initial delay can be corrected by adjusting the initial delay of the head-related transfer functions of the elevation and depression representative points between the ears so that a change equal to the change in ITD (Interaural Time Difference) when a sound directly in front of the listener (azimuth 0°) moves due to head rotation is applied to the ITD of the head-related transfer functions of the elevation and depression representative points. The amount of adjustment here varies depending on the direction of the sound source object, but the directions of representative sound source objects can be calculated in advance and stored as a table on the receiving side (i.e., on the audio presentation device 602 side), so that the table can be referenced, selected, and used according to the estimated sound source direction.

(Other embodiments)
Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

For example, the information processing system or sound reproduction system described in the above embodiments may be realized as a single device that includes all of the components, or may be realized by allocating each function to multiple devices and working together. In the latter case, the device corresponding to the information processing device may be an information processing device such as a smartphone, tablet terminal, PC, or base station. For example, in the sound reproduction system 100 that functions as a renderer that generates sound signals with added sound effects, all or part of the renderer's functions may be performed by a server. In other words, all or part of the acquisition unit 111, path calculation unit 121, output sound generation unit 131, and signal output unit 141 may reside on a server (not shown). In this case, the sound reproduction system 100 is realized by combining, for example, an information processing device such as a computer or smartphone, a sound presentation device such as a head-mounted display (HMD) or earphones worn by the user 99, and a server (not shown). The computer, sound presentation device, and server may be connected to each other so that they can communicate with each other via the same network, or may be connected via different networks. When connected via different networks, there is a high possibility of communication delays, so processing on the server may be permitted only when the computer, sound presentation device, and server are connected so that they can communicate via the same network. Furthermore, whether the server will take on all or part of the functions of the renderer may be determined depending on the amount of bitstream data accepted by the sound reproduction system 100.

Furthermore, the information processing system or sound reproduction system disclosed herein can also be realized as an information processing device that is connected to a reproduction device equipped with only a driver and that simply reproduces an output sound signal generated based on acquired sound information for the reproduction device. In this case, the information processing device may be realized as hardware equipped with dedicated circuits, or as software that causes a general-purpose processor to execute specific processes.

Furthermore, in the above embodiments, the processing performed by a specific processing unit may be performed by another processing unit. Furthermore, the order of multiple processes may be changed, or multiple processes may be performed in parallel.

Furthermore, in the above embodiments, each component may be realized by executing a software program appropriate for that component. Each component may also be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

Furthermore, general or specific aspects of the present disclosure may be realized as a system, device, method, integrated circuit, computer program, or computer-readable recording medium such as a CD-ROM. Furthermore, general or specific aspects of the present disclosure may be realized as any combination of a system, device, method, integrated circuit, computer program, and recording medium.

For example, the present disclosure may be realized as an information processing method or audio signal reproduction method executed by a computer, or as a program for causing a computer to execute an information processing method or audio signal reproduction method. The present disclosure may also be realized as a computer-readable non-transitory recording medium on which such a program is recorded.

In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would conceive, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope of the spirit of this disclosure.

In this disclosure, the encoded sound information can be rephrased as a bitstream containing a sound signal, which is information about a specific sound to be reproduced by the sound reproduction system 100, and metadata, which is information about the localization position when the sound image of the specific sound is localized at a specific position within a three-dimensional sound field. For example, the sound information may be acquired by the sound reproduction system 100 as a bitstream encoded in a specific format such as MPEG-H 3D Audio (ISO/IEC 23008-3). As an example, the encoded sound signal contains information about the specific sound to be reproduced by the sound reproduction system 100. The specific sound here refers to a sound emitted by a sound source object present in the three-dimensional sound field or a natural environmental sound, and may include, for example, mechanical sounds or the sounds of animals, including humans. In addition, if there are multiple sound source objects in the three-dimensional sound field, the sound reproduction system 100 will acquire multiple sound signals corresponding to the multiple sound source objects.

Meanwhile, metadata is information used, for example, to control acoustic processing of sound signals in the sound reproduction system 100. Metadata may also be information used to describe a scene expressed in a virtual space (three-dimensional sound field). Here, a scene is a term that refers to the collection of all elements representing three-dimensional images and acoustic events in a virtual space, modeled by the sound reproduction system 100 using metadata. In other words, the metadata here may include not only information that controls acoustic processing, but also information that controls video processing. Of course, the metadata may include information that controls only either audio processing or video processing, or information used to control both. In the present disclosure, the bitstream acquired by the sound reproduction system 100 may include such metadata. Alternatively, the sound reproduction system 100 may acquire the metadata separately from the bitstream, as described below.

The sound reproduction system 100 generates virtual sound effects by performing sound processing on the sound signal using metadata contained in the bitstream and additionally acquired position information of the interactive user 99. For example, sound effects such as early reflection sound generation, late reverberation sound generation, diffraction sound generation, distance attenuation effect, localization, sound image localization processing, or Doppler effect may be added. Information for switching all or some of the sound effects on and off may also be added as metadata.

Note that all or some of the metadata may be obtained from sources other than the audio information bitstream. For example, either the metadata controlling the audio or the metadata controlling the video may be obtained from sources other than the bitstream, or both may be obtained from sources other than the bitstream.

Furthermore, if metadata for controlling video is included in the bitstream acquired by the audio reproduction system 100, the audio reproduction system 100 may have a function for outputting the metadata that can be used to control video to a display device that displays images or a 3D video reproduction device that reproduces 3D video.

Also, as an example, the encoded metadata includes information about a three-dimensional sound field including sound source objects and obstacle objects that emit sound, as well as information about the localization position when the sound image of the sound is localized at a specific position within the three-dimensional sound field (i.e., the sound is perceived as arriving from a specific direction), i.e., information about the specific direction. Here, an obstacle object is an object that can affect the sound perceived by the user 99 by, for example, blocking or reflecting the sound emitted by the sound source object before it reaches the user 99. Obstacle objects can include not only stationary objects, but also animals such as people, or moving objects such as machines. Furthermore, when multiple sound source objects exist in a three-dimensional sound field, the other sound source objects can be obstacle objects for any sound source object. Furthermore, both non-sound source objects such as building materials or inanimate objects and sound source objects that emit sound can be obstacle objects.

The spatial information constituting the metadata may include not only the shape of the three-dimensional sound field, but also information representing the shape and position of obstacle objects present in the three-dimensional sound field, and the shape and position of sound source objects present in the three-dimensional sound field. The three-dimensional sound field may be either a closed or open space, and the metadata includes information representing the reflectance of structures that can reflect sound in the three-dimensional sound field, such as floors, walls, or ceilings, and the reflectance of obstacle objects present in the three-dimensional sound field. Here, reflectance is the ratio of the energy of reflected sound to incident sound, and is set for each frequency band of sound. Of course, the reflectance may also be set uniformly, regardless of the frequency band of the sound. Furthermore, when the three-dimensional sound field is an open space, parameters such as uniform attenuation rate, diffracted sound, or early reflected sound may be used.

In the above explanation, reflectance was mentioned as a parameter related to an obstacle object or sound source object included in the metadata, but the metadata may also include information other than reflectance. For example, metadata related to both sound source objects and non-sound source objects may include information about the material of the object. Specifically, the metadata may include parameters such as diffusion rate, transmittance, or sound absorption rate.

Information about sound source objects may include volume, radiation characteristics (directivity), playback conditions, the number and type of sound sources emitted from an object, or information specifying the sound source area within the object. Playback conditions may, for example, determine whether the sound will play continuously or trigger an event. The sound source area within an object may be determined relative to the position of the user 99 and the position of the object, or may be determined based on the object. When determined relative to the position of the user 99 and the position of the object, the surface from which the user 99 is viewing the object is used as the reference, and the user 99 can be made to perceive that sound X is coming from the right side of the object and sound Y is coming from the left side as viewed from the user 99. When determined based on the object as the reference, it is possible to fix which sound is coming from which area of the object, regardless of the direction the user 99 is looking. For example, the user 99 can be made to perceive that a high-pitched sound is coming from the right side and a low-pitched sound is coming from the left side when viewing the object from the front. In this case, if the user 99 goes around to the back of the object, the user 99 can be made to perceive that a low-pitched sound is coming from the right side and a high-pitched sound is coming from the left side as viewed from behind.

Spatial metadata can include the time to early reflections, reverberation time, or the ratio of direct sound to diffuse sound. If the ratio of direct sound to diffuse sound is zero, the user 99 will only perceive direct sound.

Furthermore, information indicating the position and orientation of the user 99 in the three-dimensional sound field may be included in the bitstream as metadata as an initial setting, or it may not be included in the bitstream. If information indicating the position and orientation of the user 99 is not included in the bitstream, the information indicating the position and orientation of the user 99 is obtained from information other than the bitstream. For example, position information of the user 99 in the VR space may be obtained from an app that provides VR content, and position information of the user 99 for presenting sound as AR may be position information obtained by a mobile device performing self-position estimation using GPS, a camera, LiDAR (Laser Imaging Detection and Ranging), or the like. Note that the sound signal and metadata may be stored in a single bitstream, or may be stored separately in multiple bitstreams. Similarly, the sound signal and metadata may be stored in a single file, or may be stored separately in multiple files.

When an audio signal and metadata are stored separately in multiple bitstreams, information indicating other related bitstreams may be included in one or some of the multiple bitstreams in which the audio signal and metadata are stored. Furthermore, information indicating other related bitstreams may be included in the metadata or control information of each bitstream of the multiple bitstreams in which the audio signal and metadata are stored. When an audio signal and metadata are stored separately in multiple files, information indicating other related bitstreams or files may be included in one or some of the multiple files in which the audio signal and metadata are stored. Furthermore, information indicating other related bitstreams or files may be included in the metadata or control information of each bitstream of the multiple bitstreams in which the audio signal and metadata are stored.

Here, the related bitstreams or files are, for example, bitstreams or files that may be used simultaneously during audio processing. Furthermore, information indicating other related bitstreams may be described collectively in the metadata or control information of one bitstream out of multiple bitstreams storing audio signals and metadata, or may be described separately in the metadata or control information of two or more bitstreams out of multiple bitstreams storing audio signals and metadata. Similarly, information indicating other related bitstreams or files may be described collectively in the metadata or control information of one file out of multiple files storing audio signals and metadata, or may be described separately in the metadata or control information of two or more files out of multiple files storing audio signals and metadata. Furthermore, a control file that collectively describes information indicating other related bitstreams or files may be generated separately from the multiple files storing audio signals and metadata. In this case, the control file does not have to store the audio signal and metadata.

Here, the information indicating the other related bitstream or file may be, for example, an identifier indicating the other bitstream, a file name indicating the other file, a URL (Uniform Resource Locator), or a URI (Uniform Resource Identifier). In this case, the acquisition unit identifies or acquires the bitstream or file based on the information indicating the other related bitstream or file. Furthermore, the information indicating the other related bitstream may be included in the metadata or control information of at least some of the bitstreams among multiple bitstreams that store audio signals and metadata, and the information indicating the other related file may be included in the metadata or control information of at least some of the files among multiple files that store audio signals and metadata. Here, the file containing information indicating the related bitstream or file may be, for example, a control file such as a manifest file used for content distribution.

This disclosure is useful when reproducing sound, such as allowing the user to perceive three-dimensional sound.

99 User 100 Sound reproduction system 101 Information processing device 102 Communication module 103 Detector 104 Driver 105 Database 111 Acquisition unit 112 Encoded sound information input unit 113 Decode processing unit 114 Sensing information input unit 121 Path calculation unit 131 Output sound generation unit 134 Generation unit 135 Synthesis unit 141 Signal output unit 300 3D video reproduction device

Claims (17)

An information processing method executed by a plurality of information processing terminals,
acquiring, in a first terminal that is one of the plurality of information processing terminals, first sound information including an acoustic signal and information about a position of a sound source object within a three-dimensional sound field;
the first sound information is information for causing the sound source object in a three-dimensional sound field to emit a reproduced sound by the acoustic signal,
converting, in the first terminal, the first sound information into second sound information for generating a representative sound arriving at a reference position from a representative point set in the three-dimensional sound field using the acoustic signal;
a step of transmitting, in the first terminal, the second sound information to a second terminal which is another information processing terminal among the plurality of information processing terminals;
detecting, at the second terminal, a position or a head direction of a user within the three-dimensional sound field;
calculating, in the second terminal, a position of a reproduction representative point corresponding to the position of the representative point based on the detected position or head direction of the user and the reference position;
generating, in the second terminal, an output sound signal using a head-related transfer function according to the calculated position of the reproduction representative point and the received second sound information. The information processing method according to claim 1 , wherein the converting step converts the reproduced sound into the representative sound by applying a time shift adjustment and a gain adjustment to the reproduced sound. The information processing method according to claim 2 , wherein the adjustment amount of the time shift adjustment is set to be 0 at the representative point. The information processing method according to claim 2 , wherein the adjustment amount of the time shift adjustment at a predetermined position is set using an adjustment amount at a position closer to the representative point than the predetermined position. 3. The information processing method according to claim 2, wherein the adjustment amount of the gain adjustment at a predetermined position is calculated by using two of the three adjacent representative points surrounding the predetermined position to calculate the adjustment amount for each of the two representative points that minimize the error, and then fixing the ratio of the calculated adjustment amounts to calculate the adjustment amount for the remaining representative point. 3. The information processing method according to claim 2, wherein the amount of gain adjustment at a predetermined position is calculated by calculating the amount of adjustment for each of two representative points located horizontally out of three adjacent representative points surrounding the predetermined position for a horizontal position obtained by removing vertical components from the predetermined position, using the two representative points that have the smallest error, and then fixing the ratio of the calculated adjustment amounts to calculate the amount of adjustment for the remaining representative point. The information processing method according to claim 2 , wherein the amount of the time shift adjustment is corrected based on an expected value of the error so as to reduce an error from a calculated correct value. The information processing method according to claim 2 , wherein the amount of gain adjustment at a predetermined position is corrected based on an expected value of the error so as to reduce an error from a calculated correct value. 3. The information processing method according to claim 2, wherein the adjustment amount of the gain adjustment at a predetermined position is set using two or more of adjustment amounts at a plurality of virtual representative points corresponding to a plurality of directions within a predetermined angle range in the horizontal direction centered on the direction of the representative point of the starting point, with the representative point for the predetermined position as the starting point. 10. The information processing method according to claim 9, wherein the adjustment amount of the gain adjustment at a predetermined position is set using all of the adjustment amounts at a plurality of virtual representative points corresponding to a plurality of directions within a predetermined angle range in the horizontal direction centered on the direction of the representative point of the starting point, with the representative point for the predetermined position as the starting point. The information processing method according to claim 9 , further comprising calculating the amount of gain adjustment at the predetermined position by averaging the amounts of adjustment at a plurality of the virtual representative points. The information processing method according to claim 9, wherein the adjustment amount of the gain adjustment at a predetermined position is calculated by taking a weighted average of adjustment amounts at a plurality of the virtual representative points based on an angle difference between the direction of each of the virtual representative points and an original direction corresponding to the virtual representative point. The information processing method according to any one of claims 2 to 12, wherein the amount of gain adjustment at a predetermined position is set to a predetermined value that does not depend on the direction of the predetermined position in a horizontal plane, the amount of adjustment of an elevation angle representative point that includes a vertical component among three adjacent representative points that surround the predetermined position. The information processing method according to claim 13 , wherein the predetermined value varies depending on the direction of the predetermined position in a vertical plane. The information processing method according to claim 14 , wherein the predetermined value is a value that gradually increases between 0 and 1 when the direction of the predetermined position in the vertical plane is from 0° to 90°. An information processing system including a first terminal and a second terminal,
The first terminal
an acquisition unit that acquires first sound information including an acoustic signal and information about a position of a sound source object in a three-dimensional sound field;
the first sound information is information for causing the sound source object in a three-dimensional sound field to emit a reproduced sound by the acoustic signal,
a conversion unit that converts the first sound information into second sound information for generating a representative sound arriving at a reference position from a representative point set in the three-dimensional sound field using the acoustic signal;
a transmitting unit that transmits the second sound information to the second terminal,
The second terminal
a detector for detecting a user's position or head direction within the three-dimensional sound field;
a calculation unit that calculates a position of a reproduction representative point corresponding to the position of the representative point based on the detected position or head direction of the user and the reference position;
an output unit in the second terminal that outputs an output sound signal using a head-related transfer function according to the calculated position of the reproduction representative point and the received second sound information. A program for causing a computer to execute the information processing method according to claim 1.