JP2018120007A - Voice signal converter, voice signal converting method and program - Google Patents

Abstract

An audio signal conversion device and the like that generate an ambisonics-type audio signal in a pseudo manner using sound pickup signals of a plurality of microphones whose positional relationships are not limited. An audio signal conversion device uses a non-directional signal W, a bi-directional signal X in an x-axis direction, using six enhancement signals that emphasize sounds arriving from six directions, front, rear, left, right, and up. It includes a conversion processing unit that generates a bi-directional signal Y in the y-axis direction and a bi-directional signal Z in the z-axis direction. , B, L, R, U, D, the conversion processing unit uses W = F + B + L + R + U + D, X = FB, Y = LR, Z = UD, signal W, bidirectional A signal X, a bidirectional signal Y, and a bidirectional signal Z are generated. [Selection] Figure 5

Description

  The present invention relates to a three-dimensional acoustic technology that can easily reproduce a change in a sound field when a space or a listener's head rotates by software.

  The video in any direction that the user wants to see is called a 360-degree video. In recent years, opportunities to view 360-degree video using video distribution services are increasing. For example, around April 2016, YouTube (registered trademark) released a service that changes sound (performs sound image localization according to the viewing direction) in response to viewing 360-degree video. Note that the sound collection / reproduction method on YouTube is based on the ambisonics format (see Non-Patent Document 1).

  As hardware for generating ambisonics-type sound, a dedicated microphone array (hereinafter also referred to as an ambisonics microphone array) is usually used. In principle, an omnidirectional microphone or a unidirectional microphone is placed at the apex of a tetrahedron, and an object that reflects or diffracts is not installed around.

Ryuichi Nishimura, “Chapter 5 Ambisonics”, [online], Journal of the Institute of Image Information and Television Engineers, vol.68, No.8, (2014), [searched on January 11, 2017], Internet <URL: https: / /www.jstage.jst.go.jp/article/itej/68/8/68_616/_pdf>

  However, the position of the microphone of the conventional ambisonics microphone array is limited to the position corresponding to the apex of the tetrahedron in principle, and the degree of freedom is low. Therefore, it is impossible to mount the ambisonic microphone array directly in the camera. Therefore, as shown in FIG. 1, a configuration in which an ambisonic microphone array 92 having a shape in which a unidirectional microphone is installed at the apex of a regular tetrahedron is installed on the upper part of the 360-degree camera 91 can be considered. In this case, there may arise a problem that the ambisonic microphone array 92 is reflected in the image of the 360-degree camera 91. At this time, if a windshield or the like covering the entire ambisonic microphone array 92 is provided, this problem is more likely to occur. In addition, such a configuration causes a problem that the sense of unity of the apparatus is lowered and portability is lowered.

  It is an object of the present invention to provide an audio signal converter, a method thereof, and a program for generating an ambisonics-type audio signal in a pseudo manner by using sound pickup signals of a plurality of microphones whose positional relationships are not limited. Objective.

  In order to solve the above-described problem, according to one aspect of the present invention, an audio signal conversion device uses six enhancement signals that enhance sounds arriving from six directions, front, rear, left, right, and up, respectively, and is omnidirectional. Signal W, x-axis direction bi-directional signal X, y-axis direction bi-directional signal Y, z-axis direction bi-directional signal Z The six emphasis signals emphasizing the incoming sound are F, B, L, R, U, D, respectively, and the conversion processing unit is W = F + B + L + R + U + D, X = FB, Y By using = LR and Z = UD, a signal W, a bidirectional signal X, a bidirectional signal Y, and a bidirectional signal Z are generated.

  In order to solve the above-described problem, according to another aspect of the present invention, an audio signal conversion method uses six emphasis signals respectively emphasizing sounds arriving from six directions, front, rear, left, right, and upper, and is omnidirectional. Signal, W, x-axis direction bi-directional signal X, y-axis direction bi-directional signal Y, z-axis direction bi-directional signal Z The six enhancement signals emphasizing the sounds coming from are respectively F, B, L, R, U, and D, and the conversion processing steps are W = F + B + L + R + U + D, X = FB, A signal W, a bidirectional signal X, a bidirectional signal Y, and a bidirectional signal Z are generated by Y = LR and Z = UD.

  Advantageous Effects of Invention According to the present invention, there is an effect that an ambisonics-type sound signal can be generated in a pseudo manner using sound pickup signals of a plurality of microphones whose positional relationships are not limited.

The figure which shows the structure which combined the ambisonics microphone array and the 360 degree camera. The figure which shows the example of the structure which embedded the some microphone directly in the housing | casing of the camera. The figure which shows the example of the structure which embedded the some microphone directly in the housing | casing of the camera. The figure for demonstrating the relationship between four omnidirectional microphones and a B format signal. The functional block diagram of the audio | voice signal converter which concerns on 1st embodiment. The figure which shows the example of the processing flow of the audio | voice signal converter which concerns on 1st embodiment. The figure for demonstrating an emphasis signal. The functional block diagram of the sound collection part classified by direction. The figure which shows the example of the processing flow of the sound collection part classified by direction.

  Hereinafter, embodiments of the present invention will be described. In the drawings used for the following description, constituent parts having the same function and steps for performing the same process are denoted by the same reference numerals, and redundant description is omitted.

<Points of first embodiment>
In the present embodiment, an ambisonics-type audio signal is generated in a pseudo manner using sound pickup signals of a plurality of microphones whose positional relationships are not limited. Therefore, a configuration in which a plurality of microphones are directly embedded in the camera casing is possible.

  Examples of configurations in which a plurality of microphones are directly embedded in the camera housing are shown in FIGS. 2A is an example in which microphones 95 are arranged on the front, side, and top surfaces of a flat housing, FIG. 2B is a front view of the cylindrical housing, and FIG. 2C is a rear view of FIG. 2B. In FIG. 2B, two microphones 95 are arranged side by side in the horizontal direction, and in FIG. 2C, two microphones 95 are arranged side by side in the vertical direction. 3A shows an example in which a microphone is arranged in a spherical casing, and FIG. 3B shows an example in which a microphone is arranged in a rectangular parallelepiped casing. In addition, these structures are examples and other arrangement | positioning may be sufficient.

  However, it is assumed that three or more microphones are used. When using four or more microphones, it is better that they are not on the same plane. Further, it is preferable that the distance between the microphones is narrow to some extent (1 to 5 cm).

<Ambisonics B format>
The Ambisonics B format will be described as a conventional method of audio generation technology for 360-degree video. The ambisonic microphone array used mainly arranges unidirectional microphones or omnidirectional microphones with uniform sensitivity at the apex of the tetrahedron. The microphone spacing is often narrow (about 1 to 4 cm). Hereinafter, a method for generating an ambisonics B format audio signal (hereinafter also referred to as a B format signal) using four omnidirectional microphones will be described because the generality of the description is not lost. Four omnidirectional microphones 93 are arranged on the origin, the X axis, the Y axis, and the Z axis (see FIG. 4). The omnidirectional microphones 93 on each axis are arranged at an equal distance when viewed from the origin.

An ambisonics B format audio signal is composed of a total of four channels of sound collected with omnidirectionality and three types of bidirectionality (X direction, Y direction, and Z direction). Four time-domain signals observed by the microphones 93 O _t (output signal of the omnidirectional microphone is placed at the origin), (the output signal of the omnidirectional microphone disposed on the X axis) A _t, B It is defined as _t (output signal of an omnidirectional microphone arranged on the Y axis) and C _t (output signal of an omnidirectional microphone arranged on the Z axis). Ambisonics B format omnidirectional time signal picked up by W _t , X-axis direction bi-directional signal X _t , Y-axis direction bi-directional signal Y _t , Z-axis direction bi-directional signal Let Z _t be the relationship.
W _t = O _t
X _t = A _t -O _t
Y _t = B _t -O _t
Z _t = C _t -O _t

  The Ambisonics B format is increasingly used because it provides a first-order differential representation of the sound field with very simple calculations and has a certain effect as a sound image localization for 360-degree video.

  However, as described above, the position of the microphone of the conventional ambisonics microphone array is limited to the position corresponding to the apex of the tetrahedron, and there is a problem that the degree of freedom is low. In addition, there are the following problems.

  (i) In order to generate Ambisonics sound, the microphones must have the same sensitivity. However, it is not easy to produce microphones with the same sensitivity as required to generate ambisonics-type sound. For this reason, many microphones are manufactured, and four microphones having the same sensitivity are selected from among them. Therefore, the manufacturing cost is increased.

  (ii) Do not place reflection / diffraction sources in the vicinity of the ambisonics microphone array. When the ambisonics microphone array is embedded directly in the camera, the camera case reflects or diffracts the sound. Therefore, from this point of view, it is impossible to mount the microphone directly in the camera.

  (iii) In principle, there is a problem that volume adjustment of sound in a specific area cannot be performed after sound collection.

<Audio signal conversion apparatus 100 according to the first embodiment>
FIG. 5 is a functional block diagram of the audio signal conversion apparatus 100 according to the first embodiment, and FIG. 6 shows a processing flow thereof.

The audio signal conversion apparatus 100 receives M sound pickup signals sm _{, t} collected by microphones 95-m of M microphones as input, and outputs the four sound pickup signals sm _{, t} to four ambisonics B. Converted into audio signals W _t , X _t , Y _t , Z _t in format format and output. Here, m is an index of a microphone, t is an index indicating discrete time, m = 1, 2,..., M, and M is an integer of 3 or more.

  The audio signal conversion apparatus 100 is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing the following processing, and is functionally configured as follows.

  The audio signal conversion apparatus 100 includes a direction-specific sound collection unit 110 and a conversion processing unit 130.

<Sound Pickup Unit 110 by Direction>
The direction-specific sound pickup unit 110 receives M sound pickup signals sm _{, t} , and uses these values to emphasize six enhancement signals F _t that emphasize sounds coming from six directions, front, rear, left, right, and top, respectively. , B _t , L _t , R _t , U _t , D _t are generated (S110, see FIG. 7, F _t , B _t , L _t , R _t , U _t , D _t in the figure are the enhancement signals F _t , B _t , L _t , R _t , U _t , and D _t indicate the directions seen from the audio signal converter 100), and output. Any existing technique may be used as the method for generating the emphasis signal, and an optimum method may be selected as appropriate in accordance with the use environment. For example, there is a noise suppression method using beam forming and a Wiener filter as an example. Since many details of this scheme have been described, details are not described here. For example, a method described in JP-A-2016-131343 can be used.

  FIG. 8 is a functional block diagram of the direction-specific sound collection unit 110 when a noise suppression method using beam forming and a Wiener filter is adopted, and FIG. 9 shows an example of the processing flow thereof.

  For example, the direction-specific sound collecting unit 110 includes a beam forming unit 111 and a Wiener filter unit 113.

The beam forming unit 111 receives M sound pickup signals sm _{, t} , and uses these values to generate six beam forming signals that are sound signals that emphasize sounds coming from six directions, front, rear, left, right, and up, and down. Output values F ′ _t , B ′ _t , L ′ _t , R ′ _t , U ′ _t , and D ′ _t are obtained (S111) and output.

The Wiener filter unit 113 uses the Wiener filter to reduce the noise from the six beamforming output values F ′ _t , B ′ _t , L ′ _t , R ′ _t , U ′ _t , and D ′ _t , respectively. Are obtained as enhancement signals F _t , B _t , L _t , R _t , U _t , D _t (S113) and output.

<Conversion processing unit 130>
The conversion processing unit 130 receives six enhancement signals F _t , B _t , L _t , R _t , U _t , and D _t and uses these values to determine the omnidirectional signal W _t and x-axis direction. generating a bi-directional signal X _t, bi-directional signals in the y-axis direction Y _t, z-axis direction of the bi-directional signal Z _t (S130), and outputs. The omnidirectional signal W _t , the x-axis direction bi-directional signal X _t , the y-axis direction bi-directional signal Y _t , and the z-axis direction bi-directional signal Z _t are obtained by the following equations.
W _t = F _t + B _t + L _t + R _t + U _t + D _t
X _t = F _t -B _t
Y _t = L _t -R _t
Z _t = U _t -D _t

<Effect>
With the above configuration, it is possible to generate ambisonics-type audio signals W _t , X _t , Y _t , Z _t by using the collected sound signals of a plurality of microphones whose positional relationships are not limited. . The generated signals W _t , X _t , Y _t , Z _t can be handled in the same manner as the B format signal. For example, you can upload to YouTube. Also, if you want to play on your own player, encode multi-channel audio, perform binaural synthesis considering the coordinate system rotation according to head movement, and generate a stereo channel audio signal corresponding to your own player. Also good.

  In the case of the present embodiment, the positional relationship between a plurality of microphones is not limited, so that a pseudo-ambisonics B format sound can be generated using a microphone embedded in the camera. As a result, the problem that the microphone is reflected on the camera and the problem that portability is impaired are solved.

  Furthermore, it is possible to make adjustments so that the sensitivity to sounds coming from six directions, front, rear, left, right, up and down is uniform without matching the sensitivity of each microphone. Therefore, it is not necessary to manufacture a large number of microphones and select four microphones having the same sensitivity among them, and the manufacturing cost can be reduced.

  In addition, since sound collection processing is performed in consideration of reflection / diffraction, an ambisonic microphone array can be mounted directly in the camera.

In this embodiment, the M collected sound signals sm _{, t} are not directly converted into B format signals, but once from the front, back, left, right, up and down directions using a noise suppression method using beam forming and a Wiener filter. An enhancement signal that emphasizes the sound is obtained, and a B format signal is obtained from the enhancement signal. In addition, when M collected sound signals sm _{, t} are directly converted into B format signals, the number of microphones M is small (for example, M is 6 or less), and a structure that causes reflection / diffraction around the microphone ( In particular, when an asymmetric structure) is installed, it is difficult to realize sound collection corresponding to the bidirectionality constituting the B format. On the other hand, even when the number of microphones M is small or when a structure is installed around the microphone, if it is a process that emphasizes only the sound coming from a single direction and suppresses the ambient noise, It can be realized relatively easily by microphone array processing. In the present embodiment, in order to obtain an emphasis signal that emphasizes sounds coming from the front, back, left, right, up and down directions once, and to obtain a B format signal from the emphasis signal, it is limited to the number of microphones M and the presence of structures around the microphone In addition, it is possible to generate an ambisonics-type audio signal in a pseudo manner relatively easily by the microphone array processing.

The audio signal conversion apparatus of this embodiment may be incorporated in an omnidirectional camera with a built-in microphone, or M sound pickup signals sm _{, t} included in the output signal of the omnidirectional camera with a built-in microphone are separated. It is good also as an input of an audio signal converter.

<Modification>
In this embodiment, M-number of the collected signal s _m, although as input _t, 6 one enhancement signal _{F t, B t, L t} , R t, U t, may enter D _t. In that case, the direction-specific sound collecting unit 110 may not be provided. For example, the output value of a speech enhancement device (see, for example, International Publication No. 2012/088684) configured as a separate device may be used as the input of the speech signal conversion device 100. Further, for example, six unidirectional microphones may be installed so as to pick up sounds arriving from six directions, front, rear, left, right, and up, respectively, and the output values thereof may be input to the audio signal converter 100. However, using the noise suppression method using the beam forming and the Wiener filter as in this embodiment can reduce the coherent noise and effectively reduce the ambient noise. Therefore, the ambisonics microphone array 92 collects sound. It is possible to generate a sound similar to that obtained when a B format signal is synthesized from the processed signal.

The audio signal conversion apparatus 100 may include a direction-specific volume adjustment unit 120 (indicated by a broken line in FIG. 5). The direction-specific volume adjustment unit 120 receives six enhancement signals F _t , B _t , L _t , R _t , U _t , and D _t and individually adjusts the volume (S120, indicated by a broken line in FIG. 6). ), The enhanced signals F _t , B _t , L _t , R _t , U _t and D _t after the volume adjustment are output. If the volume (number of 0 or more) multiplied by each region is described as α _F , α _B , α _L , α _R , α _U , α _D , the adjustment work can be performed as follows.
F _t ← α _F・ F _t
B _t ← α _B・ B _t
L _t ← α _L・ L _t
R _t ← α _R・ R _t
U _t ← α _U・ U _t
D _t ← α _D・ D _t
By providing the direction-specific volume adjustment unit 120, the volume of each of the front, rear, left, right, upper, and lower areas can be controlled. For example, it is possible to perform adjustment work such as lowering the voice of a person in the front direction to pick up sound a little, or increasing the volume when the voice of a person in a specific direction is low. Such a function for adjusting the volume in each direction cannot be realized by the conventional ambisonics sound pickup. This is because the principle is based on the idea of reproducing the field as it is. In other words, the conventional technique has a problem that, in principle, volume adjustment of sound in a specific area cannot be performed after sound collection. In this modification, it is possible to adjust the volume of sound coming from a specific area after collecting sound.

<Other variations>
The present invention is not limited to the above-described embodiments and modifications. For example, the various processes described above are not only executed in time series according to the description, but may also be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. In addition, it can change suitably in the range which does not deviate from the meaning of this invention.

<Program and recording medium>
In addition, various processing functions in each device described in the above embodiments and modifications may be realized by a computer. In that case, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on a computer, various processing functions in each of the above devices are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Further, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its storage unit. When executing the process, this computer reads the program stored in its own storage unit and executes the process according to the read program. As another embodiment of this program, a computer may read a program directly from a portable recording medium and execute processing according to the program. Further, each time a program is transferred from the server computer to the computer, processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program includes information provided for processing by the electronic computer and equivalent to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In addition, although each device is configured by executing a predetermined program on a computer, at least a part of these processing contents may be realized by hardware.

Claims (7)

Using six emphasis signals emphasizing the sound coming from six directions, front, rear, left, right, top, bottom, omnidirectional signal W, x-axis direction bi-directional signal X, y-axis direction bi-directional signal Y, including a conversion processing unit for generating a z-directional bi-directional signal Z;
The six enhancement signals that emphasize the sounds coming from the six directions of front, rear, left, right, top and bottom are F, B, L, R, U, D, respectively,
W = F + B + L + R + U + D
X = FB
Y = LR
Z = UD
To generate the signal W, the bidirectional signal X, the bidirectional signal Y, and the bidirectional signal Z.
Audio signal converter. The audio signal converter according to claim 1,
M is any integer greater than or equal to 3, and the sound collection signals collected by M microphones are input, and the sound collection unit for each direction that generates the six enhancement signals using the M sound collection signals. Including
The direction-specific sound collection unit is:
A beamforming unit that obtains six beamforming output values that are sound signals that emphasize sounds coming from six directions, front, back, left, right, up, and down, using the M collected sound signals,
Using a Wiener filter, including a Wiener filter unit that obtains six signals, each of which suppresses noise from each of the six beamforming output values, as the six enhancement signals,
Audio signal converter. The audio signal converter according to claim 1 or 2, wherein
A direction-specific volume adjustment unit that individually adjusts the volume of the six emphasized signals,
Audio signal converter. Using six emphasis signals emphasizing the sound coming from six directions, front, rear, left, right, top, bottom, omnidirectional signal W, x-axis direction bi-directional signal X, y-axis direction bi-directional signal Y, including a transformation processing step for generating a z-directional bi-directional signal Z;
The six enhancement signals that emphasize the sounds arriving from the six directions of front, rear, left, right, top, and bottom are F, B, L, R, U, and D, respectively, and the conversion processing step includes:
W = F + B + L + R + U + D
X = FB
Y = LR
Z = UD
To generate the signal W, the bidirectional signal X, the bidirectional signal Y, and the bidirectional signal Z.
Audio signal conversion method. The audio signal conversion method according to claim 4,
Sound collecting step for each direction in which M is one of an integer of 3 or more, and the sound pickup signals picked up by M microphones are input, and the six sound enhancement signals are generated using the M sound pickup signals. Including
The direction-specific sound collection step includes:
A beamforming step for obtaining six beamforming output values, which are audio signals that emphasize sounds coming from six directions, front, back, left, right, up, and down, using the M collected sound signals,
Using a Wiener filter, a Wiener filter step of obtaining six signals, each of which suppresses noise from each of the six beamforming output values, as the six enhancement signals,
Audio signal conversion method. The audio signal conversion method according to claim 4 or 5, wherein
A direction-specific volume adjustment step for individually adjusting the volume of the six emphasized signals,
Audio signal conversion method.   The program for functioning a computer as an audio | voice signal converter in any one of Claims 1-3.