JP2015061277A - Reverberation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To add reverberation to an acoustic signal of a multichannel audio system on the basis of a video image.SOLUTION: A reverberation device 1, for adding reverberation to direct sound on viewer front channels, disposed inside and in the periphery of a display unit 30 for displaying images, among the acoustic signals of the multichannel audio system, includes: a scene selection unit 11 for selecting a scene image to be a criterion of reverberation addition from the video image; a reflectivity selection unit 13 for selecting the reflectivity of each region in the scene image; an impulse response selection unit 14 for selecting an impulse response to each front channel, on the basis of the configuration of the front channel and the characteristic amount based on the reflectivity of each region; and a convolution processing unit 15 for convolution of the direct sound of the front channel with the impulse response.

Description

  The present invention relates to a reverberation adding device, and in particular, a reverberation that adds reverberation to direct sound of a viewer's front channel arranged in and around a display unit that displays a video among multi-channel sound signals. It relates to an additional device.

  For the sound system of movies and television, the conventional 2ch stereo system, 5.1ch, 7.1ch, 8.2ch, 10.2ch, etc. surround system, 22.2 multiplayer proposed as the next generation television system There is a channel sound system and the number of channels is various. In general, multi-channel audio content production tends to become complicated as the number of channels increases, and the time required for production tends to increase. Further, in the three-dimensional sound reproduction method as seen in 22.2 multi-channel sound, the control becomes more complicated than in the two-dimensional sound reproduction. Therefore, it is desired to construct an environment that allows intuitive and simple production.

  When listening to sound in real space, incoming sounds from all directions are perceived by both ears. There, a direct sound from a sound source or a reverberant sound that reaches the ear later than the direct sound due to the sound reverberating in space is perceived. As for the reverberant sound generation device, development of a device for natural reverberation generation using a virtual sound source distribution generation device is in progress (see, for example, Patent Document 1). In addition, for the purpose of reverberation control in a concert hall, development of a sound generator capable of controlling the intelligibility of sound and the feeling of sound spread is also progressing (for example, see Patent Document 2).

Japanese Patent Laid-Open No. 7-334176 Japanese Patent No. 48944422

  In the conventional two-dimensional sound system, production work such as moving a sound image between two adjacent speakers and adding reverberation was relatively easy. However, in the production of a three-dimensional sound content such as 22.2 multi-channel sound, the control of the spatial impression becomes complicated as the number of channels increases. Therefore, it is necessary to devise an efficient and effective spatial impression control technique particularly in multi-channel sound such as a three-dimensional sound system. In particular, there is no device for generating reverberation of multi-channel sound using video as a clue or estimating and generating reverberation space from an image in the production of content composed of video and sound such as television and movies.

  Accordingly, an object of the present invention made in view of such a point is to provide a reverberation adding apparatus capable of adding reverberation to a multi-channel audio signal based on an image.

  In order to solve the above-described problems, a reverberation adding device according to the present invention is a multi-channel audio signal, which is a direct sound of a front channel of a viewer arranged in and around a display unit that displays an image. A reverberation adding device that adds reverberation to the scene selection unit that selects a scene image serving as a reference for reverberation addition from the video, and a reflectance selection unit that selects the reflectance of each region of the scene image; An impulse response selection unit that selects an impulse response for the front channel based on the configuration of the front channel and a feature value based on the reflectance of each region, and a convolution that convolves the impulse response with the direct sound of the front channel A processing unit.

  Further, it is preferable that the image processing apparatus includes an area analysis unit that analyzes a boundary and a type of each area of the scene image, and the reflectance selection unit selects the reflectance of each area based on the type of each area. .

  Moreover, it is preferable that the reflectance selection unit selects the reflectance based on a material corresponding to the type of each region.

  Further, it is preferable that the scene selection unit selects an image input from a user as the scene image.

  According to the reverberation adding apparatus according to the present invention, it is possible to add reverberation to a multi-channel audio signal based on an image.

It is a figure which shows the structure of the reverberation addition apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows the process of a reverberation addition apparatus. It is a figure which shows an example which selects the reflectance of each area | region from a scene image.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Here, in the following description, a 22.2 channel sound that is a super high-definition sound will be described as an example of a multi-channel sound system, but the present invention is not limited to only a 22.2 channel sound. Please note that. The multi-channel sound system to which the present invention can be applied is not limited to one in which each channel is arranged in a plane as in the conventional two-dimensional sound system, but any sound in which each channel is arranged and expressed in three dimensions. Including methods.

  FIG. 1 is a diagram showing a configuration of a reverberation adding apparatus according to an embodiment of the present invention. The reverberation adding apparatus 1 includes a control unit 10 including a scene selection unit 11, a region analysis unit 12, a reflectance selection unit 13, an impulse response selection unit 14, and a convolution processing unit 15, a region type database 21, a reflectance database 22, The storage unit 20 includes an impulse response database 23 and an acoustic signal database 24, and is connected to the display unit 30, the operation unit 40, and the speaker 50. The reverberation adding apparatus 1 adds reverberation to the direct sound of the viewer's front channel arranged in and around the display unit 30 that displays video, among multi-channel sound signals. For example, in the case of 22.2 channel sound, the reverberation adding apparatus 1 includes the upper layer 3 channels (TpFL, TpFC, TpFR), the middle layer 5 channels (FL, FLc, FC, FRc, FR) and the lower layer 3 channels which are the front channels. Reverberation is added to the direct sound of a total of 11 channels (BtFL, BtFC, BtFR).

  Each processing part 11-15 of the control part 10 is comprised by a suitable processor, and each processing part 11-15 can be mounted with a common processor, or can be mounted as an individual processor. The storage unit 20 is a suitable storage device, and may be an external storage device through a communication interface as well as a built-in reverberation adding device 1. The display unit 30 is a suitable display, and the user of the reverberation adding apparatus 1 operates the reverberation adding apparatus 1 by the operation unit 40 which is a suitable operation interface while confirming the video displayed on the display unit 30. be able to. The speaker 50 is a speaker group corresponding to the multi-channel sound system, and the acoustic signal to which reverberation is added by the reverberation adding device 1 is reproduced from the speaker 50 corresponding to each channel.

  Hereinafter, each functional unit of the reverberation adding apparatus 1 will be described with reference to a flowchart. FIG. 2 is a flowchart showing processing of the reverberation adding apparatus 1. When producing the audio content of a certain program, the scene selection unit 11 selects a scene image as a reference for adding reverberation from the video (step S101). A scene image that is a reference for adding reverberation is an image that represents the characteristics of the acoustic space in the video. For example, indoor spaces such as concert halls, gymnasiums, and offices, and outdoor spaces such as urban areas, parks, and forests Is an image representing an acoustic space in the video. When the scene selection unit 11 selects an appropriate scene image, an acoustic space in the video is audibly reproduced by a reverberation adding process described later, and a higher sense of presence can be given to the viewer. The selected scene image can be used as it is while the same acoustic space continues in the video, and preferably a new scene image can be selected when the acoustic space in the program video is switched. The selection of the scene image allows selection of various images according to the design, such as the top image when the acoustic space is switched or an image at an arbitrary point in time during which the same acoustic space continues.

  In addition, the user can check the video on the display unit 30 by frame advance or the like, and can manually select a scene image in which the acoustic space is to be reproduced using the operation unit 40. Furthermore, the scene selection unit 11 can select not only an image in a program video but also an image such as an arbitrary photograph input by a user as a scene image. For example, the user can easily add past reverberation patterns to the current program by inputting still images with appropriate reverberation added to the same acoustic space in the past as scene images. become. In addition, depending on the experience of the program creator, it may be possible to obtain a higher sense of realism by adding reverberation using an image in another acoustic space as a scene image than the acoustic space in the program video. It is possible to respond to flexible and flexible reverberation.

  The area analysis unit 12 analyzes the boundary and type of each area of the scene image (step S102). First, the region analysis unit 12 extracts the boundary of each region of the scene image. Next, the region analysis unit 12 determines the type (texture) of each region based on the color, density, shape, area, coordinates, frequency characteristics after FFT, and the like of each region. Information on the type of region is stored in the region type database 21 in association with the color, density, shape, area, coordinates, frequency characteristics after FFT, and the like of each region. For example, the region analysis unit 12 refers to the region type database 21 and determines each region of the scene image as “road, stone wall, hedge, tree, sky” or the like. It should be noted that any known technique is used for boundary extraction and type determination of each area of the scene image, and will not be described in detail in this paper. In addition, when the boundary or type of each region is not accurately analyzed by the region analysis unit 12, the user manually corrects the boundary or type to a desired one using the operation unit 40 while confirming the video on the display unit 30. Can do.

The reflectance selection unit 13 selects the reflectance of each area of the scene image (step S103). The reflectance is a numerical value of the degree of sound reverberation in each region, and preferably a different numerical value is set for each frequency. The reflectance is stored in the reflectance database 22, and each reflectance is stored in association with the type of each region. For example, the reflectance selection unit 13 selects the reflectance of each region based on the type of each region analyzed by the region analysis unit 12. The reflectance can be stored not only in association with the type of area such as “road, stone wall, hedge, tree, sky” but also in association with the material corresponding to the type of each area. FIG. 3 is a diagram illustrating an example of selecting the reflectance of each region from the scene image. After the scene image in FIG. 3A is divided into regions of the type “road, stone wall, hedge, tree, sky” by the region analysis unit 12, the reflectance selection unit 13, as shown in FIG. It is determined that the material of each area of “road, stone wall, hedge, tree, sky” is “concrete, stone, grass, tree, omission”. The reflectance database 22 can store the reflectance of each material for each frequency as follows. For example, even for the same “road”, “concrete” roads and “soil” roads have different reflectivities, so by selecting the reflectivity based on the material corresponding to the type of each region, more appropriate reverberation can be added. It becomes possible.
{Concrete, stone, grass, wood, omission} =
{...,
500Hz: 0.6, 0.28, 0.75, 0.7, 0,
1kHz: 0.59, 0.28, 0.70, 0.65, 0,
...

  When the reflectance of each region is not correctly selected by the reflectance selection unit 13 or when the material is not correctly determined, the user manually operates the operation unit 40 while checking the image on the display unit 30. It can be modified to the desired reflectivity and material.

  The impulse response selection unit 14 selects an impulse response for the front channel based on the configuration of the front channel arranged in and around the display unit 30 and the feature amount based on the reflectance of each region (step S104). The impulse response indicates a reverberation characteristic of an acoustic signal. The impulse response is stored in the impulse response database 23, and each impulse response is stored in association with a feature amount based on the configuration of the front channel and the reflectance of each region. First, the impulse response selection unit 14 refers to the impulse response database 23 and confirms whether or not there is an impulse response corresponding to the configuration of the front channel. As described above, in the case of 22.2 channel sound, the front channels are 11 channels in total, 3 upper layers, 5 middle layers, and 3 lower channels. When an impulse response corresponding to the configuration of the front channel exists in the impulse response database 23, the impulse response selection unit 14 calculates a feature amount from the reflectance of each region, and acquires an impulse response closest to the feature amount. Note that the feature amount based on the reflectance of each region can be appropriately designed by those skilled in the art, and will not be described in detail in this paper. Further, when the configuration (physical arrangement) of the speaker 50 is different from the logical arrangement of each channel of the acoustic signal, the impulse response selection unit 14 appropriately corrects the impulse response of each channel selected based on the difference in the arrangement. It can be done.

  Further, when there is no impulse response corresponding to the impulse response database 23, the impulse response selection unit 14 generates a new impulse response by combining the impulse responses stored in the impulse response database 23, or generates an IIR (Infinite Impulse Response). Can generate a new impulse response. Thereby, the reverberation space suitable for the scene image can be generated.

  The convolution processing unit 15 convolves the impulse response selected for each channel with the direct sound of the front channel (step S105). In the present embodiment, the direct sound of the multi-channel sound is stored in the sound signal database 24. However, the convolution processing unit 15 is not limited to the built-in storage unit 20, for example, but directly from other devices using the sound signal input interface. May be received. For example, the convolution processing unit 15 convolves an impulse response for each channel with a direct sound such as a walking sound, a buzzing sound, or a bird's voice, and generates an output signal to the corresponding speaker 50 (step S106). The reverberation adding apparatus 1 according to the present embodiment can support real-time processing. In this case, as long as sound continues to be input directly to the acoustic signal input interface, the convolution processing unit 15 performs convolution processing of impulse responses and the corresponding speaker 50. Playback by is continued.

  As described above, according to the present embodiment, the scene selection unit 11 selects a scene image serving as a reference for adding reverberation from the video, and the reflectance selection unit 13 selects the reflectance of each area of the scene image, The impulse response selection unit 14 selects an impulse response for the front channel based on the configuration of the front channel and the feature quantity based on the reflectance of each region, and the convolution processing unit 15 applies the impulse response to the direct sound of the front channel. Fold it up. As a result, it is possible to add reverberation to the multi-channel audio signal based on the video. In particular, it is possible to automatically reproduce the reverberation space you want to generate simply by selecting a representative still image from the video frames, in order to reproduce the characteristics of the sound of the space according to the video depending on the skill of the creator It becomes. As a result, a reverberation space with a certain level of presence can be generated regardless of the skill level of the creator. In addition, it is possible to simplify the user's work for adding reverberation by reducing the manual adjustment for each channel as in the prior art, and to significantly reduce the time required for content production. If you want to adjust the automatically generated reverberation space according to the creator's intention, you can change it to the intended reverberation space by manual operation or manual input. It is possible to achieve both shortening of work time for space generation and securing of creativeness for further enhancing the sense of reality and the production effect.

  The area analysis unit 12 analyzes the boundary and type of each area of the scene image, and the reflectance selection unit 13 selects the reflectance of each area based on the type of each area. This makes it possible to set the reflectance based on the type of area (road, sky, etc.) that is easy for the user to understand intuitively. For example, direct reflection from the pixel value of each area, the FFT processing result (numerical string), etc. As compared with the case of obtaining the rate, there are advantages that the reflectance can be easily set and managed, and the user can easily correct the automatic processing result.

  Moreover, the reflectance selection part 13 can select a reflectance based on the raw material corresponding to the kind of each area | region. As a result, it is possible to select a more appropriate reflectance and perform correct reverberation addition for a region such as “road” where the material may be different, such as “concrete” and “soil”.

  The scene selection unit 11 can select an image input from the user as a scene image. For example, the user can easily add past reverberation patterns to the current program by inputting still images with appropriate reverberation added to the same acoustic space in the past as scene images. become. In addition, depending on the experience of the program creator, it may be possible to obtain a higher sense of realism by adding reverberation using an image in another acoustic space as a scene image than the acoustic space in the program video. It is possible to respond to flexible and flexible reverberation. Thus, generating a reverberation space similar to a video based on arbitrary image data makes it possible to listen to a sound source in the reverberation space optimal for each scene. In addition, since it gives a room for appropriate adjustment to the desired acoustic space intended by the user, it is possible to realize both the efficiency of the user's work and the securing of creativity associated with sound source production.

  Although the present invention has been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these variations and modifications are included in the scope of the present invention. For example, the functions included in each functional unit, each step, etc. can be rearranged so that there is no logical contradiction, and a plurality of functional units, steps, etc. can be combined into one or divided. It is.

DESCRIPTION OF SYMBOLS 1 Reverberation addition apparatus 10 Control part 11 Scene selection part 12 Area | region analysis part 13 Reflectivity selection part 14 Impulse response selection part 15 Convolution process part 20 Storage part 21 Area | region type database 22 Reflectance database 23 Impulse response database 24 Acoustic signal database 30 Display Part 40 Operation part 50 Speaker

Claims (4)

A reverberation adding device that adds reverberation to the direct sound of a viewer's front channel arranged in and around a display unit that displays video, among multi-channel audio signals.
A scene selection unit that selects a scene image serving as a reference for adding reverberation from the video;
A reflectance selection unit that selects the reflectance of each region of the scene image;
An impulse response selector that selects an impulse response to the front channel based on the configuration of the front channel and a feature quantity based on the reflectance of each region;
A convolution processing unit that convolves the impulse response with the direct sound of the front channel;
A reverberation adding device. An area analysis unit for analyzing the boundary and type of each area of the scene image;
The reverberation adding apparatus according to claim 1, wherein the reflectance selection unit selects the reflectance of each region based on a type of each region.   The reverberation adding apparatus according to claim 2, wherein the reflectance selection unit selects the reflectance based on a material corresponding to a type of each region.   The reverberation adding apparatus according to any one of claims 1 to 3, wherein the scene selection unit selects an image input from a user as the scene image.