JP2000295700A - Sound source localization method and apparatus using image information and storage medium storing a program for implementing the method - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To provide a sound source localization method and apparatus which can automatically convert a monaural sound image into a stereo sound image and do not give a sense of incongruity in the positional relationship between an image and a sound source in a panoramic image or the like. SOLUTION: A dividing unit 20 divides an input image with sound information.
To divide into sound information and images. The image analysis unit 40 uses the information of the image knowledge database 30 to analyze the object in the image, the movement (position) of the object, the operation of the camera, and the like from the divided image to obtain image information. Sound source separation unit 60
Separates the sound source considered to be emitted by the object from the divided sound information from the acquired image information and the information in the sound knowledge database unit 50. The sound source localization unit 70 rearranges the separated sound sources in a sound field space suitable for a video in consideration of the acquired image information and a video display method such as a panoramic image display in the playback unit 90. The rearranged sound source and the image are synthesized by the synthesis unit 80, and the synthesized image with sound information is reproduced by the reproduction unit 9
When 0, display and reproduction are performed in a predetermined video display method.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a method and an apparatus for localizing a sound source using image information.

[0002]

2. Description of the Related Art A method of generating pseudo stereo sound from monaural sound has been studied since about 1985. By using an operation device for localizing sound, the position of a sound source is moved up, down, left, right, back and forth, and a pseudo sound is generated. Research has been conducted to enable free generation of sound field space. For example, in the document “Sound image localization editing system for headphone sound contents” (Mana Okamoto et al., AES Tokyo Convention '97 Preprints, P154-157, June, 1997)
The positional relationship between the sound source and the icon indicating the listener is operated with a mouse, and the localization position of the sound source on the horizontal plane is operated by calculating the angle and distance from the positional relationship between the two. In this system, by using such a typical GUI, the operator can intuitively grasp the positional relationship between the sound source and the listener, and can generate a sound field space intended by the operator. Becomes

[0003] In recent years, many new video display methods which are not caught by an image display area such as a 4: 3 rectangular area in a TV have been studied. For example, a document “Analysis method of video using projection method and application to video handling” (Akito Akutsu et al., IEICE Transactions D-2, Vol. J.
79-D-2, no. 5, pp. 675-686, Ma
y, 1996), as shown in FIG. 10, by analyzing the movement of the camera, for example, an image obtained by photographing a town landscape while moving the camera in the horizontal direction is regarded as a panoramic image.
We propose a method to convert it into a single image. In this method, a conventional image captured in a rectangular area such as 4: 3 is converted into a panoramic image, so that an image space not captured by the rectangular area can be expressed.

[0004]

The former prior art described above is a method that allows an operator to freely arrange sound sources in a space when creating a sound field space.
For example, when converting audio in an old image recorded as a monaural sound into a pseudo stereo sound, the sound sources are separated, and the operator refers to the images one by one and adjusts the position of the sound source to the position corresponding to the image. It must be moved using the operating device.

Further, in the conventional moving image display method, the display range of the image is fixed to a rectangular area such as TV 4: 3, for example, the camera moves laterally following an automobile moving from right to left. In the case of a scene where sound is generated, the position of the sound source is treated as constant unless the distance relationship between the camera and the target changes. Therefore, when a different video display method such as the latter is used, such as the panorama expansion of this moving image, the car moves from right to left on the screen, but the sound generated by the car does not move. The unnatural state of not being produced occurs. This is a problem that occurs when only an image display method is converted when an image is converted into a panoramic video, and no conversion is performed for audio.

The problem of the present invention is that the conventional method, in which a sound source is localized manually using an operation device, takes too much time when converting a monaural sound image into a pseudo-stereo sound. Presenting a sound source localization method and apparatus that solves the disadvantage that a difference in the positional relationship between an image and a sound source occurs when a new video display method different from the conventional one is used is solved by using image information. That is.

[0007]

In order to solve the above-mentioned problems, a sound source localization method according to the present invention comprises the steps of: inputting an image with sound information; and extracting sound information and image from the input image with sound information. Procedure for dividing, and a procedure for analyzing the divided image based on information of an image knowledge database storing knowledge necessary for analyzing the image and acquiring image information,
A step of separating a sound source related to the image information from the divided sound information based on a sound knowledge database storing knowledge relating the image information and the sound information; and a method of reproducing the image information and the image when reproducing the image. Arranging the separated sound source in the sound field space, synthesizing the arranged sound source and the divided image, and reproducing and displaying the synthesized sound source and image. It is characterized by having.

Alternatively, in the procedure of analyzing the image and obtaining the image information, the image information includes characteristic information of the object displayed in the input image, position information of the object, and motion information of the camera. In the step of arranging the separated sound sources in the sound field space, the acquired characteristic information of the object, the position information of the object, and the motion information of the camera are used. The separated sound sources are arranged in a sound field space corresponding to the reproduced image in consideration of the position information of the object with respect to the reproduced image according to (1).

According to another aspect of the present invention, there is provided a sound source localization apparatus configured to input an image with sound information, and to divide sound information and an image from the input image with sound information. A dividing unit, an image knowledge database storing knowledge necessary for analyzing the image, an image analyzing unit for analyzing the divided image using information of the image knowledge database to obtain image information, and A sound knowledge database storing knowledge relating sound information and sound information, a sound source separation unit that separates a sound source related to the image information from the divided sound information using information of the sound knowledge database, and the image information A sound source localization unit that arranges the separated sound sources in a sound field space based on a reproduction method at the time of image reproduction, a synthesis unit that synthesizes the arranged sound source and the divided image, and the synthesis. The a reproducing unit which reproduces the sound source and the image display, characterized by comprising.

Alternatively, the image analysis unit acquires, as image information, characteristic information of the object displayed in the input image, position information of the object, and motion information of the camera. The unit uses the acquired characteristic information of the object, the position information of the object, and the motion information of the camera, and further takes into account the position information of the object with respect to the reproduced image due to a difference in the reproduction method at the time of image reproduction, and The separated sound source is arranged on a sound field space corresponding to an image.

A program for causing a computer to execute the procedure in the sound source localization method using the image information can be recorded on a storage medium readable by the computer, whereby the image information of the present invention can be stored. The used sound source localization method can be distributed or stored as a storage medium, and the method of the present invention can be realized using a computer.

According to the present invention, image information is obtained by analyzing an image, and the position of the sound source is specified using the information.
To automatically separate and arrange sound sources based on image information,
Compared with a method using a conventional input device, no manpower is required for conversion from monaural sound to pseudo-stereo sound.
When a video display method different from the conventional method is used, the sound source localization unit determines how the image is displayed based on the image information and the information from the playback unit, and places the sound source in a position suitable for the display method. By arranging, even when an image display method different from the conventional one is used, the positional relationship between the image and the sound source is not uncomfortable.

[0013]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing an embodiment of an apparatus to which the present invention is applied. The apparatus includes an image input unit 10 for inputting an image with sound information, a dividing unit 20 for dividing the sound information and the image from the input image with sound information, and analyzing the image to obtain information in the image. Knowledge database 30 that stores important knowledge, image knowledge database 30
Image analysis unit that analyzes what kind of object is in the image, how it is moving, how the camera is operated, etc. from the divided image using the information of 40, a sound knowledge database 50 storing image information and sound information related to the information as knowledge, and image information acquired by the image analysis unit 40 and information in the sound knowledge database 50, it is considered that an object is emitted. A sound source separation unit 60 that separates a sound source from the sound information divided by the division unit 20, image information acquired from the image analysis unit 40, and a video display method of how to display an image in the reproduction unit 90 are considered. The sound source localization unit 7 that rearranges the separated sound source in a sound field space suitable for video.
0, a synthesizing unit 80 for re-synthesizing the rearranged sound source and the image,
A playback unit 90 for displaying and playing back the synthesized image with sound information using a fixed rectangular area such as a conventional TV or a different video display method.

The present invention particularly relates to the functions and processes of the image analysis unit 40, the sound source separation unit 60, and the sound source localization unit 70.

More specifically, the image analysis section 40 analyzes the image to obtain image information using information in the image knowledge database 30. The image information in this case is, for example, in the case of the image displayed in FIG. 3, information (male and female) relating to the object (person) displayed in the image, the position of the object, and shape information of the body and face And camera movement. These are acquired by referring to the feature amount and shape information for the object stored in the image knowledge database 30, the motion feature amount of the camera, and the like.

The sound source separation section 60 obtains sound information on the object obtained by the image analysis section 40 from the sound knowledge database 50, and separates a sound source on the object from the sound information given to the image. The sound information in this case refers to information that is obtained from the sound knowledge database 50 and what kind of unique sound the object emits (a male voice model for a male). By using the information on an object acquired by the image analysis unit 40 and extracting sound information on the object from the sound knowledge database 50, a sound source unique to the object is separated from the sound information added to the image. Becomes possible.

In the sound source localization unit 70, information on what video display method is used in the reproduction unit 90, position information of the object acquired in the image analysis unit 40, motion information of the camera, etc. The sound source unique to the object separated by the sound source separation unit 60 is relocated to a position in the sound field space that is equal to the display position of the object. Reproduction unit 9
From 0, information such as whether to display an image in a fixed rectangular area or to use a different video display method is sent, so that the sound source localization unit 70 does not only use image information such as position information of an object, It is possible to generate a sound field space in consideration of a video display method.

By automatically separating sound sources based on image information automatically obtained by image analysis as described above, and by rearranging sound sources in a sound field space in consideration of an image display method, The sound source can be placed at an appropriate position even when using a new image display method that does not take the trouble of the method of arranging sound sources through human hands found in the conventional method and is not caught in a fixed image display area such as a TV. Can be arranged.

An embodiment of a sound source localization method using image information according to the present invention will be specifically described below.

<Embodiment 1> This is a case where a monaural audio image (FIG. 3) from a fixed camera capturing two men and women having a conversation is converted into a stereo audio image using the present method. FIG. In FIG. 3, although two men and women are talking on the left and right sides of the image, the viewer feels that both sounds are generated from the center of the image or the entire image because of monaural sound. In this method, such a monaural sound is pseudo-stereo-converted using image information.

FIG. 2 is an example of a flowchart showing the processing procedure of this embodiment.

First, as step 100, the image input unit 1
At 0, an input image is received. The input image is sent to the division unit 20 as step 110, and is divided into audio and images. The division unit 20 sends the image to the image analysis unit 40,
The sound is sent to the sound source separation unit 60, respectively.

Next, as a step 120, the image divided by the division unit 20 is analyzed by the image analysis unit 40. At the time of analysis, object recognition such as what kind of object is present in the image is performed, and the image knowledge database 30 is used for that purpose. Image Knowledge Database 3
0 holds, for example, human shape information, mouth position information, and the like, and recognizes an object in an image by using the information. The image analysis unit 40 also analyzes and acquires other image information such as the movement of an object and the movement of a camera. Recognition of an object and motion of an image and motion of a camera can be achieved by conventional techniques, and a detailed description thereof will be omitted. In the case of this embodiment, the image analysis unit 40
As shown in (2), there are two persons in the image, the left is a man and the right is a woman, and the movement of the mouth is recognized using the knowledge of the image knowledge database 30.

Next, in step 130, the sound source separation unit 6
At 0, the sound source associated with the object is separated from the sound input by the division unit 20 using the information of the object in the image information acquired by the image analysis unit 40. At the time of separation, a sound knowledge database 50 having knowledge relating the object and the sound, such as what kind of sound the object emits, is used. In the sound knowledge database 50, for example, a human male and a male voice model, a car and a car engine sound model, and the like are stored in association with each other.
The sound source is separated from the sound information by the sound model derived from the information of the object. In the case of this embodiment, since the person on the left side is known to be a male in step 120, a typical male voice model in the sound knowledge database 50 is used for sound source separation. Step 12
By recognizing the movement of the mouth at 0, a sound source (voice) that matches the male voice model within the time during which the mouth is moving is extracted from the image and sound information. In the case of a woman, a sound source (voice) is extracted in a similar manner.

Next, as step 140, the image analysis unit 4
In consideration of the image information acquired at 0 and the video display method at the playback unit 90, some of the sound sources separated by the sound source localization unit 70 and the remaining unseparated sound are rearranged in the sound field space. . Based on information on where the object is located in the image, how it is moving, and from which part it is emitting sound, a sound source appropriately separated according to the object is placed in the sound field space. The sound source is localized by placing it. At this time, the remaining sound that has not been separated is processed as a sound in the entire sound field space as a background sound. When arranging sound sources, a video display method in the reproducing unit 90 is taken into consideration.
For example, when considering an image captured by a camera that moves laterally to follow a car moving from right to left, when displaying an image in a rectangular area like a normal TV, the positional relationship between the car and the camera changes. Unless it does, the position of the sound source does not change. However, when this image is displayed in a panoramic manner, the car moves from right to left in the horizontally long panoramic image. In such a case, the sound source localization unit 70 moves the sound source separated by the object information of a car from right to left so as to be suitable for a video display method, in this case, a panoramic image.
In the case of the present embodiment, as shown in FIG. 5, since the camera is fixed, the sound sources are arranged so that each sound source is located at the position of the mouth of the person.

Next, in step 150, the synthesizing unit 80 synthesizes the image divided by the dividing unit 20 and the sound source arranged in the sound field space by the sound source localization unit 70, and reconstructs an image with sound.

Next, in step 160, as shown in FIG. 6, the image with sound (sound information) synthesized and reconstructed by the synthesizing unit 80 is displayed and reproduced in the reproducing unit 90.
As a display method, an image can be displayed in a fixed rectangular area such as a TV, or an image display method different from that can be selectively used. Thus, the process ends.

By performing such processing, the present method makes it possible to automatically convert a monaural sound image into a stereo sound image.

<Embodiment 2> This method is applied to a case where an image (left in FIG. 7) taken from a camera moving laterally to follow a car moving from right to left is developed into a panoramic image. 5 is an example of an embodiment in the case of converting to a stereo sound image suitable for a panoramic image using.

An example of a flowchart showing the processing procedure of this embodiment is the same as that of FIG.

The processing from step 100 to step 130 is the same as that of the first embodiment. The car in the image is extracted by the image analysis unit 40, and the sound source separation unit 60 extracts the engine sound and the like emitted by the car from the sound information. Separate and extract.

When a panoramic image is generated without considering the sound source, the image moves on the panoramic image as shown in the right side of FIG. 7, but the sound source does not move as in the case of displaying on the TV. An unnatural state will result. Thus, in the present embodiment, step 140
In FIG. 8, the sound source localization unit 70 arranges sound sources in a sound field space in consideration of a video display method (a panoramic image in the case of the present embodiment) in the reproduction unit 90, as shown in FIG.

The processing from step 150 to step 160 is the same as in the first embodiment.

As described above, by arranging the sound source in the sound field space in consideration of the video display method in the reproducing unit 90, the sound source is moved in accordance with the car moving on the panoramic image as shown in FIG. This makes it possible to generate a natural stereo sound image even in a video display method different from a display method such as a TV for directly displaying an image taken by a camera.

It goes without saying that some or all of the functions of each unit shown in FIG. 1 can be realized using a computer, or that the processing procedure shown in FIG. 2 can be executed by a computer. In addition, a program for realizing the function of each unit of the apparatus by the computer or a program for causing the computer to execute the processing procedure is stored in a storage medium readable by the computer, for example, FD (floppy disk), MO, ROM, memory card, CD, DV
D, recorded on a removable disk, provided, and distributed.

[0037]

As described above, according to the present invention, according to the present invention, image analysis is performed to extract image information, a sound source based on the image information is separated from sound information added to the image, and The sound source can be rearranged in the sound field space according to the position, movement, and image display method when playing the image, and a new stereo sound image can be automatically synthesized. It is possible to display a stereo sound image suitable for a video display method without disposing.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of an apparatus to which the present invention is applied.

FIG. 2 is a flowchart showing a processing procedure of an embodiment of the method of the present invention.

FIG. 3 is a diagram showing an example of displaying a normal monaural audio image for explaining the present invention.

FIG. 4 is a diagram for analyzing the image in the monaural audio image of FIG. 3 using an image knowledge database for explaining the present invention;
It is a figure showing the example which recognized the movement of the object and the mouth.

FIG. 5 is for explaining the present invention. FIG. 5 is a diagram illustrating an example in which male and female sound sources are arranged in a sound field space corresponding to the position of the mouth recognized by FIG. 4.

FIG. 6 is a diagram illustrating an example in which a sound source and an image arranged in a sound space are re-synthesized and displayed for explaining the present invention.

FIG. 7 is a diagram for explaining the present invention, showing an example of a sound reproduction situation when an image with a camera moving is displayed on a normal TV or the like and when it is developed into a panoramic image.

FIG. 8 is a diagram illustrating an example of rearranging sound sources in a sound field space in consideration of a video display method (panoramic image display method) for explaining the present invention.

FIG. 9 is a view for explaining the present invention, showing an example of displaying a stereo audio image in consideration of a video display method (panoramic image display) by applying the present invention.

FIG. 10 is a diagram illustrating an example of generation and display of a panoramic image according to a conventional method.

[Explanation of symbols]

 Reference Signs List 10 image input unit 20 division unit 30 image knowledge database 40 image analysis unit 50 sound knowledge database 60 sound source separation unit 70 sound source localization unit 80 synthesis unit 90 playback unit

Claims (5)

[Claims] 1. A procedure for inputting an image with sound information, a procedure for dividing sound information and an image from the input image with sound information, and an image knowledge database storing knowledge necessary for analyzing the image Analyzing the divided image based on the information and obtaining image information; anda step of converting the divided sound information into the image information based on a sound knowledge database storing knowledge relating image information and sound information. A step of separating related sound sources, a step of arranging the separated sound sources in a sound field space based on the image information and a reproduction method at the time of image reproduction, and a step of arranging the arranged sound sources and the divided images. A sound source localization method using image information, comprising: synthesizing; and reproducing and displaying the synthesized sound source and image. 2. In the step of analyzing the image and obtaining image information, the image information is obtained by acquiring characteristic information of the object displayed in the input image, position information of the object, and camera movement information. In the step of arranging the separated sound source in the sound field space, the acquired characteristic information of the object, the position information of the object, and the motion information of the camera are used. 2. The sound source localization method using image information according to claim 1, wherein the separated sound sources are arranged in a sound field space corresponding to the reproduced image, taking into account positional information of the object with respect to the reproduced image. . 3. An image input unit for inputting an image with sound information, a division unit for dividing sound information and an image from the input image with sound information, and knowledge necessary for analyzing the image. An image knowledge database, an image analysis unit that analyzes the divided image using information of the image knowledge database to obtain image information, a sound knowledge database storing knowledge relating image information and sound information, A sound source separation unit that separates a sound source related to the image information from the divided sound information by using information of a sound knowledge database; and A sound source localization unit arranged in a field space, a synthesis unit that synthesizes the arranged sound source and the divided image, and a reproduction unit that reproduces and displays the synthesized sound source and the image. A sound source localization device using image information, characterized in that: 4. The image analysis unit acquires, as image information, characteristic information of an object displayed in an input image, position information of the object, and camera movement information. The unit uses the acquired characteristic information of the object, the position information of the object, and the motion information of the camera, and further takes into account the position information of the object with respect to the reproduced image due to the difference in the reproduction method at the time of image reproduction, and reproduces The sound source localization apparatus using image information according to claim 3, wherein the separated sound sources are arranged in a sound field space corresponding to an image. 5. An image, wherein a program for causing a computer to execute a procedure in the sound source localization method using image information according to claim 1 or 2 is recorded on a storage medium readable by the computer. A storage medium storing a program for realizing a sound source localization method using information.