JPH11331827A - Television camera - Google Patents

Abstract

(57) [Problem] To provide a television camera device using a fish-eye or super-wide-angle lens and a variable directivity microphone, which can stably track a person even when no sound is generated during a video conference, and have an activity. A video camera for generating a video conference image, realizing a high-quality realistic video conference with reduced noise and echo, and providing a small, lightweight, and video camera device having no moving parts. SOLUTION: A television camera device 10 includes a fisheye or super-wide-angle lens unit 11 and a CCD imaging unit 13 in a central part, and a plurality of omnidirectional microphones 12 are arranged in a peripheral part.
A variable directional microphone is obtained by controlling the phase of the omnidirectional microphone, the direction of the sound source position (speaker direction) is determined, the direction of the sound source position is tracked, and an image of the sound source position direction (person image of the speaker) is obtained. It has a configuration to cut out and generate a video signal.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a television camera apparatus using a fish-eye or super-wide-angle lens and a variable directivity microphone, and more particularly to a camera apparatus for video conference or a monitor camera apparatus for remote monitoring. Identify the moving image of a person or the like and its direction from the image signal, identify the direction of the speaker from the audio signal, and select the moving person or the image of the speaker from the images captured with a fisheye or ultra-wide-angle lens. The present invention relates to a television camera device for cutting out and projecting.

[0002]

2. Description of the Related Art A conventional television camera apparatus captures video and audio of a conference participant using a fixed camera or a remote-controlled swing camera and a microphone, and converts the video and audio signals into a remote conference room. To a monitor television device via a network.

[0003] The image captured by the above-mentioned fixed camera has a fixed imaging area and does not move. Therefore, the image screen is monotonous. When a conference participant moves out of the imaging area, it goes out of the monitor television screen. A videoconference device using a fixed camera is not suitable for a smooth videoconference. On the other hand, a remote control-operated swing camera requires that the meeting participants operate the swing of the camera while talking, which is cumbersome, the response of the swing operation is slow, and the operability and use of the camera are slow. It is bad.

In order to automatically identify the speaker who is speaking and project the speaker, the camera is automatically turned to track the subject, or a display indicating the speaker is added and projected. A television camera system has been devised (for example, Japanese Patent Application Laid-Open No. H5-153592, Japanese Patent Application Laid-Open No. H07-15-15).
710, JP-A-6-351015, JP-A-5-207449, etc.).

FIG. 12 shows a conventional camera turning system for a video conference. (A) of the figure shows the configuration of the camera swivel system, and (B) of the figure shows the usage status. In the figure, reference numeral 12-1 denotes a first microphone device in which n unidirectional microphones are radially arranged, and 12-2.
Are second microphone devices, 12-3 and 12-, in which m sensitive unidirectional microphones are radially arranged.
Reference numeral 6 denotes a voice recognition unit that detects a microphone to which the loudest voice has been input among the first and second microphone devices. Reference numeral 12-4 denotes a microphone control unit that controls turning of the second microphone device. Reference numeral 5 denotes a camera control unit for controlling turning of the camera, and reference numeral 12-7 denotes a television camera.

[0006] The n unidirectional microphones of the first microphone device 12-1 are fixedly arranged with their directivity directed toward the center of each arc that divides the circumference of the conference table into n equal parts. Have been. Now, as shown in (B) of the figure, a voice input from a microphone whose directivity is directed in the direction of a certain circular arc 12-8 equally divided into n around the circumference of the conference table,
When the first voice recognition unit 12-3 recognizes the loudest voice, the voice recognition unit 12-3 turns the arc 12-8.
Is transmitted to the microphone control unit 12-4 and the camera control unit 12-5.

The microphone control unit 12-4 turns the entire second microphone device 12-2, and moves the second microphone device 12-2 toward the center of the arc portion 12-8 where the loudest voice is recognized. Turn the whole. The m high-sensitivity unidirectional microphones of the second microphone device 12-2 are arranged so that their directivities are directed in directions in which the circumference of the arc portion 12-8 is further equally divided by m. The second voice recognition device 12-6 recognizes a position direction 12-9 where the loudest voice is input from among m position directions obtained by dividing the arc portion 12-8 into m equal parts, and recognizes the position direction information. This is sent to the camera control unit 12-5.

The camera control unit 12-5 determines the position direction 12-9 where the loudest voice is input as the direction of the speaker based on the position direction information sent from the first and second voice recognition units. Then, the television camera 12 is moved toward the position direction 12-9.
Turn -7.

FIG. 13 is a diagram showing a conventional video conference system for displaying speaker positions. In the figure, 13-1
And 13-2 are first and second microphones, and 13-
3 is an audio circuit, 13-4 is a phase difference detection circuit, 13-5 is an arithmetic circuit, 13-6 is a position information circuit, 13-7 is a television camera, 13-8 is a video circuit, and 13-9 is a video synthesis circuit. , 13-10 is a video encoder, 13-11 is a network, 13-12 is a receiving circuit, 13-13 is a monitor television, and 13-14 is a speaker.

The first and second microphones 13-1, 13-1,
13-2 detects the phase difference of the sound input to the phase difference detection circuit 1
3-4, a video signal for calculating the position of the speaker from the phase difference, and displaying the speaker position from the position information;
Video taken by the TV camera 13-7 (video circuit 13-
8) is synthesized by a video synthesizing circuit 13-9, and the synthesized video signal is encoded and displayed on a monitor television 13-13 via a network 13-11.

[0011]

The conventional video conference system requires a preset operation of setting the microphone and the installation position of the television camera in advance for the conference participants. Cannot be changed.

Also, the microphones need microphones for the number of participants in the video conference. Also, there is a problem that a mechanical sound generated at the time of turning the television camera disturbs conversation. In addition, when the position of a speaker is detected based on the phase difference between sounds input from two microphones, the position of the speaker is sensitive to noise and sounds transmitted from speakers of a monitor television device. There are many obstacles to using it in an actual video conference with speaker sounds.

[0013] Further, since the tracking direction is determined only by voice, the tracking direction becomes unstable unless the target person utters, and when a plurality of persons exist in the conference hall, a plurality of persons are projected. Can only track one conference participant at any given time. In addition, there is a problem that the turning sound of the television camera gives a sense of tension or discomfort to the conference participants.

Further, the television camera device is installed in a conference room or the like together with a monitor television device for displaying a remote participant's conference participant and a speaker for emitting the sound, and the sound emitted from the speaker is transmitted to the television. Since an echo is generated by wrapping around the microphone of the camera device, it is necessary to incorporate an expensive echo canceller or echo suppressor that cancels or suppresses the echo.

SUMMARY OF THE INVENTION It is an object of the present invention to stably track a person even when no sound is generated during a video conference, and to generate a video of a video conference with natural and activity. In addition to specifying the position of the speaker, depending on whether the sound source input to the microphone is noise, human voice, or speaker sound of a monitor television device, different directivities are given to the respective sound sources. The purpose is to realize a high-quality, realistic video conference. It is still another object of the present invention to provide a small, lightweight, and no moving part television camera device that collects all voices and images of a conference hall or the like and extracts voices and images to be transmitted from the collected voices and images.

[0016]

According to the present invention, there is provided a television camera apparatus for use in (1) a video conference or remote monitoring, which comprises a camera having a fish-eye or super wide-angle lens, and a plurality of omni-directional cameras. Directional microphone in which directional microphones are arranged, a means for determining the direction of a sound source position by the variable directional microphone, and tracking the direction of the sound source position, and generating a video signal by cutting out a screen in the direction of the sound source position Means for performing the operation.

(2) A television camera device used for video conference or remote monitoring, wherein the camera has a fish-eye or ultra-wide-angle lens, a variable directional microphone in which a plurality of omnidirectional microphones are arranged, and A variable directional microphone determines the direction of the sound source position, and tracks the direction of the sound source position, and detects and tracks a moving image based on an inter-frame difference of an image captured by the camera of the fisheye lens or the ultra-wide-angle lens. And a means for selecting one of the means for tracking the direction of the sound source position and the means for detecting and tracking a moving image, cutting out the screen, and generating a video signal.

(3) The means for tracking the direction of the sound source position has means for detecting voiced sound, and has a configuration for cutting out a screen in the direction of the sound source position where voiced sound is detected and generating a video signal. Things.

(4) The means for tracking the direction of the sound source position has a configuration for synthesizing the output sound signal of the omnidirectional microphone so as to reduce the gain in the direction of the sound source position where no voiced sound is detected. is there.

(5) The means for tracking the direction of the sound source position has means for detecting a speaker sound signal of the monitor television device, and the output sound signal of the omnidirectional microphone and the speaker sound of the monitor television device. It has a configuration of calculating a correlation with a signal, determining a speaker position, and synthesizing an output sound signal of the omnidirectional microphone so as to suppress speaker sound output from the omnidirectional microphone. .

(6) In a television camera device provided with a means for tracking the sound source position direction and a means for detecting and tracking a moving image, a screen is cut out with priority given to the means for tracking the sound source position direction. It has a configuration for generating a video signal that emphasizes and displays a speaker.

(7) In a television camera device provided with means for tracking the direction of the sound source position and means for detecting and tracking a moving image, a screen is displayed by giving priority to the means for detecting and tracking a moving image. It has a configuration for generating a video signal that cuts out and displays a moving person with emphasis.

(8) The apparatus further comprises means for tracking the direction of the sound source position and means for detecting and tracking a moving image, and has a configuration in which one of the two tracking means is selected by key input. .

(9) A fish-eye or super-wide-angle lens camera and a plurality of omni-directional microphones are integrated, and the fish-eye or ultra-wide-angle lens is arranged at the center, and the plurality of omni-directional microphones are integrated. They are arranged in a peripheral part.

[0025]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a diagram showing a configuration and a use state of a television camera device according to an embodiment of the present invention. FIG. 1A is a plan view of a television camera device according to an embodiment of the present invention, and FIG. 1B is a cross-sectional view taken along line a1-a2. (C) of the drawing shows a use situation when the television camera device according to the embodiment of the present invention is used for a video conference.

In the figure, reference numeral 10 denotes an entire television camera device, 1
1 is a fisheye lens unit, 12 is an omnidirectional microphone, 1
3 is a CCD imaging unit, 14 is a conference participant, 15 is a conference table,
Reference numeral 16 denotes a monitor television device.

The television camera device 10 has a fisheye lens portion 11 of the camera disposed in the center thereof, and a plurality of omnidirectional microphones 12 disposed at equal intervals around the periphery. The output audio signal of each omnidirectional microphone 12 is
By synthesizing while changing the amount of delay (phase) according to the characteristics of the sound, the entire device functions as a variable directional microphone having different directivities depending on the characteristics of the sound.

Since the fisheye lens has an angle of view of about 180 degrees, as shown in FIG.
Is installed near the center of the conference table 15 on which the conference participants are seated, with one facing upward, so that images of all conference participants can be projected. Then, an image of interest such as a speaker or a moving person is cut out from the video taken by the fish-eye lens unit 11, the fish-eye lens distortion of the cut-out image is corrected, and the image is output as a video signal. It should be noted that a super wide angle lens can be used instead of the fisheye lens.

By automatically determining and switching the image to be cut out based on the position identification of the speaker or the moving person, no moving device such as a swivel table is used, so that no turning sound is generated, and the conference participants Can hold a meeting in a natural atmosphere without being conscious of the camera.

The number of omnidirectional microphones 12 to be arranged in the television camera device 10 is usually sufficient for three or four microphones. However, a sharper and more accurate directivity can be obtained for a large number of conference participants. In order to be able to respond, more omnidirectional microphones need to be arranged.

FIG. 2 is a diagram showing a configuration of a video conference system using the television camera device according to the embodiment of the present invention. FIG. 2A shows the configuration of the image processing unit, and FIG. 2B shows the configuration of the audio processing unit. 2A-1 is a fish-eye lens camera, 2A-2 is a person position detector, 2A-3
Is a person position determination unit, 2A-4 is an image cutout / distortion correction unit, 2A-5 is an image communication unit (codec), 2A-6 is a line connected to a remote partner's video conference system,
2A-7 is a monitor television device for displaying an image of the other party's conference hall.

2B-1 is a microphone bank and voice signal processing section, 2B-2 is a voiced sound detection section, 2B-3 is a speaker position calculation section, 2B-4 is a noise source position calculation section, and 2B-5 is a speaker sound. Detector, 2B-6 is speaker position calculator, 2B-7 is microphone directivity determiner, 2B-8
Is a directivity control unit, 2B-9 is a voice communication unit (codec), 2B-10 is a line connected to the other party's video conference system, 2B-11 is a sound quality improvement unit, and 2B-12 is the voice of the other party's conference room. Speaker.

An image signal photographed by the fish-eye lens camera 2A-1 is output to a person position detecting section 2A-2 and an image cutout / distortion correcting section 2A-4, and the person position detecting section 2A-
2 detects the image of the moving person and its position, and sends the position information to the person position determining unit 2A-3.

The person position determination unit 2A-3 is based on the person position information output from the person position detection unit 2A-2 and the speaker position information output from a speaker position calculation unit 2B-3 described later. Then, a person position area cut out from the image captured by the fisheye lens and projected is determined according to the set priority order.

The image cutout / distortion correction unit 2A-4 cuts out the image portion of the person image from the fisheye lens image based on the person position area information output from the person position determination unit 2A-3, and The distortion of the portion caused by the fisheye lens is corrected by a mapping conversion operation, and a video signal of a normal image without distortion is transmitted to the image communication unit 2A-5.

The image communication unit 2A-5 encodes the video signal sent from the image clipping / distortion correcting unit 2A-4 in a code format suitable for the transmission system, and transmits the video signal to the other party via the line 2A-6. Send to system. The image communication unit 2A-5 decodes the other party's video code signal received from the other party via the line 2A-6, sends the other party's video signal to the monitor television apparatus 2A-7, and -7 displays the image of the other party by the video signal.

FIG. 3 is a diagram showing the configuration of the above-mentioned person position detecting section 2A-2. In the figure, 3-1 is an A / D converter, 3-2 is a frame memory, 3-3 is an inter-frame difference detection unit, 3-4 is an intra-frame difference detection unit, and 3-5 is a moving image contour extraction unit. , 3-6 are moving person position calculation units.

An image picked up by the fish-eye lens camera 2A-1 is converted into a digital signal by an A / D converter 3-1 and stored in a frame memory 3-2 on a frame-by-frame basis. The difference from the previous frame is detected, and the moved image portion is detected. Then, the difference between adjacent image signals in one frame is detected by the intra-frame difference detection unit 3-4, and the outline of the image is detected.

The moving image contour extracting unit 3-5 extracts the contour part of the moving image by combining the outputs of the inter-frame difference detecting unit 3-3 and the intra-frame difference detecting unit 3-4. The moving person position calculating unit 3-6 includes a moving image contour extracting unit 3
-5 to calculate the position of the contour portion of the moving image output from
The position information is output to the person position determination unit 2A-3 in FIG.

The person position determination unit 2A-3 in FIG. 2 determines a person position area to be projected based on the position information of the moving person or the speaker position information described later, and extracts the information of the person position area from an image cut-out. This is sent to the distortion correction unit 2A-4. here,
The principle of clipping a display image from a fisheye lens image and correcting distortion in the image clipping / distortion correcting unit 2A-4 will be described.

FIG. 4 is a view for explaining the principle of clipping an image of a fisheye lens and correcting distortion. In the figure, 4-1 is a virtual image frame, 4-2 is a virtual hemisphere, and 4-3 is a fisheye lens imaging screen. It is assumed that the center of the fisheye lens is located at the origin O of the three-dimensional space indicated by the x, y, and z axes, and that the fisheye lens is oriented in the direction of the z axis.

The virtual hemispherical surface 4-2 is a virtual hemispherical spherical surface having a radius f if the fisheye lens is an orthographic lens and its focal length is f. It is placed on the y-plane, and its center is located at the position of the origin O.

The virtual image frame 4-1 is a frame on a virtual plane orthogonal to a line of sight vector DOV (Direction Of View) from the position (origin) of the fisheye lens toward the object to be photographed, and has a predetermined size. As will be described later, an image in this frame is cut out from the fisheye lens image and displayed on the screen of the monitor television device. That is, the frame has the same size as the frame of the image display frame of the monitor television device.

The virtual image frame 4-1 has a two-dimensional coordinate axis (p, q) having an origin at a point intersecting with the line-of-sight direction vector DOV within the virtual image frame 4-1. The point is represented by the component (p, q) of this coordinate.

The image projected by the fish-eye lens (fish-eye lens image) is obtained by the projection method of the lens (projection fomu).
la) can be divided into several types.
The fisheye lens of the orthographic projection system will be described.

It is assumed that an image seen through the virtual hemisphere 4-2 from the position where the fisheye lens is placed (origin O) is directly attached to the virtual hemisphere 4-2. Then, the image of the three-dimensional space pasted on the hemisphere 4-2 is placed on the plane of the two-dimensional space consisting of the x-axis and the y-axis in the drawing.
The image which is crushed and attached perpendicularly to the plane (from the z direction to the origin) is an image of the orthographic fisheye lens.

In practice, the image of the fisheye lens forms an image at a position where the image is inverted upside down, left and right, but since the relative positional deviation of the image distortion does not change, as described above for simplification of the description. It forms an image.

A plane in a two-dimensional space obtained by vertically crushing the image on the hemisphere 4-2 is a fisheye lens imaging screen 4-3, and an image inside the circle of the fisheye lens imaging screen 4-3 is photographed with a fisheye lens. Image. The outer square frame is
This is the outer frame of the entire imaging screen obtained from the CCD imaging device.

Since the image of the fisheye lens is distorted, it is necessary to cut out a part of the image and correct it to a normal image in order to display the image taken by the fisheye lens camera. Therefore, a frame corresponding to the frame to be displayed is assumed to be a virtual image frame 4-1.

It is assumed that the direction of the z-axis to which the fisheye lens is directed is a direction above the vertical line, and the line-of-sight direction vector D
The vertex angle (zenith angle) formed by the OV with the z-axis is φ, and the azimuth formed by the horizontal reference axis (x-axis or y-axis) is θ. The focal length (the radius of the circle of the fisheye lens image) is f, the rotation angle of the image frame is ω, and the magnification is m. The magnification factor m is m = D / f, where D is the distance from the position of the fisheye lens (origin O) to the origin in the image frame 4-1.

The point (x,
The correspondence between y) and the point (p, q) in the image frame 4-1 is represented by the following relational expression. x = R (pA-qB + mR sin φ sin θ) / (p ²
+ Q ² + m ² R ² ) ^1/2 y = R (pC-qD-mR sin φsin θ) / (p ²
+ Q ² + m ² R ² ) ^1/2 A = cosωcosθ−sinωsinθcosφ B = sinωcosθ + cosωsinθcosφ C = cosωsinθ + sinωcosθcosφ D = sinωsinθ−cosωcosθcosφ

The distortion of the fisheye lens image can be corrected using the above relational expression, and the original image can be restored and displayed on the monitor television device. That is, first, the image frame 4-1 is localized, and an area (cutout area) to be displayed on the monitor is determined. In this localization operation, the vertex angle φ, the azimuth angle θ, the rotation angle φ, and the enlargement factor m of the image frame are converted from the above-described person position determination unit 2A-3 into the image cutout / distortion correction unit 2A.
-4.

The image clipping / distortion correcting unit 2A-4 calculates the point (x, y) in the fisheye lens image plane 4-3 corresponding to the point (p, q) in the image frame 4-1 by the above-mentioned relational expression. , The color information signal at that point is read out from a video frame memory (not shown) storing the fisheye image, and output for an output having an address corresponding to the point (p, q) in the image frame 4-1. The color information signal is written into a video memory (not shown).

The color information transfer operation from the video frame memory storing the fisheye image to the output video memory having an address corresponding to the point of the image frame 4-1 is performed for all points in the image frame 4-1. By performing (p, q) and sequentially reading out and outputting the color information signals from the output video memory, the image cutout / distortion correction unit 2A-4
Can generate a video signal without distortion of a real object viewed from the origin O through the frame of the virtual image frame 4-1. (See U.S. Pat. No. 5,185,667 or the specification of Japanese Patent Application No. 10-62531 filed earlier).

Next, with reference to FIG. 2B, the person tracking by voice will be described. An output signal of a microphone bank in which a plurality of omnidirectional microphones 12 are arranged is converted into a digital signal by an audio signal processing unit 2B-1, and a voiced sound (vowel) is included by a voiced sound detection unit 2B-2. And a speaker position calculator 2B-3 calculates a speaker position for a voice signal including a voiced sound from a phase difference of an output signal of the microphone bank, and outputs information on the speaker position to a person position determiner. 2A-3 and the microphone directivity determining unit 2B-7.

For an input sound signal that does not include a voiced sound, the position of the noise source is similarly calculated from the phase difference by the noise source position calculation unit 2B-4, and the information on the noise source position is transmitted to the microphone. It is sent to the sex determination section 2B-7. Whether or not the input sound signal includes voiced sound can be detected by applying a filter coefficient of linear predictive coding (LPC) or a technique of cepstrum analysis.

Further, in order to prevent the sound transmitted from the other party via the line 2B-10 from being emitted by the speaker 2B-12, the sound is input to the microphone bank and transmitted to the other party as an echo. , The speaker sound detection unit 2B-5 detects the correlation amount between the audio signal input to the speaker 2B-12 and the audio signal input from the microphone bank, and calculates the speaker position calculation unit 2B-5.
B-6 detects the position direction of the speaker 2B-12 based on the correlation amount, and outputs the position information to the microphone directivity determination unit 2B-7.

The microphone directivity determining unit 2B-7
The directivity of the microphone bank for each sound source is determined based on the speaker position information, the noise source position information, and the speaker position information. The directivity control unit 2B-8 gives a different phase to the output signal of each omnidirectional microphone for each sound source in accordance with the directivity determined by the microphone directivity determining unit 2B-7, and synthesizes the sound sources. Different directivities are provided for each.

As a result, a high-gain directivity is produced for a sound source in the direction of a speaker including voiced sounds, and a direction of a noise source which is always input and a direction of a speaker of a monitor television apparatus are generated. By generating a low-gain directivity, it is highly sensitive to voice and low to noise and speaker sound, suppressing noise and speaker sound and reducing the occurrence of echo. Can be.

FIG. 5 is an explanatory diagram of a configuration for generating directivity for suppressing noise according to the embodiment of the present invention. In the figure, M0 to M3 are omnidirectional microphones, 5-1 is an A / D converter, 5-2 is a cross-correlation calculator, 5-3 is a sound source position calculator, and 5-4 is a speaker direction / characteristic. Storage unit, 5-5 is a filter and delay unit, 5-6 is a speech synthesis unit, 5-7
Denotes a level adjustment / adaptive filter unit, and 5-8 denotes an addition unit.

Output signals from the omnidirectional microphones M0 to M3 of the microphone bank are supplied to the A / D converter 5
The signal is A / D converted by -1 and input to the cross-correlation calculator 5-2. The cross-correlation calculation unit 5-2 obtains a mutual correlation amount and a mutual phase difference between input signals from the other omnidirectional microphones M1, M2, and M3 with respect to an input signal from the omnidirectional microphone M0, and calculates them as sound sources. Position calculation unit 5-
3

The sound source position calculation unit 5-3 calculates the sound source position based on the correlation amount and the mutual phase difference, and sets each omnidirectional microphone M0 in order to make the direction of the sound source position a negative directivity (suppression characteristic). To calculate the mutual delay value (phase amount) given to the input signal from M3, and send the calculated value to the filter and delay unit 5-5. The cross-correlation calculator 5-2 and the sound source position calculator 5-3 are the same as the noise source position calculator 2B-4 and the microphone directivity determiner 2B-8 shown in FIG.
Corresponding to

The filter and delay unit 5-5 delays the input signals from the omnidirectional microphones M0 to M3 in accordance with the delay value sent from the sound source position calculation unit 5-3 to make the phases uniform, The voice synthesizing unit 5-6 synthesizes them, the level adjustment / adaptive filter unit 5-7 relatively adjusts the level of the synthesized signal to the level of the input signal from the microphone M0, and the adder 5-8 , The inverted output signal from the level adjustment / adaptive filter section 5-7 is added to the input signal from the microphone M0 to cancel each other, thereby generating a negative directivity (suppression characteristic) for the sound source.

Therefore, the signal output from the adder 5-8 is a signal that is suppressed from unnecessary audio band signals from a noise source or the like. The filter / delay section 5-5, voice synthesis section 5-6, level adjustment / adaptive filter section 5-7, and addition section 5-8 in FIG. 5 are combined with the directivity control section 2B in FIG.
-8.

The level adjustment / filter unit 5-7 includes:
By using an adaptive filter that minimizes the level of the noise suppression sound output of the adder 5-8, a more accurate noise suppression effect can be obtained. When there are a plurality of noise sources, the same processing is sequentially performed for each noise source in order from the noise source having a large level, so that the noise sources can be simultaneously suppressed.

FIG. 6 is an explanatory diagram of a configuration for generating suppression and enhancement directivity according to the embodiment of the present invention. This configuration includes a voice detection unit, generates suppressed directivity for noise, and generates enhanced directivity for voice. In the figure, reference numeral 6-1 denotes a voiced sound detection unit, and the other configuration is the same as the configuration shown in FIG. 5, and therefore, the same components are denoted by the same reference numerals, and redundant description will be omitted. .

The output signal from the omnidirectional microphone M0 is input to the voiced sound detector 6-1 and the voiced sound detector 6-1 is input.
Detects whether a voiced sound is included in the input signal and sends the detection result to the sound source position calculation unit 5-3. In addition,
The voiced sound detection section 6-1 is the voiced sound detection section 2B- shown in FIG.
Equivalent to 2.

When a voiced sound is included, the sound source position calculation unit 5
-3 is determined to be a voice uttered by the speaker, and based on the mutual phase difference and the correlation amount of each of the omnidirectional microphones M0 to M3, the directivity in the direction is given to the filter and the delay unit 5-5. Set the delay to emphasize the gender.

The omnidirectional microphones M0 to M
The position of the speaker (sound source position) is calculated based on the mutual phase difference and the correlation amount of No. 3 and the speaker position (sound source position) is sent to the speaker direction / characteristic storage unit 5-4. Speaker direction / characteristic storage unit 5-
4 stores the direction of the speaker position and the characteristics of the sound source (speaker voice), and sends the speaker position information (sound source position information) to the above-described person position determination unit 2A-3 in FIG.

If the voiced sound detecting section 6-1 does not detect any voiced sound, it is determined that the sound is noise, and the sound source position calculating section 5-
3 sets the filter and the delay value (directivity parameter) of the delay unit 5-5 so that the energy of the synthesized signal from the sound source is minimized. The setting method is the same as the method described in the configuration of FIG.

FIG. 7 is an explanatory diagram of a configuration for generating speaker sound suppression directivity according to the embodiment of the present invention. In the figure, 7-1 is a speaker sound detection unit. The other configuration is the same as the configuration shown in FIG. 5, and therefore, the same components are denoted by the same reference symbols, and redundant description will be omitted.

The speaker sound detector 7-1 corresponds to the speaker sound detector 2B-5 shown in FIG. 2, and uses an input signal to a speaker of the monitor television device as a reference signal. When the speaker sound detection unit 7-1 detects an input signal to a speaker of a predetermined level or higher, the cross-correlation calculation unit 5-2 causes each of the omnidirectional microphones M0 to M1.
The sound source position calculation unit 5-3 calculates the mutual phase difference and the correlation amount between M3 and M3 and the speaker sound, and calculates the suppression directivity parameter of the speaker sound.

Then, as in the case of the noise suppression control,
A delay is applied to the filter and delay unit 5-5 so that the speaker sound is suppressed, and a voice synthesis unit 5-6 is provided.
Are synthesized so as to cancel each other out for the speaker sound.

FIG. 8 is an explanatory diagram of a configuration for synthesizing the speaker's voice, noise, and speaker sound according to the embodiment of the present invention. In the figure, reference numerals 8-1 and 8-2 denote synthesis units, and reference numerals 8-3 and 8-4.
Is a filter unit. As described in the above-described configurations of FIGS. 5 to 7, the speaker's voice, the noise, and the speaker's sound are distinguished as sound sources having different characteristics, and after emphasis or suppression directivity is given to each sound source. Of the extracted sounds, the speaker's voice is emphasized by the synthesis units 8-1 and 8-2, and the noise is suppressed by the synthesis unit 8-1 after passing through the filter unit 8-3. , Speaker sound is filtered by 8-
After passing through 4, the signal is suppressed by the synthesis unit 8-2, and a final synthesized sound is generated from the synthesis unit 8-2.

FIG. 9 is a diagram showing the directivity of the embodiment of the present invention with respect to speaker voice, noise, and speaker sound. 9
-1 is a speaker voice source, 9-2 is a noise source, 9-3 is a speaker sound source, 9-4 is directivity for a speaker voice source, 9-5 is directivity for a noise source, and 9-6 is a speaker sound source. 9-7 is a television camera device.

The output sound is obtained by synthesizing the extracted sound obtained by the structure shown in FIGS. 5 to 7 by the structure shown in FIG. This output sound has a strong directivity for the speaker's voice and a weak directivity for the noise and the speaker sound. Accordingly, noise is reduced, a high quality speaker's voice is generated, the wraparound of the speaker sound is reduced, and the function of the speaker sound of the present invention is equivalent to that of providing an echo canceller or an echo suppressor, and Since the wraparound can be performed by also using a configuration for generating directivity for another sound source, the same function can be performed with a much smaller amount of calculation processing than when an echo canceller or an echo suppressor is separately provided. be able to.

FIG. 10 is a flowchart of the directivity control of the variable directivity microphone according to the embodiment of the present invention.
In step 10-1, initialization of each unit of the audio processing unit of the television camera device is performed, monitoring loop processing is performed in step 10-2, detection and determination of speaker sound is performed in step 10-3, and speaker sound is detected. In this case, in step 10-4, the cross-correlation between each microphone input sound and the speaker sound is calculated, and
In step -5, a correlation parameter (phase difference, correlation amount) is calculated. In step 10-6, a delay and a coefficient are given to the output of each microphone so as to reduce the directivity, and synthesis (addition / subtraction) is performed. Return to 3.

If no speaker sound is detected, it is determined in step 10-7 whether or not the volume of the voiced sound is loud. If the sound is loud, each microphone input sound is determined in step 10-8 to track the speaker direction. In step 10-9, a correlation parameter (phase difference, correlation amount) is calculated, and in step 10-6, a delay and a coefficient are given to the output of each microphone so that the directivity increases. Combine (addition / subtraction) and return to step 10-3.

If the volume of the voiced sound is low, the cross-correlation between the microphone input sounds is calculated in step 10-10 so as to cancel the noise.
Calculates the correlation parameters (phase difference, correlation amount) at
In step 10-6, a delay and a coefficient are given to the output of each microphone to combine (addition / subtraction) so that the directivity is reduced, and the process returns to step 10-3.

Next, the integrated tracking by the image and the sound according to the present invention will be described. As described above, the combination of the tracking of the moving image by the fisheye image and the tracking of the speaker by the phase difference of the input sounds of the plurality of omnidirectional microphones enables the detection and tracking of the direction in which the moving person exists and the tracking of the speaker. The direction can be detected and tracked.

The operation of cutting out a part of persons from the whole sky image by the fish-eye lens camera can be executed simultaneously for a plurality of persons at the same time. The results can be adopted independently of each other, and it is possible to select moving person tracking priority, speaker tracking priority, and the like by setting from the outside.

The person position determining unit 2A- shown in FIG.
3 is a person position detecting unit 2A-2 or a speaker position calculating unit 2B
Based on moving person position information or speaker position information obtained from -3, tracking of a moving image or tracking in a sound source position direction is performed in accordance with a set priority order. The tracking of the moving image or the tracking in the direction of the sound source position may be selected by key input.

FIG. 11 is a diagram showing an example of a transmission image according to the embodiment of the present invention. (A) of the figure shows a case where a clipping operation is simultaneously performed on four persons in parallel to form a composite image of four screens. When each person moves without having a conversation, the individual person tracking function works, and the person is displayed in the always divided video.

Further, in the case where the person continues a conversation in a stationary state, the positioning accuracy is improved by voice tracking, so that the face portion of the person can be specified, and the upper right screen of FIG. Zooming up can be performed as in the lower left screen of (C). Naturally, if the person who is having a conversation changes or if there are a plurality of persons, the person can be zoomed up accordingly. As described above, by performing zooming on a speaker, a video having more activity can be reliably and quickly performed without a human operation, as compared with the related art.

[0085]

As described above, according to the present invention,
With a fish-eye or ultra-wide-angle lens camera and a variable directional microphone in which a plurality of omnidirectional microphones are arranged, all images and voices are collected, and images of a speaker or a moving person are cut out from the images and displayed. With this configuration, it is possible to configure a compact and simple television camera device that has no moving parts and does not give a sense of discomfort to conference participants.

Further, in specifying the sound source position direction using a plurality of omnidirectional microphones, voiced sound and correlation with speaker sound are detected, and different directivities are generated in accordance with the sound quality of the sound source, thereby producing noise. It is possible to realize a high-quality realistic video conference with little or no echo.

Further, by providing a function of tracking a moving person with a fish-eye or ultra-wide-angle lens camera and a function of tracking a speaker with a variable directional microphone, an image of a person who speaks without moving and a voice can be generated. It is possible to stably perform high-speed and high-accuracy clipping of an image of a person who moves without moving, and generate an image with activity.

Further, when used as a surveillance television camera device or the like, even if it is difficult to cut out a moving image due to backlight or insufficient lighting, etc., it is possible to detect a sound generation direction to obtain a stable cutout of a person. It is possible to set a cutout process adapted to the use situation, such as enabling tracking.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration and a use state of a television camera device according to an embodiment of the present invention.

FIG. 2 is a diagram showing a configuration of a video conference system using the television camera device according to the embodiment of the present invention.

FIG. 3 is a diagram illustrating a configuration of a person position detecting unit of the television camera device according to the embodiment of the present invention.

FIG. 4 is a diagram illustrating the principle of clipping an image of a fisheye lens and correcting distortion.

FIG. 5 is an explanatory diagram of a configuration for generating a directivity for suppressing noise according to the embodiment of the present invention;

FIG. 6 is a diagram illustrating a configuration for generating suppression and enhancement directivity according to the embodiment of the present invention.

FIG. 7 is an explanatory diagram of a configuration for generating speaker sound suppression directivity according to the embodiment of the present invention.

FIG. 8 is an explanatory diagram of a configuration for synthesizing a speaker voice, noise, and a speaker sound according to the embodiment of the present invention.

FIG. 9 is a diagram illustrating directivity of a speaker voice, noise, and speaker sound according to the embodiment of the present invention.

FIG. 10 is a flowchart of directivity control of the variable directivity microphone according to the embodiment of the present invention.

FIG. 11 is a diagram showing an example of a transmission image according to the embodiment of the present invention.

FIG. 12 is a diagram showing a conventional camera turning system for a video conference.

FIG. 13 is a diagram showing a conventional video conference system for displaying speaker positions.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 Television camera apparatus 11 Fisheye lens part 12 Non-directional microphone 13 CCD imaging part 14 Conference participant 15 Conference table 16 Monitor television apparatus

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁶ Identification symbol FI H04N 5/268 H04N 7/15 7/15 G06F 15/62 380 15/64 340B 15/66 470A

Claims (9)

[Claims] 1. A television camera device used for video conference or remote monitoring, comprising: a camera having a fisheye or super wide-angle lens; a variable directional microphone in which a plurality of omnidirectional microphones are arranged; A television camera device comprising: means for determining the direction of a sound source position and tracking the direction of the sound source position; and means for cutting out a screen in the direction of the sound source position and generating a video signal. 2. A television camera device used for video conference or remote monitoring, comprising: a fish-eye or super wide-angle lens camera; a variable directional microphone in which a plurality of omnidirectional microphones are arranged; Means for determining the direction of the sound source position and tracking the direction of the sound source position; means for detecting and tracking a moving image based on a difference between frames of an image captured by a camera of the fisheye lens or the ultra-wide-angle lens; A television camera device comprising: means for selecting one of a means for tracking a position direction and a means for detecting and tracking a moving image to cut out a screen and generate a video signal. 3. The apparatus according to claim 2, wherein the means for tracking the direction of the sound source position includes means for detecting a voiced sound, and has a configuration for cutting out a screen in the direction of the sound source position where the voiced sound is detected and generating a video signal. The television camera device according to claim 1 or 2, wherein 4. The apparatus according to claim 1, wherein the means for tracking the direction of the sound source position synthesizes an output sound signal of the omnidirectional microphone so as to reduce a gain in a direction of the sound source position where no voiced sound is detected. The television camera device according to claim 1. 5. The means for tracking the direction of a sound source position has means for detecting a speaker sound signal of a monitor television apparatus, and a means for detecting an output sound signal of the omnidirectional microphone and a speaker sound signal of the monitor television apparatus. Calculate the correlation,
5. The apparatus according to claim 1, wherein a speaker position is determined, and an output sound signal of the omnidirectional microphone is synthesized so as to suppress a speaker sound output from the omnidirectional microphone. A television camera device according to item 1. 6. A television camera device comprising: means for tracking the direction of a sound source position; and means for detecting and tracking a moving image. The television camera device according to any one of claims 2 to 5, further comprising a configuration for generating a video signal to be displayed in an emphasized manner. 7. A television camera apparatus comprising: means for tracking the direction of a sound source position; and means for detecting and tracking a moving image. 6. The television camera device according to claim 2, wherein the television camera device has a configuration for generating a video signal for highlighting and displaying a person. 8. A system comprising: means for tracking the direction of the sound source position; and means for detecting and tracking a moving image, wherein one of the two tracking means is selected by key input. Item 6. A television camera device according to any one of Items 2 to 5. 9. A fish-eye or ultra-wide-angle lens camera and a plurality of omni-directional microphones are integrated, the fish-eye or ultra-wide-angle lens is arranged at a central portion, and the plurality of omni-directional microphones are arranged at a peripheral portion. 9. The television camera device according to claim 1, wherein the television camera device is arranged.