JP2009049734A - Camera-mounted microphone and control program thereof, and video conference system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To allow a caller to accurately confirm that the caller is located within the range of microphone directivity. <P>SOLUTION: A camera-mounted microphone 1 has the microphone 11, attached to its body housing 10 and has single directivity, and a camera 12 which is attached to the body housing 10 and has an image angle, substantially corresponding to the range of single directivity of the microphone 11. As a result, the caller is photographed by the camera 12, so that the caller can confirm that he or she is located within the range of the microphone directivity. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a microphone with a camera in which a microphone and a camera are provided in a housing, a control program for the microphone with a camera, and a video conference system.

  The video conference system is a system that realizes a realistic conference by projecting video and audio of the other party's speaker on a monitor by bidirectionally communicating video and audio between remote conference rooms. Here, when the camera is directed toward a plurality of participants in the conference room, a technique for controlling the direction of the camera in accordance with the position where the microphone is capturing audio is disclosed (for example, see Patent Document 1). ).

JP 2002-171499 A

  However, when a microphone having directivity is used in a video conference system or the like, it is difficult for a speaker to know whether or not he is within the directivity range. In addition, when a plurality of directional microphones are used, if a noise source exists within the directional range of a certain microphone, it may cause difficulty in listening to sounds that require input sound from the directional microphone.

  Furthermore, when performing a video conference with the main camera equipped with a plurality of sound source direction estimation microphones facing the direction of the speaker, a correlation function between the input sounds of each sound source direction estimation microphone by voice processing was used. Estimate the direction of the sound source (for example, “Toshiyuki Morita: Examination of an automatic surveillance camera using the first wavefront sound source direction detection”), etc., and further face detection using pattern recognition by image processing (for example, “Akamatsu Shigeru: A computer-based facial recognition survey)) may be used to point the main camera toward the speaker, but the accuracy is still insufficient.

  Further, when the speaker is visually emphasized only by the video of the main camera, there is a problem that the portion where the speaker is present must be accurately extracted in the video.

  The present invention has been made to solve such problems. That is, the present invention is equipped with a camera including a unidirectional microphone provided in the main body casing and a camera provided in the main body casing and having a field angle substantially equal to the range of the unidirectionality of the microphone. It is a microphone.

  In the present invention, a microphone having a unidirectionality and a camera are provided in the same main body casing, and there is a camera having an angle of view substantially equal to the range of the unidirectionality of the microphone. The person can grasp that the image is captured by the camera and is within the directivity range of the microphone.

  Here, the unidirectionality of the microphone applied in the present invention means that a certain level or more of sound is captured mainly at a frequency corresponding to a general speaker's voice only in a region having a certain angle from the main body casing. This is a property having a gain. In addition to the case where the angle of view of the camera approximately equal to the unidirectionality of the microphone matches the range of the unidirectionality of the microphone, if the image is captured by the camera, the sound is captured by the microphone. Including the case where

  In addition, the present invention provides a microphone with a camera provided with a unidirectional microphone provided in a main body casing and a camera provided in the main body casing and having an angle of view within a range of the unidirectionality of the microphone. Is a computer-controlled program that recognizes a facial image from video captured by a camera and causes the computer to execute a step of changing the unidirectional center position of the microphone based on the recognized facial position. is there.

  In the present invention, the face image is recognized based on the video captured by the camera-equipped microphone, and the unidirectional center position of the microphone is changed to the face position. Even if the position of the person is deviated, it is possible to reliably capture the sound by the microphone.

  Here, if the face image cannot be recognized from the video captured by the camera, and if the voice is not captured by the microphone, unnecessary audio can be generated if the speaker is not within the camera angle of view. There is no need to import.

  Also, the present invention provides a microphone having a single directivity and a plurality of microphones with a camera having a camera body having an angle of view within a range of the single directivity of the microphone in a predetermined layout. In a video conference system that captures the video of participants who use microphones with the main camera, the microphone with the camera capturing the audio is identified, the shooting direction of the high-quality main camera is moved toward that position, and the audio is This is a video conference system having a control means for recognizing a face image from an image captured by a camera-equipped microphone with a camera and adjusting a shooting direction of the main camera based on the recognition result.

  In such a present invention, in a video conference system having a plurality of microphones with a camera and a main camera, the shooting direction of the main camera is moved in accordance with the position of the plurality of microphones with a camera that are capturing audio, By recognizing the face image from the video captured by the camera-equipped microphone, the shooting direction of the main camera can be accurately adjusted to the face position.

  Further, in this video conference system, a plurality of microphones with cameras sequentially capture images of participants' faces by each camera and send them to the control means, and the control means sends images of the faces of participants sent from the microphones with cameras. Are sequentially overwritten and stored, and the stored face images are used for face image recognition when the shooting direction is adjusted by the main camera. As a result, the main camera can be directed to an accurate position by always using a new face image.

  Also, the present invention provides a microphone having a single directivity and a plurality of microphones with a camera having a camera body having an angle of view within a range of the single directivity of the microphone in a predetermined layout. In a video conference system that displays video captured by a microphone camera on a monitor, a microphone with a camera with the highest audio capture level is required to display a plurality of video captured by a camera with multiple cameras side by side on a monitor. This is a video conference system having a control means for displaying the video captured in (1) larger than the video captured by a camera of another microphone with camera.

  In the present invention, when images captured by a plurality of camera-equipped microphones are displayed side by side on a monitor, the images captured by the camera-equipped microphone camera that captures the audio are displayed larger than other images. This makes it possible to make the speaker's video stand out.

  In addition, even if the video captured by the camera-equipped microphone that captures the audio is displayed brighter than the other video, the video of the speaker can be conspicuous as described above.

  In addition, when multiple microphones with cameras are connected in cascade, if multiple images captured by the camera microphones are displayed in a horizontal row on the monitor in the order of the cascade connection, the images of multiple participants are displayed in a panorama view. It becomes possible.

  Therefore, by using the camera-equipped microphone as in the present invention, it is possible to accurately recognize that the speaker is within the microphone directivity range, and using the video captured by the camera-equipped microphone camera, It is possible to accurately set the shooting direction of the main camera by accurately capturing the position. In addition, since the speaker's position can be accurately identified, it is possible to accurately identify the video to be emphasized even when emphasizing the speaker's monitor display, and to realize a realistic video conference. Become.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Configuration of microphone with camera>
FIG. 1 is a schematic diagram for explaining a microphone with a camera according to the present embodiment, where (a) is a plan view and (b) is a side view (c). This microphone 1 with a camera is used by placing a main body casing 10 on a table, and has a microphone 11 provided in the main body casing 10 and a camera 12 provided in the same main body casing 10. is doing.

  In the present embodiment, the microphone 11 has a single directivity, and has a strong sound capturing gain on one side of the main body housing 10. That is, a range having a constant voice capture gain at a frequency corresponding mainly to the voice of a general speaker is provided on one side of the main body housing 10. For this reason, it has a certain angle range as unidirectionality.

  In addition, the camera 12 of the microphone 1 with a camera according to the present embodiment has an angle of view that is substantially equal to the unidirectional range of the microphone 11. As a result, the range in which the image is captured by the camera 12 is always within the directivity range of the microphone 11.

  In the example shown in FIG. 1, the microphone 11 is disposed at the center of the upper surface of the main body housing 10 and has directivity indicated by a one-dot chain line in the drawing. On the other hand, the camera 12 is disposed on the upper surface of the main body housing 10 and has an angle of view indicated by a two-dot chain line in the drawing. As in this example, the microphone 11 and the camera 12 are arranged adjacent to each other, and the directivity angle and the angle of view are provided substantially parallel in the vertical direction and the horizontal direction, so that the unidirectionality of the microphone 11 is achieved. An angle of view of the camera 12 that is substantially equal to the range of the characteristics is realized.

  Here, the angle of view of the camera 12 that is substantially equal to the unidirectionality of the microphone 11 matches the range of the unidirectionality of the microphone 11, and the video of the speaker's face is captured by the camera 12. If so, the case where the microphone 11 can capture the voice of the speaker is included. Note that the angle of view of the camera 12 is slightly narrower than the directivity of the microphone 12 and is set so that the angle of view is completely within the directivity range of the microphone 12, so that the speaker is more sure of the microphone 12. It is also possible to devise so as to be within the directivity range.

  If such a microphone 1 with a camera is used to display an image captured by the camera on a monitor, the speaker can recognize that the image is captured by the camera and is within the directivity range of the microphone 12. On the other hand, when the camera's own image is not projected by the camera, it can be easily recognized that the microphone 12 is out of the directivity range.

<Control program for accurately capturing the voice of a speaker using the camera-equipped microphone of this embodiment>
The camera-equipped microphone control program of the present embodiment configured as described above is realized by a program process executed by a control unit to which the camera-equipped microphone is connected. Note that the control unit is provided in the system main body of the video conference system, but it can be considered that the control unit is incorporated in the main body housing of the microphone with camera.

(First control program)
The first control program is a control program for changing the direction of the directivity of the microphone based on the video in the microphone with camera in order to clear the voice of the speaker. This control program is repeatedly executed at regular time intervals.

  First, face detection is performed by image processing such as pattern recognition using video captured by a camera-equipped microphone. If the detected number of faces is an odd number, the directivity of the microphone is adjusted so that the face in the middle is the center of directivity. On the other hand, when an even number of faces are detected, the directivity of the microphone is adjusted so that the center of the faces of the two persons in the middle is the center of directivity.

  In order to change the center of directivity of the microphone, there are a case where the microphone is mechanically rotated and a case where the direction of directivity is electrically changed.

  FIG. 2 is a flowchart for explaining the flow of the first control program. First, an image is captured by a camera-equipped microphone, and face detection is performed by image processing (step S101). The number of faces detected here is set as the center coordinates on the screen of the detected faces of m, F0, F1,.

  Next, it is determined whether or not m> 0 (step S102). That is, it is determined whether or not at least one face has been detected. When m> 0 is not satisfied (when no face is detected), the process is terminated. On the other hand, if m> 0 (when one or more faces are detected), it is determined whether m is an odd number or an even number (step S103).

  When m is an odd number, control is performed so that the center of the directivity of the microphone is directed to the center coordinate of the face that is closest to the origin coordinate (0, 0) on the screen among the detected face coordinates (step S104). .

  On the other hand, when m is an even number, the coordinates between the detected face coordinates and the center coordinates of the face closest to the origin coordinate (0, 0) on the screen and the center coordinates of the next closest face are displayed. Control is performed so that the center of the directivity of the microphone is directed to (step S105). However, the process when m is an even number (step S105) may be the same as the process when m is an odd number (step S104). In this case, it is not necessary to determine whether m is an odd number or an even number (step S103). If m> 0 (step S102), the process always proceeds to step S104.

(Second control program)
The second control program does not need to reduce the gain of the microphone or mute it if the face is not present in the video as a result of detecting the face from the video captured by the camera microphone. It is a control program that prevents picking up sounds. This control program is also repeatedly executed at regular time intervals in the same manner as the first control program.

  First, face detection is performed by image processing such as pattern recognition using video captured by a camera-equipped microphone. If the face can be detected here, the first control program is executed. On the other hand, when the face cannot be detected, a process of lowering or muting the sound capturing gain by the camera microphone is performed.

  FIG. 3 is a flowchart for explaining the flow of the second control program. First, a video is captured by a camera-equipped microphone, and face detection is performed by image processing (step S201). The number of faces detected here is set as the center coordinates on the screen of the detected faces of m, F0, F1,.

  Next, it is determined whether m> 0 is satisfied (step S202). That is, it is determined whether or not at least one face has been detected. Here, if m> 0 (when one or more faces are detected), the microphone gain is initialized (step S203). On the other hand, when m> 0 is not satisfied (when no face is detected), the gain of the microphone is lowered or the output from the microphone is set to 0 (step S204).

  FIG. 4 is a flowchart for explaining processing for realizing both the first control program and the second control program. First, a video is taken in by a camera-equipped microphone, and face detection is performed by image processing (step S301). The number of faces detected here is set as the center coordinates on the screen of the detected faces of m, F0, F1,.

  Next, it is determined whether or not m> 0 (step S302). That is, it is determined whether or not at least one face has been detected. When m> 0 is not satisfied (when no face is detected), the gain of the microphone is lowered or the output from the microphone is set to 0 (step S303).

  On the other hand, when m> 0 (when one or more faces are detected), it is determined whether m is an odd number or an even number (step S304). If m is an odd number, control is performed so that the center of the directivity of the microphone is directed to the center coordinate of the face that is closest to the origin coordinate (0, 0) on the screen among the detected face coordinates (step). S305). On the other hand, when m is an even number, the coordinates between the detected face coordinates and the center coordinates of the face closest to the origin coordinate (0, 0) on the screen and the center coordinates of the next closest face are displayed. Control is made so that the center of the directivity of the microphone is directed to (step S306).

<Configuration of video conference system using microphone with camera>
FIG. 5 is a schematic diagram illustrating the configuration of a video conference system to which the microphone with a camera according to the present embodiment is applied. A video conference system 100 is centered on a system main body (control unit) 101, a monitor 102 that displays video, a main camera 103 that captures video of conference participants with high image quality, and a plurality of cameras that are arranged corresponding to the layout of the table The microphone 1 is provided.

  On the monitor 102, the video captured by the camera of the main camera 103 or the camera-equipped microphone 1 is displayed, or the video of the partner participant sent from the conference room at a remote location is displayed. These videos can be switched arbitrarily.

  The main camera 103 can perform operations such as panning, tilting, and zooming, and can capture images of all participants in the conference room at a wide angle, or can narrow down and capture a shooting range toward a specific participant. Further, the main camera 103 is provided with a microphone for estimating a sound source direction. The sound source direction can be estimated based on sound captured by the microphone, and the shooting direction can be adjusted to the direction.

  The plurality of microphones 1 with a camera are arranged in accordance with the layout of the table and the positions where the participants sit. Each microphone 1 with a camera is cascade-connected starting from the system main body 101.

  The monitor 102, the main camera 103, and the plurality of camera-equipped microphones 1 are connected to the system main body 101 and controlled by various programs executed by the system main body 101.

<Control of main camera using microphone with camera>
In the video conference system 100 as shown in FIG. 5, by using the camera-equipped microphone 1 of the present embodiment, the main camera 103 can be panned, tilted, etc. in the direction of the speaker accurately. Here, it is assumed that one microphone 1 with a camera is used for each participant.

(First main camera control method)
In the first main camera control method, the power average of the echo canceller output (here, the sound obtained by subtracting the estimated echo component from the microphone input signal in the echo canceller) for the past several seconds (for example, 1 second) exceeded the threshold. If there is a microphone with a camera, the microphone with the camera that is considered to be closest to the speaker, such as selecting the microphone with the camera with the largest average power from the microphones with a camera that exceeded the threshold, assuming that there was an utterance Decide.

  At this time, after the main camera pans / tilts the camera in the direction of the speaker by estimating the sound source direction and then performs face detection by image processing, it is assumed that the detected face is closest to the speaker from each detected face. Personal identification is performed by matching using the image of the regarded microphone with camera, the most similar face is found, and the main camera is panned and tilted in that direction.

  Each camera-equipped microphone stores images at regular time intervals as images for personal identification, and transmits the stored image to the main camera when it is deemed closest to the speaker.

  By controlling the main camera in this way, a rough shooting direction is set by estimating the sound source direction using the main camera, and when the faces of multiple participants are displayed in the captured video, the multiple faces are displayed. Then, an image similar to the image of the face captured by the camera microphone with the maximum audio capture is detected, and the shooting direction of the main camera is finely adjusted according to the position of the face. As a result, it is possible to accurately capture the video of the speaker, which could not be accurately identified only by estimating the sound source direction, with the main camera.

  FIG. 6 is a flowchart for explaining the flow of processing on the microphone with camera in the first main camera control method. For example, when the audio sampling frequency is set to 32000 Hz, this processing needs to be performed for each sample. Therefore, the processing of S401 or S403 must be started every 1/32000 seconds. Therefore, it is necessary to finish the processing of one loop (S401 to S408 or S403 to S408) within 1/32000 seconds. First, the value of the memory Mx for storing the power average of audio capturing is initialized to 0 (step S401). Here, x in the memory Mx is a number assigned sequentially from 0 for each microphone with camera.

  Next, the counter c is initialized to 0 (step S402). Next, the power for capturing audio is calculated in a predetermined sampling period (step S403). For example, in the case of a 32 kHz sampling microphone input signal, an array P having w × 32000 elements is prepared to obtain an average power Pave for w seconds (w = 1 for 1 second), and an instantaneous value is obtained for each sampling. Power P [c] = Input × Input is calculated. Here, Input is a microphone input value. When the sampling period is not 32 kHz, the above 32000 is replaced with the sampling frequency.

  Next, the average Pave of the values of all elements of the array P [c] is calculated (step S404). Then, it is determined whether or not Pave exceeds a predetermined threshold (step S405). If not, Mx = 0 (step S406), and if it exceeds Mx = Pave (step S407). ).

  Next, c = (c + 1)% (w × 32000) is performed (step S408). That is, the remainder when c + 1 is divided by w × 32000 is set as a new c. After setting a new c, the process returns to step S403, and the subsequent processing is repeated.

  FIG. 7 is a flowchart for explaining the flow of processing on the main camera side in the first main camera control method. First, Told is initialized to the current time by a timer (step S501), and Tnow is substituted for the current time by the timer (step S502).

  Next, it is determined whether or not Tnow-Told ≧ w (seconds) (step S503). Here, w is a waiting time (seconds) from the first pan and tilt of the main camera to the next pan and tilt.

  If No in step S503, the process returns to step S502, and the current time is substituted into Tnow by a timer. If Yes in step S503, Tnow is substituted for Told (step S504). Then, Xmax = 0 and Mmax = 0 are set (step S505), and the process proceeds to step S506.

  In step S506, assuming that n microphones with camera are connected, if Mmax <Mi, the process of setting Xmax = i and Mmax = Mi is performed n times. Here, Mi corresponds to Mx shown in FIG. By this processing, the number of the camera microphone with the highest average power is stored in Xmax, and the highest average power is stored in Mmax.

  Next, it is determined whether or not Xmax> 0 (step S507). If No, the process returns to step S502. If Yes, the main camera is roughly panned and tilted by detecting the sound source direction (step S508). Further, the image V is captured from the camera of the microphone with camera corresponding to the number stored in Xmax (step S509).

  Next, face detection is performed in the image captured by the main camera (step S510). Here, it is assumed that the number of detected faces is m, F0, F1,. Then, Fmax = 0 and Smax = 0 are set (step S511).

  Next, the similarity between the image of each face detected in the image captured by the main camera and the image V from the camera of the microphone with camera corresponding to the number stored in Xmax is calculated (step S512). Here, S0 to Sm are similarities (values of 0 to 1) between the image of each face and the image V. If Smax <Sj, the process of setting Fmax = j and Smax = Sj is performed m times. Thereby, the number of the image most similar to the image V among the images of each face is stored in Fmax. Here, S0 is F0, S1 is F1,..., Sm is the similarity with the face image V having Fm as the central coordinates.

  After that, fine adjustment of pan and tilt is performed so that the main camera is aligned with the center coordinate direction of the face image corresponding to Fmax (step S513).

(Second main camera control method)
In the second main camera control method, an image of a speaker when “the camera is facing the front of the camera” is stored by a camera-equipped microphone, and the main camera performs personal identification based on the image. This is a method that can improve the accuracy when the camera is punched in the direction of the speaker. This is because most speakers tend to talk to the TV, and in the environment as shown in FIG. 5, they are often almost in front of the main camera.

  Microphone with camera is a triangle area connecting two points of eyes and one point of mouth by pattern recognition using sample images of eyes and mouth (the larger the area, the more likely it is facing the front), etc. If it is considered that the image captured at regular time intervals is facing the front of the stored image, the image is overwritten and stored. However, the number of face images to be saved is not limited to one, and when a plurality of face images are saved, the face images that are more likely to face the front are preferentially saved.

  In addition, the sound of the frequency band that cannot be heard by human ears is output from the microphone with the camera that seems to be closest to the speaker, and the main camera is roughly panned by estimating the direction of the sound source using the sound in that frequency band. A method of tilting and then performing fine pan and tilt control by face detection and personal identification is also conceivable.

  FIG. 8 is a flowchart for explaining the flow of processing on the microphone with camera side (when the number of stored images is 1) in the second main camera control method. First, Told is initialized to the current time by a timer, a variable A for storing the area of a triangle composed of eyes and mouth is initialized to 0, and a saved image V is initialized to a black image (or a white image) (step) S601).

  Next, the current time is substituted into Tnow by a timer (step S602), and it is determined whether Tnow-Told ≧ w (step S603). Here, w is the time interval (seconds) until the next image is captured. If the determination is No, the process returns to step S602, and the current time is substituted into Tnow by a timer. On the other hand, if Yes, the process proceeds to step S604.

  In step S604, the value of Tnow is substituted for Told. After that, in step S605, the face image Vnow is captured by the camera-equipped microphone, and the area of the triangle formed by the eyes and mouth is defined as Anow. Next, it is determined whether or not A <Anow (step S606). If Yes, A = Anow and V = Vnow are performed (step S607). If No, the process returns to step S602.

  Thereby, A stores the area when the area of the triangle composed of the eyes and mouth is the largest among the face images captured by the camera microphone, and V stores the face image at that time. Will be.

  FIG. 9 is a flowchart for explaining the flow of processing on the camera-equipped microphone side (when the number of stored images is plural) in the second main camera control method. First, Told is initialized to the current time by a timer, the arrays A [0] to A [o] storing the area of the triangle formed by the eyes and mouth are initialized to 0, and the image V [ [0] to V [o] are initialized to black images (or white images) (step S701).

  Next, the current time is substituted into Tnow by the timer (step S702), and it is determined whether Tnow-Told ≧ w (step S703). Here, w is the time interval (seconds) until the next image is captured. If this determination is No, the process returns to step S702, and the current time is substituted into Tnow by a timer. On the other hand, if Yes, the process proceeds to step S704.

  In step S704, the value of Tnow is substituted for Told. Thereafter, in step S705, the face image Vnow is captured by the camera-equipped microphone, and the area of the triangle formed by the eyes and mouth is defined as Anow. Next, Omin = −1 and Amin = Anow are performed (step S706), and if Ai <Amin, the process of setting Omin = i and Amin = A [i] is performed o times (step S707).

  Next, it is determined whether or not Omin ≧ 0 (step S708). If No, the process returns to step S702 and the subsequent steps are repeated. If Yes, A [Omin] = Anow and V [Omin] = Vnow are set. This is performed (step S709). Then, the process returns to step S702.

  Thus, a plurality of triangular areas composed of eyes and mouths in the face image are stored in the arrays A [0] to A [o], and a plurality of the face images are stored in the images V [0] to A [o]. Will be stored.

  FIG. 10 is a flowchart for explaining the flow of processing on the main camera side in the second main camera control method. First, Told is initialized to the current time by a timer (step S801), and the current time is substituted into Tnow by the timer (step S802).

  Next, it is determined whether Tnow-Told ≧ w (step S803). Here, w is a waiting time (seconds) from the first pan and tilt of the main camera to the next pan and tilt.

  If NO in step S803, the process returns to step S802, and the current time is substituted into Tnow by a timer. If YES in step S803, the value of Tnow is substituted for Told (step S804). Thereafter, Xmax = 0 and Mmax = 0 are set (step S805), and the process proceeds to step S806.

  In step S806, assuming that n microphones with cameras are connected, if Mmax <Mi, the process of setting Xmax = i and Mmax = Mi is performed n times. Here, Mi corresponds to Mx shown in FIG. By this processing, the number of the camera microphone with the highest average power is stored in Xmax, and the highest average power is stored in Mmax.

  Next, it is determined whether or not Xmax> 0 (step S807). If No, the process returns to step S802. If Yes, the main camera is roughly panned and tilted by detecting the sound source direction (step S808). Also, the images V [0] to V [o] are captured from the camera of the microphone with camera corresponding to the number stored in Xmax (step S809).

  Next, face detection is performed in the image captured by the main camera (step S810). Here, it is assumed that the number of detected faces is m, F0, F1,. Then, Fmax = 0 and Smax = 0 are set (step S811).

  Next, the degree of similarity between the image of each face detected in the image captured by the main camera and the image V [0] to V [o] from the camera of the camera microphone corresponding to the number stored in Xmax is calculated. (Step S812). Here, S0 to Sm are averages of the similarity (value of 0 to 1) between each face image and the images V [0] to V [o]. If Smax <Sj, the process of setting Fmax = j and Smax = Sj is performed m times. Thereby, the number of the image most similar to the image V among the images of each face is stored in Fmax. Here, S0 is F0, S1 is F1,..., Sm is an average similarity with the face images V [0] to V [o] having Fm as the central coordinates.

  After that, fine adjustment of pan and tilt is performed so that the main camera is aligned with the center coordinate direction of the face image corresponding to Fmax (step S813).

<Production method for monitor output using microphone with camera>
In the video conference system 100 as shown in FIG. 5, various effects can be given to the output screen of the monitor 102 by using the camera-equipped microphone 1 of the present embodiment. Here, some examples will be described.

(First production method)
FIG. 11 is a schematic diagram for explaining the first effect method. That is, in the first production method, in the video conference system, when images captured by the cameras of the microphones with cameras are arranged and output on the monitor, the display area of the image is increased as the camera microphone with a larger power average of the echo canceller output. is doing.

  In the example shown in FIG. 11, the three horizontal images are displayed in two rows, and the image h2 displayed in the center of the upper row is the largest, next the images h1, h3, and then the images h4, h5. , H6 in that order. In other words, since the audio capture by the camera microphone capturing the video h2 is the largest, the display size is the largest accordingly. As a result, the larger the video captured by the camera-equipped microphone, which seems to be closer to the speaker.

(Second production method)
FIG. 12 is a schematic diagram for explaining the second effect method. In other words, in the second production method, in the video conference system, when images captured by the cameras of the microphones with cameras are arranged and output on the monitor, the microphone with the camera having a larger power average of the echo canceller output increases the brightness of the images. ing.

  In the example shown in FIG. 12, the three horizontal images are displayed in two rows, and the image h2 displayed in the center of the upper row is the brightest, followed by the images h1, h3, and then the images h4, h5. , H6 in that order. Thereby, the image of the microphone with a camera that seems to be closer to the speaker becomes brighter.

(Third production method)
The third presentation method is a method of changing the BGM according to the power average magnitude of the echo canceller output of the microphone with camera that seems to be closest to the speaker in the video conference system.

(Fourth production method)
FIG. 13 is a schematic diagram for explaining the fourth effect method. That is, in the fourth rendering method, when the microphones with cameras are connected in cascade from the system body in the video conference system, the images captured by the camera microphones are arranged in a horizontal row in the order closest to the system body. This is a method of creating a pseudo panoramic image by displaying.

  In the example shown in FIG. 13A, the microphones with cameras corresponding to the order of the images h1 to h5 are cascaded in order from the system main body, and the images captured by the cameras of the respective microphones with cameras are displayed side by side. Is.

  Further, in the example shown in FIG. 13B, as described above, the images captured by the camera microphones cascaded from the system main body are displayed in a horizontal row in order from the system main body. Although there is a camera-equipped mamicrophone with a large average power of echo canceller output, the image display area is enlarged.

  In the example shown in FIG. 13B, five images h1 to h5 are displayed in a horizontal row in the order of cascade connection. Among these, the image h3 is the largest, the images h2, h4, and then the image h1. , H5 in that order. In other words, since the audio capture by the microphone with a camera that captures the video h3 is the largest, the display size is the largest accordingly. As a result, the image captured by the camera-equipped microphone that seems to be close to the speaker becomes larger, and a pseudo panoramic image can be created by the horizontal display.

<Implementation effect>
By applying the camera-equipped microphone of the present embodiment to the video conference system, the speaker can easily know whether he is within the microphone directivity range. In addition, when using a plurality of microphones with a camera, if there is a noise source within the directivity range, the input sound from the directional microphone may make it difficult to hear the necessary sound. Directing directivity in the direction of the person makes the necessary sound clearer, and when there is no person nearby, the gain can be lowered or muted to avoid picking up extra noise as much as possible.

  Furthermore, when a video conference is performed with the main camera facing the direction of the speaker, the approximate shooting direction is determined by sound source direction estimation by voice processing, and the main camera is set by the face detection and personal identification by image processing. However, in order to achieve higher accuracy, it is possible to improve accuracy by using an image captured by a camera-equipped microphone that seems to be closest to the speaker.

  In addition, when the speaker is visually emphasized only with the video of the main camera, there is a problem that the part where the speaker is present must be extracted well in the video, but a plurality of microphones with cameras are used. Thus, the speaker can be visually emphasized without using the extraction process.

It is a mimetic diagram explaining a microphone with a camera concerning this embodiment. It is a flowchart explaining the flow of a 1st control program. It is a flowchart explaining the flow of a 2nd control program. It is a flowchart explaining the process which implement | achieves both a 1st control program and a 2nd control program. It is a schematic diagram explaining the structure of the video conference system to which the microphone with a camera of this embodiment is applied. It is a flowchart explaining the flow of a process by the microphone with a camera in the 1st main camera control method. It is a flowchart explaining the flow of the process by the main camera side in the 1st main camera control method. It is a flowchart explaining the flow of the process by the side with a microphone with a camera in the 2nd main camera control method (when the preservation | save number of an image is 1). It is a flowchart explaining the flow of the process by the microphone side with a camera in the 2nd main camera control method (when the preservation | save number of an image is multiple). It is a flowchart explaining the flow of the process by the side of the main camera in the 2nd main camera control method. It is a schematic diagram explaining the 1st production method. It is a schematic diagram explaining the 2nd production method. It is a schematic diagram explaining the 4th production method.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Microphone with a camera, 10 ... Main body housing, 11 ... Microphone, 12 ... Camera, 100 ... Video conference system, 101 ... System main body (control part), 102 ... Monitor, 103 ... Main camera

Claims (8)

A unidirectional microphone provided in the body housing;
A camera-equipped microphone, comprising: a camera provided in the main body casing and having a field angle substantially equal to a range of unidirectionality of the microphone. A computer-controlled microphone having a unidirectional microphone provided in a main body casing and a camera provided in the main body casing and having a field angle substantially equal to the unidirectional range of the microphone In the program to
A step of recognizing a face image from an image captured by the camera and changing a unidirectional center position of the microphone based on the recognized face position by a computer. Control program. The control of a microphone with a camera according to claim 2, wherein when a face image cannot be recognized from the video captured by the camera, the computer executes a step of preventing audio from being captured by the microphone. program. A plurality of microphones with a camera including a microphone having a unidirectionality and a camera having a field angle substantially equal to a range of the unidirectionality of the microphone in a main body casing are arranged in a predetermined layout, and the microphones with a camera are used In a video conference system that captures participants' video with the main camera,
Identify the microphone with the camera that captures the audio, move the shooting direction of the main camera toward that position, and recognize the face image from the video captured by the camera microphone that captures the audio And a control means for adjusting the shooting direction of the main camera based on the recognition result. The plurality of microphones with a camera sequentially captures images of participants' faces by each camera and sends them to the control means.
The control means sequentially overwrites and saves the face images of the participants sent from the microphones with cameras, and recognizes the face images when the stored face images are adjusted with the main camera. The video conference system according to claim 4, wherein the video conference system is used. A plurality of microphones with a camera including a microphone having a unidirectionality and a camera having a field angle substantially equal to the range of the unidirectionality of the microphone in a main body casing are arranged in a predetermined layout. In a video conference system that displays captured video on a monitor,
When displaying a plurality of images captured by a plurality of camera-equipped microphones on the monitor, images captured by a camera-equipped microphone camera with the highest audio capture level were captured by another camera-equipped microphone camera. A video conferencing system comprising control means for displaying an image larger than an image. A plurality of microphones with a camera including a microphone having a unidirectionality and a camera having a field angle substantially equal to the range of the unidirectionality of the microphone in a main body casing are arranged in a predetermined layout. In a video conference system that displays captured video on a monitor,
When displaying a plurality of images captured by a plurality of camera-equipped microphones on the monitor, images captured by a camera-equipped microphone camera with the highest audio capture level were captured by another camera-equipped microphone camera. A video conferencing system characterized by comprising control means for displaying an image brighter. A plurality of microphones with a camera including a microphone having a unidirectionality and a camera having a field angle substantially equal to the range of the unidirectionality of the microphone in a main body casing are arranged in a predetermined layout. In a video conference system that displays captured video on a monitor,
A plurality of the microphones with a camera are cascade-connected, and control means is provided for displaying a plurality of images captured by the cameras of the plurality of camera-equipped microphones in a horizontal row on the monitor in the order of the cascade connection. Video conference system.

Images