JP2016010039A - Remote conference system, video processing method, video controller, conference terminal, and program thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow for setting of the right of speaking or selection of an emphasis object, by natural movement of a conference participant, in a remote conference system.SOLUTION: Video and audio 10-1, 10-2, ..., 10-N transmitted from respective terminals 1-1, 1-2, ..., 1-N are received by a video and audio synthesis control server 2. Composite video and audio 20-1, 20-2, ..., 20-N transmitted from the video and audio synthesis control server 2 are received by a first terminal 1-1, a second terminal 1-2, ..., an N-th terminal 1-N. When a conference participant of the first terminal 1-1, i.e., a chairman terminal, performs movement for designating a desired position of a composite video displayed, the first terminal 1-1 transmits a movement detection result 11 representing a designated position to the video and audio synthesis control server 2. The video and audio synthesis control server 2 generates the composite video and audio 20-1, 20-2, ..., 20-N emphasizing the video and audio displayed at the designated position.

Description

  The present invention relates to a remote conference system, a video processing method thereof, a video control device, a conference terminal, and a program.

  2. Description of the Related Art In recent years, with the development of information communication technology, a remote conference system has been developed that allows people in remote locations to participate in a conference via a network such as the Internet. Remote conferences are not conferences where participants actually gather in one place and talk directly, such as video conferences and web conferences, but at locations (bases) where participants are separated from each other. In the meantime, this is a conference in which a conference terminal connected to each other is used to communicate a moving image and a voice to advance a discussion.

  In a remote conference using such a remote conference system, not only two participants hold a conference, but many participants at three or more locations often participate in the remote conference. When a large number of participants conduct a remote conference, it is confusing if a plurality of participants speak at a time when the speaker speaks as the conference progresses.

  As a remote conference system that addresses such a problem, there is a remote conference system described in Patent Document 1. In this teleconferencing system, one of a plurality of conference terminals is used as a moderator terminal, and the right to speak is given to the conference terminal in response to a notification of the intention to speak from the conference terminal so that the moderator can manage the progress of the conference. By setting, only a limited conference terminal can communicate video and audio.

  There is also known a remote conference system that allows a specific participant to focus on other participants and allows the conference to proceed smoothly (Patent Document 2). In this remote conference system, a request signal is transmitted from an arbitrary conference terminal to another conference terminal to emphasize and output conference information such as video and audio of the participants themselves. When the emphasis request signal is received from the conference terminal, the conference information of the participant specified by the emphasis request signal is emphasized and reproduced.

  However, the above-mentioned conventional teleconferencing system has a problem in that it is bothersome and unnatural, such as the use of buttons and keyboards on conference terminals and operations on menu screens, in the method for setting the right to speak and the method for selecting highlight targets. is there.

  The present invention has been made to solve such a problem, and an object of the present invention is to enable setting of a right to speak and selection of an emphasis target by a natural operation of a conference participant in a remote conference system. That is.

  The present invention relates to a remote conference system for communicating information including video between conference terminals installed at a plurality of bases, and a desired position of a video displayed on a predetermined conference terminal When a conference participant who uses the command is instructed by a predetermined first operation, the position detecting means for detecting the instructed position, the position detected by the position detecting means, and the predetermined one conference terminal The base displayed at the position indicated by the conference participant who uses the predetermined one conference terminal is detected from the information indicating the position of the video of the bases of the other conference terminals in the displayed video. A teleconferencing system comprising: a site detecting unit that performs the above-described operation and a video enhancement unit that enhances the video of the site detected by the site detecting unit.

  According to the present invention, in a teleconference system, it is possible to set a floor and select an emphasis target by natural actions of conference participants.

It is a figure for demonstrating the whole structure of the remote conference system which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the terminal in the remote conference system which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the video / audio synthesis / control server in the remote conference system according to the first embodiment of the present invention. It is a functional block diagram of the moderator terminal in the remote conference system which concerns on the 1st Embodiment of this invention. It is a functional block diagram of the video / audio synthesis / control server in the remote conference system according to the first embodiment of the present invention. It is a figure for demonstrating the output of the operation | movement detection means in FIG. It is a figure for demonstrating the detection method of the designated coordinate in FIG. It is a figure for demonstrating the image | video arrangement | positioning information and synthetic | combination image | video in FIG. It is a figure for demonstrating cancellation | release operation | movement of the emphasis process in the remote conference system which concerns on the 1st Embodiment of this invention. It is a flowchart which shows operation | movement of the remote conference system which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating the whole structure of the remote conference system which concerns on the 2nd Embodiment of this invention. It is a functional block diagram of the moderator terminal in the remote conference system which concerns on the 2nd Embodiment of this invention. It is a functional block diagram of terminals other than the moderator terminal in the remote conference system according to the second embodiment of the present invention. It is a flowchart which shows transmission operation | movement of the moderator terminal in the remote conference system which concerns on the 2nd Embodiment of this invention, and reception operation | movement of terminals other than moderator terminal. It is a flowchart which shows the transmission operation | movement of terminals other than the moderator terminal in the remote conference system which concerns on the 2nd Embodiment of this invention, and the reception operation of the moderator terminal.

Embodiments of the present invention will be described below with reference to the drawings.
[First Embodiment]
<Overall configuration of remote conference system>
FIG. 1 is a diagram for explaining the overall configuration of the remote conference system according to the first embodiment of the present invention.

  This teleconference system includes a first terminal 1-1, a second terminal 1-2, and N conference terminals installed at N locations (N is an integer of 3 or more) as a plurality of locations. ..., configured by connecting the N-th terminal 1-N and the video / audio synthesis / control server 2 to the network 3, and supporting the remote conference by communicating video and audio between the terminals. be able to.

Each terminal transmits video and audio (hereinafter referred to as video and audio) to the video and audio synthesis / control server 2.
The video / audio synthesis / control server 2 controls and synthesizes the received video / audio, and then transmits the synthesized video / audio to each terminal. Each terminal receives the synthesized video / audio from the video / audio synthesis / control server 2.

  That is, the video and audio 10-1 transmitted from the first terminal 1-1, the video and audio 10-2 transmitted from the second terminal 1-2,..., And the Nth terminal 1-N are transmitted. The video / audio 10-N passes through the network 3 and is received by the video / audio synthesis / control server 2. Then, the synthesized video / audio 20-1, 20-2,..., 20-N transmitted from the video / audio synthesis / control server 2 pass through the network 3, and the first terminal 1-1 and the second, respectively. Received by the terminal 1-2, ..., the Nth terminal 1-N.

  Here, 20-1, 20-2,..., 20-N may all be the same, but in the present embodiment, they differ depending on the destination terminal. That is, for example, the synthesized video / audio 20-1 transmitted to the first terminal 1-1 does not include the video / audio 10-1 from the first terminal 1-1, but only video / audio from the other terminals. Is synthesized.

  The first terminal 1-1 is a moderator terminal. The moderator terminal is a terminal used by a moderator who is one of the conference participants. The method of determining the moderator terminal is, for example, a method of registering which terminal is so by the menu operation before or after the start of the conference, or by registering the face and voice of the person acting as the host in advance. There is a method to set a terminal that can be recognized.

  The moderator terminal detects a predetermined operation for the speaker to determine the right to speak to facilitate the conference and to emphasize video and audio, and the operation detection result 11 is sent to the video and audio synthesis / control server 2. Send. As will be described in detail later, the motion detection result includes information indicating where the moderator has performed the motion indicating the synthesized video received and displayed by the first terminal 1-1. The video / audio synthesis / control server 2 controls video and audio based on the motion detection result 11. In the following description, the first terminal 1-1, the second terminal 1-2,...

<Hardware configuration of terminal>
FIG. 2 is a block diagram showing a hardware configuration of a terminal in the remote conference system according to the first embodiment of the present invention.

  The terminal 1 includes a computer and its peripheral devices, and the like, and includes a CPU 511, a memory (ROM, RAM) 512, a storage medium mounting unit 513, a network device 514, a monitor control unit 515, an input device 516, and an HDD (hard disk device). 517, a camera 521, a speaker 522, and a microphone 523 are connected by a bus line.

  The CPU 511 reads out and executes the program 520 from the HDD 517 and controls the terminal 1 as a whole. The RAM of the memory 512 is a volatile memory such as a DRAM, and serves as a work area when the CPU executes a program or an OS. The ROM of the memory 512 is a non-volatile memory that stores a boot program and the like.

  The storage medium mounting unit 513 is an interface that connects various storage media 519 in a detachable manner, and is used when reading data from the storage medium 519 and writing data into the storage medium 519. The program 520 is distributed while being stored in the storage medium 519, read from the storage medium 519, and installed in the HDD 517. The storage medium 519 is a USB memory, SD card, Memory Stick (registered trademark), multimedia card, CD-ROM (R / W), DVD-ROM (RAM, R / W), or the like.

  The network device 514 is an interface (for example, an Ethernet (registered trademark) card) for connecting the terminal 1 to the network 3. The program 520 may be installed in the HDD 517 by the network device 514 being downloaded from a server (not shown).

  The HDD 517 stores a terminal-side program 520 of the remote conference system, an OS, and various data described below. The program 520 configures a functional block (FIG. 4) to be described later in the terminal 1, and causes a flowchart (FIG. 10) to be described later to be executed.

  The input device 516 is a device for a conference participant to operate the terminal 1 using a mouse, a keyboard, a touch panel, or the like. The monitor control unit 515 draws a screen to be displayed on the display 518 with the resolution and the number of colors specified by the OS and the program 520. The display 518 is a user interface that displays a GUI (Graphical User Interface) screen under the control of the monitor control unit 515. The display 518 includes an LCD or a projector, and a touch panel may be integrally mounted on the LCD.

  The camera 521 is a device that acquires a video of a base by photographing the base where the terminal 1 is installed. The video of this base consists of the video of the terminal 1 and its surroundings, and the video of conference participants who use the terminal 1. The microphone 523 is a device that collects sound uttered by a conference participant who uses the terminal 1. Video captured by the camera 521 and audio collected by the microphone 523 are transmitted to other terminals in real time. This video is displayed on the display 518 of another terminal, and the audio is output from the speaker 522.

<Video / audio synthesis / control server>
FIG. 3 is a block diagram showing a hardware configuration of the video / audio synthesis / control server in the remote conference system according to the first embodiment of the present invention.

  The video / audio synthesis / control server 2 as a video control apparatus according to the present invention is composed of a computer and peripheral devices thereof, and includes a CPU 611, a memory (ROM, RAM) 612, a storage medium mounting unit 613, a network device 614, The input device 616 and the HDD 617 are connected via a bus line.

  The CPU 611 reads out and executes the program 620 from the HDD 617 and controls the video / audio synthesis / control server 2 as a whole. The RAM of the memory 612 is a volatile memory such as a DRAM, and serves as a work area when the CPU executes a program or OS. The storage medium mounting unit 613 is an interface that connects various storage media 619 in a detachable manner, and is used when reading data from the storage medium 619 and writing data into the storage medium 619. The program 620 is distributed in a state stored in the storage medium 619, read from the storage medium 619, and installed in the HDD 617.

  The network device 614 is an interface for connecting the video / audio synthesis / control server 2 to the network 3. The input device 616 is a device for operating the video / audio synthesis / control server 2 using a mouse, a keyboard, a touch panel, or the like.

<Functional block diagram of moderator terminal>
FIG. 4 is a functional block diagram of the moderator terminal in the remote conference system according to the first embodiment of the present invention.

  The first terminal 1-1, which is a moderator terminal, includes a video acquisition unit 101, a video transmission unit 102, an audio acquisition unit 103, an audio transmission unit 104, a sensor information acquisition unit 105, an operation detection unit 106, and a detection result transmission unit 107. Video receiving means 108, video output means 109, audio receiving means 110, and audio output means 111.

  The video acquisition unit 101 acquires the video of the presenter taken by the camera 521. The video transmission unit 102 transmits the video acquired by the video acquisition unit 101 to the video / audio synthesis / control server 2 via the network 3 by the network device 514. The voice acquisition unit 103 acquires the voice of the presenter collected by the microphone 523. The audio transmission unit 104 transmits the audio acquired by the audio acquisition unit 103 to the video / audio synthesis / control server 2 via the network 3 by the network device 514. The video acquired by the video acquisition unit 101 and the audio acquired by the audio acquisition unit 103 constitute the video / audio 10-1.

  The video receiving unit 108 receives the video transmitted from the video / audio synthesis / control server 2, and the video output unit 109 displays the video received by the video receiving unit 108 on the display 518. The audio receiving unit 110 receives the audio transmitted from the video / audio synthesis / control server 2, and the audio output unit 111 reproduces (outputs) the audio received by the audio receiving unit 110 with the speaker 522. The video received by the video receiving unit 108 and the audio received by the audio receiving unit 110 constitute the synthesized video audio 20-1.

  The sensor information acquisition means 105 acquires the distance information from the distance sensor etc. which are arrange | positioned on the front surface of the display 518 of the 1st terminal 1-1, etc. to the body element of a moderator, for example, a hand. In the motion detection unit 106, the presenter indicates a desired position on the video displayed on the display 518 from the sensor information acquired by the sensor information acquisition unit 105 and the video acquired by the video acquisition unit 101. It detects whether or not a predetermined operation has been performed (instructed) and outputs an operation detection result. This motion detection result also includes the coordinates of the indicated position. That is, the motion detection unit 106 functions as a position detection unit according to the present invention. Details of the configuration of the motion detector 106 and details of the motion detection result will be described later.

  Since terminals other than the moderator terminal do not need to include the sensor information acquisition unit 105, the operation detection unit 106, and the detection result transmission unit 107, the terminal may be configured by removing these units from the moderator terminal. Further, even if the terminal is equipped with these means, if the terminal is not a moderator, these means need not be performed.

<Video / audio synthesis / control server>
FIG. 5 is a functional block diagram of the video / audio synthesis / control server in the remote conference system according to the first embodiment of the present invention.

  The video / audio synthesis / control server 2 includes a video reception unit 201, a video synthesis unit 202, a video transmission unit 203, a voice reception unit 204, a voice synthesis unit 205, a voice transmission unit 206, a detection result reception unit 207, and an enhancement target selection unit 208. Video control means 209 and audio control means 210.

  The video receiving means 201 receives the video 1, 2,..., N transmitted from the first terminal 1-1, the second terminal 1-2,. . The video synthesizing unit 202 synthesizes the videos 1, 2,..., N (pre-combination video) received by the video receiving unit 201 to generate synthesized videos 1, 2,. Details of the operation of the video composition means 202 and details of the content of the composite video will be described later. The video transmission means 203 transmits the composite video 1, 2,..., N to the first terminal 1-1, the second terminal 1-2,.

  The voice receiving unit 204 receives the voices 1, 2,..., N transmitted from the first terminal 1-1, the second terminal 1-2,. . The voice synthesis unit 205 synthesizes the voices 1, 2,..., N (pre-synthesis voices) received by the voice reception unit 204 to generate synthesized voices 1, 2,. Details of the operation of the speech synthesizer 205 and details of the contents of the synthesized speech will be described later. The voice transmission means 206 transmits the synthesized voices 1, 2,..., N to the first terminal 1-1, the second terminal 2-2,.

  Here, the images 1, 2,..., N and the sounds 1, 2,..., N constitute the images and sounds 10-1, 10-2,. .., N and synthesized voices 1, 2,..., N constitute the synthesized video voices 20-1, 20-2,.

  The detection result receiving unit 207 receives the operation detection result transmitted from the first terminal 1-1. Emphasis target selection means 208 selects a terminal to be emphasized from videos 1, 2,..., N from the received motion detection result and video layout information supplied from video synthesis means 202, Information to be emphasized as a selection result is sent to the video control unit 209 and the audio control unit 210. That is, the emphasis target selection unit 208 functions as a site detection unit according to the present invention. The video control unit 209 and the voice control unit 210 control the operations of the video synthesis unit 202 and the voice synthesis unit 205, respectively, based on the input emphasis target information. Details of the video arrangement information and the emphasis target information will be described later.

<Details of configuration of motion detection means and details of motion detection result>
6 is a diagram for explaining the output of the motion detection means 106 in FIG. 4, and FIG. 7 is a diagram for explaining the method for detecting the designated coordinates in FIG. Here, FIG. 7A and FIG. 7B are a plan view and a side view of the relationship between the hands of the presenter and the screen of the display 518 of the presenter terminal (first terminal 1-1), respectively.

  As shown in FIG. 6, the output of the motion detection means 106, that is, the motion detection result is composed of data in which information indicating whether or not a predetermined motion has been performed and designated coordinates are associated with each other. Here, the “predetermined operation” is an operation in which the conference presenter points to a certain position of the composite image displayed on the screen 518a of the display 518 of the first terminal 1-1. Two-dimensional coordinates (x, y) representing the position on the screen. Note that the origin of the two-dimensional coordinates is, for example, the lower left corner of the screen 518a.

  This motion detection result is output at regular intervals. In the case of the figure, “FALSE” indicating that the predetermined operation is not performed at the first time and the third time is output, and “TRUE” indicating that the predetermined operation is performed at the second time and the fourth time are output. ing. In the second and fourth times, “x = 100, y = 150” and “x = 300, y = 120” are output as “designated coordinates”, respectively.

Here, the “designated coordinates” can be calculated from the positional relationship between the orientation of the presenter's hand and the screen of the display 518. That is, when the moderator's hand 40 and the screen 518a of the display 518 are in the relationship shown in FIGS. 7A and 7B, the point (x, y) on the screen 518a is obtained by the following equations [1] and [2]. It is done.
x = Z _h tan θ _h + x _h (1)
y = Z _h tanφ _h + y _h (2)

Here, the three-dimensional coordinates (x _h , y _h , z _h ) and orientation (θ _h , φ _h ) of the hand 40 are obtained from, for example, the video acquired by the video acquisition unit 101 and the information of the distance sensor.

  When there is a participant other than the moderator at the site where the first terminal 1-1 is installed, the motion detection means 106 needs to distinguish only the moderator from the moderator and detect only the moderator. There is. As a method of distinguishing, a method of predetermining the position of the presenter (e.g., determining a person closest to the center of the camera 521 as the presenter), a method of specifying the position of the presenter before the start of the conference, There is a method of identifying a moderator by face recognition.

<Details of the operation of the video composition means and details of the content of the composite video>
FIG. 8 is a diagram for explaining the video arrangement information and the composite video in FIG. Here, FIG. 8A shows the video layout information, and FIG. 8B is a diagram showing the layout on the screen of the composite video corresponding to the video layout information shown in FIG. 8A. In these figures, terminal numbers 1, 2, and 3 represent the numbers of arbitrary three terminals in FIG.

  The video arrangement information is information (layout information) indicating how the video composition unit 202 synthesizes the video of each terminal, and as shown in FIG. 8A, the terminal number and the arrangement position (the upper left coordinates, Information indicating the coordinates of the lower right corner) and information indicating the order of overlapping. The composite image displayed on the screen 518a of the display 518 based on the video arrangement information shown in FIG. 8A is as shown in FIG. 8B.

  The emphasis target selecting unit 208 can determine which terminal's video is displayed at the designated coordinates by comparing the motion detection result with the video layout information of the composite video transmitted to the moderator terminal 1-1. Since it is possible, the terminal to be emphasized is selected based on the result of the determination. That is, for example, when the designated coordinates of the motion detection result are (100, 100) for the video layout information shown in FIG. 8A, the second terminal 1-2 including the coordinates in the display area is displayed. Select as highlight target. Note that when the motion detection result does not detect a predetermined motion, the highlight target selection unit 208 does not select the highlight target, and therefore information indicating that the highlight target is not selected is transmitted as “highlight target information”. .

  The video control unit 209 and the audio control unit 210 are a video control signal for emphasizing video and audio of a terminal selected as an emphasis target, and an audio control signal for emphasizing audio, respectively. Send to 205. That is, the video control unit 209 and the audio control unit 210 function as a video enhancement unit and a voice enhancement unit according to the present invention, respectively. As a method for enhancing the video, for example, a display area is enlarged. As a method for enhancing the voice, for example, the voice from a terminal other than the terminal to be emphasized is muted. When this audio mute is performed, the right to speak is set for the conference participant who uses the emphasis target terminal. When the enhancement target is not selected, the video control signal and the audio control signal are set as they are.

  It is also possible to select an emphasis target to be selected in units of individuals rather than in units of terminals. In that case, it is necessary to include the position information of the person in the video in the video layout information obtained from the video synthesizing means 202. For example, there are a method for obtaining position information of a person, a method for fixing the position in advance, a method for using face recognition, person recognition, and the like. The emphasis target selecting means 208 selects the person displayed at the position closest to the designated coordinates of the motion detection result as the emphasis target. Examples of methods for emphasizing video and audio in individual units include a method of cutting out and displaying a target person's display area, and a method of controlling directivity by microphone array processing.

<Release emphasis processing>
FIG. 9 is a diagram for explaining the cancellation processing of the emphasis process in the remote conference system according to the first embodiment of the present invention.

  If it is desired to cancel the emphasis process, a new canceling operation may be defined and used. For example, when you want to release, tap your hand. In this case, the operation detection result is as shown in FIG. 9, for example. An operation number is added to the operation detection result shown in FIG. 6, and the video control unit 209 and the audio control unit 210 perform control corresponding to the specified emphasis target and the operation number.

  In the case of this figure, the second motion detection result represents the designation operation to be emphasized for the terminal having the designated coordinates “x = 100, y = 150” (in the case of FIG. 8B, the terminal having the terminal number 2). That is, it represents that a result of the moderator performing an operation indicating the terminal of terminal number 2 of the composite video displayed on the display 518 of the first terminal 1-1 is detected. The fourth motion detection result represents the enhancement processing canceling operation. That is, it represents that the result of the moderator performing an action of clapping his hand, for example, has been detected.

<Operation of remote conference system>
FIG. 10 is a flowchart showing the operation of the remote conference system according to the first embodiment of the present invention.

  First, in the moderator terminal (first terminal 1-1), the video acquisition unit 101, the audio acquisition unit 103, and the sensor information acquisition unit 105 acquire video, audio, and sensor information, respectively (step S1). Next, the motion detection means 106 determines whether or not a predetermined motion has been detected from the video and sensor information acquired in step S1 (step S2).

  When detecting a predetermined motion (step S2: Yes), the motion detection unit 106 calculates coordinates (designated coordinates) indicated by the predetermined motion from the video and sensor information acquired in step S1, and adds them to the motion detection result. (Step S3).

  Next, the detection result transmitting unit 107 transmits the video and audio acquired in step S1 and the motion detection result obtained in steps S2 and S3 to the video / audio synthesis / control server 2 (step S4).

  When the motion detection unit 106 does not detect the predetermined motion (step S2: No), the video and audio acquired in step S1 and information indicating that the predetermined motion has not been performed (“FALSE” in FIG. 6). The motion detection result including is transmitted to the video / audio synthesis / control server 2 (step S4).

  The video / audio synthesis / control server 2 receives the video, audio, and operation detection result from the first terminal 1-1 (step S5). Next, the highlight target selecting unit 208 includes information ("TRUE" in FIG. 6) indicating whether or not the motion detection result is detecting a predetermined motion, that is, the motion detection result is detecting a predetermined motion. Whether or not (step S6).

  As a result of the determination, if a predetermined operation is detected (step S6: Yes), a terminal to be emphasized is selected from the operation detection result and the video arrangement information from the video synthesizing means 202 (step S7). . At this time, the emphasis target information indicating the selection result is supplied to the video control unit 209 and the audio control unit 210.

  Next, the video control unit 209 and the audio control unit 210 adjust the parameters at the time of synthesis so as to emphasize the video and audio of the terminal selected in step S6 (step S8).

  The video synthesizing unit 202 and the audio synthesizing unit 205 synthesize video and audio for transmission to each terminal based on the parameters adjusted by the video control unit 209 and the audio control unit 210 (step S9).

  When the enhancement target selection unit 208 has not detected a predetermined operation (step S6: NO), the video control unit 209 and the audio control unit 210 leave the control signal parameters as they are, and the video synthesis unit 202 and the audio synthesis unit The means 205 synthesizes video and audio for transmission to each terminal using the parameter of the value as it is (step S9).

  The video transmission unit 203 and the audio transmission unit 206 transmit the video and audio synthesized in step S9 to each terminal (step S10). In each terminal, the video receiving unit 108 and the audio receiving unit 110 receive the video and audio from the video / audio synthesis / control server 2 respectively (step S11). Then, the video output unit 109 displays the video, and the audio output unit 111 reproduces the audio (step S12).

  The operations of terminals other than the moderator terminal (second terminal 1-2,..., Nth terminal 1-N) are as follows. “Obtain video and audio” (change “video, audio and sensor information” in step S1 to “video and audio”) → “send video and audio to server” (“video, audio and operation in step S4” “Detection result” is changed to “video, audio”) → “receive video and audio from server” (same as step S11) → “display and reproduce video and audio information” (same as step S12).

As described above, the remote conference system according to the first embodiment of the present invention has the following features (1) to (5).
(1) The right to speak, video enhancement, and audio enhancement can be achieved by the natural operation of the presenter.
(2) Emphasis targets can be selected in units of bases by using video layout information as layout information indicating how video from each terminal is combined.
(3) The emphasis target can be selected on an individual basis by including video position information including person position information indicating how the participant is positioned in the video from each terminal.
(4) The emphasis process can be canceled by the natural operation of the presenter.
(5) The management of the progress of the conference by the presenter becomes easy.

[Second Embodiment]
<Overall configuration of remote conference system>
FIG. 11 is a diagram for explaining the overall configuration of the remote conference system according to the second embodiment of the present invention. In this figure, the same parts as those in FIG. 1 (the entire configuration of the remote conference system according to the first embodiment) are denoted by the same reference numerals as those in FIG. 1, and the explanation of those parts is omitted unless particularly necessary.

  This remote conference system is configured by connecting a first terminal 4-1, a second terminal 4-2,..., An Nth terminal 4-N, which are N terminals, to the network 3. . The first terminal 4-1 is a moderator terminal.

  The main difference from the first embodiment is that each terminal transmits and receives video and audio to each other, and the synthesis processing performed by the video and audio synthesis / control server 2 in the first embodiment is performed in each terminal. is there.

  That is, the first terminal 4-1 transmits the video / audio 10-1 to the second terminal 4-2,..., The Nth terminal 4-N, and the second terminal 4-2,. .., 10-N are received from the Nth terminal 4-N. The second terminal 4-2 transmits the video / audio 10-2 to the first terminal 4-1, the third terminal 4-3,..., The Nth terminal 4-N, , 10-N are received from the terminal 4-1, the third terminal 4-3,..., And the Nth terminal 4-N. The N-th terminal 4-N transmits the video / audio 10-N to the first terminal 4-1,..., The N-1th terminal 4- (N−1), and the first terminal ..,..., 10- (N−1) are received from the N−1th terminal 4- (N−1).

  In addition, the first terminal 4-1 which is a moderator terminal transmits the emphasis target information 13 to the second terminal 4-2,..., The Nth terminal 4-N. The emphasis target information 13 is the same as the emphasis target information generated by the emphasis target selection means 208 of the video / audio synthesis / control server 2 of the first embodiment.

<Hardware configuration of terminal>
The hardware configuration of each terminal is the same as the hardware configuration (FIG. 2) of the terminal in the first embodiment.

<Functional block diagram of moderator terminal>
FIG. 12 is a functional block diagram of the moderator terminal in the remote conference system according to the second embodiment of the present invention.

  The first terminal 4-1, which is a moderator terminal, includes a video acquisition unit 401, a video transmission unit 402, a voice acquisition unit 403, a voice transmission unit 404, a sensor information acquisition unit 405, a motion detection unit 406, and an emphasis target selection unit 407. , Emphasis target transmission means 408, video reception means 409, video synthesis means 410, video output means 411, audio reception means 412, audio synthesis means 413, audio output means 414, video control means 415, and audio control means 416. Yes.

  Here, video acquisition means 401, video transmission means 402, audio acquisition means 403, audio transmission means 404, sensor information acquisition means 405, motion detection means 406, video reception means 409, video synthesis means 410, video output means 411, audio The receiving means 412 and the audio output means 414 are such that the video receiving means 409 receives the video 2,..., N from each other terminal, and the audio receiving means 410 receives the audio 2,. Except for receiving N, it has the same configuration and function as the means of the same name in FIG. 4 (the functional block diagram of the moderator terminal in the first embodiment). Video 2 and audio 2 constitute video / audio 10-2, and video N and audio N constitute video / audio 10-N.

  In FIG. 12, the emphasis target selection unit 407, the video synthesis unit 410, the voice synthesis unit 413, the video control unit 415, and the voice control unit 416 are the same as those in FIG. 5 (the video / sound synthesis / control server 2 in the first embodiment). In the functional block diagram of FIG.

  The emphasis target transmitting unit 408 is a unit that transmits the emphasis target information generated by the emphasis target selection unit 407 to the second terminal 4-2,..., The Nth terminal 4-N via the network 3.

<Functional block diagram of terminals other than moderator terminal>
FIG. 13 is a functional block diagram of the second terminal 4-2 which is one of terminals other than the moderator terminal in the remote conference system according to the second embodiment of the present invention. In this figure, the same means as those in FIG. 12 are denoted by the same reference numerals as those in FIG. 12, and the description thereof is omitted unless particularly necessary. Here, for convenience, the functional block diagram of the second terminal 4-2 is shown, but the functional blocks of the third terminal 4-3,..., The Nth terminal 4-N are also the second terminal. It is the same as the function block in 4-2.

  The second terminal 4-2 removes the sensor information acquisition unit 405, the motion detection unit 406, the enhancement target selection unit 407, and the enhancement target transmission unit 408 from the first terminal 4-1, and changes the enhancement target reception unit 417. It has an added configuration. However, as in the first embodiment, all the terminals may have the same configuration, and control may be performed so that unused means do not operate.

<Operation of remote conference system>
<< Transmission operation of the moderator terminal and reception operation of terminals other than the moderator terminal >>
FIG. 14 is a flowchart showing the transmission operation of the moderator terminal and the reception operation of terminals other than the moderator terminal in the remote conference system according to the second embodiment of the present invention.

  First, in the moderator terminal (first terminal 4-1), the video acquisition unit 401, the audio acquisition unit 403, and the sensor information acquisition unit 405 acquire video, audio, and sensor information, respectively (step S21). Next, the motion detection means 406 determines whether or not a predetermined motion has been detected from the video and sensor information acquired in step S21 (step S22).

  When detecting a predetermined motion (step S22: Yes), the motion detection unit 406 calculates coordinates (designated coordinates) indicated by the predetermined motion from the video and sensor information acquired in step S21, and adds them to the motion detection result. (Step S23).

  That is, steps S21 to S23 are the same as steps S1 to S3 in FIG. 10 (flow chart showing the operation of the first embodiment).

  In the next step S24, the emphasis target selecting means 407 selects the terminal to be emphasized from the motion detection result in step S23 and the video arrangement information supplied from the video synthesizing means 410, and the emphasis target is selected as the selection result. Generate information. Next, the emphasis target transmitting unit 408 transmits the audio and video acquired in step S21 and the emphasis target information generated in step S24 to other terminals (step S25). When the process proceeds directly from step S22 to step S25, that is, when a predetermined operation is not detected, the emphasis target information is information indicating that no emphasis target is selected.

  At terminals other than the moderator terminal (other terminals), the video and audio transmitted from all terminals other than the own terminal are received by the video receiving means 409 and the audio receiving means 412, respectively, and the emphasis target transmitted from the moderator terminal Information is received by the emphasis target receiving means 417 (step S26).

  Next, the video control unit 415 and the audio control unit 416 determine whether the enhancement target is selected, that is, whether the enhancement target information includes information indicating the selected terminal (step S27). .

  As a result of the determination, when the emphasis target is selected (step S27: Yes), the video control unit 415 and the audio control unit 416 emphasize the video and audio of the selected terminal at the time of each synthesis. The parameter is adjusted (step S28). Next, the video synthesizing unit 410 and the voice synthesizing unit 413 synthesize video and audio from each terminal based on the parameters adjusted by the video control unit 415 and the audio control unit 416 (step S29).

  As a result of the determination, when the emphasis target is not selected (step S27: No), the video control unit 415 and the audio control unit 416 leave the control signal parameters as they are, and the video synthesis unit 410 and the voice synthesis unit 413. Uses the parameters of the values as they are to synthesize video and audio from each terminal (step S29).

  Next, the video output unit 411 displays the synthesized video, and the audio output unit 414 reproduces the synthesized audio (step S30).

<< Transmission operation of terminals other than the moderator terminal and reception operation of the moderator terminal >>
FIG. 15 is a flowchart showing a transmission operation of a terminal other than the moderator terminal and a reception operation of the moderator terminal in the remote conference system according to the second embodiment of the present invention.

  In terminals other than the moderator terminal (other terminals), the video acquisition unit 401 and the audio acquisition unit 403 acquire video and audio, respectively (step S31), and the video transmission unit 402 and the audio transmission unit 404 respectively acquire the video acquisition unit. 401, the video and audio acquired by the audio acquisition means 403 are transmitted to each terminal (step S32).

  In the moderator terminal, the video reception unit 409 and the audio reception unit 412 respectively receive the video and audio transmitted from each terminal (step S33). Next, the video control unit 415 and the audio control unit 416 determine whether an enhancement target is selected and whether the enhancement target information includes information representing the selected terminal (step S34). This emphasis target information is the same as the emphasis target information transmitted in step S25 of FIG. Subsequent steps S34 to S37 are the same as steps S28 to S30 in FIG.

  In each of the above embodiments, the motion detection units 106 and 406 detect a predetermined motion by the presenter in a non-contact manner using the video acquired by the camera 521 and the information of the distance sensor. A predetermined operation can also be detected by detecting the position where the presenter touches with a finger or a touch pen using a mounted display. Further, a predetermined operation can be detected by detecting the line of sight of the presenter and detecting (calculating) the position on the video ahead of the line of sight.

  DESCRIPTION OF SYMBOLS 1 ... Terminal, 1-1, 4-1 ... 1st terminal, 1-2, 4-2, ..., 1-N, 4-N ... 2nd ..., Nth terminal, 106 , 406 ... motion detection means, 208, 407 ... highlighting object selection means, 209, 415 ... video control means, 210, 416 ... audio control means,

JP 2012-217068 A JP-A-6-121309

Claims (10)

A teleconferencing system that communicates information including video between conference terminals installed at multiple locations,
Position detecting means for detecting a desired position of a video displayed on one predetermined conference terminal when a conference participant using the conference terminal indicates the desired position by a predetermined first operation. When,
The predetermined one conference terminal from the position detected by the position detecting means and information indicating the position of the video of the bases of other conference terminals in the video displayed on the predetermined one conference terminal A site detection means for detecting the site displayed at the position indicated by the conference participant using
Video enhancement means for enhancing the video of the base detected by the base detection means;
A teleconferencing system. The remote conference system according to claim 1,
The remote conference system, wherein the information indicating the position of the video at the base includes information indicating the position of the video of a conference participant at the base. The remote conference system according to claim 1,
A teleconferencing system comprising voice emphasizing means for emphasizing the voice of the base detected by the base detection means. The remote conference system according to claim 1,
The first operation is a non-contact operation on the video, and the position detecting unit detects the instructed position in a non-contact manner. The remote conference system according to claim 1,
A teleconferencing system, comprising: video enhancement cancellation means for canceling enhancement of a video when a conference participant using the predetermined one conference terminal performs a second operation different from the first operation. A video processing method in a remote conference system for communicating information including video between conference terminals installed at a plurality of bases,
A position detection step of detecting a specified position of a video displayed on a predetermined one conference terminal when a conference participant using the conference terminal indicates the desired position by a predetermined operation;
The predetermined one conference terminal from the position detected in the position detection step and information indicating the positions of the video of the bases of other conference terminals in the video displayed on the predetermined one conference terminal A site detection step of detecting a site displayed at a position instructed by a conference participant using
A video enhancement step for enhancing the video of the location detected in the location detection step;
A video processing method in a remote conference system. A video control device used in a remote conference system that communicates information including video between conference terminals installed at a plurality of locations,
For each conference terminal, video transmission means for transmitting the video of the base of the other conference terminal,
Among the plurality of conference terminals, a conference participant who uses the conference terminal is transmitted from the video transmission means by a predetermined operation from a predetermined one of the conference terminals, and the video displayed on the conference terminal is displayed. A detection result receiving means for receiving a detection result of the designated position when a desired position is designated;
From the detection result received by the detection result receiving means and the information indicating the position of the video of the bases of other conference terminals in the video displayed on the predetermined one conference terminal, the predetermined one A site detection means for detecting a site displayed at a position designated by a conference participant using the conference terminal;
Video enhancement means for enhancing the video of the base detected by the base detection means;
A video control device.   The program for functioning a computer as each means of the image | video control apparatus described in Claim 7. A conference terminal used in a remote conference system that communicates information including video between conference terminals installed at multiple locations,
Video output means for displaying video of other conference terminal locations;
Position detection means for detecting the indicated position when a conference participant indicates a desired position of the video displayed on the video output means by a predetermined operation;
From the position detected by the position detecting means and the information indicating the position of the video of the bases of the other plurality of conference terminals displayed by the video output means, it is displayed at the position designated by the conference participant. A base detection means for detecting the base
Video enhancement means for enhancing the video of the base detected by the base detection means;
A conference terminal. A program for causing a computer to function as a position detection unit, a site detection unit, and a video enhancement unit of a conference terminal according to claim 9.

Images