JP2017092950A - Information processing apparatus, conference system, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus, conference system, information processing method, and program that are capable of grasping a state of a conference site in a system extracting a video region of a speaker.SOLUTION: An information processing apparatus for performing communication of video and sound with another information processing apparatus comprises: an input unit for accepting input of sound from a sound source; an imaging unit for imaging video of the sound source; a reception unit for receiving video and sound from the other information processing apparatus; an extraction unit that, when sound is input by the input unit, extracts a video region including a sound source generating the sound from the video imaged by the imaging unit to take the extracted video region as first video and, when sound is received from the other information processing apparatus by the reception unit, extracts a video region having a range wider than that of at least the first video from the video imaged by the imaging unit to take the extracted video region as second video; and a transmission unit for transmitting at least either of the first video and the second video to the other information processing apparatus.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an information processing apparatus, a conference system, an information processing method, and a program.

  Video conferencing systems that conduct remote conferences with remote locations via a communication network such as the Internet have become widespread. In this video conference system, an image (video) of a participant in the conference room is taken using a terminal device of the video conference system in a conference room where one of the parties such as a participant conducting a remote conference (video conference) exists. In addition, the voice of the utterance is input, and the video data and the voice data are transmitted to the other party's terminal device. And it displays on the display of the other party's conference room, and outputs a voice by a speaker, thereby realizing a conference between remote locations in a state close to the actual conference.

  In the video conference system, a microphone is used to acquire the voice of the conference participant, and a camera is used to acquire the video. However, since the camera has an angle of view, it is not possible to take pictures of participants outside the angle of view of the camera. In order to solve this problem, a method using a panoramic camera capable of photographing 360 degrees in all directions is known. On the other hand, since the microphone is usually omnidirectional, it is impossible to determine from which participant the voice is spoken, that is, the direction of the voice. In order to solve this problem, a method is known in which a microphone array is used to determine from which participant the voice is spoken, that is, the direction of the voice.

  As such a video conferencing system, an image corresponding to a part in which the direction of sound is specified using a microphone array is digitally cut out and displayed from an image taken using an omnidirectional camera module using a reflecting mirror. The technique to do is disclosed (patent document 1).

  In the technique described in Patent Document 1, when a participant at his / her own site speaks, the participant who speaks at his / her own site is switched and displayed on the display device at the other site. However, if the participant at his / her site is not speaking, the partner site does not specify how to display the status of the participant at his / her own site, and the overall state of his / her own site cannot be grasped. There is a point.

  The present invention has been made in view of the above, and provides an information processing apparatus, a conference system, an information processing method, and a program capable of grasping the state of a conference base even in a system that cuts out a video area of a speaker. With the goal.

  In order to solve the above-described problems and achieve the object, the present invention is an information processing apparatus that communicates video and sound with another information processing apparatus, an input unit that receives sound input of a sound source, and the sound source When a sound is input from the input unit, an image capturing unit that captures the image of the image, a receiving unit that receives the image and sound from the other information processing apparatus, and the sound input from the image captured by the image capturing unit When a video region including a sound source that emits sound is cut out to be a first video and sound is received from the other information processing apparatus by the receiving unit, at least the first video from the video captured by the imaging unit A cutout unit that cuts out a wider video area to form a second video, and a transmission unit that transmits at least one of the first video and the second video to the other information processing apparatus. That And butterflies.

  According to the present invention, it is possible to grasp the state of a conference base even in a system that cuts out a video area of a speaker.

FIG. 1 is a diagram illustrating an example of the overall configuration of a conference system according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the embodiment. FIG. 3 is a diagram illustrating an example of the arrangement of a plurality of microphones and the arrangement of a panoramic camera of the information processing apparatus according to the embodiment. FIG. 4 is a diagram illustrating an example of a functional block configuration of the information processing apparatus according to the embodiment. FIG. 5 is a diagram illustrating an arrangement example of participants who participate in the conference. FIG. 6 is a flowchart illustrating an example of a flow of mode determination processing of the information processing apparatus according to the embodiment. FIG. 7 is a diagram illustrating an overview of video display in each mode. FIG. 8 is a flowchart illustrating an example of the flow of a video cut-out operation in the two-screen switching mode of the information processing apparatus according to the embodiment. FIG. 9 is a diagram illustrating an example of screen transition in the two-screen switching mode of the information processing apparatus according to the embodiment. FIG. 10 is a diagram for explaining an example of a cut-out range for cutting out the entire video of the participant in the video conference. FIG. 11 is a diagram illustrating an example of video display when a video conference is performed between three or more locations.

  Hereinafter, embodiments of an information processing apparatus, a conference system, an information processing method, and a program according to the present invention will be described in detail with reference to FIGS. In addition, the present invention is not limited by the following embodiments, and constituent elements in the following embodiments are easily conceivable by those skilled in the art, substantially the same, and so-called equivalent ranges. Is included. Furthermore, various omissions, substitutions, changes, and combinations of the constituent elements can be made without departing from the scope of the following embodiments.

(Conference system configuration)
FIG. 1 is a diagram illustrating an example of the overall configuration of a conference system according to an embodiment. The configuration of the conference system 1 according to the present embodiment will be described with reference to FIG.

  As shown in FIG. 1, the conference system 1 according to the present embodiment includes two or more information processing devices (information processing devices 10a, 10b,...) And a conference server 20. The information processing apparatuses 10 a and 10 b can communicate with each other via the network 2 such as the Internet, and can communicate with the conference server 20. Note that the two or more information processing devices (10a, 10b,...) Illustrated in FIG. 1 are simply referred to as “information processing device 10” when referring to or collectively referring to any information processing device. 1 shows an example in which the information processing apparatus 10a is installed at the site A and the information processing apparatus 10b is installed at the site B.

  The information processing apparatus 10 is a conference terminal apparatus that establishes a session with another information processing apparatus based on the control of the conference server 20 and transmits / receives audio data and video data via the established session. . Thereby, in the conference system 1, a video conference (hereinafter sometimes simply referred to as “conference”) between a plurality of information processing devices (10a, 10b,...) Is realized.

  The conference server 20 is a server device that performs monitoring of whether or not each information processing device 10 is connected to the conference server 20, control of calling each information processing device 10 at the start of the conference, and control of information processing at the time of the conference. is there.

(Hardware configuration of information processing device)
FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the embodiment. FIG. 3 is a diagram illustrating an example of the arrangement of a plurality of microphones and the arrangement of a panoramic camera of the information processing apparatus according to the embodiment. Details of the hardware configuration of the information processing apparatus 10 according to the present embodiment will be described with reference to FIGS.

  As shown in FIG. 2, the information processing apparatus 10 according to the present embodiment includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an auxiliary storage device 204. A media drive 205, an operation button 206, a power switch 207, a network I / F 208, an image sensor I / F 209, a panoramic camera 210, an audio I / F 211, a microphone array 212, a speaker 213, An output I / F 214 and an external device I / F 216 are provided.

  The CPU 201 is an integrated circuit that controls the operation of the entire information processing apparatus 10. The ROM 202 is a non-volatile storage device that stores programs such as firmware for the information processing apparatus 10. The RAM 203 is a volatile storage device used as a work area for the CPU 201.

  The auxiliary storage device 204 is a non-volatile storage device that stores various programs for realizing the operation of the information processing apparatus 10 and various data such as video data and audio data. The auxiliary storage device 204 is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

  The media drive 205 is a device that controls reading and writing of data with respect to a recording medium 205a such as a flash memory.

  The operation button 206 is a button for performing a setting operation or the like for the information processing apparatus 10. The power switch 207 is a switch for switching on / off the power of the information processing apparatus 10.

  The network I / F 208 is an interface for communicating data using the network 2. The network I / F 208 is, for example, a NIC (Network Interface Card). The image sensor I / F 209 is an interface for transmitting video data to and from the panoramic camera 210 that captures a subject and obtains video data under the control of the CPU 201.

  The panoramic camera 210 is an imaging device that includes a lens and a solid-state imaging device that converts light into electric charges and converts an object image (video) into digital data. The panoramic camera 210 acquires video data of 360 degrees around. As described above, by acquiring 360-degree video data, it is possible to image all the participants participating in the conference around the information processing apparatus 10. The panoramic camera 210 is connected to the image sensor I / F 209. A CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) is used as the solid-state imaging device. Further, the panoramic camera 210 is installed, for example, at the center of the upper surface of the information processing apparatus 10 as shown in FIG.

  The audio I / F 211 is an interface that processes input / output of audio signals between the microphone array 212 that inputs audio and the speaker 213 that outputs audio, under the control of the CPU 201. The microphone array 212 is a sound collection device that inputs the voices of participants participating in the conference. The microphone array 212 includes a plurality of microphones, and can determine the direction of voice spoken by a conference participant, for example, under the control of the CPU 201. The speaker 213 is a device that outputs sound in accordance with the control of the CPU 201. The microphone array 212 and the speaker 213 are connected to the audio I / F 211, respectively. Moreover, the microphone array 212 includes, for example, six microphones (212a to 212f) as illustrated in FIG. For example, as illustrated in FIG. 3, the microphones 212 a to 212 f are distributed on the upper surface of the housing of the information processing apparatus 10. The microphone array 212 can determine the direction of sound based on, for example, a difference in timing at which sound is input to each of the microphones 212a to 212f. Note that the microphone array 212 is not limited to being configured with six microphones as shown in FIG. 3, and may be any one having a plurality of microphones. Further, as shown in FIG. 3, each microphone of the microphone array 212 is arranged in a distributed manner in the housing of the information processing apparatus 10, but is not limited to this, and the microphones 212 a to 212 f are included. The unit of the microphone array 212 that is included may be configured separately from the housing of the information processing apparatus 10.

  The output I / F 214 is an interface for transmitting video data to the external display device 215 under the control of the CPU 201. The external device connection I / F 216 is an interface through which external devices such as an external camera, an external microphone, and an external speaker can be electrically connected by a USB (Universal Serial Bus) cable or the like.

  The display device 215 is a display device that displays videos of participants at other sites participating in the conference. The display device 215 is, for example, a CRT (Cathode Ray Tube) display, an LCD (Liquid Crystal Display), an organic EL (Organic Electro-Luminescence) display, or the like. The display device 215 is connected to the output I / F 214 by a cable 214a. The cable 214a may be an analog RGB (VGA) signal cable or a component video cable, and may be an HDMI (registered trademark) (High-Definition Multimedia Interface) or DVI (Digital Video Interactive) signal. It may be a cable for use.

  CPU 201, ROM 202, RAM 203, auxiliary storage device 204, media drive 205, operation button 206, power switch 207, network I / F 208, image sensor I / F 209, audio I / F 211, output I / F 214, and external device I described above The / F216 is connected to be communicable with each other via a bus 217 such as an address bus and a data bus.

  Note that the hardware configuration of the information processing apparatus 10 is not limited to the configuration shown in FIG. For example, the media drive 205 may not be provided.

(Function block configuration of information processing device)
FIG. 4 is a diagram illustrating an example of a functional block configuration of the information processing apparatus according to the embodiment. FIG. 5 is a diagram illustrating an arrangement example of participants who participate in the conference. Details of the functional block configuration of the information processing apparatus 10 according to the present embodiment will be described with reference to FIGS. 4 and 5.

  As shown in FIG. 4, the information processing apparatus 10 according to the present embodiment includes a recognition unit 101, an arrangement specifying unit 102 (second specifying unit), a cutout unit 103, and a direction specifying unit 104 (first specifying unit). Section), management section 105 (switching section), designation section 106, transmission section 107, reception section 108, imaging control section 109, display control section 110, audio output control section 111, and input section 112. A storage unit 113, an operation unit 114, a communication unit 115, an imaging unit 116, a display unit 117, and an audio output unit 118.

  The recognizing unit 101 is a functional unit that recognizes a face portion of one or more participants included in an image captured by the image capturing unit 116 as a face image (hereinafter sometimes referred to as “face recognition”). The recognition unit 101 may recognize a participant's face by using a known face recognition image analysis method. By counting the number of times the recognition unit 101 has recognized the face from the image captured by the imaging unit 116, the number of participants in the conference can be known. The recognition unit 101 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  Note that the recognition unit 101 recognizes the participant by recognizing the part of the participant's face as a face image. However, the recognition unit 101 is not limited to this, and other recognition methods such as recognition by human body detection are used. It is good also as what recognizes a participant.

  The arrangement specifying unit 102 specifies the position of the participant whose face is recognized by the recognition unit 101 in the image captured by the imaging unit 116, and specifies the arrangement pattern of the participant in the conference room of the participant who participates in the conference. It is a functional part. For example, in the conference room shown in FIG. 5A, the information processing device 10 and the display device 215 are placed on the desk 40, and the vicinity of the display device 215 placed on the desk 40 (see FIG. 5A). It is assumed that the whiteboard 50 is installed in the area P7). In such a conference room state, when an image of 360 degrees omnidirectional (hereinafter, also referred to as “panoramic image”) is captured by the imaging unit 116 of the information processing apparatus 10, the arrangement specifying unit 102 recognizes the recognition unit. For the participants 60a to 60e whose faces are recognized by the user 101, the participant 60a is in the region P1, the participant 60b is in the region P2, the participant 60c is in the region P3, the participant 60d is in the region P5, and the participant 60e. An arrangement pattern indicating that is located in the region P6 is specified. In FIG. 5B, in the panoramic image captured by the image capturing unit 116, the arrangement specifying unit 102 participates in the region P2 with respect to the participants 61a to 61d whose faces are recognized by the recognition unit 101. An arrangement pattern indicating that the person 61b is located in the area P3, the participant 61c is located in the area P4, and the participant 61d is located in the area P5 is specified. That is, specifying the arrangement pattern specifies in which direction the participant is arranged with respect to the information processing apparatus 10. The arrangement specifying unit 102 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The image captured by the imaging unit 116 is sometimes referred to as a “panoramic image”. However, when an image is shown as an object captured by the imaging unit 116, it may be referred to as a “panoramic image”. However, the video is a concept including an image.

  The cutout unit 103 is a functional unit that cuts out a video region of a specific participant whose face is recognized by the recognition unit 101 or a video region including all the participants participating in the conference from the panoramic video. When extracting the video area of a specific participant from the panoramic video, for example, the clipping unit 103 selects a video area corresponding to the audio direction of the participant (sound source) specified by the direction specifying unit 104 as described later. It should be cut out. The cutout unit 103 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The direction specifying unit 104 is a functional unit that specifies the direction of audio input by the input unit 112 that is a microphone array. Specifically, the direction specifying unit 104 specifies the direction of audio based on, for example, a difference in timing at which audio is input to a plurality of microphones constituting the input unit 112 that is a microphone array. The direction specifying unit 104 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The management unit 105 is a functional unit that manages the operation mode of the information processing apparatus 10. Specifically, the management unit 105 counts the number of times the face is recognized from the panoramic image by the recognition unit 101, thereby grasping the number of participants in the conference and determining the operation mode of the information processing apparatus 10. The operation for determining the operation mode of the information processing apparatus 10 will be described later with reference to FIG. The management unit 105 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The designation unit 106 is a functional unit that designates in which display area of the display unit 117 of the information processing apparatus 10 at another base the video data corresponding to the video area cut out by the cutout unit 103 is to be displayed. Specifically, the designation unit 106 generates designation information for designating which display area to display. The designation unit 106 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The transmission unit 107 transmits the video data corresponding to the video area cut out by the cut-out unit 103 and the audio data input by the input unit 112 via the communication unit 115 and the network 2 to information processing apparatuses at other bases. 10 is a functional unit that transmits data to 10. Specifically, for example, the transmission unit 107 encodes video data and audio data and transmits the encoded data to the information processing apparatus 10 at another site. Here, a known method may be used as the encoding method. For example, H.M. H.264 / AVC, or H.264. A compression coding technique such as H.264 / SVC may be used. The transmission unit 107 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The receiving unit 108 is a functional unit that receives video data and audio data received from the information processing apparatus 10 at another site via the network 2 and the communication unit 115. Specifically, for example, the receiving unit 108 decodes received video data and audio data, sends the decoded video data to the display control unit 110, and sends the decoded audio data to the audio output control unit 111. Here, a known method may be used as a decoding method. The receiving unit 108 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The imaging control unit 109 is a functional unit that controls the operation of the imaging unit 116. Specifically, for example, the imaging control unit 109 controls the start and stop operations of imaging by the imaging unit 116 and acquires a panoramic image captured by the imaging unit 116. The imaging control unit 109 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The display control unit 110 is a functional unit that performs control to display various images on the display unit 117. The display control unit 110 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The audio output control unit 111 is a functional unit that controls the audio output unit 118 to output various types of audio. The audio output control unit 111 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The input unit 112 is a functional unit that inputs voice. The input unit 112 is realized by, for example, the microphone array 212 illustrated in FIG.

  The storage unit 113 is a functional unit that stores various programs for realizing the operation of the information processing apparatus 10, video data, audio data, and information such as the arrangement pattern specified by the arrangement specifying unit 102. The storage unit 113 is realized by, for example, the RAM 203 and the auxiliary storage device 204 illustrated in FIG.

  The operation unit 114 is a functional unit that receives various operation inputs of a user of the information processing apparatus 10 (for example, a conference participant). The operation unit 114 is realized by, for example, the operation button 206 and the power switch 207 shown in FIG. Note that the operation unit 114 is not limited to the operation button 206 and the power switch 207 illustrated in FIG. 2, and may be realized by a mouse, a keyboard, a touch panel, or the like.

  The communication unit 115 is a functional unit that performs data communication with the other information processing apparatus 10 and the conference server 20 via the network 2. The communication unit 115 is realized by, for example, the network I / F 208 shown in FIG.

  The imaging unit 116 is a functional unit that captures panoramic images or panoramic images in all directions of 360 degrees. The imaging unit 116 is realized by, for example, the panoramic camera 210 illustrated in FIG.

  The display unit 117 is a functional unit that displays various images under the control of the display control unit 110. The display unit 117 is realized by, for example, the display device 215 illustrated in FIG.

  The sound output unit 118 is a functional unit that outputs various sounds according to the control of the sound output control unit 111. The audio output unit 118 is realized by, for example, the speaker 213 illustrated in FIG.

  Note that the recognition unit 101, the arrangement specifying unit 102, the cutout unit 103, the direction specifying unit 104, the management unit 105, the specifying unit 106, the transmitting unit 107, the receiving unit 108, and the imaging control unit 109 of the information processing apparatus 10 illustrated in FIG. The display control unit 110, the audio output control unit 111, the input unit 112, the storage unit 113, the operation unit 114, the communication unit 115, the imaging unit 116, the display unit 117, and the audio output unit 118 are conceptually shown functions. However, it is not limited to such a configuration. For example, a plurality of functional units illustrated as independent functional units in the information processing apparatus 10 illustrated in FIG. 4 may be configured as one functional unit. On the other hand, in the information processing apparatus 10 illustrated in FIG. 4, the functions of one functional unit may be divided into a plurality of units and configured as a plurality of functional units.

  Also, the recognition unit 101, the arrangement specifying unit 102, the cutout unit 103, the direction specifying unit 104, the management unit 105, the specifying unit 106, the transmitting unit 107, the receiving unit 108, the imaging control unit 109, and the display control unit of the information processing apparatus 10 A part or all of the 110 and the audio output control unit 111 may be realized by a hardware circuit such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) instead of a program that is software.

(Mode decision processing)
FIG. 6 is a flowchart illustrating an example of a flow of mode determination processing of the information processing apparatus according to the embodiment. FIG. 7 is a diagram illustrating an overview of video display in each mode. The flow of the mode determination process of the information processing apparatus 10 according to the present embodiment will be described with reference to FIGS. In the following description, a video conference between two sites, that is, the information processing device 10 at the local site and the information processing device 10 at the other site is assumed.

<Step S11>
First, a participant who wants to participate in the conference operates the operation unit 114 of the information processing apparatus 10 to turn on the information processing apparatus 10 and captures a surrounding image (panoramic image) with the imaging unit 116. Operation input is performed. Upon receiving operation information for capturing a panoramic image from the operation unit 114, the imaging control unit 109 causes the imaging unit 116 to capture a panoramic image. Then, the process proceeds to step S12.

<Step S12>
The recognition unit 101 recognizes (face recognition) a face part of one or more participants included in the image captured by the imaging unit 116 as a face image. Then, the process proceeds to step S13.

<Step S13>
The arrangement specifying unit 102 specifies the position of the participant whose face is recognized by the recognition unit 101 in the panoramic image captured by the imaging unit 116, and specifies the arrangement pattern of the participant in the conference room of the participant who participates in the conference. To do. The arrangement specifying unit 102 causes the storage unit 113 to store information on the specified arrangement pattern. Then, the process proceeds to step S14.

<Step S14>
The management unit 105 grasps the number of participants in the conference by counting the number of times the recognition unit 101 has recognized the face from the panoramic image. When the number of participants in the conference is 1 (step S14: 1), the process proceeds to step S15. When the number of participants is 2 (step S14: 2), the process proceeds to step S16. When the number of participants is 3 or more (step S14: 3 or more), the process proceeds to step S17.

<Step S15>
When determining that the number of participants in the conference is one, the management unit 105 switches the operation mode of the information processing apparatus 10 to the one-screen fixed mode (first operation mode). Then, the information processing apparatus 10 ends the mode determination process.

  When the information processing apparatus 10 at its own site operates in the single screen fixed mode, there is only one participant in the conference at its own site. Send video data and audio data.

  Specifically, first, the information processing device 10 at the local site and the information processing device 10 at the other site exchange information on the operation mode with each other via the network 2. Here, the information processing apparatus 10 at its own site transmits information indicating that it operates in the one-screen fixed mode to the information processing apparatus 10 at the other site via the network 2.

  The cutout unit 103 of the information processing apparatus 10 at the local site cuts out the video area of one participant whose face is recognized by the recognition unit 101 from the panoramic video imaged by the imaging unit 116. The transmission unit 107 of the information processing apparatus 10 at the local site encodes the video data of the video area cut out by the cut-out unit 103 and the audio data input by the input unit 112, and transmits the communication unit 115 and the network 2. To the information processing apparatus 10 at the partner site.

  When the receiving unit 108 of the information processing apparatus 10 at the partner site receives video data and audio data from the information processing apparatus 10 at its own site via the network 2 and the communication unit 115, the receiving unit 108 decodes the video data and audio data. To do. The receiving unit 108 of the information processing apparatus 10 at the partner site sends the decoded video data to the display control unit 110 and sends the decoded audio data to the audio output control unit 111. Since the information processing apparatus 10 at the partner site recognizes that the information processing apparatus 10 at the partner site is operating in the one-screen fixed mode, the display control unit 110 of the information processing apparatus 10 at the partner site displays FIG. As shown to (a), the display area 300 (specific display area) in the display screen 215a of the display part 117 (display device 215) is not divided | segmented like FIG.7 (b) and (c). Then, the display control unit 110 of the information processing apparatus 10 at the partner site receives the received video data (one participant at the local site extracted by the clipping unit 103 of the information processing apparatus 10 at the local site (FIG. 7A In the example of (), the video data of the video area including the participant X) is displayed on the display area 300 as shown in FIG. In addition, the audio output control unit 111 of the information processing apparatus 10 at the partner site causes the audio output unit 118 to output the received audio data as audio.

<Step S16>
When the management unit 105 determines that the number of participants in the conference is two, the management unit 105 switches the operation mode of the information processing apparatus 10 to the two-screen fixed mode (second operation mode). Then, the information processing apparatus 10 ends the mode determination process.

  Specifically, first, the information processing device 10 at the local site and the information processing device 10 at the other site exchange information on the operation mode with each other via the network 2. Here, the information processing apparatus 10 at its own site transmits information indicating that it operates in the two-screen fixed mode to the information processing apparatus 10 at the other site via the network 2.

  The cutout unit 103 of the information processing apparatus 10 at the local site cuts out the video areas of the two participants whose faces are recognized by the recognition unit 101 from the panoramic video imaged by the imaging unit 116. The transmission unit 107 of the information processing apparatus 10 at the local site encodes the video data of the video area cut out by the cut-out unit 103 and the audio data input by the input unit 112, and transmits the communication unit 115 and the network 2. To the information processing apparatus 10 at the partner site.

  When the receiving unit 108 of the information processing apparatus 10 at the partner site receives video data and audio data from the information processing apparatus 10 at its own site via the network 2 and the communication unit 115, the receiving unit 108 decodes the video data and audio data. To do. The receiving unit 108 of the information processing apparatus 10 at the partner site sends the decoded video data to the display control unit 110 and sends the decoded audio data to the audio output control unit 111. Since the information processing device 10 at the partner site recognizes that the information processing device 10 at the partner site is operating in the two-screen fixed mode, the display control unit 110 of the information processing device 10 at the partner site is shown in FIG. As shown in (b), the display area 300 on the display screen 215a of the display unit 117 (display device 215) is divided into two to generate a divided area 300a and a divided area 300b. Then, the display control unit 110 of the information processing device 10 at the partner site receives the received video data of the two people at the own site (the participation of the two people at the own site cut out by the cutting unit 103 of the information processing device 10 at the own site) (In the example of FIG. 7B, the video data of the video area including the participants X and Y) is displayed in the divided areas 300a and 300b, respectively, as shown in FIG. 7B. In addition, the audio output control unit 111 of the information processing apparatus 10 at the partner site causes the audio output unit 118 to output the received audio data as audio.

<Step S17>
When the management unit 105 determines that the number of participants in the conference is three or more, the management unit 105 switches the operation mode of the information processing apparatus 10 to the two-screen switching mode (third operation mode). Then, the information processing apparatus 10 ends the mode determination process.

  Similar to the one-screen fixed mode and the two-screen fixed mode, the information processing apparatus 10 at the local site and the information processing apparatus 10 at the partner site exchange information on the operation mode with each other via the network 2. Here, the information processing device 10 at its own site transmits information indicating that it operates in the two-screen switching mode to the information processing device 10 at the other site via the network 2. Details of the video cut-out operation in the two-screen switching mode will be described later with reference to FIGS. In the example of FIG. 7C, the display area 300 on the display screen 215a of the display unit 117 (display device 215) of the information processing apparatus 10 at the partner site is divided into two divided areas 300a and 300b. Participant X is displayed on the screen, and the entire participant (participants V to Z) at the local site is displayed in the divided area 300b.

  The mode determination process is executed by the information processing apparatus 10 through the operations in steps S11 to S17 described above.

  Note that the mode determination process shown in FIG. 6 may be re-executed every predetermined time or when a predetermined condition is satisfied during the conference. For example, when the position of the participant whose face is recognized by the recognition unit 101 is moved in the panoramic image captured by the image pickup unit 116, the use of the face recognized by the recognition unit 101 last time is used. For example, or when a participant other than the participant whose face was previously recognized by the recognition unit 101 is recognized as a face.

  In the mode determination process shown in FIG. 6, when the number of participants is two or more, the display control unit 110 of the information processing apparatus 10 at the partner site is switched to the two-screen fixed mode or the two-screen switching mode. 7 (b) or 7 (c), the display area 300 on the display screen 215a of the display unit 117 is divided into two to generate a divided area 300a and a divided area 300b. It is not limited to. That is, even when the number of participants is two or more, the display area 300 may be processed as one screen without being divided.

(Video clipping operation in two-screen switching mode)
FIG. 8 is a flowchart illustrating an example of the flow of a video cut-out operation in the two-screen switching mode of the information processing apparatus according to the embodiment. FIG. 9 is a diagram illustrating an example of screen transition in the two-screen switching mode of the information processing apparatus according to the embodiment. FIG. 10 is a diagram for explaining an example of a cut-out range for cutting out the entire video of the participant in the video conference. With reference to FIGS. 8 to 10, the flow of the video cut-out operation in the two-screen switching mode of the information processing apparatus 10 according to the present embodiment will be described. In the following description, a video conference between two sites, that is, the information processing device 10 at the local site and the information processing device 10 at the other site is assumed.

<Step S31>
As described above, the information processing apparatus 10 at the local site and the information processing apparatus 10 at the partner site exchange information on the operation mode with each other via the network 2. Here, the information processing device 10 at its own site transmits information indicating that it operates in the two-screen switching mode to the information processing device 10 at the other site via the network 2. Since the information processing device 10 at the partner site recognizes that the information processing device 10 at the partner site is operating in the two-screen switching mode, the display control unit 110 of the information processing device 10 at the partner site is shown in FIG. As shown in (c), the display area 300 on the display screen 215a of the display unit 117 (display device 215) is divided into two to generate a divided area 300a and a divided area 300b. Then, the input unit 112 of the information processing apparatus 10 at the local site starts accepting voice input. Then, the process proceeds to step S32.

<Step S32>
When the voice is input by the input unit 112 (step S32: Yes), the process proceeds to step S33, and when the voice is not input by the input unit 112, that is, when the participant at the other party is speaking (step S32: No), the process proceeds to step S38. Here, the case where the partner site is speaking is a case where, for example, the video data and voice data of the speaker are received from the information processing apparatus 10 at the partner site by the receiving unit 108.

<Step S33>
The direction specifying unit 104 of the information processing apparatus 10 at its own site specifies the direction of the voice input to the input unit 112 when one of the three or more participants speaks. Then, control goes to a step S34.

<Step S34>
The cut-out unit 103 of the information processing apparatus 10 at the local site participates in the direction closest to the voice direction specified by the direction specifying unit 104 among the participant directions indicated by the arrangement pattern specified by the arrangement specifying unit 102. The participant is determined as a participant who speaks (current speaker), and a video area (first video) including the participant is cut out from the panoramic video captured by the imaging unit 116. Further, the cutout unit 103 continues to cut out the video area including the participant (previous speaker) (second sound source) speaking before the participant speaking in step S33. Then, the process proceeds to step S35. The cutout unit 103 cuts out the video region of the participant in the direction closest to the audio direction specified by the direction specifying unit 104 among the participant directions indicated by the arrangement pattern specified by the arrangement specifying unit 102. However, the present invention is not limited to this. That is, the cutout unit 103 may cut out the video area corresponding to the direction of the audio specified by the direction specifying unit 104 directly from the panoramic image without using the arrangement pattern.

<Step S35>
The transmission unit 107 of the information processing apparatus 10 at its own site has the current utterance extracted by the extraction unit 103 and the video data of the video regions of the previous utterers, and the current utterance input by the input unit 112. The voice data of the person is encoded and transmitted to the information processing apparatus 10 at the partner site via the communication unit 115 and the network 2. In addition, the designation unit 106 of the information processing apparatus 10 at the local site uses the video data of the video regions of the current speaker and the previous speaker extracted by the clipping unit 103 in the divided regions 300a and 300b of the partner base. The designation information for designating which display is to be generated is generated, and the transmission unit 107 transmits the designation information together with the video data to the information processing apparatus 10 at the partner site. Then, the process proceeds to step S36.

<Step S36>
When receiving the video data and the audio data from the information processing apparatus 10 at the local site via the network 2 and the communication unit 115, the reception unit 108 of the information processing apparatus 10 at the partner site decodes the video data and the audio data. The receiving unit 108 of the information processing apparatus 10 at the partner site sends the decoded video data to the display control unit 110 and sends the decoded audio data to the audio output control unit 111. Then, the process proceeds to step S37.

<Step S37>
The display control unit 110 of the information processing apparatus 10 at the other party bases the divided area that was originally displayed among the divided areas 300a and 300b in accordance with the designation of the received designation information for the received video data of the previous speaker's video area. To display continuously. Also, the display control unit 110 switches and displays the received video data of the current speaker's video area in the other divided area in accordance with the designation of the received designation information. In addition, the audio output control unit 111 of the information processing apparatus 10 at the partner site causes the audio output unit 118 to output the received audio data as audio. Then, the process proceeds to step S42.

<Step S38>
The cutout unit 103 of the information processing apparatus 10 at its own site uses a video area (first image) including all participants at its own site from the panoramic video captured by the imaging unit 116 using the arrangement pattern specified by the arrangement specifying unit 102. Cut out an example of two images. For example, as shown in FIG. 10A, two participants are located on the upper side of the desk 40 on the paper surface, and one person is on the lower side of the paper surface, surrounding the desk 40 on which the information processing apparatus 10 is placed. Consider the case where the participants are located. In this case, the cutout unit 103 is a range including all three participants, and the left participant of the two persons positioned on the upper side of the desk 40 and the lower side of the page 40 The video area of the cutout range 400a, which is the minimum range starting from the participating participant, is cut out from the panoramic video.

  Further, as shown in FIG. 10B, two participants are located on the upper side of the desk 40 on the paper 40 side, and one person on the right side of the paper 40 is surrounded by the desk 40 on which the information processing apparatus 10 is placed. Consider the case where a participant is located. In this case, the cutout unit 103 is a range that includes all three participants, and is located on the left side of the two persons located on the upper side of the desk 40 and on the right side of the paper. The video area of the cutout range 400b, which is the minimum range with the participant at the end, is cut out from the panoramic video.

  Note that the cutout unit 103 cuts out a video area including all participants at the local site from the panoramic video using the arrangement pattern, but the present invention is not limited to this. For example, the cutout unit 103 may cut out a video area wider than the video area in which only the participant who has spoken before is cut out without using the arrangement pattern, or transmits the entire panoramic video. It may be sent to the unit 107. However, when the arrangement pattern is used, the cutout unit 103 can grasp the positions of the participants at its own base, and thus has an advantage that it can cut out the video area including the entire participants.

  In addition, the cutout unit 103 continues to cut out the video area including the participant who spoke before (the previous speaker) (first sound source). Then, the process proceeds to step S39.

<Step S39>
The transmission unit 107 of the information processing apparatus 10 at its own site encodes the video data of the video regions of all the participants and the previous utterers cut out by the cut-out unit 103, and transmits the video data via the communication unit 115 and the network 2. The information is transmitted to the information processing apparatus 10 at the partner site. In addition, the designation unit 106 of the information processing apparatus 10 at the local site converts the video data of the video regions of all the participants and the previous utterances cut out by the cut-out unit 103 to any of the divided regions 300a and 300b of the counterpart site. The transmission unit 107 transmits the designation information to the information processing apparatus 10 at the partner site together with the video data. Then, the process proceeds to step S40.

<Step S40>
When the reception unit 108 of the information processing apparatus 10 at the partner site receives video data from the information processing apparatus 10 at its own site via the network 2 and the communication unit 115, the reception unit 108 decodes the video data. The receiving unit 108 of the information processing apparatus 10 at the partner site sends the decoded video data to the display control unit 110. Then, the process proceeds to step S41.

<Step S41>
The display control unit 110 of the information processing apparatus 10 at the other party bases the divided area that was originally displayed among the divided areas 300a and 300b in accordance with the designation of the received designation information for the received video data of the previous speaker's video area. To display continuously. Further, the display control unit 110 switches and displays the received video data of the entire participant's video area in the other divided area in accordance with the designation of the received designation information. Then, the process proceeds to step S42.

<Step S42>
The information processing apparatus 10 at its own site determines whether or not the video conference has ended. For example, the information processing apparatus 10 determines whether the user has performed an operation for ending the conference from the operation unit 114. When the video conference is finished (step S42: Yes), the video clipping operation is finished. When the video conference is not finished (step S42: No), the process returns to step S32.

  Through the operations in steps S31 to S42 described above, the information processing apparatus 10 executes the video cutout operation in the two-screen switching mode.

  In the above-described operation in the two-screen switching mode, the video data of the video area cut out by the cutout unit 103 is displayed in which display area (for example, the divided areas 300a and 300b) of the display unit 117 of the partner base. The designation unit 106 of the information processing apparatus 10 at the local site that is the transmission side of the video data generates the designation information in order to designate whether or not to perform the designation. However, the present invention is not limited to this. For example, the information processing apparatus 10 at the partner site that has received the video data may determine in which display area of the display unit 117 the received video data is to be displayed.

  Next, referring to FIG. 9, in the two-screen switching mode, the screen transition in the display area 300 (hereinafter simply referred to as “display area 300” in the description of FIG. 9) of the display unit 117 (display device 215) at the partner site. A specific example will be described.

  (1) of FIG. 9 shows a display example of the initial state of the display area 300. For example, the display control unit 110 displays an image of an arbitrary participant at the local site in the divided region 300a of the display region 300, and displays an image of the entire participant at the local site in the divided region 300b of the display region 300. .

  (2) of FIG. 9 shows a display example of the display area 300 when the participant X at the local site speaks from the state of (1). The display control unit 110 switches to the video of the participant X speaking at the local site in the divided region 300a (an example of the first video) while keeping the display of the video of the entire participant at the local site in the divided region 300b. Display.

  (3) of FIG. 9 shows a display example of the display area 300 when a participant Y who is different from the participant X at the local site speaks from the state of (2). The display control unit 110 continuously displays the video of the participant X who is the previous speaker in the divided area 300a, and the video of the participant Y who is currently speaking in the local area in the divided area 300b (the first video). Switch to (example) and display.

  (4) of FIG. 9 shows a display example of the display area 300 when the participant X at his / her own site speaks again from the state of (3). The display control unit 110 continuously displays the video of the participant Y who is the previous speaker in the divided area 300b, and the video of the participant X who is currently speaking in the local area in the divided area 300a (the first video). Switch to (example) and display. However, in (3) of FIG. 9, since the video of the participant X was originally displayed in the divided area 300a, the video displayed in the divided area 300a is not substantially changed.

  (5) of FIG. 9 shows a display example of the display area 300 when a participant Z different from the participants X and Y at his / her own site speaks from the state of (3). The display control unit 110 continuously displays the video of the participant Y who is the previous speaker in the divided area 300b, and the video of the participant Z who is currently speaking in the local area in the divided area 300a (the first video). Switch to (example) and display.

  (6) of FIG. 9 shows a display example of the display area 300 when the participant at the local site stops speaking due to the participant speaking at the partner site from the state of (3). The display control unit 110 continuously displays the video of the participant Y who is the previous speaker in the divided area 300b, and switches to the video of the entire participant at the local site (an example of the second video) in the divided area 300a. Display.

  As described above, when a participant at another site speaks, the cutout unit 103 of the information processing apparatus 10 at the local site cuts out a video area including the entire participant at the local site from the panoramic video, and The video region including the participant who spoke before (the previous speaker) is continuously cut out and displayed on the display region 300 of the display device 215 divided into two at the partner site. Thereby, when a participant at the partner site speaks, the participant at the partner site can know the overall atmosphere of the subject site. In addition, when a participant at the partner site is speaking, the speaker at the partner site is displayed in one divided area of the display area 300 divided into two, so that the speaker at the partner site is You can read what expression the participant who spoke last at his / her site is listening to his / her speech.

  Further, when voice is input to the input unit 112 of the information processing apparatus 10 at the local site (when a participant at the local site is speaking), the display area 300 of the display device 215 divided into two at the partner site is displayed. Each of the current speaker and the previous speaker at his / her base is displayed. As a result, the range of screen transition can be minimized. In addition, because the participant at the other site is watching the image of the divided area of the speaker at his / her own site, even if another participant at his / her own site speaks newly, Since the images are switched, the stress felt by the screen transition can be reduced.

  In addition, when the number of participants at the local site is one, the display area 300 of the display device 215 at the other site is displayed without being divided, and when the number of participants at the local site is two, The display area 300 of the display device 215 at the partner site is divided into two parts, and the two participants are fixedly displayed. In this case, since screen transition does not occur, stress due to screen transition can be reduced.

  In addition, although the imaging part 116 shall be implement | achieved by the panorama camera 210 and shall take a panoramic image or a panoramic image, it is not necessarily limited to this. That is, when the imaging range does not need to be 360 degrees in all directions, it is not necessary to use a panoramic camera. For example, even an imaging device (camera) having an angle of view that covers a necessary imaging range can be used. Good. In this case, the video may be cut out within the range of the angle of view that can be captured by the imaging apparatus.

  In addition, when no sound is input to the input unit 112 of the information processing apparatus 10 at the local site (when there is no speaker), that is, when a participant at the other site is speaking, the information processing apparatus 10 at the local site is disconnected. Although the output unit 103 cuts out a video area including all participants at the local site from the panoramic video, the present invention is not limited to this. In other words, when a participant at the partner site speaks, the participant at the partner site satisfies at least one participant, not the entire participant at the own site, to the extent that it satisfies that it knows the atmosphere of the own site. The video area including the person may be cut out. Further, for example, the cutout unit 103 may cut out a video area in a wider range than the video area in which only a specific participant is cut out, or may send the entire panoramic video to the transmission unit 107.

(Modification)
The operation of the conference system 1 according to this modification will be described focusing on differences from the operation of the conference system 1 according to the above-described embodiment. In the above-described embodiment, the video cut-out operation assuming a video conference between two sites has been described, but in this modification, an operation when a video conference is performed between three sites or more will be described.

  FIG. 11 is a diagram illustrating an example of video display when a video conference is performed between three or more locations. With reference to FIG. 11, a video cutout operation when a video conference is performed between three or more sites will be described.

  First, the information processing apparatuses 10 at each site exchange information on operation modes with each other via the network 2. Thereby, the information processing apparatus 10 at each base can recognize the operation mode of each other and can also recognize the number of participating bases. In the example shown in FIG. 11, the display state of the display screen 215a of the own site (site A) when a video conference is held between the four sites is shown. When a video conference is performed between the four bases, the display screen 215a of the local base displays display areas corresponding to the bases B to D which are other bases as shown in FIG. As shown in FIG. 11, as a display area to be displayed on the display screen 215a, a display area 301 (specific display area) corresponding to the base B and a display area 302 (specific display area) corresponding to the base C are displayed. The display area 303 (specific display area) corresponds to the site D.

  In the example of FIG. 11, the information processing apparatus 10 at its own base (base A) receives information indicating that it operates in the two-screen fixed mode from the information processing apparatus 10 at the base B, and from the information processing apparatus 10 at the base C. It is assumed that information indicating that it operates in the two-screen switching mode is received, and information indicating that it operates in the two-screen switching mode is received from the information processing apparatus 10 at the site D. Then, the display control unit 110 of the information processing apparatus 10 at the local site generates the divided areas 301a and 301b by dividing the display area 301 into two on the display screen 215a of the display unit 117 (display apparatus 215). Is divided into two to generate divided areas 302a and 302b, and the display area 303 is divided into two to generate divided areas 303a and 303b. Even when a video conference is performed between three or more sites, the video cut-out operation between the two sites is the same as the operation described with reference to FIGS.

  In the example of FIG. 11, a state in which a participant at the base (base A) is speaking is shown. That is, since the information processing apparatus 10 at the site B is operating in the two-screen fixed mode, there are two participants at the site B, and the divided regions 301a and 301b are images of the two participants, respectively. The display is fixed. Further, since the information processing apparatus 10 at the site C operates in the two-screen switching mode, there are three or more participants at the site C, and the video of the previous speaker at the site C is displayed in the divided area 302a. In the divided area 302b, an image of the entire participant at the base C is displayed. Since the information processing apparatus 10 at the site D operates in the two-screen switching mode, there are three or more participants at the site D, and the divided region 303a displays the video of all the participants at the site D. In the display area 303b, the video of the previous speaker at the site D is displayed.

  As described above, when a video conference is performed between three or more sites, the video cut-out operation between each of the two sites can be realized by applying the operations described above with reference to FIGS. As a result, the same effects as those of the above-described embodiment can be obtained with each base.

  In the above-described embodiment and modification, when at least one of the functional units of the information processing apparatus 10 is realized by executing a program, the program is provided by being incorporated in advance in a ROM or the like. A program executed by the information processing apparatus 10 according to the above-described embodiment and modification is a file in an installable format or an executable format, and is a CD-ROM (Compact Disc Read Only Memory), a flexible disk (FD). ), CD-R (Compact Disk-Recordable), DVD (Digital Versatile Disc), or other computer-readable storage media. In addition, the program executed by the information processing apparatus 10 according to the above-described embodiments and modifications is stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also good. Further, the program executed by the information processing apparatus 10 according to the above-described embodiment and modification may be configured to be provided or distributed via a network such as the Internet. In addition, the program executed by the information processing apparatus 10 according to the above-described embodiment and the modification has a module configuration including at least one of the above-described functional units. By reading the program from the storage device (for example, the ROM 202 and the auxiliary storage device 204) and executing the program, the above-described functional units are loaded on the main storage device (for example, the RAM 203) and generated. Yes.

DESCRIPTION OF SYMBOLS 1 Conference system 2 Network 10, 10a, 10b Information processing apparatus 20 Conference server 40 Desk 50 White board 60a-60e Participant 61a-61d Participant 101 Recognition part 102 Arrangement | positioning specification part 103 Cutout part 104 Direction specification part 105 Management part 106 Designation unit 107 Transmission unit 108 Reception unit 109 Imaging control unit 110 Display control unit 111 Audio output control unit 112 Input unit 113 Storage unit 114 Operation unit 115 Communication unit 116 Imaging unit 117 Display unit 118 Audio output unit 201 CPU
202 ROM
203 RAM
204 Auxiliary storage device 205 Media drive 205a Recording medium 206 Operation button 207 Power switch 208 Network I / F
209 Image sensor I / F
210 Panorama Camera 211 Audio I / F
212 Microphone array 212a to 212f Microphone 213 Speaker 214 Output I / F
214a Cable 215 Display device 215a Display screen 216 External device I / F
217 Bus 300-303 Display area 300a, 300b Divided area 301a, 301b Divided area 302a, 302b Divided area 303a, 303b Divided area 400a, 400b Cutout area P1-P7 area

JP 2010-081644 A

Claims (11)

An information processing device that communicates video and sound with other information processing devices,
An input unit for receiving sound source sound input;
An imaging unit that captures an image of the sound source;
A receiver for receiving video and sound from the other information processing apparatus;
When sound is input from the input unit, a video region including a sound source that emits the sound is cut out from the video captured by the imaging unit to be a first video, and the other information processing apparatus is configured by the reception unit. When a sound is received from, a cutout unit that cuts out a video area in a range wider than at least the first video from the video taken by the imaging unit,
A transmission unit that transmits at least one of the first video and the second video to the other information processing apparatus;
An information processing apparatus comprising: A designation unit for generating designation information for designating which display area in each of the other information processing apparatuses displays the first video and the second video cut out by the cutout unit;
The information processing apparatus according to claim 1, wherein the transmission unit also transmits the designation information when transmitting at least one of the first video and the second video. A first specifying unit that specifies a direction of sound input from the input unit;
When the sound is received from the other information processing device by the receiving unit, the cutting unit receives the sound from the other information processing device specified by the first specifying unit, by the receiving unit. A video region including the first sound source corresponding to the direction of the sound input by the input unit immediately before
The designation unit displays the video region including the first sound source cut out by the cut-out unit in a display region other than the display region where the second video is displayed in the other information processing apparatus. Generate specification information to specify,
The information processing apparatus according to claim 2, wherein the transmission unit transmits a video region including the first sound source to the other information processing apparatus together with the designation information. A recognition unit for recognizing a sound source from the video imaged by the imaging unit;
A second specifying unit that specifies an arrangement pattern indicating in which direction the sound source recognized by the recognition unit is arranged with respect to the information processing device in the video imaged by the imaging unit;
When the sound is received from the other information processing apparatus by the receiving unit, the clipping unit extracts a video region including all sound sources whose arrangement direction is specified by the arrangement pattern from the video. The information processing apparatus according to claim 1, which is cut out as an image. When the sound is input from the input unit,
A video region including a sound source in the direction of the sound input to the input unit specified by the first specifying unit is cut out from the video,
A video region including a second sound source that emits sound before the sound source is cut out from the video,
The designation unit generates designation information that designates that the video region including the second sound source is continuously displayed and the video region including the sound source in the direction of the sound input to the input unit is displayed. ,
The transmission unit transmits the video area including the second sound source, the video area including the sound source in the direction of the sound input to the input unit, and the designation information to the other information processing apparatus. The information processing apparatus described. The number of sound sources recognized by the recognition unit is obtained. When the number of sound sources is 1, the information processing apparatus is switched to the first operation mode. When the number of sound sources is 2, the information processing apparatus is When the mode is switched to the operation mode and the number of sound sources is 3 or more, a switching unit for switching to the third mode of operation is further provided,
In the first operation mode,
The cutout unit cuts out a video area including one sound source from the video,
The transmission unit transmits the video area to the other information processing apparatus;
In the second operation mode,
The cutout unit cuts out a video area including two sound sources from the video,
The information processing apparatus according to claim 4, wherein the transmission unit transmits a video area including each of the two sound sources cut out by the cutout unit to the other information processing apparatus. Every predetermined time
The recognizing unit recognizes a sound source from the video imaged by the imaging unit;
The switching unit obtains the number of sound sources recognized by the recognition unit, and based on the number of sound sources, the first operation mode, the second operation mode, or other than the first operation mode and the second operation mode The information processing apparatus according to claim 6, wherein the information processing apparatus is switched to an operation mode.   The display control part which displays the 1st picture or the 2nd picture on a display part when the 1st picture or the 2nd picture is received from the other information processor by the receiving part. The information processing apparatus according to any one of 1 to 7. Information processing apparatus according to any one of claims 1 to 8,
A server device that performs communication control between the information processing device and the other information processing device;
Conference system. An information processing method in an information processing apparatus that communicates video and sound with another information processing apparatus,
An input step for receiving sound input from the sound source;
An imaging step of imaging the image of the sound source;
A receiving step of receiving video and sound from the other information processing apparatus;
When sound is input, a video area including a sound source that has emitted the sound is cut out from the captured video as the first video, and when sound is received from the other information processing apparatus, from the captured video A step of cutting out a video area in a wider range than at least the first video to form a second video;
A transmission step of transmitting at least one of the first video and the second video to the other information processing apparatus;
An information processing method comprising: To computers that communicate video and audio with other information processing devices,
A receiving step of receiving video and sound from the other information processing apparatus;
When sound is input from an input unit that receives input of sound from a sound source, a video region including the sound source that has emitted the sound is cut out from the image captured by the image capturing unit that captures the image of the sound source. And, in the case of receiving sound from the other information processing apparatus, a step of cutting out a video area in a range wider than at least the first video from the video captured by the imaging unit to be a second video When,
A transmission step of transmitting at least one of the first video and the second video to the other information processing apparatus;
A program for running

Images