JP4957221B2 - Communication device - Google Patents

Description

  The present invention relates to a technique for transmitting and receiving video data.

2. Description of the Related Art In recent years, there are conference systems that can conduct conferences at remote locations by communicating video data and audio data via a network. In general, video data has a larger amount of information than audio data, and the amount of information in video data has further increased due to the improvement in image quality associated with the recent improvement in performance of video cameras. The load on the network is likely to increase during communication. In this situation, if communication is attempted with video and audio synchronized, the audio may be interrupted due to an increase in the network load due to the large amount of information in the video data, which is a problem when conducting a conference. It was. Therefore, in order to reduce the transmission amount of the image data, the network load is reduced by transmitting the video data only when there is a change in the video using the technology disclosed in Patent Document 1. Is also considered.
JP-A-11-225326

  However, as in Japanese Patent Application Laid-Open No. H10-260260, the determination as to whether or not there is a change in video is very heavy on the CPU (Central Processing Unit) that is a control unit of the conference system, and data processing takes time. . Therefore, when this technology is applied to a conference system, the transmission of video data will be delayed from the transmission of audio data, and if video changes frequently, a large amount of video data will be transmitted. Sometimes broke down. Furthermore, it is necessary to increase the performance of the CPU, which is very expensive.

  The present invention has been made in view of the above-described circumstances, and an object of the present invention is to provide a transmission device, a reception device, and a communication device in which sound is hardly interrupted even when image data is transmitted and received.

In order to solve the above-described problems, the present invention is a communication device that connects to a communication network and communicates with another communication device, and generates a first area image within the shooting range as first image data. Imaging means for performing, area setting means for setting a plurality of partial areas of the first area as second areas, and area generation for generating one or more third areas by combining the plurality of second areas Means, sound collection means for collecting sound from the sound source to generate sound data, sound source direction specifying means for specifying the direction of the sound source based on the sound data generated by the sound collection means, When communicating with the other communication device via the communication network, and a selection means for selecting a second area including a position corresponding to the direction of the specified sound source from the set second area. Transmission with a certain probability or more Usable bandwidth measuring means for sequentially measuring effective transmission rates every predetermined time; a first area based on the transmission rate; a second area selected by the selecting means; and the selecting means Selecting means for selecting from any one of the third areas including the second area selected by the step of generating the second video data by cutting out the video of the second area from the first video data , A video data cutout unit for cutting out the video of the third area from the first video data to generate third video data, information indicating the area selected by the selection unit, and a video corresponding to the area There is provided a communication device comprising a transmission means for transmitting data to the other communication device.

  In another preferred embodiment, the sound collecting means has a plurality of microphones, and the sound source direction specifying means adds each sound data generated when the plurality of microphones pick up sounds from the sound sources. Based on this, the direction of the sound source may be specified.

  In another preferable aspect, the video cutout unit may generate third video data compressed so as to reduce a data amount based on the transmission rate.

  In another preferable aspect, the video cutout unit may generate second video data compressed so as to reduce a data amount based on the transmission rate.

  In another preferred embodiment, receiving means for receiving information indicating an area and video data corresponding to the area from the other communication device, and receiving means receives the information indicating the area and the video data. Sometimes, the video obtained by reproducing the received video data is overwritten and displayed on the range corresponding to the received area of the video obtained by reproducing the video data immediately before. And a video data generating means for generating the fourth video data.

  According to the present invention, it is possible to provide a communication device in which sound is hardly interrupted even when image data is transmitted and received.

  Hereinafter, an embodiment of the present invention will be described.

<Embodiment>
FIG. 1 is a block diagram showing a configuration of a communication system 1 including a communication apparatus according to this embodiment of the present invention. The communication system 1 includes a communication device 100a, a communication device 100b, and a communication network 10. The communication device 100a and the communication device 100b are connected to the communication network 10 by wire or wirelessly. The communication device 100a and the communication device 100b have the same configuration. Hereinafter, when there is no need to distinguish between the communication device 100a and the communication device 100b, both are referred to as the communication device 100. Although two communication devices 100 are connected to the communication network 10 here, three or more communication devices 100 may be connected.

  In this embodiment, each communication protocol described below is used as a communication protocol. As a communication protocol for the application layer, RTP (Real-time Transport Protocol) is used for transferring audio data and video data. RTP is a communication protocol for providing a communication service for transmitting and receiving audio data and video data in end-to-end in real time, and details thereof are defined in RFC1889. In RTP, data is exchanged between communication devices by generating and transmitting / receiving RTP packets. Further, UDP (User Datagram Protocol) is used as a transport layer communication protocol, and IP (Internet Protocol) is used as a network layer communication protocol. The communication device 100a and the communication device 100b are assigned IP addresses, respectively, and are uniquely identified on the network. In addition, about UDP and IP, since it is a communication protocol generally used widely, description is abbreviate | omitted.

  Next, the configuration of the communication apparatus 100 will be described. FIG. 2 is a block diagram illustrating a configuration of the communication device 100. In the following description, when it is necessary to distinguish whether the configuration of the communication device 100 belongs to either the communication device 100a or the communication device 100b, the CPU 101 of the communication device 100a is given an alphabet like the CPU 101a. And describe.

  The CPU 101 reads out a program stored in a ROM (Read Only Memory) 102, loads it into a RAM (Random Access Memory) 103, and executes it to control each unit of the communication device 100 via the bus 110. . The RAM 103 further stores audio data output from the audio input unit 104, video data output from the video input unit 105, audio data received via the communication network 10, video data, and the like. Further, the CPU 101 functions as a work area when processing the stored data.

  The audio input unit 104 includes a microphone array and an A / D converter. The A / D converter performs analog-to-digital conversion on an analog audio signal input from the microphone array to generate digital signal audio data. The microphone array has a plurality of microphones arranged in the horizontal direction. The CPU 101 collects sound emitted from a certain sound source with a plurality of microphones, analyzes each generated sound data, calculates a difference in arrival time of the sound to each microphone, and communicates the position of the sound source. It can be specified as an angle in the horizontal direction when viewed from the device 100.

  For example, as shown in FIG. 3, when sound emitted from a specific sound source S is collected by four microphones 1041, 1042, 1043, and 1044, the distance from the sound source S to each of the microphones 1041, 1042, 1043, and 1044 Due to the difference, the arrival time of the sound will shift. For example, since the distance from the sound source S to the microphone 1041 is longer than the distance from the microphone 1044 by d, the microphone 1041 collects the sound emitted from the sound source S slower than the microphone 1044 by d / v (v: sound speed). . That is, the CPU 101 can calculate the difference in distance from the sound source S to each of the microphones 1041, 1042, 1043, and 1044 by analyzing the audio data output from the audio input unit 104 and calculating the time lag. Thus, the angle θ in the horizontal direction can be specified for the position of the sound source S. Here, θ is defined with a direction M (in the present embodiment, the front direction of the communication device 100) perpendicular to a plurality of microphones arranged in the horizontal direction as 0 degrees. It is possible to calculate not only the direction of the sound source S but also the distance, but in this embodiment, only the direction is specified. In addition, even if sound is emitted from a plurality of sound sources, the direction of each sound source can be specified, but in the present embodiment, the direction with the highest volume is specified as the direction of the sound source.

  The video input unit 105 includes an image sensor such as a CCD or a CMOS, and shoots with a predetermined image size (number of pixels) and the number of frames per unit time by the image sensor to generate video data. In the present embodiment, the image sensor of the video input unit 105 and the microphone of the audio input unit 104 are fixed to the communication device 100. By fixing in this way, the positional relationship between the sound source existing in the imaging range of the image sensor and the direction of the sound source specified by the CPU 101 is maintained. For example, as shown in FIG. 4, the angle θ in the horizontal direction is 0 degree in the center direction M of the imaging range of the image sensor, and the right side in the figure is a positive value and the left side is a negative value. Here, FIG. 4 is a cross-sectional view of the imaging range of the image sensor when cut by a plane including a plurality of microphones arranged horizontally, and the right side in the figure displays an image on the display unit 107 shown below. Sometimes it appears on the right side of the screen.

  The operation unit 106 is, for example, a keyboard or a mouse. When an operator of the communication apparatus 100 operates the operation unit 106, data representing the operation content is output to the CPU 101.

  The display unit 107 is a display device such as a liquid crystal display that displays an image on a screen, and performs display based on input image data.

  The audio output unit 108 emits input audio data, and includes a speaker and a D / A converter. The D / A converter performs digital / analog conversion on the audio data of the input digital signal to generate an audio signal of the analog signal and outputs it to the speaker. The speaker emits the input audio signal.

  The communication IF (interface) 109 is, for example, a NIC (Network Interface Card) and is connected to the communication network 10. The communication IF 109 generates RTP packets for various data such as audio data and video data to be transmitted, and transmits IP packets obtained by sequentially encapsulating according to a lower layer communication protocol to the communication network 10. Encapsulation means generating a UDP packet in which the RTP packet is written in the payload part, and further generating an IP packet in which the UDP packet is written in the payload part. Further, the communication IF 109 receives an IP packet from the communication network 10 and performs a process reverse to the above encapsulation, thereby outputting data obtained by reading the RTP packet encapsulated in the IP packet to the CPU 101.

  Here, RTP packetization of audio data and video data will be described. The RTP packet is configured by adding a header portion to the payload portion, similarly to a packet that is a data transfer unit in IP.

  In the header part, four types of data including time stamp, payload type, sequence number, and area information are set. Here, the time stamp is data indicating the time (the time elapsed since the start of communication was instructed). The payload type is data for identifying the type of communication message to the destination of the communication message. There are four types of messages used in this embodiment: a video data transmission message, an audio data transmission message, a test data transmission message, and a reception notification message. For these messages, four types of numbers “1”, “2”, “3”, and “4” are written in the payload type, respectively. The sequence number is an identifier for uniquely identifying each packet. For example, when one voice data is divided into a plurality of RTP packets and transmitted, 1, 2, 3,.・ Sequence numbers are given as shown below. The area information is information that defines which area the video data included in the RTP packet corresponds to in the video data transmission message, and information on coordinates indicating the area is written. The area includes a personal area, an active area, and an all area, details of which will be described later.

  For example, in the video data transmission message and the audio data transmission message, video data and audio data for a predetermined time are written in the payload portion, respectively. Further, test data is written in the test data transmission message, and in the reception notification message, the sequence number of the received packet is written. The test data transmission message and the reception notification message will be described later.

  Next, an operation realized when the CPU 101 of the communication apparatus 100 executes a program stored in the ROM 102 when performing a remote conference using the communication system 1 will be described.

  Here, in the present embodiment, a remote conference is performed between the room a and the room b, the communication apparatus 100a is installed in the room a, and five participants 201, 202,. It is assumed that the communication device 100b is installed in the room b, and three participants 301, 302, and 303 are participating in the conference. Each communication device 100 is installed so as to photograph a participant sitting near the desks 200 and 300 by the image sensor of the video input unit 105. Here, FIG. 5 is a diagram illustrating a shooting target of the video input unit 105a of the communication device 100a. When video data in this shooting range is transmitted to the communication device 100b, the display unit 107b of the communication device 100b. Will be displayed on the screen. FIG. 6 is a diagram illustrating a shooting target of the video input unit 105b of the communication device 100b. When video data in this shooting range is transmitted to the communication device 100a, it is displayed on the display unit 107a of the communication device 100a. It becomes a picture.

  First, the personal area is set before the start of the remote conference. Here, the setting of the personal area will be described with reference to FIGS. The participant in the room a operates the operation unit 106a while confirming the video on the display unit 107a, and designates the position of the participant in the room b. At this time, the communication device 100a may transmit the image data generated by shooting to the communication device 100b without processing, or transmit the image data close to a still image by reducing the number of frames per unit time. May be. The designation of the position is performed so as to surround the participants 301, 302, and 303 displayed on the display unit 107a with a square as shown by the broken line in FIG. The CPU 101a recognizes each square area as each personal area. At this time, the CPU 101a recognizes the personal area from the coordinates (pixel unit) of the upper left and lower right points of the square. This is when the position of the screen is represented by the number of pixels, where the upper left corner of the screen is (0, 0), and x pixels to the right and y pixels down from that point are (x, y). In addition, one personal area is recognized as upper left coordinates (xL, yL) and lower right coordinates (xR, yR). Hereinafter, the right direction of the screen is referred to as an x direction, the coordinates thereof are referred to as x coordinates, the downward direction is referred to as a y direction, and the coordinates are referred to as y coordinates.

  Then, the CPU 101a of the communication device 100a transmits information about the three personal areas recognized to the communication device 100b, and causes the CPU 101b to recognize the position of the personal area in the shooting range of the video input unit 105b of the communication device 100b. As for the room b, the operation of the communication device 100b is performed similarly to the room a, and the CPU 101a is made to recognize the position of the personal area in the shooting range of the video input unit 105a of the communication device 100a. Hereinafter, the personal area corresponding to the participant 201 is denoted as PSA 201, the personal area corresponding to the participant 202 is denoted as PSA 202, the personal area corresponding to the other participants is denoted in the same manner, and the upper left coordinate of the PSA 201 is ( xL201, yL201), and the lower right coordinates are (xR201, yL201). When there is no need to distinguish between the participants, it is simply referred to as a personal area, the upper left coordinates are (xL, yL), and the lower right coordinates are (xR, yR).

  Next, the CPU 101 of each communication device 100 generates and recognizes an active area as described below by combining a plurality of recognized personal areas, and recognizes the entire photographing range as an all area. The active area is generated by a square area including a plurality of personal areas and having a minimum size. That is, the upper left coordinate of the active area is determined by the minimum xL (xLmin) and the minimum yL (yLmin) among the upper left coordinates of the plurality of personal areas, and the lower right coordinate is the lower right coordinate of the personal area. Are determined by the maximum xR (xRmax) and yR (yRmax), the upper left coordinates of the active area are (xLmin, yLmin), and the lower right coordinates are (xRmax, yRmax).

  Here, an active area including all personal areas on the screen is referred to as an all active area, and corresponds to a portion represented by a two-dot chain line in FIGS. In the case of FIG. 5, the upper left coordinates (xLmin, yLmin) are (xL201, yL203), and the lower right coordinates (xRmax, yRmax) are (xR205, yR201). The active area can be generated by a combination of various personal areas including two adjacent personal areas. However, in the present embodiment, only the all active area is generated by the CPU 101 in the present embodiment. In the following, all active areas are also referred to as active areas.

  Since the entire area is the entire photographing range, the upper left coordinates are (0, 0) and the lower right coordinates are (xmax, ymax). Here, xmax is a number obtained by subtracting 1 from the number of pixels in the x direction of the entire shooting range, ymax is a number obtained by subtracting 1 from the number of pixels in the y direction of the entire shooting range, and the number of pixels in the shooting range is 640 × 480. In this case, xmax = 639 and ymax = 479. As a result, each CPU 101 of each communication device 100 recognizes the all area, the active area, and each personal area of the room in which each communication device 100 is installed as coordinates. If there are not a plurality of personal areas, the active area may be treated as the same as the personal area.

  When the setting of the personal area is completed, communication between video data and audio data is started. The voice data is transmitted as a voice data transmission message after being converted into an IP packet by the communication IF 109. The video data is transmitted as a video data transmission message in the form of IP packets by the communication IF 109, but the transmission method differs depending on the available bandwidth of the network. Hereinafter, transmission of video data will be described using the flowchart of FIG. 7 as an example of a case where the communication device 100a transmits to the communication device 100b. Note that the transmission from the communication device 100b to the communication device 100a is performed in the same manner as described below, and thus the description thereof is omitted.

  When transmission of video data is started, first, the CPU 101a of the communication device 100a transmits all-area video data to the communication device 100b via the communication IF 109a (S100). Since the video data need only be transmitted as a still image, only one frame may be transmitted. Next, the communication device 100a performs an available bandwidth measurement (S110).

  Here, the available bandwidth measurement will be described. The available bandwidth measurement is to measure the maximum communication bandwidth that can be used in the communication network 10 when the communication device 100a communicates with the communication device 100b via the communication network 10. Say. First, the CPU 101a determines a transmission interval when transmitting a packet. Next, the CPU 101a generates test data equivalent to the video data. The communication IF 109a encapsulates a plurality of test data transmission messages in which the test data is written in the payload part of the RTP packet, thereby encapsulating a plurality of test data transmission messages. Each IP packet is generated as an IP packet and transmitted to the communication device 100b at the determined transmission interval. At this time, the CPU 101a stores the sequence number of each transmitted RTP packet in the RAM 103a.

  When the CPU 101b of the communication device 100b recognizes the reception of the test data transmission message, a reception notification message in which the sequence number of each packet received by the communication IF 109b is written is generated as an IP packet and transmitted to the communication device 100a. The CPU 101a of the communication device 100a calculates a packet loss occurrence rate (the number of packets not received / the number of transmitted packets) in the transmission of test data from the received reception notification message and the transmitted test data transmission message.

  If the packet loss is less than the predetermined occurrence rate, the packet loss occurrence rate is recalculated by shortening the transmission interval and performing the same processing. On the other hand, if it is equal to or higher than the predetermined occurrence rate, the packet loss occurrence rate is calculated again by increasing the transmission interval and performing the same processing. Here, when the transmission interval is shortened, the transmission data amount per unit time, that is, the transmission rate is increased. On the other hand, when the transmission interval is increased, the transmission rate is decreased. When the transmission rate is smaller than the available bandwidth, the rate of occurrence of packet loss is small, and the rate of occurrence of packet loss increases as the transmission rate increases with respect to the available bandwidth. Therefore, if a lot of packet loss occurs, it means that the communication bandwidth is greatly exceeded. By performing these processes and determining the shortest transmission interval at which the packet loss is less than the predetermined occurrence rate, the transmission rate of the available bandwidth can be calculated.

  Hereinafter, the description will be continued by returning to the flowchart of FIG. The CPU 101a measures the available bandwidth and calculates the maximum value of the transmission rate that can be used in the communication network 10 (S110).

  Next, it is determined which area of video to transmit as video data to be transmitted. First, the CPU 101a determines whether the video data corresponding to the all area has a data amount that can be transmitted at the calculated transmission rate (S120). When the amount of data can be transmitted, the coordinates corresponding to the all area are written in the area information in the header part of the RTP packet, and the video data transmission message in which the video data of the all area is written in the payload part is communicated. The data is transmitted to the communication device 100b via the IF 109a (S121). The transmission of the video data is continued for a certain time.

  On the other hand, when the data amount cannot be transmitted, the CPU 101a determines whether the data amount can be transmitted at the calculated transmission rate when only the video data corresponding to the active area is cut out (S130). If transmission is possible, as described above, the coordinates corresponding to the active area are written in the area information in the header part of the RTP packet, and the video data corresponding to the active area cut out by the CPU 101a is written as the video data transmission message in the payload part. The data is transmitted to the communication device 100b via the communication IF 109a (S131). The transmission of the video data is continued for a certain time.

  Further, if the video data corresponding to the active area has a data amount that cannot be transmitted, the CPU 101a tries to transmit video data corresponding to the personal area with a smaller data amount, but there are a plurality of personal areas. In order to do this, first, a personal area is selected (S140). This is performed by selecting a personal area corresponding to a participant who is speaking (hereinafter referred to as a speaker) among the participants in the room a. As described above, the microphone array of the voice input unit 104a picks up the voice of the speaker, and the CPU 101a analyzes the voice data to specify the direction of the speaker (the angle θ in the horizontal direction). The personal area corresponding to the speaker is selected in correspondence with the coordinates of the personal area.

  Here, the correspondence between the horizontal angle θ and the coordinates of the personal area is performed as follows. Since the angle θ in the horizontal direction indicates only the horizontal direction, the angle θ can be converted into an x coordinate that is a coordinate in the horizontal direction. This conversion will be described with reference to FIG. FIG. 8 shows the photographing range viewed from the y direction. α is a light receiving surface of the image sensor of the video input unit 105, and a focal point F indicates an intersection when the end of the imaging range and the end of the light receiving surface α are respectively connected. The leftmost point O of the light receiving surface α is the origin of the x coordinate, and the x coordinate of the rightmost point is xmax. The horizontal angle θ of the speaker specified using the microphone array is substantially the same as the angle from the perpendicular FM from the focal point F to the light receiving surface α in the present embodiment. Here, the straight line from the focal point F to the direction of the speaker intersects with the point P on the light receiving surface α, and the coordinate xs can be calculated as xm + fa × tan θ. xm is a coordinate of the center M of the light receiving surface (xm = xmax / 2), and fa is a distance between the center M and the focal point F. The CPU 101a compares the coordinate xs thus obtained with the coordinates xL and xR of each personal area, and selects a personal area where xs is between xL and xR. For example, if xs is not less than xL204 and not more than xR204, the PSA 204 is selected.

  Returning to FIG. 7, the description will be continued. After selecting the personal area as described above, the CPU 101a determines whether the data amount can be transmitted at the calculated transmission rate when only the video data corresponding to the selected personal area is cut out (S150). When the amount of data that can be transmitted is the same as above, the coordinates corresponding to the personal area are written in the area information in the header part of the RTP packet, and the video data corresponding to the personal area cut out by the CPU 101a is written in the payload part. The video data transmission message is transmitted to the communication device 100b via the communication IF 109a (S151). The transmission of the video data is continued for a certain time.

  Further, if the amount of video data corresponding to the personal area cannot be transmitted, the CPU 101a reduces the amount of data by compressing the video data in the personal area to the amount of data that can be transmitted at the calculated transmission rate. The coordinates corresponding to the personal area are written in the area information in the header part of the RTP packet, and the compressed video data corresponding to the personal area is written in the payload part as a video data transmission message via the communication IF 109a. It transmits to the communication device 100b (S160). The transmission of the video data is continued for a certain time. Here, compression of video data is performed by reducing the number of frames, thinning out the number of pixels, reducing the number of colors, and the number of gradations. When such video data is transmitted and displayed on the display unit 107b of the communication device 100b as will be described later, it also changes greatly in appearance, and the available bandwidth is reduced. Participants can also judge visually.

  With these processes, it is possible to transmit video data having a data amount that can be transmitted at an appropriate transmission rate obtained by measuring the available bandwidth, but when the data amount of the video data to be transmitted changes, the RTP packet is generated. The amount of video data written in the payload portion may be changed so that the packet transmission interval is the same, or the amount of video data written in the payload portion may be the same and the transmission interval may be changed. Then, the amount of video data written in the payload portion may be changed, and the transmission interval may be changed. For these selections, the CPU 101a may select an optimum method when generating the RTP packet.

  Then, after the video data is transmitted for a certain time, the CPU 101a determines whether or not the communication is finished (S170). If the communication is not finished, the CPU 101a performs the processing from the available bandwidth measurement (S110) again. If the communication is terminated, transmission of video data is stopped. Whether or not the communication is terminated is determined by whether or not a participant in any room has operated the operation unit 106 of the communication apparatus 100 to give an instruction to terminate the communication.

  Although the video data transmission method has been described above, a description is given from the time when the communication device 100b receives the video data transmitted from the communication device 100a until it is displayed on the display unit 107b.

  First, when communication is started, the communication device 100b receives all-area video data, and the CPU 101b displays the video on the display unit 107b. Thereafter, the communication device 100b sequentially receives video data. Here, the CPU 101b recognizes which area on the screen corresponds to the video data by reading the upper left and lower right coordinates written in the area information in the header part of the RTP packet.

  Then, the CPU 101b stores the previous video data (corresponding to the image of the last frame) in the RAM 103b, and stores it in the RAM 103b so as to be positioned in the area on the screen where the video data of the transmitted data is recognized. New video data is generated by combining with the previous video data, the generated video data is output to the display unit 107b, and the video is displayed on the display unit 107b. This process is sequentially performed every time an IP packet is received from the communication device 100a.

  If it does in this way, after stopping the image | video currently displayed until just before that, it will be displayed so that an image | video may be overwritten only to the part applicable to the recognized coordinate. For example, when the video data corresponding to the PSA 202 in the personal area is received while displaying the video in the all area, the video in the all area is stopped (the still image of the frame being played back at that time) Then, the video of the portion corresponding to the PSA 202 is reproduced by overwriting the video of the stopped all area. That is, the portions other than the PSA 202 are like still images while the video is stopped. In this manner, the video of the portion corresponding to the transmitted area information is displayed on the display unit 107b of the communication device 100b while sequentially overwriting the previous video.

  The case where video data is transmitted from the communication device 100a to the communication device 100b has been described above, but these processes are also performed from the communication device 100b to the communication device 100a. Will be done in both directions. In addition, audio data generated by sound collection by the audio input unit 104 is also bidirectionally communicated in parallel with the video data.

  In this way, the communication apparatus 100 measures the bandwidth that can be used in the communication network 10, and communicates with the communication apparatus 100a for transmission performed by selecting an area that is video data having a data amount that matches the calculated transmission rate. The apparatus 100b communicates bidirectionally and displays the received video data so as to overwrite the previous video, thereby reducing packet loss and reducing audio loss in parallel audio data communication. It is possible to reduce the number of remote conferences.

  As mentioned above, although embodiment of this invention was described, this invention can be implemented in various aspects as follows.

<Modification 1>
In the embodiment, when the voice input unit 104 picks up sound emitted from a plurality of speakers, the CPU 101a calculates the direction of the maximum volume as the speaker direction. The direction may be calculated. In this case, the CPU 101a selects a plurality of personal areas. However, the CPU 101a may transmit video data obtained by extracting portions of the plurality of personal areas. Instead of transmitting video data related to a plurality of personal areas, an active area is generated from the plurality of personal areas by the method described above, and video data extracted from the active area is transmitted. Also good. In this way, even if there are a plurality of speakers, it is possible to respond, and it is possible to transmit video data in accordance with the transmission rate calculated by the CPU 101a.

<Modification 2>
In the embodiment, the personal area set by the participant is set so that the x-coordinate ranges do not overlap, but may be set so as to overlap. For example, as shown in FIG. 9, when PSA 202 and PSA 203 overlap, that is, when xL 203 becomes smaller than xR 202, CPU 101 a determines the boundary between PSA 202 and PSA 203 as the x coordinate (xR202 + xL203) / 2 of the center of the overlapping portion. Should be recognized. If it is calculated that there is a speaker in this overlapping portion without dividing the central portion, the CPU 101a generates an active area from the PSA 202 and PSA 203 as shown by a two-dot chain line in the figure, and the active area The video data may be transmitted. In this way, when there are many participants, the same effects as in the embodiment can be obtained even if it is difficult to set a personal area.

<Modification 3>
In the embodiment, the microphone 101 is used for the voice input unit 104, and the CPU 101 calculates the direction of the speaker using this. However, instead of the microphone array, a microphone is prepared in front of each participant, The CPU 101 may specify the speaker from the volume of the voice data generated by collecting the microphones. In this case, the participant only has to make the CPU 101 recognize the correspondence between each microphone and the personal area by operating the operation unit 106. In this way, a more accurate speaker position can be specified.

<Modification 4>
In the embodiment, the coordinate information of the video data to be transmitted is written in the area information in the header part of the RTP packet, but instead of the coordinates, a numerical value obtained by replacing each area with a number is written. Also good. In this case, in each communication device 100, when transmitting / receiving the coordinate information of the personal area, it is only necessary to transmit / receive by adding an area number corresponding to each personal area. In this way, the data amount of the RTP packet can be reduced, and the load on the communication network 10 can be reduced.

<Modification 5>
In the embodiment, the active area is only the all active area including all personal areas. However, if there is a bias in the personal area, the active area may be generated from a group of personal areas corresponding to the bias. . For example, in the case of the arrangement of participants as shown in FIG. 5, the left and right sides are set as a group of personal areas, the active area L is generated from the PSA 201, PSA 202, and PSA 203, and the active area R is generated from the PSA 204 and PSA 205. When the video data is transmitted, if the personal area corresponding to the direction of the speaker is selected as the PSA 202, the CPU 101a is an active area including the PSA 202 before transmitting the video data related to the PSA 202. Judgment is made from the calculated transmission rate whether or not the data amount of the video data related to the active area L can be transmitted. If transmission is possible, the video data related to the active area L is transmitted instead of transmitting the video data related to the PSA 202. What should I do? In this way, video data related to an area having an intermediate data amount can be transmitted when the video data related to the all active area has a large amount of data and the video data related to the personal area is too small. The display quality on the receiving side can be kept high.

<Modification 6>
In the embodiment, the setting of the personal area related to the participant in the room a is performed by the participant in the room b. However, the participant in the room a may set the personal area. In this case, when the personal area is set, the video data photographed by the video input unit 105a of the communication device 100a is displayed on the display unit 107a of the communication device 100a, so that the participant in the room a can make the setting while checking his / her video. can do. Here, if the video is reversed left and right so that the setting can be easily performed so that the video displayed on the display unit 107a looks like a mirror, the participant can easily perform the setting. Can do.

<Modification 7>
In the embodiment, the available bandwidth is measured, and the area that is the video data having the optimum data amount is selected and transmitted. However, the video data related to the all area is transmitted after the first transmission. , It may not be transmitted. Further, video data related to the all area may be transmitted every predetermined time, or the CPU 101 determines the similarity of the video every predetermined time, and there is a fluctuation more than a predetermined amount. You may make it transmit in a case. Here, since the video data relating to the all area may be one frame (may be still image data), the load on the communication network 10 hardly increases. Here, when the calculated transmission rate is very low and the audio data is interrupted when the video data related to the all area is transmitted, the CPU 101 then sets the transmission rate at which the video data related to the all area can be transmitted first. The video data is transmitted after waiting until

<Modification 8>
In the embodiment, the image sensor of the video input unit 105 and the microphone array of the audio input unit 104 are fixed to the communication device 100, but may be separately movable. In this case, the participant may operate the operation unit 106 to make the CPU 101 recognize the correspondence between the imaging range of the image sensor and the direction of the microphone array. If it does in this way, installation of communication apparatus 100 can be performed in various modes.

<Modification 9>
In the embodiment, depending on the calculated transmission rate, the video data related to the personal area is transmitted after being compressed to reduce the amount of data. However, the calculated transmission rate is not the video data related to the personal area. If the video data is high enough to be sent and is a little low to send video data related to the active area, the video data related to the active area may be compressed and transmitted. In this way, video data in the range of the active area can be transmitted by degrading the video to such an extent that the participant does not care.

<Modification 10>
In the embodiment, the entire shooting range is recognized as an all area. However, the all area indicates an area for transmitting video data that has not been cut out to another communication device, and thus is not necessarily the entire shooting range. May be.

<Modification 11>
In the embodiment, the available bandwidth measurement is performed at regular time intervals, but may be performed at random time intervals, or may be changed according to the status of the calculated transmission rate. May be performed every time. For example, when the state in which the transmission rate varies little continues, the CPU 101 may control to increase the interval for measuring the available bandwidth. In this way, the number of available bandwidth measurements can be reduced when a stable transmission rate continues.

<Modification 12>
In the embodiment, the video data corresponding to the selected area is cut out and transmitted to the other communication device 100. However, not only the selected area but also the video data corresponding to all the areas is generated in advance. In addition, video data corresponding to the selected area may be transmitted to another communication apparatus 100. In this way, even if the CPU 101 has a low processing capability, when the selected area is changed, it is not necessary to perform image processing at that time, and transmission can be performed simply by selecting video data.

It is a block diagram which shows the structure of a communication system. It is a block diagram which shows the structure of the communication apparatus which concerns on embodiment. It is explanatory drawing regarding calculation of the distance and direction of a microphone array and a sound source. It is explanatory drawing which shows the relationship between the imaging | photography range of an image sensor, and the direction of a sound source. It is explanatory drawing which shows the screen display in the room b. It is explanatory drawing which shows the screen display in the room a. It is a flowchart explaining the transmission process of video data. It is explanatory drawing regarding the calculation at the time of expressing a speaker's direction by x coordinate. It is explanatory drawing regarding the personal area and active area which concern on the modification 2.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Communication system, 10 ... Communication network, 100 ... Communication apparatus, 101 ... CPU, 102 ... ROM, 103 ... RAM, 104 ... Audio | voice input part, 1041, ... 1044 ... Microphone, 105 ... Image | video input part, 106 ... Operation unit 107 ... Display unit 108 ... Audio output unit 109 ... Communication IF 110 ... Bus 201 ... 205, 301, 302, 303 ... Participant 200, 300 ... Desk

Claims (5)

A communication device connected to a communication network and communicating with another communication device,
Shooting means for generating a video of a first area within the shooting range as first video data;
Area setting means for setting a plurality of ranges of a part of the first area as second areas;
Area generating means for generating one or more third areas by combining the plurality of second areas;
Sound collection means for collecting sound from a sound source and generating sound data;
Sound source direction specifying means for specifying the direction of the sound source based on the sound data generated by the sound collecting means;
Selecting means for selecting a second area including a position corresponding to the direction of the identified sound source from a plurality of set second areas;
Usable bandwidth measuring means for sequentially measuring a transmission rate at which transmission succeeds with a predetermined probability or higher when communicating with the other communication device via the communication network, every predetermined time;
Based on the transmission rate, one of the first area, the second area selected by the selecting means, and the third area including the second area selected by the selecting means is selected. A selection means;
The video of the second area is cut out from the first video data to generate second video data, and the video of the third area is cut out from the first video data to generate third video data Video data cutting means for
A communication apparatus comprising: a transmission unit configured to transmit information indicating an area selected by the selection unit and video data corresponding to the area to the other communication device. The sound collection means has a plurality of microphones,
The sound source direction specifying means, said plurality of microphones based on each audio data generated by picking up sound from a sound source, according to claim 1, characterized in that to identify the direction of the sound source Communication device. The communication apparatus according to claim 1 , wherein the video cutout unit generates third video data compressed so as to reduce a data amount based on the transmission rate. The image cut-out unit, on the basis of the transmission rate, the communication device according to any one of claims 1 to 3, characterized in that to generate the second video data compressed to reduce the amount of data . Receiving means for receiving information indicating an area and video data corresponding to the area from the other communication device;
When the receiving means receives the information indicating the area and the video data, the received video data is compared with the range corresponding to the received area of the video obtained by reproducing the video data immediately before the information. as image obtained by reproducing is displayed by overwriting, to any of the claims 1 to 4, further comprising a video data generating means for generating a fourth video data The communication device described.

Images