WO2025182639A1 - Information processing device, information processing method, and information processing system - Google Patents

Abstract

Provided are an information processing device, an information processing method, and an information processing system which make it possible to promote a smoother dialogue. A human sensing unit determines, on the basis of images captured by a plurality of cameras of speakers present in a predetermined space, whether each speaker participates in a conversation. A clarity adjustment unit adjusts the clarity of speech voice of the speaker determined to participate in the conversation by the human sensing unit, the speech voice being included in voice in the predetermined space, which voice is collected by a microphone. The transmission unit transmits the voice in the predetermined space processed by the clarity adjustment unit.

Description

Information processing device, information processing method, and information processing system

This disclosure relates to an information processing device, an information processing method, and an information processing system.

Remote communication systems are known that enable remote communication between multiple people in remote areas. In such remote communication systems, people in one remote area communicate with people in another remote area using the microphone, speaker, camera, etc. of the remote communication system.

JP 2010-191544 A

The remote communication system uses a microphone placed in one remote area to pick up the speaker's voice, and then plays the picked-up voice from a speaker placed in another remote area.

However, simply extracting and playing back human voices has the problem that even voices unrelated to the conversation are emphasized, making it difficult to concentrate on the conversation with the other person. Furthermore, completely muting surrounding sounds to emphasize the conversation can make it difficult to grasp the situation in the other person's space, and the atmosphere can become unnatural when there is no conversation.

This disclosure has been made in consideration of the above-mentioned circumstances, and provides an information processing device, information processing method, and information processing system that can promote smooth dialogue.

The information processing device disclosed herein includes a human sensing unit that determines whether each speaker is participating in a conversation based on video captured by a camera of multiple speakers present in a specified space; a clarity adjustment unit that adjusts the clarity of the speech of speakers who are participating in the conversation and that is included in the audio of the specified space picked up by a microphone and that is determined by the human sensing unit to be included; and a transmission unit that transmits the audio of the specified space processed by the clarity adjustment unit.

FIG. 1 is a block diagram of a remote communication device. 1 is a bird's-eye view of an environment in which a remote communication device is placed. FIG. 1 is a front view of an environment in which a remote communication device is placed. FIG. 10 is a diagram for explaining processing by a human sensing unit. FIG. 10 is a diagram illustrating a coordinate transformation from screen coordinates to normalized coordinates. FIG. 10 is a diagram illustrating individual sound separation processing. FIG. 10 is a diagram illustrating a multiple facing speaker balancing process. FIG. 10 is a diagram illustrating processing using a normal distribution function for outfield speech. FIG. 2 is a diagram showing an outline of data flow between the local remote communication devices according to the first embodiment. 10 is a flowchart of a transmission side process. 10 is a flowchart of a person position determination process. 10 is a flowchart of an infield/outfield determination process. 10 is a flowchart of an individual sound separation process. 10 is a flowchart of a process of adjusting clarity of individual sounds. 10 is a flowchart of a multiple opposite speaker balancing process. 10 is a flowchart of a background sound adjustment process. 10 is a flowchart of a process for adjusting infield speech sounds. 10 is a flowchart of a process for adjusting outfield speech sounds. 10 is a flowchart of a process for deriving a localization center speaker unit. FIG. 10 is a block diagram of a remote communication device according to a second embodiment. FIG. 10 is a diagram showing an outline of data flow between the local remote communication devices according to the second embodiment. FIG. 10 is a block diagram of a remote communication device according to a third embodiment. FIG. 13 is a diagram showing an outline of data flow between the local remote communication devices according to the third embodiment. FIG. 13 is a block diagram of a remote communication device according to a fifth embodiment. 13A and 13B are diagrams for explaining audio signal processing by a remote communication device according to a fifth embodiment. FIG. 20 is a diagram for explaining audio reproduction by a remote communication device according to a sixth embodiment. FIG. 20 is a diagram for explaining audio reproduction by a remote communication device according to a seventh embodiment. FIG. 10 is a hardware configuration diagram showing an example of a computer that realizes an arithmetic unit of a remote communication device that is an information processing device according to the first to seventh embodiments.

Below, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configuration will be assigned the same reference numerals, and duplicate explanations will be omitted. The explanation will be given in the following order.

1. Remote communication device according to the first embodiment 1.1. Definition of terms 1.1.1. Remote and local 1.1.2. Individual sound 1.1.3. Infield and outfield 1.1.4. Multiple facing speakers 1.2. Transmission unit 1.2.1. Human sensing unit 1.2.2. Audio separation unit 1.2.3. Acoustic signal processing unit 1.2.4. Output integration unit 1.2.5. Transmission unit 1.3. Receiving unit 1.3.1. Receiving unit 1.3.2. Audio output control unit 2. Data flow between remote communication devices according to the first embodiment 3. Remote communication processing 3.1. Transmission-side processing 3.1.1. Human position determination processing 3.1.2. Infield/outfield determination processing 3.1.3. Individual sound separation processing 3.1.4. Clarity adjustment processing 3.2. Multiple opposing speaker balancing processing (receiving side processing)
3.2.1. Background sound adjustment 3.2.2. Infield speech sound adjustment 3.2.3. Outfield speech sound adjustment 3.2.4. Localization center speaker unit derivation 4. Effect 5. Remote communication device according to the second embodiment 5.1. Acoustic signal processing unit 5.1.1. Background sound separation unit 5.1.2. Clarity adjustment unit 5.2. Output integration unit 5.3. Audio output control unit 6. Data flow between remote communication devices according to the second embodiment 7. Effect 8. Remote communication device according to the third embodiment 8.1. Acoustic signal processing unit 8.1.1. Speech synthesis unit 9. Data flow between remote communication devices according to the third embodiment 10. Effect 11. Remote communication device according to the fourth embodiment 11.1. Human sensing unit 11.2. Acoustic signal processing unit 11.3. Other voice recognition 11.4. Effect 12. 12. Remote communication device according to the fifth embodiment 12.1. Human sensing unit 12.2. Acoustic signal processing unit 12.3. Effect 13. Remote communication device according to the sixth embodiment 13.1. Audio output control unit 13.2. Effect 14. Remote communication device according to the seventh embodiment 14.1. Human sensing unit 14.2. Acoustic signal processing unit 14.3. Audio output control unit 14.4. Effect 15. Hardware configuration

1. Remote communication device according to the first embodiment
In conventional remote communication systems, when multiple people speak at the same time, the sounds can become mixed together during playback, making it difficult to hear, or the voice of the person you want to hear can be suppressed, creating an unnatural experience.In addition, conventional remote communication systems often cut out sounds other than the speech.

Figure 1 is a block diagram of a remote communication device. The remote communication device 1 according to the embodiment is able to process background sounds and spoken voices in separate systems, and further divides spoken voices into two types depending on the person being spoken to. The remote communication device 1 then processes each of the divided voices using different parameters, adjusting the ease with which the voices can be heard while maintaining a sense of connection with the remote location. Here, the sense of connection corresponds to a sense of realism that makes it feel as if you are having a face-to-face conversation with your conversation partner in the same space.

The remote communication device 1 has a transmitting unit 10 and a receiving unit 20. The remote communication device 1 shown in FIG. 1 is a device that performs two-way communication with a remote communication device 1 of a partner, and the remote communication device 1 of the partner also has a transmitting unit 10 and a receiving unit 20.

However, if two-way transmission is not performed, the remote communication device 1 on the transmitting side only needs to have at least a transmission unit 10, and the remote communication device 1 on the receiving side only needs to have at least a receiving unit 20. In other words, the information processing system that realizes remote communication in this embodiment has a remote communication device 1 as a transmitting device, a receiving device, and a remote communication device 1.

The sending remote communication device 1 and the receiving remote communication device 1 are connected via a network 7.

Figure 2A is an overhead view of the environment in which the remote communication device is placed. Figure 2B is a front view of the environment in which the remote communication device is placed. The dotted lines in Figures 2A and 2B indicate the wiring of signal lines.

The remote communication device 1 is connected to an audio interface 5. The audio interface 5 is connected to multiple microphones 3 and multiple opposed speakers 4. The audio interface 5 outputs sounds picked up from the multiple microphones 3 to the remote communication device 1. The audio interface 5 also outputs sounds output from the remote communication device 1 to the multiple opposed speakers 4.

The camera 2 captures images of the specific space in which it is placed and outputs the captured images to the remote communication device 1. The microphone 3 is an omnidirectional microphone. Multiple microphones 3 are arranged in a predetermined configuration to form a microphone array 30. It is preferable that the camera 2 and microphone array 30 are aligned in approximately the same position. In particular, it is preferable that they are aligned in a horizontal line from the position where people are lined up in Figure 2A toward the camera 2 and microphone array 30.

Furthermore, a display 6 is connected to the remote communication device 1. The display 6 displays the video received by the remote communication device 1. As shown in FIG. 2B , a speaker array 41 and a speaker array 42 of multiple opposing speakers 4 are arranged above and below the display 6, respectively.

In the following explanation, it is assumed that people who will be conversing using the remote communication device 1 are lined up in a row parallel to the display 6 at equal distances from the display 6. The distance from the display 6 to the people is, for example, approximately 2.0 m. In the following explanation, the surface of the display 6 that actually displays the image will be referred to as the "screen." Furthermore, the direction connecting the head and feet of the person being photographed, and the up and down direction of the image when projected onto the screen, will be referred to as the "vertical direction." Furthermore, the left and right directions when the person being photographed is facing forward, and the left and right directions when the image is projected onto the screen, will be referred to as the "horizontal direction." In particular, the left side of the image when projected onto the screen will be referred to as the "left," and the right side will be referred to as the "right."

<1.1. Definition of terms>
Next, definitions of terms used in the embodiments will be explained.

1.1.1. Remote and Local
In the embodiment, the remote communication device 1 connects two physically separated locations and performs remote communication between people in each space. Of the two locations where remote communication is performed, the location where the transmitting remote communication device 1 is located is referred to as the "remote" location, and the location where the transmitting remote communication device 1 is located is referred to as the "local" location. In other words, the following description will focus on the case where video and audio from the remote location are sent to the local location. The remote person who is the conversation partner of the local location is referred to as the speaker, and the conversation partner to whom the speaker wants to send a message is referred to as the listener.

<1.1.2. Individual sounds>
Individual sounds refer to the individual voices of each speaker. In many-to-many communication, where many remote people converse with many local people, it is expected that multiple people will be speaking simultaneously at one of the locations (remote). In such a situation, simply recording with an omnidirectional microphone will result in the recording of multiple voices. Therefore, the process of extracting each speaker's voice individually from the recorded voices of multiple people is called "individual sound separation."

1.1.3. Infield and outfield
The infield refers to the position of a remote speaker who is currently paying attention to the local speaker, such as someone who is currently having a conversation with the local speaker, as seen from the local speaker at that time. For example, if a conversation is taking place between a local person and a remote person, the remote person is considered to be the "infield" for the local person. The infield can also be anyone who is currently participating in the conversation.

The outfield refers to the position of people who are not paying attention to the local situation at that time, such as remote people having a conversation with other remote people, as seen from the perspective of the local speaker at that time. If the listener for the remote speaker is also remote, in other words, if the conversation is between two remote speakers, the remote listener is treated as the "outfield" for the local speaker. People in the outfield can also be said to be people who are not participating in the conversation at that time. However, remote speakers can become infield or outfield depending on changes in their behavior over time.

<1.1.4. Multiple opposing speakers>
The multiple opposed speakers 4 are a speaker system having a pair of a speaker array 41 consisting of a plurality of speaker units 411 and a speaker array 42 consisting of a plurality of speaker units 412. In this embodiment, the speaker array 41 and the speaker array 42 are arranged vertically with the screen of the display 6 in between. For example, the speaker array 41 is installed above the speaker array 42 in the image projected on the screen. The following describes an example in which the speaker array 41 is installed above the speaker array 42.

Speaker array 41 and speaker array 42 have the same number of speaker units 411 and 412. In speaker array 41, multiple speaker units 411 are arranged horizontally. Similarly, in speaker array 42, multiple speaker units 412 are arranged horizontally. Each speaker unit 411 and each speaker unit 412 are arranged at corresponding positions in the vertical direction. Below, a pair of speaker units 411 and 412 arranged at corresponding positions in the vertical direction will be referred to as speaker units 411 and 412 in the same row. By playing the same sound from speaker units 411 and 412, a sound source is virtually positioned in the center between speaker array 41 and speaker array 42.

Multiple opposed speakers 4 have the following advantages over a system that simply consists of an array of speakers. By playing back the voices of separate speakers on individual channels, multiple opposed speakers 4 make it easier to hear each individual voice than when multiple voices are mixed together on a single channel. Additionally, by localizing the sound source in the vertical center of the display 6 screen, multiple opposed speakers 4 can make the voice sound as if it is coming from approximately the position of a person's face.

However, in the first embodiment, the output direction of speaker array 41 and the output direction of speaker array 42 are opposed to each other to achieve point localization and enhance the sense of realism, but it is also possible to use a general array speaker to achieve surface localization without opposing them.

<1.2. Transmitting unit>
The transmission unit 10 is a function used in the transmitting remote communication device 1. In the following explanation, the case where sound is sent from the remote to the local will be explained, i.e., the remote will be the transmitting side and the local will be the receiving side. As shown in FIG. 1 , the transmission unit 10 has a human sensing unit 11, a voice separation unit 12, an acoustic signal processing unit 13, an output integration unit 14, and a transmission unit 15.

<1.2.1. Person position detection unit>
The human sensing unit 11 receives an input of an image captured by the camera 2 in a specific space on the remote side where the camera 2 is placed. The human sensing unit 11 then executes human sensing processing including human position determination processing and infield/outfield determination processing, which will be described below. Hereinafter, the space on the remote side captured by the camera 2 will be referred to as the "remote space."

The human sensing unit 11 detects the position of the horizontal origin in the remote space from the image of camera 2. Figure 3 is a diagram for explaining the processing of the human sensing unit. Here, as shown in the image in the upper row of Figure 3, an example will be described in which an image 60 showing three speakers 61 to 63 is displayed on display 6. For example, the human sensing unit 11 assumes that the origin is located at the horizontal center 600 of the screen when the image of camera 2 in the remote space is projected onto the screen of display 6. Next, the human sensing unit 11 sets normalized coordinates that indicate the horizontal position from the origin on the screen of display 6. For example, the human sensing unit 11 sets the distance from the horizontal edge of the screen of display 6 to the center 600 as a distance of 0.5 in normalized coordinates. In Figure 3, the human sensing unit 11 uses the x coordinate as a normalized coordinate, and sets the normalized coordinate of the right edge of the screen of display 6 to 0.5 and the normalized coordinate of the left edge to -0.5. That is, in this embodiment, the human sensing unit 11 sets normalized coordinates with the screen width set to 1.

Next, the human sensing unit 11 performs the following human position determination process. The human sensing unit 11 performs skeletal recognition of each speaker who appears in the image captured by camera 2. Then, the human sensing unit 11 acquires the normalized coordinates of the neck of each speaker who appears in the image captured by camera 2. As a result, the human sensing unit 11 can acquire normalized coordinates that indicate the position of each speaker, and can determine the lateral positional relationships between speakers. The human sensing unit 11 can perform skeletal recognition using general tools that are already commercially released. Then, the human sensing unit 11 generates a person list consisting of elements equal to the number of speakers whose normalized coordinates have been acquired. In this embodiment, the human sensing unit 11 arranges the elements in the person list in ascending order of normalized coordinate values.

The images at the bottom of Figure 3 show details of the person position determination process and the infield/outfield determination process. For example, when video 60 is acquired, the person sensing unit 11 performs skeletal recognition to obtain the normalized coordinates of speaker 61's neck 601, the normalized coordinates of speaker 62's neck 602, and the normalized coordinates of speaker 63's neck 603. The person sensing unit 11 then confirms from the acquired normalized coordinates that speakers 61, 62, and 63 are lined up from the left of video 60 in this order. The person sensing unit 11 then sets elements in the person list in the order of speaker 61, speaker 62, and speaker 63, in accordance with video 60.

Here, the human sensing unit 11 performs skeleton detection from the video, but the coordinates obtained from the image are usually not expressed in a normalized coordinate system but in screen coordinates using the number of pixels in the video. Therefore, in practice, the human sensing unit 11 obtains the screen coordinates of the speaker's position from the video in the human position determination process, and then performs coordinate conversion from those screen coordinates to normalized coordinates. Figure 4 is a diagram showing the coordinate conversion from screen coordinates to normalized coordinates. Here, the axes of the screen coordinates 621 are the u-axis and v-axis, and the axes of the normalized coordinates 622 are the x-axis and y-axis.

The human sensing unit 11 determines the position of each speaker from the image projected on the screen 620 using screen coordinates 621 having a u-axis and a v-axis, and then converts the determined position of each speaker into normalized coordinates 622 having an x-axis and a y-axis. Furthermore, if the screen width of the screen 620 is W _Scr and the screen height is H _Scr , the ranges that each coordinate system can take are 0≦u≦W _Scr and 0≦v≦H _Scr , and -0.5≦x≦0.5 and -0.5≦y≦0.5. Here, the vertical normalized coordinate system is expressed in the range from -0.5 to 0.5, with the vertical center of the image set as 0. In this case, the human sensing unit 11 performs coordinate conversion using the following equation (1). By performing coordinate conversion in this manner, processing can be performed regardless of the camera's angle of view or aspect ratio.

Next, the human sensing unit 11 executes the following infield/outfield determination process. The human sensing unit 11 estimates the head direction of each person in the video, i.e., the direction in which the speaker is facing. For example, if the normalized coordinates of p people are obtained in the human position determination process, the human sensing unit 11 detects the faces of the speakers located at each of the normalized coordinates for each of the p people, and estimates the head direction using image analysis, etc. More specifically, the human sensing unit 11 estimates the head direction by assuming that the direction of the vector in the positive direction of the normalized coordinates is 0 degrees, and estimating the angle Φ of the head direction vector from this 0-degree vector.

Here, the person sensing unit 11 has an infield range in advance for determining whether each speaker is in the infield or outfield. For example, the person sensing unit 11 defines the state in which the speaker is facing the camera 2 as 90 degrees, and defines a range around a predetermined angle of 90 degrees as the infield range. The person sensing unit 11 then determines that speakers whose head direction is within the infield range are in the infield, and determines that speakers whose head direction is not within the infield range are outfield people. Next, the person sensing unit 11 provides an infield flag item for each element in the person list and sets the initial value to False. The person sensing unit 11 then sets the infield flag to True for speakers determined to be in the infield. The person sensing unit 11 also sets the infield flag to False for speakers determined to be in the outfield. As a result, the human sensing unit 11 can set the infield flag of each speaker in the person list to Tre if the speaker is in the infield, to False if the speaker is in the outfield, and further to False if there is an oversight of head direction detection.

For example, as shown in the image at the bottom of Figure 3, the person sensing unit 11 estimates head directions 611-613 for speakers 61-63. Furthermore, here, the infield range ψ is set to 45 degrees forward or backward from the front direction 610, which is 90 degrees, and the person sensing unit 11 performs infield/outfield determination processing. In this case, because head direction 611 is included in the infield range, the person sensing unit 11 determines that speaker 61 is in the infield. Furthermore, because head directions 612 and 613 are not included in the infield range, the person sensing unit 11 determines that speakers 62 and 63 are people in the outfield. Then, the person sensing unit 11 sets the infield flag for speaker 61 in the person list to True, and the infield flags for speakers 62 and 63 to False.

Referring back to Figure 1, the explanation continues. The human sensing unit 11 then outputs the created human list to the acoustic signal processing unit 13 and the output integration unit 14.

Here, the remote space is an example of a "predetermined space," and determining whether a speaker is in the infield or outfield is an example of "determining whether a speaker is participating in a conversation." In other words, the human sensing unit 11 determines whether each speaker is participating in a conversation based on the video captured by the camera 2 that captures multiple speakers present in the predetermined space. The human sensing unit 11 also determines the position of each speaker based on the video.

<1.2.2. Audio Separation Unit>
The audio separation unit 12 receives audio input collected by the multiple microphones 3. The input to the microphone array 30 includes not only speech sounds generated by speech but also background sounds generated in the environment, such as background noise, background music (BGM), and other sounds. Therefore, the audio separation unit 12 performs audio separation processing to separate the speech sounds input from the microphone array 30 from the background sounds so that the speech sounds can be processed later to adjust their audibility. The audio separation processing can also be considered speech extraction processing. For example, the audio separation unit 12 performs audio separation using vocal extraction or technology to extract only the singing voice from music with singing voices.

In this way, the audio separation unit 12 separates background sound from audio in a specified space that corresponds to the remote space. By separating the speech audio from the background sound, it becomes easier to adjust processes such as clarity adjustment for the speech audio and background sound.

In this embodiment, an omnidirectional microphone is used as the microphone 3, but it is also possible to have each speaker wear a pin microphone as the microphone 3, and to use an omnidirectional microphone to pick up background sound. In this case, the audio separation unit 12 may use the audio picked up by the pin microphone as the speech of each speaker, and the audio picked up by the omnidirectional microphone as the background sound.

<1.2.3. Acoustic signal processing section>
The sound signal processing unit 13 includes an individual sound separation unit 131 and a clarity adjustment unit 132 .

The individual sound separation unit 131 receives input of speech sounds extracted by the speech separation process performed by the speech separation unit 12. The individual sound separation unit 131 also acquires the person list created by the person sensing unit 11. The individual sound separation unit 131 then uses the person list to perform the following individual sound separation process to separate the speech sounds of each person appearing in the video.

Figure 5 is a diagram showing the individual sound separation process. Here, we will explain the case where the following conditions are met. The camera 2 and the microphone array made up of the microphone array 30 are arranged in series in the shooting direction of the camera 2. In other words, the camera 2 and the microphone array 30 are arranged in the normal direction of the two-dimensional image shot by the camera 2, and when considering the coordinates formed by the vertical, horizontal and depth directions in the image, the vertical and horizontal coordinates of the camera 2 and the microphone array 30 match. Furthermore, the speakers in the remote space line up horizontally at positions where the vertical direction of the two-dimensional image shot by the camera 2 matches. Here, the surface that corresponds vertically and horizontally to the positions where the speakers in the remote space are lined up is called the "virtual screen."

The normalized coordinates shown in the range of -0.5 to 0.5 in Figure 5 match the horizontal coordinates of the virtual screen. The distance L between the camera 2 and the microphone 3 is known. Furthermore, x _ph in Figure 4 is the physical distance from the origin of the coordinates on the virtual screen. x _ph can be calculated from the angle of view of the camera 2 and the value of the distance L.

In this case, the individual sound separation unit 131 creates threads for the number of elements registered in the acquired person list and assigns an ID (identifier) to each thread as an attribute to identify each speaker. For example, the individual sound separation unit 131 assigns thread IDs of 1, 2, 3, ... in order from left to right in the person list. Furthermore, the individual sound separation unit 131 stores the normalized coordinate values registered in the person list in each thread as thread-specific values. Then, the individual sound separation unit 131 calculates x _ph , which is the physical distance from the origin of each speaker, using the normalized coordinates obtained in the person position determination process as coordinates on the virtual screen. Then, the individual sound separation unit 131 can calculate the angle θ to a specific person using the calculated physical distance x _ph and the distance L from the virtual screen to camera 2 using the following equation (2).

For example, for the speaker 101 in Fig. 5, the individual sound separation unit 131 acquires _x1 as the normalized coordinate. Then, the individual sound separation unit 131 calculates the physical distance from the origin to the speaker 101 from the normalized coordinate _x1 , and can calculate the angle _θ1 from the origin centered on the camera 2 and microphone 3 of the speaker 101 using the physical distance calculated in equation (2) and the distance L. The individual sound separation unit 131 can obtain the angle of each speaker for each thread by performing a calculation to find the angle θ for each thread.

Next, the individual sound separation unit 131 performs individual sound separation processing on the speech sound extracted by the speech separation by the speech separation unit 12, using beamforming, for the speech sound of each speaker. The speech sound extracted by the speech separation by the speech separation unit 12 is sound obtained from the microphone array 30, and sensitivity in the direction of each speaker is ensured. Therefore, in the case of speaker 101, the individual sound separation unit 131 sets the beam direction to angle _θ1 with respect to speaker 101 and sets the beam width to a range 102 of 15 degrees centered on angle _θ1 . Then, the individual sound separation unit 131 specifies the beam direction and beam width to suppress sounds from directions outside the specified angular range of the acquired speech sound, thereby ensuring sensitivity in the target direction and acquiring the individual sound of speaker 101.

In this way, the individual sound separation unit 131 separates individual sounds, which are the speech sounds of each speaker, from the sound in the specified space that corresponds to the remote space, based on the position of each speaker determined by the human sensing unit 11.

In the past, it was difficult to acquire individual sounds from speakers without wearing a microphone while associating them with the position of the person on the video. However, the individual sound separation unit 131 of this embodiment uses a person-specific list in which the positions of people on the video are registered in the order they appear on the video, and performs beamforming processing while maintaining that order, making it easy to pair the position of each speaker on the video with the individual sound after separation. This makes it easy to match the video and sound output positions when playing back the speech locally after transmission.

Returning to Figure 1, the explanation continues. The clarity adjustment unit 132 receives input of the individual sounds of each person present in the remote space generated by the individual sound separation unit 131. The clarity adjustment unit 132 also receives input of a person list from the individual sound separation unit 131. Furthermore, the clarity adjustment unit 132 receives input of background sounds extracted by audio separation by the audio separation unit 12.

The clarity adjustment unit 132 determines for each individual sound whether each speaker is in the infield or outfield using the infield flag in the person list. Then, the clarity adjustment unit 132 performs clarity adjustment processing on the individual sounds and background sounds according to the determination result. In detail, the clarity adjustment unit 132 performs enhancement processing on the individual sounds of people in the infield, making them easier to hear. The clarity adjustment unit 132 also performs degrade processing on the individual sounds and background sounds of people in the outfield, making them harder to hear. The clarity adjustment unit 132 then outputs the individual sounds and background sounds that have been subjected to clarity adjustment processing to the output integration unit 14.

The clarity adjustment unit 132 can use formant weighting, an equalizing filter, or a reverberation filter for clarity adjustment processing. In the case of formant weighting, the clarity adjustment unit 132 reinforces the second formant and above as an enhancement process, and suppresses the second formant and above as a degradation process. In addition, the clarity adjustment unit 132 performs equalizing filter processing, such as degrading background sounds using a low-pass filter and degrading sounds by adding reverberation.

In this way, the clarity adjustment unit 132 adjusts the clarity of the speech of speakers who are participating in the conversation as determined by the human sensing unit 11, which is included in the audio of the specified space corresponding to the remote space picked up by the microphone 3. The clarity adjustment unit 132 also performs a clarity reduction adjustment on the speech of speakers who are not participating in the conversation, which is included in the audio of the specified space corresponding to the remote space, to reduce the clarity compared to the audio pickup state by the microphone 3. The clarity adjustment unit 132 also performs a clarity reduction adjustment on the background sound separated by the audio separation unit 12 to reduce the clarity compared to other audio.

In the past, it was difficult to appropriately handle background sounds and adjust clarity depending on whether a speaker is participating in a conversation. In contrast, the clarity adjustment unit 132 determines whether the transmitted sound is background sound or speech, and even within speech, whether the speaker is in the infield or outfield, thereby making it possible to change the ease of audibility and promote smoother dialogue.

<1.2.4. Output Integration Unit>
The output integration unit 14 receives input of the individual sounds and background sounds that have been subjected to clarity adjustment processing from the clarity adjustment unit 132. The output integration unit 14 also receives input of the person list from the person sensing unit 11. Then, the output integration unit 14 executes the clarity adjustment processing described below.

Audio processing of background sounds and speech sounds is performed one channel at a time, corresponding to each sound. In other words, the output integration unit 14 receives input data equal to the number of speakers plus one channel of background sound. In other words, if the background sound is one channel of data and the number of speakers is p, the output integration unit 14 obtains (1 + P) channels of data.

Then, in order to transmit the audio locally, the output integration unit 14 integrates the sounds of all channels into a single piece of audio data. For example, the output integration unit 14 can generate a single piece of audio data using channel interleaving, which includes elements of each channel for each frame. The output integration unit 14 then outputs the generated single piece of audio data to the transmission unit 15. The output integration unit 14 also outputs the person list to the transmission unit 15, along with information that associates each element of the person list and background sound with each channel included in the audio data. Here, the output integration unit 14 may add the data of the person list to the audio data to create a single piece of data.

<1.2.5. Transmitter>
The transmitter 15 receives an input of a single piece of sound data including the individual sounds of each speaker and background sound from the output integration unit 14. The transmitter 15 also receives an input of a person list from the output integration unit 14. The transmitter 15 also receives an input of captured video from the camera 2. The transmitter 15 then transmits the single piece of sound data including the individual sounds of each speaker and background sound, the person list, and the video from the camera 2 to the receiving unit 20 of the remote communication device 1 on the local side via the network 7. The network 7 is, for example, the Internet or a LAN (Local Area Network). In this way, the transmitter 15 transmits the audio of a predetermined space corresponding to the remote space processed by the clarity adjustment unit 132.

1.3. Receiving unit
The receiving unit 20 is a function used in the receiving-side remote communication device 1, which is the local side in this embodiment. The receiving unit 20 includes a receiving section 21 and an audio output control section 22, as shown in FIG.

<3.1. Receiving section>
The receiving unit 21 receives one piece of sound data including individual sounds of each speaker in the remote space and background sounds, a person list, and video transmitted from the transmitting unit 15 of the remote communication device 1 on the remote side. The receiving unit 21 outputs the received video to the display 6. The display 6 displays the video output from the receiving unit 21 on a screen. The receiving unit 21 also outputs the sound data and the person list to the audio output control unit 22.

In this way, the receiving unit 21 receives the audio from the specified space corresponding to the remote space transmitted by the transmitting unit 15. The receiving unit 21 also receives the video along with the audio from the specified space corresponding to the remote space, and projects the video onto a screen on which speaker arrays 41 and 42 are arranged, one at each end of the video in the vertical direction when the video is displayed.

<1.3.2. Audio output control unit>
The audio output control unit 22 performs a multiple opposing speaker balancing process described below to specify the speaker units 411 and 412 that will output the background sound, the infield speech sound, and the outfield speech sound, respectively, and plays them back. The audio output control unit 22 receives sound data including the individual sound and background sound of each speaker, as well as an input of a person list, from the receiving unit 21. Hereinafter, the speech sound of people in the infield will be referred to as "infield speech sound," and the speech sound of people in the outfield will be referred to as "outfield speech sound."

Figure 6 is a diagram showing the multiple opposing speaker balancing process. The image at the top of Figure 6 shows the length of each part of speaker arrays 41 and 42, and graph 200 at the bottom shows the post-processing output sounds from speaker arrays 41 and 42 for background sound, infield speech sounds, and outfield speech sounds. Graph 200 shows the positions corresponding to speaker arrays 41 and 42 on the horizontal axis, and the volume on the vertical axis. The vertical axis of graph 200 shows a value normalized with the peak volume of infield speech sounds set to 1.

The audio output control unit 22 has information on the physical speaker array width W _spk and display width W _disp in advance. The audio output control unit 22 also has information on the number u of speaker units 411 in the speaker array 41 in advance. The number of speaker units 412 in the speaker array 42 is also u. Here, W _spk represents the distance between the leftmost speaker unit 411 and the rightmost speaker unit 411. The same applies to W _spk for the speaker units 412. Furthermore, the inter-speaker distances W _u between the speaker units 411 and between the speaker units 412 are all equal, and the display 6, speaker array 41, and speaker array 42 are aligned horizontally and centrally.

As a preliminary process common to the output adjustment processes for background sound, infield speech sound, and outfield speech sound, the audio output control unit 22 calculates the inter-speaker distance _Wu between the speaker units 411 as _Wu = _Wspk /u using known _Wspk and u. Here, since _Wu is the actual physical distance, the audio output control unit 22 aligns the processed distance to the normalized coordinate size as _{Wu_n} = _Wu / _Wdisp in order to match the position of the speaker on the video with the audio. The audio output control unit 22 also aligns the speaker array width _Wspk to the normalized coordinate size as _{Wspk_n} . The audio output control unit 22 also calculates the normalized coordinate xu of the speaker units 411 and 412.

Here, if the amplitude ratio is x, the sound pressure level is y, and the loudness ratio is z, then the following relationship holds: y = 20 log ₁₀ x and z = 2 ^y/10 . Solving this for x yields the relationship x = z ^{1/2 log} ₁₀ ^2. This means that if you want to increase the volume by z times, you just need to multiply the amplitude by z ^1.66 .

The following describes the multiple opposing speaker balancing process for background sound. In the remote communication device 1 according to this embodiment, in order to facilitate communication with a remote conversation partner while maintaining a sense of spatial connection, background sound is transmitted locally rather than being completely removed. However, if the volume of the background sound is made too high, it becomes difficult to hear infield speech. Therefore, the audio output control unit 22 further applies the following volume adjustments to the background sound, whose clarity has been reduced by the degradation process, at the timing of playback to blur the sense of positioning of the background sound.

Specifically, the audio output control unit 22 acquires background sound from the sound data. Next, the audio output control unit 22 adjusts the loudness of the background sound to 1/u and outputs it from all speaker units 411 and 412. That is, the audio output control unit 22 processes the amplitude of the background sound to be ^1.66 times (1/u), and outputs the background sound from all speaker units 411 of the speaker array 41 and all speaker units 412 of the speaker array 42.

By performing the above processing, background sound is output at the same reduced volume from all speaker units 411 and 412, as shown by curve 201 in graph 200. Here, background sound refers to sounds whose source location is unclear, such as natural environmental sounds, background music in a cafe, and office noise. By blurring the sense of positioning of the background sound, the audio output control unit 22 can reproduce the remote space with a greater sense of realism than if it were played from a single speaker unit 411 or 412.

Next, we will explain the multiple facing speaker balancing process for infield speech sounds. Since infield speech sounds are intended for local people, it is preferable to make them as easy to hear as possible. Therefore, the audio output control unit 22 performs the following process on individual infield sounds that have been remotely enhanced.

The audio output control unit 22 acquires infield speech from the sound data using the person list. Next, the audio output control unit 22 plays the acquired infield speech directly from the speaker arrays 41 and 42 without blurring the sense of localization of the speech and without adjusting the volume or volume. Here, the audio output control unit 22 identifies the speaker units 411 and 412 that are closest to the normalized coordinates of the speaker of the infield speech. In other words, if the normalized coordinates of the speaker are x _p , the audio output control unit 22 identifies the speaker units 411 and 421 that exist at the normalized coordinate x _u calculated by the following formula (3).

Then, the audio output control unit 22 outputs the infield speech sound from the identified speaker units 411 and 412. In this case, the audio output control unit 22 does not output the infield speech sound from the other speaker units 411 and 412. By performing the above processing, the infield speech sound is output from one specific speaker unit 411 and 412 at a louder volume than other sounds, as shown by curve 202 in graph 200.

Next, we will explain the multiple facing speaker balancing process for outfield speech sounds. Outfield speech does not need to be clearly audible because the local person is not directly speaking with them, but as with background sound, it is undesirable to completely remove it from the perspective of spatial connection. Also, by lowering the clarity but still being audible if you listen carefully, there is an opportunity for a local person to talk to someone in the outfield, and the communication develops and shifts to the infield. Therefore, to make infield speech relatively easier to hear and outfield speech relatively harder to hear, degrading processing is performed on individual outfield sounds remotely. Then, the audio output control unit 22 performs processing to blur the sense of positioning of the acquired outfield speech sounds.

Specifically, because the sound source position of outfield speech is clear, the audio output control unit 22 blurs the sense of localization of outfield speech by processing using a normal distribution function. Here, the normal distribution function is expressed by the following equation (4), where μ is the mean value and σ is the standard deviation.

In order to match the peak of the normal distribution function with the position of the speaker of the outfield speech, the audio output control unit 22 sets the value of μ to the position of the normalized coordinate x _u of the speaker units 411 and 412 that is closest to the normalized coordinate of the speaker that satisfies Equation (3). Furthermore, the audio output control unit 22 sets σ to W _u . FIG. 7 is a diagram showing processing using a normal distribution function for outfield speech. The audio output control unit 22 creates a function f(x) represented by a graph 211 whose peak is the position of the speaker of the outfield speech. Here, f(x) is expressed by the following Equation (5).

The x-coordinate range of the normalized distribution function is normally from -∞ to ∞, but the audio output control unit 22 imposes a restriction on the range of playback by the speaker arrays 41 and 42, for example, by playing back using speaker units 411 and 412 within the 95% confidence interval. This allows the audio output control unit 22 to output sound within a range of μ-2σ to μ+2σ (range = 2), fining the range of sound output. In this case, the audio output control unit 22 plays back one person's outfield speech from five speaker units 411 and 412 each. By narrowing the range of speaker units 411 and 412 to be played back in this way, the audio output control unit 22 can reduce the processing load.

Furthermore, depending on the value of σ, it is possible that the maximum value will be greater than 1. In that case, using f(x) to adjust the volume may result in outfield speech sounds being louder than infield speech sounds. Therefore, the audio output control unit 22 uses a scaling factor sf to reduce the overall volume by g(x) = sf/f(μ) × f(x), as shown in graph 212 in Figure 7, thereby preventing outfield speech sounds from being played louder than infield speech sounds. sf can be set to 0.8, for example.

The audio output control unit 22 then processes the amplitude of the outfield speech sound to be ^1.66 times {g(x)} according to the normalized coordinates, and reproduces the outfield speech sound from a specified number of speaker units 411 and 412 centered around the speaker position of the outfield speech sound. Here, the value of the scaling factor sf (0≦sf≦1) that defines the peak value and the speaker playback range can be changed by the user using a configuration file, etc. By performing the above processing, the outfield speech sound is output at a lower volume than the infield speech sound from the speaker units 411 and 412 within a specified range centered around the speaker position, as shown by curve 203 in graph 200 of FIG. 5.

In this way, the audio output control unit 22 performs the following processing on the multiple opposed speakers 4, which have two arrays: a speaker array 41 in which multiple speaker units 411 are arranged in a row, and a speaker array 42 in which multiple speaker units 412 are arranged in a row. Based on the position of each speaker, the audio output control unit 22 selects speaker units 411 and 412 to play back the speech of each speaker from the audio received by the receiving unit 21 in a specified space corresponding to the remote space. The audio output control unit 22 then causes the selected speaker units 411 and 412 to play back the speech of each speaker. Furthermore, for each speaker shown on the screen, the audio output control unit 22 selects speaker units 411 and 412 to play back based on the position of the speaker so that the speech is played back near the position on the screen.

In the past, it was difficult to control the playback method of multi-channel speakers depending on whether the speaker was speaking with the other party on the other side of the screen, i.e., whether they were participating in the conversation. In contrast, the audio output control unit 22 makes the voice of the infield speaker, who is the local conversation partner, relatively easier to hear, while maintaining a sense of connection with the remote space by leaving background sounds and outfield speech. In addition, the audio output control unit 22 plays background sounds and outfield speech from multiple speaker units 411 and 412, blurring the sense of sound localization and making them more difficult to hear, while making infield speech relatively easier to hear, allowing you to better concentrate on the conversation with the remote party.

2. Data flow between remote communication devices according to the first embodiment
8 is a diagram showing an outline of the data flow between the local remote communication devices according to the first embodiment. Here, too, the case where data is transmitted from the transmitting unit 10 of the remote remote communication device 1 to the receiving unit 20 of the local remote communication device 1 will be described. Here, the transmission of audio will be described.

The sending-side process 221 is a process executed by the remote communication device 1 located on the remote side. The receiving-side process 222 is a process executed by the remote communication device 1 located on the local side.

The microphone 3 inputs the audio picked up in the remote space to the remote communication device 1 (step S1). Here, for example, data from the microphone array 30 is s channel data. The camera 2 also inputs the video generated by capturing images of the remote space to the remote communication device 1 (step S2).

The video input from camera 2 is sent to the human sensing unit 11. The human sensing unit 11 uses the video captured by camera 2 to perform human sensing processing to determine the position of each person in the remote space and to determine whether the field or outfield is located (step S3).

In more detail, the human sensing unit 11 sets normalized coordinates in the horizontal direction of the image projected on the screen. Then, the human sensing unit 11 performs skeletal recognition on speakers in the image and obtains normalized coordinates of the position of each speaker. After that, the human sensing unit 11 generates a person list having elements corresponding to the order of speakers and registers the normalized coordinates of each person's position, thereby executing the human position determination process (step S31). Here, the human sensing unit 11 extracts p speakers present in the image. In other words, p elements are registered in the person list.

The human sensing unit 11 also estimates the head direction of each speaker in the video. Then, depending on whether the estimated head direction is within a predetermined infield range, the human sensing unit 11 determines whether each speaker is in the infield or outfield, and executes the infield/outfield determination process by registering an infield flag indicating the infield or outfield in the person list based on the determination result (step S32).

The audio input from microphone 3 is sent to audio separation unit 12. Audio separation unit 12 performs audio separation processing to separate speech and background sounds contained in the audio input from microphone 3 (step S4). For the s-channel data input from microphone array 30, audio separation unit 12 generates one channel of data as background sound and s-channel data as speech.

The background sound, which is one-channel data extracted by the audio separation unit 12, is sent to the clarity adjustment unit 132 of the audio signal processing unit 13. The clarity adjustment unit 132 performs a clarity adjustment process, called a degradation process, to make the background sound less audible (step S5).

Furthermore, the speech sound, which is the s-channel data extracted by the speech separation unit 12, is sent to the individual sound separation unit 131 of the acoustic signal processing unit 13. The person list is also sent to the individual sound separation unit 131. The acoustic signal processing unit 13 performs speech sound signal processing on the speech sound using the person list (step S6).

In more detail, the individual sound separation unit 131 creates p threads, which is the number of elements registered in the acquired personal list. Furthermore, the individual sound separation unit 131 stores the normalized coordinate values registered in the personal list as thread-specific values for each thread. The individual sound separation unit 131 then calculates the physical distance using the normalized coordinates of each speaker as coordinates on the virtual screen. The individual sound separation unit 131 then determines the angle θ to each speaker using the calculated physical distance and the distance from the virtual screen to camera 2. Next, the individual sound separation unit 131 performs individual sound separation processing on the speech sounds extracted by speech separation by the speech separation unit 12 using beamforming according to the angle to each speaker (step S61). As a result, the individual sound separation unit 131 generates individual sounds, which are one-channel data for each speaker. In other words, one-channel individual sound data is generated for each of the p threads.

Next, the clarity adjustment unit 132 determines whether the speaker corresponding to the individual sounds for each thread is infield or outfield using the infield flag in the person list. Then, as a clarity adjustment process, the clarity adjustment unit 132 performs an enhancement process to make the individual sounds of infield speakers easier to hear, and a degradation process to make the individual sounds of outfield speakers harder to hear (step S62).

The background sound that has undergone clarity adjustment processing is input to the output integrating unit 14. The individual sounds that have undergone speech signal processing are also input to the output integrating unit 14. As a result, the output integrating unit 14 obtains data for (1+P) channels. The output integrating unit 14 then executes output integration processing to integrate the sounds from all channels to generate a single sound data set (step S7).

The sound data and person list generated by the output integration unit 14 are sent by the transmission unit 15 via the network 7 to the local-side remote communication device 1 (step S8). By sending the person list, the normalized coordinates of each speaker determined by the person position determination process and information on whether each speaker is in the infield or outfield determined by the infield/outfield determination process are sent to the local-side remote communication device 1.

The local remote communication device 1 receives the sound data and the person list. The sound data and person list are sent to the audio output control unit 22 via the receiving unit 21. The audio output control unit 22 uses the person list to perform a multiple facing speaker balancing process on the sound data, and specifies the speaker units 411 and 412 to output the background sound, infield speech sound, and outfield speech sound for playback (step S9). In detail, the audio output control unit 22 adjusts the loudness of the background sound by dividing it by the number of pairs of speaker units 411 and 412, and plays the adjusted sound on all speaker units 411 and 412. The audio output control unit 22 also plays the infield speech sound directly on the speaker units 411 and 412 closest to the position of the speaker of the infield speech sound. In addition, the audio output control unit 22 processes the outfield speech using a normal distribution function with a scaling factor, and limits the range of speaker units 411 and 412 to be used, causing the speaker units 411 and 412 to reproduce the outfield speech.

The speaker arrays 41 and 42 output background sounds, infield speech sounds, and outfield speech sounds using the speaker units 411 and 412 specified by the audio output control unit 22 (step S10).

<3. Remote communication processing>
Next, the flow of the remote communication process will be described, which will be divided into a transmitting process by the transmitting remote communication device 1 and a multiple opposing speaker balancing process by the receiving remote communication device 1.

<3.1. Transmission side processing>
9 is a flowchart of the transmission side process. The overall flow of the transmission side process will be described with reference to FIG.

The human sensing unit 11 acquires video of the remote space captured by the camera 2. The audio separation unit 12 also acquires audio of the remote space picked up by the microphone 3 (step S11).

The person sensing unit 11 performs a person determination process using the video captured by the camera 2 to determine the normalized coordinates of each speaker's position and to create a person list with elements for the number of people and in which the normalized coordinates of each speaker are registered (step S12).

Next, the human sensing unit 11 performs an infield/outfield determination process using the video and the person list to determine whether each speaker is in the infield or outfield (step S13).

The audio separation unit 12 performs audio separation processing to separate the speech and background sounds contained in the audio input from the microphone 3 (step S14).

The clarity adjustment unit 132 receives the background sound extracted by the audio separation unit 12. The clarity adjustment unit 132 then performs a degrading process to make the background sound more difficult to hear as a clarity adjustment process (step S15).

The individual sound separation unit 131 receives input of speech extracted by the audio separation unit 12. The individual sound separation unit 131 also acquires a person-specific list from the person sensing unit 11. Next, the individual sound separation unit 131 generates threads equal to the number of elements registered in the acquired person-specific list (step S16). The individual sound separation unit 131 assigns IDs to each thread, sequentially numbered from 1, equal to the number of elements registered in the person-specific list. Here, the process is performed when the number of elements registered in the person-specific list is p.

Next, the individual sound separation unit 131 performs individual sound separation processing to separate the speech into individual sounds for each speaker (step S17).

The clarity adjustment unit 132 receives input of individual sounds from the individual sound separation unit 131. Next, the clarity adjustment unit 132 initializes i to 0 (step S18).

Next, the clarity adjustment unit 132 performs clarity adjustment processing on the individual sounds in the thread whose ID is i (step S19).

Next, the clarity adjustment unit 132 determines whether i is less than p (step S20). If i is less than p (step S20: Yes), the clarity adjustment unit 132 increments i by 1 (step S21). Then, the clarity adjustment unit 132 returns to step S19.

On the other hand, if i is greater than or equal to p (step S20: No), the clarity adjustment unit 132 outputs the background sound and individual sounds that have been subjected to clarity adjustment processing to the output integrator 14. The output integrator 14 also receives video input from the camera 2. The output integrator 14 then executes output integration processing to integrate the background sound and individual sounds to generate a single piece of sound data (step S22).

Then, the transmission unit 15 transmits the sound data generated by the output integration unit 14 and the video captured by the camera 2 to the local remote communication device 1 via the network 7 (step 23).

<3.1.1. Person position determination process>
Fig. 10 is a flowchart of the person position determination process. The process shown in Fig. 10 is an example of the process executed in step S12 in Fig. 9. Next, the flow of the person position determination process will be described with reference to Fig. 10.

The human sensing unit 11 sets normalized coordinates in the horizontal direction of the image projected on the screen. Next, the human sensing unit 11 performs skeletal recognition on the speakers in the image to identify the position of each speaker's neck and obtain the screen coordinates of the neck (step S101).

Next, the human sensing unit 11 converts the screen coordinates into normalized coordinates (step S102).

Then, the human sensing unit 11 registers the screen coordinates and normalized coordinates of each speaker in the person list (step S103). Here, the human sensing unit 11 stores the horizontal coordinates when the screen coordinates and normalized coordinates of each image are projected onto the screen of the display 6.

<3.1.2. Infield/Outfield Determination Processing>
Fig. 11 is a flowchart of the infield/outfield determination process. The process shown in Fig. 11 is an example of the process executed in step S13 in Fig. 9. Next, the flow of the infield/outfield determination process will be described with reference to Fig. 11.

The human sensing unit 11 creates an infield flag item in the person list, initializes it, and sets all items equal to the number of people present in the remote space to False (step S111).

Next, the human sensing unit 11 initializes i to 0 (step S112).

Next, the human sensing unit 11 detects the head direction of the i-th speaker, counting the first speaker from the left side of the video as 0 (step S113).

The human sensing unit 11 determines whether the head direction of the i-th speaker can be detected (step S114). If the head direction cannot be detected (step S114: No), the human sensing unit 11 proceeds to step S118.

On the other hand, if the head direction can be detected (step S114: Yes), the human sensing unit 11 determines whether the head direction is within the infield (step S115).

If the head direction is within the infield range (step S115: Yes), the person sensing unit 11 stores True in the infield flag of the element corresponding to the i-th speaker in the person list (step S116).

On the other hand, if the head direction is outside the infield range (step S115: No), the person sensing unit 11 stores False in the infield flag of the element corresponding to the i-th speaker in the person list (step S117).

Then, the human sensing unit 11 determines whether i is less than p (step S118). If i is less than p (step S118: Yes), the human sensing unit 11 increments i by 1 (step S119). Then, the human sensing unit 11 returns to step S113.

On the other hand, if i is greater than or equal to p (step S118: No), the human sensing unit 11 ends the infield/outfield determination process.

<3.1.3. Individual Sound Separation Processing>
Fig. 12 is a flowchart of the individual sound separation process. The process shown in Fig. 12 is an example of the process executed in step S17 in Fig. 9. Next, the flow of the individual sound separation process will be described with reference to Fig. 12.

The individual sound separation unit 131 creates threads for the number of people registered in the personal list. Next, the individual sound separation unit 131 stores the normalized coordinate values registered in the personal list as thread-specific values for each thread. Then, the individual sound separation unit 131 initializes i to 0 (step S121).

Next, the individual sound separation unit 131 calculates the beam direction of the i-th speaker, counting the first speaker from the left side of the video as 0 (step S122).

Next, the individual sound separation unit 131 acquires the individual sound of the i-th speaker by beamforming processing in the beam direction (step S123).

Then, the individual sound separation unit 131 determines whether i is less than p (step S124). If i is less than p (step S124: Yes), the individual sound separation unit 131 increments i by 1 (step S125). Then, the individual sound separation unit 131 returns to step S122.

On the other hand, if i is greater than or equal to p (step S124: No), the individual sound separation unit 131 terminates the individual sound separation process.

3.1.4. Clarity Adjustment Processing
Fig. 13 is a flowchart of the clarity adjustment process for individual sounds. The process shown in Fig. 13 is an example of the process executed in step S19 in Fig. 9. Next, the flow of the clarity adjustment process will be described with reference to Fig. 13.

The clarity adjustment unit 132 initializes i to 0 (step S131).

Next, the clarity adjustment unit 132 determines whether the infield flag in the person list for the ith speaker, with the first speaker from the left side of the video being number 0, is True (step S132).

If the infield flag is True (step S132: Yes), the clarity adjustment unit 132 performs enhancement processing on the individual sounds of the i-th speaker (step S133).

If the infield flag is False (step S132: No), the clarity adjustment unit 132 performs degrading processing on the individual sound of the i-th speaker (step S134).

Then, the clarity adjustment unit 132 determines whether i is less than p (step S135). If i is less than p (step S135: Yes), the clarity adjustment unit 132 increments i by 1 (step S136). Then, the clarity adjustment unit 132 returns to step S132.

On the other hand, if i is greater than or equal to p (step S135: No), the clarity adjustment unit 132 terminates the individual sound separation process.

<3.2. Multiple opposing speaker balancing process (receiving side process)>
14 is a flowchart of the multiple opposed speaker balancing process. The overall flow of the multiple opposed speaker balancing process will be described with reference to FIG. 14. Here, the case where the audio output control unit 22 controls playback using a list for each combination of speaker units 411 and 412 will be described.

The audio output control unit 22 receives input of the sound data and the person list from the receiving unit 21. The audio output control unit 22 calculates the distance between the speaker units 411 (step S321). For example, if the distance between the speaker units 411 at both ends is Wspk and there are u speaker units 411 and 412, the audio output control unit 22 can calculate the inter-speaker distance Wu between the speaker units 411 as Wu = Wspk/u.

Next, the audio output control unit 22 calculates the normalized coordinates for each pair of speaker units 411 and 412, and stores the normalized coordinates in ascending order in a speaker unit list having the same number of elements as the number of pairs of speaker units 411 and 412 (step S322).

Next, the audio output control unit 22 provides an area for storing output sounds for each element in the speaker unit list, and initializes each area with the number of channels for the background sounds and individual sounds (step S33). For example, if the background sound is one-channel data and there are p individual sounds, (1 + P) areas for storing sounds of (1 + P) channels are set in the area for storing output sounds.

Next, the audio output control unit 22 initializes i to 0 (step S34).

Next, the audio output control unit 22 determines whether the audio of the i-th channel, with the first of the (1+P) channels arranged in order in the audio data being numbered 0, is an individual sound of the speech (step S35).

If the audio on the i-th channel is background sound (step S35: No), the audio output control unit 22 performs background sound adjustment (step S36). The audio output control unit 22 then proceeds to step S40.

On the other hand, if the audio on the i-th channel is an individual sound of speech (step S35: Yes), the audio output control unit 22 determines whether the speaker of that individual sound is Uchino using the person list (step S37).

If the speaker is in the outfield (step S37: No), the audio output control unit 22 performs outfield speech sound adjustment (step S38). The audio output control unit 22 then proceeds to step S40.

If the speaker is an infield player (step S37: Yes), the audio output control unit 22 performs infield speech sound adjustment (step S39).

Then, the audio output control unit 22 stores the audio of the i-th channel in the area for storing output audio in the speaker unit list (step S40).

Next, the audio output control unit 22 determines whether i is less than p (step S41). If i is less than p (step S41: Yes), the audio output control unit 22 increments i by 1 (step S42). Then, the audio output control unit 22 returns to step S35.

On the other hand, if i is greater than or equal to p (step S41: No), the audio output control unit 22 mixes the audio stored in the area for storing output audio in the speaker unit list. Then, the audio output control unit 22 outputs the mixed sound from the speaker units 411 and 412 corresponding to the speaker unit list (step S43).

<3.2.1. Background sound adjustment>
Fig. 15 is a flowchart of the background sound adjustment process. The process shown in Fig. 15 is an example of the process executed in step S36 in Fig. 14. Next, the flow of the background sound adjustment process will be described with reference to Fig. 15.

The audio output control unit 22 adjusts the amplitude of the background sound by a factor of (1/u) (step S201), where u is the number of pairs of speaker units 411 and 412.

Then, the audio output control unit 22 assigns waveforms to the channels of all pairs of speaker units 411 and 412 (step S202).

<3.2.2. Infield speech sound adjustment>
Fig. 16 is a flowchart of the infield speech sound adjustment process. The process shown in Fig. 16 is an example of the process executed in step S39 in Fig. 14. Next, the flow of the infield speech sound adjustment process will be described with reference to Fig. 16.

The audio output control unit 22 performs localization center speaker unit derivation to identify the pair of speaker units 411 and 412 that are closest to the speaker (step S211). This localization center speaker unit derivation will be explained in detail later.

Next, the audio output control unit 22 assigns the waveform to the pair of channels of speaker units 411 and 412 that is closest to the normalized coordinates of the speaker (step S212).

3.2.3. Outfield speech adjustment
Fig. 17 is a flowchart of the outfield speech sound adjustment process. The process shown in Fig. 17 is an example of the process executed in step S38 of Fig. 14. Next, the flow of the outfield speech sound adjustment process will be described with reference to Fig. 17.

The audio output control unit 22 performs localization center speaker unit derivation to identify the pair of speaker units 411 and 412 closest to the speaker (step S221). This localization center speaker unit derivation is the same process as the process used in infield speech sound adjustment, and will be described in detail later. Here, when the positions of the pair of speaker units 411 and 412 are represented by consecutive numbers starting from the left with 0 as the first, the position of the pair closest to the speaker is called the "closest speaker unit position." Here, the number of pairs of speaker units 411 and 412 is u, and the closest speaker unit position is any one of 0 to u.

Next, the audio output control unit 22 initializes k to 0 (step S222). k is a parameter used to control the repetition of the process of determining the playback range.

Next, the audio output control unit 22 determines whether k = 0 (step S223).

If k=0 (step S223: Yes), the audio output control unit 22 sets x _spk to the position obtained by adding k to the position of the closest speaker unit (step S224). After that, the audio output control unit 22 proceeds to step S229.

If k is other than 0 (step S223: No), the audio output control unit 22 determines whether the value obtained by adding k to the nearest speaker unit position is less than or equal to u (step S225). In other words, the audio output control unit 22 determines whether the position obtained by adding k to the nearest speaker unit position does not extend beyond the right end of the speaker arrays 41 and 42.

If the value obtained by adding k to the position of the nearest speaker unit is greater than u (step S225: No), the audio output control unit 22 proceeds to step S227. On the other hand, if the value obtained by adding k to the position of the nearest speaker unit is equal to or less than u (step S225: Yes), the audio output control unit 22 sets one x _spk to the position obtained by adding k to the nearest speaker unit (step S226). Thereafter, the audio output control unit 22 proceeds to step S227.

Next, the audio output control unit 22 determines whether the value obtained by subtracting k from the nearest speaker unit position is 0 or greater (step S227). In other words, the audio output control unit 22 determines whether the position obtained by subtracting k from the nearest speaker unit position does not extend beyond the left end of the speaker arrays 41 and 42.

If the value obtained by subtracting k from the nearest speaker unit position is less than 0 (step S227: No), the audio output control unit 22 proceeds to step S229. On the other hand, if the value obtained by subtracting k from the nearest speaker unit position is 0 or greater (step S227: Yes), the audio output control unit 22 sets the other x _spk to the position obtained by subtracting k from the nearest speaker unit (step S228). Thereafter, the audio output control unit 22 proceeds to step S229.

Thereafter, the audio output control unit 22 adjusts the amplitude of the audio at position k by multiplying it by {g(x _spk )} ^1.66 (step S229). Here, if both one x _spk and the other x _spk exist, the audio output control unit 22 adjusts the amplitude of the audio at both positions k.

Then, the audio output control unit 22 determines whether k is less than a predetermined range width (step S230).

If k is equal to or less than the range width (step S230: Yes), the audio output control unit 22 increments k by 1 (step S231). Then, the audio output control unit 22 returns to step S233.

On the other hand, if k is greater than the range width (step S230: No), the audio output control unit 22 assigns the waveform to one of the u channels with a ±range width centered on the position of the closest speaker unit (step S232).

3.2.4. Derivation of the center speaker unit position
Fig. 18 is a flowchart of the process of deriving the localization center speaker unit. The process shown in Fig. 18 corresponds to an example of the process executed in step S211 of Fig. 16 and step S221 of Fig. 17. Next, the flow of the process of deriving the localization center speaker unit will be described with reference to Fig. 18.

The audio output control unit 22 sets the initial value of the shortest distance to the speaker unit 411 as the distance between the speaker units 411 (step S241).

Next, the audio output control unit 22 initializes j to 0 (step S242). Here, j is a parameter used to control the repetition of the process of determining whether the distance to the speaker for each pair of speaker units 411 and 412 is the closest speaker unit distance.

Next, the audio output control unit 22 subtracts the position of the jth speaker unit 411 when the speaker unit 411 pairs are numbered consecutively from the left, with the first being number 0, from the speaker's position (step S243).

Next, the audio output control unit 22 determines whether the subtraction result is less than the shortest distance (step S244). If the subtraction result is equal to or greater than the shortest distance (step S244: No), the audio output control unit 22 proceeds to step S246.

If the subtraction result is less than the shortest distance (step S244: Yes), the audio output control unit 22 updates the shortest distance as the subtraction result. Furthermore, the audio output control unit 22 sets j to the nearest speaker unit position (step S245).

Next, the audio output control unit 22 determines whether j is less than u, which is the number of speaker units 411 (step S246).

If j is less than u (step S246: Yes), the audio output control unit 22 increments j by 1 (step S247). Then, the audio output control unit 22 returns to step S243.

On the other hand, if j is greater than or equal to u (step S246: No), the audio output control unit 22 terminates the derivation of the localization center speaker unit.

<4. Effects>
As described above, the remote communication device 1 according to this embodiment identifies the positions of speakers from the video, estimates the head direction of each identified speaker, and determines whether each speaker is in the infield or outfield. The remote communication device 1 also separates background sound from the collected audio and acquires the individual sounds of the speakers based on the identified positions. The remote communication device 1 then reduces the volume of the background sound and outfield speech sounds and outputs them from a pair of speaker units 411 and 412 within a certain range. The remote communication device 1 also outputs the infield speech sounds from the speaker units 411 and 412 closest to the speaker's position.

In this way, by classifying speech sounds into infield speech sounds and outfield speech sounds, different acoustic processing can be performed for each. Furthermore, by blurring the sense of sound localization for background sounds and outfield speech sounds, making them harder to hear, the infield speech sounds become relatively easier to hear. This makes it easier to hear the voice of the person you're talking to while still retaining a sense of presence and atmosphere, allowing you to concentrate on the conversation. Furthermore, determining conversation participation through head direction estimation enables automatic determination of speakers participating in communication. Furthermore, by processing background sounds, infield speech sounds, and outfield speech sounds separately, signal processing based on participation level becomes possible, and multi-channel speaker control can be performed in accordance with background sounds, infield speech sounds, and outfield speech sounds. Therefore, smoother dialogue can be promoted.

5. Remote communication device according to the second embodiment
Next, a remote communication device 1 according to a second embodiment will be described. The remote communication device 1 according to the second embodiment outputs background sounds without blurring the sense of localization of sudden background sounds.

Figure 19 is a block diagram of a remote communication device according to the second embodiment. Note that in Figure 19, the same components as in Figure 1 are designated by the same reference numerals, and the following explanation will focus on the components that differ from Figure 1, and may omit explanations of the components that are the same as in Figure 1.

<5.1. Acoustic signal processing section>
The sound signal processing unit 13 according to this embodiment includes a background sound separation unit 133 in addition to an individual sound separation unit 131 and a clarity adjustment unit 132 .

<5.1.1. Background sound separation section>
The background sound separation unit 133 receives input of the background sound separated by the audio separation unit 12. The background sound separation unit 133 then performs peak detection processing to detect peaks of the background sound. Next, the background sound separation unit 133 performs DoA (Direction Of Arrival) estimation of the sudden sound to estimate the direction from which the sudden sound occurred, and performs sudden sound extraction processing to extract the sudden sound from the background sound. Here, the background sound separation unit 133 performs DoA estimation of the sudden sound by using, for example, technology for visualizing audio or a method using a cross-correlation function of audio signals obtained from the microphone array 30.

The background sound separation unit 133 then outputs information about the estimated position of the sudden sound to the output integration unit 14. The background sound separation unit 133 also outputs the extracted sudden sound to the output integration unit 14.

The background sound separation unit 133 also performs background sound extraction processing to extract sounds other than sudden sounds from the background sound as steady sounds. The background sound separation unit 133 then outputs the extracted steady sounds to the clarity adjustment unit 132. In this way, the background sound separation unit 133 separates the background sound into sudden sounds and steady sounds.

<5.1.2. Clarity adjustment section>
The clarity adjustment unit 132 receives an input of a stationary sound from the background sound separation unit 133. The clarity adjustment unit 132 then performs a degradation process on the stationary sound from the background sound to make it harder to hear. The clarity adjustment unit 132 then outputs the stationary sound from the background sound that has been subjected to clarity adjustment to the output integration unit 14.

<5.2. Output Integration Unit>
The output integrating unit 14 receives input of information about sudden sounds and the positions of the sudden sounds in the background sound from the background sound separation unit 133. The output integrating unit 14 also receives input of steady sounds in the background sound from the clarity adjustment unit 132. In this case, the output integrating unit 14 acquires individual sounds of the same number of channels as the number of people present in the remote space, as well as one channel of data for each of the sudden sounds and steady sounds in the background sound. In other words, if the number of people present in the remote space is p, the output integrating unit 14 acquires (2+p) channels of data.

The output integration unit 14 then integrates each individual sound, the sudden sound from the background sound, and the steady sound from the background sound to generate one piece of audio data. The output integration unit 14 then adds speaker information in the person list corresponding to each individual sound. The output integration unit 14 also associates the sudden sound with positional information of the sudden sound. The output integration unit 14 then outputs the sound data, the person list, and the positional information of the sudden sound to the transmission unit 15, causing it to be transmitted to the local-side remote communication device 1.

<5.3. Audio output control unit>
The audio output control unit 22 in the local remote communication device 1 receives input of audio data including each individual sound, a sudden sound from the background sound, and a steady sound from the background sound. The audio output control unit 22 also receives input of the person list and position information of the sudden sound.

The audio output control unit 22 acquires a sudden sound from the background sound and a steady sound from the background sound from the sound data. Next, the audio output control unit 22 processes the steady sound from the background sound by multiplying the amplitude by (1/u) ^1.66 so that the loudness becomes 1/u. Then, the audio output control unit 22 causes all speaker units 411 of the speaker array 41 and all speaker units 412 of the speaker array 42 to output the steady sounds with reduced loudness.

In contrast, for a sudden sound from the background sound, the audio output control unit 22 identifies the speaker units 411 and 412 that are closest to the position of the sudden sound. Then, the audio output control unit 22 causes the identified speaker units 411 and 412 to reproduce the sudden sound from the background sound as is.

In this example, the audio output control unit 22 applies the same processing to the sudden sound as to the infield speech sound in the first embodiment, and outputs the sound; however, it may also apply the same processing to the sudden sound as to the outfield speech sound in the first embodiment, and output the sound.

6. Data flow between remote communication devices according to the second embodiment
20 is a diagram showing an outline of the data flow between the local remote communication devices according to the second embodiment. Here, too, the case where data is transmitted from the transmitting unit 10 of the remote remote communication device 1 to the receiving unit 20 of the local remote communication device 1 will be described. Here, the transmission of audio will be described.

The microphone 3 inputs the audio picked up in the remote space to the remote communication device 1 (step S301). Here, for example, data from the microphone array 30 is s channel data. The camera 2 also inputs the video generated by capturing the remote space to the remote communication device 1 (step S302).

The video input from camera 2 is sent to the human sensing unit 11. The human sensing unit 11 uses the video captured by camera 2 to perform sensing processing (step S303), including human position determination processing (step S331) and infield/outfield determination processing (step S332).

The audio input from microphone 3 is sent to audio separation unit 12. Audio separation unit 12 performs audio separation processing to separate the speech audio and background sounds contained in the audio input from microphone 3 (step S304).

The background sound extracted by the audio separation unit 12 is sent to the background sound separation unit 133 of the audio signal processing unit 13. The background sound separation unit 133 executes peak detection processing to detect peaks in the background sound (step S305).

Next, the background sound separation unit 133 executes a sudden sound extraction process to estimate the DoA of the sudden sound and extract the sudden sound from the background sound (step S306). The sudden sound in the background sound and its position information are sent to the output integration unit 14.

The background sound separation unit 133 also executes a steady sound extraction process to extract steady sounds from the background sounds (step S307). The steady sounds from the background sounds are sent to the clarity adjustment unit 132.

The clarity adjustment unit 132 performs a degrading process to make the background sound less audible as clarity adjustment processing (step S308).

Furthermore, the individual sounds and person list, which are s-channel data extracted by the audio separation unit 12, are sent to the audio signal processing unit 13. The audio signal processing unit 13 performs speech signal processing (step S309), including individual sound separation processing by the individual sound separation unit 131 (step S391) and clarity adjustment processing by the clarity adjustment unit 132 (step S392).

The steady sound, which is one channel of data, the sudden sound, and the position information of the sudden sound, which is one channel of data, are input to the output integration unit 14. In addition, the individual sounds, which are p channel of data that have been subjected to speech signal processing, are also input to the output integration unit 14. As a result, the output integration unit 14 obtains (2+P) channel data. The output integration unit 14 then executes output integration processing, which integrates the sounds of all channels to generate a single sound data (step S310).

The sound data, person list, and sudden sound location information generated by the output integration unit 14 are sent by the transmission unit 15 to the local remote communication device 1 via the network 7 (step S311).

The local remote communication device 1 receives the sound data, the person list, and the positional information of the sudden sound. The sound data, the person list, and the positional information of the sudden sound are sent to the audio output control unit 22 via the receiving unit 21. The audio output control unit 22 performs a multiple facing speaker balancing process on the sound data using the person list and the positional information of the sudden sound. The audio output control unit 22 then specifies the speaker units 411 and 412 to output for each of the background sound, infield speech sound, and outfield speech sound, and plays them (step S312). At this time, the audio output control unit 22 plays the steady sound of the background sound in all speaker units 411 and 412, adjusting the loudness by dividing it by the number of pairs of speaker units 411 and 412. The audio output control unit 22 also plays the sudden sound of the steady sound as is in the speaker units 411 and 412 closest to the position of the sudden sound.

The speaker arrays 41 and 42 use the speaker units 411 and 412 specified by the audio output control unit 22 to output sudden sounds from the background sound, steady sounds from the background sound, infield speech sounds, and outfield speech sounds (step S313).

<7. Effects>
As described above, the remote communication device 1 according to the present embodiment divides background sounds into two types, sudden sounds and steady sounds, and transmits position information for the background sounds. The remote communication device 1 outputs the sudden sounds among the background sounds without blurring their localization, and outputs the steady sounds among the background sounds with blurring their localization.

This allows background sounds to be played louder than steady sounds from near the location where they were generated, preventing sudden background sounds from losing their sense of position and causing discomfort. This makes conversations more natural and promotes smoother conversations.

8. Remote communication device according to the third embodiment
Next, a remote communication device 1 according to a third embodiment will be described. The remote communication device 1 according to the third embodiment converts the speech voice picked up by the microphone 3 into a mechanical voice to make it easier to hear.

FIG. 21 is a block diagram of a remote communication device according to the third embodiment. Note that in FIG. 21, the same components as those in FIG. 1 are designated by the same reference numerals, and the following description will focus on the components that differ from FIG. 1, and may omit a description of the components that are the same as those in FIG. 1.

8.1. Acoustic signal processing section
The acoustic signal processing unit 13 according to this embodiment includes a voice synthesis unit 134 in addition to an individual sound separation unit 131 and a clarity adjustment unit 132 .

<8.1.1. Speech synthesis unit>
The speech synthesis unit 134 receives input of the individual sounds of each speaker separated by the individual sound separation unit 131. The speech synthesis unit 134 then executes a speech recognition process to recognize speech and transcribe the speech into text. Next, the speech synthesis unit 134 executes a speech synthesis process to synthesize the textual speech into text and regenerate the individual sounds. The speech synthesis unit 134 then outputs the regenerated individual sounds of each speaker to the clarity adjustment unit 132.

In this way, the speech synthesis unit 134 performs speech recognition for each individual sound to generate characters indicating the spoken content of the individual sound, and performs speech synthesis based on the characters indicating the spoken content to regenerate the speaker's speech. In this case, the clarity adjustment unit 132 makes adjustments for each individual sound regenerated by the speech synthesis unit 134 based on whether or not the speaker is participating in the conversation.

9. Data Flow Between Remote Communication Devices According to the Third Embodiment
22 is a diagram showing an outline of the data flow between the local remote communication devices according to the third embodiment. Here, too, the case where data is transmitted from the transmitting unit 10 of the remote remote communication device 1 to the receiving unit 20 of the local remote communication device 1 will be described. Here, the transmission of audio will be described.

The microphone 3 inputs the audio picked up in the remote space to the remote communication device 1 (step S401). Here, for example, data from the microphone array 30 is s channel data. The camera 2 also inputs the video generated by capturing the remote space to the remote communication device 1 (step S402).

The video input from camera 2 is sent to the human sensing unit 11. The human sensing unit 11 uses the video captured by camera 2 to perform sensing processing (step S403), including human position determination processing (step S431) and infield/outfield determination processing (step S432).

The audio input from microphone 3 is sent to audio separation unit 12. Audio separation unit 12 performs audio separation processing to separate the speech audio and background sounds contained in the audio input from microphone 3 (step S404).

The background sound extracted by the audio separation unit 12 is sent to the clarity adjustment unit 132 of the audio signal processing unit 13. The clarity adjustment unit 132 performs a degrading process to make the background sound less audible as clarity adjustment processing (step S405).

Furthermore, the individual sounds and person list, which are s-channel data extracted by the voice separation unit 12, are sent to the acoustic signal processing unit 13. The acoustic signal processing unit 13 performs speech signal processing on the speech using the person list (step S406).

In detail, the individual sound separation unit 131 creates p threads, which is the number of elements registered in the acquired personal list. Furthermore, the individual sound separation unit 131 stores the normalized coordinate values registered in the personal list as thread-specific values for each thread. Next, the individual sound separation unit 131 calculates the angle θ to each speaker. Then, the individual sound separation unit 131 performs individual sound separation processing on the speech sounds extracted by speech separation by the speech separation unit 12 using beamforming according to the angle to each speaker, and generates individual sounds for each speaker for each thread (step S461).

The individual sounds generated by the individual sound separation unit 131 are sent to the speech synthesis unit 134. The speech synthesis unit 134 performs speech recognition processing to recognize the individual sounds and transcribe the speech content into text (step S462).

Next, the speech synthesis unit 134 executes a speech synthesis process to synthesize the text of the speech and regenerate the individual sounds (step S463). The individual sounds of each speaker regenerated by the speech synthesis unit 134 are sent to the clarity adjustment unit 132.

The clarity adjustment unit 132 determines whether the speaker corresponding to the individual sounds for each thread is infield or outfield using the infield flag in the person list. The clarity adjustment unit 132 then performs clarity adjustment processing, which performs enhancement processing to make the individual sounds of infield speakers easier to hear, and degrade processing to make the individual sounds of outfield speakers harder to hear (step S464).

The background sound, which is one channel of data, is input to the output integrating unit 14. The individual sounds, which are p channel data that have been subjected to speech signal processing, are also input to the output integrating unit 14. As a result, the output integrating unit 14 obtains (1+P) channel data. The output integrating unit 14 then executes output integration processing, which integrates the sounds of all channels to generate a single sound data (step S407).

The sound data and person list generated by the output integration unit 14 are sent by the transmission unit 15 to the local remote communication device 1 via the network 7 (step S408).

The local remote communication device 1 receives the sound data and the person list. The sound data and the person list are sent to the audio output control unit 22 via the receiving unit 21. The audio output control unit 22 performs a multiple facing speaker balancing process on the sound data using the person list, and specifies the speaker units 411 and 412 to output the background sound, infield speech sound, and outfield speech sound, respectively, and plays them (step S409).

The speaker arrays 41 and 42 use the speaker units 411 and 412 specified by the audio output control unit 22 to output sudden sounds from the background sound, steady sounds from the background sound, infield speech sounds, and outfield speech sounds (step S410).

<10. Effects>
As described above, the remote communication device 1 according to this embodiment performs speech recognition to transcribe the spoken content into text, and then synthesizes the text into speech data to be played back.

If the speech recorded by the microphone 3 is played back as is, it may be difficult to hear due to differences in voice volume and articulation between people. In contrast, the remote communication device 1 of this embodiment can reduce variations in voice volume and articulation, making the speech easier to hear overall, thereby facilitating smoother communication.

11. Remote communication device according to the fourth embodiment
Next, a remote communication device 1 according to a fourth embodiment will be described. The remote communication device 1 according to the fourth embodiment performs voice recognition, such as infield/outfield determination, using a method other than head direction estimation. The remote communication device 1 according to this embodiment is also represented by the block diagram of FIG. 1. The following description will focus on the parts that are different from FIG. 1, and descriptions of parts that are the same as those in FIG. 1 may be omitted.

<11.1. Human sensing unit>
The human sensing unit 11 performs depth sensing to detect the distance of each speaker from the video. Then, under the assumption that people far from the camera 2 will not converse with remote people, the human sensing unit 11 determines that people far from the camera 2 are people in the outfield. In this way, the human sensing unit 11 determines the distance of the speaker based on the video, and determines whether the speaker is participating in the conversation based on the distance.

In this way, by determining infield and outfield using the distance from camera 2, it is possible to avoid someone passing behind you who happens to be looking at the screen while talking, being identified as a participant in a conversation with a local person. This facilitates smooth communication.

11.2. Acoustic signal processing section
The acoustic signal processing unit 13 performs speech recognition on each individual sound generated by the individual sound separation unit 131 and transcribes the speech content into text. Then, the acoustic signal processing unit 13 determines whether the speech is directed to a person on the local side based on the context of the transcribed speech content. The acoustic signal processing unit 13 can make this determination, for example, by performing AI analysis of emotions, empathy level, and listening level.

If the acoustic signal processing unit 13 determines that the speech is directed at a person on the local side, it determines that the speaker is an infield person. The acoustic signal processing unit 13 then sets the infield flag of the speaker determined to be an infield person in the person list to True, and updates the person list. The acoustic signal processing unit 13 then outputs the updated person list to the clarity adjustment unit 132.

Here, for example, the voice of a person giving a presentation while looking at slides, or the interjections of a person listening while taking notes, are also messages sent to the remote, and are preferably processed as infield. Therefore, by determining infield/outfield based on the context, the speech of a speaker who would be determined to be infield in head direction estimation but is actually participating in the conversation is transmitted to the remote and played back as the speech of an infield person. This makes it possible to process speech that is easier to hear as infield speech, even in situations where the speaker is not facing the remote but is speaking to it. This can facilitate smooth communication.

11.3. Other Voice Recognition
Additionally, when a person on the local side points to a specific position on the screen of the display 6, the local-side human sensing unit 11 extracts the specific position using motion capture, gaze estimation, or the like. Then, the local-side human sensing unit 11 transmits information about the specific position pointed to via the transmission unit 15 to the remote-side remote communication device 1.

In this case, for example, the individual sound separation unit 131 extracts the sound at that specific position as an individual sound using beamforming or the like. The clarity adjustment unit 132 applies appropriate processing to the individual sound at the specific position, such as the same processing as that applied to infield speech. The output integration unit 14 then combines this with the other individual sounds into a single data item, and transmits it, along with information about the specific position, to the local remote communication device 1 via the transmission unit 15.

The audio output control unit 22 of the local remote communication device 1 extracts individual sounds at specific positions from the sound data and performs appropriate multiple opposing speaker balancing processing, such as the same processing as for infield speech, using the information about the specific positions. The audio output control unit 22 then causes the speaker units 411 and 412 to reproduce the processed individual sounds at the specific positions.

<11.4. Effects>
In this way, by emphasizing and reproducing the sound at the specific location pointed at compared to the background sounds, it is possible to make it easier for local people to hear sounds that they want to hear, in addition to spoken voices, thereby facilitating smooth communication.

12. Remote communication device according to the fifth embodiment
Next, a remote communication device 1 according to a fifth embodiment will be described. In the first embodiment, the camera 2 and microphone 3 are aligned in the normal direction of the virtual screen, and the human sensing unit 11 determines the angle of the direction of each speaker based on normalized coordinates indicating the position of each speaker in the image captured by the camera 2. In contrast, the remote communication device 1 according to the fifth embodiment identifies the direction of a sound source using the sound captured by the microphone 3. The remote communication device 1 according to this example is also represented by the block diagram of FIG. 1. The following description will focus on the parts that are different from FIG. 1, and descriptions of parts that are the same as those in FIG. 1 may be omitted.

FIG. 23 is a block diagram of a remote communication device according to the fifth embodiment. Also, FIG. 24 is a diagram for explaining audio signal processing by the remote communication device according to the fifth embodiment.

<12.1. Sound source direction estimation section>
The remote communication device 1 according to this embodiment has a sound source direction estimating unit 16 as shown in Fig. 23. The sound source direction estimating unit 16 acquires sounds obtained by directing a beam in each direction in turn by the microphone array 30. The direction to direct the beam may be preset in the microphone 3, or the sound source direction estimating unit 16 may specify the direction to direct the beam in turn to the microphone 3.

Here, beamforming is assumed to detect only speech and sudden background sounds, and not pick up steady background sounds. Furthermore, it is assumed that no sounds are generated other than the speaker appearing on the screen and any sudden background sounds occurring within that range, i.e., no sound is heard from outside the field of view.

Next, as shown in FIG. 24, the sound source direction estimation unit 16 selects, from the sounds obtained by directing the beam in each direction, a sound whose output is greater than a threshold, and estimates the sound source direction by assuming that the sound source is located in the direction of the beam that obtained the selected sound (step S501). Step S501 in FIG. 24 shows that a large number of sound source directions have been estimated. The sound source direction estimation unit 16 then determines the sound source direction as the position where the speaker is located. In this way, the sound source direction estimation unit 16 determines the position of the speaker based on the sound in a specified space that corresponds to the remote space.

The sound source direction estimation unit 16 then notifies the person sensing unit 11 of the normalized coordinates of the position where the determined speaker is located, and creates a person list based on the normalized coordinates of the position where the speaker is located. In this case, the person sensing unit 11 uses the notified normalized coordinates to identify speakers from among the people shown in the video, performs an infield/outfield determination for each identified speaker, and registers them in the person list.

<12.2. Acoustic signal processing section>
The acoustic signal processing unit 13 acquires, from the person sensing unit 11, a person list in which normalized coordinates of the position of each speaker estimated from the voice by the sound source direction estimation unit 16 are registered. Then, the acoustic signal processing unit 13 executes speech signal processing using the acquired person list (step S502).

In more detail, the individual sound separation unit 131 generates threads for each speaker estimated from the audio registered in the person list. Then, the individual sound separation unit 131 performs individual sound separation processing using the normalized coordinates of the position of each speaker for each thread to generate individual sounds for each speaker (step S521). In this way, the individual sound separation unit 131 separates individual sounds, which are the speech sounds of each speaker, from the audio in a specified space corresponding to the remote space, based on the position of the speaker determined by the sound source direction estimation unit 16.

The individual sound separation unit 131 also separates the individual sounds from the sound source direction detected using the microphone array 30, starting from the angle on the left side of the image displayed on the display 6, thereby matching the individual sounds with the normalized coordinates indicating the position of each speaker registered in the person list. This allows the image and sound to match even after transmission to the local side.

For each thread, the clarity adjustment unit 132 performs clarity adjustment processing for the individual sounds of the speaker estimated from the audio using the infield flag of each speaker registered in the person list (step S522). In this way, the clarity adjustment unit 132 adjusts the clarity for each individual sound separated by the individual sound separation unit 131 based on whether the speaker is participating in the conversation.

<12.3. Effects>
This allows individual sounds to be generated without any restrictions on the positional relationship between the camera 2 and the microphone 3, even if it is practically difficult to install the camera 2 and the microphone 3 side by side in the normal direction of the virtual screen.

13. Remote communication device according to the sixth embodiment
Next, a remote communication device 1 according to a sixth embodiment will be described. When the width of the speaker arrays 41 and 42 is shorter than the width of the screen of the display 6, the remote communication device 1 according to this embodiment remaps the audio playback position to match the width of the display 6. The remote communication device 1 according to this example is also represented by the block diagram of Figure 1. The following description will focus on the parts that are different from Figure 1, and descriptions of parts that are the same as those in Figure 1 may be omitted.

<13.1. Audio output control unit>
25 is a diagram illustrating audio playback by a remote communication device according to a sixth embodiment. In this embodiment, as shown in FIG. 25, the speaker array width W _spk of the speaker arrays 41 and 42 is shorter than the display width W _disp, which is the width of the screen of the display 6.

In the first embodiment, the audio output control unit 22 determines the processing position by dividing the speaker array width W _spk and the speaker distance W _u by the display width W _disp in order to match the image and audio in the normalized coordinate system. However, with this method of multiple opposed speaker balancing processing, if the speaker array width W _spk is shorter than the display width W _disp , it becomes difficult to reproduce the audio of a speaker who appears on the edge of the screen of the display 6.

Therefore, the audio output control unit 22 according to the present embodiment determines the position in processing by dividing the speaker array width W _spk and the speaker distance W _u by the speaker array width W _spk . In this case, the width of the speaker arrays 41 and 42 has a length of 1 in normalized coordinates, and the width between the pair of speaker units 411 and 412 is the value obtained by dividing the speaker distance W _u by the speaker array width W _spk . In other words, the audio output control unit 22 sets the speaker array width in processing to -0.5 to 0.5 in normalized coordinates. Then, the audio output control unit 22 executes a multiple facing speaker balancing process using this distance in processing to determine the pair of speaker units 411 and 412 that will output each sound.

In this way, the audio output control unit 22 selects the speaker units 411 and 412 to play back based on the speaker's position so that the playback positions of the spoken voices in the speaker arrays 41 and 42 maintain the distance relationship between the speaker's position on the screen.

<13.2. Effects>
In this way, the remote communication device 1 according to the present embodiment remaps the audio playback position to match the width of the speaker arrays 41 and 42. This makes it possible to play back the voices of all speakers appearing on the screen. Therefore, even if the width of the speaker arrays 41 and 42 is significantly shorter than the screen of the display 6, it is possible to prevent the voices of speakers speaking at the edges of the screen from being cut off, and all speakers appearing on the screen can be played back from the speaker arrays 41 and 42.

14. Remote communication device according to the seventh embodiment
Next, a remote communication device 1 according to a seventh embodiment will be described. The multiple opposing speakers 4 virtually localize the sound source in the vertical center of the image by reproducing the same monaural sound from both the top and bottom. However, this may sound unnatural when reproducing the speech of a short speaker such as a child or a tall speaker.

Therefore, the remote communication device 1 according to this embodiment plays back audio by changing the vertical position of the sound source according to the height of the sound source. The remote communication device 1 according to this embodiment is represented by the block diagram in Figure 23. The following explanation will focus on the parts that differ from Figure 23, and explanations of parts that are the same as those in Figure 1 may be omitted. However, in this embodiment, the sound source direction estimation unit 16 does not need to estimate the position of each speaker in the normalized coordinate direction. Figure 26 is a diagram for explaining audio playback by a remote communication device according to the seventh embodiment.

<14.1. Human sensing unit>
The human sensing unit 11 acquires the vertical position of the mouth of each speaker shown in the video. For example, the human sensing unit 11 can estimate the position of the mouth through video analysis. Then, the human sensing unit 11 stores the normalized vertical coordinate Y in FIG. 26 for each speaker in the person list. In this case, as shown in FIG. 26, the human sensing unit 11 sets the vertical normalized coordinate Y with the center of the screen as the origin and the vertical coordinate between -0.5 and 0.5.

<14.2. Individual sound separation section>
Furthermore, the source of background sounds other than speech, such as sudden sounds, is not necessarily located at the center of the image. Regarding this, when identifying a sudden sound as in the second embodiment, the background sound separation unit 133 identifies the vertical position of the sudden sound. In this case, for example, the background sound separation unit 133 transmits and receives signals using a signal line connected to the microphone array 30 via the individual sound separation unit 131 and the audio interface 5. The background sound separation unit 133 can estimate the vertical position of the sound source of the sudden sound from the vertical sound intensity by causing the microphone array 30 to scan a beam. The background sound separation unit 133 then represents the position of the sudden sound in the background sound using normalized coordinates Y and transmits this information to the remote communication device 1 via the output integration unit 14 and the transmission unit 15.

Furthermore, while the configuration has been described here in which the background sound separation unit 133 operates the microphone array 30, the remote communication device 1 of the second embodiment may also be equipped with the sound source direction estimation unit 16 shown in FIG. 23. In this case, the vertical position of the sound source estimated by the sound source direction estimation unit 16 using the microphone array 30 is sent to the person sensing unit 11 and registered in the person list.

<14.3. Audio output control unit>
The audio output control unit 22 stereoizes the audio signal, for example, by using the speaker array 41 as the L channel and the speaker array 42 as the R channel. Then, the audio output control unit 22 pans the stereoized audio signal using the L channel and the R channel in accordance with the vertical normalized coordinate Y registered in the person list, and changes the sound source position so that the speech sound is reproduced at a specified vertical position.

For example, the audio output control unit 22 plays back the speech of speaker P1 in FIG. 26 using a sound source located below the center in the vertical direction. Furthermore, the audio output control unit 22 plays back the speech of speaker P1 using a sound source located above the center in the vertical direction.

Furthermore, when a sudden sound in the background sound is reproduced separately from the steady sound as in the second embodiment, the audio output control unit 22 may perform the following process. That is, the audio output control unit 22 also pans the stereo audio signal for the vertical position of the sudden sound, according to the normalized coordinate Y of the vertical position of the sudden sound notified by the individual sound separation unit 131. As a result, the audio output control unit 22 changes the sound source position so that the sudden sound is reproduced at the specified vertical position. In this way, the audio output control unit 22 stereophonizes the sound in a predetermined space that corresponds to the remote space, and adjusts the position of the sound source between the speaker array 41 and the speaker array 42.

<14.4. Effects>
In monaural playback, a virtual sound source is positioned in the vertical center of the screen. In contrast, the remote communication device 1 according to this embodiment can move the sound source position vertically to an appropriate position when there is a difference in the height of speakers or when the sound being emitted is far from the vertical center. This allows for a more natural conversation and facilitates smooth communication.

Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present disclosure. Furthermore, components from different embodiments and modifications may be combined as appropriate.

15. Hardware Configuration
FIG. 27 is a hardware configuration diagram showing an example of a computer that realizes the arithmetic unit of the remote communication device that is the information processing device according to the first to seventh embodiments.

Computer 1000 has a CPU 1100, RAM 1200, ROM (Read Only Memory) 1300, HDD (Hard Disk Drive) 1400, communication interface 1500, and input/output interface 1600. Each part of computer 1000 is connected by bus 1050.

The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400 and controls each component. For example, the CPU 1100 loads programs stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records application programs related to the present disclosure, which are an example of program data 1450.

The communication interface 1500 is an interface that allows the computer 1000 to connect to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

The input/output interface 1600 is an interface for connecting the input/output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard or mouse via the input/output interface 1600. The CPU 1100 also transmits data to output devices such as the display 6, audio interface 5, or printer via the input/output interface 1600. The input/output interface 1600 may also function as a media interface that reads programs and the like recorded on a specified recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

Note that the CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, it may also obtain these programs from another device via the external network 1550.

The above describes in detail preferred embodiments of the present disclosure with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is clear that a person with ordinary skill in the technical field of the present disclosure would be able to conceive of various modified or revised examples within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

Furthermore, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology disclosed herein may achieve other effects in addition to or in place of the above-mentioned effects that would be apparent to those skilled in the art from the description herein.

This technology can also be configured as follows:

(1)
a human sensing unit that determines whether each speaker is participating in a conversation based on images of multiple speakers present in a predetermined space captured by a camera;
an articulation adjustment unit that adjusts the articulation of speech of speakers participating in the conversation determined by the human sensing unit and included in the speech in the predetermined space collected by a microphone;
a transmitting unit that transmits the sound in the predetermined space that has been processed by the clarity adjusting unit.
(2)
the human sensing unit determines the position of each speaker based on the video;
an individual sound separation unit that separates individual sounds, which are speech sounds of each speaker, from the sound in the predetermined space based on the position of each speaker determined by the human sensing unit;
The information processing device according to (1), wherein the clarity adjustment unit adjusts clarity for each of the individual sounds separated by the individual sound separation unit based on whether or not each speaker is participating in a conversation.
(3)
a receiving unit that receives the sound in the predetermined space transmitted by the transmitting unit;
The information processing device described in (2) further includes an audio output control unit that selects, based on the position of each speaker, a speaker unit that will play the speech of each speaker from the audio in the specified space received by the receiving unit, in a speaker having two speaker arrays in which multiple speaker units are arranged in a row, and causes the selected speaker unit to play the speech of each speaker.
(4)
the receiving unit receives the video together with the audio from the predetermined space, and projects the video on a screen on which the speaker arrays are arranged, one at each end of the video in a vertical direction when the video is projected;
The information processing device described in (3), wherein the audio output control unit selects the speaker unit to be played based on the position of each speaker displayed on the screen so that the spoken audio is played near the position on the screen.
(5)
the receiving unit receives the video together with the audio from the predetermined space, and projects the video on a screen on which the speaker arrays are arranged, one at each end of the video in a vertical direction when the video is projected;
The information processing device described in (3), wherein the audio output control unit selects the speaker unit to be played back based on the position of the speaker so that the playback position of the spoken audio in the speaker array maintains the distance relationship between the positions of each speaker on the screen.
(6)
The information processing device described in any one of (1) to (5), wherein the clarity adjustment unit performs a clarity reduction adjustment to reduce the clarity of the speech of a speaker who is not participating in the conversation and is included in the audio of the specified space compared to the state of sound collection by the microphone.
(7)
further comprising an audio separation unit that separates background sounds from the audio in the predetermined space;
The information processing device according to any one of (1) to (6), wherein the clarity adjustment unit performs clarity reduction adjustment to reduce clarity of the background sound separated by the sound separation unit compared to other sounds.
(8)
a background sound separation unit that separates the background sound into a sudden sound and a steady sound,
The information processing device according to (7), wherein the clarity adjustment unit performs the clarity reduction adjustment on the stationary sound among the background sounds.
(9)
a speech synthesis unit that performs speech recognition for each of the individual sounds to generate characters indicating the speech content of the individual sounds, and performs speech synthesis based on the characters indicating the speech content to reproduce the speech of the speaker;
The information processing device according to any one of (2) to (5), wherein the clarity adjustment unit adjusts clarity for each of the individual sounds regenerated by the speech synthesis unit based on whether or not a speaker is participating in a conversation.
(10)
The information processing device according to any one of (1) to (9), wherein the human sensing unit determines the distance of the speaker based on the video and determines whether the speaker is participating in the conversation based on the distance.
(11)
a sound source direction estimation unit that determines the position of a speaker based on the sound in the predetermined space;
an individual sound separation unit separates individual sounds, which are speech sounds of each speaker, from the sound in the predetermined space based on the position of the speaker determined by the sound source direction estimation unit;
The information processing device according to any one of (1) to (10), wherein the clarity adjustment unit adjusts clarity for each of the individual sounds separated by the individual sound separation unit based on whether or not a speaker is participating in a conversation.
(12)
The information processing device according to (3), wherein the audio output control unit stereophonically converts audio in the predetermined space and adjusts the position of a sound source among the speaker arrays.
(13)
The information processing device
a human sensing step of determining whether each speaker is participating in a conversation based on an image captured by a camera capturing multiple speakers present in a predetermined space;
an articulation adjustment step of adjusting the articulation of speech of a speaker participating in the conversation determined in the person sensing step, the speech being included in the audio of the predetermined space collected by a microphone that collects audio of the predetermined space;
a transmitting step of transmitting the sound in the predetermined space processed in the clarity adjusting step.
(14)
Based on the video captured by a camera capturing multiple speakers in a predetermined space, it is determined whether each speaker is participating in the conversation, and the clarity of the speech of the speakers participating in the conversation, which is included in the audio of the predetermined space collected by a microphone that collects the audio of the predetermined space, is adjusted.
An information processing program that causes a computer to execute a process of adjusting the speech of speakers participating in the conversation and transmitting the speech in the predetermined space.
(15)
The transmitting device
a human sensing unit that determines whether each speaker is participating in a conversation based on images of multiple speakers present in a predetermined space captured by a camera;
an articulation adjustment unit that adjusts the articulation of speech of speakers participating in the conversation determined by the human sensing unit and included in the speech in the predetermined space collected by a microphone;
a transmitting unit that transmits the sound in the predetermined space processed by the clarity adjustment unit,
The receiving device
a receiving unit that receives the sound in the predetermined space transmitted by the transmitting unit;
An information processing system comprising: a speaker having two speaker arrays in which a plurality of speaker units are arranged in a row; and an audio output control unit that selects a speaker unit to play back the speech of each speaker from the audio in the specified space received by the receiving unit based on the position of each speaker, and causes the selected speaker unit to play back the speech of each speaker.

REFERENCE SIGNS LIST 1 Remote communication device 2 Camera 3 Microphone 4 Multiple opposing speakers 5 Audio interface 6 Display 7 Network 10 Transmission unit 11 Human sensing unit 12 Voice separation unit 13 Acoustic signal processing unit 14 Output integration unit 15 Transmission unit 16 Sound source direction estimation unit 20 Receiving unit 21 Receiving unit 22 Voice output control unit 30 Microphone array 41, 42 Speaker array 131 Individual sound separation unit 132 Clarity adjustment unit 133 Background sound separation unit 134 Voice synthesis unit 411, 412 Speaker unit

Claims (14)

a human sensing unit that determines whether each speaker is participating in a conversation based on images of multiple speakers present in a predetermined space captured by a camera;
an articulation adjustment unit that adjusts the articulation of speech of speakers participating in the conversation determined by the human sensing unit and included in the speech in the predetermined space collected by a microphone;
a transmitting unit that transmits the sound in the predetermined space that has been processed by the clarity adjusting unit. the human sensing unit determines the position of each speaker based on the video;
an individual sound separation unit that separates individual sounds, which are speech sounds of each speaker, from the sound in the predetermined space based on the position of each speaker determined by the human sensing unit;
The information processing device according to claim 1 , wherein the clarity adjustment unit adjusts clarity for each of the individual sounds separated by the individual sound separation unit based on whether or not each speaker is participating in a conversation. a receiving unit that receives the sound in the predetermined space transmitted by the transmitting unit;
3. The information processing device according to claim 2, further comprising: an audio output control unit that selects, based on the position of each speaker, a speaker unit that will play back the speech of each speaker from the audio in the specified space received by the receiving unit, in a speaker having two speaker arrays in which a plurality of speaker units are arranged in a row, and causes the selected speaker unit to play back the speech of each speaker. the receiving unit receives the video together with the audio from the predetermined space, and projects the video on a screen on which the speaker arrays are arranged, one at each end of the video in a vertical direction when the video is projected;
The information processing device according to claim 3 , wherein the audio output control unit selects the speaker unit to be played back based on the position of each speaker displayed on the screen so that the spoken audio is played back near the position on the screen. the receiving unit receives the video together with the audio from the predetermined space, and projects the video on a screen on which the speaker arrays are arranged, one at each end of the video in a vertical direction when the video is projected;
The information processing device according to claim 3 , wherein the audio output control unit selects the speaker unit to be played back based on the position of the speaker so that the playback position of the speech voice in the speaker array maintains a distance relationship between the positions of the speakers on the screen. The information processing device according to claim 1, wherein the clarity adjustment unit performs clarity reduction adjustment to reduce the clarity of speech of speakers not participating in the conversation included in the audio from the specified space compared to the state of sound pickup by the microphone. further comprising an audio separation unit that separates background sounds from the audio in the predetermined space;
The information processing device according to claim 1 , wherein the clarity adjustment unit performs clarity reduction adjustment on the background sound separated by the sound separation unit to reduce clarity more than other sounds. a background sound separation unit that separates the background sound into a sudden sound and a steady sound,
The information processing device according to claim 7 , wherein the clarity adjustment unit performs the clarity reduction adjustment on the stationary sound of the background sound. a speech synthesis unit that performs speech recognition for each of the individual sounds to generate characters indicating the speech content of the individual sounds, and performs speech synthesis based on the characters indicating the speech content to reproduce the speech of the speaker;
The information processing device according to claim 2 , wherein the clarity adjustment unit adjusts clarity for each of the individual sounds regenerated by the speech synthesis unit based on whether or not a speaker is participating in a conversation. The information processing device described in claim 1, wherein the human sensing unit determines the distance of the speaker based on the video and determines whether the speaker is participating in the conversation based on the distance. a sound source direction estimation unit that determines the position of a speaker based on the sound in the predetermined space;
an individual sound separation unit separates individual sounds, which are speech sounds of each speaker, from the sound in the predetermined space based on the position of the speaker determined by the sound source direction estimation unit;
The information processing device according to claim 1 , wherein the clarity adjustment unit adjusts clarity for each of the individual sounds separated by the individual sound separation unit based on whether or not a speaker is participating in a conversation. The information processing device described in claim 3, wherein the audio output control unit converts the audio from the specified space into stereo and adjusts the position of the sound source between the speaker arrays. The information processing device
a human sensing step of determining whether each speaker is participating in a conversation based on an image captured by a camera capturing multiple speakers present in a predetermined space;
an articulation adjustment step of adjusting the articulation of speech of a speaker participating in the conversation determined in the person sensing step, the speech being included in the audio of the predetermined space collected by a microphone that collects audio of the predetermined space;
a transmitting step of transmitting the sound in the predetermined space processed in the clarity adjusting step. The transmitting device
a human sensing unit that determines whether each speaker is participating in a conversation based on images of multiple speakers present in a predetermined space captured by a camera;
an articulation adjustment unit that adjusts the articulation of speech of speakers participating in a conversation determined by the human sensing unit and included in the speech in the predetermined space collected by a microphone;
a transmitting unit that transmits the sound in the predetermined space processed by the clarity adjustment unit,
The receiving device
a receiving unit that receives the sound in the predetermined space transmitted by the transmitting unit;
An information processing system comprising: a speaker having two speaker arrays in which a plurality of speaker units are arranged in a row; and an audio output control unit that selects a speaker unit to play back the speech of each speaker from the audio in the specified space received by the receiving unit based on the position of each speaker, and causes the selected speaker unit to play back the speech of each speaker.