WO2024154626A1 - Electronic apparatus and program - Google Patents

Abstract

An electronic apparatus according to the present invention comprises an acquisition unit that acquires information about one or more interlocutor candidates, a detection unit that detects the one or more interlocutor candidates on the basis of the information about the one or more interlocutor candidates, and a control unit that performs prescribed processing for one or more interlocutors selected from the one or more interlocutor candidates but does not perform the prescribed processing for interlocutor candidates not selected as interlocutors.

Description

Electronic devices and programs CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to patent application No. 2023-4631, filed in Japan on January 16, 2023, the entire disclosure of which is incorporated herein by reference.

This disclosure relates to electronic devices and programs.

In recent years, so-called remote conferences, such as web conferences or video conferences, have become widespread. In remote conferences, electronic devices (or systems including electronic devices) are used to enable communication between participants in multiple locations. For example, consider a situation in which a conference is held in an office, and at least one of the conference participants joins the remote conference at a remote home. In this case, audio and/or video of the conference in the office is acquired by, for example, an electronic device installed in the office, and transmitted to, for example, an electronic device installed in the participant's home. Also, audio and/or video at the participant's home is acquired by, for example, an electronic device installed in the participant's home, and transmitted to, for example, an electronic device installed in the office. Such electronic devices allow participants to communicate with each other without having to all gather in the same place.

Various technologies that can be applied to remote conferences such as those described above have been proposed. For example, a robot device disclosed in Patent Document 1 learns in advance from image information and finds the face of a familiar person. If it determines that the person is calling out to it, it faces the direction of the sound source. For example, Patent Document 2 proposes a technology that generates behavior patterns according to the relative distance from the user, and generates different behavior patterns for each artificial creature model even when the same external information is given. For example, Patent Document 3 discloses a robot that communicates with the nearest person by recognizing people in the vicinity or surrounding area using RFID technology. For example, Patent Document 4 discloses a communication robot that approaches nearby people and requests permission to identify them, and if permission is granted, identifies them individually and extracts and presents information of common interest.

JP 2003-266351 A JP 2004-66367 A JP 2004-216513 A JP 2007-222968 A

An electronic device (first electronic device) according to an embodiment includes:
An acquisition unit that acquires information regarding at least one interlocutor candidate;
a detection unit that detects the at least one candidate interlocutor based on information about the at least one candidate interlocutor;
a control unit that executes a predetermined process for at least one interlocutor selected from the at least one interlocutor candidate, and does not execute the predetermined process for the interlocutor candidate who is not selected as the interlocutor;
Equipped with.

An electronic device (second electronic device) according to an embodiment includes:
An acquisition unit that acquires information regarding at least one interlocutor candidate;
A selection unit that selects at least one interlocutor from the at least one interlocutor candidate;
a control unit that executes a predetermined process for the at least one interlocutor and does not execute the predetermined process for the interlocutor candidates who are not selected as the interlocutor;
Equipped with.

An electronic device (third electronic device) according to an embodiment includes:
an acquisition unit that acquires information about at least one interlocutor candidate from the first electronic device;
a detection unit that detects the at least one candidate interlocutor based on information about the at least one candidate interlocutor;
a control unit that controls at least one of the first electronic device and the second electronic device so as to execute a predetermined process for at least one interlocutor selected by the second electronic device from the at least one interlocutor candidate, and not to execute the predetermined process for the interlocutor candidate not selected as the interlocutor;
Equipped with.

A program according to an embodiment includes:
On the computer,
obtaining information about at least one potential interlocutor;
detecting the at least one candidate interlocutor based on information about the at least one candidate interlocutor;
executing a predetermined process for at least one interlocutor selected from the at least one interlocutor candidate, and not executing the predetermined process for the interlocutor candidate who is not selected as the interlocutor;
Execute the command.

FIG. 1 is a diagram illustrating an example of a usage mode of a system according to an embodiment. FIG. 2 is a functional block diagram illustrating a schematic configuration of a first electronic device according to an embodiment. 6A and 6B are diagrams illustrating an example of driving by a driving unit of the first electronic device according to an embodiment. FIG. 4 is a functional block diagram illustrating a schematic configuration of a second electronic device according to an embodiment. FIG. 4 is a functional block diagram illustrating a configuration of a third electronic device according to an embodiment. FIG. 2 is a sequence diagram illustrating a basic operation of a system according to an embodiment. FIG. 1 is a diagram illustrating an example of the operation of a system according to an embodiment. FIG. 1 is a diagram illustrating an example of the operation of a system according to an embodiment. FIG. 1 is a diagram illustrating an example of the operation of a system according to an embodiment. FIG. 1 is a diagram illustrating an example of the operation of a system according to an embodiment. FIG. 11 is a sequence diagram illustrating a basic operation of a system according to another embodiment.

In the present disclosure, the term "electronic device" may be, for example, a device that is driven by power supplied from a power system or a battery. In the present disclosure, the term "information processing device" may be, for example, a form of electronic device. In addition, the term "information processing device" may be, for example, any device that performs a predetermined process by a computer executing a program, such as a personal computer (PC), a notebook PC, a server, or a smartphone. In the present disclosure, the term "system" may include, for example, at least an electronic device and/or an information processing device. In the present disclosure, the term "user" may be, for example, a person who uses or may use an electronic device and/or an information processing device according to an embodiment (typically a human), as well as a person who uses or may use a system including an electronic device and/or an information processing device according to an embodiment. In addition, the term "user" may be, for example, a person who may enjoy the benefits of an electronic device, an information processing device, and/or a system according to an embodiment. In the present disclosure, a conference in which at least one participant participates by communication from a location different from the other participants, such as a Web conference or a video conference, is collectively referred to as a "remote conference".

For example, in order to facilitate communication in remote conferences and the like, further improvements in functionality are desired for electronic devices that enable communication. An object of the present disclosure is to provide an electronic device and program that facilitates communication. According to one embodiment, an electronic device and program that facilitates communication can be provided. Below, a system including an electronic device according to one embodiment will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing an example of how the system according to one embodiment is used. The following description assumes a situation in which interlocutor Mg remotely participates in a conference held in a conference room MR from his/her home RL, as shown in FIG. 1. Here, the conference room MR may be a closed space or may be a space such as an open space.

As shown in FIG. 1, in the conference room MR, at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf may participate in the conference as an interlocutor. Here, it is assumed that the interlocutor candidates Ma, Mb, Mc, and Md are seated around the desk in the conference room MR. It is also assumed that the interlocutor candidates Me and Mf are standing in a position slightly away from the desk in the conference room MR (for example, at a position farther away from the desk than the interlocutor candidates Ma, Mb, Mc, and Md). In the conference room MR, the interlocutor candidates who may become participants in the conference are not limited to the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf, and may include, for example, other interlocutor candidates. The interlocutor candidates Me and Mf may be seated. In the conference room MR, the interlocutor candidates may be any number of at least one person. Additionally, interlocutors other than interlocutor Mg may also participate in the conference remotely from their own homes.

In the present disclosure, an "interlocutor" may be a person who is expected to have a dialogue and/or a person who is permitted to have a dialogue, for example, in a situation such as the remote conference shown in FIG. 1. That is, an "interlocutor" may be a person who participates in a conference, for example, in a situation such as the remote conference shown in FIG. 1. Also, an "interlocutor candidate" may be a person who has not yet been permitted or set as the above-mentioned interlocutor, but who can become an interlocutor. That is, an "interlocutor candidate" may be a person who can participate in a conference, for example, in a situation such as the remote conference shown in FIG. 1, or a person who can participate in a conference but has not yet participated in the conference. In short, in one embodiment, an "interlocutor" may be permitted and/or set from among the "interlocutor candidates". Here, a "dialogue" may include, for example, a dialogue between an interlocutor Mg and at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf.

As shown in FIG. 1, the system according to an embodiment may include, for example, a first electronic device 1, a second electronic device 100, and a third electronic device 300. In FIG. 1, the first electronic device 1, the second electronic device 100, and the third electronic device 300 are shown only in schematic form. The system according to an embodiment may not include at least any of the first electronic device 1, the second electronic device 100, and the third electronic device 300, and may include devices other than the electronic devices mentioned above.

The first electronic device 1 according to one embodiment may be installed in the conference room MR. Meanwhile, the second electronic device 100 according to one embodiment may be installed in the home RL of the interlocutor Mg. The first electronic device 1 and the second electronic device 100 may be configured to be able to communicate with each other. The location of the home RL of the interlocutor Mg may be a location different from the location of the conference room MR. The location of the home RL of the interlocutor Mg may be far away from the location of the conference room MR, or may be close to the location of the conference room MR (for example, a room adjacent to the conference room MR). Furthermore, the location of the home RL of the interlocutor Mg may be within the conference room MR.

1, the first electronic device 1 according to an embodiment may be connected to the second electronic device 100 according to an embodiment, for example, via a network N. Also, as shown in FIG. 1, the third electronic device 300 according to an embodiment may be connected to at least one of the first electronic device 1 and the second electronic device 100, for example, via a network N. The first electronic device 1 according to an embodiment may be connected to the second electronic device 100 according to an embodiment, by at least one of wireless and wired. The third electronic device 300 according to an embodiment may be connected to at least one of the first electronic device 1 and the second electronic device 100, by at least one of wireless and wired. In FIG. 1, the first electronic device 1, the second electronic device 100, and the third electronic device 300 are shown by dashed lines as being connected wirelessly and/or wired via the network N. In an embodiment, the first electronic device 1 and the second electronic device 100 may be included in a remote conference system according to an embodiment. Also, the third electronic device 300 may be included in a remote conference system according to an embodiment.

In the present disclosure, the network N as shown in FIG. 1 may include various electronic devices and/or devices such as a server as appropriate. The network N as shown in FIG. 1 may also include devices such as a base station and/or a repeater as appropriate. In the present disclosure, for example, when the first electronic device 1 and the second electronic device 100 "communicate", the first electronic device 1 and the second electronic device 100 may communicate directly. In the present disclosure, for example, when the first electronic device 1 and the second electronic device 100 "communicate", the first electronic device 1 and the second electronic device 100 may communicate via at least one of other devices such as the third electronic device 300, a repeater, and/or a base station. In the present disclosure, for example, when the first electronic device 1 and the second electronic device 100 "communicate", more specifically, the communication unit of the first electronic device 1 and the communication unit of the second electronic device 100 may communicate.

The above-mentioned notation may include the same intention as above not only when the first electronic device 1 and the second electronic device 100 "communicate" with each other, but also when one "sends" information to the other and/or when the other "receives" information sent by one. Furthermore, the above-mentioned notation may include the same intention as above not only when the first electronic device 1 and the second electronic device 100 "communicate" with each other, but also when any electronic device, including the third electronic device 300, communicates with any other electronic device.

The first electronic device 1 according to one embodiment may be arranged in the conference room MR, for example, as shown in FIG. 1. In this case, the first electronic device 1 may be arranged in a position where it can acquire the voice and/or image of at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf. The first electronic device 1 outputs the voice and/or image of the interlocutor Mg, as described later. Therefore, the first electronic device 1 may be arranged so that the voice and/or image of the interlocutor Mg output from the first electronic device 1 reaches at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf. The first electronic device 1 according to one embodiment may acquire information on the gaze of the interlocutor candidates or interlocutors, such as the gaze, the direction of the gaze, and/or the movement of the gaze, of at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf, as described later. The acquisition of gaze information by the first electronic device 1 will be described further below.

The second electronic device 100 according to one embodiment may be arranged in the home RL of the interlocutor Mg, for example, in a manner as shown in FIG. 1. In this case, the second electronic device 100 may be arranged in a position where it is possible to acquire the voice and/or image of the interlocutor Mg. The second electronic device 100 may acquire the voice and/or image of the interlocutor Mg by a microphone, a headset, and/or a camera connected to the second electronic device 100. Furthermore, the second electronic device 100 according to one embodiment may acquire information on the line of sight of the interlocutor Mg, such as the line of sight of the interlocutor Mg, the direction of the line of sight, and/or the movement of the line of sight, as described below. The acquisition of line of sight information by the second electronic device 100 will be described further below.

Furthermore, the second electronic device 100 outputs the voice and/or image of at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf in the conference room MR, as described below. For this reason, the second electronic device 100 may be positioned so that the voice and/or image output from the second electronic device 100 reaches the interlocutor Mg. The voice output from the second electronic device 100 may be positioned so that it reaches the ears of the interlocutor Mg, for example, via headphones, earphones, speakers, or a headset. Furthermore, the image output from the second electronic device 100 may be positioned so that it is visually recognized by the interlocutor Mg, for example, via a display.

The third electronic device 300 may be, for example, a server-like device that relays between the first electronic device 1 and the second electronic device 100. Also, the system according to one embodiment may not include the third electronic device 300.

FIG. 1 shows only one example of a usage mode of the first electronic device 1, the second electronic device 100, and the third embodiment 300 according to an embodiment. The first electronic device 1, the second electronic device 100, and the third embodiment 300 according to an embodiment may be used in various other modes.

The remote conference system including the first electronic device 1 and the second electronic device 100 shown in FIG. 1 allows the interlocutor Mg to behave as if he or she were participating in a conference held in the conference room MR while staying at home RL. Furthermore, this remote conference system allows at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf to feel as if the interlocutor Mg is actually participating in the conference held in the conference room MR. That is, in the remote conference system including the first electronic device 1 and the second electronic device 100, the first electronic device 1 arranged in the conference room MR can play a role like an avatar of the interlocutor Mg. In this case, the first electronic device 1 may function as a physical avatar (such as a telepresence robot or a communication robot) that resembles the interlocutor Mg. Furthermore, the first electronic device 1 may function as a virtual avatar that displays an image of the interlocutor Mg or an image that resembles, for example, a character of the interlocutor Mg on the first electronic device 1. The image of the interlocutor Mg or the image of the interlocutor Mg displayed by the first electronic device 1 may be, for example, a display provided in the first electronic device 1 itself, an external display, or a 3D hologram projected by the first electronic device 1.

Next, the functional configurations of the first electronic device 1, the second electronic device 100, and the third electronic device 300 according to one embodiment will be described.

FIG. 2 is a block diagram showing a schematic configuration of the functions of the first electronic device 1 shown in FIG. 1. An example of the configuration of the first electronic device 1 according to one embodiment will be described below. As shown in FIG. 1, the first electronic device 1 may be used in the conference room MR by at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf. The second electronic device 100 described later has a function of outputting the voice, video, and/or gaze information of the interlocutor Mg acquired by the second electronic device 100 to the first electronic device 1 when the interlocutor Mg speaks. In addition, the first electronic device 1 has a function of outputting the voice and/or video of at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf acquired by the first electronic device 1 to the second electronic device 100 when the interlocutor speaks. The first electronic device 1 allows at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf to hold a remote conference or video conference in the conference room MR even if the interlocutor Mg is in a remote location. Therefore, the first electronic device 1 is also referred to as an electronic device that is "used locally" as appropriate.

The first electronic device 1 according to one embodiment may be configured to reproduce the direction of gaze of interlocutor Mg. That is, the first electronic device 1 may be capable of performing an operation that simulates the direction of gaze of interlocutor Mg. Specifically, the first electronic device 1 can cause interlocutor candidates Ma, Mb, Mc, Md, Me, Mf, and the like in the conference room MR to recognize in which direction interlocutor Mg is looking. For example, the first electronic device 1 can cause people around the first electronic device 1 in the conference room MR to recognize whether interlocutor Mg is looking at interlocutor Ma, whether interlocutor Mg is looking at interlocutor Mb, or whether interlocutor Mg is not looking at any of the other interlocutors.

The first electronic device 1 according to one embodiment may be various devices, but may be, for example, a device designed specifically for the device. For example, the first electronic device 1 according to one embodiment may have a housing with an exterior on which an illustration of a human or the like is drawn, or may have a doll-like or robot-like shape that imitates at least a part of a human or the like. The first electronic device 1 according to one embodiment may be, for example, a general-purpose smartphone, tablet, phablet, notebook computer (notebook PC or laptop), or computer (desktop). The first electronic device 1 according to one embodiment may draw at least a part of an image of a human or robot on, for example, the display of a notebook PC. The first electronic device 1 according to one embodiment may project at least a part of a human or robot as a 3D hologram. For example, when the first electronic device 1 has a shape like a robot, the direction of the gaze of the interlocutor Mg may be simulated by the movement of the eyes and/or head of the robot. For example, when the first electronic device 1 is configured to include a display that draws an image of a robot, the direction of the gaze of the interlocutor Mg may be simulated by the movement of the eyes and/or head of the drawn image of the robot.

As shown in FIG. 2, the first electronic device 1 according to one embodiment may include a control unit 10, a memory unit 20, a communication unit 30, an imaging unit 40, an audio input unit 50, an audio output unit 60, a display unit 70, a drive unit 80, an input unit 90, and a gaze information acquisition unit 92. The control unit 10 may also include, for example, an acquisition unit 12, a detection unit 14, a selection unit 16, and an identification unit 18. In one embodiment, the first electronic device 1 may not include at least some of the functional units shown in FIG. 2, or may include components other than the functional units shown in FIG. 2.

The control unit 10 may have the function of controlling and/or managing the entire first electronic device 1, including each functional unit constituting the first electronic device 1. The control unit 10 may include at least one processor, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), to provide control and processing power for executing various functions. The control unit 10 may be realized as a single processor, or as several processors, or as individual processors. The processor may be realized as a single integrated circuit (IC). The processor may be realized as multiple communicatively connected integrated circuits and discrete circuits. The processor may be realized based on various other known technologies.

The control unit 10 may include one or more processors and memories. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 10 may be either a system-on-a-chip (SoC) or a system in a package (SiP) in which one or more processors work together. The control unit 10 controls the operation of each component of the first electronic device 1.

The control unit 10 may be configured to include, for example, at least one of software and hardware resources. Furthermore, in the first electronic device 1 according to one embodiment, the control unit 10 may be configured by specific means in which software and hardware resources work together. Furthermore, in the first electronic device 1 according to one embodiment, at least one of the other functional units may also be configured by specific means in which software and hardware resources work together.

In the first electronic device 1 according to one embodiment, the control unit 10 performs various operations such as control, which will be described later. The acquisition unit 12 of the control unit 10 can perform various acquisition processes. The detection unit 14 can perform various detection processes. The selection unit 16 can perform various selection processes. The identification unit 14 can perform various identification processes. The operations performed by each of these functional units will be described later.

The storage unit 20 may function as a memory that stores various information. The storage unit 20 may store, for example, a program executed in the control unit 10 and the results of processing executed in the control unit 10. The storage unit 20 may also function as a work memory for the control unit 10. As shown in FIG. 2, the storage unit 20 may be connected to the control unit 10 by wire and/or wirelessly. The storage unit 20 may include, for example, at least one of a RAM (Random Access Memory) and a ROM (Read Only Memory). The storage unit 20 may be configured, for example, by a semiconductor memory or the like, but is not limited to this, and may be any storage device. For example, the storage unit 20 may be a storage medium such as a memory card inserted into the first electronic device 1 according to one embodiment. The storage unit 20 may also be an internal memory of a CPU used as the control unit 10, or may be connected to the control unit 10 as a separate unit.

The communication unit 30 has an interface function for wireless and/or wired communication with, for example, an external device. The communication method performed by the communication unit 30 in one embodiment may be a wireless communication standard. For example, the wireless communication standard includes cellular phone communication standards such as 2G, 3G, 4G, and 5G. For example, the cellular phone communication standards include LTE (Long Term Evolution), W-CDMA (Wideband Code Division Multiple Access), CDMA2000, PDC (Personal Digital Cellular), GSM (Registered Trademark) (Global System for Mobile communications), and PHS (Personal Handy-phone System), etc. For example, wireless communication standards include WiMAX (Worldwide Interoperability for Microwave Access), IEEE 802.11, WiFi, Bluetooth (registered trademark), IrDA (Infrared Data Association), and NFC (Near Field Communication). The communication unit 30 may include, for example, a modem whose communication method is standardized by ITU-T (International Telecommunication Union Telecommunication Standardization Sector). The communication unit 30 can support one or more of the above communication standards.

The communication unit 30 may be configured to include, for example, an antenna for transmitting and receiving radio waves and an appropriate RF unit. The communication unit 30 may wirelessly communicate with, for example, a communication unit of another electronic device via an antenna. The communication unit 30 may have a function of transmitting any information from the first electronic device 1 to another device, and/or a function of receiving any information from another device in the first electronic device 1. For example, the communication unit 30 may wirelessly communicate with the second electronic device 100 shown in FIG. 1. In this case, the communication unit 30 may wirelessly communicate with a communication unit 130 (described later) of the second electronic device 100. Thus, in one embodiment, the communication unit 30 has a function of communicating with the second electronic device 100. Also, for example, the communication unit 30 may wirelessly communicate with the third electronic device 300 shown in FIG. 1. In this case, the communication unit 30 may wirelessly communicate with a communication unit 330 (described later) of the third electronic device 300. Thus, in one embodiment, the communication unit 30 may have a function of communicating with the third electronic device 300. The communication unit 30 may also be configured as an interface such as a connector for wired connection to the outside. The communication unit 30 can be configured using known technology for wireless communication, so a detailed description of the hardware and the like is omitted.

As shown in FIG. 2, the communication unit 30 may be connected to the control unit 10 via a wired and/or wireless connection. Various information received by the communication unit 30 may be supplied to, for example, the storage unit 20 and/or the control unit 10. Various information received by the communication unit 30 may be stored in, for example, a memory built into the control unit 10. Furthermore, the communication unit 30 may transmit, for example, the results of processing by the control unit 10 and/or information stored in the storage unit 20 to the outside.

The imaging unit 40 may be configured to include an image sensor that captures images electronically, such as a digital camera. The imaging unit 40 may be configured to include an imaging element that performs photoelectric conversion, such as a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging unit 40 can capture an image of the surroundings of the first electronic device 1, for example. The imaging unit 40 may capture an image of the inside of the conference room MR shown in FIG. 1, for example. In one embodiment, the imaging unit 40 may capture images of potential interlocutors Ma, Mb, Mc, Md, Me, and Mf of a conference held in the conference room MR shown in FIG. 1, for example.

The imaging unit 40 may be configured to capture an image having a predetermined range of angle of view centered on a specific direction. For example, the imaging unit 40 according to one embodiment may capture an image centered on interlocutor candidate Mb in FIG. 1, and in which interlocutor candidate Ma and/or interlocutor candidate Md are not included in the angle of view. The imaging unit 40 may also be configured to simultaneously capture images in all directions (e.g., 360 degrees), such as the horizontal direction. For example, the imaging unit 40 according to one embodiment may capture an omnidirectional image in FIG. 1 that includes interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf.

The imaging unit 40 may convert the captured image into a signal and transmit it to the control unit 10. For this reason, the imaging unit 40 may be connected to the control unit 10 via a wired and/or wireless connection. Furthermore, a signal based on the image captured by the imaging unit 40 may be supplied to any functional unit of the first electronic device 1, such as the memory unit 20 and/or the display unit 70. The imaging unit 40 is not limited to an imaging device such as a digital camera, and may be any device that captures an image of the state inside the conference room MR shown in FIG. 1.

In one embodiment, the imaging unit 40 may capture images of the state inside the conference room MR as still images at predetermined time intervals (e.g., 15 frames per second). Also, in one embodiment, the imaging unit 40 may capture images of the state inside the conference room MR as a continuous video. Furthermore, the imaging unit 40 may be configured to include a fixed camera, or may be configured to include a movable camera.

The voice input unit 50 detects (acquires) sounds or voices around the first electronic device 1, including human voices. For example, the voice input unit 50 may convert sounds or voices detected as air vibrations, for example, by a diaphragm, into an electrical signal. Specifically, the voice input unit 50 may include an acoustic device that converts sounds into an electrical signal, such as a microphone. In one embodiment, the voice input unit 50 may detect (acquire) the voices of at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf in the conference room MR shown in FIG. 1, for example. The voices (electrical signals) detected by the voice input unit 50 may be input to the control unit 10, for example. For this reason, the voice input unit 50 may be connected to the control unit 10 by wire and/or wirelessly.

In one embodiment, the audio input unit 50 may be configured to include, for example, a stereo microphone or a microphone array. An audio input unit 50 including multiple channels, such as a stereo microphone or a microphone array, can identify (or estimate) the direction and/or position of a sound source. With such an audio input unit 50, it can be identified (or estimated) from which direction and/or position a sound detected in, for example, a conference room MR originates, based on the first electronic device 1 equipped with the audio input unit 50.

The audio input unit 50 may convert the acquired sound or voice into an electrical signal and supply it to the control unit 10. The audio input unit 50 may also supply the electrical signal (audio signal) into which the sound or voice has been converted to a functional unit of the first electronic device 1, such as the memory unit 20. The audio input unit 50 may be any device that detects (acquires) sound or voice within the conference room MR shown in FIG. 1.

The audio output unit 60 converts an electrical signal (audio signal) of sound or voice supplied from the control unit 10 into sound, and outputs the audio signal as sound or voice. The audio output unit 60 may be connected to the control unit 10 by wire and/or wirelessly. The audio output unit 60 may be configured to include a device having a function of outputting sound, such as an arbitrary speaker (loudspeaker). In one embodiment, the audio output unit 60 may be configured to include a directional speaker that transmits sound in a specific direction. The audio output unit 60 may also be configured to be able to change the directionality of the sound. The audio output unit 60 may include an amplifier or an amplification circuit that appropriately amplifies the electrical signal (audio signal).

In one embodiment, the audio output unit 60 may amplify the audio signal that the communication unit 30 receives from the second electronic device 100. Here, the audio signal received from the second electronic device 100 may be, for example, the audio signal of a speaker (e.g., interlocutor Mg shown in FIG. 1) who is speaking (currently speaking) that is received by the communication unit 30 from the second electronic device 100 of that speaker. In other words, the audio output unit 60 may output the audio signal of a speaker (e.g., interlocutor Mg shown in FIG. 1) as the voice of that speaker.

The display unit 70 may be any display device, such as a Liquid Crystal Display (LCD), an Organic Electro-Luminescence panel, or an Inorganic Electro-Luminescence panel. The display unit 70 may also be, for example, a projector that projects a 3D hologram. The display unit 70 may display various types of information, such as characters, figures, or symbols. The display unit 70 may also display objects and/or icon images that constitute various GUIs, for example, to prompt the user to operate the first electronic device 1.

The display unit 70 may be, for example, a touch screen display equipped with a touch panel function that detects input by contact with a finger or stylus of at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf.

Various data necessary for display on the display unit 70 may be supplied, for example, from the control unit 10 or the memory unit 20. For this reason, the display unit 70 may be connected to the control unit 10 or the like by wire and/or wirelessly. Furthermore, when the display unit 70 includes, for example, an LCD, it may be configured to include a backlight, etc., as appropriate.

In one embodiment, the display unit 70 may display an image based on a video signal transmitted from the second electronic device 100. As described later, the second electronic device 100 acquires, for example, the voice, video, and/or gaze information of the interlocutor Mg shown in FIG. 1 and outputs it to the first electronic device 1. The control unit 10 of the first electronic device 1 may then display, on the display unit 70, a video and/or image based on the information acquired from the second electronic device 100. For example, the display unit 70 may display an image representing the gaze direction of the interlocutor Mg based on the video and/or gaze information of the interlocutor Mg input from the control unit 10. By displaying an image representing the gaze direction of the interlocutor Mg on the display unit 70 of the first electronic device 1, for example, interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf shown in FIG. 1 can visually know the gaze state of the interlocutor Mg who is located away from the conference room MR.

The display unit 70 may directly display an image of the interlocutor Mg captured by the second electronic device 100, for example. On the other hand, the display unit 70 may display an image of a character (e.g., the gaze of an avatar or robot) that represents the direction of the gaze of the interlocutor Mg, for example. The display unit 70 may represent the gaze direction of the user of the second electronic device 100 by an image. The display unit 70 may also represent the gaze direction and/or gaze movement of the user of the second electronic device 100 by an image. In this way, the first electronic device 1 according to one embodiment may include a display unit 70 that represents the gaze and/or gaze direction of the user of the second electronic device 100 by an image.

The driving unit 80 drives a specific moving part in the first electronic device 1. The driving unit 80 may be configured to include a power source such as a servo motor that drives any moving part in the first electronic device 1. The driving unit 80 may drive any moving part in the first electronic device 1 under the control of the control unit 10. For this reason, the driving unit 80 may be connected to the control unit 10 by wire and/or wirelessly.

In one embodiment, the driving unit 80 may drive, for example, at least a part of the housing of the first electronic device 1. Furthermore, for example, when the first electronic device 1 has a shape like a doll imitating at least a part of a human or the like, or a shape like a robot, the driving unit 80 may drive at least a part of a doll or a robot. In particular, when the first electronic device 1 has a shape imitating at least a part of a human face or a shape like a robot face, the driving unit 80 may express the line of sight, line of sight direction, and/or line of sight movement of the interlocutor Mg by the physical configuration (shape) and/or movement of the doll or robot.

As described below, the second electronic device 100 acquires, for example, the voice, video, and/or gaze information of interlocutor Mg shown in FIG. 1 (by the gaze information acquisition unit 192) and outputs it to the first electronic device 1. The drive unit 80 may represent the gaze of the image of interlocutor Mg by a physical configuration (shape) and/or movement based on the video and/or gaze information of interlocutor Mg input from the first electronic device 1, as shown in FIG. 1. By the drive unit 80 of the first electronic device 1 representing the gaze of interlocutor Mg, for example, interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf shown in FIG. 1 can visually know the state of the gaze of interlocutor Mg who is in a location away from the conference room MR.

The driving unit 80 may reproduce, for example, the direction and/or movement of the gaze of the interlocutor Mg captured by the second electronic device 100 as is. On the other hand, the driving unit 80 may express, for example, the direction and/or movement of the gaze of the interlocutor Mg by the shape of a doll or robot possessed by the first electronic device 1. The driving unit 80 may express the gaze, the direction of the gaze, and/or the movement of the gaze of the user of the second electronic device 100 by a physical configuration (shape) and/or movement. As an example, when the first electronic device 1 has the shape of a robot, the direction and/or movement of the gaze of the user of the second electronic device 100 may be expressed by moving the eyes and/or the neck of the robot. In this way, the first electronic device 1 according to one embodiment may include a driving unit 80 that expresses the gaze and/or the direction of the gaze of the user of the second electronic device 100 by driving a mechanical structure.

FIG. 3 is a diagram illustrating an example of the operation of the driving unit 80 in the first electronic device 1 according to one embodiment.

As shown in FIG. 3, in one embodiment, the driving unit 80 may realize driving about at least one of the driving axes α, β, γ, δ, ε, and ζ in the first electronic device 1 having the shape of a doll or robot. For example, the driving unit 80 may express a negative movement (shaking the head from side to side) of the user of the second electronic device 100 (e.g., interlocutor Mg) by performing driving about the driving axis α in the first electronic device 1. Also, for example, the driving unit 80 may express a positive movement (nodding movement) of the user of the second electronic device 100 (e.g., interlocutor Mg) by performing driving about the driving axis β in the first electronic device 1. Also, for example, the driving unit 80 may express an undecided movement (tilting the head) of the user of the second electronic device 100 (e.g., interlocutor Mg) by performing driving about the driving axis γ in the first electronic device 1. Also, for example, the driving unit 80 may express a negative or rejection behavior (such as shaking the body from side to side) of the user of the second electronic device 100 (e.g., interlocutor Mg) by performing driving about the driving axis δ in the first electronic device 1. Also, for example, the driving unit 80 may express a polite behavior (such as bowing) of the user of the second electronic device 100 (e.g., interlocutor Mg) by performing driving about the driving axis ε in the first electronic device 1. Also, for example, the driving unit 80 may express a behavior of the user of the second electronic device 100 (e.g., interlocutor Mg) by performing driving about the driving axis ζ in the first electronic device 1.

In addition, in one embodiment, the driving unit 80 may express the movement of the eye E1 and/or the eye E2 in the face portion Fc of the first electronic device 1 shown in FIG. 3, that is, the line of sight of the user of the second electronic device 100 (e.g., the interlocutor Mg). In this case, the driving unit 80 may express the line of sight of the user of the second electronic device 100 (e.g., the interlocutor Mg) by driving at least one of the eye E1 and the eye E2 in the face portion Fc of the first electronic device 1. In one embodiment, the driving unit 80 may express the line of sight of the user of the second electronic device 100 (e.g., the interlocutor Mg) by driving the movement of at least one of the eye E1 and the eye E2 in the face portion Fc of the first electronic device 1. Specifically, the driving unit 80 may express the line of sight of the user of the second electronic device 100 (e.g., the interlocutor Mg) by, for example, moving at least one of the eye E1 and the eye E2 in the face portion Fc of the first electronic device 1 in any direction of the arrows shown in FIG. 3. The direction in which the driving unit 80 moves at least one of the eyes E1 and E2 in the face portion Fc of the first electronic device 1 is not limited to any of the directions of the arrows shown in Fig. 3. For example, the driving unit 80 may move at least one of the eyes E1 and E2 in the face portion Fc of the first electronic device 1 in a diagonal direction other than any of the directions of the arrows shown in Fig. 3.

In one embodiment, the display unit 70 may represent the gaze of the user of the second electronic device 100 (e.g., interlocutor Mg) by displaying, for example, the eye E1 and/or the eye E2 in the face portion Fc shown in FIG. 3. In one embodiment, at least one of the display unit 70 and the drive unit 80 may represent the gaze of the user of the second electronic device 100 (e.g., interlocutor Mg) by displaying at least one of the eye E1 and the eye E2 of the first electronic device 1.

As described above, various operations expressing the emotions and/or behavior of a human being, such as the interlocutor Mg, can be expressed by displaying the display unit 70 and/or driving the drive unit 80. Various known technologies may be used for the operations expressing the emotions and/or behavior of a human being, such as the interlocutor Mg, by displaying the display unit 70 and/or driving the drive unit 80. For this reason, a detailed description of the operations expressing the emotions and/or behavior of a human being, such as the interlocutor Mg, by displaying the display unit 70 and/or driving the drive unit 80 will be omitted. The first electronic device 1 according to one embodiment can perform various operations expressing the emotions and/or behavior of the interlocutor Mg by displaying the display unit 70 and/or driving the drive unit 80.

The input unit 90 shown in FIG. 2 may be configured to include any device for detecting input by a user of the first electronic device 1. For example, the input unit 90 may be configured to include at least one of various switches, various sliders, various faders, a joystick, a pad, a keyboard, a mouse, a trackball, and a touch panel. The input unit 90 may use various known technologies, so a more detailed description of the hardware, etc. will be omitted. In one embodiment, the input unit 90 may detect input by interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf shown in FIG. 1, for example.

The gaze information acquisition unit 92 acquires gaze information of the user of the first electronic device 1 (e.g., at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf). The gaze information acquisition unit 92 may acquire gaze information of the user of the first electronic device 1, such as the gaze of the user of the first electronic device 1, the direction of the gaze, and/or the movement of the gaze. The gaze information acquisition unit 92 may have a function of tracking the movement of the gaze of the user of the first electronic device 1 (e.g., at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf), such as an eye tracker. The gaze information acquisition unit 92 may be any component capable of acquiring gaze information of the user of the first electronic device 1, such as the gaze of the user of the first electronic device 1, the direction of the gaze, and/or the movement of the gaze.

The first electronic device 1 according to one embodiment may acquire gaze information of a user of the first electronic device 1 (e.g., at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf) based on the eye movement of the user captured by the imaging unit 40. In this case, the first electronic device 1 may not include a gaze information acquisition unit 92, and the imaging unit 40 may also function as the gaze information acquisition unit 92. The gaze information acquired by the gaze information acquisition unit 92 may be input to the control unit 10, for example. For this reason, the gaze information acquisition unit 92 may be connected to the control unit 10 via a wired and/or wireless connection.

In one embodiment, the first electronic device 1 may be a dedicated device as described above. Meanwhile, in one embodiment, the first electronic device 1 may include at least one of the functional units shown in FIG. 2, such as the audio output unit 60, the drive unit 80, the input unit 92, and the gaze information acquisition unit 92. In this case, the first electronic device 1 may be connected to another electronic device to supplement at least a part of the functions of the other functional units shown in FIG. 2. Here, the other electronic device may be, for example, a general-purpose smartphone, tablet, phablet, notebook computer (notebook PC or laptop), or computer (desktop).

The manner in which various actions expressing the emotions and/or behavior of a human being such as interlocutor Mg are expressed by the display unit 70 and/or the drive unit 80 in the first electronic device 1 shown in FIG. 3 may be merely considered as examples that can be envisioned. The first electronic device 1 according to one embodiment may express various actions expressing the emotions and/or behavior of a human being such as interlocutor Mg by using various configurations and/or operating modes.

FIG. 4 is a block diagram showing a schematic configuration of the second electronic device 100 shown in FIG. 1. An example of the configuration of the second electronic device 100 according to an embodiment will be described below. As shown in FIG. 1, the second electronic device 100 may be, for example, a device used by the interlocutor Mg at his/her home RL. The above-mentioned first electronic device 1 has a function of outputting the voice and/or image of at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf acquired by the first electronic device 1 to the second electronic device 100 when at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf speaks. The first electronic device 1 can express the gaze of the interlocutor Mg. In addition, the second electronic device 100 has a function of outputting the voice and/or image of the interlocutor Mg acquired by the second electronic device 100 to the first electronic device 1 when the interlocutor Mg speaks. Furthermore, the second electronic device 100 has a function of outputting the gaze information of the interlocutor Mg acquired by the second electronic device 100 to the first electronic device 1. The second electronic device 100 allows the interlocutor Mg to hold a remote conference or a video conference even when the interlocutor Mg is in a location away from the conference room MR. Therefore, the second electronic device 100 is also referred to as an electronic device "used remotely" as appropriate.

As shown in FIG. 4, the second electronic device 100 according to one embodiment may include a control unit 110, a memory unit 120, a communication unit 130, an imaging unit 140, an audio input unit 150, an audio output unit 160, a display unit 170, an input unit 190, and a gaze information acquisition unit 192. The control unit 110 may also include, for example, an acquisition unit 112, a detection unit 114, a selection unit 116, and an identification unit 118. In one embodiment, the second electronic device 100 may not include at least some of the functional units shown in FIG. 4, or may include components other than the functional units shown in FIG. 4.

The control unit 110 may have the function of controlling and/or managing the entire second electronic device 100, including each functional unit constituting the second electronic device 100. The control unit 110 may basically be configured based on the same concept as the control unit 10 shown in FIG. 2, for example. The acquisition unit 112, detection unit 114, selection unit 116, and identification unit 118 of the control unit 110 may also be configured based on the same concept as the acquisition unit 12, detection unit 14, selection unit 16, and identification unit 18 of the control unit 10 shown in FIG. 2, for example.

The storage unit 120 may function as a memory that stores various types of information. The storage unit 120 may store, for example, programs executed in the control unit 110 and results of processing executed in the control unit 110. The storage unit 120 may also function as a work memory for the control unit 110. As shown in FIG. 4, the storage unit 120 may be connected to the control unit 110 via a wired and/or wireless connection. The storage unit 120 may basically be configured based on the same concept as the storage unit 20 shown in FIG. 2, for example.

The communication unit 130 has an interface function for wireless and/or wired communication. The communication unit 130 may wirelessly communicate with, for example, a communication unit of another electronic device, for example, via an antenna. For example, the communication unit 130 may wirelessly communicate with the first electronic device 1 shown in FIG. 1. In this case, the communication unit 130 may wirelessly communicate with the communication unit 30 of the first electronic device 1. In this way, in one embodiment, the communication unit 130 has a function of communicating with the first electronic device 1. Also, for example, the communication unit 130 may wirelessly communicate with the third electronic device 300 shown in FIG. 1. In this case, the communication unit 130 may wirelessly communicate with the communication unit 330 (described later) of the third electronic device 300. In this way, in one embodiment, the communication unit 130 may have a function of communicating with the third electronic device 300. As shown in FIG. 4, the communication unit 130 may be connected to the control unit 110 in a wired and/or wireless manner. The communication unit 130 may basically have a configuration based on the same idea as the communication unit 30 shown in FIG. 2, for example.

The imaging unit 140 may be configured to include an image sensor that electronically captures images, such as a digital camera. The imaging unit 140 may capture images of the interior of the home RL shown in FIG. 1, for example. In one embodiment, the imaging unit 140 may capture images of the interlocutor Mg who participates in the conference from the home RL shown in FIG. 1, for example. The imaging unit 140 may convert the captured image into a signal and transmit it to the control unit 110. For this reason, the imaging unit 140 may be connected to the control unit 110 by wire and/or wirelessly. The imaging unit 140 may basically be configured based on the same concept as the imaging unit 40 shown in FIG. 2, for example.

The voice input unit 150 detects (acquires) sounds or voices around the second electronic device 100, including human voices. For example, the voice input unit 150 may convert sounds or voices detected as air vibrations, for example, by a diaphragm, into an electrical signal. Specifically, the voice input unit 150 may include an acoustic device that converts sounds into an electrical signal, such as an arbitrary microphone. In one embodiment, the voice input unit 150 may detect (acquire) the voice of the interlocutor Mg in the home RL shown in FIG. 1, for example. The voice (electrical signal) detected by the voice input unit 150 may be input to the control unit 110, for example. For this reason, the voice input unit 150 may be connected to the control unit 110 by wire and/or wirelessly. The voice input unit 150 may basically be configured based on the same concept as the voice input unit 50 shown in FIG. 2, for example.

The audio output unit 160 converts an electric signal (audio signal) supplied from the control unit 110 into sound, and outputs the audio signal as sound or voice. The audio output unit 160 may be connected to the control unit 110 by wire and/or wirelessly. The audio output unit 160 may be configured to include a device having a function of outputting sound, such as an arbitrary speaker (loudspeaker). In one embodiment, the audio output unit 160 may output a voice detected by the audio input unit 50 of the first electronic device 1. Here, the voice detected by the audio input unit 50 of the first electronic device 1 may be at least one of the voices of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf in the conference room MR shown in FIG. 1. The audio output unit 160 may basically be configured based on the same idea as the audio output unit 60 shown in FIG. 2, for example.

The display unit 170 may be any display device, such as a Liquid Crystal Display (LCD), an Organic Electro-Luminescence panel, or an Inorganic Electro-Luminescence panel. The display unit 170 may basically be configured based on the same concept as the display unit 70 shown in FIG. 2, for example. Various data required for display on the display unit 170 may be supplied from, for example, the control unit 110 or the memory unit 120. For this reason, the display unit 170 may be connected to the control unit 110, etc., via a wired and/or wireless connection.

The display unit 170 may be, for example, a touch screen display equipped with a touch panel function that detects input by contact with the interlocutor Mg's finger or stylus.

In one embodiment, the display unit 170 may display an image based on the video signal transmitted from the first electronic device 1. The display unit 170 may display images of interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf, etc., captured by the first electronic device 1 (its imaging unit 40), as an image based on the video signal transmitted from the first electronic device 1. By displaying images of interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf, etc. on the display unit 170 of the second electronic device 100, for example, interlocutor Mg shown in FIG. 1 can visually know the state of the interlocutor candidates in a conference room MR away from their home RL.

The display unit 170 may directly display images of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf, for example, captured by the first electronic device 1. On the other hand, the display unit 170 may display images (e.g., avatars) that characterize the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf, for example.

The input unit 190 may be configured to include any device for detecting an input by a user of the second electronic device 100. For example, the input unit 190 may be configured to include at least one of various switches, various sliders, various faders, a joystick, a pad, a keyboard, a mouse, a trackball, and a touch panel. In one embodiment, the input unit 190 may detect, for example, an input by the interlocutor Mg shown in FIG. 1.

The gaze information acquisition unit 192 acquires gaze information of the user of the second electronic device 100 (e.g., interlocutor Mg). The gaze information acquisition unit 192 may acquire gaze information of the user of the second electronic device 100, such as the gaze of the user of the second electronic device 100, the direction of the gaze, and/or the movement of the gaze. The gaze information acquisition unit 192 may have a function of tracking the movement of the gaze of the user of the second electronic device 100 (e.g., interlocutor Mg), such as an eye tracker. The gaze information acquisition unit 192 may be any component capable of acquiring gaze information of the user of the second electronic device 100, such as the gaze of the user of the second electronic device 100, the direction of the gaze, and/or the movement of the gaze.

The second electronic device 100 according to one embodiment may acquire gaze information of a user (e.g., interlocutor Mg) of the second electronic device 100 based on the eye movement of the user captured by the imaging unit 140. In this case, the second electronic device 100 may not include a gaze information acquisition unit 192, or the imaging unit 140 may also function as the gaze information acquisition unit 192. The gaze information acquired by the gaze information acquisition unit 192 may be input to the control unit 110, for example. For this reason, the gaze information acquisition unit 192 may be connected to the control unit 110 via a wired and/or wireless connection.

In one embodiment, the second electronic device 100 may be a dedicated device as described above. Meanwhile, in one embodiment, the second electronic device 100 may include some of the functional units shown in FIG. 4, for example. In this case, the second electronic device 100 may be connected to another electronic device to supplement at least some of the functions of the other functional units shown in FIG. 4. Here, the other electronic device may be, for example, a general-purpose smartphone, tablet, phablet, notebook computer (notebook PC or laptop), or computer (desktop), etc.

In particular, a smartphone or a laptop computer often has a relatively large number of the functional units shown in FIG. 4. For this reason, in one embodiment, the second electronic device 100 may be a smartphone or a laptop computer. In this case, the second electronic device 100 may be a smartphone or a laptop computer on which an application (program) for linking with the first electronic device 1 is installed.

FIG. 5 is a block diagram showing a schematic configuration of the third electronic device 300 shown in FIG. 1. An example of the configuration of the third electronic device 300 according to an embodiment will be described below. The third electronic device 300 may be installed in a location other than the home RL and the conference room MR of the interlocutor Mg, as shown in FIG. 1. The third electronic device 300 may be installed in or near the home RL of the interlocutor Mg, or in or near the conference room MR.

The first electronic device 1 has a function of transmitting audio and/or video data of the interlocutor candidates Ma, Mb, Mc, Md, Me, Mf, etc. acquired by the first electronic device 1 to the third electronic device 300 when the interlocutor candidates Ma, Mb, Mc, Md, Me, Mf, etc. speak. The third electronic device 300 may transmit the audio and/or video data received from the first electronic device 1 to the second electronic device 100. The second electronic device 100 also has a function of transmitting audio and/or video data of the interlocutor Mg acquired by the second electronic device 100 to the third electronic device 300 when the interlocutor Mg speaks. The third electronic device 300 may transmit the audio and/or video data received from the second electronic device 100 to the first electronic device 1. In this way, the third electronic device 300 may have a function of relaying between the first electronic device 1 and the second electronic device 100. The third electronic device 100 is also referred to as a "server" as appropriate.

As shown in FIG. 5, the third electronic device 300 according to one embodiment may include a control unit 310, a storage unit 320, and a communication unit 330. The control unit 310 may also include, for example, an identification unit 312 and an estimation unit 314. In one embodiment, the third electronic device 300 may not include at least some of the functional units shown in FIG. 5, or may include components other than the functional units shown in the figure.

The control unit 310 may have a function of controlling and/or managing the entire third electronic device 300, including each functional unit constituting the third electronic device 300. The control unit 310 may basically be configured based on the same concept as the control unit 10 shown in FIG. 2 or the control unit 110 shown in FIG. 4. The acquisition unit 312, detection unit 314, selection unit 316, and identification unit 318 of the control unit 310 may also be configured based on the same concept as the acquisition unit 12, detection unit 14, selection unit 16, and identification unit 18 of the control unit 10 shown in FIG. 2. The acquisition unit 312, detection unit 314, selection unit 316, and identification unit 318 of the control unit 310 may also be configured based on the same concept as the acquisition unit 112, detection unit 114, selection unit 116, and identification unit 118 shown in FIG. 4.

The storage unit 320 may function as a memory that stores various types of information. The storage unit 320 may store, for example, programs executed in the control unit 310 and results of processing executed in the control unit 310. The storage unit 320 may also function as a work memory for the control unit 310. As shown in FIG. 5, the storage unit 320 may be connected to the control unit 310 by wire and/or wirelessly. The storage unit 320 may basically be configured based on the same concept as, for example, the storage unit 20 shown in FIG. 2 or the storage unit 120 shown in FIG. 4.

The communication unit 330 has an interface function for wireless and/or wired communication. The communication unit 330 may wirelessly communicate with, for example, a communication unit of another electronic device, for example, via an antenna. For example, the communication unit 330 may wirelessly communicate with the first electronic device 1 shown in FIG. 1. In this case, the communication unit 330 may wirelessly communicate with the communication unit 30 of the first electronic device 1. Thus, in one embodiment, the communication unit 330 has a function of communicating with the first electronic device 1. Also, for example, the communication unit 330 may wirelessly communicate with the second electronic device 100 shown in FIG. 1. In this case, the communication unit 330 may wirelessly communicate with the communication unit 130 of the second electronic device 100. Thus, in one embodiment, the communication unit 330 may have a function of communicating with the second electronic device 100. As shown in FIG. 5, the communication unit 330 may be connected to the control unit 310 in a wired and/or wireless manner. The communication unit 330 may basically be configured based on the same concept as, for example, the communication unit 30 shown in FIG. 2 or the communication unit 130 shown in FIG. 4.

In one embodiment, the third electronic device 300 may be, for example, a specially designed device. On the other hand, in one embodiment, the third electronic device 300 may include, for example, some of the functional units shown in FIG. 5. In this case, the third electronic device 300 may be connected to other electronic devices to supplement at least some of the functions of the other functional units shown in FIG. 5. Here, the other electronic devices may be, for example, devices such as a general-purpose computer or server. In one embodiment, the third electronic device 300 may be, for example, a relay server, a web server, or an application server.

Next, the basic operation of the first electronic device 1 and the second electronic device 100 according to one embodiment will be described. The following description will be given assuming a situation in which a remote conference is held in a conference room MR, as shown in FIG. 1, and a participant Mg participates from his/her home RL.

That is, the first electronic device 1 according to one embodiment is installed in the conference room MR and acquires video and/or audio of at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf. The video and/or audio acquired by the first electronic device 1 is transmitted to the second electronic device 100 installed in the interlocutor Mg's home RL. The second electronic device 100 outputs the video and/or audio of at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf acquired by the first electronic device 1. This allows the interlocutor Mg to recognize the video and/or audio of at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf.

On the other hand, the second electronic device 100 according to one embodiment is installed in the home RL of the interlocutor Mg and acquires the voice of the interlocutor Mg. The second electronic device 100 also acquires information on the line of sight of the interlocutor Mg. The voice and/or line of sight information acquired by the second electronic device 100 is transmitted to the first electronic device 1 installed in the conference room MR. The first electronic device 1 outputs the voice of the interlocutor Mg received from the second electronic device 100. As a result, at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf can hear the voice of the interlocutor Mg. The first electronic device 1 also expresses the line of sight of the interlocutor Mg based on the line of sight information of the interlocutor Mg received from the second electronic device 100. As a result, at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf can visually recognize the state of the line of sight of the interlocutor Mg. Furthermore, the second electronic device 100 according to one embodiment may acquire an image of the interlocutor Mg. The video captured by the second electronic device 100 may be transmitted to the first electronic device 1 installed in the conference room MR. In this case, the first electronic device 1 may output the video of the interlocutor Mg received from the second electronic device 100.

FIG. 6 is a sequence diagram explaining the basic operation of the system according to the embodiment described above. FIG. 6 is a diagram showing the exchange of data and the like between the first electronic device 1, the second electronic device 100, and the third electronic device 300. Below, the basic operation when a remote conference or video conference is held using the system according to the embodiment will be explained with reference to FIG. 6.

In the operation shown in FIG. 6, the first electronic device 1 used locally may be used by the first user. Here, the first user may be, for example, at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf shown in FIG. 1 (hereinafter also referred to as the local user). The second electronic device 100 used remotely may be used by the second user. Here, the second user may be, for example, the interlocutor Mg shown in FIG. 1 (hereinafter also referred to as the remote user). Hereinafter, the operation performed by the first electronic device 1 may be, in more detail, performed by, for example, the control unit 10 of the first electronic device 1. In this specification, the operation performed by the control unit 10 of the first electronic device 1 may be described as the operation performed by the first electronic device 1. Similarly, the operation performed by the second electronic device 100 may be, in more detail, performed by, for example, the control unit 110 of the second electronic device 100. In this specification, the operation performed by the control unit 110 of the second electronic device 100 may be referred to as the operation performed by the second electronic device 100. In addition, the operation performed by the third electronic device 300 may be more specifically, for example, performed by the control unit 310 of the third electronic device 300. In this specification, the operation performed by the control unit 310 of the third electronic device 300 may be referred to as the operation performed by the third electronic device 300.

The operation shown in FIG. 6 may be initiated, for example, at the start of a remote conference as shown in FIG. 1. Alternatively, the operation shown in FIG. 6 may be initiated, for example, at the start of the first electronic device 1 and/or the second electronic device 100.

6 starts, the first electronic device 1 acquires information about at least one interlocutor candidate (step S11). In step S11, the first electronic device 1 may acquire at least one of the video and audio of the first user (e.g., at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf). Specifically, in step S11, the acquisition unit 12 of the first electronic device 1 may capture the video of the first user by the imaging unit 40 and acquire (or detect) the audio of the first user by the audio input unit 50. In step S11, the first electronic device 1 may acquire information about the video and/or audio of at least one of the interlocutor candidates Ma, Mb, Mc, Md, Me, and Mf shown in FIG. 1, for example. In this way, in the first electronic device 1, the acquisition unit 12 may acquire information about at least one interlocutor candidate.

Next, the first electronic device 1 transmits information (e.g., video information and/or audio information) about the candidate interlocutor to the third electronic device 300 (step S12). Specifically, in step S12, the first electronic device 1 transmits video and/or audio data from the communication unit 30 to the communication unit 330 of the third electronic device 300. Also, in step S12, the third electronic device 300 may receive (acquire) the video and/or audio data transmitted from the communication unit 30 of the first electronic device 1 via the communication unit 330.

In step S12, the control unit 10 of the first electronic device 1 may encode at least one of the video and audio of the first user before transmitting it. Here, encoding may mean compressing the video and/or audio data according to a predetermined rule and converting it into a format according to the purpose, including encryption. The first electronic device 1 may perform various known encoding methods, such as software encoding or hardware encoding. In this case, the third electronic device 300 may decode the encoded video and/or audio data received from the communication unit 30. Here, decoding may mean returning the format of the encoded video and/or audio data to the original format. The third electronic device 300 may perform various known decoding methods, such as software encoding or hardware encoding.

Next, the third electronic device 300 detects at least one interlocutor candidate based on the information on the at least one interlocutor candidate acquired in step S12 (step S13). Here, the detection unit 114 of the third electronic device 300 may detect at least one interlocutor candidate based on video and/or audio information on the at least one interlocutor candidate acquired in step S12. For example, the detection unit 114 of the third electronic device 300 may detect at least one interlocutor candidate by person recognition or face recognition based on video of the at least one interlocutor candidate. Also, the detection unit 114 of the third electronic device 300 may detect at least one interlocutor candidate by audio recognition of the at least one interlocutor candidate.

For example, in step S13, the third electronic device 300 may detect interlocutor candidates that are present at a predetermined distance from the first electronic device 1. For example, the third electronic device 300 may detect, as interlocutor candidates, persons Ma, Mb, Mc, and Md that are present within 2 m of the first electronic device 1 shown in FIG. 1. In this case, persons Me and Mf shown in FIG. 1 may not be detected as interlocutor candidates.

In step S13, the third electronic device 300 may detect interlocutor candidates based on various conditions. For example, the third electronic device 300 may detect, as interlocutor candidates, a person whose gaze information indicating a predetermined intensity or higher is detected based on the gaze information of surrounding people acquired by the gaze information acquisition unit 92 of the first electronic device 1. That is, the third electronic device 300 may detect, as interlocutor candidates, a person who is relatively gazing at the first electronic device 1 around the first electronic device 1. Also, the third electronic device 300 may detect, as interlocutor candidates, a person who is looking toward the first electronic device 1 based on the face orientation and/or gaze direction of the surrounding people acquired by the imaging unit 40 and/or the gaze information acquisition unit 92 of the first electronic device 1. That is, the third electronic device 300 may detect, as interlocutor candidates, a person who is looking toward the first electronic device 1 around the first electronic device 1 and/or a person who is looking toward the first electronic device 1. Also, for example, the third electronic device 300 may detect a person as a candidate interlocutor based on information acquired by the imaging unit 40 (and/or the line of sight information acquisition unit 92) and the voice input unit 50 of the first electronic device 1. That is, the third electronic device 300 may detect, for example, a person who is relatively gazing at the first electronic device 1 or facing the first electronic device 1 and speaking to the first electronic device 1 as a candidate interlocutor. Also, the third electronic device 300 may detect a candidate interlocutor based on the positions of surrounding people acquired by the imaging unit 40 of the first electronic device 1. For example, the third electronic device 300 may detect a person within a predetermined distance range (for example, within 5 meters) from the first electronic device 1 as a candidate interlocutor. Also, for example, the third electronic device 300 may detect a person within a predetermined distance from the person detected as a candidate interlocutor based on the above-mentioned line of sight and/or face direction as a candidate interlocutor. The third electronic device 300 may detect interlocutor candidates by recognizing face images of surrounding people acquired by the imaging unit 40 of the first electronic device 1. For example, the third electronic device 300 may identify a person who is registered in advance as a participant of the conference by face recognition and detect the person as an interlocutor candidate. The third electronic device 300 may detect interlocutor candidates by the actions of surrounding people acquired by the imaging unit 40 of the first electronic device 1. For example, the third electronic device 300 may detect as interlocutor candidates a person who performs a predetermined action (e.g., raising a hand, waving, etc.) that the person wishes to be recognized as an interlocutor. The third electronic device 300 may detect as interlocutor candidates a person who performs a predetermined action (e.g., raising a hand, waving, etc.) that the person wishes to be recognized as an interlocutor. The third electronic device 300 may detect as interlocutor candidates a person who utters a predetermined utterance (e.g., "Hello," "(user name of the first electronic device 1)," "Hey," "Can you hear me," "Excuse me," etc.) that the person wishes to be recognized as an interlocutor.

In this way, the acquisition unit 312 of the third electronic device 300 may acquire information about at least one interlocutor candidate from the first electronic device 1. Furthermore, the detection unit 314 of the third electronic device 300 may detect at least one interlocutor candidate based on the information about the at least one interlocutor candidate.

Next, the third electronic device 300 transmits information about the detected interlocutor candidates to the second electronic device 100 (step S14). For example, the third electronic device 300 may transmit information about interlocutor candidates Ma, Mb, Mc, and Md as the detected interlocutor candidates to the second electronic device 100. Specifically, in step S14, the third electronic device 300 may transmit information about the detected interlocutor candidates from the communication unit 330 to the communication unit 130 of the second electronic device 100. Also, in step S14, the second electronic device 100 may receive the information about the interlocutor candidates transmitted from the communication unit 330 of the third electronic device 300 via the communication unit 130.

When the second electronic device 100 receives information about the interlocutor candidates, it may present the information about the interlocutor candidates to the second user (e.g., interlocutor Mg). In this case, the second electronic device 100 may present at least one of the video and audio of the interlocutor candidates Ma, Mb, Mc, and Md to the second user (e.g., interlocutor Mg) from at least one of the display unit 170 and the audio output unit 160.

For example, when each person in the conference room MR shown in FIG. 1 is imaged by the imaging unit 40 of the first electronic device 1, the second electronic device 100 may display the image of each person on the display unit 170 as shown in FIG. 7. As shown in FIG. 7, the display unit 170 displays the persons Ma, Mb, Mc, and Md seated around the desk in the conference room MR. Also, as shown in FIG. 7, the display unit 170 displays the persons Me and Mf standing at a location a little away from the desk in the conference room MR. Here, as described above, it is assumed that the third electronic device 300 detects the persons Ma, Mb, Mc, and Md as interlocutor candidates, and does not detect the persons Me and Mf shown in FIG. 1 as interlocutor candidates. In this case, the second electronic device 100 displays the persons detected as interlocutor candidates and the persons not detected as interlocutor candidates in a manner that allows them to be distinguished from each other by the image displayed on the display unit 170. As an example, the second electronic device 100 may indicate that the persons Ma, Mb, Mc, and Md have been detected as interlocutor candidates by displaying, for example, an object Ob1 around or near the persons Ma, Mb, Mc, and Md, as shown in FIG. 8. In FIG. 8, the object Ob1 is not displayed around or near the persons Me and Mf, indicating that the persons Me and Mf have not been detected as interlocutor candidates.

In FIG. 8, for example, object Ob1 is displayed around or near persons Ma, Mb, Mc, and Md to indicate that the persons have been detected as interlocutor candidates. In one embodiment, the second electronic device 100 may indicate, for example, by audio guidance, that persons Ma, Mb, Mc, and Md have been detected as interlocutor candidates. In another embodiment, the second electronic device 100 may indicate, for example, by audio guidance, that persons Me and Mf have not been detected as interlocutor candidates. Specifically, the audio guidance may read out the names of persons detected as interlocutor candidates or persons not detected as interlocutor candidates.

After step S14, the selection unit 116 of the second electronic device 100 selects at least one interlocutor from the at least one interlocutor candidate (step S15). In step S15, the selection unit 116 of the second electronic device 100 may select at least one interlocutor based on an input by the user of the second electronic device 100. That is, the user of the second electronic device 100 can perform an input to select an interlocutor from the detected interlocutor candidates. In this way, in the second electronic device 100, the acquisition unit 112 may acquire information on at least one interlocutor candidate. In the second electronic device 100, the selection unit 116 may select at least one interlocutor from the at least one interlocutor candidate detected. The selection unit 116 may select at least one interlocutor based on an input by the user of the second electronic device 100.

The second electronic device 100 may select an interlocutor in various ways. For example, as shown in FIG. 9, the second electronic device 100 may prompt the user to input by displaying a context menu Cm near an object Ob1 indicating an interlocutor candidate on the display unit 170. For example, the second electronic device 100 may display the context menu Cm for the interlocutor candidate Mb by, for example, moving the pointer Pt near the interlocutor candidate Mb or by inputting a click near the interlocutor candidate Mb. Here, the second electronic device 100 may set all the people detected as interlocutor candidates as interlocutors at once. In this case, the second electronic device 100 may then exclude people who are not required as interlocutors from the interlocutors by user input.

The second electronic device 100 may select the candidate interlocutor Mb as an interlocutor by selecting "Add" based on user input in the context menu Cm shown in FIG. 9. On the other hand, the second electronic device 100 may remove person Mb from interlocutors or interlocutor candidates by selecting "Remove" based on user input in the context menu Cm shown in FIG. 9. Also, the second electronic device 100 may suspend the selection of person Mb as an interlocutor by selecting "Suspend" based on user input in the context menu Cm shown in FIG. 9.

9 shows an example in which an interlocutor is selected by detecting an input (e.g., a mouse operation and/or a click) by a user to the input unit 190 of the second electronic device 100. Here, various manners for selecting an interlocutor from interlocutor candidates can be assumed. For example, the second electronic device 100 may automatically select an interlocutor candidate that is determined by the imaging unit 140 (and/or the gaze information acquisition unit 192) to be gazed at by the user for a predetermined time (e.g., 3 seconds) as an interlocutor. In addition to the user's gaze on a certain person for a predetermined time and/or input to the input unit 190, the second electronic device 100 may select an interlocutor candidate as an interlocutor based on the detection of a predetermined command. Here, the predetermined command may be, for example, the user's voice utterance of "add" to the voice input unit 150. For example, the second electronic device 100 may select an interlocutor candidate identified by the name of a person input by the user to the voice input unit 150 as an interlocutor. Furthermore, the second electronic device 100 may display one of the interlocutor candidates in a display mode different from the other interlocutor candidates, for example by highlighting the person on the display unit 170, and may ask the user from the voice output unit 160, for example, "Do you want to add this person to the speakers?" In this case, the second electronic device 100 may select the person as an interlocutor by detecting the user's utterance of "yes" or "add" by the voice input unit 160.

Similarly, various manners in which a person is excluded from interlocutors or interlocutor candidates can be assumed. For example, the second electronic device 100 may exclude a person from interlocutors or interlocutor candidates based on the user's gaze on a certain person for a certain period of time and/or input to the input unit 190, as well as the detection of a certain command. Here, the certain command may be, for example, the user's voice utterance of "exclude" or "delete" to the voice input unit 150. In addition, the second electronic device 100 may display one of the interlocutor candidates in a display mode different from the other interlocutor candidates, such as highlighting the person on the display unit 170, and ask the user from the voice output unit 160, "Do you want to exclude this person from speakers?" In this case, the second electronic device 100 may exclude the person from interlocutors or interlocutor candidates by detecting the user's voice utterance of "yes" or "exclude" by the voice input unit 160. Furthermore, for example, if the user hardly pays attention to a certain person and/or if there is hardly any voice input to the voice input unit 160, the person may be excluded from interlocutors or interlocutor candidates, for example, based on the passage of a predetermined time. For example, the second electronic device 100 may exclude a person who is a predetermined distance or more away from the first electronic device 1 from interlocutors or interlocutor candidates. For example, the second electronic device 100 may exclude a person who behaves inappropriately in light of a predetermined standard (for example, who uses abusive language, etc.) from interlocutors or interlocutor candidates.

Similarly, various modes of putting a candidate selected from among candidate interlocutors on hold can be envisioned. For example, the second electronic device 100 may display one of the candidate interlocutors in a display mode different from the other candidate interlocutors, such as highlighting the candidate on the display unit 170, and ask the user from the voice output unit 160, "Do you want to add this person to the speakers?" In this case, the second electronic device 100 may put off adding the person to the interlocutors by detecting the user's utterance of "no" or "hold" by the voice input unit 160. Furthermore, for example, when the user is hardly gazing at a certain person and/or when there is little voice input to the voice input unit 160, the second electronic device 100 may put off adding the person to the interlocutors, for example, based on the passage of a predetermined time.

When an interlocutor is selected from the interlocutor candidates in step S15, the second electronic device 100 transmits information indicating the selected interlocutor to the third electronic device 300 (step S16). Upon receiving the information indicating the selected interlocutor, the identification unit 318 of the third electronic device 300 identifies the interlocutor from the interlocutor candidates based on the information indicating the selected interlocutor (step S17). For example, among the persons Ma, Mb, Mc, and Md detected as interlocutor candidates in step S13, persons Mc and Md are selected as interlocutors based on an input by the user of the second electronic device 100. In this case, in step S16, the second electronic device 100 transmits information indicating that the selected interlocutors are persons Mc and Md to the third electronic device 300. Then, in step S17, the third electronic device 300 identifies interlocutors Mc and Md from the interlocutor candidates Ma, Mb, Mc, and Md.

When the interlocutor is specified in step S17, the third electronic device 300 can execute different processes for the selected interlocutor and for the interlocutor candidates other than the selected interlocutor. "A person other than the selected interlocutor among the interlocutor candidates" may be an interlocutor candidate who is not selected as an interlocutor. In other words, the third electronic device 300 can execute a specific process only for the selected interlocutor. Therefore, hereafter, the third electronic device 300 executes a predetermined process for the selected interlocutor regarding the dialogue with the selected interlocutor (step S18). The operation performed in step S18 is also referred to as "a predetermined process performed on the interlocutor" or "a predetermined process on the interlocutor". After step S18, the third electronic device 300 can control the first electronic device 1 and/or the second electronic device 200 to execute different processes for the selected interlocutor and for the interlocutor candidates other than the selected interlocutor. Therefore, the third electronic device 300 may control the second electronic device 100 based on the predetermined process for the selected interlocutor (step S19). The third electronic device 300 may also control the first electronic device 1 based on a predetermined process for the selected interlocutor (step S20). In this way, the control unit 310 may control at least one of the first electronic device 1 and the second electronic device 200 to execute a predetermined process for a conversation with at least one interlocutor selected by the second electronic device 100 from at least one interlocutor candidate. The control unit 310 may also control at least one of the first electronic device 1 and the second electronic device 200 not to execute a predetermined process for an interlocutor candidate not selected as an interlocutor.

In this way, the control unit 10 of the first electronic device 1 executes a predetermined process related to a dialogue with at least one interlocutor selected from at least one interlocutor candidate. In this case, the control unit 10 may execute a predetermined process related to a dialogue with an interlocutor selected by another electronic device (the second electronic device 100) as at least one interlocutor. Also, the control unit 110 of the second electronic device 100 executes a predetermined process related to a dialogue with at least one interlocutor.

Various predetermined processes can be assumed for the process performed in steps S18 to S20 described above. For example, assume that interlocutors Mc and Md are selected from interlocutor candidates Ma, Mb, Mc, and Md as described above. In this case, the third electronic device 300 may control the second electronic device 100 so that interlocutors Mc and Md are displayed on the display unit 170 in a display mode different from that of other interlocutor candidates. For example, as shown in FIG. 10, the second electronic device 100 may indicate that persons Mc and Md have been selected as interlocutors from interlocutor candidates Ma, Mb, Mc, and Md by attaching an object Ob1 or the like different from the others to interlocutors Mc and Md. In this case, interlocutors Mc and Md may be displayed in a display mode different from that of other interlocutor candidates Ma, Mb, etc., for example, by highlighting interlocutors Mc and Md on the display unit 170. As a result, the user of the second electronic device 100 can see on the display unit 170 that persons Mc and Md have been selected as interlocutors.

Furthermore, for example, when persons Mc and Md are selected as interlocutors from among interlocutor candidates Ma, Mb, Mc, and Md, the second electronic device 100 may enlarge and display interlocutors Mc and Md on the display unit 170 as a predetermined process. In this case, the second electronic device 100 may, for example, cut out only images of interlocutors Mc and Md and display them on the display unit 170. Furthermore, the second electronic device 100 may display only interlocutors Mc and Md on, for example, a sub-display installed separately from the display unit 170. Further, for example, when persons Mc and Md are selected as interlocutors from among interlocutor candidates Ma, Mb, Mc, and Md, the second electronic device 100 may blur interlocutor candidates other than persons Mc and Md and display them on the display unit 170. Furthermore, when persons Mc and Md are selected as interlocutors from among interlocutor candidates Ma, Mb, Mc, and Md, the second electronic device 100 may remove interlocutor candidates other than persons Mc and Md from the display of the display unit 170. In this manner, the second electronic device 100 may include a display unit 170 that displays information about at least one interlocutor candidate. Furthermore, the control unit 110 of the second electronic device 100 may execute a process of displaying information about the interlocutor in a different manner from information about interlocutor candidates other than the interlocutor, as a predetermined process related to the dialogue with the interlocutor.

Furthermore, for example, when persons Mc and Md are selected as interlocutors from among interlocutor candidates Ma, Mb, Mc, and Md as interlocutors as a predetermined process, the second electronic device 100 may display on the display unit 170 that the interlocutor Mc or Md is speaking, based on the speech of the interlocutor Mc or Md. In this case, when a person other than interlocutors Mc or Md speaks, the second electronic device 100 may not display on the display unit 170 that the person is speaking.

Furthermore, for example, as a predetermined process, the second electronic device 100 may output from the audio output unit 160 a voice indicating that interlocutors Mc and Md have been selected when persons Mc and Md are selected from among interlocutor candidates Ma, Mb, Mc, and Md. In this manner, the second electronic device 100 may be provided with an audio output unit 160 that outputs information about at least one interlocutor candidate by audio. The control unit 110 of the second electronic device 100 may execute a process of outputting information about the interlocutor in a voice that is different from information about persons other than the interlocutor among the interlocutor candidates, as a predetermined process related to the dialogue with the interlocutor.

Also, for example, when the persons Mc and Md are selected as interlocutors from among the interlocutor candidates Ma, Mb, Mc, and Md, the third electronic device 300 may control the first electronic device 1. For example, when the first electronic device 1 has a structure having a shape of a face of a doll or a robot, the first electronic device 1 may cause the doll or robot to perform a predetermined action toward the persons Mc or Md as a predetermined process. Specifically, when the person Mg is speaking or looking toward the person Mc or Md, the first electronic device 1 may drive the driving unit 80 to move the face or look toward the interlocutors Mc and Md. In this case, the first electronic device 1 may be configured not to look toward or look toward the interlocutors Mc or Md even if the person Mg speaks or looks toward a person other than the person Mc or Md. In this way, the control unit 310 of the third electronic device 300 may execute a process of controlling the other electronic device (the first electronic device 1) as a predetermined process related to the dialogue with the interlocutor, so that the interlocutor behaves differently from the interlocutor candidates other than the interlocutor. Also, the control unit 310 of the third electronic device 300 may execute a process of controlling the other electronic device (the second electronic device 100) as a predetermined process related to the dialogue with the interlocutor, so that the interlocutor behaves differently from the interlocutor candidates other than the interlocutor.

The operation shown in FIG. 6 may be repeatedly executed from the start at an appropriate timing. Also, for example, in a situation as shown in FIG. 10, it is assumed that interlocutor Mg is conversing with interlocutors Mc and Md. Then, for example, it is assumed that person Me and/or Mf shown in FIG. 10 approaches the first electronic device 1. In such a situation, it is assumed that the operation shown in FIG. 6 is started. In this case, for example, in step S13, if person Me and/or Mf exists within a predetermined distance from the first electronic device 1, they are detected as interlocutor candidates. In this case, in step S14, the third electronic device 300 transmits information about the new interlocutor candidate Me and/or Mf to the second electronic device 100. Therefore, in step S15, the user of the second electronic device 100 can select whether or not to select the new interlocutor candidate Me and/or Mf as an interlocutor. In this case, the second electronic device 100 may inform the user of the second electronic device 100 of the existence of the new interlocutor candidate Me and/or Mf. For example, the second electronic device 100 may output a predetermined voice or sound from the audio output unit 160 to notify the presence of the new interlocutor candidate Me and/or Mf. Also, the second electronic device 100 may highlight or display a pop-up on the display unit 170 to notify the presence of the new interlocutor candidate Me and/or Mf.

According to the system of one embodiment, in communication between an electronic device used remotely and an electronic device used locally, a user can select an interlocutor with whom he or she intends to communicate. Generally, humans perform turn-taking almost unconsciously based on cultural background, etc. In addition, generally, humans can focus conscious resources on turn-taking with an interlocutor by mutually recognizing one or more interlocutors with whom they share turn-taking. However, if there is no human element involved, it is difficult to distinguish whether a detected person is an interlocutor or not. For this reason, it is expected that physical and/or processing resources will be allocated to a person who would not be recognized as an interlocutor by a human being by determining that the person is a speaker. As a result, there is a risk that turn-taking between the user and the interlocutor will be hindered by presenting the user with the results of processing that recognizes a person who is not an interlocutor. However, according to the system of one embodiment described above, a user can select an interlocutor from interlocutor candidates presented by the system, and physical and/or processing resources can be concentrated only on the person the user recognizes as an interlocutor. Therefore, according to the system of one embodiment, communication can be performed smoothly.

Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art would easily be able to make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to cause logical inconsistencies, and multiple components or steps can be combined into one or divided. Although the embodiments of the present disclosure have been described mainly with respect to the device, the embodiments of the present disclosure can also be realized as a method including steps executed by each component of the device. The embodiments of the present disclosure can also be realized as a method, a program executed by a processor or the like included in the device, or a storage medium or storage medium on which a program is recorded. It should be understood that these are also included in the scope of the present disclosure.

The above-described embodiments are not limited to implementation as a system. For example, the above-described embodiments may be implemented as a control method for a system, or as a program executed in a system. For example, the above-described embodiments may be implemented as at least one of the first electronic device 1, the second electronic device 100, and the third electronic device 300. The above-described embodiments may be implemented as a control method for at least one of the first electronic device 1, the second electronic device 100, and the third electronic device 300. Furthermore, the above-described embodiments may be implemented as a program executed by at least one of the first electronic device 1, the second electronic device 100, and the third electronic device 300, or as a storage medium or recording medium on which the program is recorded.

After step S14 of the operation shown in FIG. 6, the second electronic device 100 selected an interlocutor from the interlocutor candidates (step S15). Also, an example was described in which, when selecting an interlocutor from the interlocutor candidates, the second electronic device 100 prompts the user to select from among "add," "remove," and "reserve" the interlocutor. On the other hand, as shown in FIG. 11, after step S14, the second electronic device 100 may select a "non-interlocutor" from the interlocutor candidates (step S21) and transmit information indicating the "non-interlocutor" to the third electronic device 300 (step S22). In this case, in step S23, the third electronic device 300 may identify the "non-interlocutor" from the interlocutor candidates. Thereafter, a predetermined process related to the dialogue with the interlocutor may be performed on the interlocutor without performing it on the "non-interlocutor."

In the above-described embodiment, the third electronic device 300 relays the communication between the first electronic device 1 and the second electronic device 100. However, at least one of the first electronic device 1 and the second electronic device 100 may execute some or all of the functions executed by the third electronic device 300. For example, the detection unit 14 of the first electronic device 1 may detect at least one interlocutor candidate based on information about at least one interlocutor candidate. The control unit 10 of the first electronic device 1 may transmit information about at least one interlocutor candidate detected by the detection unit 14 to another electronic device (e.g., the second electronic device 100). The identification unit 18 of the first electronic device 1 may identify an interlocutor selected by the other electronic device (e.g., the second electronic device 100) as at least one interlocutor from the at least one interlocutor candidate detected by the detection unit 14. The control unit 10 of the first electronic device 1 may transmit information about an interlocutor identified by the identification unit 18 as at least one interlocutor to the other electronic device (e.g., the second electronic device 100). The control unit 10 of the first electronic device 1 may execute a predetermined process related to a dialogue with the interlocutor identified by the identification unit 18 as at least one interlocutor. Furthermore, the detection unit 14 of the first electronic device 1 may detect at least one interlocutor candidate based on information regarding the line of sight of the at least one interlocutor candidate acquired by the line of sight information acquisition unit 92.

The selection unit 116 of the second electronic device 100 may select at least one non-interlocutor from at least one candidate interlocutor based on input by the user. In this case, the control unit 110 of the second electronic device 100 may not execute a predetermined process for at least one non-interlocutor. The second electronic device 100 may also include a gaze information acquisition unit 192 that acquires information regarding the gaze of the user of the second electronic device 100. In this case, the selection unit 116 of the second electronic device 100 may select at least one interlocutor based on information regarding the user's gaze.

The first electronic device 1 according to the above-described embodiment has been described as being operated by the user of the second electronic device 100. For example, the system according to the above-described embodiment has been described as an aspect in which an interlocutor is selected from interlocutor candidates by the user of the second electronic device 100. However, the first electronic device 1 may perform at least partially autonomous operations without the intervention of partial or full operations by the user of the second electronic device 100. For example, in one embodiment, the first electronic device 1 (or the third electronic device 300) may autonomously (for example, according to a predetermined algorithm) select an interlocutor from interlocutor candidates.

1 First electronic device 10 Control unit 12 Acquisition unit 14 Detection unit 16 Selection unit 18 Identification unit 20 Memory unit 30 Communication unit 40 Imaging unit 50 Audio input unit 60 Audio output unit 70 Display unit 80 Driving unit 90 Input unit 92 Line-of-sight information acquisition unit 100 Second electronic device 110 Control unit 112 Acquisition unit 114 Detection unit 116 Selection unit 118 Identification unit 120 Memory unit 130 Communication unit 140 Imaging unit 150 Audio input unit 160 Audio output unit 170 Display unit 190 Input unit 192 Line-of-sight information acquisition unit 300 Third electronic device 310 Control unit 312 Acquisition unit 314 Detection unit 316 Selection unit 318 Identification unit 320 Memory unit 330 Communication unit N Network

Claims (14)

An acquisition unit that acquires information regarding at least one interlocutor candidate;
a detection unit that detects the at least one candidate interlocutor based on information about the at least one candidate interlocutor;
a control unit that executes a predetermined process for at least one interlocutor selected from the at least one interlocutor candidate, and does not execute the predetermined process for the interlocutor candidate who is not selected as the interlocutor;
An electronic device comprising: The electronic device according to claim 1, wherein the control unit executes the predetermined process for a party selected by another electronic device as the at least one party. The electronic device according to claim 1, wherein the detection unit detects a person within a predetermined distance range from the electronic device as the interlocutor candidate. a gaze information acquisition unit for acquiring information regarding the gaze of the at least one interlocutor candidate,
The electronic device according to claim 1 , wherein the detection unit detects the at least one interlocutor candidate based on information regarding a line of sight of the at least one interlocutor candidate acquired by the line of sight information acquisition unit. A structure having a shape of a face of a doll or a robot,
The control unit performs, as the predetermined processing, an operation of directing a face or a line of sight of the structure to the interlocutor and not directing a face or a line of sight of the structure to the interlocutor candidate who is not selected as the interlocutor.
2. The electronic device according to claim 1. An acquisition unit that acquires information regarding at least one interlocutor candidate;
A selection unit that selects at least one interlocutor from the at least one interlocutor candidate;
a control unit that executes a predetermined process for the at least one interlocutor and does not execute the predetermined process for the interlocutor candidates who are not selected as the interlocutor;
An electronic device comprising: The electronic device according to claim 6, wherein the selection unit selects the at least one interlocutor based on an input by a user. The selection unit selects at least one non-interlocutor from the at least one interlocutor candidate based on an input by a user;
The electronic device according to claim 6 , wherein the control unit does not execute the predetermined process for the at least one non-interacting party. a line-of-sight information acquisition unit that acquires information about a line of sight of a user of the electronic device,
The electronic device according to claim 6 , wherein the selection unit selects the at least one interlocutor based on information regarding a line of sight of the user. a display unit for displaying information about the at least one interlocutor candidate;
The electronic device according to claim 6 , wherein the control unit executes, as the predetermined processing, a process of displaying information about the at least one interlocutor in a manner different from information about the interlocutor candidates who are not selected as the interlocutor. a voice output unit that outputs information about the at least one interlocutor candidate by voice;
The electronic device according to claim 6 , wherein the control unit executes, as the predetermined processing, a process of outputting information about the at least one interlocutor in a voice different from information about the interlocutor candidates not selected as the interlocutor. The electronic device according to claim 6, wherein the control unit executes, as the predetermined process, a process of controlling other electronic devices so that the at least one interlocutor operates differently from the interlocutor candidate who is not selected as the interlocutor. an acquisition unit that acquires information about at least one interlocutor candidate from the first electronic device;
a detection unit that detects the at least one candidate interlocutor based on information about the at least one candidate interlocutor;
a control unit that controls at least one of the first electronic device and the second electronic device so as to execute a predetermined process for at least one interlocutor selected by the second electronic device from the at least one interlocutor candidate, and not to execute the predetermined process for the interlocutor candidate not selected as the interlocutor;
An electronic device comprising: On the computer,
obtaining information about at least one potential interlocutor;
detecting the at least one candidate interlocutor based on information about the at least one candidate interlocutor;
executing a predetermined process for at least one interlocutor selected from the at least one interlocutor candidate, and not executing the predetermined process for the interlocutor candidate who is not selected as the interlocutor;
A program to execute.

Images