JP2019061629A - Information processing apparatus, information processing method, program, display control device, display control method, program, and information processing system - Google Patents

Abstract

To enable a remote communication support system to provide a user with a video from a viewpoint suitable for the behaviors and psychological characteristics of the users.SOLUTION: There is provided an information processing apparatus comprising a learning unit that, when a three-dimensional line is designated, which is according to a three-dimensional selected position in a three-dimensional virtual space corresponding to a selected position based on a user's designation and a three-dimensional installation position of an imaging device including the three-dimensional selected position in an imaging range, and the imaging device corresponding to the three-dimensional line is designated as a viewpoint position of the user, calculates a browsing condition for the user for a person corresponding to an object present at the selected position on the basis of the three-dimensional selected position and the three-dimensional installation position, and calculates the relationship between the browsing condition and the number of times of designation of the viewpoint position.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, an information processing method, a program, a display control apparatus, a display control method, a program, and an information processing system.

  2. Description of the Related Art In recent years, softphones implemented by application software have become widespread in place of telephones implemented by hardware as in the past. Since the softphone is implemented by software, the function addition to the softphone and the cooperation between the softphone and other application software can be realized relatively easily. Therefore, various application technologies related to soft phones have been proposed.

  For example, Patent Document 1 discloses a system including a plurality of cameras for photographing the floor in an overhead manner and face recognition when touching a person appearing in the photographed image on the display screen of the photographed image obtained from the camera. There is disclosed a technique of specifying the person by the above, acquiring the telephone number of the person, and calling out the person.

Unexamined-Japanese-Patent No. 2007-208863 Unexamined-Japanese-Patent No. 2004-152276

Gaver et al., One is not enough: multiple views in a media space, in Proceedings of INTERCHI '93, pp. 335-341, 1993.

  However, although Patent Document 1 describes that the system is provided with the function of switching a plurality of cameras, it is specifically described what specification the user interface for the camera switching operation is. There is no mention of that at all.

  Here, the importance of the camera switching function and its operation in the remote communication support system will be described.

  Just before making a call to a callee at a remote location, the user of the call can know the presence information of the callee by viewing the video of the remote location. Presence information conveys the situation of the other party, and it is preferable that the information be high in both quantity and quality. When the quality of the presence information is improved, not only the rough situation grasp such as whether or not the callee is present, but also the callee who is busy or is about to leave soon, etc. More advanced situational understanding of As a result, both the calling user's ease of talking to the callee is improved, while the callee's callee is less likely to be called when the call is not likely to be called. Useful effects can be obtained.

  Then, in order to effectively perform the above-mentioned advanced situation grasping, that is, “watching” behavior in a remote environment, not the presence or absence of the callee, the physical condition of the other party is diversified. System functions that can be observed It is desirable, but not sufficient, that the telecommunication support system comprises a plurality of cameras to realize its function.

  Because, in the switching operation of a plurality of cameras installed at remote locations, it is necessary for the operation user to see which camera at which remote location is currently being viewed by which camera, and the target / position to view. This is because the existence of a problem (Spatial Discontinuities) in which the mutual positional relationship between a plurality of cameras is not known, such as whether to select a camera, has been pointed out in the past (Non-Patent Document 1).

  Furthermore, for example, in Patent Document 2, the user learns the tendency of the function selection operation to be executed next from the information of the device function selection operation history executed by the user in association with the time when the user performs the input. A technique for predicting and supporting function selection operations by the U.S. Pat.

  However, Patent Document 2 is a technology that targets, in a certain information device operated by a user, information in which a functional unit in a chainable function series, which is manually defined by a certain device developer, is selected. On the other hand, the present invention, as described later, is a technology directed to designation information of "position" corresponding to real space. In particular, based on “the positional relationship between the camera and the subject person”, the system learns position specification history information according to a psychological characteristic of viewpoint preference for each user. Then, according to the position designated by the user and the viewpoint preference characteristic, the system is a technique related to a user interface for recommending the position of the viewpoint predicted to be optimum for each user to the user.

  Therefore, the present invention solves the problem of confusion without being aware of the mutual positional relationship between the plurality of cameras while performing intuitive and efficient operation, so that the calling user can receive a call from a remote place. The present invention provides a technique for further enhancing the effect of the conventional invention, which provides a user interface that enables a user's situation to be easily and highly comprehended. An object of the present invention is to enable a remote communication support system to provide a user with an image of a viewpoint suitable for each user's behavior and psychological characteristics by learning processing of viewpoint selection behavior based on position information. .

  According to the present invention, a three-dimensional line corresponding to the three-dimensional selection position in the three-dimensional virtual space corresponding to the selection position based on the user specification and the three-dimensional installation position of the imaging device including the three-dimensional selection position in the imaging range. Is specified, and when the imaging device corresponding to the three-dimensional line is specified as the viewpoint position of the user, the object corresponding to the selected position is supported based on the three-dimensional selected position and the three-dimensional installation position. There is provided an information processing apparatus, comprising: a learning unit that calculates a viewing condition of the user with respect to a person who is to be and calculates a relationship between the viewing condition and the number of times of designation of the viewpoint position.

  The learning unit may calculate the viewing condition in which the number of designations of the viewpoint position is maximum.

  The learning unit may calculate a distance between the user and the person in at least a three-dimensional virtual space as the viewing condition based on the three-dimensional selection position and the three-dimensional installation position.

  The learning unit may calculate a relational expression between the distance and the number of designations of the viewpoint position, and may calculate the distance at which the designation number of the viewpoint position is maximum, based on the relational expression.

  The learning unit uses an angle of a viewpoint position of the user with respect to the person in at least a three-dimensional virtual space as the viewing condition based on the three-dimensional selection position, the three-dimensional installation position, and information of the body direction of the person. It may be calculated.

  The learning unit may acquire the body direction of the person based on the information of the installation direction of the desk in the real space corresponding to the three-dimensional virtual space linked to the person.

  The learning unit may acquire a body direction of the person based on a face detection process of the person or a three-dimensional human body model creation process on an image captured by the imaging device.

  The learning unit may acquire the body direction of the person based on sensor data detected by a sensor device provided on the body of the person.

  The learning unit may calculate a relational expression between the angle and the number of times of designation of the viewpoint position, and calculate the angle at which the number of designation of the viewpoint position is maximum based on the relational expression.

  The learning unit determines a distance between the user and the person in a three-dimensional virtual space and a viewpoint of the user with respect to the person based on the three-dimensional selection position, the three-dimensional installation position, and information of the body direction of the person. An angle of position is calculated as the viewing condition, and a regression equation including the distance and the angle and the number of times of designation of the viewpoint position as variables is calculated, and the number of times of designation of the viewpoint position is determined to be maximum based on the regression equation. And the angle may be calculated.

  Further, according to the present invention, three according to the three-dimensional selection position in the three-dimensional virtual space corresponding to the selection position based on the user specification and the three-dimensional installation position of the imaging device including the three-dimensional selection position in the imaging range. When a dimensional line is specified and an imaging device corresponding to the three-dimensional line is specified as the viewpoint position of the user, an object existing at the selected position based on the three-dimensional selected position and the three-dimensional installation position An information processing method is provided, including: calculating a viewing condition of the user with respect to a person corresponding to; and calculating a relationship between the viewing condition and the designated number of times of the viewpoint position.

  Further, according to the present invention, a computer is divided into a three-dimensional selection position in a three-dimensional virtual space corresponding to a selection position specified by a user and a three-dimensional installation position of an imaging device including the three-dimensional selection position in an imaging range. When a corresponding three-dimensional line is specified and an imaging device corresponding to the three-dimensional line is specified as the viewpoint position of the user, based on the three-dimensional selection position and the three-dimensional installation position, A program for functioning as an information processing apparatus is provided, comprising: a learning unit that calculates a viewing condition of the user with respect to a person corresponding to an existing object, and calculates a relationship between the viewing condition and the number of designations of the viewpoint position. Ru.

  Further, according to the present invention, three according to the three-dimensional selection position in the three-dimensional virtual space corresponding to the selection position based on the user specification and the three-dimensional installation position of the imaging device including the three-dimensional selection position in the imaging range. A dimensional line is specified, an imaging device corresponding to the three-dimensional line is specified as a viewpoint position of the user, and an object existing at the selected position is supported based on the three-dimensional selected position and the three-dimensional installation position. A display control unit including a display control unit configured to control a display according to the relationship when the viewing condition of the user with respect to the subject person is calculated and the relationship between the viewing condition and the number of designations of the viewpoint position is calculated; Provided.

  The display control unit selects a three-dimensional line according to the viewing condition in which the number of times of designation of the viewpoint position is the largest, and corresponds to a two-dimensional line corresponding to the selected three-dimensional line and another three-dimensional line The display mode of at least one of the two-dimensional line corresponding to the selected three-dimensional line and the two-dimensional line corresponding to the other three-dimensional line is changed so that the two-dimensional line and the three-dimensional line are distinguishably viewed. It is also good.

  The display control unit calculates a viewpoint position in a three-dimensional line according to the viewing condition where the number of times of designation of the viewpoint position is maximum, and performs control so that a captured image corresponding to the calculated viewpoint position is displayed. May be

  Further, according to the present invention, three according to the three-dimensional selection position in the three-dimensional virtual space corresponding to the selection position based on the user specification and the three-dimensional installation position of the imaging device including the three-dimensional selection position in the imaging range. A dimensional line is specified, an imaging device corresponding to the three-dimensional line is specified as a viewpoint position of the user, and an object existing at the selected position is supported based on the three-dimensional selected position and the three-dimensional installation position. A display control method is provided, including controlling display according to the relationship when the user's viewing condition for the person who is being calculated and the relationship between the viewing condition and the designated number of times of the viewpoint position are calculated; Be done.

  Further, according to the present invention, a computer is divided into a three-dimensional selection position in a three-dimensional virtual space corresponding to a selection position specified by a user and a three-dimensional installation position of an imaging device including the three-dimensional selection position in an imaging range. A corresponding three-dimensional line is specified, and an imaging device corresponding to the three-dimensional line is specified as the viewpoint position of the user, and exists at the selected position based on the three-dimensional selected position and the three-dimensional installation position. A display control unit configured to control display according to the relationship when a viewing condition of the user with respect to a person corresponding to an object is calculated and a relationship between the viewing condition and the number of designations of the viewpoint position is calculated; A program for functioning as a control device is provided.

  Further, according to the present invention, three according to the three-dimensional selection position in the three-dimensional virtual space corresponding to the selection position based on the user specification and the three-dimensional installation position of the imaging device including the three-dimensional selection position in the imaging range. When a dimensional line is specified and an imaging device corresponding to the three-dimensional line is specified as the viewpoint position of the user, an object existing at the selected position based on the three-dimensional selected position and the three-dimensional installation position And a learning unit that calculates a viewing condition of the user with respect to a person corresponding to the user, and calculates a relationship between the viewing condition and the designated number of times of the viewpoint position; and a display control unit that controls display according to the relationship An information processing system is provided.

  As described above, according to the present invention, in the remote communication support system, the user can operate intuitively and efficiently while solving the problem of cognitive confusion without knowing the mutual positional relationship between the plurality of cameras in the remote location. Can improve the quality of use of the user interface of the system that allows the calling user to easily and highly understand the situation of the remote called party. Specifically, when there are a plurality of viewpoint candidates of the viewing position that can be selected by the user with respect to the remote location designated by the user, the number of viewpoint candidates is narrowed down in advance, thereby causing the user to accompany the selection action. If the cognitive load of the subject is reduced or the accumulation of learning data is increased and the prediction accuracy is improved, the user can designate the desired position and leave the subsequent determination of the viewed position to the system, reducing the number of operation steps. , The effort of the user's designated input action is reduced.

It is an explanatory view showing an example of rough composition of an information processing system concerning one embodiment. It is a block diagram showing an example of the hardware constitutions of the terminal unit concerning one embodiment. It is a block diagram showing an example of functional composition of a terminal unit concerning one embodiment. It is an explanatory view for explaining an example of a display screen displayed in an eyebrow mode. It is an explanatory view for explaining an example of a display screen displayed in proximity mode. It is an explanatory view for explaining the 1st example of a position in a proximity image pick-up picture specified by a user. It is explanatory drawing for demonstrating the 2nd example of the position in the proximity | contact captured image designated by the user. It is explanatory drawing for demonstrating the 1st example of the three-dimensional virtual space corresponding to a center office. FIG. 9 is an explanatory diagram for describing an example of selection of an object arranged in the three-dimensional virtual space shown in FIG. 8; It is explanatory drawing for demonstrating the 2nd example of the three-dimensional virtual space corresponding to a center office. FIG. 11 is an explanatory diagram for describing an example of selection of an object arranged in the three-dimensional virtual space shown in FIG. 10; It is an explanatory view for explaining an example of a display screen displayed in conversation mode. It is a transition diagram for demonstrating an example of transition of a display mode. It is an explanatory view for explaining an example of a camera line in a three-dimensional virtual space corresponding to a center office. It is an explanatory view for explaining an example of a camera line displayed on a display screen of a terminal unit. It is an explanatory view for explaining some examples of touch input which specifies a position on a camera line, and a feedback display to it. FIG. 15 is an explanatory diagram for describing an example of selection of an object arranged in the three-dimensional virtual space shown in FIG. 14; It is an explanatory view for explaining an example of a display screen of terminal unit 100 in case multiple camera lines exist near. It is a block diagram showing an example of the software configuration of the terminal unit concerning one embodiment. It is a block diagram showing an example of the hardware constitutions of the information management server concerning one embodiment. It is a block diagram showing an example of functional composition of an information management server concerning one embodiment. It is a flow chart which shows an example of a rough flow of information processing concerning one embodiment. It is a flow chart which shows an example of a rough flow of starting processing concerning one embodiment. It is a figure which shows an example of the display which applied the function of the modification 1 with respect to FIG. It is a figure which shows the example which made the quality of the picked-up image high and low corresponded to the display form of a camera line. It is explanatory drawing for demonstrating an example of the preference prediction camera line displayed on the display screen of a terminal device. It is a figure which shows an example of the data used for the learning process of the point designation data of a viewpoint. It is a horizontal plane angle classification conceptual diagram around a person. It is a figure which shows an example of the statistical data of the browsing person angle in the viewpoint position designation operation log of a certain user A. FIG. It is a figure which shows an example of the statistical data of the browsing person angle in the viewpoint position designation operation log of a certain user B. FIG. It is a flow chart which shows an example of the rough flow of study and recommendation processing concerning this embodiment.

  The present invention will now be described more fully with reference to the accompanying drawings, in which exemplary embodiments of the invention are shown. In the present specification and the drawings, components having substantially the same functional configuration will be assigned the same reference numerals and redundant description will be omitted.

  From then on, <1. Schematic Configuration of Information Processing System>, <2. Configuration of Terminal Device>, <3. Configuration of Information Management Server>, <4. Processing Flow> The embodiment of the present invention will be described in the order of

<< 1. Schematic configuration of information processing system >>
First, a schematic configuration of an information processing system according to an embodiment of the present invention will be described with reference to FIG. FIG. 1 is an explanatory view showing an example of a schematic configuration of an information processing system according to the present embodiment. Referring to FIG. 1, the information processing system exists, for example, over a plurality of bases. In this example, the information processing system exists across the center office 10 and the satellite office 20 (or the home office 20). The center office 10 is a relatively large office, and the satellite office 20 (or home office and 20) is a relatively small office.

  The information processing system includes, in the center office 10, a camera 11, a microphone 13, a sensor 15, a media distribution server 17, an information management server 200 (an information processing apparatus), and a LAN (Local Area Network) 19. The information processing system also includes a terminal device 100 (display control device), a display 21 and a LAN 23 in the satellite office 20 (or home office and 20). The information processing system further includes a PBX (Private Branch eXchange) 40.

(Camera 11)
The camera 11 captures an area in the direction in which the camera 11 is directed (that is, the imaging direction). In the center office 10, a plurality of cameras 11 are installed. And each installed camera 11 images a part or whole of a center office from each installation position. Thus, in the information processing system, the center office is imaged from various positions. The captured image generated through the camera 11 may be a still image or a moving image (i.e., an image).

  Also, the camera 11 can automatically change its direction, for example. Furthermore, the camera 11 has, for example, a zoom function. The zoom function may be an optical zoom function or a digital zoom function.

  Also, the camera 11 may be capable of changing its position. As an example, the camera 11 may be movable by dolly. That is, the camera 11 may be movable along the rails. In this case, the camera 11 may move back and forth by controlling the motor to move along the rails. Thus, even with one camera 11, it is possible to generate a captured image in which a subject is captured from different positions.

  In addition, when the camera 11 can change the position, the zoom function may be a zoom function by changing the position of the camera 11. As one example, the zoom function may be a dolly zoom function. For example, zooming in may be performed by moving the camera 11 toward the subject, and zooming out may be performed by moving the camera 11 in a direction away from the subject. Note that the dolly zoom does not have to be a fine zoom like optical zoom or digital zoom. For example, in zoom-in, a captured image in which a subject is captured in a larger size may be generated, and in zoom-out, a captured image in which a subject is captured in a smaller size may be generated.

(Microphone 13)
The microphone 13 collects sound around the microphone 13. In the center office 10, a plurality of microphones 13 are installed. And each installed microphone 13 collects the sound around each installation position in a center office. As described above, in the information processing system, sounds at various positions in the center office 10 are collected.

(Sensor 15)
Sensor 15 may include various types of sensors. For example, the sensor 15 includes a seat sensor that determines whether a person is in a seat. The seat sensor is installed in each seat and determines whether a person is sitting in each seat. The seat sensor is, for example, any sensor that can detect a press.

(Media distribution server 17)
The media distribution server 17 distributes media (for example, audio, video, etc.) to the terminal device in response to the request.

(Information management server 200)
The information management server 200 manages various information used in the information processing system. That is, the information management server 200 stores the various information and updates the various information in a timely manner.

  For example, the information management server 200 manages parameters related to the camera 11, the microphone 13, and the sensor 15. Specifically, for example, the information management server 200 stores and updates information such as the installation position of the camera 11, the imaging direction (for example, the direction perpendicular to the lens of the camera), and the zoom ratio as parameters of the camera 11. .

  Also, for example, the information management server 200 manages data of a three-dimensional virtual space corresponding to a real space. The three-dimensional virtual space is, for example, a three-dimensional virtual space imitating the center office 10. In addition, an object is arranged in the three-dimensional virtual space. For example, the object corresponds to a person. Then, the object is arranged at a three-dimensional virtual position in the three-dimensional virtual space corresponding to the position of each seat of the center office 10. That is, when the person is sitting in the seat, the object is placed at a three-dimensional virtual position corresponding to the position where the person would be. As an example, the object is a cylindrical object. The three-dimensional virtual space will be described later.

(LAN 19)
The LAN 19 is a network that connects the devices in the center office 10. The LAN 19 also connects devices in the center office 10 to devices outside the center office 10 via the external network 30. The external network 30 includes, for example, the Internet.

(Terminal device 100)
The terminal device 100 is used by a user. For example, the terminal device 100 provides the user with a function for performing communication such as telephone calls and mails. The terminal device 100 is a tablet terminal as an example. The terminal device 100 may be another device having a display function and a communication function, such as a smartphone, a PC (Personal Computer), and a telephone with a display, instead of the tablet terminal.

(Display 21)
The display 21 displays one of the screens. For example, the display 21 displays a screen including a captured image generated through the camera 11. Thereby, many persons including the user of the terminal device 100 can view the state of the center office 10 through the display 21.

  The display 21 also outputs, for example, any sound. For example, the display 21 outputs the sound collected by the microphone 13. Thereby, many persons including the user of the terminal device 100 can hear the sound of the center office 10 through the display 21.

(LAN23)
The LAN 23 is a network connecting devices in the satellite office 20 (or home office 20). Further, the LAN 23 connects each device in the satellite office 20 to a device outside the satellite office 20 via the external network 30.

(PBX 40)
The PBX 40 enables communication between devices via the external network 30. For example, in the H. It operates according to H.323 or SIP (Session Initiation Protocol).

  Specifically, for example, the PBX 40 stores communication identification information (for example, a telephone number) and an IP (Internet Protocol) address in association with each other. Then, in response to the request, the PBX 40 converts communication identification information into an IP address, and provides the IP address to the request source.

  The PBX 40 may be connected to the LAN 19 or the LAN 23.

<< 2. Terminal Configuration >>
Then, with reference to Drawing 2-Drawing 19, an example of composition of terminal unit 100 concerning this embodiment is explained.

<2-1. Hardware configuration >>
First, an example of the hardware configuration of the terminal device 100 according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing an example of the hardware configuration of the terminal device 100 according to the present embodiment. Referring to FIG. 2, the terminal device 100 includes a central processing unit (CPU) 801, a read only memory (ROM) 803, a random access memory (RAM) 805, a bus 807, a storage device 809, a communication interface 811, a camera 813 and a microphone. 815, a speaker 817 and a touch panel 820 are provided.

  The CPU 801 executes various processes in the terminal device 100. The ROM 803 also stores programs and data for causing the CPU 801 to execute processing in the terminal device 100. The RAM 805 temporarily stores programs and data when the CPU 801 executes processing.

  The bus 807 connects the CPU 801, the ROM 803, and the RAM to one another. Further, a storage device 809, a communication interface 811, a camera 813, a microphone 815, a speaker 817, and a touch panel 820 are connected to the bus 807. The bus 807 includes, for example, a plurality of types of buses. As one example, the bus 807 includes a high speed bus connecting the CPU 801, the ROM 803 and the RAM 805, and one or more other buses slower than the high speed bus.

  The storage device 809 stores data to be temporarily or permanently stored in the terminal device 100. The storage device 809 may be, for example, a magnetic storage device such as a hard disk, or an EEPROM (Electrically Erasable and Programmable Read Only Memory), a flash memory, a MRAM (Magnetoresistive Random Access Memory). The memory may be a non-volatile memory such as a ferroelectric random access memory (FeRAM) and a phase change random access memory (PRAM).

  The communication interface 811 is communication means included in the terminal device 100, and communicates with an external device via a network (or directly). The communication interface 811 may be an interface for wireless communication, and in this case, may include, for example, a communication antenna, an RF circuit, and other circuits for communication processing. The communication interface 811 may be an interface for wired communication, and in this case, may include, for example, a LAN terminal, a transmission circuit, and other circuits for communication processing.

  The camera 813 captures an image of a subject. The camera 813 includes, for example, an optical system, an imaging device, and an image processing circuit.

  The microphone 815 collects surrounding sounds. The microphone 815 converts ambient sound into an electrical signal and converts the electrical signal into digital data.

  The speaker 817 outputs sound. The speaker 817 converts digital data into an electrical signal and converts the electrical signal into sound.

  The touch panel 820 includes a touch detection surface 821 and a display surface 823.

  The touch detection surface 821 detects a touch position on the touch panel 820. More specifically, for example, when the user touches the touch panel 820, the touch detection surface 821 senses the touch, generates an electric signal according to the position of the touch, and detects the electric signal at the touch position. Convert to information. The touch detection surface 821 may be formed, for example, according to an arbitrary touch detection method such as an electrostatic capacitance method, a resistive film method, or an optical method.

  The display surface 823 displays an output image (that is, a display screen) from the terminal device 100. The display surface 823 can be realized using, for example, a liquid crystal, an organic light-emitting diode (OLED), a cathode ray tube (CRT), or the like.

2-2. Functional configuration >>
Next, an example of a functional configuration of the terminal device 100 according to the present embodiment will be described. FIG. 3 is a block diagram showing an example of a functional configuration of the terminal device 100 according to the present embodiment. Referring to FIG. 3, the terminal device 100 includes a communication unit 110, an input unit 120, an imaging unit 130, a sound collection unit 140, a display unit 150, an audio output unit 160, a storage unit 170, and a control unit 180.

(Communication unit 110)
The communication unit 110 communicates with other devices. For example, the communication unit 110 is directly connected to the LAN 23 and communicates with each device in the satellite office 20. The communication unit 110 also communicates with each device in the center office 10 via the external network 30 and the LAN 19. Specifically, for example, the communication unit 110 communicates with the camera 11, the microphone 13, the sensor 15, the media distribution server 17, and the information management server 200. The communication unit 110 may be implemented by, for example, the communication interface 811.

(Input unit 120)
The input unit 120 receives an input by the user of the terminal device 100. Then, the input unit 120 provides the input result to the control unit 180.

  For example, the input unit 120 detects the position designated by the user on the display screen. More specifically, for example, the input unit 120 is mounted by the touch detection surface 821 and detects a touch position on the touch panel 820. Then, the input unit 120 provides the detected touch position to the control unit 180.

(Imaging unit 130)
The imaging unit 130 captures an image of a subject. For example, the imaging unit 130 captures an area in the front direction of the terminal device 100. In this case, the imaging unit 130 captures an image of the user of the terminal device 100. The imaging unit 130 provides the control unit 180 with an imaging result (that is, a captured image). The imaging unit 130 may be mounted by, for example, the camera 813.

(Sound collecting unit 140)
The sound collecting unit 140 collects sounds around the terminal device 100. For example, the sound collection unit 140 collects the voice of the user of the terminal device 100. The sound collection unit 140 provides the control unit 180 with the sound collection result (i.e., audio data). The sound collection unit 140 may be implemented by the microphone 815, for example.

(Display unit 150)
The display unit 150 displays an output image (that is, a display screen) from the terminal device 100. The display unit 150 displays a display screen according to control by the control unit 180. Note that the display unit 150 can be mounted by, for example, the display surface 823.

(Voice output unit 160)
The voice output unit 160 outputs voice from the terminal device 100. The voice output unit 160 outputs voice in accordance with control by the control unit 180. The audio output unit 160 may be implemented by, for example, the speaker 817.

(Storage unit 170)
The storage unit 170 stores programs and data for the operation of the terminal device 100. The storage unit 170 may be implemented by, for example, the storage device 809.

  For example, the storage unit 170 stores data of a three-dimensional virtual space corresponding to a real space. Specifically, for example, since the information management server 200 stores data of the three-dimensional virtual space corresponding to the center office 10, the control unit 180 can communicate with the three-dimensional virtual space via the communication unit 110. Get data Then, the storage unit 170 stores the acquired data of the three-dimensional virtual space.

(Control unit 180)
The controller 180 provides various functions of the terminal device 100. Control unit 180 includes real space information providing unit 181, position acquisition unit 183, object selection unit 185, ID acquisition unit 187, telephone unit 189, camera line control unit 191, camera line selection unit 193, and camera line recommendation unit 195. . The control unit 180 can be implemented by, for example, the CPU 801, the ROM 803, and the RAM 805.

(Real space information provider 181)
The real space information provision unit 181 provides the user of the terminal device 100 with information of the real space.

  For example, the real space information provision unit 181 causes the display unit 150 to display a display screen of a captured image of the real space. More specifically, for example, the captured image is a captured image generated through the camera 11 in the real space (center office 10). The captured image may be a captured image generated by the camera 11 or may be a captured image generated by processing the captured image generated by the camera 11. Further, the display screen is a screen including the captured image in part or in whole.

  The real space information provision unit 181 acquires a captured image generated by the camera 11 via the communication unit 110. Then, the real space information provision unit 181 generates a display screen including the captured image, and causes the display unit 150 to display the display screen.

  Also, for example, the captured image is a captured image generated through a selected one of the plurality of imaging devices in the real space. More specifically, for example, the captured image is a captured image generated through one camera 11 selected from the plurality of cameras 11 disposed in the center office 10. The specific method regarding how the user selects the camera 11 will be described later. Since the camera 11 can be selected, the user can view a captured image from a desired position. And when specifying the position of a captured image so that it may mention later, a user can specify a position with a more desirable captured image.

  Also, for example, the display screen includes a captured image according to the display mode. More specifically, for example, in the first display mode, the display screen includes a first captured image obtained by capturing a first area of the real space, and in the second display mode, the first area. A narrower second region includes the second captured image captured. That is, the real space information provision unit 181 causes the display unit 150 to display the first captured image in the first display mode, and displays the second captured image on the display unit 150 in the second display mode. Let

  More specifically, for example, the first captured image is a captured image corresponding to a first zoom ratio. The second captured image is a captured image corresponding to a second zoom ratio that is larger than the first zoom ratio. For example, the real space information providing unit 181 requests the camera 11 to perform zoom (optical zoom, digital zoom, or zoom by changing the position of the imaging device (for example, zoom by dolly)) via the communication unit 110. Thus, the captured image corresponding to the first zoom factor or the captured image corresponding to the second zoom factor is acquired. Alternatively, the real space information provision unit 181 may generate a captured image corresponding to the first zoom ratio or a captured image corresponding to the second zoom ratio by digital zoom on the captured image generated by the camera 11 . Note that the zoom ratio here does not have to be a precise value such as 1.5 times or 2 times, and may be a value that directly or indirectly indicates the extent to which the subject appears in the captured image. . For example, particularly when zoom by changing the position of the camera 11 (for example, zoom in and zoom out by dolly) is used, the zoom factor is not a minute value such as 1.5 times or 2 times, but the size of the object. Something that directly indicates the degree of size (eg, a parameter that indicates the degree of approximate size of the subject, etc.), or that indirectly indicates the degree of size of the subject (eg, the position of the camera 11 on the rail , Etc.). The captured image corresponding to the first zoom ratio is a captured image in which the subject is captured smaller, and the captured image corresponding to a second zoom ratio larger than the first zoom ratio is captured in a larger object. The captured image is sufficient.

  As an example, the display screen includes a bird's-eye view image generated by the camera 11 in imaging at a zoom ratio of X (for example, X = 1) in the bird's-eye mode, and in the proximity mode It includes a close-up captured image generated by the camera 11 in imaging at Y> X). That is, the overhead image is a captured image in which a wider area of the center office 10 is captured, and the close-up captured image is a captured image in which a narrower area of the center office 10 is captured. Hereinafter, this example will be described with reference to FIGS. 4 and 5.

-Display Screen Displayed in Eyebrow Mode FIG. 4 is an explanatory diagram for describing an example of a display screen displayed in the eyelid mode. Referring to FIG. 4, a display screen 60 displayed in the overhead mode is shown. The display screen 60 includes an eyelid captured image 61, a button image 63, a presence icon 65, a balloon image 67, and a map image 69.

  The bird's-eye view pickup image 61 is, for example, a pickup image generated by the camera 11 by imaging at a zoom ratio of X times. As an example, X = 1. That is, the overhead view captured image 61 is a captured image generated by the camera 11 by imaging without zooming.

  Also, for example, when the user specifies the position of the overhead view captured image 61, the real space information provision unit 181 switches the display mode from the overhead mode to the proximity mode. More specifically, for example, when the user touches the position of the eyelid captured image 61 and the touch position corresponding to the eyelid captured image 61 is detected, the real space information provision unit 181 brings the display mode closer to the eyelid mode. Switch to mode.

  The button image 63 is an image for selecting another camera 11. For example, when the user specifies the position of the button image 63, the real space information provision unit 181 acquires a bird's-eye view captured image generated by another camera 11, and causes the display unit 150 to display the bird's-eye view captured image. More specifically, for example, when the user touches the position of the button image 63 and the touch position corresponding to the button image 63 is detected, the real space information providing unit 181 generates an image generated by another camera 11. A captured image is acquired, and the overhead view captured image is displayed on the display unit 150. For example, when the position of the button image 63A is designated by the user, the camera 11 located on the left side of the current camera 11 is selected. In addition, when the position of the button image 63B is designated by the user, the camera 11 located on the right side of the current camera 11 is selected. Then, the real space information provision unit 181 acquires the overhead view captured image generated by the selected camera 11, and causes the display unit 150 to display the overhead view captured image.

  Further, the presence icon 65 is an icon indicating the degree of busyness of the person shown in the captured image 61. For example, the presence icon 65 changes color according to the busyness of the person. As an example, the presence icon indicates that the degree of busyness is high when it is red, it indicates that the degree of busyness is normal when it is yellow, and that the degree of busyness is low when it is blue. As described later, since it can be known where in the captured image 61 a person should be shown, it is possible to display such an icon.

  For example, as described above, the display screen 60 includes, for example, information related to a person shown in the overhead view captured image 61 (hereinafter, referred to as “person related information”). The person related information includes, for example, state information indicating the state of the person. As described above, the state information is the presence icon 65 as an example. The person related information may include a state history indicating the state of the person at two or more points in time. As an example, the state history information may include a history of busyness of a person shown in the captured image 61. That is, on the display screen 60, a history of busyness of a person may be displayed. The real space information provision unit 181 acquires, for example, the person-related information or information necessary for displaying the person-related information from the information management server 200 via the communication unit 110.

  Such person-related information allows the user to more accurately grasp the situation in which the person is placed. In addition, the state information allows the user to more accurately determine whether to contact a person. In addition, since the state history allows the user to grasp not only the instantaneous state of the user but also the state of the user in a certain period, the user can more accurately determine whether or not a person may be contacted. be able to.

  In addition, the balloon image 67 is an image including character information presented by a person shown in the captured image 61. The balloon image 67 is also an example of person-related information.

  Further, the map image 69 is an image showing a map of the center office 10. The map image 69 further indicates the camera 11 in use by an icon. Note that the map image 69 may be omitted, especially if only one or a few cameras 11 are installed in the center office 10.

Display Screen Displayed in Proximity Mode FIG. 5 is an explanatory diagram for describing an example of a display screen displayed in the proximity mode. Referring to FIG. 5, a display screen 70 displayed in the proximity mode is shown. The display screen 70 includes a close-up captured image 71, a button image 73 and a map image 75.

  The close-up captured image 71 is, for example, a captured image generated by the camera 11 by imaging at a zoom ratio of Y (Y> X). As an example, Y = 1.5. That is, the overhead view captured image 61 is a captured image generated by the camera 11 by capturing at a 1.5 × zoom.

  The button image 73 is an image for switching the display mode from the proximity mode to the overhead mode. For example, when the user specifies the position of the button image 73, the real space information provision unit 181 switches the display mode from the proximity mode to the overhead mode. More specifically, for example, when the user touches the position of button image 73 and the touch position corresponding to button image 73 is detected, real space information provision unit 181 changes the display mode from the proximity mode to the overhead mode. Switch.

  Further, the map image 75 is an image showing a map of the center office 10 as in the case of the map image 69 in the bird's-eye view mode. The map image 75 further shows the camera 11 in use. For example, in the proximity mode, the icon of the camera in use in the map image 75 is displayed at a more advanced position to symbolically indicate that the zoom is performed. It should be noted that the map image 75 may be omitted as well as the map image 69 in the overhead mode, especially if only one or a few cameras 11 are installed in the center office 10.

  The display screen 70 displayed in the proximity mode may also include person-related information such as a presence icon and a balloon image.

  As described above, according to the display mode, the user can view the general situation of the real space by displaying a captured image in which a wider area is captured, and it is easy to find a specific person. Can. Then, by displaying a captured image in which a narrower area is captured, the user can more easily specify the position of a specific person. Also, since only the display mode is switched, only easy operation is required for the user. In addition, since the captured image in which the wider area is captured and the captured image in which the narrower area is captured have different zoom ratios, the user intuitively and easily grasps the relationship between these captured images. be able to. Therefore, even if the display mode is switched, the user can easily find a specific person and designate the position of the specific person.

-Other real space information As mentioned above, the real space information provision part 181 provides the visual information of real space by making the display part 150 display the display screen of the captured image of real space. Furthermore, the real space information provision unit 181 may also provide auditory information of real space. That is, the real space information provision unit 181 may cause the sound output unit 160 to output the sound of the sound data obtained by sound collection in the real space.

  For example, the real space information provision unit 181 selects the microphone 13 close to the selected camera 11. Then, the real space information provision unit 181 acquires voice data obtained by sound collection at the center office 10 from the microphone 13 via the communication unit 110. Then, the real space information provision unit 181 may cause the sound output unit 160 to output the sound of the acquired sound data.

(Position acquisition unit 183)
The position acquisition unit 183 acquires the position in the captured image specified by the user on the display screen of the captured image in the real space. More specifically, for example, when the input unit 120 detects a position in the captured image specified by the user on the display screen of the captured image in real space, the position acquisition unit 183 acquires the position. Then, the position acquisition unit 183 provides the object selection unit 185 with the position in the captured image.

  As an example, when acquiring any position in the close-up captured image 71, the position acquisition unit 183 provides the position to the object selection unit 185. Hereinafter, this example will be described with reference to FIGS. 6 and 7.

  FIG. 6 is an explanatory diagram for describing a first example of the position in the close-up captured image specified by the user. Referring to FIG. 6, the proximity captured image 71 included in the display screen displayed in the proximity mode and the hand 3 of the user are shown. Further, a person 77 appearing in the close-up captured image 71 is shown. Then, the user touches the position of the person 77 with the hand 3 to specify the position of the person 77 in the close-up captured image 71. In this case, the input unit 120 detects the above-mentioned position of the person 77 in the close-up captured image 71, and the position acquisition unit 183 acquires the position. Then, the input unit 120 provides the acquired position to the object selection unit 185.

  FIG. 7 is an explanatory diagram for describing a second example of the position in the close-up captured image specified by the user. Referring to FIG. 7, persons 77A and 77B appearing in the close-up captured image 71 are shown. Then, the user touches the position of the person 77A with the hand 3 to specify the position of the person 77A in the close-up captured image 71. In this case, the input unit 120 detects the position of the person 77A in the close-up captured image 71, and the position acquisition unit 183 acquires the position. Then, the input unit 120 provides the acquired position to the object selection unit 185.

(Object selection unit 185)
The object selection unit 185 selects an object arranged in the three-dimensional virtual space corresponding to the real space based on the position in the acquired captured image. For example, when the position acquisition unit 183 acquires a position in the captured image specified by the user on the display screen of the captured image in the real space, the object selection unit 185 corresponds to the real space based on the position. Select an object placed in the 3D virtual space.

  The object is an object disposed at a three-dimensional virtual position in the three-dimensional virtual space corresponding to the position in the captured image. Also, for example, the captured image is a captured image included in the display screen in the second mode (for example, the proximity mode).

  As an example, when the position acquisition unit 183 acquires the position in the close-up captured image generated by the camera 11, the object selection unit 185 selects one of the objects arranged in the three-dimensional virtual space corresponding to the center office 10. Select an object corresponding to the above position. The object selection unit 185 acquires, for example, data of a three-dimensional virtual space corresponding to the center office 10 from the storage unit 170.

-One Example when One Object is Arranged Hereinafter, with reference to FIG. 8 and FIG. 9, a concrete example when one object is arranged in the three-dimensional virtual space will be described.

  FIG. 8 is an explanatory diagram for explaining a first example of a three-dimensional virtual space corresponding to the center office 10. As shown in FIG. Referring to FIG. 8, a three-dimensional virtual space 90 corresponding to the center office 10 is shown. Further, in the three-dimensional virtual space 90, an object 91 is disposed. The object 91 corresponds to a person (for example, Mr. A). Then, the object is arranged at a three-dimensional virtual position corresponding to the position of the seat of the person (for example, Mr. A) of the center office 10. That is, the object 91 is placed at a three-dimensional virtual position corresponding to the position where the person would be present when the person is sitting in the seat. In this example, the object 91 is a cylindrical object. The cylindrical object is a cylindrical object with radius R and height H. The radius R and the height H are, for example, predetermined. The data of the three-dimensional virtual space 90 also includes information related to each camera 11. For example, the information related to each camera 11 includes a three-dimensional virtual position corresponding to the installation position of each camera 11, an imaging direction (for example, a direction perpendicular to the lens of the camera), an angle of view and the like.

  FIG. 9 is an explanatory diagram for explaining an example of selection of an object arranged in the three-dimensional virtual space shown in FIG. Referring to FIG. 9, the positional relationship in the horizontal plane in the three-dimensional virtual space 90 is shown for easy understanding. Specifically, an object 91 disposed in the three-dimensional virtual space 90, a three-dimensional virtual position (hereinafter referred to as a "virtual camera position") O corresponding to the installation position of the camera 11 used for imaging, and imaging of the camera 11 An axis y corresponding to a direction (e.g., a direction perpendicular to the lens of the camera) and an axis x orthogonal to the axis y are shown. In this example, for easy understanding, it is assumed that the camera 11 is installed such that the imaging direction of the camera 11 is parallel to the horizontal plane.

  Furthermore, the angle of view θ of the camera 11 is also shown. Further, a virtual surface 93 is shown which is perpendicular to the axis y corresponding to the imaging direction of the camera 11 and has a width corresponding to the angle of view θ. Also, the virtual plane 93 is separated from the virtual camera position O by the distance I. The virtual surface 93 is a square surface and has the same aspect ratio as the captured image. That is, the virtual surface 93 is a surface corresponding to a captured image.

  The object selection unit 185 converts, for example, the position in the captured image specified by the user as shown in FIG. 6 into a three-dimensional virtual position A. Then, the object selection unit 185 specifies an object that intersects a straight line passing the virtual camera position O and the three-dimensional virtual position A. In this example, the object selection unit 185 specifies the object 91. Then, the object selection unit 185 selects the object 91.

  In the example of FIG. 9, for example, the position in the captured image converted into the three-dimensional virtual position between the three-dimensional virtual position B and the three-dimensional virtual position D in the virtual surface 93 is the user in the captured image. When designated by the, the object 91 is selected. Note that such a position is generally a position where a person corresponding to the object 91 is shown in the captured image.

  The distance I is determined so that the virtual plane 93 is located between the virtual camera position O and the object 91. As an example, the distance I is the focal length of the camera 11, but of course is not limited thereto.

  In addition, although the method of selecting an object by focusing on the horizontal surface of the three-dimensional virtual space 90 has been described for easy explanation, naturally, even if a vertical direction (for example, z axis) is added, Can be converted into a three-dimensional virtual position to specify the object 91 from the three-dimensional virtual position.

  For example, as described above, an object corresponding to a position in a captured image is selected. In the example described above, the object corresponding to the position in the captured image is specified by converting the position in the captured image into a three-dimensional virtual position, but a method for specifying the object corresponding to the position in the captured image Not limited to this.

  As an example, the object selecting unit 185 may project the object 91 on the virtual plane 93 with the virtual camera position O as an origin, and convert the projection range of the object 91 into a range in the captured image. Then, the object selection unit 185 may select the object 91 when the position in the captured image specified by the user is included in the range.

  As another example, a rendering image is generated by rendering the three-dimensional virtual space 90 using the virtual camera position O, the axis y, and the angle of view θ, and the object 91 in the rendering image is captured. The range in the captured image corresponding to the object 91 may be obtained from the range. Then, the object selection unit 185 may select the object 91 when the position in the captured image specified by the user is included in the range.

-Selection of Object Considering the State Further, for example, the object 91 arranged in the three-dimensional virtual space 90 corresponds to state information indicating any one of a plurality of states. For example, the object 91 corresponds to a person. Then, the object 91 corresponds to state information (hereinafter referred to as “presence / absence information”) indicating whether or not the person is sitting in the seat. The presence / absence information indicates that a person is sitting in a seat or a person is not sitting in a seat. The object selection unit 185 acquires, for example, the determination result as to whether or not the person corresponding to the object 91 is sitting in the seat from the sensor 15 via the communication unit 110. And the said determination result becomes presence or absence information.

  Then, for example, the object selecting unit 185 selects the object corresponding to the state information indicating the predetermined state among the plurality of states, which is the object arranged in the three-dimensional virtual space. For example, the object selecting unit 185 selects an object arranged in the three-dimensional space, which corresponds to state information indicating a state in which a person is sitting in a seat. That is, when it is determined by the sensor 15 that a person is sitting in the seat, the object selection unit 185 can select an object corresponding to the person, but the sensor 15 determines that the person is not sitting in the seat When it is determined, the object corresponding to the person is not selected.

  In this way, since the object is selected according to the state of the person, it is possible to avoid selecting an object that should not be selected. For example, it is possible to prevent the object corresponding to the person from being selected when there is no person.

-Example in which two objects are arranged In addition, two or more objects may be arranged in the three-dimensional virtual space. Hereinafter, a specific example in which two objects are arranged will be described with reference to FIG.

  FIG. 10 is an explanatory diagram for explaining a second example of the three-dimensional virtual space corresponding to the center office 10. As shown in FIG. Referring to FIG. 10, a three-dimensional virtual space 90 corresponding to the center office 10 is shown. Further, in the three-dimensional virtual space 90, an object 91A and an object 91B are arranged. The object 91A corresponds to a certain person (for example, Mr. A), and is arranged at a three-dimensional virtual position corresponding to the position of the seat of the certain person in the center office 10. Further, the object 91 B is arranged at a three-dimensional virtual position corresponding to a certain person (for example, Mr. B) and corresponding to the position of the seat of the certain person in the center office 10. Similar to the example of FIG. 8, the object 91 is a cylindrical object with a radius R and a height H.

  When two or more objects are arranged in the three-dimensional virtual space in this manner, the plurality of objects may correspond to positions in the captured image specified by the user. For example, in the example of FIG. 10, both the object 91A and the object 91B may correspond to the position in the captured image specified by the user. As an example, as shown in FIG. 7, when two persons can appear in a captured image, when the user designates a position where the two persons overlap in the captured image, the two persons are designated. Corresponding two objects may correspond to the above position.

  Therefore, in such a case (that is, when the plurality of objects correspond to the position of the captured image specified by the user), the object selection unit 185 selects one of the plurality of objects. select. That is, when there are a plurality of objects respectively arranged in the three-dimensional virtual position in the three-dimensional virtual space corresponding to the position in the captured image specified by the user, the object selecting unit 185 Select one of the objects.

  For example, the captured image is generated through an imaging device in real space. Then, the object selection unit 185 selects an object closer to the three-dimensional virtual position in the three-dimensional virtual space corresponding to the imaging device among the plurality of objects. More specifically, for example, the captured image is generated through the camera 11 in the center office 10. Then, when there are a plurality of objects corresponding to the position in the captured image specified by the user, the object selecting unit 185 selects the three-dimensional virtual position (that is, the virtual camera position O) corresponding to the installation position of the camera 11 Select a close object. Hereinafter, a specific example will be described with reference to FIGS. 10 and 11 regarding this point.

  FIG. 11 is an explanatory diagram for explaining an example of selection of an object arranged in the three-dimensional virtual space shown in FIG. Referring to FIG. 11, the positional relationship in the horizontal plane in the three-dimensional virtual space 90 is shown for easy understanding. Specifically, an object 91A and an object 91B arranged in the three-dimensional virtual space 90 are shown. Also, as in FIG. 9, a virtual camera position O, an axis y, an axis x, an angle of view θ, and a virtual surface 93 are shown. Also in this example, as in FIG. 9, for easy understanding, it is assumed that the camera 11 is installed so that the imaging direction of the camera 11 is parallel to the horizontal plane.

  For example, as shown in FIG. 7, the position in the captured image is designated by the user. In this case, according to the method described with reference to FIG. 9, the position in the captured image is converted into a three-dimensional virtual position between the three-dimensional virtual position B ′ and the three-dimensional virtual position D. In the above, both the object 91A and the object 91B are specified as objects corresponding to the positions in the captured image. Then, the object selection unit 185 selects an object 91A closer to the virtual camera position O among the object 91A and the object 91B.

  When the position in the captured image specified by the user is converted into a three-dimensional virtual position between the three-dimensional virtual position B and the three-dimensional virtual position B ′, the object 91A is the above-described captured image It is identified and selected as the object corresponding to the position in the When the position in the captured image specified by the user is converted to a three-dimensional virtual position between the three-dimensional virtual position D and the three-dimensional virtual position D ′, the object 91 B is the above-described captured image It is identified and selected as the object corresponding to the position in the

  By selecting one object from a plurality of objects as described above, it is possible to prevent a plurality of objects from being selected and an error (an error due to acquiring a plurality of communication IDs) occurring in the subsequent processing. be able to. Further, by selecting an object close to the virtual camera position corresponding to the camera 11 among a plurality of objects, for example, even when people overlap in the captured image, it corresponds to the person appearing in the foreground Objects to be selected. Thus, an object corresponding to the person intended by the user is selected.

  As described above, the object selection unit 185 selects an object. Then, the object selection unit 185 provides the ID acquisition unit 187 with identification information of the selected object (hereinafter referred to as “object ID”). The object ID may be identification information of a person corresponding to the selected object, or may be a mere number assigned to the selected object.

(ID acquisition unit 187)
The ID acquisition unit 187 acquires identification information corresponding to the selected object. For example, the identification information is identification information for communication (hereinafter referred to as “communication ID”) corresponding to the selected object. The communication ID is, for example, a telephone number.

  Specifically, for example, when the object selection unit 185 selects an object, the ID acquisition unit 187 acquires the object ID of the selected object. Then, the object selection unit 185 transmits the object ID to the information management server 200 via the communication unit 110, and acquires a communication ID corresponding to the object ID. The communication ID is a communication ID of a person corresponding to the selected object.

  For example, as described above, the ID acquisition unit 187 acquires a communication ID corresponding to the selected object. The communication ID is a communication ID of the communication device of the person corresponding to the object (that is, the person appearing at the position of the captured image specified by the user). Then, the ID acquisition unit 187 provides the acquired communication ID to the telephone unit 189.

  As described above, when the position in the captured image is designated by the user, the object corresponding to the position is selected, and the communication ID corresponding to the object is acquired. This enables the user to contact the target person with an intuitive operation. Further, regardless of how the person is shown in the captured image, the object corresponding to the person is selected and the communication ID is acquired, so that the person can be more reliably contacted.

(Telephone part 189)
The telephone unit 189 provides a function for making a call. For example, the telephone unit 189 provides a softphone function.

  For example, when the telephone unit 189 acquires the communication ID provided by the ID acquisition unit 187, the telephone unit 189 uses the communication ID to make a call. More specifically, for example, when the telephone unit 189 acquires a communication ID, the telephone unit 189 provides the communication ID to the PBX 40 via the communication unit 110, and acquires an IP address from the PBX 40. Then, the telephone unit 189 executes a series of sequences for establishing a session with the communication apparatus having the IP address (that is, the communication apparatus at the destination).

  In this manner, the telephone unit 189 makes a call for a call to a person appearing at a position in the captured image specified by the user on the display screen. That is, the telephone unit 189 makes a call to the communication device of the person.

  When voice data from the communication apparatus of the other party of the telephone call is received by the communication unit 110, the telephone unit 189 causes the voice output unit 160 to output the voice of the voice data. In addition, the telephone unit 189 causes the communication unit 110 to transmit the audio data provided by the sound collection unit 140 to the communication device at the other end of the telephone. Also, the telephone unit 189 causes, for example, the communication unit 110 to also transmit a captured image provided by the imaging unit 130 (for example, a captured image in which a user of the terminal device 100 is shown) to the communication device of the other party of the telephone.

  Further, the telephone unit 189 causes the display unit 150 to display a display screen at the time of a telephone call. For example, a display mode for displaying a display screen at the time of telephone call is called a conversation mode. In this case, when the telephone unit 189 acquires the communication ID provided by the ID acquisition unit 187, the telephone unit 189 switches the display mode from the proximity mode to the conversation mode. Hereinafter, a concrete example of the display screen in the conversation mode will be described with reference to FIG.

  FIG. 12 is an explanatory diagram for describing an example of a display screen displayed in the conversation mode. Referring to FIG. 12, a display screen 80 displayed in the conversation mode is shown. The display screen 80 includes the other-side captured image 81, the button image 83, and the own-side captured image 85.

  The other party side captured image 81 is, for example, a captured image acquired from the communication device of the other party of the telephone. For example, when the captured image from the communication device of the other party of the telephone call is received by the communication unit 110, the telephone unit 189 uses the captured image as the other-side captured image 81.

  Also, the button image 63 is an image for ending the telephone call. For example, when the user specifies the position of the button image 83, the telephone unit 189 terminates the call. More specifically, for example, when the user touches the position of the button image 83 and the touch position corresponding to the button image 83 is detected, the telephone unit 189 executes a sequence for call termination including disconnection of the session. Do. Also, for example, the telephone unit 189 switches the display mode from the conversation mode to the eyebrow mode.

  The self-side captured image 85 is a captured image provided by the imaging unit 130.

(Other: Transition of display mode)
Here, a specific example of the transition of the habit mode, the proximity mode, and the conversation mode will be described with reference to FIG.

  FIG. 13 is a transition diagram for explaining an example of transition of the display mode. Referring to FIG. 13, for example, when connection processing with the camera 11, the microphone 13, the sensor 15, the information management server 200, and the like is performed at the time of activation of the software, the display mode is the bird's eye mode 301 (ENTRY).

  In the overhead mode 301, overhead mode processing is performed (DO). Then, when the user specifies the position of the overhead view captured image 61, mode change processing is performed (EXIT), and the display mode is switched from the overhead mode 301 to the proximity mode 303. The mode change process includes the zoom process of the camera 11 (ENTRY).

  In the proximity mode 303, proximity mode processing is performed (DO). Then, when the user designates the position where the person in the close-up shot image 71 is taken, mode change processing is performed (EXIT), and the display mode is switched from the close mode 303 to the conversation mode 305. The mode change process in this case includes a process for a call (ENTRY). Further, when the user designates the position of the button image 73, mode change processing is performed (EXIT), and the display mode is switched from the proximity mode 303 to the overhead mode 301. The mode change process in this case includes the connection process described above (ENTRY).

  In the conversation mode 305, conversation mode processing is performed (DO). Further, when the user designates the position of the button image 83, mode change processing is performed (EXIT), and the display mode is switched from the conversation mode 305 to the overhead mode 301. The mode change process in this case includes the connection process described above (ENTRY).

(Others: Another example of the display screen)
-A captured image according to the display mode As described above, for example, in the first display mode (for example, the eyelid mode), the real space information provision unit 181 is the first image in which the first region of the real space is imaged. A captured image (for example, a bird's-eye view captured image) is displayed on the display unit 150. Further, in the second display mode (for example, the proximity mode), the real space information provision unit 181 is a second captured image (for example, the proximity captured image) in which the second area narrower than the first area is captured. ) Is displayed on the display unit 150. Then, in the above-described example, the first captured image in which the first region of the real space is captured is a captured image corresponding to the first zoom ratio, and the second image is narrower than the first region. The second captured image obtained by capturing the area of is a captured image corresponding to a second zoom ratio that is larger than the first zoom ratio. However, the first captured image and the second captured image are not limited to this.

  For example, the first captured image is a captured image generated through the first imaging device of the plurality of imaging devices in real space, and the second captured image is the image of the plurality of imaging devices. It may be a captured image generated through the second imaging device. In this case, for example, the real space information provision unit 181 selects the camera 11 that captures a wide area of the center office 10 in the overhead mode, and uses the captured image generated by the camera 11 as the overhead captured image get. Then, in the proximity mode, the real space information provision unit 181 selects a camera 11 (for example, a camera further ahead) that images a narrower area of the center office 10, and the image generated by the camera 11 An image is acquired as a close-up image.

  Thereby, depending on the arrangement of the camera 11, it becomes easier to specify the position of the person. In addition, the camera 11 may not request the zoom. Therefore, as in the case of requesting an optical zoom or a zoom by dolly, for example, requests from individual terminal devices do not compete, and no waiting state occurs in any of the terminal devices. In addition, as in the case of using digital zoom, for example, the amount of processing does not increase.

-A captured image generated by imaging under more free conditions In the above-described example, although an example in which the display mode is switched has been described, the display screen is not limited to this. For example, instead of switching the display mode, a captured image may be generated by imaging under more free conditions, and a display screen including the captured image may be displayed.

  For example, the captured image of the real space may be a captured image corresponding to the selected zoom ratio of the plurality of zoom ratios. In this case, for example, the real space information provision unit 181 requests the camera 11 via the communication unit 110 for the zoom factor designated by the user via the input unit 120. As a result, the camera 11 changes the zoom factor according to the request, and provides the terminal device 100 with a captured image generated by imaging at the changed zoom factor. Then, the real space information providing unit 181 causes the display unit 150 to display a display screen including the provided captured image. Then, when the user designates a position in the captured image, the position acquisition unit 183 acquires the position. Then, the position acquisition unit 183 provides the object selection unit 185 with the position in the captured image.

  Thus, the user can specify a fine zoom factor to display a desired captured image. Thus, the user can more easily specify the position of a specific person.

  Note that, as described above, the zoom ratio here does not have to be a precise value such as 1.5 times or 2 times, and directly or indirectly indicates the degree of the size of the subject in the captured image. It is a thing. For example, particularly when zoom by changing the position of the camera 11 (for example, zoom in and zoom out by dolly) is used, the zoom factor is not a minute value such as 1.5 times or 2 times, but the size of the object. Something that directly indicates the degree of size (eg, a parameter that indicates the degree of approximate size of the subject, etc.), or that indirectly indicates the degree of size of the subject (eg, the position of the camera 11 on the rail , Etc.).

(Camera line control unit 191)
The camera line control unit 191 performs arithmetic processing of various parameters related to a camera line, which will be described later, display control of the camera line, and acquisition of a user input to the camera line. Here, “camera line” is a coined word in the embodiment of the present invention, and is a line indicating a display element and an input element of the user interface displayed on the display screen of the terminal device 100.

  More specifically, as described above, the position acquisition unit 183 acquires the selected position based on the specification of the user, but the camera line corresponds to the selected position in the three-dimensional virtual space 90 corresponding to the center office 10 Can be a line (hereinafter, also referred to as “three-dimensional line”) according to the three-dimensional selection position to be selected and the three-dimensional installation position of the camera 11 that includes the three-dimensional selection position in the imaging range (described later with FIG. 14) To do). In the following description, although the position of the object corresponding to the position designated by the user is used as an example of the selected position based on the designation by the user, the position itself designated by the user may be used.

  In the present embodiment, a straight line is described as an example of the “line”, but the “line” may be a bent line. Also, for example, the line corresponding to the three-dimensional selection position and the three-dimensional installation position may be the entire line having the selection position and the installation position at both ends, or a line at both ends being the selection position and the installation position. It may be part of In the following description, “camera line” can also mean a line (two-dimensional line) obtained by projecting a three-dimensional line onto a corresponding position on the display screen of a captured image.

  As described above, the data of the three-dimensional virtual space 90 also includes information related to each camera 11. For example, information related to each camera 11 is a three-dimensional virtual position corresponding to the installation position of each camera 11, an imaging direction (for example, a direction perpendicular to the lens of the camera including the optical axis), an angle of And so on. The terminal device 100 can use the information to identify which camera 11 includes the three-dimensional selection position in the imaging range. The information may be stored in the information management server 200, and the terminal device 100 may appropriately acquire the information through the communication unit 110 when a situation requiring the information occurs.

  The camera line is projected and superimposed on the display screen of the terminal device 100, for example, the bird's-eye view image 61 or the map image 69 in FIG. 4 (described later with reference to FIG. 15). The user of the terminal device 100 can view a camera line displayed on the display screen to obtain a camera capable of capturing a three-dimensional selection position from an angle different from that of the camera capturing the captured image currently being viewed. Intuitively know whether or not the camera exists, and if there is a relative positional relationship between the camera, the camera that captures the image being viewed, and the three-dimensional selection position. be able to.

  FIG. 14 is an explanatory diagram for describing an example of a camera line in the three-dimensional virtual space corresponding to the center office 10.

  Referring to FIG. 14, a three-dimensional virtual space 90 corresponding to the center office 10 is shown. Further, in the three-dimensional virtual space 90, an object 91A and an object 91B are arranged. The object 91A corresponds to a certain person (for example, Mr. A), and is arranged at a three-dimensional virtual position corresponding to the position of the seat of the certain person in the center office 10. Further, the object 91 B is arranged at a three-dimensional virtual position corresponding to a certain person (for example, Mr. B) and corresponding to the position of the seat of the certain person in the center office 10. Similar to the example of FIG. 8 and FIG. 9, the object 91 is a cylindrical object with a radius R and a height H. Also, three-dimensional gravity center positions 92A and 92B of the object 91A and the object 91B are illustrated.

  Furthermore, in the three-dimensional virtual space 90, virtual cameras 93C, virtual cameras 93D, and virtual cameras 93E are disposed, and virtual camera positions (for example, center positions of lenses of virtual cameras) 94C, 94D, respectively corresponding thereto. There is 94E. Further, line segments respectively having the three-dimensional gravity center position 92A of the object 91A and the virtual camera positions 94C, 94D, 94E at both ends, that is, camera lines 95C, 95D, 95E are illustrated. Further, a camera line 95F having the three-dimensional gravity center position 92B of the object 91B and the virtual camera position 94C at both ends is also shown for reference. In this example, the object 91B is not included in the imaging range of the virtual cameras 93C and 93E, but is included only in the imaging range of the virtual camera 93D, so only one camera line for the object 91B is present. I did not.

  FIG. 15 is an explanatory diagram for describing an example of a camera line displayed on the display screen of the terminal device 100.

  Referring to FIG. 15, a display screen 80 displayed in the eyebrow mode or the proximity mode is shown. The display screen 80 includes a captured image 81, a map image 84, and camera lines 87C, 87D, 87E, 86C, 86D, and 86E projected and displayed superimposed. In addition, although the mark on the ring is superimposed and drawn near the person image corresponding to the object according to the position designated by the captured image 81, the object according to the designated position is on the screen It is the result of feedback display to the user who is operating about which of.

  The captured image 81 is an image similar to the overhead captured image 61 or the proximity captured image 71. The map image 84 is an image similar to the map image 69 or the map image 75. That is, the display of the camera line can be realized in the same manner in either the overhead mode or the proximity mode.

  The camera line 87 is superimposed and displayed in the vicinity of the image of the object by an operation of the user specifying the image of the object in the captured image 81 (for example, an operation of touching the position of the object). When an object is designated on the map image 84, the camera line 86 is superimposed and displayed. At this time, the camera line drawn on the display screen 80 is a line where the camera line in the three-dimensional virtual space 90 corresponding to the center office 10 is projected onto the two-dimensional plane of the captured image 81 or the map image 84, Two-dimensional line segments having the object and the camera shown in the captured image 81 at both ends become drawing elements.

  Note that at least one of the camera and the object may not necessarily be projected inside the captured image 81. At this time, a portion of the three-dimensional line that is not projected inside the captured image 81 may or may not be drawn out of the captured image 81. The user corresponds to the camera line by estimating the extension destination of the line segment drawn in the inside of the captured image 81 even if the part of the three-dimensional line that is not projected inside the captured image 81 is not drawn You can roughly grasp the position of the camera or object.

  In addition, at least one of the camera and the object may not necessarily be projected inside the display screen 80. At this time, a portion of the three-dimensional line which is not projected inside the display screen 80 may not be drawn. Even if a portion of the three-dimensional line that is not projected inside the display screen 80 is not drawn, the user corresponds to the camera line by estimating the extension destination of the line segment drawn inside the display screen 80. You can roughly grasp the position of the camera or object.

  Furthermore, the camera line can be not only a display element but also an input element. Therefore, the user can designate the camera 11 in the real space corresponding to the designated and inputted camera line by touching any one of the camera lines and performing the designation operation. At this time, the captured image 81 is switched to the overhead captured image generated by the designated camera 11 in response to the designating operation. This is realized by the real space information provision unit 181 acquiring the overhead view captured image generated by the designated camera 11 and displaying the overhead view captured image on the display unit 150.

  The user can intuitively know the relative positional relationship between the designated object and the switching destination camera by the camera line before the camera switching operation. In addition, the camera line itself functions as a controller for input operation, whereby an intuitive and easy input method of camera switching operation is realized.

  For example, a camera switching button is provided in the terminal device 100, and a conventional camera switching operation input method is assumed in which when the button is pressed, the camera 11 switches in the order specified by the system. When such an input method is used, the user can not know which camera has been switched to until he / she sees a captured image after camera switching, and the above-described mutual positional relationship of the plurality of cameras is not understood and cognitively Problems (Spatial Disruptions) are likely to occur. According to the input method according to the present embodiment using a camera line, it is possible to solve the problems of such a conventional camera switching input method.

  Furthermore, the camera line also has an operation input advantage in addition to the advantages described above. For example, the user touches an arbitrary one of the camera lines displayed after designating input of an object to perform designating operation, and then further touches an arbitrary position on the designated camera line to perform designating operation. The zoom factor after switching can be designated and input.

  For example, if the middle point of the camera line is touched and specified, the viewpoint position specification is performed such that the object is viewed from the position of the middle point between the object and the camera at both ends of the line segment of the camera line. Not only it is regarded as being, it switches to the camera corresponding to the specified camera line, but it is displayed on the captured image 81 as a close-up captured image having a zoom ratio of 2 times. Similarly, if a position about one third of the object side end point on the camera line is touched and designated, a close-up captured image with a zoom ratio of 3 times that of the camera line corresponds to the captured image 81 Is displayed.

  Note that, as described above, at least one of the camera and the object may not necessarily be projected inside the captured image 81. At this time, among the camera side end point and the object side end point in the camera line, the position of the end point projected to the outside of the captured image 81 may be used as it is for calculating the zoom ratio. For example, it may be used to calculate the zoom factor after being corrected to the intersection of the camera line in the captured image and the end of the captured image.

  In addition, at least one of the camera and the object may not necessarily be projected inside the display screen 80. At this time, among the camera side end point and the object side end point in the camera line, the position of the end point projected to the outside of the display screen 80 may be used as it is for calculation of the zoom ratio. For example, the correction may be made to the zoom ratio after being corrected to the intersection of the camera line on the display screen 80 and the end of the display screen 80).

  The above two input methods to the camera line, that is, a touch input for selecting and designating a camera to be switched from among a plurality of camera lines, and for designating a zoom ratio for one selected camera line The touch input can be integrated into one touch input and efficiently performed instead of a plurality of touch inputs.

  For example, when the user first presses the image area of an object in the captured image 81 with a finger, a camera line corresponding to the camera 11 capable of capturing the object is superimposed on the display screen 80. At this time, the user can drag the finger from the position of the object on the screen to the desired position on the camera line without releasing the finger from the display screen, that is, the touch panel 820, and then release the finger from the touch panel 820. It is possible to perform a designation input of a camera for capturing a captured image to be viewed by a touch input (drag) and a designation input of a magnification to the system by a touch input (drag).

  In this way, by introducing the method according to the present embodiment, the input that conventionally required three steps of designating input operations such as designating an object, designating a camera desired to be switched, and designating a zoom magnification of that camera What can be done with one touch input brings about a large usability improvement to the user who uses the remote video transmission application having a multiple camera switching function also from the viewpoint of efficiency.

  FIG. 16 is an explanatory diagram for describing some examples of touch input for specifying a position on a camera line and feedback display therefor. FIG. 16 shows the camera line displayed on the display screen 80 and the user's hand making a touch input thereto.

  In the example shown in FIG. 16A, when the user designates the position on the camera line by tapping or dragging, the user changes the color or density of the partial area corresponding to the position on the camera line. Returns feedback information about the specified input position.

  In the example shown in FIG. 16B, the user performs an input operation to move the end point of the camera line (by the user performing the input operation to change the length of the camera line), the end point on the camera line Specify the position of. The touch input method at this time is preferably dragging. Note that the maximum length that can be specified by the camera line may be displayed in advance.

  The example shown in FIG. 16C is the same as FIG. 16B in that the user designates the position on the camera line by performing an input operation to change the length of the camera line. However, in the example shown in FIG. 16C, pressing is performed by pressing and holding (long press) on the camera line rather than dragging the end point of the camera line to adjust the length of the camera line. The endpoints are moved according to the length of time. Note that feedback display may be performed in which the position of the partial area is changed as illustrated in FIG. 16A according to the length of the pressing time.

  FIG. 17 is an explanatory diagram for describing an example of selection of an object arranged in the three-dimensional virtual space shown in FIG. Referring to FIG. 17, the positional relationship in the horizontal plane in the three-dimensional virtual space 90 is shown for easy understanding. Specifically, the virtual camera position of the virtual camera 93E corresponding to the object 91A disposed in the three-dimensional virtual space 90, the virtual camera position O of the camera 11A currently used for imaging, and the camera 11B serving as the switching destination camera. A camera line 95E of the camera 11B is shown at 94E. Further, as in FIG. 9, an axis y, an axis x, an angle of view θ, and a virtual surface 93 are shown. Also in this example, as in FIG. 9, for easy understanding, it is assumed that the camera 11 is installed so that the imaging direction of the camera 11 is parallel to the horizontal plane.

  For example, as shown in FIG. 7, the position in the captured image is designated by the user. In this case, according to the method described with reference to FIG. 9, the camera line control unit 191 sets the camera line (camera line 95E in this example) corresponding to the above position in the captured image on the virtual plane 93 Projective transformation processing to E is performed, and a camera line is displayed on the display unit 150. When the user touches the AE area, the camera line control unit 191 inputs the camera line corresponding to the touch specification and the information on the two-dimensional designated position on the camera line via the input unit 120. And obtain a three-dimensional virtual position corresponding to the two-dimensional designated position by conversion.

  The obtained three-dimensional virtual position is information on the user's viewpoint position in the three-dimensional virtual space 90 desired by the user. As described above, the switching process to the camera 11 corresponding to the user's viewpoint position and the magnification of the camera 11 The terminal device 100 including the camera line control unit 191 performs specified zoom processing, and displays a close-up captured image corresponding to the captured image 81 on the display unit 150 or displays a new camera line in the close-up captured image. To be done.

  As described above, the camera line control unit 191 can perform arithmetic processing of various parameters related to the camera line, display control of the camera line, and acquisition of a user input to the camera line.

(Camera line sorting unit 193)
When a plurality of camera lines exist (for example, when a plurality of camera lines exist in the vicinity), the camera line selection unit 193 generates a designation input error of the camera line drawn on the display screen 80 or In order to solve the problem of difficulty in designating input, a process of narrowing down to one camera line optimal for the condition from among a plurality of camera lines (nearby) is presented to the user.

  FIG. 18 is an explanatory diagram for describing an example of a display screen of the terminal device 100 when there are a plurality of camera lines in the vicinity. In the following description, processing performed when camera lines are in proximity to each other will be described as an example, but a plurality of camera lines may be in the vicinity regardless of whether a plurality of camera lines are in proximity to each other. The same processing may be uniformly performed.

  Referring to FIG. 18, a captured image 81 displayed in the eyelid mode or the proximity mode is shown. Although the object corresponding to the position designated by the user and the three camera lines corresponding thereto are superimposed and displayed, since the three display positions are very close to each other, the user can select one of them as desired. It is difficult to select a camera line and perform touch input, and in some cases, there is a possibility that a camera line which is not intended may be erroneously selected and input. In addition, there is also a problem that the visibility of each camera line displayed when there are a plurality of camera lines in the vicinity before the input is impaired. The camera line sorting unit 193 performs information processing to solve the above-mentioned problems.

  Specifically, the camera line selection unit 193 compares a plurality of camera lines existing in the vicinity according to the camera line evaluation condition described later, and selects one camera line with the highest evaluation as an optimum camera line. , And processing for displaying only one of the camera lines on the display screen 80.

  Hereinafter, some examples of comparison processing corresponding to the camera line evaluation condition will be shown.

  The first camera line evaluation condition is the “distance” between the object corresponding to the position designated by the user and the camera 11. The distance is synonymous with the total length of the camera line, and in FIG. 14, the total length of the camera line 95C (or 95D, 95E), that is, between two points of the three-dimensional position 92A and 94C (or 94D, 95E) It is obtained by computing the distance of As a high-low standard of comparative evaluation, for example, the evaluation may be made higher as the camera line has a larger distance value. The reason is that if the performance of the camera 11 is the same, the viewing angle of the captured image 81 becomes wider as the distance is larger, and it is possible to grasp more widely up to the situation around the position / object designated by the user. This is because it is considered that the effect to be enjoyed by the user is more excellent, for example, the range of the zoom factor which can be designated by the user becomes wider and the degree of freedom by which the user can decide the viewpoint becomes larger.

  The second camera line evaluation condition is the “angle” between the camera line and the horizontal surface (floor surface) in the three-dimensional virtual space 90 corresponding to the center office 10. The angles are, in FIG. 14, three points 94C-92A-94C 'of the point 94C' obtained by orthogonally projecting the three-dimensional positions 92A and 94C and 94C on the horizontal surface (floor surface) of the three-dimensional virtual space 90. It is obtained by computing the angle of by the cosine theorem or the like. As a high / low standard of the comparative evaluation, for example, it is preferable that the evaluation is made higher as the camera line whose angle value is smaller and closer to zero. The reason is that the smaller the angle between the floor and the horizontal plane, the closer the rays of the camera line become horizontal, and the way the target person specified by the user looks natural from an interpersonal viewpoint (while taking a picture from above at a high angle) And the zoom expression is as natural as walking and approaching (while on the other hand, zooming from above at a high angle results in an unnatural feeling like gliding through the sky), etc. It is because it is thought that the effect to be enjoyed by is better.

  The third camera line evaluation condition is the “performance” of the camera 11 corresponding to each camera line, and is effective when the plurality of cameras 11 have different models and properties. The performance is, for example, the hardware performance of the camera, and the number of pixels of the imaging device (the higher is the evaluation) and the area of the imaging device (the larger one) Rated high), the angle of view of the lens (wide is high), the F-number of the lens (small is high), shutter speed (wide is high), maximum frame rate (large is high) ), The maximum ISO sensitivity (the larger is the high evaluation), the presence or absence of the optical zoom mechanism (the higher evaluation is), the photographing possible distance (the wide distance range is high evaluation) and the like. The reason for the high and low criteria of the comparative evaluation is that it is considered that the higher the rating of the camera, the higher the possibility that the high-quality captured image can be provided to the user.

  Further, the camera line sorting unit 193 determines whether a plurality of camera lines obtained by the camera line control unit 191 exist in the vicinity and the above-described sorting process should be performed. The determination as to whether or not the image is present in the vicinity is, for example, that a display area of a plurality of camera lines drawn on the display screen 80 is overlapped, or the distance between the display areas of the plurality of camera lines is predetermined. If the distance is less than or equal to the two-dimensional distance (for example, when the distance is less than 7 mm or 44 pixels), the corresponding camera line group is in a close positional relationship and the above-described sorting process should be performed. It may be determined that In addition, when the angle formed between the plurality of camera lines in the three-dimensional virtual space 90 is less than or equal to a predetermined value (for example, less than 10 degrees), the corresponding camera line group is in the vicinity positional relationship. It may be determined that the sorting process should be performed.

  As described above, the camera line selection unit 193 compares a plurality of camera lines existing in the vicinity according to the camera line evaluation condition described later, and selects one camera line with the highest evaluation as an optimum camera line. Processing can be performed. Then, the camera line sorting unit 193 can perform a process of displaying only the one camera line on the display screen 80.

(Camera line recommendation unit 195)
The camera line recommendation unit 195 (display control unit) causes the viewpoint preference learning unit 231 of the information management server 200 described later to calculate the user's viewing conditions for the person corresponding to the object present at the selected position based on the user specification. When the relationship between the viewing condition and the number of times of designation of the viewpoint position is calculated, the display according to the relationship calculated by the viewpoint preference learning unit 231 is controlled. Here, the viewing conditions are not particularly limited. However, in the example described later, it is assumed that the viewing condition is at least one of the distance between the person and the user in the three-dimensional virtual space and the angle of the viewpoint position of the user with respect to the person in the three-dimensional virtual space. As display control according to the relationship calculated by the viewpoint preference learning unit 231, various display controls are assumed.

  For example, the camera line recommendation unit 195 selects a three-dimensional line according to the viewing condition in which the number of designations of the viewpoint position is the largest (for example, the three-dimensional line closest to the viewing condition in which the number of designations of the viewpoint positions is the largest). And (2) selecting a two-dimensional line corresponding to the selected three-dimensional line so that the two-dimensional line corresponding to the selected three-dimensional line and the two-dimensional line corresponding to the other three-dimensional line The display mode of at least one of two-dimensional lines corresponding to another three-dimensional line is changed.

  More specifically, in the camera line recommendation unit 195, the user is selected from among a plurality of camera lines corresponding to the object, which is presented when the user specifies an object corresponding to a person present at a certain position. A camera line (hereinafter referred to as a “preference expected camera line”) that is most preferably designated is selected, and the user changes the mode so as to be distinguished from other camera lines and presents it on the display screen of the terminal device 100. The selection of the preference prediction camera line is prediction data predicted by statistical processing by a viewpoint preference learning unit 231 of the information management server 200 described later and transmitted to the camera line recommendation unit 195 (for example, the number of times of designation of viewpoint position is maximum). Viewing conditions)).

  FIG. 26 is an explanatory diagram for describing an example of a preference prediction camera line displayed on the display screen of the terminal device 100. The components of the reference numerals other than the reference numeral 89 are the same as those of FIG. On the captured image 81 or the map image 84, three superimposed camera lines are displayed.

  The camera line recommendation unit 195 receives prediction data (for example, viewing conditions where the number of times of specifying the viewpoint position is maximum) from the viewpoint preference learning unit 231 of the information management server 200 described later, and the viewpoint preference of the information management server 200 described later The prediction data is received from the learning unit 231, and it is specified that 87E (or 86E) which is one of the three camera lines is the preference prediction camera line. Then, the camera line recommendation unit 195 performs a process of changing the display mode so that the user can visually identify the preference prediction camera line and the other camera lines.

  In the example of FIG. 26, the camera line recommendation unit 195 displays a star-shaped identification symbol 89 only in the vicinity of the preference prediction camera line. As an example of the other mode change performed by the camera line recommendation unit 195, processing for displaying only the other camera lines other than the preference prediction camera line as semi-transparent or displaying no other camera line is performed. Conceivable.

  Alternatively, the camera line recommendation unit 195 does not cause the user to designate the camera line, but a three-dimensional line according to the viewing condition in which the number of designations of the viewpoint position is the largest (for example, the number of designations of the viewpoint positions is the maximum The viewpoint position in the three-dimensional line of the condition closest to the viewing condition may be calculated, and automatic control may be performed so that the captured image corresponding to the calculated viewpoint position is displayed. That is, when the user designates an object corresponding to a person present at a certain position, the camera line recommendation unit 195 automatically determines the preference prediction camera line and the position on that line as the position of the viewpoint, and the terminal device The captured image 81 from the position of the determined viewpoint may be switched and displayed on the display screen 80 by communicating with the information management server 200 at 100.

<2-3. Software configuration>
Next, an example of the software configuration of the terminal device 100 according to the present embodiment will be described. FIG. 19 is a block diagram showing an example of the software configuration of the terminal device 100 according to the present embodiment. Referring to FIG. 19, the terminal device 100 includes an operating system (OS) 840 and application software. Then, the terminal device 100 includes, as application software, a soft phone 851, an ultra-realism client 853, and a call origination control function 855.

  The OS 840 is software that provides basic functions for operating the terminal device 100. The OS 840 executes each application software.

  The softphone 851 is application software for making a call using the terminal device 100. The telephone unit 189 may be implemented by, for example, the softphone 851.

  The ultra-realism client 853 is application software for providing the terminal device 100 with information of real space. The real space information provision unit 181 may be implemented by, for example, the ultra-realism client 853.

  Note that the ultra-realism client 853 may acquire state information indicating the state of a person who is in the real space (for example, the center office 10), and may provide the softphone 851 via the OS. Then, the softphone 851 may control call origination based on the state information.

  In addition, the telephone transmission control function 855 is application software for acquiring a communication ID of a communication device of a person shown in a captured image in a display screen. The position acquisition unit 183, the object selection unit 185, and the ID acquisition unit 187 may be implemented by the telephone transmission control function 855.

  When the call origination control function 855 acquires the communication ID, it provides the softphone 851 via the OS. Then, the softphone 851 makes a call by using the communication ID.

<< 3. Information Management Server Configuration >>
Subsequently, an example of the configuration of the information management server 200 according to the present embodiment will be described with reference to FIGS. 20 and 21.

<3-1. Hardware configuration >>
First, an example of the hardware configuration of the information management server 200 according to the present embodiment will be described with reference to FIG. FIG. 20 is a block diagram showing an example of the hardware configuration of the information management server 200 according to the present embodiment. Referring to FIG. 20, the information management server 200 includes a CPU 901, a ROM 903, a RAM 905, a bus 907, a storage device 909, and a communication interface 911.

  The CPU 901 executes various processes in the information management server 200. The ROM 903 also stores programs and data for causing the CPU 901 to execute processing in the information management server 200. The RAM 905 temporarily stores programs and data when the CPU 901 executes processing.

  The bus 907 interconnects the CPU 901, the ROM 903 and the RAM. Further, a storage device 909 and a communication interface 911 are connected to the bus 907. The bus 907 includes, for example, a plurality of types of buses. As one example, the bus 907 includes a high-speed bus connecting the CPU 901, the ROM 903 and the RAM 905, and one or more other buses slower than the high-speed bus.

  The storage device 909 stores data to be temporarily or permanently stored in the information management server 200. The storage device 909 may be, for example, a magnetic storage device such as a hard disk, or a non-volatile memory such as an EEPROM, a flash memory, an MRAM, an FeRAM, and a PRAM.

  The communication interface 911 is a communication unit included in the information management server 200, and communicates with an external device via a network (or directly). The communication interface 911 may be an interface for wireless communication, and in this case, may include, for example, a communication antenna, an RF circuit, and other circuits for communication processing. The communication interface 911 may be an interface for wired communication, and in this case, may include, for example, a LAN terminal, a transmission circuit, and other circuits for communication processing.

<3-2. Functional configuration >>
Next, an example of a functional configuration of the information management server 200 according to the present embodiment will be described. FIG. 21 is a block diagram showing an example of a functional configuration of the information management server 200 according to the present embodiment. Referring to FIG. 21, the information management server 200 includes a communication unit 210, a storage unit 220, and a control unit 230. Furthermore, the control unit 230 includes a viewpoint preference learning unit 231.

(Communication unit 210)
The communication unit 210 communicates with other devices. For example, the communication unit 210 is directly connected to the LAN 19 and communicates with each device in the center office 10. Specifically, for example, the communication unit 210 communicates with the camera 11, the microphone 13, the sensor 15, and the media distribution server 17. The communication unit 210 also communicates with each device in the satellite office 20 via the external network 30 and the LAN 23. Specifically, for example, the communication unit 210 communicates with the terminal device 100 and the display 21. The communication unit 210 may be implemented by, for example, the communication interface 911.

(Storage unit 220)
The storage unit 220 stores a program and data for the operation of the information management server 200.

  In particular, in the present embodiment, the storage unit 220 stores various information used in the information processing system.

  As a first example, the storage unit 220 stores parameters related to the camera 11, the microphone 13, and the sensor 15. The specific content of the parameter is as described above.

  As a second example, the storage unit 220 stores data of a three-dimensional virtual space corresponding to a real space. The three-dimensional virtual space is, for example, a three-dimensional virtual space imitating the center office 10. The specific content of the three-dimensional virtual space is as described above.

  As a third example, the storage unit 220 stores person related information. The person-related information is, for example, person-related information of a person at the center office 10. The specific content of the person-related information is as described above.

  As a fourth example, the storage unit 220 stores the object ID of the object arranged in the three-dimensional virtual space and the communication ID in association with each other. The specific contents of the object ID and the communication ID are as described above.

  As a fifth example, the storage unit 220 links the user identification information (which may be the communication ID corresponding to each user), and records the history information of the position of the object designated by each user and the viewpoint. Furthermore, the storage unit 220 also stores information on the body direction in the space of the person corresponding to the object specified by each user. The information on the body direction in the space of the person may be determined in advance as a direction to face the desk of each person in the three-dimensional virtual space imitating the center office 10 described above. Also, for example, a gyro sensor is attached to each person in the center office 10, a three-dimensional model of each person is generated from image data of a plurality of cameras 11, a face detection process is used, etc. Information on the body direction may be acquired.

(Control unit 230)
The control unit 230 provides various functions of the information management server 200.

  For example, the control unit 230 provides various information used in the information processing system in response to a request. For example, in response to a request from the terminal device 100, the control unit 230 includes data of a three-dimensional virtual space, person-related information, a communication ID corresponding to an object ID, and parameters related to the camera 11, the microphone 13 and the sensor 15. provide.

  Also, for example, the control unit 230 appropriately updates various information used in the information processing system. The control unit 230 updates the information automatically or in response to a manual instruction.

(Viewpoint preference learning unit 231)
The viewpoint preference learning unit 231 (learning unit) responds to the three-dimensional selection position in the three-dimensional virtual space corresponding to the selection position based on the user specification and the three-dimensional installation position of the imaging device including the three-dimensional selection position in the imaging range. When the three-dimensional line is designated by the user and the imaging device corresponding to the three-dimensional line designated by the user is designated by the terminal device 100 as the viewpoint position of the user, the three-dimensional selection position and the three-dimensional installation position are used. Then, the user's viewing condition for the person corresponding to the object present at the selected position is calculated, and the relationship between the viewing condition and the designated number of viewpoint positions is calculated. As described above, the viewing conditions are the distance between the person and the user in the three-dimensional virtual space (hereinafter, also referred to as "reading distance"), the angle of the viewpoint position of the user with respect to the person in the three-dimensional virtual space (hereinafter, "reading It may also be “person angle”.

  For example, the viewpoint preference learning unit 231 learns history information of position designation of a viewpoint for each user, and when a certain user designates an object existing at a certain position, the body direction in the space of the person corresponding to the object , And predict the position of the viewpoint that the user will most preferably designate. In addition, the viewpoint preference learning unit 231 transmits the information on the predicted viewpoint position (for example, the viewing condition in which the number of designations of the viewpoint position is maximum) to the camera line recommendation unit 195 of the terminal device 100 via the communication unit 210. Send to

  Hereinafter, learning processing and prediction processing of viewpoint designation data performed by the viewpoint preference learning unit 231 will be described.

  First, FIG. 27 shows an example of data used for learning processing of position designation data of a viewpoint according to the present embodiment. Data in FIG. 27 is, for example, data in a table format stored in storage unit 220, and includes items (fields) as user identification information, camera identification information, viewpoint position, object identification information, object position, viewing distance, person It has a body direction angle, a viewing person angle, and the like.

  The user identification information is identification information of a person of a user operating a certain terminal device 100A, and is, for example, "Taro Yamada" in lines 1 to 3 in FIG.

  The camera identification information identifies the plurality of cameras 11 described above, and three cameras of A, B, and C are stored in the example of FIG.

  The viewpoint position is a three-dimensional position of a virtual viewpoint in a real space or a corresponding three-dimensional virtual space, and is represented by, for example, three-dimensional coordinate information of X, Y, Z. The method of calculating the viewpoint position may be the same as the method of calculating the viewpoint position on the camera line described above. Since the camera line has positional information in a three-dimensional space, a virtual viewpoint position which is one point on the camera line can also be represented by, for example, three-dimensional coordinate information of X, Y, Z.

  The object identification information may be the same information as the object ID described above, and is “Jiro Satou” or “Saburo Tanaka” in the example of FIG. The object position is, for example, a three-dimensional center position of an object corresponding to the object ID, and is represented by, for example, three-dimensional coordinate information of X, Y, Z.

  The viewing distance is the distance between the viewpoint position and the object position, and is described in millimeter units in the example of FIG. The human body orientation angle is an arbitrary one reference direction uniquely determined in advance by the administrator of the system or the like in the global coordinate system in the real world or the corresponding three-dimensional virtual space, and in the global coordinate system. And the body direction that each object is pointing at, is the angle made between the two vectors. The body direction is, for example, the direction in which the person is facing in the horizontal plane, and the person body direction angle formed with the reference direction can be determined for each object. For example, the object body direction angle of the object “Jiro Satou” in the first line of FIG. 27 is 90 °, and the object “Saburo Tanaka” in the third line is 270 °. It can be seen that there is a relationship of body orientation of 180 degrees upside down. In addition, although the vector of the said body direction and the said reference direction illustrated here by the angle of 0-360 degrees in case it exists on the horizontal surface which is a two-dimensional plane for description, the vector of the said body direction and the said reference direction A may be present in a three-dimensional space instead of a two-dimensional plane, and even in such a case, the human body direction angle can be similarly determined between two vectors.

The viewing person angle is an angle formed by the vector of the body direction and the vector of the camera line of the camera 11 corresponding to the camera identification information. That is, this is information indicating from which direction the person corresponding to the object is viewed. If the angle is limited to 0 to 360 ° on the horizontal plane for the sake of explanation, for example, the first line in FIG. The person corresponding to the object “Jiro Satou” viewed the photographed image of the body by the user through the captured image 81 from the viewpoint at an angle of 45 ° clockwise (or counterclockwise) from the front of the body with the camera A become. Also, for example, the object “Jiro Satou” in the second line of FIG.
The person corresponding to B is viewed by the user from behind the body (180 °) with the camera B.

  In the learning processing of position specification data of a viewpoint according to the present embodiment, the viewpoint preference learning unit 231 calculates a viewing condition in which the number of designations of the viewpoint position is maximum. That is, the viewpoint preference learning unit 231 may calculate the viewing distance in at least the three-dimensional virtual space as the viewing condition based on the three-dimensional selection position and the three-dimensional installation position, or the three-dimensional selection position and the three-dimensional installation. Based on the position and the information on the body direction of the person, the viewing person angle of the viewpoint position of the user with respect to the person at least in the three-dimensional virtual space may be calculated as the viewing condition.

  The viewing person angle may be calculated in any way. For example, the viewpoint preference learning unit 231 may acquire the body direction of the person based on the information of the installation direction of the desk in the real space corresponding to the third virtual space linked to the person (in the desk installation direction Information may be the person's body direction). Alternatively, the viewpoint preference learning unit 231 may acquire the body direction of the person based on the face detection process of the person or the three-dimensional human body model creation process on the captured image by the imaging device. Alternatively, the viewpoint preference learning unit 231 may detect the body of the person based on sensor data (for example, an acceleration detected by an acceleration sensor, an angular velocity detected by a gyro sensor, etc.) detected by a sensor device provided on the person's body. You may get the direction.

  For example, statistical data processing is performed using the user identification information and the information on the viewing person angle from the data table of FIG. That is, for each piece of user identification information, the distribution of the number of histories of the designation operation of the browsing person angle is obtained.

  The data of statistical processing is divided for each user identification information because behavior and psychological characteristics of viewpoint preference are considered to be different for each user. One user C prefers to view or zoom from the front of the body where the other person's face can be seen when looking at the state of the person of interest in the remote location through the captured image 81, while another user D is the other person's body The user prefers viewing from the side of the body in which the tilt or the operation state of the keyboard or mouse is easy to view, and another user E may prefer viewing from the back of the body where the contents of the display screen viewed by the other can be confirmed. Such preference of viewpoint selection (viewpoint preference) is considered to be different for each user, and the tendency also appears in the data of the viewing person angle to be statistically processed. In statistical processing of viewing person angle, as shown in the horizontal plane angle classification conceptual diagram around the person in FIG. 28, for example, a value between −22 ° and 22 ° is 0 degree item, between 23 ° and 67 °. The processing may be performed by classifying and aggregating data for each fixed angle width such as a 45-degree item.

  FIG. 29 shows an example of statistical data of the viewing person angle in the viewpoint position specification operation history of a certain user A. The horizontal axis of the graph indicates the viewing person angle of the 45 ° wide angle classification shown in FIG. 28, and the vertical axis indicates the number of viewpoint position designation operations. Referring to the graph of FIG. 29, the user A has a value (number of designated times) of 180 degrees of angle classification much more than that of other angles, and the viewing person angle is 180 degrees, that is, from behind the body of the person of interest It can be seen that the user has behavior and psychological characteristics that favor watching and zooming. In addition, there is a tendency to prefer to browse from the right side rather than the left side of the person of interest. Further, FIG. 30 illustrates an example of statistical data of the viewing person angle in the viewpoint position specification operation history of a certain user B. The horizontal and vertical axes of the graph are the same as in FIG. Referring to the graph of FIG. 30, the user B has 45 or 315 degrees of angle classification much more than other angle values, and the viewing person angle is 45 or 315 degrees, that is, the body of the person of interest It can be seen that the user has behavior and psychological characteristics that favor viewing while viewing and zooming diagonally from the front.

  When the user selects an object of a person of interest from the statistical data in FIGS. 29 and 30, if there are a plurality of camera lines corresponding to the object, the user A is on the body rear side of the object of the person of interest The probability of selecting a certain camera line is the highest, and the user B has the highest probability of selecting the camera line on the diagonally forward side of the body of the object of the person of interest. As described above, the viewpoint preference learning unit 231 can predict the camera line with the highest probability of being selected by each user by the statistical processing of the viewpoint position designation operation history data.

  In the above, statistical processing is performed with the viewing person angle as an explanatory variable and the number of viewpoint position designation times as a target variable. That is, the case has been described in which the viewpoint preference learning unit 231 calculates the relational expression between the browsing person angle and the number of viewpoint position specification times, and calculates the browsing person angle at which the viewpoint position specification frequency is maximum based on the calculated relational expression. . However, the type of explanatory variable is not limited to the viewing person angle. For example, statistical processing may be performed using the explanatory variable as the viewing distance described above and the number of viewpoint position designation times as the target variable. That is, the viewpoint preference learning unit 231 may calculate a relational expression between the browsing distance and the number of viewpoint position designation times, and calculate the browsing distance at which the viewpoint position designation number is maximum based on the calculated relational expression.

  Since the value of the viewing distance with the highest probability of being specified can be predicted for each user, the above-described zoom factor specification operation can be automated, and the user's viewpoint change input can be made simpler. That is, since the user simply designates the position to look at and then the determination of the position to look after is left to the system, the number of operation steps is reduced and the effort of the user's designated input action is reduced.

  Furthermore, the viewpoint preference learning unit 231 can perform probability prediction of the user's viewpoint selection with higher accuracy by combining and using a plurality of types of data related to the position specification of the viewpoint. For example, the viewpoint preference learning unit 231 calculates the viewing distance and the viewing person angle as the viewing condition based on the three-dimensional selection position, the three-dimensional installation position, and the information on the body direction of the person, and the viewing distance and the viewing person angle It is also possible to calculate a regression distance including the viewpoint position specification frequency as a variable, and calculate a viewing distance and a browsing person angle at which the viewpoint position specification frequency is maximum, based on the calculated regression expression. In the following, as an example, prediction processing of a designated position of a viewpoint by statistical processing using the above-mentioned browsing person angle and browsing distance as explanatory variables will be described.

  The viewpoint preference learning unit 231 can also calculate, using the position designation data of the viewpoint for each user in the storage unit 220, a regression equation for predicting the number of viewpoint position designation times from the data of the viewing person angle and the viewing distance. Since both the viewing person angle and the viewing distance, which are explanatory variables, and the number of viewpoint position designation times, which is the objective variable, are quantitative data, the viewpoint preference learning unit 231 performs the processing corresponding to known multiple regression analysis. Can be calculated. Furthermore, the viewpoint preference learning unit 231 arbitrarily changes the values of the viewing person angle and the viewing distance from the calculated regression equation, and predicts the viewing person angle and the viewing distance in which the calculated number of viewpoint position designation times takes a maximum value. Do. When multiple collinearity is found between the viewing person angle and the viewing distance, which are explanatory variables, the viewpoint preference learning unit 231 determines only univariate data of either the viewing person angle or the viewing distance as described above. An operation of switching to a method of performing statistical processing and prediction used may be performed.

  As described above, the viewpoint preference learning unit 231 calculates data of the viewing person angle, the viewing distance, and the number of viewpoint position designation times from the history data of position designation of the viewpoint for each user, and performs statistical processing to obtain a viewpoint as a target variable. It is possible to calculate the value of the viewing person angle or the viewing distance that maximizes the number of position designations, and predict the position of the viewpoint of the preference that is most easily specified by a certain user.

<< 4. Flow of processing >>
Subsequently, an example of the information processing according to the present embodiment will be described with reference to FIG. FIG. 22 is a flowchart showing an example of a schematic flow of information processing according to the present embodiment.

  In step S401, the position acquisition unit 183 determines whether the position in the captured image on the display screen of the captured image in the real space is designated by the user. More specifically, for example, the position acquisition unit 183 determines whether the position in the close-up captured image is designated by the user. If the above position is designated, the process proceeds to step S403. Otherwise, the process repeats step S401.

  In step S403, the position acquisition unit 183 acquires the position in the captured image specified by the user.

  In step S405, the object selection unit 185 acquires data of the three-dimensional virtual space corresponding to the real space from the storage unit 170.

  In step S407, an object arranged in the three-dimensional virtual space is selected based on the acquired position in the captured image.

  In step S409, the camera line control unit 191 acquires information of the camera 11 that includes the acquired position or an object corresponding to the position in the imaging range.

  In step S411, the camera line control unit 191 calculates a parameter that configures a camera line corresponding to the camera 11 from the information of the camera 11 that includes the acquired position or an object corresponding to the position in the imaging range. .

  In step S413, the camera line selection unit 193 determines whether a plurality of camera lines exist in the vicinity. If there are a plurality of camera lines in the vicinity, the process proceeds to step S415. If not, the process proceeds to step S417.

  In step S415, the camera line selection unit 193 compares a plurality of camera lines existing in the vicinity according to the above-described camera line evaluation condition, and selects one camera line having the highest evaluation as an optimal camera line.

  In step S417, the camera line control unit 191 causes the display screen 80 to display a camera line corresponding to the acquired position or an object corresponding to the position.

  In step S419, the camera line control unit 191 determines whether one of the camera lines or the position on the camera line is designated by the user. If the camera line or the position is specified, the process proceeds to step S423. Otherwise, the process repeats step S401.

  In step S421, the camera line control unit 191 acquires a captured image of the camera corresponding to the designated camera line, and switches the display image to the captured image of the current camera at the designated zoom ratio, thereby displaying the display screen 80. Displayed on top, the process ends. Note that the corresponding camera line may be automatically recalculated and redrawn on the display screen after the camera switching. In that case, the process may be resumed from S409.

  As described above, the information processing according to the present embodiment is performed. Furthermore, an example of the activation process performed before the start of the information processing will be described with reference to FIG.

  FIG. 23 is a flowchart showing an example of a schematic flow of activation processing according to the present embodiment.

  In step S501, a process of activating the softphone 851 is performed. Thus, the softphone 851 is activated.

  In step S503, registration processing for the softphone 851 is performed. For example, as one of the registration processing of the softphone 851, registration (for example, SIP REGISTRATION) in the PBX 40 is performed.

  In step S505, the process of activating the ultra-realistic client is performed. For example, the camera 11, the microphone 13, the sensor 15, the media distribution server 17, the information management server 200, and the like used in the ultra-realism client are specified.

  In step S507, the crawling mode processing of the ultra-realism client is executed. Then, the series of activation processing ends.

  FIG. 31 is a flowchart showing an example of a schematic flow of learning and recommendation processing according to the present embodiment.

  In step S601, the position acquisition unit 183 of the terminal device 100 determines whether the position in the captured image is designated by the user on the display screen of the captured image in the real space. If the above position is designated, the process proceeds to step S603. Otherwise, the process repeats step S601.

  In step S603, the control unit 180 of the terminal device 100 calculates a camera line corresponding to the acquired position by calculation, and displays the camera line on the display screen 80.

  In step S605, the viewpoint preference learning unit 231 of the information management server 200 stores the position specification data of the above-mentioned viewpoint linked to the identification information of the user who has specified the position, which is stored in the storage unit 170. It is determined whether there is an amount that can perform statistical prediction processing with sufficient prediction accuracy. If there is a sufficient amount of position designation data of the viewpoint, the process proceeds to step S 607. If not, the process proceeds to step S609. In the above-mentioned example of the multiple regression analysis, for example, the number of data (number of samples) needs to be larger by two or more than the number of explanatory variables. As an example, if the explanatory variable is two types of viewing person angle and viewing distance, it can be said that the number of data is at least 4 or more and insufficient. In addition, for example, the threshold of a sufficient number of data numbers may be set to any value such as 20.

  In step S 607, the viewpoint preference learning unit 231 of the information management server 200 performs prediction processing of the preference prediction camera line described above or the viewpoint on the camera line using position designation data of the viewpoint, and the camera line recommendation of the terminal device 100 The information is sent to the part 195. The camera line recommendation unit 195 that has received the information performs processing to change the display mode on the display screen 80 of the predicted preference camera line as described above.

  In step S609, the camera line control unit 191 of the terminal device 100 determines whether one of the camera lines (including the preference predicted camera line) or the position on the camera line is designated by the user. If the camera line or position is specified, the process proceeds to step S611. Otherwise, the process repeats step S601.

  In step S611, the viewpoint preference learning unit 231 of the information management server 200 receives the information on the camera line or the position designated by the user in step S609 from the terminal device 100, and stores the information in the storage unit 170. In the storage unit 170, history information of position designation of a previous viewpoint for each user is recorded, and the present information may be added to the end. Further, the data format of the storage may be in accordance with the format of the position specification data of the viewpoint described in FIG. 27 described above.

  This completes the series of learning and recommendation processes.

  Although the preferred embodiments of the present invention have been described above with reference to the accompanying drawings, it goes without saying that the present invention is not limited to such examples. It will be apparent to those skilled in the art that various changes or modifications can be conceived within the scope of the appended claims, and of course these also fall within the technical scope of the present invention. It is understood.

  For example, although the example in which the three-dimensional virtual space of the center office is prepared as the three-dimensional virtual space corresponding to the real space has been described, the present invention is not limited thereto. For example, a plurality of three-dimensional spaces may be prepared. As an example, a three-dimensional virtual space corresponding to real space may be prepared for each of a plurality of offices. For example, a three-dimensional virtual space may also be provided for satellite offices, home offices, other center offices, etc. In this case, the three-dimensional virtual space of each office may be a three-dimensional virtual space having a size corresponding to the size of each office. In addition, the object selection unit of the terminal device may acquire data of a desired three-dimensional virtual space among a plurality of three-dimensional virtual spaces. Also, a three-dimensional virtual space other than the office may be prepared.

  Further, although the example has been described in which the object corresponding to the person is selected only when the person is sitting in the seat, the present invention is not limited thereto. For example, the object may be selected even when the person is not seated in the seat. As an example, when a person is sitting in a seat, the communication ID of the communication device installed in the seat is acquired, and when the person is not sitting in the seat, the ID for communication of the portable terminal of the person May be acquired.

  Although the example in which the communication ID is a telephone number has been described, the present invention is not limited to this. The communication ID may be an ID other than a telephone number. As one example, the communication ID may be a softphone ID other than the telephone number. As another example, the communication ID may be an ID for communication other than telephone. For example, the communication ID may be a mail address or an ID for a short message. In this case, a mail may be sent or a short message may be sent using the communication ID.

  Further, although an example has been described in which the communication ID corresponding to the object is acquired when the object is selected, the present invention is not limited to this. For example, when an object is selected, any identification information corresponding to the object may be acquired. As one example, when an object is selected, any identification information of a person corresponding to the object may be obtained. Then, for example, the communication ID may be acquired from the identification information.

  Further, although the example in which the object (object selected by the object selection unit) disposed in the three-dimensional virtual space corresponding to the real space corresponds to a person and is a cylindrical object has been described, the present invention is not limited thereto . For example, the object may not be a cylindrical object but an object of another shape. Also, for example, an object may correspond to something other than a person. As an example, an object may correspond to a region of real space. Specifically, for example, the object may be arranged at a three-dimensional virtual position corresponding to the seat and corresponding to the position of the seat. Then, the communication ID of the communication device installed in the seat corresponds to the object, and when the object is selected, the communication ID may be acquired. Also, the object may correspond to an area wider than the seat, and may exist over a three-dimensional virtual area corresponding to the area of the area. Then, the communication ID of the communication device installed in the area corresponds to the object, and when the object is selected, the communication ID may be acquired.

  In addition, although an example in which the position of the captured image on the display screen is specified by a touch by the user has been described, the present invention is not limited thereto. For example, the position of the captured image may be designated by the user using an input unit other than the touch panel. For example, the position of the captured image may be specified by clicking with a mouse, or may be specified using another input means such as a button or a keyboard.

  Also, an example has been described in which the captured image generated by the camera, the audio data generated by the microphone, and the determination result by the sensor are directly provided to the terminal device by the camera, the microphone and the sensor, respectively. Is not limited to this. For example, these data may be provided by another device. As an example, any server (for example, media distribution server) may obtain these data and provide these data to the terminal device.

  Moreover, although the example in which the functions such as the position acquisition unit, the object selection unit, and the ID acquisition unit are provided by the terminal device has been described, the present invention is not limited thereto. For example, these functions may be provided by devices other than the terminal device. As an example, these functions may be provided by any server.

  Moreover, although the example in which the display screen of the captured image of real space was displayed by the terminal device was demonstrated, this invention is not limited to this. For example, the display screen may be displayed by another device. As one example, the display screen may be displayed by a display installed at a satellite office. Then, the user may designate the position of the captured image included in the display image on the display.

  In addition, the processing steps in the information processing of the present specification may not necessarily be performed in chronological order according to the order described in the flowchart. For example, the processing steps in the information processing may be performed in an order different from the order described as the flowchart, or may be performed in parallel.

  In addition, it is possible to create a computer program for causing hardware such as a CPU, a ROM, and a RAM built in an information processing apparatus (for example, a terminal apparatus) to exhibit the same function as each configuration of the information processing apparatus. A storage medium storing the computer program is also provided.

<< 5. Modified example >>
<5-1. Variation 1: Notification Function of Camera Line Sorting Processing>
As a modification 1, the camera line sorting unit 193 sorts an optimal one camera line from a plurality of camera line candidates, and displays only the optimal one camera line. It may have a function of displaying the total number of the plurality of camera line candidates around. FIG. 24 shows an example of a display in which the function of the first modification is applied to FIG. For example, when it is determined that three camera lines are present in the vicinity and the selection processing to an optimal one camera line is performed, as shown in FIG. In addition to the display on the display screen 80, the number "3" may be displayed in the vicinity of the one optimal camera line on the display screen 80.

  With this function, the user can intuitively know whether the camera line displayed on the display screen 80 is a camera line subjected to sorting processing (conversely, if there is no notification function, the user can Can not know whether each camera line has been sorted or not). Furthermore, when the display portion of the total number of the plurality of camera line candidates is touched, the selection processing may be canceled and mode switching may be performed such that all the plurality of camera line candidates are displayed.

5-2. Modified example 2: Acquisition management function of performance information under camera line evaluation condition>
As a second modification, when the camera line evaluation condition described above is the “performance” of the camera 11, the camera line selection unit 193 has a function of evaluating the performance and managing the performance information obtained as a result of the evaluation. It may be done. For example, the camera line sorting unit 193 causes all the cameras 11 included in the system according to the present embodiment to perform short-time trial shooting, obtains the image, and automatically analyzes the data of each obtained image. The performance information of each camera 11 may be estimated. Alternatively, the camera line selection unit 193 may acquire information on performance from data of Exif (Exchangeable image file format) embedded in a captured image of each camera 11 without performing image analysis processing.

  In addition, the camera line sorting unit 193 has a crawler program function that automatically crawls Web pages and collects information, and has a function of collecting information on the performance of each camera 11 such as camera specifications from the Web. It may be Furthermore, the data of the performance information acquired as described above may be stored and managed in cooperation with the information management server 200. With this function, when the camera line evaluation condition is the performance of the camera 11, the workload for the user to manually input and register the performance information of each camera 11 to the system according to the present embodiment is reduced.

5-3. Modification 3: Display Function of Image Quality Related Information to Camera Line>
As a third modification, the camera line control unit 191 may have a function of changing the display of the corresponding camera line according to the quality of the image captured by each camera 11. For example, in FIG. 15, the shooting pixel number of the camera 11C corresponding to the camera line 87C, the camera line 87D, and the camera line 87E in the captured image 81 is 320 × 240 pixels, and the shooting pixel number of the camera 11D is 1920 × 1200 pixels, When the shooting pixel count of the camera 11E is 640 × 480 pixels, the comparison of the quality of the shot image is camera 11D> camera 11E> camera 11C.

  However, before the camera switching process is performed, the user can not predict such differences in image quality. Therefore, as shown in FIG. 25, by drawing the high and low of the quality of the captured image in accordance with the display form of the camera line, the user selects the camera line before the camera switching process, from the camera of the switching destination. It is possible to intuitively understand the quality of the obtained photographed image, and it is possible to select the camera which can obtain the photographed image of higher quality and perform the switching operation. In the example shown in FIG. 25, the height of the captured image quality is expressed by the thickness of the camera line, but the height of the captured image quality may be expressed by the color of the camera line or the display transparency, for example. .

5-4. Variation 4: Application of the Invention Technique to Video Interpolation Viewpoint of Free Viewpoint Video>
As a camera to be switched in the present embodiment, an imaging apparatus mainly having hardware is assumed. However, as a fourth modification, the present embodiment relates to any video interpolation generated in free viewpoint video technology. The virtual imaging viewpoint position corresponding to the viewpoint may be regarded as one of the cameras 11. Thus, the present embodiment can also be applied to a remote video transmission application system having free viewpoint video technology in which viewpoint switching can be performed not discretely but continuously.

  In the complete free viewpoint video technology, the camera line may also be generated arbitrarily, but even in that case, the above-described sorting process by the camera line sorting unit 193 of the present embodiment may be made effective. For example, the selection process is applied to each predetermined spatial range, and an optimum camera line is obtained for the spatial range, whereby any camera line is narrowed down to camera lines corresponding to high quality viewpoint candidates, and viewpoint selection is performed. There is a possibility of realizing a viewpoint selection input method with higher operability than in the case where the possibility of operation is infinite.

5-5. Variation 5: Position of Each Configuration>
In the above, the example of the functional block which each of the terminal device 100 and the information management server 200 has was demonstrated concretely. However, the above-mentioned example is only an example of the functional block which each of the terminal device 100 and the information management server 200 has. For example, in place of the terminal device 100, the information management server 200 may have the functional blocks (for example, the camera line recommendation unit 195 and the like) described above as having the terminal device 100, and in the above Instead of the information management server 200, the terminal device 100 may have a functional block (for example, the viewpoint preference learning unit 231 or the like) described as the management server 200 has.

11 Camera 13 Microphone 15 Sensor 17 Media Distribution Server 19, 23 LAN (Local Area Network)
21 Display 30 External Network 40 PBX (Private Branch eXchange)
60 Display screen (俯瞰 mode)
61 俯瞰 captured image 63, 73, 83 button image 70 display screen (proximity mode)
71 Close-up image 80 Display screen (conversation mode)
81 Other-Side Captured Image 85 Own-Side Captured Image 90 Three-Dimensional Virtual Space 91 Object 93 Virtual Surface 100 Terminal Device 180 Control Unit 181 Real Space Information Providing Unit 183 Position Acquisition Unit 185 Object Selection Unit 187 ID Acquisition Unit 189 Telephone Unit 191 Camera Line Control unit (line control unit)
193 Camera Line Sorting Unit (Line Sorting Unit)
195 camera line recommendation unit 200 information management server 210 communication unit 220 storage unit 230 control unit 231 viewpoint preference learning unit

Claims (18)

A three-dimensional line is designated according to the three-dimensional selection position in the three-dimensional virtual space corresponding to the selection position based on the user's specification and the three-dimensional installation position of the imaging device including the three-dimensional selection position in the imaging range. When an imaging device corresponding to a dimensional line is designated as the viewpoint position of the user, the user's action for the person corresponding to the object present at the selected position is performed based on the three-dimensional selected position and the three-dimensional installation position. And a learning unit that calculates a viewing condition and calculates a relationship between the viewing condition and the number of times of designation of the viewpoint position.
Information processing device. The learning unit calculates the viewing condition in which the number of designations of the viewpoint position is maximum.
An information processing apparatus according to claim 1. The learning unit calculates a distance between the user and the person in at least a three-dimensional virtual space as the viewing condition based on the three-dimensional selection position and the three-dimensional installation position.
An information processing apparatus according to claim 1. The learning unit calculates a relational expression between the distance and the number of designations of the viewpoint position, and calculates the distance at which the designation number of the viewpoint position is maximum, based on the relational expression.
The information processing apparatus according to claim 3. The learning unit uses an angle of a viewpoint position of the user with respect to the person in at least a three-dimensional virtual space as the viewing condition based on the three-dimensional selection position, the three-dimensional installation position, and information of the body direction of the person. calculate,
An information processing apparatus according to claim 1. The learning unit acquires the body direction of the person based on the information of the installation direction of the desk in the real space corresponding to the tertiary virtual space linked to the person.
The information processing apparatus according to claim 5. The learning unit acquires a body direction of the person based on a face detection process of the person or a three-dimensional human body model creation process on the image captured by the imaging device.
The information processing apparatus according to claim 5. The learning unit acquires the body direction of the person based on sensor data detected by a sensor device provided on the body of the person.
The information processing apparatus according to claim 5. The learning unit calculates a relational expression between the angle and the number of designations of the viewpoint position, and calculates the angle at which the designation number of the viewpoint position is maximum based on the relational expression.
The information processing apparatus according to claim 5. The learning unit determines a distance between the user and the person in a three-dimensional virtual space and a viewpoint of the user with respect to the person based on the three-dimensional selection position, the three-dimensional installation position, and information of the body direction of the person. An angle of position is calculated as the viewing condition, and a regression equation including the distance and the angle and the number of times of designation of the viewpoint position as variables is calculated, and the number of times of designation of the viewpoint position is determined to be maximum based on the regression equation. Calculate the distance and the angle
An information processing apparatus according to claim 1. A three-dimensional line is designated according to the three-dimensional selection position in the three-dimensional virtual space corresponding to the selection position based on the user's specification and the three-dimensional installation position of the imaging device including the three-dimensional selection position in the imaging range. When an imaging device corresponding to a dimensional line is designated as the viewpoint position of the user, the user's action for the person corresponding to the object present at the selected position is performed based on the three-dimensional selected position and the three-dimensional installation position. Calculating a viewing condition, and calculating a relationship between the viewing condition and the designated number of times of the viewpoint position;
Information processing method. Computer,
A three-dimensional line is designated according to the three-dimensional selection position in the three-dimensional virtual space corresponding to the selection position based on the user's specification and the three-dimensional installation position of the imaging device including the three-dimensional selection position in the imaging range. When an imaging device corresponding to a dimensional line is designated as the viewpoint position of the user, the user's action for the person corresponding to the object present at the selected position is performed based on the three-dimensional selected position and the three-dimensional installation position. And a learning unit that calculates a viewing condition and calculates a relationship between the viewing condition and the number of times of designation of the viewpoint position.
A program for functioning as an information processing apparatus. A three-dimensional line is designated according to the three-dimensional selection position in the three-dimensional virtual space corresponding to the selection position based on the user's specification and the three-dimensional installation position of the imaging device including the three-dimensional selection position in the imaging range. An imaging device corresponding to a dimensional line is designated as a viewpoint position of the user, and based on the three-dimensional selection position and the three-dimensional installation position, viewing conditions of the user for a person corresponding to an object present at the selection position. And a display control unit that controls display according to the relationship when the relationship between the viewing condition and the number of times of designation of the viewpoint position is calculated.
Display control device. The display control unit selects a three-dimensional line according to the viewing condition in which the number of times of designation of the viewpoint position is the largest, and corresponds to a two-dimensional line corresponding to the selected three-dimensional line and another three-dimensional line The display mode of at least one of the two-dimensional line corresponding to the selected three-dimensional line and the two-dimensional line corresponding to the other three-dimensional line is changed so that the two-dimensional line and the two-dimensional line are distinguishably viewed.
The display control device according to claim 13. The display control unit calculates a viewpoint position in a three-dimensional line according to the viewing condition in which the number of times of designation of the viewpoint position is maximum, and performs control so that a captured image corresponding to the calculated viewpoint position is displayed. ,
The display control device according to claim 13. A three-dimensional line is designated according to the three-dimensional selection position in the three-dimensional virtual space corresponding to the selection position based on the user's specification and the three-dimensional installation position of the imaging device including the three-dimensional selection position in the imaging range. An imaging device corresponding to a dimensional line is designated as a viewpoint position of the user, and based on the three-dimensional selection position and the three-dimensional installation position, viewing conditions of the user for a person corresponding to an object present at the selection position. When the relationship between the viewing condition and the number of times of designation of the viewpoint position is calculated, the display according to the relationship is controlled.
Display control method including. Computer,
A three-dimensional line is designated according to the three-dimensional selection position in the three-dimensional virtual space corresponding to the selection position based on the user's specification and the three-dimensional installation position of the imaging device including the three-dimensional selection position in the imaging range. An imaging device corresponding to a dimensional line is designated as a viewpoint position of the user, and based on the three-dimensional selection position and the three-dimensional installation position, viewing conditions of the user for a person corresponding to an object present at the selection position. And a display control unit that controls display according to the relationship when the relationship between the viewing condition and the number of times of designation of the viewpoint position is calculated.
A program for functioning as a display control device. A three-dimensional line is designated according to the three-dimensional selection position in the three-dimensional virtual space corresponding to the selection position based on the user's specification and the three-dimensional installation position of the imaging device including the three-dimensional selection position in the imaging range. When an imaging device corresponding to a dimensional line is designated as the viewpoint position of the user, the user's action for the person corresponding to the object present at the selected position is performed based on the three-dimensional selected position and the three-dimensional installation position. A learning unit that calculates a viewing condition and calculates a relationship between the viewing condition and the number of times of designation of the viewpoint position;
A display control unit that controls display according to the relationship;
An information processing system having

Images