JP2002034009A - Bidirectional recognition system and method and recording medium - Google Patents

Abstract

(57) [Summary] [Problem] To make it easier to check the user status. SOLUTION: An image of a user photographed by a video camera is taken into a computer main body (S1) and compression-encoded (S2). If the office view display window is minimized or exists behind another application window, it is determined that the user is not looking at the office view (S3). The determined user state is transferred to the host server device via the network together with the compressed image data (S4). The user terminal receives an image of another user from the server device 12 (S5), decompresses the image (S6), and displays the image on the display device 44 (S7). Further, the usage status of another user is displayed on the office view so that the status of the user can be understood from the received usage status information of the user (S8). If the user does not indicate termination,
The process after S1 is repeated (S9).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a two-way cognitive system, method, and storage medium in which a user recognizes each other mainly by images in a virtual system formed on a computer network.

[0002]

2. Description of the Related Art Hitherto, each organization such as a company, an organization, and a public institution employs an employment worker (hereinafter referred to as a worker) who engages in office work, sales work, and technical work that does not require large facilities. The office space is owned or leased by a lease agreement, and the workers usually arrive and gather in the office space by the commencement of employment using public transportation or private cars, and the determined working hours It was common to work collectively within. And
By equipping office space with telephones, copiers, facsimile machines, computers, computer networks, etc., efficiency of various tasks has been improved.

[0003] This collective work practice is a relatively new phenomenon in the history of mankind that has been adopted for efficient operation of factories since the Industrial Revolution. In recent years, commuting hell and air pollution due to the increase of private cars,
While the negative effects of collective work have become more pronounced, collective work is not necessarily an indispensable form of work for organizations and their workers as a result of the availability of communication infrastructure such as the Internet and various communication technologies. Is coming. Rather, the idea of distributed work, in which workers belonging to the same organization work at their home or a preferred place near the home to carry out the purpose work of the organization as a whole, has been attracting attention.

In general, in order to realize distributed work in an organization such as a company, one room of each worker's home is used as a work space (this is called a home office) and a plurality of home offices distributed to remote locations are used. With a communication line,
A method using a home communication terminal device such as a telephone and a facsimile, and a communication application system is adopted. In this way, necessary communication means between each worker is secured. The communication line includes, for example, a subscribed telephone network, an ISDN network, and a line dedicated to the Internet, and the communication application system includes an electronic mail system, WWW (Wor).
ld WideWeb) system and a video conference system.

Workers use satellite offices and telecottages prepared by their organizations and local governments instead of home offices. In the case of sales and maintenance work, workers use private cars and vehicles used for customer visits. Use public transportation seats as temporary work locations (mobile offices).

[0006]

Conventionally, various systems have been proposed for the purpose of smoothly performing such distributed work. FIG. 12 shows White P in the United States.
ine Software InC. 1 shows a client screen of a multipoint video conference system provided by the company. This system mainly realizes a real-time conference using images between workers at remote locations on a personal computer.

[0007] By using such a system, the so-called two-way awareness system is realized by exchanging images between distributed workers, preventing the feeling of isolation and alienation, and energizing the workers. Communication can be promoted.

However, if existing tools developed for video conference communication are used as they are, it is impossible to exchange awareness information in real time, which cannot be determined only by video, so that psychological resistance (one-sided monitoring There is a problem of anxiety about whether or not That is, it can be determined from the video information that the user is located near the terminal device (in this case, a personal computer) and uses the terminal device,
It was not possible to determine whether another user is using the system or whether another software is being used. Unlike a remote monitoring system, in the case of a two-way awareness system, from the viewpoint of reciprocity, information as to whether a user is viewing other users' awareness information is important. However, the conventional system does not have the function of such a viewpoint.

The present invention has been made in view of such a problem, and realizes a smooth communication opportunity support by exchanging a user's usage status in real time, thereby enhancing a reciprocity. And a storage medium.

[0010]

A two-way cognitive system according to the present invention is a two-way cognitive system comprising a terminal device installed for each of a plurality of users and a communication line, wherein the terminal device comprises: A video acquisition unit for acquiring a video of a user, a transmission unit for transmitting the acquired video data to another user, a reception display unit for receiving and displaying a video of another user, and determining a user's system usage status It is characterized by comprising a use status judging means, a notifying means for notifying another user of the use status judgment result, and a use status display means for displaying the use status of another user.

[0011] A two-way recognition method according to the present invention is a method in which users mutually recognize each other by a terminal device and a communication line installed in each of a plurality of users. A transmitting step of transmitting the obtained video data to another user, a receiving and displaying step of receiving and displaying a video of the other user, a usage status determining step of determining a user's system usage status, and a usage status determination result And a usage status display step of displaying the usage status of the other user.

A storage medium according to the present invention stores a program for executing the above-described interactive recognition method.

[0013]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a schematic block diagram of a virtual office system according to an embodiment of the present invention. A host server device 12 is installed in a main office (a headquarters office of various organizations such as a company introducing a distributed office) 10. The host server device 12 is connected to a LAN (Local Area Network) in the main office 10.
k) Connect to 14. The user terminals 16 and 18 for workers working in the main office 10 are computers 2
0, user terminal software 22, telephone 24,
This is realized by combining the video camera 26 and other devices. The telephone set 24 can be connected to the host server device 12 via a PSTN line (general subscriber telephone line) or an ISDN line, and the computer 20 is connected to the host server device 12 via the LAN 14.

Reference numerals 30, 32, and 34 denote user terminal devices for users who work in the home office. A computer 36, user terminal device software 38, telephone 40, video camera 42, and virtual office display device 4 are provided, respectively.
4 and other devices are combined. When the display device 44 is not used, the display 76 is not used.
To display the virtual office. The telephone 40 can be connected to the host server device 12 via a PSTN line (general subscriber telephone line) or an ISDN line. The display device 44
The monitor 36 is connected to the computer 36 and displays its screen.

FIG. 2 shows an example of a hardware configuration of the host server device 12. BP (Basic Platform)
m) 12a is a server device constituted by a PC (personal computer). SPU (Signal p
processing Unit) 12b is a high-performance DS
P (Digital Signal processo)
r) and is built into the BP extension interface. The SPU 12b is used for signal processing of images and sounds on the server side. B
The P12a is connected to the LAN 14 in the main office 10 through the LAN board. Such a configuration itself is well-known, and other well-known configurations can be adopted.

FIG. 3 is a schematic block diagram showing the software configuration of the host server device 12. The software installed in the host server device 12 includes a software program developed using the C ++ language or the like and an existing software program, and includes an OS (operator).
Windows NT as an Attaching System)
(Registered trademark of Microsoft Corporation in the United States).
In FIG. 3, reference numeral 50 denotes a server manager, and 52 denotes an SPU.
An access library, 54 and 56 are drivers, 58 is a mail transmission unit, and 60 is a DLL (Dynamic Link).
(Library), 62 is a dynamic Web server, 64 is a database connector, 66 is Windows NT
(Registered trademark of Microsoft Corporation, USA), 68 is a database.

FIG. 4 shows an example of a hardware configuration of the user terminal device 30. As is well known, the computer 36 includes a main body 70, a mouse 72, a keyboard 74, and a display 7
6, a microphone 78, a speaker 80, and a modem 82 (or terminal adapter). Virtual office display device 4
4 comprises a video monitor device similar to the display,
Used to display a virtual office screen by connecting to the main body 70. The video camera 42 is also connected to the main body 70.
A mouse 72, a keyboard 74, a display 76, a microphone 78, a speaker 80, and a modem 82 are connected to corresponding input / output terminals of the main body 70.

FIG. 5 shows an example of the installation status of each device shown in FIG. 4 in a home office. Here, the display device 44 functions as a dual monitor system of the computer 36 together with the display 76 of the computer 36 used by the user for work.

FIG. 6 shows software 2 for a user terminal device.
2 shows a schematic configuration. The software for the user terminal device includes a software program developed using the C ++ language and the like and an existing software program, and employs Windows 98 (registered trademark of Microsoft Corporation, USA) as the OS. 22a is Windows
ws / Dialog part, 22b is a program component part, 22c is a signal line, 22d is an HTML part, 22e
Is an upper browser (component) section.

Next, the operation of this embodiment will be described. To simplify the explanation, it is assumed that each user in the home office has already completed the connection to the host server device 12 using their own user terminal devices 30, 32, and has started working.

FIG. 7 shows a virtual office screen (hereinafter referred to as an office view) including awareness information displayed on the display device 44 or 76. 90 is an image showing the work status of another user. Reference numeral 92 denotes a work status data display unit for displaying the work status data of the user as text information, where the destination and schedule of the user are displayed. Reference numeral 94 denotes the user's private office space. The combination of these three is a virtual private room office for this user. In FIG. 7, nine private offices are displayed on the same screen, but the number of private offices may be larger or smaller. The space for displaying these private offices is changed to a private office area 96.
Call. In the private room office area 96, awareness information of users including the user is displayed. As the awareness information, image data 90 of the user, text data 92 indicating the situation of the user, and the like are displayed. The space between the private offices is a virtual corridor. The user work situation image 90 displayed as a part of the private room office is an image captured by the camera 42 provided in the user terminal devices 30 to 34 used by the user.

A common area 98 is on the right side of the private office area 96. In the common area 98, icons 100 to 108 respectively indicating the A meeting room, the B meeting room, the research room, the refresh room, and the video news room are displayed.

Normally, when the user is at work, an office view as shown in FIG. 7 is displayed on the display device 44 or 76 of the virtual office, and information indicating the presence at an appropriate position is displayed.

The operation of this embodiment will be described. FIG. 8 shows an operation flowchart of a part relating to acquisition and display of image data and video control of the user terminal device.

The image photographed by the video camera 42 is taken into the main body 70 as image data in the QCIF format (176 × 144 pixels) by the video input device of the computer main body 70 (S1), and JPEG or H.264. 26
3 is compressed and coded by an image coding method (S
2).

The use status of the user is determined from the state of the user's office view display window (S3). Specifically, if the office view display window is minimized or is behind another application window, the user is not looking at the office view (i.e., seeing other users' awareness information). Not determined). The window status is determined by the API of the operating system on which the application is running.
(Application Program Inte
rface) or the like. The determined user state is transferred to the host server device 12 via the network together with the compressed image data (S4).

The user terminal receives the image of another user from the server device 12 (S5), decompresses it (S6), and displays it on the display device 44 or 76 (S7). Further, the usage status of the user is displayed on the office view so that the status of the user can be understood from the received usage status information of the user (S8). If the user does not instruct termination, S1
The subsequent steps are repeated (S9).

FIG. 9 is a display example of the user usage status when another user is not looking at the office view, and FIG. 10 is a display example of the user usage status when another user is looking at the office view. Show. When looking at the office view, as shown in FIG. 10, an explicit frame 110 is displayed around the user's image. This makes it possible to determine whether or not another user is present and operating the terminal from the video data, and further, whether or not another user is looking at the office view from the state use status display of the explicit frame 110. Or (that is,
Whether or not they are looking at their own awareness information).

As described above, in this embodiment, it is possible to notify other users whether or not the user is looking at the interactive awareness system by a simple method of grasping the status of the application display window of the user.
Recognition is improved by displaying the awareness display on the other user's terminal screen, and psychological resistance to feel that they are being monitored unilaterally without knowing (video Problems arising in a two-way awareness system that uses a Web browser).

FIG. 11 shows another operation flowchart of the present embodiment. It is assumed that the user is constantly displaying awareness information on the display device 44 in the environment shown in FIG. Also in this case, the user's equipment installation environment (camera 4
2 and the installation position of the display device 44) cannot know whether or not the user is looking at the awareness display only by the posture of the user obtained from the video.

The image photographed by the video camera 42 is taken into the main body 70 as image data in the QCIF format (176 × 144 pixels) by the video input device of the computer main body 70 (S11). The orientation of the user's face is recognized from the captured image data (S12). For example, the method proposed by Hagiwara et al. (Yokoo, Hagiwara, "Extraction of Multiple Face Regions from Natural Images Using Genetic Algorithm", IEICE 117-C, 9, pp. 1245-1252, 199).
7)), a method of detecting a face area and judging the face direction from the cut out face area by a neural network or the like can be used. Or, Suenaga, "Reading People-Computer Vision for Human Interfaces", Journal of the Institute of Electronics, Information and Communication Engineers, pp. 800-804,19
A method introduced in 95/8 or the like may be used.

It is determined whether or not the user is looking at the office view from the obtained information on the face orientation (S1).
3). Specifically, in the user environment shown in FIG. 5, a face facing the front means that the user is working using a personal computer, and a face facing the side looks at the office view. Means a situation. The relationship between the face direction and the situation of the user is determined according to the installation situation of the camera 42 and the display device 44 of the user, and it is assumed that the user has registered in advance.

The obtained image data is stored in JPEG or H.264. 2
Compression coding using an image coding method such as S.63 (S1
4) The user status information and the compressed image data are transferred to the host server device 12 via the network (S1).
5).

The user terminal receives an image of another user from the server device 12 (S16), decompresses the image (S17),
It is displayed on the display device 44 (S18). Further, the user's situation can be understood from the received user usage state information,
Display the usage status of other users on the office view (S
19). If the user does not instruct termination, S11 and subsequent steps are repeated (S20).

The operation shown in FIG. 11 has an effect that, in addition to the effect of the operation shown in FIG. 8, the situation of the user can be determined even when the awareness information is always displayed on the dedicated display 44. Further, since the usage status is determined from the video data for mutual exchange, no special additional device is required.

In the operation shown in FIG. 8, the usage status of another user is schematically displayed as a frame of video data. However, the present invention is not limited to this. May be something. Instead of displaying the data schematically, the received video data may be processed and displayed. Specifically, when another user is not looking at the interactive awareness system, a spatial filter process is performed on the decoded video data of the user, and a process such as blurring a video to be actually displayed is performed. As a result, the image of the user in a situation where the interactive awareness system is not actively used is displayed so that the user can easily understand the general state,
A more natural expression becomes possible as an awareness system, and reciprocity increases. In addition, it matches the feeling that one does not want to be seen by others when he is not looking at others. In addition, a method of changing information on colors (display as black and white data) as well as changing spatial characteristics may be applied.

In the operation shown in FIG. 11, the situation of the user is determined from the direction of the face, but the present invention is not limited to this. For example, the situation of the user can be determined by detecting the line of sight.
In this case, video data having a resolution higher than the image size used for communication is acquired, and a line of sight is detected from the high resolution data. The gaze can be detected by various methods proposed in the related art. It is determined whether or not the user is looking at the display unit of the awareness system based on the obtained gaze direction. An image having a resolution (for example, QCIF size) on a communication line is generated from the obtained high-resolution video by resolution conversion and used as a replacement image. When using gaze detection, whether the user is viewing the awareness information even when the awareness information is displayed on a part of the display 76 or when the awareness information display device 44 is juxtaposed on the display 76 Can be determined.

Although the case where the operation of the user terminal is realized by software of a personal computer has been described, the present invention is not limited to this. For example, the display device may be configured as a self-contained dedicated terminal and connected to a network independently. In that case, the signal processing and communication processing of various users are all processed by the terminal.

In order to operate various devices so as to realize the functions of the above-described embodiments, a computer connected to the various devices or a computer in the system has a program code of software for realizing the functions of the above-described embodiments. And the computer (CPU or MPU) of the apparatus or system is operated by operating the various devices in accordance with the stored programs.

In this case, the program code itself of the software realizes the function of the above-described embodiment, and the program code itself and means for supplying the program code to the computer, for example,
A storage medium storing such a program code constitutes the present invention. As a storage medium for storing such a program code, for example, a floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, C
A D-ROM, a magnetic tape, a nonvolatile memory card, a ROM, and the like can be used.

When the computer executes the supplied program code, not only the functions of the above-described embodiment are realized, but also the OS (Operating System) or other operating system running on the computer. Even when the functions of the above-described embodiments are realized in cooperation with application software and the like, it goes without saying that such program codes are included in the embodiments of the invention according to the present application.

Further, the supplied program code is:
After being stored in the memory provided in the function expansion board of the computer or the function expansion unit connected to the computer, the CPU or the like provided in the function expansion board or the function expansion unit performs one of the actual processing based on the instruction of the program code. It is needless to say that a case where the functions of the above-described embodiments are realized by performing all or part of the processes and executing the processing is also included in the invention according to the present application.

[0044]

As can be easily understood from the above description, according to the present invention, in a two-way awareness system, a user's usage status is exchanged in real time to realize smoother support for communication opportunities. In addition, it is possible to provide a two-way awareness system that enhances reciprocity.

[Brief description of the drawings]

FIG. 1 is a schematic block diagram of an embodiment of the present invention.

FIG. 2 is a diagram showing a hardware configuration of a host server device 12.

FIG. 3 is a diagram showing a software configuration of the host server device 12.

FIG. 4 is a diagram showing a device configuration of user terminal devices 30 to 34.

FIG. 5 is a diagram illustrating an example of an arrangement of user terminal devices 30 to 34;

FIG. 6 is a configuration diagram of user terminal software.

7 is an example of a screen called an office view displayed on the virtual office display device 44. FIG.

FIG. 8 is an operation flowchart of the embodiment.

FIG. 9 is a user status display example.

FIG. 10 is another user status display example.

FIG. 11 is another operation flowchart of the embodiment.

FIG. 12 is an example of a client screen of a conventional multipoint video conference system.

[Explanation of symbols]

10: Main office 12: Host server device 12a: BP (Basic Platform) 12b: SPU (Signal processing)
Unit) 14: LAN 16, 18: User terminal device 20: Computer 22: Software for user terminal device 22a: Windows / Dialog unit 22b: Program component unit 22c: Signal line 22d: HTML unit 22e: Upper browser (component) unit 24: Telephone 26: Video camera 30, 32, 34: User terminal device 36: Computer 38: Software for user terminal device 40: Telephone 42: Video camera 42a, 42b: Video camera 44: Virtual office display device 50: Server manager 52: SPU access library 54, 56: driver 58: mail transmission unit 60: DLL (Dynamic Link Library)
ry) 62: Dynamic Web server 64: Database connector 66: Windows NT (registered trademark of Microsoft Corporation, USA) 68: Database 70: Main body 72: Mouse 74: Keyboard 76: Display 78: Microphone 80: Speaker 82: Modem (or Terminal adapter) 90: Image showing work status of another user 92: Work status data display section 94: Private room office space 96: Private room office area 98: Common area 100: Icon indicating meeting room A 102: Showing meeting room B Icon 104: Icon indicating a laboratory 106: Icon indicating a refresh room 108: Icon indicating a video news room 110: Frame indicating that the user is looking at the office view

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Yoshihisa Tadokoro, 3-30-2 Shimomaruko, Ota-ku, Tokyo Inside Canon Inc. (72) Takanori Miyazaki, 3-30-2 Shimomaruko, Ota-ku, Tokyo F term in Canon Inc. (reference) 5C022 AA01 AB62 AB65 AC01 AC13 AC69 AC75 CA00 5C064 AA02 AA06 AB03 AB04 AC04 AC14 AD02 AD08 AD09

Claims (19)

[Claims] 1. A two-way cognitive system including a terminal device installed in each of a plurality of users and a communication line, wherein the terminal device includes: a video obtaining unit configured to obtain a video of a user; Transmitting means for transmitting video data to another user; receiving and displaying means for receiving and displaying the video of another user; usage determining means for determining a user's system usage status; and transmitting the usage status determination result to another user. An interactive cognitive system comprising: a notifying unit for notifying a user and a usage status display unit for displaying a usage status of another user. 2. The interactive cognitive system according to claim 1, wherein the use status determination unit determines whether the user is watching video data of another user. 3. The interactive recognition system according to claim 1, wherein the use status determination unit uses a state of a video data display window displayed on the terminal device. 4. The interactive cognitive system according to claim 1, wherein the use status judging unit uses the video data obtained by the video obtaining unit. 5. The interactive recognition system according to claim 1, wherein said usage status determination means includes means for recognizing a direction of a user's face, and determines the usage status of the user based on the recognition result. 6. The interactive recognition system according to claim 1, wherein said use status determination means includes means for recognizing a user's line of sight, and determines the use status of the user based on the recognition result. 7. The interactive recognition system according to claim 1, wherein the usage status display means displays whether or not another user is viewing information including the video data of the user. 8. The interactive cognitive system according to claim 1, wherein the use status display means schematically displays the use status of another user. 9. The interactive recognition system according to claim 1, wherein said usage status display means processes the received video data of another user in accordance with a usage status determination result. 10. A method for mutually recognizing a user through a terminal device and a communication line installed in each of a plurality of users, the method comprising: acquiring a video of the user; A transmitting step of transmitting to another user, a receiving and displaying step of receiving and displaying an image of another user, a utilization state determining step of determining a user's system usage state, and a notification of notifying the usage state determination result to another user. An interactive recognition method, comprising: a step of displaying a usage state of another user; 11. The interactive recognition method according to claim 10, wherein the use status determining step determines whether a user is watching video data of another user. 12. The use status judging step uses a state of a video data display window.
2. The interactive recognition method according to 1. 13. The interactive recognition method according to claim 10, wherein the use status determination step uses video data obtained in the video acquisition step. 14. The interactive recognition method according to claim 10, wherein the use status determining step includes a step of recognizing a direction of the user's face, and determining the use status of the user from the recognition result. 15. The interactive recognition method according to claim 10, wherein the use status determination step includes a step of recognizing a user's line of sight, and the use status of the user is determined from the recognition result. 16. The interactive recognition method according to claim 10, wherein the use status display step displays whether or not another user is viewing information including the video data of the user. 17. The usage status display step, wherein the usage status of another user is schematically displayed.
Interactive recognition method described in 1. 18. The interactive recognition method according to claim 10, wherein in the use status display step, the received video data of another user is processed according to a use status determination result. 19. A storage medium storing a program for executing the interactive recognition method according to claim 10. Description: