JP2025114374A - Information processing system, output method, and program - Google Patents

Abstract

[Problem] To notify a user operating an avatar placed in one virtual space of information about another virtual space. [Solution] An information processing system transmits visibility information of avatars placed in virtual spaces to terminal devices via a network, in which a virtual conference server (1) has an avatar management unit that places multiple avatars of a first user (lecturer) in different virtual spaces, a virtual display control unit (26) that generates information about virtual spaces not displayed on the first user's terminal device, and a communication control unit (23) that transmits information about the virtual spaces to the first user's terminal device. [Selected Figure] Figure 5

Description

The present invention relates to an information processing system, an output method, and a program.

By communicating with a server via a PC (personal computer), users can participate in online meetings and take e-learning courses. It is also possible for participants to display avatars placed in a virtual space on a VR (virtual reality) terminal device, allowing them to hold meetings and other events in the virtual space.

Technology has been devised that allows participants wearing terminal devices to take lessons in a virtual space (see, for example, Patent Document 1). Patent Document 1 discloses a configuration in which one instructor controls multiple avatars to conduct multiple lessons in a virtual space.

However, conventional technology has the problem of not having a function to notify a user operating an avatar placed in one virtual space of information about the other virtual space. For example, if a user who is viewing a virtual space from the viewpoint of an avatar placed in one virtual space wants to be called into the other virtual space, there is no function to notify the user.

In consideration of the above-mentioned problems, the present invention provides a technology that notifies a user operating an avatar placed in one virtual space of information related to the other virtual space.

In consideration of the above-mentioned problems, the present invention provides an information processing system that transmits view information of avatars placed in a virtual space to a terminal device via a network, and includes an avatar management unit that places multiple avatars of a first user in different virtual spaces, a virtual display control unit that generates information about the virtual space that is not displayed on the terminal device of the first user, and a communication control unit that transmits information about the virtual space to the terminal device of the first user.

This technology allows a user operating an avatar placed in one virtual space to be notified of information about another virtual space.

FIG. 1 is a diagram illustrating a configuration example of a virtual space system. FIG. 2 is a diagram illustrating a hardware configuration of an example of a virtual conference server. FIG. 1 is a diagram illustrating a hardware configuration of an example of a notebook PC. FIG. 1 is a diagram illustrating a hardware configuration of an example of VR goggles. FIG. 2 is a hardware configuration diagram of an example of a VR operation controller. FIG. 1 is an example of a functional block diagram illustrating the functions of a virtual conference server, a notebook PC, VR goggles, and a VR operation controller in a virtual space system, divided into blocks. FIG. 10 is a sequence diagram illustrating an example of a process performed by a lecturer to start a conference in a virtual conference space. FIG. 10 is a sequence diagram illustrating an example of a process for a user to participate in a conference in a virtual conference space. FIG. 10 is a diagram showing an example of a menu list screen displayed in the virtual conference space. FIG. 10 is a diagram showing an example of notebook PC display data of a notebook PC displayed in a virtual conference space. FIG. 1 is a diagram illustrating avatars arranged in three groups held in a virtual conference space. 10 is a diagram illustrating an example of correspondence information between groups, participants, and instructor avatars. 10 is an example of a diagram showing the field of view of an instructor avatar Z1 in group 1. FIG. FIG. 10 is a diagram showing an example of the field of view screen of a lecturer avatar on which a notification message indicating that the discussion is stalled is displayed. FIG. 10 is a diagram illustrating the instructor pressing a notification message. FIG. 10 is a diagram showing an example of a field of view screen on which a switch now button and a switch later button are displayed. This is an example of a diagram showing the field of view screen of group 2 displayed by the VR goggles by pressing the Now switch button. This is an example of a diagram showing the field of view screen of group 1 displayed by the VR goggles later by pressing the switch button. 13 is an example of a diagram showing the field of view screen of avatar E (Aomori-san) of group 2 when the instructor later presses the switch button. 10 is an example of a sequence diagram illustrating a process in which an output control unit displays a notification message on the VR goggles of the instructor. 10 is an example of a sequence diagram illustrating a process when a lecturer responds to a notification message. 10A and 10B are diagrams showing examples of statements made by avatars in a group; FIG. 10 is a diagram showing an example of a view screen of an instructor avatar of Group 1 on which an inquiry notification message is displayed. FIG. 10 is a diagram showing an example of a field of view screen on which a plurality of notification messages are simultaneously displayed. 10 is an example of a sequence diagram illustrating a process in which an output control unit notifies an instructor avatar of an inquiry from a group that is not currently displaying a view screen on the VR goggles and is not outputting audio data. FIG. 10 is a diagram showing an example of an avatar switching button displayed on the instructor's VR goggles. FIG. 10 is a diagram showing an example of a view screen displaying a list of instructor avatars. 10 is an example of a sequence diagram illustrating a process of switching the view screen displayed by the instructor's VR goggles. FIG. 10 is a diagram showing an example of a field of view screen including a notification button with a mark added. 10 is an example of a sequence diagram illustrating a process in which the instructor causes the VR goggles to display a notification message again. FIG. 10 is a diagram showing an example of the field of view screen of the instructor avatar of Group 1 on which the inquiry notification message is displayed again.

Below, a virtual space system and the output method performed by the virtual space system will be described as an example of an embodiment of the present invention, with reference to the drawings.

<Terminology>
View information is information that visualizes the state of the virtual space as seen from the viewpoint of an avatar. In this embodiment, view information is described using the term view screen. The VR goggles monitor the user's viewpoint using a gyro sensor and an acceleration sensor, and a 3D model constructed in the virtual space in the direction of the user's viewpoint is displayed on the VR goggles' display. Alternatively, the user can control the user's viewpoint using a VR operation controller. A 3D model is data that represents a three-dimensional surface using polygons or the like with three-dimensional coordinates. 3D models in this embodiment include desks, electronic whiteboards, and other users' avatars.

A virtual space is a virtual space that recreates, for example, a 360° three-dimensional space on a computer. The term virtual space is a place where people interact and engage in social activities over a network, just as they would in the real world, and is likened to a physical space. In this embodiment, a conference room is primarily recreated in the virtual space, but any three-dimensional space can be recreated in the virtual space. Furthermore, in this embodiment, the virtual space can also be two-dimensional, as it is sufficient for planar display data to be displayed.

A pointer is a type of cursor that indicates the input position on a computer screen, and is, for example, an arrow-shaped icon. Pointers are usually arrow-shaped, but the shape of the cursor can change depending on the state of the computer or the position of the mouse pointer.

In this embodiment, the display device that allows the user to view the virtual space is referred to as VR goggles. VR goggles may be AR (Augmented Reality) goggles or MR (Mixed Reality) goggles.

An avatar is a participant's alter ego generated using computer graphics. By wearing VR goggles, participants can perceive each participant's avatar in the virtual meeting space as if they were actually in real space.

<System configuration example>
1 is a system configuration diagram of a virtual space system 300. In the virtual space system 300, a virtual conference server 1 is communicably connected to Wi-Fi routers 3, 7, and 8 via a network N. Notebook PCs 4, 11, and 14 can also be communicably connected to the Wi-Fi routers 3, 7, and 8. The network N is, for example, the Internet, but may also be an on-premise network such as an in-house network.

The virtual conference server 1 is an information processing system having one or more information processing devices. The virtual conference server 1 stores data for the virtual conference space 8 in memory and executes a virtual conference app. Hereinafter, the service provided by the virtual conference app executed by the virtual conference server 1 will be referred to as the virtual conference service. The virtual conference space 8 is, for example, a 360° three-dimensional space that simulates a conference room in a virtual space. VR goggles 5, 12, and 15 (an example of a terminal device) display any display data 9 in the virtual space, and conference participants can view any viewpoint of the virtual conference space 8 using the VR goggles 5, 12, and 15. Participants can view the screens of devices located in the conference room as if they were actually in the conference room.

The virtual conference space 8 has a screen acquired from the notebook PCs 4, 11, and 14 and synchronized with the screen of a notebook PC (hereafter referred to as a virtual notebook PC). The screen data of the notebook PCs 4, 11, and 14 is referred to as PC display data 9a and 9b. The virtual conference space 8 also manages virtual electronic whiteboard display data 9c and 9d as display data for a virtual electronic whiteboard. When there is no distinction between the PC display data 9a and 9b and the virtual electronic whiteboard display data 9c and 9d, the data displayed in the virtual conference space 8 is referred to as display data 9.

The virtual conference space 8 has rooms, and conference participants (lecturer Z, users A and B) can enter each room using avatars. In Figure 1, there are rooms A and B. The display data 9 in rooms A and B can be shared by participants who enter those rooms. PC display data 9a and 9b can be controlled so that only the participant's own data is visible.

In Figure 1, instructor Z is using a laptop PC 4, VR goggles 5, and a VR operation controller 6. The laptop PC 4 and VR goggles 5 are connected to a Wi-Fi router 3 via wireless LAN so that they can communicate with each other. The VR goggles 5 and VR operation controller 6 are connected via Bluetooth (registered trademark; hereafter omitted). The laptop PC 4 and VR goggles 5 are also connected via Bluetooth. A virtual conference app is running on the laptop PC 4.

User A uses a laptop PC 11, VR goggles 12, and a VR operation controller 13 in an environment where a Wi-Fi router 7 is installed. User B uses a laptop PC 14, VR goggles 15, and a VR operation controller 16 in an environment where a Wi-Fi router 8 is installed.

User A's and User B's laptop PCs 11, 14 and VR goggles 12, 15 are connected to Wi-Fi routers 7, 8 via wireless LAN, and the VR goggles 12, 15 and VR operation controllers 13, 16 are connected via Bluetooth. The laptop PCs 11, 14 and VR goggles 12, 15 are also connected via Bluetooth. A virtual conference app is running on User A's and User B's laptop PCs 11, 14.

<Hardware configuration example>
<<Virtual conference server and laptop PC>>
Fig. 2A is a diagram showing an example of the hardware configuration of the virtual conference server 1. As shown in Fig. 2A, the virtual conference server 1 is constructed by a computer 500 and includes a CPU 501, a ROM 502, a RAM 503, a hard disk (HD) 504, a hard disk drive (HDD) controller 505, a display 506, an external device connection I/F (Interface) 508, a network I/F 509, a bus line 510, a keyboard 511, an optical drive 514, and a media I/F 516.

Of these, the CPU 501 controls the overall operation of the virtual conference server 1. The ROM 502 stores programs used to drive the CPU 501, such as IPL. The RAM 503 is used as a work area for the CPU 501. The HDD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from the HDD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menu, window, text, or image. The external device connection I/F 508 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory or a mouse. The network I/F 509 is an interface for data communication using a network. The bus line 510 is an address bus, data bus, or the like, for electrically connecting the various components, such as the CPU 501, shown in FIG. 2A.

The keyboard 511 is a type of input means having multiple keys used to input characters, numbers, various instructions, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), etc. The media I/F 516 controls reading and writing (storing) of data from a storage medium 515, such as a flash memory.
<<Notebook PC>>
2B is a diagram showing an example of the hardware configuration of the notebook PCs 4, 11, and 14. As shown in Fig. 2B, the notebook PCs 4, 11, and 14 are constructed by a computer 600, and include a CPU 521, a ROM 522, a RAM 523, a HD (Hard Disk) 524, a HDD (Hard Disk Drive) controller 525, a display 526, an external device connection I/F (Interface) 528, a network I/F 529, a bus line 530, a keyboard 531, a pointing device 532, and a media I/F 536.

Of these, the CPU 521 controls the overall operation of the notebook PC 4, 11, or 14. The ROM 522 stores programs used to drive the CPU 521, such as IPL. The RAM 523 is used as a work area for the CPU 521. The HDD 524 stores various data, such as programs. The HDD controller 525 controls the reading and writing of various data from the HDD 524 under the control of the CPU 521. The display 526 displays various information, such as a cursor, menus, windows, text, or images. The external device connection I/F 528 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I/F 529 is an interface for data communication using a network. The bus line 530 is an address bus, data bus, etc., for electrically connecting the various components, such as the CPU 521, shown in FIG. 2B.

The keyboard 531 is a type of input means equipped with multiple keys used to input characters, numbers, various instructions, etc. The pointing device 532 is a type of input means used to select and execute various instructions, select processing targets, move the cursor, etc. The media I/F 536 controls the reading and writing (storage) of data from and to the recording media 535, such as flash memory.

<<VR goggles>>
3 is a diagram showing the hardware configuration of the VR goggles 5, 12, and 15. The VR goggles 5, 12, and 15 each include a CPU 80, a main memory 81, a ROM 82, a display controller 86, a wireless LAN controller 88, an audio codec 90, and a video codec 93, all of which are connected via a bus 94.

The CPU 80 executes and processes the OS and control processing programs read from the ROM 82 into the main memory 81. The main memory 81 includes DRAM and is used as a work area for the CPU 80, etc.

ROM 82 is pre-loaded with the OS, system programs that run when the power is turned on, and programs that control the VR goggles 5, 12, and 15.

A UART 83 is connected to the CPU 80. The UART 83 is an interface that exchanges serial data between the CPU 80 and the Bluetooth module 84, and is composed of a FIFO (First In, First Out) and a shift register, etc.

The Bluetooth module 84 consists of an RF section and a baseband section, and is connected to an antenna 85, performing wireless communication in accordance with the Bluetooth standard.

The display controller 86 performs D/A conversion of character data, graphic data, image data, etc., and controls the display of this data on the LCD 87.

The wireless LAN controller 88 executes a communication protocol compliant with the IEEE (Institute of Electrical and Electronics Engineers) 802.11ax standard, controlling communication with other devices by sending and receiving radio waves via the antenna 89.

Audio signals input from microphone 91 are converted into audio data by an A/D (Analog/Digital) conversion circuit, and this audio data is encoded by audio codec 90 using the AAC (Advanced Audio Coding) method. Furthermore, AAC-encoded data received from an external device is decoded by audio codec 90, converted to analog by a D/A conversion circuit, and the resulting audio is output from speaker 92. Video codec 93 decodes compressed video data (for example, a method compliant with ITU-T Recommendation H.264) received from an external device. Data is exchanged between the above components via bus 94.

<<VR operation controller>>
4 is a hardware configuration diagram of the VR operation controllers 6, 13, and 16. The VR operation controllers 6, 13, and 16 each have a UART 117, a main memory 111, a ROM 112, a 6-axis acceleration/angular velocity sensor 113, a menu display button 114, a pointer display button 115, and a confirm button 116, all of which are connected to a CPU 110.

The CPU 110 executes and processes the control processing program read from the ROM 112 into the main memory 111. The main memory 111 includes DRAM and is used as a work area for the CPU 110, etc.

ROM 112 is pre-loaded with a system program that runs when the power is turned on, and a program that transmits information via Bluetooth when the menu display button 114, pointer display button 115, and confirm button 116 are pressed.

The six-axis acceleration/angular velocity sensor 113 outputs measurement data on acceleration and angular velocity. The UART 117 is an interface that exchanges serial data between the CPU 110 and the Bluetooth module 118, and is composed of a FIFO, shift register, etc. The Bluetooth module 118 is composed of an RF section and a baseband section, and is connected to an antenna 119, which performs wireless communication in accordance with the Bluetooth standard.

<About the function>
Next, the functional configuration of the virtual space system 300 will be described with reference to Fig. 5. Fig. 5 is an example of a functional block diagram that explains the functions of the virtual conference server 1, the notebook PCs 4, 11, and 14, the VR goggles 5, 12, and 15, and the VR operation controllers 6, 13, and 16 in the virtual space system 300, divided into blocks.

<<Virtual conference server>>
The virtual conference server 1 includes a virtual conference control unit 20, a whiteboard generation and management unit 21, a screen synchronization control unit 22, a communication control unit 23, a conference data storage unit 24, a room management unit 41, an output control unit 43, a display data generation unit 25, a virtual display control unit 26, a pointer position calculation unit 27, a user authentication unit 28, a user information storage unit 29, a voice recognition unit 42, and an avatar management unit 44. Each of these units included in the virtual conference server 1 is a function or a means for performing a function that is realized when any of the components shown in FIG. 2A operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 to the RAM 503.

The virtual conference control unit 20 associates conference participants and devices connected to the conference (notebook PCs 4, 11, 14, VR goggles 5, 12, 15) with conference identification information (conference ID) and stores them in the conference data storage unit 24. The virtual conference control unit 20 runs a remote conference with each device for each conference ID. The virtual conference control unit 20 also stores virtual electronic whiteboard display data 9c, 9d, PC display data 9a, 9b, audio data received from devices connected to the virtual conference, and so on in the conference data storage unit 24.

The whiteboard generation and management unit 21 generates a virtual electronic whiteboard within the virtual conference space 8 and manages the virtual electronic whiteboard display data 9c and 9d.

When a participant uses a physical electronic whiteboard, the screen synchronization control unit 22 synchronizes the electronic whiteboard screen data with the virtual electronic whiteboard display data 9c, 9d so that they are the same. The screen synchronization control unit 22 operates as an application layer.

The communication control unit 23 is a means for transmitting information about the group to instructor Z's VR goggles 5. The communication control unit 23 is also a means for transmitting avatar visibility information generated by the virtual display control unit 26 to the instructor's VR goggles 5. Specifically, the communication control unit 23 transmits and receives data between the laptop PCs 4, 11, and 14 and the VR goggles 5, 12, and 15.

The conference data storage unit 24 stores virtual electronic whiteboard display data 9c, 9d, PC display data 9a, 9b, and audio data received from devices connected to the virtual conference. The conference data storage unit 24 also stores notification messages between groups.

The room management unit 41 manages conferences in the virtual conference space 8 in units called rooms. Multiple conferences can be held simultaneously in the virtual conference space 8. One conference is held in one room, and a virtual conference space 8 is created for each room.

The output control unit outputs information about the virtual space where field of view information is not displayed on the instructor's VR goggles 5 to the instructor's VR goggles 5. Specifically, the output control unit 43 analyzes user operations and causes the display data generation unit 25 to generate appropriate notification messages for the conference. For example, if no one speaks during a certain period of time in the conference, the output control unit 43 sends a notification message to instructor Z. Furthermore, if instructor Z is busy, the output control unit 43 sends a notification message to the group participants on behalf of instructor Z.

The display data generation unit 25 generates the menus displayed in the virtual conference space 8, the pointer 170 displayed by the VR operation controllers 6, 13, and 16, and the like.

The virtual display control unit 26 is a means for displaying a pointer 170 indicating virtual electronic whiteboard display data in the virtual conference space 8 based on operation information transmitted from the VR operation controllers 6, 13, and 16. Specifically, the virtual display control unit 26 displays various display data in the virtual conference space 8. The virtual display control unit 26 displays in the virtual conference space 8 the menu and pointer 170 generated by the display data generation unit 25, virtual electronic whiteboard display data 9c and 9d related to the virtual electronic whiteboard generated by the whiteboard generation and management unit 21, and PC display data 9a and 9b related to the laptop PCs 4, 11, and 14. The virtual display control unit 26 transmits the display data for the virtual conference space 8 to the VR goggles 5, 12, and 15 via the communication control unit 23, and the virtual conference space 8 is displayed on the VR goggles 5, 12, and 15.

The pointer position calculation unit 27 calculates the position (in the virtual conference space 8) of the pointer 170 indicated by the orientation and movement of the VR operation controllers 6, 13, and 16.

The user authentication unit 28 compares the username and password received via the communication control unit 23 with the username and password stored in the user information storage unit 29 to authenticate users participating in the virtual conference.

The user information storage unit 29 stores the usernames and passwords of users participating in the virtual conference.

The voice recognition unit 42 recognizes the utterances (voice) of users corresponding to each avatar in the virtual conference and records them together with the user's identification information. The voice data and recognition results are associated and stored in the conference data storage unit 24. Voice recognition may be performed by a dedicated server.

The avatar management unit 44 is a means for placing multiple instructor avatars Z1 to Z3 of a first user (an example of an instructor) in different virtual spaces. Specifically, the avatar management unit 44 generates avatars corresponding to users A and B, and an avatar corresponding to instructor Z. Multiple avatars are generated for instructor Z. The avatar management unit 44 manages the correspondence between groups and avatars.

<<Notebook PC>>
The notebook PCs 4, 11, and 14 each have a virtual conference connection unit 30, a remote desktop control unit 31, a wireless LAN communication control unit 32, a Bluetooth communication control unit 33, a display control unit 34, and a data storage unit 35. Each of these units in the notebook PCs 4, 11, and 14 is a function or a means for performing the function, which is realized when any of the components shown in Fig. 2B operates in response to an instruction from the CPU 501 in accordance with a program (virtual conference application) loaded from the HD 504 to the RAM 503.

The virtual conference connection unit 30 connects to the virtual conference service of the virtual conference server 1 and transmits the authentication data, which is a username and password. The virtual conference connection unit 30 also manages the URL and conference ID of the virtual conference space 8 received from the virtual conference service, and connects to the URL of the virtual conference space 8 to participate in the virtual conference. Furthermore, the virtual conference connection unit 30 transmits the URL and conference ID of the virtual conference space 8 to the VR goggles 5, 12, and 15 via the Bluetooth communication control unit 33.

The remote desktop control unit 31 transmits screen data from the notebook PCs 4, 11, and 14 to the virtual conference service for display in the virtual conference space 8.

The wireless LAN communication control unit 32 communicates with the virtual conference server 1 via Wi-Fi routers 3, 7, and 8.

The Bluetooth communication control unit 33 performs communication with the VR goggles 5, 12, and 15 in accordance with the Bluetooth standard. The display control unit 34 controls the display of screen data on the display (LCD).

The data storage unit 35 stores the address (URL) of the virtual conference service provided by the virtual conference server 1, the URL of the virtual conference space 8 received from the virtual conference service, the conference ID, the username and password of the user using the notebook PC 4, 11, 14, etc.

<<VR goggles>>
The VR goggles 5, 12, and 15 each have a display control unit 100, a wireless LAN communication control unit 101, a Bluetooth communication control unit 102, a virtual conference connection unit 103, a button information transfer unit 104, and a data storage unit 105. Each of these units in the VR goggles 5, 12, and 15 is a function or a means for performing a function that is realized when any of the components shown in Fig. 3 operates in response to an instruction from the CPU 80 in accordance with a program (virtual conference application) loaded from the ROM 82 to the main memory 81.

The display control unit 100 controls the display of display data sent from the virtual conference service on the LCD 87. The wireless LAN communication control unit 101 communicates with the virtual conference server 1 via Wi-Fi routers 3, 7, and 8.

The Bluetooth communication control unit 102 communicates with the VR operation controllers 6, 13, and 16 in accordance with the Bluetooth standard. The virtual conference connection unit 103 uses the URL and conference ID of the virtual conference service sent from the laptop PC 4 to perform control for connecting to the virtual conference service executed by the virtual conference server 1.

The button information transfer unit 104 receives button press information from the VR operation controllers 6, 13, and 16, and transmits information about the pressed button to the virtual conference server 1 via the wireless LAN communication control unit 101.

The data storage unit 105 stores the address (URL) of the virtual conference service provided by the virtual conference server 1, the conference ID, the username and password of the user using the VR goggles 5, 12, 15, etc.

<<VR operation controller>>
The VR operation controllers 6, 13, and 16 have a Bluetooth communication control unit 120, a button information transmission unit 121, and a direction/movement detection unit 122. These functions of the VR operation controllers 6, 13, and 16 are functions or means that are realized by operating in accordance with instructions from the CPU 110 in accordance with the program loaded from the ROM 112 to the main memory 111 shown in FIG.

The Bluetooth communication control unit 120 communicates with the VR goggles 5, 12, and 15 in accordance with the Bluetooth standard. The button information transmission unit 121 passes press information for the menu display button 114, pointer display button 115, and confirm button 116 to the Bluetooth communication control unit 120.

The orientation/movement detection unit 122 detects the orientation (posture in three-dimensional space) and movement of the VR operation controllers 6, 13, 16 from the acceleration and angular velocity data obtained from the 6-axis acceleration/angular velocity sensor 113, and passes this information to the Bluetooth communication control unit 120.

<Operation or Processing>
Next, referring to Figure 6, we will explain the procedure until instructor Z and user A use the laptop PCs 4, 11 and VR goggles 5, 12 to participate in the virtual conference space 8 on the virtual conference server 1 and start a conference in the virtual space. Figure 6 is a sequence diagram explaining the process until instructor Z starts a conference in the virtual conference space 8.

When Lecturer Z, the conference organizer, launches the virtual conference app on his laptop PC 4, the virtual conference connection unit 30 connects to the virtual conference service on the virtual conference server 1 and displays a screen for entering a username and password. In the following, no distinction is made between the virtual conference server 1 and the virtual conference service, and the explanation will mainly focus on the virtual conference service. The URL of the virtual conference service is pre-registered in the laptop PC 4 as setting data for the virtual conference app.

When instructor Z enters his/her username and password, the virtual conference connection unit 30 sends the username and password to the virtual conference service via the wireless LAN communication control unit 32. The virtual conference service authenticates the user and sends an authentication OK response to the laptop PC 4.

S1: When the virtual conference connection unit 30 receives an authentication OK response via the wireless LAN communication control unit 32, the display control unit 34 of the virtual conference app running on the laptop PC 4 displays a start button and a join button for the virtual conference. Since instructor Z is the organizer, he selects the start button.

S2: The virtual conference connection unit 30 sends a virtual conference start command to the virtual conference service via the wireless LAN communication control unit 32. This transmission uses, for example, HTTP.

S3: When the virtual conference control unit 20 of the virtual conference service receives this start command via the communication control unit 23, it generates a virtual conference space 8 and a conference ID.

S4: The communication control unit 23 sends the URL of the virtual conference space 8 and the conference ID to the notebook PC 4.

S5: When the wireless LAN communication control unit 32 of the notebook PC 4 receives this data, the virtual conference connection unit 30 connects to the URL of the virtual conference space 8 via the wireless LAN communication control unit 32 and sends a conference participation command including the received conference ID.

S6: The virtual conference control unit 20 of the virtual conference service compares this conference ID with the conference ID generated in step S3, and if they match, allows the connection between the laptop PC 4 and the virtual conference space 8 (participation in the virtual conference corresponding to the conference ID).

S7: The virtual conference control unit 20 sends a response OK to the notebook PC 4 via the communication control unit 23.

S8: When the wireless LAN communication control unit 32 of the notebook PC 4 receives the response OK, the virtual conference connection unit 30 transmits the URL and conference ID of the virtual conference space 8 to instructor Z's VR goggles 5 via Bluetooth via the Bluetooth communication control unit 33.

S9: When the Bluetooth communication control unit 102 of the VR goggles 5 receives this data, the virtual conference connection unit 103 connects to the URL of the virtual conference space 8 via the wireless LAN communication control unit 101 and sends a conference participation command including the received conference ID.

S10: The virtual conference control unit 20 of the virtual conference service compares this conference ID with the conference ID generated in step S3, and if they match, authorizes connection between the VR goggles 5 and the virtual conference space 8 (participation in the virtual conference corresponding to the conference ID).

S11: The virtual conference control unit 20 sends a response OK to the VR goggles 5 via the communication control unit 23.

S12: The virtual display control unit 26 of the virtual conference service converts the 3D model of the virtual conference space 8 into a view screen visible from the instructor's avatar and transmits it to the VR goggles 5 via the communication control unit 23. The view screen may include display data 9.

S13: When the wireless LAN communication control unit 101 of the VR goggles 5 receives the screen information of the field of view screen, the display control unit 100 displays the received field of view screen of the virtual conference space 8 on the LCD 87.

S14: Next, the display control unit 34 of the virtual conference app on the notebook PC 4 displays a remote desktop selection button.

S15: Instructor Z presses the selection button on the remote desktop.

S16: The remote desktop control unit 31 transmits the screen data of the notebook PC 4 to the virtual conference service via the wireless LAN communication control unit 32.

S17: The virtual display control unit 26 of the virtual conference service displays the screen data received by the communication control unit 23 in the virtual conference space 8 (PC display data 9a).

S18: The virtual display control unit 26 of the virtual conference service sends the PC display data 9a to the VR goggles 5.

S19: The display control unit 100 of the VR goggles 5 displays this PC display data 9a on the LCD 87.

With this, instructor Z has joined the virtual conference, and the field of view screen of the virtual conference space 8 is displayed on the VR goggles 5 (screen data from the laptop PC 4 is also displayed). Note that the virtual display control unit 26 of the virtual conference service controls the screen data of the laptop PC 4 displayed in the virtual conference space 8 so that it is only displayed on instructor Z's VR goggles 5.

<<Users A and B participate in a virtual conference>>
Next, instructor Z sends the URL of the virtual conference space 8 and the conference ID from his/her laptop PC 4 to the laptop PCs of the other conference participants by email or instant message. When user A, a conference participant, starts the virtual conference app on his/her laptop PC 11, the virtual conference connection unit 30 connects to the virtual conference service and displays a screen for entering a username and password. The URL of the virtual conference service is pre-registered in laptop PC 11 as setting data for the virtual conference app.

When User A enters their username and password, the virtual conference connection unit 30 sends this data to the virtual conference service. The virtual conference service authenticates the user and sends an authentication OK response to the notebook PC 11. The virtual conference app on the notebook PC 11 displays a button to start the virtual conference and a button to join.

Figure 7 is a sequence diagram that explains the process up to when user A joins a conference in virtual conference space 8.

S21: User A is a participant, so he selects the join button.

S22: The display control unit 34 of the notebook PC 11 displays a screen for inputting the URL of the virtual conference space 8 and the conference ID.

S23: User A enters the URL and conference ID of virtual conference space 8 notified by instructor Z and presses the OK button.

S24: The virtual conference connection unit 30 sends a connection command including this conference ID to the URL of the virtual conference space 8 via the wireless LAN communication control unit 32.

S25: The virtual conference control unit 20 of the virtual conference service compares this conference ID with the conference ID generated in step S3, and if they match, allows the notebook PC 11 to connect to the virtual conference space 8.

S26: The virtual conference control unit 20 sends a response OK to the notebook PC 11 via the communication control unit 23.

S27: Next, the Bluetooth communication control unit 33 of the notebook PC 11 transmits the URL and conference ID of the virtual conference space 8 to user A's VR goggles 12 via Bluetooth.

S28: When the Bluetooth communication control unit 102 of the VR goggles 12 receives this data, the virtual conference connection unit 103 sends a connection command including this conference ID to the URL of the virtual conference space 8 via the wireless LAN communication control unit 101.

S29: The virtual conference control unit 20 of the virtual conference service compares this conference ID with the conference ID generated in step S3, and if they match, allows the connection between the VR goggles 12 and the virtual conference space 8.

S30: The virtual conference control unit 20 sends a response OK to the VR goggles 12 via the communication control unit 23.

S30-2: The virtual display control unit 26 of the virtual conference service converts the 3D model of the virtual conference space 8 into a view screen visible from User A's avatar and transmits it to the VR goggles 12 via the communication control unit 23. The view screen may include display data 9.

S30-3: When the wireless LAN communication control unit 101 of the VR goggles 12 receives the screen information of the field of view screen, the display control unit 100 displays the received field of view screen of the virtual conference space 8 on the LCD 87.

S31: When the virtual conference connection unit 30 of the virtual conference app running on the notebook PC 11 connects to the virtual conference space 8, the display control unit 34 displays a remote desktop selection button.

S32: User A selects the remote desktop selection button.

S33: The remote desktop control unit 31 of the notebook PC 11 transmits the screen data of the notebook PC 11 to the virtual conference service via the wireless LAN communication control unit 32.

S34: The virtual display control unit 26 of the virtual conference service displays the screen data of the notebook PC 11 in the virtual conference space 8 (PC display data 9b).

S35: The virtual display control unit 26 of the virtual conference service transmits the PC display data 9b to the VR goggles 12.

S36: The display control unit 100 of the VR goggles 12 displays the PC display data 9b on the LCD 87.

The virtual display control unit 26 of the virtual conference service controls the screen data of the notebook PC 11 displayed in the virtual conference space 8 so that it is only displayed on User A's VR goggles 12.

Other participants in the conference perform the same operations as User A above, connecting their laptops and VR goggles to the virtual conference space 8.

<Operation of the VR control controller>
When the instructor Z or the users A and B press the menu display button 114 on the VR operation controller 6, 13, 16, the Bluetooth communication control unit 120 of the VR operation controller 6, 13, 16 transmits a menu display command to the VR goggles 5, 12, 15 via Bluetooth.

The wireless LAN communication control unit 101 of the VR goggles 5, 12, and 15 sends this command to the virtual conference service. When the virtual conference service receives this command, the display data generation unit 25 generates a menu list related to the virtual conference and displays it in the virtual conference space 8. An example of this menu list is shown in Figure 8.

Figure 8 shows the menu list screen 200 displayed in the virtual conference space 8. The menu list screen 200 has a participant list button 201, a whiteboard generation button 202, and a settings button 203. The participant list button 201 is a button that displays a list of participants currently attending a conference identified by a conference ID. The whiteboard generation button 202 is a button for generating a virtual electronic whiteboard in the virtual conference space 8. The settings button 203 is a button that displays various setting screens such as language and font size.

Furthermore, when instructor Z or users A and B press the pointer display button 115 on the VR operation controller 6, 13, 16, the Bluetooth communication control unit 120 of the VR operation controller 6, 13, 16 transmits a pointer display command including information about the orientation and movement of the device to the VR goggles 5, 12, 15 via Bluetooth. The wireless LAN communication control unit 101 of the VR goggles 5, 12, 15 transmits this command to the virtual conference service. When the virtual conference service receives this command, the pointer position calculation unit 27 calculates the display position of the pointer 170 based on the orientation and movement information of the VR operation controller 6, 13, 16. The virtual display control unit 26 displays the pointer 170 at the calculated display position in the virtual conference space 8.

While the user is pressing the pointer display button 115, the Bluetooth communication control unit 120 of the VR operation controller 6, 13, 16 periodically (e.g., every 100 milliseconds) sends a pointer display command including information about the orientation and movement of its own device to the VR goggles 5, 12, 15. The wireless LAN communication control unit 101 of the VR goggles 5, 12, 15 sends this command to the virtual conference service. The pointer position calculation unit 27 calculates the display position of the pointer 170 based on the orientation and movement information of the VR operation controller 6, 13, 16. When the pointer 170 moves and the display of the virtual conference space 8 changes, the virtual display control unit 26 of the virtual conference service sends display data of the virtual conference space 8 to the VR goggles 5, 12, 15. Therefore, the pointer 170 is controlled to move within the virtual conference space 8.

When instructor Z or users A and B press the confirm button 116 on the VR operation controller 6, 13, 16, the Bluetooth communication control unit 120 of the VR operation controller 6, 13, 16 sends a confirm command to the VR goggles 5, 12, 15 via Bluetooth. The wireless LAN communication control unit 101 of the VR goggles 5, 12, 15 sends this command to the virtual conference service. When the virtual conference service receives this command, it determines that the menu item corresponding to the display position of the pointer 170 has been selected (pressed), and performs the operation according to that menu.

When instructor Z (or user A or B) moves the pointer 170 to the whiteboard generation button 202 shown in Figure 8 and presses the confirm button 116, the VR operation controllers 6, 13, and 16 send a confirm command to the virtual conference service via the VR goggles 5, 12, and 15. When the virtual conference service receives this command, the whiteboard generation and management unit 21 generates a virtual electronic whiteboard, and the virtual display control unit 26 displays it in the virtual conference space 8. Note that the screen when the virtual electronic whiteboard is generated is blank. In other words, blank virtual electronic whiteboard display data is displayed in the virtual conference space 8.

<Generation of virtual electronic whiteboard display data>
On this virtual electronic whiteboard, users can display screen data from the laptop PCs 4, 11, and 14, as well as any display content (photos, graphics, etc.). In Fig. 1, the screens of two virtual electronic whiteboards (virtual electronic whiteboard display data 9c and virtual electronic whiteboard display data 9d) are displayed. The virtual electronic whiteboard display data 9c displays screen data from the laptop PC 14 operated by User B, who is participating in the virtual conference. The virtual electronic whiteboard display data 9d displays display data that is not screen data from the laptop PCs 4, 11, and 14 (keyboard input, voice recognition results, handwritten data, etc.).

The following describes the process and operation for displaying the screen data of user B's laptop 14 on the virtual electronic whiteboard. When user B launches the virtual conference app on his or her laptop 14 and performs the same operations as user A above, the laptop 14 and VR goggles 15 connect to the virtual conference space 8. When user B then selects the remote desktop selection button displayed on the laptop 14, the screen data of the laptop 14 is sent to the virtual conference service, and the screen data of the laptop 14 is displayed in the virtual conference space 8.

Figure 9 shows the notebook PC display data 210 of the notebook PC 14 displayed in the virtual conference space 8. In the notebook PC display data 210, a movement bar 211 is added to the screen display area 212 where the screen data of the notebook PC 14 in the virtual conference space 8 is displayed. The virtual electronic whiteboard display data 9c is displayed in the screen display area 212. When user B presses the pointer display button 115 on the VR operation controller 16 and holds the pointer 170 over the movement bar 211, the screen display area 212 of the notebook PC 14 becomes movable. When user B moves the pointer 170 while pressing the pointer display button 115, the screen display area 212 of the notebook PC 14 also moves accordingly.

When User B performs this operation to overlay the screen display area 212 of the laptop PC 14 onto the virtual electronic whiteboard (blank virtual electronic whiteboard display data), the virtual display control unit 26 of the virtual conference service copies the screen data of the laptop PC 14 onto the virtual electronic whiteboard screen, and the screen data of the laptop PC 14 (virtual electronic whiteboard display data 9c) is displayed in the virtual conference space 8. The virtual display control unit 26 also deletes the screen display area 212 and the movement bar 211 that were displaying the screen data of the laptop PC 14.

<Group work in a virtual meeting space>
Next, group work in the virtual conference space 8 will be described with reference to Figure 10. Group work is work in which multiple people form a group, discuss and create work based on a theme, and finally present their conclusions or deliverables. Group work may also be simply called a meeting. Figure 10 shows a schematic diagram of avatars arranged in three groups 1 to 3 held in the virtual conference space 8.

Participants are divided into three groups and conduct group work. The virtual space for each group corresponds to virtual conference space 8. Avatars A to K are the avatars of participants (an example of a second user). Hereinafter, users will be referred to as participants in the group work. Avatars A to D are assigned to Group 1 (an example of a first virtual space), avatars E to H are assigned to Group 2 (an example of a second virtual space), and avatars I to K are assigned to Group 3. Each group may have one or more avatars. This assignment is made, for example, when participants log in to the virtual conference service. There is one instructor Z (an example of a first user). The same instructor Z moderates the group work for groups 1 to 3. Although instructor Z also participates as an avatar, they are referred to as instructor avatars Z1 to Z3 to distinguish them from each other.

For ease of explanation, the VR goggles used by users in Group 2 will be referred to as VR goggles 12, and the VR goggles used by users in Group 3 will be referred to as VR goggles 15.

Each avatar A-K is created when participants first use the virtual conference service, customizing their appearance by selecting from several templates, such as face and clothing, to suit their preferences. In addition, while participating in a conference, the participant's name will be displayed above each avatar A-K.

In the real world, instructor Z can only participate in one group at a time, but in virtual conference space 8, it is possible to generate multiple avatars of himself, so we will explain an example scene in which instructor avatars Z1 to Z3 are present in each group at the same time.

As shown in Figure 11, the avatar management unit 44 manages groups, participants, and instructor avatars in association with each other. Figure 11 shows association information between groups, participants, and instructor avatars. First, participants 1 to 11 log in to the group assigned to them, so participants 1 to 11 are associated with the group. Instructor Z places his or her own instructor avatars 1 to 3 in each group. Instructor avatars 1 to 3 correspond to instructor avatars Z1 to Z3 in Figure 10. The avatar management unit 44 places avatars A to K of participants 1 to 11 in the virtual space corresponding to each group (virtual conference space 8 prepared for each group), and places instructor avatars Z1 to Z3 in the corresponding virtual space.

The virtual display control unit 26 displays the avatars in the group, the instructor avatar, the desk, the electronic blackboard (
A field of view screen is generated by perspectively projecting 3D models such as virtual electronic whiteboard display data 9c, 9d) and a notebook PC (PC display data 9a, 9b). The field of view screen is displayed on the VR goggles 5, 12, 15 of each participant and instructor Z. Regarding audio data, the communication control unit 23 regards the participants and instructor Z associated with a group as one session and transmits the audio of a participant in the group to all participants in the same group and instructor Z. The communication control unit 23 transmits instructor Z's audio data to the VR goggles of all participants in the group. Instructor Z may also be able to specify any participant to send audio data (private conversation).

Only one group's audio data and view screen are sent to Instructor Z's VR goggles 5 at a time. As described below, Instructor Z can switch between these groups at will. Of multiple groups (multiple virtual spaces), the group for which the view screen and audio data are sent to the instructor's VR goggles 5 is sometimes referred to as the group that the instructor is "moderating." Moderating can also be referred to as the instructor "operating" the VR goggles 5. The instructor avatar of the group for which the view screen is currently being displayed and audio data is being output on the VR goggles 5 is called the "master." The instructor always operates the master.

Instructor Z may also be able to set up his/her VR goggles 5 to simultaneously transmit audio data and view screens from multiple groups. Instructor Z may also be able to set up his/her VR goggles 5 to transmit audio data and different view screens from different groups.

Instructor avatars Z1-Z3 may or may not be displayed to participants in groups for which audio data and a visual display screen are not transmitted to instructor Z's VR goggles 5. By not displaying them, participants can see that instructor Z is currently moderating another group. Alternatively, even if instructor avatars Z1-Z3 are displayed, they may be displayed in a manner that shows that instructor Z is currently moderating another group.

Figure 12 shows the field of view of instructor avatar Z1 in Group 1. The field of view of instructor avatar Z1 is the field of view screen displayed on instructor Z's VR goggles 5. More specifically, the field of view screen is an image projected in the direction instructor Z is looking in the virtual 360-degree virtual space in which Group 1 is constructed. Therefore, the field of view screen can display any 360-degree space. Avatars A to D appear in the field of view of the instructor avatar. The following explanation assumes that instructor Z is initially operating instructor avatar Z1 in Group 1. That is, the field of view screen of Group 1 is displayed on instructor Z's VR goggles 5, and audio data for Group 1 is output.

In addition to showing avatars A to D doing group work, the view screen also displays an avatar switch button 131 and a notification button 132. The avatar switch button 131 is a button for switching the view screen displayed on instructor Z's VR goggles 5 and the audio data output to that of another group. The notification button 132 is a button that indicates that there is a notification for instructor avatar Z1 of the group whose VR goggles 5 are currently displaying the view screen and outputting audio data. The view screen that can be seen by the other instructor avatars Z2 and Z3 may be the same as in Figure 12 (however, the avatars in view and the state of the notification button will differ).

<Notification to the instructor avatar when no conversation is taking place in Group 2>
For example, even if there is a group where no conversation is taking place, if instructor Z is not the moderator of that group, instructor Z will not be able to notice. In such cases, it would be convenient if a notification to that effect was displayed on the field of view screen.

One method for detecting that no conversation has taken place for a certain period of time is for the output control unit 43 to use the voice recognition results of the voice recognition unit 42. When an avatar speaks within a group, the voice recognition unit 42 recognizes the voice data and can therefore determine that a speech has taken place within the group. If the voice recognition unit 42 is unable to recognize the voices of participants within a group for a certain period of time, it can determine that no speech has taken place within the group during that time.

If the output control unit 43 detects that no conversation has taken place within a group for a certain period of time, it requests the virtual display control unit 26 of the virtual conference server to add a notification message to the field of view of instructor Z's VR goggles 5, indicating that the discussion has stalled. The virtual display control unit 26 receives the request to add a notification message and displays the notification message on the field of view of the group chaired by instructor Z. In this way, instructor Z can confirm from the notification message on his or her field of view that the discussion has stalled (no comments) in other groups that are currently displayed on the field of view but not outputting audio data.

Figure 13 shows an example of the field of view screen of instructor avatar Z1, displaying a notification message 133 indicating that the discussion is stalled. The notification message 133 reads "The discussion is stalled. Group 2" at the top of the field of view screen. Instructor Z confirms from the notification message 133 that the discussion in Group 2 is stalled, and presses the notification message 133. The notification message 133, "The discussion is stalled. Group 2," may be output by voice (the notification message 133 may or may not be displayed).

Figure 14 is a diagram illustrating instructor Z pressing the notification message 133. Instructor Z operates the VR operation controller 6, moves the pointer 170 to the notification message 133, and presses the confirm button 116. When the notification message 133 is pressed, the virtual display control unit 26 displays the now switch button 134 and the later switch button 135.

Figure 15 is an example of a field of view screen displaying the Switch Now button 134 and Switch Later button 135. The Switch Now button 134 and Switch Later button 135 are displayed on the field of view screen for Group 1 displayed by Instructor Z's VR goggles 5, so Instructor Z can display the field of view screen for Group 2 by pressing the Switch Now button 134. Figure 16 shows the case where the Switch Now button 134 has been pressed, and Figure 17 shows the case where the Switch Later button 135 has been pressed.

Figure 16 shows the field of view screen for Group 2 displayed by the VR goggles 5 upon pressing the Switch Now button 134. Because instructor Z pressed the Switch Now button 134 displayed in response to the notification message for Group 2, the virtual display control unit 26 displays the field of view screen for Group 2 on the VR goggles 5. In a setting where instructor avatar Z2 is not displayed in groups that are not moderated, the virtual display control unit 26 places instructor avatar Z2 in Group 2. The virtual display control unit 26 displays a 3D model of Group 2 on the VR goggles 5. Therefore, this field of view screen is the field of view screen seen by instructor avatar Z2 of Group 2. Therefore, avatars E to H are displayed. Note that if no conversation has taken place in Group 3 for a certain period of time, a list of Groups 2 and 3 is displayed by pressing the Switch Now button 134, and instructor Z may select Group 2 or 3 to display from the list.

Figure 17 shows the field of view screen for Group 1 displayed on the VR goggles 5 when the "later" switch button 135 is pressed. Since Instructor Z is still moderating Group 1, this field of view screen displays a 3D model of Group 1 on the VR goggles 5. Because the "later" switch button 135 was pressed (an example of postponing a response), the virtual display control unit 26 erases the notification message 133 and displays a mark 136 on the notification button 132 indicating that a notification is available. Note that when the "later" switch button 135 is pressed, the output control unit 43 stores the notification message in the conference data storage unit 24. The conference data storage unit 24 stores the notification message itself, the name of the group that did not speak for a certain period of time, the time, etc.

In addition, after a certain period of time has passed since the notification message was displayed as shown in Figure 14, the virtual display control unit 26 will hide the notification message 133 and switch to displaying a mark 136 indicating that a notification is available on the notification button 132, even if instructor Z does not later press the switch button 135.

Figure 18 shows the field of view screen of avatar E (Aomori-san) in group 2 when instructor Z presses the "later" switch button 135. As an example, Figure 18 shows the field of view screen of avatar E, Aomori-san, in group 2. Because instructor Z pressed the "later" switch button 135, the output control unit 43 outputs audio data so that it can be heard from the direction of instructor avatar Z2. The virtual display control unit 26 may move the mouth of instructor avatar Z2 or display a statement from instructor avatar Z2 in a speech bubble or the like. The content of the statement is, for example, "I will join the discussion later." "I will join the discussion later" is a standard phrase generated by the output control unit. Because instructor Z's voice will be transmitted to group 1, making it a standard phrase prevents instructor Z from making any statements. A selection of standard phrases is displayed on the field of view screen of group 1, from which instructor Z may select.

As a result, avatars E to H in Group 2, who have not spoken for a certain period of time, can share with the other avatars that the instructor will join the discussion later.

In Figure 18, instructor avatar Z2 is displayed in group 2. In this way, if the switching button 135 is pressed later, the virtual display control unit 26 may temporarily display instructor avatar Z2. Alternatively, the virtual display control unit 26 may display an avatar for the output control unit (an avatar for a bot) and output a statement from instructor Z as audio data in the form of a message. Alternatively, the virtual display control unit 26 may output only the generated audio without displaying instructor avatar Z2 or an avatar for a bot.

In addition, in Figures 14 to 18, the output control unit 43 detects a delay in speech within the group, but the output control unit 43 may also detect a delay in speech for each participant and notify the instructor avatar Z1.

<Operation or Processing>
19 is a sequence diagram illustrating the process in which the output control unit 43 displays a notification message on the VR goggles 5 of instructor Z. As a premise for the explanation of FIG. 19, instructor avatar Z1 of instructor Z is placed in group 1, and one or more participant avatars are placed in group 2. If no participant utterance is detected in group 2, the virtual display control unit 26 generates a message indicating that no participant utterance is detected in group 2, and the communication control unit 23 transmits this message to the instructor's VR goggles 5 that are displaying the field of view information for group 1.

S51: The output control unit 43 is always on standby for conversations between avatars in each group. The output control unit 43 has a timer for each group, and resets the timer when there is conversation. When the timer times out, it determines that a certain period of time has passed without any conversation.

S52: If there is no conversation for a certain period of time, a request to generate a notification message indicating that the discussion is stalling is sent to the display data generation unit 25.

S53: Upon receiving the request, the display data generation unit 25 generates a notification message indicating that the discussion is stalled.

S54: The display data generation unit 25 requests the virtual display control unit 26 to generate a field of view screen onto which the notification message is synthesized.

S55: The virtual display control unit 26 generates a field of view screen that combines the notification message.

S56: The virtual display control unit 26 transmits screen information of the field of view screen onto which the notification message has been combined to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 32 of the VR goggles 5 receives the screen information of the field of view screen, and the display control unit 34 displays it on the LCD 87 of the VR goggles 5 (see Figure 13).

FIG. 20 is a sequence diagram illustrating the process when instructor Z responds to a notification message.
S61: Instructor Z points the pointer 170 to the notification message 133 and presses the confirm button 116 (see FIG. 14).

S62-1: The VR operation controller 6 receives a press of the confirmation button 116, and the button information transmission unit 121 transmits a notification message to the VR goggles 5 via the Bluetooth communication control unit 120 to the effect that the confirmation button 116 has been pressed.
S62-2: The wireless LAN communication control unit 101 of the VR goggles 5 transmits to the virtual conference server 1 a notification that the notification message has been pressed.

S63: The communication control unit 23 of the virtual conference server receives the notification message indicating that the button has been pressed, and the virtual conference control unit 20 sends a request to generate the Now switch button 134 and Later switch button 135 to the display data generation unit 25.

S64: In response to this request, the display data generation unit 25 generates a now switch button 134 and a later switch button 135.

S65: The display data generation unit 25 also requests the virtual display control unit 26 to generate a field of view screen including a now switch button 134 and a later switch button 135.

S66: The virtual display control unit 26 combines the now switch button 134 and the later switch button 135 into the current instructor avatar's field of view screen.

S67: The virtual display control unit 26 transmits screen information of a field of view screen on which the Now switch button 134 and Later switch button 135 are combined to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen on which the Now switch button 134 and Later switch button 135 are combined (see Figure 15).

Next, we will explain what happens when instructor Z presses the Switch Now button 134 on the field of view screen. In step S70, instructor Z moves the pointer 170 to the Switch Now button 134. When the VR goggles 5 of the instructor that output the notification message 133 accepts an operation to display field of view information for group 2, the virtual display control unit 26 generates field of view information for instructor avatar Z2 of instructor Z who is placed in group 2, and the communication control unit 23 sends this information to instructor Z's VR goggles 5.

S71: Instructor Z positions the pointer 170 on the Now switch button 134 and presses the Confirm button 116.

S72-1: The VR operation controller 6 receives a press of the confirmation button 116, and the button information transmission unit 121 transmits to the VR goggles 5 via the Bluetooth communication control unit 120 a message indicating that the switch now button 134 has been pressed.
S72-2: The wireless LAN communication control unit 101 of the VR goggles 5 immediately transmits to the virtual conference server 1 a message that the switching button 134 has been pressed.

S73: The communication control unit 23 of the virtual conference server receives the notification that the switch now button 134 has been pressed, and the virtual conference control unit 20 sends a request to switch the field of view screen specifying group 2 to the virtual display control unit 26.

S74: In response to this request, the virtual display control unit 26 switches the field of view screen of the instructor avatar to the field of view screen of group 2. In detail, the field of view screen is an image projected in the direction that instructor Z is looking in the virtual 360° virtual space in which group 2 is constructed.

S75: The virtual display control unit 26 transmits screen information of the field of view screen for group 2 to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen for group 2 (see Figure 16).

Next, we will explain what happens when instructor Z presses the "later" switch button 135 on the field of view screen in Figure 15. In step S80, instructor Z moves the pointer 170 to the "later" switch button 135. If instructor Z's VR goggles 5, which are displaying the notification message 133, receive an operation to postpone responding to the notification message rather than an operation to display the field of view information for group 2, the output control unit 43 generates audio data indicating that the response will be made later. The communication control unit 23 transmits the audio data to the VR goggles 12 of group 2.

S81: Instructor Z later aligns the pointer 170 with the switch button 135 and presses the confirm button 116.

S82-1: The VR operation controller 6 accepts the pressing of the confirmation button 116, and the button information transmission unit 121 later transmits to the VR goggles 5 via the Bluetooth communication control unit 120 that the switching button 135 has been pressed.
S82-2: The wireless LAN communication control unit 101 of the VR goggles 5 later transmits to the virtual conference server 1 a message that the switching button 135 has been pressed.

S83: The communication control unit 23 of the virtual conference server later receives information that the switch button 135 has been pressed, and the virtual conference control unit 20 sends a request to the display data generation unit 25 to generate a notification button 132 with the mark 136 added.

S84: In response to this request, the display data generation unit 25 generates a notification button 132 with a mark 136 added.

S85: The display data generation unit 25 also requests the virtual display control unit 26 to generate a field of view screen including the notification button 132 to which the mark 136 has been added.

S86: The virtual display control unit 26 composites the notification button 132 with the mark 136 added onto the current field of view screen of the instructor avatar Z1.

S87: The virtual display control unit 26 transmits screen information of the field of view screen onto which the notification button 132 with the mark 136 added has been superimposed to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen onto which the notification button 132 with the mark 136 added has been superimposed (see FIG. 17).

S88: Also, since the "later" switch button 135 was pressed, the virtual conference control unit 20 requests the output control unit 43 to utter the utterance "I will join the discussion later."

S89: In response to this request, the output control unit 43 generates voice data for the utterance "I will join the discussion later." The output control unit 43 also stores the notification message 133 in the conference data storage unit 24.

S90: The output control unit 43 transmits the audio data to the VR goggles 12 of each participant in Group 2 via the communication control unit 23.

S91: The wireless LAN communication control unit 101 of the VR goggles 12 receives the audio data and outputs the audio "I will join the discussion later" from the speaker 92. Note that although the VR goggles 12 outputs the audio "I will join the discussion later", similar content may be displayed using linguistic information. In this case, a notification message is synthesized on the field of view screen of Group 2.

<Major Effects>
In the virtual space system 300 of this embodiment, in a situation where instructor Z participates in group work and places instructor avatars Z1 to Z3 that represent instructor Z in each group, notifications according to the status of groups that instructor Z is not moderating are sent to instructor avatar Z1, so that instructor Z can understand the status of groups for which instructor Z is not currently displayed on the VR goggles 5 and for which no audio data is being output.

Second Embodiment
In this embodiment, a virtual conference server 1 is described in which a group not being moderated notifies the instructor avatar Z1 of the group 1 being moderated that there is an inquiry such as a question.

Figure 21 shows an example of a statement made by an avatar in a group. In Figure 21, avatar J (Mr. Chiba) from Group 3 makes a statement 161 to instructor avatar Z3: "Mr. Nakazawa, is what I explained correct?" The content of statement 161 in the speech bubble in Figure 21 is for explanatory purposes and does not need to be actually displayed.

Instructor Z's VR goggles 5 display the field of view for Group 1, and the audio for Group 1 is output from the speaker 92, so Instructor Z is unaware of what Group 3 is saying.

The voice recognition unit 42 constantly recognizes the speech of each group and transmits the speech data resulting from the recognition to the output control unit 43. "Nakazawa-san" included in speech 161 is the name of instructor Z (or just the first or last name would be acceptable), and when the output control unit 43 detects that speech 161 containing instructor Z's name has been made, it determines that an inquiry has been made to instructor Z. The output control unit 43 issues a request to the virtual display control unit 26 of the virtual conference server 1 to add a notification message indicating that an inquiry has been made to the field of view screen of instructor Z's group 1. The virtual display control unit 26 receives the request to add the notification message and synthesizes the notification message on the field of view screen of instructor Z's group 1.

Figure 22 is an example of the field of view screen of the instructor avatar of Group 1 on which a notification message for an inquiry is displayed. A notification message 137 reading "We have a question for you. From Chiba, Group 3" is displayed at the top of the field of view screen. The "from Chiba, Group 3" in the notification message 137 indicates that an inquiry has been made by Chiba of Group 3. Note that the voice recognition unit 42 identifies the speaker based on which VR goggles 15 the voice data is from, so it can determine the participant who said "Nakazawa-san." The notification message 137 reading "We have a question for you. From Chiba, Group 3" may be output by voice (the notification message 137 may or may not be displayed).

Note that avatar J's statement may include "Nakazawa-san" even if it is not an inquiry, so it is preferable for the output control unit 43 to analyze the context, determine whether it is an inquiry, and notify instructor avatar Z1 if it is an inquiry.

In this way, instructor Z can confirm that there has been an inquiry from another group whose VR goggles 5 are not currently displaying a view screen or outputting audio, by checking the notification message 137 superimposed on the view screen currently displayed on the VR goggles 5.

As for the action taken by instructor Z in response to the notification message 137 in Figure 22, similar to Figure 15 in the first embodiment, instructor Z can select whether to switch immediately or later by selecting the switch now button 134 or the switch later button 135.

It is also possible to simultaneously display multiple notification messages 133, 137 on the field of view screen, as shown in Figure 23. In Figure 23, the output control unit 43 displays notification message 133 indicating that the discussion in Group 2 is stalled, and notification message 137 indicating that there is an inquiry from Group 3.

When multiple notification messages 133, 137 are displayed simultaneously, the older ones may be displayed at the top, or after a certain period of time they may be displayed in a highlighted color such as red or may flash.

<Operation or Processing>
24 is a sequence diagram illustrating the process in which the output control unit 43 notifies the instructor avatar of an inquiry from a group that is not currently displaying a field of view screen or outputting audio data on the VR goggles 5. As a premise of FIG. 24, the instructor's instructor avatar Z1 is placed in group 1, and one or more participant avatars are placed in group 3. If a participant in group 3 makes a comment about instructor Z, the virtual display control unit 26 generates a notification message indicating that there is an inquiry from the participant, and the communication control unit 23 sends this to the instructor's VR goggles 5 that are displaying the field of view information for group 1.

S101: For example, Mr. Chiba from Group 3 spoke.

S102: The microphone 91 in Chiba's VR goggles captures what he says and converts it into voice data. The wireless LAN communication control unit 101 in the VR goggles 15 transmits the voice data to the virtual conference server.

S103: The communication control unit 23 of the virtual conference server receives the voice data and uses the voice recognition unit 42 to recognize the voice (convert it into text). The voice recognition unit 42 notifies the output control unit 43 of the name of the group that made the comment and the name of the speaker.

The output control unit 43 determines whether the statement is an inquiry to the instructor (whether the instructor's first or last name is included), and if the statement is an inquiry to instructor Z, the following processing is performed. That is, if the last name or first name of instructor Z is included among the participants in group 3, the virtual display control unit 26 generates a notification message indicating that there is an inquiry from a participant in another group, and the communication control unit 23 sends this to the instructor's VR goggles 5 displaying the field of view information for group 1.

S104: The output control unit 43 requests the display data generation unit 25 to generate a notification message indicating that an inquiry has been received. The output control unit 43 also attaches the name of the group that made the comment and the name of the person who made the comment.

S105: In response to this request, the display data generation unit 25 generates a notification message indicating that an inquiry has been made.

S106: The display data generation unit 25 also requests the virtual display control unit 26 to generate a field of view screen that combines a notification message indicating that an inquiry has been received. This field of view screen is the field of view screen of the group that is currently being displayed on instructor Z's VR goggles 5.

S107: The virtual display control unit 26 generates a view screen in which a notification message indicating that an inquiry has been received is superimposed on the view screen of the current instructor avatar.

S108: The virtual display control unit 26 transmits screen information of the field of view screen onto which a notification message indicating that an inquiry has been received to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen onto which a notification message indicating that an inquiry has been received (see Figure 22).

<Major Effects>
According to this embodiment, in addition to the effects of the first embodiment, one instructor Z places instructor avatars Z1 to Z3 in multiple groups, and even if an inquiry comes from a group that is not displaying the field of view screen on the VR goggles 5, the inquiry can be noticed and responded to quickly.

[Third embodiment]
In this embodiment, an explanation will be given of the alter ego switching of the instructor avatar by pressing the alter ego switching button 131. Alter ego switching refers to the instructor Z arbitrarily switching which group's field of view screen is displayed on the VR goggles 5.

Figure 25 shows an example of the avatar switching button 131 displayed on instructor Z's VR goggles 5. Instructor Z moves the pointer 170 and presses the avatar switching button 131. This causes the virtual display control unit 26 to display a field of view screen that combines a list 141 of switchable instructor avatars, as shown in Figure 26.

Figure 26 shows a view screen displaying a list 141 of instructor avatars. The list 141 of instructor avatars displays instructor avatar buttons 141a-141c for all instructor avatars assigned to each group. The instructor avatar of the group whose view screen is currently being displayed on the VR goggles 5 and whose audio data is being output is called the "master." The master is indicated by icon 143 on the instructor avatar button 141a. The view screen and audio data of the group with icon 143 are being sent to instructor Z's VR goggles 5 (instructor Z is moderating). The instructor avatar buttons 141a-141c also display the names of the groups in which instructor avatars Z1-Z3 are assigned. These are registered in the correspondence information in Figure 11.

When instructor Z selects, for example, the instructor avatar button 141b for group 2 from the list of instructor avatars 141, the virtual display control unit 26 switches the field of view screen of instructor Z's VR goggles 5 to the field of view screen seen by instructor avatar Z2 for group 2.

<Operation or Processing>
FIG. 27 is a sequence diagram illustrating the process of switching the view screen displayed by the instructor Z's VR goggles 5.

S121: Instructor Z moves the pointer 170 to the ghost switch button 131 while pressing the pointer display button 115.

S121-2: While instructor Z is pressing the pointer display button 115, the Bluetooth communication control unit 120 of the VR operation controller 6 periodically (for example, every 100 milliseconds) transmits information regarding the movement of the pointer (information regarding the orientation and movement of the device) to the VR goggles 5.
S122: The wireless LAN communication control unit 101 of the VR goggles 5 transmits a position calculation request based on information regarding the movement of the pointer to the virtual conference service.

S123: The communication control unit 23 of the virtual conference service receives the position calculation request. The pointer position calculation unit 27 of the virtual conference service calculates the display position of the pointer 170 based on the orientation and movement information of the VR operation controller 6.

S124: The pointer position calculation unit 27 requests the display data generation unit 25 to generate a pointer 170 along with the calculated position.

S125: In response to this request, the display data generation unit 25 generates a pointer 170 at the calculated position.

S126: The display data generation unit 25 also requests the virtual display control unit 26 to generate a field of view screen that combines the moved pointer 170. This field of view screen is the field of view screen of the group currently displayed on instructor Z's VR goggles 5.

S127: The virtual display control unit 26 generates a field of view screen in which the moved pointer 170 is superimposed on the current field of view screen of the instructor avatar.

S128: The virtual display control unit 26 transmits screen information of the field of view screen onto which the moved pointer 170 is superimposed to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen onto which the moved pointer 170 is superimposed (see Figure 25).

S129: Instructor Z presses the confirm button 116 with the pointer 170 overlapping the ghost switch button 131.

S130-1: The Bluetooth communication control unit 120 of the VR operation controller 6 transmits a confirmation command to the VR goggles 5 via Bluetooth.
S130-2: The wireless LAN communication control unit 101 of the VR goggles 5 sends this command to the virtual conference service.

S131: When the communication control unit 23 of the virtual conference service receives this confirmation command, the virtual conference control unit 20 requests the display data generation unit 25 to generate a list 141 of instructor avatars.

S132: In response to this request, the display data generation unit 25 generates a list 141 of instructor avatars. The avatar management unit 44 manages the correspondence between groups 1 to 3 and instructor avatars Z1 to Z3, and also manages information about which group the instructor avatar is currently assigned to. The display data generation unit 25 obtains information about instructor avatars Z1 to Z3 assigned to each group from the avatar management unit 44.

S133: The display data generation unit 25 also requests the virtual display control unit 26 to generate a field of view screen that combines the list 141 of instructor avatars. This field of view screen is the field of view screen of the group currently displayed on instructor Z's VR goggles 5.

S134: The virtual display control unit 26 generates a field of view screen in which the list of instructor avatars 141 is superimposed on the field of view screen of the current instructor avatar.

S135: The virtual display control unit 26 transmits screen information of the field of view screen onto which the list of instructor avatars 141 has been superimposed to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen onto which the list of instructor avatars 141 has been superimposed (see Figure 26).

In step S136, instructor Z operates the VR operation controller 6 to move the pointer 170 to the list of instructor avatars 141. This process may be similar to steps S121 to S128.

S137: Instructor Z presses the confirm button 116 with the pointer 170 overlapping the instructor avatar button 141b. In this way, when the VR goggles 5 accepts the selection of an avatar from the list of instructor Z's avatars placed in groups 1 to 3, the virtual display control unit 26 generates field of view information for the group in which the selected avatar is placed, and the communication control unit 23 transmits the field of view information to instructor Z's VR goggles 5.

S138-1: The Bluetooth communication control unit 120 of the VR operation controller 6 transmits a confirmation command together with the identification information of the selected instructor avatar button 141b to the VR goggles 5 via Bluetooth.
S138-2: The wireless LAN communication control unit 101 of the VR goggles 5 transmits this confirmation command to the virtual conference service.

S139: When the communication control unit 23 of the virtual conference service receives the identification information and confirmation command of the lecturer avatar button 141b, the virtual conference control unit 20 requests the virtual display control unit 26 to specify a group and switch the field of view screen.

S140: The virtual display control unit 26 switches to the field of view screen of the specified group.

S141: The virtual display control unit 26 transmits screen information of the field of view screen of the switched-to group to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen of the switched-to group.

<Major Effects>
According to this embodiment, when instructor Z's instructor avatars Z1 to Z3 are arranged in groups 1 to 3, instructor Z can switch the field of view screen of any group.

[Fourth embodiment]
In this embodiment, the re-display of the notification message by pressing the notification button 132 will be described.

Figure 28 shows a field of view screen including a notification button 132 with a mark 136 added. When lecturer Z presses the notification button 132, the virtual display control unit 26 obtains the contents of the notification message from the conference data storage unit 24, and the virtual display control unit 26 generates a field of view screen with the notification message 133 superimposed on it, as shown in Figure 22.

<Operation or Processing>
29 is a sequence diagram illustrating the process in which instructor Z causes the VR goggles 5 to display the notification message again. When the VR operation controller 6 of instructor Z receives an operation to display the notification message, the communication control unit 23 transmits the notification message obtained from the conference data storage unit 24 to instructor Z's VR goggles 5, which are displaying the field of view information of group 1.

S151: Instructor Z presses the confirm button 116 with the pointer 170 overlapping the notification button 132.

S152-1: The Bluetooth communication control unit 120 of the VR operation controller 6 transmits a confirmation command to the VR goggles 5 via Bluetooth, along with a notice that the notification button 132 has been pressed.
S152-2: The wireless LAN communication control unit 101 of the VR goggles 5 transmits this confirmation command to the virtual conference service.

S153: When the communication control unit 23 of the virtual conference service receives the confirmation command, the virtual conference control unit 20 requests the virtual display control unit 26 to re-display the notification message.

S154: In response to this request, the virtual display control unit 26 obtains from the conference data storage unit 24 a notification message indicating that instructor Z has not pressed the switch button 134 now. "Not pressing the switch button 134 now" means that instructor Z is not responding to inquiries from the group (is not displaying the field of view screen for that group).

S155: The virtual display control unit 26 generates a view screen in which the notification message is superimposed on the view screen of the current instructor avatar.

S156: The virtual display control unit 26 transmits screen information of the field of view screen onto which the notification message has been combined to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen onto which the notification message has been combined.

Figure 30 is an example of the field of view screen of the instructor avatar of group 1 on which the inquiry notification message 137 is displayed again. The explanation of Figure 30 will mainly focus on the differences from Figure 22. If instructor Z selects the switch button 135 later when the notification message 133 is displayed, the notification message 133 may also be displayed. Figure 30 shows the time when the notification message 137 was first displayed. This allows instructor Z to notice that time has passed since the first display and respond promptly.

Also, even if instructor Z does not press the notification button 132, the output control unit 43 may automatically display the notification message 137 when a certain amount of time has passed since the first display time of the notification message 137. In this way, even if instructor Z forgets that he or she has not responded to the notification message 137, the output control unit 43 can remind him or her.

<Major Effects>
According to this embodiment, it is possible to display again a notification message regarding a group for which instructor Z has postponed taking action.

[Other application examples]
The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

For example, in this embodiment, instructor Z's VR goggles 5 switch between displaying the field of view screen for each group, but the field of view screen may also be displayed on instructor Z's laptop PC 4, for example. Also, the field of view screens of instructor avatars Z1 to Z3 for each group may be displayed constantly on a large display such as an electronic whiteboard.

Furthermore, the content of the notification message sent when instructor Z is not moderating may be anything, not just a message that the discussion is stalling or an inquiry. For example, the notification may be that the discussion is getting heated, or the output control unit 43 may detect and notify that the discussion is moving in a direction that is different from the correct answer.

In addition, in this embodiment, instructor Z's VR goggles 5 output audio data for the group displaying the field of view screen, but the VR goggles 5 may display a different group for the field of view screen and output audio data. For example, the VR goggles 5 could display the field of view screen for group 1 but output audio data for group 2. Instructor Z can select the group for displaying the field of view screen and the group for outputting audio data.

In addition, in this embodiment, when instructor Z's VR goggles 5 displayed a notification message and the switch now button 134 was pressed, the field of view screen for group 2 was displayed, but the VR goggles 5 may also split the LCD 87 to simultaneously display the field of view screens for groups 1 and 2. In this case, it is conceivable to temporarily switch the audio data to group 2, or have instructor Z select it.

Furthermore, when the switching button 135 is pressed later, the field of view screen displayed by the VR goggles 5 may remain at Group 1, while only the audio data may be switched to Group 2 or 3. In this case, the VR goggles 5 may display on the LCD 87 which group's audio data is being output. Instructor Z may be able to freely select the group for which audio data will be output.

In addition, in this embodiment, participants selected the virtual electronic whiteboard display data by operating a VR operation controller, but voice operation may also be used.

In addition, although this embodiment does not provide detailed information about an actual electronic whiteboard, if there is an electronic whiteboard in the real space, the screen of the electronic whiteboard in the real space can be synchronized with the virtual conference space 8.

Furthermore, the configuration examples shown in Figure 5 and elsewhere are divided according to main functions to make it easier to understand the processing performed by the virtual space system 300. The method of dividing the processing units or their names does not limit the present invention. The processing performed by the virtual space system 300 can also be divided into even more processing units depending on the processing content. It can also be divided so that one processing unit includes even more processing.

Furthermore, each function of the embodiments described above can be realized by one or more processing circuits. Here, the term "processing circuit" as used herein includes processors programmed to perform each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to perform each of the functions described above.

Embodiments of the present invention provide significant improvements in computer power and functionality. These improvements allow users to utilize computers that provide more efficient and robust interaction with tables, the way information is stored and presented in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of more efficient, powerful, and robust user interfaces. Such user interfaces provide better interaction between humans and machines.
<Additional Notes>
[Appendix 1]
An information processing system that transmits visibility information of an avatar placed in a virtual space to a terminal device via a network,
an avatar management unit that arranges a plurality of avatars of a first user in different virtual spaces;
a virtual display control unit that generates information about the virtual space that is not displayed on the terminal device of the first user;
a communication control unit that transmits information about the virtual space to the terminal device of the first user;
An information processing system having the above.
[Appendix 2]
an avatar of the first user is located in a first virtual space, and one or more avatars of second users are located in a second virtual space;
If no utterance of the second user is detected in the second virtual space,
the virtual display control unit generates a message indicating that no utterances of the second user are detected in the second virtual space;
The information processing system according to claim 1, wherein the communication control unit transmits the message to the terminal device of the first user that is displaying visibility information of the first virtual space.
[Appendix 3]
an avatar of the first user is located in a first virtual space, and one or more avatars of second users are located in a second virtual space;
If the statement of the second user in the second virtual space is related to the first user,
the virtual display control unit generates a message indicating that there is an inquiry from the second user;
The information processing system according to claim 1, wherein the communication control unit transmits the message to the terminal device of the first user that is displaying visibility information of the first virtual space.
[Appendix 4]
If the second user's comment in the second virtual space includes the first name or last name of the first user,
the virtual display control unit generates a message indicating that there is an inquiry from the second user;
The information processing system described in Appendix 3, wherein the communication control unit transmits the message to the terminal device of the first user that is displaying field of view information of the first virtual space.
[Appendix 5]
When the terminal device of the first user who output the message receives an operation to display view information of the second virtual space,
the virtual display control unit generates visibility information of an avatar of the first user arranged in the second virtual space;
The information processing system according to any one of appendices 2 to 4, wherein the communication control unit transmits the visibility information to the terminal device of the first user.
[Appendix 6]
When the terminal device of the first user that displayed the message receives an operation to postpone the response to the message instead of an operation to display the view information of the second virtual space,
an output control unit that generates voice data indicating that the message will be addressed later;
The information processing system according to any one of claims 2 to 4, wherein the communication control unit transmits the voice data to the terminal device of the second user.
[Appendix 7]
When the terminal device of the first user that displayed the message receives an operation to postpone the response to the message instead of an operation to display the view information of the second virtual space,
The output control unit stores the message in a storage unit,
When the terminal device of the first user receives an operation to display the message,
An information processing system as described in Appendix 6, wherein the communication control unit transmits the message obtained from the memory unit to the terminal device of the first user displaying field of view information of the first virtual space.
[Appendix 8]
When a selection of an arbitrary avatar is received from a list of avatars of the first user arranged in the virtual space displayed by the terminal device of the first user,
the virtual display control unit generates view information of the virtual space in which the selected avatar is placed;
The information processing system according to any one of claims 1 to 7, wherein the communication control unit transmits the visibility information to the terminal device of the first user.

1 Virtual conference server 2 Electronic whiteboard 4, 11, 14 Notebook PC
5, 12, 15 VR goggles 6, 13, 16 VR operation controller 8 Virtual meeting space 9 Display data

JP 2009-145883 A

Claims (10)

An information processing system that transmits visibility information of an avatar placed in a virtual space to a terminal device via a network,
an avatar management unit that arranges a plurality of avatars of a first user in different virtual spaces;
a virtual display control unit that generates information about the virtual space that is not displayed on the terminal device of the first user;
a communication control unit that transmits information about the virtual space to the terminal device of the first user;
An information processing system having the above. an avatar of the first user is located in a first virtual space, and one or more avatars of second users are located in a second virtual space;
If no utterance of the second user is detected in the second virtual space,
the virtual display control unit generates a message indicating that no utterances of the second user are detected in the second virtual space;
The information processing system according to claim 1 , wherein the communication control unit transmits the message to the terminal device of the first user that is displaying the field of view information of the first virtual space. an avatar of the first user is located in a first virtual space, and one or more avatars of second users are located in a second virtual space;
If the statement of the second user in the second virtual space is related to the first user,
the virtual display control unit generates a message indicating that there is an inquiry from the second user;
The information processing system according to claim 1 , wherein the communication control unit transmits the message to the terminal device of the first user that is displaying the field of view information of the first virtual space. If the second user's comment in the second virtual space includes the first name or last name of the first user,
the virtual display control unit generates a message indicating that there is an inquiry from the second user;
The information processing system according to claim 3 , wherein the communication control unit transmits the message to the terminal device of the first user that is displaying the field of view information of the first virtual space. When the terminal device of the first user who output the message receives an operation to display view information of the second virtual space,
the virtual display control unit generates visibility information of an avatar of the first user arranged in the second virtual space;
5. The information processing system according to claim 2, wherein the communication control unit transmits the visibility information to the terminal device of the first user. When the terminal device of the first user that displayed the message receives an operation to postpone the response to the message instead of an operation to display the view information of the second virtual space,
an output control unit that generates voice data indicating that the message will be addressed later;
5. The information processing system according to claim 2, wherein the communication control unit transmits the voice data to the terminal device of the second user. When the terminal device of the first user that displayed the message receives an operation to postpone the response to the message instead of an operation to display the view information of the second virtual space,
The output control unit stores the message in a storage unit,
When the terminal device of the first user receives an operation to display the message,
The information processing system according to claim 6 , wherein the communication control unit transmits the message obtained from the memory unit to the terminal device of the first user that is displaying field of view information of the first virtual space. When a selection of an arbitrary avatar is received from a list of avatars of the first user arranged in the virtual space displayed by the terminal device of the first user,
the virtual display control unit generates view information of the virtual space in which the selected avatar is placed;
The information processing system according to claim 1 , wherein the communication control unit transmits the visibility information to the terminal device of the first user. An output method performed by an information processing system that transmits visual field information of an avatar placed in a virtual space to a terminal device via a network, comprising:
placing a plurality of avatars of the first user in different virtual spaces;
A process of generating information about the virtual space that is not displayed on the terminal device of the first user;
transmitting information about the virtual space to the terminal device of the first user;
Output method. An information processing system that transmits visual field information of an avatar placed in a virtual space to a terminal device via a network,
an avatar management unit that arranges a plurality of avatars of a first user in different virtual spaces;
a virtual display control unit that generates information about the virtual space that is not displayed on the terminal device of the first user;
a communication control unit that transmits information about the virtual space to the terminal device of the first user;
A program to function as a