JP2024110135A - Information processing system and program - Google Patents

Abstract

To provide an information processing system capable of expanding a width of use of a virtual space, and a program.SOLUTION: An information processing system comprises: a control unit 310 which provides a first user with a first view which is a view of a virtual space; a control unit 510 which provides a second user with a second view which is a view of a composite real space where a virtual object is arranged in a real space; an MR-side input acceptance unit 816 which accepts an input of location information indicative of a current location of the second user in the real space; and a storage unit 611 which stores information indicative of a correspondence between points in the real space and points in the virtual space. The control unit 310 provides the first user with the first view on a prescribed event being performed in a specific location in the virtual space. When the location information indicates that the second user is present in a location corresponding to the specific location, the control unit 510 provides the second user with the second view reflected with the prescribed event being performed in the specific location in the virtual space according to the location information.SELECTED DRAWING: Figure 9

Description

The present invention relates to an information processing system and a program.

Technologies that provide users with mixed reality (MR) spaces and virtual reality (VR) spaces have been known (see, for example, Patent Documents 1 and 2).

Patent No. 6860488 Patent No. 6800599

However, there has long been a demand for expanding the range of uses for virtual space.

The present invention was made in consideration of the above circumstances, and aims to broaden the range of uses for virtual space.

According to one embodiment shown in the present disclosure,
a first control means for providing a first view, the first view being a view of a virtual space, to a first user;
a second control means for providing a second user with a second view, the second view being a view of a mixed reality space in which a virtual object is arranged in a real space;
an input receiving means for receiving position information indicating a current position of the second user in the real space;
a correspondence relationship storage means for storing information indicating a correspondence relationship between positions in the real space and the virtual space,
the first control means provides the first user with the first view of a predetermined event taking place at a specific position in the virtual space;
The second control means is
providing the second view to the second user according to the location information;
An information processing system is provided that, when the location information indicates that the second user is in a location corresponding to the specific location, provides the second user with the second view reflecting the specified event taking place at the specific location in the virtual space.

This invention can broaden the range of uses for virtual space.

FIG. 1 is a diagram illustrating a schematic configuration of an information processing system. FIG. 1 is a diagram showing a schematic configuration of a VR system. FIG. 1 is a diagram showing a schematic configuration of an MR system. FIG. 1 is a diagram conceptually illustrating a virtual space. 1 is a diagram showing a YZ cross section of a field of view in a virtual space as viewed from an X direction. 1 is a diagram showing an XZ cross section of a field of view in a virtual space as viewed from a Y direction. FIG. 2 is a diagram illustrating a schematic configuration of a controller. FIG. 2 is a diagram illustrating a mixed reality space and a virtual space. FIG. 2 is a diagram illustrating a functional configuration of the information processing system. 1A and 1B are diagrams illustrating a real object and a virtual object corresponding to the real object. 13 is a flowchart illustrating an example of a process executed by the VR system. 13 is a flowchart illustrating an example of a process executed by the MR system. 13 is a flowchart showing an example of a process for reflecting the behavior of an MR user in a virtual space. 13 is a flowchart illustrating an example of a process for reflecting the behavior of a VR user in a mixed reality space. 1A and 1B are diagrams for explaining how an event in a virtual space is reflected in a real space. 1A and 1B are diagrams for explaining how an event in a real space is reflected in a virtual space;

The following describes an embodiment of the present invention with reference to the drawings.

[First embodiment]
<Configuration of Information Processing System>
As shown in FIG. 1, the information processing system 100 of the present embodiment includes a VR (Virtual Reality) system 200, an MR (Mixed Reality) system 400, a server 600, and an external device 700. The VR system 200 is configured to be able to communicate with the server 600, the MR system 400, and the external device 700 via a network 2. The MR system 400 is configured to be able to communicate with the server 600, the VR system 200, and the external device 700 via the network 2. The number of MR systems 400 constituting the information processing system 100 is not limited to one, and may be multiple. The number of VR systems 200 constituting the information processing system 100 is not limited to one, and may be multiple. In this embodiment, communication between the VR system 200 and the MR system 400 via the network 2 is performed via the server 600, but may be performed without the server 600. The network 2 may be composed of, for example, the Internet, a mobile communication system (e.g., 3G, 4G, 5G, LTE (Long Term Evolution), etc.), Wi-Fi (registered trademark), Bluetooth (registered trademark), other communication lines, or a combination of these.

(VR system configuration)
2, the VR system 200 includes a VR device 210, a computer 300, a detection device 260, a display 270, and a controller 280. The VR device 210 includes a display 211, a gaze sensor 212, a first camera 213, a second camera 214, a microphone 215, a speaker 216, and a sensor 217. Hereinafter, a user who uses the VR device 210 is referred to as a VR user.

The computer 300 can be connected to the Internet or other network 2, and can communicate with, for example, a server 600, a computer of the MR system 400, and other computers connected to the network 2. Examples of other computers include computers of other VR systems 200 and external devices 700.

The VR device 210 is worn on the head of a VR user and can provide the VR user with a virtual space during operation. The VR device 210 can be, for example, a so-called head-mounted display equipped with a display, or a head-mounted device equipped with a smartphone or other terminal having a display. The VR device 210 displays, for example, an image for the right eye and an image for the left eye on the display 211. When each eye of the VR user views the respective image, the VR user can recognize the image as a three-dimensional image based on the parallax between the two eyes.

The display 211 is realized, for example, as a non-transparent display device. The display 211 is arranged on the main body of the VR device 210 so as to be located, for example, in front of both eyes of the VR user. Therefore, when the VR user visually recognizes the three-dimensional image displayed on the display 211, he or she can become immersed in the virtual space. Note that the display 211 may also be realized by a display provided on a so-called smartphone or other terminal.

The detection device 260 detects the movement of a VR user who uses the VR device 210. The detection device 260 may have a position tracking function for detecting the movement of the VR device 210, thereby detecting the movement of the VR user. Specifically, the detection device 260 may have, for example, a sensor that reads light (e.g., infrared light) from the VR device 210 as a sensor for detecting the movement of the VR user, and detect the position, inclination, etc. of the VR device 210 in real space. In this case, the VR device 210 may have multiple light sources not shown. Furthermore, each light source may be realized, for example, by an LED (Light Emitting Diode) that emits infrared light.

The detection device 260 may also be realized by, for example, a camera. Specifically, the detection device 260 may have an image sensor (for example, an image sensor that acquires an RGB image, an image sensor that acquires a black and white image, or a depth sensor, etc.) as a sensor for detecting the movement of the VR user. In other words, the movement of the VR user may be detected by a camera. The detection device 260 may be, for example, a depth camera equipped with a device that emits a predetermined light such as infrared light or a predetermined pattern of light, and an image sensor (for example, a depth sensor), and may detect the reflected light of the light emitted from the device by the image sensor, and may detect the position, posture, etc. of the VR user based on the output from the image sensor. The detection device 460 may also be equipped with such a depth camera and a camera that can acquire an RGB image, and may detect the position, posture, etc. of the VR user based on the output from these cameras. The detection device 260 may also be, for example, a stereo camera equipped with multiple image sensors, and may detect the position, posture, etc. of the VR user based on the output from the multiple image sensors. Note that detecting the position, posture, etc. of the VR user may be detecting the position, posture, etc. of the VR user's body, or may be detecting the position, tilt, etc. of the VR device 210.

The VR system 200 may have one or more types of detection devices as the detection device 260, and may have multiple detection devices of each type. Furthermore, the VR system 200 does not have to have the detection device 260. A part of the detection device 260 may be configured by the computer 300 or the like. For example, analysis of the output from the image sensor (e.g., image recognition, etc.) may be performed by the computer 300 or the like.

In addition to the various sensors described above, the detection device 260 may have a sensor capable of detecting the position, inclination, etc. of the detection device 260 itself. Specifically, for example, the detection device 260 may have an angular velocity sensor (e.g., a three-axis angular velocity sensor), an acceleration sensor (e.g., a three-axis acceleration sensor), or a geomagnetic sensor (e.g., a three-axis geomagnetic sensor). In addition, the output from these sensors may be sent to the computer 300, etc., and may be used, for example, when a predetermined process is performed based on the output from an image sensor provided in the detection device 260.

The VR device 210 may also include a sensor 217 instead of or in addition to the detection device 260 as a detection means for detecting the movement of the VR user. The VR device 210 may use the sensor 217 to detect the position and inclination of the VR device 210 itself. The sensor 217 may be, for example, an angular velocity sensor (e.g., a three-axis angular velocity sensor), an acceleration sensor (e.g., a three-axis acceleration sensor), or a geomagnetic sensor (e.g., a three-axis geomagnetic sensor). The VR device 210 may also have one or more types of sensors as the sensor 217, and may have multiple sensors of each type. As an example, when an angular velocity sensor is used as the sensor 217, the VR device 210 can detect the angular velocity around the three axes of the VR device 210 in real space over time. The VR device 210 can then calculate the change over time in the angles around the three axes of the VR device 210 based on each angular velocity, and can further calculate the inclination of the VR device 210 based on the change over time in the angles. The sensor 217 may be, for example, an image sensor. The position, posture, etc. of the VR user may be detected based on the output from the image sensor. In other words, the position, posture, etc. of the VR user may be detected based on information from a camera that is provided in the VR device 210 and captures the surroundings of the VR device 210. In other words, tracking of the VR device 210 may be performed by an outside-in method or an inside-out method.

The gaze sensor 212 detects the direction in which the gaze of the right and left eyes of the VR user is directed. That is, the gaze sensor 212 detects the gaze of the VR user (in other words, the movement of the eyes). The detection of the gaze direction is realized, for example, by a known eye tracking function. The gaze sensor 212 is realized by a sensor having the eye tracking function. The gaze sensor 212 may include a sensor for the right eye and a sensor for the left eye. The gaze sensor 212 may be, for example, a sensor that irradiates the right and left eyes of the VR user with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris in response to the irradiated light. In this case, the gaze sensor 212 can detect the gaze of the VR user based on each detected rotation angle.

The first camera 213 photographs the lower part of the VR user's face. More specifically, the first camera 213 photographs the nose, mouth, etc. of the VR user. The second camera 214 photographs the eyes, eyebrows, etc. of the VR user. Here, the VR user side of the housing of the VR device 210 is defined as the inside of the VR device 210, and the opposite side of the housing of the VR device 210 from the VR user is defined as the outside of the VR device 210. The first camera 213 may be disposed outside the VR device 210, and the second camera 214 may be disposed inside the VR device 210. The images photographed by the first camera 213 and the second camera 214 are input to the computer 300. Note that the first camera 213 and the second camera 214 may be realized as a single camera, and the face of the VR user may be photographed by this single camera.

The microphone 215, serving as a sound input means, converts the voice of the VR user into a sound signal (in other words, an electrical signal) and outputs it to the computer 300. The speaker 216, serving as a sound output means, converts the sound signal into sound and outputs it to the VR user. Note that the VR device 210 may include earphones instead of the speaker 216 as a sound output means.

The display 270 displays an image similar to the image displayed on the display 211. This allows users other than the VR user wearing the VR device 210 to view the same image (in other words, a virtual space) as the VR user. The image displayed on the display 270 does not need to be a three-dimensional image, and may be, for example, an image for the right eye or an image for the left eye displayed on the VR device 210. Examples of the display 270 include a liquid crystal display and an organic EL display.

The controller 280 is connected to the computer 300 by wire or wirelessly. The controller 280 accepts input operations related to instructions from the VR user to the computer 300. The controller 280 also accepts input operations by the VR user to control the position and movement of a virtual object placed in a virtual space. The controller 280 may be configured to be held by the VR user, for example. The controller 280 may also be configured to be wearable on the body or a part of the clothing of the VR user, for example. Specifically, the controller 280 may be, for example, a glove type controller. The controller 280 may also be configured to be able to output at least one of vibration, sound, and light based on a signal transmitted from the computer 300.

The controller 280 may also have multiple light sources. Each light source may be realized, for example, by an LED that emits infrared light. The detection device 260 may then read the infrared light from the controller 280 and detect the position and inclination of the controller 280 in real space. In other words, the detection device 260 may have a position tracking function that detects the movement of the controller 280, thereby detecting the movement of the VR user.

Note that the VR system 200 does not necessarily have to include the controller 280.

The information processing system 100 may also have a sensor 286 that detects the movement of the VR user. The sensor 286 may be, for example, an angular velocity sensor, an acceleration sensor, or a geomagnetic sensor. The sensor 286 may also be provided in the controller 280 (see FIG. 7). The information processing system 100 may have one or more types of sensors as the sensor 286, and may have multiple sensors of each type. The device including the sensor 286 (for example, the controller 280 or a specific camera) and the computer 300 may be connected to each other, for example, wirelessly. The information acquired by the sensor 286 may also be transmitted to the computer 300, for example, by wireless communication.

The position and movement of the VR user may also be detected by the VR device 210 being equipped with a device that performs a specific wireless communication (e.g., short-range wireless communication such as Wi-Fi communication, Bluetooth communication, or UWB (Ultra Wide Band) communication), and the device performing wireless communication with surrounding devices to obtain position information of the VR device 210.

The position and movement of the VR user may also be detected using an external device 700 (and a device that tracks the movement of the VR user in cooperation with the device via short-range wireless communication or the like) that can track the movement of the VR user, such as a device worn by the VR user (for example, a wearable device such as a watch-type, wristband-type, ring-type, or clothing-type device, or an implantable device, etc.). These external devices 700 may also be used as the controller 280. The position and movement of the VR user may also be detected by a GPS sensor or the like.

(Configuration of computer 300)
2, the computer 300 includes, as its main components, a processor 301, a memory 302, a storage 303, an input/output interface 304, and a communication interface 305. The components are connected to each other via a bus.

The processor 301 controls the operation of the VR device 210. The processor 301 reads a program from the storage 303 and loads it into the memory 302. The processor 301 executes the loaded program. The processor 301 may be configured to include, for example, one or more of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), and an FPGA (Field-Programmable Gate Array).

Memory 302 is a main storage device. Memory 302 is composed of storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). Memory 302 provides a working area for processor 301 by temporarily storing programs and various data that processor 301 reads from storage 303. Memory 302 also temporarily stores various data generated while processor 301 is operating according to a program, various data input to computer 300, etc.

Storage 303 is an auxiliary storage device. Storage 303 is composed of a storage device such as a flash memory or a hard disk drive (HDD). Storage 303 stores programs for providing services in information processing system 100. Storage 303 also stores various data for providing services in information processing system 100. The data stored in storage 303 includes data for defining a virtual space and data related to virtual objects.

The storage 303 may be implemented as a removable storage device such as a memory card. Also, instead of the storage 303 built into the computer 300, programs and data stored in an external storage device may be used.

The input/output interface 304 is an interface through which the computer 300 accepts data input, and is also an interface through which the computer 300 outputs data. The input/output interface 304 may transmit or receive data between the VR device 210, the detection device 260, and the display 270. It may also transmit or receive data between the display 211, the gaze sensor 212, the first camera 213, the second camera 214, the microphone 215, the speaker 216, and the sensor 217 included in the VR device 210.

The input/output interface 304 may transmit or receive data to and from the controller 280. For example, the input/output interface 304 may receive input of signals output from the controller 280 and the sensor 286. The input/output interface 304 may also transmit to the controller 280 an instruction output from the processor 301. The instruction may instruct the controller 280 to vibrate, output sound, emit light, or the like. Upon receiving the instruction, the controller 280 performs vibration, output sound, emit light, or the like in accordance with the instruction.

Note that each of the VR device 210, the detection device 260, the display 270, the controller 280, and the sensor 286 may be connected to the computer 300 via a wired or wireless connection.

The communication interface 305 controls the transmission and reception of various data with other computers (e.g., the server 600, the computer 500, or other computers 300, etc.) via the network 2.

The processor 301 accesses the storage 303, loads a program stored in the storage 303 into the memory 302, and executes a series of instructions contained in the program. The processor 301 also sends a signal to the VR device 210 via the input/output interface 304 to provide a virtual space.

The computer 300 may be provided outside the VR device 210, or a part or the whole of the computer 300 may be built into the VR device 210. Also, a portable terminal (e.g., a smartphone) including a display 211 may function as at least a part of the computer 300. Note that when there are multiple VR devices 210, the computer 300 may be provided for each VR device 210, or may be shared by the multiple VR devices 210.

(Coordinate System)
A real coordinate system, which is a coordinate system in real space, is set in advance in the VR system 200. The real coordinate system has three reference directions (axes) that are parallel to the vertical direction in real space, the horizontal direction perpendicular to the vertical direction, and the front-rear direction perpendicular to both the vertical and horizontal directions.

The position and inclination of the VR device 210 in real space can be detected by the detection device 260 and the sensor 217. The detected inclination of the VR device 210 corresponds to, for example, each of the inclinations around the three axes of the VR device 210 in the real coordinate system. The computer 300 sets a uvw field of view coordinate system for the VR device 210 based on the inclination of the VR device 210 in the real coordinate system (see FIG. 4). The uvw field of view coordinate system set for the VR device 210 corresponds to the viewpoint coordinate system when a VR user wearing the VR device 210 views an object in a virtual space.

Now, the uvw field of view coordinate system will be explained. The computer 300 sets a three-dimensional uvw field of view coordinate system with the head of the VR user wearing the VR device 210 as its center (origin). More specifically, the computer 300 tilts the horizontal, vertical, and front-to-rear directions that define the real coordinate system around each axis by the tilt of the VR device 210 around each axis in the real coordinate system, and sets the three newly obtained directions as the pitch axis (u-axis), yaw axis (v-axis), and roll axis (w-axis) of the uvw field of view coordinate system in the VR device 210.

For example, when a VR user wearing the VR device 210 stands upright and looks straight ahead, the processor 301 sets a field of view coordinate system for the VR device 210 that is parallel to the real coordinate system. In this case, the horizontal direction, vertical direction, and front-to-back directions in the real coordinate system coincide with the pitch axis (u-axis), yaw axis (v-axis), and roll axis (w-axis) of the field of view coordinate system in the VR device 210.

After the uvw field of view coordinate system is set in the VR device 210, the detection device 260 or the sensor 217 can detect the tilt of the VR device 210 in the set uvw field of view coordinate system based on the movement of the VR device 210. In this case, the detection device 260 or the sensor 217 detects the pitch angle, yaw angle, and roll angle of the VR device 210 in the uvw field of view coordinate system as the tilt of the VR device 210. The pitch angle represents the tilt angle of the VR device 210 around the pitch axis in the uvw field of view coordinate system. The yaw angle represents the tilt angle of the VR device 210 around the yaw axis in the uvw field of view coordinate system. The roll angle represents the tilt angle of the VR device 210 around the roll axis in the uvw field of view coordinate system.

Based on the detected inclination of the VR device 210, the computer 300 sets the uvw field of view coordinate system of the VR device 210 after the VR device 210 moves to the VR device 210. The relationship between the VR device 210 and the uvw field of view coordinate system of the VR device 210 is always constant, regardless of the position and inclination of the VR device 210. When the position and inclination of the VR device 210 change, the position and inclination of the uvw field of view coordinate system of the VR device 210 in the real coordinate system change in conjunction with the change in the position and inclination.

In addition, the detection device 260 may identify the position of the VR device 210 in real space as a relative position with respect to the detection device 260. Furthermore, the processor 301 may determine the origin of the uvw field of view coordinate system of the VR device 210 in real space (actual coordinate system) based on the identified relative position.

(Virtual space)
The virtual space will be further described with reference to FIG. 4. FIG. 4 is a diagram conceptually showing one mode of expressing the virtual space 11 according to an embodiment. The virtual space 11 has a spherical structure covering the entire 360-degree direction of the center 12. In FIG. 4, in order to avoid complicating the description, the upper half of the celestial sphere in the virtual space 11 is illustrated. Each mesh is defined in the virtual space 11. The position of each mesh is defined in advance as a coordinate value in the XYZ coordinate system, which is a global coordinate system defined in the virtual space 11. The computer 300 associates each partial image constituting the panoramic image 13 (still image, video, etc.) that can be deployed in the virtual space 11 with each corresponding mesh in the virtual space 11.

In the virtual space 11, an XYZ coordinate system is defined with a specific point as the origin 12. The XYZ coordinate system is, for example, parallel to the real coordinate system. The horizontal direction, vertical direction (up-down direction), and front-to-back direction in the XYZ coordinate system are defined as the X-axis, Y-axis, and Z-axis, respectively. Therefore, the X-axis of the XYZ coordinate system is parallel to the horizontal direction of the real coordinate system, the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the vertical direction of the real coordinate system, and the Z-axis (front-to-back direction) of the XYZ coordinate system is parallel to the front-to-back direction of the real coordinate system.

When the VR device 210 is started up, i.e., in the initial state of the VR device 210, the virtual camera 14 is placed at a predetermined position (e.g., the center) in the virtual space 11. The processor 301 also displays an image captured by the virtual camera 14 on the display 211 of the VR device 210. The virtual camera 14 moves within the virtual space 11 in conjunction with the movement of the VR device 210 in the real space. This allows changes in the tilt and position of the VR device 210 in the real space to be reproduced in the virtual space 11 in the same way.

A uvw field of view coordinate system is defined for the virtual camera 14, as in the case of the VR device 210. The uvw field of view coordinate system of the virtual camera 14 in the virtual space 11 is defined so as to be linked to the uvw field of view coordinate system of the VR device 210 in real space (actual coordinate system). Therefore, when the inclination of the VR device 210 changes, the inclination of the virtual camera 14 also changes accordingly. Note that the virtual camera 14 may move in the virtual space 11 in conjunction with the movement of the VR user in real space, but in this embodiment, it does not move in the virtual space 11 even if the VR user moves in real space.

The processor 301 of the computer 300 determines the viewing area 15 in the virtual space 11 based on the position and inclination of the virtual camera 14. The viewing area 15 corresponds to the area of the virtual space 11 that is viewed by a VR user wearing the VR device 210. In other words, the position of the virtual camera 14 can be said to be the viewpoint of the VR user in the virtual space 11.

The uvw field of view coordinate system of the VR device 210 is equal to the viewpoint coordinate system when the VR user views the display 211. The uvw field of view coordinate system of the virtual camera 14 is linked to the uvw field of view coordinate system of the VR device 210. Therefore, the line of sight of the VR user detected by the gaze sensor 212 can be regarded as the line of sight of the VR user in the uvw field of view coordinate system of the virtual camera 14.

(Visibility area)
The field of view 15 will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram showing a YZ cross section of the field of view 15 in the virtual space 11 as viewed from the X direction. Fig. 6 is a diagram showing an XZ cross section of the field of view 15 in the virtual space 11 as viewed from the Y direction.

As shown in FIG. 5, the field of view 15 in the YZ cross section includes an area 18. The area 18 is defined by the position of the virtual camera 14, the reference line of sight 16, and the YZ cross section of the virtual space 11. The processor 301 defines the range including the polar angle α centered on the reference line of sight 16 in the virtual space as the area 18.

As shown in FIG. 6, the field of view 15 in the XZ cross section includes area 19. Area 19 is defined by the position of virtual camera 14, reference line of sight 16, and the XZ cross section of virtual space 11. Processor 301 defines a range including azimuth angle β centered on reference line of sight 16 in virtual space 11 as area 19. Polar angles α and β are determined according to the position of virtual camera 14 and the inclination (direction) of virtual camera 14.

The VR system 200 provides the VR user with a field of view in the virtual space 11 by displaying a field of view image 17 on the display 211 based on a signal from the computer 300 (see FIG. 4). The field of view image 17 is an image that corresponds to the portion of the panoramic image 13 that corresponds to the field of view area 15. When the VR user moves the VR device 210 worn on the head, the virtual camera 14 also moves in conjunction with the movement. As a result, the position of the field of view area 15 in the virtual space 11 changes. As a result, the field of view image 17 displayed on the display 211 is updated to an image of the panoramic image 13 that is superimposed on the field of view area 15 in the direction in which the VR user is facing in the virtual space 11. The VR user can view the desired direction in the virtual space 11.

While wearing the VR device 210, the VR user can only view the panoramic image 13 displayed in the virtual space 11, without being able to see the real world. Therefore, the information processing system 100 can give the VR user a highly immersive feeling in the virtual space 11.

The virtual camera 14 may include two virtual cameras, i.e., a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. In this case, an appropriate parallax is set for the two virtual cameras so that the VR user can recognize the three-dimensional virtual space 11. The virtual camera 14 may also be realized by a single virtual camera. In this case, an image for the right eye and an image for the left eye may be generated from an image obtained by the single virtual camera.

(controller)
An example of the controller 280 will now be described with reference to FIG.

As shown in FIG. 7, the controller 280 may include a right controller 280R and a left controller (not shown). In this case, the right controller 280R is operated by the right hand of the VR user. The left controller is operated by the left hand of the VR user. In one aspect, the right controller 280R and the left controller are configured as separate devices. Therefore, the VR user can freely move both the right hand holding the right controller 280R and the left hand holding the left controller. In another aspect, the controller 280 may be an integrated controller that accepts operation from both hands. The right controller 280R will be described below.

The right controller 280R includes a grip 281, a frame 282, a top surface 283, buttons 284 and 285, a sensor (e.g., a motion sensor) 286, an infrared LED 287, buttons 288 and 289, and an analog stick 290. The grip 281 is configured to be held by the right hand of a VR user. For example, the grip 281 can be held by the palm and three fingers (middle finger, ring finger, and little finger) of the VR user's right hand.

Button 284 is disposed on the side of grip 281 and is operable by the middle finger of the right hand. Button 285 is disposed on the front of grip 281 and is operable by the index finger of the right hand. Buttons 284 and 285 are also equipped with switches that detect the movement of the user pressing buttons 284 and 285. Buttons 284 and 285 may also be configured as trigger-type buttons.

The sensor 286 is built into the housing of the grip 281. The sensor 286 detects the movement of the VR user. Specifically, the sensor 286 detects the movement of the VR user's hand. For example, the sensor 286 detects the rotation speed and number of rotations of the hand. Note that the controller 280 does not necessarily have to include the sensor 286.

A number of infrared LEDs 287 are arranged along the circumferential direction of the frame 282. When a program using the controller 280 is being executed, the infrared LEDs 287 emit infrared light in accordance with the progress of the program. The infrared light emitted from the infrared LEDs 287 can be used to detect the positions and attitudes (tilt, direction) of the right controller 280R and the left controller.

The top surface 283 is equipped with buttons 288, 289 and an analog stick 290. The buttons 288, 289 are operated by the thumb of the VR user's right hand. The buttons 288, 289 are also equipped with switches that detect the movement of the user pressing the buttons 288, 289. The analog stick 290 is equipped with operations in any direction within 360 degrees from the initial position (neutral position). Such operations include, for example, operations for moving an object placed in the virtual space 11. The analog stick 290 is also equipped with a sensor that detects the movement of the user operating the analog stick 290.

Information regarding the VR user's input operations (in other words, the VR user's movements) detected by the controller 280 is sent to the computer 300.

As shown in FIG. 7, the controller 280 defines the yaw, roll, and pitch directions for, for example, the right hand of a VR user. Specifically, for example, when a VR user extends his or her thumb and index finger, the direction in which the thumb extends is defined as the yaw direction, the direction in which the index finger extends is defined as the roll direction, and the direction perpendicular to the plane defined by the yaw direction axis and the roll direction axis is defined as the pitch direction.

(Configuration of MR System)
3, the MR system 400 includes an MR device 410, a computer 500, a detection device 460, a display 470, and a controller 480. The MR device 410 includes a display 411, a gaze sensor 412, a camera 413, a microphone 415, a speaker 416, and a sensor 417. Hereinafter, a user who uses the MR device 410 is referred to as an MR user 6.

The computer 500 can be connected to the Internet or other network 2, and can communicate with, for example, a server 600, a computer of the VR system 200, and other computers connected to the network 2. Examples of other computers include computers of other MR systems 400 and external devices 700.

The MR device 410 is worn on the head of the MR user 6 and can provide a mixed reality space to the MR user 6 during operation. The MR device 410 may be, for example, a glasses-type device equipped with a display (for example, so-called MR glasses). The MR device 410 may also be a contact lens-type device, etc.

The display 411 is realized as, for example, a transmissive display device. The transmissive display 411 may temporarily function as a non-transmissive display device by adjusting its transmittance. The display 411 is disposed on the main body of the MR device 410 so as to be located, for example, in front of both eyes of the MR user 6. Therefore, the MR user 6 is provided with a display in which a virtual object displayed by the display 411 is superimposed on the real space seen through the transmissive display 411. That is, the MR device 410 provides the MR user 6 with a view of a mixed reality space in which a virtual object is disposed in the real space. In other words, the MR user 6 can simultaneously see a virtual object and an object existing in the real space (hereinafter referred to as a "real object") by the MR device 410.

The detection device 460 detects the movement of the MR user 6 who uses the MR device 410. The detection device 460 may have a position tracking function for detecting the movement of the MR device 410, thereby detecting the movement of the MR user 6. Specifically, the detection device 460 may have, for example, a sensor that reads light (e.g., infrared light) from the MR device 410 as a sensor for detecting the movement of the MR user 6, and detect the position, inclination, etc. of the MR device 410 in the mixed reality space. In this case, the MR device 410 may have multiple light sources not shown. Furthermore, each light source may be realized, for example, by an LED (Light Emitting Diode) that emits infrared light.

The detection device 460 may also be realized by, for example, a camera. Specifically, the detection device 460 may have an image sensor (for example, an image sensor that acquires an RGB image, an image sensor that acquires a black and white image, or a depth sensor, etc.) as a sensor for detecting the movement of the MR user 6. In other words, the movement of the MR user 6 may be detected by a camera. The detection device 460 may be, for example, a depth camera equipped with a device that emits a predetermined light such as infrared light or a predetermined pattern of light, and an image sensor (for example, a depth sensor), and may detect the reflected light of the light emitted from the device by the image sensor, and may detect the position, posture, etc. of the MR user 6 based on the output from the image sensor. The detection device 460 may also be equipped with such a depth camera and a camera capable of acquiring an RGB image, and may detect the position, posture, etc. of the MR user 6 based on the output from these cameras. Furthermore, the detection device 460 may be, for example, a stereo camera equipped with multiple image sensors, and may detect the position, posture, etc. of the MR user 6 based on the output from the multiple image sensors. Note that detection of the position, posture, etc. of the MR user 6 may be detection of the body position, posture, etc. of the MR user 6, or detection of the position, tilt, etc. of the MR device 410.

The MR system 400 may have one or more types of detection devices as the detection device 460, and may have multiple detection devices of each type. For example, the MR system 400 may have multiple cameras (in other words, multiple sensors), and the multiple cameras may function as the detection device 460. In this embodiment, the MR system 400 is described as having three detection devices 460, and each detection device 460 is described as having a depth camera and a camera capable of acquiring RGB images. A part of the detection device 460 may be configured by the computer 500 or the like. For example, analysis of the output from the image sensor (e.g., image recognition, etc.) may be performed by the computer 500 or the like. The MR system 400 may not have the detection device 460.

In addition to the various sensors described above, the detection device 460 may have a sensor capable of detecting the position, inclination, etc. of the detection device 460 itself. Specifically, for example, the detection device 460 may have an angular velocity sensor (e.g., a three-axis angular velocity sensor), an acceleration sensor (e.g., a three-axis acceleration sensor), or a geomagnetic sensor (e.g., a three-axis geomagnetic sensor). In addition, the output from these sensors may be sent to the computer 500 or the like, and may be used, for example, when a predetermined process is performed based on the output from an image sensor provided in the detection device 460.

Also, as will be understood from the configuration of the MR device 410 described below, the MR device 410 of another MR user 6 may be used as the detection device 460.

The MR device 410 may also include a sensor 417 instead of or in addition to the detection device 460 as a detection means for detecting the movement of the MR user 6. The MR device 410 may use the sensor 417 to detect the position and inclination of the MR device 410 itself. The sensor 417 may be, for example, an angular velocity sensor (e.g., a three-axis angular velocity sensor), an acceleration sensor (e.g., a three-axis acceleration sensor), or a geomagnetic sensor (e.g., a three-axis geomagnetic sensor). The MR device 410 may also include one or more types of sensors as the sensor 417, and may include multiple sensors of each type.

The gaze sensor 412 detects the direction in which the right and left eyes of the MR user 6 are directed. That is, the gaze sensor 412 detects the gaze (in other words, the movement of the eyes) of the MR user 6. The detection of the gaze direction is realized, for example, by a known eye tracking function. The gaze sensor 412 is realized by a sensor having the eye tracking function. The gaze sensor 412 may include a sensor for the right eye and a sensor for the left eye. The gaze sensor 412 may be, for example, a sensor that irradiates the right and left eyes of the MR user 6 with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris in response to the irradiated light. In this case, the gaze sensor 412 can detect the gaze of the MR user 6 based on each detected rotation angle.

The camera 413 captures the surroundings (e.g., in front) of the MR user 6. The MR device 410 may include, for example, a depth camera and a camera capable of acquiring RGB images as the camera 413. Based on the output from the camera 413, the computer 500 may detect the shape of objects around the MR device 410 (in other words, the MR user 6) and the relative distance between the MR device 410 and the surrounding objects. The position and inclination of the MR device 410 may also be detected based on the output from the camera 413. In other words, tracking of the MR device 410 may be performed by an outside-in method or an inside-out method.

The microphone 415 as a sound input means converts the voice of the MR user 6 into a sound signal (in other words, an electrical signal) and outputs it to the computer 500. The speaker 416 as a sound output means converts the sound signal into sound and outputs it to the MR user 6. Note that the MR device 410 may include earphones as a sound output means instead of the speaker 416.

The display 470 displays, for example, an image similar to the image displayed on the display 411. This allows users other than the MR user 6 wearing the MR device 410 to view an image similar to that of the MR user 6. The image displayed on the display 470 does not need to be a three-dimensional image, and may be, for example, an image for the right eye or an image for the left eye displayed by the MR device 410. Examples of the display 470 include a liquid crystal display and an organic EL display. The display 470 may display an image showing a view of the mixed reality space provided to the MR user 6 by the MR device 410.

The controller 480 is connected to the computer 500 by wire or wirelessly. The controller 480 accepts input operations related to instructions from the MR user 6 to the computer 500. The controller 480 also accepts input operations by the MR user 6 to control the position and movement of a virtual object placed in the mixed reality space. The controller 480 may be configured to be held by the MR user 6, for example. The controller 480 may also be configured to be worn on a part of the body or clothing of the MR user 6, for example. The controller 480 may also be configured to be able to output at least one of vibration, sound, and light based on a signal transmitted from the computer 500.

The controller 480 may also have multiple light sources. Each light source may be realized, for example, by an LED that emits infrared light. The detection device 460 may then read the infrared light from the controller 480 and detect the position and inclination of the controller 480 within the mixed reality space. In other words, the detection device 460 may have a position tracking function that detects the movement of the controller 480, thereby detecting the movement of the MR user 6.

The MR system 400 does not have to have the controller 480. The controller 480 may have, for example, some or all of the components of the controller 280.

The information processing system 100 may have a sensor 486 that detects the movement of the MR user 6. The sensor 486 may be, for example, an angular velocity sensor, an acceleration sensor, or a geomagnetic sensor. The sensor 486 may also be provided in the controller 480, for example. The information processing system 100 may have one or more types of sensors as the sensor 486, and may have multiple sensors of each type. The device including the sensor 486 (for example, the controller 480 or a specific camera) and the computer 500 may be connected to each other, for example, wirelessly. The information acquired by the sensor 486 may be transmitted to the computer 300, for example, by wireless communication.

The position and movement of the MR user 6 may also be detected by the MR device 410 being equipped with a device that performs predetermined wireless communication (e.g., short-range wireless communication such as Wi-Fi communication, Bluetooth communication, or UWB communication) and having the device perform wireless communication with surrounding devices to obtain position information of the VR device 210.

The position and movement of the MR user 6 may also be detected using an external device 700 (and a device that tracks the movement of the MR user 6 in cooperation with the device via short-range wireless communication or the like) that can track the movement of the MR user 6, such as a device worn by the MR user 6 (for example, a wearable device such as a watch type, wristband type, ring type, or clothing type, or an implantable device, etc.). These external devices 700 may also be used as the controller 480. The position and movement of the MR user 6 may also be detected by a GPS sensor or the like.

(Configuration of computer 500)
3, the computer 500 includes, as its main components, a processor 501, a memory 502, a storage 503, an input/output interface 504, and a communication interface 505. The components are connected to each other via a bus.

The processor 501 controls the operation of the MR device 410. The processor 501 reads a program from the storage 503 and loads it in the memory 502. The processor 501 executes the loaded program. The processor 501 may be configured to include, for example, one or more of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), and an FPGA (Field-Programmable Gate Array).

Memory 502 is a main storage device. Memory 502 is composed of storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). Memory 502 provides a working area for processor 501 by temporarily storing programs and various data that processor 501 reads from storage 503. Memory 502 also temporarily stores various data generated while processor 501 is operating according to a program, various data input to computer 500, etc.

Storage 503 is an auxiliary storage device. Storage 503 is composed of a storage device such as a flash memory or a hard disk drive (HDD). Storage 503 stores programs for providing services in information processing system 100. Storage 503 also stores various data for providing services in information processing system 100. Data stored in storage 503 includes data for defining a mixed reality space and data related to virtual objects.

The storage 503 may be implemented as a removable storage device such as a memory card. Also, instead of the storage 503 built into the computer 500, programs and data stored in an external storage device may be used.

The input/output interface 504 is an interface through which the computer 500 accepts data input, and is also an interface through which the computer 500 outputs data. The input/output interface 504 may transmit or receive data between the MR device 410, the detection device 460, and the display 470. The input/output interface 504 may also transmit or receive data between the display 411, the gaze sensor 412, the camera 413, the microphone 415, the speaker 416, and the sensor 417 included in the MR device 410.

The input/output interface 504 may transmit or receive data to and from the controller 480. For example, the input/output interface 504 may receive input of signals output from the controller 480 and the sensor 486. The input/output interface 504 may also transmit to the controller 480 an instruction output from the processor 501. The instruction may instruct the controller 480 to vibrate, output sound, emit light, or the like. Upon receiving the instruction, the controller 480 performs vibration, output sound, emit light, or the like in accordance with the instruction.

Note that each of the MR device 410, the detection device 460, the display 470, the controller 480, and the sensor 486 may be connected to the computer 500 via a wired or wireless connection.

The communication interface 505 controls the transmission and reception of various data with other computers (e.g., the server 600, the computer 300, or other computers 500, etc.) via the network 2.

The processor 501 accesses the storage 503, loads a program stored in the storage 503 into the memory 502, and executes a series of instructions contained in the program. The processor 501 also sends a signal to the MR device 410 via the input/output interface 504 to provide a mixed reality space.

The computer 500 may be provided outside the MR device 410, or a part or the whole of the computer 500 may be built into the MR device 410. Furthermore, when there are multiple MR devices 410, the computer 500 may be provided for each MR device 410, or may be used commonly for the multiple MR devices 410. Furthermore, in this case, a part of the computer 500 may be built into each of the multiple MR devices 410. In this embodiment, an example will be described in which multiple MR devices 410 are connected to the computer 500.

Methods for acquiring the position and inclination of the MR device 410 in real space (in other words, mixed reality space) are well known, and can be detected in a similar manner to the method for the VR device 210 described above, so a detailed explanation is omitted.

(Server configuration)
The server 600 may transmit a program to the computer 300. The server 600 may also transmit a program to the computer 500. The server 600 also enables communication between the computer 300 and the computer 500. The server 600 also enables communication between the computer 300 and other computers 300. For example, when the information processing system 100 provides a service in which multiple VR users can participate, each computer 300 of the VR user may communicate with the other computers 300 via the server 600, thereby enabling multiple VR users to share experiences in the same virtual space. Note that each computer 300 may communicate with the other computers 300 without going through the server 600.

As shown in FIG. 1, the server 600 includes a processor 601, a memory 602, a storage 603, an input/output interface 604, and a communication interface 605. Each component is connected to each other via a bus.

The processor 601 controls the operation of the entire server 600. The processor 601 reads a program from the storage 603 and loads it into the memory 602. The processor 601 executes the loaded program. The processor 601 may be configured to include, for example, one or more of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), and an FPGA (Field-Programmable Gate Array).

Memory 602 is a main storage device. Memory 602 is composed of storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). Memory 602 provides a working area for processor 601 by temporarily storing programs and various data that processor 601 reads from storage 603. Memory 602 also temporarily stores various data generated while processor 601 is operating according to a program, various data input to server 600, etc.

Storage 603 is an auxiliary storage device. Storage 603 is composed of a storage device such as a flash memory or a HDD (Hard Disk Drive). Storage 603 stores programs for providing services in information processing system 100. Storage 603 also stores various data for providing services in information processing system 100. Data stored in storage 603 includes data for defining a virtual space, data for defining a mixed reality space, data related to virtual objects, and the like.

The storage 603 may be implemented as a removable storage device such as a memory card. Also, instead of the storage 603 built into the server 600, programs and data stored in an external storage device may be used.

The input/output interface 604 is an interface through which the server 600 accepts data input, and is also an interface through which the server 600 outputs data. The input/output interface 604 can transmit or receive data between, for example, input devices such as a mouse or keyboard, and output devices such as a display.

The communication interface 605 controls the sending and receiving of various data with other computers (e.g., computer 300, computer 500, etc.) via network 2.

The external device 700 may be any device capable of communicating with the computer 300, the computer 500, or the server 600. The external device 700 may be, for example, a device capable of communicating with the computer 300 via the network 2, or a device capable of communicating with the computer 300 via short-range wireless communication or a wired connection. The external device 700 may be, for example, a device capable of communicating with the computer 500 via the network 2, or a device capable of communicating with the computer 500 via short-range wireless communication or a wired connection. The external device 700 may be, for example, a device capable of communicating with the server 600 via the network 2. Examples of the external device 700 include, but are not limited to, smart devices, PCs (Personal Computers), and peripheral devices of the computers 300 and 500.

<Control of Information Processing System>
In the MR system 400 of this embodiment, the MR device 410 provides the MR user 6 with a view of a mixed reality space in which a virtual object is arranged in a real space, similar to a known MR device. In the VR system 200 of this embodiment, the VR device 210 provides the VR user with a view of a virtual space, similar to a known VR device. In the MR system 400 of this embodiment, the MR user 6 is capable of operating a virtual object in the mixed reality space. In the VR system 200 of this embodiment, the VR user is capable of operating a virtual object in the virtual space. In addition, in the information processing system 100 of this embodiment, the movement of the VR user using the VR system 200 is reflected in a virtual object included in the view of the mixed reality space provided by the MR system 400. In the information processing system 100 of this embodiment, the movement of the MR user 6 using the MR system 400 is reflected in a virtual object included in the view of the virtual space provided by the VR system 200. In this embodiment, the motion of the VR user and the like are reflected in the virtual object in the virtual space and the mixed reality space in real time. In addition, in this embodiment, the motion of the MR user 6 and the like are reflected in the virtual object in the virtual space and the mixed reality space in real time.

In the following, the configuration according to the present embodiment is applied to a service (in other words, an application) that enables communication between an MR user 6 and a VR user. Specifically, the configuration according to the present embodiment is applied to a service that enables an MR user 6 in a conference room as a specific real space and a VR user in a location away from the conference room to hold a conference while simultaneously viewing an object such as a product mockup as a virtual object. The configuration according to the present embodiment can also be applied to a system that enables a student to take school classes from home. The configuration according to the present embodiment can also be applied to a system that enables a student to participate in a home party held at a specific home from another location. The application of the configuration according to the present embodiment is not limited to these services. In the following, the configuration is described assuming that a plurality of detection devices 460 are installed in advance at predetermined positions in the conference room as shown in FIG. 8. In the following, the configuration is described assuming that three MR users 6, MR user 6A, MR user 6B, and MR user 6C, exist in the conference room. In the following, the avatar object of the VR user is called a VR avatar 25. Additionally, the avatar object of MR user 6 is called MR avatar 26. Specifically, the avatar object of MR user 6A is called MR avatar 26A, the avatar object of MR user 6B is called MR avatar 26B, and the avatar object of MR user 6C is called MR avatar 26C.

With reference to FIG. 8, the field of view provided to the MR users 6A, 6B, and 6C in the real conference room 21 (in other words, the mixed reality space 21) and the field of view provided to the VR user who enters the virtual space 11 simulating the conference room 21 through the VR device 210 will be described. FIG. 8 is a schematic diagram showing the state of the real conference room 21 and the state of the virtual space 11 when the MR users 6A, 6B, and 6C and the VR user are using the service. The upper part of FIG. 8 shows the state of the real conference room 21, and the lower part of FIG. 8 shows the state of the virtual space 11. Note that in FIG. 8, the VR avatar 25 is wearing the VR device 210, but this is for ease of explanation, and in reality, the VR avatar 25 does not need to wear the VR device 210 (virtual VR device 210). Also, in FIG. 8, the MR avatar 26 is not wearing an MR device 410 (virtual MR device 410), but the MR avatar 26 may be wearing an MR device 410.

As shown in FIG. 8, a desk 8 and the like are arranged in the conference room 21 as real objects. Three detection devices 460 are also arranged in the conference room 21. Three MR users 6A, 6B, and 6C are also present in the conference room 21. On the other hand, no VR users are present in the conference room 21. Here, a VR avatar 25 and a mock-up virtual object 30 (hereinafter also referred to as a "virtual model 30") are displayed as virtual objects on the display 411 of the MR device 410 used by each of the MR users 6A, 6B, and 6C. That is, the MR device 410 displays the VR avatar 25 and the virtual model 30 in the real conference room 21 visible through the transparent display 411. Therefore, the MR users 6A, 6B, and 6C are provided with a view as if the VR avatar 25 and the virtual model 30, which do not actually exist in the conference room 21, exist in the conference room 21.

The virtual space 11 that the VR user sees through the VR device 210 is modeled after the conference room 21, and includes a virtual object 31 of a desk 8 that exists in the real conference room 21. The virtual space 11 also contains MR avatars 26A, 26B, 26C of the MR users 6A, 6B, 6C, a virtual model 30, and the like. This provides the VR user with a view as if they were in the conference room 21 where the MR users 6A, 6B, 6C are located.

In this way, in the information processing system 100 of this embodiment, the MR user 6 can see other MR users 6 and real objects that are actually present in the same place through the transparent display 411. In addition, from the viewpoint of the MR user 6, the VR avatar 25 of the VR user that appears as a virtual object can be seen. Therefore, the MR user 6 can communicate with the VR user and other MR users 6 without losing the sense of being in the real world. In addition, the VR user is in a different place from the MR user 6 in the real world, and in such a case, the shape of the room where the VR user is and the place where the MR user 6 is often different. Therefore, if the VR user also uses the MR device 410 or the like to display the MR avatar 26 in the real space, there is a risk that the display will be unnatural. In this embodiment, the VR user can enter the virtual space 11 and communicate with the MR user 6, so that such an unnatural feeling can be prevented. In addition, because the virtual space 11 provided to the VR user mimics the location of the MR user 6, when the VR user or MR user 6 moves around or moves a virtual object in the virtual space 11 or real space, respectively, the movements of the avatar or virtual model 30 displayed to the other party become natural. In addition, when the VR user or MR user 6 operates a virtual object such as the virtual model 30, it becomes easy for them to imagine how it will look to the other party, enabling smooth communication.

The functional configuration of the information processing system 100 will be described below with reference to FIG. 9. Note that the functional configuration shown in FIG. 9 is merely an example. Each of the VR system 200, the MR system 400, and the server 600 may have at least some of the functions of the other devices. In other words, some or all of the functional blocks of the computer 300, the computer 500, and the server 600 in this embodiment may be provided by the computer 300, the computer 500, the server 600, or other devices. In addition, each device such as the computer 300, the computer 500, and the server 600 does not have to be realized by an integrated device, and may be realized by, for example, multiple devices connected via a network or the like.

In the present embodiment, the processor 301, the processor 501, or the processor 601 executes a program stored in the information processing system 100 to perform each process described below. However, at least a part of the process described below and performed by the processor 301 may be executed by a processor other than the processor 301. At least a part of the process described below and performed by the processor 501 may be executed by a processor other than the processor 501. At least a part of the process described below and performed by the processor 601 may be executed by a processor other than the processor 601. In other words, the computer that executes the program in this embodiment may be any computer including the computer 300, the computer 500, and the server 600, or may be realized by a combination of multiple devices.

FIG. 9 is a block diagram showing the functional configuration of the information processing system 100. As shown in FIG. 9, the computer 300 (in other words, the VR system 200) functions as a control unit 310 and a storage unit 311 through cooperation between the processor 301, memory 302, storage 303, input/output interface 304, and communication interface 305. The computer 500 (in other words, the MR system 400) functions as a control unit 510 and a storage unit 511 through cooperation between the processor 501, memory 502, storage 503, input/output interface 504, and communication interface 505. The server 600 functions as a control unit 610 and a storage unit 611 through cooperation between the processor 601, memory 602, storage 603, input/output interface 604, and communication interface 605.

The control unit 510 of the MR system 400 includes a virtual space generation unit 810, a coordinate definition unit 812, an MR side input reception unit 816, a user information acquisition unit 818, an MR side object control unit 820, a display control unit 840, a sound control unit 845, and a communication control unit 850.

The virtual space generating unit 810 generates a virtual space (in other words, virtual space data representing a virtual space) based on a predetermined real space. In other words, the virtual space generating unit 810 generates a virtual space that imitates a predetermined real space. Here, the predetermined real space is a predetermined place that exists in the real world, and may be, for example, a specific room such as a conference room or a school classroom. The predetermined real space may also be, for example, a school or a specific building such as a house. The predetermined real space may also be, for example, a specific town. In other words, the predetermined real space does not have to be a space separated by walls or the like. Furthermore, simulating a predetermined real space means that the generated virtual space has a structure similar to the basic structure of the real space. Specifically, for example, a virtual space that imitates a real-world conference room can be one in which the shape of the virtual room as a virtual space (for example, the shape of the walls, floor, etc.) is approximately the same as the shape of the conference room in the real world, and virtual objects corresponding to real objects such as desks and chairs that exist in the conference room in the real world are arranged. In this case, virtual objects such as desks, chairs, walls, and floors do not need to be exact copies of the shapes of real objects, and may be simplified in shape, pattern, color, etc. Also, real objects such as posters on the wall or small items placed on a desk or floor may exist that do not appear in the virtual world (real objects that do not have corresponding virtual objects).

In this embodiment, as described above, the specified real space is a specific conference room 21, and a virtual space 11 that mimics this specific conference room 21 is generated.

A known method can be used to generate a virtual space that imitates a real space, and is not particularly limited, but may be, for example, as follows. That is, the virtual space generation unit 810 generates the virtual space 11 based on information from a sensor that can detect the shape and position of real objects such as walls, floors, desks, and chairs that constitute the real space. Specifically, the virtual space generation unit 810 may obtain three-dimensional information of the real space based on information from an image sensor (e.g., an image sensor that obtains RGB images, an image sensor that obtains black and white images, or a depth sensor) provided in the detection device 460 or the MR device 410, and generate the virtual space. In addition, the virtual space generation unit 810 may use information from, for example, an acceleration sensor, an angular velocity sensor, or a geomagnetic sensor to generate the virtual space 11.

The virtual space generation unit 810 stores data representing the generated virtual space 11 (hereinafter referred to as "virtual space data"), for example, in the storage unit 611, which serves as a virtual space data storage unit.

The virtual space generation unit 810 may generate virtual space data representing the virtual space 11 in advance before the VR user or MR user 6 starts using the service. Also, the virtual space generation unit 810 may generate virtual space data representing the virtual space 11 in real time, for example, while the VR user or MR user 6 is using the service.

The coordinate definition unit 812 defines the coordinates of the real space (in other words, the coordinates of the mixed reality space 21, hereinafter referred to as "mixed reality coordinates"). The coordinate definition unit 812 also defines the coordinates of the virtual space 11 (hereinafter referred to as "virtual space coordinates").

The mixed reality coordinates have three mutually orthogonal axes, for example, the x-axis, y-axis, and z-axis. The virtual space coordinates have three mutually orthogonal axes, for example, the x-axis, y-axis, and z-axis. The coordinate definition unit 812 defines (in other words, associates) each coordinate for each point of the virtual space 11 (in other words, the virtual conference room) that corresponds to (in other words, indicates the same point) each point of the real space (in other words, the real conference room 21) so that the mixed reality coordinates indicating each point of the real space and the virtual space coordinates indicating each point of the virtual space correspond one-to-one. For example, when the virtual space generation unit 810 generates a virtual space that imitates the real space, the coordinate definition unit 812 defines the virtual space coordinates associated with the mixed reality coordinates. At this time, the coordinate definition unit 812 may associate the mixed reality coordinates with the virtual space coordinates based on, for example, a feature point of the conference room detected by the detection device 460 or the MR device 410. Furthermore, the coordinate definition unit 812 may associate the mixed reality coordinates with the virtual space coordinates, for example, based on the position of the detection device 460 or the position of a marker or the like that has been installed in advance in the conference room.

In addition, the coordinate definition unit 812 stores information regarding the correspondence between mixed reality coordinates and virtual space coordinates (in other words, the correspondence between each point in the real space and each point in the virtual space) in the memory unit 611, which functions as a correspondence storage unit.

The method for defining mixed reality coordinates and virtual space coordinates can be any known method, and is not particularly limited.

The timing for defining the mixed reality coordinates and the virtual space coordinates (in other words, the timing for associating the mixed reality coordinates with the virtual space coordinates) may be, for example, when the virtual space generating unit 810 generates a virtual space that imitates the real space, or when the MR device 410 starts displaying a virtual object.

The generation of the virtual space and the definition of each coordinate may be performed in advance, for example, by a dedicated device serving as an external device 700 capable of scanning the real space with high accuracy, and the generated data may be stored in the storage unit 611.

The MR side input receiving unit 816 receives input from the MR user 6. In other words, the MR side input receiving unit 816 acquires input information from the MR user 6. Specifically, the MR side input receiving unit 816 receives the output of the detection device 460, the controller 480, the camera 413, the device worn by the MR user, the gaze sensor 412, the sensor 417, the sensor 486, and the image sensor provided in the detection device 460 and the MR device 410 as input from the MR user 6. In other words, the MR side input receiving unit 816 receives the output of the detection means (e.g., the detection device 460, the controller 480, the camera 413, the gaze sensor 412, the sensor 417, the sensor 486, and the image sensor provided in the detection device 460 and the MR device 410) that detects the movement of the MR user 6 as input from the MR user 6. As described below, these devices and sensors are intended to obtain information about the movements of the MR user 6, which is used to control virtual objects in the virtual space 11 and the mixed reality space 21, and the MR user 6 can input information to the MR system 400 to move the virtual objects, etc., by his or her own movements.

Specifically, the MR side input receiving unit 816 may receive, for example, an output from the camera or camera 413 serving as the detection device 460 as an input from the MR user 6. More specifically, the MR side input receiving unit 816 may receive, for example, position information indicating the position of the MR user 6 in the mixed reality space 21, detected by image recognition from an image captured by an image sensor provided in each of these cameras, as an input from the MR user 6. In other words, the MR side input receiving unit 816 may receive, for example, an output from a position detection means that detects the position of the MR user 6 in the mixed reality space 21 as an input from the MR user 6. The position detection means may detect the position of the MR user 6 and obtain the position information based on, for example, wireless communication (e.g., Wi-Fi communication, Bluetooth communication, or UWB communication) between the MR device 410 and a specific device of the MR system 400 (e.g., the detection device 460 or a beacon (not shown)), or positioning using a GPS (Global Positioning System) sensor provided in the MR device 410. The position information indicating the position of the MR user 6 can also be considered information regarding the movement of the MR user 6.

The MR side input receiving unit 816 may also receive, as input from the MR user 6, information regarding the movement of the MR user 6 detected by image recognition from an image captured by a camera serving as the detection device 460 or an image sensor provided in the camera 413. For example, the MR side input receiving unit 816 may receive, as input from the MR user 6, information regarding the hand movement of the MR user 6 detected by image recognition, or information regarding the inclination or facing direction of the MR user 6.

The MR side input receiving unit 816 may also receive, as input from the MR user 6, information regarding the movements of the MR user 6 obtained using, for example, a device worn by the MR user 6 (for example, a wearable device such as a watch-type, wristband-type, ring-type, or clothing-type device, or an implantable device) capable of tracking the movements of the MR user 6 (and a device that tracks the movements of the MR user 6 in cooperation with the device via short-range wireless communication, etc.).

The MR side input receiving unit 816 may also receive, for example, information indicating the operation of the MR user 6 detected by the controller 480 (in other words, information regarding the movement of the MR user 6) as input from the MR user 6.

The MR side input receiving unit 816 may also receive output data (in other words, information regarding the movement of the MR user 6) regarding the inclination or direction of the MR device 410 from a sensor (e.g., an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, etc.) provided in the detection device 460 or the MR device 410 as input from the MR user 6.

The user information acquisition unit 818 acquires information about the MR user 6 present in the mixed reality space 21. Specifically, for example, the user information acquisition unit 818 acquires identification information that enables each MR user 6A, 6B, and 6C in the mixed reality space 21 to be identified (in other words, identification information that enables each MR device 410 to be identified). For example, the user information acquisition unit 818 may communicate with the MR device 410 worn by each user and acquire information about the user using the MR device 410 from the MR device 410 (for example, information about an account logged into the MR device 410, etc.) as the identification information. Also, for example, the user information acquisition unit 818 may communicate with the MR device 410 worn by each user and acquire information specific to each MR device 410 that enables each MR device 410 to be identified from the MR device 410 as the identification information. Also, for example, the user information acquisition unit 818 may acquire the identification information by identifying the MR user 6 in the mixed reality space 21 through image recognition from an image captured by an image sensor provided in the detection device 460 or the MR device 410. The user information acquisition unit 818 also associates the information acquired by the MR side input acceptance unit 816 with the identification information. In other words, the user information acquisition unit 818 performs control so that it is possible to determine which MR user 6 the information belongs to when information about the movement of the MR user 6 is used in controlling a virtual object, which will be described later.

The MR side object control unit 820 controls virtual objects in the mixed reality space 21.

The MR-side object control unit 820 places virtual objects in the mixed reality space 21. Specifically, the MR-side object control unit 820 places, for example, the VR avatar 25 of the VR user, and objects that are given predetermined changes by at least one of the VR user and the MR user 6. In this embodiment, as shown in FIG. 8, the MR-side object control unit 820 places the VR avatar 25 and a mock-up virtual object 30 (hereinafter also referred to as the "virtual model 30") that can be moved by the VR user and the MR user 6 in the mixed reality space 21.

The MR-side object control unit 820 places a virtual object corresponding to the virtual object at a position in the mixed reality space 21 corresponding to the position where the virtual object in the virtual space 11 is placed, for example. Here, the virtual object corresponding to the virtual object, i.e., the virtual object in the mixed reality space 21 corresponding to the virtual object in the virtual space 11, may have the same form as the virtual object in the virtual space 11 (in other words, it represents the same object), or may have a different form. In this embodiment, a virtual model 30 having the same form as the virtual model 30 in the virtual space 11 is placed in the mixed reality space 21. Also, a VR avatar 25 having the same form as the VR avatar 25 in the virtual space 11 is placed in the mixed reality space 21.

The placement of the VR avatar 25 in the mixed reality space 21 can also be described as follows. That is, the MR-side object control unit 820 can also be said to place the VR avatar 25 at a position in the mixed reality space 21 corresponding to the position of the VR user in the virtual space 11. Here, the position of the VR user in the virtual space 11 is, for example, the position of the VR avatar 25 in the virtual space 11. Note that in content in which the VR user enters the virtual space 11 from a first-person perspective, there may be cases in which the VR user (the VR avatar 25) cannot be seen (specifically, only a part of the user's body, such as the hands, or even a part of the user cannot be seen), but even in such cases, the VR user in the virtual space 11 recognized by the computer 300 can be said to be the VR avatar 25, and the position of the VR user in the virtual space 11 recognized by the computer 300 can be said to be the position of the VR avatar 25 in the virtual space.

The appearance (in other words, the form) of the VR avatar 25 may imitate the appearance of the VR user, but it does not have to imitate the appearance. For example, the appearance of the VR avatar 25 may imitate the appearance of a specific animal. The VR avatar 25 placed in the virtual space 11 and the VR avatar 25 placed in the mixed reality space 21 may have different appearances. With this configuration, the VR user can display his/her avatar in the mixed reality space 21 while hiding the form of his/her avatar in the virtual space 11. The appearance of the VR avatar 25 placed in the virtual space 11 may be an appearance that is visible to the VR user operating the VR avatar 25, or an appearance that is visible to other VR users operating other VR avatars 25 in the virtual space 11 in which the VR avatar 25 exists.

The MR-side object control unit 820 may also place a virtual object in the mixed reality space 21, for example, as follows:

That is, for example, when placing the VR avatar 25, the MR-side object control unit 820 determines where in the mixed reality space 21 the VR avatar 25 should be placed based on the position information indicating the position of the VR user (in other words, the VR avatar 25) in the virtual space 11. Specifically, for example, when the VR-side object control unit 920 described later places the VR avatar 25 in the virtual space 11, it transmits information regarding the position of the VR avatar 25 in the virtual space 11 (for example, the coordinate value of the virtual space coordinates) to the MR-side object control unit 820 via the server 600. Then, based on the information, the MR-side object control unit 820 places the VR avatar 25 at a position in the mixed reality space 21 corresponding to the position of the VR user in the virtual space 11 (for example, the position of the coordinate value of the mixed reality coordinates corresponding to the coordinate value of the virtual space coordinates). As a result, the VR avatar 25 is placed at the same position in the conference room in the virtual space 11 and the mixed reality space 21.

In other words, when placing a virtual object (e.g., a VR avatar 25 or a virtual model 30, etc.) in the mixed reality space 21, the MR side object control unit 820 may determine the position in the mixed reality space 21 to place the virtual object based on position information (e.g., position information transmitted by the VR side object control unit 920) indicating the position of the virtual object (e.g., a VR avatar 25 or a virtual model 30, etc.) in the virtual space corresponding to the virtual object.

Also, for example, when placing a virtual object in the mixed reality space 21, the MR-side object control unit 820 may determine at what position in the mixed reality space 21 to place the virtual object based on information detected by a predetermined sensor of the MR system 400. Specifically, for example, a predetermined marker may be set in advance in the real space, and the MR-side object control unit 820 may place the virtual model 30 as a virtual object at the position of the marker photographed by the image sensor provided in the detection device 460 or the MR device 410. Also, the MR-side object control unit 820 may place the virtual model 30 as a virtual object at a position indicated by the MR user 6. The instruction may be made by a predetermined gesture (for example, a gesture of pointing at the point where the virtual model 30 is to be placed), or may be to place the marker, etc. In addition, the MR-side object control unit 820 may detect a flat surface in the mixed reality space 21 from an image captured by an image sensor provided in the detection device 460 or the MR device 410, and place the virtual model 30 on the flat surface.

When arranging a virtual object in the mixed reality space 21, the MR-side object control unit 820 receives object data indicating the form of the virtual object, and arranges the virtual object based on the object data. Here, the MR-side object control unit 820 may receive the object data, for example, from the control unit 610 of the server 600, or from the VR-side object control unit 920 of the VR system 200. Specifically, when arranging the VR avatar 25 in the mixed reality space 21, the MR-side object control unit 820 may receive object data indicating the form of the VR avatar 25 from the control unit 610, and arrange the VR avatar 25 in the form indicated by the object data in the mixed reality space 21. When arranging the VR avatar 25 in the mixed reality space 21, the MR-side object control unit 820 may receive object data indicating the form of the VR avatar 25 from the VR-side object control unit 920, and arrange the VR avatar 25 in the form indicated by the object data in the mixed reality space 21.

The display control unit 840 controls the image display on the display 411 of the MR device 410. The display control unit 840 generates an image for displaying a virtual object in the mixed reality space 21 placed by the MR side object control unit 820 at the position placed by the MR side object control unit 820. The display control unit 840 also causes the image to be displayed on the display 411. This provides the MR user 6 with a view of the mixed reality space in which the virtual object is placed at the desired position in the real space.

The display control unit 840 also controls the image display on the display 470. For example, the display control unit 840 causes the display 470 to display an image similar to the image displayed on the display 411. Note that the display control unit 840 may also cause the display 470 to display an image showing a view of the mixed reality space provided to the MR user 6 by the MR device 410.

When the microphone 415 of the MR device 410 detects speech from the MR user 6, the sound control unit 845 acquires sound data corresponding to the speech. The sound control unit 845 also transmits the acquired sound data to the computer 300, etc. via the network 2. When the sound control unit 845 receives sound data from the computer 300 via the network 2, it outputs sound (speech) corresponding to the sound data from the speaker 416. This allows the MR user 6 to communicate with the VR user, for example, by phone. The MR side input receiving unit 816 may also receive sound data related to the speech of the MR user 6 detected by the microphone 415 as input from the MR user 6.

The communication control unit 850 can communicate with the server 600, the computer 300, and other information communication devices via the network 2. For example, the communication control unit 850 transmits information used by the server 600 or the computer 300 to the server 600 or the computer 300. The communication control unit 850 also receives information used by the computer 500 from the server 600 or the computer 300, for example.

The control unit 310 of the VR system 200 includes a VR side input receiving unit 916, a VR side object control unit 920, a virtual camera control unit 930, a display control unit 940, a sound control unit 945, and a communication control unit 950.

The VR side input receiving unit 916 receives input from the VR user. In other words, the VR side input receiving unit 916 acquires input information from the VR user. Specifically, the VR side input receiving unit 916 receives outputs from the detection device 260, the controller 280, the first camera 213, the second camera 214, a wearable device worn by the VR user, the gaze sensor 212, the sensor 217, the sensor 286, and image sensors provided in the detection device 260 and the VR device 210 as input from the VR user. In other words, the VR-side input receiving unit 916 receives the output of detection means for detecting the movements of the VR user (e.g., the detection device 260, the controller 280, the first camera 213, the second camera 214, a wearable device worn by the VR user, the gaze sensor 212, the sensor 217, the sensor 286, and the image sensors provided in the detection device 260 and the VR device 210) as input from the VR user. As described below, these devices and sensors are for obtaining information about the movements of the VR user that is used to control virtual objects in the virtual space 11 and the mixed reality space 21, and the VR user can input information for moving the virtual objects, etc., to the VR system 200 through his or her own movements.

Specifically, the VR-side input receiving unit 916 may receive, for example, an output from a camera serving as the detection device 260 or a camera provided in the VR device 210 that captures the surroundings of the VR device 210 as an input from the VR user. More specifically, the VR-side input receiving unit 916 may receive, as an input from the VR user, position information indicating the position of the VR user in real space, detected by image recognition from an image captured by an image sensor provided in each of these cameras. In other words, the VR-side input receiving unit 916 may receive, as an input from the VR user, an output from a position detection means that detects the position of the VR user in real space. The position detection means may detect the position of the VR user and acquire the position information based on, for example, positioning based on wireless communication (e.g., Wi-Fi communication, Bluetooth communication, or UWB communication) between the VR device 210 and a specific device of the VR system 200 (e.g., the detection device 260 or a beacon (not shown)), or positioning using a GPS (Global Positioning System) sensor provided in the VR device 210. The position information indicating the position of the VR user can also be considered information regarding the movement of the VR user.

The VR-side input receiving unit 916 may also receive, as input from the VR user, information about the VR user's movements detected by image recognition from images captured by an image sensor provided in a camera serving as the detection device 260 or a camera provided in the VR device 210 that captures the surroundings of the VR device 210. For example, the VR-side input receiving unit 916 may receive, as input from the VR user, information about the VR user's hand movements detected by image recognition, or information about the VR user's inclination or facing direction.

The VR side input receiving unit 916 may also receive, as input from a VR user, information regarding the user's movements obtained using, for example, a device worn by the VR user (e.g., a wearable device such as a watch-type, wristband-type, ring-type, or clothing-type device, or an implantable device) capable of tracking the movements of the VR user (and a device that tracks the movements of the MR user 6 in cooperation with the device via short-range wireless communication, etc.).

The VR-side input receiving unit 916 may also receive, for example, information indicating the VR user's operation detected by the controller 280 (in other words, information regarding the VR user's movements) as input from the VR user.

The VR-side input receiving unit 916 may also receive output data (in other words, information regarding the VR user's movements) regarding the inclination or direction of the VR device 210 from a sensor (e.g., an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, etc.) provided in the detection device 260 or the VR device 210 as input from the VR user.

The control unit 310 controls the virtual space 11. The control unit 310 acquires virtual space data indicating the virtual space 11 stored in the storage unit 611, and defines the virtual space 11 to be provided to the VR user based on the virtual space data.

The VR side object control unit 920 controls virtual objects in the virtual space 11.

The VR-side object control unit 920 places a virtual object in the virtual space 11 indicated by the virtual space data. The object data indicating the virtual object is stored, for example, in the storage unit 611 or the storage unit 311 as an object data storage unit. The VR-side object control unit 920 then places the virtual object in the virtual space 11 using the object data.

Specifically, the VR-side object control unit 920 places, for example, in a virtual conference room, MR avatars 26A, 26B, 26C of MR users 6A, 6B, 6C, virtual objects to which predetermined changes are applied by at least one of the VR user and MR user 6, etc. In this embodiment, the VR-side object control unit 920 places, in the virtual space 11, the VR avatar 25, the MR avatar 26, and a mock-up virtual object 30 (hereinafter also referred to as the "virtual model 30") that can be moved by the VR user and the MR user 6. Note that a virtual object 31 corresponding to a desk 8 as a real object, etc., is also placed in the virtual space 11.

For example, when a VR system 200 (in other words, computer 300) used by a VR user is connected to a VR system 200 (in other words, computer 300) used by another VR user via network 2, the VR side object control unit 920 may place a VR avatar 25 of the other VR user in the virtual space 11. The VR avatar 25 of the other VR user can be operated by the other VR user.

The VR-side object control unit 920 places the MR avatar 26 of the MR user 6 at a position in the virtual space 11 that corresponds to the position of the MR user 6 in the mixed reality space 21. In other words, the VR-side object control unit 920 places the MR avatars 26A, 26B, and 26C so that the positions of the MR users 6A, 6B, and 6C in the real conference room 21 are the same as the positions of the MR avatars 26A, 26B, and 26C in the virtual conference room.

The appearance (in other words, the form) of the MR avatar 26 may imitate the appearance of the MR user 6, but it does not have to imitate it. For example, the appearance of the MR avatar 26 may imitate the appearance of a specific animal. When placing a virtual object in the mixed reality space 21, the VR-side object control unit 920 receives object data indicating the form of the virtual object and places the virtual object based on the object data. Here, the VR-side object control unit 920 may receive the object data from the control unit 610 of the server 600, or may receive it from the MR-side object control unit 820 of the MR system 400. Specifically, for example, when placing the MR avatar 26 in the virtual space 11, the VR-side object control unit 920 may receive object data indicating the form of the MR avatar 26 from the control unit 610 or the MR-side object control unit 820, and place the MR avatar 26 in the form indicated by the object data in the virtual space 11. The appearance of the MR avatar 26 may be preset by the MR user 6, and when the MR avatar 26 is displayed on the VR device 210, it may be displayed with the appearance set by the MR user 6. In addition, when the appearance of the MR avatar 26 imitates the appearance of the MR user 6, the VR-side object control unit 920 may place the MR avatar 26 in the virtual space 11 (e.g., generated by the control unit 510) based on an image of the MR user 6 acquired by a specific camera such as the camera 413 or the camera serving as the detection device 460.

The virtual camera control unit 930 places the virtual camera 14 in the virtual space 11. The virtual camera control unit 930 also controls the position of the virtual camera 14 in the virtual space 11 and the inclination (direction) of the virtual camera 14. In this embodiment, the virtual camera control unit 930 places the virtual camera 14 at the eye position of the VR avatar 25 in the virtual space 11. In other words, the virtual camera control unit 930 links the position of the virtual camera 14 to the position of the VR avatar 25, and moves the virtual camera 14 in the virtual space 11 when the VR avatar 25 moves in the virtual space 11. The position of the virtual camera 14 does not have to be linked to the position of the VR avatar 25. For example, it may be possible to move the VR avatar 25 in the virtual space 11 while keeping the virtual camera 14 fixed in a predetermined position.

The display control unit 940 controls the image display on the display 211 of the VR device 210. The display control unit 940 defines the field of view 15 according to the position and tilt of the virtual camera 14 (in other words, the tilt of the head of the VR user wearing the VR device 210). The display control unit 940 also generates a field of view image 17 to be displayed on the display 211 based on the defined field of view area 15. The field of view image 17 generated by the display control unit 940 is output to the VR device 210.

The display control unit 940 also controls the image display on the display 270. For example, the display control unit 940 causes the display 270 to display an image similar to the image displayed on the display 211.

When the sound control unit 945 detects speech from the VR user using the microphone 215 from the VR device 210, it acquires sound data corresponding to the speech. The sound control unit 945 also transmits the acquired sound data to the computer 500 or other VR users' computers 300 via the network 2. When the sound control unit 945 receives sound data from the computer 500 or other users' computers 300 via the network 2, it outputs sound (speech) corresponding to the sound data from the speaker 216. This allows the VR user to communicate with the MR user 6, for example, by phone call. The VR side input receiving unit 916 may also receive sound data related to the VR user's speech detected by the microphone 215 as input from the VR user.

The communication control unit 950 can communicate with the server 600, the computer 500, other VR users' computers 300, and other information communication devices via the network 2. The communication control unit 950 transmits, for example, information used by the server 600, the computer 500, or other VR users' computers 300 to the server 600, the computer 500, or other VR users' computers 300. The communication control unit 850 also receives, for example, information used by the computer 300 from the server 600, the computer 500, or other VR users' computers 300.

(Integration of mixed reality and virtual space)
In this embodiment, the virtual objects displayed by the MR device 410 and the virtual objects displayed by the VR device 210 are controlled based on input from the MR user 6 acquired by the MR side input receiving unit 816.

Specifically, the MR-side object control unit 820 applies a predetermined change to a virtual object in the mixed reality space 21, specifically the virtual model 30, based on input from the MR user 6 received by the MR-side input receiving unit 816. Here, the predetermined change may be a change to move a virtual object in the mixed reality space 21, such as moving the virtual object, changing the inclination or direction of the virtual object, or changing the relative positional relationship between parts of the virtual object. In addition, the predetermined change may be a change to the color of the virtual object, a change to information displayed by the virtual object, or a change to the form of the virtual object.

More specifically, the MR-side object control unit 820 moves the virtual model 30 in the mixed reality space 21 based on, for example, information on the movement of the MR user 6 received by the MR-side input receiving unit 816. Specifically, for example, when a gesture of the MR user 6 carrying the virtual model 30 is detected by a detection means (for example, the detection device 460 or the camera 413, etc.) and the detection means outputs information related to the gesture, the MR-side input receiving unit 816 receives the output as an input from the MR user 6. Also, based on the input received by the MR-side input receiving unit 816, the MR-side object control unit 820 moves the virtual model 30 in the mixed reality space 21 according to the instruction from the MR user 6 by the gesture. Note that the gesture of the MR user 6 that causes a predetermined change to the virtual model 30 as a virtual object does not have to be one that touches the virtual model 30 in the mixed reality space 21. For example, it may be possible to rotate the virtual object by holding a hand toward the virtual model 30 and waving the hand sideways.

The MR-side object control unit 820 may also make a predetermined change to the virtual object based on, for example, information about the operation of the MR user 6 on the controller 480 (in other words, information about the movement of the MR user 6) received by the MR-side input receiving unit 816. The MR-side object control unit 820 may also make a predetermined change to the virtual object based on, for example, sound data about the speech of the MR user 6 received by the MR-side input receiving unit 816.

The VR-side object control unit 920 also applies predetermined changes to virtual objects in the virtual space 11 based on input from the MR user 6 received by the MR-side input receiving unit 816.

Specifically, the VR-side object control unit 920 applies a change to the virtual model 30 in the virtual space 11 similar to the change applied to the virtual model 30 in the mixed reality space 21 based on, for example, input from the MR user 6 received by the MR-side input receiving unit 816. That is, for example, as described above, when the virtual model 30 in the mixed reality space 21 moves due to a gesture of the MR user 6 to carry the virtual model 30, the VR-side object control unit 920 moves the virtual model 30 in the virtual space 11 in the same manner.

The method of imparting a change to the virtual model 30 in the virtual space similar to the change imparted to the virtual model 30 in the mixed reality space 21 is not particularly limited, but may be, for example, as follows. That is, for example, when the MR-side object control unit 820 moves the virtual model 30 (in other words, the virtual object) in the mixed reality space 21 based on an input from the MR user 6 (in other words, at a predetermined opportunity), it may send information indicating the position and tilt of the virtual model 30 after the movement (in other words, information regarding the state of the virtual model 30) to the VR-side object control unit 920. Then, the VR-side object control unit 920 may place the virtual model 30 at the position indicated by the information in the virtual space 11, or place the virtual model 30 at the tilt indicated by the information. Also, for example, the communication control unit 850 may send information based on the input from the MR user 6 (for example, information on the movement amount, movement trajectory, inclination, etc. of the virtual model 30 calculated (in other words, acquired) by the control unit 510 based on the input) to the communication control unit 950, and the VR-side object control unit 920 may move a virtual object such as the virtual model 30 based on the information. That is, in this embodiment, the VR-side object control unit 920 makes a predetermined change to the virtual object in the virtual space 11 based on the input from the MR user 6 accepted by the MR-side input acceptance unit 816, but "based on the input from the MR user 6" is not limited to the VR-side object control unit 920 itself receiving the input and controlling the virtual object, and may be any virtual object in the virtual space 11 that ultimately makes a predetermined change in response to the input.

The VR-side object control unit 920 also moves the MR avatar 26 in the virtual space 11 based on, for example, input from the MR user 6 received by the MR-side input receiving unit 816. Specifically, the VR-side object control unit 920 moves the MR avatar 26 based on, for example, information on the movement of the MR user 6 received by the MR-side input receiving unit 816. For example, the VR-side object control unit 920 moves the MR avatar 26 based on position information indicating the position of the MR user 6 in the mixed reality space 21, which the MR-side input receiving unit 816 receives as input from the MR user 6. That is, the VR-side object control unit 920 places the MR avatar 26 at a position in the virtual space 11 corresponding to the position of the MR user 6 in the mixed reality space 21 based on the position information, and when the MR user 6 moves in the mixed reality space 21, places the MR avatar 26 at a position in the virtual space 11 corresponding to the position after the movement.

The VR-side object control unit 920 also moves the MR avatar 26 based on, for example, information on the movements of each part of the MR user 6's body (e.g., hand movements, head movements, eye movements, changes in facial expressions, etc.) and information on the inclination and direction of the MR user 6, which are received by the MR-side input receiving unit 816 as input from the MR user 6. That is, the VR-side object control unit 920 reflects the movements of each part of the MR user 6's body (in other words, posture) and the inclination and direction of the MR user 6 in the MR avatar 26 in the virtual space 11 based on each of these pieces of information. That is, in this embodiment, the VR-side object control unit 920 controls the movement of the MR avatar 26 so that the movement of the MR avatar 26 in the virtual space 11 imitates the movement of the MR user 6 detected by a detection means for detecting the movement of the MR user 6. In other words, the VR-side object control unit 920 controls the movement of the MR avatar 26 so that the movement of at least part of the body (e.g., hands, feet, head, eyes, mouth, etc.) of the MR avatar 26 is linked to the movement of the MR user 6.

In addition, at least a part of the movement of the MR avatar 26 may be realized by the MR user 6 operating the controller 480 (e.g., operating an analog stick, a button, etc.). That is, for example, the VR-side object control unit 920 may move a part of the body of the MR avatar 26 based on information indicating the MR user 6's operation of the controller 480 as input from the MR user 6.

Note that at least one of the position and movement of the MR avatar 26 in the virtual space 11 does not have to be linked to the position or movement of the MR user 6 in the mixed reality space 21. That is, for example, even if the MR user 6 moves in the mixed reality space 21, the MR avatar 26 in the virtual space 11 may not move in response to this. Also, for example, even if the MR user 6 moves his/her hand in the mixed reality space 21, the MR avatar 26 in the virtual space 11 may not make the same movement.

In addition, in this embodiment, the virtual objects displayed by the VR device 210 and the virtual objects displayed by the MR device 410 are controlled based on input from the VR user acquired by the VR side input receiving unit 916.

Specifically, the VR-side object control unit 920 applies a predetermined change to a virtual object in the virtual space 11, specifically the virtual model 30, based on input from the VR user received by the VR-side input receiving unit 916.

More specifically, the VR-side object control unit 920 makes a predetermined change to the virtual model 30 in the virtual space 11 based on, for example, information indicating the VR user's operation on the controller 280 (in other words, information regarding the VR user's movement) received by the VR-side input receiving unit 916. Specifically, for example, the VR-side object control unit 920 moves the virtual model 30 in the virtual space 11 according to instructions from the VR user via operations on the controller 280.

The VR-side object control unit 920 may also make a predetermined change to a virtual object based on, for example, a gesture of the VR user detected by a detection means. The VR-side object control unit 920 may also make a predetermined change to a virtual object based on, for example, sound data related to the speech of the VR user received by the VR-side input receiving unit 916.

The MR-side object control unit 820 also applies predetermined changes to virtual objects in the mixed reality space 21 based on input from the VR user received by the VR-side input receiving unit 916.

Specifically, the MR-side object control unit 820 applies a change to the virtual model 30 in the mixed reality space 21 similar to the change applied to the virtual model 30 in the virtual space 11 based on, for example, input from the VR user received by the VR-side input receiving unit 916. That is, for example, as described above, when the virtual model 30 in the virtual space 11 moves due to the VR user's operation on the controller 280, the MR-side object control unit 820 moves the virtual model 30 in the mixed reality space 21 in the same manner.

The method of imparting a change to the virtual model 30 in the mixed reality space 21 similar to the change imparted to the virtual model 30 in the virtual space 11 is not particularly limited, but may be, for example, as follows. That is, for example, when the VR-side object control unit 920 moves the virtual model 30 (in other words, the virtual object) in the virtual space 11 based on an input from the VR user (in other words, at a predetermined opportunity), it may send information indicating the position and tilt of the virtual model 30 after the movement (in other words, information regarding the state of the virtual model 30) to the MR-side object control unit 820. Then, the MR-side object control unit 820 may place the virtual model 30 at the position indicated by the information in the mixed reality space 21, or place the virtual model 30 at the tilt indicated by the information. Also, for example, the communication control unit 950 may send information based on the input from the VR user (for example, information on the movement amount, movement trajectory, inclination, etc. of the virtual model 30 calculated (in other words, acquired) by the control unit 310 based on the input) to the communication control unit 850, and the MR-side object control unit 820 may move a virtual object such as the virtual model 30 based on the information. That is, in this embodiment, the MR-side object control unit 820 makes a predetermined change to a virtual object in the mixed reality space 21 based on the input from the VR user received by the VR-side input receiving unit 916, but "based on the input from the VR user" does not necessarily mean that the MR-side object control unit 820 itself receives the input and controls the virtual object, but may mean that the virtual object in the mixed reality space 21 ultimately makes a predetermined change in response to the input.

The VR-side object control unit 920 also makes a predetermined change to the VR avatar 25 in the virtual space 11 based on input from the VR user received by the VR-side input receiving unit 916. Specifically, the VR-side object control unit 920 makes a predetermined change to the VR avatar 25 in the virtual space 11 based on, for example, information indicating the VR user's operation on the controller 280 (in other words, information regarding the VR user's movements) received by the VR-side input receiving unit 916. Specifically, for example, the VR-side object control unit 920 moves the VR avatar 25 in the virtual space 11 in accordance with instructions from the VR user via operation on the controller 280.

The VR-side object control unit 920 also moves the VR avatar 25 based on, for example, information on the movements of each part of the VR user's body (e.g., hand movements, head movements, eye movements, changes in facial expressions, etc.) and information on the VR user's inclination and facing direction, which the VR-side input receiving unit 916 receives as input from the VR user. That is, the VR-side object control unit 920 reflects the movements of each part of the VR user's body (in other words, posture) and the VR user's inclination and facing direction in the VR avatar 25 in the virtual space 11 based on each of these pieces of information. That is, in this embodiment, the VR-side object control unit 920 controls the movement of the VR avatar 25 so that the movement of the VR avatar 25 in the virtual space 11 imitates the movement of the VR user detected by a detection means for detecting the movement of the VR user. In other words, the VR-side object control unit 920 controls the movement of the VR avatar 25 so that the movement of at least part of the body (e.g., hands, feet, head, eyes, mouth, etc.) of the VR avatar 25 is linked to the movement of the VR user.

The MR-side object control unit 820 also imparts to the VR avatar 25 in the mixed reality space 21 a change similar to the change imparted to the VR avatar 25 in the virtual space 11 based on, for example, input from the VR user received by the VR-side input receiving unit 916. That is, for example, as described above, when the VR avatar 25 in the virtual space 11 moves due to the VR user's operation on the controller 280, the MR-side object control unit 820 moves the VR avatar 25 in the mixed reality space 21 in the same manner. Note that the method of imparting to the VR avatar 25 in the mixed reality space 21 a change similar to the change imparted to the VR avatar 25 in the virtual space 11 is not particularly limited, but may be the same as in the case of the virtual model 30 described above, for example.

The position of the VR avatar 25 in the virtual space 11 and the position in the mixed reality space 21 may move in conjunction with the movement of the VR user in real space. That is, for example, the VR-side object control unit 920 may move the VR avatar 25 based on position information indicating the position of the VR user in real space as an input from the VR user.

Furthermore, the movement of the VR avatar 25 in the virtual space 11 and a part of the VR avatar 25 in the mixed reality space 21 may be realized by an operation on the controller 280. That is, for example, the VR side object control unit 920 and the MR side object control unit 820 may move a part of the body of the VR avatar 25 based on information indicating the VR user's operation on the controller 280 as an input from the VR user.

In this embodiment, the results of actions taken on real objects that actually exist in the conference room 21 as a specified real space can also be shared between the VR user and the MR user 6. In this regard, the following describes an example of control of a real object on which a specific piece of writing can be made (e.g., a blackboard or whiteboard). As shown in FIG. 8, a blackboard 35 is placed as a real object in the mixed reality space 21, and a virtual blackboard 36 is placed as a virtual object corresponding to the blackboard 35 in the virtual space 11 at a position corresponding to the blackboard 35.

In this embodiment, the action (in other words, the result of the action) taken by the MR user 6 on the blackboard 35 as a real object in the mixed reality space 21 is reflected on the blackboard 35 in the mixed reality space 21. Also, in this embodiment, the action is reflected on the virtual blackboard 36 in the virtual space 11.

In the mixed reality space 21, the MR user 6 can make a predetermined entry on the blackboard 35. The entry action on the blackboard 35 by the MR user 6 is detected by a detection means for detecting the entry action (hereinafter referred to as the "MR side entry detection means"), and the output from the MR side entry detection means (in other words, information on the entry action) is received by the MR side input reception unit 816 as an input from the MR user 6. The MR side entry detection means may be, for example, a detection means capable of detecting the movement of the hand of the MR user 6. Specifically, the MR side entry detection means may be, for example, a pen-type controller as the controller 480. The MR side entry detection means may be, for example, a camera as the detection device 460, the camera 413, or the like. The MR side entry detection means may be, for example, a device worn by the MR user 6. The MR-side entry detection means may be a sensor provided on the blackboard 35 (e.g., a touch sensor or a sensor that reads reflected infrared light from an infrared light emitting device provided on the blackboard 35). In other words, the blackboard 35 may be capable of communicating with the computer 500, and information regarding the writing action of the MR user 6 detected by the sensor provided on the blackboard 35 may be received by the MR-side input receiving unit 816 as input from the MR user 6.

The writing action of the MR user 6 is reflected on the blackboard 35 in the mixed reality space 21. In other words, when the MR user 6 performs a writing action, the blackboard 35 changes to a state in which the writing content of the MR user 6 is displayed (for example, a picture of a moon, a picture of a heart, and the word "Sample" in FIG. 10(a)) as shown in FIG. 10(a). For example, the blackboard 35 may be an electronic blackboard that electronically displays (in other words, reflects) the writing content of the MR user 6, and when the MR user 6 performs a writing action, it may change to a state in which the writing content is displayed. Specifically, for example, the writing surface of the blackboard 35 itself may be a display, and the display may display the writing content. The blackboard 35 may also be provided with a projector that projects an image onto the writing surface, and the projector may project the writing content onto the writing surface. In other words, the reflection of the user's behavior in the mixed reality space 21 (in other words, a change in the mixed reality space 21 seen through the MR device 410 due to the behavior of the MR user 6) may be realized by some change (for example, a change in the display content) occurring in the real object itself. When the written content is electronically displayed, the computer of the blackboard 35 may control the display based on the detection result of a sensor equipped in the blackboard, or the computer 500 may control the display based on the detection result of the MR side writing detection means. That is, for example, the control unit 510 of the computer 500 or the like may function as a control means (hereinafter referred to as "blackboard display control means") that controls the display of the blackboard 35 based on the input from the MR user 6 received by the MR side input receiving unit 816. Specifically, the blackboard display control means may control the image displayed by the display or projector of the blackboard 35 to be an image showing the written content based on the input from the MR user 6. The writing action of the MR user 6 on the blackboard 35 in the mixed reality space 21 may be reflected by the MR user 6 actually writing characters or pictures on the blackboard 35 with a writing implement such as chalk. In other words, the action of actually writing characters or pictures on the blackboard 35 with a writing implement may be detected by the MR-side writing detection means as the writing action of the MR user 6.

For example, the MR-side object control unit 820 may generate a virtual object (hereinafter, referred to as the "entry object 37") indicating the contents entered by the MR user 6 and place it at the position of the blackboard 35 in the mixed reality space 21 (see FIG. 10(a)). Specifically, the MR-side object control unit 820 may generate the entry object 37 based on the input from the MR user 6 received by the MR-side input receiving unit 816. When the blackboard 35 is viewed through the MR device 410, the state in which the blackboard 35 displays the entry contents (in other words, the entry object 37) may be visually recognized. In other words, the reflection of the user's actions in the mixed reality space 21 (in other words, the change in the mixed reality space 21 seen through the MR device 410 due to the actions of the MR user 6) may be realized by adding a predetermined virtual object to the real object.

In addition, the VR-side object control unit 920 applies a predetermined change to the virtual object corresponding to the real object based on the input from the MR user 6 received by the MR-side input receiving unit 816. Specifically, as shown in FIG. 10(b), the VR-side object control unit 920 controls the virtual blackboard 36 to reflect the input from the MR user 6, and to display the contents written by the MR user 6. In other words, the VR-side object control unit 920 reflects a change in a real object (e.g., the blackboard 35) in the mixed reality space 21 based on the behavior of the MR user 6 in a virtual object (e.g., the virtual blackboard 36) in the virtual space 11 corresponding to the real object. Specifically, for example, when information on the writing action of the MR user 6 is input to the MR side input receiving unit 816, the VR side object control unit 920 may place the writing object 38 indicating the writing contents written by the MR user 6 at the position of the virtual blackboard 36 in the virtual space 11 (see FIG. 10B). In addition, the VR side object control unit 920 may control the virtual blackboard 36 to display the writing contents by changing the form of the virtual blackboard 36 itself. In addition, the virtual blackboard 36 may be a virtual object that displays an image displayed by the display or projector of the blackboard 35, and the VR side object control unit 920 may display the image on the virtual blackboard 36. In addition, the MR system 400 may be provided with a camera that captures the blackboard 35 in the mixed reality space 21, the virtual blackboard 36 may be a virtual object that displays an image captured by the camera, and the VR side object control unit 920 may display the image on the virtual blackboard 36.

In addition, in this embodiment, the action (in other words, the result of the action) taken by the VR user in the virtual space 11 on the virtual blackboard 36 serving as a virtual object is reflected on the virtual blackboard 36 in the virtual space 11. In addition, in this embodiment, the action is reflected on the blackboard 35 in the mixed reality space 21.

In the virtual space 11, the VR user can make predetermined entries on the virtual blackboard 36. Furthermore, the VR user's writing action on the virtual blackboard 36 is detected by a detection means for detecting the writing action (hereinafter referred to as "VR side writing detection means"), and the output from the VR side writing detection means is received by the VR side input receiving unit 916 as an input from the VR user. The VR side writing detection means may be, for example, a detection means capable of detecting the movement of the VR user's hands. Specifically, the VR side writing detection means may be, for example, a controller 280 or the like. Furthermore, the VR side writing detection means may be a camera or the like as the detection device 260. Furthermore, the VR side writing detection means may be, for example, a device worn by the VR user.

The writing action of the VR user is reflected on the blackboard 35 in the mixed reality space 21. In other words, when the VR user performs a writing action, the blackboard 35 changes to a state in which the contents written by the VR user are displayed. In other words, the input from the VR user received by the VR side input receiving unit 916 is reflected on the blackboard 35. The input from the VR user may be reflected on the blackboard 35, for example, as follows. That is, for example, the MR side object control unit 820 may place a writing object 37 indicating the contents written by the VR user at the position of the blackboard 35 in the mixed reality space 21. In addition, the blackboard display control means may control the image displayed by the display or projector of the blackboard 35 to be an image indicating the contents written by the VR user, based on the input from the VR user. In other words, the MR-side object control unit 820 or the blackboard display control means can function as a reflection means that reflects input from a VR user received by the VR-side input receiving unit 916 in a real object in the mixed reality space 21.

The writing action of the VR user is also reflected on the virtual blackboard 36 in the virtual space 11. In other words, when the VR user performs a writing action, the virtual blackboard 36 changes to a state in which the writing content written by the VR user is displayed. In other words, the input from the VR user received by the VR side input receiving unit 916 is reflected on the virtual blackboard 36. Specifically, the VR side object control unit 920 applies a predetermined change to the virtual blackboard 36 based on the input from the VR user received by the VR side input receiving unit 916. More specifically, the VR side object control unit 920 reflects the input from the VR user on the virtual blackboard 36, and controls the virtual blackboard 36 to a state in which the writing content written by the VR user is displayed. Specifically, for example, when information on the writing action of the VR user is input to the VR-side input receiving unit 916, the VR-side object control unit 920 may place an entry object 38 indicating the contents written by the VR user at the position of the virtual blackboard 36 in the virtual space 11. Also, the VR-side object control unit 920 may control the virtual blackboard 36 to display the contents written by the VR user, for example, by changing the form of the virtual blackboard 36 itself. Also, the virtual blackboard 36 may be a virtual object that displays an image displayed by the display or projector of the blackboard 35, and the VR-side object control unit 920 may display the image on the virtual blackboard 36. Also, the MR system 400 may be provided with a camera that captures the blackboard 35 in the mixed reality space 21, the virtual blackboard 36 may be a virtual object that displays an image captured by the camera, and the VR-side object control unit 920 may display the image on the virtual blackboard 36. In other words, the input from the VR user received by the VR-side input receiving unit 916 may be reflected in a real object in the mixed reality space 21, thereby causing a change in the virtual blackboard 36 based on the input from the VR user.

As described above, in this embodiment, actions taken by the MR user 6 on real objects and actions taken by the VR user on virtual objects corresponding to the real objects are reflected in the mixed reality space 21 and the virtual space 11. The reflection of each action in the mixed reality space 21 or the virtual space 11 may be as follows:

That is, for example, when the MR user 6 points to an arbitrary position in the mixed reality space 21 with a predetermined pointer such as a laser pointer, the pointed position (hereinafter referred to as the "pointed position") may be known in the virtual space 11. Specifically, for example, in the mixed reality space 21, the detection device 460 or the camera 413 of the MR system 400 may function as a pointed position detection means for detecting the pointed position. Then, information on the pointed position detected by the pointed position detection means may be received by the MR side input reception unit 816 as an input from the MR user 6. In addition, the VR side object control unit 920 may place a virtual object representing light from a laser pointer at a position in the virtual space 11 corresponding to the pointed position in the mixed reality space 21. In other words, the control unit 310 may control the display of the virtual space 11 in the VR device 210 so that the position in the virtual space 11 corresponding to the pointed position in the mixed reality space 21 is illuminated in a predetermined color representing light from a laser pointer.

Also, for example, when a VR user points to an arbitrary position in the virtual space 11 with a predetermined pointer such as a virtual laser pointer, the pointed position may be known in the mixed reality space 21. Specifically, for example, when information on the operation of the VR user on the controller 280 for operating the virtual laser pointer (in other words, information on the movement of the VR user) is input to the VR side input receiving unit 916, the VR side object control unit 920 may place a virtual object showing light from the virtual laser pointer at the position pointed to by the operation of the VR user (in other words, the position pointed to by the virtual laser pointer). In other words, the control unit 310 may control the display of the virtual space in the VR device 210 so that the position in the virtual space 11 pointed to by the operation of the virtual VR user is illuminated in a predetermined color representing light from the virtual laser pointer. Furthermore, the control unit 510 or the like may control a position in the mixed reality space 21 corresponding to a designated position in the virtual space 11 (specifically, a designated position by the VR user) to be illuminated in a predetermined color representing light from a virtual laser pointer. Specifically, the MR-side object control unit 820 may place a virtual object representing light from a virtual laser pointer at a position in the mixed reality space 21 corresponding to the designated position in the virtual space 11. Furthermore, for example, when the designated position by the VR user is a predetermined position on the virtual blackboard 36, the blackboard display control means may control, based on an input from the VR user, an image displayed by the display or projector of the blackboard 35 to be an image in which a position corresponding to the predetermined position on the virtual blackboard 36 is illuminated (i.e., an image indicating that a predetermined place is being pointed to by a laser pointer). Also, for example, the detection device 460, the MR device 410, or the like may be equipped with a device that emits laser light, and the control unit 510 may control the device in response to input from the VR user received by the VR-side input receiving unit 916 to direct the laser light at a position in the mixed reality space 21 that corresponds to the indicated position in the virtual space 11.

Furthermore, the behavior of the MR user 6 or the VR user may be reflected in the mixed reality space 21 and the virtual space 11 as follows. That is, for example, a device capable of inputting or outputting sound as a real object (e.g., an electronic piano) may be placed in the mixed reality space 21, and the behavior of the MR user 6 or the VR user related to the device may be reflected in the mixed reality space 21 and the virtual space 11. In the following, it is assumed that an electronic piano is placed as a real object in the mixed reality space 21, and a virtual electronic piano is placed as a virtual object corresponding to the electronic piano in the virtual space 11 at a position corresponding to the electronic piano.

The electronic piano may be capable of communicating with the computer 500, for example. When the MR user 6 operates the electronic piano in the mixed reality space 21, information about the operation (for example, sound data corresponding to the operation) may be sent to the computer 300 via the computer 500. The sound control unit 945 may output a sound from the speaker 216 based on the information. As a result, for example, when the MR user 6 operates a key of the electronic piano (for example, a key of the note "C"), a sound corresponding to the operated key (for example, the note "C") may be output from the speaker 216. In other words, the MR side input receiving unit 816 may receive an output (for example, sound data corresponding to the operation) from the electronic piano (in other words, a sensor that detects an operation on the key, etc.) as a detection means for detecting the operation of the MR user 6 as an input from the MR user 6, and the control unit 310 (for example, the sound control unit 945) may output a sound corresponding to the operation of the MR user 6 on the electronic piano to the speaker 216 based on the input. The sound control unit 945 may also control the direction from which the sound is emitted so that the sound corresponding to an operation on the electronic piano sounds as if it is being output from a virtual electronic piano in the virtual space 11.

In addition, when the VR user operates the virtual electronic piano in the virtual space 11, information on the operation may be sent to the computer 500. The sound control unit 845 may output a sound from the speaker 416 based on the information. As a result, for example, when the VR user operates a key (e.g., a key of the note "C") of the virtual electronic piano, a sound corresponding to the operated key (e.g., the note "C") may be output from the speaker 416. In other words, the VR side input receiving unit 916 may receive an output from a detection means (e.g., the detection device 260 or the controller 280) that detects the operation of the VR user on the virtual electronic piano as an input from the VR user, and the control unit 510 (e.g., the sound control unit 845) may output a sound corresponding to the operation of the VR user on the virtual electronic piano to the speaker 416 based on the input. The control unit 510 may output a sound corresponding to the operation of the VR user on the virtual electronic piano to the speaker of the electronic piano.

When the MR user 6 operates the electronic piano, the sound corresponding to the operation may be output in the mixed reality space 21 by the speaker of the electronic piano or by the speaker 416. Also, when the VR user operates the virtual electronic piano, the sound corresponding to the operation may be output in the virtual space 11 by the speaker 216.

<Processing Executed by Information Processing System>
Next, the flow of processing executed by the information processing system 100 will be described with reference to FIGS.

An example of the processing performed by the VR system 200 is described below with reference to FIG. 11.

In step S101, the processor 301 of the computer 300 identifies virtual space data and defines the virtual space 11.

In step S102, the processor 301 places the VR avatar 25 of the VR user in the virtual space 11.

In step S103, the processor 301 acquires position information indicating the position of the MR user 6 in the mixed reality space 21 from the processor 501 of the computer 500, which is connected to the computer 300 via the network 2 and controls the mixed reality space 21 corresponding to the virtual space 11.

In step S104, the processor 301 places the MR avatar 26 of the MR user 6 at a position in the virtual space 11 that corresponds to the position of the MR user 6 in the mixed reality space 21 based on the position information acquired in step S103.

In step S105, the processor 301 places the virtual model 30 as a virtual object in the virtual space 11. The processor 301 places the virtual model 30 at a position in the virtual space 11 that corresponds to the position of the virtual model 30 in the mixed reality space 21. Note that placing at a corresponding position may mean that the position in the mixed reality space 21 is determined first, or that the position in the virtual space 11 is determined first.

In step S106, the processor 301 accepts input from the VR user. Specifically, the processor 301 accepts, for example, the output of a detection means that detects the movement of the VR user as input from the VR user.

In step S107, the processor 301 makes a predetermined change to the virtual object in the virtual space 11 based on the input from the VR user. Specifically, the processor 301 moves the VR avatar 25 or the virtual model 30 in the virtual space 11 based on, for example, the input from the VR user. For example, if the input is related to the VR avatar 25, the processor 301 moves the VR avatar 25. Also, for example, if the input is related to the virtual model 30, the processor 301 moves the virtual model 30. Also, the processor 301 transmits information related to the input from the VR user to the processor 501.

In step S108, the processor 301 acquires information regarding the input from the MR user 6 that was accepted by the processor 501.

In step S109, the processor 301 makes a predetermined change to the virtual object in the virtual space 11 based on the input from the MR user 6 acquired in step S108. Specifically, the processor 301 moves the MR avatar 26 or the virtual model 30 in the virtual space 11, for example, based on the input from the MR user 6. For example, if the input is related to the MR avatar 26, the processor 301 moves the MR avatar 26. Also, if the input is related to the virtual model 30, the processor 301 moves the virtual model 30.

Next, an example of the processing performed by the MR system 400 will be described with reference to FIG. 12.

In step S201, the processor 501 of the computer 500 defines the mixed reality space 21.

In step S202, the processor 501 detects the position of the MR user 6 in the mixed reality space 21. The processor 501 also transmits position information indicating the detected position of the MR user 6 to the processor 301 of the computer 300 that is connected to the computer 500 via the network 2 and that controls the virtual space 11 that corresponds to the mixed reality space 21.

In step S203, the processor 501 acquires location information indicating the location of the VR avatar 25 in the virtual space 11 from the processor 301.

In step S204, the processor 501 places the VR avatar 25 at a position in the mixed reality space 21 that corresponds to the position of the VR avatar 25 in the virtual space 11, based on the position information acquired in step S203. Note that placing at a corresponding position may mean that the position in the virtual space 11 is determined first, or that the position in the mixed reality space 21 is determined first.

In step S205, the processor 501 places the virtual model 30 as a virtual object in the mixed reality space 21. The processor 501 places the virtual model 30 at a position in the mixed reality space 21 that corresponds to the position of the virtual model 30 in the virtual space 11.

In step S206, the processor 501 accepts input from the MR user 6. Specifically, the processor 501 accepts, for example, the output of a detection means that detects the movement of the MR user 6 as input from the MR user 6.

In step S207, the processor 501 applies a predetermined change to the virtual object in the mixed reality space 21 based on the input from the MR user 6. Specifically, the processor 501 moves the virtual model 30 in the mixed reality space 21, for example, based on the input from the MR user 6. The processor 501 also transmits information regarding the input from the MR user 6 to the processor 301.

In step S208, the processor 501 acquires information regarding the input from the VR user that was accepted by the processor 301.

In step S209, the processor 501 makes a predetermined change to the virtual object in the mixed reality space 21 based on the input from the VR user acquired in step S208. Specifically, the processor 501 moves the VR avatar 25 or the virtual model 30 in the mixed reality space 21 based on, for example, the input from the VR user. For example, if the input is related to the VR avatar 25, the processor 501 moves the VR avatar 25. Also, if the input is related to the virtual model 30, the processor 501 moves the virtual model 30.

Next, an example of a process for reflecting the behavior of the MR user 6 in the virtual space 11 will be described with reference to FIG. 13.

In step S301, the processor 501 receives an input from the MR user 6. Specifically, the processor 501 receives, as an input from the MR user 6, the output of a detection means that detects the behavior of the MR user 6 with respect to a specific real object (e.g., a blackboard 35, a laser pointer, or an electronic piano, etc.).

In step S302, the processor 501 transmits information regarding the input from the MR user 6 received in step S301 to the processor 301 (in other words, the VR system 200).

In step S303, the processor 301 makes a predetermined change to the virtual object corresponding to the specific real object on which the MR user 6 performed an action, based on the information transmitted in step S302. For example, when the MR user 6 performs a writing action to write a predetermined entry on the blackboard 35, the processor 301 changes the virtual blackboard 36 corresponding to the blackboard 35 to a state in which the predetermined entry is displayed.

Next, an example of a process for reflecting the actions of a VR user in the mixed reality space 21 will be described with reference to FIG. 14.

In step S351, the processor 301 accepts an input from a VR user. Specifically, the processor 301 accepts, as an input from the VR user, the output of a detection means that detects the VR user's behavior with respect to a virtual object (e.g., a virtual blackboard 36 or a virtual electronic piano) that corresponds to a specific real object.

In step S352, the processor 301 transmits information regarding the input from the VR user received in step S351 to the processor 501 (in other words, the MR system 400).

In step S353, the processor 501 reflects the action performed by the VR user on a virtual object corresponding to a specific real object in the specific real object based on the information transmitted in step S352. For example, when the VR user performs a writing action to write a specific entry on the virtual blackboard 36, the processor 501 changes the blackboard 35 to a state in which the specific entry is displayed.

Second Embodiment
Next, a second embodiment of the present invention will be described. The information processing system 100 of the second embodiment may adopt the contents described in the first embodiment to the extent that no contradiction occurs. For example, the information processing system 100 of the second embodiment may include a part or all of the configuration of the information processing system of the first embodiment. In the following, the description of the matters described in the first embodiment will be omitted or simplified.

The following describes an example in which the information processing system 100 is applied to a service in which events taking place in the virtual space 11 are reflected in the mixed reality space 21. Specifically, the following describes an information processing system 100 in which an MR user 6 who is at a position in the real space corresponding to a specific location (in front of store B in this case) in the virtual space 11 that mimics a location that actually exists (in front of store B in this case) can experience an event (in this case, a live music performance) taking place at the specific location (in front of store B in this case) via the MR device 410 (here, the user can watch the live music performance).

The coordinate definition unit 812 stores information about the correspondence between the coordinates measured by the GPS sensor (in other words, the position detection means) and the virtual space coordinates (in other words, the correspondence between points in the real space and points in the virtual space 11) in the storage unit 611, which serves as a correspondence storage unit. In other words, in this embodiment, information indicating the correspondence between positions in the real space (in other words, the mixed reality space) and the virtual space 11 is stored in advance in the correspondence storage unit.

The MR side input receiving unit 816 receives input from the position detection means. The position detection means acquires position information indicating the position of the MR user 6 using a GPS sensor provided in the MR device 410. In other words, the MR side input receiving unit 816 accepts input of position information indicating the current position of the MR user 6 in real space (e.g., position information acquired using a GPS sensor).

The control unit 310 of the VR system 200 controls the execution of a predetermined event that is performed at a specific position in the virtual space 11. In this embodiment, the predetermined event is a predetermined event (specifically, a music concert). Specifically, the VR-side object control unit 920 places an event-related object 50 as a virtual object related to the predetermined event at a specific position in the virtual space 11, as shown in FIG. 15. More specifically, the VR-side object control unit 920 places, for example, an object of a character appearing in a music concert (hereinafter referred to as a "performing character") or an object of a stage for the music concert as an event-related object 50 in front of a store B in the virtual space 11. The VR-side object control unit 920 may place the event-related object 50 based on an instruction from a VR user (in other words, an input from the VR user received by the VR-side input receiving unit 916 (for example, an operation on the controller 280)). The VR-side object control unit 920 may also place the event-related object 50 based on an instruction from the operator of this service.

The VR-side object control unit 920 also moves the event-related object 50 (e.g., a featured character) and controls the progress of the event (in other words, progresses the event). The VR-side object control unit 920 may move the featured character based on the VR user's instructions (in other words, input from the VR user received by the VR-side input receiving unit 916 (e.g., an operation on the controller 280)). The movements of the featured characters may be prepared in advance, and the VR-side object control unit 920 may move the featured character by playing back the prepared data. In other words, the featured character may be a VR avatar 25 or a non-player character, etc.

In other words, in the information processing system 100 of this embodiment, the VR system 200 (in other words, the control unit 310) provides a view related to a music live performance as a specified event to a VR user (in other words, a VR user controlling a character at a specific location) who is at a specific location (in other words, in front of virtual store B) in the virtual space 11 (in other words, virtual town A) in which the user can move around by controlling his or her own character (e.g., VR avatar 25). Here, the VR user may be a different user from the user controlling the character appearing in the music live performance (in other words, for example, a spectator at the music live performance), or may be the same user (in other words, for example, a performer at the music live performance).

The event-related objects 50 are provided with information (in other words, a tag) indicating that they are objects related to a specific event (in other words, a specific event). The event-related objects 50 are also provided with position information indicating the position of the event-related objects 50 in virtual space (in other words, coordinates). Note that the position information of the event-related objects 50 may be provided individually to each event-related object 50, or may be provided collectively to the event, etc.

15, the MR-side object control unit 820 places a virtual object corresponding to the event-related object 50 at a position in the mixed reality space 21 corresponding to the position at which the event-related object 50 is placed in the virtual space 11. Here, the virtual object corresponding to the event-related object 50, i.e., the virtual object in the mixed reality space 21 corresponding to the event-related object 50 in the virtual space 11, may have the same form as the event-related object 50 in the virtual space 11 (in other words, it may represent the same object), or may have a different form. In this embodiment, an event-related object 50 of the same form as the event-related object 50 in the virtual space 11 (specifically, a character appearing in a music live show, a stage for a music live show, etc.) is placed in the mixed reality space 21.

The MR-side object control unit 820 also moves the event-related object 50 (e.g., a character that appears) to progress the event that is being executed in the mixed reality space 21. Here, the progress of the event by the MR-side object control unit 820 (specifically, control of moving the character that appears, etc.) and the progress of the event by the VR-side object control unit 920 (specifically, control of moving the character that appears, etc.) are linked. That is, the MR-side object control unit 820 reflects the event taking place in the virtual space 11 to the mixed reality space 21. In other words, in this embodiment, the event occurring in the virtual space 11 is also reflected in the mixed reality space 21 in real time. Specifically, the MR-side object control unit 820 controls the event-related object 50 in the mixed reality space 21 to move in the same way as the event-related object 50 in the virtual space 11. Here, "reflected in real time" includes a case where there is a slight difference in the occurrence of the event (in other words, the progress of the event), and may be preceded by either the progress of the event in the virtual space 11 or the progress of the event in the mixed reality space 21. In addition, "reflected" here does not mean that the event-related object 50 in the mixed reality space 21 does not have to move in exactly the same way as the event-related object 50 in the virtual space 11. For example, some of the movements of the appearing characters in the virtual space 11 may not be reflected in the appearing characters displayed in the mixed reality space 21. In other words, the control unit 510 of the MR system 400 may provide the MR user 6 with a view that can be determined to be related to the same event as the view provided to the VR user.

In this embodiment, when a live music concert is held in front of store B in town A in virtual space 11, the live music concert is held in front of store B in town A in mixed reality space 21. That is, the MR-side object control unit 820 places an event-related object 50 relating to a specific event taking place at a specific position in virtual space 11 at a position in mixed reality space 21 corresponding to the specific position, and moves the event-related object 50 so as to move in accordance with the movement in virtual space 11. Then, in response to position information input to the MR-side input receiving unit 816, the display control unit 840 causes the display 411 of the MR device 410 to display the event-related object 50 placed at the position in mixed reality space 21 indicated by the position information. Specifically, when location information indicating that the MR user 6 is in front of the store B in the real space is input to the MR side input receiving unit 816, which receives input of location information acquired using a GPS sensor (specifically, location information indicating the location of the MR user 6), the control unit 510 causes the display 411 to display an event-related object 50 relating to a live music performance taking place in front of the store B in the virtual space 11. This provides the MR user 6 with a view of the mixed reality space 21 in which the event-related object 50 is placed in front of the store B in the real space (in other words, a live music performance taking place in the virtual space 11 in front of the store B in the real space). That is, when the location information indicates that the MR user 6 is in a position in the real space corresponding to a specific position in the virtual space 11, the control unit 510 provides the MR user 6 with a view of the mixed reality space 21 reflecting an event taking place at the specific position in the virtual space 11.

The control unit 510 may include a reward granting unit that grants a reward to the MR user 6 based on the MR user 6 visiting a location in real space that corresponds to the specific location. The reward may be, for example, a virtual object (e.g., an item that a character can equip, etc.) that the user who has acquired the reward can use in the game (in other words, in the service), an electronic ticket that can be exchanged for a real object in real space, or a predetermined point owned by the user in the service. In addition, "based on visiting" may mean granting a reward by directly determining whether or not the user has visited, or may be granting a reward when a specific action that can be performed by visiting (e.g., starting or finishing watching a live music performance, or continuing for a predetermined period of time or more) is performed. With this configuration, the MR user 6 can be motivated to visit a location in real space that corresponds to a specific location.

In this manner, in this embodiment, a view of the mixed reality space 21 reflecting a predetermined event occurring at a specific position in the virtual space 11 is provided to the MR user 6, but the predetermined event is not limited to a live music performance. For example, the predetermined event may be an event in which a character operated by a specific user (in other words, an event-related object 50 related to the predetermined event) moves. Specifically, for example, the VR system 200 provides a game in which the character can be moved in the virtual space 11 simulating town A. In this case, the VR-side object control unit 920 causes the character to perform a predetermined action (e.g., an action such as jumping or running) in the virtual space 11 simulating town A based on the instruction of the VR user as the specific user (in other words, an input from the VR user received by the VR-side input receiving unit 916 (e.g., an operation on the controller 280)). Further, the MR-side object control unit 820 places the character at a position in the mixed reality space 21 corresponding to the position where the character is placed in the virtual space 11, and causes the character to perform an action in the mixed reality space 21 similar to the action of the character in the virtual space 11. That is, in this case, for example, when a VR user as the specific user is playing a game in which a character is moved in the virtual space 11 simulating a town A, if the MR user 6 visits the town A in the real space, the control unit 510 reflects the movement of the character in the game (in other words, in the virtual space 11) moved by the specific user in the view of the mixed reality space 21 provided via the display 411 of the MR device 410. That is, in this example, the virtual object operated by the specific user moves in the same manner in the view of the virtual space 11 provided to the specific user in the VR device 210 of the VR user as the specific user and the view of the mixed reality space 21 provided to the MR user 6 in the MR device 410 of the MR user 6 in the town A in the real space. This configuration makes it possible to replicate the movements of characters on the game screen in a real city, improving the entertainment value of the service.

Note that the MR users 6 who can receive a view of the mixed reality space 21 reflecting a specific event occurring at a specific position in the virtual space 11 may be limited. Specifically, the storage unit 511 functions as a rights information storage unit that stores information regarding the right to receive the view (hereinafter referred to as "rights information"), and the control unit 510 may provide the view to the MR user 6 if the MR user 6 has the right, and may not provide the view to the MR user 6 if the MR user 6 does not have the right. In addition, the right may be granted to the MR user 6 by, for example, the MR user 6 paying a price (for example, a payment of cash, or consumption of the assets held by the MR user 6 (for example, virtual currency, etc.) within the service). Specifically, for example, the right to watch a live music performance may be purchased within the service. More specifically, for example, the rights information may store information regarding the presence or absence of tickets for a live music performance that can be purchased within the service, and the control unit 510 may control so that an MR user 6 who has the ticket can watch the live music performance, and an MR user 6 who does not have the ticket cannot watch the live music performance. In other words, the rights information may store information regarding the possession of an item (e.g., a ticket, etc.) related to the right to receive the view, and the control unit 510 may provide the view to the MR user 6 when the MR user 6 has the item. Note that the right (in other words, the item) may be obtained by completing a specified mission within the service, etc.

The control unit 510 may also provide the MR user 6 with a view of the mixed reality space 21 reflecting a predetermined event occurring at a specific position in the virtual space 11 and related to a VR user selected by the MR user 6 from among multiple users. In other words, the MR side input receiving unit 816 may receive an input by the MR user 6 related to an operation (e.g., an operation on the controller 480) for selecting which of the multiple users a predetermined event related to should be reflected, and the control unit 510 may reflect the predetermined event related to the VR user selected by the operation in the view provided to the MR user 6. The operation may be, for example, an operation in which the MR user 6 selects a user from among his/her friends to receive the reflection of the event. In addition, the predetermined events related to all friends are reflected, and the operation may be an operation of registering another user as a friend, etc. In other words, in the service of this embodiment, the MR user 6 may be provided with a view of the mixed reality space 21 reflecting a predetermined event related to his/her friend. Also, for example, the operation may be an operation in which the MR user 6, such as a professional player, selects a user from among multiple players for which a predetermined event is to be reflected. That is, for example, when a player selected by the MR user 6 (in other words, a specific VR user) is playing the above-mentioned game in which the character is moved in the virtual space 11 simulating the town A, the control unit 510 may reflect the movement of the character moved by the player in the game (in other words, in the virtual space 11) in the view of the mixed reality space 21 provided to the MR user 6. In other words, the MR user 6 may be provided with a view of the mixed reality space 21 reflecting a predetermined event related to a user associated with the MR user 6 among multiple users. Also, for example, when an entertainer selected by the MR user 6 is performing a live show in the virtual space 11, the control unit 510 may reflect the live show in the view of the mixed reality space 21 provided to the MR user 6. Note that, when there are multiple events related to a specific user, it may be possible to select a specific event related to the specific user as the event to be reflected.

In this way, in the service of this embodiment, each user (e.g., MR user 6 or VR user) may be able to register other users (e.g., VR user or MR user 6) as friends. Each user can register other users as friends, for example, based on a predetermined operation on the controller 280, 480. Registering as a friend can also be said to be a user associating another user with himself/herself. Registering as a friend can also be said to be bookmarking a specific user other than himself/herself. Registering as a friend can also be said to be making information about a certain user (specifically, a friend) (e.g., information about the login state, etc.) easier to call up than information about other users (specifically, non-friends). Note that there may be a state in which a certain user registers another user as a friend, but the other user does not register the certain user as a friend. That is, there may be a state in which the certain user unilaterally follows the other user. There may also be a state in which the certain user and the other user follow each other. Note that there may not be a state in which one-sided following exists. In other words, in this embodiment, a "friend" may refer only to other users with whom a reciprocal follow relationship is established from the perspective of a certain user, or may refer to other users with whom a one-sided follow relationship is established.

The control unit 510 may also provide the MR 6 user with a view of the mixed reality space 21 reflecting a predetermined event taking place at a specific position in the virtual space 11 and having an attribute corresponding to the attribute set for the MR user 6. Specifically, for example, each MR user 6 may be assigned an attribute related to the user's hobbies and preferences, more specifically, an attribute such as "music lover" or "comedy lover" (or "like a specific entertainer"). The control unit 510 may, for example, reflect a predetermined event with the attribute "music" for an MR user 6 with the attribute "music lover", and not reflect a predetermined event with the attribute "music" for an MR user 6 without the attribute "music lover". The control unit 510 may, for example, reflect a predetermined event with the attribute "comedy" for an MR user 6 with the attribute "comedy lover", and not reflect a predetermined event with the attribute "comedy" for an MR user 6 without the attribute "comedy lover". The attributes set for the MR users 6 may be set by each MR user 6 himself/herself (e.g., input of his/her own preferences via the controller 480, etc.), or may be set automatically by AI (artificial intelligence) (e.g., according to the user's actions within the service, etc.).

As described above, the control unit 510 may reflect in the mixed reality space 21 a specific event selected by the MR user 6 among multiple events taking place in the virtual space 11 (for example, an event related to a specific user selected by the MR user 6, an event having an attribute corresponding to the MR user 6's own attribute set by the MR user 6, an event related to a ticket purchased by the MR user 6, etc.). In other words, the control unit 510 may not reflect in the mixed reality space 21 an event not selected by the MR user 6 among multiple events taking place in the virtual space 11. In this way, when an unselected event is not reflected in the mixed reality space 21, there may be an event that is reflected regardless of the user's selection. In other words, "not reflecting an unselected event in the mixed reality space 21" means that there is at least one event that is reflected in the mixed reality space 21 if a selection is made, and is not reflected in the mixed reality space 21 if no selection is made. The selection of an event may be possible before the specific event selected by the MR user 6 occurs (in other words, starts), or may be possible after the specific event occurs (in other words, while it is taking place).

The virtual space 11 in which an event takes place does not have to be a replica of the real space in which the event is reflected. For example, an area corresponding to a square in front of store B in town A that actually exists may be prepared in the virtual space 11, and a specific event that takes place in that area may be reflected in the MR device 410 of an MR user 6 who is in that square in real space. In this case, the memory unit 611 serving as a correspondence storage unit may store, for example, information indicating the correspondence between the area and the square. Similarly, the virtual space 11 in which real-space events are reflected, as described in the third embodiment, does not have to be a replica of the real space.

In addition, the virtual space 11 in which the event takes place may be one that imitates the topography of the real space in which the event is reflected, but may have a different landscape. For example, in this embodiment, the virtual space 11 may be one that imitates the topography of a town A that actually exists, but may have a different streetscape. Specifically, the virtual space 11 may be one that reproduces an area corresponding to town A in the past (for example, the Edo period), and points in the real space and points in the virtual space 11 may be associated so that each point in the reproduced area and each point in the current town A in the real space indicate the same place on the map (in other words, the same place with the same longitude and latitude). Then, a specific event that takes place at a specific position in the reproduced area may be reflected in the MR device 410 of the MR user 6 that is at a position corresponding to the specific position in the real space. Note that the virtual space 11 in which the event in the real space is reflected, as described in the third embodiment, may also be one that imitates the topography of the real space, but may have a different landscape.

The view of the mixed reality space 21 related to a predetermined event does not have to be provided by the MR device 410 that superimposes a virtual object on the real space seen through the transparent display 411, but may be provided by, for example, a smartphone that displays an image of the real space captured by a camera with a virtual object superimposed on the display 411. In other words, the view of the mixed reality space 21 may be provided by, for example, displaying an event-related object 50 on a transparent display, or by displaying an image of the real space captured with the event-related object 50 superimposed on the display. In other words, a predetermined system including a smartphone that displays an image of the real space captured with a virtual object superimposed on the display may have at least a part of the configuration of the MR system 400. In other words, the MR system 400 in this embodiment or other embodiments can be interpreted as the predetermined system (for example, a smartphone). In this specification, the user of the predetermined system also corresponds to the MR user 6.

In addition, the view of the virtual space 11 relating to a specific event does not have to be provided by a head-mounted display as the VR device 210, but may be provided, for example, by displaying an image of the virtual space 11 on a PC monitor, a television, or the like. In other words, a specific system comprising a PC or a game device, or these devices and a specific display, may have at least a part of the configuration of the VR system 200. In other words, the VR system 200 in this embodiment or other embodiments can be interpreted as being the specific system. In addition, in this specification, a user of the specific system also corresponds to a VR user.

[Third embodiment]
Next, a third embodiment of the present invention will be described. The information processing system 100 of the third embodiment may adopt the contents described in the first or second embodiment to the extent that no contradiction occurs. For example, the information processing system 100 of the third embodiment may include a part or all of the configuration of the information processing system of the first embodiment. Also, the information processing system 100 of the third embodiment may include a part or all of the configuration of the information processing system of the second embodiment. In the following, the matters described in the first or second embodiment will be omitted or simplified.

In the following, an example will be described in which the information processing system 100 is applied to a service in which an event taking place in real space is reflected in the virtual space 11. Specifically, in the following, a case will be described in which an event (hereinafter, as an example, a "music live performance") taking place at a specific position (hereinafter, as an example, "in front of store B" in town A) in a specific place (hereinafter, as an example, "town A") in real space is performed at a position corresponding to the specific position in the virtual space 11 that imitates the specific place (here, in front of store B (in other words, in front of a virtual object that imitates store B)). That is, in the second embodiment, an event taking place in the virtual space 11 is reflected in real space, whereas in the third embodiment, an event taking place in real space is reflected in the virtual space 11. Note that the information processing system 100 may be configured to have both the configuration according to the second embodiment and the configuration according to the third embodiment, and to reflect an event taking place in the virtual space 11 in real space and to reflect an event taking place in the virtual space 11.

The coordinate definition unit 812 stores information about the correspondence between the coordinates measured by the GPS sensor (in other words, the position detection means) and the virtual space coordinates (in other words, the correspondence between points in the real space and points in the virtual space 11) in the storage unit 611, which serves as a correspondence storage unit. In other words, in this embodiment, information indicating the correspondence between positions in the real space (in other words, the mixed reality space) and the virtual space 11 is stored in advance in the correspondence storage unit.

In this embodiment, as shown in FIG. 16, a detection device 460 is placed in front of store B as a specific location. The detection device 460 is a camera, and is configured to capture an image of the area in front of store B as a specific location in real space. Note that multiple detection devices 460 (in other words, cameras) may be placed in front of store B. The detection device 460 (in other words, cameras) in this embodiment may not be intended to detect the movements of the MR user 6, but may simply be intended to capture images of a specific event.

In this embodiment, the MR system 400 may or may not include an MR device 410. For example, the MR system 400 may include a detection device 460 and a computer 500. The computer 500 may be provided outside the detection device 460, or a part or the whole of the computer 500 may be built into the detection device 460. In other words, the MR system 400 in this embodiment or other embodiments can be interpreted as a camera serving as the detection device 460 installed in front of store B in this embodiment.

The detection device 460 also includes a GPS sensor and is capable of acquiring location information indicating the location of the detection device 460. The MR side input acceptance unit 816 also accepts input of location information acquired using the GPS sensor. That is, the MR side input acceptance unit 816 accepts input of location information indicating a specific location. The location information acquired using the GPS sensor can also be said to be location information indicating the location where a specific event photographed by the detection device 460 is taking place. Furthermore, when the detection device 460 photographs a specific person such as the MR user 6 (in other words, when detecting the movement of the specific person), the location information acquired using the GPS sensor can also be said to be location information indicating the current location of the specific person.

The control unit 310 reflects a predetermined event taking place at a specific location in real space in the virtual space 11. In this embodiment, the predetermined event is a predetermined event (specifically, a live music concert). That is, the control unit 310 performs control so that a live music concert taking place in front of store B in real town A is also held in front of store B in the virtual space 11.

Specifically, the MR side input receiving unit 816 (in other words, the control unit 510) receives as input, for example, an image captured by an image sensor provided in a camera serving as the detection device 460. In this case, the image is an image of a live music performance as a predetermined event. The communication control unit 850 also sends the image and information indicating a specific position where the image was acquired to the communication control unit 950, and the control unit 310 displays the image at a position corresponding to the specific position in the virtual space 11 (i.e., in front of store B) based on this information, as shown in FIG. 16. In other words, when the VR user (in other words, the VR avatar 25) is in a position in the virtual space 11 corresponding to a specific position in the real space, the control unit 310 displays the image (i.e., an image of an event taking place at the specific position) on the display 211. This allows a VR user (in other words, a VR user operating a character at a specific position) in a virtual space 11 (in other words, a virtual town A) where the user can move around by operating his/her own character to be provided with a view of an event taking place at the specific position in real space when the VR user (in other words, a VR avatar 25) is at a position in the virtual space 11 corresponding to a specific position in real space, based on an image of a specific event captured by a specific camera, the control unit 310 provides the VR user with a view of the virtual space 11 reflecting an event taking place at the specific position in real space. This allows the VR user to share an experience with a person at the specific position in real space. In this embodiment, information regarding the correspondence between the coordinates measured by the GPS sensor and the virtual space coordinates is stored in the correspondence storage unit as information indicating the correspondence between points in real space and points in virtual space 11. However, for example, information regarding the correspondence between identification information that enables identification of detection device 460 installed at a specific position and coordinates indicating a position in virtual space 11 corresponding to the specific position may be stored as information indicating the correspondence between points in real space and points in virtual space 11, and a predetermined event may be reflected at a position in virtual space 11 corresponding to the specific position based on the information. In other words, the identification information that enables identification of detection device 460 may function as information indicating a specific position.

The image related to the predetermined event that the control unit 310 causes to be displayed on the display 211 may be captured by the camera 413 of the MR device 410 worn by the predetermined MR user 6. In other words, the MR device 410 may function as the detection device 460 of this embodiment. In other words, the MR device 410 may be equipped with a GPS sensor for acquiring location information related to a specific location where the predetermined event is performed and a camera for acquiring images related to the predetermined event. In this case, the MR user 6 wearing the MR device 410 may be a performer or the like of a predetermined event as the predetermined event, or may be a spectator or the like. In addition, the acquisition of location information and the acquisition of images related to the predetermined event may be performed by different MR devices 410 (in other words, MR devices 410 used by different users).

In addition, a predetermined event taking place at a specific position in the real space may be reflected in the virtual space 11 as follows. That is, the control unit 310 may not display an image of a predetermined event as described above, but may cause an avatar of a person related to the predetermined event (here, a performer of a live music performance) to perform an action related to the predetermined event. Specifically, the MR user 6 in the first embodiment may be read as the person, and the avatar of the person as the MR avatar 26 may be moved in the virtual space 11. That is, for example, the MR side input receiving unit 816 receives the output of a detection means that detects the movement of the person as an input from the person. Specifically, for example, the MR side input receiving unit 816 may receive information on the movement of the MR user 6 detected by image recognition from an image captured by an image sensor provided in a camera as the detection device 460 as an input from the person. In addition, the user information acquisition unit 818 acquires identification information that enables the person to be identified. Specifically, the user information acquisition unit 818 may acquire the identification information by identifying the MR user 6 in the mixed reality space 21 by image recognition from an image captured by an image sensor included in the detection device 460. Then, the VR-side object control unit 920 places the person's avatar (in other words, the MR avatar 26) at a position in the virtual space 11 corresponding to a specific position in the real space. In addition, the VR-side object control unit 920 moves the person's avatar in the virtual space 11 based on, for example, an input from the person accepted by the MR-side input acceptance unit 816. Specifically, the VR-side object control unit 920 moves the MR avatar 26 based on, for example, information on the movement of the MR user 6 accepted by the MR-side input acceptance unit 816. That is, the VR-side object control unit 920 controls the movement of the person's avatar in the virtual space 11 so that the movement of the person's avatar in the virtual space 11 imitates the movement of the person detected by a detection means for detecting the movement of the person in the real space. This allows a predetermined event taking place at a specific location in real space to be reflected in virtual space 11. Even with this configuration, when a VR user (in other words, a VR avatar 25) is at a location in virtual space 11 that corresponds to a specific location in real space, a view of virtual space 11 reflecting an event taking place at that specific location in real space can be provided to the VR user based on an image of the predetermined event captured by a specific camera. Specifically, a view of virtual space 11 in which a predetermined event is reproduced by an avatar of a person captured by a specific camera can be provided to the VR user.

The detection of the movement of the person may be performed by the camera 413 of the MR device 410 worn by a specific MR user 6, or may be performed by the controller 480, a device worn by the MR user 6, the gaze sensor 412, the sensor 417, or the sensor 486. The MR user 6 wearing the MR device 410 may be a performer of a specific event as a specific event, or may be an audience member. When detecting the movement of a person by image recognition from an image captured by an image sensor equipped in a specific camera and reflecting it in an avatar in the virtual space 11, only the avatar corresponding to the performer (in other words, a specific person identified based on image recognition or the presence or absence of a specific device that can track the user's movement) among multiple people captured by the camera may appear in the virtual space 11 (the avatars of people other than the specific person may not appear), or avatars (in other words, 3D models) may be created in real time for all people captured by the camera, and the avatars of all people may appear in the virtual space 11.

The display 411 of the MR device 410 of the MR user 6 (e.g., a performer or spectator of the specific event) who is in front of the store B as a specific location where a specific event is taking place may display the VR avatar 25 of the VR user watching the specific event in the virtual space 11. For example, when location information indicating that the MR user 6 is in a specific location is input to the MR side input receiving unit 816 of the MR device 410, the control unit 510 of the MR device 410 may acquire information of the VR avatar 25 at a position in the virtual space 11 corresponding to the specific location via the communication control unit 850 and display the VR avatar 25 on the display 411. That is, when the location information input to the MR side input receiving unit 816 indicates that the MR user 6 is in the specific location, the control unit 510 may display an image of the VR avatar 25 at a position in the virtual space corresponding to the specific location on the display. With this configuration, a person at a location where a specific event is taking place in the real space can see the reaction of the VR user in the virtual space 11. Specifically, for example, when a specific live performance (e.g., a music concert or comedy concert) is taking place in real space, the performers or spectators of the live performance as MR users 6 can view the VR avatars 25 (in other words, the spectators in the virtual space 11) via the display 411 of the MR device 410. This allows the performers or spectators of the live performance as MR users 6 to visually grasp how many spectators have gathered, including the virtual space 11, and how each spectator is reacting. In addition, for example, it becomes possible for a VR user and an MR user 6 who are friends to watch a specific event together.

In this case, the MR device 410 on which the VR avatar 25 is displayed and the device that captures the specified event (e.g., a camera or MR device 410 installed in front of store B) may be separate devices or may be the same device (specifically, the MR device 410 worn by a single MR user 6).

The view of the mixed reality space 21 provided to the MR user 6 at a specific position where a specific event is occurring does not have to be provided by the MR device 410 that superimposes a virtual object (specifically, a VR avatar 25) on the real space seen through the transparent display 411, but may be provided by, for example, a smartphone that displays an image of the real space captured by a camera with a virtual object (specifically, a VR avatar 25) superimposed on it. That is, the information processing system 100 according to this embodiment can be configured to display, for example, a VR avatar 25 (e.g., an audience member's avatar) at a position in the virtual space 11 corresponding to the specific position on the display of the smartphone when a performer performs a specific live performance while taking a selfie using his/her smartphone at a specific position in the real space. In this case, the display of the smartphone or the like may display an image in which only the VR avatar 25 is displayed, or an image of the VR avatar 25 with the virtual space 11 (for example, an image of a position in the virtual space 11 corresponding to the specific position) as the background, rather than an image of the mixed reality space 21 in which the VR avatar 25 is superimposed on an image of the real space.

The control unit 310 may include a reward granting unit that grants a reward to a VR user based on the VR user (in other words, the VR avatar 25) visiting a position in the virtual space 11 that corresponds to a specific position where a specific event is taking place. The reward may be, for example, a virtual object (e.g., an item that a character can equip, etc.) that the user who has acquired the reward can use in the game (in other words, in the service), an electronic ticket that can be exchanged for a real object in the real space, or a specific point that the user owns in the service. In addition, "based on visiting" may be a reward that is granted by directly determining whether or not the user has visited, or a reward that is granted when a specific action that can be performed by visiting (e.g., starting or finishing watching a live music performance, or performing for a specific period of time or more) is performed. With this configuration, the VR user can be motivated to visit a position in the virtual space 11 that corresponds to a specific position.

Note that the VR users who can receive the view of the virtual space 11 reflecting a specific event occurring at a specific position in the real space may be limited. Specifically, the storage unit 311 functions as a rights information storage unit that stores information regarding the right to receive the view (hereinafter referred to as "rights information"), and the control unit 310 may provide the view to the VR user if the VR user has the right, and may not provide the view to the VR user if the VR user does not have the right. In addition, the right may be granted to the VR user by, for example, the VR user paying a price (for example, a payment of cash, or consumption of the VR user's assets (for example, virtual currency, etc.) within the service). Specifically, for example, the right to watch a live music performance may be purchased within the service. More specifically, for example, information regarding the presence or absence of tickets for the live music performance that can be purchased within the service is stored as the rights information, and the control unit 310 may control so that a VR user who has the ticket can watch the live music performance and a VR user who does not have the ticket cannot watch the live music performance. In other words, information regarding possession of an item (e.g., a ticket) related to the right to receive the view is stored as the rights information, and the control unit 310 may provide the view to a VR user if the VR user has the item. Note that the right (in other words, the item) may be obtained by completing a specified mission within the service, etc.

The control unit 310 may also provide the VR user with a view of the virtual space 11 reflecting a predetermined event occurring at a specific position in the real space and related to the MR user 6 selected by the VR user from among multiple users. In other words, the VR side input receiving unit 916 may receive an input by the VR user related to an operation (e.g., an operation on the controller 280) for selecting which of the multiple users a predetermined event related to should be reflected, and the control unit 310 may reflect the predetermined event related to the MR user 6 selected by the operation in the view provided to the VR user. The operation may be, for example, an operation by the VR user to select a user from among his/her friends to receive the reflection of the event. In addition, the predetermined events related to all friends are reflected, and the operation may be an operation to register another user as a friend, etc. In other words, in the service of this embodiment, the VR user may be provided with a view of the virtual space 11 reflecting a predetermined event related to his/her friend. In other words, the VR user may be provided with a view of the virtual space 11 that reflects a specific event related to a user associated with the VR user among multiple users. Also, for example, when an entertainer selected by a VR user is performing live in real space, the control unit 310 may reflect the live performance in the view of the virtual space 11 provided to the VR user. Note that, when there are multiple events related to a specific user, it may be possible to select a specific event related to the specific user as the event to be reflected.

The control unit 310 may also provide the VR user with a view of the virtual space 11 reflecting a predetermined event occurring at a specific position in the real space and having an attribute corresponding to the attribute set for the VR user. Specifically, for example, each VR user may be assigned an attribute related to the user's hobbies and preferences, more specifically, an attribute such as "music lover" or "comedy lover" (or "like a specific entertainer"). The control unit 310 may, for example, reflect a predetermined event with the attribute "music" for a VR user with the attribute "music lover" and not reflect a predetermined event with the attribute "music" for a VR user without the attribute "music lover". The control unit 310 may, for example, reflect a predetermined event with the attribute "comedy" for a VR user with the attribute "comedy lover" and not reflect a predetermined event with the attribute "comedy" for a VR user without the attribute "comedy lover". The attributes set for a VR user may be set by each VR user themselves (e.g., by inputting their own preferences via the controller 280), or may be set automatically by AI (artificial intelligence) (e.g., based on the user's actions within the service, etc.).

As described above, the control unit 310 may reflect in the virtual space 11 a specific event selected by the VR user among multiple events taking place in the real space (for example, an event related to a specific user selected by the VR user, an event having an attribute corresponding to the VR user's own attribute set by the VR user, an event related to a ticket purchased by the VR user, etc.). In other words, the control unit 310 may not reflect in the virtual space 11 an event not selected by the VR user among multiple events taking place in the real space. In this way, when an unselected event is not reflected in the virtual space 11, there may be an event that is reflected regardless of the user's selection. In other words, "not reflecting an unselected event in the virtual space 11" means that there is at least one event that is reflected in the virtual space 11 when a selection is made, and is not reflected in the virtual space 11 when no selection is made. Note that the selection of an event may be made before the occurrence (in other words, start) of a specific event selected by the VR user, or may be made after the occurrence (in other words, while the specific event is taking place).

The information processing system 100 of the second embodiment and the information processing system 100 of the third embodiment can also be applied to games that utilize both real space and virtual space. A specific example is shown below.

(Exploration game)
For example, the information processing system 100 may be applied to a game in which a character moving in the virtual space 11 is searched for in the real world (hereinafter, referred to as a "search game"). Specifically, for example, first, a virtual space 11 that imitates a theme park existing in the real world (in other words, a virtual space 11 corresponding to a theme park) is prepared. In addition, the VR-side object control unit 920 moves a predetermined character (hereinafter, referred to as a "search target character"; in other words, a search target object) so as to move around in the virtual space 11. The search target character may be one that is operated by a VR user (in other words, one that moves based on the operation of the VR user), or one that moves without the operation of a user. In other words, the search target character may be a VR avatar 25 or a so-called non-player character. In addition, the appearance of the search target character may be, for example, the appearance of a character belonging to a theme park (for example, a character that symbolizes a theme park, etc.).

The MR system 400 (in other words, the control unit 510) displays the search target character on the display 411 when the MR user 6 is in a position in real space corresponding to the position where the search target character is located in the virtual space 11. In other words, the MR system 400 does not display the search target character on the display 411 when the MR user 6 is not in a position in real space corresponding to the position where the search target character is located in the virtual space 11. In other words, this search game is a game in which the player searches for the search target character moving around in the virtual space 11 in the real space.

In addition, the MR side input receiving unit 816 receives an input related to a specific operation by the MR user 6 in a situation where the MR user 6 is in a position in the real space corresponding to the position where the search target character exists in the virtual space 11 (in other words, a situation where the search target character is displayed on the display 411). The specific operation may be, for example, an operation of taking a photo of the search target character shown on the display 411 (in other words, an operation of storing an image). In other words, the specific operation here can also be an operation related to storing information indicating that the search target character has been found. Note that, in this case, the information indicating that the search target character has been found is a photo of the search target character, but it may also be a flag that is set when the search target character is found. In addition, the information may be automatically stored without the intervention of a specific operation. In other words, the information indicating that the search target character has been found may be stored based on the MR user 6 visiting a position in the real space corresponding to the position where the search target character exists in the virtual space 11. That is, the control unit 510 stores information indicating that the operation target character has been discovered in the storage unit 511 based on the MR user 6 visiting that position in real space.

The control unit 510 also grants a bonus to the MR user 6 based on the MR user's visit to the position in real space (in other words, when information indicating that the search target character has been discovered is stored). The bonus may be an item that can be used in the virtual space 11. The bonus may also be an electronic certificate or trophy indicating that the search target character has been discovered. The bonus may also be an electronic ticket that can be exchanged for a real object (e.g., merchandise of the search target character) in real space (e.g., at a real theme park).

In addition, in this search game, the VR user may be able to transmit information to the MR user 6 to help with the search. The VR user may be the user who operates the search target character, or may be a user different from the user. Specifically, for example, the VR user operates his/her own VR avatar 25 in the virtual space 11 to search for the search target character in the virtual space 11 simulating a theme park. The VR side input receiving unit 916 also receives an input related to an operation (e.g., an operation on the controller 280) by the VR user to send sighting information of the search target character (e.g., information on the place where the sighting occurred. In other words, information that helps with the search) to the MR user 6. The control unit 510 also receives the sighting information via the communication control unit 850, and causes a predetermined output means to execute an output that helps with the search based on the sighting information. Specifically, for example, the control unit 510 causes the display 411 to display a display informing the user of the location in the virtual space 11 where the search target character has been sighted, and causes the speaker 416 to output a sound informing the user of the location where the search target character has been sighted. This configuration makes it possible for an MR user 6 who is actually at a theme park and a VR user who is not actually at the theme park (for example, at home) to cooperate with each other to enjoy a game.

(Survival game)
Also, for example, the information processing system 100 may be applied to a survival game in which a user playing in the virtual space 11 and a user playing in the real space can compete against each other. Specifically, for example, first, a venue where a user on the real space side plays and a virtual space 11 simulating the venue (in other words, a virtual space 11 simulating the venue) are prepared. The venue may be a so-called survival game field or a predetermined area such as a city. Also, a user on the real space side wears an MR device 410, and a user on the virtual space 11 side wears a VR device 210.

The control unit 310 of the VR system 200 controls the VR avatar 25 of the VR user based on the input from the VR user received by the VR side input receiving unit 916. Specifically, the control unit 310 moves the VR avatar 25 within the virtual space 11 based on the input from the VR user.

The control unit 310 of the VR system 200 receives the input from the MR user 6 received by the MR side input receiving unit 816 via the communication control unit 950 and controls the MR avatar 26. Specifically, the control unit 310 moves the MR avatar 26 in the virtual space 11 based on the position information (i.e., the position information indicating the position of the MR user 6 in the real space) acquired by the GPS sensor (in other words, the position detection means) equipped in the MR device 410. More specifically, the control unit 310 moves the MR avatar 26 so that the MR avatar 26 is located at a position in the virtual space 11 corresponding to the position of the MR user 6 in the venue on the real space side. That is, on the display 211 of the VR device 210, a display is provided in which the MR 6 user appears to be present at a position in the virtual space 11 (in other words, within the venue) corresponding to the position of the MR user 6 in the mixed reality space 21. In addition, in order to obtain the position of the MR user 6 more accurately, height information of the MR device 410 may be obtained via a sensor 417 of the MR device 410, etc.

The control unit 510 of the MR system 400 also moves the VR avatar 25 so that the VR avatar 25 is positioned at a position in the mixed reality space 21 that corresponds to the position at which the VR avatar 25 is located in the virtual space 11. That is, the display 411 of the MR device 410 provides a display that makes it appear as if the VR user is present at a position in the mixed reality space 21 (in other words, within the venue) that corresponds to the position at which the VR user is located in the virtual space.

The VR-side input receiving unit 916 also receives inputs related to attacks by the VR user against the MR user 6 (in other words, the MR avatar 26). The MR-side input receiving unit 816 also receives inputs related to attacks by the MR user 6 against the VR user (in other words, the VR avatar 25). Note that an attack by the VR user or MR user 6 against an opponent may involve, for example, firing a gun at the opponent, and the input operation related to the attack may be performed via a gun-shaped controller 280, 480, etc.

The control unit 310 also determines whether an attack by the VR user against the MR user 6 hits the MR user 6. Specifically, the control unit 310 determines that an attack by the VR user hits the MR user 6 when, for example, the VR user performs an input operation while the gun held by the VR avatar 25 is pointed at the MR avatar 26 and a bullet fired from the gun hits the MR avatar 26 directly. The control unit 310 also determines whether an attack by the MR user 6 against the VR user hits the VR user. Specifically, the control unit 310 determines that an attack by the MR user 6 hits the VR user when, for example, the MR user performs an input operation while the gun held by the MR avatar 26 is pointed at the VR avatar 25 and a bullet fired from the gun hits the VR avatar 25 directly. Here, the direction of the gun held by the MR avatar 26 is linked to the direction of the controller 480 held by the MR user 6, for example, and the aim of the gun of the MR avatar 26 is aligned with a position in the virtual space 11 corresponding to the position where the MR user 6 aims the controller 480 in the mixed reality space 21. That is, the control unit 310 determines that an attack by the MR user 6 hits the VR user when the MR user 6 performs an input operation while aiming the gun-shaped controller 480 at the VR avatar 25 in the mixed reality space 21. Note that an attack by the VR user against the MR user 6 can be made similar to that of a normal shooting game. Note that a part or all of the determination of an attack by the MR user 6 against the VR user or an attack by the VR user against the MR user 6 may be made by the control unit 510 or the control unit 610, etc. In other words, the determination unit that determines whether an MR user 6 has attacked a VR user, or the determination unit that determines whether a VR user has attacked an MR user 6, may be realized by any of the control unit 310, the control unit 510, and the control unit 610.

Note that, although an example has been described here in which a VR user and an MR user 6 fight as enemies, the survival game (in other words, a battle game) may allow the VR user and the MR user 6 to play cooperatively. That is, for example, in a survival game in which multiple teams (e.g., two teams) made up of multiple users can compete against each other, one of the multiple teams may be made up of only VR users (in other words, users on the virtual space 11 side), and the other team may be made up of only MR users 6 (in other words, users on the real space side). Also, one team may be able to include both a VR user and an MR user 6.

(Escape Game)
Also, for example, the information processing system 100 may be applied to an escape game (in other words, a puzzle-solving game) in which a user on the real space side and a user on the virtual space 11 side cooperate to play. Specifically, for example, first, a venue where the user on the real space side plays and a virtual space 11 simulating the venue (in other words, a virtual space 11 simulating the venue) are prepared. The venue may be a specified area such as a city center, or may be a specific facility. The user on the real space side wears the MR device 410, and the user on the virtual space 11 side wears the VR device 210.

A user wearing the MR device 410 and a user wearing the VR device 210 cooperate to accomplish a set mission. Specifically, the MR device 410 and the VR device 210 display virtual objects related to the mission at corresponding positions. The virtual object may be, for example, an object related to the reception of an action performed by the VR user and the MR user 6 in cooperation. Specifically, when the VR user performs a predetermined action (for example, a hand-waving operation) on a predetermined device as the virtual object displayed on the VR device 210, and the MR user 6 performs a predetermined action (for example, a hand-waving operation. Note that the action may be the same as or different from the action performed by the VR user), the predetermined device may be started (in other words, a game (event) may progress), or the like. The virtual object may also be, for example, an item that can be acquired by at least one of the VR user or the MR user 6, such as a treasure chest. The virtual object may also be an object that includes information about solving the puzzle (for example, a sentence about the puzzle or hints for solving the puzzle).

In addition, in a game executed using such an information processing system 100, in which an MR user 6 and a VR user cooperate to play, a second action related to the progress of the game by the MR user 6 or the VR user may be possible on the condition that one of the MR user 6 and the VR user has performed a first action related to the progress of the game. In this case, for example, the first action is an action that can only be performed by the one user and cannot be performed by the other user. Also, for example, the second action is an action that can only be performed by the other user and cannot be performed by the one user. Specifically, for example, when the MR system 400 (in other words, the control unit 510) detects that the MR user 6 has visited a specific place in the real space (for example, when GPS information indicating that the specific place has been visited is input to the MR side input reception unit 816), the VR system 200 (in other words, the control unit 310) may receive an input related to the second action by the VR user. In other words, the second action by the VR user may be an action that can be executed based on the MR user 6 visiting a specific place in the real space. Also, for example, when the VR system 200 (in other words, the control unit 310) detects that the VR user has visited a specific place in the virtual space 11, the MR system 400 (in other words, the control unit 510) may accept an input related to the second action by the MR user 6. In other words, the second action by the MR user 6 may be an action that can be executed based on the VR user visiting a specific place in the virtual space 11. Also, for example, the VR device 210 may display a predetermined display related to the progress of the game (for example, a display of a hint related to solving a puzzle) on the display 211 only while the MR user 6 is pointing the camera of the smartphone as the MR device 410 at a specific place in the real space in the real space. Also, the virtual space 11 may represent the past of the mixed reality space 21, and the VR system 200 may receive input related to a predetermined action (e.g., an action of placing an item at a specific position in the virtual space 11) as a first action by the VR user in the virtual space 11 (in other words, the past). Then, when the VR user performs a first action in the virtual space 11 (in other words, the past), the MR system 400 may receive input related to a predetermined action (e.g., an action of going to a position in the mixed reality space 21 corresponding to the specific position and finding the item) as a second action by the MR user 6 in the real space (in other words, the present), and may be configured to progress the game (in other words, an event) based on the input.

The present invention is not limited to the above-described embodiment, and can be modified in various ways without departing from the spirit of the invention. Within the scope of the invention, the components can be freely combined, any component can be modified, or any component can be omitted. The process flow described in this specification is merely an example, and the order and configuration of each process can be different.

<Additional Notes>
The matters described in the above embodiment can also be described as follows.

(Appendix 1)
A first control means (e.g., a control unit 310) for providing a first view, which is a view of a virtual space, to a first user;
A second control means (e.g., a control unit 510) for providing a second user with a second view, which is a view of a mixed reality space in which a virtual object is arranged in a real space;
An input receiving unit (e.g., an MR side input receiving unit 816) that receives position information indicating a current position of the second user in the real space;
A correspondence relationship storage means (e.g., a storage unit 611) that stores information indicating a correspondence relationship between positions in the real space and the virtual space,
the first control means provides the first user with the first view of a predetermined event taking place at a specific position in the virtual space;
The second control means is
providing the second view to the second user according to the location information;
When the location information indicates that the second user is in a location corresponding to the specific location, the information processing system provides the second user with the second view reflecting the specified event taking place at the specific location in the virtual space.
According to this configuration, when the second user is at a predetermined position in the real space, it is possible to provide the second user with a view of the mixed reality space reflecting a predetermined event taking place at a specific position in the virtual space corresponding to the predetermined position. Therefore, it is possible to make an event taking place at a specific position in the virtual space visible at a position corresponding to the specific position in the real space. This expands the range of uses of the virtual space.

(Appendix 2)
the first control means moves a virtual object related to the predetermined event in the virtual space based on an operation of the first user;
The second control means reflects a movement of a virtual object moved by an operation of the first user in the second view, and provides the second user with the second view in which the predetermined event is reflected.
According to this configuration, the second user can see in the real space the movement of the virtual object that the first user moves in the virtual space, which provides the user with an unprecedented experience and expands the range of uses for the virtual space.

(Appendix 3)
The information processing system described in Appendix 1 is provided with a reward granting means (e.g., a control unit 510) that grants a reward to the second user based on the second user visiting a location in the real space that corresponds to the specific location.
According to this configuration, a reward can be given to the second user who visits a position in the real space where a specific event taking place in the virtual space is reflected. Therefore, the second user can be motivated to visit a position in the real space that corresponds to a specific position, and the range of uses of the virtual space can be expanded.

(Appendix 4)
The information processing system according to any one of appendices 1 to 3, wherein the specified event reflected in the second view is an event selected by the second user from a plurality of events taking place in the virtual space.
According to this configuration, the second user can select the event to be reflected from among a plurality of events by himself/herself, which can prevent the second user from having too many events reflected in the mixed reality space provided to the second user, causing the processing load to become too heavy, or preventing the second user's field of vision from becoming cluttered.

(Appendix 5)
a right information storage means (e.g., a storage unit 511) for storing information regarding a right to be provided with the second view reflecting the predetermined event;
The right is granted to the second user in exchange for a fee;
The information processing system according to any one of appendices 1 to 3, wherein the second control means provides the second user with the second view in which the specified event is reflected, when the second user has the right.
According to this configuration, it is possible to charge a fee to a second user who wishes to view a specific event in the real space, thereby expanding the range of uses for the virtual space.

(Appendix 6)
The second control means provides the second user with the second view reflecting the predetermined event related to the first user selected by the second user from among a plurality of users.
According to this configuration, the second user can select which of the multiple users using the information processing system the second user wants to receive events related to, thereby expanding the range of uses of the virtual space.

(Appendix 7)
The information processing system according to any one of appendices 1 to 3, wherein the second control means provides the second user with the second view in which the predetermined event having an attribute corresponding to an attribute set for the second user is reflected.
According to this configuration, it is possible to provide a view that reflects events suitable for the second user in accordance with the attributes of the second user, thereby expanding the range of uses for the virtual space.

(Appendix 8)
A first control means (e.g., a control unit 310) for providing a first view, which is a view of a virtual space, to a first user;
A second control means (e.g., a control unit 510) for providing a second user with a second view, which is a view of a mixed reality space in which a virtual object is arranged in a real space;
An input receiving unit (e.g., an MR side input receiving unit 816) that receives position information indicating a current position of the second user in the real space;
a correspondence relationship storage unit (e.g., a storage unit 611) that stores information indicating a correspondence relationship between positions in the real space and the virtual space; and a program that causes a computer of an information processing system to function as the second control unit,
the first control means provides the first user with the first view of a predetermined event taking place at a specific position in the virtual space;
The second control means is
providing the second view to the second user according to the location information;
A program for providing the second user with the second view reflecting the specified event taking place at the specific location in the virtual space when the location information indicates that the second user is in a location corresponding to the specific location.
According to this configuration, it is possible to achieve the same effects as the information processing system described in Supplementary Note 1.

11 Virtual space, 21 Mixed reality space, 25 VR avatar, 26 MR avatar, 30 Virtual model, 35 Blackboard, 36 Virtual blackboard, 50 Event-related object, 100 Information processing system, 200 VR system, 210 VR device, 211 Display, 212 Gaze sensor, 213 First camera, 214 Second camera, 215 Microphone, 216 Speaker, 217 Sensor, 260 Detection device, 270 Display, 280 Controller, 286 Sensor, 300 Computer, 301 Processor, 302 Memory, 303 Storage, 304 Input/output interface, 305 Communication interface, 310 Control unit, 311 Storage unit, 400 MR system, 410 MR device, 411 Display, 412 Gaze sensor, 413 Camera, 415 Microphone, 416 Speaker, 417 Sensor, 460 Detection device, 470 Display, 480 Controller, 486 Sensor, 500 Computer, 501 Processor, 502 Memory, 503 Storage, 504 Input/Output interface, 505 Communication interface, 510 Control unit, 511 Storage unit, 600 Server, 601 Processor, 602 Memory, 603 Storage, 604 Input/Output interface, 605 Communication interface, 610 Control unit, 611 Storage unit, 700 External device, 810 Virtual space generation unit, 812 Coordinate definition unit, 816 MR side input reception unit, 818 User information acquisition unit, 820 MR side object control unit, 840 Display control unit, 845 Sound control unit, 850 Communication control unit, 916 VR side input reception unit, 920 VR side object control unit, 930 Virtual camera control unit, 940 Display control unit, 945 Sound control unit, 950 Communication control unit

Claims (8)

a first control means for providing a first view, the first view being a view of a virtual space, to a first user;
a second control means for providing a second user with a second view, the second view being a view of a mixed reality space in which a virtual object is arranged in a real space;
an input receiving means for receiving position information indicating a current position of the second user in the real space;
a correspondence relationship storage means for storing information indicating a correspondence relationship between positions in the real space and the virtual space,
the first control means provides the first user with the first view of a predetermined event taking place at a specific position in the virtual space;
The second control means is
providing the second view to the second user according to the location information;
When the location information indicates that the second user is in a location corresponding to the specific location, the information processing system provides the second user with the second view reflecting the specified event taking place at the specific location in the virtual space. the first control means moves a virtual object related to the predetermined event in the virtual space based on an operation of the first user;
2 . The information processing system according to claim 1 , wherein the second control means reflects a movement of a virtual object moved by an operation of the first user in the second view, and provides the second user with the second view in which the predetermined event is reflected. The information processing system according to claim 1 , further comprising a reward granting means for granting a reward to the second user on the basis that the second user has visited a position in the real space that corresponds to the specific position. The information processing system according to any one of claims 1 to 3, wherein the specified event reflected in the second view is an event selected by the second user from a plurality of events taking place in the virtual space. a right information storage means for storing information regarding a right to receive the second view reflecting the predetermined event,
The right is granted to the second user in exchange for a fee;
The information processing system according to any one of claims 1 to 3, wherein the second control means provides the second view reflecting the specified event to the second user when the second user has the right. The information processing system according to any one of claims 1 to 3, wherein the second control means provides the second user with the second view reflecting the predetermined event related to the first user selected by the second user from among a plurality of users. The information processing system according to any one of claims 1 to 3, wherein the second control means provides the second user with the second view reflecting the predetermined event having an attribute corresponding to an attribute set for the second user. a first control means for providing a first view, the first view being a view of a virtual space, to a first user;
a second control means for providing a second user with a second view, the second view being a view of a mixed reality space in which a virtual object is arranged in a real space;
an input receiving means for receiving position information indicating a current position of the second user in the real space;
a correspondence storage unit configured to store information indicating a correspondence relationship between positions in the real space and the virtual space,
the first control means provides the first user with the first view of a predetermined event taking place at a specific position in the virtual space;
The second control means is
providing the second view to the second user according to the location information;
A program for providing the second user with the second view reflecting the specified event taking place at the specific location in the virtual space when the location information indicates that the second user is in a location corresponding to the specific location.

Images