JP2026018315A - Head-mounted display system - Google Patents

Abstract

A head-mounted display system is provided that allows one user to easily check an image displayed to another user.
[Solution] A head-mounted display system 100 according to the present disclosure includes a first HMD 10 worn by a first user U1 and a second HMD 20 worn by a second user U2. The first HMD 10 includes a display unit 11, a communication unit 12 that acquires from the second HMD 20 a second image IM2 that the second HMD 20 displays to the second user U2, a position detection unit 13 that detects the positional relationship between the first HMD 10 and the second HMD 20, an orientation detection unit 15 that detects the orientation of the first HMD 10, and a control unit 14.
[Selected Figure] Figure 1

Description

This disclosure relates to a head-mounted display system.

The system disclosed in Patent Document 1 processes the operation of a head-mounted display (HMD) user to participate in a virtual reality (VR) scene. The system provides a first view of the VR scene to a first HMD of a first user and receives a request from a second user requesting to participate in the VR scene provided to the first HMD. The system further obtains real-world position and orientation data of a second HMD relative to the first HMD, and then provides a second view of the VR scene based on this data. The system also controls the first and second views independently through changes in their respective positions and orientations while viewing the VR scene.

Special Publication No. 2019-522856

The inventors of the present invention have found the following problems.
The system disclosed in Patent Literature 1 aims to allow other users to participate in a VR scene of one user (host user). In such a system, a first and a second user can participate in a common VR scene by performing multiple processes, such as receiving and responding to requests. Therefore, one user cannot easily check the image displayed to the other user.

This disclosure has been made in consideration of the above-mentioned problems, and provides a head-mounted display system that allows one user to easily check the image displayed to the other user.

The head-mounted display system according to the present disclosure comprises:
a first HMD (head mounted display) worn by a first user;
a second HMD worn by a second user;
The first HMD includes:
a display unit that displays a first video to the first user;
a communication unit that acquires, from the second HMD, a second image that the second HMD displays to the second user;
a position detection unit that detects a positional relationship between the first HMD and the second HMD;
an orientation detection unit that detects an orientation of the first HMD;
a control unit,
The display unit is configured to enable the first user to view the real space within the field of view, and when the control unit determines, based on the positional relationship and the orientation of the first HMD, that the second user is located within the field of view of the first user, the control unit displays the second image around the real image of the second user viewed by the first user on the display unit.

This disclosure provides a head-mounted display system that allows one user to easily view the image displayed to another user.

1 is a block diagram showing a configuration example of a head-mounted display system according to a first embodiment. FIG. 1 is a block diagram showing a specific example of a head-mounted display system according to a first embodiment. FIG. 10 is a block diagram showing another specific example of the head-mounted display system according to the first embodiment. 10 is a flowchart illustrating an example of a control method for the head-mounted display system according to the first embodiment. 10 is a flowchart illustrating a modified example of the control method for the head-mounted display system according to the first embodiment. 3A and 3B are schematic diagrams showing examples of images that can be viewed through the display units of the first and second HMDs. 10 is a schematic diagram showing a specific example of an image that can be viewed through the display unit of the first HMD. FIG. 10 is a schematic diagram showing a specific example of an image that can be viewed through the display unit of the first HMD. FIG. FIG. 2 is a schematic diagram showing the relationship between the first and second HMDs and the first and second images. 3A and 3B are schematic diagrams showing examples of positions and orientations of first and second HMDs. FIG. 10 is a schematic diagram showing another example of the positions and orientations of the first and second HMDs. FIG. 2 is a schematic diagram showing a specific example of a first image. 3A and 3B are schematic diagrams showing examples of images displayed by the first and second HMDs to the first and second users, respectively; FIG. 2 is a schematic diagram showing a specific example of a first image. FIG. 2 is a schematic diagram showing a specific example of a first image. FIG. 2 is a schematic diagram showing a specific example of a first image. FIG. 2 is a schematic diagram showing a specific example of a first image. FIG. 2 is a schematic diagram showing a specific example of a first image. FIG. 2 is a schematic diagram showing a specific example of a first image. FIG. 2 is a schematic diagram showing a specific example of a first image. FIG. 2 is a schematic diagram showing a specific example of a first image. 3A and 3B are schematic diagrams showing examples of images displayed by the first and second HMDs to the first and second users, respectively; FIG. 2 is a schematic diagram showing a specific example of a first image. FIG. 2 is a schematic diagram showing a specific example of a first image. FIG. 2 is a schematic diagram showing a specific example of a first image. FIG. 2 is a schematic diagram showing a specific example of a first image. FIG. 2 is a schematic diagram showing a specific example of a first image.

Specific embodiments of the present invention will be described in detail below with reference to the drawings. However, the present invention is not limited to the following embodiments. Furthermore, the following description and drawings have been simplified as appropriate for clarity.

First Embodiment
<Configuration example of a head-mounted display system>
An example of the configuration of a head-mounted display system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of a head-mounted display system according to the first embodiment.

As shown in FIG. 1, the head-mounted display system 100 includes a first HMD (head-mounted display) 10 and a second HMD 20. The first HMD 10 and the second HMD 20 may be wearable devices capable of displaying images. The shapes of the first HMD 10 and the second HMD 20 are not limited, and may be, for example, head-mounted, goggle-type, or eyeglass-type devices that can be worn by a user on their head. The head-mounted HMD may have the same configuration as, for example, a helmet or a hat. The first HMD 10 and the second HMD 20 may be, for example, smart glasses or AR (Augmented Reality) glasses. The first HMD 10 may be worn, for example, on the head of a first user U1 shown in FIG. 6. The second HMD 20 may be worn, for example, on the head of a second user U2 shown in FIG. 6. The first HMD 10 and the second HMD 20 may be, for example, a fully immersive HMD (non-transparent HMD) or a transparent HMD, and it is preferable that they are transparent HMDs.

Examples of see-through HMDs include video see-through HMDs and optical see-through HMDs. Video see-through HMDs use an imaging device to capture real-time images of the real space within the user's field of view on a display unit, overlaying another image onto the image. Optical see-through HMDs use an optical system to overlay an image onto the real space within the user's field of view, as viewed by the user through the display unit. The optical system is, for example, a prism or a half mirror.

The first HMD 10 includes a display unit 11, a communication unit 12, a position detection unit 13, a control unit 14, and an orientation detection unit 15.

The communication unit 12 acquires from the second HMD 20 a second image that the second HMD 20 displays to the second user U2. The position detection unit 13 detects the positional relationship between the first HMD 10 and the second HMD 20. The orientation detection unit 15 detects the orientation of the first HMD 10. The control unit 14 determines whether the second user U2 is located within the field of view of the first user U1 based on the positional relationship between the first HMD 10 and the second HMD 20 and the orientation of the first HMD 10.

The control unit 14 may display an image showing the real space within the field of view of the first user U1 on the display unit 11. In such a case, if the control unit 14 determines that the second user U2 is located within the field of view of the first user U1, the control unit 14 causes the display unit 11 to display the second image around the real image of the second user U2 within the image showing the real space within the field of view of the first user U1.

The display unit 11 may also use an optical system (not shown) to allow the first user U1 to view the real space within the first user U1's field of view through the display unit 11. In such a case, when the control unit 14 determines that the second user U2 is located within the first user U1's field of view, it causes the display unit 11 to display the second video around the real image of the second user U2 that can be viewed through the display unit 11.

As a result, the first user U1 can easily check the second image displayed to the second user U2.

The first HMD 10 may further include a gaze detection unit that detects the gaze of the first user U1. The gaze detection unit includes, for example, a light source and an optical sensor. The light source supplies light other than visible light, such as infrared light or ultraviolet light, to the pupil of the first user U1. The optical sensor captures the light reflected from the pupil of the first user U1. The gaze detection unit can detect the gaze of the first user U1 based on this captured light. If the gaze of the first user U1 is directed toward a specific target, it can be determined that the first user U1 is gazing at the specific target. The gaze level, which will be described later, can be calculated based on this detected gaze of the first user U1.

The first HMD 10 may further include a biometric information acquisition unit that acquires biometric information of the first user U1. This biometric information is, for example, information whose value changes depending on the psychological state. The gaze level, which will be described later, can be calculated using this information whose value changes. Examples of such information whose value changes include heart rate, pulse waves, and brain waves. The biometric information acquisition unit may include, for example, a heart rate sensor, a pulse wave sensor, an electroencephalograph, etc.

<Video transparent type>
A specific example of the head-mounted display system according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the specific example of the head-mounted display system according to the first embodiment.

The head-mounted display system 100A shown in FIG. 2 is a specific example of the head-mounted display system 100 shown in FIG. 1. The head-mounted display system 100A includes a first HMD 10A and a second HMD 20A. The first HMD 10A and the second HMD 20A are video-transmitting HMDs.

The first HMD 10A includes a display unit 11A, a communication unit 12A, a position detection unit 13A, a control unit 14A, an orientation detection unit 15A, and an imaging unit 16.

The second HMD 20A may have the same configuration as the first HMD 10A. Specifically, the second HMD 20A includes a display unit 21, a communication unit 22, a position detection unit 23, a control unit 24, an orientation detection unit 25, and an imaging unit 26. In the head-mounted display system 100A, the configuration of the first HMD 10A is the same as that of the second HMD 20A. The configuration of the first HMD 10A will be described in detail below, but the display unit 21, communication unit 22, position detection unit 23, control unit 24, orientation detection unit 25, and imaging unit 26 have the same configuration as the display unit 11A, communication unit 12A, position detection unit 13A, control unit 14A, orientation detection unit 15A, and imaging unit 16, respectively.

When the first user U1 wears the first HMD 10A, the imaging unit 16 captures an image of the real space within the field of view of the first user U1 through the display unit 11A. The imaging unit 16 generates a real-space image that represents this captured real space. The imaging unit 16 includes an imaging element such as a CMOS (Complementary Metal-Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor.

The display unit 11A is a specific example of the display unit 11 shown in FIG. 1 and is, for example, a flat panel display such as a liquid crystal monitor or an organic EL (Electro-Luminescence) monitor. When the first HMD 10A is worn by, for example, the first user U1 shown in FIG. 6, the display unit 11A is positioned in front of the head of the first user U1. The display unit 11A displays a first image, which includes a real-space image generated by the imaging unit 16. The first image is generated by superimposing another image on the real-space image generated by the imaging unit 16.

The communication unit 12A is a specific example of the communication unit 12 shown in FIG. 1 and has an interface capable of communicating with other devices, including the second HMD 20A. This interface can be a known interface such as wireless LAN or Bluetooth (registered trademark). The communication unit 12A can receive content from other devices (not shown). The communication unit 12A can also acquire visual information of the second user U2 from the second HMD 20A by communicating with the communication unit 22 of the second HMD 20A. This visual information of the second user U2 is information about the image displayed on the second HMD 20A and viewed by the second user U2. In other words, the visual information is information about the second image displayed on the display unit 21 or the second content included in the second image. The communication unit 12A communicates with the communication unit 22 of the second HMD 20A, allowing the first HMD 10A and the second HMD 20A to share visual information and preferably also auditory information. The communication unit 12A acquires from the second HMD 20A a second image to be displayed by the second HMD 20A to the second user U2.

The position detection unit 13A is a specific example of the position detection unit 13 shown in FIG. 1 and detects the positional relationship between the first HMD 10A and the second HMD 20A. The position detection unit 13A may detect the positional relationship between the first HMD 10A and the second HMD 20A using a known position detection means. Examples of such position detection means include "outside-in tracking," in which the first HMD 10A acquires position information for the first HMD 10A and the second HMD 20A via the communication unit 12A, based on position information detected by a device other than the first HMD 10A or the second HMD 20A using a camera sensor or other device, and "inside-out tracking," in which the first HMD 10A acquires position information for the other HMD (the second HMD 20A) using a camera sensor (not shown) mounted on the first HMD 10A. The positional relationship between the first HMD 10A and the second HMD 20A includes the distance between the first HMD 10A and the second HMD 20A, etc.

The orientation detection unit 15A is a specific example of the orientation detection unit 15 shown in FIG. 1 and detects the orientation of the first HMD 10A. The orientation detection unit 15A may detect the orientation of the first HMD 10A using, for example, a gyro, acceleration sensor, or magnetic sensor.

The control unit 14A is a specific example of the control unit 14 shown in FIG. 1, and can be configured with a processor such as a CPU (Central Processing Unit). In other words, the control unit 14A can have the functions of a computer. The control unit 14A can execute programs stored in a storage device such as an internal memory (not shown) and perform various processes. The control unit 14A can, for example, obtain input from a first user U1, etc., and display any video or content on the display unit 11A.

The control unit 14A of the first HMD 10A causes the display unit 11A to display a first image. The first image includes the aforementioned real space image and first content. The first image is generated by overlaying the first content on the aforementioned real space image. The control unit 14A obtains the first content from another device (not shown) via the communication unit 12A and generates the first image. The first image may include multiple contents. In this case, the multiple contents displayed are referred to as a first content group. The first content may also be stored in a memory unit within the first HMD 10A (not shown). Similarly, the control unit 24 of the second HMD 20A causes the display unit 21 to display a second image. The second image includes the second content. Similar to the first image, the second image may be generated by superimposing the second content onto a real-space image that shows the real space within the field of view of the second user U2 through the display unit 21 when the second user U2 is wearing the second HMD 20A. The first content and the second content may be of a wide variety, including, for example, movies, music, photos, comics, animations, computer games, videos, websites, etc. The first and second images may also include icon images, etc. for operation. Furthermore, a background image may be displayed as a common background for the first content or second content and the icon images, etc.

The control unit 14A determines whether the second user U2 is located within the field of view of the first user U1 based on the positional relationship between the first HMD 10A and the second HMD 20A detected by the position detection unit 13A and the orientation of the first HMD 10A detected by the orientation detection unit 15A. If the second user U2 is located within the field of view of the first user U1, the first user U1 will be able to see the real image of the second user U2 via the display unit 11A. The control unit 14A estimates the position of the real image of the second user U2 on the display unit 11A based on the positional relationship and the orientation of the first HMD 10. If the control unit 14A determines that the second user U2 is located within the field of view of the first user U1, it displays the second image around the real image of the second user U2 in the first image on the display unit 11A. Furthermore, the control unit 14A may change at least one of the size, resolution, and transparency of the second image based on the distance between the first HMD 10A and the second HMD 20A detected by the position detection unit 13A. When the control unit 14A determines that the second user U2 is located within the field of view of the first user U1, the control unit 14A may cause the display unit 11A to display the first content at a position separated from the real image of the second user U2 in the first image. Furthermore, the control unit 14A may cause the display unit 11A to display only the second content included in the second image around the real image of the second user U2 in the first image.

The control unit 14A may also acquire a first gaze degree indicating the degree to which the first user U1 is gazing at the first content. The control unit 14A can calculate and acquire the first gaze degree, for example, based on the result of detecting the first user U1's gaze or the first user U1's biometric information. The first HMD 10 may be equipped with the gaze detection unit and biometric information acquisition unit described above. The gaze detection unit may detect the first user U1's gaze. The biometric information acquisition unit may acquire the first user U1's biometric information. The first gaze degree may be calculated, for example, based on the length of time the first user U1's gaze is directed toward the first content. The first gaze degree may also be calculated, for example, based on whether the first user U1's biometric information is a value indicating a psychological state corresponding to the first user U1 gazing at the first content.

Similarly, the control unit 14A may acquire a second gaze degree indicating the degree to which the second user U2 is gazing at the second content. The control unit 14A may acquire the second gaze degree from the second HMD 20A, for example, via the communication unit 12A. The control unit 24 of the second HMD 20A can calculate and acquire the second gaze degree based on the result of detecting the line of sight of the second user U2 and the biometric information of the second user U2. The second HMD 20A may be equipped with the above-mentioned gaze detection unit and biometric information acquisition unit. The gaze detection unit may detect the line of sight of the second user U2. The biometric information acquisition unit may acquire the biometric information of the second user U2. The second gaze degree can be calculated, for example, according to the length of time that the line of sight of the second user U2 is directed at the second content. Furthermore, the second gaze level can be determined, for example, depending on whether the biometric information of the second user U2 has a value indicating a psychological state corresponding to the second user U2 gazing at the second content.

The control unit 14A may change at least one of the size, resolution, and transparency of the first and second content based on the first or second degree of attention. The control unit 14A may change at least one of the size, resolution, and transparency of the first and second content, for example, by comparing the first or second degree of attention with a predetermined value. For example, when the first degree of attention is higher than the predetermined value, the first user U1 is gazing at the first content. In such a case, the control unit 14A may lower at least one of the resolution and transparency of the second image and display the second image so as not to interfere with viewing. Furthermore, for example, when the second degree of attention is higher than the predetermined value, the second user U2 is concentrating on viewing the second content. In such a case, the control unit 14A may increase at least one of the resolution and transparency of the second image and display the second image so that the first user U1 can understand the content that the second user U2 is gazing at.

Furthermore, when the control unit 14A determines that the real image of the second user U2 is located within the central region of the first image, it may change at least one of the size, resolution, and transparency of the first and second content. The central region of the first image may be a predetermined region that encompasses the center of the first image. When the first image is displayed on the entire surface of the display unit 11A, the central region of the first image becomes the central region of the display unit 11A. Furthermore, since the first image is generated by superimposing another image on the aforementioned real space image, the central region of the first image becomes a predetermined region at the center of the aforementioned real space image. Furthermore, when the control unit 14A determines that the real image of the second user U2 is located outside the central region of the first image, it may change at least one of the size, resolution, and transparency of the first and second content as the real image of the second user U2 in the first image approaches the central region of the first image. For example, if the control unit 14A determines that the real image of the second user U2 is located within the central region of the first image, the first user U1 is gazing at the second user U2. In such a case, the control unit 14A may increase the resolution of the second image, decrease its transparency, or both, before displaying the second image in order to grasp the situation of the second user U2 with higher accuracy. Furthermore, for example, if the control unit 14A determines that the real image of the second user U2 is located outside the central region of the first image, the first user U1 is not gazing at the second user U2. In such a case, the control unit 14A does not need to grasp the situation of the second user U2 with high accuracy, and may decrease the resolution of the second image, increase its transparency, or both, before displaying the second image. In this embodiment, the control unit 14A determines whether the second user U2 is located within the field of view of the first user U1 based on the positional relationship between the first HMD 10A and the second HMD 20A detected by the position detection unit 13A and the orientation of the first HMD 10A detected by the orientation detection unit 15A. However, the control unit 14A may also determine whether the second user U2 is located within the field of view of the first user U1 based on a real space image generated by the imaging unit 16.

<Optical transmission type>
Another specific example of the head mounted display system according to the first embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing another specific example of the head mounted display system according to the first embodiment.

The head-mounted display system 100B shown in FIG. 3 is a specific example of the head-mounted display system 100 shown in FIG. 1. The head-mounted display system 100B includes a first HMD 10B and a second HMD 20B. The first HMD 10B and the second HMD 20B are optically transmissive HMDs.

The first HMD 10B includes a display unit 11B, a communication unit 12A, a position detection unit 13A, a control unit 14B, and an orientation detection unit 15A.

The second HMD 20B may have the same configuration as the first HMD 10B. Specifically, the second HMD 20B includes a display unit 21B, a communication unit 22, a position detection unit 23, a control unit 24B, and an orientation detection unit 25. In the head-mounted display system 100B, the configuration of the first HMD 10B is the same as that of the second HMD 20B. The configuration of the first HMD 10B is described in detail below, but the display unit 21B and control unit 24B have the same configuration as the display unit 11B and control unit 14B, respectively.

Display unit 11B is a specific example of display unit 11 shown in FIG. 1. Display unit 11B guides at least a portion of visible light from the real space within the field of view of first user U1 to the eyes of first user U1. This allows first user U1 to view the real space within the field of view of first user U1 through display unit 11B. Display unit 11B uses an optical system to superimpose a third image, described below, onto the real space within the field of view of first user U1 as viewed through display unit 11B. The optical system is a prism, a half mirror, or the like.

Control unit 14B is a specific example of control unit 14 shown in Figure 1. Control unit 14B has the same configuration as control unit 14A, except that it displays a third image instead of the first image.

The control unit 14B causes the display unit 11B to display the third image. The control unit 14B may acquire the third image from another device (not shown) via the communication unit 12A. The third image, like the first image, includes the first content. Unlike the first image, the third image does not include the real space image generated by the imaging unit 16, as described above. When the control unit 14B determines that the second user U2 is located within the field of view of the first user U1, it causes the display unit 11B to display the second image around the real image of the second user U2 visible through the display unit 11B. The control unit 14B may also change at least one of the size, resolution, and transparency of the second image based on the distance between the first HMD 10A and the second HMD 20A detected by the position detection unit 13A. When the control unit 14B determines that the second user U2 is located within the field of view of the first user U1, the control unit 14B may cause the display unit 11B to display the first content at a position separated from the real image of the second user U2 that is visible through the display unit 11B. Furthermore, the control unit 14B may cause the display unit 11B to display only the second content included in the second video in the periphery of the real image of the second user U2 that is visible through the display unit 11B.

Furthermore, when the control unit 14B determines that the real image of the second user U2 visible through the display unit 11B is located within the central region of the display unit 11B, it may change at least one of the size, resolution, and transparency of the first and second content. The central region of the display unit 11B may be a predetermined region that encompasses the center of the display unit 11B. Because the first user U1 views the real space within their field of view through the display unit 11B, the central region of the display unit 11B is the same as the predetermined region at the center of the real space image. Furthermore, when the control unit 14B determines that the real image of the second user U2 is located outside the central region of the real space image, it may change at least one of the size, resolution, and transparency of the first and second content as the real image of the second user U2 visible through the display unit 11B approaches the central region of the real space image.

<Example of a method for controlling a head-mounted display system>
Next, an example of a control method for the video see-through type head mounted display system 100A shown in Fig. 2 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of a control method for the head mounted display system according to the first embodiment.

The control unit 14A displays the first image on the display unit 11A of the first HMD 10A (step ST11). The display unit 11A displays the first image, and the first user U1 can view the first image. The position detection unit 13A also begins detecting the positional relationship between the first HMD 10A and the second HMD 20A. The position detection unit 13A may continue to detect the positional relationship between the first HMD 10A and the second HMD 20A from step ST11 to step ST14, which will be described later. The orientation detection unit 15A also begins detecting the orientation of the first HMD 10A. The orientation detection unit 15A may continue to detect the orientation of the first HMD 10A from step ST11 to step ST14, which will be described later.

Next, the control unit 14A determines whether the second user U2 is located within the field of view of the first user U1 (step ST12). Specifically, the control unit 14A determines whether the second user U2 is located within the field of view of the first user U1 based on the positional relationship between the first HMD 10A and the second HMD 20A detected by the position detection unit 13A and the orientation of the first HMD 10A detected by the orientation detection unit 15A. More specifically, the control unit 14A estimates whether the second HMD 20A is located in the direction in which the first HMD 10A is facing from the first HMD 10A based on the positional relationship between the first HMD 10A and the second HMD 20A and the orientation of the first HMD 10A. If the control unit 14A estimates that the second HMD 20A is located in the direction of the first HMD 10A from the first HMD 10A, it determines that the second user U2 is located within the field of view of the first user U1. If the control unit 14A determines that the second user U2 is not located within the field of view of the first user U1 (step ST12: NO), it repeats step ST12.

When the control unit 14A determines that the second user U2 is located within the field of view of the first user U1 (step ST12: YES), the control unit 14A further displays a second image on the display unit 11A of the first HMD 10A (step ST13). Specifically, the communication unit 12A acquires the second image as visual information from the second HMD 20A, and the control unit 14A displays the acquired second image around the real image of the second user U2 within the first image on the display unit 11A.

The control unit 14A then determines whether the second user U2 is located outside the field of view of the first user U1 (step ST14). Specifically, if the control unit 14A estimates, based on the positional relationship between the first HMD 10A and the second HMD 20A and the orientation of the first HMD 10A, that the second HMD 20A is located in the direction in which the first HMD 10A is facing, it determines that the second user U2 is not located outside the field of view of the first user U1 (step ST14: NO), and repeats steps ST13 and ST14.

When the control unit 14A estimates that the second HMD 20A is not located in the direction of the first HMD 10A from the first HMD 10A and determines that the second user U2 is located outside the field of view of the first user U1 (step ST14: YES), the control unit 14A stops displaying the second image on the display unit 11A of the first HMD 10A (step ST15). In other words, the control unit 14A displays only the first image on the display unit 11A of the first HMD 10A. The communication unit 12A may stop acquiring the second image as visual information from the second HMD 20A.

As described above, according to the control method described above, the first user U1 can view the second image displayed around the real image of the second user U2 in the first image. This allows the first user U1 to easily view the second image.

Furthermore, in steps ST12 and ST14 above, the control unit 14 may make the above-mentioned determinations based on the distance between the first HMD 10 and the second HMD 20, in addition to the position of the second user relative to the field of view of the first user. A control method for making such determinations will be described with reference to Figure 5. Figure 5 is a flowchart showing a modified version of the control method shown in Figure 4. In this modified version of the control method shown in Figure 5, steps ST11, ST22, ST13, ST24, and ST15 are executed in this order.

In step ST22, the control unit 14A determines whether the second user U2 is located within the field of view of the first user U1 and whether the distance between the first HMD 10A and the second HMD 20A is less than or equal to a predetermined distance. The positional relationship between the first HMD 10A and the second HMD 20A detected by the position detection unit 13A includes the distance between the first HMD 10A and the second HMD 20A. The control unit 14A can determine the distance between the first HMD 10A and the second HMD 20A from the positional relationship between the first HMD 10A and the second HMD 20A detected by the position detection unit 13A. The predetermined distance may be set in advance. For example, the predetermined distance may be set in advance depending on whether the usage environment of the first HMD 10A and the second HMD 20A is indoors or outdoors. The control unit 14A determines whether the second user U2 is located within the field of view of the first user U1 and whether the distance between the first HMD 10A and the second HMD 20A is equal to or less than a predetermined distance. If the control unit 14A determines that the second user U2 is located within the field of view of the first user U1 and that the distance between the first HMD 10A and the second HMD 20A is equal to or less than the predetermined distance (step ST22: YES), the control unit 14A proceeds to step ST13. On the other hand, if the control unit 14A determines that the second user U2 is not located within the field of view of the first user U1 or that the distance between the first HMD 10A and the second HMD 20A is not equal to or less than the predetermined distance (step ST22: NO), the control unit 14A repeats step ST22.

In step ST24, the control unit 14A determines whether the second user U2 is located outside the field of view of the first user U1, or whether the distance between the first HMD 10A and the second HMD 20A exceeds a predetermined distance. If the control unit 14A determines that the second user U2 is located outside the field of view of the first user U1, or that the distance between the first HMD 10A and the second HMD 20A exceeds a predetermined distance (step ST24: YES), the control unit 14A proceeds to step ST15. On the other hand, if the control unit 14A determines that the second user U2 is not located outside the field of view of the first user U1, or that the distance between the first HMD 10A and the second HMD 20A does not exceed a predetermined distance (step ST24: NO), steps ST13 and ST24 are repeated.

In addition, in the control method shown in FIG. 4, the control unit 14A of the first HMD 10A and the like executes steps ST11 to ST15 described above, but the control unit 24 of the second HMD 20A may also execute steps ST11 to ST15 in parallel with steps ST11 to ST15 described above. That is, in parallel with steps ST11 to ST15 described above, the control unit 24 of the second HMD 20A may cause the display unit 21 to display the first image and the like, or stop the display. Similarly, in the control method shown in FIG. 5, the control unit 24 of the second HMD 20A may also execute steps ST11, ST22, ST13, ST24, and ST15 described above.

The control methods shown in Figures 4 and 5 can also be applied to the optically transmissive head-mounted display system 100B shown in Figure 3. In the control method applied to the head-mounted display system 100B shown in Figure 3, the first user U1 views the real space within his or her field of view through the display unit 11B, and in each step, the control unit 14B appropriately displays a third image on the display unit 11B. Also, in step ST13, the control unit 14B displays the second image in the periphery of the real image of the second user visible through the display unit 11B.

(Specific example of processing)
<Processing by optical transmission type HMD>
Next, a specific example of steps ST11 to ST13 of the processing by the head-mounted display system 100B shown in Fig. 3 will be described with reference to Figs. 6 to 8. As described above, the head-mounted display system 100B includes a first HMD 10B and a second HMD 20B, which are optically transmissive HMDs. Note that in the following specific example, the first HMD 10B is an optically transmissive HMD, and the first user U1 can view the real space in front of them through the display unit 11B of the first HMD 10B.

In step ST11, the first user U1 views the real space image R1 through the display unit 11B. Furthermore, the control unit 14B shown in FIG. 3 causes the third image IM31 shown in FIG. 6 to be superimposed on the real space image R1 and displayed on the display unit 11B. The first user U1 views the real space image R1 through the display unit 11B and the third image IM31 superimposed on the display surface 11c of the display unit 11B. The third image IM31 includes the first content C1. The third image IM31 may include multiple copies of the first content C1. When the third image IM31 includes only one copy of the first content C1, the first content C1 may be displayed across the entire third image IM31 or may be positioned near the center of the third image IM31. The first user U1 is viewing the third video IM31, and the second user U2 is viewing the second video M21.

In step ST12, the control unit 14B determines that the second user U2 is located within the field of view of the first user U1 (ST12: YES) and proceeds to step ST13.

Next, in step ST13, as shown in FIG. 8, the control unit 14B causes the display unit 11B to display the second image IM21 in the vicinity of the real image of the second user U2 visible through the display unit 11B. The control unit 14B may also cause the third image IM31 to be displayed in the display unit 11B at a position spaced apart from the real image of the second user U2 and the second image IM21. Note that the control unit 14B may also display the second content C2, rather than the second image IM21, in the vicinity of the real image of the second user U2 visible through the display unit 11B. This allows the first user U1 to easily check the first content C1 and the second content C2.

In this example, the third image IM31 is separated when the second image IM21 is displayed in step ST13. However, as shown in FIG. 7, the second image IM21 may not be displayed, and the third image IM31 may be separated from the second user U2 in the real space image R1. This allows the first user U1 to easily confirm the real image of the second user U2 and the first content C1 on the display surface 11c.

<Processing by video see-through HMD>
Next, specific examples of steps ST11 to ST13 of the processing by the head-mounted display system 100A shown in Figure 2 will be described. As described above, the head-mounted display system 100A includes a first HMD 10A and a second HMD 20A, which are video-transmissive HMDs. Note that in the following specific examples, the first HMD 10A is a video-transmissive HMD, and the display unit 11A of the first HMD 10A displays a real-space image R1 that indicates at least the real space in front of the first user U1.

<First Specific Example>
A first specific example of steps ST11 to ST13 will be described with reference to FIGS.

In step ST11, the control unit 14A shown in FIG. 2 causes the display unit 11A of the first HMD 10A to display the first image IM1 shown in FIG. 9. Similarly, the control unit 24 shown in FIG. 2 causes the display unit 21 of the second HMD 20A to display the second image IM2 shown in FIG. 9. The first image IM1 includes a real space image R1 generated by the imaging unit 16, which is within the field of view of the first user U1 through the display unit 11A when the first user U1 is wearing the first HMD 10A. The first image IM1 may further include first content. The first image IM1 is displayed with the first content C1 superimposed on the real space image R1. The first image IM1 may also be displayed with a background image (not shown) superimposed on the real space image R1, with the first content C1 displayed within the background image. The first user U1 can view the first image IM1, and the second user U2 can view the second image IM2.

As shown in FIG. 10, the first user U1 may face a direction that prevents the second user U2 from being visible. In such a case, in step ST12, the control unit 14A determines that the second user U2 is not located within the field of view of the first user U1 based on the positional relationship between the first HMD 10A and the second HMD 20A and the orientation of the first HMD 10A (ST12: NO). Therefore, the control unit 14A continues to display the first image IM1 on the display unit 11A.

As shown in FIG. 11, the first user U1 may face a direction in which the second user U2 can be seen. In such a case, in step ST12, the control unit 14A determines that the second user U2 is located within the field of view of the first user U1 based on the positional relationship between the first HMD 10A and the second HMD 20A and the orientation of the first HMD 10A (ST12: YES). When the second user U2 is located within the field of view of the first user U1, the first user U1 can see the second user U2 within the real space image R1 in the first image IM1. In step ST13, the control unit 14A displays the second image IM2, which the communication unit 12A has acquired as visual information from the second HMD 20A, around the real image of the second user U2 in the first image IM1a. As shown in FIG. 12, the first image IM1a is displayed with the second image IM2 superimposed around the real image of the second user U2 in the first image IM1 shown in FIG. 9. The first user U1 can check the second image IM2 displayed around the real image of the second user U2 in the first image IM1a. This allows the first user U1 to easily check the second image IM2.

<Second Specific Example>
Next, a second specific example of steps ST11 to ST13 will be described with reference to FIGS.

In step ST11, the control unit 14A shown in FIG. 2 causes the display unit 11A of the first HMD 10A to display the first image IM1b shown in FIG. 13. Similarly, the control unit 24 shown in FIG. 2 causes the display unit 21 of the second HMD 20A to display the second image IM2b shown in FIG. 13. The first image IM1b shows the real space image R1 and first content C1 generated by the imaging unit 16. The second image IM2b shows the real space image R2 and second content C2 generated by the imaging unit 26. The first user U1 is viewing the first image IM1b, and the second user U2 is viewing the second image IM2b.

In step ST12, the control unit 14A determines that the second user U2 is located within the field of view of the first user U1 (ST12: YES) and proceeds to step ST13.

In this embodiment, in step ST13, the second image IM21 is not displayed, and the first content C1 is displayed at a position in the first image IM1c that is spaced apart from the real image of the second user U2. In step ST13, the control unit 14A displays the first image IM1c shown in FIG. 14 on the display unit 11A. When the second user U2 is located within the field of view of the first user U1, the first image IM1c shows the real space image R1 and the first content C1 in front of the first user U1, and the second user U2 is displayed in the real space image R1. In other words, the first user U1 can see the second user U2 who exists in the real space displayed on the display unit 11A. In the first image IM1c, the first content C1 is displayed at a position spaced apart from the real image of the second user U2. This allows the first user U1 to easily confirm the real image of the second user U2 and the first content C1 in the first image IM1c.

In step ST13 of this example, the control unit 14A causes the display unit 11A to display the first content C1 at a position separated from the real image of the second user U2 in the first image IM1c. However, the control unit 14A may also stop the display of the first content C1 by the display unit 11A. This allows the first user U1 to easily confirm the real image of the second user U2 in the first image IM1c.

<Third Specific Example>
Next, a third specific example of steps ST11 to ST13 will be described with reference to FIGS.

Steps ST11 and ST12 are performed in the same manner as in the second specific example described above. Next, in step ST13, the control unit 14A causes the display unit 11A to display the first image IM1d shown in FIG. 15. When the second user U2 is located within the field of view of the first user U1, the first image IM1d shows a real space image R1, first content C1, and second content C2 in front of the first user U1. The second user U2 is displayed in the real space image R1. In other words, the first user U1 can see the second user U2 existing in the real space displayed on the display unit 11A. The control unit 14A causes the display unit 11A to display the first content C1 at a position separated from the real image of the second user U2 in the first image IM1d, while displaying the second content C2 in the periphery of the real image of the second user U2 in the first image IM1d. This allows the first user U1 to easily confirm the first content C1 that he or she is viewing and the second content C2 that the second user U2 is viewing within the first video IM1d.

<Fourth Specific Example>
Next, a fourth specific example of steps ST11 to ST13 will be described with reference to FIGS.

Steps ST11 and ST12 are executed in the same manner as in the third specific example described above. Next, in step ST13, as in the third specific example, the control unit 14A displays the second content C2 in the periphery of the real image of the second user U2 in the first image IM1, and displays the first content C1 at a position separated from the real image of the second user U2 in the first image IM1. Also, in this step ST13, the control unit 14A changes the size of the first content C1 based on the first gaze degree of the first user U1.

For example, when the first gaze level, which indicates the degree to which the first user U1 is gazing at the first content C1, is higher than a predetermined value, the control unit 14A enlarges the size of the first content C1 shown in FIG. 15. The control unit 14A causes the display unit 11A to display the first image IM1e shown in FIG. 16. When the second user U2 is located within the field of view of the first user U1, the first image IM1e shows the real space image R1, the first content C1e, and the second content C2 in front of the first user U1. The second user U2 is displayed in the real space image R1. The first user U1 can see the second user U2 present in the real space displayed on the display unit 11A. The first content C1e is the same as the first content C1 shown in FIG. 15, except for its size. The size of the first content C1e is larger than the size of the first content C1. This allows the first user U1 to view the first content C1e and easily confirm the first content C1 that they are viewing.

For example, when the first degree of attention, which indicates the degree to which the first user U1 is gazing at the first content C1, is lower than a predetermined value, the control unit 14A reduces the size of the first content C1 shown in FIG. 15. The control unit 14A displays the first content C1f shown in FIG. 17. The first content C1f is the same as the first content C1 shown in FIG. 15 except for its size. The size of the first content C1f is smaller than the size of the first content C1. This allows the first user U1 to easily check parts of the first image IM1f other than the first content C1f, such as the second user U2 and the second content C2 that the second user U2 is watching. In this embodiment, the size of the first content C1 is enlarged or reduced when it is determined that the second user U2 is located within the field of view of the first user U1. However, if the first gaze level of the first user U1 is higher or lower than a predetermined value, the size of the first content C1 displayed in the first image IM1 may be enlarged or reduced in step ST11 before the first image IM1 is displayed, regardless of whether the second user U2 is located within the field of view of the first user U1.

<Fifth Specific Example>
Next, a fifth specific example of steps ST11 to ST13 will be described with reference to FIGS.

Steps ST11 and ST12 are executed in the same manner as in the third specific example described above. Next, in step ST13, as in the third specific example, the control unit 14A displays the second content C2 in the periphery of the real image of the second user U2 in the first image IM1, and displays the first content C1 at a position separated from the real image of the second user U2 in the first image IM1. Also, in this step ST13, the control unit 14A changes the resolution of the first content C1 based on the first gaze degree of the first user U1.

For example, the control unit 14A may increase the resolution of the first content C1 shown in FIG. 15 when the first gaze level, which indicates the degree to which the first user U1 is gazing at the first content C1, is higher than a predetermined value. The control unit 14A causes the display unit 11A to display the first image IM1g shown in FIG. 18. When the second user U2 is located within the field of view of the first user U1, the first image IM1g shows the real space image R1, the first content C1g, and the second content C2 in front of the first user U1. The second user U2 is displayed in the real space image R1. In other words, the first user U1 can see the second user U2 existing in the real space displayed on the display unit 11A. The first content C1g is the same as the first content C1 shown in FIG. 15 except for the resolution. The resolution of the first content C1g is higher than the resolution of the first content C1. This allows the first user U1 to view the first content C1g and easily confirm the first content C1 that they are viewing.

For example, the control unit 14A may lower the resolution of the first content C1 shown in FIG. 15 when the first gaze level, which indicates the degree to which the first user U1 is gazing at the first content C1, is lower than a predetermined value. The control unit 14A causes the display unit 11A to display the first image IM1h shown in FIG. 19. When the second user U2 is located within the field of view of the first user U1, the first image IM1h shows the real space image R1, the first content C1h, and the second content C2 in front of the first user U1. The second user U2 is displayed in the real space image R1. In other words, the first user U1 can see the second user U2 existing in the real space displayed on the display unit 11A. The first content C1h is the same as the first content C1 shown in FIG. 15 except for the resolution. The resolution of the first content C1h is lower than the resolution of the first content C1. This allows the first user U1 to lower the resolution of the first content C1, which they do not pay much attention to, making it easier for them to see the surrounding areas of the first content C1h in the first image IM1h, such as the second user U2 and the second content C2 that the second user U2 is viewing.

In addition, in step ST13 of this example, the control unit 14A may change the transparency of the first content C1 based on the first gaze degree of the first user U1. Specifically, in step ST13, the control unit 14A may lower the transparency of the first content C1 shown in FIG. 15 if the first gaze degree of the first user U1 is higher than a predetermined value. Also, the control unit 14A may increase the transparency of the first content C1 shown in FIG. 15 if the first gaze degree of the first user U1 is lower than a predetermined value. In this embodiment, if it is determined that the second user U2 is located within the field of view of the first user U1, the resolution or transparency of the first content C1 is changed. However, if the first gaze degree of the first user U1 is higher or lower than a predetermined value, the resolution or transparency of the first content C1 displayed in the first image IM1 may be changed in step ST11, and the first image IM1 may be displayed, regardless of whether the second user U2 is located within the field of view of the first user U1.

<Sixth Specific Example>
Next, a sixth specific example of steps ST11 to ST13 will be described with reference to FIGS.

Steps ST11 and ST12 are executed in the same manner as in the third specific example described above. Next, in step ST13, as in the third specific example, the control unit 14A displays the second content C2 in the periphery of the real image of the second user U2 in the first image IM1, and displays the first content C1 at a position separated from the real image of the second user U2 in the first image IM1. Also, in this step ST13, the control unit 14A changes at least one of the size, resolution, and transparency of the second content C2 based on a second gaze level indicating the degree to which the second user U2 is gazing at the second content C2.

For example, when the second gaze level, which indicates the degree to which the second user U2 is gazing at the second content C2, is higher than a predetermined value, the control unit 14A reduces the transparency of the second content C2 shown in FIG. 15. The control unit 14A causes the display unit 11A to display the first image IM1i shown in FIG. 20. When the second user U2 is located within the field of view of the first user U1, the first image IM1i shows the real space image R1, the first content C1, and the second content C2i in front of the first user U1. The second user U2 is displayed in the real space image R1. In other words, the first user U1 can see the second user U2 existing in the real space displayed on the display unit 11A. The second content C2i is the same as the second content C2 shown in FIG. 15 except for the transparency. The transparency of the second content C2i is lower than the transparency of the second content C2. This allows the first user U1 to visually recognize the second content C2i and easily confirm the second content C2 being viewed by the second user U2.

For example, when the second gaze level, which indicates the degree to which the second user U2 is gazing at the second content C2, is lower than a predetermined value, the control unit 14A increases the transparency of the second content C2 shown in FIG. 15. The control unit 14A causes the display unit 11 to display the first image IM1j shown in FIG. 21. When the second user U2 is located within the field of view of the first user U1, the first image IM1j shows the real space image R1, the first content C1, and the second content C2j in front of the first user U1. The second user U2 is displayed in the real space image R1. In other words, the first user U1 can see the second user U2 existing in the real space displayed on the display unit 11A. The second content C2j is the same as the second content C2 shown in FIG. 15 except for the transparency. The transparency of the second content C2j is higher than the transparency of the second content C2. This allows the first user U1 to view the second content C2j, easily confirm the second content C2 that the second user U2 is viewing, and understand that the second user U2 is not focusing on the second content C2j.

<Seventh Specific Example>
Next, a seventh specific example of steps ST11 to ST13 will be described with reference to FIGS. 22 to 24 and 27.

In step ST11, the control unit 14A shown in FIG. 2 displays the first image IM1k shown in FIG. 22 on the display unit 11A of the first HMD 10A. Similarly, the control unit 24A shown in FIG. 2 displays the second image IM2b shown in FIG. 22 on the display unit 21 of the second HMD 20A. The first image IM1k shows the real space image R1 in front of the first user U1, the first content C11, and the third content C13. The second image IM2b shows the real space image R2 in front of the second user U2, and the second content C2. The first user U1 can view the first image IM1k, and the second user U2 can view the second image IM2b.

In step ST12, the control unit 14A determines that the second user U2 is located within the field of view of the first user U1 (ST12: YES) and proceeds to step ST13.

In step ST13, the control unit 14A changes at least one of the size, resolution, and transparency of the first content C11 and the third content C13 based on the attention level of the first user U1. The attention level of the first user U1 described above includes two attention levels that indicate the degree to which the first user U1 is gazing at the first content C11 and the third content C13, respectively.

For example, if the first user U1's degree of attention to the first content C11 shown in FIG. 22 is higher than the first user U1's degree of attention to the third content C13, the control unit 14A increases the size of the first content C1 relative to the third content C13. The control unit 14A displays the first image IM1m shown in FIG. 23 on the display unit 11. When the second user U2 is located within the first user U1's field of view, the first image IM1m shows the real space image R1, the first content C11m, and the third content C13m in front of the first user U1. The second user U2 is displayed in the real space image R1. In other words, the first user U1 can see the second user U2 present in the real space displayed on the display unit 11A. The first content C11m is the same as the first content C11 shown in FIG. 22, except for the size. The third content C13m is the same as the third content C13 shown in FIG. 22 except for its size. The size of the first content C11m relative to the third content C13m is larger than the size of the first content C11 relative to the third content C13 shown in FIG. 22. This allows the first content C11m, which is larger in size relative to the third content C13, to be displayed for the first content C11 that receives a high level of attention. The first user U1 can easily view the first content C11m and easily confirm the first content C1 that he or she is viewing.

For example, if the first user U1's degree of attention to the first content C11 shown in FIG. 22 is higher than the first user U1's degree of attention to the third content C13, the control unit 14A may display the first content C11 so that the third content C13 is transparent. The control unit 14A displays the first image IM1n shown in FIG. 24 on the display unit 11. When the second user U2 is located within the first user U1's field of view, the first image IM1n shows the real space image R1, the first content C11m, and the third content C13m in front of the first user U1. The second user U2 is displayed in the real space image R1. In other words, the first user U1 can see the second user U2 present in the real space displayed on the display unit 11A. The third content C13n is the same as the third content C13m shown in FIG. 23, except for the transparency. The transparency of the third content C13n may be set so that the third content C13n is transparent to the first content C11m. The third content C13n is positioned closer to the front of the page than the first content C11m, and a portion of the third content C13n overlaps a portion of the first content C11m. This overlapping portion of the first content C11m transmits a portion of the third content C13n. This allows the first content C11m to be displayed so that the first content C11m is transparent to the third content C13n, making it easier for the first user U1 to view the first content C11m and easily confirm the first content C1 that they are viewing.

In step ST13, the control unit 14A may display at least one of the first content C11 and the third content C13, and stop displaying the remaining content, based on the gaze level of the first user U1 described above. Furthermore, the control unit 14A may display the second content C2 in the vicinity of the real image of the second user U2 in the first image IM1k.

For example, if the first user U1's degree of attention to the first content C11 shown in FIG. 22 is higher than the first user U1's degree of attention to the third content C13, the control unit 14A displays the first content C11 while stopping the display of the third content C13. The control unit 14A causes the display unit 11 to display the first image IM1r shown in FIG. 27. If the second user U2 is located within the first user U1's field of view, the first image IM1r shows the real space image R1, the first content C11, and the second content C2 in front of the first user U1. The second user U2 is displayed in the real space image R1. The first user U1 can see the second user U2 present in the real space displayed on the display unit 11A. The control unit 14A displays the first content C11 while stopping the display of the third content C13. The control unit 14A also displays the second content C2 in the vicinity of the real image of the second user U2 in the first image IM1r. This displays only the first content C11 that the first user U1 is paying close attention to, and displays the second content C2 in the vicinity of the real image of the second user U2. This makes it easy to confirm the second user U2 and the second content C2 that the second user U2 is viewing, along with the first content C11 that the first user U1 is paying attention to.

<Eighth Specific Example>
Next, an eighth specific example of steps ST11 to ST13 will be described with reference to FIGS.

Steps ST11 and ST12 are executed in the same manner as in the third specific example described above. Next, in step ST13, as in the third specific example, the control unit 14A displays the second content C2 in the periphery of the real image of the second user U2 in the first image IM1, and displays the first content C1 at a position separated from the real image of the second user U2 in the first image IM1. Also, in this step ST13, if the control unit 14A determines that the real image of the second user U2 in the first image IM1b is located within the central region of the first image IM1b, it changes at least one of the size, resolution, and transparency of the first content C1 and the second content C2.

For example, if the control unit 14A determines that the real image of the second user U2 in the first image IM1p shown in FIG. 25 is not located within the central region AC of the first image IM1p, it increases the transparency of the second content C2p. The central region AC may be, for example, an area that includes the center point PC of the first image IM1p, and may be arbitrarily determined in advance. The control unit 14A displays the first image IM1p shown in FIG. 25 on the display unit 11A. When the second user U2 is located within the field of view of the first user U1, the first image IM1p shows the real space image R1, the first content C1, and the second content C2p in front of the first user U1. The second user U2 is displayed in the real space image R1. The first user U1 can see the second user U2 existing in the real space displayed on the display unit 11A. The second content C2p is the same as the second content C2 shown in FIG. 15 except for its transparency. The transparency of the second content C2p is higher than that of the second content C2. This allows the first user U1 to view the second content C2p and easily confirm the second content C2 being viewed by the second user U2.

For example, if the control unit 14A determines that the real image of the second user U2 in the first image IM1q shown in FIG. 26 is located within the central region AC of the first image IM1q, it reduces the transparency of the second content C2q. The central region AC may be any region located at the predetermined center point PC of the first image IM1q. The control unit 14A causes the display unit 11A to display the first image IM1q shown in FIG. 26. When the second user U2 is located within the field of view of the first user U1, the first image IM1p shows the real space image R1, the first content C1, and the second content C2q in front of the first user U1. The second user U2 is displayed in the real space image R1. In other words, the first user U1 can see the second user U2 present in the real space of the display unit 11A. The second content C2q is the same as the second content C2 shown in FIG. 15, except for the transparency. The transparency of the second content C2q is lower than the transparency of the second content C2. When the real image of the second user U2 is located within the central region AC of the first image IM1q, it is highly likely that the first user U1 is gazing at the second user U2. Therefore, the first user U1 can easily confirm the second content C2 being viewed by the second user U2 by viewing the second content C2q, which has low transparency. Furthermore, the control unit 14A may increase the size of the second content C2, increase the resolution of the second content C2, or decrease the transparency of the second content C2 as the real image of the second user U2 approaches the central region AC of the first image IM1q. This makes it easier for the second user U2 to confirm the second content C2 being viewed by the second user U2 as the second user U2 approaches the center of the first user U1's field of view. As described above, in the third to eighth specific examples, the control unit 14A displays the second content C2, C2i, C2j, C2p, and C2q in the vicinity of the real image of the second user U2 in the first images IM1d-IM1j and IM1p-IM1r. Simultaneously, the control unit 14A displays the first content C1, C1e-C1h, and C11 at positions spaced apart from the real image of the second user U2 in the first images IM1d-IM1j and IM1p-IM1r. Simultaneously, the control unit 14A changes the size, resolution, and transparency of the first content C1, etc. or the second content C2, etc. Alternatively, the control unit 14A may change the size, resolution, and transparency of the first content C1, etc. or the second content C2, etc. without moving the position of the first content C1, etc. in the first image IM1, etc.

(Other embodiments, etc.)
The head-mounted display systems 100, 100A, and 100B according to the above-described embodiments can be expressed as programs executed by the control units 14, 14A, 14B, 24, and 24B. The above-described control units 14, 14A, 14B, 24, and 24B include a processor and memory (not shown). The configuration of the control units 14, 14A, 14B, 24, and 24B described in the above-described embodiments is realized by the processor reading and executing a program stored in the memory. In other words, this program is a control program that causes the processor to function as the head-mounted display system 100, 100A, and 100B, or as a part thereof.

The above-described program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, optical disk storage, and magnetic storage devices. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable medium or communication medium includes electrical, optical, or other forms of propagated signals.

Furthermore, as described above in the various embodiments regarding the processing procedures in head-mounted display systems 100, 100A, and 100B, the present disclosure may also be embodied as a control method for head-mounted display systems 100, 100A, and 100B. Furthermore, the above-described program can be considered a control program for causing head-mounted display systems 100, 100A, and 100B to execute such a control method.

Note that the present invention is not limited to the above-described embodiments and can be modified as appropriate without departing from the spirit of the present invention. The present invention may also be implemented by appropriately combining the above-described embodiments or examples thereof. For example, the head-mounted display system 100 shown in FIG. 1 includes a first HMD 10 and a second HMD 20, but may also include one or more additional HMDs. Similarly, head-mounted display systems 100A and 100B may also include one or more additional HMDs.

Furthermore, the configuration of the second HMD 20 in the head-mounted display system 100 may be different from the configuration of the first HMD 10. If at least one of the first HMD 10 and the second HMD 20 has the same configuration as the first HMD 10 shown in FIG. 1, the user wearing the other of the first HMD 10 and the second HMD 20 can confirm the displayed image. The same applies to the head-mounted display systems 100A and 100B. For example, the head-mounted display system 100A may include either the first HMD 10B or the second HMD 20B. Similarly, the head-mounted display system 100B may include either the first HMD 10A or the second HMD 20A.

Furthermore, if the first HMD 10 and the second HMD 20 are fully immersive head-mounted displays, the present invention can be implemented by switching them to the above-mentioned video-transparent head-mounted displays. For example, when the control unit 14 determines that the second user is located within the first user's field of view, it causes the display unit 11 to display the real space image R1 (switching to video-transparent type), and causes the real image of the second user and the second image to be displayed around the second user. Furthermore, when the second user is detected within a predetermined distance range of the first user, the control unit 14 causes the display unit 11 to display the peripheral image (switching to video-transparent type), and when it is determined that the second user is located within the field of view, it causes the second image to be displayed around the second user.

100, 100A, 100B Head-mounted display system 10, 10A, 10B First HMD
20, 20A, 20B Second HMD
11, 11A, 11B, 21 Display units 11c, 21c Display surfaces 12, 12A, 22 Communication units 13, 13A, 23 Position detection units 14, 14A, 14B, 24 Control units 15, 15A, 25 Orientation detection unit 16 Imaging unit U1 First user U2 Second user IM1, IM1a to IM1k, IM1m to IM1q First images IM2, IM2b Second images IM31 Third images R1, R2 Real space images C1, C1e to C1h, C11, C11m First content C2, C2i, C2j, C2p, C2q Second content C13, C13m, C13n Third content AC Central region PC Central point

Claims (5)

a first HMD (head mounted display) worn by a first user;
a second HMD worn by a second user;
The first HMD includes:
a display unit that displays a first video to the first user;
a communication unit that acquires, from the second HMD, a second image that the second HMD displays to the second user;
a position detection unit that detects a positional relationship between the first HMD and the second HMD;
an orientation detection unit that detects an orientation of the first HMD;
a control unit,
the display unit is configured to be able to view a real space within a visual field of the first user, and when the control unit determines that the second user is located within the visual field of the first user based on the positional relationship and the orientation of the first HMD, the control unit displays the second video around a real image of the second user viewed by the first user on the display unit.
Head-mounted display system. the control unit changes at least one of a size, a resolution, and a transparency of the second image based on a distance between the first HMD and the second HMD.
The head-mounted display system according to claim 1 . the first video includes a first content;
the second video includes second content;
When the control unit determines that the second user is located within the field of view of the first user based on the positional relationship and the orientation of the first HMD,
displaying the first content at a position separated from the real image of the second user on the display unit, and displaying the second content around the real image of the second user on the display unit, by the display unit;
The head-mounted display system according to claim 1 or 2. the first video includes a first content;
the second video includes second content;
the first HMD acquires a first gaze degree indicating a degree to which the first user gazes at the first content;
the second HMD acquires a second gaze degree indicating a degree to which the second user gazes at the second content;
When the control unit determines that the second user is located within the field of view of the first user based on the positional relationship and the orientation of the first HMD, the control unit changes at least one of a size, a resolution, and a transparency of the first and second contents based on the first gaze degree or the second gaze degree.
The head-mounted display system according to claim 1 or 2. the first video includes a first content;
the second video includes second content;
the control unit changes at least one of a size, a resolution, and a transparency of the first and second contents as the real image of the second user on the display unit approaches a central region of the display unit.
The head-mounted display system according to claim 1 or 2.