JP6496391B1 - Display system and display method - Google Patents

Abstract

An object of the present invention is to provide a display system in which a user using an HMD displays an image different from the displayed image without performing complicated operations and operations. In a system including a display device used by being mounted on a user's head, a background image is displayed on the display screen of the display device according to the orientation of the display device, and the specified orientation is supported. When the visual axis and a predetermined object in the virtual space overlap, a second image different from the first image is displayed superimposed on the first image. [Selection] Figure 2

Description

  The present invention relates to a display system and a display method using a display device mounted on a user's head.

  In recent years, there is a VR (Virtual Reality) technology using a head mounted display (hereinafter referred to as HMD) or the like. For example, a background image that forms a virtual world captured by a virtual camera is displayed on the HMD, and the user feels as if he / she has entered the virtual world (so-called immersion).

  However, the image resolution of the HMD is low, and the user cannot visually recognize the details of the displayed image. Further, when a complicated input operation is requested, there is a risk that the user's immersive feeling may be impaired.

  An object of at least one embodiment of the present invention is to provide a display system in which a user using an HMD displays an image different from the displayed image without performing complicated operations and operations.

  According to a non-limiting viewpoint, the display system according to the present invention is a display system including a display device that is used by being worn on a user's head, and an orientation specifying unit that specifies the orientation of the display device; First display means for displaying a first image indicating the state of the virtual space according to the specified orientation on the display screen of the display device, and a visual axis corresponding to the specified orientation are predetermined objects in the virtual space. And a second display means for displaying a second image different from the first image on the display area generated by being superimposed on the first image.

  According to a non-limiting aspect, the display method according to the present invention is a display method executed in a system including a display device used by being worn on the user's head, and specifies the orientation of the display device. And displaying the first image indicating the state of the virtual space according to the specified orientation on the display screen of the display device, and the visual axis corresponding to the specified orientation is a predetermined object in the virtual space. Is displayed on the display area generated by superimposing on the first image, the second image different from the first image is displayed.

  Each embodiment of the present invention solves one or more deficiencies.

It is a block diagram which shows the structure of the display system corresponding to at least 1 of embodiment of this invention. It is a flowchart of the execution process corresponding to at least 1 of embodiment of this invention. It is a block diagram which shows the structure of the display system corresponding to at least 1 of embodiment of this invention. It is a block diagram which shows the structure of the display apparatus corresponding to at least 1 of embodiment of this invention. It is a figure which shows the orthogonal coordinate system for specifying a user's attitude | position when the display apparatus corresponding to at least 1 of embodiment of this invention is mounted | worn with a head. It is a block diagram which shows the structure of the computer apparatus corresponding to at least 1 of embodiment of this invention. It is a block diagram which shows the structure of the display system corresponding to at least 1 of embodiment of this invention. It is an example of the execution screen corresponding to at least 1 of embodiment of this invention. It is a figure showing the concept of virtual space corresponding to at least 1 of embodiment of this invention. It is a flowchart of the execution process corresponding to at least 1 of embodiment of this invention. It is a figure which shows an example of the image imaged with the virtual camera corresponding to at least 1 of embodiment of this invention. It is a figure which shows an example of the image imaged with the some virtual camera corresponding to at least 1 of embodiment of this invention. It is a figure which shows an example of the image imaged with the some virtual camera corresponding to at least 1 of embodiment of this invention.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. Hereinafter, the description regarding the effect is one aspect of the effect of the embodiment of the present invention, and is not limited to what is described here. In addition, the order of the processes constituting the flowchart described below is out of order as long as no contradiction or inconsistency occurs in the process contents.

[First embodiment]
The outline of the first embodiment of the present invention will be described. Hereinafter, as a first embodiment of the present invention, a display system including a display device that is used by being worn on the user's head will be described as an example.

  FIG. 1 is a block diagram showing a configuration of a display system corresponding to at least one of the embodiments of the present invention. The display system 1 includes at least an orientation specifying unit 101, a first display unit 102, and a second display unit 103.

  The orientation specifying unit 101 has a function of specifying the orientation of the display device. The 1st display part 102 has a function which displays the 1st image which shows the state of virtual space on the display screen of a display apparatus according to the specified direction. The second display unit 103 is different from the first image on the display area generated by being superimposed on the first image when the visual axis corresponding to the specified orientation overlaps with a predetermined object in the virtual space. It has a function of displaying the second image.

  The execution process in the first embodiment of the present invention will be described. FIG. 2 is a flowchart of execution processing corresponding to at least one of the embodiments of the present invention.

  In the display system 1, the orientation identifying unit 101 identifies the orientation of the display device used by being mounted on the user's head (step S1). The display system 1 displays the first image indicating the state of the virtual space on the display screen of the display device according to the specified orientation (step S2). When the visual axis corresponding to the specified orientation overlaps with a predetermined object in the virtual space, the display system 1 is different from the first image on the display area generated by being superimposed on the first image. The image is displayed (step S3), and the process ends.

  As one aspect of the first embodiment, a second image different from the first image can be displayed on the display area generated by superimposing the first image without performing complicated operations and operations. it can.

  In the first embodiment, “attaching to the head” means, for example, putting on the head like a hat, attaching to the face like glasses, or attaching to a part of the head. It is a concept that includes the case of covering the head by extending from a part of the body. “Display device” is, for example, a display device capable of visually recognizing images with both eyes or a single eye. It includes a transmission type that can be seen. “Direction” is a concept including not only the direction and angle of an object, but also a direction related to the amount of change in unit time, such as acceleration. The “virtual space” is a space defined by a program and data, for example, and is a virtual space generated by a computer. The “first image” refers to, for example, an image indicating the state of the virtual space. The “visual axis” refers to an axis extending in the visual line direction from the viewpoint of the virtual camera, for example.

[Second Embodiment]
The outline of the second embodiment of the present invention will be described. Hereinafter, as a second embodiment of the present invention, a display system including a display device used by being mounted on the user's head will be described as an example.

  With regard to the configuration of the display system, the contents described in FIG. 1 can be adopted within a necessary range. Regarding the flowchart of the execution process, the contents described in FIG. 2 can be adopted within a necessary range.

  As an example of the second embodiment, when the state where the visual axis overlaps with a predetermined object in the virtual space continues for a predetermined time, the second display unit 103 displays the second image on the display area. It is preferable.

  As one aspect of the second embodiment, a second image different from the first image can be displayed on the display area generated by superimposing the first image without performing complicated operations and operations. it can.

  As one aspect of the second embodiment, when a state where the visual axis overlaps with a predetermined object in the virtual space continues for a predetermined time, a second image is displayed, which is not intended by the user. The display of images can be suppressed, and only the second image that the user intends to display can be displayed, thereby improving convenience.

  In the second embodiment, “mounted on the head”, “display device”, “orientation”, “virtual space”, “first image”, and “visual axis” are respectively the first embodiment. The contents described in can be adopted within a necessary range.

[Third embodiment]
The outline of the third embodiment of the present invention will be described. Hereinafter, as a third embodiment of the present invention, a display system including a display device used by being worn on the user's head will be described as an example.

  With regard to the configuration of the display system, the contents described in FIG. 1 can be adopted within a necessary range. Regarding the flowchart of the execution process, the contents described in FIG. 2 can be adopted within a necessary range.

  As an example of the third embodiment, the state in which the visual axis overlaps with the first object in the virtual space continues for a predetermined time, so that the second display unit 103 corresponds to the first object on the display area. After the two images are displayed, when the state where the visual axis overlaps with the second object in the virtual space continues for a predetermined time, the second display unit 103 displays the second image corresponding to the first object on the display area. Instead, it is preferable to display a second image corresponding to the second object.

  As one aspect of the third embodiment, a second image different from the first image can be displayed on the display area generated by superimposing the first image without performing complicated operations and operations. it can.

  As one aspect of the third embodiment, when the state where the visual axis overlaps with a predetermined object in the virtual space continues for a predetermined time, a second image is displayed, which is not intended by the user. The display of images can be suppressed, and only the second image that the user intends to display can be displayed, thereby improving convenience.

  As one aspect of the third embodiment, when the visual axis continuously overlaps with the second object for a predetermined time after displaying the second image corresponding to the first object, the first image corresponding to the first object is displayed. By displaying the second image corresponding to the second object instead of the two images, the display of the second image can be updated simply by overlaying the visual axis on the object, and the operability for the user is further improved. Can do. Since a plurality of second images are not displayed, it contributes to a reduction in processing load.

  In the third embodiment, “mounted on the head”, “display device”, “orientation”, “virtual space”, “first image”, and “visual axis” are respectively the first embodiment. The contents described in can be adopted within a necessary range.

[Fourth embodiment]
Next, an outline of the fourth embodiment of the present invention will be described. As a fourth embodiment of the present invention, a display system including a display device used by being worn on the user's head will be described as an example.

  FIG. 3 is a block diagram showing a configuration of a display system corresponding to at least one of the embodiments of the present invention. FIG. 3A is a diagram illustrating an example of the configuration of the display system. The display system 1 includes a display device 201 equipped with a sensor for detecting movement and orientation, an irradiation device 202 that emits light, a computer device 203 that generates or processes an image displayed on the display device 201, It is comprised from the controller 204 carrying a sensor.

  The display device 201 is preferably a goggle type display device that is attached to the head so as to cover the field of view. The display may be a transmissive display having a high transmittance or a non-transmissive display having a low transmittance. In addition, the display device 201 can specify the posture of the user using a plurality of mounted sensors. The configuration of the display device 201 will be described later.

  The irradiation device 202 is a device that irradiates a laser beam over a wide range. When a laser beam is irradiated around the irradiation device 202, the laser beam is detected by an optical sensor mounted on the display device 201 and the controller 204, and the positions of the display device 201 and the controller 204 are specified. This position specifying method is called “Lighthouse method”. In order to specify the position more accurately, it is preferable that at least two irradiation devices 202 are installed at a predetermined interval.

  The computer device 203 can be connected to the display device 201 and the controller 204 by communication. Communication may be wired or wireless. The computer device 203 may be a device independent of the display device 201 or may be an integrated device incorporated in the display device 201.

  The controller 204 is an input device that includes a button and a trigger and receives an input by a user operation. In addition, an optical sensor is mounted, and the laser beam emitted from the irradiation device 202 can be sensed. Furthermore, the touch sensor is mounted and the input by a user's contact can be received. The controller 204 is used by being gripped by a pair of left and right hands, but may use only one of them.

  FIG. 3B is a diagram illustrating an example regarding the configuration of a display system different from the display system illustrated in FIG. The display system illustrated in FIG. 3B includes a display device 251 including a light emitting unit serving as a light source, a photographing device 252 that captures light emitted from the light emitting unit serving as a light source of the display device 251 and the controller 254, and a display device. The computer apparatus 253 which produces | generates or processes the image displayed on 251 and the controller 254 provided with the light emission part used as a light source are comprised.

  The display device 251 is preferably a goggle-type display device that is attached to the head so as to cover the field of view, like the display device 201. The display may be a transmissive display having a high transmittance or a non-transmissive display having a low transmittance.

  The photographing device 252 is a device for identifying the positions of the display device 251 and the controller 254 by photographing light emitted from a light emitting unit serving as a light source of the display device 251 and the controller 254.

  The computer device 253 can be connected to the display device 251 and the controller 254 by communication. Communication may be wired or wireless.

  The controller 254 is an input device that includes buttons and receives an input by a user operation. A light emitting unit serving as a light source is provided, and the position of the controller 254 is specified by the photographing device 252 photographing the light emitted from the controller 254. The controller is gripped by the hand and preferably used in pairs, but one controller may be used.

  In the following description, in the embodiment of the present invention, the positions of the display device 201 and the controller 204 are specified by the method shown in FIG. 3A, but the present invention is not limited to this method. For example, a method shown in FIG. 3B or a method of specifying the positions of the display device 201 and the controller 204 by wearing on a part of the body other than the head may be used.

  The image displayed on the display device 201 is an image generated or processed by the computer device 203 based on a value measured by a sensor mounted on the display device 201 and a user input operation to the controller 204. .

  Here, the configuration of the display device 201 will be described. FIG. 4 is a block diagram showing a configuration of a display device corresponding to at least one of the embodiments of the present invention. The display device 201 includes a control unit 11, a RAM (Random Access Memory) 12, a storage unit 13, an imaging unit 14, a graphics processing unit 15, a display unit 16, a sensor unit 18, a communication interface 19, an interface unit 20, and a lens 21. Each connected by an internal bus.

  The control unit 11 includes a CPU (Central Processing Unit) and a ROM (Read Only Memory). The control unit 11 executes a program stored in the storage unit 13 and controls the display device 201. The RAM 12 is a work area for the control unit 11. The storage unit 13 is a storage area for storing programs and data.

  The control unit 11 reads the program and data from the RAM 12 and the storage unit 13 and performs processing. The control unit 11 processes the program and data loaded in the RAM 12 to output a drawing command to the graphics processing unit 15. Further, the control unit 11 performs processing based on various values measured by the sensor unit 18 or transmits data to the computer apparatus 3 via the interface unit 20.

  The imaging unit 14 outputs an image captured through the lens 21 to the graphics processing unit 15. When the field of view is covered by mounting the display device 201 on the head, information outside the device cannot be confirmed, and thus the image is captured by the imaging unit 14 on the display screen 17 provided in the display unit 16 described later. By displaying the information, the user can recognize the real space and ensure safety.

  The graphics processing unit 15 performs drawing of one image for each frame. One frame time of an image is, for example, 1/30 second. The graphics processing unit 15 is responsible for a part of arithmetic processing related to drawing and has a role of distributing the load of the entire system.

  The display unit 16 includes a display screen 17 and is installed inside the goggles of the display device 201 that covers the user's field of view. For example, organic EL or inorganic EL is used for the display screen 17, but the display screen 17 is not limited to this, and a terminal device including a display such as a smartphone may be incorporated.

  The sensor unit 18 includes at least a proximity sensor 18a, an infrared sensor 18b, a gyro sensor 18c, and an acceleration sensor 18d. The proximity sensor 18a is installed in the goggle portion of the display device 201, and is used to determine whether or not the user touches the face. A plurality of infrared sensors 18b are installed in the display device 201, and are used to identify the position of the display device 201 by sensing the light emitted from the irradiation device 202. The gyro sensor 18c and the acceleration sensor 18d are used for specifying the posture of the user.

  Here, a method for specifying the posture of the user will be described with reference to the drawings. FIG. 5 is a diagram illustrating an orthogonal coordinate system for specifying the posture of the user when the display device is mounted on the head, corresponding to at least one of the embodiments of the present invention.

  As shown in the figure, XYZ coordinates are defined around the head of the user wearing the display device 201. The vertical direction in which the user stands upright is the Y axis (yaw angle), the direction orthogonal to the Y axis, and the direction connecting the center of the display screen 17 of the display device 201 and the user's head is the Z axis (roll angle), Y axis A direction orthogonal to the Z axis is defined as an X axis (pitch angle). The gyro sensor 18c detects an angle (inclination) around each axis, and the acceleration sensor 18d detects the movement of the display device 201 and displays it on the display screen 17 according to the detected angle and the detected movement. Change the image. The display device 201 can identify the user's posture using values measured by a plurality of sensors (hereinafter referred to as posture information).

  Subsequently, the communication interface 19 can be connected to the communication line 5 wirelessly or by wire, and can receive data via the communication line 5. Data received via the communication interface 19 is loaded into the RAM 12 and arithmetic processing is performed by the control unit 11.

  The interface unit 20 is mainly connected to the computer device 203 and can receive data such as images processed or generated by the computer device 203. In addition, the measurement values of various sensors acquired by the sensor unit 18 can be transmitted to the computer device 203.

  Next, the computer device 203 will be described. FIG. 6 is a block diagram showing a configuration of a computer device corresponding to at least one of the embodiments of the present invention. The computer device 203 includes a control unit 31, a RAM 32, a storage unit 33, a sound processing unit 34, a graphics processing unit 35, a DVD / CD-ROM drive 36, a communication interface 37, an interface unit 38, and a frame memory 39. Connected by internal bus.

  The control unit 31 includes a CPU and a ROM. The control unit 31 executes a program stored in the storage unit 33 and controls the computer device 203. In addition, the control unit 31 includes an internal timer that measures time. The RAM 32 is a work area for the control unit 31. The storage unit 33 is a storage area for storing programs and data.

  The control unit 31 reads the program and data from the RAM 32 and performs processing. The control unit 31 outputs a drawing command to the graphics processing unit 35 by processing the program and data loaded in the RAM 32.

  The sound processing unit 34 is connected to the sound output device 40. When the control unit 31 outputs a sound output instruction to the sound processing unit 34, the sound processing unit 34 outputs a sound signal to the sound output device 40. The sound output device 40 is, for example, a headphone, and a user may wear it on the head together with the display device 201, or a speaker may be incorporated in the display device 201.

  The graphics processing unit 35 draws one image for each frame. One frame time of an image is, for example, 1/90 second. The graphics processing unit 35 is responsible for a part of calculation processing related to drawing and has a role of distributing the load of the entire system. The graphics processing unit 35 of the computer device 203 is included in the display device 201 in order to generate an image to be displayed on the display screen 17 based on the position information and orientation information of the display device 201 and the position information of the controller 204. It requires higher performance than the graphics processing unit 15.

  The DVD / CD-ROM drive 36 reads data from the disk 41, loads it into the RAM 32, and performs arithmetic processing by the control unit 31. The type of the disc 41 is not limited to a DVD or a CD, but may be a Blu-ray (registered trademark) disc or other standards.

  The communication interface 37 can be connected to the communication line 5 wirelessly or by wire, and can receive data via the communication line 5. Data received via the communication interface 37 is loaded into the RAM 32 in the same manner as data read from the disk 41, and arithmetic processing is performed by the control unit 31.

  The interface unit 38 is mainly connected to the display device 201 and can transmit data such as an image processed or generated by the computer device 203.

  In the above description, the display device 201 and the computer device 203 are described as separate devices. However, the display device 201 may perform all processing of the computer device 203. In this case, the computer device 203 is unnecessary, and the controller 204 is connected to the display device 201 by communication.

  Next, the configuration of the display system in the embodiment of the present invention will be described. FIG. 7 is a block diagram showing a configuration of a display system corresponding to at least one of the embodiments of the present invention. The display system 1 includes an angle information measuring unit 301, a position and orientation specifying unit 302, an information transmitting unit 303, an information receiving unit 304, a line-of-sight specifying unit 305, an intersection determining unit 306, a time measuring unit 307, an elapsed time determining unit 308, and a virtual camera. It includes at least a position specifying unit 309, a sub window image generation unit 310, a controller input acceptance determination unit 311, a display image generation unit 312, an image transmission unit 313, an image reception unit 314, and an image display unit 315.

  The angle information measurement unit 301 has a function of measuring angle information from the attitude of the display device 201. The position / orientation specifying unit 302 has a function in which an optical sensor mounted on the display device 201 senses a laser beam emitted from the irradiation device 202 and specifies the position and orientation of the display device 201. The information transmission unit 303 has a function of transmitting the angle information measured in the display device 201 and the specified position and orientation information to the computer device 203.

  The information receiving unit 304 has a function of receiving information transmitted from the information transmitting unit 303. The line-of-sight specifying unit 305 has a function of specifying the user's line of sight in the virtual space based on the information received by the information receiving unit 304. The intersection determination unit 306 has a function of determining whether or not the object arranged in the virtual space and the line of sight specified by the line-of-sight specifying unit 305 intersect.

  The time measuring unit 307 has a function of measuring the time during which the object and the line of sight have crossed continuously after it is determined that the object and the line of sight intersect. The elapsed time determination unit 308 has a function of determining whether or not the time measured by the time measuring unit 307 has passed a predetermined time. The virtual camera position specifying unit 309 has a function of specifying the position of a virtual camera that can capture an object that intersects the line of sight specified by the line of sight specifying unit 305.

  The sub window image generation unit 310 has a function of generating an image captured from the position specified by the virtual camera position specifying unit 309. The controller input reception determination unit 311 has a function of determining whether or not an input from the controller 204 has been received. The display image generation unit 312 has a function of generating an image to be displayed.

  The image transmission unit 313 has a function of transmitting the display image generated by the display image generation unit 312 to the display device 201. The image receiving unit 314 has a function of receiving the display image transmitted by the image transmitting unit 313. The image display unit 315 has a function of displaying the display image received by the image receiving unit 314 on the display screen 17.

  Subsequently, an execution process in the fourth embodiment of the present invention will be described. An example of the fourth embodiment of the present invention is a system for browsing content. More specifically, the content is composed of a plurality of sheets of paper, and is data related to a comic that progresses according to the development of the story, and is a virtual that represents a single comic by a user operation for each comic. It is a display system that can enter the space.

  FIG. 8 is an example of an execution screen corresponding to at least one of the embodiments of the present invention. In FIG. 8A, the display screen 17 displays the status of one frame of a cartoon, and a character 401 and a sub window 402 are displayed. The sub window 402 displays an image captured by a virtual camera that is different from the virtual camera that captures the image displayed on the display screen 17. In FIG. 8, the face of the character 401 is enlarged and displayed.

  FIG. 8B is an example of an execution screen in which the user changes the direction of the display device 201 by moving the line of sight (hereinafter also referred to as the visual axis) to the left in the situation of FIG. As the user moves his / her head, the virtual space on the left side of FIG. 8A is displayed on the display screen 17 for the user. In this case, the sub window 402 remains displayed in a predetermined area on the display screen 17, and the position does not change according to the movement of the user's head. The predetermined area may be a partial area included in the display screen 17 and is not limited to the lower right of the display screen 17.

  The sub-window 402 may be designed so that it is displayed in association with a single frame of a comic so that the sub-window 402 is displayed whenever a virtual space representing the single frame of the comic is entered. That is, the image displayed on the display screen 17 may have, for example, a flag for displaying the sub window 402 as attribute information. Or you may display according to the content of an image so that the subwindow 402 may be displayed when the content which an image shows is a specific scene.

  The sub window 402 may be designed to be displayed when a predetermined condition is satisfied. The predetermined condition is, for example, when an input operation is accepted by the controller 204, or by matching the direction specified from the orientation of the display device 201 with the visual axis of the virtual camera, and the visual axis intersects with an object in the virtual space. For example, it may be determined that it has been performed.

  Next, generation of an image to be displayed on the display screen in the fourth embodiment of the present invention will be described. When the program is executed, the control unit 31 of the computer device 203 causes the graphics processing unit 35 to output a virtual space defined by the program. Further, the position of the user wearing the display device 201 on the head and the position of the controller 204 are specified based on the position and direction in which the light beam emitted from the irradiation device 202 is received. Further, the attitude of the display device 201 is specified by the measurement values of various sensors of the sensor unit 18 of the display device 201.

  The virtual space defined by the program will be described. FIG. 9 is a diagram showing the concept of the virtual space corresponding to at least one of the embodiments of the present invention. When the virtual space 70 defined by the program is represented by a hemispherical space as illustrated, the visual field of the user 80 wearing the display device 201 on the head corresponds to the visible region 81. That is, an image corresponding to the visible region 81 is displayed on the display screen 17 of the display device 201. For example, when the user 80 turns to the left, the visible region 81 moves to the left according to the movement of the user 80. Similarly, in the vertical direction, for example, when the user 80 faces upward, the visible region 81 moves upward in accordance with the movement of the user 80.

[Execution process]
FIG. 10 is a flowchart of execution processing corresponding to at least one of the embodiments of the present invention. First, the display apparatus 201 measures angle information from the attitude of the display apparatus 201 (step S11). The angle information is measured by various sensors included in the sensor unit 18 of the display device 201.

  Next, when the optical sensor mounted on the display device 201 senses the laser beam emitted from the irradiation device 202, the display device 201 specifies the position and orientation of the display device 201 (step S12). The processing of step S11 and step S12 may be performed at predetermined intervals, and may be transmitted to the computer device 203 at every measurement or specific time to generate a display image to be displayed on the display screen 17 of the display device 201.

  Next, the display apparatus 201 transmits the angle information measured in step S11 and the information related to the orientation and position specified in step S12 to the computer apparatus 203 (step S13). Next, the computer device 203 receives the information (step S14), and based on the angle information measured in step S11 and the information on the position and orientation specified in step S12, the computer device 203 displays the user's line of sight in the virtual space. Specify (step S15).

  The user's line of sight in the virtual space is based on the angle information (information about the tilt with respect to the horizontal plane), the information about the position, and the information about the direction (information about which direction on a plane parallel to the horizontal plane). This is the direction in which it is estimated that the user is looking in the virtual space in the visible region 81 specified in the above. That is, the specified direction corresponds to the visual axis of the virtual camera in the virtual space. For example, the viewpoint of the virtual camera is set at the position of the display device 201, and based on the orientation of the display device 201, the direction of the line of sight on the plane parallel to the horizontal plane in the virtual world is determined from the viewpoint. The inclination of the line of sight with respect to the horizontal plane can be specified based on the orientation information of the display device 201. Furthermore, by performing perspective transformation from the viewpoint of the virtual camera onto the virtual screen, an image corresponding to the user's line of sight can be generated. In this way, without causing the user to be conscious, in the virtual world, an object that exists in the direction that the user is assumed to be viewing is displayed on the display device 201, and the convenience of the user is improved. It becomes possible to immerse in the virtual world more.

  Incidentally, the display device 201 may have a function of tracking the line of sight of the user wearing the display device 201. In this case, the line of sight can be calculated from the movement of the user's eyeball.

  Next, the computer device 203 determines whether or not the user's line of sight identified in step S15 intersects with the object arranged in the virtual space (step S16). The object is, for example, a character, an enemy character, an item, or an object displayed in the virtual space, which can be arranged in the virtual space. Crossing means that a line of sight intersects with a boundary surface forming an object, and is a concept including contact between a line and a surface. For the intersection determination between the line segment and the object in the three-dimensional space, for example, a method for obtaining the intersection of the object and the ray (ray) in the ray tracing method can be used. Other methods are also possible, and in the embodiment of the present invention, the method of intersection determination is not limited.

  If it is determined that the user's line of sight intersects with a predetermined object in the virtual space (YES in step S16), the computer device 203 counts time with an internal timer provided in the control unit 31 of the computer device 203. Is started (step S17).

  Next, the computer device 203 determines whether or not a predetermined time has elapsed after the predetermined object and the line of sight have continued to intersect based on the timing started in step S17 (step S18). If it is determined that the predetermined time has elapsed (YES in step S18), the computer apparatus 203 specifies the position of the virtual camera that can image the object that intersects the line of sight specified in step S15 (step S19). ). If the predetermined time has not elapsed (NO in step S18), the sub window is not displayed. When the subwindow is already displayed, the existing subwindow may be kept displayed.

  The identification of the position of the virtual camera in step S19 will be described. The virtual camera whose position is specified here is a virtual camera (hereinafter referred to as a sub-virtual camera) that is different from a virtual camera (hereinafter referred to as a main virtual camera) that captures an image corresponding to the direction of the line of sight. The position of the virtual camera specified in step S15 is the position of the sub-virtual camera and may be any position as long as the intersected object is captured by the virtual camera. For example, an image corresponding to the line of sight is displayed. What is necessary is just to have a different viewpoint or a different visual axis from the virtual camera to be imaged.

  Next, the computer apparatus 203 captures an object with the sub virtual camera from the position specified in step S19, and generates an image to be displayed on the sub window (step S20).

  If it is determined that the user's line of sight does not intersect with a predetermined object in the virtual space (NO in step S16), it is determined whether a predetermined input operation from the controller has been accepted (step S21). . If it is determined that a predetermined input operation has been received from the controller (YES in step S21), the process proceeds to step S19 to specify the position of the sub virtual camera. If it is determined that a predetermined input operation has not been received from the controller (NO in step S21), a sub window image is not generated.

  Next, an image corresponding to the line-of-sight direction specified in step S15 is generated (step S22). The image corresponding to the direction of the line of sight is an image captured by the main virtual camera. Next, the computer apparatus 203 transmits the sub window image generated in step S20 and the image corresponding to the line-of-sight direction generated in step S22 to the display apparatus 201 (step S23).

  The display device 201 receives the image transmitted in step S23 (step S24). The display device 201 displays the image received in step S24 (step S25) and ends.

  A display mode of an image captured by the virtual camera on the display screen 17 will be described. FIG. 11 is a diagram illustrating an example of an image captured by a virtual camera, corresponding to at least one of the embodiments of the invention. FIG. 11A is a schematic diagram illustrating a state in which an object is imaged by a virtual camera. The main virtual camera 98 is a virtual camera that captures an image corresponding to the direction of the user's line of sight. FIG. 11B is a diagram illustrating an image captured in the state of FIG. In FIG. 11A, when the object 91 and the object 92 are imaged, the visual axis α does not intersect any object, and thus the subwindow is not displayed on the display screen 17. The point 90 representing the line of sight may be displayed or not displayed.

  FIG. 12 is a diagram illustrating an example of an image captured by a plurality of virtual cameras corresponding to at least one of the embodiments of the invention. FIG. 12A is a schematic diagram showing a state where the visual axis α of the main virtual camera 98 and the object 92 intersect. FIG. 12B is a diagram illustrating a plurality of images captured in the state of FIG. Since the visual axis α intersects the object 92, the object 92 is imaged from a viewpoint different from that of the main virtual camera 98 by the sub virtual camera 99 that generates an image displayed in the sub window. An image captured by the sub virtual camera 99 is displayed in the sub window 93.

  As shown in the figure, when the sub-virtual camera 99 is picked up from different viewpoints, a part of the image picked up by the main virtual camera 98 can be enlarged and displayed so that the user can visually recognize so far. The missing feature point 94 can be displayed so as to be visible.

  FIG. 13 is a diagram illustrating an example of an image captured by a plurality of virtual cameras corresponding to at least one of the embodiments of the invention. FIG. 13A shows a state where the visual axis α of the main virtual camera 98 intersects the object 92. FIG. 13B is a diagram showing a plurality of images captured in the state of FIG. Since the visual axis α intersects the object 92, the object 92 is imaged from a visual axis different from that of the main virtual camera 98 by the sub virtual camera 99 that generates an image displayed in the sub window. An image captured by the sub virtual camera 99 is displayed in the sub window 93.

  As shown in the drawing, since the imaging by the sub virtual camera 99 is performed from a visual axis different from that of the main virtual camera 98, the feature point 95 that the user could not visually recognize in the image captured by the main virtual camera 98 is obtained. It can be displayed so as to be visible.

  The execution process may be repeatedly executed at a predetermined timing. The predetermined timing can be performed, for example, at every frame rate of the screen or at a multiple of the frame rate.

  In step S25, the display apparatus 201 displays the received subwindow image and an image corresponding to the direction of the line of sight in a superimposed manner. In step S25, if the subwindow has already been displayed, the image of the subwindow generated in step S20 may be displayed in place of the displayed subwindow image to update the display content.

  In the fourth embodiment, the light beam for specifying the positions of the display device 201 and the controller 204 is a laser beam, for example, an infrared ray. However, the light beam is not limited to this, and is a visible ray. Alternatively, other invisible light may be used.

  In the fourth embodiment, the sub-window is displayed when the line of sight continues to cross a predetermined object for a predetermined time. However, the line of sight and the predetermined line are not determined without determining whether the predetermined time has elapsed. The sub window may be displayed only by the determination regarding the intersection with the object.

  In the fourth embodiment, the position where the subwindow is displayed is not mentioned. However, when a predetermined condition is satisfied such as when the subwindow is overlapped with the predetermined object, the position of the subwindow is changed or the subwindow is changed. You may make it not display.

  In the fourth embodiment, the image displayed on the sub-window is an image captured by the virtual camera, but the present invention is not limited to this. That is, it may not be an image captured by a virtual camera, but may be an image related to an object that intersects the user's visual axis. For example, invisible information such as the attribute information of the object and the psychological state of the object may be displayed. Information that is not related to an object that intersects the visual axis may be displayed. For example, an option for moving to the next scene may be displayed by intersecting the user's visual axis and the object.

  In the fourth embodiment, the image to be displayed in the sub-window is an image captured by the virtual camera, but an image stored in the storage unit 33 of the computer device 203 may be displayed. For example, when a main frame of a cartoon is displayed by the main virtual camera, if you adjust the line of sight to the hero character, a part of the hero character's body is displayed in the sub-window so that the comic story progresses. Also good. Also, audio may be output in accordance with the display of the subwindow.

  As one aspect of the fourth embodiment, a predetermined area of the display screen is different from the first image on the display area generated by superimposing the first image without performing complicated operations and operations. A second image can be displayed.

  As one aspect of the fourth embodiment, when the state where the visual axis overlaps with a predetermined object in the virtual space continues for a predetermined time, a second image that is not intended by the user is displayed by displaying the second image. The display of images can be suppressed, and only the second image that the user intends to display can be displayed, thereby improving convenience.

  As one aspect of the fourth embodiment, when the visual axis continuously overlaps the second object for a predetermined time after displaying the second image corresponding to the first object, the first object corresponding to the first object is displayed. By displaying the second image corresponding to the second object instead of the two images, the display of the second image can be updated simply by overlaying the visual axis on the object, and the operability for the user is further improved. Can do. In addition, since a plurality of sub-windows are not displayed, the processing load is also reduced.

  As one aspect of the fourth embodiment, by displaying an image related to a predetermined object intersecting the user's visual axis in the sub-window, the user can know information related to the predetermined object without performing an input operation. In addition, there is an advantage that a place where the user's visual axis is hit can be recognized.

  As one aspect of the fourth embodiment, by displaying an enlarged image of an object with overlapping visual axes in a sub-window, information that could not be displayed conventionally can be given to the user, improving user convenience. be able to. Moreover, interest in objects in the virtual space can be collected, which contributes to giving a deeper immersive feeling to the user.

  In the fourth embodiment, “mounted on the head”, “display device”, “orientation”, “virtual space”, “first image”, and “visual axis” are respectively the first embodiment. The contents described in can be adopted within a necessary range.

  In the fourth embodiment, “enlargement” means, for example, displaying an image of a part of the object with the same or a larger number of pixels as the number of pixels representing the imaged object.

[Appendix]
The above description of the embodiments described the following invention so that a person having ordinary knowledge in the field to which the invention belongs can carry out the invention.

[1] A display system including a display device used by being worn on a user's head,
Orientation specifying means for specifying the orientation of the display device;
First display means for displaying a first image indicating the state of the virtual space on the display screen of the display device according to the specified orientation;
A second image different from the first image is displayed on the display area generated by overlapping the first image when the visual axis corresponding to the specified orientation overlaps with a predetermined object in the virtual space. A display system comprising two display means.

[2] The method according to [1], wherein the second display unit displays a second image on a display area when the state where the visual axis overlaps with a predetermined object in the virtual space continues for a predetermined time. Display system.

[3] After the state where the visual axis overlaps the first object in the virtual space continues for a predetermined time, the second display means displays the second image corresponding to the first object on the display area. ,
When the state in which the visual axis overlaps with the second object in the virtual space continues for a predetermined time, the second display means replaces the second image corresponding to the first object on the display area by replacing the second image. The display system according to [2], wherein a second image corresponding to the object is displayed.

[4] The display system according to any one of [1] to [3], wherein the second image is an image related to the predetermined object.

[5] The display system according to [4], wherein the second image is an enlarged image of a predetermined object.

[6] A display method executed in a display system including a display device used by being worn on a user's head,
Identifying the orientation of the display device;
Displaying a first image indicating the state of the virtual space on the display screen of the display device according to the specified orientation;
When the visual axis corresponding to the specified orientation overlaps with a predetermined object in the virtual space, a second image different from the first image is displayed on the display area generated by being superimposed on the first image. And display method to execute.

1 Display System 11 Control Unit 12 RAM
DESCRIPTION OF SYMBOLS 13 Storage part 14 Imaging part 15 Graphics processing part 16 Display part 17 Display screen 18 Sensor part 18a Sensor 18b Infrared sensor 18c Gyro sensor 18d Acceleration sensor 19 Communication interface 20 Interface part 21 Lens 201 Display apparatus 202 Irradiation apparatus 203 Computer apparatus 204 Controller 251 Display device 252 Imaging device 253 Computer device 254 Controller 31 Control unit 32 RAM
33 Storage Unit 34 Sound Processing Unit 35 Graphics Processing Unit 36 CD-ROM Drive 37 Communication Interface 38 Interface Unit 39 Frame Memory 40 Sound Output Device 41 Disc 5 Communication Line

Claims (6)

A display system including a display device used by being worn on a user's head,
Orientation specifying means for specifying the orientation of the display device;
First display means for displaying a first image indicating the state of the virtual space on the display screen of the display device according to the specified orientation;
A second image different from the first image is displayed on the display area generated by overlapping the first image when the visual axis corresponding to the specified orientation overlaps with a predetermined object in the virtual space. Two display means ,
The display system is a display system in which a position displayed on the display screen does not change in accordance with the orientation specified by the orientation specifying means . 2. The display system according to claim 1, wherein the second display unit displays a second image on a display area when a state where the visual axis overlaps with a predetermined object in a virtual space continues for a predetermined time. . The state in which the visual axis overlaps with the first object in the virtual space continues for a predetermined time, so that the second display unit displays the second image corresponding to the first object on the display area.
When the state in which the visual axis overlaps with the second object in the virtual space continues for a predetermined time, the second display means replaces the second image corresponding to the first object on the display area by replacing the second image. The display system according to claim 2, wherein a second image corresponding to the object is displayed. The display system according to claim 1, wherein the second image is an image related to the predetermined object. The display system according to claim 4, wherein the second image is an enlarged image of a predetermined object. A display method executed in a display system including a display device used by being worn on a user's head,
Identifying the orientation of the display device;
Displaying a first image indicating the state of the virtual space on the display screen of the display device according to the specified orientation;
When the visual axis corresponding to the specified orientation overlaps with a predetermined object in the virtual space, a second image different from the first image is displayed on the display area generated by being superimposed on the first image. run the door,
The display method, wherein the display area does not change a position displayed on the display screen in accordance with the orientation of the specified display device .

Images