JP2017139000A - Display device and method of controlling display device - Google Patents

Abstract

To improve operability when performing an operation based on a virtual index displayed as an image. According to a head-mounted display device HM, a display drive unit and an optical image display unit that output an image light L superimposed on external light and visually recognize the image, and an input auxiliary image are arranged on the image. An input auxiliary image processing unit 146 to be displayed on the display drive unit and the optical image display unit, and an input detection unit 142 that receives a position instruction operation corresponding to the input auxiliary image, and the input auxiliary image processing unit 146 includes input assistance for an image. The aspect of the image can be changed. [Selection] Figure 3

Description

  The present invention relates to a display device and a control method for the display device.

  2. Description of the Related Art Conventionally, there has been known a device that allows a user to recognize a virtual input area by a display device and perform an input operation on the virtual input area (see, for example, Patent Document 1). The apparatus of Patent Literature 1 displays a landscape in front of the user and a keyboard image serving as a virtual input device in a superimposed manner by an HMD (head mounted display) attached to the user's head. When the user moves his / her finger according to the keyboard, the position of the finger is detected as an input operation on the keyboard.

JP 2002-318652 A

The apparatus described in Patent Document 1 detects a surface existing in front of the user with a camera, generates an image of a virtual input device in accordance with the surface, and displays the virtual input device superimposed on the captured image of the camera. Since this virtual input device is fixed, the user needs to perform an operation in accordance with the displayed virtual input device. In other words, there is a restriction that the user must perform an operation on the surface detected by the apparatus.
The present invention has been made in view of the above-described circumstances, and an object thereof is to improve operability when an operation is performed based on a virtual index displayed as an image.

In order to achieve the above object, the present invention provides display means for superimposing and outputting image light on external light to visually recognize the image, and display control means for arranging an operation area on the image and causing the display means to display the display means. Input means for receiving a position indicating operation corresponding to the operation area, and the display control means changes a mode of the operation area in the image.
According to the present invention, since the image is displayed by the image light output to be superimposed on the external light, the user can see the external scene by the external light together with the operation area displayed by the image light. You can operate by looking at the place. And by changing the mode of the displayed operation area, it is possible to change the display mode of the operation area according to the status of the operation, to realize comfortable operability, and to improve operability. it can.

Further, the present invention is the display device described above, wherein the display control unit changes a mode of the operation area according to a distance to a virtual input surface on which the position instruction operation is performed.
ADVANTAGE OF THE INVENTION According to this invention, an operation area is displayed in the aspect corresponding to the position which a user operates, and more comfortable operativity is realizable.

In addition, the present invention provides the display device described above, further comprising a distance detection unit that detects a distance to the virtual input surface, and the display control unit performs the operation according to the distance detected by the distance detection unit. The aspect of the area is changed.
According to the present invention, since the distance to the virtual input surface on which the user operates is detected, the operation of inputting or setting the distance to the virtual input surface can be omitted, and the operability is further improved. be able to.

Moreover, the present invention is the display device described above, wherein the distance detecting unit detects a surface visually recognized by the external light and detects a distance to the surface.
According to the present invention, since the surface that the user may use for the operation is detected, the distance to the virtual input surface can be detected more accurately.

Further, the present invention is the display device described above, wherein the display control unit arranges the operation area including an input auxiliary image for guiding the position indicating operation, and changes a size of the input auxiliary image. And
According to the present invention, it is possible to easily and accurately perform fine input operations such as character input by changing the size of the input auxiliary image serving as an index of user operation.

Also, the present invention is the above display device, wherein the display unit forms the image with a plurality of pixels arranged in a matrix, and the display control unit changes the size of the input auxiliary image. The number of pixels constituting the input auxiliary image is changed.
According to the present invention, when the size of the input auxiliary image is changed by changing the number of pixels constituting the input auxiliary image, the resolution decreases as the size increases or the visibility decreases as the size decreases. Etc. can be prevented.

Further, the present invention is the above display device, wherein the display control unit displays an image including a plurality of auxiliary lines as the input auxiliary image by the display unit, and changes the interval of the auxiliary lines. And
According to the present invention, it is possible to more easily and accurately perform a fine input operation such as character input by changing the interval between the auxiliary lines of the input auxiliary image.

Further, the present invention is the above-described display device, wherein the display control unit causes the display unit to display an image of a virtual keyboard for inputting characters by the position instruction operation as the input auxiliary image, It is characterized in that at least one of the display size, display color, and display area of the keyboard is changed.
According to the present invention, it is possible to easily input characters by displaying a virtual keyboard. Then, by changing the display size, display color, or display area of the virtual keyboard, the user can perform an input operation more comfortably.

Further, the present invention is the display device described above, wherein the display control means changes the display mode of the operation area at a position corresponding to the operation position received by the input means.
According to the present invention, the operation position detected by the display device can be indicated to the user by the change in the display mode of the operation area. Further, by changing the display mode of the operation area, it is possible to easily perform an operation following this operation, and to further improve the operability.

Further, the present invention is the display device, wherein the display device is a head-mounted display device that allows a user to visually recognize an image, and the display means displays an image using display image data. An image light generation unit that generates and emits light; and an optical member that guides the emitted image light to the user's eyes, and the external light passes through the optical member and is used together with the image light. It is characterized by being incident on a person's eyes.
According to the present invention, an input operation is more easily performed by a see-through display device that guides image light to a user's eye by an optical member and transmits external light to the user's eye through the optical member. It becomes possible.

In order to achieve the above object, the present invention provides a control method for a see-through type display device that outputs image light superimposed on external light and visually recognizes the image, and arranges an operation area on the image, A position instruction operation at a position corresponding to the operation area is accepted, and the mode of the operation area in the image is variable.
According to the present invention, since the image is displayed by the image light that is output to be superimposed on the external light, the user can view the external scene together with the operation area displayed by the image light, and can view an arbitrary place of the external scene. The operation can be performed. And since the mode of the displayed operation area is changed, the display mode of the operation area can be changed according to the state of the operation, comfortable operability can be realized, and operability can be improved.

  According to the present invention, it is possible to realize a comfortable operability by changing the display mode of the operation area according to the operation state, and to improve the operability.

1 is an external view of a head-mounted display device according to an embodiment of the present invention. It is a figure which shows schematic structure of the optical system of a head mounting type display apparatus. It is a functional block diagram of a control system of a head mounted display device. It is explanatory drawing which shows the example of the input auxiliary | assistant image displayed by a head mounted display apparatus, and input operation, (A) shows the example using a virtual keyboard, (B) and (C) show an input auxiliary line. An example of using a virtual input board is shown. It is explanatory drawing which shows the example of a display of an input auxiliary image, (A) shows the example in case a user gazes at a distant place, (B) shows the example in case a user gazes near. It is a flowchart which shows operation | movement of the head mounted display apparatus HM. It is explanatory drawing which shows the example of a display of an input auxiliary image, (A) shows the state during input operation, (B) shows the state by which the input content was decided. It is explanatory drawing which shows another example of a display of an input auxiliary image. FIG. 8A shows an example in which the designated position during the input operation is highlighted, and FIG. 8B shows an example in which the display area and the input auxiliary image are displayed side by side.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is an external view of a head-mounted display device according to an embodiment of the present invention.
The head-mounted display device HM is a head-mounted display device mounted on the user's head and is called a head mounted display (HMD). The head-mounted display device HM of the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene.

As shown in FIG. 1, the head-mounted display device HM includes an image display unit 20 that is mounted on the user's head and a control device (controller) 10 that controls the image display unit 20. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, and a right optical image display unit 26.
The right optical image display unit 26 and the left optical image display unit 28 are located in front of the user's right eye and left eye, respectively, and are connected and integrated with each other at a position corresponding to the user's eyebrow. The right holding unit 21 extends from the end ER of the right optical image display unit 26, and the left holding unit 23 extends from the end EL of the left optical image display unit 28.

The right holding part 21 is located on the side of the right side of the user, and its tip engages with the right ear of the user. Moreover, the left holding part 23 is located on the side of the left side of the user, and the tip thereof engages with the user's left ear. A head-mounted display is provided by the tips of the right holding unit 21 and the left holding unit 23 and a nose pad (not shown) provided at the lower end of the joint of the right optical image display unit 26 and the left optical image display unit 28. The device HM is supported.
A right display driving unit 22 is built in the base end of the right holding unit 21, and a left display driving unit 24 is built in the base end of the left holding unit 23. With the right display drive unit 22 and the left display drive unit 24, the head-mounted display device HM causes the user's right eye and left eye to visually recognize an image.

A dimming plate 20A is provided on the front side, that is, the front side of the right optical image display unit 26 and the left optical image display unit 28. The light control plate 20A can be attached to and detached from the right optical image display unit 26 and the left optical image display unit 28, and a plurality of types of light control plates 20A can be exchanged and mounted. The light control plate 20A is a thin plate-like optical element that has substantially no light transmission, is nearly transparent, has a light amount attenuated to transmit light, has a specific wavelength attenuated or reflected, etc. Various types can be used, and the amount of external light incident on the right optical image display unit 26 and the left optical image display unit 28 from the outside can be adjusted by appropriately selecting the optical characteristics of the light control plate 20A. In the present embodiment, a case will be described in which at least a light control plate 20A having such a light transmittance that a user wearing the head-mounted display device HM can visually recognize an outside scene is used. The light control plate 20A also has a function of protecting the right light guide plate 261 and the left light guide plate 262, which will be described later, from damage, adhesion of dirt, and the like.
Further, a camera 61 is disposed at the boundary between the right optical image display unit 26 and the left optical image display unit 28 on the front surface of the head-mounted display device HM. The position of the camera 61 is approximately the center of the front surface of the head-mounted display device HM, and is an intermediate position between the left and right eyes of the user.

  The shooting direction of the camera 61, that is, the angle of view, images at least a part of the outside view in the direction of the user's field of view when the head-mounted display device HM is worn, in other words, the front-side direction of the head-mounted display device HM. It is a direction, and is preferably a direction in which an outside scene in a range visually recognized by the user through the surface 262A is photographed. More preferably, the range including the range visually recognized through the surface 262A is captured and a range wider than this range is photographed. The camera 61 shoots a still image intermittently or captures a moving image and outputs captured image data. In the present embodiment, the camera 61 is exemplified as one digital camera, but a stereo video camera including a plurality of digital cameras may be employed.

FIG. 2 is a diagram illustrating a schematic configuration of an optical system in the image display unit 20. FIG. 2 is a plan view showing the configurations of the left display drive unit 24 and the left light guide plate 262 that allow the user's left eye LE to visually recognize a virtual image. Since each configuration for allowing the user's left eye and right eye to visually recognize a virtual image is bilaterally symmetric, only the left display drive unit 24 and the left light guide plate 262 will be described here.
The left display driving unit 24 of the image display unit 20 includes a left backlight 222 having a light source such as an LED and a diffusion plate, and a transmissive left LCD disposed on the optical path of light emitted from the diffusion plate of the left backlight 222. (Liquid Crystal Display: liquid crystal display) 242 and a left projection optical system 252 including a lens group that guides the image light L transmitted through the left LCD 242 are provided.
The left projection optical system 252 is configured by a collimating lens that converts the image light L emitted from the left LCD 242 to light beams in a parallel state. The image light L that has passed through the left projection optical system 252 enters the left light guide plate 262. The left light guide plate 262 is a prism formed with a plurality of reflecting surfaces that reflect the image light L, and the image light L is guided to the left eye LE side through a plurality of reflections inside the left light guide plate 262. The image light L reflected by the surface 262A located in front of the left eye LE is emitted from the right optical image display unit 26 toward the left eye LE, and this image light L forms an image on the retina of the left eye LE. Let the user see the image.
The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “light guide unit”. As long as this light guide unit forms a virtual image in front of the user's eyes using image light, any method can be used. For example, a diffraction grating or a transflective film may be used. .

  The surface 262A is formed of a half mirror. For this reason, not only the image light L reflected by the surface 262A but also the external light OL transmitted through the light control plate 20A is incident on the user's left eye LE. That is, the head-mounted display device HM superimposes the image light L of the internally processed image and the external light OL on the eyes of the user, and for the user, the light control plate of the head-mounted display device HM. The outside scene can be seen through 20A, and the image by the image light L is visually recognized over the outside scene. For this reason, it can be said that the head-mounted display device HM is a see-through display device.

  The image light L guided to both eyes of the user of the head-mounted display device HM forms an image on the retina of the user, so that the user visually recognizes the virtual image. Here, the visibility of the outside scene and the virtual image for the user is affected by the balance of the light amount of the external light OL and the image light L. When the light amount of the external light OL is larger than the image light L, the external scene can be seen well while the head is visible. The visibility of the virtual image processed (displayed) by the part-mounted display device HM is low. On the other hand, when the light amount of the image light L is larger than the external light OL, the visibility of the virtual image processed (displayed) by the head-mounted display device HM is high, but the visibility of the outside scene is low.

Returning to FIG. 1, the head-mounted display device HM includes a connection unit 40 that connects the image display unit 20 to the control device 10. The connection unit 40 includes a main body cord 48 connected to the control device 10, and two right cords 42 and a left cord 44 that branch from the main body cord 48 via a connecting member 46. The right cord 42 is inserted into the housing of the right holding unit 21 and connected to the right display driving unit 22. The left cord 44 is inserted into the housing of the left holding unit 23 and connected to the left display driving unit 24.
The image display unit 20 and the control device 10 transmit various signals via the connection unit 40. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

The control device 10 is a device for operating the head-mounted display device HM. The control device 10 includes a lighting unit 12, a touch pad 14, a cross key 16, and a power switch 18.
The lighting unit 12 includes an indicator composed of an LED or the like, and notifies the operation state (for example, power ON / OFF, etc.) of the head-mounted display device HM according to the light emission state. The touch pad 14 detects a contact operation with a user's finger and outputs a signal corresponding to the detected content. The cross key 16 is a key corresponding to the up / down / left / right directions, and outputs a signal corresponding to the pressing operation. The power switch 18 is a switch for switching the power state of the head-mounted display device HM.

FIG. 3 is a functional block diagram showing the configuration of the control system of the head-mounted display device HM.
The control device 10 includes an input information acquisition unit 110, a storage unit 120, a power supply 130, a control unit 140, an interface 180, and transmission units (Tx) 51 and 52. The aforementioned units are connected to each other by a bus (not shown).

  The input information acquisition unit 110 acquires a signal corresponding to an input from the touch pad 14, the cross key 16, the power switch 18, and the like, for example. The storage unit 120 stores programs and data using a semiconductor memory element, a magnetic storage device, or the like. The power source 130 includes, for example, a secondary battery and supplies power to each unit of the head-mounted display device HM.

  The control unit 140 includes a CPU, a ROM, a RAM, and the like, and controls each unit of the head-mounted display device HM by executing a program stored in the ROM or the storage unit 120. The control unit 140 executes the above-described program, thereby allowing the input detection unit 142 (input unit), the distance detection unit 144 (distance detection unit), the input auxiliary image processing unit 146 (display control unit), the image processing unit 160, the sound. It also functions as the processing unit 170 and the display control unit 190.

  The image processing unit 160 generates a signal to be supplied to the image display unit 20 based on content input via the interface 180 and transmits the signal to the image display unit 20. When an analog signal is input via the interface 180, the image processing unit 160 performs A / D conversion processing to generate digital image data (Data), and generates a clock signal (PCLK) and a vertical synchronization signal ( VSync) and a horizontal synchronizing signal (HSync). In addition, when digital image data is input to the interface 180, the image processing unit 160 performs various processes such as frame rate conversion and resolution conversion, and the processed digital image data is converted into a clock signal, a vertical synchronization signal, and a horizontal signal. Output sync signal.

A signal output from the image processing unit 160 is input to the right display drive unit 22 of the image display unit 20 through the transmission unit 51 and input to the left display drive unit 24 through the transmission unit 52.
The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 uses the control signal to turn on / off the right LCD 241 by the right LCD control unit 211, turn on / off the right backlight 221 by the right backlight control unit 201, and control the left LCD. The left display drive unit 22 and the left display drive unit 24 are controlled by individually controlling the ON / OFF of the left LCD 242 by the unit 212 and the ON / OFF of the left backlight 222 by the left backlight control unit 202. Each controls the generation and emission of image light. For example, the display control unit 190 may cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, generate only one image light, or neither may generate image light.
The display control unit 190 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. In addition, the display control unit 190 transmits control signals for the right backlight control unit 201 and the left backlight control unit 202, respectively.

The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and supplies the acquired audio signal to the right earphone 32 and the left earphone 34 of the image display unit 20 via the connection unit 40.
The interface 180 is an interface for connecting various external devices OA that are content supply sources to the control device 10. The interface 180 includes various connectors and interface circuits that are wired to the external device OA of the head-mounted display device HM, and / or a wireless communication unit and interface circuit that transmit and receive radio signals to and from the external device OA. Yes. The interface 180 can include, for example, an interface conforming to the USB standard, an interface for a memory card, a wireless communication interface conforming to a wireless LAN or Bluetooth (registered trademark) standard, another short-range wireless communication interface, or the like. Examples of the device OA include a personal computer PC, a mobile phone terminal, and a game terminal. In addition, a connector for inputting an analog signal may be provided in the interface 180.

The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and a camera 61. And a right earphone 32 and a left earphone 34.
The right display driving unit 22 includes a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221, a right LCD control unit 211 and a right LCD 241 that function as display elements, A projection optical system 251.
The reception unit 53 receives a signal input from the transmission unit 51. The right backlight control unit 201 drives the right backlight 221 based on the control signal input to the reception unit 53. The right LCD control unit 211 drives the right LCD 241 based on the clock signal, vertical synchronization signal, horizontal synchronization signal, and right-eye image data input to the reception unit 53. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix, and is driven by the right LCD control unit 211 to draw various images.

Similarly, the left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222, a left LCD 242 and a left LCD 242 that function as display elements, A projection optical system 252.
The right backlight control unit 201, the right LCD control unit 211, the right backlight 221, and the right LCD 241 are collectively referred to as an “image light generation unit” on the right side, and the left backlight control unit 202 and the left LCD control unit 212. The left backlight 222 and the left LCD 242 are collectively referred to as the “image light generation unit” on the left side. Further, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, the right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”, and the right optical image display unit. 26 and the left optical image display unit 28 are also simply referred to as “optical image display unit”.
The display drive unit and the image light generation unit constitute the display means of the present invention. The display means may include an optical image display unit or a light guide unit.

The reception unit 54 receives a signal input from the transmission unit 52. The left backlight control unit 202 drives the left backlight 222 based on the control signal input to the reception unit 54. The left LCD control unit 212 drives the left LCD 242 based on the clock signal, vertical synchronization signal, horizontal synchronization signal, and left-eye image data input to the reception unit 54. The left LCD 242 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix, and is driven by the left LCD control unit 212 to draw various images.
In the following description, a virtual image visually recognized by the user with the head-mounted display device HM is referred to as an “image”. This image is recognized by forming an image formed on the right LCD 241 and the left LCD 242 on the retina of the user. In addition, letting the user visually recognize a virtual image by the head-mounted display device HM is referred to as “display”.

In the head-mounted display device HM of the present embodiment, characters and figures are displayed by performing operations based on the input auxiliary images in the shooting range of the camera 61 in a state where the input auxiliary images for inputting characters and figures are displayed. Can be entered.
This operation is performed by moving an indicator such as a user's finger, a pen-shaped or stick-shaped operation device, or a light source such as a laser pointer or LED within the shooting range of the camera 61.
The input auxiliary image is an image displayed by the image display unit 20 in order to input a character or a figure with an indicator, and is, for example, a virtual keyboard on which characters are arranged, a ruled line for inputting handwritten characters, or the like. The input auxiliary image will be described in detail later.

The control unit 140 includes an input detection unit 142. The input detection unit 142 acquires captured image data of the camera 61 and analyzes the captured image data to detect the position and / or movement of the indicator. The storage unit 120 stores operation detection data 122 including the shape and color characteristics of the indicator. The input detection unit 142 extracts an image of the indicator from the captured image data based on the operation detection data 122 and detects the indication position of the indicator. The indication position of the indicator is, for example, the tip of the user's finger, the tip of the operation device, the light emission position of the light source, and the like. Furthermore, the input detection unit 142 detects the indication position of the indicator from a plurality of still image captured image data or a plurality of frames constituting the moving image data, and the plurality of indication positions are detected. By comparing, the time change of the designated position, in other words, the movement or locus of the designated position is detected.
The operation detection data 122 includes data indicating the movement of the indicated position or the condition of the locus. This data is, for example, data indicating the direction of movement of the designated position, the magnitude of the movement, the speed of movement, the time of a continuous movement, the shape of the locus, and the like. The input detection unit 142 detects that a specific operation has been performed when the movement or locus of the designated position detected from the captured image data satisfies the condition included in the operation detection data 122. The input detection unit 142 may detect a plurality of types of operations using a plurality of different conditions included in the operation detection data 122.

In addition, the control unit 140 includes a distance detection unit 144 that detects a distance to an object located in front of the image display unit 20, that is, in the shooting direction of the camera 61. The distance detection unit 144 detects a distance to a front object (target object), for example, using a distance measurement function for adjusting the focus of the camera 61. Note that the distance detection unit 144 may include an infrared sensor or an ultrasonic sensor and detect the distance using these sensors.
In the present embodiment, the distance detection unit 144 detects the distance to the surface located in front of the image display unit 20. The detected surface may be a fixed surface or a surface of a movable object, for example, a wall surface, a side surface or top surface of furniture fixtures, a floor surface, the ground, a book or paper, or a user's palm. For example, the surface of the object to be included.
The surface detected by the distance detection unit 144 is not limited to a flat surface, and may be a curved surface. For example, the distance detection unit 144 performs distance detection in a plurality of directions in front of the image display unit 20, and detects the distance to this surface when it is clear that one surface exists from the detected distance. . In this case, it is desirable that the distance detection unit 144 can recognize that it is one surface, and for example, it is preferable that the curved surface is not an extremely high curved surface. The distance detection unit 144 also recognizes the front surface of the image display unit 20 by extracting the surface from the captured image data captured by the camera 61 together with the distance detection result in front of the image display unit 20. May be.

Further, the control unit 140 includes an input auxiliary image processing unit 146. The input auxiliary image processing unit 146 generates data for displaying the input auxiliary image and outputs the data to the display control unit 190 when performing an input operation using the indicator, and the right display driving unit 22 and the left display driving unit. 24.
The storage unit 120 stores input auxiliary image data 124 including image data and vector data for displaying the virtual keyboard and ruled lines, for example. The input auxiliary image data 124 may include data for displaying a plurality of types of input auxiliary images. In this case, the input auxiliary image processing unit 146 selects and uses data from the input auxiliary image data 124. To do.

FIG. 4 is an explanatory diagram illustrating an example of an input auxiliary image and an input operation displayed by the head-mounted display device HM. FIG. 4A shows an example using a virtual keyboard, and FIGS. 4B and C show examples using input auxiliary lines. 4A to 4C are shown as images visually recognized by the user.
In the example shown in FIG. 4A, the virtual keyboard 301 is displayed under the control of the input auxiliary image processing unit 146. The virtual keyboard 301 has characters arranged in the same manner as a general QWERTY keyboard, and a frame for selecting each character is displayed.
Since the user can visually recognize the virtual keyboard 301 and the outside scene in an overlapping manner, when the finger F is used as an indicator as in the example of FIG. Operate so that it overlaps with. In the example of FIG. 4A, the tip of the finger F is at a position where it overlaps the character “O”. Here, for example, when an operation such as holding the finger F stationary for a predetermined time, bending the tip of the finger F (for example, the first joint), or moving the tip of the finger F in a specific direction, a character overlapping the tip of the finger F is performed. ("O" in the example in the figure) is selected and input.
In the head-mounted display device HM, the input detection unit 142 detects an operation with the finger F based on the captured image data of the camera 61, and the display position of the virtual keyboard 301 in the user's field of view (the position where the user visually recognizes). The character selected by the operation of the finger F is specified by comparing the detected position of the tip of the finger F.

  In the example shown in FIG. 4B, the virtual input board 302 in which ruled lines are arranged at equal intervals in the vertical direction and the horizontal direction is displayed under the control of the input auxiliary image processing unit 146. The virtual input board 302 is used when characters and figures are drawn and input by the movement of the indicator. For example, as shown in FIG. 4B, characters are input one by one into the square formed by the ruled line of the virtual input board 302. When the virtual input board 302 is displayed, the user draws a character or a figure by moving an indicator (a finger F in this example) that is visually recognized while being superimposed on the virtual input board 302 in the square. The input detection unit 142 detects the position of the tip of the finger F, compares it with the display position of the virtual input board 302, and determines that the trajectory of the finger F in one square indicates a specific character. Determines that the character is the input character. In the example of FIG. 4B, the letter “W” is input to the upper left corner of the virtual input board 302, and the letter “O” is drawn on the adjacent box.

Further, as shown in FIG. 4C, the input auxiliary image processing unit 146 can display a virtual input board 303 having more cells. The virtual input board 303 has a larger number of ruled lines and a smaller interval between the ruled lines than the virtual input board 302 of FIG. For this reason, each square is small and the number of squares is large. The mode of input operation is the same as that of the virtual input board 302. When a character or figure is drawn at the tip of the finger F in the square, the character or figure is input as an input to the square.
Image data or vector data for displaying the virtual keyboard 301 and the virtual input boards 302 and 303 is stored in the storage unit 120 as input auxiliary image data 124.

  Further, the area where the virtual keyboard 301 and the virtual input boards 302 and 303 which are input auxiliary images are displayed corresponds to an area that can be operated by the user, that is, an operation area. The input auxiliary image may be arranged in the maximum range in which the right display drive unit 22 and the left display drive unit 24 can form an image on the right LCD 241 and the left LCD 242, that is, the entire displayable region. Corresponds to the operation area. Further, the input auxiliary image may be arranged in a part of the displayable area.

By the way, the user wearing the head-mounted display device HM sees the virtual image visually recognized by the right display driving unit 22 and the left display driving unit 24 and the outside scene so that the appearance of the virtual image is affected by the outside scene. is there. Specifically, the size of the virtual image looks different depending on the distance to the outside scene visually recognized by the user.
FIGS. 5A and 5B are explanatory diagrams showing an example of display of the input auxiliary image. FIG. 5A shows an example in which the user gazes far away, and FIG. 5B shows an example in which the user gazes near. 5A and 5B are examples in which virtual input boards 302 and 303 are displayed in the entire displayable area of the image display unit 20. In addition, reference numeral VA in the figure indicates the user's field of view.

Like the surface 262A illustrated in FIG. 2, the size of the half mirror surface that reflects the image light L toward the user's eyes in the right light guide plate 261 and the left light guide plate 262 does not change, and the size of the right LCD 241 and the left LCD 242 is large. It does not change. Therefore, the relative size of the image formed by the image light L with respect to the entire field of view of the user does not change. On the other hand, when the user is gazing at a distance, a wide range is accommodated in the user's field of view, and when the user is gazing near, the majority of the field of view is occupied by a small range.
For example, as shown in FIG. 5A, when the user is gazing at a distant landscape or the like, the outside scene that overlaps the virtual input board 302 is a very wide range of scenery. Therefore, the user feels that the virtual input board 302 itself is large.
FIG. 5B is an example in which the user is gazing at the desk D located near the user, and the top surface of the desk D is visible in almost the entire visual field VA. In this case, the outside scene overlapping the virtual input board 303 is almost the size of the desk D, which is smaller than the example of FIG. For this reason, it seems to the user that the virtual input board 303 is the same size as the desk D.
Regarding such a difference in appearance, the distance to the place where the user gazes in the outside scene visible to the user is a dominant factor.

Therefore, the head-mounted display device HM changes the display mode of the input auxiliary image according to the distance to the outside scene that the user is gazing at.
Specifically, as shown in FIG. 5A, when the outside scene that the user gazes at is far away, a virtual input board 302 with a wide ruled line interval is displayed as an input auxiliary image, and FIG. When the user is gazing at the near view, the virtual input board 303 having a narrow ruled line interval is displayed as the input auxiliary image.

In this case, the input auxiliary image processing unit 146 selects an input auxiliary image suitable for the distance detected by the distance detection unit 144 from the input auxiliary images that can be displayed based on the input auxiliary image data 124, and displays the display control unit. It is displayed by 190. The number of input auxiliary images is arbitrary. For example, the distance detected by the distance detection unit 144 may be divided into a plurality of stages, and a plurality of input auxiliary images corresponding to each stage may be displayed.
The distance detection unit 144 detects the distance to the object in the user's visual field direction by using the distance detection function described above. The distance detection unit 144 may include, for example, a mechanism for detecting a user's line of sight, and may detect a distance to an object positioned in the user's line of sight.

  According to this configuration, by changing the mode of the input auxiliary image corresponding to the size of the input auxiliary image felt for the user, the user can perform an input operation without feeling uncomfortable, and the operability is improved. Can do.

  Further, the distance detection unit 144 detects the distance to the virtual input surface. In the example of FIG. 5B, the user's field of view includes the top surface of the desk D, and the distance detection unit 144 detects the distance to the desk D. In this case, it is conceivable that the user performs an operation of placing a finger on the top of the desk D and drawing a character or a figure. The same applies when the user's field of view has a wall or floor. Therefore, the distance detection unit 144 detects the distance to the surface, thereby appropriately detecting the distance to the object that the user is gazing for the input operation, and changing the mode of the input auxiliary image more appropriately. be able to. For this reason, the surface on which the distance detection unit 144 detects the distance is referred to as a virtual input surface as a surface on which the user performs an input operation.

  Alternatively, the distance detection unit 144 may detect the distance to the indicator, and the input auxiliary image processing unit 146 may determine the display mode of the input auxiliary image based on the distance to the indicator. Since the virtual input surface described above is a surface on which the indicator is operated, the distance to the indicator can be regarded as the distance to the virtual input surface. That is, the distance detection unit 144 does not detect the surface itself that can be a virtual input surface, but indirectly detects the distance to the virtual input surface by detecting the distance to the indicator. Since the input detection unit 142 is configured to detect the indicator from the captured image data of the camera 61, the distance detection unit 144 can easily detect the indicator. Therefore, it is useful when it is difficult for the distance detection unit 144 to detect the distance to the surface. In addition to the indicator, the distance detection unit 144 detects the distance to an object (whether or not having a specific surface is not limited) that exists in the user's field of view, and the distance to the virtual input surface is detected. It may be a distance.

FIG. 6 is a flowchart showing the operation of the head-mounted display device HM, and particularly shows the operation related to the input operation using the input auxiliary image.
When the head-mounted display device HM is turned on and an input start is instructed by an operation on the touch pad 14 or the cross key 16 of the control device 10 (step S11), the control unit 140 operates as shown in FIG. To start.
First, the distance detection unit 144 detects the distance to the virtual input surface (step S12). Next, the input auxiliary image processing unit 146 determines the display mode of the input auxiliary image based on the distance detected by the distance detection unit 144, and outputs the image data to the display control unit 190 (step S13). As a result, the input auxiliary image is displayed by the right display driving unit 22 and the left display driving unit 24. Here, the input detection unit 142 starts detecting an input operation based on the captured image data of the camera 61.

  Thereafter, the input detection unit 142 detects an input operation until the input content is confirmed (step S15). When the input content is confirmed (step S15; Yes), the process is executed based on the confirmed content. (Step S16). For example, the control unit 140 shifts to a state in which the control device 10 executes the function of the web browser and inputs a search keyword to the web search engine by the web browser, and displays the virtual input board 302. When a plurality of characters are input by an operation using the virtual input board 302, the control unit 140 transmits a character string including the input characters to the search engine as a search keyword, and displays the search result using a web browser. I do.

  The control unit 140 determines whether or not the input operation is instructed by operating the touch pad 14 or the cross key 16 (step S17), and until the input operation is instructed (step S17; No). Returning to step S15, the display of the input auxiliary image is continued. When an instruction to end the input operation is given (step S17; Yes), the input auxiliary image processing unit 146 stops the display of the input auxiliary image (step S18), and the process ends.

FIG. 7 is an explanatory diagram showing a display example of an input auxiliary image, where (A) shows a state during an input operation, and (B) shows a state in which the input content is confirmed.
7A and 7B illustrate an operation in which the control unit 140 executes a web browser and inputs a search keyword to be input to the search engine as described above.
In FIG. 7A, the virtual input board 302 is displayed in a part of the displayable area, arranged in the virtual input board 302, the search character string input box 305 is displayed, and the virtual input board 302 is used to enter the input box 305. This is the state where the input is performed. In this example, the control unit 140 recognizes the drawn character by moving the finger F so as to draw a character in accordance with the square of the virtual input board 302, and the recognized character is applied to each square of the virtual input board 302. Is displayed.

  When the input content based on the virtual input board 302 is confirmed, the input auxiliary image processing unit 146 ends the display of the virtual input board 302. In the example shown in FIG. 7B, instead of the virtual input board 302, an input content display unit 306 that displays input content is switched and displayed. In addition, a character string composed of characters displayed on the input content display unit 306 is in a state of being input to the input box 305. When the input content display unit 306 is displayed and an instruction for performing an input operation again is input by the touch pad 14, the cross key 16, or the like, the virtual input board 302 is switched and displayed again.

In the example of FIG. 7, the entire background of the virtual input board 302 is displayed in a dark color (for example, dark gray). As described with reference to FIG. 2, the visibility with which the user visually recognizes the outside scene is the amount of the external light OL that passes through the light control plate 20 </ b> A and enters the user's eyes, the right light guide plate 261, and the left light guide plate. It is influenced by the balance with the light quantity of the image light L (FIG. 2) which injects into a user's eyes from 262. For this reason, if the virtual input board 302 is displayed in a dark color, the user can easily view the outside scene. As a result, the effect of changing the display mode of the virtual input board 302 corresponding to the distance to the object to be watched by the user becomes more prominent, and further improvement in operability can be expected. Only a part of the virtual input board 302 may be displayed in a dark color.
Also, in order to prevent a sense of discomfort when switching between the virtual input board 302 and the input content display unit 306, the input content display unit 306 is an image having a dark background, like the virtual input board 302. In the example of FIG. 7, the ruled lines and frame lines of the virtual input board 302 and the input content display unit 306 are displayed in a darker color than the background color, and the input characters have a high brightness ( For example, white). In addition, the ruled lines and the frame lines may be colored with a higher brightness than the background, such as displaying the ruled lines and the frame lines in gray with the background black. The background, ruled lines, frame lines, characters, and the like are not limited to achromatic colors, and may be displayed in colors with higher saturation.

While the input auxiliary image processing unit 146 displays the input auxiliary image and the input detection unit 142 detects an operation, the control unit 140 intermittently performs distance detection by the distance detection unit 144 and is detected. The display mode of the input auxiliary image may be changed based on the distance. In this case, when the object to be watched by the user changes during the operation by the indicator, the display mode of the input auxiliary image can be adapted to the change, and good operability can be maintained.
Furthermore, when it is detected that the user's line-of-sight direction has moved during the display of the input auxiliary image by the mechanism for detecting the change in the captured image data of the camera 61 and the user's line-of-sight, the position in the user's line-of-sight direction after the movement is detected. The distance to the object to be detected may be detected by the distance detection unit 144, and the display mode of the input auxiliary image may be changed.
When the display mode of the input auxiliary image is changed during the display of the input auxiliary image as in the above example, the display of the input auxiliary image is temporarily stopped and the state where the input auxiliary image is not displayed is maintained for a predetermined time. Then, the changed input auxiliary image may be displayed.

In addition, when it is detected that the direction of the user's line of sight has moved during the display of the input auxiliary image by the mechanism that detects the change in the captured image data of the camera 61 or the user's line of sight, the display of the input auxiliary image is stopped. When the user's line-of-sight direction returns to the original position or the predetermined range from the original position, the input auxiliary image may be displayed again in the original display mode.
Further, when the input auxiliary image is displayed in a part of the displayable area, when it is detected that the user's line of sight moves while the input auxiliary image is displayed, the movement in the line of sight is within a predetermined range. If present, the display position of the input auxiliary image in the displayable area can be changed in a direction that cancels the movement in the line-of-sight direction. In this case, when the line of sight moves due to fluctuations in the user's line of sight or head position, the input auxiliary image is stably displayed at the same position, making the input operation easier without feeling the effects of fluctuations. Become. When the image display unit 20 can detect the posture using a gyro sensor or the like, the display position of the input auxiliary image may be moved as described above based on the detected change in the posture of the image display unit 20.

  The input auxiliary image processing unit 146 may change the display mode of the input auxiliary image in accordance with the operation of the touch pad 14 or the cross key 16. The display mode of the input auxiliary image is determined by the input auxiliary image processing unit 146 according to the distance to the virtual input surface detected by the distance detection unit 144. When it is desired to change the input auxiliary image according to the user's intention, the input auxiliary image may be changed by operating the touch pad 14 or the cross key 16 of the control device 10. Further, the user inputs a distance to the virtual input surface by operating the touch pad 14, the cross key 16 or the indicator detected by the input detection unit 142, and the input auxiliary image processing unit 146 inputs based on the input distance. The display mode of the auxiliary image may be determined / changed.

The input auxiliary image processing unit 146 not only changes the number of ruled lines of the virtual input boards 302 and 303 but also changes the display size of the virtual input boards 302 and 303 as input auxiliary images to the distance detected by the distance detection unit 144. It may be changed accordingly.
In this case, the input auxiliary image processing unit 146 changes the image data of the input auxiliary image to be output to the display control unit 190, so that the right LCD 241 and the left LCD 242 form an image with a plurality of pixels arranged in a matrix. The number of pixels constituting the input auxiliary image can be changed.

  The input auxiliary image processing unit 146 may determine the type of the indicator based on the captured image data captured by the camera 61, and may change the display mode of the input auxiliary image according to the type of the indicator. For example, when the indicator is a user's finger, a virtual input board 302 with a wide ruled line interval is displayed as an input auxiliary image, and when the indicator is a bar-shaped operation device, a virtual input board 303 is displayed. May be. Since the tip of the operation device can be made thinner than the user's finger, fine input operations are possible when using the operation device, and when using the finger, the spacing between the ruled lines is widened to increase the size of characters and figures. Recognizability can be ensured and operability can be expected to improve.

The display mode of the input auxiliary image is not limited to the above example.
FIG. 8 is an explanatory diagram showing another display example of the input auxiliary image. FIG. 8A shows an example in which the designated position during the input operation is highlighted, and FIG. 8B shows an example in which the display area and the input auxiliary image are displayed side by side.
In the example of FIG. 8A, the virtual input board 302 is displayed in a part of the displayable area as in FIG. 7A, and the search character string input box 305 is displayed side by side on the virtual input board 302. This is a state in which input to the input box 305 is performed using the virtual input board 302. In the example of FIG. 8A, the designated area 302A including the tip of the finger F that is the designated position is highlighted. For example, when the virtual input board 302 is displayed with a dark background, the indication area 302A has no background color and is in a “watermark” display state, and the tip of the finger F is more easily visible. ing. In this case, the input auxiliary image processing unit 146 sets a range of a predetermined area including the designated position detected by the input detection unit 142 in the designated area 302A, and changes the display state of the designated area 302A to other virtual input boards 302. The display is different from the part. In addition to the watermark display shown in the figure, the instruction area 302A is a color that makes it easier to visually recognize the outside scene than the background color of the virtual input board 302, or a method of making the frame of the instruction area 302A a highly saturated color that is easy to visually recognize. Also mentioned.

In each of the display examples described above, the input auxiliary image is displayed at the center of the displayable area. However, the input auxiliary image may be arranged near the end of the displayable area, or may be displayed together with other images. Good.
In the example of FIG. 8B, an application screen 307 and a virtual input board 309 are both displayed. The application screen 307 is a screen for displaying the processing result of the application program executed by the control unit 140. For example, when the control unit 140 executes a web browser and inputs to a search engine, an input box 308 for inputting a search keyword is arranged on the application screen 307, and a search result is displayed in a list below the input box. In this example, the application screen 307 is displayed with a light background so that the visibility is higher than that of the outside scene.
The virtual input board 309 is arranged below the application screen 307 so as not to hide the display of the application screen 307. The virtual input board 309 may be arranged next to the application screen 307, or a plurality of virtual input boards 309 may be arranged around the application screen 307.
The virtual input board 309 is displayed as an auxiliary input image similarly to the virtual input board 302 (FIG. 8A), and has a dark background so as to ensure the visibility of the outside scene. The input auxiliary image processing unit 146 can change the interval and the number of ruled lines of the virtual input board 309 according to the distance detected by the distance detection unit 144. As a result, the user can easily perform an input operation with the user's fingers, operation device, or the like while looking at the outside scene, and the result of the application program executed according to the content of the input operation can be visually recognized on the application screen 307.

As described above, according to the head-mounted display device HM according to the embodiment to which the present invention is applied, the display driving unit and the optical image display unit that output the image light L superimposed on the external light and visually recognize the image. An input auxiliary image processing unit 146 that arranges the input auxiliary image in the image and displays the input auxiliary image on the display drive unit and the optical image display unit, and an input detection unit 142 that receives a position instruction operation corresponding to the input auxiliary image, The input auxiliary image processing unit 146 can change the mode of the input auxiliary image in the image. According to this configuration, since the image is displayed by the image light L output superimposed on the outside light, the user can view the outside scene together with the input auxiliary image displayed by the image light L. You can perform operations. And since the mode of the displayed input auxiliary image can be changed, the display mode of the input auxiliary image can be changed according to the state of operation, and comfortable operability can be realized, thereby improving operability. be able to.
Further, since the input auxiliary image processing unit 146 can change the mode of the input auxiliary image according to the distance to the virtual input surface where the position instruction operation is performed, the input auxiliary image processing unit 146 is input in a mode corresponding to the position where the user performs the operation. Auxiliary images can be displayed for more comfortable operability.

  The head-mounted display device HM includes a distance detection unit 144 that detects a distance to the virtual input surface, and the input auxiliary image processing unit 146 displays an input auxiliary image according to the distance detected by the distance detection unit 144. Since the aspect can be changed, an operation of inputting or setting the distance to the virtual input surface can be omitted, and the operability can be further improved.

Further, the input auxiliary image processing unit 146 can arrange the input auxiliary image including the virtual input boards 302 and 303 for guiding the operation of the indicator and can change the size of the input auxiliary image. By changing the size of the input auxiliary image, it is possible to easily and accurately perform fine input operations such as character input.
Further, the right LCD 241 and the left LCD 242 of the image display unit 20 form an image with a plurality of pixels arranged in a matrix, and the input auxiliary image processing unit 146 can change the number of pixels constituting the input auxiliary image. Therefore, when the size of the input auxiliary image is changed, it is possible to prevent a decrease in resolution associated with the increase in size and a decrease in visibility associated with the decrease in size.

Further, the input auxiliary images such as the virtual input boards 302, 303, and 309 include a plurality of auxiliary lines, and the input auxiliary image processing unit 146 can change the interval of the auxiliary lines included in the input auxiliary images, and character input and the like. This makes it possible to perform detailed input operations more easily and accurately.
Further, the input auxiliary image processing unit 146 changes the display mode of the input auxiliary image at a position corresponding to the operation position received by the input detection unit 142, so that the detected operation position can be shown to the user. In addition, by changing the display mode of the input auxiliary image, it is possible to easily perform the operation following this operation, and to further improve the operability.

The above-described embodiment shows an example to which the present invention is applied, and the configuration of the present invention and the scope of application of the present invention are not limited to the configuration of the above-described embodiment. For example, the input auxiliary image is not limited to the virtual input boards 302, 303, and 309, and may be any image that assists the user's input operation. The input auxiliary image is not limited to ruled lines, and guides the operation using other graphics. There may be.
In the above-described embodiment, as an example of the head-mounted display device, the configuration including the image display unit 20 worn by the user like glasses is described as an example, but the present invention is limited to this. Instead of the image display unit 20, for example, an image display unit worn like a hat or an image display unit incorporated in a helmet or the like may be provided. Further, the earphones 32 and 34 may adopt an ear hook type or a headband type, or may be omitted.

  Furthermore, in the above embodiment, the image display unit 20 and the control device 10 are separated and connected by way of the connection unit 40, but the control device 10 and the image display unit 20 are integrated. It is also possible to adopt a configuration that is configured on the user's head. Further, the control device 10 and the image display unit 20 are connected by a longer cable or a wireless communication line. As the control device 10, a notebook computer, a tablet computer or a desktop computer display, a game machine, a portable phone, You may use the display etc. which a portable electronic device containing a smart phone and a portable media player, and another dedicated apparatus have.

  In the above embodiment, for example, the image light generation unit includes the left and right backlights 221, 222, the left and right backlight control units 201, 202, the left and right LCDs 241, 242 and the left and right LCD control units 211, 212. It was described as a configuration comprising The aspect of this embodiment is merely an example, and the image light generation unit may include a configuration unit for realizing another method together with the configuration unit described above or instead of the configuration unit. For example, the image light generation unit may include a self-luminous display such as an organic EL (Organic Electro-Luminescence) panel and a control unit that controls the light emission. In addition, for example, the image generation unit may use LCOS (registered trademark) (Liquid crystal on silicon), a digital micromirror device (DMD), or the like instead of the LCD. In this case, the image generation unit includes a light source such as an LED, and generates light by modulating light emitted from the light source with LCOS or DMD.

  Further, for example, the present invention can be applied to a laser retinal projection type head mounted display. In other words, the image generation unit includes a laser light source and an optical system that guides the laser light source to the user's eye, makes the laser light enter the user's eye, scans the retina, and forms an image on the retina. You may employ | adopt the structure which makes an image visually recognize. When a laser retina projection type head-mounted display is employed, the “image light emitting area in the image light generation unit” can be defined as an image area recognized by the user's eyes.

  The optical system that guides the image light generated by the image generation unit to the user's eye in the head-mounted display includes an optical member that transmits external light that is incident on the apparatus from the outside, and enters the user's eye together with the image light. Configuration can be adopted. Moreover, you may use the optical member which is located ahead of a user's eyes and overlaps a part or all of a user's visual field. Further, a scanning optical system that scans a laser beam or the like to obtain image light may be employed. Further, the optical member is not limited to guiding the image light inside the optical member, and may have only a function of guiding the image light by being refracted and / or reflected toward the user's eyes.

In addition, the present invention can be applied to a display device using MEMS display technology.
The head-mounted display device HM of the above embodiment includes left and right backlights 221, 222, left and right backlight control units 201, 202, left and right LCDs 241, 242 and left and right LCD control units 211, 212. The image light generation unit provided is used to guide the light generated by the image light generation unit to the eyes of the user by the right light guide plate 261 and the left light guide plate 262.
On the other hand, a scanning optical system using a MEMS mirror may be employed. That is, as an image display element, a signal light forming unit, a scanning optical system having a MEMS mirror that scans light emitted from the signal light forming unit, and an optical member on which a virtual image is formed by light scanned by the scanning optical system May be provided. In this configuration, the light emitted from the signal light forming unit is reflected by the MEMS mirror, enters the optical member, is guided through the optical member, and reaches the virtual image forming surface. When the MEMS mirror scans the light, a virtual image is formed on the virtual image forming surface, and the user recognizes the image when the virtual image is captured by the user's eyes. The optical component in this case may be one that guides light through a plurality of reflections, such as the right light guide plate 261 and the left light guide plate 262 of the above embodiment, and the virtual image forming surface is the surface 262A. It may be a half mirror surface.

  Further, the optical member for visually recognizing the image by the image light may be a hologram reflecting light guide as well as a light guide having a semi-transmissive surface such as the light guide plates 261 and 262. In this case, as a light source that emits image light, a configuration including a laser diode or an organic EL element with high coherence can be given. In addition, a transflective film or a hologram reflection system is provided on the surface of an optical member that does not have a function of guiding light to the inside located in the user's field of view, and this surface reflects image light toward the user's eyes. It is also possible to do.

Further, for example, the indicator may be a device worn on the user's finger in addition to the user's finger or operation device, and the input detection unit 142 may detect the position of this device. Specifically, the configuration may be such that the user wears a ring-type instrument on his / her finger and the head-mounted display device HM takes a picture with the camera 61 to detect the position and inclination of the ring-type instrument. In this case, the image of the ring-type device can be extracted from the captured image data of the camera 61, and the position and inclination of the ring-type device can be detected based on the shape (ellipse or circle) and size of the image of the ring-type device. . Alternatively, a configuration may be adopted in which a glove-like detection device is attached to the user's hand, and the head-mounted display device HM detects the position and inclination of the detection device.
The display device of the present invention is not limited to the head-mounted display device described above, and can be applied to various display devices such as a flat panel display and a projector.
Furthermore, it is not limited to the structure which memorize | stores the program which the control part 140 performs in the memory | storage part 120, for example, the said program is a computer-readable portable recording medium, the memory | storage device externally connected to the control apparatus 10, or The program may be stored in a device connected to the control device 10 via a wired or wireless communication line, and the control device 10 may read and execute the program as necessary. Of course, the detailed configuration of the other head-mounted display device HM can be arbitrarily changed.

  DESCRIPTION OF SYMBOLS 10 ... Control apparatus, 14 ... Touch pad, 16 ... Cross key, 20 ... Image display part, 20A ... Light control board, 22 ... Right display drive part, 24 ... Left display drive part, 26 ... Right optical image display part, 28 DESCRIPTION OF SYMBOLS ... Left optical image display part, 61 ... Camera, 120 ... Memory | storage part, 122 ... Operation detection data, 124 ... Input auxiliary image data, 140 ... Control part, 142 ... Input detection part (input means), 144 ... Distance detection part (Distance detection means), 146 ... input auxiliary image processing section (display control means), 160 ... image processing section, 190 ... display control section, 221 ... right backlight, 222 ... left backlight, 241 ... right LCD, 242 ... Left LCD, 251 ... right projection optical system, 252 ... left projection optical system, 261 ... right light guide plate (optical member), 262 ... left light guide plate (optical member), 262A ... surface, 301 ... virtual keyboard, 302, 303, 3 9 ... Virtual input board, 302A ... Instruction area, 305, 308 ... Input box, 306 ... Input content display section, 307 ... Application screen, M ... Head-mounted display device (display device), L ... Image light, OL ... Outside light.

Claims (11)

Display means for superimposing and outputting image light on external light to visually recognize the image;
Display control means for arranging an operation area on the image and displaying the operation area on the display means;
Input means for receiving a position instruction operation corresponding to the operation area,
The display control means can change a mode of the operation area in the image;
A display device. The display device according to claim 1,
The display control means, wherein the display control unit changes a mode of the operation area according to a distance to a virtual input surface on which the position instruction operation is performed. The display device according to claim 2,
A distance detecting means for detecting a distance to the virtual input surface;
The display control unit changes a mode of the operation area according to the distance detected by the distance detection unit. The display device according to claim 3,
The distance detecting means detects a surface visually recognized by the external light and detects a distance to the surface. The display device according to any one of claims 1 to 4,
The display control means arranges the operation area including an input auxiliary image for guiding the position indicating operation, and changes at least one of a size, a display color, and a display area of the input auxiliary image. apparatus. The display device according to claim 5,
The display means forms the image with a plurality of pixels arranged in a matrix,
The display control means, when changing the size of the input auxiliary image, changes the number of pixels constituting the input auxiliary image. The display device according to claim 6,
The display control unit displays an image including a plurality of auxiliary lines as the input auxiliary image by the display unit, and changes an interval between the auxiliary lines. The display device according to any one of claims 1 to 7,
The display control means causes the display means to display an image of a virtual keyboard for performing character input by the position instruction operation as the input auxiliary image, and at least one of display size, display color, and display area of the virtual keyboard. A display device characterized by changing the above. A display device according to any one of claims 1 to 8,
The display device, wherein the display control unit changes a display mode of the operation area at a position corresponding to the operation position received by the input unit. The display device according to any one of claims 1 to 9,
The display device is a head-mounted display device that allows a user to visually recognize an image,
The display means includes
An image light generation unit that generates and emits image light representing an image using display image data;
An optical member for guiding the emitted image light to the eyes of the user;
Including
The display device, wherein the external light passes through the optical member and enters the user's eyes together with the image light. A control method for a see-through display device that outputs image light superimposed on external light and visually recognizes the image,
An operation area is arranged on the image, and a position instruction operation at a position corresponding to the operation area is accepted.
Changing the mode of the operation area in the image;
A control method of a display device characterized by the above.

Images