JP2013190941A - Information input output device, and head-mounted display device - Google Patents

Abstract

To improve convenience.
An information input / output device controls information to be superimposed on a captured image based on an image capturing unit that captures an image and a hand shape included in the captured image captured by the image capturing unit. And a processing unit that outputs an image on which information is superimposed.
[Selection] Figure 7

Description

  The present invention relates to an information input / output device and a head-mounted display device.

  In recent years, a technique for displaying a three-dimensional (hereinafter referred to as 3D) character formed of polygons on a marker printed on paper or the like is known (see, for example, Patent Document 1).

Japanese Patent No. 4696184

  However, in the technique described in Patent Document 1, for example, a sheet called a marker has to be prepared separately, and it is difficult to easily display a character anywhere. As described above, the technique described in Patent Document 1 has a problem that convenience is poor.

  The present invention has been made to solve the above problems, and an object thereof is to provide an information input / output device and a head-mounted display device that can improve convenience.

  In order to solve the above problem, an embodiment of the present invention provides an information to be superimposed on the captured image based on an imaging unit that captures an image and a hand shape included in the captured image captured by the imaging unit. And a processing unit for outputting an image obtained by superimposing the information on the captured image.

  According to one embodiment of the present invention, the information input / output device described above, a display unit that displays an image output from the processing unit, and at least the imaging unit and the display unit are mounted on a user's head. It is a head-mounted display device characterized by comprising a possible mounting part.

  According to the present invention, convenience can be improved.

It is a perspective view of the head mounted display by this embodiment. It is the perspective view which looked at the head mounted display in this embodiment from the back side. It is a figure which shows the mounting | wearing form of the head mounted display in this embodiment. It is a horizontal sectional view of a display body in this embodiment. It is a perspective view which shows the remote control in this embodiment. It is a perspective view which shows the stereo earphone in this embodiment. It is a functional block diagram of the head mounted display in this embodiment. It is a functional block diagram of the process part in this embodiment. It is a figure which shows an example of the network connection of the head mounted display in this embodiment. It is a schematic diagram which shows the imaging range at the time of right eye mounting | wearing of the head mounted display in this embodiment. It is a schematic diagram which shows the imaging range at the time of left eye mounting | wearing of the head mounted display in this embodiment. It is a figure which shows switching of the imaging range and display range of the head mounted display in this embodiment. It is a flowchart which shows the switching process of the imaging range and display range of a head mounted display in this embodiment. It is a 1st figure of the flowchart which shows the process of the input / output device in this embodiment. It is a 2nd figure of the flowchart which shows the process of the input / output device in this embodiment. It is the 3rd figure of the flowchart which shows the process of the input / output device in this embodiment. It is a 1st figure which shows an example of operation | movement of the head mounted display in this embodiment. It is a figure which shows an example of determination of the back of the hand in this embodiment. It is a 2nd figure which shows an example of operation | movement of the head mounted display in this embodiment. It is a 3rd figure which shows an example of operation | movement of the head mounted display in this embodiment. It is a 4th figure which shows an example of operation | movement of the head mounted display in this embodiment. It is a 5th figure which shows an example of operation | movement of the head mounted display in this embodiment. It is a figure which shows an example in case the user is wearing gloves in this embodiment.

Hereinafter, an information input / output device and a head-mounted display device according to an embodiment of the present invention will be described with reference to the drawings.
In the following description, an XYZ orthogonal coordinate system is set as necessary, and the positional relationship of each part will be described with reference to this XYZ orthogonal coordinate system. A predetermined direction in the horizontal plane is defined as an X-axis direction, a direction orthogonal to the X-axis direction in the horizontal plane is defined as a Y-axis direction, and a direction orthogonal to each of the X-axis direction and the Y-axis direction (that is, a vertical direction) is defined as a Z-axis direction. Further, the rotation (inclination) directions around the X axis, Y axis, and Z axis are the θX, θY, and θZ directions, respectively.

  FIG. 1 is a perspective view of the head mounted display of the present embodiment. FIG. 2 is a perspective view of the head mounted display as viewed from the back side. FIG. 3 is a diagram showing a mounting form of the head mounted display of the present embodiment.

The head-mounted display 1 (head-mounted display device) is a monocular type equipped with a display body 20 and a headband 40 (mounting part) that is mounted on the head of a user (user) and supports the display body 20. It is a head mounted display. The head mounted display 1 of the present embodiment can be used with both eyes as shown in FIG. 3A shows a state where the user is viewing the display unit 60 with the right eye (right eye), and FIG. 3B shows a state where the user is viewing with the left eye (left eye). Here, the headband 40 attaches the display body 20 to the user's head.
As shown in FIG. 2, the display main body 20 and the headband 40 are configured to be detachable via a connecting pin 41.
1 and 2, the longitudinal direction of the display main body 20 is defined as the Y-axis direction, and the direction in which the headband 40 sandwiches the user's head is defined as the X-axis direction.

Hereinafter, the configuration of each part of the head mounted display 1 will be described in detail.
The display main body 20 includes a device main body 21 that includes a main circuit and includes an operation unit and various interfaces, and a display unit 60 that is connected to the tip of the device main body 21.

  The apparatus main body 21 has a substantially plate-shaped casing 21A (FIG. 4). In the present embodiment, an end portion (+ Y side end portion) on the side where the connection portion with the headband 40 is provided in the apparatus main body portion 21 is a base end portion, and an end portion on the opposite side of the base end portion ( The end on the -Y side) is the tip. Further, in a state where the apparatus main body 21 is attached to the headband 40, the headband 40 side (+ X side) of the apparatus main body 21 is the inside, and the opposite side (−X side) from the headband 40 is the outside.

As shown in FIG. 1, a main switch 28, a touch switch 34, and a sound collecting microphone 24 are arranged on the outer surface of the apparatus main body 21 along the longitudinal direction of the housing 21A.
The main switch 28 is a switch for performing an on / off operation of the display main body 20. The touch switch 34 is a touch panel that can perform various operations of the head mounted display 1 by touching the surface with a finger or the like. The sound collection microphone 24 is an external microphone that collects environmental sounds.

  As shown in FIG. 2, an ear speaker 23, an audio connector 26, and a headband hinge 32 having a connection hole 31 are provided on the base end side of the inner surface of the apparatus main body 21. A heart rate sensor 137 is provided at the center of the inner surface of the apparatus main body 21. A call microphone 37 is provided at the tip of the inner side surface of the apparatus main body 21.

  The ear speaker 23 is disposed in the vicinity of the user's ear. Audio information is transmitted from the ear speaker 23 to the user. The audio connector 26 is an audio input / output terminal to which, for example, the earphone shown in FIG. 6 is connected. The headband hinge 32 is a joint part with the headband 40. The voice of the user is input to the call microphone 37.

  The heart rate sensor 137 is a sensor that measures a user's heart rate by contacting the surface of the user's face. The heart rate sensor 137 includes a light emitting unit including a light emitting diode and the like, and a light receiving unit that detects light reflected inside the user's skin. The heart rate sensor 137 counts the heart rate by detecting a change in the amount of reflected light due to a change in blood flow. The heart rate sensor 137 is disposed near the user's eyes, but if the configuration is such that infrared light is emitted from the light emitting unit, the user will not feel glare.

A USB connector 25, an operation switch 30, and a video connector 27 are provided on the side end surface of the apparatus main body 21 on the base end side.
The USB connector 25 is a connection terminal of a USB (Universal Serial Bus) device. In the present embodiment, for example, a remote controller (remote controller) 140 shown in FIG. 5 is connected.

The operation switch 30 is a pointing device such as a trackball or a stick. The operation switch 30 is provided so as to face the screen displayed on the display unit 60. As a result, the left-right direction of the operation on the operation switch 30 matches the left-right direction of the screen, so that the user can intuitively operate the operation switch 30 while looking at the screen.
The video connector 27 is a video input / output terminal.

  As shown in FIG. 2, the headband 40 includes a pair of head pads (mounting members) 46 and 47 that sandwich the user's head, a first headband 43, a second headband 44, and a rotation mechanism 56. , 57.

  The first headband 43 is an elastic member having an arc shape as a whole. A joint portion 43 a for bending the first headband 43 is provided on the top of the first headband 43. At both ends of the first headband 43, bearing portions 43b and 43c constituting the rotation mechanisms 56 and 57 are provided, respectively. Bearing portions 43d and 43e to which the head pads 46 and 47 are connected are provided further on the front end side of the band than the bearing portions 43b and 43c.

  The second headband 44 is an elastic member having an arc shape as a whole. A joint portion 44 a for bending the second headband 44 is provided on the top of the second headband 44. At both ends of the second headband 44, shaft members 44b and 44c constituting rotation mechanisms 56 and 57 are provided.

  In the present embodiment, the second headband 44 has a configuration in which the surface of the spring member 48 made of a metal such as stainless steel is covered with a flexible material such as a resin. The spring member 48 of the second headband 44 generates a spring force that holds the user's head. The second headband 44 is formed wider than the first headband 43. The formation part of the joint part 44a is a mark display part 49 formed wider than the other band parts. A product tag or the like is attached to the mark display section 49 by sticker or printing. The entire second headband 44 may be made of metal.

  The head pad 46 includes a plate-like support plate 46a and an elastic member 46b having an arched cross section provided on one surface side of the support plate 46a. On the surface of the support plate 46a opposite to the elastic member 46b, a substantially hexagonal column-shaped connecting pin 41 is erected in a posture perpendicular to the surface. The connecting pin 41 is pivotally supported by a bearing portion 43 d provided at one end of the first headband 43. As a result, the head pad 46 can rotate around the connecting pin 41.

  The head pad 47 includes a plate-like support plate 47a and an elastic member 47b having an arcuate cross section provided on one surface of the support plate 47a. A shaft member 47c is provided on the surface of the support plate 47a opposite to the elastic member 47b. The shaft member 47 c is pivotally supported by a bearing portion 43 e provided at the tip of the first headband 43. Thereby, the head pad 47 can rotate around the shaft member 47c.

FIG. 4 is a horizontal sectional view of the display body 20.
As shown in FIG. 4, the apparatus body 21 includes a plate-like circuit board 29 extending along the longitudinal direction of the housing 21 </ b> A and a battery 33. A control circuit, a power supply circuit, and the like (not shown) are mounted on the circuit board 29, and are electrically connected to each part of the display main body 20 via a wiring (not shown).

  A display panel 36 made of a liquid crystal panel and a backlight 35 are arranged inside the touch switch 34 exposed on the outer surface of the apparatus main body 21. In the present embodiment, the display image on the display panel 36 is displayed through the touch switch 34. The display panel 36 and the backlight 35 may be an organic EL panel or an electrophoresis panel.

  The headband hinge 32 is a ball joint composed of a concavely curved housing portion 32a provided in the housing 21A and a spherical portion 32b fitted to the housing portion 32a. The spherical portion 32b has a spherical side surface portion and two plane portions formed in parallel to each other so as to sandwich the side surface portion. A connecting hole 31 is formed so as to penetrate the two flat portions vertically. The connection hole 31 is formed in a hexagonal shape when viewed in the axial direction. The display main body 20 and the headband 40 are connected by inserting the connection pin 41 of the headband 40 into the connection hole 31.

  By providing the headband hinge 32, the display main body 20 can be rotated in the A direction (around the X axis centering on the headband hinge 32) shown in FIG. In the present embodiment, the rotatable range of the display body 20 is about 270 °. By the rotation operation around the X axis, a switching function between the form in which the image is observed with the right eye shown in FIG. 3A and the form in which the image is observed with the left eye shown in FIG. 3B is realized.

  Further, since the headband hinge 32 is a ball joint, the display main body 20 can be swung in the B direction (around the Z axis centering on the headband hinge 32) shown in FIG. By this swinging operation, the position of the display main body 20 relative to the user's eyes and ears can be adjusted.

  The heart rate sensor 137 in the vicinity of the headband hinge 32 is provided so as to protrude from the inner surface of the apparatus main body 21, and can contact the surface of the user's face when the head mounted display 1 is worn. The battery 33 may be a rechargeable battery or a disposable battery.

  As shown in FIGS. 1 and 2, the display unit 60 is connected to the distal end portion of the apparatus main body unit 21. In the present embodiment, the headband 40 side of the display unit 60 is the inside, and the side opposite to the headband 40 is the outside. The display unit 60 is an arm member that has a curved shape in a top view (Z-axis view), and has a shape that curves inward from the connecting portion with the apparatus main body 21 toward the distal end side. A finder opening 67 is provided on the inner surface of the display unit 60. A camera 64 is provided on the outer surface of the display unit 60.

  As shown in FIG. 4, the display unit 60 is connected to the apparatus main body 21 via a display hinge 61. The display hinge 61 is a ball joint composed of a concave curved housing 61a formed in the housing 60A of the display 60 and a spherical portion 61b formed in the apparatus main body 21 and fitted in the housing 61a. .

  The spherical portion 61b of the apparatus main body 21 has an inclined surface extending obliquely with respect to the longitudinal direction of the casing 21A formed at the front end of the outer surface of the apparatus main body 21, and the normal direction of the inclined surface (Y in FIG. 4). It is provided so as to protrude in the “axial direction”. The display unit 60 can be freely rotated around the Y ′ axis with respect to the spherical portion 61 b of the display hinge 61.

  The spherical portion 61b of the display hinge 61 is formed with a through hole 61c that penetrates the spherical portion 61b in the height direction (Y′-axis direction). The inside of the display unit 60 and the inside of the apparatus main body unit 21 are communicated with each other through the through hole 61c. A cable (not shown) is inserted through the through hole 61c. The circuit board 29 and each part of the display unit 60 are electrically connected via the inserted cable.

  Inside the display unit 60, there are a backlight 62, a display panel 63, a camera 64, a prism 65, a reflection mirror 66, a front light 68, a front speaker 70, an imaging lens 71, and an image sensor 72. And are provided. That is, the display unit 60 includes a backlight 62, a display panel 63, a camera 64, a prism 65, a reflection mirror 66, a front light 68, a front speaker 70, an imaging lens 71, and an image sensor 72.

  The prism 65 has a configuration in which a substantially triangular first prism 65a and a second prism 65b are bonded to each other on a top view (Z-axis view). A display panel 63 made of a liquid crystal panel is provided at a position facing one of the other two surfaces of the first prism 65a other than the bonding surface. A backlight 62 that transmits and illuminates the display panel 63 is disposed on the back surface of the display panel 63. A reflection mirror 66 is disposed at a position facing the other surface of the first prism 65a. The reflection mirror 66 is positioned substantially in front of the finder opening 67.

  Of the two surfaces other than the bonding surface of the second prism 65 b, one surface is a viewfinder eyepiece disposed in the viewfinder opening 67. An imaging element 72 is disposed opposite to the other surface of the second prism 65b via the imaging lens 71.

  In the display unit 60, the image displayed on the display panel 63 is emitted from the finder opening 67 through the first prism 65a, the reflection mirror 66, and the second prism 65b, and is observed by the user. The eye of the user looking at the finder opening 67 is imaged on the image sensor 72 via the second prism 65 b and the imaging lens 71. The eye image acquired through the image sensor 72 is used for analyzing the direction of the user's line of sight, blinking, and facial expression. That is, the imaging element 72 functions as an imaging unit that images the user's eyes. The backlight 62 is also used as an illuminating device for imaging the eye using the image sensor 72.

The camera 64 (imaging unit) has, for example, an image sensor with 5 to 10 million pixels, and is configured to be able to perform an autofocus operation.
In the case of this embodiment, the imaging direction of the camera 64 is set so as to coincide with the user's line-of-sight direction. That is, the camera 64 captures an image (video) in the viewing direction of the user wearing the head mounted display 1. The adjustment between the imaging direction and the line-of-sight direction may be performed mechanically or by image processing. For example, the camera 64 is configured to be able to acquire a wide-angle image, and a part of the electrically captured image is extracted as an image in the line-of-sight direction based on the information on the line-of-sight direction acquired from the user's eye image. 63 is displayed. Thereby, it is possible to capture and display a video (image) in front of the user without providing an adjustment mechanism in the camera 64.

  The front light 68 is, for example, an LED (Light Emitting Diode) light. The front light 68 may include red, green, and blue light emitting elements, and may be configured to emit any color at any timing. The front light 68 may be used as a device that displays information to the outside depending on the light emission color and the light emission timing, or may be used as an illumination device when taking an image with the camera 64.

FIG. 5 is a perspective view showing a remote controller attached to the head mounted display 1 of the present embodiment.
The remote controller 140 includes a housing 140A, a touch pad 141, an operation switch 147, a USB connector 142, indicators 143a, 143b, and 143c, a mounting clip 144, and an indicator switch 150.

  The remote controller 140 is electrically connected to the apparatus main body 21 by connecting one terminal of the USB cable 148 to the USB connector 142 and connecting the other terminal to the USB connector 25 of the apparatus main body 21. A commercially available USB cable can be used as the USB cable 148. In the present embodiment, a plurality of clips 149 are provided in the middle of the USB cable 148. The plurality of clips 149 can be used to fix the USB cable 148 along the second headband 44, for example.

  The remote control 140 incorporates a communication circuit for performing wireless communication with the display main body 20 and other electronic devices. As a communication method of the communication circuit, for example, Bluetooth (registered trademark), wireless LAN, infrared communication, UWB (Ultra Wide Band), Transfer Jet, or the like can be used. With such a communication circuit, communication with the display main body 20 is possible even when the USB cable 148 is not connected.

  The remote control 140 may incorporate a G sensor, an acceleration sensor, and a direction sensor. Information detected by these sensors can also be communicated to the display body 20 via the communication circuit described above.

  The touch pad 141 is provided on one main surface of the housing 140A. The touch pad 141 is an operation unit for operating the display main body 20 when the user touches. A display panel and a backlight may be provided in the housing 140 </ b> A on the back side of the touch pad 141, similarly to the touch switch 34 of the apparatus main body 21.

  The operation switch 147 is a switch for manually operating the remote controller 140 and the display main body 20. The operation switch 147 may be provided as necessary. The indicator switch 150 provided on the side surface of the housing 140A is a switch for performing a predetermined operation using the indicators 143a to 143c.

The USB connector 142 is used not only for connection with the apparatus main body 21 but also for connection with a charging terminal of the remote control 140 or a PC when the remote control 140 has a built-in rechargeable battery.
Further, the USB connector 142 can also be used as a power supply terminal to the apparatus main body 21 when the remote control 140 has a built-in battery. By making the battery of the remote control 140 usable as an auxiliary power source, the head mounted display 1 can reduce the battery capacity of the device main body 21 and reduce the size and weight of the device main body 21.

  Indicators 143a, 143b, and 143c are arranged around the touch pad 141. In the present embodiment, the indicator 143a is arranged at a position along one side of the touch pad 141, and the indicator 143b and the indicator 143c are arranged on the opposite side of the touch pad 141 with respect to the indicator 143a.

  The indicators 143a to 143c include light emitting elements such as LEDs. In the present embodiment, the indicator 143a includes a green light emitting element, the indicator 143b includes an orange light emitting element, and the indicator 143c includes a blue light emitting element. These light emitting elements are configured to be capable of pattern light emission at controlled timing. The pattern light emission of the indicators 143a to 143c can be turned on and off by operating the indicator switch 150 on the side surface of the housing 140A.

  The attachment clip 144 is provided on the surface of the housing 140A opposite to the touch pad 141. In the case of this embodiment, the attachment clip 144 is an S-shaped spring plate member having one end fixed to the housing 140A. With the attachment clip 144, the remote controller 140 can be attached to a user's clothes, belt, or the like.

  Further, the head mounted display 1 may attach the remote control 140 to the headband 40 by sandwiching the vicinity of the head pad 47 with the attachment clip 144. By attaching the display main body 20 to the head pad 46 of the headband 40 and attaching the remote controller 140 to the head pad 47 side, it is possible to reduce the imbalance in the weight balance in the head mounted display 1.

  The indicators 143a to 143c are configured to blink in a predetermined pattern when the user operates the operation switch 147 with the user holding the remote control 140. For example, the color of the indicator 143a is green, the indicator 143b is orange, and the indicator 143c is blue. Can be recognized. As a result, the operation state of the remote controller 140 can be transmitted to the main body side without performing communication by Bluetooth (registered trademark), which will be described later, between the remote controller 140 and the display main body 20, that is, while suppressing power consumption to a minimum.

FIG. 6 is a perspective view showing a stereo earphone attached to the head mounted display of the present embodiment.
The stereo earphone 100 includes a connector 101, a cable 102, a first speaker 103, a second speaker 104, a sound collecting microphone 105, and a plurality of clips 106.

  The connector 101 is provided at one end of the cable 102. The connector 101 is, for example, a general 4-pole φ3.5 mm mini plug. The breakdown of the four poles is the sound collection microphone 105, the first speaker 103, the second speaker 104, and the ground (GND). The cable 102 is bifurcated in the vicinity of the connector 101, and a first speaker 103 is provided at the end of the branched cable. A second speaker 104 and a sound collecting microphone 105 are provided at the other end of the cable 102. The plurality of clips 106 are arranged on the cable 102 at a predetermined interval.

  The first speaker 103 of the stereo earphone 100 is attached to the ear of the user on the side where the display main body 20 is arranged, and the second speaker 104 is attached to the ear on the opposite side to the first speaker 103. At this time, the cable 102 can be fixed to the second headband 44 by the clip 106.

  Further, stereo recording can be performed by the sound collecting microphone 105 of the stereo earphone 100 and the sound collecting microphone 24 provided on the outer surface of the housing 21 </ b> A in the display main body 20. For example, if the display main body 20 is arranged on the right eye side as shown in FIG. 3A, the sound collecting microphone 24 of the display main body 20 collects sounds on the right side of the user and is attached to the left ear. The sound collecting microphone 105 collects the sound on the left side of the user. Note that the right channel sound is output from the first speaker 103 of the stereo earphone 100, and the left channel sound is output from the second speaker 104.

  On the other hand, when the display main body 20 is arranged on the left eye side as shown in FIG. 3B, the sound collecting microphone 24 of the display main body 20 collects the sound of the left side of the user and is attached to the right ear. The sound collecting microphone 105 collects the sound on the right side of the user. The left channel sound is output from the first speaker 103 of the stereo earphone 100, and the right channel sound is output from the second speaker 104.

FIG. 7 is a functional block diagram of the head mounted display 1.
The head mounted display 1 includes various electric circuits centering on the processing unit 123. The head mounted display 1 includes, for example, a backlight 62, a display panel 63, a camera 64, an image sensor 72, a decoder 121, a flash memory 122, a processing unit 123, an LCD driver 125, a BL driver 126, a memory 127, an encoder 129, and a main switch. 28, an operation switch 30, a touch switch 34, a BT communication circuit 130, a WiFi communication circuit 131, and a 3G / LTE communication circuit 138.
In the present embodiment, the camera 64, the image sensor 72, the processing unit 123, the memory 127, the encoder 129, the main switch 28, the operation switch 30, the touch switch 34, the BT communication circuit 130, the WiFi communication circuit 131, and 3G / LTE. The communication circuit 138 corresponds to the input / output device 200 (information input / output device). That is, the input / output device 200 includes a camera 64, an image sensor 72, a processing unit 123, a memory 127, an encoder 129, a main switch 28, an operation switch 30, a touch switch 34, a BT communication circuit 130, a WiFi communication circuit 131, and 3G / An LTE communication circuit 138 is provided.

The processing unit 123 is, for example, a CPU (Central Processing Unit), and is connected to various circuits of the head mounted display 1 and comprehensively controls the head mounted display 1. Details of the processing unit 123 will be described later.
In the case of the present embodiment, the encoder 129, the decoder 121, the power supply circuit 120, the operation switch 30, the flash memory 122, the BT communication circuit 130, the WiFi communication circuit 131, the acceleration sensor 132, the geomagnetic sensor 133, the front light 68, and 3G / LTE communication. The circuit 138, laser oscillator 73, angular velocity sensor 134, GPS sensor 135, temperature / humidity sensor 136, heart rate sensor 137, BL driver 126, memory 127, main switch 28, and touch switch 34 are connected to the processing unit 123. .

  The encoder 129 encodes (encodes) the audio signal and the video signal into audio data and video data of a predetermined method. Connected to the encoder 129 are a camera 64, an image sensor 72, a sound collection microphone 24, a call microphone 37, an audio connector 26, and a video connector 27.

  The encoder 129 encodes the audio signal supplied from the sound collection microphone 24, the call microphone 37, and the audio connector 26, and supplies the encoded audio signal to the processing unit 123. The encoder 129 encodes (encodes) a video (image) signal input from the camera 64, the image sensor 72 that captures the user's eyes, and the video connector 27, and processes the encoded video (image) signal. 123.

  The decoder 121 decodes (decodes) audio data and video data into audio signals and video signals. An LCD driver 125, a speaker amplifier 162, an audio connector 26, and a video connector 27 are connected to the decoder 121. That is, the decoder 121 decodes the audio data into an audio signal, and supplies the decoded audio signal to the speaker amplifier 162 or the audio connector 26. The decoder 121 decodes (decodes) video (image) data into a video (image) signal, and supplies the decoded video signal to the LCD driver 125 or the video connector 27.

The LCD driver 125 is a circuit that generates a driving signal for a liquid crystal panel, for example, and is connected to the display panel 36 and the display panel 63. That is, the LCD driver 125 generates a drive signal for the liquid crystal panel based on the video (image) signal supplied from the processing unit 123 via the decoder 121, and the generated drive signal for the liquid crystal panel is displayed on the display panel 36 or This is supplied to the display panel 63.
The display panel 63 (display unit) is, for example, a liquid crystal display panel, and displays a video (image) based on a drive signal supplied from the LCD driver 125. The display panel 63 displays, for example, an image captured by the camera 64 and a menu screen.

  The BL driver 126 is a circuit that generates a drive signal for backlight, for example, and is connected to the backlight 35 and the backlight 62. That is, the BL driver 126 generates a backlight driving signal based on the control signal supplied from the processing unit 123, and supplies the generated backlight driving signal to the backlight 35 and the backlight 62.

The battery 33 supplies power to each part of the head mounted display 1 via the power supply circuit 120.
The power supply circuit 120 controls the power supplied from the battery 33 (for example, voltage conversion, supply on / off) based on the control signal supplied from the processing unit 123.

  The speaker amplifier 162 is, for example, a circuit that amplifies an audio signal and outputs the amplified signal to the speaker, and is connected to the ear speaker 23 and the front speaker 70. The speaker amplifier 162 amplifies the audio signal supplied from the processing unit 123 via the decoder 121 and supplies the amplified audio signal to the ear speaker 23 or the front speaker 70.

In the present embodiment, the ear speaker 23 and the front speaker 70 are assumed to use monaural sound, and the ear speaker 23 and the front speaker 70 output a sound in which left and right audio signals are synthesized.
The memory 127 stores a control program executed by the processing unit 123.

The main switch 28 is a switch for turning on and off the entire power supply, and supplies a control signal based on the operation to the processing unit 123 when operated by a user.
The operation switch 30 is a switch for performing a pointing operation on the screen, and supplies a control signal based on the operation to the processing unit 123 when operated by a user.
The touch switch 34 is a switch that performs various operations by a touch operation, and supplies a control signal based on the operation to the processing unit 123 when operated by a user.
The display main body 20 includes a main switch 28, an operation switch 30, and a touch switch 34. The display main body 20 can be mounted on either the right side or the left side of the user's body. For this reason, the main display switch 20, the operation switch 30, and the touch switch 34 of the display main body 20 can be mounted on either the right side or the left side of the user's body, and operate the information input / output device.

The BT communication circuit 130 is a communication circuit for performing Bluetooth (registered trademark) communication with other devices.
The WiFi communication circuit 131 (communication unit) is a communication circuit for performing wireless LAN communication (IEEE 802.11) with other devices.
The 3G / LTE communication circuit 138 is a communication circuit for performing mobile communication with other devices.
The BT communication circuit 130, the WiFi communication circuit 131, and the 3G / LTE communication circuit 138 transmit / receive information such as audio data and video (image) data to / from other devices.

The acceleration sensor 132 (G sensor) detects acceleration information (for example, acceleration and gravity) and is used for detecting the tilt of the head mounted display 1. The acceleration sensor 132 supplies the detected acceleration information to the processing unit 123.
The geomagnetic sensor 133 detects azimuth information based on geomagnetism, and is used for detecting the direction of the head mounted display 1. The geomagnetic sensor 133 supplies the detected azimuth information to the processing unit 123.

The angular velocity sensor 134 (gyro sensor) detects the angular velocity and is used for detecting the rotation of the head mounted display 1. The angular velocity sensor 134 supplies the detected angular velocity to the processing unit 123.
The GPS sensor 135 is used for positioning detection using GPS (Global Positioning System), and measures the latitude and longitude of the head mounted display 1. The GPS sensor 135 supplies the measured latitude and longitude of the head mounted display 1 to the processing unit 123.

The temperature / humidity sensor 136 detects the temperature and humidity of the environment. The temperature / humidity sensor 136 supplies the detected temperature and humidity of the environment to the processing unit 123.
The heart rate sensor 137 contacts the user's cheek and detects the user's heart rate. That is, the heart rate sensor 137 detects whether or not the head mounted display 1 is worn by the user. The heart rate sensor 137 supplies the detected user's heart rate to the processing unit 123.
As described above, the laser oscillator 73 irradiates the target (object) corresponding to a predetermined position in the image displayed on the display panel 63 with laser light (spot light).

  The camera 64 (imaging unit) is configured to be able to perform an autofocus operation, and includes an optical system 641 and an image sensor 642. The camera 64 has a relatively deep depth of field, and can be handled as a so-called pan focus from 1 m (meters) to infinity, for example. Therefore, even if the camera 64 is focused on a subject at infinity, for example, a hand within 1 m can be detected even if the hand is not precisely in focus at this stage. It has become.

The optical system 641 includes, for example, a plurality of lenses, and forms an image that is a subject image on the light receiving surface of the image sensor 642. In the optical system 641, the lens is driven based on a control signal supplied from the processing unit 123 and the image is focused on the image sensor 642 after the autofocus operation is resumed.
The image sensor 642 is an image sensor such as a CCD image sensor (Charge Coupled Device Image Sensor), for example, and captures an image formed by the optical system 641. For example, the image sensor 642 converts an object image formed by the optical system 641 into an electric signal and outputs the electric signal to the encoder 129 as image data.

The processing unit 123 comprehensively controls the head mounted display 1 as described above. For example, the processing unit 123 stores the audio data supplied from the sound collection microphone 24, the call microphone 37, or the audio connector 26 via the encoder 129 in the flash memory 122. For example, the processing unit 123 converts the audio data supplied from the sound collection microphone 24, the call microphone 37, or the audio connector 26 via the encoder 129 to the BT communication circuit 130, the WiFi communication circuit 131, or the 3G / LTE communication circuit 138. To other devices.
Further, the processing unit 123 stores, for example, video (image) data supplied from the camera 64, the image sensor 72, or the video connector 27 via the encoder 129 in the flash memory 122. For example, the processing unit 123 displays video (image) data supplied from the camera 64, the image sensor 72, or the video connector 27 via the encoder 129 on the display panel 63 via the LCD driver 125. For example, the processing unit 123 converts video (image) data supplied from the camera 64, the image sensor 72, or the video connector 27 via the encoder 129 into the BT communication circuit 130, the WiFi communication circuit 131, or the 3G / LTE communication circuit 138. Send to other devices via

  When playing back video data, the processing unit 123 receives video (image) data recorded in the flash memory 122 or video (image) data input from the encoder 129 via the decoder 121 and the LCD driver 125. It supplies to the display panels 36 and 63. Thereby, the video based on the video data is displayed on the display panel 36 or the display panel 63 to which the video signal is input. Further, the processing unit 123 causes the video signal output from the decoder 121 to the video connector 27 to be output to an external device via the video connector 27.

Further, when displaying an image, the processing unit 123 turns on the backlight 35 for the display panel 36 and the backlight 62 for the display panel 63 as necessary. That is, the processing unit 123 supplies a control signal to the BL driver 126, and the BL driver 126 lights the backlights (35, 62) individually.
Further, the processing unit 123 detects an operation based on, for example, a control signal, and executes an operation defined in the control program.

In addition, the processing unit 123 controls information (for example, a character and a cursor mark) to be superimposed on the captured image based on the hand shape included in the captured image captured by the camera 64, and also displays information on the captured image. Output the superimposed image. That is, the processing unit 123 controls the process of superimposing information on the captured image based on this hand mode, and outputs an image in which information is superimposed on the captured image.
For example, the processing unit 123 determines the left and right hands (whether the right hand or the left hand) is included in the captured image based on the captured image, and changes the information to be superimposed based on the determined determination result. Specifically, for example, the processing unit 123 displays a right-hand character when it is a right hand, and displays terminal information (system information) of the head mounted display 1 on the display panel 63. For example, the processing unit 123 displays a character for the left hand when it is the left hand, and displays information about the user (personal information) on the display panel 63.

  Further, for example, the processing unit 123 determines the front and back of the hand included in the captured image (whether it is the back of the hand or the palm) based on the captured image, and changes the information to be superimposed based on the determined determination result. To do. Specifically, for example, when the palm 123 is the palm (the back of the hand), the processing unit 123 causes the display panel 63 to display a predetermined character superimposed on the palm image area. For example, when the processing unit 123 is the back of the hand (the front of the hand), the display unit 63 superimposes a cursor (mark) indicating the indicated position on the indicated position (for example, the position of the fingertip) indicated by the hand. Display.

Further, the processing unit 123 determines whether or not the hand included in the captured image is a user's hand based on the captured image, and changes the information to be superimposed based on the determined determination result. Specifically, when the processing unit 123 determines that the hand included in the captured image is the hand of the user (person), the processing unit 123 displays information associated with the user (person) (for example, personal information of the user) on the display panel. 63 is displayed. In addition, when the processing unit 123 determines that the hand included in the captured image is not a user's hand (is another person's hand), the processing unit 123 acquires information (for example, personal information) associated with the owner of the hand, The acquired information is displayed on the display panel 63.
Further, the processing unit 123 controls the laser light irradiation by the laser oscillator 73 based on, for example, a video (image) captured by the camera 64.

In the present embodiment, the processing unit 123 includes a pattern recognition unit 201, an input control unit 202, an AF (autofocus) control unit 203, a menu selection processing unit 204, a display control unit 205, an eye detection unit 206, and a display switching unit. 207 and an angle detection unit 208.
FIG. 8 is a functional block diagram illustrating an example of the processing unit 123 in the present embodiment.
Hereinafter, each unit of the processing unit 123 in the present embodiment will be described with reference to FIGS. 7 and 8.

The pattern recognition unit 201 (recognition unit) recognizes the shape of the human hand included in the image captured by the camera 64 and recognizes the hand form. Here, the aspect of the hand is, for example, the state of the left and right of the hand (right hand or left hand), the front and back of the hand (back of hand or palm). Further, in the present embodiment, the head mounted display 1 is, for example, an operation of the head mounted display 1 (for example, an operation such as indicating a predetermined position on the screen of the display panel 63, selecting a menu, and inputting information). This is done by recognizing the shape.
The pattern recognition unit 201 uses, for example, pattern matching that determines whether or not a predetermined hand shape pattern matches a pattern included in the image, and the human hand and the left and right sides of the hand (right hand or Recognize (left hand). Also, the pattern recognition unit 201 recognizes the front and back of the hand (back of the hand or palm of the hand) based on whether there is a palm print (hand wrinkle) in the pattern area recognized as a human hand, for example. In this way, the pattern recognition unit 201 recognizes the hand mode based on the captured image captured by the camera 64.

Note that a predetermined hand shape pattern may be stored in the memory 127. Here, the hand shape indicates a pattern region of the hand shape in the captured image captured by the camera 64.
When recognizing a human hand, the pattern recognition unit 201 supplies information related to the recognized human hand to the input control unit 202 and the angle detection unit 208. In addition, the information regarding a human hand is information, such as position information on the imaged image, the size of the hand shape, the position of the finger, the length of the finger, and the direction of the finger. Also, when the pattern recognition unit 201 recognizes a human hand, the pattern recognition unit 201 supplies the recognized hand mode to the input control unit 202.

  The AF control unit 203 (focusing unit) drives the optical system 641 of the camera 64 to focus an image such as a subject image on the image sensor 642. The AF control unit 203 performs control to drive the optical system 641 so as to focus on a predetermined pixel position in an image captured by the camera 64 based on a control command supplied from the input control unit 202.

  The menu selection processing unit 204 (operation unit) operates input according to the designated position indicated by the hand calculated by the input control unit 202 described later. For example, the menu selection processing unit 204 selects a plurality of operation menus (for example, icons, etc.) displayed on the display panel 63 according to the pointing position by hand. For example, the menu selection processing unit 204 performs an operation process such as selecting at least one operation menu in accordance with a manual pointing position. That is, the menu selection processing unit 204 operates the operation menu according to the instruction position by hand.

  The display control unit 205 performs control to display a video (image) and various displays on the display panel 63 and the display panel 36. For example, the display control unit 205 performs control to display a mark such as a cursor or a character on the display panel 63 at a position designated by the input control unit 202 in a video (image) captured by the camera 64. The display control unit 205 supplies video (image) data to be displayed on the display panel 63 to the decoder 121 via the display switching unit 207.

  The eye detection unit 206 detects whether or not the head mounted display 1 is attached to the user based on an image captured by the image sensor 72 that captures the user's eye. Further, the eye detection unit 206 detects the left and right positions where the display unit 60 is mounted, for example, by performing pattern matching of the user's eyes based on the image captured by the image sensor 72. That is, the eye detection unit 206 detects whether the display unit 60 (operation switch 30) is attached to the user, for example, whether the display unit 60 is on the right side or the left side of the user's body. The eye detection unit 206 supplies information indicating whether the display unit 60 is attached to either the left or right of the detected user to the display switching unit 207 and the input control unit 202.

  The display switching unit 207 displays the image information supplied from the display control unit 205 according to information on whether the display unit 60 is mounted on either the left or right side of the user detected by the eye detection unit 206. Switch the display so that you can see it. Details of the display switching process will be described later. The display switching unit 207 supplies the image data switched according to the information on whether the display unit 60 is mounted on either the left or right side of the user detected by the eye detection unit 206 to the LCD driver 125 via the decoder 121. By doing so, it is displayed on the display panel 63.

  The angle detection unit 208 detects an angle indicated by the hand direction included in the captured image as the hand angle based on the captured image captured by the camera 64. Specifically, the angle detection unit 208 detects the hand angle based on the information regarding the human hand supplied from the pattern recognition unit 201. The angle detection unit 208 detects the angle of the hand, for example, by detecting the angle of the finger orientation with respect to the horizontal direction of the captured image. Note that the hand angle detected by the angle detection unit 208 is used to determine whether or not the hand is a user's hand to be described later.

  The input control unit 202 (control unit) controls whether to execute a predetermined process according to the size of the hand recognized by the pattern recognition unit 201. Here, the predetermined process is, for example, an autofocus process, which is a process for focusing the optical system 641 on the AF control unit 203, or a process for selecting an operation menu by the menu selection processing unit 204.

  The input control unit 202 causes the above-described predetermined processing to be executed when the size of the hand shape is equal to or larger than a predetermined size. Here, the “predetermined predetermined size” is the size of the hand corresponding to a predetermined distance from the camera 64 in the image captured by the camera 64. In general, when the head mounted display 1 is mounted on the head, the user's hand is not farther than 1 m even when the user's hand is farthest. Therefore, the “predetermined predetermined size” of the hand shape is a size corresponding to the hand within 1 m, for example. For example, the “predetermined predetermined size” indicates that the number of pixels in the horizontal direction of the pattern recognized as the shape of the hand by the pattern recognition unit 201 is the number of pixels of the camera 64, the angle of view of the lens, the hand and the head. For example, if the width of the hand is 200 pixels or more, the value calculated from the distance is the user's “hand”.

Also, the input control unit 202 controls whether or not to execute the above-described predetermined processing according to the size of the largest hand shape when the pattern recognition unit 201 recognizes the shape of a plurality of hands. To do. This is because, for example, when there are a plurality of “hands” within 1 m, the input control unit 202 determines the nearest hand as the user's hand.
For example, when an image captured by the camera 64 is displayed on the display panel 63 that displays an image, the input control unit 202 sets the optical system 641 to the AF-shaped control unit 203 against the background of the hand shape. Control to focus. That is, in this case, since there is a high possibility that the user is pointing the background portion by hand, the input control unit 202 performs control to focus the optical system 641 on the hand-shaped background. Here, the background indicates an area excluding the hand shape in the image displayed on the display panel 63.

Further, for example, when the image picked up by the camera 64 is not displayed on the display panel 63 that displays an image, the input control unit 202 focuses the optical system 641 on the hand shape on the AF control unit 203. Let That is, in this case, since the user is likely to perform selection processing by the menu selection processing unit 204, it is necessary to recognize the shape of the hand in detail. Control for focusing the optical system 641 is performed. Here, focusing the optical system 641 on the shape of the hand means focusing (focusing) on the pixel position (pixel area) of the hand shape in the image captured by the camera 64. The optical system 641 is driven.
Further, the input control unit 202 determines whether or not the hand is holding an object based on the shape of the hand recognized by the pattern recognition unit 201, for example. When the input control unit 202 determines that the hand is in a state of holding an object, the input control unit 202 causes the AF control unit 203 to focus the optical system 641 on the shape of the hand. In this case, since there is a high possibility that the user is pointing to an object held by the hand, the input control unit 202 causes the AF control unit 203 to focus the optical system 641 on the shape of the hand.

  In addition, the input control unit 202 operates, for example, when the pattern recognition unit 201 recognizes the shape of a plurality of hands and when the shape of the left and right hands is recognized, the operation switch detected by the eye detection unit 206 30 selects one of the left and right hand shapes according to the left or right position on the user. The input control unit 202 controls whether or not to execute the above-described predetermined processing according to the selected size of the left or right hand shape.

  For example, when the head mounted display 1 is used for the right eye as shown in FIG. 3A, the touch switch 34 and the operation switch 30 are also arranged on the right side, so that the user is right-handed. Is likely. In this case, for example, the user's left hand is used for supporting the body, and the right hand is likely to be used for detailed operations and instructions. Therefore, the input control unit 202 gives priority to the right hand. select.

  Further, the input control unit 202 determines an instruction position indicated by the user's hand based on the position of the hand recognized by the pattern recognition unit 201 and the direction and length of a predetermined portion (for example, the index finger) of the hand. calculate. The “hand position” here is a position on the captured image captured by the camera 64. For example, the input control unit 202 calculates a position extended from the tip of the hand (for example, the tip of the index finger) in the direction of the index finger according to the length of the index finger as the indication position. Specifically, for example, the input control unit 202 uses the position extended by multiplying the length of the index finger from the tip of the hand in the direction of the index finger by a predetermined factor (for example, 2 times) as the indication position. calculate. Details of the processing for calculating the designated position will be described later.

The input control unit 202 displays a cursor (mark) indicating the designated position at a position corresponding to the calculated designated position in the display image of the display panel 63 that displays the image. For example, the input control unit 202 supplies the calculated designated position to the display control unit 205 to display a cursor (mark) at a position corresponding to the designated position on the display panel 63.
Further, the input control unit 202 performs control for causing the menu selection processing unit 204 to operate an input in accordance with the calculated designated position. For example, the input control unit 202 controls the menu selection processing unit 204 to select one of a plurality of operation menus according to the designated position.

  The input control unit 202 determines whether the hand included in the captured image is the right hand or the left hand based on the hand aspect supplied to the pattern recognition unit 201, and performs imaging based on the determined determination result. Control to change the information to be superimposed on the image is performed. For example, the input control unit 202 causes the display control unit 205 to display the character for the right hand and the terminal information (system information) of the head mounted display 1 on the display panel 63 when it is the right hand. The input control unit 202 also causes the display control unit 205 to display information about the left hand character and the user (personal information) on the display panel 63 when the user has the left hand.

  Further, the input control unit 202 determines whether the hand included in the captured image is the palm or the back of the hand, and performs control to change the information to be superimposed based on the determined determination result. For example, when the input control unit 202 is a palm, for example, the input control unit 202 causes the display control unit 205 to display a predetermined character on the display panel 63 by superimposing a predetermined character on the palm image area. For example, when the input control unit 202 is the back of the hand, the display control unit 205 displays the cursor by superimposing a cursor (mark) indicating the designated position on the designated position (for example, the position of the fingertip) designated by the hand. It is displayed on the panel 63.

  Further, the input control unit 202 determines whether or not the hand included in the captured image is a user's hand, and performs control to change the information to be superimposed based on the determined determination result. For example, when the input control unit 202 determines that the hand included in the captured image is the hand of the user (person), the input control unit 202 informs the display control unit 205 of information (for example, the user) Personal information) is displayed on the display panel 63. Further, for example, when the hand included in the captured image is another person's hand, the input control unit 202 transmits information (for example, personal information) associated with the owner of the hand via, for example, the WiFi communication circuit 131. Get. Then, the input control unit 202 causes the display control unit 205 to display the acquired information on the display panel 63.

FIG. 9 is a diagram illustrating an example of a network connection of the head mounted display 1.
In this figure, the head mounted display is denoted as HMD. The HMD 1 is a head mounted display used by a user (person), and the HMD 500 is a head mounted display used by another person. In this figure, only elements necessary for connection are shown, and the others are omitted.

  In FIG. 9, the head mounted display 1 (HMD1) is connected to the HMD 500 and the server 210 via the network 220. The HMD 1 includes a touch switch 34, a display unit 63, a camera 64, a geomagnetic sensor 133, a processing unit 123, a flash memory 122, and communication interfaces (131, 138). Here, the communication interface is, for example, the WiFi communication circuit 131 and the 3G / LTE communication circuit 138.

  The HMD 500 is a head mounted display having the same communication means as the HMD 1. Similar to the HMD 1, the HMD 500 includes a touch switch 534, a display panel 563 (display unit), a camera 564, a geomagnetic sensor 533, a processing unit 523, a flash memory 522, and communication interfaces (531, 538). Each function part of HMD500 is equivalent to each function part of HMD1.

The server 210 includes a processing unit 211, a storage unit 212, and a communication interface 213. The processing unit 211 receives image data (moving image, still image, line drawing, etc.) received from each terminal (HMD1, HMD500) via the communication interface 213, audio data, environmental information of each terminal, and requests from each terminal. The communication record is recorded (stored) in the storage unit 212. The processing unit 211 reads the image data recorded in the storage unit 212 in response to a request from each terminal, and transmits the read image data to each terminal via the communication interface 213.
The storage unit 212 records (stores) communication records such as image data (moving images and still images) received from each terminal, audio data, environment information of each terminal, and requests from each terminal.
The communication interface 213 includes, for example, a WiFi communication circuit, a 3G communication circuit, an LTE communication circuit, a wired LAN circuit, and the like.

  The HMD 1 connects to the network 220 via the communication interface (131, 138), and the HMD 1 transmits information associated with the owner of the hand from the HMD 500 or the server 210 via the communication interface (131, 138). get.

Next, the operation of the head mounted display 1 in this embodiment will be described.
First, the imaging range of the camera 64, which is a premise for describing the operation of the head mounted display 1 in the present embodiment, will be described with reference to FIGS.
FIG. 10 is a schematic diagram showing an imaging range when the right eye is mounted on the head mounted display 1 in the present embodiment.
In this figure, the display unit 60 is disposed below the direction of the angle α with respect to the line-of-sight direction when facing the front of the right eye. In this case, the camera 64 images the range of the imaging range P. The imaging range P includes the upper half range Q and the lower half range R of the imaging range P when the right eye is worn.

FIG. 11 is a schematic diagram showing an imaging range when the left eye of the head mounted display 1 in this embodiment is mounted.
In this figure, the display unit 60 is disposed below the direction of the angle α with respect to the line-of-sight direction when facing the front of the left eye. In this case, the camera 64 images the range of the imaging range P. The imaging range P includes an upper half range R and a lower half range Q when the right eye is mounted in the imaging range P.

Next, with reference to FIG. 12 and FIG. 13, switching processing of the imaging range and display range in the head mounted display 1 will be described.
FIG. 12 is a diagram illustrating switching of the imaging range and the display range of the head mounted display 1 in the present embodiment.
In FIG. 12, the imaging range P indicates the entire imaging range of the camera 64. Here, a camera 64 (imaging element 642) having an aspect ratio of 4 to 3 is used to arrange the camera so that the imaging range is vertically long.
FIG. 13 is a flowchart showing the switching process of the imaging range and the display range of the head mounted display 1 in this embodiment.

  In FIG. 13, first, the processing unit 123 of the head mounted display 1 selects the range Q as a video (image) reading range (display range) on the assumption that the user is first viewing the display panel 63 with the right eye. (Step S1). That is, as shown in FIG. 10, the processing unit 123 reads the upper range Q (upper half) as viewed from the user in the imaging range P, and displays it on the display panel 63 by the display control unit 205 and the display switching unit 207. Display.

Next, the processing unit 123 determines whether or not the user is observing with the right eye (step S2). Here, the processing unit 123 detects information regarding the eyes being observed by the user (information indicating whether the display unit 60 is mounted on either the left or right side of the user). That is, the eye detection unit 206 of the processing unit 123 detects information related to the eye being observed by the user based on an image obtained by photographing the eye including the user's eyelid using the image sensor 72.
When it is determined that the user is observing with the right eye (step S2: Yes), the processing unit 123 proceeds with the process to step S6. Further, when the processing unit 123 determines that the user is not observing with the right eye (observing with the left eye) (step S2: No), the processing proceeds to step S3. In this case, the user is wearing the head mounted display 1 as shown in FIG.

  Next, in step S <b> 3, the processing unit 123 selects the range R as a video (image) readout range (display range). The range R is a frame indicated by a dotted line on the lower side in FIG. 12, but when the head mounted display 1 is used with the left eye as shown in FIG. 11, the upper side as viewed from the user in the imaging range P Will be read out. That is, as shown in FIG. 11, the processing unit 123 reads the upper range (upper half) range R as viewed from the user in the imaging range P, and displays the range R via the display control unit 205 and the display switching unit 207. Output to panel 63.

  Next, the processing unit 123 inverts the reading range in the imaging range P upside down (step S4). That is, the processing unit 123 rotates the video (image) of the camera in the reading range by 180 degrees because the camera 64 is rotated by 180 degrees.

  Next, the processing unit 123 rotates the display image by 180 degrees (step S5). That is, the display switching unit 207 of the processing unit 123 determines whether or not to rotate the image supplied from the display control unit 205 by 180 degrees according to the information about the eyes observed by the user detected by the eye detection unit 206. Switch.

Next, the processing unit 123 changes the image reading range in accordance with the distance measurement value and the zoom value to the subject (step S6). That is, the processing unit 123 changes the image reading range (display range) in the imaging range P according to the zoom magnification of the subject and the camera. For example, when the display unit 60 is observed with the right eye (when the head mounted display 1 is used with the right eye), the processing unit 123 causes the display panel 63 to display the readout range Q shown in FIG. Yes. In this case, when the user gives a zooming instruction to enlarge a part of the subject using the operation switch 30 or the like, the processing unit 123 first measures the distance to the subject using a known distance measuring technique using the camera 64. Distance. When the distance to the subject is sufficiently long, the processing unit 123 reads the range S (FIG. 12) in the center of the screen as the reading range because the human eye looking at the distance also looks at the front of the face. It is displayed on the panel 63. Further, when the distance to the subject is extremely close, the processing unit 123 reads the range T (FIG. 12) and displays it on the display panel 63 because the human eye is close to the center. The processing unit 123 changes the reading range T to the range S substantially continuously according to the distance to the subject.
The processing unit 123 does not move the reading range particularly when it is desired to display a wide range, that is, when zooming is on the wide side.

For example, when the display unit 60 is observed with the left eye (when the head mounted display 1 is used with the left eye), the processing unit 123 displays the readout range R illustrated in FIG. 12 on the display panel 63. I am letting. Also in this case, as described above, the processing unit 123 reads the range U (FIG. 12) when the subject is zoomed and the distance to the subject is sufficiently long, and when the distance to the subject is very close. Reads out the range V (FIG. 12) and outputs it to the display panel 63.
As described above, the head mounted display 1 according to the present embodiment reduces the difference between the range that the user is viewing and the range that is captured by the camera 64, and is easy to use (good for convenience) with no sense of incongruity during use. Can be provided.

  Next, the processing unit 123 sets the left and right sounds of the speakers (the first speaker 103 and the second speaker 104) (step S7). That is, the processing unit 123 determines the speaker (the first speaker 103, the second speaker) according to the information related to the eyes observed by the user (information indicating whether the display unit 60 is mounted on either the left or right side of the user). The left and right sounds of the speaker 104) are set.

  Next, the processing unit 123 sets the left and right sounds of the stereo microphone (sound collection microphone 105, sound collection microphone 24) (step S8). That is, the processing unit 123 determines the stereo microphone (the sound collecting microphone 105, the sound collecting microphone 105) according to the information related to the eyes observed by the user (information indicating whether the display unit 60 is mounted on either the left or right side of the user). The left and right sounds of the sound microphone 24) are set.

  Next, the processing unit 123 displays the switched image data on the display panel 63 (step S9). The display switching unit 207 displays the switched image data on the display panel 63 by supplying the switched image data to the LCD driver 125 via the decoder 121.

As described with reference to FIGS. 10 to 13, the head mounted display 1 in the present embodiment does not require the display main body 20 and the display unit 60 to be disposed in front of the user's eyes, and is disposed below the front. Therefore, it is possible to place the call microphone 37 relatively near the mouth. For this reason, the head mounted display 1 in this embodiment can record a sound clearly.
Further, the head mounted display 1 according to the present embodiment reads out an appropriate range and displays it on the display panel 63 in both cases of observing the display unit 60 with the right eye and observing the display unit 60 with the left eye. be able to. Therefore, the head mounted display 1 in this embodiment can provide a highly convenient system. Furthermore, the head-mounted display 1 according to the present embodiment provides an easy-to-use system without parallax because the direction observed by the user matches the center of the shooting range of the camera 64 even when the distance to the subject is short. can do.

Next, the operation of the input / output device 200 provided in the head mounted display 1 will be described.
14A, 14B, and 14C are flowcharts showing processing of the input / output device 200 according to the present embodiment.
In this figure, first, as shown in FIG. 15 (B), the camera 64 captures an image G2 in a horizontally long range, and the entire imaging range is read and displayed on the display panel 63 of the display unit 60. A description will be given of the case.

  First, the input / output device 200 determines whether an image from the camera 64 is displayed (step S101). That is, the input control unit 202 of the processing unit 123 determines whether an image captured by the camera 64 is displayed on the display panel 63 of the display unit 60. When the image captured by the camera 64 is displayed on the display panel 63 of the display unit 60 (step S101: Yes), the input control unit 202 advances the process to step S102. Moreover, the input control part 202 advances a process to step S117 (FIG. 14B), when the image imaged with the camera 64 is not displayed on the display panel 63 of the display part 60 (step S101: No). When an image captured by the camera 64 is not displayed on the display panel 63 of the display unit 60, the display panel 63 includes menu icons I1 to I9 as shown in FIG. An image G1 is displayed. Here, the image G <b> 1 is an image that does not depend on the image captured by the camera 64.

  As shown in FIG. 14B, in step S117, the input / output device 200 determines whether or not there is a hand of a predetermined size or larger in an image captured by the camera 64. That is, the input / output device 200 determines whether or not the user's “hand” can be recognized by the camera 64 based on the size of the hand. In this case, first, for example, the pattern recognition unit 201 of the processing unit 123 recognizes a human hand included in the image captured by the camera 64 and supplies information related to the recognized hand to the input control unit 202. Here, it is not necessary to focus on the hand of the person, and it is sufficient that the pattern recognition unit 201 can roughly recognize the hand. The input control unit 202 determines whether or not the user's “hand” can be recognized based on the information regarding the hand supplied from the pattern recognition unit 201. For example, the input control unit 202 determines that the number of pixels in the horizontal direction of the pattern recognized as a hand by the pattern recognition unit 201 is calculated from the number of pixels of the camera 64, the angle of view of the lens, and the distance between the hand and the head. For example, if the width of the hand is 200 pixels or more (for example, a hand within 1 m), the user's “hand” is determined.

  When the size of the hand shape is equal to or larger than a predetermined size (step S117: Yes), the input control unit 202 advances the process to step S118. Further, the input control unit 202 ends the process when the size of the hand shape is less than a predetermined size (step S117: No).

Next, in step S <b> 118, the input / output device 200 focuses on the nearest “hand”. That is, the input control unit 202 focuses the optical system 641 on the hand shape closest to the AF control unit 203. This is because the input control unit 202 determines the closest hand as the user's “hand” when there are a plurality of “hands” within 1 m.
Further, the input control unit 202 gives priority to either hand when both the right hand and the left hand of the user are present. Here, when the head mounted display 1 is used for the right eye as shown in FIG. 3A, the touch switch 34, the operation switch 30 and the like are also arranged on the right side, and the user may be right-handed. High nature. In this case, the left hand of the user is used to support the body, and the right hand is likely to be used for detailed operations and instructions. Therefore, the input control unit 202 prioritizes the right hand. Align (focus the optical system 641 on the AF control unit 203).

Next, the input / output device 200 recognizes the shape of the hand by a known pattern recognition process (step S119). That is, the pattern recognition unit 201 recognizes the shape of the hand by a known pattern recognition process, and supplies information related to the recognized hand shape to the input control unit 202.
Next, the input / output device 200 detects whether an operation unit such as the operation switch 30 is operated (step S120). That is, the input control unit 202 detects whether an operation unit such as the operation switch 30 is operated. Since the user's hand should be placed on the operation switch 30 or the like when the operation switch 30 or the like is being operated (step S120: Yes), the input control unit 202 may show the other person's hand Therefore, priority is given to the operation of the operation switch 30 or the like, and the process proceeds to step S121. Moreover, the input control part 202 advances a process to step S122, when the operation switch 30 grade | etc., Is not operated (step S120: No).

In step S121, the input / output device 200 performs a predetermined display on the display panel 63 when the operation switch 30 or the like is operated, and performs processing according to the operation when the operation switch 30 or the like is operated.
On the other hand, in step S122, the input / output device 200 displays a cursor (cursor M1 in FIG. 15A) in a superimposed manner on the screen corresponding to the position of the index finger. That is, for example, as illustrated in FIG. 15, the input control unit 202 causes the display control unit 205 to display the cursor M1 at a position corresponding to the extending direction of the index finger that is twice the length of the index finger. Here, as shown in FIG. 15A, it can be seen that the envelope icon I8 is selected.

Here, the user's hand H1 is photographed on the camera 64 as shown in FIG. 15B, and the input control unit 202 is on the extension line of the user's index finger as shown by the broken line by pattern recognition. Then, a position P1 (position of distance L2 from the fingertip) obtained by extending the index finger length L1 that can be measured on the screen by about twice is calculated. The input control unit 202 causes the display control unit 205 to display a cursor M1 at a position on the display image of the display panel 63 corresponding to the position P1, as shown in FIG. That is, the input control unit 202 indicates the indicated position (position) indicated by the user's hand based on the position of the hand recognized by the pattern recognition unit 201 and the direction and length of a predetermined part of the hand (for example, the index finger). P1) is calculated. The input control unit 202 displays a cursor M1 (mark) indicating the designated position (position P1) at the position on the display image of the display panel 63 corresponding to the calculated designated position.
The input control unit 202 calculates the position P1 based on the following equations (1) and (2).

X3 = X1 + 2 × (X1-X2) (1)
Y3 = Y1 + 2 × (Y1-Y2) (2)

Here, the variable X1 and the variable Y1 correspond to the coordinates (X1, Y1) of the tip of the index finger in FIG. 15B, and the variables X2 and Y2 correspond to the coordinates (X2, Y2) of the base of the index finger. . Variable X3 and variable Y3 correspond to the coordinates (X3, Y3) of cursor M1.
For example, when the index finger is pointed substantially parallel to the optical axis of the camera 64, the index finger is shown short on the camera 64, so the cursor M1 is near the tip of the index finger. Further, for example, when the index finger is placed so as to be orthogonal to the optical axis of the camera 64, the cursor M1 is displayed at the most distant place. As described above, the position, tilt, and direction of the index finger can be freely changed by the user to point (point) the position on the screen.
Here, since the menu screen (image G1) as shown in FIG. 15A is displayed on the display panel 63, the input / output device 200 displays the image of the camera 64 shown in FIG. 15B. It is not displayed on the panel 63.

  Next, the input / output device 200 determines whether or not the finger has performed a predetermined action (step S123). Here, the predetermined operation is, for example, an operation in which a circle is drawn with a finger or the cursor M1 is stationary on an icon (for example, icon I8) for a predetermined time. That is, the menu selection processing unit 204 determines whether or not the finger has performed a predetermined action. If it is determined that the finger has performed a predetermined action (step S123: Yes), the menu selection processing unit 204 advances the process to step S124. Moreover, the menu selection process part 204 complete | finishes a process, when it determines with the finger | toe not performing predetermined | prescribed operation | movement (step S123: No).

In step S124, the input / output device 200 performs a predetermined process corresponding to the case where the icon is selected. That is, the menu selection processing unit 204 selects, for example, an envelope icon I8, and the processing unit 123 executes processing corresponding to the selected envelope icon I8.
Note that the menu selection processing unit 204 can also perform operations such as menu selection by detecting that the thumb and index finger are formed in a specific pattern or changing its shape. For example, even when the menu selection processing unit 204 is forced to select two choices such as “YES / NO” on the menu screen or the like, the user makes a choice between the thumb and the index finger to select one option. Setting (selection) may be performed by detecting an OK sign.

  In this way, when the hand is in focus, the camera 64 can read fine hand movements, so the input / output device 200 can read the finger pattern in detail. A fine icon as shown in FIG. 15A can be selected. Therefore, when the input control unit 202 focuses the optical system 641 on the hand shape, the input control unit 202 optically selects the number of operation menus that can be selected by the menu selection processing unit 204 with respect to the hand-shaped background described later. More display is made on the display panel 63 than when the system 641 is focused.

Next, a case where an image from the camera 64 is displayed will be described.
Again, as shown in FIG. 14A, in step S102, the input / output device 200 determines whether or not the image captured by the camera 64 has a hand of a predetermined size or larger in the same manner as in step S117. judge. That is, the input control unit 202 determines whether or not the size of the hand shape is greater than or equal to a predetermined size based on the information about the hand supplied from the pattern recognition unit 201. When the size of the hand shape is equal to or larger than a predetermined size (step S102: Yes), the input control unit 202 advances the process to step S104. Moreover, the input control part 202 advances a process to step S103, when the magnitude | size of the shape of a hand is less than predetermined predetermined magnitude | size (step S102: No).

  Next, in step S <b> 103, the input / output device 200 adjusts the focus and exposure on the closest subject. That is, the input control unit 202 performs control to focus the optical system 641 on the subject close to the AF control unit 203, and the processing unit 123 performs control to adjust the exposure of the close subject to the camera 64. . It should be noted that a subject that is extremely close to the subject (for example, a distance that a finger cannot enter, such as 5 cm) that cannot be focused due to the restriction of the optical system 641 may be dust on the lens. The unit 202 causes the AF control unit 203 to focus on the closest subject that can be focused next. Further, the input control unit 202 may perform control for focusing on a person when a person is detected by a so-called face recognition technique. The AF control unit 203 uses a known means for focusing (focusing). The input control unit 202 performs control to focus on the closest subject, and then proceeds to step S115.

  On the other hand, in step S104, the input / output device 200 determines whether or not an object is held in the hand. That is, the input control unit 202 determines whether or not the hand has an object based on the shape of the hand recognized by the pattern recognition unit 201, for example. When the input control unit 202 determines that the hand has an object (step S104: Yes), the process proceeds to step S105, and when it is determined that the hand does not have an object (step S104: No). Then, the process proceeds to step S106.

  Next, in step S105, the input / output device 200 focuses on a subject near the hand. In this case, if the user has an object in his hand and it is reflected in the camera 64, it is estimated that the object he has in his hand is what he wants to convey to other communication terminals (other devices). it can. Therefore, when the input control unit 202 determines that the hand is in a state of holding an object, the input control unit 202 causes the AF control unit 203 to focus the optical system 641 on a subject in the shape of the hand or in the vicinity of the hand. The input control unit 202 performs control to focus on the shape of the hand or a subject in the vicinity of the hand, and then proceeds to step S109.

On the other hand, in step S106, the input / output device 200 detects whether the recognized hand is a palm. That is, the input control unit 202 determines whether or not the recognized hand is a palm based on the hand mode recognized by the pattern recognition unit 201, for example. When the input control unit 202 determines that the recognized hand is the palm (step S106: Yes), the process proceeds to step S201 (FIG. 14C) and determines that the recognized hand is not the palm (the back of the hand). If so (step S106: No), the process proceeds to step S107. As shown in FIG. 16, the pattern recognition unit 201 recognizes a pattern when there is no wrinkle on the back of the hand on the basis of the wrinkle of the palm. In this case, the pattern recognition unit 201 may recognize the front and back of the hand (whether it is the palm of the hand or the back of the hand) in consideration of the degree of bending of the finger.
FIG. 16 shows an example in which the hand H7 displayed in the image G7 is the back of the hand.

  On the other hand, since it is determined to be the back of the hand, that is, when the user points the object with the finger of the hand, in step S107, the input / output device 200 focuses on the background while avoiding the user's hand. In this case, since there is a high possibility that the user indicates the background portion by hand, the input control unit 202 performs control to focus the optical system 641 on the hand-shaped background. For example, as shown in FIG. 17, in the case where one of several products is pointed out, an image in focus on the background rather than telling the image in focus on the hand to the other person. Because it is important to communicate. As a result, the input / output device 200 can communicate in a manner that is easy for a partner to transmit information (such as an image).

  Note that when transmitting an image to another communication terminal, for example, when transmitting at a low resolution such as VGA (Video Graphics Array: 640 × 480 pixels), and the depth of field and resolution of the camera 64 If it is determined that the background is not blurred even if the hand is in focus, the input / output device 200 may skip the process of step S107. Further, the input / output device 200 may execute the process of step S107 only when the resolution when transmitting an image to another communication terminal is high (for example, 800 × 600 pixels, high vision (1080i), or the like). .

Next, assuming that there are a plurality of hands, the input / output device 200 recognizes the largest hand pattern (hand shape) (step S108). That is, the input control unit 202 recognizes the largest hand shape based on the information about the hand supplied from the pattern recognition unit 201. Here, the reason for recognizing the largest hand is that when a plurality of hands are present in the screen, the largest hand is the closest and it is highly likely that it is the user's hand.
In step S118, the closest hand is recognized. However, since the hand is not focused in step S108, it is not possible to determine which hand is the closest even if there are a plurality of hands. Therefore, the input control unit 202 determines (recognizes) the largest hand as the user's hand. Further, the processing when the right hand and the left hand are detected is the same as that in step S118.

  Next, the input / output device 200 emphasizes the contour of the largest hand (step S109). That is, the input control unit 202 emphasizes the recognized contour of the hand pattern and causes the display control unit 205 to display it. This is a process for clearly displaying a hand that is out of focus as much as possible. Further, the input control unit 202 may emphasize the outline of the hand in red or the like when the hand is difficult to see due to the background.

  Next, as in step S120, the input / output device 200 detects whether an operation unit such as the operation switch 30 is operated (step S110). When the operation switch 30 or the like is operated (step S110: Yes), the input control unit 202 gives priority to the operation of the operation switch 30 and advances the process to step S111. Moreover, the input control part 202 advances a process to step S112, when the operation switch 30 grade | etc., Is not operated (step S110: No).

  In step S111, as in step S121, the input / output device 200 performs a predetermined display on the display panel 63 when the operation switch 30 or the like is operated, and follows the operation when the operation switch 30 or the like is operated. Process.

On the other hand, in step S112, the input / output device 200 detects the index finger, and displays a cursor (cursor M2 in FIG. 17) on the display panel 63 at the tip of the detected index finger. Here, the input control unit 202 detects a predetermined finger (for example, an index finger) determined in advance based on the information regarding the hand supplied from the pattern recognition unit 201. For example, as shown in FIG. 17, the input control unit 202 displays the cursor M2 at a position extended from the detected tip of the index finger by multiplying the length of the finger by a predetermined coefficient (for example, two times). 205 is displayed.
In FIG. 17, an image G <b> 3 indicates an image captured by the camera 64 (image sensor 642), and a hand H <b> 2 indicates a hand captured by the camera 64.

  Next, the input / output device 200 adjusts the focus and exposure to the background indicated by the cursor (step S113). For example, the input control unit 202 controls the AF control unit 203 to focus the optical system 641 on the vicinity of the background indicated by the cursor M <b> 2 in FIG. 17, and the processing unit 123 controls the camera 64. Control is performed to adjust the exposure in the vicinity of the background indicated by the cursor M2. In other words, the input control unit 202 focuses the optical system 641 on the background at a position extended from the detected tip of the index finger by multiplying the length of the finger by a predetermined coefficient (for example, two times). This makes it possible to clearly capture the background rather than the front finger, so that the head mounted display 1 can provide a clear image (clear image) to other communication terminals that are performing communication. Can be displayed. Therefore, the input / output device 200 can reduce the possibility of communication errors.

Next, the input / output device 200 displays a cursor / trajectory on the display panel 63 according to the operation (step S114). That is, the input control unit 202 causes the display control unit 205 to continuously display the cursor or display the locus of the cursor according to the movement of a predetermined finger (for example, the index finger).
In this case, since the finger is not strictly focused, it is difficult to control the head mounted display 1 based on a fine finger movement. Therefore, the input / output device 200 performs input control limited to menus that can be controlled by finger image recognition when the finger is not focused. Here, menus that can be controlled by finger image recognition are, for example, limited items such as drawing a line drawing in addition to displaying a cursor.

Next, the head mounted display 1 (input / output device 200) sequentially transmits images to other communication terminals that perform communication (step S115), and also displays the result of the above input control on the display panel 63. Displayed (step S116). The input / output device 200 ends the process after the process of step S116.
In the present embodiment, the input / output device 200 repeatedly executes this input control process (the process from step S101 to step S124).

Next, an example in which the head mounted display 1 uses the images of the camera 64 divided into upper and lower parts will be described.
In the flowchart of FIG. 12, an example in which the upper half is used as the reading area of the camera 64 has been described. However, FIG. 18 shows a state where the display is observed with the right eye as shown in FIG. . In FIG. 18, the head mounted display 1 and another communication terminal of the communication partner display an image in the range Q of the imaging range P of the camera 64 on the display panel 63.

  For example, when the contrast between the hand and the background is low, such as in the evening, the head mounted display 1 displays the cursor M3 on the fingertip of the hand H3 as shown in FIG. May be. That is, the input control unit 202 calculates the position of the tip of the hand H3 as the designated position when the hand H3 is included in the image of the display area (for example, the range Q). Then, the input control unit 202 controls the display control unit 205 to display a cursor (for example, the cursor M3 in FIG. 18) at a position in the display image of the display panel 63 corresponding to the calculated designated position.

Next, a process when it is determined in step S106 that the recognized hand is a palm will be described.
As shown in FIG. 14C, in step S201, the input / output device 200 focuses on the palm of the hand. That is, the input control unit 202 causes the AF control unit 203 to focus the optical system 641 on the palm.

  Next, the input / output device 200 determines whether or not the hand shown in the captured image is the hand of the user (user) (step S202). Here, the head-mounted display 1 is mounted on the user's head, and when considering ergonomics, for example, the user's hand is imaged from the direction as shown in FIG. There is nothing. Therefore, the angle detection unit 208 detects the angle between the vector indicating the direction of the fingertip of the user's index finger and the horizontal direction in the image as the hand angle, and supplies the detected hand angle to the input control unit 202. Based on the hand angle supplied from the angle detection unit 208, the input control unit 202 determines whether or not the hand included in the captured image is the user's hand. The input control unit 202 determines that the hand is the user's hand when the angle of the hand detected by the angle detection unit 208 is between 0 degrees and 180 degrees, for example.

  If the input control unit 202 determines that the hand is the user's hand (step S202: Yes), the process proceeds to step S203. If the input control unit 202 determines that it is not the user's hand (the hand of another person) (step S202: No), the process proceeds to step S210.

  Next, in step S203, the input / output device 200 determines whether or not the user's hand is the right hand. The input control unit 202 determines whether or not the user's hand is the right hand based on the hand mode recognized by the pattern recognition unit 201. In this case, the input control unit 202 determines that the user's hand is the right hand when the angle of the hand detected by the angle detection unit 208 is between 90 degrees and 180 degrees, for example. Good. This is because it is difficult for a human right hand to place the right hand in an angle range other than 90 degrees to 180 degrees when viewed from his / her direction.

For example, in FIG. 19A, since the angle θ1 with respect to the horizontal direction of the finger vector F1 of the hand H4 included in the image G4 is an angle range between 90 degrees and 180 degrees, the input control unit 202 judge. For example, in FIG. 19B, since the angle θ2 with respect to the horizontal direction of the finger vector F2 of the hand H5 included in the image G5 is an angle range between 0 degrees and 90 degrees, the input control unit 202 Judge with left hand.
If the input control unit 202 determines that the user's hand is the right hand (step S203: Yes), the process proceeds to step S204. If the input control unit 202 determines that the user's hand is the left hand (step S203: No), the process proceeds to step S218.

  Next, in step S204, the input / output device 200 displays the character C1. The character C1 (first character) is formed of a so-called 3D polygon, and is stored in advance in the memory 127 or the flash memory 122, for example. The input control unit 202 reads the image information of the character C1 from the memory 127 or the flash memory 122, and displays it on the display panel 63 so as to be superimposed on the palm image area on the display control unit 205, as shown in FIG. Let In addition, when the user turns (rotates) the right hand, the input / output device 200 controls the character C1 to change its direction substantially in unison with the movement of the right hand. Note that the character C1 itself may be able to independently operate or speak according to a program created in advance.

  Next, the input / output device 200 displays the device information (property) of the head mounted display 1 (step S205). That is, the input control unit 202 causes the display control unit 205 to display system information D1 (device information) on the display panel 63, as shown in FIG. The system information D1 is, for example, information such as remaining battery capacity, memory usage, radio wave intensity in WiFi communication, and system version information. The character C1 is a character when representing device information of the head mounted display 1. For example, the input / output device 200 displays the system information D1 at a position to the right of the character C1. Here, the input / output device 200 may output the system information D1 by sound so that the character C1 is speaking (speaking).

  Next, the input / output device 200 determines whether or not the left hand is added (step S206). That is, the input control unit 202 further determines whether or not a hand is added to the image (in the video) and the hand is a “left hand” opposite to the right hand that has been detected first. If the input control unit 202 determines that the left hand has been added (step S206: Yes), the process proceeds to step S207. If the input control unit 202 determines that the left hand is not added (step S206: No), the process proceeds to step S209.

  Next, in step S207, the input / output device 200 determines whether or not the newly detected left hand is the same size as the user's hand. That is, the input control unit 202 determines whether or not the detected left hand is the user's left hand based on the size of the hand. If the input control unit 202 determines that the detected left hand is the user's left hand (step S207: Yes), the process proceeds to step S208. If the input control unit 202 determines that the detected left hand is not the user's left hand (is the other person's left hand) (step S207: No), the process proceeds to step S209.

  Next, in step S208, the input / output device 200 adjusts the orientation of the character C1 in accordance with the rotation of the user's left hand. Specifically, the input control unit 202 performs control to change the orientation with the character C1 when the user turns the left hand. For example, in FIG. 19A, the character C1 is facing forward, but when the user turns the left hand to the left, the input / output device 200 allows the character C1 only with the angle of the user's right hand unchanged. Can be controlled to turn (rotate) counterclockwise, for example, in the opposite direction by 180 degrees.

Next, in step S209, the input / output device 200 determines whether there is another person's right hand or left hand. That is, the input control unit 202 determines whether another person's hand is included in the captured image. In this case, as shown in FIG. 20, the input control unit 202 determines that the angle θ3 of the vector F3 indicating the direction of the finger of the hand H6 with respect to the horizontal direction in the image G6 is in the range of about 180 degrees to 360 degrees and If the “hand” is larger than a predetermined size as described above, it is determined that the “hand” is the “other hand”. For the hands other than the user, the angle to be presented varies depending on the positional relationship of the other party, and therefore it is difficult to determine whether the hand is the right hand or the left hand, so the input control unit 202 is controlled by the pattern recognition unit 201. Whether the right hand or the left hand is determined based on the recognized hand mode.
If the input control unit 202 determines that the other person's hand is included in the captured image (step S209: Yes), the process proceeds to step S210. If the input control unit 202 determines that the other person's hand is not included in the captured image (step S210) The processing is terminated at S209: No).

  Next, in step S210, the input / output device 200 has determined that there is another person in the vicinity, and therefore checks whether there is a WiFi signal to attempt communication. For example, the processing unit 123 uses the WiFi communication circuit 131 to determine whether or not there is a WiFi signal (WiFi communication radio wave). When there is a WiFi signal (WiFi communication radio wave) (step S210: Yes), the processing unit 123 proceeds with the process to step S211. When there is no WiFi signal (step S210: No), the processing unit 123 ends the process.

  Next, in step S211, the input / output device 200 detects a partner having the strongest radio wave intensity. Normally, there may be a plurality of WiFi communication radio waves, but here, the hand of the other party who is the communication partner is imaged with a size larger than a predetermined size. Therefore, the detected communication partner who is the owner of the other person's hand should be extremely close, and the processing unit 123 determines that the communication partner (terminal) having the highest radio wave intensity is the other person's hand.

  Next, the input / output device 200 determines whether or not the communication partner (terminal) with the strongest detected radio wave intensity can communicate (step S212). That is, the processing unit 123 determines whether or not a communication partner (terminal) having the strongest radio wave intensity can communicate via the WiFi communication circuit 131. When the communication partner (terminal) having the strongest radio wave intensity is communicable (step S212: Yes), the processing unit 123 proceeds with the process to step S213, and when communication is impossible (step S212: No), End the process.

Next, in step S213, for example, when the communication partner is the HMD 500 shown in FIG. 9 which is the same device as the head mounted display 1 (HMD1), the input / output device 200 determines the personal information of the user of the HMD 500 and the character C3. Receive the information. For example, the processing unit 123 acquires personal information (D3) and character C3 information (information associated with the hand owner) stored in the flash memory 522 of the HMD 500 via the WiFi communication circuit 131.
Here, the communication between the HMD 1 and the HMD 500 may be performed in an ad hoc mode in which devices directly communicate with each other, or may be performed via the server 210 through an Internet line.

  Next, the input / output device 200 causes the display unit 60 to display the character C3 (third character) set by the user of the HMD 500 together with the personal information of the user of the HMD 500 (step S214). That is, as shown in FIG. 20, the input control unit 202 of the processing unit 123 displays an image obtained by superimposing the character C3 acquired via the WiFi communication circuit 131 on the image area of the hand H6 on the display panel 63. 205 is displayed. Further, as shown in FIG. 20, the input control unit 202 displays personal information D3 of the user of the HMD 500 (for example, the handle name of the user of the HMD 500, the date of birth, what he likes, dislikes) on the display panel 63. Display. This display content may be selectable by the user.

Next, the input / output device 200 determines whether or not data exchange is desired (step S215). Here, the processing unit 123 determines whether or not the HMD 1 and the HMD 500 wish to exchange personal information with each other, and if the exchange of personal information is desired (step S215: Yes), the processing proceeds to step S216. Proceeding and exchanging personal information according to screen settings (not shown). Moreover, the process part 123 complete | finishes a process, when not exchanging personal information (step S215: No).
In step S216, the exchanged personal information is recorded (stored) in the memory 127 (527) of the HMD 1 and the HMD 500, the storage unit 212 on the server 210, or the like.

  Next, the input / output device 200 determines whether or not there is still a hand in the captured image (step S217). The processing unit 123 determines whether there are other hands in the captured image. If there are other hands (step S217: Yes), the processing unit 123 returns the processing to step S202 and repeats the above-described processing. Moreover, the process part 123 complete | finishes a process, when there is no other hand (step S217: No).

  On the other hand, if it is determined in step S203 that the user is the left hand of the user, the input / output device 200 executes a series of processes from step S218 to step S222.

  In step S218, the input / output device 200 displays the character C2. This character C2 (second character) is formed of a so-called 3D polygon, and is stored in advance in the memory 127 or the flash memory 122, for example. The input control unit 202 reads the image information of the character C2 from the memory 127 or the flash memory 122, and displays it on the display panel 63 so as to superimpose the palm image region on the display control unit 205 as shown in FIG. Let Similarly to the case of the right hand, the input / output device 200 controls the character C1 to change its direction substantially integrally with the movement of the left hand when the user turns (rotates) the left hand. Note that the character C2 itself may be able to operate independently or speak according to a program created in advance.

  Next, the input / output device 200 displays information (D2) such as the user's personal information (personal information), action schedule, and environment information (step S219). That is, as shown in FIG. 19B, the input control unit 202 causes the display control unit 205 to display the above-described information D2 including personal information on the display panel 63. The character C2 is a character when representing the personal information of the user. For example, the input / output device 200 displays the information D2 described above at a position to the right of the character C2. Here, the input / output device 200 may output the information D2 by sound so that the character C2 is speaking (speaking).

  Next, the input / output device 200 determines whether or not the right hand is added (step S220). That is, the input control unit 202 further determines whether or not a hand is added to the image (in the video) and the hand is a “right hand” opposite to the left hand that has been detected first. If the input control unit 202 determines that the right hand has been added (step S220: Yes), the process proceeds to step S221. If the input control unit 202 determines that the right hand is not added (step S220: No), the process proceeds to step S209.

  Next, in step S221, the input / output device 200 determines whether or not the newly detected right hand is the same size as the user's hand. That is, the input control unit 202 determines whether or not the detected right hand is the user's right hand based on the size of the hand. If the input control unit 202 determines that the detected right hand is the user's right hand (step S221: Yes), the process proceeds to step S222. Further, when the input control unit 202 determines that the detected right hand is not the user's right hand (is the other person's right hand) (step S221: No), the process proceeds to step S209.

  Next, in step S222, the input / output device 200 adjusts the orientation of the character C2 in accordance with the rotation of the user's right hand. Specifically, when the user turns the right hand, the input control unit 202 performs control to change the orientation with the character C2, and advances the process to step S209 described above.

By the way, in this embodiment, gloves are regularly used for work, etc., and when it is difficult to recognize hand wrinkles, or when it is difficult to recognize hand wrinkles in a bright environment, a marker is written on the gloves. You may use things.
FIG. 21 is a diagram illustrating an example when the user is wearing gloves in the present embodiment. As shown in FIG. 21 (A), the glove worn by the user has a marker MK1 indicating “LB” as an initial of “Left Back” indicating the back of the left hand and the tip of the fingernail on the back of the left hand. It has a fingertip marker MK2 to represent. In FIG. 21A, the user's hand H8 included in the image G8 is the back of the left hand and the glove GV1A is worn.

  As shown in FIG. 21 (B), the glove worn by the user has a marker MK3 indicating “LP” which is the initial of “Left Palm” representing the palm of the left hand on the palm of the left hand. FIG. 21B shows a state where the user's hand H9 included in the image G9 is the palm of the left hand and the glove GV1B is worn.

As shown in FIG. 21, when the hand included in the captured image is wearing a glove displaying predetermined identification information (for example, a marker such as “LP” or “LB”), the pattern recognition unit 201 Based on the identification information included in the hand image area recognized based on the captured image, the hand mode (front and back of the hand, left and right of the hand, etc.) is recognized. In this case, the processing unit 123 may determine whether or not the user's hand is based on the identification information.
In FIG. 21A, the fingertip marker MK2 is described on the glove GV1A, so that the pattern recognition unit 201 can easily recognize when the cursor is displayed and identify the back of the hand.

As described above, the input / output device 200 according to the present embodiment displays the information to be superimposed on the captured image based on the camera 64 that captures the image and the aspect of the hand included in the captured image captured by the camera 64. And a processing unit 123 that controls and outputs an image in which information is superimposed on the captured image.
Thereby, the input / output device 200 according to the present embodiment can superimpose information by switching the information to be superimposed according to the hand mode. For example, since the input / output device 200 according to the present embodiment recognizes a hand and uses it as a marker, it is not necessary to prepare a separate sheet called a marker, and easily displays information such as a character anywhere according to the form of the hand. It is possible. Thus, the input / output device 200 in the present embodiment can improve convenience.

  In the present embodiment, the left and right hands (right hand and left hand) are included in the hand mode. The processing unit 123 determines whether the right hand or the left hand is included in the captured image based on the captured image captured by the camera 64, and changes the information to be superimposed based on the determined determination result. That is, the processing unit 123 can change the information to be superimposed depending on whether the right hand or the left hand. Thereby, the input / output device 200 in the present embodiment can improve convenience.

  In the present embodiment, the hand aspect includes the front and back of the hand (the palm and the back of the hand). Based on the captured image captured by the camera 64, the processing unit 123 determines whether the hand included in the captured image is the palm or the back of the hand, and changes the information to be superimposed based on the determined determination result. . That is, the processing unit 123 can change the information to be superimposed depending on whether the palm is the palm or the back of the hand. Thereby, the input / output device 200 in the present embodiment can improve convenience.

  In the present embodiment, when the processing unit 123 determines that the hand is a palm, the processing unit 123 superimposes information on the palm image area of the captured image. As a result, the palm is determined and information such as an image is superimposed and displayed on the palm, so that the input / output device 200 according to the present embodiment can improve convenience.

  In the present embodiment, when the processing unit 123 determines that the hand included in the captured image is a palm, the processing unit 123 superimposes the image of a predetermined character as information to be superimposed. Since the palm is determined and an image of a predetermined character is superimposed on the palm, the input / output device 200 in this embodiment can provide easy-to-understand information by the character to the user. Therefore, the input / output device 200 according to the present embodiment can improve convenience.

Further, in the present embodiment, when the processing unit 123 determines that the hand included in the captured image is the back of the hand, the instruction position is set to the instruction position indicated by the hand corresponding to the determined back of the captured image. Is superimposed as information to be superimposed (for example, the cursor M1).
Thus, by determining the back of the hand, the input / output device 200 in the present embodiment can also be used as an instruction device (pointing device). Therefore, the input / output device 200 in the present embodiment can improve convenience.

Moreover, in this embodiment, the process part 123 determines whether the hand contained in a captured image is a user's hand based on a captured image, and changes the information to superimpose based on the determined determination result. .
As a result, the information to be displayed can be changed depending on whether it is the user's hand or another person's hand, so that the input / output device 200 in this embodiment can improve convenience.

In addition, the input / output device 200 according to the present embodiment includes an angle detection unit 208 that detects an angle indicated by the hand direction included in the captured image as the hand angle based on the captured image. The processing unit 123 determines whether the hand is a user's hand based on the hand angle detected by the angle detection unit 208.
Thereby, the process part 123 can detect a user's hand correctly with a simple structure.

Moreover, in this embodiment, the process part 123 determines whether it is a user's hand based on the magnitude | size of the hand contained in a captured image.
Thereby, the process part 123 can detect a user's hand correctly with a simple structure.

In the present embodiment, the processing unit 123 includes the hand included in the captured image in the image area of the hand recognized based on the captured image when wearing a glove on which predetermined identification information is displayed. It is determined whether or not the user's hand is based on the identification information.
Thereby, for example, when it is difficult to recognize a wrinkle of a hand due to work that regularly uses gloves or a bright environment, the input / output device 200 according to the present embodiment can accurately determine whether or not the hand is a user's hand. it can.

Moreover, in this embodiment, the process part 123 superimposes the information linked | related with the user as information to superimpose, when it determines with the hand contained in a captured image being a user's hand.
Thereby, when it is determined that the user's hand, appropriate information is displayed, so the input / output device 200 according to the present embodiment can improve convenience.

Further, in the present embodiment, when the processing unit 123 determines that the hand included in the captured image is not the user's hand (the hand of another person), the owner of the hand that is determined not to be the user's hand as the information to be superimposed The information associated with is superimposed. In addition, the input / output device 200 includes a communication unit (for example, a WiFi communication circuit 131, a 3G / LTE communication circuit 138, etc.) that communicates information, and the processing unit 123 passes the communication unit to the owner of the hand. Get associated information.
Accordingly, the input / output device 200 according to the present embodiment can appropriately display information on a person (other person) other than the person whose hand is included in the captured image, and thus can improve convenience. .

In the present embodiment, a memory 127 that stores information associated with the hand owner acquired by the WiFi communication circuit 131 is provided.
As a result, a record (history) in which information is exchanged with a person other than the person (other person) can be left, and this information can be effectively used later. Therefore, the input / output device 200 according to the present embodiment can improve convenience.

Further, the input / output device 200 according to the present embodiment includes a pattern recognition unit 201 that recognizes a hand mode based on a captured image captured by the camera 64. The processing unit 123 controls information to be superimposed on the captured image based on the hand mode recognized by the pattern recognition unit 201.
Since the pattern recognition unit 201 recognizes the hand mode based on the captured image, the processing unit 123 can control information to be superimposed on the captured image according to the hand mode with a simple configuration. Therefore, the input / output device 200 according to the present embodiment can improve convenience.

Further, in the present embodiment, the pattern recognition unit 201 applies an image area of the hand recognized based on the captured image when the hand included in the captured image is wearing a glove displaying predetermined identification information. Based on the identification information included, the hand mode is recognized.
Thereby, the input / output device 200 in the present embodiment can accurately recognize the hand mode with a simple configuration.

The head mounted display 1 according to the present embodiment includes the input / output device 200 described above, the display panel 63 that displays an image output from the processing unit 123, and at least the camera 64 and the display panel 63 on the user's head. And a headband 40 that can be worn.
Thereby, since the head mounted display 1 in this embodiment is provided with the input / output device 200 mentioned above, there exists an effect similar to the input / output device 200 mentioned above, and can improve the convenience.

The present invention is not limited to the above embodiments, and can be modified without departing from the spirit of the present invention.
For example, in each of the above embodiments, the pattern recognition unit 201 and the angle detection unit 208 have been described as being included in the processing unit 123. However, the present invention is not limited to this, and the pattern recognition unit 201 and the angle detection unit are not limited thereto. 208 may be provided outside the processing unit 123.

In each of the above embodiments, the input / output device 200 is an example of an operation unit, and the menu selection processing unit 204 performs an operation of selecting one of a plurality of operation menus (a plurality of menu icons I1 to I9). However, the present invention is not limited to this. For example, the input / output device 200 may include an operation unit that performs operations such as handwritten character input and information input using a keyboard displayed on the screen in addition to menu selection.
Further, in each of the embodiments described above, the input / output device 200 has been described with respect to a mode in which a cursor is displayed. However, an AR (Augmented Reality) pen may be displayed instead of the cursor. Further, the input / output device 200 may be configured to change the color and shape of the cursor according to an operation, for example.

In each of the above-described embodiments, the embodiment using the WiFi communication circuit 131 as an example of the communication communication interface (communication unit) has been described. However, the present invention is not limited to this. For example, the BT communication circuit 130 may communicate using the Bluetooth (registered trademark) communication standard, or the 3G / LTE communication circuit 138 may communicate.
Further, the information exchanged in step S216 of FIG. 14C described above is not limited to the information displayed on the display panel 63 as shown in FIG. In FIG. 20, limited information is displayed for easy viewing. However, in data communication, for example, detailed information such as an address and a mail address may be communicated and recorded in the memory 127.

Moreover, the marker described (displayed) on the glove is not limited to the example of FIG. For example, a more complicated pattern may be drawn on the glove marker, and the shape of the character displayed by the marker and the displayed information may be changed. Further, as a complicated pattern, for example, a barcode or a QR code (registered trademark) may be printed (displayed) to include more information.
Thus, for example, in an event or the like, for example, by changing a male glove and a female glove, the input / output device 200 can match the gender of the user without setting the head mounted display 1 in advance. It is also possible to display different characters. Further, the input / output device 200 may recognize the shape of the hand and change the character type, posture, and the like.

In each of the above embodiments, a mode has been described in which a cursor is displayed when the back of the hand is observed, and a character is displayed when the palm is observed. However, the present invention is not limited to this. For example, the processing unit 123 may be configured to display four types of characters and different information for the palm and back, and the right and left hands. Further, the processing unit 123 may be configured to display the character by rotating it 180 degrees in the same manner when the hand is rotated 180 degrees from the palm of the hand to the back of the hand.
Furthermore, the processing unit 123 may perform not only the user's hand but also the hand other than the user to identify the left and right hands and the front and back of the hand. The processing unit 123 may change the information to be superimposed based on this identification.

  The head mounted display 1 and the input / output device 200 described above have a computer system therein. The process of the input / output device 200 described above is stored in a computer-readable recording medium in the form of a program, and the above process is performed by the computer reading and executing the program. Here, the computer-readable recording medium means a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

  DESCRIPTION OF SYMBOLS 1 ... Head mounted display, 40 ... Headband, 63 ... Display panel, 64 ... Camera, 123 ... Processing part, 127 ... Memory, 131 ... WiFi communication circuit, 200 ... Input / output device, 201 ... Pattern recognition part, 202 ... Input Control unit, 204 ... menu selection processing unit, 208 ... angle detection unit

Claims (17)

An imaging unit that captures an image;
A control unit that controls information to be superimposed on the captured image based on a hand shape included in the captured image captured by the imaging unit, and outputs an image in which the information is superimposed on the captured image. An information input / output device. The aspect of the hand includes a right hand and a left hand,
The processor is
The information to be superimposed is changed based on the determined determination result by determining whether the hand included in the captured image is a right hand or a left hand based on the captured image. The information input / output device described in 1. The aspect of the hand includes the palm and the back of the hand,
The processor is
The information to be superimposed is changed based on the determined determination result by determining whether a hand included in the captured image is a palm or a back of the hand based on the captured image. Alternatively, the information input / output device according to claim 2. The processor is
The information input according to claim 3, wherein, when it is determined that the hand included in the captured image is a palm, the information to be superimposed is superimposed on the palm image area of the captured image. Output device. The processor is
5. The information input / output device according to claim 3, wherein, when it is determined that the hand included in the captured image is a palm, an image of a predetermined character is superimposed as the information to be superimposed. The processor is
When it is determined that the hand included in the captured image is the back of the hand, the mark indicating the indicated position is superimposed on the indicated position indicated by the hand corresponding to the determined back of the hand in the captured image as the information to be superimposed The information input / output device according to claim 3, wherein the information input / output device is superimposed. The processor is
The information to be superimposed is changed based on the determined determination result based on the determined determination result based on the determination result whether or not the hand included in the captured image is a user's hand. The information input / output device according to claim 6. A detection unit that detects, based on the captured image, an angle indicated by a hand direction included in the captured image as a hand angle;
The processor is
The information input / output device according to claim 7, wherein it is determined whether the hand is a user's hand based on an angle of the hand detected by the detection unit. The processor is
The information input / output apparatus according to claim 7 or 8, wherein it is determined whether or not the hand is a user's hand based on a size of a hand included in the captured image. The processor is
Based on the identification information included in the image area of the hand recognized based on the captured image when the hand included in the captured image is wearing a glove displaying predetermined identification information. The information input / output device according to claim 7, wherein the information input / output device determines whether the hand is a hand. The processor is
The information associated with the user is superimposed as the information to be superimposed when it is determined that the hand included in the captured image is a user's hand. The information input / output device according to any one of the above. The processor is
When it is determined that the hand included in the captured image is not a user's hand, the information to be superimposed is superimposed on information associated with the owner of the hand that is determined not to be the user's hand. The information input / output device according to any one of claims 7 to 10. A communication unit that communicates information;
The processor is
The information input / output device according to claim 12, wherein information associated with the owner of the hand is acquired via the communication unit. The information input / output device according to claim 13, further comprising a storage unit that stores information associated with the owner of the hand acquired via the communication unit. A recognition unit for recognizing the form of the hand based on the captured image;
The processor is
The information input / output device according to any one of claims 1 to 14, wherein information to be superimposed on the captured image is controlled based on the form of the hand recognized by the recognition unit. The recognition unit
When the hand included in the captured image is wearing a glove on which predetermined identification information is displayed, based on the identification information included in the image area of the hand recognized based on the captured image, the hand The information input / output device according to claim 15, wherein the information input / output device is recognized. The information input / output device according to any one of claims 1 to 16,
A display unit for displaying an image output by the processing unit;
A head-mounted display device comprising: a mounting unit capable of mounting at least the imaging unit and the display unit on a user's head.

Images