WO2017221809A1 - Display device and gesture input method - Google Patents

Abstract

Provided is a display device, comprising a display unit, a detection unit, a mode control unit, a storage unit, a storage control unit, and a processing unit. The detection unit comprises a detection region which is in a different position from the display unit, and is capable of differentiating and detecting two or more predetermined gestures. The mode control unit commences an acceptance mode of accepting a gesture input, and ends the acceptance mode after a prescribed interval has elapsed. After the commencement of the acceptance mode, the storage control unit carries out a control which causes the storage unit to store, for a series of gestures which are executed a plurality of times, input information which indicates an input which has been assigned to the gestures, in sequence beginning with the gesture which has been first detected by the detection unit. After the ending of the acceptance mode, the processing unit carries out a prescribed process, using the input information which is stored in the storage unit of each of the plurality of gestures which configure the series of gestures.

Description

Display device and gesture input method

The present invention relates to a display device capable of gesture input.

Gesture input is to operate a display device (for example, a terminal device or a game machine) by gesture or hand gesture. For example, in the case of a smartphone, gesture input can be performed by touching the screen.

∙ You may not want to touch the screen when the user's hand is dirty. Therefore, gesture input without using a screen has been proposed. For example, Patent Literature 1 discloses a display device that includes a display, a motion reception unit, and a display control unit that controls display information displayed on the display according to the motion received by the motion reception unit.

In Patent Document 1, the motion accepting unit accepts only the motion of the hand moving in the left direction and the motion of the hand moving in the right direction, and can perform only a simple operation. For this reason, since input corresponding to a plurality of key operations cannot be performed, for example, personal authentication by password input cannot be performed.

Personal authentication can be performed using devices such as a hardware keyboard, fingerprint authentication device, and vein sensor. However, when the device is used for personal authentication of a portable terminal or wearable terminal, the device must be carried, which is inconvenient. When personal authentication is performed using a barcode or QR code (registered trademark), there is a risk of loss or theft of the barcode or QR code. When performing personal authentication by voice recognition, there is a risk that someone else will ask for your password.

* Further improvement of gesture input without using a screen is desired so that passwords can be entered.

JP 2010-1229069 A

An object of the present invention is to provide a display device that improves gesture input without using a screen, and a gesture input method applied to the display device.

The display device according to one aspect of the present invention includes a display unit, a detection unit, a mode control unit, a storage unit, a storage control unit, and a processing unit. The detection unit has a detection region at a position different from that of the display unit, and can detect and distinguish two or more predetermined gestures. The mode control unit starts a reception mode for receiving a gesture input, and ends the reception mode after a predetermined period. The storage control unit, for a series of gestures executed a plurality of times after the start of the reception mode, input information indicating inputs pre-assigned to the gestures in order from the gesture first detected by the detection unit Is stored in the storage unit. The processing unit performs a predetermined process using the input information of each of a plurality of gestures constituting the series of gestures stored in the storage unit after the acceptance mode ends.

The above and other objects, features and advantages of the present invention will become apparent from the following detailed description and the accompanying drawings.

It is a perspective view which shows the structural structure of HMD which concerns on this embodiment. It is a front view which shows the structural structure of HMD which concerns on this embodiment. It is a schematic sectional drawing which shows the structure of the display unit with which HMD which concerns on this embodiment is equipped. It is a figure which shows the structure of the proximity sensor with which HMD which concerns on this embodiment is equipped. It is a block diagram which shows the electrical structure of HMD which concerns on this embodiment. It is a front view at the time of mounting HMD concerning this embodiment. It is a side view at the time of mounting HMD concerning this embodiment. It is a partial top view at the time of mounting | wearing with HMD which concerns on this embodiment. It is a figure which shows an example of the image which a user visually recognizes through a see-through type image display part. It is a figure which shows an example of the output of the proximity sensor with which HMD which concerns on this embodiment is equipped. It is explanatory drawing explaining the relationship between a gesture and input information. FIG. 10 is a waveform diagram showing changes in output levels of focus elements RA to RD when gesture 1 is made. FIG. 9 is a waveform diagram showing changes in the output levels of focus elements RA to RD when gesture 2 is made. FIG. 7 is a waveform diagram showing changes in the output levels of focus elements RA to RD when gesture 3 is made. FIG. 10 is a waveform diagram showing changes in the output levels of focus elements RA to RD when gesture 4 is made. It is a flowchart explaining the operation | movement at the time of gesture input in HMD which concerns on this embodiment. FIG. 6 is a waveform diagram showing changes in the output levels of focus elements RA to RD when a series of gestures (gesture 2, gesture 1, gesture 4) is performed for password input. FIG. 10 is a waveform diagram showing changes in the output levels of the focus elements RA to RD when a gesture 11 is made to switch to the next screen. FIG. 11 is a waveform diagram showing changes in the output levels of focus elements RA to RD when a series of gestures (gesture 11, gesture 12, gesture 11) indicating “bye-bye” is performed. It is a flowchart explaining the operation | movement at the time of the gesture input in the modification 2 of this embodiment. FIG. 6 is a waveform diagram showing changes in the output levels of focus elements RA to RD when a gesture is made to switch to the next screen. It is a flowchart explaining the operation | movement at the time of the gesture input in the modification 3 of this embodiment. FIG. 16 is a screen diagram illustrating a first example of a screen displayed on an image display unit in Modification 5 of the present embodiment. In Modification 5 of this embodiment, it is a screen figure which shows the 2nd example of the screen displayed on the image display part. In Modification 5 of this embodiment, it is a screen figure which shows the 3rd example of the screen displayed on the image display part.

Hereinafter, an embodiment according to the present invention will be described with reference to the drawings. In addition, the structure which attached | subjected the same code | symbol in each figure shows that it is the same structure, The description is abbreviate | omitted suitably.

The display device according to the present embodiment is, for example, a wearable terminal (for example, a head mounted display (HMD) or a wristwatch type terminal) or a smart terminal (for example, a smartphone or a tablet terminal). In this specification, a head mounted display (HMD) will be described as an example.

FIG. 1 is a perspective view showing a structural configuration of the HMD 100 according to the present embodiment. FIG. 2 is a front view showing a structural configuration of the HMD 100 according to the present embodiment. FIG. 3 is a schematic cross-sectional view showing the configuration of the display unit 104 provided in the HMD 100 according to the present embodiment. FIG. 4 is a diagram illustrating a configuration of the proximity sensor 105 provided in the HMD 100 according to the present embodiment. FIG. 5 is a block diagram showing an electrical configuration of the HMD 100 according to the present embodiment. Hereinafter, the right side and the left side of the HMD 100 refer to the right side and the left side for the user wearing the HMD 100.

The structural configuration of the HMD 100 will be described. With reference to FIG.1 and FIG.2, HMD100 which concerns on this embodiment is provided with the flame | frame 101 which is an example of the head mounting member for mounting | wearing a head. The frame 101 includes a front part 101a to which two spectacle lenses 102 are attached, and side parts 101b and 101c extending rearward from both ends of the front part 101a. The two spectacle lenses 102 attached to the frame 101 may or may not have refractive power (optical power, reciprocal of focal length).

The cylindrical main body 103 is fixed to the front part 101 a of the frame 101 at the upper part of the right eyeglass lens 102 (which may be the left side according to the user's dominant eye etc.). The main body 103 is provided with a display unit 104. A display control unit 104DR (FIG. 5) that performs display control of the display unit 104 based on an instruction from a control processing unit 121 described later is disposed in the main body unit 103. Note that a display unit may be disposed in front of both eyes as necessary.

Referring to FIG. 3, the display unit 104 includes an image forming unit 104A and an image display unit 104B. The image forming unit 104A is incorporated in the main body unit 103, and includes a light source 104a, a one-way diffusing plate 104b, a condenser lens 104c, and a display element 104d. On the other hand, the image display unit 104B, which is a so-called see-through type display member, is generally plate-shaped and is disposed so as to extend downward from the main body unit 103 and parallel to one eyeglass lens 102 (FIG. 1). The eyepiece prism 104f, the deflecting prism 104g, and the hologram optical element 104h are included.

The light source 104a has a function of illuminating the display element 104d. For example, the peak wavelength of light intensity and the half width of the light intensity are 462 ± 12 nm (blue light (B light)), 525 ± 17 nm (green light (G light )), 635 ± 11 nm (red light (R light)), and is composed of RGB integrated light emitting diodes (LEDs) that emit light in three wavelength bands.

The display element 104d displays an image by modulating the light emitted from the light source 104a in accordance with image data, and is configured by a transmissive liquid crystal display element having pixels that serve as light transmitting regions in a matrix. Is done. Note that the display element 104d may be of a reflective type.

The eyepiece prism 104f totally reflects the image light from the display element 104d incident through the base end face PL1 by the opposed parallel inner side face PL2 and outer side face PL3, and passes through the hologram optical element 104h to the user's pupil. On the other hand, it transmits external light and guides it to the user's pupil, and is formed of, for example, an acrylic resin together with the deflecting prism 104g. The eyepiece prism 104f and the deflection prism 104g are joined by an adhesive with the hologram optical element 104h sandwiched between inclined surfaces PL4 and PL5 inclined with respect to the inner surface PL2 and the outer surface PL3.

The deflection prism 104g is joined to the eyepiece prism 104f, and becomes a substantially parallel flat plate integrated with the eyepiece prism 104f. In addition, if the spectacle lens 102 (FIG. 1) is mounted between the display unit 104 and the user's pupil, it is possible for a user who normally uses spectacles to observe an image.

The hologram optical element 104h diffracts and reflects the image light (light having a wavelength corresponding to the three primary colors) emitted from the display element 104d, guides it to the pupil B, enlarges the image displayed on the display element 104d, and enlarges the user's pupil. It is a volume phase type reflection hologram guided as a virtual image. The hologram optical element 104h has, for example, three wavelength ranges of 465 ± 5 nm (B light), 521 ± 5 nm (G light), and 634 ± 5 nm (R light) with a peak wavelength of diffraction efficiency and a wavelength width of half the diffraction efficiency. The light is diffracted (reflected). Here, the peak wavelength of diffraction efficiency is the wavelength at which the diffraction efficiency reaches a peak, and the wavelength width at half maximum of the diffraction efficiency is the wavelength width at which the diffraction efficiency is at half maximum of the diffraction efficiency peak. is there.

In the display unit 104 having such a configuration, light emitted from the light source 104a is diffused by the unidirectional diffusion plate 104b, condensed by the condenser lens 104c, and incident on the display element 104d. The light incident on the display element 104d is modulated for each pixel based on the image data input from the display control unit 104DR, and is emitted as image light. Thereby, a color image is displayed on the display element 104d. The image light from the display element 104d enters the eyepiece prism 104f from its base end face PL1, is totally reflected a plurality of times by the inner side face PL2 and the outer side face PL3, and enters the hologram optical element 104h. The light incident on the hologram optical element 104h is reflected there, passes through the inner side surface PL2, and reaches the pupil B. At the position of the pupil B, the user can observe an enlarged virtual image of the image displayed on the display element 104d, and can visually recognize it as a screen formed on the image display unit 104B.

On the other hand, since the eyepiece prism 104f, the deflecting prism 104g, and the hologram optical element 104h transmit almost all of the external light, the user can observe the external image (real image) through these. Therefore, the virtual image of the image displayed on the display element 104d is observed so as to overlap with a part of the external image. In this way, the user of the HMD 100 can simultaneously observe the image provided from the display element 104d and the external image via the hologram optical element 104h. In addition, when the display unit 104 is in a non-display state, the image display unit 104B is transparent and can observe only an external image. In this embodiment, the display unit is configured by combining a light source, a liquid crystal display element, and an optical system. However, instead of the combination of the light source and the liquid crystal display element, a self-luminous display element (for example, an organic EL display) is used. Element) may be used. Further, instead of a combination of a light source, a liquid crystal display element, and an optical system, a transmissive organic EL display panel having transparency in a non-light emitting state may be used.

With reference to FIGS. 1 and 2, on the front surface of the main body 103, a proximity sensor 105 disposed near the center of the frame 101 and a lens 106a of the camera 106 disposed near the side face the front. It is provided to face.

In this specification, the “proximity sensor” refers to a proximity range in front of a detection surface of a proximity sensor in order to detect that an object, for example, a part of a human body (such as a hand or a finger) is close to the user's eyes The signal is output by detecting whether or not it exists within the detection area. The proximity range may be set as appropriate according to the characteristics and preferences of the user. For example, the proximity range from the detection surface of the proximity sensor can be within a range of 200 mm. If the distance from the proximity sensor is 200 mm or less, the user can put a palm or a finger into or out of the user's field of view with the arm bent, so that the hand, finger, or pointing tool (for example, a rod-shaped member) It is possible to easily perform an operation by a gesture using the symbol, and the possibility of erroneous detection of a human body or furniture other than the user is reduced.

There are two types of proximity sensors: passive type and active type. A passive proximity sensor has a detection device that detects invisible light and electromagnetic waves emitted from an object when the object approaches. As a passive proximity sensor, there are a pyroelectric sensor that detects invisible light such as infrared rays emitted from an approaching human body, an electrostatic capacitance sensor that detects a change in electrostatic capacitance between the approaching human body, and the like. The active proximity sensor includes a projection device that projects invisible light and sound waves, and a detection device that receives invisible light and sound waves that have been reflected back to the object. Active proximity sensors include infrared sensors that project infrared rays and receive infrared rays reflected by objects, laser sensors that project laser beams and receive laser beams reflected by objects, and project ultrasonic waves. Then, there is an ultrasonic sensor that receives ultrasonic waves reflected by an object. Note that a passive proximity sensor does not need to project energy toward an object, and thus has excellent low power consumption. An active proximity sensor is easy to improve the certainty of detection. For example, even when a user wears a glove that does not transmit detection light emitted from a human body such as infrared light, Can detect movement. A plurality of types of proximity sensors may be combined.

Referring to FIG. 4, in this embodiment, a pyroelectric sensor including a plurality of pyroelectric elements arranged in a two-dimensional matrix is used as the proximity sensor 105. The proximity sensor 105 includes four pyroelectric elements RA, RB, RC, and RD arranged in two rows and two columns, and receives invisible light such as infrared light emitted from the human body as detection light. A corresponding signal is output from each of the pyroelectric elements RA to RD. The outputs of the pyroelectric elements RA to RD change in intensity according to the distance from the light receiving surface of the proximity sensor 105 to the object, and the intensity increases as the distance decreases.

Referring to FIGS. 1 and 2, the right sub-body portion 108-R is attached to the right side portion 101b of the frame 101, and the left sub-body portion 108-L is attached to the left side portion 101c of the frame 101. Is attached. The right sub body 108-R and the left sub body 108-L have an elongated plate shape.

The main main body 103 and the right sub main body 108-R are connected to each other by a wiring HS so that signals can be transmitted. The right sub main body 108-R is connected to the control unit CTU via a cord CD extending from the rear end thereof. It is connected to the.

Next, the electrical configuration of the HMD 100 will be described. Referring to FIG. 5, the HMD 100 includes a control unit CTU, a display unit 104, a display control unit 104DR, a proximity sensor 105, and a camera 106. The control unit CTU includes a control processing unit 121, an operation unit 122, a storage unit 125, a battery 126, and a power supply circuit 127.

The display control unit 104DR is a circuit that is connected to the control processing unit 121 and controls the image forming unit 104A of the display unit 104 according to the control of the control processing unit 121 to form an image on the image forming unit 104A. The image forming unit 104A is as described above.

The camera 106 is an apparatus that is connected to the control processing unit 121 and generates an image of a subject under the control of the control processing unit 121. The camera 106 is, for example, an image forming optical system that forms an optical image of a subject on a predetermined image forming surface, and a light receiving surface that matches the image forming surface. An image sensor that converts the image sensor into a digital signal processor (DSP) that performs known image processing on the output of the image sensor to generate an image (image data). The imaging optical system includes one or more lenses, and includes the lens 106a as one of them. The camera 106 outputs the generated image data to the control processing unit 121.

The proximity sensor 105 is connected to the control processing unit 121. The proximity sensor 105 is as described above, and outputs the output to the control processing unit 121.

The operation unit 122 is connected to the control processing unit 121 and is a device that inputs a predetermined instruction, such as power on / off, to the HMD 100, for example, one or a plurality of switches assigned a predetermined function Etc.

The battery 126 is a battery that accumulates electric power and supplies the electric power. The battery 126 may be a primary battery or a secondary battery. The power supply circuit 127 is a circuit that supplies power supplied from the battery 126 to each part of the HMD 100 that requires power at a voltage corresponding to each part.

The storage unit 125 is a circuit that is connected to the control processing unit 121 and stores various predetermined programs and various predetermined data under the control of the control processing unit 121. Examples of the various predetermined programs include control processing programs such as a control program that controls each unit of the HMD 100 according to the function of each unit, and a gesture processing program that determines a gesture based on the output of the proximity sensor 105. included. The storage unit 125 includes, for example, a ROM (Read Only Memory) that is a nonvolatile storage element, an EEPROM (Electrically Erasable Programmable Read Only Memory) that is a rewritable nonvolatile storage element, and the like. The storage unit 125 includes a RAM (Random Access Memory) that serves as a working memory of the control processing unit 121 that stores data generated during the execution of the predetermined program.

The control processing unit 121 controls each unit of the HMD 100 according to the function of each unit, determines a predetermined gesture set in advance based on the output of the proximity sensor 105, and executes processing according to the determination result. Is. The control processing unit 121 includes, for example, a CPU (Central Processing Unit) and its peripheral circuits. In the control processing unit 121, a control processing program is executed, so that a control unit 1211, a gesture processing unit 1212, and a processing unit 1213 are functionally configured. Note that some or all of the control unit 1211, the gesture processing unit 1212, and the processing unit 1213 may be configured by hardware.

The control unit 1211 controls each unit of the HMD 100 according to the function of each unit. The control unit 1211 has functions of a mode control unit 1214 and a storage control unit 1215. These functions will be described later.

The gesture processing unit 1212 determines a predetermined gesture set in advance based on the outputs of the plurality of pyroelectric elements in the proximity sensor 105, in this embodiment, the four pyroelectric elements RA to RD. The gesture processing unit 1212 notifies the processing unit 1213 of the determination result. The gesture processing unit 1212 and the proximity sensor 105 constitute a detection unit 128. The detection unit 128 has a detection area SA (FIGS. 7A, 7B, and 8) at a position different from that of the image display unit 104B (an example of the display unit), and distinguishes and detects two or more predetermined gestures.

The processing unit 1213 performs a predetermined process (for example, password authentication) using the determination result of the gesture processing unit 1212. Details of the processing unit 1213 will be described later.

The basic operation for detecting a gesture in the HMD 100 will be described. FIG. 6 is a front view when the HMD 100 according to the present embodiment is mounted. FIG. 7A is a side view when the HMD 100 according to the present embodiment is mounted. FIG. 7B is a partial top view in this case. In these figures, the hand HD of the user US is also shown. FIG. 8 is a diagram illustrating an example of an image visually recognized by the user through the see-through type image display unit 104B. FIG. 9 is a diagram illustrating an example of the output of the proximity sensor 105 provided in the HMD 100 according to the present embodiment. 9A shows the output of the pyroelectric element RA, FIG. 9B shows the output of the pyroelectric element RB, FIG. 9C shows the output of the pyroelectric element RC, and FIG. (D) shows the output of the pyroelectric element RD. The horizontal axis of each figure in FIG. 9 is time, and the vertical axis thereof is the output level (intensity). Here, the gesture input is an operation in which at least the hand HD or finger of the user US enters or leaves the detection area SA of the proximity sensor 105, and the gesture processing unit of the control processing unit 121 of the HMD 100 via the proximity sensor 105. 1212 can be detected.

Referring to FIG. 8, screen 104i of image display unit 104B is arranged so as to overlap with effective visual field EV of the user's eye facing image display unit 104B (here, positioned within effective visual field EV). The The detection area SA of the proximity sensor 105 is in the visual field of the user's eye facing the image display unit 104B. Preferably, the detection area SA is located within the stable field of view of the user's eye or the field inside thereof (within about 90 ° horizontal and within about 70 ° vertical), and more preferably located inside the stable field of view. The proximity sensor 105 may be installed with its arrangement and orientation adjusted so as to overlap the effective visual field EV or the inner visual field (horizontal within about 30 °, vertical within about 20 °).

FIG. 8 shows an example in which the detection area SA overlaps the screen 104i. In this way, by setting the detection region SA of the proximity sensor 105 within the visual field of the eye of the user US while the user US is wearing the frame 101 that is the head mounting member on the head, the screen While observing the hand HD through 104i, the approach and retraction of the hand to the detection area SA of the proximity sensor 105 can be surely visually recognized without moving the eye. In particular, by setting the detection area SA of the proximity sensor 105 within the stable visual field or the inner visual field, it is possible to reliably perform gesture input while recognizing the detection area SA even when the user observes the screen. . Further, by making the detection area SA of the proximity sensor 105 within the effective visual field EV or the visual field inside the effective visual field EV, gesture input can be performed more reliably. If the detection area SA overlaps the screen 104i, gesture input can be performed more reliably. When the proximity sensor 105 has a plurality of pyroelectric elements RA to RD as in the present embodiment, the entire light receiving area of the plurality of pyroelectric elements RA to RD is regarded as one light receiving unit, and It is assumed that the maximum detection range is the detection area SA. As shown in FIG. 8, when the detection area SA of the proximity sensor 105 is set to overlap the screen 104i, an image showing the detection area SA is displayed on the screen 104i (for example, the range of the area SA is displayed by a solid line). ), The user can surely recognize the detection area SA, so that the operation by the gesture can be more reliably performed.

Next, the basic principle of gesture detection will be described. If there is nothing in front of the user US when the proximity sensor 105 is operating, the proximity sensor 105 does not receive invisible light as detection light, so the gesture processing unit 1212 of the control processing unit 121 It is determined that no gesture is performed. On the other hand, as shown in FIGS. 7A and 7B, when the user US's own hand HD is approached in front of the user US, the proximity sensor 105 detects invisible light emitted from the hand HD, and the proximity based on the invisible light is detected. In response to the output signal from the sensor 105, the gesture processing unit 1212 determines that a gesture has been performed. In the following description, it is assumed that a gesture is performed with the hand HD of the user US. However, the gesture may be performed by the user US using an indicator made of a material capable of emitting invisible light. You may go.

As described above, the proximity sensor 105 has four pyroelectric elements RA to RD arranged in two rows and two columns (see FIG. 4). Therefore, when the user US brings the hand HD close to the front of the HMD 100 from either the left, right, up, or down directions, the output timings of signals detected by the pyroelectric elements RA to RD are different.

For example, referring to FIG. 7A, FIG. 7B and FIG. 8, in the case of a gesture in which the user US moves the hand HD from the right to the left in front of the HMD 100, the invisible light emitted from the hand HD is close The light enters the sensor 105. In this case, the pyroelectric elements RA and RC first receive invisible light. Therefore, referring to FIGS. 4 and 9, first, the signals of pyroelectric elements RA and RC rise, and the signals of pyroelectric elements RB and RD rise after a delay. Thereafter, the signals of the pyroelectric elements RA and RC fall, and the signals of the pyroelectric elements RB and RD fall after a delay. The gesture processing unit 1212 detects the timing of this signal, and the gesture processing unit 1212 determines that the user US has made a gesture by moving the hand HD from right to left.

This embodiment will be described using twelve gestures as examples of two or more predetermined gestures. Two or more means a plurality and is not limited to twelve. FIG. 10 is an explanatory diagram for explaining the relationship between a gesture and input information. FIG. 10 includes arrows indicating the hand movements of each of the 12 gestures, and input information indicating inputs pre-assigned to each of the 12 gestures.

The gesture 1 will be described with reference to FIG. 4, FIG. 10, and FIG. FIG. 11 is a waveform diagram showing changes in the output levels of the focus elements RA to RD when the gesture 1 is performed. In this waveform diagram, the horizontal axis indicates time, and the vertical axis indicates the output level. The threshold values th are all the same value.

The gesture 1 is a gesture in which the hand HD enters the detection area SA from the upper side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the lower side of the detection area SA. When the user performs gesture 1, the output levels of the pyroelectric elements RA and RB exceed the threshold value th, and later, the output levels of the pyroelectric elements RC and RD exceed the threshold value th, and The output levels of the pyroelectric elements RA and RB become equal to or lower than the threshold value th, and the output levels of the pyroelectric elements RC and RD become equal to or lower than the threshold value th later. When the gesture processing unit 1212 detects such a change in output level, the gesture processing unit 1212 (FIG. 5) determines that the gesture 1 has been made. The input information indicating the input previously assigned to the gesture 1 is “1”. The user can input the number 1 to the HMD 100 by making the gesture 1.

The gesture 2 will be described with reference to FIG. 4, FIG. 10, and FIG. FIG. 12 is a waveform diagram showing changes in the output levels of the focus elements RA to RD when the gesture 2 is performed. The horizontal axis and vertical axis of this waveform diagram are the same as the horizontal axis and vertical axis of the waveform diagram of FIG.

The gesture 2 is a gesture in which the hand HD enters the detection area SA from the lower side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the upper side of the detection area SA. When the user performs gesture 2, the output levels of the pyroelectric elements RC and RD exceed the threshold value th, and later, the output levels of the pyroelectric elements RA and RB exceed the threshold value th, and The output levels of the pyroelectric elements RC and RD become the threshold value th or less, and the output levels of the pyroelectric elements RA and RB become the threshold value th or later after this. When the gesture processing unit 1212 (FIG. 5) detects such a change in output level, the gesture processing unit 1212 determines that the gesture 2 has been made. The input information indicating the input previously assigned to the gesture 2 is “2”. The user can input the number 2 to the HMD 100 by making the gesture 2.

The gesture 3 will be described with reference to FIGS. 4, 10, and 13. FIG. 13 is a waveform diagram showing changes in the output levels of the focus elements RA to RD when the gesture 3 is performed. The horizontal axis and vertical axis of this waveform diagram are the same as the horizontal axis and vertical axis of the waveform diagram of FIG.

The gesture 3 is a gesture in which the hand HD enters the detection area SA from the right side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the lower side of the detection area SA. When the user performs gesture 3, the output levels of the pyroelectric elements RA and RC exceed the threshold value th, and later, the output levels of the pyroelectric elements RB and RD exceed the threshold value th, and The output levels of the pyroelectric elements RA and RB become equal to or lower than the threshold value th, and the output levels of the pyroelectric elements RC and RD become equal to or lower than the threshold value th later. When the gesture processing unit 1212 (FIG. 5) detects such a change in output level, the gesture processing unit 1212 determines that the gesture 3 has been made. The input information indicating the input previously assigned to the gesture 3 is “3”. The user can input the number 3 to the HMD 100 by making the gesture 3.

The gesture 4 will be described with reference to FIG. 4, FIG. 10, and FIG. FIG. 14 is a waveform diagram showing changes in the output levels of the focus elements RA to RD when the gesture 4 is performed. The horizontal axis and vertical axis of this waveform diagram are the same as the horizontal axis and vertical axis of the waveform diagram of FIG.

The gesture 4 is a gesture in which the hand HD enters the detection area SA from the lower side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the right side of the detection area SA. When the user performs gesture 4, the output levels of the pyroelectric elements RC and RD exceed the threshold value th, and later, the output levels of the pyroelectric elements RA and RB exceed the threshold value th, and The output levels of the pyroelectric elements RB and RD become the threshold value th or less, and the output levels of the pyroelectric elements RA and RC become the threshold value th or later after this. When the gesture processing unit 1212 (FIG. 5) detects such a change in output level, the gesture processing unit 1212 determines that the gesture 4 has been made. The input information indicating the input previously assigned to the gesture 4 is “4”. The user can input the number 4 to the HMD 100 by making the gesture 4.

Waveform diagrams are omitted for Gesture 5 to Gesture 12. Regarding these gestures, description of waveform changes is omitted (the same concept as gestures 1 to 4).

The gesture 5 will be described with reference to FIGS. 4 and 10. The gesture 5 is a gesture in which the hand HD enters the detection area SA from the right side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the upper side of the detection area SA. The input information indicating the input previously assigned to the gesture 5 is “5”. The user can input the number 5 to the HMD 100 by making the gesture 5.

Explain gesture 6. The gesture 6 is a gesture in which the hand HD enters the detection area SA from the upper side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the right side of the detection area SA. The input information indicating the input previously assigned to the gesture 6 is “6”. The user can input the number 6 to the HMD 100 by making the gesture 6.

Explain gesture 7. The gesture 7 is a gesture in which the hand HD enters the detection area SA from the left side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the lower side of the detection area SA. The input information indicating the input previously assigned to the gesture 7 is “7”. The user can input the number 7 to the HMD 100 by making the gesture 7.

Explain gesture 8. The gesture 8 is a gesture in which the hand HD enters the detection area SA from the lower side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the left side of the detection area SA. The input information indicating the input previously assigned to the gesture 8 is “8”. The user can input the number 8 to the HMD 100 by making the gesture 8.

Explain gesture 9. The gesture 9 is a gesture in which the hand HD enters the detection area SA from the left side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the upper side of the detection area SA. The input information indicating the input previously assigned to the gesture 9 is “9”. The user can input the number 9 to the HMD 100 by making the gesture 9.

The gesture 10 will be described. The gesture 10 is a gesture in which the hand HD enters the detection area SA from the upper side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the left side of the detection area SA. The input information indicating the input previously assigned to the gesture 10 is “10”. The user can input the number 0 to the HMD 100 by making the gesture 10.

The gesture 11 will be described. The gesture 11 is a gesture in which the hand HD enters the detection area SA from the left side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the right side of the detection area SA. The input information indicating the input previously assigned to the gesture 11 is a “command to switch to the next screen”. The user can input a “command to switch to the next screen” to the HMD 100 by making the gesture 11.

The gesture 12 will be described. The gesture 12 is a gesture in which the hand HD enters the detection area SA from the right side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the left side of the detection area SA. The input information indicating the input previously assigned to the gesture 12 is a “command to switch to the previous screen”. The user can input a “command to switch to the previous screen” to the HMD 100 by making the gesture 12.

Input information (“0”) to input information (“9”) are elements constituting a password. This element is not limited to numbers but may be alphabets or the like.

The combinations of the gestures 1 to 12 and the input information are not limited to the example shown in FIG. 10, and may be different combinations (for example, the input information (“1” may be assigned to the gesture 11)). The gestures that can be detected by the proximity sensor 105 are not limited to the gestures 1 to 12, and arbitrary input information may be assigned to gestures other than these gestures.

In the HMD 100 according to the present embodiment, an operation when a gesture input is performed will be described. FIG. 15 is a flowchart for explaining this operation. Referring to FIG. 5, the user operates operation unit 122 to turn on the power. The display control unit 104DR displays a password input screen on the image display unit 104B for password authentication. The password will be described by taking the number of digits as an example.

FIG. 16 is a waveform diagram showing changes in the output levels of the focus elements RA to RD when a series of gestures (gesture 2, gesture 1, gesture 4) is performed for password input. Referring to FIGS. 4, 5, and 16, the user inputs a password by making a gesture. For example, the user password is “214”. The user places a hand in front of the proximity sensor 105 and performs gesture 2 (FIG. 10) to which input information (“2”) is assigned. When the output level of any of the pyroelectric elements RA to RD first exceeds the threshold th, the mode control unit 1214 starts a reception mode for accepting a gesture input (step S1 in FIG. 15). In this case, the acceptance mode is started when the output levels of the pyroelectric elements RC and RD exceed the threshold value th. The mode control unit 1214 ends the reception mode after a predetermined period (for example, 3 seconds) has elapsed since the start of the reception mode. When the acceptance mode ends, the user cannot make a gesture input.

The gesture processing unit 1212 determines from which of the upper side, the lower side, the left side, and the right side the hand has entered the detection area SA (FIGS. 7A, 7B, and 8). Here, since the output level of the pyroelectric elements RC and RD exceeds the threshold th earlier than the output level of the pyroelectric elements RA and RB, it is determined to be lower. The storage control unit 1215 stores information indicating “lower side” in the storage unit 125 (step S2 in FIG. 15).

The gesture processing unit 1212 determines from which of the upper side, the lower side, the left side, and the right side the hand has exited the detection area SA. Here, since the output level of the pyroelectric elements RA and RB is equal to or lower than the threshold th after the output level of the pyroelectric elements RC and RD, it is determined as the upper side. The storage control unit 1215 stores information indicating “upper side” in the storage unit 125 (step S3 in FIG. 15).

The gesture processing unit 1212 determines the type of gesture using the results of step S2 and step S3 (step S4 in FIG. 15). The types of gestures are the 12 gestures described in FIG. The storage unit 125 stores in advance a table (hereinafter referred to as the table of FIG. 10) indicating the correspondence between the gestures 1 to 12 and the input information assigned to them.

The gesture processing unit 1212 reads out the determination results of step S2 and step S3 stored in the storage unit 125. Here, the determination result of step S2 is “lower side”, and the determination result of step S3 is “upper side”. Accordingly, since the hand enters the detection area SA from the lower side of the detection area SA and exits the detection area SA from the upper side of the detection area SA, the gesture processing unit 1212 determines that the gesture is 2. Note that the gesture processing unit 1212 performs error processing when it is determined that none of the gestures 1 to 12 corresponds. As a result, the display control unit 104DR causes the image display unit 104B to display a screen for prompting a correct gesture.

The storage control unit 1215 refers to the table in FIG. 10 and stores the input information assigned to the gesture determined in step S4 in the storage unit 125 (step S5 in FIG. 15). Here, the input information (“2”) assigned to gesture 2 is stored in storage unit 125.

The gesture processing unit 1212 determines whether or not a predetermined period has elapsed from the start of the reception mode (step S1 in FIG. 15) (step S6 in FIG. 15). If the predetermined period has not elapsed (No in step S6), the gesture processing unit 1212 determines whether any of the output levels of the pyroelectric elements RA to RD has exceeded the threshold value th ( Step S7 in FIG. That is, the gesture processing unit 1212 stands by until the next gesture is made. When the gesture processing unit 1212 determines that all the output levels of the pyroelectric elements RA to RD are equal to or lower than the threshold th (No in step S7), the gesture processing unit 1212 performs the process of step S6.

When the gesture processing unit 1212 determines that the output level of any one of the pyroelectric elements RA to RD exceeds the threshold th (Yes in step S7), the gesture processing unit 1212 performs the process of step S2. To do. Here, since the user performs the gesture 1 to which the input information (“1”) is assigned, the gesture processing unit 1212 performs the process of step S2.

The gesture processing unit 1212 determines from which of the upper side, the lower side, the left side, and the right side the hand has entered the detection area SA. Here, since the output level of the pyroelectric elements RA and RB exceeds the threshold th earlier than the output level of the pyroelectric elements RC and RD, it is determined to be the upper side. The storage control unit 1215 stores information indicating “upper side” in the storage unit 125 (step S2).

The gesture processing unit 1212 determines from which of the upper side, the lower side, the left side, and the right side the hand has exited the detection area SA. Here, since the output levels of the pyroelectric elements RC and RD are equal to or lower than the threshold th after the output levels of the pyroelectric elements RA and RB, it is determined that the output is lower. The storage control unit 1215 stores information indicating “lower side” in the storage unit 125 (step S3).

The gesture processing unit 1212 determines the type of gesture using the results of step S2 and step S3 (step S4). More specifically, the gesture processing unit 1212 reads the determination results of step S2 and step S3 stored in the storage unit 125. Here, the determination result of step S2 is “upper side”, and the determination result of step S3 is “lower side”. Accordingly, since the hand enters the detection area SA from the upper side of the detection area SA and exits the detection area SA from the lower side of the detection area SA, the gesture processing unit 1212 determines that the gesture is 1.

The storage control unit 1215 refers to the table in FIG. 10 and stores the input information assigned to the gesture determined in step S4 in the storage unit 125 (step S5). Here, the input information (“1”) assigned to gesture 1 is stored in storage unit 125.

Detailed description of the gesture 4 is omitted, but the input information (“4”) assigned to the gesture 4 is stored in the storage unit 125 in the same manner as the gesture 2 and the gesture 1. The storage control unit 1215 stores input information (“2”), input information (“1”), and input information (“4”) in the storage unit 125. As described above, the storage control unit 1215 first detects a series of gestures (gesture 2, gesture 1, gesture 4) executed a plurality of times by the detection unit 128 after the start of the reception mode (step S1 in FIG. 15). The storage unit 125 is controlled to store input information indicating an input previously assigned to the gesture in order from the gesture (gesture 2).

The mode control unit 1214 ends the reception mode after elapse of a predetermined period (for example, 3 seconds) after starting the reception mode. Therefore, the user must make a series of gestures including gesture 2, gesture 1, and gesture 4 within a predetermined period.

When a predetermined period has elapsed from the start of the reception mode (Yes in step S6), the processing unit 1213 stores each input information (of gesture 2) of a plurality of gestures constituting a series of gestures stored in the storage unit 125. A predetermined process is performed using the input information (“2”), the input information of the gesture 1 (“1”), and the input information of the gesture 4 (“4”)) (step S8 in FIG. 15). Since the password input screen is displayed on the image display unit 104B (that is, the password authentication mode), the processing unit 1213 performs password authentication using the input information stored in the storage unit 125 as a predetermined process. Here, the processing unit 1213 performs password authentication by using the input information (“2”), the input information (“1”), and the input information (“4”), assuming that the input password is “214”.

When the processing unit 1213 determines that the password authentication has failed, it performs error processing. As a result, the display control unit 104DR causes the image display unit 104B to display a screen indicating that password authentication has failed.

When the processing unit 1213 determines that the password authentication is successful, the display control unit 104DR switches the screen displayed on the image display unit 104B from the password input screen to the initial screen.

Explain about switching from the initial screen to the next screen. In the password input, a series of three gesture inputs are executed, but in the input for instructing switching of the screen, one gesture input is executed.

FIG. 17 is a waveform diagram showing changes in the output levels of the focus elements RA to RD when the gesture 11 is made to switch to the next screen. With reference to FIGS. 4, 5, and 17, when the user performs gesture 11 (FIG. 10), an instruction to switch to the next screen is input to HMD 100. More specifically, as in the case of password input, when the output level of any of the pyroelectric elements RA to RD first exceeds the threshold th, the mode control unit 1214 sets a reception mode for accepting gesture input. Start (step S1 in FIG. 15). In this case, the acceptance mode is started when the output levels of the pyroelectric elements RB and RD exceed the threshold value th. As in the case of password input, the mode control unit 1214 ends the reception mode after a predetermined period (for example, 3 seconds) has elapsed since the start of the reception mode.

As in the case of password input, the gesture processing unit 1212 determines from which of the upper side, the lower side, the left side, and the right side the hand has entered the detection area SA (FIGS. 7A, 7B, and 8). . Here, since the output level of the pyroelectric elements RB and RD exceeds the threshold th earlier than the output level of the pyroelectric elements RA and RC, it is determined as the left side. The storage control unit 1215 stores information indicating “left side” in the storage unit 125 (step S2 in FIG. 15).

As in the case of password input, the gesture processing unit 1212 determines from which of the upper side, the lower side, the left side, and the right side the hand has exited the detection area SA. Here, since the output levels of the pyroelectric elements RA and RC are equal to or lower than the threshold th after the output levels of the pyroelectric elements RB and RD, it is determined that the output is on the right side. The storage control unit 1215 stores information indicating “right side” in the storage unit 125 (step S3 in FIG. 15).

As in the case of password input, the gesture processing unit 1212 determines the type of gesture using the results of step S2 and step S3 (step S4 in FIG. 15). More specifically, the gesture processing unit 1212 reads the determination results of step S2 and step S3 stored in the storage unit 125. Here, the determination result of step S2 is “left side”, and the determination result of step S3 is “right side”. Therefore, since the hand enters the detection area SA from the left side of the detection area SA and exits the detection area SA from the right side of the detection area SA, the gesture processing unit 1212 determines that it is the gesture 11.

As in the case of password input, the storage control unit 1215 refers to the table in FIG. 10 and stores the input information assigned to the gesture determined in step S4 in the storage unit 125 (step S5 in FIG. 15). . Here, the input information (“command to switch to the next screen”) assigned to the gesture 11 is stored in the storage unit 125.

As in the case of password input, the gesture processing unit 1212 determines whether or not a predetermined period has elapsed since the start of the reception mode (step S1 in FIG. 15) (step S6 in FIG. 15). If the predetermined period has not elapsed (No in step S6), the gesture processing unit 1212 determines whether any of the output levels of the pyroelectric elements RA to RD has exceeded the threshold value th ( Step S7 in FIG. When the gesture processing unit 1212 determines that the output level of any of the pyroelectric elements RA to RD is equal to or lower than the threshold th (No in step S7), the gesture processing unit 1212 performs the process of step S6. Here, since only the gesture 11 is made, the gesture processing unit 1212 performs the process of step S6.

When a predetermined period has elapsed from the start of the acceptance mode (Yes in step S6), the processing unit 1213 performs a predetermined process (step S8 in FIG. 15). The processing unit 1213 generates a command for switching to the next screen as a predetermined process. With this command, the display control unit 104DR switches the screen displayed on the image display unit 104B from the initial screen to the next screen. Hereinafter, although detailed description is omitted, each time the gesture 11 is performed, the screen is switched to the next screen, and each time the gesture 12 is performed, the screen is switched to the previous screen.

The main effect of this embodiment will be described. Referring to FIGS. 5, 15, and 16, detection unit 128 does not detect a gesture using the screen displayed on image display unit 104 </ b> B, but detects detection area SA ( 7A, 7B, and 8), and two or more predetermined gestures are distinguished and detected. The processing unit 1213 performs password authentication (an example of a predetermined process) using input information (that is, three input information) of each of the three gestures constituting the series of gestures (step S8). For this reason, it is possible to perform a process that requires a plurality of inputs such as password authentication using gesture input without using a screen. Therefore, according to this embodiment, it is possible to improve gesture input without using a screen.

Referring to FIG. 5, in the present embodiment, two or more predetermined gestures are distinguished and detected by a detection unit 128 including a proximity sensor 105 and a gesture processing unit 1212. The detection unit 128 is not limited to this configuration. For example, the detection unit 128 may include a camera 106 (two-dimensional imaging device) and an image processing unit that performs predetermined image processing on an image captured by the camera 106 and recognizes a gesture.

About the modification of this embodiment, the point different from this embodiment is mainly demonstrated. As modifications, there are Modification 1 to Modification 5. The first modification can be input by an intuitive operation. For example, when a series of gestures is a hand movement indicating “bye-bye”, an input of “command to switch to initial screen” is made. The instruction is not limited to this, and may be, for example, “an instruction to cancel the previous input”.

The movement of the hand indicating “bye-bye” is performed by the gesture 11 shown in FIG. 10, then the gesture 12, and then the gesture 11, and vice versa. In some cases, gesture 11 is made, and then gesture 12 is made. The former will be described as an example.

FIG. 18 is a waveform diagram showing changes in the output levels of the focus elements RA to RD when a series of gestures (gesture 11, gesture 12, and gesture 11) indicating “bye-by” is performed. Referring to FIG. 5, FIG. 15, and FIG. 18, in the first modification, the processes in steps S1 to S7 are performed as in the case of password input. When a predetermined period has elapsed from the start of the reception mode (Yes in step S6), the processing unit 1213 performs a predetermined process (step S8). The processing unit 1213 generates a command for switching to the initial screen as a predetermined process. With this command, the display control unit 104DR switches the screen to be displayed on the image display unit 104B from the current screen to the initial screen.

Modification 2 will be described. As shown in FIG. 17, in this embodiment, the predetermined period (that is, the valid period of the reception mode) is fixed. The time required for a series of three gestures for password input is set as a predetermined period. For this reason, when the processing unit 1213 performs a predetermined process using input information obtained by one gesture input or two gesture inputs (for example, generation of a command for switching to the next screen), There is a waiting time before starting. Therefore, in the second modification, when the state where the start of the next gesture is not detected continues, the acceptance mode is terminated without waiting for the elapse of a predetermined period, and a predetermined process is performed.

FIG. 19 is a flowchart for explaining the operation when a gesture is input in the second modification of the present embodiment. FIG. 20 is a waveform diagram showing changes in the output levels of the focus elements RA to RD when the gesture 11 is made to switch to the next screen. The flowchart shown in FIG. 19 is different from the flowchart shown in FIG. 15 in that step S9 is added between steps S6 and S7.

Referring to FIGS. 5, 19, and 20, after detecting unit 128 detects gesture 11 within a predetermined period, gesture processing unit 1212 determines whether the predetermined period has elapsed since the start of the reception mode (step S <b> 1). It is determined whether or not (step S6). If the predetermined period has not elapsed (No in step S6), the gesture processing unit 1212 determines whether the non-detection period in which the start of the next gesture has not been detected has reached a predetermined value ( Step S9). The non-detection period is a period in which all output levels of the pyroelectric elements RA to RD are, for example, the threshold value th or less. A period in which all the output levels of the pyroelectric elements RA to RD are 0 may be a non-detection period. The predetermined value is a value smaller than a predetermined period (for example, 3 seconds) (for example, 0.5 seconds), and takes into account the time required for the start of the next gesture after the end of one gesture. Is set.

When the gesture processing unit 1212 determines that the non-detection period has not reached a predetermined value (No in step S9), the gesture processing unit 1212 performs the process of step S7. On the other hand, when the gesture processing unit 1212 determines that the non-detection period has reached a predetermined value (Yes in step S9), the mode control unit 1214 ends the reception mode. Thereby, the processing unit 1213 performs a predetermined process without waiting for the elapse of the predetermined period (step S8). Here, since the gesture 11 is performed, the storage unit 125 stores the input information (“command to switch to the next screen”) illustrated in FIG. 10. The processing unit 1213 generates a command for switching to the next screen as a predetermined process. Thereby, the display control unit 104DR switches the screen displayed on the image display unit 104B to the next screen.

As described above, according to the second modification, the waiting time until the start of the predetermined process after the end of the gesture can be shortened.

Modification 3 will be described. As shown in FIG. 16, in the present embodiment, the length of the predetermined period is fixed, but the third modification can extend the predetermined period.

As a series of gestures, there are a first series gesture composed of a first number of gestures and a second series gesture composed of a second number of gestures greater than the first number. In password authentication, the number of passwords for a user is a first number (for example, 3), and the number of special passwords (for example, a password for accessing a screen dedicated to a serviceman) is a second number (for example, 4). The user inputs a password by making a first series of gestures (that is, a series of gestures composed of three gestures). The service person inputs a password by making a second series of gestures (that is, a series of gestures composed of four gestures).

Modification 3 assumes that the initial predetermined period (that is, the initial value of the predetermined period) is longer than the time required for the first series of gestures, but shorter than the time required for the second series of gestures. This makes it difficult for the user to access the screen dedicated to the service person.

The first part of the second series of gestures is composed of a third number of gestures equal to or less than the first number. In Modification 3, a command for extending a predetermined period is assigned in advance to the input information of the first part. The first part of the second series of gestures is, for example, two gestures configured by the first and second gestures.

Suppose that the password for accessing the screen dedicated to the serviceman is a four-digit password including, for example, “00” at the beginning (for example, 0012). A command for extending a predetermined period is assigned to the input information (“00”). By repeating gesture 10 shown in FIG. 10 twice, when “00” is input, the predetermined period is extended, and a time for inputting the remaining number “12” is secured.

Specified period will be extended for input of 4-digit password including 00. For this reason, when there are a plurality of screens dedicated to the serviceman, a four-digit different password including 00 can be assigned to each screen (that is, a dedicated password can be given to each screen).

The input information to which a command for extending a predetermined period is assigned is not limited to one, and may be plural. For example, a command for extending a predetermined period may be assigned to each of the input information (“00”) and the input information “99”.

FIG. 21 is a flowchart for explaining the operation when a gesture is input in the third modification of the present embodiment. The flowchart shown in FIG. 21 is different from the flowchart shown in FIG. 19 in that steps S10 and S11 are added between steps S5 and S6.

Referring to FIGS. 5 and 21, after step S5, processing unit 1213 determines whether or not the predetermined process based on the input information stored in storage unit 125 generates an extension command for a predetermined period. Is determined (step S10). When input information other than “00” is stored in the storage unit 125, the processing unit 1213 determines that the predetermined process is not a process for generating an extension command for a predetermined period (No in step S10), and the gesture processing unit 1212 performs the process of step S6.

When the input information of “00” is stored in the storage unit 125, the processing unit 1213 determines that the predetermined process is a process of generating an extension command for a predetermined period (Yes in step S10), and the processing unit 1213 Generate an extension command for a predetermined period. Thereby, the mode control part 1214 extends a predetermined period (step S11). Then, the gesture processing unit 1212 performs step S7.

Modification 4 will be described. In the third modification, the input information assigned to the first part of the second series of gestures is a command for extending the predetermined period. On the other hand, in the fourth modification, input information assigned to one or more predetermined gestures is used as a command for extending a predetermined period.

When there are a large number of gestures in a series or when the user has slowed down the gestures, the series of gestures cannot be completed within the predetermined period even if the predetermined period is relatively long (for example, 5 seconds). Therefore, when the user makes one or more gestures as a command for extending the predetermined period during the series of gestures, the processing unit 1213 extends the predetermined period.

More specifically, for example, the password is “924845”. “00” is a command for extending the predetermined period. The command for extending the predetermined period is not limited to a plurality of digits, and may be a single digit. In this case, the predetermined period is extended by one gesture.

The user performs the gesture 9 to which the input information (“9”) is assigned, the gesture 2 to which the input information (“2”) is assigned, and the gesture 4 to which the input information (“4”) is assigned. The gesture 10 assigned (“0”) is repeated twice. As a result, the processing unit 1213 extends the predetermined period (for example, extends 5 seconds from the current time). The user performs the gesture 8 to which the input information (“8”) is assigned, the gesture 4 to which the input information (“4”) is assigned, and the gesture 5 to which the input information (“5”) is assigned.

The processing unit 1213 includes input information (“9”, “2”, “4”, “0”, “0”, “8”, “4”, “5”) stored in the storage unit 125. The processing unit 1213 performs processing excluding “0” and “0.” The processing unit 1213 uses the remaining input information ((“9”, “2”, “4”, “8”, “4”, “5”)) as a password. As a password authentication.

Modification 5 will be described. The user must finish a series of gestures within a predetermined period (eg, 3 seconds). Modification 5 shows the remainder of the predetermined period on the image display unit 104B (FIG. 5). Thereby, the user can adjust the speed of the gesture so that the time required for a series of gestures falls within a predetermined period.

Modification 5 displays information indicating the remainder of the predetermined period on the image display unit 104B illustrated in FIG. 5 during the predetermined period. FIG. 22 is a screen diagram illustrating a first example of a screen displayed on the image display unit 104B in Modification 5 of the present embodiment. FIG. 23 is a screen diagram illustrating a second example of a screen displayed on the image display unit 104B in Modification 5 of the present embodiment.

Referring to FIG. 5 and FIG. 22, when mode control unit 1214 starts a reception mode for accepting a gesture input, display control unit 104DR causes image display unit 104B to display screen SC1. The screen SC1 includes a number indicating the remaining time (= predetermined period−the time elapsed since the start of the reception mode) and characters indicating that a gesture input is received. The remaining time is information indicating the remaining of the predetermined period.

Referring to FIG. 5 and FIG. 23, when the predetermined period is extended in the modified example 3, the display control unit 104DR displays the screen SC2 on the image display unit 104B. Screen SC2 includes the remaining time and characters indicating that the predetermined period has been extended.

According to the modified example 5, the user of the HMD 100 can adjust the speed of the gesture so that the time required for a series of gestures falls within a predetermined period.

The display control unit 104DR may cause the image display unit 104B to display information indicating the time elapsed since the start of the reception mode instead of the information indicating the remainder of the predetermined period, or display both of them on the image display unit 104B. You may let them. The time elapsed from the start of the reception mode is the time elapsed from the start of the predetermined period.

Although the information indicating the remainder of the predetermined period is expressed in numbers, it is not limited to this. For example, as illustrated in FIG. 24, the display control unit 104DR causes the image display unit 104B to display a screen SC3 including a white region and a gray region. The white area indicates the time that has elapsed since the start of the reception mode, and the display control unit 104DR increases the area of the white area as the time increases. The gray area indicates the remaining time of the predetermined period, and the display control unit 104DR decreases the area of the gray area as the time becomes shorter. The screen SC3 shown in FIG. 24 includes both information (gray area) indicating the remainder of the predetermined period and information (white area) indicating the time elapsed since the start of the reception mode.

(Summary of embodiment)
The display device according to the embodiment has a display unit, a detection unit having a detection region at a position different from the display unit, capable of distinguishing and detecting two or more predetermined gestures, and a reception mode for receiving a gesture input And a gesture that is first detected by the detection unit with respect to a mode control unit that terminates the reception mode after a predetermined period, a storage unit, and a series of gestures that are executed a plurality of times after the reception mode is started. In order from the above, a storage control unit for controlling the storage unit to store input information indicating an input pre-assigned to the gesture, and the storage unit stored in the storage unit after completion of the reception mode, A processing unit that performs predetermined processing using the input information of each of a plurality of gestures constituting a series of gestures.

The detection unit does not detect a gesture using the screen displayed on the display unit, but has a detection area at a position different from the display unit, and detects two or more predetermined gestures separately. The display device according to the embodiment performs a predetermined process using input information (that is, a plurality of input information) of a plurality of gestures constituting a series of gestures. For this reason, a process (predetermined process) that requires a plurality of inputs such as password authentication can be performed using gesture input without using a screen. Therefore, according to the display device according to the embodiment, gesture input without using a screen can be improved.

The detection unit includes, for example, a plurality of pyroelectric elements arranged in a two-dimensional matrix, and a gesture processing unit that determines a gesture based on each output of the plurality of pyroelectric elements. The display device is, for example, a wearable terminal that can be worn on the head or arm. A wearable terminal is a terminal device that can be worn on a part of a body (for example, a head or an arm).

In the above configuration, the input information of each of the two or more gestures indicates an element constituting a password, and the processing unit performs password authentication as the predetermined processing.

The elements constituting the password are, for example, numbers and alphabets. According to this configuration, password authentication can be performed by gesture input without using a screen.

In the above configuration, when the detection unit determines that a non-detection period in which the start of the next gesture has not been detected has reached a predetermined value after detecting a gesture within the predetermined period, the processing unit Performs the predetermined process using the input information stored in the storage unit without waiting for the predetermined period to elapse.

The predetermined value is a value smaller than a predetermined period (for example, 3 seconds) (for example, 0.5 seconds). The time required for a series of gestures is set as a predetermined period. For this reason, for example, when the processing unit performs a predetermined process using input information from one gesture input (for example, generation of a command to switch to the next screen), the process waits until the predetermined process starts after the gesture ends. Time occurs. According to this configuration, when the non-detection period in which the start of the next gesture has not been detected reaches a predetermined value, the reception mode is terminated and predetermined processing is performed without waiting for the predetermined period to elapse. . For this reason, waiting time can be shortened.

In the above configuration, a command for extending the predetermined period is assigned in advance to the input information assigned to one or more predetermined gestures, and the one or more commands are stored in the storage unit during the predetermined period. When the input information of the gesture is stored, the mode control unit extends the predetermined period.

According to this configuration, the predetermined period can be extended during the predetermined period. Therefore, even when the number of gestures is large or when the user has slowed down the gestures, the series of gestures can be completed within a predetermined period.

In the above configuration, as the series of gestures, a first series of gestures configured by a first number of gestures, and a second series of gestures configured by a second number of gestures greater than the first number, And the first part of the second series of gestures is composed of a third number of gestures equal to or less than the first number, and the command is preliminarily applied to the input information of the first part. When the input information of the first part is stored in the storage unit, the mode control unit extends the predetermined period.

For example, in password authentication, the number of passwords for users is the first number, and the number of special passwords (for example, passwords for accessing screens dedicated to service personnel) is the second number. The user inputs a password by making a first series of gestures. The service person makes a second series of gestures and inputs a password. This configuration assumes that the initial predetermined period (that is, the initial value of the predetermined period) is longer than the time required for the first series of gestures but shorter than the time required for the second series of gestures. This makes it difficult for the user to access the screen dedicated to the service person. In this configuration, a command for extending a predetermined period is assigned in advance to the input information of the first part of the second series of gestures. When the second series of gestures is started and the input information of the first part is stored in the storage unit, the mode control unit extends the predetermined period. This ensures time for performing the remaining part of the second series of gestures. Therefore, according to this configuration, the processing unit uses the input information of each of the plurality of gestures constituting the second series of gestures stored in the storage unit to perform predetermined processing (for example, dedicated to a serviceman To access the screen).

In the above configuration, a display control unit that causes the display unit to display at least one of information indicating the rest of the predetermined period and information indicating a time elapsed since the start of the reception mode during the predetermined period. Further prepare.

For example, assume that the predetermined period is 3 seconds and 1.4 seconds have elapsed since the start of the reception mode. The remainder of the predetermined period is 1.6 seconds. The time elapsed from the start of the reception mode (that is, the start of the predetermined period) is 1.4 seconds. According to this configuration, the user of the display device can adjust the speed of the gesture so that the time required for a series of gestures falls within a predetermined period.

The gesture input method according to the embodiment is directed to a display device including a display unit and a detection unit that has a detection region at a position different from the display unit and can distinguish and detect two or more predetermined gestures. A method for inputting a gesture, the first step of starting a reception mode for receiving a gesture input, and ending the reception mode after a predetermined period of time, and a series of steps executed a plurality of times after the start of the reception mode A second step of controlling the storage unit to store input information indicating an input pre-assigned to the gesture in order from the gesture first detected by the detection unit, and the reception mode. After completion of the operation, it is stored in the storage unit, and the plurality of gestures constituting the series of gestures Using the input information Les, and a third step of the predetermined processing.

The gesture input method according to the embodiment defines the display device according to the embodiment from the viewpoint of the method, and has the same effects as the display device according to the embodiment.

This application is based on Japanese Patent Application No. 2016-123744 filed on June 22, 2016, the contents of which are included in the present application.

In order to express the present invention, the present invention has been properly and fully described through the embodiments with reference to the drawings. However, those skilled in the art can easily change and / or improve the above-described embodiments. It should be recognized that this is possible. Therefore, unless the modifications or improvements implemented by those skilled in the art are at a level that departs from the scope of the claims recited in the claims, the modifications or improvements are not covered by the claims. To be construed as inclusive.

According to the present invention, a display device and a gesture input method can be provided.

Claims (9)

A display unit;
A detection unit having a detection region at a different position from the display unit, and capable of distinguishing and detecting two or more predetermined gestures;
A mode control unit that starts a reception mode for receiving a gesture input, and ends the reception mode after a predetermined period;
A storage unit;
After a start of the acceptance mode, for a series of gestures executed a plurality of times, input information indicating inputs previously assigned to the gestures in order from the gesture first detected by the detection unit is stored in the storage unit. A storage control unit for controlling the storage;
A display device comprising: a processing unit that performs predetermined processing using the input information of each of a plurality of gestures constituting the series of gestures stored in the storage unit after the acceptance mode ends. The input information of each of the two or more gestures indicates an element constituting a password,
The display device according to claim 1, wherein the processing unit performs password authentication as the predetermined processing. When the detection unit determines that a non-detection period in which the start of the next gesture has not been detected has reached a predetermined value after detecting a gesture within the predetermined period, the processing unit The display device according to claim 1, wherein the predetermined process is performed using the input information stored in the storage unit without waiting for the elapse of a period. A command for extending the predetermined period is pre-assigned to the input information assigned to one or more predetermined gestures, and the storage unit stores the command of the one or more gestures during the predetermined period. The display device according to any one of claims 1 to 3, wherein when the input information is stored, the mode control unit extends the predetermined period. As the series of gestures, there are a first series gesture composed of a first number of gestures and a second series gesture composed of a second number of gestures larger than the first number,
The first part of the second series of gestures is composed of a third number of gestures equal to or less than the first number, and the command is pre-assigned to the input information of the first part. And
The display device according to claim 4, wherein the mode control unit extends the predetermined period when the input information of the first part is stored in the storage unit. The display control part which displays on the display part at least one among the information which shows the remaining of the predetermined period during the predetermined period, and the information which has passed since the start of the acceptance mode. The display device according to any one of 1 to 5. The detection unit includes a plurality of pyroelectric elements arranged in a two-dimensional matrix, and a gesture processing unit that determines a gesture based on each output of the plurality of pyroelectric elements. A display device according to claim 1. The display device according to any one of claims 1 to 7, wherein the display device is a wearable terminal. A method for inputting gestures to a display device comprising: a display unit; and a detection unit having a detection region at a position different from the display unit and capable of distinguishing and detecting two or more predetermined gestures,
A first step of starting a reception mode for receiving a gesture input and ending the reception mode after a predetermined period;
For a series of gestures executed a plurality of times after the start of the acceptance mode, input information indicating inputs pre-assigned to the gestures is stored in the storage unit in order from the gesture first detected by the detection unit. A second step of controlling
A gesture comprising: a third step of performing a predetermined process using the input information of each of a plurality of gestures constituting the series of gestures stored in the storage unit after the acceptance mode ends. input method.

Images