WO2018092674A1 - Display apparatus and gesture input method - Google Patents

Abstract

This display apparatus has: a display unit; a detection unit; a mode control unit; and a processing unit. The detection unit has a detection region at a positon different from that of the display unit, and can discriminate and detect two or more predetermined gestures. The mode control unit executes a reception mode for receiving gesture inputs for a prescribed period of time. The processing unit executes, after the reception mode has ended, a prescribed process by using input information allocated to a last detected gesture among one or more of the gestures detected by the detection unit during the period of the reception mode.

Description

Display device and gesture input method

The present invention relates to a display device capable of gesture input and a gesture input method.

Gesture input is to operate a display device (for example, a terminal device or a game machine) by gesture or hand gesture. For example, in the case of a smartphone, gesture input can be performed by touching the screen.

∙ You may not want to touch the screen when the user's hand is dirty. Therefore, gesture input without using a screen has been proposed. For example, Patent Literature 1 discloses a display device that includes a display, a motion reception unit, and a display control unit that controls display information displayed on the display according to the motion received by the motion reception unit. In gesture input without using a screen, a detection unit (hereinafter referred to as a detection unit) that detects a gesture is provided at a position different from the display unit on which the screen is displayed. The motion reception unit is a detection unit.

⿟ Wrong gesture input. That is, a gesture that is not intended by the user may be detected by the detection unit. For example, the movement of the hand that the user moves unconsciously may be detected by the detection unit. In particular, in the case of a head-mounted display, since the detection unit is not visible to the user, an erroneous gesture input is likely to occur.

Suppose that gesture input is possible during the reception mode period for accepting gesture input, and that gesture input is not possible during periods other than this period. If an incorrect gesture is input during a certain reception mode, the gesture cannot be input unless the next reception mode is waited. This is stressful for the user. In other words, when a wrong gesture is input, if a waiting time occurs to re-input the gesture, it is stressful for the user.

JP 2010-1229069 A

An object of the present invention is to provide a display device and a gesture input method capable of performing gesture input without waiting for the next reception mode period even if an incorrect gesture input is performed during the reception mode period for receiving gesture input. Is to provide.

In order to realize the above-described object, a display device reflecting one aspect of the present invention includes a display unit, a detection unit, a mode control unit, and a processing unit. The detection unit has a detection region at a position different from that of the display unit, and can detect and distinguish two or more predetermined gestures. The mode control unit executes a reception mode for receiving a gesture input for a predetermined period. The processing unit has input information pre-assigned to the last detected gesture among the one or more gestures detected by the detection unit during the reception mode after the reception mode ends. Is used to perform predetermined processing.

The advantages and features afforded by one or more embodiments of the invention will be more fully understood from the detailed description and accompanying drawings provided below. The detailed description and the accompanying drawings are given by way of example only and are not intended as a definition of the limitations of the invention.

It is a perspective view which shows the structural structure of HMD which concerns on this embodiment. It is a front view which shows the structural structure of HMD which concerns on this embodiment. It is a schematic sectional drawing which shows the structure of the display unit with which HMD which concerns on this embodiment is equipped. It is a figure which shows the structure of the proximity sensor with which HMD which concerns on this embodiment is equipped. It is a block diagram which shows the electrical structure of HMD which concerns on this embodiment. It is a front view at the time of mounting HMD concerning this embodiment. It is a side view at the time of mounting HMD concerning this embodiment. It is a partial top view at the time of mounting | wearing with HMD which concerns on this embodiment. It is a figure which shows an example of the image which a user visually recognizes through a see-through type image display part. It is a figure which shows an example of the output of the proximity sensor with which HMD which concerns on this embodiment is equipped. In this embodiment, it is explanatory drawing explaining the relationship between a gesture and input information. FIG. 10 is a waveform diagram showing changes in output levels of focus elements RA to RD when gesture 1 is made. FIG. 9 is a waveform diagram showing changes in the output levels of focus elements RA to RD when gesture 2 is made. It is a flowchart explaining the operation | movement when the gesture input is made in the 1st aspect of HMD which concerns on this embodiment. FIG. 10 is a waveform diagram showing changes in the output levels of focus elements RA to RD provided in the first mode when a gesture input is made to switch the screen to the next screen. It is explanatory drawing explaining an example of the screen containing the image which shows whether it is reception mode, and the image which shows reception time. In the 2nd mode of HMD concerning this embodiment, it is a flow chart explaining operation at the time of gesture input. FIG. 12 is a waveform diagram showing changes in the output levels of focus elements RA to RD provided in the second mode when a gesture input is made to switch the screen to the next screen. It is explanatory drawing explaining an example of the screen containing the image which shows whether it is reception mode, and the image which shows the time regarding a non-detection state. In the 3rd mode of HMD concerning this embodiment, it is a flow chart explaining operation when gesture input is performed. FIG. 10 is a waveform diagram showing changes in the output levels of focus elements RA to RD provided in the third mode when a gesture input is made to switch the screen to the next screen. It is explanatory drawing explaining an example of the screen containing the image which shows whether it is reception mode, the image which shows reception time, and the image which shows the time regarding a non-detection state.

Hereinafter, one or more embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the disclosed embodiments. Configurations denoted by the same reference numerals in the drawings indicate the same configuration, and description thereof will be omitted as appropriate.

The display device according to the present embodiment is, for example, a wearable terminal (head mounted display (HMD), wristwatch type terminal, etc.), or a smart terminal (smart phone, tablet terminal, etc.). In this specification, a head mounted display (HMD) will be described as an example.

FIG. 1 is a perspective view showing a structural configuration of the HMD 100 according to the present embodiment. FIG. 2 is a front view showing a structural configuration of the HMD 100 according to the present embodiment. FIG. 3 is a schematic cross-sectional view showing the configuration of the display unit 104 provided in the HMD 100 according to the present embodiment. FIG. 4 is a diagram illustrating a configuration of the proximity sensor 105 provided in the HMD 100 according to the present embodiment. FIG. 5 is a block diagram showing an electrical configuration of the HMD 100 according to the present embodiment. Hereinafter, the right side and the left side of the HMD 100 refer to the right side and the left side for the user wearing the HMD 100.

The structural configuration of the HMD 100 will be described. With reference to FIG.1 and FIG.2, HMD100 which concerns on this embodiment is provided with the flame | frame 101 which is an example of the head mounting member for mounting | wearing a head. The frame 101 includes a front part 101a to which two spectacle lenses 102 are attached, and side parts 101b and 101c extending rearward from both ends of the front part 101a. The two spectacle lenses 102 attached to the frame 101 may or may not have refractive power (optical power, reciprocal of focal length).

The cylindrical main body 103 is fixed to the front part 101 a of the frame 101 at the upper part of the right eyeglass lens 102 (which may be the left side according to the user's dominant eye etc.). The main body 103 is provided with a display unit 104. In the main body 103, a display control unit 104DR (FIG. 5) that controls display of the display unit 104 based on an instruction from a control processing unit 121 (FIG. 5) described later is arranged. Note that a display unit may be disposed in front of both eyes as necessary.

Referring to FIG. 3, the display unit 104 includes an image forming unit 104A and an image display unit 104B. The image forming unit 104A is incorporated in the main body unit 103, and includes a light source 104a, a one-way diffusing plate 104b, a condenser lens 104c, and a display element 104d. On the other hand, the image display unit 104B, which is a so-called see-through type display member, is generally plate-shaped and is disposed so as to extend downward from the main body unit 103 and parallel to one eyeglass lens 102 (FIG. 1). The eyepiece prism 104f, the deflecting prism 104g, and the hologram optical element 104h are included.

The light source 104a has a function of illuminating the display element 104d. For example, the peak wavelength of light intensity and the half width of the light intensity are 462 ± 12 nm (blue light (B light)), 525 ± 17 nm (green light (G light )), 635 ± 11 nm (red light (R light)), and is composed of RGB integrated light emitting diodes (LEDs) that emit light in three wavelength bands.

The display element 104d displays an image by modulating the light emitted from the light source 104a in accordance with image data, and is configured by a transmissive liquid crystal display element having pixels that serve as light transmitting regions in a matrix. Is done. Note that the display element 104d may be of a reflective type.

The eyepiece prism 104f totally reflects the image light from the display element 104d incident through the base end face PL1 by the opposed parallel inner side face PL2 and outer side face PL3, and passes through the hologram optical element 104h to the user's pupil. On the other hand, external light is transmitted and guided to the user's pupil. The eyepiece prism 104f is formed of, for example, an acrylic resin together with the deflecting prism 104g. The eyepiece prism 104f and the deflection prism 104g are joined by an adhesive with the hologram optical element 104h sandwiched between inclined surfaces PL4 and PL5 inclined with respect to the inner surface PL2 and the outer surface PL3.

The deflection prism 104g is joined to the eyepiece prism 104f, and becomes a substantially parallel flat plate integrated with the eyepiece prism 104f. In addition, if the spectacle lens 102 (FIG. 1) is mounted between the display unit 104 and the user's pupil, it is possible for a user who normally uses spectacles to observe an image.

The hologram optical element 104h diffracts and reflects the image light (light having a wavelength corresponding to the three primary colors) emitted from the display element 104d, guides it to the pupil B, enlarges the image displayed on the display element 104d, and enlarges the user's pupil. It is a volume phase type reflection hologram guided as a virtual image. The hologram optical element 104h has, for example, three wavelength ranges of 465 ± 5 nm (B light), 521 ± 5 nm (G light), and 634 ± 5 nm (R light) with a peak wavelength of diffraction efficiency and a wavelength width of half the diffraction efficiency. It is made to diffract (reflect) light. Here, the peak wavelength of diffraction efficiency is the wavelength at which the diffraction efficiency reaches a peak, and the wavelength width at half maximum of the diffraction efficiency is the wavelength width at which the diffraction efficiency is at half maximum of the diffraction efficiency peak. is there.

In the display unit 104 having such a configuration, light emitted from the light source 104a is diffused by the unidirectional diffusion plate 104b, condensed by the condenser lens 104c, and incident on the display element 104d. The light incident on the display element 104d is modulated for each pixel based on the image data input from the display control unit 104DR, and is emitted as image light. Thereby, a color image is displayed on the display element 104d. The image light from the display element 104d enters the eyepiece prism 104f from its base end face PL1, is totally reflected a plurality of times by the inner side face PL2 and the outer side face PL3, and enters the hologram optical element 104h. The light incident on the hologram optical element 104h is reflected there, passes through the inner side surface PL2, and reaches the pupil B. At the position of the pupil B, the user can observe an enlarged virtual image of the image displayed on the display element 104d, and can visually recognize it as a screen formed on the image display unit 104B.

On the other hand, since the eyepiece prism 104f, the deflecting prism 104g, and the hologram optical element 104h transmit almost all of the external light, the user can observe the external image (real image) through these. Therefore, the virtual image of the image displayed on the display element 104d is observed so as to overlap with a part of the external image. In this way, the user of the HMD 100 can simultaneously observe the image provided from the display element 104d and the external image via the hologram optical element 104h. In addition, when the display unit 104 is in a non-display state, the image display unit 104B is transparent and can observe only an external image. In this embodiment, the display unit is configured by combining a light source, a liquid crystal display element, and an optical system. However, instead of the combination of the light source and the liquid crystal display element, a self-luminous display element (for example, an organic EL display) is used. Element) may be used. Further, instead of a combination of a light source, a liquid crystal display element, and an optical system, a transmissive organic EL display panel having transparency in a non-light emitting state may be used.

With reference to FIGS. 1 and 2, on the front surface of the main body 103, a proximity sensor 105 disposed near the center of the frame 101 and a lens 106a of the camera 106 disposed near the side 101b are disposed forward. It is provided so as to face.

In the present specification, the “proximity sensor” means a proximity range in front of the detection surface of the proximity sensor in order to detect that an object, for example, a part of a human body (hand, finger, etc.) is in front of the user's eyes. The signal is output by detecting whether or not it exists within the detection area. The proximity range may be set as appropriate according to the characteristics and preferences of the user. For example, the proximity range from the detection surface of the proximity sensor can be within a range of 200 mm. If the distance from the proximity sensor is within 200 mm, the user can put his / her palm or finger into / out of the user's field of view with his / her arm bent, so that the hand, finger or pointing tool (for example, a rod-shaped member) It is possible to easily perform an operation by a gesture using the symbol, and the possibility of erroneous detection of a human body or furniture other than the user is reduced.

There are two types of proximity sensors: passive type and active type. A passive proximity sensor has a detection device that detects invisible light or electromagnetic waves emitted from an object when the object approaches. As a passive proximity sensor, there are a pyroelectric sensor that detects invisible light such as infrared rays emitted from an approaching human body, and a capacitance sensor that detects a change in electrostatic capacitance between the approaching human body and the like. The active proximity sensor includes a projection device that projects invisible light (or sound waves), and a detection device that receives invisible light (or sound waves) reflected back from an object. As an active proximity sensor, an infrared sensor that projects infrared rays and receives infrared rays reflected by an object, a laser sensor that projects laser beams and receives laser light reflected by an object, and projects ultrasonic waves There is an ultrasonic sensor that receives ultrasonic waves reflected by an object. Note that a passive proximity sensor does not need to project energy toward an object, and thus has excellent low power consumption. An active proximity sensor is easy to improve the certainty of detection. For example, even when a user wears a glove that does not transmit detection light emitted from a human body such as infrared light, Can detect movement. A plurality of types of proximity sensors may be combined.

Referring to FIG. 4, in this embodiment, a pyroelectric sensor including a plurality of pyroelectric elements arranged in a two-dimensional matrix is used as the proximity sensor 105. The “right side” and “left side” in FIG. 4 refer to the right side and the left side for the user wearing the HMD 100. The proximity sensor 105 includes four pyroelectric elements RA, RB, RC, and RD arranged in two rows and two columns, and receives invisible light such as infrared light emitted from the human body as detection light. A corresponding signal is output from each of the pyroelectric elements RA to RD. The outputs of the pyroelectric elements RA to RD change in intensity according to the distance from the light receiving surface of the proximity sensor 105 to the object, and the intensity increases as the distance decreases.

Referring to FIGS. 1 and 2, the right sub-body portion 108-R is attached to the right side portion 101b of the frame 101, and the left sub-body portion 108-L is attached to the left side portion 101c of the frame 101. Is attached. The right sub body 108-R and the left sub body 108-L have an elongated plate shape.

The main main body 103 and the right sub main body 108-R are connected to each other by a wiring HS so that signals can be transmitted. The right sub main body 108-R is connected to the control unit CTU via a cord CD extending from the rear end thereof. It is connected to the.

Next, the electrical configuration of the HMD 100 will be described. Referring to FIG. 5, the HMD 100 includes a control unit CTU, a display unit 104, a display control unit 104DR, a proximity sensor 105, and a camera 106. The control unit CTU includes a control processing unit 121, an operation unit 122, a storage unit 125, a battery 126, and a power supply circuit 127.

The display control unit 104DR is a circuit that is connected to the control processing unit 121 and controls the image forming unit 104A of the display unit 104 according to the control of the control processing unit 121 to form an image on the image forming unit 104A. The image forming unit 104A is as described above.

The camera 106 is an apparatus that is connected to the control processing unit 121 and generates an image of a subject under the control of the control processing unit 121. The camera 106 is, for example, an image forming optical system that forms an optical image of a subject on a predetermined image forming surface, and a light receiving surface that matches the image forming surface. An image sensor that converts the image sensor into a digital signal processor (DSP) that performs known image processing on the output of the image sensor to generate an image (image data). The imaging optical system includes one or more lenses, and includes the lens 106a as one of them. The camera 106 outputs the generated image data to the control processing unit 121.

The proximity sensor 105 is connected to the control processing unit 121. The proximity sensor 105 is as described above, and outputs the output to the control processing unit 121.

The operation unit 122 is connected to the control processing unit 121 and is a device that inputs a predetermined instruction, such as power on / off, to the HMD 100, for example, one or a plurality of switches assigned a predetermined function Etc.

The battery 126 is a battery that accumulates electric power and supplies the electric power. The battery 126 may be a primary battery or a secondary battery. The power supply circuit 127 is a circuit that supplies power supplied from the battery 126 to each part of the HMD 100 that requires power at a voltage corresponding to each part.

The storage unit 125 is a circuit that is connected to the control processing unit 121 and stores various predetermined programs and various predetermined data under the control of the control processing unit 121. Examples of the various predetermined programs include control processing programs such as a control program for controlling each unit of the HMD 100 according to the function of each unit, and a gesture processing program for determining a gesture based on the output of the proximity sensor 105. Is included. The storage unit 125 includes, for example, a ROM (Read Only Memory) that is a nonvolatile storage element and an EEPROM (Electrically Erasable Programmable Read Only Memory) that is a rewritable nonvolatile storage element. The storage unit 125 includes a RAM (Random Access Memory) that serves as a working memory of the control processing unit 121 that stores data generated during the execution of the predetermined program.

The control processing unit 121 controls each unit of the HMD 100 according to the function of each unit, determines a predetermined gesture set in advance based on the output of the proximity sensor 105, and executes processing according to the determination result. Is. The control processing unit 121 includes, for example, a CPU (Central Processing Unit) and its peripheral circuits. In the control processing unit 121, a control processing program is executed, so that a control unit 1211, a gesture processing unit 1212, and a processing unit 1213 are functionally configured. Note that part or all of the functions of the control unit 1211 may be realized by processing by a DSP (Digital Signal Processor) instead of or by processing by the CPU. In addition, some or all of the functions of the control unit 1211 may be realized by processing using a dedicated hardware circuit instead of or together with processing by software. What has been described above also applies to the gesture processing unit 1212 and the processing unit 1213.

The control unit 1211 controls each unit of the HMD 100 according to the function of each unit. The control unit 1211 has functions of a mode control unit 1214 and a storage control unit 1215. These functions will be described later.

The gesture processing unit 1212 determines a predetermined gesture set in advance based on the outputs of the plurality of pyroelectric elements in the proximity sensor 105, in this embodiment, the four pyroelectric elements RA to RD. The gesture processing unit 1212 notifies the processing unit 1213 of the determination result. The gesture processing unit 1212 and the proximity sensor 105 constitute a detection unit 128. The detection unit 128 has a detection area SA (FIGS. 7A, 7B, and 8) at a position different from that of the image display unit 104B (an example of the display unit), and distinguishes and detects two or more predetermined gestures.

The processing unit 1213 performs a predetermined process (for example, sends a command to switch the screen to the next screen to the display control unit 104DR) using the determination result of the gesture processing unit 1212. Details of the processing unit 1213 will be described later.

The basic operation for detecting a gesture in the HMD 100 will be described. FIG. 6 is a front view when the HMD 100 according to the present embodiment is mounted. FIG. 7A is a side view when the HMD 100 according to the present embodiment is mounted. FIG. 7B is a partial top view when the HMD 100 according to the present embodiment is mounted. 7A and 7B also show the hand HD of the user US. FIG. 8 is a diagram illustrating an example of an image visually recognized by the user through the see-through type image display unit 104B. FIG. 9 is a diagram illustrating an example of the output of the proximity sensor 105 provided in the HMD 100 according to the present embodiment. 9A shows the output of the pyroelectric element RA, FIG. 9B shows the output of the pyroelectric element RB, FIG. 9C shows the output of the pyroelectric element RC, and FIG. (D) shows the output of the pyroelectric element RD. The horizontal axis of each figure in FIG. 9 is time, and the vertical axis thereof is the output level (intensity). Here, the gesture input is an operation in which at least the hand HD or finger of the user US enters or leaves the detection area SA of the proximity sensor 105, and the gesture processing of the control processing unit 121 of the HMD 100 is performed via the proximity sensor 105. The part 1212 can detect it.

Referring to FIG. 8, screen 104i of image display unit 104B is arranged so as to overlap with effective visual field EV of the user's eye facing image display unit 104B (here, positioned within effective visual field EV). The The detection area SA of the proximity sensor 105 is in the visual field of the user's eye facing the image display unit 104B. Preferably, the detection area SA is in the stable eye-field of the user's eye or in the field of view inside thereof (within about 90 ° horizontally and within about 70 ° vertically). More preferably, the detection area SA is positioned so as to overlap with the effective field EV or the field inside thereof (within about 30 ° in the horizontal and within about 20 ° in the vertical) located inside the stable focus field. By installing the proximity sensor 105 by adjusting the arrangement and orientation of the proximity sensor 105, the detection area SA can be set to those positions.

FIG. 8 shows an example in which the detection area SA overlaps the screen 104i. It is set so that the detection area SA of the proximity sensor 105 is located in the visual field of the eyes of the user US while the user US is wearing the frame 101 that is a head mounting member on the head. Accordingly, it is possible to surely visually recognize the approach and withdrawal of the hand to the detection area SA of the proximity sensor 105 without moving the eye while observing the hand HD through the screen 104i. In particular, by setting the detection area SA of the proximity sensor 105 within the stable visual field or the inner visual field, it is possible to reliably perform gesture input while recognizing the detection area SA even when the user observes the screen. . Further, by making the detection area SA of the proximity sensor 105 within the effective visual field EV or the visual field inside the effective visual field EV, gesture input can be performed more reliably. If the detection area SA overlaps the screen 104i, gesture input can be performed more reliably. When the proximity sensor 105 has a plurality of pyroelectric elements RA to RD as in the present embodiment, the entire light receiving area of the plurality of pyroelectric elements RA to RD is regarded as one light receiving unit, and It is assumed that the maximum detection range is the detection area SA. As shown in FIG. 8, when the detection area SA of the proximity sensor 105 is set to overlap the screen 104i, an image showing the detection area SA is displayed on the screen 104i (for example, the range of the area SA is displayed with a solid line). Is preferred. Thereby, since the user can recognize the detection area SA with certainty, the operation by the gesture can be more reliably performed.

Next, the basic principle of gesture detection will be described. Assume that nothing is in front of the user US when the proximity sensor 105 is operating. In this case, since the proximity sensor 105 does not receive invisible light as detection light, the gesture processing unit 1212 of the control processing unit 121 determines that no gesture is performed. On the other hand, as shown in FIGS. 7A and 7B, when the user US's own hand HD is approached in front of the user US, the proximity sensor 105 detects invisible light emitted from the hand HD. The gesture processing unit 1212 determines that a gesture has been performed according to an output signal from the proximity sensor 105 based on the gesture processing unit 1212. In the following description, the gesture is performed using the hand HD of the user US. However, the gesture may be performed using a finger or other part, or an indication tool made of a material that can emit invisible light is used. May be used for gestures.

As described above, the proximity sensor 105 has four pyroelectric elements RA to RD arranged in two rows and two columns (see FIG. 4). Therefore, when the user US brings the hand HD close to the front of the HMD 100 from either the left, right, up, or down directions, the output timings of signals detected by the pyroelectric elements RA to RD are different.

For example, referring to FIG. 7A, FIG. 7B and FIG. 8, in the case of a gesture in which the user US moves the hand HD from the right to the left in front of the HMD 100, the invisible light emitted from the hand HD is close The light enters the sensor 105. In this case, the pyroelectric elements RA and RC first receive invisible light. Therefore, referring to FIGS. 4 and 9, first, the signals of pyroelectric elements RA and RC rise, and the signals of pyroelectric elements RB and RD rise after a delay. Thereafter, the signals of the pyroelectric elements RA and RC fall, and the signals of the pyroelectric elements RB and RD fall after a delay. The gesture processing unit 1212 detects the timing of this signal, and the gesture processing unit 1212 determines that the user US has made a gesture by moving the hand HD from right to left.

This embodiment will be described by taking four gestures as an example of two or more predetermined gestures. Two or more means a plurality and is not limited to four. FIG. 10 is an explanatory diagram illustrating the relationship between gestures and input information in the present embodiment. FIG. 10 includes arrows indicating the movement of each hand of the four gestures and input information pre-assigned to each of the four gestures. The gesture processing unit 1212 shown in FIG. 5 determines gestures 1 to 4 using the timing when the output levels of the pyroelectric elements RA to RD exceed the threshold value th.

The gesture 1 will be described with reference to FIG. 4, FIG. 10, and FIG. FIG. 11 is a waveform diagram showing changes in the output levels of the focus elements RA to RD when the gesture 1 is performed. In this waveform diagram, the horizontal axis indicates time, and the vertical axis indicates the output level. The threshold values th are all the same value.

The gesture 1 is a gesture in which the hand HD enters the detection area SA from the left side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the right side of the detection area SA. When the user performs gesture 1, the output level of the pyroelectric elements RB and RD exceeds the threshold value th, and later, the output level of the pyroelectric elements RA and RC exceeds the threshold value th, and The output levels of the pyroelectric elements RB and RD become the threshold value th or less, and the output levels of the pyroelectric elements RA and RC become the threshold value th or later after this. When the gesture processing unit 1212 (FIG. 5) detects such a change in output level, the gesture processing unit 1212 determines that the gesture 1 has been made. The input information previously assigned to the gesture 1 is a “command to switch to the next screen”. The user can input a “command to switch to the next screen” to the HMD 100 by performing the gesture 1.

The gesture 2 will be described with reference to FIG. 4, FIG. 10, and FIG. FIG. 12 is a waveform diagram showing changes in the output levels of the focus elements RA to RD when the gesture 2 is performed. The horizontal axis and vertical axis of this waveform diagram are the same as the horizontal axis and vertical axis of the waveform diagram of FIG.

The gesture 2 is a gesture in which the hand HD enters the detection area SA from the right side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the left side of the detection area SA. When the user performs gesture 2, the output levels of the pyroelectric elements RA and RC exceed the threshold value th, and later, the output levels of the pyroelectric elements RB and RD exceed the threshold value th, and The output levels of the pyroelectric elements RA and RC become the threshold value th or less, and the output levels of the pyroelectric elements RB and RD become the threshold value th or later after this. When the gesture processing unit 1212 (FIG. 5) detects such a change in output level, the gesture processing unit 1212 determines that the gesture 2 has been made. The input information previously assigned to the gesture 2 is a “command to switch to the previous screen”. The user can input a “command to switch to the previous screen” to the HMD 100 by performing the gesture 2.

Explain gesture 3 and gesture 4. Waveform diagrams are omitted for these gestures. In addition, description of waveform change is omitted for these gestures (the concept is the same as for gesture 1 and gesture 2).

Referring to FIGS. 4 and 10, in gesture 3, hand HD enters detection area SA from above detection area SA (FIGS. 7A, 7B, and 8), and detection area SA from below detection area SA. It is a gesture to exit. The input information previously assigned to the gesture 3 is a “command to switch to the last screen”. The user can input a “command to switch to the last screen” to the HMD 100 by performing the gesture 3.

Explain gesture 4. The gesture 4 is a gesture in which the hand HD enters the detection area SA from the lower side of the detection area SA (FIGS. 7A, 7B, and 8) and exits the detection area SA from the upper side of the detection area SA. The input information previously assigned to the gesture 4 is a “command to switch to the first screen”. The user can input a “command to switch to the first screen” to the HMD 100 by performing the gesture 4.

The combination of the gestures 1 to 4 and the input information is not limited to the example shown in FIG. For example, input information (“command to switch to the previous screen” may be assigned to gesture 1. Gestures that can be detected by proximity sensor 105 are not limited to gestures 1 to 4. Arbitrary input information may be assigned.

Referring to FIG. 5, in the present embodiment, two or more predetermined gestures are distinguished and detected by a detection unit 128 including a proximity sensor 105 and a gesture processing unit 1212. The detection unit 128 is not limited to this configuration. For example, the detection unit 128 may include a camera 106 (two-dimensional imaging device) and an image processing unit that performs predetermined image processing on an image captured by the camera 106 and recognizes a gesture.

There are several reasons why wrong gestures are input. FIG. 7B shows a state where the hand HD is the right hand and the hand HD is in the detection area SA. For example, it is assumed that the gesture 1 is performed in a state where the hand HD is located on the right side of the detection area SA. In order to make the gesture 1, the user US must first move the hand HD to the left side of the detection area SA. This operation must be done outside the detection area SA.

However, if the operation is erroneously performed in the detection area SA, the detection unit 128 detects a gesture for moving the hand HD from right to left (that is, gesture 2). Therefore, since a gesture that is not intended by the user US is detected by the detection unit 128, an incorrect gesture is input.

Further, it is assumed that the user US has a habit of touching hair. Since the HMD 100 is worn on the head of the user US, touching the hair causes a wrong gesture input.

As described below, in this embodiment, even if an incorrect gesture is input, the gesture input can be immediately performed again.

The HMD 100 according to the present embodiment has a first aspect, a second aspect, and a third aspect. The operation of these modes will be described by taking as an example a gesture input (gesture 1) for switching the screen displayed on the image display unit 104B to the next screen.

In the first mode, the mode control unit 1214 shown in FIG. 5 ends the reception mode when the length of time that has elapsed since the reception mode started reaches a predetermined value. An example of the predetermined value is 5 seconds. In the first aspect, the period of the reception mode is fixed (5 seconds). The user operates the operation unit 122 (for example, a cross key provided in the operation unit 122) to set a predetermined value in the mode control unit 1214 in advance. Thereby, the user can determine the length of the period of the reception mode.

FIG. 13 is a flowchart for explaining the operation when a gesture is input in the first mode. FIG. 14 is a waveform diagram showing changes in the output levels of the focus elements RA to RD provided in the first mode when a gesture is input to switch the screen to the next screen. Referring to FIGS. 4, 5, 13, and 14, it is assumed that the user places a hand in front of proximity sensor 105 and performs gesture 1, but erroneously performs gesture 2 due to the above-described cause. When the output levels of pyroelectric elements RA to RD all exceed threshold value th, mode control unit 1214 starts a reception mode for accepting a gesture input (step S1 in FIG. 13 and time t1 in FIG. 14).

At this time, the display control unit 104DR includes an image indicating whether the mode is the reception mode and an image indicating the reception time on the screen displayed on the image display unit 104B. FIG. 15 is an explanatory diagram for explaining an example of the screen 10-1 including these images. The image 11 has a circular shape, and uses the color of the image 11 to indicate whether or not the reception mode is set. In the acceptance mode, the color of the image 11 is blue, and when the acceptance mode is finished, the color of the image 11 is red. Here, the color of the image 11 is blue.

Image 12 shows reception time. This is information indicating the remaining time in the reception mode. More specifically, the mode control unit 1214 has a function of a countdown timer. When the reception mode starts, the mode control unit 1214 sets the countdown timer to 5 seconds and starts the timer. The display control unit 104DR changes the image 12 to an image indicating 5 seconds. When the time indicated by the countdown timer is 4 seconds, 3 seconds, 2 seconds, 1 second, and 0 seconds, the display control unit 104DR displays the image 12 as an image indicating 4 seconds, an image indicating 3 seconds, and 2 seconds. The image is an image showing 1 second and an image showing 0 second. When the time indicated by the countdown timer reaches 0 seconds, the mode control unit 1214 ends the acceptance mode (step S8).

4, 5, 13, and 14, the gesture processing unit 1212 detects the hand HD from any of the upper side, the lower side, the left side, and the right side from the detection area SA (FIGS. 7A, 7B, It is determined whether it has entered FIG. Here, since the output level of the pyroelectric elements RA and RC exceeds the threshold th earlier than the output level of the pyroelectric elements RB and RD, it is determined as the right side. The storage control unit 1215 stores information indicating “right side” in the storage unit 125 (step S2 in FIG. 13).

The gesture processing unit 1212 determines from which of the upper side, the lower side, the left side, and the right side the hand HD has exited the detection area SA. Here, since the output levels of the pyroelectric elements RB and RD are equal to or lower than the threshold th after the output levels of the pyroelectric elements RA and RC, it is determined as the left side. The storage control unit 1215 stores information indicating “left side” in the storage unit 125 (step S3 in FIG. 13).

The gesture processing unit 1212 determines the type of gesture using the results of step S2 and step S3 (step S4 in FIG. 13). The types of gestures are the four gestures described in FIG. The storage unit 125 stores in advance a table (hereinafter referred to as the table of FIG. 10) indicating the correspondence between the gestures 1 to 4 and the input information assigned to them.

The gesture processing unit 1212 reads out the determination results of step S2 and step S3 stored in the storage unit 125. Here, the determination result of step S2 is “right side”, and the determination result of step S3 is “left side”. Accordingly, since the hand HD enters the detection area SA from the right side of the detection area SA and exits the detection area SA from the left side of the detection area SA, the gesture processing unit 1212 determines that the gesture is 2. Note that the gesture processing unit 1212 performs error processing when it is determined that none of the gestures 1 to 4 corresponds. As a result, the display control unit 104DR causes the image display unit 104B to display a screen for prompting a correct gesture.

The storage control unit 1215 refers to the table in FIG. 10 and stores the input information assigned to the gesture determined in step S4 in the storage unit 125 (step S5 in FIG. 13). Here, the input information assigned to the gesture 2 (“command to switch to the previous screen”) is stored in the storage unit 125.

The display control unit 104DR includes a character image (not shown) indicating “an instruction to switch to the previous screen has been input” on the screen 10-1 shown in FIG. The user recognizes that he / she made a wrong gesture by looking at the character image.

The mode control unit 1214 determines whether or not the length of time that has elapsed since the start of the reception mode has reached 5 seconds (a predetermined value) (step S6 in FIG. 13). When the elapsed time has not reached 5 seconds (No in step S6), the mode control unit 1214 determines whether or not the output levels of the pyroelectric elements RA to RD all exceed the threshold th. (Step S7 in FIG. 13). That is, the mode control unit 1214 waits until the next gesture is made. When the mode control unit 1214 determines that all the output levels of the pyroelectric elements RA to RD do not satisfy the condition of exceeding the threshold value th (No in step S7), the mode control unit 1214 performs step S6. Process.

As described above, the user recognizes that he / she made a wrong gesture, so the gesture is redone. Here, it is assumed that the user intended gesture (gesture 1).

When the mode control unit 1214 determines that the output levels of the pyroelectric elements RA to RD all exceed the threshold th (Yes in step S7), the gesture processing unit 1212 performs the process of step S2.

The gesture processing unit 1212 determines from which of the upper side, the lower side, the left side, and the right side the hand HD has entered the detection area SA. Here, since the output level of the pyroelectric elements RB and RD exceeds the threshold th earlier than the output level of the pyroelectric elements RA and RC, it is determined as the left side. The storage control unit 1215 stores information indicating “left side” in the storage unit 125 (step S2).

The gesture processing unit 1212 determines from which of the upper side, the lower side, the left side, and the right side the hand HD has exited the detection area SA. Here, since the output levels of the pyroelectric elements RA and RC are equal to or lower than the threshold th after the output levels of the pyroelectric elements RB and RD, it is determined that the output is on the right side. The storage control unit 1215 stores information indicating “right side” in the storage unit 125 (step S3).

The gesture processing unit 1212 determines the type of gesture using the results of step S2 and step S3 (step S4). More specifically, the gesture processing unit 1212 reads the determination results of step S2 and step S3 stored in the storage unit 125. Here, the determination result of step S2 is “left side”, and the determination result of step S3 is “right side”. Accordingly, since the hand HD has entered the detection area SA from the left side of the detection area SA and has exited the detection area SA from the right side of the detection area SA, the gesture processing unit 1212 determines that the gesture is 1.

The storage control unit 1215 refers to the table in FIG. 10 and stores the input information assigned to the gesture determined in step S4 in the storage unit 125 (step S5). Here, the input information (“command to switch to the next screen”) assigned to gesture 1 is stored in storage unit 125.

The display control unit 104DR includes a character image (not shown) indicating “an instruction to switch to the next screen has been input” on the screen 10-1 shown in FIG. The user sees this character image and knows that he has made the intended gesture. The user waits for the end of the reception mode without making a gesture during the period of the reception mode.

The mode control unit 1214 ends the reception mode when the length of time that has elapsed since the start of the reception mode has reached 5 seconds (a predetermined value) (Yes in step S6) (step in FIG. 13). S8, time t2 in FIG. The display control unit 104DR sets the color of the image 11 included in the screen 10-1 (FIG. 15) displayed on the image display unit 104B to red, and sets the image 12 to an image indicating 0 second.

The processing unit 1213 stores the input information of the gesture 1 in the input information (here, the input information of the gesture 2 and the input information of the gesture 1) stored in the storage unit 125. A predetermined process is performed by using them (step S9 in FIG. 13). In other words, among the one or more gestures detected by the detection unit 128 during the reception mode, predetermined processing is performed using the input information previously assigned to the last detected gesture. Since the last detected gesture is gesture 1, as a predetermined process, the processing unit 1213 sends a command to switch to the next screen to the display control unit 104DR. Thereby, the display control unit 104DR switches the screen 10-1 displayed on the image display unit 104B to the next screen.

When the user starts a gesture after the completion of the predetermined process (step S9), the gesture processing unit 1212 determines that all the output levels of the pyroelectric elements RA to RD have exceeded the threshold value th. The control unit 1214 starts the reception mode (step S1, time t3). Here, the first gesture is a gesture intended by the user (for example, gesture 1). Therefore, the next gesture is not performed during this acceptance mode period. When the acceptance mode ends (step S8, time t4), the processing unit 1213 performs a predetermined process (step S9). Since the last detected gesture is gesture 1, as a predetermined process, the processing unit 1213 sends a command to switch to the next screen to the display control unit 104DR. Thereby, the display control unit 104DR switches the screen displayed on the image display unit 104B to the next screen.

The main effect of the first aspect will be described. Referring to FIGS. 5 and 14, processing unit 1213 preliminarily applies gesture 1 in each of the reception mode period defined from time t1 to time t2 and the reception mode period defined from time t3 to time t4. The assigned input information (command to switch the screen to the next screen) is sent to the display control unit 104DR. In this way, the processing unit 1213 performs a predetermined process using the input information previously assigned to the last detected gesture among the one or more detected gestures during the period of the reception mode. For this reason, since the last gesture should just be the gesture which a user intends, if it is the period of reception mode, the user can repeat gesture input many times. Therefore, according to the first aspect, even if an incorrect gesture input is made during the period of the reception mode for accepting the gesture input, the gesture input can be performed without waiting for the period of the next reception mode. This effect also occurs in the second and third aspects.

Referring to FIGS. 5 and 15, display control unit 104DR causes image display unit 104B to display image 12 indicating the reception time during the reception mode. The image 12 is information indicating the remaining time in the reception mode. Therefore, according to the first aspect, the user can recognize when the period of the reception mode ends. This effect also occurs in the second and third aspects.

Referring to FIG. 14, mode control unit 1214 sets the reception mode when the levels of pyroelectric elements RA to RD output all exceed threshold value th (time t1, time t3) in a period other than reception mode. Start. Consider a case where the acceptance mode is started when the output level of some of the pyroelectric elements RA to RD exceeds a threshold value. In this case, after the start of the acceptance mode, when the gesture is stopped before the output levels of the pyroelectric elements RA to RD all exceed the threshold value th, the gesture is not determined, but the acceptance mode is continued. . For this reason, when the gesture is performed again, the acceptance mode may end before the gesture ends. According to the first aspect, since the reception mode is started when the output values of the pyroelectric elements RA to RD all exceed the threshold value th during the period other than the reception mode, such inconvenience does not occur. . This effect also occurs in the third aspect.

Referring to FIG. 5, as described in step S <b> 5 of FIG. 13, every time a gesture is detected by the detection unit 128, the display control unit 104 </ b> DR is a character indicating input information assigned in advance to the detected gesture. A screen including an image (for example, a character image indicating “a command to switch to the previous screen has been input”) is displayed on the image display unit 104B. Thereby, the user can determine whether the gesture he / she made is wrong.

Thus, the display control unit 104DR and the image display unit 104B function as a notification unit. Every time a gesture is detected by the detection unit 128, the notification unit notifies input information pre-assigned to the detected gesture.

There are an aspect 1 that uses a screen, an aspect 2 that uses sound, and an aspect 3 that uses both as notification parts. Aspect 1 includes a display unit (image display unit 104B) and a display control unit 104DR that causes the display unit (image display unit 104B) to display a screen showing input information pre-assigned to the detected gesture. Aspect 2 includes an audio processing unit that converts input information pre-assigned to the detected gesture into an audio signal, an amplifier that amplifies the audio signal, and a speaker that outputs the amplified audio signal as audio. The audio processing unit is realized by a combination of hardware such as a CPU, RAM, and ROM, and a program that converts input information into an audio signal. Aspect 3 includes Aspect 1 and Aspect 2.

The second aspect will be described mainly with respect to differences from the first aspect. In the second mode, when the mode control unit 1214 shown in FIG. 5 changes during the reception mode from the detection state in which the gesture is detected by the detection unit 128 to the non-detection state in which no gesture is detected, no detection is performed. Measurement of the time of the state is started, and when the length of time of the non-detection state reaches a predetermined value, the period of the reception mode is ended. An example of the predetermined value is 2 seconds. That is, when the period of the non-detection state continues for 2 seconds, the reception mode ends. In the second aspect, the period of the reception mode is not fixed. The user operates the operation unit 122 to set a predetermined value in the mode control unit 1214 in advance. Thereby, the user can determine the length of time of the non-detection state which is a condition for terminating the reception mode.

FIG. 16 is a flowchart for explaining the operation when a gesture is input in the second mode. FIG. 17 is a waveform diagram showing changes in the output levels of the focus elements RA to RD provided in the second mode when a gesture input is made to switch the screen to the next screen. The flowchart shown in FIG. 16 differs from the flowchart shown in FIG. 13 in the following two points. In step S1 of FIG. 16, a screen 10-2 shown in FIG. 18 is displayed on the image display unit 104B instead of the screen 10-1 shown in FIG. Step S10 in FIG. 16 is executed instead of step S6 in FIG.

Referring to FIG. 4, FIG. 5, FIG. 16, and FIG. 17, it is assumed that the user places a hand in front of proximity sensor 105 and performs gesture 1, but makes gesture 2 incorrectly for the above-described reasons. When the output levels of pyroelectric elements RA to RD all exceed threshold value th, mode control unit 1214 starts a reception mode for accepting a gesture input (step S1 in FIG. 16, time t1 in FIG. 17). At this time, the display control unit 104DR includes an image indicating whether or not the reception mode is set and an image indicating the time related to the non-detection state on the screen displayed on the image display unit 104B. FIG. 18 is an explanatory diagram for explaining an example of the screen 10-2 including these images. The image 11 indicates whether or not the reception mode is set as described in the screen 10-1 shown in FIG. Here, since it is a reception mode, the color of the image 11 is blue.

Image 13 shows the time related to the non-detection state. This is information indicating the remaining time in the reception mode. More specifically, it is assumed that when at least one of the output levels of the pyroelectric elements RA to RD exceeds the threshold th, the detection unit 128 detects a gesture. When the output levels of the pyroelectric elements RA to RD are all equal to or lower than the threshold th, the detection unit 128 is in a non-detection state in which no gesture is detected. The mode control unit 1214 has a function of a countdown timer, and when the detection state is changed to the non-detection state (time t10, time t11, time t12 in FIG. 17), the countdown timer is set to 2 seconds. Start the timer. The display control unit 104DR changes the image 13 to an image indicating 2 seconds. When the time indicated by the countdown timer is 1 second and 0 seconds, the display control unit 104DR changes the image 13 to an image indicating 1 second and an image indicating 0 second. When the time indicated by the countdown timer reaches 0 seconds, the mode control unit 1214 ends the reception mode.

In the second mode, step S10 is executed after step S5. More specifically, when one hand moves out of the detection area SA (FIGS. 7A, 7B, and 8) by the end of one gesture, the detection state changes to the non-detection state. This change occurs during the period of step S3. When the mode control unit 1214 changes from the detection state to the no detection state, the mode control unit 1214 sets the countdown timer to 2 seconds and starts the timer. The mode control unit 1214 determines whether or not the time indicated by the timer has reached 0 seconds. That is, the mode control unit 1214 determines whether or not the length of time of the non-detection state has reached a predetermined value (2 seconds) (step S10).

When the mode control unit 1214 determines that the length of time of the non-detection state has not reached a predetermined value (No in step S10), the mode control unit 1214 performs the process of step S7.

When the mode control unit 1214 determines that the length of time of the non-detection state has reached a predetermined value (Yes in step S10, time t2, time t4 in FIG. 17), the mode control unit 1214 Terminates the acceptance mode (step S8).

As described above, according to the second aspect, the reception mode does not end unless the length of time of the non-detection state reaches 2 seconds (predetermined value). Therefore, according to the second aspect, it is possible to prevent the reception mode from ending while the user is making a gesture.

The third aspect will be described. The third aspect is an aspect in which the first aspect and the second aspect are combined. In the mode control unit 1214 illustrated in FIG. 5, a first value that is an upper limit value for the period of the reception mode is set. Even if the period of the reception mode has not reached the upper limit value, the mode control unit 1214 ends the reception mode when the length of time of the non-detection state reaches a predetermined second value. The second value is smaller than the first value. The first value takes 5 seconds as an example. The second value takes 2 seconds as an example. The user operates the operation unit 122 to set the first value and the second value in the mode control unit 1214 in advance. Thereby, the user can determine the first value and the second value which are the conditions for terminating the reception mode.

FIG. 19 is a flowchart for explaining the operation when a gesture is input in the third mode. FIG. 20 is a waveform diagram showing changes in the output levels of the focus elements RA to RD provided in the third mode when a gesture input is made to switch the screen to the next screen. The flowchart shown in FIG. 19 differs from the flowchart shown in FIG. 13 in the following two points. In step S1 of FIG. 19, a screen 10-3 shown in FIG. 21 is displayed on the image display unit 104B instead of the screen 10-1 shown in FIG. If step S6 is No, step S10 is executed. Step S10 is the same as step S10 in FIG.

Referring to FIG. 4, FIG. 5, FIG. 19, and FIG. 20, it is assumed that the user places a hand in front of proximity sensor 105 and makes gesture 1, but makes gesture 2 by the above-mentioned cause. When the output levels of pyroelectric elements RA to RD all exceed threshold value th, mode control unit 1214 starts a reception mode for accepting a gesture input (step S1 in FIG. 19 and time t1 in FIG. 20). At this time, the display control unit 104DR includes, on the screen displayed on the image display unit 104B, an image indicating whether or not the reception mode is set, an image indicating the reception time, and an image indicating the time related to the non-detection state. FIG. 21 is an explanatory diagram for explaining an example of the screen 10-3 including these images. The image 11 indicates whether or not the reception mode is set as described in the screen 10-1 shown in FIG. Here, since it is a reception mode, the color of the image 11 is blue.

The image 12 shows the reception time as described in the screen 10-1 shown in FIG. This is information indicating the remaining time in the reception mode. Here, the image 12 is an image showing 5 seconds.

The image 13 shows the time relating to the non-detection state as described in the screen 10-2 shown in FIG. This is information indicating the remaining time in the reception mode. When the non-detection state starts during the reception mode (time t10, time t11, and time t12 in FIG. 20), the display control unit 104DR changes the image 13 to an image indicating 2 seconds. Here, since this is not the case, the time portion indicated by the image 13 is left blank.

In the third mode, mode control unit 1214 determines whether or not the length of time that has elapsed since the start of the reception mode has reached a predetermined value (first value) (FIG. 19). Step S6). When the length of the elapsed time has not reached the predetermined value (first value) (No in step S6), the mode control unit 1214 determines that the length of time in the non-detection state is predetermined. It is determined whether or not a predetermined value (second value) has been reached (step S10).

When the length of time of the non-detection state does not reach the predetermined value (second value) (No in step S10), the mode control unit 1214 outputs the levels of the pyroelectric elements RA to RD. It is determined whether or not all have exceeded the threshold value th (step S7 in FIG. 19).

When the length of time of the non-detection state has reached a predetermined value (second value) (Yes in step S10), the mode control unit 1214 ends the reception mode (step S8).

The main effect of the third aspect will be described. With reference to FIG. 5 and FIG. 20, the first value is set to be relatively large (for example, 5 seconds) in order to allow the user a margin for gesture input. The processing unit 1213 performs a predetermined process after the end of the reception mode (step S9 in FIG. 19). If the first gesture input is correct, if the acceptance mode ends 5 seconds after the acceptance mode starts, a relatively large waiting time occurs after the first gesture ends until the predetermined processing is performed. (For example, 4 seconds). This is wasted time for the user. Therefore, if the length of time in the non-detection state reaches a second value (for example, 2 seconds), the mode control unit 1214 ends the reception mode even before it reaches 5 seconds after the reception mode starts. (Time t4).

(Summary of embodiment)
The display device according to the first aspect of the embodiment includes a display unit, a detection unit having a detection region at a position different from the display unit, and capable of distinguishing and detecting two or more predetermined gestures, and a gesture input A mode control unit that executes a reception mode for reception for a predetermined period, and after the reception mode ends, the last detected in one or more of the gestures detected by the detection unit during the period of the reception mode A processing unit that performs predetermined processing using input information pre-assigned to the gesture.

The display device performs a predetermined process using the input information previously assigned to the last detected gesture among the one or more detected gestures during the period of the reception mode in which the gesture input is received (for example, And a command to switch the screen displayed on the display unit to the next screen). For this reason, since the last gesture should just be the gesture which a user intends, if it is the period of reception mode, the user can repeat gesture input many times. Therefore, according to the display device according to the first aspect of the embodiment, even if an incorrect gesture input is made during the period of the reception mode for accepting the gesture input, the gesture input is performed without waiting for the next period of the reception mode. It becomes possible.

The display device is, for example, a wearable terminal. A wearable terminal is a terminal device that can be worn on a part of a body (for example, a head or an arm).

As will be described below, the display device according to the first aspect of the embodiment has a mode in which the period of the reception mode is fixed (first mode) and a mode in which the period of the reception mode is not fixed ( There are a second aspect) and a combination of these two aspects (third aspect).

In the above configuration, the mode control unit ends the reception mode when the length of time that has elapsed since the reception mode started reaches a predetermined value.

This is a mode in which the period of the reception mode is fixed (first mode). If the predetermined value is, for example, 5 seconds, the period of the reception mode is 5 seconds.

The above configuration further includes an operation unit capable of performing an operation of setting the predetermined value.

According to this configuration, since the user can set a predetermined value, the user can determine the length of the period of the reception mode.

In the above configuration, when the mode control unit changes from the detection state in which the gesture is detected by the detection unit to the non-detection state in which the gesture is not detected during the reception mode, Time measurement is started, and when the length of time in the non-detection state reaches a predetermined value, the reception mode is terminated.

This is a mode in which the period of the reception mode is not fixed (second mode). In the second aspect, the reception mode does not end unless the length of time of the non-detection state reaches a predetermined value (for example, 2 seconds). Therefore, according to the second aspect, it is possible to prevent the reception mode from ending while the user is making a gesture.

The above configuration further includes an operation unit capable of performing an operation of setting the predetermined value.

According to this configuration, the user can set a predetermined value. Therefore, the user can determine the length of time of the non-detection state that is a condition for terminating the reception mode.

In the above configuration, the mode control unit ends the reception mode when the length of time that has elapsed since the reception mode started reaches a first predetermined value, and the mode control unit The second value is a predetermined value smaller than the first value, and the gesture is not detected from the detection state in which the gesture is detected by the detection unit during the reception mode. When the state is changed, the measurement of the time of the non-detection state is started, and when the length of the time of the non-detection state reaches the second value, the time elapsed after the reception mode is started The reception mode is terminated even before the length reaches the first value.

This is a mode combining the first mode and the second mode (third mode). The first value is set to be relatively large (for example, 5 seconds) in order to allow the user a margin for gesture input. The processing unit performs a predetermined process after the acceptance mode ends. If the first gesture input is correct, if the acceptance mode ends 5 seconds after the acceptance mode starts, a relatively large waiting time occurs after the first gesture ends until the predetermined processing is performed. (For example, 4 seconds). This is wasted time for the user. Therefore, if the length of time of the non-detection state reaches a second value (for example, 2 seconds), the mode control unit ends the reception mode even before reaching 5 seconds after the reception mode starts. .

The above configuration further includes an operation unit capable of performing an operation of setting the first value and the second value.

According to this configuration, the user can set the first value and the second value. Therefore, the user can determine the first value and the second value that are the conditions for ending the acceptance mode.

The above configuration further includes a display control unit that displays information indicating the remaining time in the reception mode on the display unit during the reception mode.

This configuration allows the user to recognize when the acceptance mode ends.

In the above configuration, the detection unit includes a plurality of pyroelectric elements arranged in a two-dimensional matrix, and a gesture processing unit that determines a gesture based on each output of the plurality of pyroelectric elements.

This is an example of a detection unit.

In the above configuration, the gesture processing unit determines the gesture using a timing at which an output value of each of the plurality of pyroelectric elements exceeds a predetermined threshold, and the mode control unit When the output values of the plurality of pyroelectric elements all exceed the threshold during a period other than the acceptance mode, the acceptance mode is started.

This configuration is an example of a condition for starting the reception mode. This configuration has the following effects with respect to the modes (first mode and third mode) in which the reception mode is ended when a predetermined time has elapsed since the start of the reception mode. Consider a case where the acceptance mode is started when the output values of some of the pyroelectric elements exceed a threshold value. In this case, after the start of the reception mode, when the gesture is stopped before all the output values of the plurality of pyroelectric elements exceed the threshold value, the gesture is not determined, but the reception mode continues. For this reason, when the gesture is performed again, the acceptance mode may end before the gesture ends. According to this configuration, since the reception mode is started when the output values of the plurality of pyroelectric elements all exceed the threshold value during the period other than the reception mode, such inconvenience does not occur.

The above configuration further includes a notification unit that notifies the input information indicating an input pre-assigned to the detected gesture each time the gesture is detected by the detection unit.

This configuration can notify the user of input information every time the user makes a gesture. Thereby, the user can determine whether the gesture he / she made is wrong.

A gesture input method according to a second aspect of the embodiment includes a display unit, and a detection unit that has a detection region at a position different from the display unit and can detect and detect two or more predetermined gestures. A method for performing gesture input to a display device, wherein a first step of executing a reception mode for receiving a gesture input for a predetermined period of time, and detection by the detection unit during the reception mode after the reception mode ends A second step of performing a predetermined process using input information pre-assigned to the last detected gesture among the one or more gestures performed.

The gesture input method according to the second aspect of the embodiment defines the display device according to the first aspect of the embodiment from the viewpoint of the method, and has the same operation as the display device according to the first aspect of the embodiment. Has an effect.

Although embodiments of the present invention have been illustrated and described in detail, it is merely exemplary and illustrative and not limiting. The scope of the invention should be construed by the language of the appended claims.

The entire disclosure of Japanese Patent Application No. 2016-225078 filed on November 18, 2016 is incorporated herein by reference in its entirety.

According to the present invention, a display device and a gesture input method can be provided.

Claims (13)

A display unit;
A detection unit having a detection region at a different position from the display unit, and capable of distinguishing and detecting two or more predetermined gestures;
A mode control unit for executing a reception mode for receiving a gesture input for a predetermined period;
After the acceptance mode ends, among the one or more gestures detected by the detection unit during the acceptance mode period, input information pre-assigned to the last detected gesture is used for predetermined And a processing unit that performs the process. The display device according to claim 1, wherein the mode control unit terminates the reception mode when a length of time that has elapsed since the reception mode started reaches a predetermined value. The display device according to claim 2, further comprising an operation unit capable of performing an operation of setting the predetermined value. The mode control unit measures the time of the non-detection state when the detection unit changes from a detection state where the gesture is detected to a non-detection state where the gesture is not detected during the reception mode. The display device according to claim 1, wherein the display mode is started, and when the length of time of the non-detection state reaches a predetermined value, the reception mode is ended. The display device according to claim 4, further comprising an operation unit capable of performing an operation of setting the predetermined value. The mode control unit ends the reception mode when the length of time that has elapsed since the reception mode has started reaches a first predetermined value,
The mode control unit sets a predetermined value smaller than the first value as a second value, and detects the gesture from a detection state in which the gesture is detected by the detection unit during the reception mode. When the non-detection state is changed, measurement of the time of the non-detection state is started, and when the length of time of the non-detection state reaches the second value, the reception mode is started. The display device according to claim 1, wherein the reception mode is ended even before the length of time elapsed since the time reaches the first value. The display device according to claim 6, further comprising an operation unit capable of performing an operation for setting the first value and the second value. The display device according to any one of claims 1 to 7, further comprising a display control unit configured to display information indicating the remaining time in the reception mode on the display unit during the reception mode. The detection unit includes a plurality of pyroelectric elements arranged in a two-dimensional matrix, and a gesture processing unit that determines a gesture based on each output of the plurality of pyroelectric elements. A display device according to claim 1. The gesture processing unit determines the gesture using a timing at which an output value of each of the plurality of pyroelectric elements exceeds a predetermined threshold value,
10. The display device according to claim 9, wherein the mode control unit starts the reception mode when output values of the plurality of pyroelectric elements all exceed the threshold during a period other than the reception mode. The display device according to any one of claims 1 to 10, wherein the display device is a wearable terminal. The reporting unit according to any one of claims 1 to 11, further comprising a notification unit that notifies the input information indicating an input pre-assigned to the detected gesture every time the gesture is detected by the detection unit. Display device. A method for inputting gestures to a display device comprising: a display unit; and a detection unit having a detection region at a position different from the display unit and capable of distinguishing and detecting two or more predetermined gestures,
A first step of executing a reception mode for accepting a gesture input for a predetermined period;
After the acceptance mode ends, among the one or more gestures detected by the detection unit during the acceptance mode period, input information pre-assigned to the last detected gesture is used for predetermined A gesture input method comprising: a second step of performing the process.

Images