JP2018109911A - Display device, instruction detection device, and image display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display device capable of improving the accuracy of an operation of the display device by a user's gesture.SOLUTION: A display device 1 includes: a display part 12 for displaying an image; a motion vector detection part 115 for specifying a direction in which an object moves at a display surface side of display means; and a detection operation specification part 117 for specifying an indication position being a position indicated by the object in the display part 12 on the basis of the direction with the object moving therein, specified by the motion vector detection part 115.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a display device that acquires a user's gesture motion and controls a display image, an instruction detection device, and an image display method.

  Nowadays, a display device is mounted with a motion sensor and a virtual button is realized by software, so that a user can operate a video with a finger gesture while looking at the display device. However, the size of the user's finger and the distance between the motion sensor and the finger are different for each user. Therefore, even if a plurality of users perform the same gesture, the display device may determine that a different gesture is performed for each user. As a method for improving the recognition accuracy of a gesture, Patent Document 1 discloses a method for determining an action based on a ratio between a finger size and a gesture input area.

JP 2014-203367 A

  When the user operates the display device with a gesture, it is assumed that the user stands in various positions with respect to the display device. When the position where the user stands is different, the movement of the user's finger may be different even in the same operation. However, in the technique disclosed in Patent Document 1, the movement of the user's finger is not considered. Therefore, when the user performs a gesture from a different direction with respect to the display device, the gesture may be recognized as a different gesture although the gesture is for performing the same operation. As a result, there has been a problem that the display device may perform an operation different from the intention of the user who tried to operate the display device with a gesture.

  Accordingly, the present invention has been made in view of the above-described points, and an object thereof is to provide a display device, an instruction detection device, and an image display method that can improve the accuracy of operation of the display device by a gesture.

  The display device of the present invention is based on display means for displaying an image, direction specifying means for specifying the direction in which the object moves on the display surface side of the display means, and the direction in which the object specified by the direction specifying means moves. And position specifying means for specifying an indicated position which is a position pointed to by the object in the display means.

  An instruction detection apparatus according to the present invention is an instruction detection apparatus that detects an instruction to a display device that displays an image, the direction specifying unit specifying a direction in which an object moves on the display surface side of the display device, and the direction specifying unit Based on the moving direction of the object specified by the user, position specifying means for specifying an indicated position that is a position pointed to by the object in the display device; and the display device that specifies the indicated position specified by the position specifying means. And transmitting means for transmitting to.

  The image display method of the present invention is a computer-executed step of specifying a direction in which an object moves on the display screen side of a display device, and a display means for displaying an image based on the direction in which the object moves. A step of specifying an indicated position which is the indicated position; and a step of processing an image to be displayed on the display means based on the indicated position.

  According to the present invention, there is an effect that the accuracy of operation of the display device by the gesture can be improved.

It is a figure which shows the outline | summary of an instruction | indication detection system. It is a figure for demonstrating the difference in recognition of the operation | movement of the finger by a user's position. It is a figure for demonstrating a user's operation | movement. It is a figure for demonstrating the fixed positioning calculation performed with respect to a user's operation | movement. It is a block diagram which shows the structure of a display apparatus. It is a flowchart of a fixed positioning calculation. It is a figure for demonstrating the outline | summary of the 2nd Embodiment of this invention. It is a figure for demonstrating the calculation in a XZ plane. It is a figure for demonstrating the calculation in a ZY plane. It is a block diagram which shows the structure of a display apparatus. It is a flowchart of a fixed positioning calculation. It is a figure for demonstrating the outline | summary of the 3rd Embodiment of this invention. It is a figure for demonstrating the difference of an instruction | indication position. It is a block diagram which shows the structure of a display apparatus. It is a flowchart (1) of fixed positioning calculation. It is a flowchart (2) of fixed positioning calculation. It is a flowchart (1) which shows a process in case a several person operate | moves simultaneously. It is a flowchart (2) which shows a process in case a several person operate | moves simultaneously. It is a flowchart which shows the arbitration process in case a several person operate | moves simultaneously.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
<First Embodiment>
[Outline of display device 1]
The operation of the display device in the first embodiment of the present invention will be described. In the first embodiment, the display device 1 specifies the direction of the gesture performed by the user based on the positional relationship between the user's finger that has entered the recognition area of the motion sensor and the motion sensor. And the display apparatus 1 specifies the content of operation by specifying the instruction | indication position in a user's display apparatus 1 based on the direction of the specified gesture.

  The display device 1 performs a rotation process of the motion vector corresponding to the gesture performed by the user, and converts the motion into a fixed position. The fixed position in the present embodiment is, for example, a position directly in front of the display unit of the display device 1, but may be a position other than the directly front. Moreover, in the following description, it is called fixed positioning calculation that the display device 1 performs coordinate conversion of the operation of the user's finger and converts it into an operation positioned in front of the display unit of the display device 1.

  FIG. 1 is a diagram showing an overview of the instruction detection system S. The instruction detection system S includes a display device 1, a video signal generator 2, and a motion sensor device 3. In FIG. 1, it is assumed that a plurality of users operate the display device 1 using gestures.

  The display device 1 includes a control unit 11 and a display unit 12. The control unit 11 is an instruction detection device that specifies a user's gesture based on coordinate information indicating the position of an object acquired from the motion sensor device 3 and controls the display unit 12. The motion sensor device 3 is an object detection unit that detects the position of an object on the display surface side of the display unit 12 of the display device 1. The control unit 11 notifies the display unit 12 of the position indicated by the user gesture specified based on the coordinate information. Further, the control unit 11 controls the video signal generator 2 to receive a video signal from the video signal generator 2. The control unit 11 inputs the received video signal to the display unit 12 and causes the display unit 12 to display an image based on the video signal.

  The display unit 12 displays videos such as moving images and still images output from the video signal generator 2. Although the control unit 11 is separated from the display unit 12 in FIG. 1, the control unit 11 may be configured integrally with the display unit 12. The display device 1 may include a motion sensor device 3.

  FIG. 2 is a diagram for explaining a difference in recognition of finger movement depending on the position of the user. In FIG. 2, the fingers of the user U1, the user U2, and the user U3 are present in the detectable area 7, which is an area where the motion sensor device 3 recognizes the finger, and indicates the user instruction area 8 inside the display device 1. Assumes that.

  First, each user moves his / her finger in the direction indicated by the solid line arrow toward the user instruction area 8. The motion sensor device 3 detects the movement of each user's finger. The control unit 11 of the display device 1 determines the movement of the finger detected by the motion sensor device 3 as a linear movement that is different for each user. In the present embodiment, the control unit 11 converts the motion determined as a linear motion that is different for each user into the same motion as long as it is a finger within the detectable region 7. In addition, the motion conversion process according to the present embodiment is not limited to the straight motion, but can be applied to other motions such as a motion of rotating the hand and a motion of moving the hand in the horizontal and vertical directions.

  FIG. 3 is a diagram for explaining a user's operation. FIG. 3 is a plan view when each device and user is viewed from the top to the bottom. The finger of the user U1 enters the detectable region 7 from the entry position P1 (X1, Z1), performs the operation indicated by the motion vector L2, and stops at the operation stop position P3 (X3, Z3). In FIG. 3, a wrist center T0 indicates the center of the user's wrist.

  FIG. 4 is a diagram for explaining the fixed positioning calculation performed for the user's operation. The control unit 11 of the display device 1 passes the center position P0 (X0, Z0) and the center line L1 and the motion vector based on the position where the center line L1 parallel to the Z axis intersects with the extension line of the motion vector L2. An angle θ1 formed by L2 is calculated. The control unit 11 of the display device 1 calculates a coordinate-converted motion vector L3 by performing a rotation process on the motion vector L2 by an angle θ1 with the position where the center line L1 and the motion vector L2 intersect as the rotation center. And the control part 11 of the display apparatus 1 processes the operation | movement of the user U1 as the motion vector L3 after coordinate transformation.

In this embodiment, the process of converting the state before the fixed positioning calculation shown in FIG. 3 into the state after the fixed positioning calculation shown in FIG. 4 is performed using the block diagram of FIG. 5 and the flowchart of FIG. I will explain.
Hereinafter, the configuration and operation of the display device 1 will be described in detail.

[Configuration and Operation of Display Device 1]
FIG. 5 is a block diagram illustrating a configuration of the display device 1. The display device 1 includes a control unit 11, a display unit 12, and a storage unit 13. The control unit 11 includes a detection coordinate input unit 111, a hand position information detection unit 112, a finger position information detection unit 113, an intrusion angle detection unit 114, a motion vector detection unit 115, a coordinate conversion unit 116, and a detection operation. The identification unit 117, the drawing unit 118, the image holding unit 119, the image selection unit 120, the enlargement / reduction unit 121, and the synthesis unit 122 are included. The solid line in FIG. 5 indicates the flow of the video signal, and the broken line indicates the flow of the sensor signal and the control signal.

First, the operation of each block of the display device 1 will be described.
The control unit 11 is a control unit that controls the display device 1 and is, for example, a CPU (Central Processing Unit). The control unit 11 executes a program stored in the storage unit 13, which will be described later, to thereby detect a detection coordinate input unit 111, a hand position information detection unit 112, a finger position information detection unit 113, an intrusion angle detection unit 114, a motion vector detection Functions as a unit 115, a coordinate conversion unit 116, a detection operation specifying unit 117, a drawing unit 118, an image holding unit 119, an image selection unit 120, an enlargement / reduction unit 121, and a composition unit 122.

  The display unit 12 is, for example, a liquid crystal panel that displays an image. The display unit 12 displays videos such as moving images and still images output from the video signal generator 2. The storage unit 13 is a storage medium including a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage unit 13 stores a program executed by the control unit 11.

  The detection coordinate input unit 111 checks the center position and the coordinates of the detectable region 7 at the time of initialization. Specifically, the detected coordinate input unit 111 confirms the center position P0 (X0, Z0), and confirms the coordinates of the boundary of the detectable region 7 with the center position P0 (X0, Z0) as a reference. The detection coordinate input unit 111 notifies the hand position information detection unit 112 of the confirmed center position P0 (X0, Z0) and the detectable region 7. Thereafter, the detection coordinate input unit 111 notifies the hand position information detection unit 112 of coordinate information indicating the position of the object acquired from the motion sensor device 3.

  The detection coordinate input unit 111 may acquire the moving direction of the object from the motion sensor device 3 when the motion sensor device 3 detects the moving direction of the object. The detection coordinate input unit 111 acquires, for example, a motion vector L2 indicating the moving direction of the object in FIG. 3 and notifies the motion vector detection unit 115 of it.

The hand position information detection unit 112 detects the position of the finger of the user performing the gesture based on the coordinate information indicating the position of the object acquired from the detection coordinate input unit 111. Further, the hand position information detection unit 112 may detect the tip position of a predetermined pointing bar in addition to the user's finger.
The hand position information detection unit 112 notifies the finger position information detection unit 113 and the intrusion angle detection unit 114 of coordinate information indicating the position of the object acquired from the detection coordinate input unit 111, and also notifies the motion vector detection unit 115. The information indicating the position of the detected hand is notified.

  The finger position information detection unit 113 detects the finger position based on the coordinate information regarding the user's finger acquired from the hand position information detection unit 112. Then, the finger position information detection unit 113 notifies the motion vector detection unit 115 of information indicating the detected finger position.

  The intrusion angle detection unit 114 detects the intrusion position P1 (X1, Z1) of the finger based on the coordinate information regarding the user's finger acquired from the hand position information detection unit 112 and the detectable region 7. Then, the intrusion angle detection unit 114 calculates the center line L1 based on the center position P0 (X0, Z0) and the detectable region 7. The intrusion angle detection unit 114 calculates the angle θ1 based on the center line L1, the intrusion position P1 (X1, Z1), and the operation stop position P3 (X3, Z3). The intrusion angle detection unit 114 transmits the calculated angle θ1 to the coordinate conversion unit 116.

  The motion vector detection unit 115 functions as direction specifying means for specifying the direction in which the object moves on the display surface side of the display unit 12. The object that moves on the display surface side of the display unit 12 is, for example, a user's finger or a predetermined pointer. The direction of the line connecting the finger entry position P1 (X1, Z1) and the operation stop position P3 (X3, Z3) corresponds to the direction in which the object moves. The motion vector detection unit 115 notifies the detected coordinate input unit 111 of the specified moving direction, for example, as a vector.

  The motion vector detection unit 115 calculates a motion vector L2 indicating the direction in which the object moves based on the coordinate information regarding the user's finger acquired from the hand position information detection unit 112 and the finger position information detection unit 113. Specifically, the motion vector detection unit 115 calculates the motion vector L2 based on the fingertip position P2 (X2, Z2) that is the position of the tip of the finger that gives an instruction and the motion stop position P3 (X3, Z3). To do. Then, the motion vector detection unit 115 notifies the coordinate conversion unit 116 of information indicating the motion vector L2. The motion vector detection unit 115 may notify the intrusion angle detection unit 114 of the motion vector L2.

The motion vector detection unit 115 may specify the outer shape of at least a part of the object. The motion vector detection unit 115 specifies, for example, the tip of the user's finger or the center of the wrist.
When the motion vector detection unit 115 acquires the motion vector L2 from the detection coordinate input unit 111, the motion vector detection unit 115 does not calculate the motion vector L2 and notifies the coordinate conversion unit 116 of the motion vector L2 acquired from the detection coordinate input unit 111. To do.

  The coordinate conversion unit 116 is a straight line in the direction in which the center line and the finger move, with the center line passing through the center position of the display unit 12 and perpendicular to the display unit 12 intersecting with the straight line in the direction in which the finger moves. The direction in which the object moves is converted by rotating the direction in which the object moves by an angle θ1 formed by. Specifically, the coordinate conversion unit 116 performs coordinate conversion using the angle θ1 acquired from the intrusion angle detection unit 114 and the motion vector L2 acquired from the motion vector detection unit 115, and the coordinate vector after the coordinate conversion L3 Is calculated. Then, the coordinate conversion unit 116 notifies the detected motion specifying unit 117 of the post-coordinate conversion motion vector L3.

  The detection operation specifying unit 117 functions as a position specifying unit that specifies the indicated position that is the position pointed to by the object on the display unit 12 based on the moving direction of the object specified by the motion vector detecting unit 115. The detection operation specifying unit 117 specifies the user's operation by specifying the designated position. Specifically, the detection operation specifying unit 117 specifies the operation of the user U1 based on the designated position indicated by the coordinate-converted motion vector L3 acquired from the coordinate conversion unit 116. In the present embodiment, it is assumed that the detection operation specifying unit 117 specifies that the user has performed an operation in which the user's index finger is moved straight by the distance of the coordinate vector after the coordinate conversion.

  If the specified operation is image selection, the detection operation specifying unit 117 notifies the image selection unit 120 of image selection instruction information. If the specified operation is image enlargement or reduction, the detection operation specifying unit 117 transmits enlargement / reduction instruction information to the enlargement / reduction unit 121. The detection operation specifying unit 117 transmits drawing instruction information to the drawing unit 118 if drawing of an image is necessary due to the specified operation.

  When the motion vector detection unit 115 specifies at least a part of the outer shape of the object, the detection operation specifying unit 117 specifies the indicated position based on the moving direction of the object having the specified outer shape. For example, the detection operation specifying unit 117 specifies the designated position based on the moving direction of the finger tip specified by the motion vector detecting unit 115. Further, the detection operation specifying unit 117 may specify the designated position based on the angle of the moving direction of the object with respect to the screen of the display unit 12. Specifically, the detection operation specifying unit 117 specifies the designated position based on an angle θ1 formed by the center line L1 and a straight line obtained by extending the motion vector L2.

  The drawing unit 118 performs a process corresponding to the designated position specified by the detection operation specifying unit 117 on the image displayed on the display unit 12. Specifically, the drawing unit 118 performs drawing based on information related to the drawing instruction acquired from the detection operation specifying unit 117, and transmits the image to the combining unit 122.

  The image holding unit 119 acquires an image such as a moving image or a still image from the video signal generator 2. The number of images acquired by the image holding unit 119 is determined according to the amount of memory held by the storage unit 13 of the display device 1. Further, the image holding unit 119 may transmit a control signal indicating completion to the video signal generator 2 at a time when image acquisition is completed. Thereafter, the image holding unit 119 transmits the instructed image among the images to be held in response to an instruction from the image selection unit 120. Note that the image holding unit 119 may acquire an image from the video signal generator 2 at a time when the image selection unit 120 instructs.

  The image selection unit 120 acquires an image from the image holding unit 119 based on the instruction information acquired from the detection operation specifying unit 117. Then, the image selection unit 120 transmits the image acquired from the image holding unit 119 to the enlargement / reduction unit 121. The enlargement / reduction unit 121 performs enlargement or reduction based on the instruction information acquired from the detection operation specifying unit 117. When the instruction information is not acquired, the enlargement / reduction unit 121 transmits the instruction information to the synthesis unit 122 without performing enlargement or reduction.

The combining unit 122 combines the image acquired from the drawing unit 118 and the image acquired from the enlargement / reduction unit 121 and transmits the combined image to the display unit 12.
Hereinafter, the immobilization calculation for the operation performed by the user will be described in detail.

[Fixed operation processing for user actions]
In the following description, it is assumed that the display device 1 is displaying video. Hereinafter, the operation in which the control unit 11 of the display device 1 performs the fixed value calculation on the operation performed by the user U1 and specifies the user's operation will be described with reference to the flowchart of FIG. FIG. 6 is a flowchart of the fixed positioning calculation according to the first embodiment of the present invention. In the flowchart of FIG. 6, the description will be made assuming that the motion sensor device 3 notifies the detected coordinate input unit 111 of the coordinate information.

  In step S100, the detected coordinate input unit 111 confirms the center position P0 (X0, Z0) of the detectable area 7 and the recognizable area. The detected coordinate input unit 111 notifies the hand position information detecting unit 112 of the confirmed detectable region 7 and the center position P0 (X0, Z0) of the recognizable region. In step S <b> 101 and step S <b> 102, the hand position information detection unit 112 detects information indicating the user's finger based on the coordinate information indicating the position of the object acquired from the detection coordinate input unit 111. The hand position information detection unit 112 repeats steps S101 and S102 until a finger is detected.

  In step S103, the hand position information detection unit 112 uses the information indicating the user's finger acquired from the detection coordinate input unit 111 for the finger detected in step S101 and step S102, such as the center of the wrist and the tip of the index finger. Coordinate information indicating the outer shape of the hand is detected. The hand position information detection unit 112 notifies the finger position information detection unit 113, the intrusion angle detection unit 114, and the motion vector detection unit 115 of coordinate information indicating the detected hand shape.

In step S <b> 104, the intrusion angle detection unit 114 determines the intrusion position P <b> 1 (X <b> 1, Z <b> 1) into the detectable region 7 based on the coordinate information regarding the user's finger acquired from the hand position information detection unit 112 and the detectable region 7. Detect and notify the coordinate converter 116.
In step S <b> 105, the finger position information detection unit 113 detects coordinates related to the tip of the finger based on the coordinate information related to the user's finger acquired from the hand position information detection unit 112, and notifies the motion vector detection unit 115.

In step S106, the motion vector detection unit 115 identifies the fingertip position P2 (X2, Z2) based on the coordinates relating to the tip of the finger acquired from the finger position information detection unit 113 in step S105, and sends it to the intrusion angle detection unit 114. Notice.
In step S107, the intrusion angle detection unit 114 receives the center position P0 (X0, Z0), the intrusion position P1 (X1, Z1), and the fingertip position P2 (X2, Z2) acquired by the operations from step S100 to step S106. Based on the above, an angle θ1 that is an intrusion angle of the user's finger is calculated. The intrusion angle detection unit 114 notifies the coordinate conversion unit 116 of the detected angle θ1.

  In step S108 and step S109, the motion vector detection unit 115 confirms the completion of the operation while the user moves the finger from the fingertip position P2 (X2, Z2) to the operation stop position P3 (X3, Z3). In step S109, when the motion has not been completed (No in S109), the motion vector detection unit 115 returns to step S108 and confirms the completion of the motion. Then, after confirming the completion of the operation (Yes in S109), the motion vector detection unit 115 calculates the motion vector L2, and notifies the coordinate conversion unit 116 of the motion vector L2.

  In step S110 and step S111, when the coordinate conversion unit 116 has not acquired the motion vector L2 from the motion vector detection unit 115, the coordinate conversion unit 116 acquires the fingertip position P2 (X2, Z2) and the motion stop position P3 (X3, Z3). To do. The coordinate conversion unit 116 acquires the operation start coordinates and the operation end coordinates by acquiring the fingertip position P2 (X2, Z2) and the operation stop position P3 (X3, Z3). Note that the coordinate conversion unit 116 may acquire the angle θ1 from the intrusion angle detection unit 114 in step S110 and step S111. When the coordinate conversion unit 116 acquires the angle θ1 in steps S110 and S111, the coordinate conversion unit 116 omits step S113.

In step S112, the coordinate conversion unit 116 calculates a motion vector L2 based on the motion start coordinates and motion end coordinates acquired in steps S110 and S111. When the coordinate conversion unit 116 has acquired the motion vector L2 in step S110 and step S111, step S112 is omitted.
In step S113, the coordinate conversion unit 116 acquires the angle θ1 from the intrusion angle detection unit 114. The coordinate conversion unit 116 omits step S113 when the angle θ1 is acquired in step S110 and step S111.

  In step S114, the coordinate conversion unit 116 calculates a coordinate-converted motion vector L3 based on the motion vector L2 and the angle θ1. The post-coordinate conversion motion vector L3 is the same as the operation when the user moves his / her finger while positioned in front of the display unit 12. Then, the coordinate conversion unit 116 notifies the detected motion specifying unit 117 of the post-coordinate conversion motion vector L3.

  In step S115, the detection operation specifying unit 117 specifies the operation of the user's finger based on the coordinate-transformed motion vector L3. In the present embodiment, the detection operation specifying unit 117 specifies that the user has operated the index finger for the distance indicated by L3 from the front of the display unit 12.

  In step S116, the detection operation specifying unit 117 notifies an instruction to another block based on the detected operation. When the operation for selecting a display image is specified, the detection operation specifying unit 117 notifies the image selecting unit 120 of image selection instruction information. When the operation for enlarging or reducing the display image is specified, the detection operation specifying unit 117 notifies the enlargement / reduction unit 121 of enlargement / reduction instruction information. The detection operation specifying unit 117 notifies the drawing unit 118 of a drawing instruction when an operation for drawing on the display image is specified.

  In step S117, the motion vector detection unit 115 determines whether or not the user continues to operate. When the user continues the operation (Yes in S117), the motion vector detection unit 115 returns to step S108 and confirms the operation. If the user does not continue the operation (No in S117), the motion vector detection unit 115 ends the detection operation.

[Effect of the first embodiment]
As described above, in the present embodiment, the detection operation specifying unit 117 of the display device 1 is positioned in front of the display unit 12 based on the angle of the user's finger that has entered the detectable region 7. It converts into the operation | movement of a case, and specifies a user's instruction | indication. By doing so, the detection operation specifying unit 117 of the display device 1 improves the probability that the gesture operation performed by the user can be specified regardless of the position of the user with respect to the display unit 12. Can do.

<Second Embodiment>
In the second embodiment, the control unit 11 of the display device 1 is based on the distance between the motion sensor device 3 and the display unit 12 and the angle at which the user's finger enters the recognizable region of the motion sensor device 3. The location indicated on the display unit 12 by the user's finger gesture is specified.

  FIG. 7 is a diagram for explaining the outline of the second embodiment of the present invention. The display device 1 further includes a distance measuring sensor 15. The control unit 11 of the display device 1 calculates an inter-sensor distance L4 that connects the motion sensor device 3 and the distance measuring sensor 15 with a straight line. The display unit 12 is located outside the detectable region 7 where the motion sensor device 3 recognizes the user's operation. The recognition outside distance L5 indicates the distance between the detectable region 7 and the display unit 12.

  As described above, when the display unit 12 is located outside the detectable region 7, the control unit 11 of the display device 1 cannot express the position indicated by the user with the coordinates inside the detectable region 7. Therefore, the control unit 11 of the display device 1 calculates the recognition outside distance L5 that is the distance between the detectable region 7 and the display unit 12, and calculates the external coordinates of the detectable region 7. The detection operation specifying unit 117 of the display device 1 according to the present embodiment calculates the recognition outside distance L5 based on the distance between the display unit 12 and the motion sensor device 3, and recognizes the recognition outside distance L5 to the coordinates inside the detectable region 7. Is added to specify the designated position indicated by the user.

[Calculation method for specifying the indicated position pointed to by the user]
The control unit 11 of the display device 1 performs an operation for detecting the position pointed to by the user by dividing the XYZ three-dimensional space into an XZ plane and a ZY plane. FIG. 8 is a diagram for explaining calculation in the XZ plane according to the second embodiment of the present invention. The control unit 11 obtains the recognition outside distance L5, the inter-sensor distance L4, and the angle θ2 in the XZ plane, and calculates the indicated coordinate error δ1. Then, the control unit 11 calculates the X coordinate of the user designated position based on the designated coordinate error δ1.

  First, the control unit 11 of the display device 1 calculates the inter-sensor distance L4 using the distance measuring sensor 15. Next, the control unit 11 calculates the distance measurement sensor entry position P4 (X4, Z4) based on the distance measurement sensor position P5 (X5, Z5), the inter-sensor distance L4, and the detectable region 7. Further, the control unit 11 calculates the angle θ2.

  The controller 11 calculates the recognition outside distance L5 based on the inter-sensor distance L4, the angle θ2, and the distance measurement sensor intrusion position P4 (X4, Z4). The control unit 11 calculates the designated coordinate error δ1 based on the recognition outside distance L5 and the angle θ1. The detection operation specifying unit 117 of the display device 1 calculates the user instruction position P7 (X7, Z7) by adding the instruction coordinate error δ1 and the recognition outside distance L5 to the recognition boundary intersection P6 (X6, Z6).

  FIG. 9 is a diagram for explaining calculation on the ZY plane according to the second embodiment of the present invention. The control unit 11 calculates Y7, which is the Y coordinate of the user-instructed position, based on the recognition outside distance L5 obtained by calculation on the XZ plane, the recognition boundary intersection Y6, and the fingertip position P2 (Z2, Y2). . Then, the control unit 11 calculates the user instruction position P7 (Z7, Y7) on the ZY plane. The detection operation specifying unit 117 sets the intersection point between the display device 1 and the line extended from the fingertip position P2 (X2, Y2, Z2), which is the tip of the object, in the direction in which the object moves, to the user specified position P7 (X7, Y7). , Z7).

The operation of the second embodiment will be described below using the block diagram of FIG. 10 and the flowchart of FIG.
FIG. 10 is a block diagram showing the configuration of the display device 1 according to the second embodiment of the present invention. The display device 1 according to the second embodiment further includes a distance measuring sensor 15, a device distance detection unit 123, and an indicated coordinate calculation unit 124, as compared with the display device 1 according to the first embodiment.

  The device distance detection unit 123 measures the first distance between the motion sensor device 3 and the display unit 12. Specifically, the device distance detection unit 123 uses the distance measuring sensor 15 to measure the inter-sensor distance L4 that is the first distance. The indicated coordinate calculation unit 124 is based on the angle θ1 and the angle θ2 acquired from the intrusion angle detection unit 114, the inter-sensor distance L4 acquired from the device distance detection unit 123, and the motion vector L2 acquired from the motion vector detection unit 115. To calculate the coordinates indicated by the user.

  The detection operation specifying unit 117 specifies the designated position based on the coordinates indicated by the user calculated by the designated coordinate calculating unit 124 based on the first distance measured by the device distance detecting unit 123. The intrusion angle detection unit 114 performs both the operation of the first embodiment and the operation of notifying the indicated coordinate calculation unit 124 of the inter-sensor distance L4 detected by the device distance detection unit 123.

  FIG. 11 is a flowchart of the fixed positioning calculation according to the second embodiment of the present invention. In the following description, operations different from those of the first embodiment will be described, and similar operations will be omitted as appropriate.

  In step S200, the device distance detection unit 123 acquires the distance measurement sensor position P5 (X5, Z5) and the coordinates of the distance measurement sensor 15 in order to calculate the inter-sensor distance L4 that is the distance to the motion sensor device 3. . The device distance detection unit 123 includes a straight line connecting the distance measurement sensor position P5 (X5, Z5) and the distance measurement sensor 15, and a straight line passing through the distance measurement sensor position P5 (X5, Z5) and parallel to the Z axis. The formed angle θ2 is calculated. Next, the device distance detection unit 123 calculates the recognition outside distance L5 by calculating the distance measurement sensor intrusion position P4 (X4, Z4) based on the detectable region 7.

  In step S201, the designated coordinate calculation unit 124 calculates the user designated position X7. First, the designated coordinate calculation unit 124 acquires the angle θ1 calculated based on the fingertip position P2 (X2, Z2) and the center line L1 of the user's finger from the intrusion angle detection unit 114, and recognizes the recognition outside distance L5 and tan θ1. Is added to calculate the designated coordinate error δ1. Next, the designated coordinate calculation unit 124 calculates the recognition boundary intersection P6 (X6, Z6) based on the fingertip position P2 (X2, Z2) of the user's finger and the detectable area 7, and the designated coordinate error δ1 is stored in X6. Is added to calculate the user instruction position X7.

  In step S202, the designated coordinate calculation unit 124 calculates the user designated position Y7. First, the designated coordinate calculation unit 124 indicates the tip of the user's index finger based on Z2 that is the Z-axis value of the fingertip position P2, Z6 that is the Z-axis value of the recognition boundary intersection P6, and the recognition outside distance L5. A ratio between the distance from the fingertip position P2 to the recognition boundary intersection P6 and the distance from the fingertip position P2 indicating the tip of the tip of the user's index finger to the display unit 12 is calculated. Next, the designated coordinate calculation unit 124 determines the user designated position Y7 based on Y2 which is the Y-axis value of the fingertip position P2 and Y6 which is the Y-axis value of the recognition boundary intersection P6 according to the distance ratio. calculate.

  The detection operation specifying unit 117 specifies the user's operation based on the user instruction position P7 (X7, Y7) calculated in step S201 and step S202 described above. Specifically, the detection operation specifying unit 117 includes the object specified based on the inter-sensor distance L4 and the position at which the object is detected in the detectable region 7 where the motion sensor device 3 can detect the object, and the display unit 12. The indicated position is specified based on the second distance. The second distance corresponds to the recognition outside distance L5. The detection operation specifying unit 117 notifies the drawing unit 118 of the specified pointing position. By doing in this way, the drawing unit 118 of the display device 1 can process the content instructed by the user through the gesture.

(Modification)
In the above description, the device distance detection unit 123 detects the inter-sensor distance L4, which is the first distance, using the distance measuring sensor 15, but the device distance detection unit 123 detects the distance between the object and the display unit 12. May be measured. The distance measuring sensor 15 includes, for example, an infrared distance measuring sensor, and detects the distance and azimuth of the object with respect to the display unit 12. The device distance detection unit 123 functions as a measurement unit that measures the distance between the object and the display unit 12 based on the distance and direction of the object detected by the distance measuring sensor 15. The detection operation specifying unit 117 specifies the designated position based on the distance between the object specified by the device distance detecting unit 123 and the display unit 12.

[Effects of Second Embodiment]
As described above, the control unit 11 of the display apparatus 1 performs the detection operation regardless of the position of the motion sensor device 3 and the user by performing the calculation in consideration of the distance between the motion sensor device 3 and the display unit 12. The specifying unit 117 can specify the designated position indicated by the user. By doing in this way, the user can perform control by gesture irrespective of an apparatus or a user's position. For example, the user can enlarge or reduce the image displayed on the display unit 12 of the display device 1 around the intended coordinates.

<Third Embodiment>
In the third embodiment, the control unit 11 of the display device 1 calculates the coordinates indicated by the user based on the angle at which the user's finger enters the recognizable region of the motion sensor device 3, or always has a fixed position. It is determined whether or not the operation is converted into operation. For example, when the angle at which an object such as a finger enters exceeds a threshold, the control unit 11 of the display device 1 calculates the coordinates indicated by the user based on the moving direction of the object, or converts the movement to a fixed position. Judge whether to do.

  FIG. 12 is a diagram for explaining the outline of the third embodiment of the present invention. In FIG. 12, it is assumed that the user U <b> 2 is located in front of the display device 1 and the user U <b> 1 is located in the vicinity of the display device 1. Then, it is assumed that the user U1 points to the area 8a and the user U2 points to the area 8b with an index finger and performs an operation of changing the indicated distance. At this time, the user U1 moves the finger in the vertical direction, that is, the Z-axis direction with respect to the display unit 12 with the fingertip not directed toward the display unit 12 of the display device 1. On the other hand, the user U2 can perform the same operation as the user U1 by moving the finger in the vertical direction, that is, in the Z-axis direction with respect to the display unit 12 with the fingertip directed toward the display unit 12 of the display device 1. is assumed. Therefore, the control unit 11 of the display device 1 performs the fixed positioning calculation described in the first embodiment and the second embodiment according to the relationship between the position of the user and the position of the display unit 12. judge.

  FIG. 13 is a diagram for explaining a difference in designated position according to the third embodiment of the present invention. When the user performs the operation L6 shown in FIG. 13, the detection operation specifying unit 117 of the display device 1 does not specify the user-specified position P7 (X7, Y7, Z7) but the user-specified position P8 (X2, Y2). , Z7).

  When it is determined that the fixed positioning calculation is to be performed, the detection operation specifying unit 117 determines that the object moves in the direction in which the object moves starting from the tip position of the object when the moving direction is the region 8a that is the first range. The user designated position P7 (X7, Y7, Z7) that is the intersection of the extended line and the display unit 12 is specified as the designated position. In addition, when the moving direction of the object is the region 8b that is the second range different from the first range, the detection operation specifying unit 117 extends in the direction orthogonal to the display unit 12 with the tip position of the object as a starting point. The intersection of the line and the display unit 12 is specified as the user instruction position P8 (X2, Y2, Z7). The detection operation specifying unit 117 is orthogonal to the display unit 12 with the tip position of the object as the starting point when the line extending in the moving direction of the object from the tip position of the object does not intersect the display area of the display unit 12. The intersection of the line extended in the direction and the display unit 12 is specified as the designated position.

The control unit 11 of the display device 1 according to the present embodiment calculates an area indicated by the user using the method described in the second embodiment, and determines whether the area indicated by the user is inside the display area of the display unit 12. Based on this, it is determined whether or not to perform the fixed positioning calculation. The user can set an area for determining whether or not to perform the fixed positioning calculation. For example, the user specifies an arbitrary area in the display area of the display unit 12 and sets so as not to perform the fixing operation when pointing to the inside of the area specified by the user.
Hereinafter, the configuration and operation of the display device 1 according to the third embodiment will be described with reference to the block diagram of FIG. 14 and the flowcharts of FIGS. 15 and 16.

[Configuration and Operation of Display Device 1 According to Third Embodiment]
FIG. 14 is a block diagram showing the configuration of the display device 1 according to the third and fourth embodiments of the present invention. The display device 1 further includes a calculation method determination unit 125. The calculation method determination unit 125 determines whether or not to perform a fixed positioning calculation.

  The calculation method determination unit 125 acquires an angle θ1 that is an intrusion angle of the user U1's finger from the intrusion angle detection unit 114 when the user determines an area for determining whether or not to perform the fixed positioning calculation. Based on the angle θ1, the calculation method determination unit 125 determines whether or not the indicated position indicated by the user points to the inside of the area specified by the user. Then, the calculation method determination unit 125 determines whether or not to perform the fixed positioning calculation based on whether or not the designated position indicates the inside of the region.

  When the calculation method determination unit 125 determines based on whether or not the region pointed to by the user is inside the display region of the display unit 12, the user instruction in FIG. The position P8 (X2, Y2, Z7) and the user instruction position P7 (X7, Y7, Z7) are acquired. Then, when the user instruction position P7 (X7, Y7, Z7) is in the display area of the display unit 12, the calculation method determination unit 125 uses the user instruction position P7 (X7, Y7, Z7) as coordinates indicated by the user. adopt. When the user instruction position P7 (X7, Y7, Z7) is outside the display area of the display unit 12, the calculation method determination unit 125 adopts the user instruction position P8 (X2, Y2, Z7) as coordinates indicated by the user. To do.

  Further, the calculation method determination unit 125 notifies the coordinate conversion unit 116 and the detection operation specifying unit 117 that the fixed positioning calculation is not performed when the user designated position P8 (X2, Y2, Z7) is adopted. The calculation method determination unit 125 notifies the coordinate conversion unit 116 and the detection operation specifying unit 117 to perform the same coordinate conversion as in the second embodiment when the user designated position P7 (X7, Y7, Z7) is adopted. To do.

FIG. 15 is a flowchart (1) of the fixed positioning calculation according to the third embodiment of the present invention. FIG. 16 is a flowchart (2) of the fixed positioning calculation according to the third embodiment of the present invention. In the following description, an operation different from the operation of the first embodiment and the operation of the second embodiment will be described, and the same operation will be omitted as appropriate.
Hereinafter, the operation of the display device 1 according to the third embodiment will be described with reference to FIGS. 15 and 16.

  In step S300, the calculation method determination unit 125 confirms the determination mode. The determination mode can be selected by the user using a menu or the like.

  In step S301, the calculation method determination unit 125 calculates the angle θ1 acquired from the intrusion angle detection unit 114, the user instruction position P7 (X7, Y7, Z7) and the user instruction position P8 (X2, X7) acquired from the instruction coordinate calculation unit 124. Based on Y2, Z7), the user-instructed position is adopted and the fixed position calculation is determined.

  In step S302, the calculation method determination unit 125 determines the user instruction position based on the determination result performed in step S301. Based on the determination result, the calculation method determination unit 125 notifies the coordinate conversion unit 116 and the detection operation specifying unit 117 whether or not to perform the fixed positioning calculation. The operations from step S108 to step S112 are the same as those in the first embodiment and the second embodiment, and are therefore omitted.

  In step S303, the coordinate conversion unit 116 determines whether or not to perform the fixed positioning calculation based on the determination result acquired from the calculation method determination unit 125 in step S302. When the user-designated position is within the display area of the display unit 12 and the fixed position calculation is performed (Yes in S303), the coordinate conversion unit 116 proceeds to step S113. If the user instruction position is outside the display area of the display unit 12 and the fixed positioning calculation is not performed (No in S303), the coordinate conversion unit 116 proceeds to Step S115.

  In step S113, the detection operation specifying unit 117 indicates that the area pointed by the user's finger is not a place where the tip of the user's finger is extended, but a place where the user's finger is extended in the vertical direction of the display device, that is, in the Z-axis direction. Specify as location. Hereinafter, the operations from step S114 to step S117 are the same as those in the first embodiment and the second embodiment, and therefore will be omitted.

[Effect of the third embodiment]
As described above, in the present embodiment, the control unit 11 of the display device 1 determines the relationship between the position of the display unit 12 and the position of the user in the area indicated by the user's finger. Then, when the user is present at a close position such as right next to the display unit 12, the control unit 11 determines that the user's gesture operation is the same as the operation when the user is located in front. By doing in this way, control part 11 can carry out judgment of gesture operation which does not give a sense of incongruity to a user, even if it exists in a position where it is assumed that the user does not point display part 12 from diagonally. .

<Fourth Embodiment>
In the fourth embodiment, the control unit 11 of the display device 1 determines which operation is adopted when a plurality of users are present in the recognition area of the motion sensor device 3 and perform a gesture operation. The arbitration process is performed. Hereinafter, the arbitration process performed by the control unit 11 of the display device 1 will be described with reference to the block diagram of FIG. 14, the flowcharts of FIGS. 17, 18, and 19. In the present embodiment, the motion vector detection unit 115 stores the operation start time and the operation completion time of the specified operation.

  FIG. 17 is a flowchart (1) showing a process when a plurality of persons operate simultaneously according to the fourth embodiment of the present invention. FIG. 18 is a flowchart (2) showing a process when a plurality of persons operate simultaneously according to the fourth embodiment of the present invention. In the following description, an operation different from the operation of the first embodiment, the operation of the second embodiment, and the operation of the third embodiment will be described, and the same operation will be omitted as appropriate.

  In step S400, when the motion vector detection unit 115 specifies motion vectors of a plurality of persons at the same time, the motion vector detection unit 115 determines whether to adopt each motion vector using a flowchart of FIG. In step S401, the motion vector detection unit 115 returns to step S110 if there is a motion vector to be employed, and returns to step S108 if there is no motion vector to be employed.

  FIG. 19 is a flowchart illustrating an arbitration process when a plurality of persons operate simultaneously according to the fourth embodiment of the present invention. The motion vector detection unit 115 performs an arbitration process for all the motion vectors specified by the motion vector detection unit 115 in step S400 described above.

  In step S411, the motion vector detection unit 115 determines whether or not the identified motion order is first. Specifically, the motion vector detection unit 115 searches the order of the stored operation start times, and determines whether or not the order of the operation start times is first. The motion vector detection unit 115 proceeds to step S412 when the order of operations is first, and proceeds to step S416 when the order of operations is not first.

  In step S412, the motion vector detection unit 115 determines whether there is another motion vector having the same rank. Specifically, the motion vector detection unit 115 determines whether there are motion vectors having the same operation start time. If there is no other motion vector of the same order, that is, if it is determined that the motion vector has the earliest motion start, the motion vector detection unit 115 proceeds to step S415, and if there is another motion vector of the same order. The process proceeds to step S413.

  In step S413, the motion vector detection unit 115 searches for the order of operation completion times. The motion vector detection unit 115 determines that the operation has been performed to the end when the order of the operations is the lowest or when the identified operation continues the operation, and proceeds to step S414. The motion vector detection unit 115 proceeds to step S415 when the order of operations is not the lowest.

  In step S414, the motion vector detection unit 115 employs the motion vector subjected to the arbitration process. Then, the motion vector detection unit 115 notifies the designated coordinate calculation unit 124 of information on the adopted motion vector. In step S415, the motion vector detection unit 115 deletes the motion vector that has undergone the arbitration process without adopting the motion vector.

  In step S115, when the motion vector detection unit 115 specifies the direction in which the plurality of objects move, the detection operation specifying unit 117 determines the movement based on the direction in which the selected object moves based on the order in which the plurality of objects start moving. Identify the indicated position. Specifically, the detection operation specifying unit 117 specifies the indicated position based on the moving direction of the object that has started moving the earliest among the plurality of objects. For example, when there are a plurality of objects that have started to move most quickly, the detection operation specifying unit 117 specifies the designated position based on the direction in which the object that has continued to move the longest among the plurality of objects moves. Specifically, the detection operation specifying unit 117 specifies the designated position based on the moving direction of the object that moved last among the plurality of objects moving simultaneously.

[Effect of the fourth embodiment]
As described above, the motion vector detection unit 115 of the display device 1 according to the present embodiment uses the user's operation start time and operation completion time when a plurality of users exist in the recognition area of the motion sensor device. Based on this, it is determined whether or not the gesture operation is adopted. The motion vector detection unit 115 of the display device 1 employs the gesture with the earliest start of operation and erases the second and subsequent gestures, so that it is easy to determine which of the plurality of users has adopted the gesture. Can be determined. In addition, when the user performs gesture operations at the same time, the motion vector detection unit 115 can perform mediation processing that does not give the user a sense of incongruity by adopting the longer gesture operation.

  In the above, it demonstrated based on embodiment of this invention and some embodiment. New embodiments resulting from any combination of these are also included in the embodiments of the present invention. The effect of the new embodiment produced by the combination has the effect of the original embodiment. The technical scope of the present invention is not limited to the scope described in the above embodiment, and various modifications and changes can be made within the scope of the gist.

  For example, in the above description, the method in which the display device 1 specifies the indicated position by performing the rotation process of the motion vector corresponding to the gesture performed by the user has been described. However, the indicated position is specified based on the motion vector. The method to do is not limited to this. For example, the display device 1 may specify the position where the extension line of the motion vector and the display unit 12 intersect as the designated position.

DESCRIPTION OF SYMBOLS 1 ... Display apparatus 12 ... Display part 115 ... Motion vector detection part 117 ... Detection operation | movement specific | specification part 118 ... Drawing part 123 ... Device distance detection part

Claims (18)

Display means for displaying an image;
Direction specifying means for specifying the direction in which the object moves on the display surface side of the display means;
Position specifying means for specifying an indicated position which is a position pointed to by the object on the display means based on a moving direction of the object specified by the direction specifying means;
A display device comprising: The position specifying means specifies the indicated position based on an angle of a moving direction of the object with respect to a screen of the display means.
The display device according to claim 1. The position specifying means specifies, as the indicated position, an intersection between a line extending in the direction in which the object moves from the tip of the object and the display means,
The display device according to claim 1. The object passes through the center position of the display means and is perpendicular to the display means, and the position where the straight line in the direction in which the object moves intersects the rotation center. Further comprising conversion means for converting the direction of movement of the object by rotating the direction of movement;
The position specifying means specifies, as the indicated position, an intersection between a direction in which the object moves after the conversion means converts and the display means.
The display device according to claim 3. When the moving direction of the object is in the first range, the position specifying means uses an intersection of a line extending in the moving direction of the object from the tip position of the object and the display means as the indicated position. When the direction in which the object moves is a second range different from the first range, a line extended in a direction perpendicular to the display means from the tip position of the object and the display means An intersection point is specified as the indicated position,
The display device according to claim 3 or 4. The position specifying means is orthogonal to the display means starting from the tip position of the object when a line extending in the moving direction of the object from the tip position of the object does not intersect the display area of the display means A point of intersection of the line extended in the direction and the display means is specified as the indicated position,
The display device according to claim 5. The position specifying means specifies the indicated position based on a motion vector of the object starting from a position where the object enters a detectable region where the movement of the object can be detected.
The display device according to any one of claims 1 to 6. The direction specifying means specifies an outer shape of at least a part of the object;
The position specifying means specifies the indicated position based on a direction in which an object of the specified outer shape moves,
The display device according to claim 1. The image processing apparatus further comprises processing execution means for executing processing corresponding to the designated position specified by the position specifying means for the image displayed on the display means.
The display device according to claim 1. A measuring means for measuring a first distance between the object detecting means for detecting the position of the object on the display surface side of the display means and the display means;
The position specifying means specifies the indicated position further based on the first distance measured by the measuring means;
The display device according to any one of claims 1 to 9. The position specifying means specifies the object and the display specified based on the first distance measured by the measuring means and a position where the object detecting means detects the object in a detectable region where the object can be detected. The pointing position is specified based on a second distance from the means;
The display device according to claim 10. A measuring unit for measuring a distance between the object and the display unit;
The position specifying means specifies the indicated position further based on the distance measured by the measuring means;
The display device according to any one of claims 1 to 9. The position specifying means, when the direction specifying means specifies the direction in which the plurality of objects move, based on the direction in which the selected object moves based on the order in which the plurality of objects started moving, Characterized by specifying
The display device according to any one of claims 1 to 12. The position specifying means specifies the indicated position based on a direction in which an object that has started moving most quickly among the plurality of objects moves.
The display device according to claim 13. The position specifying means specifies the indicated position based on a moving direction of an object that has continued to move the longest among the plurality of objects when there are a plurality of objects that have started moving the earliest. ,
The display device according to claim 14. The position specifying means specifies the indicated position based on a moving direction of an object that moved last among the plurality of objects moving simultaneously,
The display device according to claim 13. An instruction detection device that detects an instruction to a display device that displays an image,
Direction specifying means for specifying a direction in which an object moves on the display surface side of the display device;
Position specifying means for specifying an indicated position, which is a position pointed to by the object in the display device, based on a moving direction of the object specified by the direction specifying means;
Transmitting means for transmitting the indicated position specified by the position specifying means to the display device;
An instruction detection device comprising: The computer runs,
Identifying the direction in which the object moves on the display surface side of the display device;
Identifying a pointing position that is a position pointed to by the object on a display means for displaying an image based on a moving direction of the object;
Processing an image to be displayed on the display means based on the indicated position;
An image display method characterized by comprising:

Images