JP2013171529A - Operation input device, operation determination method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an operation input device capable of accurately determining an input operation without requiring an operation of causing a finger to directly touch on a touch panel or the like and hence with less burden imposed on an operator.SOLUTION: An input operation device acquires a position of a finger of a right hand 103 with a camera 201, which is front surface imaging means provided in a portable phone 101, and sets, in the air in the vicinity of the finger, an operation area which corresponds to a screen 203 of the portable phone 101, so as to cause a cursor on the screen 203 to be moved correspondingly to a position of the finger of an operator in the operation area, or to cause an icon to be highlighted and specified. For example, in Fig. 2, an operation of the operator is focused on an icon 205. In this state, when a decision button 202 is depressed by a finger of a left hand 102, for example, an operation indicated by the currently focused icon is decided and executed.

Description

  The present invention relates to an operation input device, an operation determination method, and a program. More specifically, the present invention relates to an operation input device, an operation determination method, and a program that determine an operation of an operator from an image captured by a camera or the like.

  When inputting data in a portable terminal including a computer or a mobile phone, a game machine, or the like, a keyboard, a touch panel, a mouse, or the like is usually used. In particular, devices such as smartphones, tablet PCs, and portable game machines that have been widely used in recent years can be operated intuitively by providing a touch panel function (see, for example, Patent Document 1).

  In addition, with the aim of further improving convenience related to the operation of the operator without wearing special equipment, the operator is provided with an operation means corresponding to a keyboard, a mouse, etc. For example, by defining a virtual plane G or a virtual keyboard H, an operator can perform a click operation, a drag operation, a double click operation, or the like as a mouse operation on a display image, a desired line, figure, Alternatively, a technique that enables drawing and inputting of characters and the like has been proposed (see, for example, Patent Document 2).

Special table 2009-522669 JP 2004-258714 A

  However, when a device such as a touch panel is used, the finger directly touches the touch panel when operating with the finger. Therefore, the screen is permanently soiled due to adhesion of fingerprints or oil, and the field of view is hidden by the operating finger. There are problems such as obstruction and frictional stress on the fingertip during the drag operation (mainly resistance film pressure sensitive type). Moreover, since it is necessary to provide a touch panel sensor, a touch panel cost is required, and there is a problem that a glove with a dedicated conductive property is required for use of a capacitive touch panel of a mainstream technology in a smartphone or the like.

  Further, when the technique of the cited document 2 is used, an input operation can be performed in a non-contact manner, but a large apparatus such as an image is projected on a screen, so that the cost is increased or the structure is enlarged. There is a problem.

  The present invention has been made in view of such a problem, and does not require an operation of directly touching a finger such as a touch panel, and can accurately determine an operation input with less burden on the operator. An object of the present invention is to provide an operation determination method and a program.

  In order to achieve such an object, the invention according to claim 1 is an operation input device, which is disposed in front of the display means, the display means for displaying an image viewed by the operator, and the display means. Front imaging means for imaging a part of the operator on the front surface, and extracting a part of the operator on the front surface of the display means from the image read by the front imaging means, and displaying the operator part on the display means Display means according to the positional relationship with the display means, and a front control means for moving the sign in response to a part of the operator's movement, and a display means for the sign arranged at any position of the device housing And a signal output means for outputting a predetermined signal when the operation is confirmed by the confirmation means.

  The invention according to claim 2 is an operation input device, wherein a display means for displaying an image viewed by the operator, and a part of the operator located on the back of the display means are imaged on the back of the display means. A rear imaging unit that extracts the part of the operator on the rear side of the display unit from the image read by the rear imaging unit, and displays on the display unit according to the positional relationship between the part of the operator and the display unit And a rear control means for moving the sign in response to a partial movement of the operator, and an operation related to the position on the display surface of the sign display means arranged at any position on the device housing And a signal output means for outputting a predetermined signal when an operation is confirmed by the confirmation means.

  According to a third aspect of the present invention, in the operation input device according to the second aspect of the present invention, a front imaging unit that is disposed in front of the display unit and images a part of an operator in front of the display unit, and a front imaging unit A part of the operator on the back of the display means is extracted from the image read by the display means, and a display according to the positional relationship between the part of the operator and the display means is displayed on the display means. It is further characterized by further comprising front surface control means for moving the marking in response to the movement of.

  According to a fourth aspect of the present invention, in the operation input device according to the third aspect, the region is defined by the operator's image read by the front imaging means, and a part of the operator is displayed on the display means. It is further characterized by further comprising area setting means for presetting a control area capable of controlling the predetermined sign.

  According to a fifth aspect of the present invention, in the operation input device according to the fourth aspect, the control area is an area that does not cover an area defined by connecting the viewpoint of the operator and each vertex of the display surface of the display means. It is characterized by.

  The invention according to claim 6 is an operation input device, which is a display means for displaying an image viewed by the operator, and is arranged in front of the display means and images a part of the operator in front of the display means. A front imaging unit that extracts the part of the operator on the back of the display unit from the image read by the front imaging unit, and displays on the display unit the position corresponding to the positional relationship between the part of the operator and the display unit And a control means for moving the sign corresponding to the movement of a part of the operator, and a part of the operator located on the back of the display means, A confirmation means for confirming an operation related to a position on the display surface of the display means of the sign when performed, and a signal output means for outputting a predetermined signal when the operation is confirmed by the confirmation means, To do.

  The invention described in claim 7 includes display means for displaying an image viewed by the operator, and front image pickup means that is arranged in front of the display means and images a part of the operator in front of the display means. A method for performing an operation input using the operation input device, wherein a part of the operator in front of the display means is extracted from the image read by the front imaging means, and the operator part and the display means are displayed on the display means. The sign is displayed according to the front control step for moving the sign in response to the movement of a part of the operator and the confirming means arranged at any position of the device housing. A confirmation step for confirming an operation related to a position on the display surface of the display means, and a signal output step for outputting a predetermined signal when the operation is confirmed in the confirmation step are provided.

  According to an eighth aspect of the present invention, there is provided display means for displaying an image viewed by the operator, and rear image pickup means that is arranged on the back surface of the display means and images a part of the operator on the back surface of the display means. A method of performing an operation input using the operation input device, wherein a part of the operator on the back of the display means is extracted from the image read by the back imaging means, and a part of the operator and the display means are displayed on the display means. The display is displayed in accordance with the positional relationship between the back and the rear control step for moving the sign in response to part of the movement of the operator, and the confirmation means arranged at any position of the device housing. A confirmation step for confirming an operation related to a position on the display surface of the display means, and a signal output step for outputting a predetermined signal when the operation is confirmed in the confirmation step are provided.

  The invention according to claim 9 is a display means for displaying an image viewed by the operator, a front image pickup means that is arranged in front of the display means and images a part of the operator in front of the display means, and a display A method of performing an operation input by an operation input device provided with a rear imaging means for imaging a part of an operator on the back of the means, the operator located on the back of the display means from an image read by the front imaging means A front control step of extracting a part of the display, displaying a sign according to the positional relationship between the part of the operator and the display means on the display means, and moving the sign in response to the movement of the part of the operator; A confirmation step of confirming an operation related to the position of the display means on the display surface when a predetermined operation is performed using a part of the operator on the back surface of the display means, which is imaged by the back imaging means And the operation is confirmed in the confirmation step. When, characterized in that a signal output step of outputting a predetermined signal.

  According to a tenth aspect of the present invention, there is provided display means for displaying an image browsed by an operator, front image pickup means arranged on the front face of the display means for picking up a part of the operator on the front face of the display means, and apparatus A program for causing an operation input device including a determination unit arranged at any position of a housing to execute an operation input method, the operation input method being based on an image read by a surface imaging unit The operator is extracted in part, and a display according to the positional relationship between the part of the operator and the display is displayed on the display means, and the sign is moved in response to the movement of the part of the operator. A front control step, a confirmation step for confirming an operation related to the position on the display surface of the display means by the confirmation means, and a signal output step for outputting a predetermined signal when the operation is confirmed in the confirmation step. Characterized by To.

  According to the eleventh aspect of the present invention, there is provided display means for displaying an image browsed by an operator, confirmation means arranged at any position of the apparatus housing, and a part of the operator on the back of the display means. A program for causing an operation input device having a rear imaging means to perform an operation input method to execute an operation input method, wherein the operation input method is to select a part of an operator on the back of the display means from an image read by the rear imaging means. A back control step for extracting, displaying a sign according to the positional relationship between a part of the operator and the display means on the display means, and moving the sign corresponding to the movement of the part of the operator; and a confirmation means And a confirmation step for confirming an operation related to the position on the display surface of the display means of the sign, and a signal output step for outputting a predetermined signal when the operation is confirmed in the confirmation step. .

  According to a twelfth aspect of the present invention, there is provided display means for displaying an image to be viewed by an operator, front image pickup means that is disposed in front of the display means and images a part of the operator in front of the display means, and display A program for causing an operation input device to perform an operation input method with a back imaging means for imaging a part of an operator on the back of the means, the operation input method being displayed from an image read by the front imaging means A part of the operator on the back of the means is extracted, a display according to the positional relationship between the part of the operator and the display means is displayed on the display means, and a sign corresponding to the movement of the part of the operator is displayed. The front control step for moving the image and the position of the indication on the display surface of the display means when a predetermined operation is performed using a part of the operator on the back of the display means, which is imaged by the back imaging means A confirmation step for confirming the operation to be performed. Program, characterized in that was.

It is a figure for demonstrating an example of the operation input system of this embodiment. It is a figure which shows typically operation | movement of the operation input system of this embodiment. It is a figure which shows typically operation | movement of the operation input system of this embodiment. It is a figure for demonstrating recognition of the fingertip of this embodiment. It is a block diagram which shows the hardware constitutions of the input device concerning one Embodiment of this invention. It is a functional block diagram for implement | achieving the input method concerning one Embodiment of this invention. It is a flowchart of the process of this embodiment. It is a figure for demonstrating correction | amendment of the image in the finger recognition concerning one Embodiment of this invention. It is a figure which shows the operation method of another one Embodiment of this invention. It is a figure which shows the operation method of another one Embodiment of this invention. It is a figure which shows the operation method of another one Embodiment of this invention. It is a figure which shows one Embodiment which applied this invention to operation of a personal computer. It is a figure which shows one Embodiment which applied this invention to operation of an automatic teller machine. It is a figure which shows the example which uses the out camera as a confirmation button in one Embodiment of this invention. It is a figure which shows the example which uses the out camera as a confirmation button in one Embodiment of this invention. It is a figure which shows the example of the operation input system concerning another one Embodiment of this invention. It is a figure which shows the example of the operation input system concerning another one Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the operation input apparatus concerning this embodiment. It is a figure which shows the operation method of another one Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a diagram for explaining an example of an operation input system according to the present embodiment. In this embodiment, the operator's left hand 102 holds the mobile phone 101 as the operation input means, and the mobile phone 101 is operated with the finger of the right hand 103. Of course, the mobile phone is held with the right hand and operated with the left hand. You can also. As shown in FIG. 1, the operator can specify an icon displayed on the screen or move the cursor while keeping the finger of the right hand 103 floating in the air without touching the screen of the mobile phone 101. Can do. On the other hand, the left hand 102 holds the mobile phone, and presses a confirmation button arranged at an arbitrary position of the mobile phone 101 with the finger, thereby confirming the operation and position specified with the finger of the right hand 103. In this way, the mobile phone 101 can be operated.

  FIG. 2 is a diagram schematically showing the operation of the operation input system of the present embodiment. With reference to FIG. 2, the operation principle of this embodiment will be described. The position of the finger of the right hand 103 is acquired by the camera 201 which is a front surface imaging means provided in the smartphone or the mobile phone 101, and the operation area is placed in the air so as to correspond to the screen 203 of the mobile phone 101 near the position of the finger. When the finger is moved in accordance with the setting and the position of the operator's finger in the operation area, the cursor on the screen 203 is moved or the icon is highlighted and designated. For example, in FIG. 2, the operator's operation is focused on the icon 205. In this state, when the confirm button 202 is pressed with the finger of the left hand 102, for example, the operation indicated by the currently focused icon is confirmed and executed. For example, when the cursor 205 is an icon of an application, the application is executed by confirming the operation.

  Furthermore, in addition to cursor movement and icon selection, operations that can be normally performed with a touch panel or a mouse, such as dragging, can be performed with the hand that does not confirm the operation. Specifically, the same operation as dragging with the mouse can be performed by continuously pressing the confirm button 202 with the cursor placed on the desired icon, moving the icon to the desired position, and then speaking the confirm button 202. However, the present invention is not limited to this, and any gesture can be used according to the technology in this technical field. In addition, for example, as shown in FIG. 19, the display on the screen can be sucrose linked with a so-called flip gesture that shakes the palm up, down, left, and right, and the object to be moved is displayed instead of a single object. It can be a group unit or it can move smoothly by applying inertia.

  In this embodiment, a so-called in-camera built in the mobile phone 101 is used as the camera 201. However, the present invention is not limited to this, and any camera that can capture a part of the operator, such as an external simple camera, is used. For example, various imaging devices can be used. In addition, the confirmation button 202 can recognize the confirmation operation without using the camera 201 such as using one of the keyboards or using a specific touch sensor according to the configuration of the device or system. Any one can be used.

  That is, in this embodiment, a desired operation is specified by the position and movement of the finger by recognizing the position of the operator's finger using a camera built in the apparatus or a camera that is easily attached, and is provided separately. By confirming the operation with the confirmed key or button, it is possible to input the operation easily and surely without contact. In this embodiment, it is described that the operation is specified by the operator's finger. However, the present invention is not limited to this, and any part can be used as long as it is a part of the operator's body. Alternatively, a worn instrument can be included as part of the operator. As shown in FIG. 3, the mobile phone 101 can be operated sideways. In the example of the operation input shown in FIG. 3, the position of the finger of the operator's hand 103 is specified by the camera on the right side of the apparatus, the position and movement of the finger are determined based on the operation area 301, and the icon 302 of the corresponding position is determined. Focus by zooming in. In this case, the XY coordinates of the gesture operation determination by the camera are changed in accordance with the screen display of the target screen by determining the vertical holding and the horizontal holding corresponding to the sensor built in the terminal. For this reason, it is possible to accurately recognize the tip of the finger by cutting out a moving image acquired by camera shooting to an aspect ratio that roughly matches the operation target screen.

  Further, as shown in FIG. 9, when the operator operates by holding the lower part of the mobile phone, the confirmation button 901 can be provided at the lower part of the apparatus.

  Here, the work area formed by the virtual plane that serves as a reference for recognizing the position and movement of the finger does not obstruct the screen displayed on the screen 203 even if the operator moves his / her hand in the operation input operation. Set to ensure visibility during the operation. For example, depending on the operation environment, as shown in FIG. 10 and FIG. 11, an operation area is set so as not to interrupt the line of sight of the operator's screen, so that a desired operation can be performed by moving a finger while checking the display on the screen. Can be identified. In general, the operation area is generally located at the upper part of the target screen in the same area. However, if the target screen is small, it is desirable that the operation area is larger, and conversely if the operation target screen is large. It is easier to operate by reducing the determination image area. Therefore, in this embodiment, an optimum operation determination area is set and recorded in advance in the usage environment, or a dedicated position designation calibration sheet is first captured by the camera, or one hand is used. You can also give instructions.

  FIG. 4 is a diagram for explaining fingertip recognition according to the present embodiment. An operator's finger in the predetermined or set gesture control area is captured by the camera 201 to generate a finger image. As shown in FIG. 4, a finger contour image is extracted from the generated finger image, and the tip and bone are recognized by binarizing the image. Here, any method known in this technical field can be used for the recognition of the contour and the tip. Based on the recognized position and movement of the tip of the finger, the cursor on the screen is moved or an icon is designated.

  Further, when recognizing the tip of the finger, it is necessary to consider distortion of the finger image. FIG. 8 is a diagram for explaining image correction in finger recognition according to the present embodiment. Since the camera of this embodiment is installed at the upper end or the right end of the mobile phone 101 as described above, the image of the operator's finger has a certain distortion as shown in FIG. 8 in addition to the normal lens distortion. May occur. For example, an image having a barrel aberration as shown in FIG. 8A or an image having a shrinkage aberration as shown in FIG. 8B, before and after extracting the finger contour image. , (C) so that the trajectory of the movement operation such as dragging in the work area matches the actual operation.

(System configuration of this embodiment)
FIG. 5 is a block diagram showing a hardware configuration of the input device according to the embodiment of the present invention. The CPU 501 is connected to a display 203, a camera 201, and a confirmation button 202, and inputs / outputs images and inputs signals. Although not shown here, the functions of a normal device, for example, a mobile phone, a telephone and communication function, and a personal computer include a storage device such as a hard disk, a mouse, etc. Only the important functions are described.

  For the image obtained by photographing with the camera 201, the CPU 501 extracts the image of the finger of the right hand that is a part of the operator, which is a feature of the present embodiment. The mobile phone 101 of this embodiment generally includes a CPU 501, executes a program stored in the ROM 502 or the like on the RAM 503, and outputs a processing result based on an image input from the camera 201 to the display 203 or the like. In the present embodiment, the display 203 can output various images provided by various applications that the operator wants to experience, in addition to the icon focused by the operation of the operator's finger described above, It is also possible to display information for assisting operation input.

  FIG. 6 is a block diagram illustrating an example of a functional module of a program processed in the CPU 501 of the mobile phone 101 according to the present embodiment. As shown in FIG. 6, the processing in this system is executed by an image reading unit 601, an image extraction unit 602, an image position calculation unit 603, an operation determination unit 604, and a sign control unit 605. In this embodiment, the processing from receiving an image from the camera 203 to outputting data is executed by these five modules. However, the present invention is not limited to this, and other modules may be used. It can be processed with few modules.

(Operation determination process of this embodiment)
The operation determination process of this embodiment will be described with reference to FIGS. FIG. 7 is a flowchart of the processing of this embodiment. The position of the finger of the right hand 103 that is part of the operator photographed by the camera 203 or the like is determined, and the cursor on the display is determined based on the relative position and shape of these positions and corresponding points on the display. Move the icon, identify the icon, or determine that the operation is constant. First, data captured by the camera 203 or the like is read in the image reading unit 601 in FIG. 6, and an image of the operator's finger is extracted from the data by the image extracting unit 602 (S701 and 702). Here, it is also possible to read out the position and size of an effective motion area previously stored in the storage means or the like and use only the movement of the finger within the area. After the finger is extracted, the sign is displayed by displaying the cursor at the position on the display 203 corresponding to the position of the finger or by focusing the icon by any method known in the art. The display is controlled by the control unit 605 and controlled to move with the movement of the finger. Therefore, the operator can specify the operation by controlling the image displayed on the display surface by moving the finger according to the movement of the displayed cursor or the display form of the icon (S703).

  When the operation determination unit 604 recognizes that the operator has pressed the confirm button 202, the operation determination unit 604 determines that the operation specified in step S703 has been performed, and notifies the CPU that the operation has been executed together with the specified operation content. (S704 and 705), and the subsequent processing is performed based on the confirmed operation (S706). For example, if the cursor is moved, processing such as enabling input in the moved input field is performed, and if the application icon is specified, the specified application is executed.

(Other application examples)
In the present embodiment, the present invention has been described mainly using an operation input on a mobile phone as an example, but the operation input device, operation determination method, and program of the present invention can be applied to various other devices and systems. Figures 12 and 13 show one such application. FIG. 12 is a diagram showing an embodiment in which the present invention is applied to the operation of a personal computer. The personal computer can be operated without a mouse by recognizing and operating the finger of the operator's right hand 103 with a built-in or external web camera 1201 or the like. In particular, when a portable personal computer such as a notebook computer is used on the go, the mouse cannot be used in many cases, and the attached pointer such as a mouse pad is effective when the operability is lowered.

  FIG. 13 is a diagram showing an embodiment in which the present invention is applied to the operation of an automated teller machine. An input operation can be performed by installing a camera at an appropriate position of the housing 1301 and operating the screen 1302 with a finger of the right hand 103. In this case, the operation can be confirmed by pressing the provided keyboard with the left hand 102 or the like. Thereby, according to the cash dispenser to which the present invention is applied, it is possible to prevent a problem that an operation mark remains on a button such as a specific password by operation of a password or the like and a security hole is formed. In other words, in normal touch panel operations, the operator directly presses the target, but in the case of operations such as passwords and passwords, a third party sees the input or records it with a movie, etc. As a result, secret information can be easily leaked. However, according to the present invention, the pointing operation is performed as a series of flow operations, and the input is determined by the operation of a very small movement toward a button or sensor at another position. It is very difficult to tell if is entered. It is also possible to completely hide the input timing by covering the decision operation buttons. Therefore, it is possible to provide a very effective operation method even when using a cash dispenser or a security lock.

  As described above, according to the present embodiment, an input operation can be accurately determined without requiring an operation of directly touching a finger such as a touch panel without placing a burden on the operator.

(Second Embodiment)
In the first embodiment described above, a confirmation key or button arranged at an appropriate position of the device is used for the confirmation of the operation. In this embodiment, on the surface opposite to the screen of the mobile phone or the like, that is, the back surface. The user confirms the operation using the attached camera, so-called out camera. 14 and 15 show an operation example of this embodiment. FIG. 18 is a block diagram illustrating a hardware configuration of the operation input device according to the present embodiment. Compared to the first embodiment, in addition to the in-camera 201, an out-camera 1501, which is a rear imaging means, is built in, but the rest is basically the same as the configuration of the first embodiment.

  In the operation of this embodiment, as shown in FIG. 14, the control of the cursor and the icon with the finger of the right hand is the same as that of the first embodiment, but the left hand operates the out camera on the back of the apparatus and operates. The confirmation process is instructed. As shown in FIG. 15, the specific operation of the left hand can be confirmed by closing the lens of the out camera 1401 with the finger of the left hand 102, but is not limited to this, and the out camera 1401 is used. The operation can be confirmed in various ways.

  As described above, according to the present embodiment, the two cameras (out-camera and in-camera) that are usually installed in the mobile phone are allocated to different operations of selection and determination, so that the touch panel sensor can be substituted and the cost can be reduced. It is also possible to prevent contamination by not touching the screen.

(Third embodiment)
In the first and second embodiments described above, operation control such as movement of a cursor and instruction of an icon is determined based on the movement of a finger captured by an in-camera in front of the display unit. In the present embodiment, the display unit Operation control is performed based on the movement of the finger captured by the out-camera on the back of the camera. That is, as shown in FIG. 16, the movement of the finger is imaged by the out camera 1501, and the cursor on the screen is moved in accordance with the position and movement of the finger in the operation area 1502 set on the back of the apparatus. Or highlight an icon.

  As shown in FIG. 17, for example, the icon 1701 on the screen 203 is specified and highlighted by the movement of the finger. In this state. When the confirm button 202 is pressed, the designation of the icon 1701 is confirmed. If the icon 1701 is an app icon, the app is executed. As described above, by providing the operation area on the back surface, the operator can grasp the operation area to extend the screen, and intuitive operation becomes easier. Here, the operation can be determined by appropriately adjusting the determination method used in the first and second embodiments described above.

  As described above, according to the present embodiment, the operation area is set on the other side of the display, and the operation input is performed by placing the cursor by the finger tip recognition on the target screen or by enlarging or reducing the focused icon. It can be performed. As a result, when the target screen of a mobile terminal, tablet PC, etc. is small, the screen is not blocked by the operating finger, so that the operation can be performed while ensuring the full field of view, and there is no interposition between the operator and the screen. Therefore, the operation can be performed while the distance between the two is short, and the touch panel sensor can be omitted, so that the cost can be saved.

Claims (12)

Display means for displaying an image viewed by the operator;
A front imaging means that is arranged in front of the display means and images a part of the operator in front of the display means;
A part of the operator in front of the display means is extracted from the image read by the front imaging means, and an indication corresponding to the positional relationship between the part of the operator and the display means is displayed on the display means. Front-side control means for displaying and moving the sign in response to a partial movement of the operator;
A determination unit that is disposed at any position of the apparatus housing and determines an operation related to a position on the display surface of the display unit of the indication;
An operation input device comprising: signal output means for outputting a predetermined signal when the operation is confirmed by the confirmation means. Display means for displaying an image viewed by the operator;
A rear imaging means disposed on the rear surface of the display means for imaging a part of the operator on the rear surface of the display means;
A part of the operator behind the display means is extracted from the image read by the back imaging means, and an indication corresponding to the positional relationship between the part of the operator and the display means is displayed on the display means. Back surface control means for displaying and moving the sign corresponding to the movement of a part of the operator;
A determination unit that is disposed at any position of the apparatus housing and determines an operation related to a position on the display surface of the display unit of the indication;
An operation input device comprising: signal output means for outputting a predetermined signal when the operation is confirmed by the confirmation means. A front imaging means that is disposed in front of the display means and images a part of the operator in front of the display means;
A part of the operator behind the display means is extracted from the image read by the front imaging means, and an indication corresponding to the positional relationship between the part of the operator and the display means is displayed on the display means. The operation input device according to claim 2, further comprising: front surface control means for displaying and moving the sign in response to a partial movement of the operator.   An area that is defined by an image of the operator read by the front imaging means, and in which a control area in which a part of the operator can control a predetermined sign displayed on the display means is set in advance The operation input device according to claim 3, further comprising setting means.     The operation input device according to claim 4, wherein the control area is an area that does not cover an area defined by connecting the viewpoint of the operator and each vertex of the display surface of the display unit. Display means for displaying an image viewed by the operator;
A front imaging means that is disposed in front of the display means and images a part of the operator in front of the display means;
A part of the operator behind the display means is extracted from the image read by the front imaging means, and an indication corresponding to the positional relationship between the part of the operator and the display means is displayed on the display means. Front-side control means for displaying and moving the sign in response to a partial movement of the operator;
An operation related to the position of the indication on the display surface of the display means when a predetermined operation is performed by imaging a part of the operator located on the back face of the display means and on the back surface of the display means A confirmation means for confirming
An operation input device comprising: signal output means for outputting a predetermined signal when the operation is confirmed by the confirmation means. An operation is performed by an operation input device including a display unit that displays an image viewed by an operator and a front imaging unit that is disposed in front of the display unit and images a part of the operator on the front side of the display unit. Input method,
A part of the operator in front of the display means is extracted from the image read by the front imaging means, and an indication corresponding to the positional relationship between the part of the operator and the display means is displayed on the display means. A front control step for displaying and moving the indication in response to a partial movement of the operator;
A confirmation step of confirming an operation related to a position on the display surface of the display means of the indication by a confirmation means arranged at any position of the apparatus housing;
A signal output step of outputting a predetermined signal when the operation is confirmed in the confirmation step. An operation is performed by an operation input device including a display unit that displays an image viewed by an operator and a rear imaging unit that is disposed on the rear surface of the display unit and images a part of the operator on the rear surface of the display unit. Input method,
A part of the operator behind the display means is extracted from the image read by the back imaging means, and an indication corresponding to the positional relationship between the part of the operator and the display means is displayed on the display means. A back control step for displaying and moving the indication in response to a partial movement of the operator;
A confirmation step of confirming an operation related to a position on the display surface of the display means of the indication by a confirmation means arranged at any position of the apparatus housing;
A signal output step of outputting a predetermined signal when the operation is confirmed in the confirmation step. Display means for displaying an image viewed by an operator, front image pickup means arranged on the front face of the display means and picking up a part of the operator on the front face of the display means, and on the back face of the display means A method of performing an operation input by an operation input device including a rear imaging means for imaging a part of the operator,
A part of the operator behind the display means is extracted from the image read by the front imaging means, and an indication corresponding to the positional relationship between the part of the operator and the display means is displayed on the display means. A front control step for displaying and moving the indication in response to a partial movement of the operator;
When a predetermined operation is performed using a part of the operator on the back surface of the display means, which is imaged by the back surface imaging means, an operation related to the position of the indication on the display surface of the display means is performed. A confirmation step to confirm;
A signal output step of outputting a predetermined signal when the operation is confirmed in the confirmation step. Any one of a display means for displaying an image viewed by an operator, a front imaging means arranged in front of the display means for imaging a part of the operator in front of the display means, and an apparatus housing A program for causing an operation input device having a determination means arranged at a position to execute an operation input method, the operation input method comprising:
A part of the operator in front of the display means is extracted from the image read by the front imaging means, and an indication corresponding to the positional relationship between the part of the operator and the display means is displayed on the display means. A front control step for displaying and moving the indication in response to a partial movement of the operator;
A confirmation step of confirming an operation related to a position of the display on the display surface of the display means by the confirmation means;
A signal output step of outputting a predetermined signal when the operation is confirmed in the confirmation step. Display means for displaying an image viewed by the operator, confirmation means arranged at any position of the apparatus housing, and rear imaging means for imaging a part of the operator on the back of the display means A program for causing an operation input device provided to execute an operation input method, the operation input method comprising:
A part of the operator behind the display means is extracted from the image read by the back imaging means, and an indication corresponding to the positional relationship between the part of the operator and the display means is displayed on the display means. A back control step for displaying and moving the indication in response to a partial movement of the operator;
A confirmation step for confirming an operation related to a position of the display on the display surface of the display means by the confirmation means;
A signal output step of outputting a predetermined signal when the operation is confirmed in the confirmation step. Display means for displaying an image viewed by an operator, front image pickup means arranged on the front face of the display means and picking up a part of the operator on the front face of the display means, and on the back face of the display means A program for causing an operation input device having a rear imaging means for imaging a part of the operator to execute an operation input method, the operation input method comprising:
A part of the operator behind the display means is extracted from the image read by the front imaging means, and an indication corresponding to the positional relationship between the part of the operator and the display means is displayed on the display means. A front control step for displaying and moving the indication in response to a partial movement of the operator;
When a predetermined operation is performed using a part of the operator on the back surface of the display means, which is imaged by the back surface imaging means, an operation related to the position of the indication on the display surface of the display means is performed. A confirmation step to confirm;
A program characterized by comprising:

Images