JP2012073659A - Operation determination device, fingertip detection device, operation determination method, fingertip detection method, operation determination program, and fingertip detection program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an operation determination device which determines that click operation is performed from a gesture which is easy for a user to make.SOLUTION: When a fingertip 50 enters a nearly stationary state, a rectangular range 52 is set which includes the latest fingertip 50 in the center (S2000). When the fingertip 50 which is as large as a predetermined past number is present within a certain distance from the latest fingertip 50, it is determined that the fingertip 50 is in the nearly stationary state. It is considered that a single click is made when the fingertip 50 disappears from the rectangular range 52 within a certain time TC after the setting of the rectangular range 52 and is not present within the rectangular range 52 (S2002), and then appears within the rectangular range 52 again and is present within the rectangular range 52 (S2004).

Description

  The present invention relates to an operation determination device that determines whether or not a click operation has been performed when operating a device in a non-contact manner using a gesture, and a related technology.

  The present invention also relates to an operation determination device that determines whether or not a pseudo button operation has been performed when operating a device in a non-contact manner using a gesture and related technology.

  Furthermore, the present invention relates to a fingertip detection apparatus that detects an image of a human hand and detects a fingertip from a captured image, and a related technology.

  The cursor operation device of Patent Document 1 detects a user gesture in a three-dimensional space with a camera, and controls a cursor for operating a device in a non-contact manner. That is, this cursor operation device detects a fingertip pointing to the screen and displays the cursor at a position on the screen corresponding to the position of the fingertip. Then, when the cursor stays on the button displayed on the screen for a certain period of time, the cursor operation device determines that a click operation has been performed and considers that the button has been pressed.

JP 2004-258837 A

  However, the user must keep the fingertip stationary for a certain time for the click operation, and it is often difficult to perform it in the three-dimensional space.

  Also, the position of the fingertip can be obtained only with the accuracy of the image from the camera. Therefore, when the camera image has a low resolution and the screen on which the cursor is displayed has a high resolution, the cursor cannot be moved smoothly, resulting in discontinuous movement.

  Therefore, an object of the present invention is to provide an operation determination device that determines that a click operation has been performed with a gesture that is easy for the user and related technology.

  Another object of the present invention is to provide an operation determination device and related technology for determining that a button operation has been performed in a pseudo manner by a gesture that is easy for the user.

  Still another object of the present invention is to detect the position of the fingertip with high accuracy from a low resolution image obtained by photographing a human finger, and to reduce the storage capacity and increase the processing speed. It is an object of the present invention to provide a fingertip detection device and related technology that can be realized.

  According to the first aspect of the present invention, an operation determination device includes a photographing unit that photographs a user's hand moved in a three-dimensional space, and a portion corresponding to the fingertip of the user from an image obtained by photographing. Detecting means for detecting; setting means for setting a predetermined range including a portion corresponding to the detected fingertip; and whether the portion corresponding to the fingertip has disappeared after the setting means has set the predetermined range. First judgment means for judging whether or not the part corresponding to the fingertip has appeared in the predetermined range after the first judgment means judges that the part corresponding to the fingertip has disappeared from the predetermined range Second determining means for determining whether or not the second determining means determines that a click operation has been performed when it is determined that a portion corresponding to the fingertip has appeared in the predetermined range. .

  According to this configuration, the user can perform a non-contact click by simply performing a gesture of folding a finger from a state where one finger is raised and returning to the original state again in a three-dimensional space. The operation can be performed. In this way, a click operation can be performed with a gesture that is easy for the user.

  In such a human gesture, it has been verified by experiments of the present inventor that the fingertip returns almost to its original position after the finger is bent. Therefore, by detecting such a gesture, it can be determined stably and surely, that is, with high reproducibility, whether or not a click operation has been performed.

  In this operation determination device, the setting means sets the predetermined range when the detected movement of the portion corresponding to the fingertip is within a certain range.

  According to this configuration, when the movement of the portion corresponding to the fingertip is within a certain range, it is assumed that the user intends to perform a click operation, the predetermined range is set, and this is used as a trigger. It can be determined whether or not the user has made a gesture corresponding to the click operation. For this reason, it is not always necessary to determine whether or not the user has made the gesture, and the processing load can be reduced.

  In the operation determination device, when the setting unit does not determine that the portion corresponding to the fingertip has disappeared from the predetermined range within a first predetermined time after the predetermined range is set. The predetermined range is canceled.

  According to this configuration, if the first determination means does not determine that the portion corresponding to the fingertip has disappeared from the predetermined range within the first predetermined time, it is considered that the user does not intend to perform a click operation. Therefore, it can be determined that a click operation is performed against the user's intention. In other words, in general, a click operation is performed quickly, but by appropriately setting the first predetermined time, even if the movement is the same, a movement other than the gesture intended for the click operation can be excluded and erroneous determination can be prevented. .

  In this operation determination device, the setting means does not determine that the portion corresponding to the fingertip has appeared in the predetermined range within the first predetermined time after the predetermined range is set. In the case, the predetermined range is canceled.

  According to this configuration, if the second determination means does not determine that the portion corresponding to the fingertip has appeared in the predetermined range within the first predetermined time, it is considered that the user does not intend to perform a click operation. Therefore, it can be determined that a click operation is performed against the user's intention. In other words, in general, a click operation is performed quickly, but by appropriately setting the first predetermined time, even if the movement is the same, a movement other than the gesture intended for the click operation can be excluded and erroneous determination can be prevented. .

  Further, in the operation determination device, the setting means includes the second determination means within a second predetermined time after the first determination means determines that the portion corresponding to the fingertip has disappeared from the predetermined range. However, when it is not determined that the portion corresponding to the fingertip has appeared in the predetermined range, the predetermined range can be canceled.

  According to this configuration, when the second determination unit does not determine that the portion corresponding to the fingertip has appeared in the predetermined range within the second predetermined time, it is considered that the user does not intend to perform the click operation. Therefore, it can be determined that a click operation is performed against the user's intention. That is, in general, a click operation is performed quickly, but by appropriately setting the second predetermined time, even if the movement is the same, movement other than the gesture intended for the click operation can be excluded to prevent erroneous determination. .

  The operation determining apparatus is configured to determine whether the portion corresponding to the fingertip has disappeared after the second determining portion determines that the portion corresponding to the fingertip has appeared in the predetermined range. And the third determining means determines whether or not the portion corresponding to the fingertip has appeared in the predetermined range after determining that the portion corresponding to the fingertip has disappeared from the predetermined range. The determination unit determines that a double-click operation has been performed when the fourth determination unit determines that a portion corresponding to the fingertip has appeared in the predetermined range.

  According to this configuration, in the three-dimensional space, the user folds the finger from the state where one finger is raised, returns it to the original state, and further folds the finger again. A non-contact double-click operation can be performed only by performing the gesture of returning to the original standing state again, that is, by performing the gesture for the click operation (single click operation) twice. In this way, a double click operation can be performed with a gesture that is easy for the user.

  Such a double click operation is performed by performing the single click operation twice. However, it has been verified by experiments of the present inventor that the folded fingertip returns to almost the same position both times. Yes. For this reason, by detecting such a gesture, it can be determined stably and reliably, that is, with high reproducibility, whether or not a double-click operation has been performed.

  In this operation determination device, the setting means does not determine that the portion corresponding to the fingertip has disappeared from the predetermined range within the first predetermined time after the predetermined range is set. In the case, the predetermined range is canceled.

  According to this configuration, if the third determination means does not determine that the portion corresponding to the fingertip has disappeared from the predetermined range within the first predetermined time, it is considered that the user does not intend to perform a double-click operation. Thus, it can be determined that a double-click operation has been performed against the user's intention. That is, in general, a double-click operation is performed quickly. However, by appropriately setting the first predetermined time, even if the movement is the same, a movement other than a gesture intended for the double-click operation is excluded, and an erroneous determination is made. Can be prevented.

  In this operation determination device, the setting means does not determine that the portion corresponding to the fingertip has appeared in the predetermined range within the first predetermined time after the predetermined range is set. In the case, the predetermined range is canceled.

  According to this configuration, if the fourth determination unit does not determine that the portion corresponding to the fingertip has appeared in the predetermined range within the first predetermined time, it is considered that the user does not intend to perform a double click operation. Thus, it can be determined that a double-click operation has been performed against the user's intention. That is, in general, a double-click operation is performed quickly. However, by appropriately setting the first predetermined time, even if the movement is the same, a movement other than a gesture intended for the double-click operation is excluded, and an erroneous determination is made. Can be prevented.

  Further, in the operation determination device, the setting means includes the fourth determination means within a third predetermined time after the third determination means determines that the portion corresponding to the fingertip has disappeared from the predetermined range. However, when it is not determined that the portion corresponding to the fingertip has appeared in the predetermined range, the predetermined range can be canceled.

  According to this configuration, if the fourth determination means does not determine that the portion corresponding to the fingertip has appeared in the predetermined range within the third predetermined time, it is considered that the user does not intend to perform a double click operation. Thus, it can be determined that a double-click operation has been performed against the user's intention. In other words, in general, a double-click operation is performed quickly, but by appropriately setting the third predetermined time, even if it is the same movement, a movement other than a gesture intended for the double-click operation is excluded, and an erroneous determination is made. Can be prevented.

  In the operation determination device, the setting means may include the third determination means within a fourth predetermined time after the second determination means determines that the portion corresponding to the fingertip has appeared in the predetermined range. If it is not determined that the part corresponding to the fingertip has disappeared from the predetermined range, the predetermined range is canceled.

  According to this configuration, if the third determination unit does not determine that the portion corresponding to the fingertip has disappeared from the predetermined range within the fourth predetermined time, it is considered that the user does not intend to perform a double-click operation. Thus, it can be determined that a double-click operation has been performed against the user's intention. That is, in general, a double-click operation is performed quickly, but by appropriately setting the fourth predetermined time, even if it is the same movement, a movement other than a gesture intended for the double-click operation is excluded, and an erroneous determination is made. Can be prevented.

  In this operation determination device, the setting unit cancels the predetermined range when the fourth determination unit does not determine that a portion corresponding to the fingertip has appeared in the predetermined range within the fourth predetermined time. .

  According to this configuration, if the fourth determination unit does not determine that the portion corresponding to the fingertip has appeared in the predetermined range within the fourth predetermined time, it is considered that the user does not intend to perform a double-click operation. Thus, it can be determined that a double-click operation has been performed against the user's intention. That is, in general, a double-click operation is performed quickly, but by appropriately setting the fourth predetermined time, even if it is the same movement, a movement other than a gesture intended for the double-click operation is excluded, and an erroneous determination is made. Can be prevented.

  Further, in the operation determination device, the setting means includes the fourth determination means within a fifth predetermined time after the third determination means determines that the portion corresponding to the fingertip has disappeared from the predetermined range. However, when it is not determined that the portion corresponding to the fingertip has appeared in the predetermined range, the predetermined range can be canceled.

  According to this configuration, if the fourth determination unit does not determine that the portion corresponding to the fingertip has appeared in the predetermined range within the fifth predetermined time, it is considered that the user does not intend to perform a double-click operation. Thus, it can be determined that a double-click operation has been performed against the user's intention. In other words, in general, a double-click operation is performed quickly. By appropriately setting the fifth predetermined time, even if the movement is the same, a movement other than a gesture intended for the double-click operation is excluded, and an erroneous determination is made. Can be prevented.

  According to the second aspect of the present invention, the operation determination device includes a photographing unit that photographs the user's hand moved in the three-dimensional space, and a portion corresponding to the user's finger from the image obtained by the photographing. In the case where the detection means for detecting and one portion corresponding to the user's finger are detected this time, if two portions corresponding to the user's finger have been detected last time, a pseudo button operation is performed. Determination means for determining whether or not.

  According to this configuration, the user can perform a button operation in a pseudo manner only by raising another finger and raising two fingers from a state where one finger is raised. In this way, it is possible to perform a pseudo button operation with a gesture that is easy for the user.

  Also, since one finger is always photographed before and after the pseudo button operation, the cursor can be displayed at the position on the screen corresponding to the fingertip, and pointing and pseudo button operation can be performed with one hand. it can.

  Here, in the present specification and claims, “pseudo button operation” is not an operation of an actual button of a device such as a pointing device, but is equivalent to information or a case of operating an actual button. This means giving instructions to the computer by gestures in a three-dimensional space.

  According to the third aspect of the present invention, the operation determination device includes a photographing unit that photographs the user's hand moved in the three-dimensional space, and a portion corresponding to the user's finger from the image obtained by the photographing. In the case where two detection parts corresponding to the user's finger are detected at this time and one part corresponding to the user's finger has been detected last time, a pseudo button operation is performed. Determination means for determining whether or not.

  According to this configuration, the user can perform a button operation in a pseudo manner by simply closing another finger and raising one finger from a state where two fingers are raised. In this way, it is possible to perform a pseudo button operation with a gesture that is easy for the user.

  Also, since one finger is always photographed before and after the pseudo button operation, the cursor can be displayed at the position on the screen corresponding to the fingertip, and pointing and pseudo button operation can be performed with one hand. it can.

  According to a fourth aspect of the present invention, an operation determination device includes a photographing unit that photographs a user's hand moved in a three-dimensional space, and a portion corresponding to the fingertip of the user from an image obtained by photographing. A detecting means for detecting, and a determining means for determining that a button is simulated when two portions corresponding to the fingertip of the user are detected and the distance between the two points is smaller than a predetermined value; .

  According to this configuration, the user can perform a button operation in a pseudo manner only by bringing the fingertips of two fingers close to each other. In this way, it is possible to perform a pseudo button operation with a gesture that is easy for the user.

  In addition, a cursor can be displayed between two detected fingertips (for example, a midpoint), and pointing and pseudo button operations can be performed with one hand.

  According to the fifth aspect of the present invention, the fingertip detection device detects the contour of the hand image from the photographing means for photographing the user's hand moved in the three-dimensional space and the image obtained by the photographing. A contour detection means; a finger detection means for detecting a pixel corresponding to the fingertip of the hand (hereinafter referred to as “fingertip pixel”) from the contour; and a pixel surrounding the fingertip pixel (hereinafter referred to as “peripheral”). Correction means for correcting the coordinates of the fingertip pixels detected by the fingertip detection means with a decimal point accuracy using pixel values and coordinates of “pixels”).

  According to this configuration, since an approximate fingertip is determined from the contour and correction is performed using only the surrounding pixels, the storage capacity for processing can be reduced and the processing speed can be improved.

  Further, since the correction is executed with decimal point accuracy, the position of the fingertip can be determined with high accuracy even when the imaging means has a relatively low resolution. As a result, even when the cursor is displayed at a position corresponding to the fingertip on the screen having a higher resolution than the photographing means, the movement can be smoothed.

  Furthermore, since the correction is performed based on the pixel values of the surrounding pixels, the corrected fingertip can be determined not at the contour line but at the position inside the finger, and closer to the position recognized by the user as the fingertip. As a result, when the cursor is displayed at a position on the screen corresponding to the fingertip, the user can easily control the cursor with the fingertip. In general, when a person moves his / her finger toward the belly of a finger, the convex point on the contour of the finger is not recognized as the fingertip, and the center of the finger pad at the tip of the finger is defined as the fingertip. Recognize.

  In this fingertip detection device, the correction value calculation means calculates the weighted average of the coordinates of the fingertip pixel and the surrounding pixels by using the pixel values of the fingertip pixel and the surrounding pixels as weights, and corrects the coordinates of the fingertip pixels. To do.

  In the fingertip detection apparatus, the correction value calculation means performs the correction so that the corrected fingertip position exceeds the range of the surrounding pixels.

  According to this configuration, the position of the fingertip after correction can be determined to be a constant position or a substantially constant position of the belly of the finger tip, and the position of the fingertip detected by the contour detection means is not stable However, the position of the fingertip after correction can be made as constant as possible, or the difference in the position of the fingertip can be kept as small as possible. As a result, when the cursor is displayed at a position on the screen corresponding to the fingertip, the cursor can be stabilized.

  Further, the corrected fingertip can be made closer to the position that the user recognizes as the fingertip. As a result, when the cursor is displayed at a position on the screen corresponding to the fingertip, it becomes easier for the user to control the cursor with the fingertip.

  In the fingertip detection device, the correction value calculation means uses the pixel values of the fingertip pixel and the surrounding pixels as weights to the coordinates of the fingertip pixel and the surrounding pixels, and weights the fingertip pixels and the surrounding pixels. The coordinate of the fingertip pixel can also be corrected by calculating the sum of coordinates and dividing the sum by the maximum value of the pixel values of the fingertip pixel and the surrounding pixels.

  In the above fingertip detection device, the correction value calculation means includes a distance between the fingertip pixel and the surrounding pixels located in an oblique direction of the fingertip pixel, and a horizontal direction or a vertical direction of the fingertip pixel and the fingertip pixel. The coordinates of the surrounding pixels are defined such that the distance between the surrounding pixels and the surrounding pixels is equal.

  According to this configuration, when using the pixel values, the pixel values of all the surrounding pixels are treated equally, and correction can be performed without deviation depending on the direction.

  According to the sixth aspect of the present invention, an operation determination method includes a step of photographing a user's hand moved in a three-dimensional space, and a portion corresponding to the user's fingertip is detected from an image obtained by the photographing. A step of setting a predetermined range including a portion corresponding to the detected fingertip, a step of determining whether or not a portion corresponding to the fingertip has disappeared after setting the predetermined range, The step of determining whether or not the portion corresponding to the fingertip has disappeared from the predetermined range and then determining whether or not the portion corresponding to the fingertip has appeared in the predetermined range; Determining whether or not an appearance has occurred includes determining that a click operation has been performed when it is determined that a portion corresponding to the fingertip has appeared in the predetermined range.

  According to this structure, there exists an effect similar to the operation determination apparatus by the said 1st viewpoint.

  According to a seventh aspect of the present invention, an operation determination method includes a step of photographing a user's hand moved in a three-dimensional space, and a portion corresponding to the user's finger is detected from an image obtained by the photographing. In the case where one portion corresponding to the user's finger is detected this time and two portions corresponding to the user's finger have been detected last time, it is determined that a pseudo button operation has been performed. Including the steps of:

  According to this structure, there exists an effect similar to the operation determination apparatus by the said 2nd viewpoint.

  According to an eighth aspect of the present invention, an operation determination method includes a step of photographing a user's hand moved in a three-dimensional space, and a portion corresponding to the user's finger is detected from an image obtained by the photographing. When two portions corresponding to the user's finger are detected this time and one portion corresponding to the user's finger has been detected last time, it is determined that a pseudo button operation has been performed. Including the steps of:

  According to this structure, there exists an effect similar to the operation determination apparatus by the said 3rd viewpoint.

  According to a ninth aspect of the present invention, an operation determination method includes a step of photographing a user's hand moved in a three-dimensional space, and a portion corresponding to the user's fingertip is detected from the image obtained by the photographing. And, when two portions corresponding to the user's fingertips are detected, and when the distance between the two points is smaller than a predetermined value, determining that a button operation has been performed in a pseudo manner. Including.

  According to this structure, there exists an effect similar to the operation determination apparatus by the said 4th viewpoint.

  According to a tenth aspect of the present invention, a fingertip detection method includes a step of photographing a user's hand moved in a three-dimensional space, and a step of detecting an outline of the hand image from an image obtained by the photographing. And detecting a pixel corresponding to the fingertip of the hand (hereinafter referred to as “fingertip pixel”) from the contour, and a pixel surrounding the fingertip pixel (hereinafter referred to as “peripheral pixel”). And correcting the coordinates of the fingertip pixel detected by the step of detecting the fingertip pixel with the decimal point accuracy using the pixel value and coordinates of

  According to this structure, there exists an effect similar to the fingertip detection apparatus by the said 5th viewpoint.

  According to an eleventh aspect of the present invention, a computer program causes a computer to execute the operation determination method according to the sixth aspect.

  According to a twelfth aspect of the present invention, a computer program causes a computer to execute the operation determination method according to the seventh aspect.

  According to a thirteenth aspect of the present invention, a computer program causes a computer to execute the operation determination method according to the eighth aspect.

  According to a fourteenth aspect of the present invention, a computer program causes a computer to execute the operation determination method according to the ninth aspect.

  According to a fifteenth aspect of the present invention, a computer program causes a computer to execute the fingertip detection method according to the tenth aspect.

(A) It is explanatory drawing of pseudo button operation by embodiment of this invention. (B) It is explanatory drawing of the 1st-3rd example of pseudo button operation by embodiment of this invention. 1 is a diagram illustrating an electrical configuration of an image processing apparatus 1 according to an embodiment of the present invention. It is a flowchart which shows an example of the flow of the image analysis process by MCU3 of FIG. It is a flowchart which shows an example of the flow of the detection process of the inflection point of FIG.3 S5. It is a flowchart which shows an example of the flow of the process which determines the shape of the outline image of step S7 of FIG. It is a flowchart which shows an example of the flow of a process which calculates the outline information of step S161 of FIG. It is a flowchart which shows an example of the flow of a process which detects the fingertip of step S163 of FIG. 6 is a flowchart illustrating an example of a flow of processing for determining “par” in step S165 of FIG. 5. It is a flowchart which shows an example of the flow of a process which determines "pick" of step S167 of FIG. It is a flowchart which shows an example of the flow of a process which determines "go" of step S169 of FIG. It is a flowchart which shows an example of the flow of a process which determines the one finger | toe of step S171 of FIG. (A) It is a flowchart which shows the flow of the 1st example of the fingertip correction value calculation process of step S377 of FIG. 9, and step S447 of FIG. (B) It is a flowchart which shows the flow of the 2nd example of the fingertip correction value calculation process of step S377 of FIG. 9, and step S447 of FIG. (C) It is a flowchart which shows the flow of the 3rd example of the fingertip correction value calculation process of FIG.9 S377 and FIG.11 S447. (D) It is a flowchart which shows the flow of the 4th example of the fingertip correction value calculation process of step S377 of FIG. 9, and step S447 of FIG. (A) It is a flowchart which shows the flow of the 1st example of the pseudo | simulation button operation determination process of FIG.3 S9. (B) It is a flowchart which shows the flow of the 2nd example of the pseudo | simulation button operation determination process of FIG.3 S9. (C) It is a flowchart which shows the flow of the 3rd example of the pseudo | simulation button operation determination process of FIG.3 S9. It is explanatory drawing of the deflection angle (theta) j detected by step S3 of FIG. It is explanatory drawing of the inflection point detected by step S5 of FIG. 3, and the fingertip detected by step S163 of FIG. (A) It is explanatory drawing of the fingertip correction value calculation process of Fig.12 (a) and FIG.12 (c). (B) It is explanatory drawing of the fingertip correction value calculation process of FIG.12 (b) and FIG.12 (d). It is explanatory drawing of the divisor in a fingertip correction value calculation process. (A) It is a flowchart which shows the flow of the 4th example of the pseudo button operation determination process of FIG.3 S9. (B) It is a flowchart which shows the flow of the 5th example of the pseudo button operation determination process of FIG.3 S9. (C) It is a flowchart which shows the flow of the 6th example of the pseudo | simulation button operation determination process of FIG.3 S9. (A) It is explanatory drawing of the 4th example (single click) of pseudo button operation by embodiment of this invention. (B) It is explanatory drawing of the 4th example (double-click) of pseudo button operation by embodiment of this invention. It is a flowchart which shows a part of flow of the 7th example of the pseudo | simulation button operation determination process of FIG.3 S9. It is a flowchart which shows another part of the flow of the 7th example of the pseudo | simulation button operation determination process of FIG.3 S9.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals, and the description thereof is incorporated. In the present specification, “h” added to the end of a number indicates a hexadecimal number.

  FIG. 1A and FIG. 1B are explanatory diagrams of pseudo button operations according to the embodiment of the present invention. With reference to FIG. 1A, the image processing apparatus 1 described later obtains the positions of the fingertips a and b by photographing a user's (person) hand and analyzing the obtained image. In this example, the fingertip a is the fingertip of the index finger, and the fingertip b is the fingertip of the thumb.

  Referring to FIG. 1B, in the first example, an electronic device 9 described later receives the position of the fingertip a from the image processing apparatus 1, and places the cursor at a corresponding position on the screen of the monitor 13 described later. indicate. When the image processing apparatus 1 detects a transition from a state in which the fingertip a and the fingertip b are present in the photographed image to a state in which only the fingertip a is present in the photographed image, for example, two fingers ( From the state where the thumb and index finger) are reflected, one finger (thumb) that is not related to the cursor position is folded, and only one other finger (index finger) that controls the cursor position is reflected When the transition is detected, it is determined that the user has performed a pseudo button operation, and the electronic device 9 is notified that the button operation has been performed in a pseudo manner. Then, the electronic device 9 executes processing corresponding to the button operation.

  In the second example, the electronic device 9 receives the position of the fingertip a from the image processing apparatus 1 and displays a cursor at a corresponding position on the screen of the monitor 13. Further, when the image processing apparatus 1 detects a transition from a state where only the fingertip a exists in the captured image to a state where the fingertip a and the fingertip b exist in the captured image, for example, the image processing apparatus 1 relates to the position of the cursor. One finger (thumb) is not folded and only one other finger (forefinger) that controls the cursor position is shown, and two fingers (thumb and forefinger) are shown When the transition is detected, it is determined that the user has performed a pseudo button operation, and the electronic device 9 is notified that the button operation has been performed in a pseudo manner. Then, the electronic device 9 executes processing corresponding to the button operation.

  In the third example, the electronic device 9 receives the position of the midpoint between the position of the fingertip a and the position of the fingertip b from the image processing apparatus 1 and displays a cursor at a corresponding position on the screen of the monitor 13. Further, the image processing apparatus 1 calculates the distance between the fingertip a and the fingertip b, and detects that the distance is equal to or less than a certain value, that is, the fingertip a and the fingertip b approach each other. When the distance is less than or equal to a certain value, it is determined that the user has performed a pseudo button operation, and notifies the electronic device 9 that the pseudo button operation has been performed. Then, the electronic device 9 executes processing corresponding to the button operation.

  Here, in the first to third examples, it can be considered that the button operation (a state in which the button is pressed) is performed when it is determined that the button operation is performed in a pseudo manner. Alternatively, it can be considered that a click operation (operation of pressing and releasing a button) has been performed.

  When it is determined that the button operation is performed in a pseudo manner, it is determined that the button is pressed, and the operation that the button is released is determined as follows.

  In the first example, when the image processing apparatus 1 detects a transition from a state where only the fingertip a is present in the captured image to a state where the fingertip a and the fingertip b are present in the captured image, the user is simulated. It is determined that the operation of releasing the button is performed, and the electronic device 9 is notified that the operation of speaking the button is performed.

  In the second example, when the image processing apparatus 1 detects a transition from a state in which the fingertip a and the fingertip b exist in the captured image to a state in which only the fingertip a exists in the captured image, the user performs a simulation. It is determined that the operation of releasing the button is performed, and the electronic device 9 is notified that the operation of releasing the button is performed.

  In the third example, the image processing apparatus 1 calculates the distance between the fingertip a and the fingertip b, and after the distance has become a certain value or less, the image processing apparatus 1 again confirms that the distance has exceeded the certain value. When detected, it is determined that the user has performed an operation of releasing the button in a pseudo manner, and the electronic device 9 is notified that the operation of releasing the button has been performed in a pseudo manner.

  Note that the “pseudo button operation” is not an actual button operation of a device such as a pointing device, but the same information or command as when operating an actual button is used by a gesture in a three-dimensional space. Means to give to the computer.

  FIG. 2 is a diagram showing an electrical configuration of the image processing apparatus 1 according to the embodiment of the present invention. With reference to FIG. 1, the image processing apparatus 1 includes an MCU (Micro Controller Unit) 3 and an image sensor 5. The MCU 3 includes a memory 7. The image sensor 5 can be used in either color or gray scale, and any number of pixels can be used. In the present embodiment, the image sensor 5 is gray scale and has a relatively low resolution of 64 × 64 pixels. Is used. As the image sensor 5, a CMOS image sensor, a CCD image sensor, or the like can be employed.

  The memory 7 of the MCU 3 is a semiconductor memory such as a RAM, a ROM, and a flash memory, and is used as a program storage area, a data storage area, a work area, a temporary storage area, and the like. The MCU 3 executes various processes shown in the flowcharts described later in accordance with the computer program stored in the memory 7. Depending on the specifications, as a recording medium, in addition to the memory 7, an optical disc such as a CD (including CD-ROM, Video-CD) and a DVD (including DVD-Video, DVD-ROM, DVD-RAM), a semiconductor A memory card, a memory cartridge, a USB memory, a hard disk, a magneto-optical disk, a magnetic tape, a flexible disk, and the like equipped with a memory can be connected to the MCU 3. In addition, computer programs and data can be stored in these.

  The image processing apparatus 1 is used by being connected to an electronic device 9. The electronic device 9 includes a computer 11 and a monitor 13 that displays the processing result. The electronic device 9 is, for example, a television, a video game machine, a car navigation, a personal computer, or the like, but is not particularly limited. In the present embodiment, it is assumed that a monitor 13 having a higher resolution than that of the image sensor 5 is used.

  The MCU 3 controls the image sensor 5 according to a program stored in the memory 7. The image sensor 5 shoots a subject (in this embodiment, a human hand) that moves or is moved in a three-dimensional space under the control of the MCU 3, and outputs the obtained image to the MCU 3. The MCU 3 analyzes the image received from the image sensor 5 according to the program stored in the memory 7, detects the fingertip, and indicates the position, information on the presence / absence of a pseudo button operation, etc. 11 is output. Then, the computer 11 of the electronic device 9 displays a cursor at a position on the screen of the monitor 13 corresponding to the position of the fingertip, and when notified that a button operation has been performed, executes a process corresponding thereto. To do. In addition, the computer 11 processes other information received from the image processing apparatus 1 in accordance with the specifications of the electronic device 9, and executes unique information processing.

  Next, details of the processing of the MCU 3 of the image processing apparatus 1 will be described using a flowchart.

  FIG. 3 is a flowchart showing an example of the flow of image analysis processing by the MCU 3 of FIG. Referring to FIG. 3, in step S <b> 1, MCU 3 extracts the contour line of the image of the subject (a hand in the present embodiment) from the image data output from image sensor 5. In this process, the coordinates (XO [j], YO [j]) of each pixel Pj (j = 0 to J−1) on the contour line are obtained.

  Here, in the present embodiment, the value J indicates the total number of pixels Pj on the contour line. Further, it is assumed that the contour line extraction process is executed counterclockwise.

  As an algorithm for extracting the contour line, a known algorithm can be used. For example, in a method using a mask made up of 3 × 3 small pixel regions, the mask is sequentially moved to extract the contour line, although the center of the mask is always kept on the contour line.

  In step S3, the MCU 3 calculates the deflection angle θj of the vector Vj of each pixel Pj on the extracted contour line.

  FIG. 14 is an explanatory diagram of the deviation angle θj detected in step S3 of FIG. With reference to FIG. 4, attention is now focused on the three pixels Pj-1, Pj, and Pj + 1 on the contour line of the subject. Also, the contour line is scanned counterclockwise. A vector Vj of the pixel Pj is defined as a vector from the pixel Pj toward the next pixel Pj + 1. The deflection angle θj of the vector Vj of the pixel Pj is defined as an angle formed by the vector Vj with respect to the vector Vj−1 of the pixel Pj−1 immediately before the pixel Pj. The sign of the counterclockwise deflection angle θj is positive, and the sign of the clockwise deflection angle θj is negative.

  Returning to FIG. 3, in step S5, the MCU 3 detects the inflection point of the extracted contour line. The inflection point is a point where the direction of the outline changes, and a convex point and a concave point are defined.

  FIG. 15 is an explanatory diagram of the inflection point detected in step S5 of FIG. Referring to FIG. 15, inflection points I0 to I8 are given as points at which the direction of the extracted outline 21 changes. Inflection points I0, I2, I4, I6 and I8 are convex points. Inflection points I1, I3, I5 and I7 are concave points.

  Returning to FIG. 3, in step S <b> 7, the MCU 3 determines the shape (goo, choki, par, one finger, indefinite) of the extracted contour line (contour image). In step S9, the MCU 3 determines whether or not a pseudo button operation has been performed. In step S11, the MCU 3 determines the result of step S7 (the barycentric coordinates, area, and shape information of the contour image (flag SH described later), the corrected fingertip coordinates, and the midpoint coordinates of the two fingertips) and the step. The result of S9 (a pseudo button flag described later or a single click flag and a double click flag described later) is transmitted to the computer 11 of the electronic device 9. The MCU 3 repeats steps S1 to S11 each time it receives one frame of image data from the image sensor 5.

  FIG. 4 is a flowchart showing an example of the flow of the inflection point detection process in step S5 of FIG. Referring to FIG. 4, in step S21, MCU 3 substitutes 0 for variables DA, ADA, j, and CF []. In step S23, the MCU 3 acquires from the memory 7 the deviation angle θj of the pixel Pj on the contour detected in step S1 of FIG. In step S25, the MCU 3 adds the deviation angle θj to the current value of the variable DA and substitutes it for the variable DA. In step S27, the MCU 3 obtains the absolute value of the variable DA and assigns it to the variable ADA.

  In step S29, the MCU 3 determines whether or not the value of the variable ADA has become 135 degrees. If the value is 135 degrees, the MCU 3 determines that the direction of the contour has changed, regards the pixel Pj as an inflection point, and performs step S31. Otherwise, go to step S39. In step S31, the MCU 3 determines whether or not the sign of the variable DA is positive. If it is positive, the process proceeds to step S33, and if negative, the process proceeds to step S35. In step S33, the MCU 3 assigns 01 indicating that the inflection point is a convex point to the convex / concave flag CF [j]. On the other hand, in step S35, the MCU 3 assigns 10 indicating that the inflection point is a concave point to the convex / concave flag CF [j].

  In step S37, the MCU 3 substitutes 0 for the variable DA. In step S39, the MCU 3 increments the variable j by one. In step S41, the MCU 3 determines whether or not the value of the variable j is equal to the value J. If j = J, the process returns. Otherwise, the process proceeds to step S23.

  FIG. 5 is a flowchart showing an example of the processing flow for determining the shape of the contour image in step S7 of FIG. With reference to FIG. 5, MCU3 calculates the information regarding an outline in step S161. In step S163, the MCU 3 detects a pixel corresponding to the fingertip of the hand from the contour image.

  Again referring to FIG. 15, convex points I0, I2, I4, I6 and I8 are collectively referred to as convex points As (s = 0, 1,..., S), and concave points I1, I3, I5. And I7 are collectively referred to as concave points Bt (t = 0, 1,..., T). The MCU 3 obtains a line segment 23 that connects two pixels located a certain number of pixels Np away from the convex point As, and calculates the length Lv of the perpendicular line 25 drawn from the convex point As to the line segment 23. When the length Lv exceeds the predetermined value CL, the MCU 3 sets the convex point As as a candidate for the fingertip.

  Then, the MCU 3 determines whether or not there is a concave point Bt on both sides or one side of the convex point As that is a fingertip candidate, and if it exists, regards the convex point As as the fingertip.

  Also, the MCU 3 calculates the tilt angle θv necessary for determining “par” and “goo”. The inclination angle θv is an angle formed by a perpendicular line 25 drawn from the convex point As regarded as the fingertip to the line segment 23 with a line 27 parallel to the X axis of the difference image.

  Returning to FIG. 5, in step S <b> 165, the MCU 3 determines whether or not the shape of the contour image is a “par” hand shape. In step S167, the MCU 3 determines whether or not the shape of the contour image is the shape of a hand that is “stiff”. In step S169, the MCU 3 determines whether or not the shape of the contour image is a “goo” hand shape. In step S171, the MCU 3 determines whether or not the shape of the contour image is the shape of a hand with one finger raised, and returns.

  FIG. 6 is a flowchart illustrating an example of a flow of processing for calculating the contour information in step S161 in FIG. Referring to FIG. 6, in step S191, MCU 3 substitutes 0 for variables XG, YG, and j. In step S193, the MCU 3 substitutes the variable XG for the value obtained by adding the X coordinate XO [j] of the pixel Pj on the contour line to the current value of the variable XG. Also, the MCU 3 substitutes the variable XG for the value obtained by adding the Y coordinate YO [j] of the pixel Pj on the contour line to the current value of the variable YG. In step S195, the MCU 3 increments the variable j by one. In step S197, the MCU 3 determines whether or not the value of the variable j has become the value J. If the value has become the value J, the process proceeds to step S199. Otherwise, the process proceeds to step S193.

  In step S199, the MCU 3 assigns the value obtained by dividing the value of the variable XG by the value J to the variable Xg. Further, the MCU 3 substitutes the value of the variable YG divided by the value J for the variable Yg. The coordinates (Xg, Yg) indicate the barycentric coordinates of the contour image.

  In step S201, the MCU 3 substitutes 0 for variables j, LS, and L []. In step S203, the MCU 3 calculates a distance L [j] between the barycentric coordinates (Xg, Yg) and the coordinates (XO [j], YO [j]) of each pixel Pj on the contour line by the following equation. To do.

L [j] ← √ ((Xg−XO [j]) ² + (Yg−YO [j]) ² )

  In step S205, the MCU 3 substitutes a value obtained by adding the value of the variable L [j] to the current value of the variable LS for the variable LS. In step S207, the MCU 3 increments the variable j by one. In step S209, the MCU 3 determines whether or not the value of the variable j has become the value J. If the value has become the value J, the process proceeds to step S211. Otherwise, the process proceeds to step S203.

  In step S211, the MCU 3 calculates the average value AVL of the distance L [j] by dividing the value of the variable LS by the value J. In step S213, the MCU 3 counts the contour line and the pixels in the contour line, calculates the area of the contour image, and returns.

  FIG. 7 is a flowchart illustrating an example of a flow of processing for detecting a fingertip in step S163 of FIG. Referring to FIG. 7, in step S241, MCU 3 substitutes 0 for variables j and FP []. In step S243, the MCU 3 determines whether or not the flag CF [j] is set to 01 indicating a convex point. If the determination is affirmative, the process proceeds to step S245. If the determination is negative, the process proceeds to step S249.

  In step S245, the MCU 3 determines whether or not there is a pixel in which 10 indicating the concave point is set in the flag CF [j] on both sides or one side of the pixel Pj that is the convex point. If the case pixel Pj is regarded as a fingertip candidate, the process proceeds to step S247, and if not, the process proceeds to step S249. In step S247, the MCU 3 sets 1 indicating that the pixel Pj is a fingertip candidate to the flag FP [j].

  In step S249, the MCU 3 increments the variable j by one. In step S251, the MCU 3 determines whether or not the value of the variable j has become the value J. When the value becomes the value J, the MCU 3 proceeds to step S253, otherwise proceeds to step S243.

  In step S253, the MCU 3 assigns 0 to variables j, q, TF [], θv [], XT, YT, Q, and θv. In step S255, the MCU 3 determines whether or not 1 indicating a fingertip candidate is set in the flag FP [j]. If the determination is affirmative, the process proceeds to step S257; otherwise, the process proceeds to step S271.

  In step S257, the MCU 3 calculates the length Lv of the perpendicular 25 drawn from the pixel Pj to the line segment 23 connecting the pixel Pj + Np and the pixel Pj−Np (see FIG. 15). In step S259, the MCU 3 determines whether or not the length Lv exceeds the predetermined value CL. If the length Lv exceeds, the MCU 3 regards the pixel Pj as a fingertip and proceeds to step S261. Otherwise, the process proceeds to step S271.

  In step S261, the MCU 3 sets 1 indicating that the pixel Pj is the fingertip to the fingertip flag TF [j]. In step S263, the MCU 3 substitutes the X coordinate of the pixel Pj for the variable XT [q]. Further, the MCU 3 substitutes the Y coordinate of the pixel Pj for the variable YT [q]. The coordinates (XT [q], YT [q]) are the coordinates of the fingertip. In step S265, the MCU 3 calculates the inclination angle θv of the perpendicular 25 (see FIG. 15). In step S267, the MCU 3 substitutes the tilt angle θv for the variable θv [q]. The tilt angle θv [q] is the tilt angle of the finger corresponding to the fingertip indicated by the coordinates (XT [q], YT [q]). In step S269, the MCU 3 increments the variable q by one.

  In step S271, the MCU 3 increments the variable j by one. In step S273, the MCU 3 determines whether or not the value of the variable j has become the value J. If the value has become the value J, the process proceeds to step S275. Otherwise, the process proceeds to step S255. In step S275, the MCU 3 assigns the value of the variable q to the variable Q and returns. The value of variable Q represents the number of detected fingertips.

  FIG. 8 is a flowchart showing an example of a process flow for determining “par” in step S165 of FIG. With reference to FIG. 3, in step S301, the MCU 3 determines whether or not the value of the variable Q (number of fingertips) is 3 or more, the process proceeds to step S303 if it is 3 or more, and otherwise returns. In step S303, the MCU 3 substitutes 0 for the variables q and m. In step S305, the MCU 3 calculates the absolute value of the difference between the array θv [q] and the array θv [q + 1], and substitutes it for the variable θd. The variable θd represents the angle formed between adjacent fingers.

  In step S307, the MCU 3 determines whether or not the value of the variable θd is less than 100 degrees. If the determination is affirmative, the process proceeds to step S309. If the determination is negative, the process proceeds to step S311. In step S309, the MCU 3 increments the variable m by one. In step S311, the MCU 3 increments the variable q by one.

  In step S313, the MCU 3 determines whether or not the value of the variable q has reached the value Q. If the value has become the value Q, the process proceeds to step S315, otherwise the process proceeds to step S305.

  In step S315, the MCU 3 determines whether the value of the variable m is 6 or less and 3 or more. If the determination is affirmative, the MCU 3 determines that the shape of the hand is “par” and proceeds to step S317. Return in case of judgment. The value of the variable m is the number of occurrences of the case where the angle θd formed by the adjacent fingers is less than 100 degrees. In step S317, the MCU 3 sets the flag SH to 10h indicating “par”, and proceeds to step S9 in FIG.

  FIG. 9 is a flowchart illustrating an example of a process flow for determining “pick” in step S167 of FIG. Referring to FIG. 9, in step S341, MCU 3 proceeds to step S343 if the value of variable Q (number of fingertips) is 2 or 3, and otherwise returns.

  In step S343, the MCU 3 substitutes 0 for the variables q, p, SH, XL, YL, XS, and YS. In step S345, the MCU 3 calculates a distance Lgt0 between the center-of-gravity coordinates (Xg, Yg) of the contour image and the fingertip coordinates (XT [q], YT [q]). In step S347, the MCU 3 determines whether or not the distance Lgt0 is equal to or larger than the average value AVL (see step S211 in FIG. 6) multiplied by 1.1, that is, the distance Lgt0 is 110% or more of the average value AVL. If the determination is affirmative, the process proceeds to step S349. If the determination is negative, the process proceeds to step S359.

  In step S349, the MCU 3 calculates a distance Lgt1 between the center-of-gravity coordinates (Xg, Yg) of the contour image and the fingertip coordinates (XT [q + 1], YT [q + 1]). The fingertip of coordinates (XT [q + 1], YT [q + 1]) is the fingertip next to the fingertip of coordinates (XT [q], YT [q]). In step S351, the MCU 3 determines whether or not the distance Lgt1 is greater than or equal to the value AVL multiplied by 1.1, that is, determines whether or not the distance Lgt1 is 110% or more of the average value AVL. In the case of determination, the process proceeds to step S353, and in the case of negative determination, the process proceeds to step S359.

  In step S353, the MCU 3 calculates the absolute value of the difference between the array θv [q] and the array θv [q + 1] (that is, the angle formed by the adjacent finger and the finger) and substitutes it in the variable θd. In step S355, the MCU 3 determines whether or not the value of the variable θd is less than 90 degrees. If the determination is affirmative, the process proceeds to step S357. If the determination is negative, the process proceeds to step S359. In step S357, the MCU 3 increments the variable p by one.

  In step S359, the MCU 3 increments the variable q by one. In step S361, the MCU 3 determines whether or not the value of the variable q has become the value Q. If the value has become the value Q, the process proceeds to step S363, otherwise the process proceeds to step S345. In step S363, the MCU 3 determines that the shape of the hand is “pick” when the value of the variable p is 1 or 2, proceeds to step S365, and returns otherwise.

  In step S365, the MCU 3 determines whether or not the value of the variable Q is 2, in the case of 2, that is, in the case where two fingers are detected, the process proceeds to step S369, otherwise the process proceeds to step S367. In step S367, the MCU 3 sets the flag SH to 21h indicating “pick”, and proceeds to step S9 in FIG. Note that the value 21h indicates that three fingers are detected, but one of them is noise that is not a finger.

  In step S369, the MCU 3 sets the flag SH to 20h indicating “pick”. Note that the value 20h indicates that two fingers are detected and does not include noise as in step S367. In step S371, the MCU 3 determines whether or not the distance Lgt0 is greater than the distance Lgt1, that is, determines which of the detected two fingers is longer. If the distance Ltg0 is large, the process proceeds to step S373. Otherwise, the process proceeds to step S375.

  In step S373, the MCU 3 substitutes coordinates XT [0] and YT [0] for the coordinates XL and YL, respectively. Also, the MCU 3 substitutes coordinates XT [1] and YT [1] for the coordinates XS and YS, respectively. On the other hand, in step S375, the MCU 3 substitutes coordinates XT [1] and YT [1] for the coordinates XL and YL, respectively. Also, the MCU 3 substitutes coordinates XT [0] and YT [0] for the coordinates XS and YS, respectively. Here, the coordinates (XL, YL) are the coordinates of the fingertip of the long finger, and the coordinates (XS, YS) are the coordinates of the fingertip of the short finger.

  In step S377, the MCU 3 calculates a correction value for each of the coordinates (XL, YL) and the coordinates (XS, YS). In step S379, the MCU 3 adds the correction value to the coordinates (XL, YL) to obtain the corrected coordinates. The MCU 3 adds the correction value to the coordinates (XS, YS) to obtain the corrected coordinates. In step S381, the MCU 3 calculates the midpoint coordinates of the corrected coordinates of the coordinates (XL, YL) and the corrected coordinates of the coordinates (XS, YS). Then, the MCU 3 proceeds to step S9 in FIG.

  FIG. 10 is a flowchart illustrating an example of a process flow for determining “go” in step S169 of FIG. Referring to FIG. 10, in step S391, the MCU 3 determines whether or not the value of the variable Q (number of fingertips) is 2 or more, and proceeds to step S393 in the case of a positive determination, and in the case of a negative determination. Return.

  In step S393, the MCU 3 substitutes 0 for the variable j. In step S395, the MCU 3 determines that the distance L [j] (see step S203 in FIG. 6) is equal to or greater than the average value AVL (see step S211 in FIG. 6) multiplied by 0.6, and the value AVL. Whether or not the distance L [j] is 60% or more of the average value AVL and 140% or less of the average value AVL. If the determination is affirmative, the process proceeds to step S397. If the determination is negative, the process returns.

  In step S397, the MCU 3 increments the variable j by one. In step S399, the MCU 3 determines whether or not the value of the variable j has become the value J. If the value J has reached the value J, the hand shape is determined to be “Goo”, and the process proceeds to step S401. Otherwise, the process proceeds to step S395. Proceed to In step S401, the MCU 3 sets the flag SH to 30h indicating “Goo”, and proceeds to step S9 in FIG.

  FIG. 11 is a flowchart illustrating an example of a process flow for determining one finger of step S171 in FIG. Referring to FIG. 11, in step S421, MCU 3 determines whether or not the value of variable Q (number of fingertips) is 1 or less, and proceeds to step S423 if affirmative determination is made, and if negative determination is performed. The process proceeds to step S447. In step S447, the MCU 3 sets the flag SH to 00h indicating “indefinite” and returns.

  On the other hand, in step S423, the MCU 3 substitutes 0 for the variables j, s, Dc [], Dca, and Dcm. In step S425, the MCU 3 determines whether or not 1 indicating a fingertip candidate (that is, a convex point) is set in the flag FP [j]. If the determination is affirmative, the process proceeds to step S427. Otherwise, the MCU 3 proceeds to step S433. Proceed to

  In step S427, the MCU 3 calculates a distance D between the barycentric coordinates (Xg, Yg) and the convex pixel Pj. In step S429, the MCU 3 substitutes the distance D into the array Dc [s]. In step S431, the MCU 3 increments the variable s by one. In step S433, the MCU 3 increments the variable j by one. In step S435, the MCU 3 determines whether or not the value of the variable j has become the value J. If the value has become the value J, the process proceeds to step S437. Otherwise, the process proceeds to step S425.

  In step S437, an average value Dca of the distance Dc [s] is calculated. That is, the average value Dca is the average value of the distances from the center of gravity to each convex point. In step S439, the maximum value Dcm is detected from the distance Dc [s]. That is, the maximum value Dcm is the maximum of the distances from the center of gravity to each convex point. In step S441, the MCU 3 determines whether or not the maximum value Dcm is equal to or greater than a value obtained by multiplying the average value Dca by 1.3. If the determination is affirmative, the MCU 3 regards that one finger is standing. The process proceeds to S443, and in the case of a negative determination, the process proceeds to Step S447.

  In step S443, the MCU 3 sets the flag SH to 40h indicating “a state where one finger is standing”. In step S445, the MCU 3 substitutes the X coordinate of the pixel Pj of the convex point corresponding to the maximum value Dcm for the coordinate XH. Further, the MCU 3 substitutes the Y coordinate of the pixel Pj of the convex point corresponding to the maximum value Dcm for the coordinate YH. The coordinates (XH, YH) are the coordinates of one fingertip in a standing state. In step S447, the MCU 3 calculates a correction value (Xc, Yc) of the coordinates (XH, YH). In step S449, the MCU 3 adds the correction value (Xc, Yc) to the coordinate (XH, YH) to obtain the corrected coordinate (Xf, Yf). Then, the MCU 3 returns.

  Next, the coordinates (XL, YL) and (XS, YS) of the pixel indicating the fingertip obtained in steps S373 and S375 of FIG. 9, and the coordinates of the pixel indicating the fingertip obtained in step S445 of FIG. A method of recalculating XH, YH) with high accuracy exceeding the resolution of the image sensor 5 will be described. Hereinafter, the coordinates of the pixel P4 indicating the fingertip before recalculation are expressed as coordinates (Xb, Yb). Then, the correction value (Xc, Yc) of the coordinates (Xb, Yb) is obtained as follows. In the present embodiment, first to fourth examples will be given as correction value calculation methods.

  FIG. 12A is a flowchart showing a flow of a first example of the fingertip correction value calculation process in step S377 in FIG. 9 and step S447 in FIG. FIG. 12B is a flowchart showing the flow of the second example of the fingertip correction value calculation process in step S377 in FIG. 9 and step S447 in FIG. FIG. 12C is a flowchart showing the flow of the third example of the fingertip correction value calculation process in step S377 in FIG. 9 and step S447 in FIG. FIG. 12D is a flowchart showing the flow of the fourth example of the fingertip correction value calculation process in step S377 in FIG. 9 and step S447 in FIG.

  FIG. 16A is an explanatory diagram of the fingertip correction value calculation process of FIGS. 12A and 12C. FIG. 16B is an explanatory diagram of the fingertip correction value calculation process of FIGS. 12B and 12D.

  With reference to FIGS. 16A and 16B, the coordinates (Xb, Yb) of the fingertip pixel P4 before correction (before recalculation) are set as the origin (0, 0). In the figure, the horizontal right direction is positive on the x axis, and the vertical downward direction is positive on the y axis. Then, eight pixels P0, P1, P2, P5, P6, P7 and P8 surrounding the fingertip pixel P4 before correction are used. The pixels P0 to P8 have pixel values (that is, luminance values) p0 to p8, respectively.

  As shown in FIG. 16A, in the first and third examples, the coordinates of the pixels P1, P7, P3, and P5 on the top, bottom, left, and right of the fingertip pixel P4 at the origin (0, 0) are coordinates (0 , -1), (0, 1), (-1, 0) and (1, 0). The coordinates of the upper left pixel P0, upper right pixel P2, lower left pixel P6, and lower right pixel P8 of the fingertip pixel P4 are coordinates (−√2 / 2, −√2 / 2), (√2 / 2, − √2 / 2), (−√2 / 2, √2 / 2) and (√2 / 2, √2 / 2).

  On the other hand, as shown in FIG. 16B, in the second and fourth examples, the coordinates of the surrounding pixels P0, P1, P2, P5, P6, P7 and P8 of the fingertip pixel P4 at the origin (0, 0) are , Coordinates (-1, -1), (0, -1), (1, -1), (-1, 0), (1, 0), (-1, 1), (0, 1), respectively. ) And (1,1).

  In the following formulas, symbols “*” and “/” indicate multiplication and division, respectively.

  With reference to FIG. 12A, in the first example, in steps S361 and S363, the MCU 3 calculates the correction value Xc of the X coordinate Xb and the correction value Yc of the Y coordinate Yb by the following equations. The value Cs is a constant. The constant Cs is a value greater than 1.

Xc = (A / C) * Cs
Yc = (B / C) * Cs

A = −R * p0 + 0 * p1 + R * p2 + (− 1) * p3 + 0 * p4 + 1 * p5 + (− R) * p6 + 0 * p7 + R * p8
B = −R * p0 + (− 1) * p1 + (− R) * p2 + 0 * p3 + 0 * p4 + 0 * p5 + R * p6 + 1 * p7 + R * p8
C = p0 + p1 + p2 + P3 + p4 + p5 + p6 + p7 + p8
R = √2 / 2

  Therefore, the corrected X coordinate Xf and Y coordinate Yf of the fingertip are as follows.

Xf = Xb + Xc
Yf = Yb + Yc

  Referring to FIG. 12B, in the second example, in steps S371 and S373, the MCU 3 calculates the correction value Xc of the X coordinate Xb and the correction value Yc of the Y coordinate Yb by the following equations. The value Cs is a constant.

Xc = (D / C) * Cs
Yc = (E / C) * Cs

D = -1 * p0 + 0 * p1 + 1 * p2 + (-1) * p3 + 0 * p4 + 1 * p5 + (-1) * p6 + 0 * p7 + 1 * p8
E = -1 * p0 + (-1) * p1 + (-1) * p2 + 0 * p3 + 0 * p4 + 0 * p5 + 1 * p6 + 1 * p7 + 1 * p8
C = p0 + p1 + p2 + P3 + p4 + p5 + p6 + p7 + p8

  Therefore, the corrected X coordinate Xf and Y coordinate Yf of the fingertip are as follows.

Xf = Xb + Xc
Yf = Yb + Yc

  Referring to FIG. 12C, in the third example, in steps S381 and S383, the MCU 3 calculates the correction value Xc of the X coordinate Xb and the correction value Yc of the Y coordinate Yb by the following equations. The value pmax indicates the maximum value among the pixel values p0 to p8.

Xc = A / pmax
Yc = B / pmax

A = −R * p0 + 0 * p1 + R * p2 + (− 1) * p3 + 0 * p4 + 1 * p5 + (− R) * p6 + 0 * p7 + R * p8
B = −R * p0 + (− 1) * p1 + (− R) * p2 + 0 * p3 + 0 * p4 + 0 * p5 + R * p6 + 1 * p7 + R * p8
R = √2 / 2

  Therefore, the corrected X coordinate Xf and Y coordinate Yf of the fingertip are as follows.

Xf = Xb + Xc
Yf = Yb + Yc

  Referring to FIG. 12D, in the fourth example, in steps S391 and S393, the MCU 3 calculates the correction value Xc of the X coordinate Xb and the correction value Yc of the Y coordinate Yb by the following equations. The value pmax indicates the maximum value among the pixel values p0 to p8.

Xc = D / pmax
Yc = E / pmax

D = -1 * p0 + 0 * p1 + 1 * p2 + (-1) * p3 + 0 * p4 + 1 * p5 + (-1) * p6 + 0 * p7 + 1 * p8
E = -1 * p0 + (-1) * p1 + (-1) * p2 + 0 * p3 + 0 * p4 + 0 * p5 + 1 * p6 + 1 * p7 + 1 * p8

  Therefore, the corrected X coordinate Xf and Y coordinate Yf of the fingertip are as follows.

Xf = Xb + Xc
Yf = Yb + Yc

  Here, in the fingertip correction value calculation process in step S377 of FIG. 9, the MCU 3 sets the coordinates (XL, YL) and (XS, YS) as the coordinates (Xb, Yb) of the fingertip P4, and sets the correction values. calculate. In the fingertip correction value calculation process in step S447 in FIG. 11, the correction value is calculated using the coordinates (XH, YH) as the coordinates (Xb, Yb) of the fingertip P4.

  The fourth example will be supplementarily described with reference to FIG.

  In the above formula of the fourth example, the coordinates (0, 0), (-1, -1), (0, -1) of the fingertip pixel P4 and the surrounding pixels P0, P1, P2, P5, P6, P7 and P8. , (1, -1), (-1, 0), (1, 0), (-1, 1), (0, 1) and (1, 1) corresponding pixel values p4, p0, p1 , P2, P3, p5, p6, p7 and p8, and the sum D thereof is calculated. Then, the correction value Xc is obtained by dividing the sum D by the value pmax.

  In this way, a correction value Xc with accuracy having a decimal point is obtained. Note that the coordinate Xb is an integer value because it is obtained in units of pixels. Therefore, by adding the correction value Xc to the coordinate Xb, a coordinate value with decimal point precision can be obtained. These are also true for the Y coordinate.

  In addition, by adding the correction value Xc obtained by weighting with the pixel value to the X coordinate Xb before correction, the X coordinate Xb can be corrected to the larger pixel value (luminance value) (depending on the pixel value). Weight). This is also true for the Y coordinate. That is, by obtaining the inflection point from the declination angle θj, the pixel corresponding to the fingertip is specified from the pixels on the contour line of the hand (Steps S373 and S375 in FIG. 9 and Step S445 in FIG. 11), and further, the pixel value By adding the above, the coordinates of the fingertip are corrected at a position having a larger pixel value around the pixel specified on the contour line. This is because it is possible to more appropriately specify the fingertip by regarding the position of the tip portion of the image of the finger reflected in the image and having a larger pixel value (luminance value) as the fingertip.

  Further, the sum D is divided by the maximum value pmax among the pixel values p0 to p8. That is, the divisor is not a fixed value but a variable value. This is because the distance from the image sensor 5 to the hand is not constant. That is, when the distance from the image sensor 5 to the hand is short, the finger appears large in the image, while when the distance from the image sensor 5 to the hand is long, the finger appears small in the image, the correction value This is because Xc is set to an appropriate value according to the distance from the image sensor 5 to the hand. The same applies to the correction value Yc.

  For example, when the divisor is a fixed value, if the fixed value is too small, it may be a relatively reasonable value if the distance from the image sensor 5 to the hand is long, but if it is close, the correction value is large. It becomes too much. On the other hand, when the divisor is a fixed value, if the fixed value is too large, it may be a relatively reasonable value when the distance from the image sensor 5 to the hand is short, but if it is far, it is corrected. The value becomes too small. In addition, it is difficult to set a proper divisor as a fixed value for both cases where the distance from the image sensor 5 to the hand is long and close. This is because even when the same finger is photographed at the same distance, the pixel value (luminance value) varies depending on the environment (lighting, outside light, etc.) in which the image sensor 5 is installed, and the finger color, This is because the contour, the shape of the surface, and the like vary from person to person.

  A weighted average can be obtained by dividing by the value C (= p0 + p1 + p2 + P3 + p4 + p5 + p6 + p7 + p8). However, for the following reason, it is preferable that the divisor is not the value C but the maximum value pmax.

  Referring to FIG. 17, an actual human fingertip 30 generally has an arc shape with a certain point being a vertex (referred to as “real vertex”) 32 in plan view. On the other hand, when detecting the fingertip from a relatively low resolution image and determining the fingertip as in the present embodiment, it is difficult to always detect the actual vertex 32 as the fingertip. Therefore, the detected fingertip is the actual vertex 32, the left point 34 of the actual vertex 32, or the right point 36.

  In this case, when the sum D is divided by the value C to obtain a correction value, the correction value cannot exceed the range of 9 × 9 pixels centered on the pixel P4 in FIG. The same applies to the sum E. Therefore, for example, the corrected position of the real vertex 32 in FIG. 17 is the point 38, the corrected position of the point 34 is the point 40, and the corrected position of the point 36 is the point 42. Thus, the corrected fingertip position is different in these three points. If a different part is recognized as a fingertip to detect the same fingertip, the cursor placed at the detected fingertip position is not stable.

  Therefore, even when the fingertip obtained from the contour is not constant, the sums D and E are calculated with the maximum value pmax in order to make the corrected position as constant as possible, or to keep the difference in the fingertip position in the smallest possible range. In other words, the correction value exceeds the range of 9 × 9 pixels, and the corrected position is gathered near the point 44 in FIG. 17, that is, near the center of the belly of the finger tip 30.

  The first example will be supplementarily described with reference to FIG. In this example, unlike the fourth example, the weighting targets of the pixels P0, P2, P6, and P8 are coordinates (−√2 / 2, −√2 / 2), (√2 / 2, −√, respectively. 2/2), (−√2 / 2, √2 / 2) and (√2 / 2, √2 / 2). This is due to the following reason.

  This is because all pixel values are treated equally when the distance from the center of the fingertip pixel P4 before correction is set to the same value for all surrounding pixels and the pixel values are weighted. For example, the distance between the pixel P4 and the pixel P5 is “1”, but when the coordinates of the pixel P2 are calculated as (1, −1), the distance between the pixel P4 and the pixel P2 is “√2”. And an imbalance occurs depending on the positions of surrounding pixels. By setting the distance from the center of the fingertip pixel P4 before correction to the same value for all the surrounding pixels, the correction value can be made uneven depending on the direction.

  In this example, A / C and B / C are multiplied by a constant Cs. This is the same as the reason that the divisor is not set to the value C and the weighted average is not taken in the fourth example. In other words, even when the fingertip obtained from the contour is not constant, the correction value is multiplied by a constant Cs in order to make the corrected position as constant as possible, or to keep the difference in the fingertip position in the smallest possible range. The range of 9 × 9 pixels is exceeded, and the corrected positions are gathered at point 44 in FIG. Note that A / C and B / C can also be used as correction values without multiplying by the constant Cs.

  Here, the constant Cs is determined by experiments and trial and error. The constant Cs can also be dynamically changed according to the detected finger width. For example, the constant Cs is determined by calculating the width of the finger from the image and multiplying the width by a predetermined number (decimal value). This predetermined number is determined by experiments and trial and error. For example, this predetermined number is 2/3. Note that, for example, a distance between left and right pixels that are a fixed number of pixels away from each other on the left and right of the fingertip pixel before correction is obtained and used as the finger width.

  The reason why the weighted average is obtained by dividing the sums A and B by the value C is to obtain accuracy correction values Xc and Yc having decimal points. Note that the coordinate Xb is an integer value because it is obtained in units of pixels. Therefore, by adding the correction value Xc to the coordinate Xb, a coordinate value with decimal point precision can be obtained. These are also true for the Y coordinate.

  Further, the reason for performing weighting by the pixel value is the same as in the fourth example.

  A supplementary explanation will be given of the second example. The reason for multiplying by the constant Cs, the reason for taking the weighted average, and the reason for weighting by the pixel value are the same as in the first example.

  A supplementary explanation will be given of the third example. The reason for taking the weighted average by dividing the sums A and B by the maximum value pmax is to obtain precision correction values Xc and Yc having decimal points. Note that the coordinate Xb is an integer value because it is obtained in units of pixels. Therefore, by adding the correction value Xc to the coordinate Xb, a coordinate value with decimal point precision can be obtained. These are also true for the Y coordinate.

  Also, the weighting targets of the pixels P0, P2, P6, and P8 are coordinates (−√2 / 2, −√2 / 2), (√2 / 2, −√2 / 2), and (−√2), respectively. / 2, √2 / 2) and (√2 / 2, √2 / 2) are the same as in the first example.

  Further, the reason why the pixel value is weighted, the reason why the divisor is a variation value, and the reason why the weighted average is not obtained by dividing by the maximum value pmax are the same as in the fourth example.

  FIG. 13A is a flowchart showing the flow of a first example of the pseudo button operation determination process in step S9 of FIG. 3 (corresponding to the first example of FIG. 1B). Referring to FIG. 13A, in step S621, the MCU 3 turns off a pseudo button flag indicating whether or not a pseudo button operation has been performed. In step S623, the MCU 3 determines whether or not 40h is set in the current flag SH indicating the shape of the hand, and if 40h is set (that is, the state of “one finger”), the step Proceed to S627, otherwise return.

  In step S627, the MCU 3 determines whether or not 20h is set in the previous flag SH indicating the shape of the hand, and when 20h is set (that is, in the “shrink” state), the process proceeds to step S629. Otherwise, return. In step S629, the MCU 3 turns on the pseudo button flag to indicate that the button operation has been performed in a pseudo manner.

  FIG. 13B is a flowchart showing the flow of the second example of the pseudo button operation determination process in step S9 of FIG. 3 (corresponding to the second example of FIG. 1B). Referring to FIG. 13B, in step S601, the MCU 3 turns off a pseudo button flag indicating whether or not a pseudo button operation has been performed. In step S603, the MCU 3 determines whether or not 20h is set in the current flag SH indicating the shape of the hand. If 20h is set (that is, the state of “shrink”), the process proceeds to step S607. Proceed, otherwise return.

  In step S607, the MCU 3 determines whether or not 40h is set in the previous flag SH indicating the shape of the hand. If 40h is set (that is, the state of “one finger”), step S609 is performed. Proceed to, otherwise return. In step S609, the MCU 3 turns on the pseudo button flag to indicate that the button operation has been performed in a pseudo manner.

  FIG. 13C is a flowchart showing the flow of the third example of the pseudo button operation determination process in step S9 of FIG. 3 (corresponding to the third example of FIG. 1B). With reference to FIG.13 (c), MCU3 turns off the pseudo button flag which shows the presence or absence of pseudo button operation in step S641. In step S643, the MCU 3 determines whether or not 20h is set in the current flag SH indicating the shape of the hand. If 20h is set (that is, the state of “shrink”), the process proceeds to step S645. Proceed, otherwise return.

  In step S645, the MCU 3 calculates a distance DC between the coordinate of one fingertip after correction and the coordinate of the other fingertip after correction. In step S647, the MCU 3 determines whether or not the distance DC is smaller than the certain value CC. If the distance DC is smaller, the MCU 3 determines that one fingertip has approached the other fingertip, and proceeds to step S649. Otherwise, the MCU 3 returns. In step S649, MCU 3 turns on the pseudo button flag to indicate that the button operation has been performed in a pseudo manner.

  Here, in FIGS. 13A to 13C and FIGS. 18A to 18C, which will be described later, the current flag SH refers to one frame of image data received from the image sensor 5 this time. It means the flag set based on the above. The previous flag SH means a flag set based on one frame of image data received from the image sensor 5 last time.

  13 (a) to 13 (c), a click operation (an operation for pressing and releasing a button) is performed when it is determined that a button operation has been performed in a pseudo manner (that is, the pseudo button flag is turned on). 4 shows an example of processing when it is assumed that has been performed.

  Next, referring to FIG. 18A to FIG. 18C, it is determined that the button operation has been performed in a pseudo manner (that is, the pseudo button flag is turned on). An example of processing that is considered to have been performed) will be described. Note that turning off the pseudo button flag is regarded as a state in which the button is released.

  FIG. 18A is a flowchart showing the flow of a fourth example of the pseudo button operation determination process in step S9 of FIG. 3 (corresponding to the first example of FIG. 1B). Referring to FIG. 18A, in step S701, the MCU 3 determines whether or not a pseudo button flag indicating whether or not a pseudo button operation has been performed, the process proceeds to step S709 if it is on, and if it is off. The process proceeds to step S703. Note that the pseudo button flag is set to off at the time of system initialization.

  In step S703, the MCU 3 determines whether or not the current flag SH indicating the shape of the hand is set to 40h, and if 40h is set (that is, the state of “one finger”), the step Proceed to S705, otherwise return.

  In step S705, the MCU 3 determines whether or not 20h is set in the previous flag SH indicating the shape of the hand. If 20h is set (that is, the state of “shrink”), the process proceeds to step S707. Otherwise, return. In step S707, the MCU 3 turns on the pseudo button flag to indicate that the button has been pseudo-pressed.

  On the other hand, in step S709, the MCU 3 determines whether or not 20h is set in the current flag SH, and when 20h is set (that is, in the “shock” state), the process proceeds to step S711, otherwise Assumes that the button press state is maintained and returns. In step S711, the MCU 3 turns off the pseudo button flag to indicate that the button has been released in a pseudo manner.

  FIG. 18B is a flowchart showing the flow of the fifth example of the pseudo button operation determination process in step S9 of FIG. 3 (corresponding to the second example of FIG. 1B). Referring to FIG. 18B, in step S731, the MCU 3 determines whether or not a pseudo button flag indicating whether or not a pseudo button operation is performed is on. If it is on, the MCU 3 proceeds to step S739, and if it is off. The process proceeds to step S733. Note that the pseudo button flag is set to off at the time of system initialization.

  In step S733, the MCU 3 determines whether or not the current flag SH indicating the shape of the hand is set to 20h. If 20h is set (that is, the state of “shrink”), the process proceeds to step S735. Proceed, otherwise return.

  In step S735, the MCU 3 determines whether or not 40h is set in the previous flag SH indicating the shape of the hand. If 40h is set (that is, the state of “one finger”), step S737 is performed. Proceed to, otherwise return. In step S737, the MCU 3 turns on the pseudo button flag to indicate that the button has been pseudo-pressed.

  On the other hand, in step S739, the MCU 3 determines whether or not the current flag SH is set to 40h. If 40h is set (that is, “one finger” state), the process proceeds to step S741. Otherwise, the button press state is assumed to be maintained and the process returns. In step S741, the MCU 3 turns off the pseudo button flag to indicate that the button has been pseudo released.

  FIG. 18C is a flowchart showing the flow of the sixth example of the pseudo button operation determination process in step S9 of FIG. 3 (corresponding to the third example of FIG. 1B). Referring to FIG. 18C, in step S761, the MCU 3 determines whether or not a pseudo button flag indicating whether or not a pseudo button operation is performed is on. If it is on, the MCU 3 proceeds to step S771, and if it is off. The process proceeds to step S763. Note that the pseudo button flag is set to off at the time of system initialization.

  In step S763, the MCU 3 determines whether or not 20h is set in the current flag SH indicating the shape of the hand. If 20h is set (that is, the state of “shrink”), the process proceeds to step S765. Proceed, otherwise return.

  In step S765, the MCU 3 calculates a distance DC between the coordinate of one fingertip after correction and the coordinate of the other fingertip after correction. In step S767, the MCU 3 determines whether or not the distance DC is smaller than the certain value CC. If the distance DC is smaller, the MCU 3 determines that one fingertip has approached the other fingertip, and proceeds to step S769, and otherwise returns. In step S769, the MCU 3 turns on the pseudo button flag to indicate that the button has been pressed in a pseudo manner.

  On the other hand, in step S771, the MCU 3 determines whether or not 20h is set in the current flag SH. If 20h is set (that is, the state of “pick”), the process proceeds to step S773. Otherwise, the button press state is assumed to be maintained and the process returns.

  In step S <b> 773, the MCU 3 calculates a distance DC between the coordinate of one fingertip after correction and the coordinate of the other fingertip after correction. In step S775, the MCU 3 determines whether or not the distance DC is equal to or greater than a certain value CC. If the distance DC is equal to or greater than CC, the MCU 3 assumes that the button has been released in a pseudo manner, and proceeds to step S777. It is assumed that it has been done and returns. In step S777, the MCU 3 turns off the pseudo button flag to indicate that the button has been pseudo released.

  By the way, in the above, the 1st-3rd example of pseudo button operation was demonstrated with reference to FIG.1 (b). Hereinafter, a fourth example of the pseudo button operation will be described. In the fourth example, single click and double click are determined.

  FIG. 19A is an explanatory diagram of a fourth example (single click) of the pseudo button operation according to the embodiment of the present invention. Referring to FIG. 19A, when a change in steps S2000 to S2004 is detected within a certain time TC, it is considered that a single click has been performed.

  Specifically, when the fingertip 50 is almost stationary, a rectangular range (for example, 3 pixels × 3 pixels) 52 around the latest fingertip 50 is set (step S2000). For example, when there are a predetermined number of fingertips 50 within a certain distance RD from the latest fingertip 50, it is determined that the fingertips 50 are substantially stationary.

  Within a predetermined time TC from the setting of the rectangular range 52, the fingertip 50 disappears (exits) from the rectangular range 52, and the fingertip 50 does not exist in the rectangular range 52 (step S2002). When the fingertip 50 appears (enters) at 52 and the fingertip 50 is present in the rectangular area 52 (step S2004), it is considered that a single click has been performed. For example, the total value of the pixel values (luminance values) in the rectangular range 52 is compared with a predetermined threshold, and if the total value exceeds the threshold, it is determined that the fingertip 50 exists in the rectangular range 52, while the total When the value is equal to or smaller than the threshold value, it is determined that the fingertip 50 does not exist in the rectangular range 52.

  In addition, for example, within a certain time TC, the user bends the third joint (joint of the base) of the index finger, folds the finger from the state where the index finger is extended, and returns to the original state (state where the index finger is extended). It is considered that a single click has been made when making the gesture of returning. In this case, the states of the first and second joints are arbitrary. Also, for example, within a certain time TC, the user bends the second joint of the index finger and folds the finger (the third joint is fixed) from the state in which the index finger is extended, and again the original state (state in which the index finger is extended) ), It is considered that a single click has been made. In this case, the state of the first joint is arbitrary. Here, the joint closest to the fingertip is called the first joint, the next joint is called the second joint, and the next joint is called the third joint.

  In the human gesture as in this example, it has been verified by experiments of the present inventor that the fingertip returns almost to the original position after the finger is bent. For this reason, it is possible to determine whether a single click has been performed stably and reliably, that is, with high reproducibility, by the method described with reference to FIG.

  FIG. 19B is an explanatory diagram of a fourth example (double click) of the pseudo button operation according to the embodiment of the present invention. Referring to FIG. 19B, when a change in steps S2100 to S2108 is detected within a certain time TC, it is considered that a double click has been performed. The fixed time TC in this case is the same as the fixed time TC when determining a single click.

  Specifically, when the fingertip 50 is almost stationary, a rectangular range (for example, 3 pixels × 3 pixels) 52 around the latest fingertip 50 is set (step S2100). The determination method of the state in which the fingertip 50 is substantially stationary is the same as in the case of single click.

  Within a predetermined time TC from the setting of the rectangular range 52, the fingertip 50 disappears (exits) from the rectangular range 52, and the fingertip 50 does not exist in the rectangular range 52 (step S2102). The fingertip 50 appears (enters) in 52, the fingertip 50 exists in the rectangular range 52 (step S2104), the fingertip 50 disappears (exits) from the rectangular range 52 again, and the fingertip 50 enters the rectangular range 52. 50 is not present (step S2106), and when the fingertip 50 again appears (enters) in the rectangular area 52 and the fingertip 50 is present in the rectangular area 52 (step S2108), double Assume that a click has been made. The determination of whether or not the fingertip 50 is present in the rectangular area 52 is the same as in the case of single click.

  The double click is executed by the user performing the single click gesture twice. In addition, the double click is a single click performed twice, and therefore the folded fingertip returns to almost the same position both times. This point has also been verified by the present inventors. For this reason, it is possible to determine whether double-clicking has been performed stably and reliably, that is, with high reproducibility, by the method described with reference to FIG.

  20 and 21 are flowcharts showing the flow of the seventh example of the pseudo button operation determination process in step S9 of FIG. 3 (corresponding to the fourth example of FIGS. 19A and 19B). . Referring to FIG. 20, in step S1200, MCU 3 sets a single click flag that is set to ON when it is determined that a single click has been performed, and a double click flag that is set to ON when it is determined that a double click has been performed. , Turn off.

  In step S1202, the MCU 3 determines whether or not the stay flag is on. If the stay flag is on, the MCU 3 proceeds to step S1250 in FIG. 21, and if it is off, the MCU 3 proceeds to step S1204. The stay flag is a flag that is set to ON when it is determined that the fingertip 50 is substantially stationary, that is, when the rectangular range 52 in FIGS. 19A and 19B is set. Accordingly, the processing in steps S1204 to S1220 after the negative determination is made in step S1202 is processing for setting the rectangular range 52. On the other hand, the processing of steps S1250 to S1284 after the affirmative determination is made in step S1202 is processing for determining the states of steps S2002, S2004, and S2102 to S2108 in FIGS. 19A and 19B. .

  In step S1204, the MCU 3 sets 0 to the variable k. In step S1206, the MCU 3 calculates a distance DIS between the latest fingertip 50 and the past fingertip 50. The distance DIS is calculated by the following equation.

DIS = √ (XD ² + YD ² )
XD = Xf [0] −Xf [k + 1]
YD = Yf [0] −Yf [k + 1]

  Here, in step S449 of FIG. 11, the coordinates (Xf, Yf) of the corrected fingertip 50 are calculated. In this case, the corrected coordinates (Xf, Yf) of the fingertip 50 are stored in the queue, and a total of N pieces of current and past data are stored (N is an integer of 2 or more. In this embodiment, N = 10). The queue is composed of arrays Xf [0] to Xf [9] for substituting X coordinates Xf and arrays Yf [0] to Yf [9] for substituting Y coordinates Yf. The latest coordinates (Xf, Yf) are always stored in the arrays Xf [0] and Yf [0].

  Therefore, in step S1206, the latest fingertip 50 coordinates (Xf [0], Yf [0]) and (k + 1) past fingertip 50 coordinates (Xf [k + 1], Yf [k + 1]) The distance DIS between is calculated.

  In step S1208, the MCU 3 determines whether or not the distance DIS is smaller than the fixed distance RD. If the distance DIS is smaller, the process proceeds to step S1210. Otherwise, the MCU 3 determines that the fingertip 50 is not stationary. Proceed to In step S1210, the MCU 3 increments the variable k by one. In step S1212, the MCU 3 determines whether or not the value of the variable k has become 9, the process proceeds to step S1214 if it has reached 9, otherwise the process returns to step S1206.

  The determination of k = 9 in step S1212 means that all the distances between the latest fingertip 50 and the past nine fingertips 50 exist within a certain distance RD. That is, it means that the fingertip 50 is almost stationary. For this reason, in step S1214, the MCU 3 turns on the stay flag. On the other hand, if there is even one past fingertip 50 whose distance DIS is not within the fixed distance RD, it is determined that the fingertip 50 is not stationary. Therefore, in step S1220, the MCU 3 turns off the stay flag and returns.

  In step S1216 following step S1214, the MCU 3 sets a rectangular range 52 centered on the coordinates (Xf [0], Yf [0]) of the latest fingertip 50 (steps S2000 and S2000 in FIG. 19A). This corresponds to step S2100 in FIG. In step S1218, the MCU 3 sets the timer TM and returns. This timer TM is for measuring the above-mentioned fixed time TC (see FIGS. 19A and 19B). This fixed time TC is determined by experiments and trial and error.

  Referring to FIG. 21, in step S1250, the MCU 3 determines whether or not the value of the timer TM has reached a certain time TC. If the certain time TC has been reached, the MCU 3 proceeds to step S1278 and has not reached the certain time. If yes, go to Step S1252.

  In step S1252, the MCU 3 determines whether or not the disappearance flag is on. If the flag is on, the process proceeds to step S1262. If the flag is off, the process proceeds to step S1254. The disappearance flag is a flag that is turned on when the state in which the fingertip 50 is present in the rectangular range 52 is changed to a state in which the fingertip 50 does not exist.

  In step S1254, the MCU 3 determines whether or not the fingertip 50 is not present in the rectangular range 52, returns if it exists, and proceeds to step S1256 if it does not exist. In step S1256, the MCU 3 returns with the disappearance flag turned on (corresponding to steps S2002 in FIG. 19A and steps S2102 and S2106 in FIG. 19B).

  If the disappearance flag is on, in step S1262, the MCU 3 determines whether or not the fingertip 50 exists in the rectangular range 52, returns if not, and proceeds to step S1264 if it exists (FIG. 19A). Step S2004 and steps S2104 and S2108 in FIG. 19B).

  In step S1264, the MCU 3 determines whether or not the temporary click flag is on. If the temporary click flag is on, the process proceeds to step S1272, and if it is off, the process proceeds to step S1266. The temporary click flag is a flag that is turned on when the user's gesture satisfies a single click condition (steps S2000 to S2004 in FIG. 19A). However, since the double click includes a single click operation (steps S2100 to S2104 in FIG. 19B), there is a possibility that the double click is performed until a predetermined time TC elapses. For this reason, it is necessary to finally determine whether single-clicking or double-clicking after a certain time TC has elapsed. Therefore, when the single click condition is satisfied, the fact is held until the final decision by the temporary click flag.

  In step S1266, the MCU 3 turns on the temporary click flag because the single click condition is satisfied. In step S1268, the MCU 3 turns off the disappearance flag and returns. This is because an affirmative determination is made in step S1262.

  On the other hand, in step S1272, the MCU 3 turns on the double click flag. This is because the change in steps S2106 and S2108 of FIG. 19B is further detected in the state where the temporary click flag is on (positive determination in step S1264, positive determination in step S1252, and step S1262). Affirmative). In step S1274, the MCU 3 releases the timer TM. In step S1276, the MCU 3 turns off the stay flag, the disappearance flag, and the temporary click flag, and returns. Because double-clicking has been decided, this is the final decision. Note that turning off the stay flag corresponds to releasing the rectangular range 52, that is, resetting the click operation determination.

  Now, after a positive determination is made in step S1250, the MCU 3 proceeds to step S1278 to make a final decision. In step S1278, the MCU 3 determines whether or not the temporary click flag is on. If the temporary click flag is on, the process proceeds to step S1280. If the temporary click flag is off, the MCU 3 proceeds to step S1282. In step S1280, the MCU 3 turns on the single click flag and proceeds to step S1282.

  In step S1282, the MCU 3 releases the timer TM. In step S1284, the MCU 3 turns off the stay flag, the disappearance flag, and the temporary click flag, and returns. Note that turning off the stay flag corresponds to releasing the rectangular range 52, that is, resetting the click operation determination.

  The single click flag and the double click flag are transmitted to the computer 11 in step S11 of FIG.

  As described above, according to the present embodiment, the user can artificially change the state where one finger is raised to another state where the other finger is raised and the two fingers are raised. Button operation can be performed. In this manner, a button operation can be performed in a pseudo manner by a gesture that is easy for the user (first example in FIG. 1B).

  Also, since one finger is always photographed before and after the pseudo button operation, the cursor can be displayed at the position on the screen corresponding to the fingertip, and pointing and pseudo button operation can be performed with one hand. Yes (first example of FIG. 1B).

  In addition, according to the present embodiment, the user can perform a button operation in a pseudo manner by simply closing another finger and raising one finger from a state where two fingers are raised. Can do. In this way, it is possible to perform a pseudo button operation with a gesture that is easy for the user (second example in FIG. 1B).

  Also, since one finger is always photographed before and after the pseudo button operation, the cursor can be displayed at the position on the screen corresponding to the fingertip, and pointing and pseudo button operation can be performed with one hand. Yes (second example in FIG. 1B).

  Furthermore, according to the present embodiment, the user can perform a pseudo button operation only by bringing the fingertips of two fingers close to each other. In this manner, a button operation can be performed in a pseudo manner with a gesture that is easy for the user (third example in FIG. 1B).

  In addition, a cursor can be displayed between two detected fingertips (for example, the middle point), and pointing and pseudo button operations can be performed with one hand (third example in FIG. 1B). .

  Furthermore, according to the present embodiment, the user simply performs a gesture in which the finger is folded from the state in which the finger is raised in the three-dimensional space, and then returned to the original state. A non-contact click operation can be performed. Thus, a click operation can be performed with a gesture that is easy for the user (fourth example in FIG. 19A).

  In such a human gesture, it has been verified by experiments of the present inventor that the fingertip returns almost to its original position after the finger is bent. Therefore, by detecting such a gesture, it can be determined stably and surely, that is, with high reproducibility, whether or not a click operation has been performed.

  Further, according to the present embodiment, the user folds his / her finger from the state where one finger is raised in the three-dimensional space, returns the finger to the original state, and again, the finger again. The contactless double-click operation can be performed only by performing the gesture of returning to the original standing state again, that is, by performing the gesture for single-click operation twice. In this way, a double-click operation can be performed with a gesture that is easy for the user (fourth example in FIG. 19B).

  Such a double click operation is a single click operation performed twice, and it has been verified by experiments of the present inventor that the folded fingertip returns to almost the same position both times. . For this reason, by detecting such a gesture, it can be determined stably and reliably, that is, with high reproducibility, whether or not a double-click operation has been performed.

  Further, in the present embodiment, the MCU 3 performs a click operation (single click or double click) by the user when the movement of the fingertip 50 is within the certain range DIS (positive determination in steps S1208 and S1212 in FIG. 20). The rectangular range 52 is set (step S1216 in FIG. 20), and using this as a trigger, it can be determined whether or not the user has made a gesture corresponding to the click operation. For this reason, it is not always necessary to determine whether or not the user has made the gesture, and the processing load can be reduced.

  Furthermore, the MCU 3 does not determine that the fingertip 50 has disappeared from the rectangular range 52 within a predetermined time TC after the rectangular range 52 is set (step S1254 in FIG. 21), or the fingertip has appeared in the rectangular range 52. Is not determined (step S1262 in FIG. 21), the stay flag is turned off and the rectangular range 52 is released (step S1284 in FIG. 21).

  Thus, when it is not determined that the fingertip 50 has disappeared from the rectangular range 52 within the predetermined time TC, or when it is not determined that the fingertip has appeared in the rectangular range 52, the user performs a click operation (single click or double click). Therefore, it can be determined that the click operation has been performed against the user's intention. That is, in general, a click operation is performed quickly. However, by appropriately setting the predetermined time TC, a motion other than a gesture intended for the click operation can be excluded even if the motion is the same, thereby preventing erroneous determination.

  Furthermore, according to the present embodiment, an approximate fingertip is determined from the contour (steps S373 and S375 in FIG. 9 and step S445 in FIG. 11), and correction is performed using only the surrounding pixels (FIG. 12A). ) To FIG. 12 (d)), the storage capacity for processing can be reduced and the processing speed can be improved.

  Further, since the correction is performed with decimal point accuracy, the position of the fingertip can be determined with high accuracy even when the image sensor 5 has a relatively low resolution. As a result, even when the cursor is displayed at a position corresponding to the fingertip on the screen of the monitor 13 having a resolution higher than that of the image sensor 5, the movement can be made smooth.

  Furthermore, since the correction is performed based on the pixel values of the surrounding pixels, the corrected fingertip can be determined not at the contour line but at the position inside the finger, and closer to the position recognized by the user as the fingertip. As a result, when the cursor is displayed at a position on the screen of the monitor 13 corresponding to the fingertip, the user can easily control the cursor with the fingertip. In general, when a person moves his / her finger toward the belly of a finger, the convex point on the contour of the finger is not recognized as the fingertip, and the center of the finger pad at the tip of the finger is defined as the fingertip. Recognize.

  Further, in the present embodiment, correction is performed so that the corrected fingertip position exceeds the range of surrounding pixels (9 × 9 pixel range). For this reason, the position of the fingertip after correction can be determined as a constant position or a substantially constant position of the belly of the finger tip, and the fingertip detected in steps S373 and S375 in FIG. 9 and step S445 in FIG. Even when the position of the fingertip is not stable, the corrected fingertip position can be made as constant as possible, or the difference in fingertip position can be kept as small as possible. As a result, when the cursor is displayed at a position on the screen of the monitor 13 corresponding to the fingertip, the cursor can be stabilized.

  Further, the corrected fingertip can be made closer to the position that the user recognizes as the fingertip. As a result, when the cursor is displayed at a position on the screen of the monitor 13 corresponding to the fingertip, it is easier for the user to control the cursor with the fingertip.

  Furthermore, in the present embodiment, the distance between the fingertip pixel P4 and the surrounding pixels (P0, P2, P6, P8) located in the diagonal direction of the fingertip pixel P4, the horizontal direction of the fingertip pixel P4 and the fingertip pixel P4, or The coordinates of the surrounding pixels (P0, P2, P6, P8) are defined so that the distances to the surrounding pixels (P1, P3, P5, P7) positioned in the vertical direction are equal (FIG. 16A )reference). As a result, when using pixel values as weights, the pixel values of all surrounding pixels are treated equally, and correction can be performed without deviation depending on the direction.

  Furthermore, according to the present embodiment, the electronic device 9 can execute information processing using the shape of the subject (goo, choki, par, one finger) as an input.

  Furthermore, according to the present embodiment, the finger is specified based on the pixel between the fingertip and the web (see steps S257 and S259 in FIG. 7 and FIG. 15), and the finger is not specified by the center of gravity of the contour image. Compared with the case where the finger is specified by the center of gravity, the finger can be specified with high accuracy (FIG. 7), and the shape of the hand can be determined with higher accuracy (FIGS. 8 and 9). In general, the center of gravity of the contour image and the center of gravity of the subject often do not match. In this case, if a finger is specified by the center of gravity, the accuracy of recognition is lowered.

  In addition, the shape of the hand is determined based on the distance from the center of gravity to the pixel corresponding to the fingertip, as well as the angle between the adjacent finger and the finger obtained based on the pixel between the fingertip and the web. Therefore (steps S345 to S351 in FIG. 9), more various hand shapes can be determined (FIG. 9). For example, it can be determined that the shape of the hand is “pick”.

  Further, the shape of the hand can be determined (FIG. 10) by a simple calculation such as obtaining the distance from the center of gravity to the pixel on the contour line (step S203 in FIG. 6). For example, when the distance from the center of gravity to the pixel on the contour line is within a certain range, it can be determined that the shape of the hand is “Goo”.

  Furthermore, according to the present embodiment, the number of detected fingertips is not determined based only on the result of step S275 in FIG. 7, but the barycentric coordinates (Xg, Yg) and the convex point (FIG. 4) are further determined. In step S33), the distance D to the coordinates of the pixel Pj is taken into account (steps S427 and S441 in FIG. 11), and it is determined whether or not only one finger is standing. For this reason, the state where only one finger stands can be detected with higher accuracy.

  Furthermore, according to the present embodiment, the area of the subject on the image (step S213 in FIG. 6) correlates with the distance between the image sensor 5 and the subject. The distance between them can be easily determined.

  The present invention is not limited to the above-described embodiment, and can be implemented in various modes without departing from the gist thereof. For example, the following modifications are possible.

  (1) In the above description, the shapes of “goo”, “choki”, “par”, and “one finger” are recognized, but the present invention is not limited to this.

  (2) The image processing apparatus 1 can also include an infrared light emitting diode that blinks intermittently. In this case, the image sensor 5 executes a photographing process at the time of emitting the infrared light and at the time of turning off, generates a difference image between the image at the time of emitting light and the image at the time of turning off, and outputs the difference image to the MCU 3. The MCU 3 detects a hand from the difference image. Thus, the background image can be easily removed by a simple process such as difference.

  In this case, when the image sensor 5 serially outputs each pixel (for example, gray scale) of the difference image, the MCU 3 binarizes the pixels received from the image sensor 5 by sequentially comparing with a predetermined threshold value. And stored in the memory 7. And MCU3 performs the various processes mentioned above based on the difference image after binarization. When executing the fingertip correction value calculation processing shown in FIGS. 12A to 12D, the MCU 3 sequentially acquires the pixel values p0 to p8 from the image sensor 5 and substitutes them in the calculation formula. The correction value (Xc, Yc) is calculated. Note that the image sensor 5 holds the difference image data for a certain period of time (until the next shooting process) even after outputting the difference image for binarization by the MCU 3, so that the fingertip correction value calculation process Then, necessary pixels can be acquired therefrom.

  As described above, the memory capacity can be reduced by storing the binarized difference image in the memory 7. Also, in the fingertip correction value calculation processing, instead of processing 3 × 3 pixels (for example, gray scale) stored in the memory 7, output pixels from the image sensor 5 are sequentially processed, which also has a memory capacity. Can be reduced. Of course, the difference image output from the image sensor 5 can be stored in the memory 7 as it is and processed.

  (3) In the above, with the fingertip pixel P4 as the origin (0, 0), the coordinate values of the surrounding pixels are set, the correction value (Xc, Yc) is calculated, and the coordinate value (Xb, Yb) of the fingertip pixel P4 is calculated. To obtain corrected coordinates (Xf, Yf) (FIG. 16 (a) and FIG. 16 (b)). However, the origin of the coordinates used in the equations of the first to fourth examples in FIG. 12 is not limited to this. For example, the coordinates obtained in steps S373 and S375 in FIG. 9 and the coordinates obtained in step S445 in FIG. 11 are used as they are as the coordinates of the fingertip pixel P4 to determine the coordinates of the surrounding pixels, and the correction coordinates (Xf, Yf) are determined. You can ask for it. In this case, the values (Xc, Yc) obtained in the steps of FIGS. 12A to 12D are the correction coordinates (Xf, Yf) themselves, and no addition is necessary.

  (4) In the fourth example of FIG. 19A, when a gesture corresponding to a single click operation is not detected within a predetermined time TC, the rectangular range 52 is canceled and the single click operation determination is reset (FIG. 19). 21 step S1284).

  However, if the MCU 3 does not determine that the fingertip 50 has appeared in the rectangular range 52 within the predetermined time TC2 after determining that the fingertip 50 has disappeared from the rectangular range 52 (Yes in step S1254), the rectangular range 52 is set. It may be canceled and the single click operation determination may be reset. It is the same as FIG. 21 in that the MCU 3 releases the rectangular range 52 when it is not determined that the fingertip 50 has disappeared from the rectangular range 52 within a predetermined time TC after the rectangular range 52 is set.

  As described above, if the MCU 3 does not determine that the fingertip 50 has appeared in the rectangular range 52 within the predetermined time TC2, it is considered that the user does not intend to perform the click operation, and the user clicks against the user's intention. It is possible to prevent the operation from being determined. That is, in general, a click operation is performed quickly. However, by appropriately setting the predetermined time TC2, even if the movement is the same, movement other than the gesture intended for the click operation can be excluded, and erroneous determination can be prevented.

  Further, in the fourth example of FIG. 19B, when the gesture corresponding to the double click operation is not detected within the predetermined time TC, the double click operation determination is reset (step S1284 in FIG. 21).

  However, if the MCU 3 does not determine that the fingertip 50 has appeared in the rectangular range 52 within a predetermined time TC3 after determining that the fingertip 50 has disappeared from the rectangular range 52 after the temporary click flag is turned on, The rectangular range 52 may be canceled and the double-click operation determination may be reset. Note that the MCU 3 releases the rectangular range 52 when the MCU 3 does not determine that the fingertip 50 has disappeared from the rectangular range 52 within a predetermined time TC after setting the rectangular range 52 after the temporary click flag is turned on. Is the same as FIG. The determination until the temporary click flag is turned on (single click determination) is the same as in FIG. 21 or the above modification.

  In this way, if the MCU 3 does not determine that the fingertip 50 has appeared in the rectangular range 52 within the predetermined time TC3, it is considered that the user does not intend to perform a double-click operation, contrary to the user's intention. It can be determined that a double-click operation has been performed. In other words, in general, a double-click operation is performed quickly, but by setting the predetermined time TC3 appropriately, even if the movement is the same, movements other than the gesture intended for the double-click operation are excluded, and erroneous determination is prevented. it can.

  If the MCU 3 determines that the fingertip 50 has appeared in the rectangular range, that is, if the fingertip 50 does not determine that the fingertip 50 has disappeared from the rectangular range 52 within a predetermined time TC4 after the provisional click flag is turned on, The range 52 may be canceled and the double-click operation determination may be reset. The determination until the temporary click flag is turned on (single click determination) is the same as in FIG. 21 or the above modification.

  In this way, if the MCU 3 does not determine that the fingertip 50 has disappeared from the rectangular range 52 within the predetermined time TC4, it is considered that the user does not intend to perform a double-click operation, contrary to the user's intention. It can be determined that a double-click operation has been performed. In other words, in general, the double-click operation is performed quickly, but by setting the predetermined time TC4 appropriately, even if the movement is the same, the movement other than the gesture intended for the double-click operation is excluded to prevent erroneous determination. it can.

  Further, in this case, if the MCU 3 does not determine that the fingertip 50 has appeared again in the rectangular area 52 after the disappearance of the fingertip 50 within the predetermined time TC4 after turning on the temporary click flag, the rectangular area 52 is canceled. The double-click operation determination may be reset.

  In this way, if the MCU 3 does not determine that the fingertip 50 has appeared in the rectangular range 52 within the predetermined time TC4, it is considered that the user does not intend to perform a double-click operation, contrary to the user's intention. It can be determined that a double-click operation has been performed. In other words, in general, the double-click operation is performed quickly, but by setting the predetermined time TC4 appropriately, even if the movement is the same, the movement other than the gesture intended for the double-click operation is excluded to prevent erroneous determination. it can.

  Instead of such determination, the MCU 3 determines that the fingertip 50 has appeared in the rectangular range 52 within the predetermined time TC5 after the disappearance of the fingertip 50 within the predetermined time TC4 after turning on the temporary click flag. If not, the rectangular range 52 can be canceled and the double-click operation determination can be reset.

  In this way, if the MCU 3 does not determine that the fingertip 50 has appeared in the rectangular range 52 within the predetermined time TC5, it is considered that the user does not intend to perform a double-click operation, contrary to the user's intention. It can be determined that a double-click operation has been performed. In other words, in general, a double-click operation is performed quickly, but by setting the predetermined time TC5 appropriately, even if the movement is the same, the movement other than the gesture intended for the double-click operation is excluded to prevent erroneous determination. it can.

  Here, the predetermined times TC2 to TC5 are determined by experiments, trial and error, and the like.

  (5) The computer 11 can also perform all or part of the processing performed by the MCU 3. In short, which process is to be performed can be arbitrarily determined according to the specification.

  (6) Processing steps that describe a program for causing the MCU 3 to perform various processes do not necessarily have to be processed in time series in the order described in the flowchart, and are executed in parallel or individually. It also includes processing.

  (7) In the present specification and claims, the means does not necessarily mean a physical means, but includes a case where the function of each means is realized by software. Further, the function of one means may be realized by two or more physical means, or the functions of two or more means may be realized by one physical means.

  The present invention is applicable to the field of pointing devices that perform pointing by displaying a cursor on a display device.

  The present invention can also be used in the field of image analysis in which a fingertip is detected by image analysis.

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 3 ... MCU, 5 ... Image sensor, 7 ... Memory, 9 ... Electronic device, 11 ... Computer, 13 ... Monitor.

Claims (30)

Photographing means for photographing a user's hand moved in a three-dimensional space;
Detecting means for detecting a portion corresponding to the fingertip of the user from an image obtained by photographing;
Setting means for setting a predetermined range including a portion corresponding to the detected fingertip;
First setting means for determining whether or not a portion corresponding to the fingertip has disappeared after the setting means has set the predetermined range;
Second determining means for determining whether or not a portion corresponding to the fingertip has appeared in the predetermined range after the first determining means determines that the portion corresponding to the fingertip has disappeared from the predetermined range;
An operation determination apparatus comprising: determination means for determining that a click operation has been performed when the second determination means determines that a portion corresponding to the fingertip has appeared in the predetermined range. The operation determination device according to claim 1, wherein the setting unit sets the predetermined range when a motion of a portion corresponding to the detected fingertip is within a certain range.
According to this configuration, when the movement of the portion corresponding to the fingertip is within a certain range, it is assumed that the user intends to perform a click operation, the predetermined range is set, and this is used as a trigger. It can be determined whether or not the user has made a gesture corresponding to the click operation. For this reason, it is not always necessary to determine whether or not the user has made the gesture, and the processing load can be reduced.   The setting means cancels the predetermined range when the first determination means does not determine that the portion corresponding to the fingertip has disappeared from the predetermined range within a first predetermined time after the predetermined range is set. The operation determination device according to claim 1 or 2.   The setting means determines the predetermined range when the second determination means does not determine that a portion corresponding to the fingertip has appeared in the predetermined range within the first predetermined time after the predetermined range is set. The operation determination device according to claim 3, wherein the operation determination device is released.   The setting means includes: a portion corresponding to the fingertip within a second predetermined time after the first determining means determines that the portion corresponding to the fingertip has disappeared from the predetermined range. The operation determination device according to claim 3, wherein when the predetermined range is not determined to have appeared, the predetermined range is canceled. Third determination means for determining whether or not the portion corresponding to the fingertip has disappeared after the second determination means determines that the portion corresponding to the fingertip has appeared in the predetermined range;
Fourth determining means for determining whether or not a portion corresponding to the fingertip has appeared in the predetermined range after the third determining means determines that the portion corresponding to the fingertip has disappeared from the predetermined range; Further comprising
The operation determination according to claim 1 or 2, wherein the determination unit determines that a double-click operation has been performed when the fourth determination unit determines that a portion corresponding to the fingertip has appeared in the predetermined range. apparatus.   The setting means determines the predetermined range when the third determination means does not determine that the portion corresponding to the fingertip has disappeared from the predetermined range within the first predetermined time after the predetermined range is set. The operation determination device according to claim 6, wherein the operation determination device is released.   The setting means determines the predetermined range when the fourth determination means does not determine that a portion corresponding to the fingertip has appeared in the predetermined range within the first predetermined time after the predetermined range is set. The operation determination device according to claim 7, wherein the operation determination device is released.   The setting means includes a portion in which the fourth determination means corresponds to the fingertip within a third predetermined time after the third determination means determines that the portion corresponding to the fingertip has disappeared from the predetermined range. The operation determination device according to claim 7, wherein when the predetermined range is not determined to have appeared, the predetermined range is canceled.   In the setting means, the third determination means corresponds to the fingertip within a fourth predetermined time after the second determination means determines that the portion corresponding to the fingertip has appeared in the predetermined range. The operation determination device according to claim 6, wherein the predetermined range is canceled when it is not determined that the portion has disappeared from the predetermined range.   The said setting means cancels the said predetermined range, when the said 4th determination means does not judge that the part corresponded to the said fingertip appeared in the said predetermined range within the said 4 predetermined time. Operation determination device.   The setting means includes: a part corresponding to the fingertip within the fifth predetermined time after the third determining means determines that the part corresponding to the fingertip has disappeared from the predetermined range. The operation determination device according to claim 10, wherein when the predetermined range is not determined to have appeared, the predetermined range is canceled. Photographing means for photographing a user's hand moved in a three-dimensional space;
Detecting means for detecting a portion corresponding to the user's finger from an image obtained by photographing;
A determination means for determining that a button operation has been performed in a pseudo manner when two portions corresponding to the user's finger were detected last time when one portion corresponding to the user's finger was detected this time; An operation determination apparatus comprising: Photographing means for photographing a user's hand moved in a three-dimensional space;
Detecting means for detecting a portion corresponding to the user's finger from an image obtained by photographing;
A determination means for determining that a button operation has been performed in a pseudo manner when one portion corresponding to the user's finger was detected last time when two portions corresponding to the user's finger were detected this time; An operation determination apparatus comprising: Photographing means for photographing a user's hand moved in a three-dimensional space;
Detecting means for detecting a portion corresponding to the fingertip of the user from an image obtained by photographing;
An operation determination device comprising: a determination unit that determines that a button is simulated when two portions corresponding to the fingertip of the user are detected and a distance between the two points is smaller than a predetermined value . Photographing means for photographing a user's hand moved in a three-dimensional space;
Contour detecting means for detecting the contour of the hand image from an image obtained by photographing;
Fingertip detection means for detecting a pixel of a part corresponding to the fingertip of the hand (hereinafter referred to as “fingertip pixel”) from the contour;
Correction means for correcting the coordinates of the fingertip pixels detected by the fingertip detection means with decimal point accuracy using pixel values and coordinates of pixels around the fingertip pixels (hereinafter referred to as “peripheral pixels”); A fingertip detection device comprising:   The correction value calculation means calculates the weighted average of the coordinates of the fingertip pixel and the surrounding pixels using the pixel values of the fingertip pixel and the surrounding pixels as weights, and corrects the coordinates of the fingertip pixels. Fingertip detection device.   The fingertip detection device according to claim 16 or 17, wherein the correction value calculation means performs the correction so that a corrected fingertip position exceeds a range of the surrounding pixels.   The correction value calculating means calculates the sum of the weighted coordinates of the fingertip pixel and the surrounding pixels by using the pixel values of the fingertip pixel and the surrounding pixels as weights to the coordinates of the fingertip pixel and the surrounding pixels. The fingertip detection device according to claim 16, wherein the coordinate of the fingertip pixel is corrected by dividing the sum by a maximum value of pixel values of the fingertip pixel and the surrounding pixels.   The correction value calculating means includes a distance between the fingertip pixel and the surrounding pixel located in an oblique direction of the fingertip pixel, and the surrounding pixel located in a horizontal direction or a vertical direction of the fingertip pixel and the fingertip pixel. The fingertip detection device according to any one of claims 16 to 19, wherein the coordinates of the surrounding pixels are defined so that the distance between them is equal. Photographing a user's hand moved in three-dimensional space;
Detecting a portion corresponding to the fingertip of the user from an image obtained by photographing;
Setting a predetermined range including a portion corresponding to the detected fingertip;
Determining whether the portion corresponding to the fingertip has disappeared after setting the predetermined range;
The step of determining whether or not the portion corresponding to the fingertip has disappeared from the predetermined range, and then determining whether or not the portion corresponding to the fingertip has appeared in the predetermined range. When,
The step of determining whether or not the operation includes a step of determining that a click operation has been performed when it is determined that a portion corresponding to the fingertip has appeared in the predetermined range. Photographing a user's hand moved in three-dimensional space;
Detecting a portion corresponding to the user's finger from an image obtained by photographing;
In the case where one portion corresponding to the user's finger is detected this time, when two portions corresponding to the user's finger were detected last time, it is determined that a button operation has been performed in a pseudo manner; Operation determination method including Photographing a user's hand moved in three-dimensional space;
Detecting a portion corresponding to the user's finger from an image obtained by photographing;
In the case where two portions corresponding to the user's finger are detected this time, when one portion corresponding to the user's finger has been detected last time, it is determined that a pseudo button operation has been performed; Operation determination method including Photographing a user's hand moved in three-dimensional space;
Detecting a portion corresponding to the fingertip of the user from an image obtained by photographing;
A step of determining that a button operation has been performed in a pseudo manner when two portions corresponding to the fingertip of the user are detected and the distance between the two points is smaller than a predetermined value. . Photographing a user's hand moved in three-dimensional space;
Detecting an outline of the hand image from an image obtained by photographing;
Detecting a pixel corresponding to the fingertip of the hand (hereinafter referred to as “fingertip pixel”) from the contour;
The coordinates of the fingertip pixel detected by the step of detecting the fingertip pixel are corrected with decimal point accuracy using pixel values and coordinates of pixels around the fingertip pixel (hereinafter referred to as “ambient pixels”). And a fingertip detection method.   A computer program for causing a computer to execute the operation determination method according to claim 21.   A computer program for causing a computer to execute the operation determination method according to claim 22.   A computer program for causing a computer to execute the operation determination method according to claim 23.   A computer program for causing a computer to execute the operation determination method according to claim 24.   A computer program for causing a computer to execute the fingertip detection method according to claim 25.

Images