JP5532315B2 - Image processing apparatus, image processing method, program, and electronic apparatus - Google Patents

Description

  The present invention relates to an image processing device, an image processing method, a program, and an electronic device. For example, the user's hand is detected from an image obtained by imaging, and the operation of the electronic device is controlled according to the detected posture or gesture. The present invention relates to an image processing apparatus, an image processing method, a program, and an electronic apparatus that are suitable for use in the case of the above.

  There is a skin detection technique for detecting an area where skin is exposed (hereinafter referred to as a skin area) such as a face or a hand from an image obtained by imaging a person (see, for example, Patent Document 1). .

  In this skin detection technique, a first image obtained by imaging a subject (person) irradiated by an LED (light emitting diode) that outputs light having a wavelength λ1 in the near infrared region and a near infrared region different from the wavelength λ1. And a second image obtained by imaging the subject irradiated by the LED that outputs light having the wavelength λ2. Then, the skin area is detected based on the difference in luminance between the first image and the second image.

  Note that the wavelengths λ1 and λ2 in the near-infrared region are determined using the reflection characteristics of human skin.

  FIG. 1 shows reflection characteristics with respect to human skin. This reflection characteristic is general regardless of the difference in human skin color (race), sunburn, and the like. In the figure, the horizontal axis indicates the wavelength of the irradiation light irradiated on the human skin, and the vertical axis indicates the reflectance of the irradiation light irradiated on the human skin.

  It is known that the reflectance of irradiated light radiated to human skin decreases rapidly from around 900 nm with a peak at around 800 nm, and rises again around 1000 nm as a local minimum. Specifically, for example, the reflectance of reflected light obtained by irradiating human skin with 870 nm light is 63%, and the reflectance of reflected light obtained by irradiating 950 nm light is 50%. %.

  This is peculiar to human skin, and it is known that the change in reflectance is moderate in the vicinity of 800 to 1000 nm in objects other than human skin (for example, hair and clothes).

  The wavelengths λ1 and λ2 are determined using such reflection characteristics.

JP 2006-47067 A

  If the above-described skin detection technology is employed in an electronic device (for example, a television receiver), it is determined that the user is present while detecting human skin, and a predetermined operation is performed. When it is no longer possible to detect this, it is possible to perform operation control according to the presence or absence of the user, such as determining that there is no user and suspending a predetermined operation.

  Furthermore, this skin detection technology is applied so that the user can use his / her hand to create a predetermined posture (posture) or move (gesture). It is conceivable that the electronic device detects and executes a predetermined operation according to the posture or gesture.

  However, when such an electronic device is to be realized, it may occur that a user's movement that is not intended for operation is detected as an operation by a predetermined posture or an operation by a gesture and malfunctions. Need to increase accuracy.

  Also, when trying to realize such an electronic device, it is necessary to always irradiate the range where the user can exist with the irradiation light of the wavelengths λ1 and λ2, so that a large amount of power is consumed. I need it.

  The present invention has been made in view of such a situation, and enables a user's intention to perform an operation using a posture or a gesture to be detected with high accuracy while realizing power saving. .

An image processing apparatus according to a first aspect of the present invention is an image processing apparatus that detects a skin region representing human skin from an image, a first irradiation unit that irradiates a subject with light of a first wavelength, A second irradiating means for irradiating the subject with second light having a wavelength longer than the first wavelength; and reflected light from the subject when the subject is irradiated with light having the first wavelength. Generating means for generating a first image based on the reflected light from the subject when the subject is irradiated with light of the second wavelength; and The skin area is detected by comparing a luminance difference value between corresponding pixels of the image and the second image with a threshold value, and the one having the highest average brightness among the detected one or more skin areas It is detected as a hand region, at least the skin area or before Image processing means for determining an operation mode based on one time-series detection result of one of the hand regions, and only in the active mode, the subject based on at least one of the time-series detection results of the skin region or the hand region Identification means for identifying at least one of a posture operation or a gesture operation by a user, and the image processing means associates the operation mode with the first and second irradiation means with a predetermined frame rate R. Operating in standard mode, operating the first and second irradiating means in association with the predetermined frame rate R, active mode operating the identifying means, and the first and second irradiating means A first sleep mode that operates in association with a frame rate lower than a predetermined frame rate R; The second does not operate the illumination means, the first second sleep mode to operate in association with a lower frame rate than the irradiation means and said predetermined frame rate R, or stop mode to terminate the operation
However, if the detected state of the hand region without movement continues for a predetermined time, the operation mode is determined to be the active mode.

  The image processing means may determine that there is no movement when the detected time-series change of the center-of-gravity coordinates of the hand region is equal to or less than a threshold value.

  In the second sleep mode, the image processing unit is further configured to detect the movement of the user based on a luminance difference value between corresponding pixels of the two first images captured before and after in time. The presence or absence can be detected.

  The first wavelength has a value in the range of 630 nm to 1000 nm, the second wavelength has a value in the range of 900 nm to 1100 nm, and the second wavelength is larger than the first wavelength. be able to.

An image processing method according to a first aspect of the present invention includes a first irradiating unit that irradiates a subject with light having a first wavelength, and irradiating the subject with second light having a longer wavelength than the first wavelength. Generating a first image on the basis of reflected light from the subject when the subject is irradiated with light of the first wavelength, and the light of the second wavelength is Generation means for generating a second image based on reflected light from the subject when the subject is irradiated, and skin representing human skin based on the first image and the second image In an image processing method of an image processing apparatus including an image processing unit for detecting a region, a luminance difference value between corresponding pixels of the first image and the second image by the image processing unit is compared with a threshold value. the detected skin region, one or more of the skin area by Detecting the one with the highest average luminance as a hand region, and setting the operation mode based on at least one time-series detection result of the skin region or the hand region , A standard mode that operates in association with a frame rate R, the first and second irradiation means operate in association with the predetermined frame rate R, and at least one time series of the skin region or the hand region Based on the detection result, the active mode for identifying at least one of the posture operation or the gesture operation by the user who is the subject, and the first and second irradiation means are associated with a frame rate lower than the predetermined frame rate R The first sleep mode is activated, the second irradiation means is not operated, and the first irradiation means is moved forward. Second sleep mode to operate in association with a lower frame rate than a predetermined frame rate R, or will be determined in any of the stop mode to terminate the operation, the absence of detected motion of the hand region which is predetermined If you duration comprises the step of determining the operation mode to the active mode.

A program according to a first aspect of the present invention includes a first irradiating unit that irradiates a subject with light having a first wavelength, and a second irradiator that irradiates the subject with second light having a longer wavelength than the first wavelength. A first image is generated based on reflected light from the subject when the subject is irradiated with light of the first wavelength, and the light of the second wavelength is the subject Generating means for generating a second image based on the reflected light from the subject when irradiated on the skin, and a skin region representing human skin based on the first image and the second image. A program for controlling an image processing apparatus comprising an image processing means for detecting the skin region by comparing a luminance difference value between corresponding pixels of the first image and the second image with a threshold value The average brightness is the highest among the one or more skin areas Detecting an object as a hand region , and associating the first and second irradiation means with a predetermined frame rate R based on at least a time-series detection result of one of the skin region and the hand region And operating the first and second irradiation means in association with the predetermined frame rate R, and based on at least one time-series detection result of the skin region or the hand region, An active mode for identifying at least one of a posture operation or a gesture operation by a user who is the subject, and a first sleep in which the first and second irradiation means are operated in association with a frame rate lower than the predetermined frame rate R. Mode, the second irradiation means is not operated, and the first irradiation means is moved to the predetermined frame rate. Second sleep mode to operate in association with a lower frame rate than R, or will be determined in any of the stop mode to terminate the operation, if the absence of a detected motion of the hand region which continues for a predetermined time to execute the process including the step of determining the operation mode to the active mode to the computer of the image processing apparatus.

In the first aspect of the present invention, a skin area is detected by comparing a luminance difference value between corresponding pixels of the first image and the second image with a threshold, and an average of one or more skin areas having the highest luminance is detected as a hand region, based on the detection result of one of the time series of at least the skin region or hand region, the operation mode is, corresponding to the first and second illumination means to a predetermined frame rate R The standard mode to be operated is operated, the first and second irradiation means are operated in association with a predetermined frame rate R, and the subject is based on at least one time-series detection result of the skin region or the hand region. Active mode for identifying at least one of a posture operation or a gesture operation by a user, and the first and second irradiation means are lower than a predetermined frame rate R A first sleep mode that operates in association with a frame rate; a second sleep mode in which the first irradiation unit is operated in association with a frame rate lower than a predetermined frame rate R without operating the second irradiation unit; Alternatively, the mode is determined to be one of stop modes for ending the operation, but the operation mode is determined to be the active mode when the detected state of no movement of the hand region continues for a predetermined time.

The electronic device according to the second aspect of the present invention irradiates the subject with first irradiating means for irradiating the subject with light having the first wavelength and second light having a wavelength longer than the first wavelength. A first image is generated based on reflected light from the subject when the subject is irradiated with light of the first wavelength and the second wavelength of light, and the light of the second wavelength is Generation means for generating a second image based on reflected light from the subject when the subject is irradiated, and a luminance difference value between corresponding pixels of the first image and the second image as a threshold value The skin area is detected by comparing with the above , and among the detected one or more skin areas , the highest average luminance is detected as a hand area, and at least one time series of the skin area or the hand area is detected. The screen that determines the operation mode based on the detection result Only in the active mode and the processing means, at least one of a posture operation or a gesture operation by the user as the subject is identified based on at least a time-series detection result of one of the skin region and the hand region, and the identification result is Identification means for generating a corresponding operation signal, and operation signal response means for executing a predetermined process in accordance with the generated operation signal, wherein the image processing means changes the operation mode to the first and second modes. A standard mode in which the irradiation unit is operated in association with a predetermined frame rate R; an active mode in which the first and second irradiation units are operated in association with the predetermined frame rate R and the identification unit is operated; A frame rate lower than the predetermined frame rate R for the first and second irradiation means. A first sleep mode in which the second irradiation unit is not operated, and a second sleep mode in which the first irradiation unit is operated in association with a frame rate lower than the predetermined frame rate R; Alternatively, one of the stop modes for ending the operation is determined. However, when the detected state of no movement of the hand region continues for a predetermined time, the operation mode is determined to be the active mode.

In the first aspect of the present invention, a skin area is detected by comparing a luminance difference value between corresponding pixels of the first image and the second image with a threshold, and an average of one or more skin areas having the highest luminance is detected as a hand region, based on the detection result of one of the time series of at least the skin region or hand region, the operation mode is, corresponding to the first and second illumination means to a predetermined frame rate R The standard mode to be operated is operated, the first and second irradiation means are operated in association with a predetermined frame rate R, and the subject is based on at least one time-series detection result of the skin region or the hand region. Active mode for identifying at least one of a posture operation or a gesture operation by a user, and the first and second irradiation means are lower than a predetermined frame rate R A first sleep mode that operates in association with a frame rate; a second sleep mode in which the first irradiation unit is operated in association with a frame rate lower than a predetermined frame rate R without operating the second irradiation unit; Alternatively, the mode is determined to be one of stop modes for ending the operation, but the operation mode is determined to be the active mode when the detected state of no movement of the hand region continues for a predetermined time . Only in the active mode, at least one of the posture operation or the gesture operation by the user as the subject is identified based on the detection result of at least one of the skin region or the hand region, and an operation signal corresponding to the identification result is generated. The predetermined processing is executed according to the operation signal.

  According to the first aspect of the present invention, it is possible to detect a gesture or gesture operation with high accuracy while realizing power saving.

  According to the second aspect of the present invention, it is possible to detect an operation by a posture or a gesture with high accuracy and control an electronic device while realizing power saving.

It is a figure which shows the reflective characteristic of a human skin. It is a block diagram which shows the structural example of the operation detection apparatus to which this invention is applied. It is a figure which shows the state transition of an operation detection apparatus. It is a flowchart explaining a state transition process. It is a flowchart explaining the process of standby mode. It is a figure which shows the frame rate in the process of standby mode. It is a figure which shows the frame rate in the process of 1st sleep mode. It is a figure which shows the frame rate in the process of 2nd sleep mode.

  Hereinafter, the best mode for carrying out the invention (hereinafter referred to as an embodiment) will be described in detail with reference to the drawings.

<1. Embodiment>
[Configuration example of operation detection device]
FIG. 2 shows an example of the configuration of an operation detection apparatus according to an embodiment of the present invention. The operation detection device 10 detects a human hand that is the detection target 20 from a captured image, and controls a predetermined operation according to at least one of a posture or a gesture by the hand.

  That is, if the operation detection device 10 is incorporated in an electronic device to be controlled, the user can operate the electronic device by performing a predetermined posture or gesture with his / her hand.

  For example, if the operation detection device 10 is incorporated in a television receiver, the user can switch channels, adjust the volume, or turn off the power by performing a predetermined posture or gesture by his / her hand. . Of course, it is possible to incorporate the operation detection device 10 into an electronic device other than a television receiver, such as a personal computer or a cellular phone. However, in the following embodiment, the operation detection device 10 is typically incorporated in a television receiver. An example of the case will be described.

  The operation detection apparatus 10 includes a control unit 11, an LED control unit 12, an irradiation light source (LED) 13-1, an irradiation light source (LED) 13-2, an imaging unit 14, an imaging control unit 15, an image processing unit 16, a memory 17, And an operation control unit 18.

  The control unit 11 controls the operation of each unit of the operation detection device 10 in an integrated manner. Further, the control unit 11 switches the operation mode of the operation detection device 10 based on the mode switching signal notified from the image processing unit 16. Note that the operation detection device 10 is set to operate in one of the operation modes of the standby mode, the active mode, the first sleep mode, the second sleep mode, or the stop mode. Details of the operation mode will be described later with reference to FIG.

  The LED control unit 12 controls the light emission timing and the irradiation level of the irradiation light sources 13-1 and 13-2 according to the control from the control unit 11.

  The irradiation light source 13-1 is composed of an LED, and emits irradiation light whose emission spectrum peak has a wavelength λ1 in the near-infrared region (hereinafter referred to as light of wavelength λ1) under the control of the LED control unit 12. The irradiation light source 13-2 is composed of an LED, and emits irradiation light whose emission spectrum peak has a wavelength λ2 in the near-infrared region (hereinafter referred to as light having a wavelength λ2) under the control of the LED control unit 12. The emitted irradiation light is irradiated to a subject (in this case, a user of an electronic device in which the operation detection device 10 is incorporated). The number of LEDs constituting each light source is appropriately determined so as to uniformly irradiate the subject in a use state of a device in which the operation detection device is incorporated.

  If the wavelength λ1 is a value in the range of 630 nm to 1000 nm, the wavelength λ2 is a value in the range of 900 nm to 1100 nm, and the wavelength λ2 is longer than the wavelength λ1 (λ1 <λ2), skin detection is performed. Is possible. However, when the user feels uncomfortable by irradiating visible light, the wavelength λ1 may be set in the near infrared region of 800 nm to 1000 nm.

The imaging unit 14 incorporates a condenser lens and an imaging device such as a CMOS, and generates an image by receiving reflected light from the subject in accordance with control from the imaging control unit 15. Hereinafter, an image captured and generated when the irradiation light source 13-1 emits λ1 irradiation light is referred to as a first image, and an image is captured when the irradiation light source 13-2 emits λ2 irradiation light. The generated image is a second image.
The imaging unit 14 may be provided with a band pass filter that transmits only the near infrared region.

  The imaging control unit 15 controls the imaging timing of the imaging unit 14, the luminance amplification gain, and the like according to the control from the control unit 11. In addition, the imaging control unit 15 outputs the first and second images generated by the imaging unit 14 to the image processing unit 16.

  The image processing unit 16 generates a difference image having a difference value d (= Y1-Y2) between the luminances Y1 and Y2 of corresponding pixels of the first image and the second image as a pixel value, and the difference value d Is binarized by comparing with a predetermined threshold value to generate a binarized image. Based on the binarized image, the skin area of the subject is specified. In an example of a television receiver, a user's face is often detected as a skin area in addition to a hand toward the imaging unit for a user to perform a gesture or a posture operation as a skin area. In view of this, the image processing unit 16 further selects an area having the highest average luminance for the operation because it is presented near the irradiation light source of the operation detection device incorporated in the television receiver among the areas specified as the skin area. The operation skin area to be used (in the present embodiment, the hand area) is specified, and the barycentric coordinates thereof are calculated. The generated binarized image is recorded in the memory 17 together with coordinate information indicating the position of the hand region and its center of gravity. Further, the image processing unit 16 generates a mode switching signal based on the binary image recorded in the memory 17 and notifies the control unit 11 of the mode switching signal.

  In the above example, the difference in luminance value is used. However, if a difference in reflectance between the skin region and the other region can be detected, a luminance value ratio or the like may be used.

  In the example incorporated in the television receiver of this embodiment, the skin area used for the operation is the hand area. However, when operating a small device such as a mobile phone in the vicinity, the finger area is used. In some cases, only the tip of the finger may be used based on the luminance value determined by the distance to the irradiation light source. Further, in the case of a mobile phone, when the user's face is not detected by the imaging unit, a part of the above processing can be omitted.

  Even in the case of a television receiver, a gesture operation may be performed based on a face area having a low luminance value.

  The operation control unit 18 detects at least one of a posture or a gesture by the user based on the binarized image recorded in the memory 17 and generates an operation signal based on the detection result. This operation signal is notified to the electronic device to be controlled in which the operation detection device 10 is incorporated, and the electronic device executes a predetermined operation.

[Types of operation modes]
Next, the operation mode of the operation detection device 10 will be described. FIG. 3 shows a mode transition of the operation detection device 10.

  As described above, there are five types of operation modes, standby mode M1, active mode M2, first sleep mode M3, second sleep mode M4, and stop mode M5, and operation detection device 10 has five types of operation modes. It operates in the state set to one of these.

  When the power is turned on, the operation detection device 10 is set to the standby mode M1. In the standby mode M1, imaging is performed at a predetermined frame rate R (for example, a value of 30 fps to 60 fps, hereinafter assumed to be 60 fps). Thereby, the first and second images can be obtained at the frame rate R / 2 (in this case, 30 fps), and the difference image and the binarized image are also generated at the frame rate R / 2. . Further, in the standby mode M1, a hand area is detected based on the binarized image.

In the standby mode M1, when a hand region that is an operation skin region used for gesture or posture operation is detected and the movement is stationary for a predetermined time (for example, 0.5 seconds), the user uses the hand. In order to perform a gesture or the like from now on, it is determined that the hand is stationary toward the operation detection device 10, and the operation mode is changed to the active mode M2. Thus, only when it is determined that the hand is stationary toward the operation detection device 10, for example, when a person who is not a user crosses in front of the operation detection device 10 by changing to the active mode M <b> 2. Even if the hand is detected by chance, the active mode M2 is not changed.
If the user expresses an intention to perform a gesture operation using a hand, there may be a person who shakes his / her hand toward the television receiver without stopping his / her hand. In this case, the hand region or the center of gravity of the hand region remains within a predetermined range of the image (for example, within 2/3 of the horizontal width of the captured image or within 1/2 of the screen area of the captured image) for a predetermined time or more. In this case, the operation mode may be changed to the active mode M2. In addition, if the hand is stationary, the intention of the operation is judged in a relatively short time, and if the hand is waving, the judgment is made in a relatively long time (several seconds). Can be improved.

  In the case of a mobile phone, if you hold your finger in front of the imaging unit, it is considered that the gesture operation is basically intended, so the operation used for operation somewhere within the imaging range for a predetermined time. If there is a skin area, the mode may transition to the active mode M2.

  In the active mode M2, based on the detected shape and movement of the hand region, a gesture performed by the user using the hand is detected, and the operation of the electronic device is controlled according to the detection result. In the active mode M2, when a time during which no hand area is detected passes a predetermined time (for example, 10 seconds), the operation mode is changed to the standby mode M1.

  In the standby mode M1, when a predetermined time (for example, 1 minute) elapses without changing to the active mode M2, the operation mode is changed to the first sleep mode M3.

In the first sleep mode M3, the frame rate is lower than that in the standby mode M1. For example, when halved, the first and second images are captured at a frame rate R / 4 (in this case, 15 fps), and the difference image and the binarized image are also generated at the frame rate R / 4. Will be. When a skin area having a predetermined area or more is detected based on the binarized image, the operation mode is changed to the standby mode M1. On the other hand, when a predetermined time (for example, 3 minutes) has elapsed without detecting a skin area larger than the predetermined area, the operation mode is changed to the second sleep mode M4.
In this embodiment, the frame rate is reduced in the first sleep mode M3. However, the frame rate may be similarly reduced in the standby mode M1. In the standby mode M1, it is only necessary to detect the intention of the user to start the gesture or posture operation from the movement of the skin area such as the hand as described above. Therefore, the high speed as the active mode M2 is not necessarily required. There is no problem.

  In the second sleep mode M4, only the wavelength λ1 having a shorter wavelength and a higher reflection intensity on the human skin is irradiated. This is because one of the easy-to-obtain and inexpensive general image sensors has high sensitivity in the visible light region of 400 nm to 800 nm, and the sensitivity suddenly drops in the near-infrared region having a longer wavelength. Because it is. In an example of a typical image sensor, the sensitivity at 870 nm is about twice the sensitivity at 950 nm.

  The light emission efficiency of the LED used for the light source is generally higher at 870 nm than at 950 nm, and the light emission efficiency is about double in the typical LED example. Therefore, overall, the power consumption of the 950 nm light source LED is about 4 times larger than that of the 870 nm light source LED, so if the λ2 on the long wavelength side is turned off, the power consumption is reduced to about 1/5. Reduced.

  At this time, only the first image is captured at a frame rate R / 4 (in this case, 15 fps). And the difference value of the brightness | luminance of the corresponding pixels of the 1st image imaged continuously in time series is calculated, and the number of the pixels where the difference value is not 0 (namely, pixel of a region with a motion) May be transitioned to the first sleep mode M3. On the other hand, the predetermined number of pixels in which the difference value of the luminance between corresponding pixels of the first image continuously captured is 0 (that is, the pixel in the non-motion region) is detected in a predetermined number or more. When time (for example, 10 minutes) has elapsed, it may be determined that there is a possibility that there is no user in front of the operation detection device 10, and the operation mode may be changed to the stop mode M5.

  However, even if there is actually no movement of the user, it is possible to watch the TV still, so it changes to standby mode every few minutes that there is no human face, for example a skin area It is even better if the user confirms that there is no user and then shifts to the stop mode M5.

  In the above example, in the second sleep mode M4, only the light source having the wavelength λ2 is turned off. However, when the brightness of the captured image captured by the imaging device when both the λ1 and λ2 are turned off is equal to or higher than a predetermined value. May be determined to be used in a bright environment and both λ1 and λ2 may be turned off. This is because, when the light is used in a bright place, it is possible to detect a moving pixel in the image from the difference value of the corresponding brightness of the images continuously captured in time series without turning on λ1. . In this way, power consumption can be further reduced.

  As described above, the second sleep mode is a mode for detecting a user's movement. When the user starts a gesture or posture operation, in the case of this embodiment, there is usually a movement for the user to put his hand toward the television receiver. Therefore, the second sleep mode may be called a “skin detection trigger mode” for starting skin detection. Therefore, when it is determined that there is a movement, the apparatus may directly enter the standby mode.

  The same applies when the present invention is mounted on a handheld mobile device such as a mobile phone. In other words, since the operation of inserting a finger in front of the image sensor of a mobile phone is not a normal user-friendliness, if any movement of the user is detected by the image sensor, it can be used as a trigger to set the skin detection standby mode to save power. Responsiveness can be improved while realizing.

  In the stop mode M5, the operation control unit 18 outputs an operation signal for instructing the electronic device to be controlled to turn off the power. Further, the power supply of the operation detection device 10 itself is also turned off. Therefore, there is no transition from the stop mode M5 to another operation mode, and when the power is turned on again, the standby mode M1 is set.

  In the stop mode, it is also conceivable to turn off only the operation detection device 10 itself without turning off the power supply of the electronic device on which the operation detection device is mounted. Which one to select depends on the product form of the electronic device. For example, when a user's finger is detected by an imaging means placed on the back side of a liquid crystal screen on a mobile phone, there is no user in front of the image sensor even during normal use, so there is no movement, It is not always appropriate to turn off the device. Depending on the electronic device, the stop mode itself may not be provided.

[Description of operation]
Next, the state transition process between the above-described operation modes will be described with reference to FIG. FIG. 4 is a flowchart for explaining the state transition process.

  This state transition process is started when the power is turned on. In step S1, the control unit 11 sets the operation mode to the standby mode M1. In response to this, each unit of the operation detection device 10 executes processing in the standby mode M1 in which imaging is performed at a frame rate R as shown in FIG.

  FIG. 5 is a flowchart detailing the processing in the standby mode M1.

  In step S <b> 11, the irradiation light source 13-1 emits light having the wavelength λ <b> 1 according to the control of the LED control unit 12. In step S <b> 12, the imaging unit 14 receives reflected light from the subject according to control from the imaging control unit 15, generates a first image, and outputs the first image to the image processing unit 16.

  In step S <b> 13, the irradiation light source 13-2 emits light having a wavelength λ <b> 2 according to the control of the LED control unit 12. In step S <b> 14, the imaging unit 14 receives reflected light from the subject according to control from the imaging control unit 15, generates a second image, and outputs the second image to the image processing unit 16.

  In step S15, the image processing unit 16 generates a difference image in which the difference value d (= Y1-Y2) between the luminances Y1 and Y2 between corresponding pixels of the first image and the second image is a pixel value. . Further, in step S16, the image processing unit 16 binarizes the difference value d, which is each pixel value of the difference image, with a predetermined threshold value, and generates a binarized image indicating the skin region and other regions. Generate.

  In step S <b> 17, the image processing unit 16 performs labeling on each skin region on the binarized image. Further, in step S18, the average brightness of each labeled skin area is calculated from the original first or second image, the skin area having the highest average brightness is determined as the hand area, and the barycentric coordinates of the hand area are determined. calculate. Then, the binarized image generated at the frame rate R / 2 and the coordinate information indicating the hand region and the center of gravity are recorded in the memory 17.

  Returning to FIG. In step S2, the image processing unit 16 keeps the user's hand stationary for a predetermined time (for example, 0.5 seconds) based on the binarized image sequentially recorded in the memory 17 and the coordinate information indicating the hand region and the center of gravity. It is determined whether or not. Here, being stationary includes the case where the amount of change in the center of gravity of the hand region is within a predetermined threshold. In addition to using the barycentric coordinates, for example, combining the method such as the rate of change of the area of the skin area within 10% or the use of the rate of change of the ellipticity when the skin area is approximated by a circle. Thus, the determination accuracy of the stationary state may be increased.

  If it is determined in step S2 that the user's hand is stationary for a predetermined time or more, the process proceeds to step S3.

  In step S <b> 3, the image processing unit 16 outputs a mode switching signal instructing the transition to the active mode M <b> 2 to the control unit 11. In response to the mode switching signal, the control unit 11 sets the operation mode to the active mode M2. In response to this, each unit of the operation detection device 10 executes processing in the active mode M2 in which imaging is performed at the frame rate R as in the standby mode M1.

  In the process of the active mode M2, in addition to the process of the standby mode M1, the operation control unit 18 uses the binarized images sequentially recorded in the memory 17 and the coordinate information indicating the hand area and the center of gravity. At least one of a posture and a gesture performed using the is detected, and the operation of the electronic device is controlled according to the detection result.

  In step S <b> 4, the image processing unit 16 determines whether or not a predetermined time (for example, 10 seconds) has passed without detecting a hand region based on the binarized images sequentially recorded in the memory 17. Here, if the hand region is detected, the process returns to step S3, and the process of the active mode M2 is continued. On the other hand, when the predetermined time has passed without detecting the hand region, the process returns to step S1, and the operation mode is changed to the standby mode M1.

  In step S2 in the standby mode M1, if a predetermined time (for example, 1 minute) has passed without transitioning to the active mode M2, the process proceeds to step S5.

  In step S <b> 5, the image processing unit 16 outputs a mode switching signal that instructs the transition to the first sleep mode M <b> 3 to the control unit 11. In response to the mode switching signal, the control unit 11 sets the operation mode to the first sleep mode M3. In response to this, each unit of the operation detection device 10 executes processing in the first sleep mode M3 in which imaging is performed at a frame rate R / 2 that is half that of the standby mode M1, as shown in FIG.

  In step S <b> 6, the image processing unit 16 determines whether or not a skin area having a predetermined area or more is detected based on the binarized images sequentially recorded in the memory 17. If it is determined that a skin area larger than the predetermined area has been detected, the process returns to step S1, and the operation mode is shifted to the standby mode M1. On the other hand, when a predetermined time (for example, 3 minutes) has elapsed without detecting a skin area of a predetermined area or more, the process proceeds to step S7.

  In step S <b> 7, the image processing unit 16 outputs a mode switching signal instructing the transition to the second sleep mode M <b> 4 to the control unit 11. In response to the mode switching signal, the control unit 11 sets the operation mode to the second sleep mode M4. In response to this, each unit of the operation detection device 10 performs processing in the second sleep mode M4 that captures only the first image at a frame rate R / 4 that is 1/4 of the standby mode M1, as shown in FIG. Run.

  In step S <b> 6, the image processing unit 16 calculates a luminance difference value between corresponding pixels of the first image captured continuously, and a pixel whose difference value is not 0 (i.e., in an area with motion). It is determined whether or not the number of pixels is equal to or greater than a predetermined number. Then, when it is determined that the number of pixels in the region with movement is equal to or greater than the predetermined number, the process is returned to step S5, and the operation mode is changed to the first sleep mode M3. On the other hand, if it is determined that the number of pixels in the region with movement is less than the predetermined number, the process proceeds to step S9.

  In step S <b> 9, the image processing unit 16 has a predetermined number of pixels whose luminance difference value between corresponding pixels of the first image continuously captured is 0 (that is, pixels in a region having no motion). It is determined whether or not a predetermined time (for example, 10 minutes) has elapsed in the state detected above. Here, the process returns to step S7 until a predetermined time elapses. Then, when it is determined that the predetermined time has elapsed while the area without motion is detected, the process proceeds to step S10.

  In step S <b> 10, the image processing unit 16 outputs a mode switching signal instructing the transition to the stop mode M <b> 5 to the control unit 11. In response to the mode switching signal, the control unit 11 sets the operation mode to the stop mode M5. In response to this, the operation control unit 18 of the operation detection device 10 outputs an operation signal for instructing the electronic device to be controlled to turn off the power. Further, the power supply of the operation detection device 10 itself is also turned off. The state transition process is thus completed.

  As described above, according to the state transition process of the operation detection device 10, since the posture and the gesture are not detected in the standby mode M1, but detected in the active mode M2, it is possible to suppress malfunction due to erroneous detection.

  Further, in the standby mode M1, an area having the highest average luminance among the skin areas is detected as an operation skin area used for the operation, and is active only when it is within a predetermined range of the image for a predetermined time. Since the mode is changed to the mode M2, a motion that is not intended to be operated by the user or a motion of a person other than the user who crosses the front of the apparatus is not erroneously detected as a motion for gesture or posture. That is, the user may perform a posture or a gesture after performing a natural motion such as holding a hand or shaking a hand.

  In the first sleep mode M3, since the frame rate is lower than that in the standard mode M1, the power required for the irradiation light sources 13-1 and 13-2 to emit the irradiation light can be suppressed. Therefore, power saving is realized.

  Furthermore, in the second sleep mode M4, in order to achieve the same luminance level in the image sensor, the wavelength λ1 is emitted that has lower power consumption and higher reflection intensity than the irradiation light source 13-2 that emits the wavelength λ2. Since only the irradiation light source 13-1 operates, further power saving can be realized as compared with the first sleep mode M3.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  The program may be processed by a single computer, or may be distributedly processed by a plurality of computers. Furthermore, the program may be transferred to a remote computer and executed.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

  DESCRIPTION OF SYMBOLS 10 Operation detection apparatus, 11 Control part, 12 LED control part, 13, LED, 14 Imaging part, 15 Imaging control part, 16 Image processing part, 17 Memory, 18 Operation control part

Claims (7)

In an image processing apparatus for detecting a skin region representing human skin from an image,
First irradiation means for irradiating the subject with light of a first wavelength;
A second irradiation means for irradiating the subject with second light having a wavelength longer than the first wavelength;
When the first image is generated based on the reflected light from the subject when the subject is irradiated with the light of the first wavelength, and the subject is irradiated with the light of the second wavelength Generating means for generating a second image based on the reflected light from the subject;
The skin area is detected by comparing a luminance difference value between corresponding pixels of the first image and the second image with a threshold, and the average luminance is the highest among the detected one or more skin areas. Image processing means for detecting a high one as a hand region and determining an operation mode based on at least a time-series detection result of one of the skin region or the hand region;
An identification means for identifying at least one of a posture operation or a gesture operation by a user as the subject based on at least a time-series detection result of one of the skin region and the hand region only in the active mode,
The image processing means changes the operation mode,
A standard mode in which the first and second irradiation means are operated in association with a predetermined frame rate R;
An active mode in which the first and second irradiation means are operated in association with the predetermined frame rate R, and the identification means is operated;
A first sleep mode in which the first and second irradiation means are operated in association with a frame rate lower than the predetermined frame rate R;
A second sleep mode in which the second irradiation unit is not operated and the first irradiation unit is operated in association with a frame rate lower than the predetermined frame rate R;
Or
Stop mode to end operation
To decide either
An image processing apparatus that determines the operation mode to be the active mode when the detected state in which the hand region does not move continues for a predetermined time. Wherein the image processing means, if the time-series change of the center coordinates of the detected the hand regions is below the threshold, the image processing apparatus according to claim 1 for determining the absence of the motion. In the second sleep mode, the image processing unit is further configured to detect the movement of the user based on a luminance difference value between corresponding pixels of the two first images captured before and after in time. The image processing apparatus according to claim 1 , wherein presence or absence is detected. The first wavelength is a value in the range of 630 nm to 1000 nm;
The second wavelength is in the range of 900 nm to 1100 nm;
The image processing apparatus according to claim 1 , wherein the second wavelength is larger than the first wavelength. First irradiation means for irradiating the subject with light of a first wavelength;
A second irradiation means for irradiating the subject with second light having a wavelength longer than the first wavelength;
When the first image is generated based on the reflected light from the subject when the subject is irradiated with the light of the first wavelength, and the subject is irradiated with the light of the second wavelength Generating means for generating a second image based on the reflected light from the subject;
In an image processing method of an image processing apparatus comprising: an image processing unit that detects a skin region representing a human skin based on the first image and the second image;
By the image processing means,
Detecting the skin region by comparing a luminance difference value between corresponding pixels of the first image and the second image with a threshold;
One or more of the skin area, to detect an average brightness highest as hand area,
Based on at least the time series detection result of one of the skin area or the hand area, the operation mode is
A standard mode in which the first and second irradiation means are operated in association with a predetermined frame rate R;
The first and second irradiation means are operated in association with the predetermined frame rate R, and at least based on a time-series detection result of one of the skin region or the hand region, by the user who is the subject An active mode that identifies at least one of a posture operation or a gesture operation,
A first sleep mode in which the first and second irradiation means are operated in association with a frame rate lower than the predetermined frame rate R;
A second sleep mode in which the second irradiation unit is not operated and the first irradiation unit is operated in association with a frame rate lower than the predetermined frame rate R;
Or
Stop mode to end operation
The image processing method will be determined either, if the absence of detected motion of the hand region which continues for a predetermined time comprises the step of determining the operation mode to the active mode. First irradiation means for irradiating the subject with light of a first wavelength;
A second irradiation means for irradiating the subject with second light having a wavelength longer than the first wavelength;
When the first image is generated based on the reflected light from the subject when the subject is irradiated with the light of the first wavelength, and the subject is irradiated with the light of the second wavelength Generating means for generating a second image based on the reflected light from the subject;
A program for controlling an image processing apparatus comprising: image processing means for detecting a skin region representing human skin based on the first image and the second image,
Detecting the skin region by comparing a luminance difference value between corresponding pixels of the first image and the second image with a threshold;
One or more of the skin area, to detect an average brightness highest as hand area,
Based on at least the time series detection result of one of the skin area or the hand area, the operation mode is
A standard mode in which the first and second irradiation means are operated in association with a predetermined frame rate R;
The first and second irradiation means are operated in association with the predetermined frame rate R, and at least based on a time-series detection result of one of the skin region or the hand region, by the user who is the subject An active mode that identifies at least one of a posture operation or a gesture operation,
A first sleep mode in which the first and second irradiation means are operated in association with a frame rate lower than the predetermined frame rate R;
A second sleep mode in which the second irradiation unit is not operated and the first irradiation unit is operated in association with a frame rate lower than the predetermined frame rate R;
Or
Stop mode to end operation
While determining in any one of the absence of detected motion of the hand region which causes a computer to execute an image processing apparatus processing including the step of determining the operation mode to the active mode when continued for a predetermined time program. First irradiation means for irradiating the subject with light of a first wavelength;
A second irradiation means for irradiating the subject with second light having a wavelength longer than the first wavelength;
When the first image is generated based on the reflected light from the subject when the subject is irradiated with the light of the first wavelength, and the subject is irradiated with the light of the second wavelength Generating means for generating a second image based on the reflected light from the subject;
The skin area is detected by comparing a luminance difference value between corresponding pixels of the first image and the second image with a threshold, and the average luminance is the highest among the detected one or more skin areas. Image processing means for detecting a high one as a hand region and determining an operation mode based on at least a time-series detection result of one of the skin region or the hand region;
Only in the active mode, at least one of a posture operation or a gesture operation by the user as the subject is identified based on at least one time-series detection result of the skin region or the hand region, and an operation signal corresponding to the identification result Identifying means for generating
An operation signal corresponding means for performing a predetermined process according to the generated operation signal,
The image processing means changes the operation mode,
A standard mode in which the first and second irradiation means are operated in association with a predetermined frame rate R;
An active mode in which the first and second irradiation means are operated in association with the predetermined frame rate R, and the identification means is operated;
A first sleep mode in which the first and second irradiation means are operated in association with a frame rate lower than the predetermined frame rate R;
A second sleep mode in which the second irradiation unit is not operated and the first irradiation unit is operated in association with a frame rate lower than the predetermined frame rate R;
Or
Stop mode to end operation
To decide either
The electronic device that determines the operation mode as the active mode when the detected state of the hand region without movement continues for a predetermined time.

Images