JP2019169004A - Sight line detection system and sight line detection program - Google Patents

Abstract

To suppress power consumption.SOLUTION: The sight line detection system includes: an irradiation device that irradiates infrared rays in a target range S20; an imaging device that captures an image of the infrared rays reflected from the target range S22; a detection unit that detects a line of sight of a person within the target range based on the images S26; and a control unit that determines whether or not a person is included in the target range based on the image S28, and when determining that a person is not included in the target range, sets a first period in which the irradiation device is caused to stop the irradiation of the infrared rays S15.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a line-of-sight detection system and a line-of-sight detection program, for example, a line-of-sight detection system and a line-of-sight detection program using infrared rays.

  There is known a method of detecting the direction of the line of sight of a person within a target range by irradiating the target range with infrared rays, imaging the target range with an infrared imaging device, and analyzing the captured image. (For example, patent document 1).

JP-A-2015-000156

  However, in order to detect the line-of-sight direction, it is required that the intensity of the infrared ray reflected from the target range is a certain level or more. For this reason, the power consumption of the irradiation apparatus which irradiates infrared rays will become large.

  The present invention has been made in view of the above problems, and an object thereof is to suppress power consumption.

  The present invention relates to an irradiation device that irradiates a target range with infrared rays, an imaging device that picks up the infrared image reflected from the target range, and a detection unit that detects a gaze of a person within the target range based on the images And determining whether or not a person is included in the target range based on the image, and determining that a person is not included in the target range, the first stop the irradiation device to stop the infrared irradiation And a control unit for one period.

  In the above configuration, when the control unit determines whether or not a person is included in the target range in the first period, and determines that a person is included in the target range, the irradiation It can be set as the 2nd period which makes an apparatus irradiate the said infrared rays, make the said imaging device pick up an image, and makes the said detection part detect the said eyes | visual_axis.

  In the above configuration, in the first period, the control unit causes the irradiation device to irradiate the infrared ray, causes the imaging device to pick up the image, and causes the irradiation device to emit the infrared ray after imaging. When the irradiation is stopped, it is determined whether or not a person is included in the target range based on an image captured by the imaging device immediately before, and when it is determined that a person is included in the target range, the first It can be set as the structure made into 2 periods.

  In the above configuration, in the second period, the control unit causes the irradiation device to irradiate the infrared light, causes the imaging device to capture an image, and irradiates the image after the imaging, for each second interval shorter than the first interval. The apparatus can be configured to stop the irradiation of the infrared rays and to cause the detection unit to detect the line of sight.

  The said structure WHEREIN: When it determines with the said control part not including the person more than the 3rd period longer than the said 1st interval in the said target range, it can be set as the structure made into the said 1st period.

  The said structure WHEREIN: The said control part can be set as the structure made into the said 1st period, when it determines with the person not being contained in the said target range for the 3rd period or more.

  In the above configuration, the control unit may be configured such that when the second period is set as the first period and the fourth period is not passed, the second period is set as the first period and when the fourth period or more is passed. The first interval can be shortened.

  In the above configuration, when the detection unit detects a line of sight of a person within the target range, the control unit determines that a person is included in the target range, and the detection unit is within the target range. When a person's line of sight is not detected, it can be determined that no person is included in the target range.

  According to the present invention, a computer detects a line of sight of a person within the target range based on the infrared image reflected from the target range captured by the imaging device when the irradiation device irradiates the target range with infrared rays; A step of determining whether or not a person is included in the target range based on an image, and when determining that a person is not included in the target range, the step of setting the irradiation apparatus to stop the infrared irradiation And a line-of-sight detection program for executing

  According to the present invention, power consumption can be suppressed.

FIG. 1 is a schematic diagram of a line-of-sight detection system according to the first embodiment. FIG. 2 is a block diagram of the eye gaze detection system according to the first embodiment. FIG. 3A is a block diagram illustrating another example of the visual line detection system according to the first embodiment, and FIG. 3B is a block diagram of a computer. FIG. 4 is a functional block diagram illustrating processing of the processor and the like in the first embodiment. FIG. 5 is a flowchart illustrating the processing of the processor in the first embodiment. FIG. 6 is a diagram illustrating each process with respect to time in the first embodiment. FIG. 7 is a flowchart illustrating an example of step S15 of FIG.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a schematic diagram of a line-of-sight detection system according to the first embodiment. As shown in FIG. 1, a virtual target range 52 is provided in front of the visual line detection system 50, and a virtual visual line detection range 58 is provided behind the visual line detection system 50. The line-of-sight directions 56a to 56c of the persons 54a to 54c are indicated by arrows. The line-of-sight detection system 50 detects the line-of-sight direction 56a of the person 54a in the target range 52 (for example, the coordinates of the point 60 in the line-of-sight detection range 58 in the line-of-sight direction 56a). The person 54b is included in the target range 52, but the line-of-sight direction 56b does not face the line-of-sight detection range 58. Therefore, the line-of-sight detection system 50 does not detect the line-of-sight direction 56b. Although the line-of-sight direction 56c of the person 54c faces the line-of-sight detection range 58, the person 54c is outside the target range 52. Therefore, the line-of-sight detection system 50 does not detect the line-of-sight direction 56c. The line-of-sight detection range 58 is a two-dimensional plane such as a display or a three-dimensional structure such as a monument.

  FIG. 2 is a block diagram of the eye gaze detection system according to the first embodiment. As shown in FIG. 2, the line-of-sight detection system 50 includes an irradiation device 10, an imaging device 12, a processor 14, and a memory 16. The irradiation device 10 is, for example, an LED (Light Emitting Diode) that emits infrared rays, and emits infrared rays toward the target range 52. The imaging device 12 is an infrared camera including an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and captures an infrared image in the target range 52. The processor 14 is a CPU (Central Processing Unit), for example, and functions as a detection unit and a control unit in cooperation with software such as a program. The processor 14 outputs information on the detected line-of-sight direction to the outside. The memory 16 is, for example, a volatile memory or a nonvolatile memory, and stores a program and / or data.

  FIG. 3A is a block diagram illustrating another example of the visual line detection system according to the first embodiment, and FIG. 3B is a block diagram of a computer. As shown in FIG. 3A, in the visual line detection system 50, the visual line detection module 51 includes the irradiation device 10 and the imaging device 12, and does not include a processor that functions as a detection unit and a control unit. The line-of-sight detection module 51 is connected to the computer 18. The computer 18 functions as a detection unit and a control unit in cooperation with software such as a program. The program is stored in a storage medium such as an optical disk, and is installed in the computer from the storage medium. The program may be implemented on a computer online. Other configurations are the same as those in FIG.

  As shown in FIG. 3B, the computer 18 includes a processor 20, a memory 22, an input / output device 24, an interface 26, and an internal bus 28. The processor 20 is a CPU, for example, and functions as a detection unit and a control unit in cooperation with software such as a program. The line-of-sight detection program causes the computer 18 to execute each step of the detection unit and the control unit. The memory 22 is, for example, a volatile memory or a nonvolatile memory, and stores a program and / or data. The input / output device 24 is, for example, a keyboard, a mouse, and a display. The interface 26 is an interface for connecting to the irradiation device 10 and the imaging device 12. The interface 26 may function as an interface for outputting detection information to the outside.

  FIG. 4 is a functional block diagram illustrating processing of the processor and the like in the first embodiment. As shown in FIG. 4, the processor 14 or 20 functions as the detection unit 32 and the control unit 34 in cooperation with the program.

  The line-of-sight detection method of the detection unit 32 is the same as that in Patent Document 1, for example. The detection unit 32 recognizes the eyes of the person 54a in the target range 52 from the image captured by the imaging device 12, and recognizes the center position of the pupil in the eye. An image of visible light in the target range 52 may be used for recognizing the center position of the pupil. The detection unit 32 recognizes the reflection of infrared rays from the cornea of the human eye using the infrared image within the target range 52. For example, a range having an infrared intensity greater than or equal to a predetermined threshold in the eye is recognized as the cornea. The cornea is recognized as a circular shape, for example. The detection unit 32 calculates the line-of-sight direction 56a of the person 54a based on the distance between the center position of the pupil and the cornea and the radius of curvature of the cornea.

  FIG. 5 is a flowchart illustrating the processing of the processor in the first embodiment. FIG. 6 is a diagram illustrating each process with respect to time in the first embodiment. As shown in FIGS. 5 and 6, at time t0, the irradiation device 10 stops (ie, turns off) infrared irradiation, and is a stop period. At time t1, the control unit 34 causes the irradiation device 10 to emit (that is, turn on) infrared rays (step S10). Thereby, for example, the control unit 34 outputs a control signal instructing irradiation of infrared rays to the irradiation apparatus 10.

  The irradiation apparatus 10 irradiates the target range 52 with infrared rays based on the control signal. The control unit 34 causes the imaging device 12 to image the target range 52 (step S12). For example, the control unit 34 outputs a control signal instructing imaging to the imaging device 12. The imaging device 12 captures an image of the target range 52 by capturing infrared light reflected from the target range 52 based on the control signal. The imaging device 12 outputs image information regarding the captured image to the detection unit 32.

  At time t2, the control unit 34 causes the irradiation device 10 to stop emitting infrared rays (step S14). For example, the control unit 34 outputs a control signal instructing to stop the infrared irradiation to the irradiation device 10. The irradiation device 10 stops the infrared irradiation based on the control signal. The time between the times t1 and t2 is, for example, 1 ms to 100 ms.

  The detection unit 32 detects the line-of-sight direction 56a of the person 54a within the target range 52 (step S16). For example, the detection unit 32 acquires image information from the imaging device 12. The detection unit 32 calculates the line-of-sight direction 56a of the person 54a based on, for example, an image captured by the imaging device 12. The detection information includes information indicating the line-of-sight direction 56a (for example, coordinates within the line-of-sight detection range 58). When the line-of-sight direction cannot be detected within the target range 52, information indicating that the line-of-sight direction cannot be detected is included.

  The control unit 34 determines whether or not a person is included in the target range 52 (step S18). For example, the control unit 34 acquires detection information from the detection unit 32. When the information that cannot detect the line-of-sight direction is included in the detection information, or the information that indicates the line-of-sight direction is not included in the detection information, the control unit 34 determines No, and the information that cannot detect the line-of-sight direction is included in the detection information. If it is not included or information indicating the line-of-sight direction is included, it is determined as Yes.

  For example, in FIG. 1, when there is no person in the target range 52 like the person 54c, and when the gaze direction 56b of the person 54b in the target range 52 cannot be detected like the person 54b, the control unit 34 determines No. When the line-of-sight direction 56a faces the line-of-sight detection range 58 like the person 54a, the control unit 34 determines Yes. Further, when a plurality of people are included in the target range 52, it may be determined as Yes when at least one gaze direction in the target range 52 is detected. The control unit 34 may determine whether a person is included in the image of the target range 52 from the image information without using the detection information.

  When No in step S18, the control unit 34 stands by (step S15). The waiting period is, for example, 1 second or longer. Then, it returns to step S10 and performs step S10 to step S18. The interval T1 from step S18 to S10 is, for example, 1 second or longer.

  When Yes in step S18, the control unit 34 sets the operation period. At time t3, the control unit 34 causes the irradiation device 10 to emit infrared rays (step S20). The control unit 34 causes the imaging device 12 to image the target range 52 (step S22). The control unit 34 causes the irradiation device 10 to stop emitting infrared rays (step S24). The detection unit 32 detects the line-of-sight direction of the person within the target range 52 (step S26). The control unit 34 determines whether or not a person is included in the target range 52 (step S28). Details of the processes of steps S20, S22, S24, S26, and S28 are the same as those of steps S10, S12, S14, S16, and S18, respectively, and a description thereof is omitted.

  When Yes in step S28, the control unit 34 outputs detection information to the outside (step S30). The control unit 34 may output the detection information acquired from the detection unit 32 as it is, or may process and output the detection information. Thereafter, the process returns to step S20. The interval T2 from step S20 to S30 is, for example, 1/30 second.

  When No in step S28, the control unit 34 determines whether or not a person is included in the target range 52 a predetermined number of times or more or a predetermined time or more (step S32). For example, the control unit 34 determines that no person is included in the target range 52 when no person is included in the target range 52 for five seconds continuously in step S32. When No, it returns to Step S20 and performs Steps S20 to S28. When Yes in step S32, the control unit 34 determines whether or not the process is finished (step S34). When yes, the process ends. When No, the process returns to step S15 at time t4. Thereby, it becomes a stop period.

  As described above, the control unit 34 sets the stop period when a person is not included in the target range 52 for a predetermined period (for example, 5 seconds) in the operation period, and sets the person in the target range 52 for each predetermined period (for example, every second). Determine whether it is included. When a person is included, the control unit 34 sets the operation period.

[Comparative Example 1]
As Comparative Example 1, an example in which the irradiation apparatus 10 irradiates the target range 52 constantly will be described. In order to detect the line-of-sight direction, it is required that the intensity of infrared rays reflected from the target range 52 be at least a certain level. For example, when the target range 52 is away from the irradiation device 10 or when an obstacle such as glass exists between the target range 52 and the irradiation device 10, the intensity of infrared rays reflected by the imaging device 12 becomes small. For this reason, if it is going to increase the emitted light intensity of the irradiation apparatus 10, the power consumption and the emitted-heat amount of the irradiation apparatus 10 will increase. Thereby, the cooling device which enlarges the irradiation apparatus 10 and / or cools the irradiation apparatus 10 is used. Therefore, the irradiation device 10 is increased in size and / or cost. It is also conceivable to improve the performance of the imaging device 12 without increasing the light emission amount of the irradiation device 10. In this case, the imaging device 12 is increased in size and / or cost. Thus, in Comparative Example 1, the line-of-sight inspection system is increased in size and / or cost.

  In Comparative Example 1, wasteful power is consumed when infrared rays are irradiated even though there is no person in the target range 52. Therefore, in order to determine whether or not a person is included in the target range 52, it is conceivable to provide a human sensor that senses a person. However, providing a human sensor increases the size and / or cost of the line-of-sight detection system.

[Effect of Example 1]
For example, in the first embodiment, when the imaging device 12 captures 30 images per second and the irradiation device 10 irradiates the infrared rays in order to capture one image, the irradiation device 10 per second. The irradiation time of infrared rays is 450 msec. Compared with the comparative example 1 in which the irradiation apparatus 10 always irradiates infrared rays, in Example 1, the time for the irradiation apparatus 10 to irradiate infrared rays is 45%. Furthermore, in the stop period, if the standby period in step S15 is 1 second, the time for which the irradiation apparatus 10 irradiates infrared rays per second is 15 milliseconds, for example. Therefore, compared with the comparative example 1, in Example 1, the time which the irradiation apparatus 10 irradiates infrared rays can be 1.5%. Since the power consumption of the irradiation device 10 can be suppressed, even if the emission intensity of the irradiation device 10 is increased while the imaging device 12 captures an image, there is no need for a cooling device. Further, it is not necessary to improve the performance of the imaging device 12. Therefore, it is possible to reduce the size and / or cost of the line-of-sight detection system as compared with Comparative Example 1. Further, since it is determined whether or not a person is included in the target range 52 based on the image captured by the imaging device 12, it is not necessary to provide a human sensor, and the line-of-sight detection system can be downsized and / or reduced in cost. Is possible. If the visible light is hardly included in the light irradiated by the irradiation device 10, the person will not feel discomfort even if the irradiation device 10 repeatedly irradiates and stops infrared rays.

  According to the first embodiment, the irradiation apparatus 10 irradiates the target range 52 with infrared rays as in step S20 of FIG. As in step S <b> 22, the imaging device 12 captures an infrared image reflected from the target range 52. As in step S <b> 26, the detection unit 32 detects the line of sight of a person within the target range 52 based on the image. As in step S <b> 28, the control unit 34 determines whether a person is included in the target range 52 based on the image. As in step S <b> 15, when the control unit 34 determines that no person is included in the target range 52, the control unit 34 sets a stop period (first period) in which the irradiation apparatus 10 stops the infrared irradiation.

  In the line-of-sight detection system of FIG. 3A, the line-of-sight detection program converts the line-of-sight detection program into an infrared image reflected from the target range 52 captured by the imaging device 12 by the irradiation device 10 irradiating the target range 52 with infrared rays. Based on this, the line of sight of the person within the target range 52 is detected. The line-of-sight detection program determines whether or not a person is included in the target range 52 based on the image in the computer 18, and if it is determined that no person is included in the target range 52, an infrared ray is transmitted to the irradiation device 10. This is a stop period in which irradiation is stopped.

  Thereby, when there is no person in the target range 52, the irradiation apparatus 10 can be turned off. Therefore, the power consumption of the irradiation apparatus 10 can be suppressed. Moreover, since the control part 34 determines the presence or absence of the person in the object range 52 based on an image, size reduction and / or cost reduction are attained.

  Moreover, like step S18, the control unit 34 determines whether or not a person is included in the target range 52 during the stop period. When determining that the person is included in the target range 52, the control unit 34 causes the irradiation device 10 to emit infrared rays as in step S20, and causes the imaging device 12 to capture an image as in step S22. The operation period (second period) in which the detection unit 32 detects the line of sight as in S26. Thereby, in the stop period, when a person returns within the target range 52, the operation period can be restored.

  Further, in the stop period, the control unit 34 causes the irradiation device 10 to emit infrared rays at each interval T1 (first interval) (step S10), causes the imaging device 12 to capture an image (step S12), and after imaging, The irradiation device 10 is stopped from emitting infrared rays (step S14), and it is determined whether or not a person is included in the target range 52 based on the image captured by the imaging device 12 immediately before (step S18). When the control unit 34 determines that a person is included in the target range 52, the control unit 34 sets the operation period. Thus, in the stop period, the irradiation device 10 is irradiated with infrared rays periodically or aperiodically, and the target range 52 is imaged. Thereby, the control unit 34 can determine whether a person is included in the target range 52 during the stop period.

  The irradiation apparatus 10 may always irradiate the target range 52 with infrared rays during the operation period. However, in this case, the power consumption of the irradiation apparatus 10 increases.

  Therefore, as illustrated in FIG. 7, the control unit 34 causes the irradiation device 10 to emit infrared rays at intervals T2 (second intervals) shorter than the intervals T1 during the operation period (step S20), and images are captured on the imaging device 12. After the imaging (step S22), the irradiation device 10 is stopped from emitting infrared rays (step S24), and the detection unit 32 is caused to detect the line of sight (step S26). Thereby, the power consumption of the irradiation apparatus 10 in an operation period can be suppressed.

  As in step S <b> 32, when it is determined that the target range 52 does not include a person for a predetermined period (third period) or more, the control unit 34 sets a stop period. Thereby, even when the detection unit 32 cannot detect a person due to, for example, the person in the target range 52 facing backward or blinking, it is possible to suppress the stop period.

  The predetermined period is preferably not less than the interval T1. Thereby, a stop period can be set appropriately.

  The control unit 34 may determine whether a person is included in the target range 52 based on an image captured by the imaging device 12 separately from the detection result of the detection unit 32. In this case, the process of the control unit 34 becomes complicated.

  Therefore, when the detection unit 32 detects the line of sight of a person in the target range 52, the control unit 34 determines that a person is included in the target range 52, and the detection unit 32 determines that the person in the target range 52 When the line of sight is not detected, it is determined that no person is included in the target range 52. Thereby, the process of the control part 34 can be simplified.

  When determining the presence / absence of a person in the target range 52 based on the presence / absence of the line-of-sight detection result, if the person in the target range 52 is looking outside the line-of-sight detection range 58, the control unit 34 includes the person in the target range 52. It is determined that it is not. Therefore, the determination in step S32 is performed. Thereby, when a person is included in the target range 52, it is possible to further suppress the suspension period.

  FIG. 7 is a flowchart illustrating an example of step S15 of FIG. As shown in FIG. 7, the control unit 34 determines whether or not a period T4 (for example, 5 seconds) or more has elapsed since the operation period became the stop period (step S40). If No, the system waits for a predetermined period (for example, 1 second) as standby 1 (step S42). When Yes, as standby 2, the system waits for a period shorter than step S42 (for example, 0.5 seconds) (step S44). There is a possibility that a person returns to the target range 52 immediately after shifting from the operation period to the stop period. Moreover, although the person is included in the target range 52, there is a possibility that the line of sight is only deviated for a long time.

  Therefore, the control unit 34 shortens the interval T1 when the operation period is not the post-period T4 (fourth period) and the operation period is the post-period T4 or more after the operation period is the post-period T4. Thereby, it can return to an operating period quickly.

  Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and changes can be made within the scope of the gist of the present invention described in the claims. It can be changed.

DESCRIPTION OF SYMBOLS 10 Irradiation device 12 Imaging device 14 Processor 18 Computer 32 Detection part 34 Control part 50 Gaze detection system 51 Gaze detection module 52 Target range 54a-54c Person 56a-56c Gaze direction

Claims (9)

An irradiation device for irradiating the target area with infrared rays;
An imaging device that captures the infrared image reflected from the target range;
A detection unit for detecting a gaze of a person within the target range based on the image;
A first period in which it is determined whether or not a person is included in the target range based on the image, and when it is determined that a person is not included in the target range, the irradiation apparatus stops the infrared irradiation. A control unit, and
A line-of-sight detection system comprising:   In the first period, the control unit determines whether or not a person is included in the target range. When the control unit determines that a person is included in the target range, the infrared ray is transmitted to the irradiation device. The line-of-sight detection system according to claim 1, wherein a second period is set in which the image capturing apparatus captures an image and the detection unit detects the line of sight.   In the first period, the control unit causes the irradiation device to irradiate the infrared ray at each first interval, causes the imaging device to take an image, causes the irradiation device to stop emitting the infrared ray after imaging, When it is determined whether or not a person is included in the target range based on an image captured by the imaging device immediately before, and it is determined that a person is included in the target range, the second period is set. Item 3. The eye gaze detection system according to item 2.   In the second period, the control unit causes the irradiation device to irradiate the infrared light at every second interval shorter than the first interval, causes the imaging device to capture an image, and causes the irradiation device to capture the infrared light after imaging. The line-of-sight detection system according to claim 3, wherein the irradiation is stopped and the line of sight is detected by the detection unit.   The visual axis detection system according to claim 4, wherein the control unit determines the first period when it is determined that no person is included in the target range for a third period longer than the first interval.   The line-of-sight detection system according to any one of claims 1 to 4, wherein the control unit sets the first period when it is determined that no person is included in the target range for a third period or more.   The control unit may be configured such that when the second period is set as the first period and the fourth period is not passed, the first period is set as the first period, and the first period is set as compared with the case where the second period is exceeded. The line-of-sight detection system according to claim 3, wherein the interval is shortened.   The control unit determines that a person is included in the target range when the detection unit detects the line of sight of a person in the target range, and the detection unit determines the line of sight of the person in the target range. The line-of-sight detection system according to any one of claims 1 to 7, wherein when it is not detected, it is determined that no person is included in the target range. On the computer,
Detecting a line of sight of a person in the target range based on the infrared image reflected from the target range captured by the imaging device when the irradiation device irradiates the target range with infrared rays;
It is determined whether or not a person is included in the target range based on the image, and when it is determined that a person is not included in the target range, the irradiation apparatus is configured to stop the infrared irradiation. Steps,
Gaze detection program that executes

Images