JP2005261728A - Line-of-sight direction recognition apparatus and line-of-sight direction recognition program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a line-of-sight direction recognition apparatus and a line-of-sight direction recognition program capable of reducing burdens on an observer and calibrating the line of sight. <P>SOLUTION: The line-of-sight direction recognition apparatus is provided with an object detection means for detecting an object in the range of the visual field of the observer, an eyeball direction detection means for observing the eyeball of the observer and detecting its optical axis direction, that is the calculative direction of the line of sight, and a line-of-sight direction recognition computer for processing data obtained from them and calibrating the direction of the line of sight. For instance, it is defined that the range of the visual field of the observer is indicated by an image 60. At the time of indicating the calculative movement of the visual field by an arrow 66 in the image 60, when an automobile 62 moving in the same direction for the same amount is present near the line of sight as indicated by an arrow 64, the automobile 62 is defined as a line-of-sight candidate object with a high possibility to be a line-of-sight candidate empirically, the disagreement Δ<SB>2</SB>between the position and the calculative position of the line of sight is calibrated, and the direction from the eyeball of the observer to the position of the line-of-sight candidate object is defined as the direction of the line of sight. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a gaze direction recognizing device and a gaze direction recognizing program for detecting the optical axis direction of an eyeball of an observer and calibrating this to obtain the direction of the gaze of the observer.

  Conventionally, for example, various eye cameras have been provided as devices for detecting a so-called line of sight (visual axis) for detecting which position on an observation surface an observer is observing. Since the detected line of sight is the object of interest of the observer, it can be used to specify the in-focus object of the camera using this, or can be used to specify the object of image processing.

  In Patent Document 1, an axis in the direction of a gazing store, which is observed by a photographer (observer) through a camera finder system, a so-called line of sight (visual axis) is obtained when the observer's eyeball is illuminated. A technique for detecting using a reflection image of an eyeball is disclosed. Furthermore, due to individual differences among observers, the optical axis of the reflected light from the cornea of the anterior segment of the eyeball does not necessarily coincide with the center position of the pupil, and the gaze (visual axis) obtained from the reflected image of the eyeball and the actual observation Discloses a method for correcting the occurrence of an error with the gaze (visual axis) that the person is gazing at.

  That is, in order to correct individual differences in the line of sight, the observer is allowed to see two targets having different positions, and the line of sight is corrected based on the rotation angle of the eyeball of the observer calculated at that time. For example, when two targets are provided in the finder and the observer selects “calibration mode”, the two targets blink sequentially. When the first target blinks, the observer gazes at it, and when the second first target blinks, the observer shifts his gaze to watch. In this way, the observer can see two targets having different positions, the rotation angle of the eyeball at that time can be obtained, and the line of sight can be corrected.

JP-A-6-34874

  By executing the calibration mode as described above, the line of sight of the observer can be corrected, but this places a burden on the observer. That is, the observer needs to perform a calibration work in advance before entering the normal observation state or the photographing state, and in that work, as in Patent Document 1, follow the instructions for the blinking target, etc. You have to watch. Also. If the calibration is not successful or is different from the value in the previous calibration, the calibration will be performed more carefully.

  As described above, in the prior art, the observer is required to perform a special calibration operation for the correction and calibration of the line of sight, which places a burden on the observer and is inconvenient.

  An object of the present invention is to provide a gaze direction recognizing apparatus and a gaze direction recognizing program that can solve the problems of the prior art and reduce the burden on the observer and perform gaze calibration.

1. Principle of the present invention The present invention aims to make the calibration process easier and to make the observer almost unaware that the calibration is being performed. And, to try to use the empirical fact that the observer's line of sight can be judged by looking at the movement of the observer's face and eyes and the face or eyes facing It is. For example, if the observer's face moves from right to left and follows the line of sight, the car is just moving from right to left. Yes. That is, it can be seen that the viewing direction of the observer is in the vehicle. Also, if a person is writing with a pen, the person's line of sight is at the tip of the pen. That is, at this time, the line of sight of the observer is at the pen tip.

  In this way, what an observer wants to see can often be judged from experience by considering the observation environment surrounding the observer. Then, if the observer's line of sight is in the vicinity of the sight line candidate object in experience, it is assumed that the observer's line of sight is directed toward the line of sight candidate object, and the reflected image of the observer's actual eyeball is observed When the line-of-sight direction obtained by this is different from this, it can be calibrated. In this way, the line of sight can be calibrated to the observer in the normal observation state, and it is not necessary to require a special calibration operation from the observer. The present invention is based on the above idea.

2. In order to achieve the above object, a gaze direction recognition apparatus according to the present invention is a normal gaze direction recognition apparatus that detects the direction of the observer's line of sight by detecting the optical axis direction of the eyeball of the observer. And a calibrating unit for calibrating the direction of the line of sight.

  The gaze direction recognizing device according to the present invention is a gaze direction recognizing device that detects the direction of the observer's line of sight by detecting the direction of the optical axis of the observer's eyeball. An object detection means for detecting the position of a sight candidate object that is likely to be a sight candidate from experience, an eyeball direction detection means for detecting the optical axis direction of the observer's eyeball, Calibration means for calibrating the detected eyeball optical axis direction from the observer's eyeball to the position of the candidate object of sightline as the sightline direction, and calibrating the sightline direction in a normal observation state It is characterized by doing.

  In addition, it is preferable that the line-of-sight candidate object includes an object having a movement correlated with the movement of the observer's eyeball. Moreover, it is preferable that the line-of-sight candidate object includes a movement mark in display means that the observer is looking at. Moreover, it is preferable to use an object with a high possibility of an observer's gaze together with the movement mark in the display means correlated with the movement of the eyeball of the observer as the line-of-sight candidate object. Further, the line-of-sight candidate object preferably includes a pen tip written by the observer. Moreover, it is preferable to use the eye gaze candidate object together with an object that is highly likely to be watched by the observer, together with a pen tip correlated with the movement of the eyeball of the observer.

  The line-of-sight candidate object is based on a comparison between the size of the object in the range of the observer's visual field and the distribution of the stay points in the optical axis direction of the eyeball of the observer detected by the eyeball direction detection means. Are preferably detected.

  In addition, the line-of-sight candidate object has a size of a human skin color region in the range of the observer's visual field and a distribution of stay points in the optical axis direction of the eyeball of the observer detected by the eyeball direction detecting means. It is preferably a human face that is detected based on matching.

  In the gaze direction recognizing device according to the present invention, after the gaze direction is calibrated with respect to the eyeball optical axis direction, the characteristics of the frequent object in which the gaze direction of the observer may stay frequently and the eye may turn is acquired by learning. It is preferable to include learning means for performing update and updating means for updating the position of the line-of-sight candidate object based on the learning result of the frequent object.

  The gaze direction recognizing device according to the present invention is a gaze direction recognizing device that detects the direction of the observer's line of sight by detecting the direction of the optical axis of the observer's eyeball. Position specifying means, an object detection means for specifying the position of a line-of-sight candidate object specified in the range of the visual field of the observer, and an eyeball direction detection means for detecting the optical axis direction of the observer's eyeball And a calibrating unit that calibrates the direction from the observer's eyeball toward the position of the sight candidate object with respect to the detected eyeball optical axis direction, and the direction of the sightline in a normal observation state. Is calibrated.

  Moreover, it is preferable that the designated line-of-sight candidate object is the face of the closest person facing the face of the observer. Further, it is preferable that the specified line-of-sight candidate object is a display unit that faces the face of the observer.

  A gaze direction recognition program according to the present invention is a gaze direction recognition program that is executed in a gaze direction recognition device that detects an optical axis direction of an eyeball of an observer and obtains the direction of the gaze of the observer. Detecting a position of a visual line candidate target object in the range of the visual field, and detecting a position of the visual line candidate target object that is likely to be a visual line candidate from experience, and an eyeball of the observer An eyeball direction detecting step for detecting the optical axis direction, and a calibration step for performing calibration with respect to the detected eyeball optical axis direction, with the direction from the observer's eyeball toward the position of the candidate eyesight target line of sight. The direction of the line of sight is calibrated in the normal observation state.

  A gaze direction recognition program according to the present invention is a gaze direction recognition program that is executed in a gaze direction recognition device that detects an optical axis direction of an eyeball of an observer and obtains the direction of the gaze of the observer. Identifying a position of a visual line candidate object within the range of the visual field of the object, the object detecting process identifying the position of the visual line candidate object specified in the visual field of the observer, and the eyeball of the observer An eyeball direction detecting step for detecting the optical axis direction, and a calibration step for performing calibration with respect to the detected eyeball optical axis direction, with the direction from the observer's eyeball toward the position of the candidate eyesight target line of sight. The direction of the line of sight is calibrated in the normal observation state.

  According to at least one of the above-described configurations, the position of a visual line candidate object that is likely to be a visual line candidate from experience is detected in the range of the visual field of the observer, and is compared with the optical axis direction of the observer's eyeball. The direction from the eyeball toward the position of the visual candidate object is calibrated as the visual line direction, and the visual line calibration is performed in the normal observation state. Here, the optical axis direction of the eyeball of the observer means a calculation line of sight obtained by projecting a parallel light beam from the light source onto the anterior eye portion of the eyeball and obtaining from the corneal reflection image by the reflected light from the cornea and the imaging position of the pupil ( (Visual axis) direction. Therefore, when there is a line-of-sight candidate that is likely to be a line-of-sight candidate in the range of the observer's visual field using the normal observation state of the observer, the line-of-sight is regarded as being viewed by the observer. Can be calibrated. In this way, the observer is hardly made aware that the calibration is performed, and the burden on the observer is small.

  With at least one of the above-described configurations, the line-of-sight candidate object can move in correlation with the movement of the eyeball of the observer. For example, a pen tip written by an observer. For example, a movement mark in the display means that the observer is looking at. In addition, other objects such as objects that do not move can be used together with objects that are highly likely to be watched by the observer. By using a plurality of line-of-sight candidate objects among those that may be watched by the observer, the line-of-sight direction can be calibrated more accurately.

  With at least one of the above-described configurations, the size of the object in the range of the observer's visual field can be compared with the optical axis direction of the observer's eyeball, that is, the distribution of the distribution of stay points of the calculated line of sight (visual axis). Based on this, a line-of-sight candidate object is detected. Since the observer's line of sight has a distribution in the range of the size of the object, when the object corresponding to the size of the distribution in the line of sight is in the range of the observer's field of view, the observer The line of sight can be calibrated as if looking at it. In this case, even a target object that does not move can be detected as a line-of-sight candidate target object.

  The line of sight based on the comparison between the size of the human skin color region in the range of the visual field of the observer and the size of the distribution of the stay points of the visual line of sight (visual axis) calculated by the observer by at least one of the above-described configurations Candidate objects are detected. For example, even if the observer is looking at the face of the other person, since the features of the person's face are various, it is often difficult to detect the position of the line-of-sight candidate object using that feature as a clue. . Therefore, if the skin color of a person is detected, the position of the person's face can be easily detected. That is, when the human skin color region is within the range of the visual field of the observer, the observer can calibrate the line of sight as if he / she is looking at it. In this case, even the face of a person who does not move can be detected as the sight line candidate object.

  With at least one of the above-described configurations, the characteristics of the frequent object that the observer often looks at in the state where the line-of-sight direction is calibrated and the accuracy of the line-of-sight direction is ensured are learned. Based on the learning result, the position of the line-of-sight candidate object is updated. By using the frequent object in this manner, it is possible to correct a change with time in the accuracy of the calibration of the gaze direction, and it is possible to perform the calibration of the gaze direction more accurately.

  Further, according to at least one of the above-described configurations, the position of the line-of-sight candidate object specified in the range of the visual field of the observer is detected, compared with the optical axis direction of the observer's eyeball, and the line-of-sight candidate object is detected from the observer's eyeball. The direction toward the object position is calibrated as the line-of-sight direction, and the line-of-sight calibration is performed in the normal observation state. The designation may be determined in advance from the observation environment surrounding the observer, or an object that is likely to be viewed by the observer may be identified and designated from the analysis of the screen. Therefore, when there is a line-of-sight candidate that is likely to be a line-of-sight candidate in the range of the observer's visual field using the normal observation state of the observer, the line-of-sight is regarded as being viewed by the observer. Can be calibrated. In this way, the observer is hardly made aware that the calibration is performed, and the burden on the observer is small.

  With at least one of the above-described configurations, the line-of-sight candidate object can be the face of the closest person facing the face of the observer. In addition, the line-of-sight candidate object can be the display means that faces the viewer's face.

  As described above, according to the gaze direction recognition device and the gaze direction recognition program according to the present invention, the gaze is calibrated using the normal observation state of the observer, so that the burden on the observer can be reduced. .

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following, an eyeball camera that detects the optical axis direction of the eyeball and an imaging camera that images the observer's field of view are both attached to the observer's head under a predetermined positional relationship, or an omnidirectional camera. However, other configurations and arrangements may be used. For example, as in a camera finder system, a system that shows the observer's field of view and detects the optical axis direction of the eyeball may be combined therewith. In addition, the eyeball camera and the imaging camera may be installed in different locations as long as the mutual positional relationship is clear in advance.

  FIG. 1 shows a state when the gaze direction of the observer 2 is calibrated using the gaze direction recognition device 10, and a block diagram of the gaze direction recognition device 10 is shown. The line-of-sight direction recognition device 10 includes an eyeball camera 12, an imaging camera 14, and a line-of-sight direction recognition computer 20 that are attached to the head of the observer 2 in a predetermined positional relationship. In FIG. 1, the observer 2 puts the eyeball camera 12 and the imaging camera 14 on the head, but other than that, the observer 2 is in a normal observation state and can freely act on his / her own will. In the example of FIG. 1, the exhibit on the bulletin board 6 in front of you is seen.

  The eyeball camera 12 is an optical device for observing the eyeball three-dimensionally and obtaining the optical axis direction of the eyeball. As this eyeball camera 12, what was disclosed by patent document 1, for example can be used. That is, it has a light source that projects a parallel light beam onto the anterior segment of the eyeball, and an optical system that captures the corneal reflection image by the reflected light from the cornea and the image formation position of the pupil. The cornea reflection image detected by the eyeball camera 12 and the data of the pupillary imaging position are sent to the line-of-sight direction recognition computer 20, and the optical axis direction of the eyeball, that is, the calculated line-of-sight direction is obtained. In FIG. 1, a calculated line-of-sight direction 16 from the eyeball 4 of the observer 2 toward a position on the bulletin board 6 is shown.

  The imaging camera 14 is a camera that images an observer's field of view, and can be configured by, for example, a commercially available CCD camera. Since the observer's 2 field of view changes when the observer 2 moves the head, it is desirable to be fixed to the head. However, the observer 2 detects a change in the position of the head of the observer 2, particularly the eyeball 4, If the imaging range can be changed accordingly, it is not necessarily fixed to the head. Further, in order to associate the visual line direction 16 calculated by the eyeball camera 12 with the visual field of the observer, the relative positional relationship between the imaging camera 14 and the eyeball camera 12 needs to be clarified in advance. In FIG. 1, the imaging camera 14 and the eyeball camera 12 are fixedly arranged in a predetermined positional relationship. In addition, it is desirable to adjust the imaging camera 14 so that the imaging direction and range of the imaging camera 14 substantially match the range of the observer's visual field. The adjustment can be performed by an electrical operation inside the imaging camera 14, and the mounting position on the head may be changed. In FIG. 1, the range captured by the imaging camera 14 is shown as the range 18 of the visual field of the observer 2.

  The line-of-sight direction recognition computer 20 includes a CPU 22, an eyeball camera I / F (24) that communicates signals with the eyeball camera 12, an imaging camera I / F (26) that communicates signals with the imaging camera 14, a keyboard, and the like. Input unit 28, an output unit 30 such as a display, and a storage device 32 for storing programs and the like, which are connected to each other via an internal bus. Moreover, you may have a communication control function connected to an external network. The line-of-sight direction recognition computer 20 can be configured by a general computer having a storage device.

  The CPU 22 is based on the data received from the eyeball camera 12, based on the data received from the imaging camera 14 and the eyeball direction detection unit 40 having a function of obtaining the optical axis direction of the eyeball, that is, the calculated line-of-sight direction 16. An object detection unit 42 having a function of obtaining the position of the line-of-sight candidate object in the range 18, and a calibration unit having a function of configuring the line-of-sight direction when the calculated line-of-sight direction and the position of the line-of-sight candidate object do not match 48. These functions can be realized by software, and specifically, can be realized by executing a recognition program in the corresponding line-of-sight direction. Moreover, you may comprise so that a part of each function may be implement | achieved by hardware.

  The eyeball camera I / F (24) is an interface circuit for communicating signals with the eyeball camera. For example, it can be configured by a circuit for performing serial transmission / reception of data or parallel transmission / reception. Similarly, the imaging camera I / F (26) is an interface circuit for performing signal communication with the imaging camera 14.

  The operation of the line-of-sight direction recognition computer 20, in particular, each function of the CPU 22, will be described below based on a specific line-of-sight direction recognition procedure. The reference numerals shown in FIG. 1 are used. First, the observer 2 sets the eyeball camera 12 and the imaging camera 14 on the head. In the setting, the light source of the eyeball camera 12 irradiates the anterior eye part, and the imaging range of the imaging camera 14 approximately matches the range of the observer's visual field. The observer 2 has done so far, and thereafter, the gaze direction recognition device 10 automatically proceeds with the procedure.

  First, the optical axis direction of the eyeball 4 of the observer 2, that is, the calculated line-of-sight direction is obtained by the function of the eyeball direction detection unit 40 of the CPU 22 (eyeball direction detection step). Specifically, the eyeball direction detection unit 40 instructs the eyeball camera 12 to irradiate the anterior eye part of the eyeball 4 with a parallel light beam from the light source built in the eyeball camera 12 as described above, and reflect the cornea by the optical system. An image and a pupil imaging position are imaged. The imaged data is transmitted via the eyeball camera I / F (24). The eyeball direction detection unit 40 processes the transmitted imaging data and calculates the optical axis direction from the direction in which the shape of the eyeball 4 is directed by a known eyeball direction calculation calculation. To do.

  Moreover, the visual field of the observer 2 is imaged by the function of the visual field imaging module 44 of the object detection unit 42 (visual field imaging step). Specifically, the visual field imaging module 44 instructs the imaging camera 14 to image the visual field. The imaged data is transmitted via the imaging camera I / F (26). The visual field imaging module 44 acquires this as the visual field range 18 of the observer 2.

  Then, the position of the line-of-sight candidate object is detected in the range 18 of the visual field of the observer 2 (object detection process). Here, the line-of-sight candidate object is an object in the range 18 of the field of view of the observer 2 and an object that has a high possibility of being a line-of-sight candidate determined in advance.

  As the line-of-sight candidate object, an object having a movement correlated with the movement of the eyeball of the observer is raised. FIG. 2 is a diagram for explaining that the automobile 62 is a line-of-sight candidate object. FIG. 2 shows a screen 60 that captures the range of the field of view of the observer, and in this field of view, a single car 62 is going up the hill. The movement of the automobile 62 is indicated by an arrow 64. On the other hand, the movement of the line of sight of the eyeball calculated using the eyeball camera 12 is indicated by an arrow 66. In this example, the position of the visual line of sight of the eyeball is not on the automobile 62. However, the direction and amount of movement of the arrow 66 are substantially the same as the direction of movement of the arrow 64. In such a case, it can be assumed from the experience that the observer 2 is following the movement of the automobile 62 with his eyes.

  In FIG. 2, the field of view is described as being stationary. If the face of the observer moves, it may not necessarily be a still image, but the calculated gaze direction and the relative movement of the gaze candidate target are invariant. I will explain. The same applies to FIGS. 4 to 10 below.

  As described above, when there is an object having a motion correlated with the motion of the eyeball 4 of the observer 2 in the range 18 of the field of view of the observer 2, it can be set as a line-of-sight candidate object. Specifically, in order to extract an object having a correlation with the movement of the eyeball, a method of subtracting two image data at an arbitrary time interval can be used. That is, if there is movement, the position changes and remains without being erased by subtraction between the two image data. For example, the movement of the eyeball is converted to 5 cm / sec on the screen 60, subtracting two image data captured at an interval of 1 second, and the eyeball movement is the largest among the remaining images without being erased. An object having a high correlation may be detected. In this case, an object moved by a distance of 5 cm may be detected. In this way, the automobile 62 can be specified as the line-of-sight candidate object, and its position is obtained on the screen 60.

  Other line-of-sight candidate objects that have a movement that correlates with the movement of the observer's eyeball include the pen tip that the observer writes, the cursor that moves on the display screen that the observer is viewing, For example, the movement mark of the display means.

  FIG. 3 is a diagram showing a state where the observer 2 is writing on the sheet with the pen tip 72 using the pen 70. Elements similar to those in FIG. 1 are denoted by the same reference numerals. FIG. 4 shows a screen 80 that captures the range of the visual field of the observer in this case. The movement of the nib 72 is indicated by an arrow 84 in the field of view. The movement of the eyeball 4 about the calculated line of sight is indicated by an arrow 86. Even in this case, the position of the visual line of sight of the eyeball is not on the pen tip 72. However, the direction and amount of movement of the arrow 86 are substantially the same as the direction of movement of the arrow 84. In such a case, it can be assumed from the experience that the observer 2 is following the movement of the nib 72 with his eyes. In this way, the pen tip 72 can be specified as the line-of-sight candidate object, and its position is obtained on the screen 60.

  Returning to the gaze direction recognition procedure again, when the eyeball direction is detected and the position of the gaze candidate object is detected, gaze calibration is performed by the function of the calibration unit 48 (gaze calibration step). In addition, as long as it is before a visual line calibration process, the order of an eyeball position detection process and a target object detection process may be any first. In the calibration step, the calibration unit 48 superimposes the calculated line-of-sight position of the eyeball 4 and the position of the line-of-sight candidate object on the screen for the range of the visual field of the observer, and determines whether there is a match or mismatch between them. Is done. Whether the image data matches or not can be determined by a method such as subtraction of data.

When there is a discrepancy, the calculated line-of-sight direction is calibrated with the direction from the observer's eyeball toward the position of the line-of-sight candidate object as the line-of-sight direction. In FIG. 2, the difference in the position of the line of sight on the screen 60 is Δ ₂ . Rotation angle of the eyeball 4 corresponding to the delta ₂ is weight calibration. Similarly, in FIG. 4, the rotation angle of the eyeball 4 corresponding to delta ₄ is weight calibration. Thus, the gaze direction can be calibrated while the observer 2 is in the normal observation state. It should be noted that the processing on the program can be simplified by simply performing the calculation of Δ = 0 without determining the coincidence / non-coincidence.

  5 and 6 are diagrams schematically showing how the gaze direction calibration is performed. FIG. 5 shows screens 90 and 100 before the line-of-sight direction calibration is performed, and FIG. 6 shows screens 91 and 101 after the line-of-sight direction calibration is performed. In each figure, (a) is an image at time t, and (b) is an image at time t + Δt. Each screen shows that there are a car and two trees in the range of the observer's field of view. Here, when the optical axis direction of the observer's eyeball, that is, where the calculated line of sight is, is shown on the same screen, it is in the line of sight range 94 at time t and in the line of sight range 104 at time t + Δt. . The movement from the line-of-sight range 94 to the line-of-sight range 104 corresponds to the movement of the image range 92 at time t to the image range 102 at time t + Δt for the automobile. On the other hand, the two trees do not move and do not correspond to the movements of the line-of-sight ranges 92 and 102.

  The line-of-sight ranges 94 and 104 do not coincide with the vehicle image ranges 92 and 102, but are highly correlated with their movements. In other words, the range 92 and 102 of the image of the automobile has a movement in the visual field and has a high correlation with the movement of the line of sight. Further, there is no such correlation between two trees that are other objects in the range of the field of view of the observer. Therefore, as described above, the automobile can be the line-of-sight candidate object. FIG. 6 is a diagram illustrating the line-of-sight ranges 95 and 105 after the correlation between the image range 92 and 102 of the automobile and the line-of-sight ranges 94 and 104 is obtained and the line-of-sight direction is calibrated based thereon. In this way, the line-of-sight ranges 95 and 105 can be made to substantially coincide with the vehicle image ranges 92 and 102 by performing the line-of-sight direction calibration.

  In the above example, the line-of-sight candidate object has a correlation with the movement of the eyeball of the observer. That is, the line-of-sight candidate object moves. Even if the object does not move, it is possible to make a line-of-sight candidate object that is highly likely to be watched by the observer as follows.

  As described with reference to FIGS. 5 and 6, when the object has a certain size, the direction of the observer's line of sight is not a single point but stays with a certain spread and often has a distribution. Therefore, the size of the object in the range of the observer's field of view is compared with the distribution of the stay points in the observer's calculated line of sight, and the line-of-sight candidate object is identified based on that, and its position is detected. can do. That is, when an object corresponding to the size of the line-of-sight distribution is within the range of the observer's field of view, it is assumed that the observer is looking at it, and this is set as the line-of-sight candidate object. In the following, the distribution of staying points in the observer's calculated line of sight will be described as the line of sight range.

  FIG. 7 is a diagram illustrating a state in which a line-of-sight object is specified by collating the distribution of the line-of-sight range and the size of the object, the position is detected, and the line-of-sight direction is calibrated. FIG. 7A is a view showing the field of view of the observer before calibration, and shows that four people 110, 112, 114, 116 hold the cards 120, 122, 124, 126 on their right hands, respectively. . At this time, there are eight line-of-sight ranges 130 that are distributions in the line of sight calculated by the observer. Here, for example, paying attention to the leftmost line-of-sight range, and automatically collating an object having a size corresponding to the width of the distribution by image processing, several candidates appear. When the line-of-sight directions are calibrated for each of them, it is determined whether the remaining seven line-of-sight ranges are associated with objects having a size corresponding to the spread of the distribution. If the eight line-of-sight ranges 130 are associated with objects having sizes corresponding to the spreads of the respective distributions, the observer can see them.

  FIG. 7B is a diagram showing the line-of-sight range 131 when the line-of-sight direction is calibrated as shown by the arrows in FIG. In this way, the spread of the distribution of the eight line-of-sight ranges is associated with the faces of the four people 110, 112, 114, and 116 and the sizes of the four cards 120, 122, 124, and 126, respectively. Yes. Therefore, the line-of-sight candidate objects are the faces of these four people and the four cards, and the calibration of the line-of-sight direction is indicated by the arrow in FIG.

  In this way, the line-of-sight object can be specified by comparing the distribution of the line-of-sight range with the size of the object, the position thereof can be detected, and the line-of-sight direction can be calibrated. Although a certain amount of time is required to obtain the distribution of the line-of-sight range, it can be performed in a much shorter time than the moving object described with reference to FIGS. 2, 4, and 5-6. Therefore, even an object that does not move can be identified as a line-of-sight candidate object, and its position can be detected. For collation between the distribution of the line-of-sight range and the size of the object, in addition to quantitative collation such as the area of the line-of-sight range and the area of the object, collation between each shape is used. be able to. It is also possible to combine quantity matching and shape matching.

  In addition, the human face can be identified as a line-of-sight candidate object based on a comparison between the size of the person's skin color area in the range of the observer's field of view and the distribution of the stay point distribution of the line of sight calculated by the observer. . In general, since the characteristics of a person's face are various, it is often difficult to automatically detect the position of the person's face by image processing using that characteristic as a clue. Therefore, if the skin color of a person is detected, the position of the person's face can be easily detected.

  FIG. 8 is a diagram illustrating a state in which a human face is specified as a visual line candidate target object by collating the size of the human skin color area with the spread of the distribution of the visual line range of the observer. In FIG. 8, human skin color regions 140, 142, and 144 in the range of the visual field of the observer are schematically shown. In addition, the observer's line-of-sight ranges 150, 152, and 154 are three in this way, and none of them coincides with the human skin-colored areas 140, 142, and 144. Is correlated to human skin color regions 140, 142, and 144. Therefore, the line-of-sight direction can be calibrated by obtaining the human skin color regions 140, 142, and 144 as the line-of-sight candidate objects and obtaining the correlation between them and the line-of-sight ranges 150, 152, and 154.

  In this way, the line-of-sight object can be identified by comparing the spread of the line-of-sight range distribution with the size of the human skin color area, the position can be detected, and the line-of-sight direction can be calibrated. The human skin color area can be automatically detected by image processing, and can be detected instantaneously without requiring time. Therefore, even a face of a person who does not move can be identified as a line-of-sight candidate object, and its position can be detected.

  In addition to this, when a visual feature point can be detected within the range of the visual field of the observer, this can be set as a line-of-sight candidate object. For example, it detects a characteristic point of a pattern such as a characteristic edge, edge, or corner of a pattern, or a place where contrast, brightness difference, hue difference, etc. are large, and the line of sight is considered to be highly likely to be observed by an observer. Can be a candidate object.

  Further, the identification of the line-of-sight candidate object, the detection of the position thereof, and the subsequent calibration of the line-of-sight direction can be performed based on a plurality of line-of-sight candidate objects. For example, in the examples of FIGS. 7 and 8, a plurality of line-of-sight candidate objects are used, and the line-of-sight direction is calibrated using a comprehensive correlation between these objects and a plurality of line-of-sight ranges. In addition, a plurality of different methods can be used in combination as the method for specifying the line-of-sight candidate object. For example, the line-of-sight candidate object detection method for the moving object described in FIGS. 2, 4, and 5-6, and the line-of-sight candidate that can be detected without the movement described in FIGS. The object detection method can be used in combination. Thus, the accuracy of the gaze direction calibration can be further improved by appropriately combining a plurality of gaze candidate objects and a plurality of detection methods.

  In addition, a learning means can be used to cope with the temporal change in the calibration of the gaze direction. There are many cases where an observer often sees, that is, there is something that looks well during a time period in which a change over time becomes a problem. The feature of the frequent object where the line of sight frequently accumulates is recognized, and the position of the line-of-sight candidate object is updated. In this case, since the gaze direction cannot be calibrated unless the frequent object is accurately captured, the characteristics of the frequent object are learned by some method while the gaze direction accuracy is ensured. The change with time is corrected based on the above. As some method for ensuring the accuracy of the line-of-sight direction, the line-of-sight direction calibration in the embodiment described above can be used, and the line-of-sight direction calibration may be performed by other methods. In any case, after the line-of-sight direction is calibrated, the characteristics of the frequent object can be acquired by learning, and the position of the line-of-sight candidate object can be appropriately updated based on the learning result.

  In the above, based on the direction and amount of the eyeball movement of the observer, the size and shape of the line-of-sight range, the color of the object, etc., the line-of-sight candidate object is automatically identified using image processing technology, etc. Direction calibration can also be performed automatically. Also, for static objects not included in these, if the sight line candidate object can be identified empirically by the observation environment, the position is detected by designating it on the screen, and the line of sight is based on it. Directional calibration can be performed.

  For example, the line of sight candidate object can be identified by such an observation environment, for example, the face of the person closest to the face of the observer, the bulletin board ahead of the face of the observer, display means, etc. can give.

  FIG. 9 shows a screen 160 that captures the range of the field of view when the observer is in a conference room or the like. The observer's field of view includes a painting 162 on the wall, the opponent's face 164 ahead of the observer's face, and the like. The calculated visual field position 166 of the observer's eye is far from the painting 162 and does not coincide with the face 164 of the other party. However, the face 164 of the opponent is partly included in the vicinity 168 of the visual line of calculation of the observer's eyeball. Here, the vicinity of the line of sight is a range in which, for example, the viewing angle is set to 30 degrees, and if the range is within that range, the observer is likely to be watching. Thus, from the screen 160, the observer is more likely to see the opponent's face 164 than the painting 162. Therefore, on the screen 160, the opponent's face 164 can be designated as a line-of-sight candidate object, and the mismatch between the position and the calculated line-of-sight position 166 can be calibrated. Although the line-of-sight candidate object can be specified on the image, it may be specified by other specifying means or the like as long as it is within the range of the visual field of the observer, not on the image.

  In this way, the position of the visual line of sight of the observer's eyeball is calculated within the range of the visual field of the observer, and if there are no objects that are candidates for the visual line in experience, the area near the line of sight is sequentially expanded. Then, the object that becomes the line-of-sight candidate that first enters the line-of-sight vicinity region can be set as the line-of-sight candidate object. The enlargement of the line-of-sight vicinity region can be realized, for example, by sequentially increasing the viewing angle in increments of 0 to 5 degrees.

  FIG. 10 shows a case where there is a bulletin board 172, a sample 174 used for explanation, and two sight line candidates on the screen 170 that captures the range of the visual field of the observer. In this example, the calculated gaze position 176 of the eyeball of the observer does not match the bulletin board 172 or the sample 174. However, if the vicinity 178 of the line of sight is sequentially enlarged, the sample 174 is included in the range first. Therefore, it is possible to designate the sample 174 as a sight line candidate target, and to calibrate the sight line based on this. As described above, when there are a plurality of line-of-sight candidates in the range of the visual field of the observer, the vicinity of the line-of-sight can be enlarged sequentially, and the object first included in the vicinity of the line-of-sight can be designated as the line-of-sight candidate object.

  When the line-of-sight candidate object can be narrowed down to a small number in advance according to the observation environment, this can be given to the object detection unit 42, and the line-of-sight candidate object can be automatically specified by the method of sequentially expanding the line-of-sight vicinity as described above. For example, (closest person face, bulletin board, sample) is given as a candidate. In addition, when it is not possible to narrow down items that are candidates for the line of sight in advance, it is possible to designate what can be determined as a line-of-sight candidate object based on experience by the user viewing the screen later. When the line-of-sight candidate object is designated in this way, the object detection unit 42 acquires it, detects its position, and based on this, the line-of-sight direction is automatically calibrated.

  In addition, regarding the designation of the above-mentioned line-of-sight candidate object, it has been described that the position of the observer's line of sight is one point, but as described above, the line of sight of the observer is assumed to be a line-of-sight range having a spread distribution. Range matching can be used. Further, the gaze direction can be calibrated by combining the method for automatically detecting the line-of-sight candidate object and the method for designating the line-of-sight candidate object.

  In the above description, the eyeball camera and the imaging camera have been described as being attached to the observer's head. The detection of the optical axis of the observer's eyeball and the imaging of the range of the observer's visual field can be performed using an omnidirectional camera 180 as shown in FIG. FIG. 11 is a plan view showing the positional relationship between the observer 2, its eyeball 4, the bulletin board 6, and the omnidirectional camera 180. The omnidirectional camera 180 has a predetermined imaging range angle, and performs 360-degree rotational scanning in the arrow direction shown in FIG. That is, in the imaging range 182, the head and the eyeball 4 of the observer 2 are imaged, and the visual field direction of the observer 2 and the optical axis direction of the eyeball 4, that is, the calculation are calculated by image processing analysis means (not shown) from the data. Is calculated. Then, the bulletin board 6 in the visual field direction of the observer 2 can be imaged by the imaging range 184 or the like, and this can be used as data of the visual field range of the observer 2. By using the omnidirectional camera 180, the load on the observer 2 can be further reduced.

It is a block diagram of a gaze direction recognition device in an embodiment of the present invention. In embodiment of this invention, it is a figure for description of a visual line candidate target object and calibration of a visual line direction. In embodiment of this invention, it is a figure explaining the line-of-sight candidate target object when the observer is writing on the sheet | seat with a pen. FIG. 4 is a diagram for explaining the line-of-sight candidate object and the line-of-sight direction calibration in the case of FIG. 3. In embodiment of this invention, it is a figure which shows typically a mode that calibration of a gaze direction is performed. In embodiment of this invention, it is a figure which shows typically a mode that calibration of a gaze direction is performed. In embodiment of this invention, it is a figure explaining a mode that a sight line direction is calibrated based on collation with the breadth of distribution of a sight line range, and the magnitude | size of a target object. In embodiment of this invention, it is a figure which shows a mode that a person's face is pinpointed as a visual line candidate target object by collation with the magnitude | size of the area | region of a person's skin color, and the breadth of distribution of an observer's visual line range. In embodiment of this invention, when an observer is in a conference room etc., it is a figure for description of a visual line candidate target object about the image of the range of the field of view. In embodiment of this invention, it is explanatory drawing when there exists a target object used as two eyes | visual_axis candidates. In embodiment of this invention, it is a top view when using an omnidirectional camera.

Explanation of symbols

  2 observers, 4 eyeballs, 6,172 bulletin board, 10 gaze direction recognition device, 12 eyeball camera, 14 imaging camera, 16 calculation gaze direction (eyeball optical axis direction), 18 field of view range, 20 gaze direction recognition computer, 22 CPU, 40 Eyeball direction detection unit, 42 Object detection unit, 44 Field of view imaging module, 46 Eye gaze candidate object identification module, 48 Calibration unit, 60, 80, 90, 91, 100, 101, 160, 170 Range of field of view Screen, 64, 66, 84, 86 arrow indicating movement, range of 92, 102 images, 94, 95, 104, 105, 130, 150, 152, 154 line of sight, 110, 112, 114, 116 people, 120 , 122, 124, 126 cards, 140, 142, 144 Skin color area of people, 166, 176 Position of line of sight, 168 , 178 Near the line of sight, 180 omnidirectional camera.

Claims (15)

In a gaze direction recognition device that detects the optical axis direction of the eyeball of the observer and obtains the direction of the gaze of the observer,
A gaze direction recognition device comprising calibration means for calibrating the gaze direction in a normal observation state. In a gaze direction recognition device that detects the optical axis direction of the eyeball of the observer and obtains the direction of the gaze of the observer,
A position detection means for a line-of-sight candidate object in the range of the field of view of the observer, and an object detection means for detecting the position of the line-of-sight candidate object that is likely to be a line-of-sight candidate from experience;
Eyeball direction detection means for detecting the optical axis direction of the eyeball of the observer;
Calibration means for calibrating with respect to the detected eyeball optical axis direction, with the direction from the observer's eyeball toward the position of the eyesight candidate object as the eyesight direction,
And a gaze direction recognition device that calibrates the gaze direction in a normal observation state. In the gaze direction recognition device according to claim 2,
The line-of-sight direction recognition apparatus characterized in that the line-of-sight candidate object includes an object having a movement correlated with the movement of the eyeball of the observer. In the gaze direction recognition device according to claim 2,
The line-of-sight direction recognition apparatus, wherein the line-of-sight candidate object includes a movement mark in display means that is viewed by an observer. In the gaze direction recognition device according to claim 4,
An eye-gaze direction recognition apparatus characterized in that an eye-gaze candidate object is used together with a movement mark in display means correlated with the movement of the eyeball of the observer and an object that is likely to be watched by the observer. In the gaze direction recognition device according to claim 2,
The line-of-sight direction recognition apparatus, wherein the line-of-sight candidate object includes a pen tip written by an observer. In the gaze direction recognition device according to claim 6,
An eye-gaze direction recognition apparatus characterized in that, as the eye-gaze candidate object, an object that is highly likely to be watched by the observer is used together with a pen tip correlated with the movement of the eyeball of the observer. In the gaze direction recognition device according to claim 2,
The line-of-sight candidate object is detected based on a comparison between the size of the object in the range of the observer's visual field and the distribution of the stay points in the optical axis direction of the observer's eyeball detected by the eyeball direction detection means. An eye-gaze direction recognition device. In the gaze direction recognition device according to claim 2,
The line-of-sight candidate object is used for comparing the size of the human skin color region in the range of the visual field of the observer and the distribution of the stay points in the optical axis direction of the eyeball of the observer detected by the eyeball direction detecting means. A gaze direction recognizing device characterized in that it is a human face detected based on the gaze direction. In the gaze direction recognition device according to claim 2,
Learning means for acquiring, by learning, characteristics of a frequent object in which the gaze direction of the observer may stay frequently after the gaze direction is calibrated with respect to the optical axis direction of the eyeball;
Updating means for updating the position of the line-of-sight candidate object based on the learning result of the frequent object;
A line-of-sight recognition device comprising: In a gaze direction recognition device that detects the optical axis direction of the eyeball of the observer and obtains the direction of the gaze of the observer,
A position detection unit for a line-of-sight candidate object in the range of the field of view of the observer, the object detection unit for specifying the position of the line-of-sight candidate object specified in the range of the field of view of the observer;
Eyeball direction detection means for detecting the optical axis direction of the eyeball of the observer;
Calibration means for calibrating with respect to the detected eyeball optical axis direction, with the direction from the observer's eyeball toward the position of the eyesight candidate object as the eyesight direction,
And a gaze direction recognition device that calibrates the gaze direction in a normal observation state. In the gaze direction recognition device according to claim 11,
The line-of-sight direction recognition apparatus, wherein the specified line-of-sight candidate object is the face of the person closest to the face of the observer. In the gaze direction recognition device according to claim 11,
The line-of-sight direction recognition apparatus, wherein the specified line-of-sight candidate object is display means facing the face of the observer. A gaze direction recognition program that is executed in a gaze direction recognition device that detects the direction of the optical axis of the eyeball of the observer and obtains the direction of the gaze of the observer,
A step of detecting the position of the line-of-sight candidate object in the range of the field of view of the observer, the object detecting step detecting the position of the line-of-sight candidate object that is likely to be a line-of-sight candidate from experience;
An eyeball direction detection step of detecting the optical axis direction of the eyeball of the observer;
A calibration step for calibrating the detected eyeball optical axis direction from the observer's eyeball toward the position of the line-of-sight candidate object as the line-of-sight direction;
Is executed, and the direction of the line of sight is calibrated in the normal observation state. A gaze direction recognition program that is executed in a gaze direction recognition device that detects the direction of the optical axis of the eyeball of the observer and obtains the direction of the gaze of the observer,
A step of identifying a position of the line-of-sight candidate object in the range of the observer's visual field, the object detecting step identifying the position of the line-of-sight candidate object specified in the field of view of the observer;
An eyeball direction detection step of detecting the optical axis direction of the eyeball of the observer;
A calibration step for calibrating the detected eyeball optical axis direction from the observer's eyeball toward the position of the line-of-sight candidate object as the line-of-sight direction;
Is executed, and the direction of the line of sight is calibrated in the normal observation state.

Images