JP2019000135A - Eye gaze measurement device - Google Patents
Eye gaze measurement device Download PDFInfo
- Publication number
- JP2019000135A JP2019000135A JP2017114762A JP2017114762A JP2019000135A JP 2019000135 A JP2019000135 A JP 2019000135A JP 2017114762 A JP2017114762 A JP 2017114762A JP 2017114762 A JP2017114762 A JP 2017114762A JP 2019000135 A JP2019000135 A JP 2019000135A
- Authority
- JP
- Japan
- Prior art keywords
- image
- face
- line
- unit
- center
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Eye Examination Apparatus (AREA)
Abstract
【課題】簡易な構成で、近似計算を行うことなく、精度よく視線計測を行うことができる視線計測装置を提供する。【解決手段】被観察者の顔を撮像する画像撮像部１２と、前記被観察者の目に対して光を照射する照射部１３と、前記撮像手段によって撮像された前記顔を表す顔画像から、前記顔画像上の前記顔の目の角膜反射像と、前記顔画像上の前記顔の目の瞳孔中心位置と、予め定められた３次元眼球モデルとに基づいて、カメラ座標系における３次元の視線ベクトルを計算する視線ベクトル計算部４６と、を含み、画像撮像部と照射部との位置関係、画像撮像部と目との位置関係、及び画像撮像部に関する焦点距離パラメータが、画像撮像部の撮像方向と照射部の光照射方向とが同軸であるとみなすための予め定められた制約条件を満たす。【選択図】図１An eye gaze measuring apparatus is provided that can accurately perform eye gaze measurement with a simple configuration and without performing approximate calculation. An image pickup unit that picks up an image of the face of the person to be observed, an irradiation unit that irradiates light to the eyes of the person to be observed, and a face image that represents the face picked up by the image pickup means. 3D in the camera coordinate system based on a cornea reflection image of the face eye on the face image, a pupil center position of the face eye on the face image, and a predetermined three-dimensional eyeball model A line-of-sight vector calculation unit 46 for calculating a line-of-sight vector of the image, and a positional relationship between the image capturing unit and the irradiation unit, a positional relationship between the image capturing unit and the eye, and a focal length parameter related to the image capturing unit. A predetermined constraint condition for satisfying that the imaging direction and the light irradiation direction of the irradiating unit are considered to be coaxial is satisfied. [Selection] Figure 1
Description
本発明は、視線計測装置に係り、特に、顔を撮像した画像から、視線ベクトルを計測する視線計測装置に関する。 The present invention relates to a line-of-sight measurement apparatus, and more particularly to a line-of-sight measurement apparatus that measures a line-of-sight vector from an image obtained by capturing a face.
従来より、３次元眼球モデルを用いた角膜反射像方式の視線計測装置が知られている（非特許文献１、２）。この角膜反射像方式の視線計測装置では、カメラと光源の位置を離して設置している。 2. Description of the Related Art Conventionally, a corneal reflection image type visual line measurement device using a three-dimensional eyeball model is known (Non-Patent Documents 1 and 2). In this corneal reflection image type visual line measuring device, the camera and the light source are separated from each other.
非特許文献１では、角膜曲率中心座標を求めるために、カメラと光源との間の距離、角膜曲率中心から角膜までの距離、及びカメラと目との間の距離に応じた近似計算を行っている。 In Non-Patent Document 1, in order to obtain a corneal curvature center coordinate, an approximate calculation is performed according to the distance between the camera and the light source, the distance from the corneal curvature center to the cornea, and the distance between the camera and the eyes. Yes.
非特許文献２では、複数の光源を用いて角膜反射像を取得している。 In Non-Patent Document 2, a corneal reflection image is acquired using a plurality of light sources.
上記非特許文献１に記載の技術では、近似計算を行っているため、近似計算による誤差が生じ、視線計測の精度が低下してしまう。 In the technique described in Non-Patent Document 1, since approximate calculation is performed, an error due to approximate calculation occurs, and the accuracy of line-of-sight measurement decreases.
また、上記非特許文献２に記載の技術では、角膜反射像を得るために、複数の光源が必要となるため、装置コストがかかってしまう。 Further, the technique described in Non-Patent Document 2 requires a plurality of light sources in order to obtain a corneal reflection image, resulting in an increase in apparatus cost.
本発明は、上記の問題点を解決するためになされたもので、簡易な構成で、近似計算を行うことなく、精度よく視線計測を行うことができる視線計測装置を提供することを目的とする。 The present invention has been made to solve the above-described problems, and an object of the present invention is to provide a gaze measurement apparatus that can accurately perform gaze measurement without performing approximate calculation with a simple configuration. .
上記の目的を達成するために本発明に係る視線計測装置は、被観察者の顔を撮像する撮像手段と、前記被観察者の目に対して光を照射する光照射手段と、前記撮像手段によって撮像された前記顔を表す顔画像から、前記顔画像上の前記顔の目の角膜反射像と、前記顔画像上の前記顔の目の瞳孔中心位置と、予め定められた３次元眼球モデルとに基づいて、カメラ座標系における３次元の視線ベクトルを計算する視線ベクトル計算手段と、を含み、前記撮像手段と前記光照射手段との位置関係、前記撮像手段と前記目との位置関係、及び前記撮像手段に関するパラメータが、前記撮像手段の撮像方向と前記光照射手段の光照射方向とが同軸であるとみなすための予め定められた制約条件を満たす。 In order to achieve the above object, an eye gaze measurement apparatus according to the present invention includes an imaging unit that images the face of the person to be observed, a light irradiation unit that irradiates light to the eyes of the person to be observed, and the imaging unit. From the face image representing the face imaged by, a corneal reflection image of the face eye on the face image, a pupil center position of the face eye on the face image, and a predetermined three-dimensional eyeball model Line-of-sight vector calculation means for calculating a three-dimensional line-of-sight vector in the camera coordinate system, and a positional relationship between the imaging means and the light irradiation means, a positional relationship between the imaging means and the eyes, And the parameter relating to the imaging unit satisfies a predetermined constraint condition for regarding that the imaging direction of the imaging unit and the light irradiation direction of the light irradiation unit are coaxial.
本発明によれば、前記撮像手段と前記光照射手段との位置関係、前記撮像手段と前記目との位置関係、及び前記撮像手段に関するパラメータが、前記撮像手段の撮像方向と前記光照射手段の光照射方向とが同軸であるとみなすための予め定められた制約条件を満たす。 According to the present invention, the positional relationship between the imaging unit and the light irradiating unit, the positional relationship between the imaging unit and the eye, and the parameters relating to the imaging unit include the imaging direction of the imaging unit and the light irradiating unit. A predetermined constraint condition for assuming that the light irradiation direction is coaxial is satisfied.
また、前記被観察者の目に対して光照射手段から光を照射しているときに、撮像手段によって、前記顔を表す顔画像を撮像する。視線ベクトル計算手段によって、顔画像から、顔画像上の前記顔の目の角膜反射像と、前記顔画像上の前記顔の目の瞳孔中心位置と、予め定められた３次元眼球モデルとに基づいて、カメラ座標系における３次元の視線ベクトルを計算する。 Further, a face image representing the face is picked up by the image pickup means when light is irradiated from the light irradiation means to the eyes of the person to be observed. Based on a corneal reflection image of the eye of the face on the face image, a pupil center position of the eye of the face on the face image, and a predetermined three-dimensional eyeball model from the face image by the gaze vector calculation means. Then, a three-dimensional line-of-sight vector in the camera coordinate system is calculated.
以上説明したように、本発明の視線計測装置によれば、前記撮像手段の撮像方向と前記光照射手段の光照射方向とが同軸であるとみなすための予め定められた制約条件を満たすことにより、簡易な構成で、近似計算を行うことなく、精度よく視線計測を行うことができる、という効果が得られる。 As described above, according to the line-of-sight measurement device of the present invention, by satisfying a predetermined constraint condition for considering that the imaging direction of the imaging unit and the light irradiation direction of the light irradiation unit are coaxial. With the simple configuration, it is possible to obtain an effect that the line-of-sight measurement can be accurately performed without performing an approximate calculation.
以下、図面を参照して本発明の実施の形態を詳細に説明する。なお、本実施の形態では、撮像された顔画像から、視線ベクトルを推定する視線計測装置に本発明を適用した場合を例に説明する。 Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, a case where the present invention is applied to a line-of-sight measurement apparatus that estimates a line-of-sight vector from a captured face image will be described as an example.
＜本発明の実施の形態の概要＞
カメラと光源が同軸であれば、角膜曲率中心座標を求めるのに近似計算が不要で精度が向上する。
<Outline of Embodiment of the Present Invention>
If the camera and the light source are coaxial, approximate calculation is not required to obtain the corneal curvature center coordinates, and the accuracy is improved.
一方、カメラと光源が同軸でない場合、近似計算を行わずに、厳密計算を行うことができるが、その場合の従来技術では、複数の光源を必要としている。本発明の実施の形態では、光源は１箇所とする。 On the other hand, when the camera and the light source are not coaxial, exact calculation can be performed without performing approximate calculation, but the conventional technique in that case requires a plurality of light sources. In the embodiment of the present invention, the number of light sources is one.
カメラと光源が同軸でない場合でも、カメラと光源と撮像対象の位置関係、およびカメラのピクセル単位で表わされる焦点距離ｆがある制約条件を満たせば、実用的には同軸として扱える。 Even if the camera and the light source are not coaxial, if the positional relationship between the camera, the light source and the imaging target, and the focal length f expressed in pixel units of the camera satisfy certain constraints, it can be practically handled as coaxial.
そこで、本発明の実施の形態では、光源と撮像部と撮像対象の位置関係が、同軸とみたすための制約条件を満たす光学系を用いる。 Therefore, in the embodiment of the present invention, an optical system is used in which the positional relationship between the light source, the imaging unit, and the imaging target satisfies a constraint condition for assuming that the light source, the imaging unit, and the imaging target are coaxial.
＜視線計測装置の構成＞
図１に示すように、本発明の実施の形態に係る視線計測装置１０は、対象とする被験者の顔を含む画像を撮像するＣＣＤカメラ等からなる画像撮像部１２と、被験者の目に対して光を照射する照射部１３と、画像処理を行うコンピュータ１４と、ＣＲＴ等で構成された出力部１６とを備えている。
<Configuration of eye gaze measurement device>
As shown in FIG. 1, the line-of-sight measurement device 10 according to the embodiment of the present invention has an image capturing unit 12 including a CCD camera or the like that captures an image including a subject's face as a target, and the subject's eyes. An irradiation unit 13 that irradiates light, a computer 14 that performs image processing, and an output unit 16 that includes a CRT or the like are provided.
画像撮像部１２は、１つのカメラであり、照射部１３は、例えば、１つの近赤外ＬＥＤである。本実施の形態では、画像撮像部１２の撮像方向と照射部１３の照射方向とが同軸ではないが、同軸とみなされるように、以下の（１）式に示す制約条件を満たす配置となっている（図２）。 The image capturing unit 12 is one camera, and the irradiation unit 13 is, for example, one near infrared LED. In the present embodiment, the imaging direction of the image capturing unit 12 and the irradiation direction of the irradiating unit 13 are not coaxial, but are arranged so as to satisfy the constraint condition shown in the following equation (1) so as to be regarded as coaxial. (Fig. 2).
・・・（１）
... (1)
ただし、Ｌは、画像撮像部１２から角膜曲率中心へ向かう直線と角膜との交点と、画像撮像部１２との間の距離であり、ｒは、角膜曲率半径であり、ｆは、画像撮像部１２のピクセル単位の焦点距離である。 However, L is the distance between the intersection of the straight line from the image capturing unit 12 toward the corneal curvature center and the cornea and the image capturing unit 12, r is the corneal curvature radius, and f is the image capturing unit. The focal length is 12 pixels.
上記（１）式の導出は、以下の通りである。 The derivation of the above equation (1) is as follows.
図２において、画像撮像部１２の両側に配置した照明Ｓ1とＳ2による２つの角膜反射像中心点Ｐ1とＰ2の距離がカメラ画像で1pixel以下であれば、角膜反射像中心点Ｐ1と角膜反射像中心点Ｐ2は同じ画素上もしくは隣り合う画素上で観測されることになる。これは、角膜表面に映る２つの照明を区別できないということであり、画像撮像部１２からは２つの照明が同じ場所にあるとみなせるということである。従って２つの照明の間にある画像撮像部１２も照明と同じ場所にあるとみなせることになる。まとめると、画像撮像部１２と照明と角膜曲率中心が同軸上にあるとみなせる条件は、角膜反射像中心点Ｐ1とＰ2の画像座標での距離が1pixel以下になることである。 In FIG. 2, if the distance between two corneal reflection image center points P 1 and P 2 by illuminations S 1 and S 2 arranged on both sides of the image pickup unit 12 is 1 pixel or less in the camera image, the corneal reflection image center point P 1 and the corneal reflection image center point P 2 are observed on the same pixel or on adjacent pixels. This means that the two illuminations reflected on the cornea surface cannot be distinguished, and the image capturing unit 12 can consider that the two illuminations are in the same place. Therefore, the image capturing unit 12 between the two illuminations can be regarded as being in the same place as the illumination. In summary, the condition that the image capturing unit 12, the illumination, and the corneal curvature center can be regarded as coaxial is that the distance between the corneal reflection image center points P 1 and P 2 at the image coordinates is 1 pixel or less.
ＣはＳ1Ｓ2の中間にあるものとし、距離ＣＳ1とＣＳ2をそれぞれｘとする。角膜表面を球と仮定したので、Ｐ1における垂線は角膜曲率中心を通るとともに、角ＣＰ1Ｓ1の２等分線になる。従って、以下の式(２)が成立する。 C is assumed to be in the middle of S 1 S 2 , and distances CS 1 and CS 2 are each x. Since the corneal surface is assumed to be a sphere, the perpendicular at P 1 passes through the center of corneal curvature and becomes a bisector of the angle CP 1 S 1 . Therefore, the following formula (2) is established.
・・・(２)
... (2)
角ＣＰ2Ｓ2においても同様のことが言えるため、以下の式(３)を得る。 Since the same can be said for the angle CP 2 S 2 , the following expression (3) is obtained.
・・・(３)
... (3)
三角形Ｍ1ＡＭ2と三角形Ｐ1ＡＰ2の相似条件から、以下の式(５)が得られる。 From the similarity condition of the triangle M 1 AM 2 and the triangle P 1 AP 2 , the following equation (5) is obtained.
・・・(５)
... (5)
角膜反射像Ｐ1の三次元ベクトルを
、画像座標を
とする。角膜反射像Ｐ2についても同様に定義する。ピクセル単位で表される焦点距離をｆとすると、透視投影における
の関係式、および
の関係式(６)が得られる。
The three-dimensional vector of the cornea reflection image P 1
, The image coordinates
And The corneal reflection image P 2 is defined similarly. Let f be the focal length expressed in pixels.
The relational expression, and
The following relational expression (6) is obtained.
・・・(６)
... (6)
なので、角膜反射像中心点Ｐ1とＰ2の画像座標での距離は以下の式(７)で表される。
Therefore, the distance at the image coordinates of the cornea reflection image center points P 1 and P 2 is expressed by the following equation (7).
・・・(７)
... (7)
式(５)を式(７)に代入し、以下の式(８)を得る。 Substituting equation (5) into equation (7), the following equation (8) is obtained.
・・・(８)
... (8)
最後に、角膜反射像中心点Ｐ1とＰ2の画像座標での距離が1pixel以下なので以下の式(１０)を得る。 Finally, since the distance at the image coordinates of the cornea reflection image center points P 1 and P 2 is 1 pixel or less, the following expression (10) is obtained.
・・・(９)
・・・(１０)
... (9)
... (10)
コンピュータ１４は、ＣＰＵ、後述する視線計測処理ルーチンのプログラムを記憶したＲＯＭ、データ等を記憶するＲＡＭ、及びこれらを接続するバスを含んで構成されている。このコンピュータ１４をハードウエアとソフトウエアとに基づいて定まる機能実現手段毎に分割した機能ブロックで説明すると、図１に示すように、コンピュータ１４は、画像撮像部１２から出力される濃淡画像である顔画像を入力する画像入力部２０と、画像入力部２０の出力である顔画像の時系列から、角膜反射法（図３参照）を用いて、カメラ座標系における視線ベクトルを計算する角膜反射法視線検出部２８とを備えている。なお、角膜反射法視線検出部２８が、視線ベクトル計算手段の一例である。 The computer 14 includes a CPU, a ROM that stores a program for a visual line measurement processing routine that will be described later, a RAM that stores data and the like, and a bus that connects these. If the computer 14 is described with functional blocks divided for each function realizing means determined based on hardware and software, the computer 14 is a grayscale image output from the image capturing unit 12 as shown in FIG. A cornea reflection method for calculating a line-of-sight vector in the camera coordinate system from the image input unit 20 for inputting a face image and a time series of the face image as an output of the image input unit 20 using the cornea reflection method (see FIG. 3). And a line-of-sight detection unit 28. The corneal reflection gaze detection unit 28 is an example of a gaze vector calculation unit.
画像入力部２０は、例えば、Ａ／Ｄコンバータや１画面の画像データを記憶する画像メモリ等で構成される。 The image input unit 20 includes, for example, an A / D converter and an image memory that stores image data for one screen.
角膜反射法視線検出部２８は、図１に示すように、眼球モデル記憶部４０と、角膜反射像位置推定部４２と、角膜曲率中心計算部４４と、視線ベクトル計算部４６とを備えている。 As shown in FIG. 1, the corneal reflection gaze detection unit 28 includes an eyeball model storage unit 40, a corneal reflection image position estimation unit 42, a corneal curvature center calculation unit 44, and a gaze vector calculation unit 46. .
眼球モデル記憶部４０には、顔形状モデル座標系における眼球中心座標Ｅと、角膜曲率に応じた球体及び眼球を表す球体の位置関係及びサイズと、瞳孔中心座標と角膜曲率中心座標との距離ｓ、角膜曲率中心座標と眼球中心座標との距離ｔ、角膜反射像中心座標と角膜曲率中心座標との距離ｒとが記憶されている（図４参照）。 The eyeball model storage unit 40 stores the eyeball center coordinate E in the face shape model coordinate system, the positional relationship and size of the sphere and the sphere representing the eyeball according to the corneal curvature, and the distance s between the pupil center coordinate and the corneal curvature center coordinate. The distance t between the corneal curvature center coordinates and the eyeball center coordinates and the distance r between the corneal reflection image center coordinates and the corneal curvature center coordinates are stored (see FIG. 4).
角膜反射像位置推定部４２は、顔画像から、顔画像上の角膜反射像中心の２次元座標を計算し、顔画像上の角膜反射像中心の２次元座標と、顔形状モデル座標系における眼球中心座標Ｅとから、カメラ位置Ｃから角膜反射像中心Ｐへ向かう３次元ベクトルｐを推定する。 The cornea reflection image position estimation unit 42 calculates the two-dimensional coordinates of the center of the cornea reflection image on the face image from the face image, and the two-dimensional coordinates of the center of the cornea reflection image on the face image and the eyeball in the face shape model coordinate system. From the center coordinate E, a three-dimensional vector p from the camera position C to the corneal reflection image center P is estimated.
角膜曲率中心計算部４４は、３次元ベクトルｐと、角膜反射像中心Ｐと角膜曲率中心Ａとの距離ｒとを用いて、以下の式に従って、カメラ位置Ｃから角膜曲率中心Ａへ向かう３次元ベクトルａを推定する。 The corneal curvature center calculation unit 44 uses the three-dimensional vector p and the distance r between the corneal reflection image center P and the corneal curvature center A to perform a three-dimensional operation from the camera position C to the corneal curvature center A according to the following equation. Estimate the vector a.
視線ベクトル計算部４６は、顔画像から、顔画像上の瞳孔中心（見かけの瞳孔中心）Ｂの２次元座標を計算し、顔画像上の瞳孔中心（見かけの瞳孔中心）Ｂの２次元座標と、３次元ベクトルａと、角膜反射像中心Ｐと角膜曲率中心Ａとの距離ｒと用いて、カメラ位置Ｃから見かけの瞳孔中心Ｂへ向かう３次元ベクトルｂを求める。ここで、図５に示すように、見かけの瞳孔中心Ｂの候補は２か所存在するため、Zがカメラ側になる方(=小さい方)を、見かけの瞳孔中心Ｂとして選べばよい。 The line-of-sight vector calculation unit 46 calculates the two-dimensional coordinates of the pupil center (apparent pupil center) B on the face image from the face image, and the two-dimensional coordinates of the pupil center (apparent pupil center) B on the face image. Using the three-dimensional vector a and the distance r between the corneal reflection image center P and the corneal curvature center A, a three-dimensional vector b from the camera position C to the apparent pupil center B is obtained. Here, as shown in FIG. 5, there are two candidates for the apparent pupil center B. Therefore, the direction where Z is closer to the camera side (= the smaller one) may be selected as the apparent pupil center B.
そして、視線ベクトル計算部４６は、３次元ベクトルｂと、３次元ベクトルｐとを用いて、角膜反射像中心Ｐと見かけの瞳孔中心Ｂとの距離ｕを計算する。視線ベクトル計算部４６は、３次元ベクトルｐと、３次元ベクトルｂとを用いて、以下の式に従って、３次元ベクトルｐと、３次元ベクトルｂとの角度εを計算する。 The line-of-sight vector calculation unit 46 calculates the distance u between the corneal reflection image center P and the apparent pupil center B using the three-dimensional vector b and the three-dimensional vector p. The line-of-sight vector calculation unit 46 uses the three-dimensional vector p and the three-dimensional vector b to calculate the angle ε between the three-dimensional vector p and the three-dimensional vector b according to the following equation.
視線ベクトル計算部４６は、角膜反射像中心Ｐと角膜曲率中心Ａとの距離ｒと、角膜反射像中心Ｐと見かけの瞳孔中心Ｂとの距離ｕ（＝||ｂ−ｐ||）とを用いて、以下の式に従って、３次元ベクトルｐと、見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルとの角度θを計算する。 The line-of-sight vector calculation unit 46 calculates the distance r between the corneal reflection image center P and the corneal curvature center A and the distance u (= || b−p ||) between the corneal reflection image center P and the apparent pupil center B. The angle θ between the three-dimensional vector p and the three-dimensional vector from the apparent pupil center B toward the corneal curvature center A is calculated according to the following equation.
視線ベクトル計算部４６は、３次元ベクトルｐと３次元ベクトルｂとの角度εの計算結果と、３次元ベクトルｐと見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルとの角度θの計算結果とを用いて、以下の式に従って、見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルと、真の瞳孔中心Ｂ´から眼球中心Ｅへ向かう３次元ベクトルとの角度ψを計算する。以下の式で、ｒは角膜曲率半径、ｓはＡＢ’間の距離、ｎ1は、空気の屈折率、ｎ2は角膜の内側にある水晶体の屈折率である。 The line-of-sight vector calculation unit 46 calculates the angle θ between the calculation result of the angle ε between the three-dimensional vector p and the three-dimensional vector b and the three-dimensional vector p and the three-dimensional vector from the apparent pupil center B toward the corneal curvature center A. Using the calculation results, an angle ψ between a three-dimensional vector from the apparent pupil center B to the corneal curvature center A and a three-dimensional vector from the true pupil center B ′ to the eyeball center E is calculated according to the following formula. To do. In the following formula, r is the radius of curvature of the cornea, s is the distance between AB ′, n 1 is the refractive index of air, and n 2 is the refractive index of the crystalline lens inside the cornea.
視線ベクトル計算部４６は、３次元ベクトルｐと見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルとの角度θと、見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルと真の瞳孔中心Ｂ´から眼球中心Ｅへ向かう３次元ベクトルとの角度ψとの和により、視線ベクトルの角度（＝θ＋ψ）を計算する。 The line-of-sight vector calculation unit 46 calculates the angle θ between the three-dimensional vector p and the three-dimensional vector from the apparent pupil center B toward the corneal curvature center A, and the true three-dimensional vector from the apparent pupil center B toward the corneal curvature center A. The angle of the line-of-sight vector (= θ + ψ) is calculated from the sum of the angle ψ with the three-dimensional vector from the pupil center B ′ toward the eyeball center E.
視線ベクトル計算部４６は、視線ベクトルの角度（＝θ＋ψ）と、カメラ位置Ｃから角膜反射像中心Ｐへ向かう３次元ベクトルｐと、カメラ位置Ｃから見かけの瞳孔中心Ｂへ向かう３次元ベクトルｂとに基づいて、視線ベクトルｄを求めて、出力部１６により出力する。ただし、-(θ＋ψ)でも式が成立するため、視線ベクトルｄのｙ成分が３次元ベクトルｐのｙ成分よりも大きくなる方を選ぶか、ｒｄがＢに近い方を選べばよい。 The line-of-sight vector calculation unit 46 includes a line-of-sight vector angle (= θ + ψ), a three-dimensional vector p from the camera position C to the corneal reflection image center P, and a three-dimensional vector b from the camera position C to the apparent pupil center B. Based on the above, the line-of-sight vector d is obtained and output by the output unit 16. However, since − (θ + ψ) also holds the equation, it may be selected that the y component of the line-of-sight vector d is larger than the y component of the three-dimensional vector p or that rd is close to B.
＜視線計測装置の動作＞
次に、視線計測装置１０の動作について説明する。まず、照射部１３により近赤外の光を被験者の目に照射しているときに、画像撮像部１２で被験者の顔画像を連続的に撮像する。
<Operation of eye gaze measurement device>
Next, the operation of the line-of-sight measurement device 10 will be described. First, when the near-infrared light is irradiated to the subject's eyes by the irradiation unit 13, the face image of the subject is continuously captured by the image capturing unit 12.
そして、コンピュータ１４において、撮像された顔画像毎に、図６に示す視線計測処理ルーチンを実行する。まず、ステップＳ１１８において、画像撮像部１２で撮像された顔画像を取得する。 Then, the computer 14 executes the line-of-sight measurement processing routine shown in FIG. 6 for each captured face image. First, in step S118, the face image captured by the image capturing unit 12 is acquired.
そして、ステップＳ１２０において、顔画像から、顔画像上の角膜反射像中心の２次元座標を計算し、顔画像上の角膜反射像中心の２次元座標と、顔形状モデル座標系における仮の眼球中心座標Ｅとから、カメラ位置Ｃから角膜反射像中心Ｐへ向かう３次元ベクトルｐを推定する。 In step S120, the two-dimensional coordinates of the corneal reflection image center on the face image are calculated from the face image, and the two-dimensional coordinates of the corneal reflection image center on the face image and the temporary eyeball center in the face shape model coordinate system are calculated. From the coordinate E, a three-dimensional vector p from the camera position C to the corneal reflection image center P is estimated.
ステップＳ１２２では、上記ステップＳ１２０で推定された３次元ベクトルｐと、角膜反射像中心Ｐと角膜曲率中心Ａとの距離ｒとを用いて、カメラ位置Ｃから角膜曲率中心Ａへ向かう３次元ベクトルａを推定する。 In step S122, using the three-dimensional vector p estimated in step S120 and the distance r between the corneal reflection image center P and the corneal curvature center A, the three-dimensional vector a heading from the camera position C to the corneal curvature center A. Is estimated.
ステップＳ１２４では、顔画像から、顔画像上の瞳孔中心（見かけの瞳孔中心）Ｂの２次元座標を計算する。そして、ステップＳ１２６では、顔画像上の瞳孔中心（見かけの瞳孔中心）Ｂの２次元座標と、上記ステップＳ１２２で推定された３次元ベクトルａと、角膜反射像中心Ｐと眼球中心Ｅとの距離ｒと用いて、カメラ位置Ｃから見かけの瞳孔中心Ｂへ向かう３次元ベクトルｂを求める。 In step S124, the two-dimensional coordinates of the pupil center (apparent pupil center) B on the face image are calculated from the face image. In step S126, the two-dimensional coordinates of the pupil center (apparent pupil center) B on the face image, the three-dimensional vector a estimated in step S122, and the distance between the corneal reflection image center P and the eyeball center E are calculated. Using r, a three-dimensional vector b from the camera position C to the apparent pupil center B is obtained.
そして、ステップＳ１２８では、上記ステップＳ１２４で求めた３次元ベクトルｂと、上記ステップＳ１２０で推定された３次元ベクトルｐとを用いて、角膜反射像中心Ｐと見かけの瞳孔中心Ｂとの距離ｕを計算する。また、３次元ベクトルｐと、３次元ベクトルｂとを用いて、３次元ベクトルｐと、３次元ベクトルｂとの角度εを計算する。また、角膜反射像中心Ｐと角膜曲率中心Ａとの距離ｒと、角膜反射像中心Ｐと見かけの瞳孔中心Ｂとの距離ｕ（＝||ｂ−ｐ||）とを用いて、３次元ベクトルｐと、見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルとの角度θを計算する。また、３次元ベクトルｐと３次元ベクトルｂとの角度εの計算結果と、３次元ベクトルｐと見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルとの角度θの計算結果とを用いて、見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルと、真の瞳孔中心Ｂ´から眼球中心Ｅへ向かう３次元ベクトルとの角度ψを計算する。 In step S128, the distance u between the corneal reflection image center P and the apparent pupil center B is calculated using the three-dimensional vector b obtained in step S124 and the three-dimensional vector p estimated in step S120. calculate. Further, the angle ε between the three-dimensional vector p and the three-dimensional vector b is calculated using the three-dimensional vector p and the three-dimensional vector b. In addition, the distance r between the corneal reflection image center P and the corneal curvature center A and the distance u (= || b−p ||) between the corneal reflection image center P and the apparent pupil center B are three-dimensional. The angle θ between the vector p and the three-dimensional vector from the apparent pupil center B toward the corneal curvature center A is calculated. Also, the calculation result of the angle ε between the three-dimensional vector p and the three-dimensional vector b and the calculation result of the angle θ between the three-dimensional vector p and the three-dimensional vector from the apparent pupil center B to the corneal curvature center A are used. Then, the angle ψ between the three-dimensional vector from the apparent pupil center B to the corneal curvature center A and the three-dimensional vector from the true pupil center B ′ to the eyeball center E is calculated.
そして、上記で計算された、３次元ベクトルｐと見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルとの角度θと、見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルと真の瞳孔中心Ｂ´から眼球中心Ｅへ向かう３次元ベクトルとの角度ψとの和により、視線ベクトルの角度を計算する。 The angle θ between the three-dimensional vector p calculated above and the three-dimensional vector from the apparent pupil center B toward the corneal curvature center A, and the three-dimensional vector from the apparent pupil center B toward the corneal curvature center A The angle of the line-of-sight vector is calculated based on the sum of the angle ψ with the three-dimensional vector from the true pupil center B ′ toward the eyeball center E.
そして、ステップＳ１３０では、上記ステップＳ１２８で計算された、視線ベクトルの角度と、カメラ位置Ｃから角膜反射像中心Ｐへ向かう３次元ベクトルｐと、カメラ位置Ｃから見かけの瞳孔中心Ｂへ向かう３次元ベクトルｂとに基づいて、視線ベクトルｄを求め、出力部１６により出力して、視線計測処理ルーチンを終了する。 In step S130, the line-of-sight vector angle calculated in step S128, the three-dimensional vector p from the camera position C to the corneal reflection image center P, and the three-dimensional direction from the camera position C to the apparent pupil center B are calculated. Based on the vector b, the line-of-sight vector d is obtained and output by the output unit 16, and the line-of-sight measurement processing routine is terminated.
以上説明したように、本発明の実施の形態に係る視線計測装置によれば、画像撮像部の撮像方向と照射部の光照射方向とが同軸であるとみなすための予め定められた制約条件を満たすように配置し、角膜反射像を用いて、視線ベクトルを計算することにより、画像撮像部の撮像方向と照射部の光照射方向とが同軸でない場合であっても、簡易な構成で、近似計算を行うことなく、精度よく視線計測を行うことができる。 As described above, according to the line-of-sight measurement apparatus according to the embodiment of the present invention, the predetermined constraint condition for regarding the imaging direction of the image imaging unit and the light irradiation direction of the irradiation unit as being coaxial is determined. Even if the image capturing direction of the image capturing unit and the light irradiation direction of the irradiating unit are not coaxial by calculating the line-of-sight vector using the corneal reflection image, it is approximated with a simple configuration. Gaze measurement can be performed with high accuracy without performing calculations.
また、画像撮像部と照射部とが同軸として扱えるため、角膜曲率中心推定計算に近似計算が含まれず、角膜曲率中心の推定精度が向上すると共に、角膜曲率中心推定の計算量を削減することができる。また、１つの光源で視線計算が可能になり装置コストが下げられる。 In addition, since the image capturing unit and the irradiation unit can be treated as coaxial, approximate calculation is not included in the corneal curvature center estimation calculation, the estimation accuracy of the corneal curvature center is improved, and the calculation amount of the corneal curvature center estimation can be reduced. it can. Further, the line of sight can be calculated with one light source, and the apparatus cost can be reduced.
１０ 視線計測装置
１２ 画像撮像部
１３ 照射部
１４ コンピュータ
１６ 出力部
２０ 画像入力部
２８ 角膜反射法視線検出部
４０ 眼球モデル記憶部
４２ 角膜反射像位置推定部
４４ 角膜曲率中心計算部
４６ 視線ベクトル計算部
DESCRIPTION OF SYMBOLS 10 Eye gaze measuring device 12 Image pick-up part 13 Irradiation part 14 Computer 16 Output part 20 Image input part 28 Corneal reflection method gaze detection part 40 Eyeball model memory | storage part 42 Corneal reflection image position estimation part 44 Corneal curvature center calculation part 46 Gaze vector calculation part
Claims (3)
前記被観察者の目に対して光を照射する光照射手段と、
前記撮像手段によって撮像された前記顔を表す顔画像から、前記顔画像上の前記顔の目の角膜反射像と、前記顔画像上の前記顔の目の瞳孔中心位置と、予め定められた３次元眼球モデルとに基づいて、カメラ座標系における３次元の視線ベクトルを計算する視線ベクトル計算手段と、
を含み、
前記撮像手段と前記光照射手段との位置関係、前記撮像手段と前記目との位置関係、及び前記撮像手段に関するパラメータが、前記撮像手段の撮像方向と前記光照射手段の光照射方向とが同軸であるとみなすための予め定められた制約条件を満たす
視線計測装置。 Imaging means for imaging the face of the subject;
A light irradiating means for irradiating light to the eyes of the person to be observed;
From a face image representing the face imaged by the imaging means, a corneal reflection image of the face eye on the face image, a pupil center position of the face eye on the face image, and a predetermined 3 Line-of-sight vector calculation means for calculating a three-dimensional line-of-sight vector in the camera coordinate system based on the three-dimensional eyeball model;
Including
The positional relationship between the imaging unit and the light irradiation unit, the positional relationship between the imaging unit and the eye, and the parameters relating to the imaging unit are such that the imaging direction of the imaging unit and the light irradiation direction of the light irradiation unit are coaxial. A line-of-sight measurement device that satisfies a predetermined constraint for being considered to be.
ただし、Ｌは、前記撮像手段から角膜曲率中心へ向かう直線と角膜との交点と、前記撮像手段との間の距離、ｒは、角膜曲率半径、ｆは、前記撮像手段のピクセル単位の焦点距離である。 The line-of-sight measurement apparatus according to claim 1, wherein the constraint condition is expressed by the following expression.
Where L is the distance between the intersection of the straight line from the imaging means to the corneal curvature center and the cornea and the imaging means, r is the corneal curvature radius, and f is the focal length in pixel units of the imaging means. It is.
前記顔の目の角膜反射像と、前記３次元眼球モデルとに基づいて、前記角膜反射像の３次元位置を推定する角膜反射像位置推定手段と、
前記角膜反射像の３次元位置に基づいて、角膜曲率中心の３次元位置を計算する角膜曲率中心計算手段と、を含み、
前記角膜反射像の３次元位置と、前記角膜曲率中心の３次元位置とに基づいて、前記カメラ座標系における３次元の視線ベクトルを計算する請求項１又は２記載の視線計測装置。 The line-of-sight vector calculating means includes:
Corneal reflection image position estimation means for estimating a three-dimensional position of the corneal reflection image based on the corneal reflection image of the face eye and the three-dimensional eyeball model;
Corneal curvature center calculating means for calculating the three-dimensional position of the corneal curvature center based on the three-dimensional position of the corneal reflection image,
The line-of-sight measurement device according to claim 1, wherein a three-dimensional line-of-sight vector in the camera coordinate system is calculated based on a three-dimensional position of the corneal reflection image and a three-dimensional position of the corneal curvature center.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2017114762A JP6785723B2 (en) | 2017-06-09 | 2017-06-09 | Line-of-sight measuring device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2017114762A JP6785723B2 (en) | 2017-06-09 | 2017-06-09 | Line-of-sight measuring device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| JP2019000135A true JP2019000135A (en) | 2019-01-10 |
| JP6785723B2 JP6785723B2 (en) | 2020-11-18 |
Family
ID=65005490
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| JP2017114762A Active JP6785723B2 (en) | 2017-06-09 | 2017-06-09 | Line-of-sight measuring device |
Country Status (1)
| Country | Link |
|---|---|
| JP (1) | JP6785723B2 (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2020162759A (en) * | 2019-03-28 | 2020-10-08 | 株式会社豊田中央研究所 | Line-of-sight measuring device |
| JP2021049130A (en) * | 2019-09-25 | 2021-04-01 | 株式会社豊田中央研究所 | Eyeball structure estimation device |
| CN113662506A (en) * | 2021-09-26 | 2021-11-19 | 温州医科大学 | Method, device, medium and electronic device for measuring corneal surface morphology |
| JP2022035177A (en) * | 2020-08-20 | 2022-03-04 | 株式会社Ｊｖｃケンウッド | Distance calculation device, distance calculation method, and distance calculation program |
| CN120167884A (en) * | 2023-12-20 | 2025-06-20 | 北京字跳网络技术有限公司 | Method for determining corneal center and pupil distance and related equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH09505756A (en) * | 1994-07-26 | 1997-06-10 | ルナー コーポレイション | Radiography gantry with collision avoidance software |
| JP2006051403A (en) * | 2005-10-31 | 2006-02-23 | Hitachi Medical Corp | Fluoroscopic and radiographic apparatus |
| US20080049185A1 (en) * | 2006-08-22 | 2008-02-28 | Southwest Research Institute | Eye location and gaze detection system and method |
| JP2010259605A (en) * | 2009-05-01 | 2010-11-18 | Nippon Hoso Kyokai <Nhk> | Gaze measurement apparatus and gaze measurement program |
| JP2011217904A (en) * | 2010-04-08 | 2011-11-04 | Toshiba Corp | X-ray diagnostic apparatus |
| JP2016051318A (en) * | 2014-08-29 | 2016-04-11 | アルプス電気株式会社 | Illumination imaging apparatus |
-
2017
- 2017-06-09 JP JP2017114762A patent/JP6785723B2/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH09505756A (en) * | 1994-07-26 | 1997-06-10 | ルナー コーポレイション | Radiography gantry with collision avoidance software |
| JP2006051403A (en) * | 2005-10-31 | 2006-02-23 | Hitachi Medical Corp | Fluoroscopic and radiographic apparatus |
| US20080049185A1 (en) * | 2006-08-22 | 2008-02-28 | Southwest Research Institute | Eye location and gaze detection system and method |
| JP2010259605A (en) * | 2009-05-01 | 2010-11-18 | Nippon Hoso Kyokai <Nhk> | Gaze measurement apparatus and gaze measurement program |
| JP2011217904A (en) * | 2010-04-08 | 2011-11-04 | Toshiba Corp | X-ray diagnostic apparatus |
| JP2016051318A (en) * | 2014-08-29 | 2016-04-11 | アルプス電気株式会社 | Illumination imaging apparatus |
| CN106137117A (en) * | 2014-08-29 | 2016-11-23 | 阿尔卑斯电气株式会社 | Lighting camera device |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2020162759A (en) * | 2019-03-28 | 2020-10-08 | 株式会社豊田中央研究所 | Line-of-sight measuring device |
| EP3725211A3 (en) * | 2019-03-28 | 2020-12-30 | Aisin Seiki Kabushiki Kaisha | Line-of-sight measurement device |
| US11571125B2 (en) | 2019-03-28 | 2023-02-07 | Aisin Corporation | Line-of-sight measurement device |
| JP7283841B2 (en) | 2019-03-28 | 2023-05-30 | 株式会社豊田中央研究所 | Gaze measurement device |
| JP2021049130A (en) * | 2019-09-25 | 2021-04-01 | 株式会社豊田中央研究所 | Eyeball structure estimation device |
| JP7255436B2 (en) | 2019-09-25 | 2023-04-11 | 株式会社豊田中央研究所 | Eyeball structure estimation device |
| JP2022035177A (en) * | 2020-08-20 | 2022-03-04 | 株式会社Ｊｖｃケンウッド | Distance calculation device, distance calculation method, and distance calculation program |
| JP7593008B2 (en) | 2020-08-20 | 2024-12-03 | 株式会社Ｊｖｃケンウッド | Distance calculation device, distance calculation method, and distance calculation program |
| CN113662506A (en) * | 2021-09-26 | 2021-11-19 | 温州医科大学 | Method, device, medium and electronic device for measuring corneal surface morphology |
| CN113662506B (en) * | 2021-09-26 | 2023-08-04 | 温州医科大学 | Method, device, medium and electronic equipment for measuring corneal surface morphology |
| CN120167884A (en) * | 2023-12-20 | 2025-06-20 | 北京字跳网络技术有限公司 | Method for determining corneal center and pupil distance and related equipment |
| WO2025130787A1 (en) * | 2023-12-20 | 2025-06-26 | 北京字跳网络技术有限公司 | Method for determining cornea center and interpupillary distance and related device |
Also Published As
| Publication number | Publication date |
|---|---|
| JP6785723B2 (en) | 2020-11-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6800091B2 (en) | Line-of-sight measuring device and program | |
| US11068056B2 (en) | Wearable eye tracking system with slippage detection and correction | |
| US10878237B2 (en) | Systems and methods for performing eye gaze tracking | |
| US10902635B2 (en) | Line-of-sight detection device | |
| JP6785723B2 (en) | Line-of-sight measuring device | |
| TWI624170B (en) | Image scanning system and method thereof | |
| JP7283841B2 (en) | Gaze measurement device | |
| JP6677522B2 (en) | Information processing apparatus, control method for information processing apparatus, and program | |
| JP2016173313A (en) | Visual line direction estimation system, visual line direction estimation method and visual line direction estimation program | |
| US11054659B2 (en) | Head mounted display apparatus and distance measurement device thereof | |
| JP2015090560A (en) | Image processing apparatus, and image processing method | |
| JP2014052758A (en) | Sight line measurement method | |
| US20170278269A1 (en) | Corneal reflection position estimation system, corneal reflection position estimation method, corneal reflection position estimation program, pupil detection system, pupil detection method, pupil detection program, gaze detection system, gaze detection method, gaze detection program, face orientation detection system, face orientation detection method, and face orientation detection program | |
| JP6288770B2 (en) | Face detection method, face detection system, and face detection program | |
| JP5515018B2 (en) | Environment recognition device | |
| JP7255436B2 (en) | Eyeball structure estimation device | |
| JP6452235B2 (en) | Face detection method, face detection device, and face detection program | |
| US10386930B2 (en) | Depth determining method and depth determining device of operating body | |
| JP2015111128A (en) | Position attitude measurement device, position attitude measurement method, and program | |
| JP6468755B2 (en) | Feature point detection system, feature point detection method, and feature point detection program | |
| WO2016159255A1 (en) | Mouth region detection device and mouth region detection method | |
| KR101348903B1 (en) | Cornea radius estimation apparatus using cornea radius estimation algorithm based on geometrical optics for eye tracking and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| A621 | Written request for application examination |
Free format text: JAPANESE INTERMEDIATE CODE: A621 Effective date: 20190403 |
|
| A977 | Report on retrieval |
Free format text: JAPANESE INTERMEDIATE CODE: A971007 Effective date: 20200226 |
|
| A131 | Notification of reasons for refusal |
Free format text: JAPANESE INTERMEDIATE CODE: A131 Effective date: 20200310 |
|
| A521 | Request for written amendment filed |
Free format text: JAPANESE INTERMEDIATE CODE: A523 Effective date: 20200501 |
|
| TRDD | Decision of grant or rejection written | ||
| A01 | Written decision to grant a patent or to grant a registration (utility model) |
Free format text: JAPANESE INTERMEDIATE CODE: A01 Effective date: 20200929 |
|
| A61 | First payment of annual fees (during grant procedure) |
Free format text: JAPANESE INTERMEDIATE CODE: A61 Effective date: 20201027 |
|
| R150 | Certificate of patent or registration of utility model |
Ref document number: 6785723 Country of ref document: JP Free format text: JAPANESE INTERMEDIATE CODE: R150 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |