JP2014149794A - Line of sight analyzer - Google Patents

Abstract

[PROBLEMS] In the conventional gaze analysis, even if the hand moves greatly, if the gaze movement combining head movement and eye movement is almost stationary, the part of the gaze before the hand moves There is a possibility of misrecognizing that you keep watching.
Disclosed is a gaze analyzing apparatus for analyzing a gaze of a subject. The line-of-sight analysis apparatus includes a head movement calculation unit, an eye movement calculation unit, a line-of-sight direction calculation unit, a target movement calculation unit, and a viewpoint calculation unit. The head movement calculation unit calculates the rotation angle of the head due to the rotation of the head, the eye movement calculation unit calculates the rotation angle of the eyeball due to the movement of the eye, and the gaze direction calculation unit calculates the calculated rotation movement angle. The movement angle in the gaze direction of the subject is calculated based on the calculated rotation angle, the target motion calculation unit calculates a movement angle equivalent to the gaze movement due to the motion of the target object, and the viewpoint calculation unit is calculated by the gaze direction calculation unit The movement of the viewpoint based on the line of sight of the subject is calculated based on the calculated movement angle and the movement angle calculated by the target motion calculation unit.
[Selection] Figure 1

Description

  The present invention relates to a line-of-sight analysis apparatus and the like. In particular, the present invention relates to an apparatus for analyzing a line of sight in consideration of movement of a subject's head, movement of an eyeball, and movement of an object to be viewed.

  Conventionally, when evaluating the movement of the line of sight, the movement of the line of sight that combines the eye movement and the head movement has been analyzed using an eye movement measurement device and a head movement measurement device (see, for example, Patent Document 1). .

  On the other hand, with the widespread use of mobile terminals, the proportion of reading behavior that reads characters displayed on the display screen of mobile terminals has increased.

Japanese Examined Patent Publication No. 4-46570

  Usually, when visually recognizing the display screen of the mobile terminal, the mobile terminal is held by hand. For this reason, it is considered that hand movement is also involved in reading behavior such as characters displayed on the display screen of the mobile terminal. However, no studies have been conducted that take such hand movements into account. Therefore, even if the hand holding the mobile device moves greatly, if the movement of the line of sight combining the head movement and eye movement is almost stationary, the part that was gazing before the hand moved There is a possibility of misrecognizing that he keeps watching.

  As one embodiment of the present invention, a line-of-sight analysis device for analyzing the line of sight of a subject who turns his line of sight toward an object is disclosed. The line-of-sight analysis device includes a head movement calculation unit, an eye movement calculation unit, a line-of-sight direction calculation unit, a target movement calculation unit, and a viewpoint calculation unit. The head movement calculation unit calculates the rotational movement angle of the head due to the rotational movement of the head. The head movement calculation unit may calculate the rotational movement angle of the head due to the movement movement of the head. The eye movement calculation unit calculates the rotation angle of the eyeball due to the eye movement. The line-of-sight direction calculation unit calculates a movement angle in the direction of the line of sight of the subject based on the rotational movement angle calculated by the head movement calculation unit and the rotation angle calculated by the eye movement calculation unit. The target motion calculation unit calculates a movement angle corresponding to the movement of the line of sight when observed from the subject based on the movement of the target object. The viewpoint calculation unit calculates the movement of the viewpoint by the line of sight toward the object by the subject based on the movement angle calculated by the line-of-sight direction calculation unit and the movement angle calculated by the target motion calculation unit.

  As one embodiment of the present invention, a line-of-sight analysis method by a line-of-sight analysis apparatus that analyzes the line of sight of a subject who turns his line of sight toward an object is disclosed. The line-of-sight analysis apparatus includes a head movement calculation unit, an eye movement calculation unit, a line-of-sight direction calculation unit, and a target movement calculation unit. The head movement calculation unit calculates the rotational movement angle of the head due to the rotational movement of the head. The head movement calculation unit may calculate the rotational movement angle of the head due to the movement movement of the head. The eye movement calculation unit calculates the rotation angle of the eyeball due to the eye movement. The line-of-sight direction calculation unit calculates a movement angle in the direction of the line of sight of the subject based on the rotational movement angle calculated by the head movement calculation unit and the rotation angle calculated by the eye movement calculation unit. The target motion calculation unit calculates a movement angle corresponding to the movement of the line of sight when observed from the subject based on the movement of the target object. The line-of-sight analysis method includes calculating the movement of the viewpoint by the line of sight toward the object by the subject based on the movement angle calculated by the line-of-sight direction calculation unit and the movement angle calculated by the target motion calculation unit.

  According to the present invention, since the movement of the object is detected, for example, the movement of the object that occurs when reading the display content when the object is a mobile terminal is calculated, converted into the movement of the line of sight, The movement of the object converted into the movement of the line of sight can be combined with the result of the calculation of the head movement and the eye movement. Thereby, the movement of the line of sight can be evaluated by displaying the head movement, the eye movement, the movement of the object, etc. that contribute to the movement of the line of sight when reading the display content of the portable terminal or the like.

Functional block diagram of a line-of-sight analysis apparatus according to an embodiment of the present invention The figure for demonstrating the principle of the magnetic sensor which can be used for one Embodiment of this invention The figure for demonstrating the parameter showing head movement in one Embodiment of this invention. The figure for demonstrating one principle which detects eye movement in one Embodiment of this invention. An example figure of a mode that movement of a portable terminal is detected in one embodiment of the present invention. The figure for demonstrating calculation of the parameter showing the exercise | movement of a portable terminal in one Embodiment of this invention. The flowchart of the process of the visual axis analyzer which concerns on one Embodiment of this invention. The figure for demonstrating one principle which detects a gaze point by the gaze analyzer which concerns on one Embodiment of this invention. 1 is a diagram showing an example of viewpoint movement by a line-of-sight analysis apparatus according to an embodiment of the present invention.

  Hereinafter, embodiments for carrying out the present invention will be described as embodiments. The present invention is not limited to the embodiments described below, and can be implemented with various modifications.

(Embodiment 1)
FIG. 1 is a functional block diagram of a visual line analyzer according to an embodiment of the present invention. The line-of-sight analysis apparatus 100 includes a head movement detection unit 101, a head movement calculation unit 102, an eye movement detection unit 103, an eye movement calculation unit 104, an object movement detection unit 105, and an object movement calculation unit 106. And a gaze direction calculation unit 107 and a viewpoint calculation unit 108.

  The head movement detection unit 101 detects head movement. The head is the head of the subject. The subject is a subject whose viewpoint position is measured by the line-of-sight analysis apparatus 100. The head movement of the subject's head is detected by the head movement detection unit 101. The subject's movement of the eyeball is detected by an eyeball detection unit 103 described below. Further, when the target object is a portable terminal or a book to be grasped and visually recognized by the subject, the motion of the target object gripped by the subject is also detected by the target object motion detection unit 105 described later. Thereby, the motion of the subject's hand is indirectly detected.

  The head movement detection unit 101 detects the movement of the subject's head using a magnetic sensor or an acceleration sensor. FIG. 2 is a diagram for explaining a principle of detecting movement of the head or the like by the magnetic sensor. The operation of the magnetic sensor is based on the principle of electromagnetic induction in which an electromotive force is generated when a conductor crosses the magnetic field. As shown in FIG. 2, an AC magnetic field 203 is generated by applying an AC voltage to a source coil 202 composed of coils wound in three orthogonal directions by a drive circuit 201. Similarly, when the sensor coil 205 wound in three orthogonal directions moves in the magnetic field 203, the electromotive force of each of the coils wound in the three directions of the sensor coil 205 changes in accordance with the movement. By detecting this electromotive force by the detection circuit 206 and analyzing it by the computer 204, it is possible to detect and output the linear acceleration in the horizontal, vertical and longitudinal directions and the angular acceleration of the roll angle, pitch angle and yaw angle. .

  As long as the sensor can measure the acceleration in the rotation direction and the linear direction, the acceleration sensor may be a sensor using another principle. Moreover, the head movement detection part 101 should just be able to measure the position of a test subject's head continuously. For example, the head of the subject may be photographed from a plurality of locations by optical means, and the position of the head may be measured. Further, the position of the subject's head may be measured using ultrasonic reflection or the like.

The head movement calculation unit 102 calculates the amount of movement due to the movement of the head of the subject based on the result of detection of the head movement of the subject by the head movement detection unit 101. Specifically, when a magnetic sensor is used, if the acceleration output by the magnetic sensor is α, the displacement amount is x, and the time is t, then α = d ² x / dt ² and the acceleration α is the displacement amount. Therefore, the amount of movement can be obtained by integrating the acceleration α twice over time. Further, the current position can be calculated by adding this amount of movement to a position that is a reference for head movement (for example, a position of a sensor or the like at t = 0).

  Further, when the head movement detection unit 101 can continuously measure the position of the subject's head by optical means or the like, by comparing the current measurement result with the previous measurement result, The amount of movement due to the movement of the subject's head is calculated.

  Examples of the movement amount calculated by the head movement calculation unit 102 include a horizontal rotation amount Hψ, a vertical rotation amount Hφ, and an inclination amount Hθ of the subject's head. Here, “left and right”, “up and down” and “front and back” are directions orthogonal to each other. These directions can be defined as appropriate, but the up and down directions should match the vertical direction, the left and right directions should match the left and right directions for the subject, and the front and back directions should match the front and back directions for the subject. Good. In this case, the object can be positioned in the forward direction as viewed from the subject.

  Further, when the upper body of the subject can be freely moved, as in Embodiment 2 described later, the horizontal movement amount Hx, the vertical movement amount Hy, and the longitudinal movement amount Hz of the head are calculated. You can also In the first embodiment, a case where the subject sits deeply on the sofa or a case where the upper body cannot be moved freely because there are people on the left and right in a train seat or the like will be mainly described.

  FIG. 3 shows an example of the definition of Hψ, Hφ, and Hθ. As shown in FIG. 3A, when a sensor such as a magnetic sensor is installed on the subject's parietal portion 301 as shown in FIG. 3A, from the roll angle, pitch angle, and yaw angle detected by the sensor, as shown in FIG. The angle that the subject rotates in the horizontal direction around the top of the head 301 is Hψ, and the angle that the subject tilts the head back and forth is Hφ, as shown in FIG. The angle inclined to can be defined as Hθ. It should be noted that the center of measurement of Hψ is the crown 301, and the center of measurement of Hφ and Hθ can be the position 302 of the neck portion of the subject.

  The eye movement detection unit 103 detects the eye movement of the subject. For this detection, a normal eye movement measuring device can be used. For example, an eye movement measuring device using a corneal reflection method can be used.

  FIG. 4 is a diagram for explaining the detection principle of eye movement using the corneal reflection method. As for the eye movement measuring device, products based on various detection principles are commercially available, and any of them can be used.

  The corneal reflection method is a method of detecting the rotation angle of the eyeball by measuring reflected light by photographing the eyeball portion with a camera or the like. First, as shown in FIG. 4B, infrared light is irradiated to the eye part of the subject by the spot lamp 405 to photograph the eye part of the observer. FIG. 4A shows an example of a captured image 401. Referring to FIG. 4 (b), the infrared light that has passed through the subject's iris and cornea 404 and entered the eyeball 403 is reflected by the retina on the opposite side of the cornea 404 of the eyeball 403, and the pupil is again constricted by the iris. The light exits from the portion and is photographed as a bright white pupil image 410. On the other hand, the infrared light irradiated to the eyes of the observer is reflected on the surface of the cornea and is also photographed as a bright white virtual image 402. Since the rotation center O of the eyeball is different from the rotation center O ′ of the corneas 404 and 407, when the eyeball 403 rotates, the position of the white circle 402 corresponding to the virtual image 408, that is, the position of the reflected light spot from the cornea is shifted. Detected as momentum 409. On the other hand, for the movement of the body and body instead of the eye movement, the center of the previous pupil image 410, that is, the center of the pupil and the reflected light point from the cornea simultaneously move in the same direction and amount, so that the head The eye movement amount 409 can be measured without being influenced by such movement.

  The eye movement calculation unit 104 calculates a horizontal line-of-sight movement angle and a vertical line-of-sight movement angle from the eye movement detected by the eye movement detection unit 103. The calculation results are Xe and Ye, respectively.

  The line-of-sight direction calculation unit 107 calculates the direction of the line of sight of the subject. Xe and Ye indicate directions based on the subject's head. Therefore, by considering Hψ, Hφ, and Hθ in addition to Xe and Ye, for example, the line-of-sight movement angle on the coordinate axis with reference to the neck portion 302 that is the center of movement of the subject's head can be calculated. .

Specifically, the line-of-sight direction calculation unit 107 calculates the line-of-sight movement angle (Fx, Fy) by the following equation.
Fx = Xe '+ Hψ
Fy = Ye '+ Hφ
However, (Xe ′, Ye ′) is defined as (Xe ′, Ye ′) = (Xe, Ye) rot (Hθ), and is a vector obtained by rotating the vector (Xe, Ye) by Hθ. This represents the movement angle of the line of sight in the horizontal and vertical directions when the head is tilted by Hθ.

  That is, the rotation angle of head movement can be directly converted because Hψ corresponds to the horizontal eyeball rotation angle and Hφ corresponds to the vertical eyeball rotation angle. On the other hand, Hθ corresponds to the inclination of the neck. The eye rotation angle corresponds to so-called Torsion (rotational movement), but in general, Torsion hardly occurs as a human eye movement. However, when the neck is tilted, the meaning of the eyeball rotation angle in the horizontal direction and the vertical direction toward the target object is different. Therefore, in one embodiment of the present invention, Hθ with respect to the detected eyeball rotation angle. Therefore, the coordinate axis is rotated, converted into a horizontal component and a vertical component with respect to the object, and added.

  The object motion detection unit 105 detects the motion of the object visually recognized by the subject. If the object is a mobile phone, for example, as shown in FIG. 5, a sensor 603 is installed on the back surface of the mobile terminal 600 held by the subject with the left hand 501, and a signal is transmitted via the wiring 502 connected to the sensor 603. The movement of the mobile phone 600 can be detected by receiving. As the sensor 603, a magnetic sensor based on the above principle can be used.

  The object motion calculation unit 106 calculates the amount of movement of the object from the result of detection of the object motion by the object motion detection unit 105. Usually, as shown in FIG. 5, the subject holds the portable terminal on a plane substantially orthogonal to the line of sight. Therefore, when the object has a flat display surface like a portable terminal, the display surface is defined as an XY-plane, and Δx and Δy, which are movement amounts of the object in the XY-plane, are defined. Calculated and converted to a moving angle observed by the subject as will be described later. Further, the object motion calculation unit 106 can calculate Δz that is the amount of movement of the object in the Z-axis direction perpendicular to the XY-plane and the position in the XYZ coordinates. Hereinafter, a method for converting the movement amount of the hand into the movement angle observed from the subject will be described.

Note that the three-axis directions detected by the magnetic sensor depend on the position of the source and the mounting position of the sensor, and are not necessarily parallel to the XY-plane of the surface when the portable terminal is gripped. Therefore, first, the XY-plane of the display surface of the mobile terminal is determined. For this purpose, the subject is moved in the X direction and the Y direction by an arbitrary amount while holding the portable terminal, and the coordinates detected by the magnetic sensor are converted into the XY-plane from the coordinate position acquired at that time. The following equation f is calculated. In general, the plane can be determined from the coordinates of three points. there,
(X, Y, Z) = f (Mx, My, Mz)
Based on the relational expression, the coordinate in the three-axis direction detected by the magnetic sensor is converted into the coordinate in the three-axis direction composed of the XY-plane and the Z-axis perpendicular thereto. Mx, My, and Mz are movement amounts output from the magnetic sensor, and the movement amounts are horizontal, vertical, and depth movement amounts of the magnetic sensor with respect to the magnetic source. These movement amounts can be converted into movement amounts on the XYZ coordinate axes by the conversion formula f obtained by the above means.

After determining the XY-plane, the process of converting the movement amounts Δx and Δy in the X and Y directions of the mobile terminal into the movement angles observed from the subject will be described by taking Δy as an example. FIG. 6 shows the principle of converting the movement amount when the subject grips the portable terminal 600 having the display surface 601 and the back surface 602 to which the sensor 603 is attached into the movement of the line of sight (movement angle when observed from the subject). It is a figure explaining about. In general, when looking at a portable terminal, the portable terminal is held so as to be perpendicular to the eyes, that is, a perpendicular line drawn from the line of sight. In this way, it is possible to reduce the distortion of the screen that is most viewed to every corner of the screen. Therefore, Δy is considered to be a movement that occurs when reading a line feed and a sentence in vertical writing, and a movement that occurs at the time of line breaking in horizontal writing. D is the distance between the eyes and the display surface. The value of D can be calculated from the calculation result of the position of the head by the head movement calculation unit 102 and the calculation result of the position of the target by the target movement calculation unit 106. Hnφ is an angle in the Y direction among the movement angles observed from the subject in the horizontal direction. Δy is converted into Hnφ indicating the amount obtained by converting the movement amount of the hand into the movement angle (unit: radians) observed from the subject.
Hnφ = tan ⁻¹ (Δy / D)
As for Δx, Hnψ = tan ⁻¹ (Δx / D)
Can be converted into a movement angle Hnψ (unit is radians) observed from the subject in the horizontal direction. It should be noted that the movement Δz in the depth direction of the object can be ignored in a situation where the object such as a portable terminal is gripped, because it hardly changes due to the movable range of the arm. However, in the case of analyzing the line of sight more accurately, Δz and Hz (the depth movement amount of the head, which will be described later) may be added to D described above to correct the distance between the eyes and the mobile terminal.

  Further, the object calculation unit 106 calculates a movement angle when the movement of the object is observed from the subject, so that the object is photographed from a camera attached to the subject's glasses, for example, and the latest photographing result and immediately before The moving angle of the object can also be calculated from the shooting result of. In this case, the object motion detection unit 105 acquires an image signal captured by a camera attached to the subject's glasses or the like.

  The viewpoint calculation unit 108 obtains information on the viewpoint of the subject from the movement amount (Fx, Fy) of the line of sight calculated by the line-of-sight direction calculation unit 107 and the amount of movement of the object calculated by the object movement calculation unit 106. calculate.

In one embodiment of the present invention, the movement angle when an object such as a portable terminal is observed from a subject is calculated. Therefore, the viewpoint calculation unit 108 converts Δx and Δy to Hnψ and Hnφ as described above,
Gx = Fx + Hnψ
Gy = Fy + Hnφ
Thus, the direction of the line of sight, which is the direction of the viewpoint that the subject faces the object, is calculated. The viewpoint calculation unit 108 can calculate the viewpoint as an intersection of a straight line extending in parallel with the vector (Gx, Gy) from the position of the eyeball and the object. In addition, the viewpoint calculation unit 108 can calculate the movement (movement amount, movement direction) of the viewpoint according to the time change of the vector (Gx, Gy).

  FIG. 7 is a flowchart for explaining the flow of processing in the line-of-sight analysis apparatus 100. In step S701, the head movement of the subject is detected by the head movement detection unit 101. In step S702, the eye movement of the subject is detected by the eye movement detection unit 103. In step S703, the target movement detection unit 105 detects the target. Detect the movement of objects.

  After the head movement of the subject is detected by the head movement detection unit 101, the process proceeds to step S704, and the head movement calculation unit 102 calculates Hψ, Hφ, and Hθ. Further, as in the second embodiment, the movement angles Ex and Ey in the horizontal direction and the vertical direction may be calculated.

  After the eye movement of the subject is detected by the eye movement detection unit 103, the process proceeds to step S705, and the eye movement calculation unit 104 calculates Xe and Ye.

  After the movement of the object is detected by the object movement detection unit 105, the process proceeds to step S706, and the object movement calculation unit 106 calculates Hnψ and Hnφ.

  After the processes of steps S704, S705, and S706, the processes of steps S704, S705, and S706 are integrated and the process proceeds to step S707. The line-of-sight direction calculation unit 107 calculates Fx and Fy, and the viewpoint calculation unit 108 Gx and Gy are calculated as information on the viewpoint of the subject.

  Thereafter, the process proceeds to step S708. If the entire process is not terminated, the process returns to the beginning, and the processes in steps S701 to S707 are executed again.

  Before performing the processing shown in the flowchart of FIG. 7, it is preferable to perform calibration as follows. The output of the head movement detection device depends on the mounting state of a sensor such as a magnetic sensor mounted on the head, and the coordinate axes of the three-dimensional orthogonal coordinates and the rotation coordinates do not necessarily coincide with the horizontal and vertical directions of the eye movement. Shake the part horizontally and vertically to calibrate the coordinate system. In order to detect the movement of the object, as described in the determination of the XY-plane, the subject is moved in the X direction and the Y direction by an arbitrary amount in order while holding the portable terminal such as the object. Then, from the coordinate position acquired at that time, an equation f for converting the coordinates detected by the magnetic sensor into the XY-plane on the portable terminal is calculated. Next, the direction of the line of sight is calculated according to the above-described equations for obtaining Gx and Gy from the respective calibration outputs of the head movement, eye movement, and XY-plane. Then, the subject is caused to gaze at the designated point on the mobile terminal screen, and the directions obtained as Gx and Gy are translated so that the line of sight is displayed there.

  As described above, in the present embodiment, the line of sight of the subject can be analyzed in consideration of the motion of the object in addition to the motion of the subject's head and eyeballs.

(Embodiment 2)
In the first embodiment of the present invention, the description has been mainly made on the assumption that the subject cannot freely move the upper body including the head. In the second embodiment of the present invention, analysis of the line of sight of the subject when the subject can freely move the upper body will be described.

In the present embodiment, the head movement calculation unit also calculates the horizontal movement amount Hx, vertical movement amount Hy, and front-rear movement amount Hz of the head. By moving the head in the horizontal direction, the vertical direction, and the front-rear direction, the object moves in the opposite direction with respect to the head. Therefore, using the distance D from the subject's head to the object (for example, the display surface of a mobile phone), the horizontal and vertical movement angles Ex and Ey are expressed as follows: Ex = tan ⁻¹ (Hx / D)
Ey = tan ⁻¹ (Hy / D)
It becomes. Regarding the movement amount Hz in the depth direction, for example, when holding and viewing the mobile terminal, it can be almost ignored due to the limitation of the movable range of the arm. However, it can be used as a visual line movement amount in the depth direction when observing a stereoscopic image or during an operation of actually driving a car. For more accurate measurement, it is preferable to correct the distance between the eyes and the object by adding the amount of movement Δz and Hz of the object in the front-rear direction to the aforementioned D.

Therefore, the gaze calculation in the viewpoint calculation unit 108 is
Gx = Fx + Hnψ + Ex
Gy = Fy + Hnφ + Ey
Is calculated as

(Embodiment 3)
As a third embodiment of the present invention, the definition of a gazing point based on the line-of-sight speed will be described.

  It is known that the eye movement is always small and finely moved by fixation micromotion. Head movements and hand movements are not completely stationary in the space, but always move slightly to maintain posture and position. Therefore, the line of sight obtained by adding these is always in some motion and is not completely stationary. However, when the object to be viewed is captured in the fovea of the retina having the highest visual acuity, it is necessary to keep the line of sight almost stationary at a point in the visual field for a certain period of time. This state is called gaze, and a limited range of objects to be watched is called gaze. However, in general, the line of sight constantly fluctuates for the reasons described above, and the point that a person is consciously trying to capture visual information from that part of the field of view is considered a point of sight, and the line of sight that is constantly moving In order to extract a gaze point from movement, it is necessary to define the range of the gaze point.

Eye movements that occur even when staring at a single point are important movements for obtaining information from the outside world, and should be considered as a gaze component when eye movements are performed only with fixation movements. It is. In addition, jumping movement and tracking movement are known as components of eye movement. It is known that the visual characteristics are reduced during the jumping movement but not so much during the following movement. The following movement occurs when following the motion target, and can be considered as an eye movement for trying to better capture the information of the motion target. However, as the speed of the motion target increases, the relative position of the eyeball relative to the target gradually delays. It is known that the eye movement speed of the follow-up movement that can follow the movement target faithfully with little delay is about 5 ° per second. Therefore, as a method for separating the gazing point from the eye movement, a “substantial gazing point” is defined as shown in FIG. 8 with a threshold speed Vth of 5 ° per second. That is, the distance Dth to reach the next sample point at a speed of Vth from a certain gazing point 801 is determined as follows.
Dth = Vth × sample time interval As shown in FIG. 8, when a next sample point 802 exists in a circle 804 having a radius of Dth 803, it is assumed that the point is included in the current gaze point, and the substantial gaze point is When it does not change and exists outside the circle 804, a substantially new gaze point is set.

  The above describes the extraction of the gazing point, focusing on the eyeball movement. From the viewpoint of the movement of the visual target on the retina, the eye movement, the head movement, or the hand movement The phenomenon that the visual target moves on the retina remains the same. Therefore, in the embodiment of the present invention, the above-described gaze point definition is defined with respect to the speed of change of the line-of-sight directions Gx and Gy including the eyeball rotation angle, head movement, and movement of the mobile terminal held by the hand. Is applied to separate the gaze point from the line-of-sight directions Gx and Gy. The conventional method of calculating the gaze from eye movements and head movements to extract the gazing point is a hand holding the mobile terminal while moving the terminal by hand movement or in a train. Even if the eye movement moves greatly, if the line of sight calculated from the eye movement and the head movement is not moving, there is a disadvantage that the gazing point is extracted. By applying the embodiment of the present invention, there is a great feature that it is possible to extract only when the line of sight considering the movement of the hand in addition to the eye movement and the head movement gazes at the visual target.

(Embodiment 4)
Using the line-of-sight analysis apparatus according to Embodiment 4 of the present invention, the gazing point is extracted as described above, and the movement of the line of sight is caused by any of the head movement of the subject, the eye movement of the subject, and the movement of the object. Can be determined. For example, when the viewpoint calculated by the viewpoint calculation unit 108 moves, it contributes most to the movement amount of the gazing point among the calculation results of the head movement calculation unit 102, the eye movement calculation unit 104, and the object movement calculation unit 106. Determine the result. Therefore, it is possible to determine whether the movement of the line of sight is caused by the head movement, the eye movement, or the movement of the object.

  In other words, the position of the subject's viewpoint on the surface of the object can be displayed by obtaining the intersection of the surface of the object and a straight line extending in the direction of the viewpoint. Therefore, the head movement amount / the movement amount of the object is converted into an eye movement amount, and is combined with the eye movement amount to calculate an integrated line of sight movement. The gaze point is extracted from the calculation result, and the gaze time is calculated. In addition to calculating the movement speed from the movement trajectory between gazing points, the movement amount by eye movement, the movement amount by converting head movement to eye movement amount, and the movement amount by converting hand movement to eye movement amount are calculated. Along with data that understands time-series changes in movement, head movement, and hand movement, it is output as a gaze analysis output.

  FIG. 9 shows an example in which when the object is a display of a portable terminal, the position of the viewpoint is superimposed on the presentation image displayed on the display. The displayed images vary in image quality and resolution, such as images of electronic books and mobile devices. Once image conversion is performed, it is converted into image quality and resolution that can be displayed by combining eye movement, head movement, and hand object movement. It is a thing.

  FIG. 9 shows an example in which the movement of the gazing point and the line of sight is projected on a display image of a mobile terminal or the like and displayed on the display unit of the line-of-sight analysis apparatus in accordance with the gazing point extraction and the viewpoint movement determination. In FIG. 9, “◯” 901, 905, 907, 909, 911, 914, and 917 represent gazing points, indicating that the subject was gazing at the position indicated by “◯”. Solid lines 902, 904, 910, and 913 indicate the movement of the line of sight generated by the head movement, the alternate long and short dash lines 906 and 912 indicate the movement of the line of sight generated by the eye movement, and the broken line 903 is generated by the movement of the object. Indicates the movement of the line of sight.

  Therefore, the movement trajectory between the gazing points indicates where the movement between the gazing points is performed in the image. The gazing point can be displayed in different colors according to the gazing time, and the movement locus between the gazing points can be displayed in different colors according to the moving speed. Further, as described above, it is possible to display the head movement converted into the amount of eye movement by the solid line, the eye movement amount by the alternate long and short dash line, and the movement of the object converted to the eye movement amount by the dotted line. Furthermore, for example, the eye movement is a green line, the head movement converted to an eye movement is a red line, and the movement of the object is converted to an eye movement amount, and the blue line is parallel to the movement trajectory between the above-mentioned gazing points. To know where the eye movement, head movement, and movement of the object were performed during the movement between the gazing points from the output image obtained by combining the movements of the eye, head, and object. Can do.

In addition, the display mode of “◯” can be changed depending on the duration of the gazing point. For example, depending on the duration,
Blue <100ms
100ms ≦ yellow <200ms
200ms ≦ cyan <300ms
300ms ≦ magenta <400ms
500ms ≦ blue green <600ms
600ms ≦ green <700ms
700ms ≦ red <800ms
The display color of “◯” can be changed as 800 ms ≦ white. The inequality “blue <100 ms” means that if the duration of the gazing point is less than 100 ms, the display is performed in blue. The same applies to other inequalities.

Also, the movement trajectory between gazing points can be color-coded as follows.
Blue <100deg / sec
100deg / sec ≦ yellow <200deg / sec
200deg / sec ≦ cyan <300deg / sec
300deg / sec ≦ magenta <400deg / sec
500deg / sec ≦ blue green <600deg / sec
600deg / sec ≦ green <700deg / sec
700deg / sec ≦ red <800deg / sec
800deg / sec ≦ white

  As described above, as the line-of-sight analysis output, the movement amount by eye movement, the movement amount by converting head movement to eye movement amount, the movement amount of hand movement to eye movement amount, their start time, operation end time, and eye movement Calculates the gaze point position and gaze time calculated from the line of sight that combines the movement amount, the movement amount converted from the head movement into eye movement amount, the movement amount converted from the movement amount of the object into eye movement amount, and between the fixation points. The amount of movement due to eye movement occupying the line of sight, the movement amount obtained by converting head movement into eye movement amount, and the ratio of movement amount to eye movement amount of hand movement can be output for use in human interface research.

  In each embodiment of the present invention, regarding the movement of the line of sight when visual information is acquired from the mobile terminal, not only the eye movement and the head movement that have been performed so far, but also the role of the movement of the object is displayed. Can contribute to research and development in this field. The visual line analysis output is calculated from the movement amount by the eye movement described above, the movement amount obtained by converting the head movement into the eye movement amount, the movement amount obtained by converting the movement amount of the object into the eye movement amount, and the visual line obtained by combining these. In addition to the position of the gazing point and the gazing time, the movement amount of the eye movement occupying the line of sight between the gazing points, the movement amount obtained by converting the head movement into the eye movement amount, and the ratio of the movement amount obtained by converting the movement amount of the object into the eye movement amount. The output can be used to develop new human interfaces.

DESCRIPTION OF SYMBOLS 100 Gaze analysis apparatus 101 Head movement detection part 102 Head movement calculation part 103 Eye movement detection part 104 Eye movement calculation part 105 Object movement detection part 106 Object movement calculation part 107 Gaze direction calculation part 108 View point calculation part

Claims (7)

A line-of-sight analysis device that analyzes the line of sight of a subject who turns his line of sight to an object,
A head movement calculation unit for calculating a rotational movement angle of the head by a rotational movement of the head;
An eye movement calculation unit that calculates a rotation angle of the eyeball due to eye movement;
A line-of-sight direction calculation unit that calculates a movement angle in the direction of the line of sight of the subject based on the rotation movement angle calculated by the head movement calculation unit and the rotation angle calculated by the eye movement calculation unit;
A target motion calculation unit that calculates a movement angle when the movement of the target object is observed from the subject; and
A line of sight having a viewpoint calculation unit that calculates a movement of the viewpoint by the line of sight toward the object by the subject based on the movement angle calculated by the line-of-sight direction calculation unit and the movement angle calculated by the target motion calculation unit. Analysis equipment. The head movement calculation unit calculates a movement amount due to the parallel movement of the head;
The line-of-sight analysis device according to claim 1, wherein the line-of-sight direction calculation unit converts the movement amount due to the parallel movement of the head calculated by the head movement calculation unit into a movement angle when observed from the subject.   The line-of-sight analysis apparatus according to claim 1, wherein the movement of the viewpoint calculated by the viewpoint calculation unit is displayed by being superimposed on the image of the object.   The line-of-sight analysis apparatus according to claim 3, wherein a display mode superimposed on the image of the object is changed according to a movement speed of the viewpoint calculated by the viewpoint calculation unit.   The line-of-sight analysis apparatus according to claim 3 or 4, wherein when the speed of movement of the viewpoint calculated by the viewpoint calculation unit is less than a threshold value, the viewpoint is superimposed on the image of the target object as a gazing point.   Depending on whether the movement of the viewpoint calculated by the viewpoint calculation unit is due to movement of the head, movement of the eyeball, or movement due to movement of the object, The line-of-sight analysis apparatus according to claim 3, wherein a display mode superimposed on the display is changed. It is a line-of-sight analysis method by a line-of-sight analysis device that analyzes the line of sight of a subject who turns his line of sight to an object,
The line of sight analyzer is
A head movement calculation unit for calculating a rotational movement angle of the head by a rotational movement of the head;
An eye movement calculation unit that calculates a rotation angle of the eyeball due to eye movement;
A line-of-sight direction calculation unit that calculates a movement angle in the direction of the line of sight of the subject based on the rotation movement angle calculated by the head movement calculation unit and the rotation angle calculated by the eye movement calculation unit;
A target motion calculation unit that calculates a movement angle when the movement of the object is observed from the subject,
A line-of-sight analysis method including calculating a movement of a viewpoint by a line of sight directed toward the object by the subject based on the movement angle calculated by the line-of-sight direction calculation unit and the movement angle calculated by the target motion calculation unit.

Images