JPH0583248B2 - - Google Patents

Description

[Detailed description of the invention]

[0001]

[Industrial Application Field] The present invention relates to a line-of-sight direction detection device, and in particular, regarding information regarding the direction in which a person looks, that is, the line-of-sight direction. This invention relates to a detection device.

[0002]

[Prior Art] In technical fields that require so-called telexistence, in which information existing in a remote location is photographed by remotely controlling a man-machine interface or a camera, information on the direction of a person's line of sight is used in various control systems. One of the key challenges is to design a line-of-sight detector that transmits data to and interacts with the direction in which it is viewed. In particular, it is strongly desired to realize stable and sensitive line-of-sight linkage. That is, it is necessary to design an optimal filter.

[0003] FIG. 6 is a diagram illustrating an example of a device that detects the line of sight direction using a conventional method. Referring to FIG. 6, a self-propelled robot 1 includes a camera 2 and a manipulator 3 having six degrees of freedom: left and right, up and down, and left and right rotation.
and is remotely guided by commands from the command system 4. In the command system 4, movements 5 of a person's eyes and head are detected by a head orientation detection unit 7 and an eyeball orientation detection unit 8 provided on glasses 6, and hand movements are detected by an operation unit 9. and the respective detection outputs are provided to the transmission section 10. These detection outputs are transmitted from the transmission section 10 to the self-propelled robot 1 by wireless or the like. The self-propelled robot 1 moves according to the detection output of the operating section 9, and the manipulator 3
operates, and images in the directions detected by the detection units 7 and 8 are captured by the camera 2 and transmitted to the command system 4. The captured image is displayed on the monitor 11. Ideally, a stable continuous landscape is reproduced as shown in 12.

[0004] FIG. 7 is a diagram showing an example of a landscape that is actually reproduced according to eye and head movements using a conventional method. In the monitor 11 shown in FIG. 6, high-frequency noise contained in the detection outputs of the head movement detection section 7 and the eyeball orientation detection section 8 is removed by a low-pass filter, and the detection outputs are processed according to a specific method. It is displayed as an image viewed at a glance. However, due to the involuntary movements involved in eye movements and their unstable behavior, landscapes unrelated to intention are reproduced. Therefore,
A stable image can be reproduced by significantly lowering the detection sensitivity of the head orientation detector 7 and the eyeball orientation detector 8, but this will result in a time lag between the direction you want to view and the reproduced image. come to perceive.

[0005] FIG. 8 is a diagram showing an algorithm of a head and eyeball detection system using the head orientation detection unit and the eyeball orientation detection unit shown in FIG. 6. Referring to FIG. 8, the detection unit 21 includes the head orientation detection unit 7 and the eyeball orientation detection unit 8 shown in FIG. ) and the eyeball orientation coordinates (Ex, Ey). Noise included in these displacement information is removed by the filter section 22, and each displacement information is changed to Hx→Hx ₁ , Hy→
Hy ₁ , Hz→Hz ₁ , Ex→Ex ₁ , Ey→Ey ₁ . The line-of-sight direction calculation unit 23 synthesizes the filtered head and eyeball displacement information. This is to uniformly calculate the viewing direction in the screen system when the displacement of the eyeball is a displacement with respect to the head, and the head is a displacement with respect to the screen system. Each component of the line of sight is shown in Figure 6.
The signal is transmitted to the self-propelled robot 1 via the transmitter 24 included in the transmitter 10 shown in FIG.

[0006] FIG. 9 is a diagram for explaining an example of a waveform detected from head movement and eyeball movement. The detection output of the head orientation detection section 7 of the observer 5 shown in FIG. 6 has a waveform 31 shown in FIG. 9a, and the detection output of the eyeball orientation detection section 8 has a waveform 34 shown in FIG. 9b.
become that way. When these detection outputs are passed through low-pass filters 32 and 35, respectively, smooth waveforms 33 and 36 are obtained. Although the head orientation detection waveform 33 becomes considerably smooth, there is still room for smoothing the eyeball orientation detection waveform 36. In order to achieve stable and sensitive interlocking,
Filtering that takes into account the behavior of the eyes and head is required.

【0077】

[Problems to be Solved by the Invention] As mentioned above, the conventional line of sight detection method simply includes low-pass filters 32 and 35 as shown in FIG. Although smoothing has been achieved to some extent for However, there were problems such as visual discomfort.

[0008] Therefore, the main purpose of the present invention is to develop a gaze direction system that incorporates a gaze detection algorithm that includes an optimal filter, based on the knowledge about human eyeball and head movement control and coordination obtained through experiments. An object of the present invention is to provide a detection device.

【0009】

[Means for Solving the Problems] The present invention is a line-of-sight direction detecting device, which includes a head direction detecting means for detecting the amount and direction of movement of the head, and an eye direction detecting means for detecting the amount and direction of movement of the eyeball. a detection means; a first filter means for removing minute change components included in the displacement components of the detection outputs of the head orientation detection means and the eyeball orientation detection means; A head movement state determination means for determining the movement state of the head according to the displacement component; and a head movement state determination means for determining the movement state of the head, and a change in the orbit of the eyeball and a movement direction of the eyeball according to the displacement component in the direction of the eyeball included in the output of the first filter means. eyeball direction calculation means for calculating the eyeball direction calculation means; a second filter means for removing unnecessary components associated with involuntary movements according to the eyeball movement directions calculated by the eyeball direction calculation means for each eyeball movement direction; It is configured to include a movement information calculation means for calculating movement information of the line of sight according to the output of the state determination means and the output of the second filter means.

【0010】

[Function] The line-of-sight direction detection device according to the present invention has the following features:
The first filter removes minute change components included in the detected head orientation detection output and eyeball orientation detection output, determines the movement state of the head according to the head orientation displacement component, and determines the eyeball orientation. The trajectory change of the eyeball and the movement direction of the eyeball are calculated according to the displacement component, and unnecessary components accompanying involuntary movements according to the direction of eyeball movement are removed by the second filter means, thereby achieving a stable head movement state. The line of sight movement information is calculated according to the eyeball movement direction.

[0011]

Embodiment FIG. 1 is a diagram for explaining the operating status of the visual line detection direction device according to the present invention. Figure 1
, a movement 41 of the eyeball and a movement 42 of the head are shown, and the movement of the eyeball and the head is shown when the observer looks in the horizontal, diagonal, and vertical directions.
A time delay 43 occurs between the eyeball and head movements at the points 44, 45 where the direction of the line of sight changes. On the other hand, it has been found that when the line of sight moves in a fixed direction, the movement of the eyeballs contains a saccadic component and exhibits unstable behavior, but the movement of the head is relatively smooth46. There is. In particular, the instability of the eyeball varies depending on the direction of movement.

[0012] FIG. 2 is a diagram for explaining the difference in instability of head and eye movements due to differences in line of sight.
Head movements 47, 49 and eye movements 48, 50
is detected when the observer is instructed to fixate on a single point. It can be seen that the behavior is clearly different between head movement 47 and eyeball movement 48 in a sitting position, and head movement 49 and eyeball movement 50 in a three-dimensional position. Also, the eyeballs don't stay on one point.

[0013] FIG. 3 is a diagram showing the configuration of an embodiment of the present invention, and FIG. 4 is a flowchart for explaining the operation of the stabilizing section shown in FIG. 3.

[0014] First, referring to FIG. 3, the detection unit 51 includes a head orientation detection unit 52 and an eyeball orientation detection unit 53. The head orientation detection unit 52 detects head orientation components (Hx, Hy, Hz) and provides them to the filter unit 54. The filter section 54 detects the head direction component (Hx,
Noise components are removed from the signal (Hy, Hz) and provided to the displacement amount detection section 56. The displacement amount detection unit 56 detects the position change per unit time of each component of the head orientation component (Hx, Hy, Hz), that is, the velocity ΔHx,
Calculate ΔHy and ΔHz. Also, from the velocity displacement components ΔHx, ΔHy, ΔHz, the direction ΔXhx = ΔHx/Δt,
Calculate ΔVhy=ΔHy/Δt and ΔVhz=ΔHz/Δt.

[0015] Similarly, the eyeball orientation detection unit 53 detects eyeball orientation components (Ex, Ey) and provides them to the filter unit 55. The filter 55 removes noise components from the eyeball direction components (Ex, Ey) and provides the result to the displacement detection section 57 . The displacement amount detection unit 57 detects the position change per unit time of each component of the eyeball orientation component (Ex, Ey), that is, the velocity ΔEx, ΔEy.
Calculate. Also, the velocity displacement components ΔEx, ΔEy
Direction from ΔVex=ΔEx/Δt, ΔVey=ΔEy/Δt,
Calculate Δte=tan ^-1 (ΔEy/ΔEx).

[0016] The determining unit 58 sequentially selects head or eye movement information. Here, the operation of the determination unit 58 will be explained in more detail with reference to FIG. 4. In step SP1 (abbreviated as SP in the diagram),
Based on the detected speed change of the head, it is determined whether the head is stationary or moving. If it is determined that the head is moving, the direction of the eyeballs is determined in step SP2. Based on the change, it is determined whether the trajectory of the eye movement has changed. If there is a change in the eye movement trajectory, proceed to step SP3.
The direction of the eyeball is detected based on the speed change information of the eyeball. At step SP4, filters with different bands for horizontal, vertical, and diagonal directions are sequentially selected according to the detected direction of movement of the eyeball, and noise included in the component in the direction of movement is removed. In step SP5, the direction in which the head moves and the direction in which the eyeballs move are compared, and if the directions of movement are parallel to each other, head movement information is selected in step SP7, and if they are not parallel, the information is selected in step SP6. Eye movement information is selected. The line-of-sight direction calculation unit 59 shown in FIG.
In SP8, head or eyeball movements are sequentially selected and the direction of the line of sight is calculated.

[0017] FIG. 5 is a diagram showing the results of a detection experiment using a line-of-sight direction detection device including a line-of-sight detection algorithm according to the present invention. When observing the schematic diagram 61 of a house displayed on the display shown in FIG.
3 is becoming unstable. In other words, it can be seen that the linearity of eye movement is poor. On the other hand, Fig. 5
As shown in b, when the trajectory of the line of sight is detected using the present invention, the line of sight 63 becomes smooth. For comparison, the results using the conventional algorithm are shown in Figure 5c. Obviously, due to the unstable behavior of eye movements, the conventional algorithm cannot produce the composite value of the head and eyeball trajectories, i.e., the line of sight trajectory6.
4 becomes a mess as it is. On the contrary,
In this invention, when the line of sight 63 moves as shown in FIG. 5b, the trajectory is switched to the trajectory of the head that shows stable behavior, and information on changes in the trajectory of the eyeballs is used to change the direction of the line of sight. Since filters with different bands are adaptively switched depending on the direction of the line of sight, it is possible to extract stable and sensitive line-of-sight direction information that could not be achieved with conventional line-of-sight direction detection.

[0018]

Effects of the Invention As described above, according to the present invention, based on information on the movement direction, movement speed, movement start information, etc. of the head and eyeballs from the detector, Since the filters are adaptively switched, a stable and sensitive interlocking system can be realized against unstable eyeball behavior including involuntary movements. Therefore, when reproducing an image in the direction in which it is viewed, detection errors can be greatly reduced compared to conventional interlocking.

[Brief explanation of the drawing]

FIG. 1 is a diagram for explaining the operating status of a line-of-sight direction detection device according to the present invention.

FIG. 2 is a diagram for explaining differences in instability of head and eye movements due to differences in posture.

FIG. 3 is a diagram showing the configuration of an embodiment of the present invention.

FIG. 4 is a flow diagram for explaining the operation of the determination section shown in FIG. 3;

FIG. 5 is a diagram showing the results of a detection experiment using a line-of-sight direction detection device including a line-of-sight detection algorithm.

FIG. 6 is a diagram illustrating an example of a device that detects the line of sight direction using a conventional method.

FIG. 7 is a diagram showing an example of a landscape actually reproduced according to the back position of the eyes and head using a conventional method.

FIG. 8 is a diagram for explaining problems in linking the direction of the line of sight and the acquired scenery using a conventional line of sight direction detection algorithm.

FIG. 9 is a diagram for explaining an example of a waveform detected from head movement and eyeball movement.

[Explanation of symbols]

51 Detection section 52 Head orientation detection unit 53 Eyeball orientation detection unit 54, 55 Filter section 58 Judgment section 59 Gaze direction calculation unit.

Claims (1)

[Claims] 1. Head orientation detection means for detecting the amount of movement and direction of the head; eyeball direction detection means for detecting the amount and direction of eyeball movement; a first filter means for removing minute change components included in the displacement component of the detection output; a head for determining the movement state of the head according to a displacement component in the head direction included in the output of the first filter means; part movement state determination means; eyeball direction calculation means for calculating the orbital change of the eyeball and the movement direction of the eyeball according to the displacement component of the eyeball direction included in the output of the first filter means; a second filter means for removing unnecessary components associated with involuntary movements according to the calculated eyeball movement directions for each eyeball movement direction;
and a line-of-sight direction detecting device, comprising movement information calculation means for calculating line-of-sight movement information according to the determination output of the head movement state determination means and the output of the second filter means.