JP2023058161A - Data processing device, eye movement data processing system, data processing method, and program - Google Patents

Abstract

A data processing device, an eye movement data processing system, a data processing method, and a program are provided for processing data from an imaging device that images an eye in medical care.
A data processing device (100) according to the present disclosure is a processing device that processes data from an imaging device (400) that images an eye in medical care. The data processing device 100 includes an I/O interface 110 and a control section 120 . The I/O interface 110 receives images of the eyes of the subject 2 captured by the first imaging unit 411 and the second imaging unit 412 held in the housing 401 of the imaging device 400 worn on the head of the subject 2. Data C is input. The control unit 120 derives the displacement amount between the head of the subject 2 and the housing 401 based on the image data C input through the I/O interface 110 .
[Selection drawing] Fig. 3

Description

The present invention relates to a data processing device, an eye movement data processing system, a data processing method, and a program.

Traditionally, otolaryngologists, neurologists, and neurosurgeons have focused on diagnosing dizziness, balance dysfunction, and other conditions by examining how eyeballs move in response to stimuli to the eyes, head, or ears. Balance function tests are widely used to determine whether Patent Literature 1 discloses a nystagmus analysis system in which a subject wears goggles (imaging device) incorporating a camera and a head position sensor to capture eye movements of a subject. Patent Literature 2 discloses an eye-tracking system that tracks the subject's line of sight with a camera fixed at a location independent of the subject.

Japanese Patent Application Laid-Open No. 2020-18704 Japanese Patent Application Publication No. 2019-519859

In the nystagmus analysis system disclosed in Patent Literature 1, the subject's head is moved while wearing the goggles, so the position of the goggles may shift from the position of the first worn goggles. Accurate measurements cannot be made if the goggles are misaligned. In addition, in order to avoid the position of the goggles being shifted, it is conceivable to employ the eye-tracking system disclosed in Patent Document 2. etc.) cannot be employed.

The present disclosure has been made to solve such problems, and provides a data processing device, an eye movement data processing system, a data processing method, and a program for processing data from an imaging device that images the eye in medical care. intended to provide

A data processing apparatus according to the present disclosure processes data from an imaging device that images the eye in medicine. The data processing device includes an input unit for inputting image data of the subject's eyes captured by an imaging unit held in a housing of an imaging device worn on the subject's head, and an input unit for inputting image data of the subject's eyes. a deriving unit that derives the displacement between the subject's head and the housing based on the image data.

An eye movement data processing system according to the present disclosure processes eye movement data in balance testing. An eye movement data processing system includes an imaging device that images an eyeball of a subject, and a data processing device that receives data from the imaging device and processes the data. The imaging device includes a housing mounted on the subject's head, and an imaging unit held in the housing for imaging the subject's eyes. It includes an input unit to which image data of a person's eyes is input, and a derivation unit to derive the displacement amount between the subject's head and the housing based on the image data input by the input unit.

A data processing method according to the present disclosure is a method of processing data from an imaging device that images the eye in medicine. The data processing method includes a step of inputting image data of the subject's eye captured by an imaging unit held in a housing of an imaging device worn on the subject's head, and a step of inputting image data of the subject's eye, and and deriving the amount of displacement between the subject's head and the housing.

A program according to the present disclosure is executed by a data processing device that processes data from an imaging device that images the eye in medicine. The program includes a step of inputting image data of the subject's eyes captured by an imaging unit held in a housing of an imaging device worn on the subject's head, and based on the input image data, and deriving an amount of displacement between the subject's head and the housing.

According to the present disclosure, it is possible to derive the displacement amount between the head of the subject and the housing from the image of the eye captured by the imaging unit.

1 is a schematic diagram showing the configuration of an eye movement data processing system according to Embodiment 1; FIG. 1 is a schematic diagram for explaining the configuration of an imaging device according to Embodiment 1; FIG. 1 is a block diagram showing the overall configuration of an eye movement data processing system according to Embodiment 1; FIG. It is a schematic diagram for demonstrating an example of nystagmus examination. FIG. 10 is a diagram for explaining processing for deriving the amount of deviation between the subject's head and the housing in the eye movement data processing system according to the first embodiment; 4 is a diagram showing eye image data processed by the data processing device according to Embodiment 1. FIG. 4 is a schematic diagram for explaining processing for correcting eyeball positions and head angular velocity data in image data in the data processing apparatus according to Embodiment 1. FIG. 4 is a flowchart for explaining data processing in the data processing device according to Embodiment 1; FIG. 12 is a block diagram for explaining processing for estimating feature points using an estimation model in the data processing device according to Embodiment 2; FIG. 11 is a block diagram for explaining a learning stage of an estimation model in the data processing device according to Embodiment 2;

Embodiments of the present disclosure will be described in detail with reference to the drawings. The same or corresponding parts in the drawings are given the same reference numerals, and the description thereof will not be repeated.

[Embodiment 1]
An eye movement data processing system and imaging device according to Embodiment 1 will be described with reference to FIGS. 1 to 3. FIG. Note that the eye movement data processing system 10 described below is just an example. It is not limited to data processing systems.

The operator 1 can diagnose the dizziness of the subject 2 by using the eye movement data processing system 10 . Note that the "operator" is any person who uses the eye movement data processing system 10, such as an operator such as a doctor belonging to a clinic, general hospital, or university hospital, or a teacher or student at a medical university. good too. The medical department to which the operator belongs is not limited to departments specializing in vertigo treatment such as ophthalmology and otolaryngology, but may be other departments such as internal medicine and dentistry. The “subject” may be any person to be diagnosed by the eye movement data processing system 10, such as patients in clinics, general hospitals, university hospitals, and subjects in medical colleges. "Vertigo" refers to vertigo in which the world in front of you spins around, dizziness in which you feel like you are floating, and fainting dizziness in which the world in front of you turns black. Including the state in which an abnormality occurs.

As shown in FIG. 1 , an eye movement data processing system 10 according to Embodiment 1 includes a data processing device 100 . A display 300 , an imaging device 400 , a keyboard 501 and a mouse 502 are connected to the data processing device 100 . Display 300 is an example of a display device. Also, the keyboard 501 and the mouse 502 are examples of input devices.

Diagnosis of vertigo is generally made by observing nystagmus (an involuntary movement of the eyeballs that rhythmically moves). Nystagmus includes spontaneous nystagmus that occurs spontaneously in the absence of any stimulus and evoked nystagmus that occurs when a stimulus is applied. Further, induced nystagmus includes head positional nystagmus induced when the position of the head is displaced and head positional nystagmus induced when the position of the body is displaced. With regard to induced nystagmus, it is known that when a physiological rotational stimulus is applied to the head, the eye moves in the opposite direction to the head in order to stabilize the visual field. It is also called VOR (Vestibulo Ocular Reflex).

Specifically, in the eye movement data processing system 10, in order to observe the nystagmus of the subject 2, the eyeball of the subject 2 is imaged by the imaging device 400, and the imaged image data is processed by the data processing device 100. , storage, and display. Therefore, the data processing device 100 is connected to the imaging device 400 . The imaging device 400 is a goggle-shaped device that is worn on the head of the subject 2, and is a device that captures an image of the eyeball of the subject 2 and acquires image data of eye movement for use in diagnosing dizziness. As shown in FIG. 1, an operator 1 performs a nystagmus examination with an imaging device 400 mounted on a subject 2, and the image data of the eyeball movement of the subject 2 obtained at that time is taken as Input to the data processing device 100 . The data processing device 100 processes the image data acquired by the imaging device 400 and provides the operator 1 with information necessary for diagnosing dizziness.

The imaging device 400 shown in FIG. 2A is in a state where a light shielding cover 402 is attached to the front side. Wiring 403 for connecting to the data processing apparatus 100 is provided on the upper surface of the housing 401 . Note that the connection between the imaging device 400 and the data processing device 100 is not limited to a wired connection, and may be wireless as long as a sufficient communication speed can be secured for transmitting image data.

The imaging device 400 shown in FIG. 2B is shown disassembled into an upper housing 401a, a lower housing 401b, and an eyepiece 401c that contacts the subject 2. FIG. A first imaging unit 411 of an infrared imaging device for imaging the right eye of the subject 2 and a second imaging unit 412 of the infrared imaging device for imaging the left eye of the subject 2 are provided in the lower housing 401b. It is On the other hand, although not shown, the arithmetic processing unit 420 shown in FIG. 3 is provided in the upper housing 401a. In the present disclosure, a configuration in which the imaging device 400 is provided with the first imaging unit 411 and the second imaging unit 412 will be described. Alternatively, it may be configured to image both eyes.

The eyepiece 401 c has an opening 401 d so that the eyeball of the subject 2 can be imaged by the first imaging section 411 and the second imaging section 412 while covering the anterior segment of the subject 2 . The eyepiece 401c is made of a synthetic resin or soft rubber having appropriate flexibility and resilience so that the eyepiece 401c is brought into close contact with the face of the subject 2 when worn.

The light shielding cover 402 is provided with a magnet, for example, so that it can be easily attached to and detached from the imaging device 400 . When the light-shielding cover 402 is removed from the imaging device 400 , the subject 2 can look ahead through the hot mirror 410 and can see indices and the like emitted from the visual stimulus signal processing device 600 . The hot mirror 410 is a glass or resin plate coated with a coating that transmits visible light and reflects near-infrared light. optical components. The first imaging unit 411 and the second imaging unit 412 capture images of the subject's 2 eyeball reflected by the hot mirror 410 .

As shown in FIG. 3, the imaging device 400 processes the image data A from the first imaging unit 411 and the image data B from the second imaging unit 412 in the arithmetic processing unit 420 to obtain image data C as a data processing device. Sending to 100. The first imaging unit 411 and the second imaging unit 412 can capture images at 60 frames/second or 240 frames/second. Further, the infrared imaging elements used for the first imaging unit 411 and the second imaging unit 412 are, for example, CMOS (Complementary Metal Oxide Semiconductor) sensors capable of imaging infrared rays, CCD (Charge Coupled Device), and the like.

The imaging device 400 is provided with a head sensor 450 for detecting the movement of the head of the subject 2 in the housing 401 as shown in FIG. The head sensor 450 is composed of nine sensors, for example, an acceleration sensor, an angular velocity sensor, and a geomagnetic sensor provided in three axial directions. The acceleration sensor can detect the orientation of the head of the subject 2 by detecting gravitational acceleration. The angular velocity sensor can detect the angular velocity of the subject's 2 head. The geomagnetic sensor can detect the orientation (orientation) of the subject's 2 head. Based on the measurement signal from the head sensor 450, the arithmetic processing unit 420 calculates the head angle, the head angular velocity, and the like. In the imaging device 400, a configuration in which the head sensor 450 is provided will be described. may

The arithmetic processing unit 420 synchronizes the image data A captured by the first imaging unit 411, the image data B captured by the second imaging unit 412, and the external signal from the visual stimulus signal processing device 600 to generate image data. Arithmetic processing to generate C is performed. Therefore, the arithmetic processing unit 420 is an arithmetic entity that executes image data processing, and is an example of a computer, and is configured by, for example, a CPU (Central Processing Unit), an FPGA (Field-Programmable Gate Array), or the like. Further, the arithmetic processing unit 420 has memories such as a RAM (Random Access Memory) for storing images and the like, and a ROM (Read Only Memory) for storing programs and the like. In addition to the configuration for generating the image data C, the arithmetic processing unit 420 also has a configuration that is executed as a communication unit that transmits the generated image to the outside.

Here, the visual stimulus signal processing device 600 is a laser device as shown in FIG. 3, and is a device for generating laser points on the screen and displaying them as visual targets. The visual stimulus signal processing device 600 can transmit and receive data to and from the data processing device 100 .

When performing a nystagmus examination used for diagnosing dizziness, it is necessary to displace the position of the head of the subject 2 while the subject 2 is wearing the imaging device 400 as shown in FIG. . FIG. 4 is a schematic diagram for explaining an example of nystagmus examination. Nystagmus examination is performed under predetermined conditions. For example, in the inspection of spontaneous nystagmus, the operator 1 fixes the head of the subject 2 and makes the subject 2 gaze at the front, and diagnoses dizziness based on the eye movement of the subject 2 at that time. do. In the examination of head position nystagmus, as shown in FIG. 4, the operator 1 displaces the position of the head of the subject 2 to various states, and the eyeballs of the subject 2 induced at that time are displaced. Diagnose vertigo based on exercise.

At this time, the position of the imaging device 400 may deviate from the position of the imaging device 400 attached first. Accurate measurement cannot be performed if the position of the imaging device 400 deviates from the initial position. Therefore, in the data processing device 100, the amount of deviation between the head of the subject 2 and the imaging device 400 (housing 401) is derived from the obtained eye image data, and if the deviation is within a correctable range, the amount of deviation is calculated. Correct the eye movement data based on If the derived amount of deviation exceeds the correctable range, the data processing apparatus 100 notifies the deviation of the housing 401 as a measurement error through the display 300, a speaker (not shown), or the like. Note that the display 300, speaker, etc. function as a notification unit that notifies the user of the measurement error. Alternatively, if the derived deviation amount exceeds the correctable range, the data processing device 100 associates it with the eye movement data derived at that timing, and stores data indicating that fact in the storage device 700 (for example, SSD (Solid State Drive), HDD (Hard Disk Drive), etc.).

The data processing device 100 includes an I/O interface 110, a control section 120, and an image processing section 130, as shown in FIG. The I/O interface 110 is an input unit to which image data of the subject's eyes captured by the imaging units (the first imaging unit 411 and the second imaging unit 412) held in the housing 401 of the imaging device 400 is input. function as Furthermore, the I/O interface 110 inputs and outputs signals, data, etc. between the display 300 and the input device 500 as well.

The control unit 120 derives the amount of deviation between the subject 2 and the housing 401 from the amount of movement of the feature points, and based on the derived amount of deviation, the control unit 120 executes processing such as correcting the eye movement data. is. Specifically, the data processing device 100 is an example of a computer, and for example, a CPU, an FPGA, or the like corresponds to the control unit 120 . The control unit 120 also has a memory such as a RAM for storing images and a ROM for storing programs.

The image processing unit 130 is configured with a GPU (Graphics Processing Unit) or the like, and displays an image of the image data C on the display 300, or superimposes an image showing the contour and center of the pupil of the eyeball on the image of the image data C. and displayed on the display 300.

The data processing device 100 may have a media reader. The media reading device accepts storage media storing various programs and data, and can read programs and data from the storage media. Storage media include CDs (Compact Disks), SD cards (Secure Digital cards), USB memories (Universal Serial Bus memories), and the like. In the present embodiment, the program stored in the storage medium may be read by the media reading device and stored in the memory instead of being stored in the ROM or the like. Of course, data processing apparatus 100 may download the program via a network and store the program in memory.

Next, the data processing device 100 derives the amount of deviation between the head of the subject 2 and the housing 401, and based on the derived amount of deviation, corrects the eye movement data and the like. do. FIG. 5 is a diagram for explaining processing for deriving the amount of deviation between the subject's 2 head and the housing 401 in the eye movement data processing system 10 according to the first embodiment. In the example shown in FIG. 5, based on the N-th frame image (N is a natural number) of the eye of the subject 2 and the N−1-th frame image, the head and housing of the subject 2 401 is derived. The derivation of the amount of deviation between the head of the subject 2 and the housing 401 need not be performed in units of one frame, but may be performed in units of several frames or in units of several tens of frames.

First, the imaging device 400 acquires image data of the eye of the subject 2 in the N−1th frame with the first imaging unit 411 and the second imaging unit 412 (S1). The image data of the eye is generated as image data C by combining the image data A of the first imaging unit 411 and the image data B of the second imaging unit 412 in the (N−1)th frame in the arithmetic processing unit 420, and data processing is performed. sent to the device 100.

The imaging device 400 acquires head angular velocity data from the head sensor 450 at the time of imaging the N−1th frame (S2). The head angular velocity data is generated as data H in the arithmetic processing unit 420 based on the signal of the head sensor 450 acquired at the timing of imaging the N−1th frame, and sent to the data processing device 100 .

The data processing device 100 detects feature points of the subject 2 from the acquired image data (image data C) (S3). When detecting feature points from the image data of the eye of the subject 2, for example, at least one of the inner and outer corners of the eye of the subject 2 is set as the feature point. Concretely, it demonstrates using a figure. FIG. 6 is a diagram showing eye image data processed by the data processing apparatus 100 according to the first embodiment. In FIG. 6, image data C obtained by combining the image data A of the right eye of the subject 2 captured by the first imaging unit 411 and the image data B of the left eye of the subject 2 captured by the second imaging unit 412 is Illustrated.

A process of specifying the inner corner of the eye as the singular point of the left eye of the subject 2 by the data processing device 100 will be described with reference to FIG. 6 . It is assumed that the control unit 120 stores in advance a plurality of human eye inner corner shape patterns. First, when the control unit 120 acquires the image data C of the (N−1)th frame, it detects whether or not there is a shape in the image data C that matches the pre-stored shape pattern of the inner corner of the eye. When the control unit 120 finds a shape that matches the shape pattern of the inner corner of the eye in the image data C, the control unit 120 specifies the inner corner region R as shown in FIG. The control unit 120 outputs the upper left coordinate r of the identified inner corner region R as position data of the inner corner (feature point) of the left eye of the subject 2 . In the image data C, for example, x and y coordinates are set with the upper left coordinates being (0, 0). Note that the coordinates may be set for each of the image data A and the image data B. FIG. For example, the x, y coordinates of image data A are set with the upper left coordinates of image data C as (0, 0), and the x, y coordinates of image data B are set with the upper right coordinates of image data C as (0, 0). set.

Here, the algorithm for specifying the region matching the shape pattern of the inner corner of the eye from the image data C is, for example, calculating the degree of matching (similarity) with the shape pattern for the entire image data C, calculating the degree of matching (similarity) degree) to find the place with the highest degree of matching (similarity). As a method of calculating the degree of matching (similarity), for example, there is SAD (Sum of Absolute Difference). degrees). Note that the algorithm for identifying the region that matches the shape pattern of the inner corner of the eye from the image data C is not limited to this, and any known algorithm can be applied.

Next, the control unit 120 determines the displacement amount between the head of the subject 2 and the housing 401 from the change from the feature point of the previous frame (N-2 frame) to the feature point of the current frame (N-1 frame). is derived (S4). Note that when N=1, there is no previous frame (N−2 frames), so the control unit 120 does not derive the displacement amount between the head of the subject 2 and the housing 401 .

Further, the control unit 120 corrects the position of the eyeballs in the image data C and the head angular velocity data from the derived amount of deviation between the head of the subject 2 and the housing 401 (S5). In the case of N=1, the previous frame (N-2 frames) does not exist, so the control unit 120 cannot derive the displacement amount between the head of the subject 2 and the housing 401. No corrections are made to the eye position and head angular velocity data.

The control unit 120 acquires and stores the eyeball position and head angular velocity data in the corrected image data C (S6). Note that when N=1, there is no previous frame (N-2 frames), so the control unit 120 acquires the eyeball position and head angular velocity data in the first image data C and stores them.

Next, the imaging device 400 acquires an eye image of the subject 2 in the N-th frame with the first imaging unit 411 and the second imaging unit 412 (S11). The image of the eye is generated as image data C by combining the image data A of the first imaging unit 411 and the image data B of the second imaging unit 412 in the N-th frame in the arithmetic processing unit 420, and is sent to the data processing device 100. Sent.

The imaging device 400 acquires head angular velocity data from the head sensor 450 at the time of imaging the Nth frame (S12). The head angular velocity data is generated as data H in the arithmetic processing unit 420 based on the signal of the head sensor 450 acquired at the timing of imaging the Nth frame, and sent to the data processing device 100 .

The data processing device 100 detects feature points of the subject 2 from the acquired image data (image data C) (S13). When detecting feature points from the eye image of the subject 2, for example, at least one of the inner and outer corners of the eye of the subject 2 is set as the feature point. As described with reference to FIG. 6, when the control unit 120 finds a shape that matches the shape pattern of the inner corner of the eye in the image data C, the inner corner region R is specified. The control unit 120 outputs the upper left coordinate r of the identified inner corner region R as position data of the inner corner (feature point) of the left eye of the subject 2 .

Next, the control unit 120 derives the displacement amount between the head of the subject 2 and the housing 401 from the change from the feature point of the previous frame (N−1 frame) to the feature point of the current frame (N frame). (S14). Furthermore, the control unit 120 corrects the position of the eyeballs in the image data C and the head angular velocity data from the derived amount of deviation between the head of the subject 2 and the housing 401 (S15). FIG. 7 is a schematic diagram for explaining the process of correcting the eyeball position and head angular velocity data in the image data C in the data processing apparatus 100 according to the first embodiment. In addition, in FIG. 7, illustration other than the left eye of the subject 2 and the area R is omitted for the sake of easy understanding of the explanation.

FIG. 7A is a diagram for explaining the process of correcting the position of the eyeball in the image data C based on the amount of deviation between the head of the subject 2 and the housing 401. FIG. As shown in FIG. 7A, the control unit 120 sets the coordinate r(N-1) of the N-1th frame to (A, B) and the coordinate r(N) of the N-th frame to (A+a, B+b). ), the displacement amount between the head of the subject 2 and the housing 401 can be derived as (a, b). Based on the derived shift amount (a, b), the control unit 120 converts the coordinates (a, b) of the N-1th frame image data C to the coordinates (0, 0) of the Nth frame image data C. , the position of the eyeball in the image data C is corrected.

FIG. 7B is a diagram for explaining processing for correcting head angular velocity data based on the amount of deviation between the head of subject 2 and housing 401 . As shown in FIG. 7B, the control unit 120 receives the angle θ between the coordinate r(N−1) of the N−1th frame and the coordinate r(N) of the Nth frame at the center of rotation O1. A displacement amount between the head of the examiner 2 and the housing 401 is derived. The control unit 120 obtains a correction value of the angular velocity from the time of one frame and the deviation amount θ, and increases or decreases the correction value with respect to the head angular velocity data. It should be noted that the shift amount θ is positive when clockwise and negative when counterclockwise. Further, in order to correct the head angular velocity data accurately, it is necessary to consider the difference between the rotation center O of the head angular velocity and the rotation center O1.

The control unit 120 further calculates the horizontal eyeball angle (right), the vertical eyeball angle (right), and the eyeball rotation angle (right) from the first image 41, and the horizontal eyeball angle (left) and the vertical eyeball angle (right) from the second image 42. left) and eyeball rotation angle (left) are obtained by calculation. Specifically, the control unit 120 obtains the outline and center position of the pupil of each eyeball detected for each frame of the first image 41 and the second image 42, and calculates the eyeball horizontal angle (right, left), The vertical eyeball angle (right, left) and the eyeball rotation angle (right, left) are calculated. The control unit 120 uses values obtained by correcting the position of the eyeball in the image data C based on the displacement amount to determine the horizontal eyeball angle (right, left), the vertical eyeball angle (right, left), and the eyeball rotation angle (right, left). left) is calculated, the displacement between the subject 2 and the housing 401 can be ignored from the measurement results.

Note that the control unit 120 does not correct the position of the eyeball in the image data C based on the displacement amount, but rather calculates the calculated eyeball horizontal angle (right, left), eyeball vertical angle (right, left), and eyeball rotation angle. (Right, left) may be corrected by applying the deviation amount. Similarly, the displacement between the subject 2 and the housing 401 can be ignored from the measurement results. Here, the eyeball horizontal angle (right, left), the eyeball vertical angle (right, left), and the eyeball rotation angle (right, left) are examples of eye movement data. The eye movement data may include data on at least one movement of the subject's 2 eyeballs in the horizontal direction, the vertical direction, and the rotational direction.

However, when the displacement amount between the head of the subject 2 and the housing 401 is large, the eye movement data cannot be calculated. Therefore, the control unit 120 determines whether or not the derived deviation amount is within the correctable range. may be urged. Specifically, a process for determining whether or not the derived deviation amount is within a correctable range will be described using a flowchart. FIG. 8 is a flow chart for explaining data processing in the data processing device 100 according to the first embodiment.

First, the control unit 120 starts processing the N−1th frame (step S101). The control unit 120 receives image data (image data C) of the eye of the subject 2 in the N−1th frame imaged by the first imaging unit 411 and the second imaging unit 412 (step S102). If the image data (image data C) of the eye of subject 2 has not been input (NO in step S102), the control unit 120 returns the process to step S102 to obtain the image data of the eye of subject 2 ( The state in which the image data C) is input continues.

When image data (image data C) of the eye of subject 2 is input (YES in step S102), control unit 120 identifies the inner corner of eye of subject 2 as a feature point in image data C (step S103). ). The control unit 120 searches for the inner corner region of the subject 2 from the image data C by, for example, image pattern matching, and specifies the region as a feature point.

The control unit 120 stores, in the storage unit, the coordinates of the upper left corner of the region of the inner corner of the eye identified as the feature point as the position of the inner corner of the eye (step S104). The position of the inner corner of the eye may be the upper left coordinate of the inner corner of the eye, and may be the lower right coordinate of the inner corner of the eye, the center coordinates of the inner corner of the eye, the barycentric coordinates of the inner corner of the eye, or the like.

Next, the control unit 120 starts processing the Nth frame (step S105). The control unit 120 receives image data (image data C) of the eye of the subject 2 in the N-th frame imaged by the first imaging unit 411 and the second imaging unit 412 (step S106). If the image data (image data C) of the eye of subject 2 is not input (NO in step S106), the control unit 120 returns the process to step S106 to obtain the image data (image data C) of the eye of subject 2. C) continues to be entered.

When image data (image data C) of the eye of subject 2 is input (YES in step S106), control unit 120 identifies the inner corner of eye of subject 2 as a feature point in image data C (step S107). ). The control unit 120 searches for the inner corner region of the subject 2 from the image data C by, for example, image pattern matching, and specifies the region as a feature point.

The control unit 120 derives the deviation amount (deviation amount of the inner corner of the eye) of the position of the inner corner of the eye specified as the feature point from the N-th frame image with respect to the position of the inner corner of the eye specified as the feature point from the N-1th frame image. (step S108). Note that the deviation amount of the inner corner of the eye may include the deviation amount of the angular velocity in addition to the deviation amount of the x and y coordinates of the image.

The control unit 120 determines whether or not the deviation amount of the inner corner of the eye is equal to or less than a threshold value (step S109). Here, the threshold value is set within a correctable range. For example, the correctable range is a distance at which a part of the pupil or iris pattern of the eyeball of the subject 2 does not fall outside the imaging area.

When it is determined that the amount of deviation of the inner corner of the eye is equal to or less than the threshold (YES in step S109), the control unit 120 corrects the eye movement data based on the amount of deviation of the inner corner of the eye (step S110). Here, the control unit 120 does not correct the eye movement when the deviation amount of the inner corner of the eye is zero. Note that the control unit 120 may correct the position of the eyeball in the image data based on the amount of deviation of the inner corner of the eye, and calculate the eye movement data from the corrected position of the eyeball. Further, when the processing load is heavy due to image processing, the control unit 120 does not correct eye movement data based on the amount of inner corner of the eye displacement in step S110, outputs only the inner corner of the eye displacement amount, and outputs the eye movement data after the examination. may be corrected. Thereby, the control unit 120 can suppress the possibility of adversely affecting the vertigo diagnosis function, which is the main function. In certain cases (for example, when the processing power of the control unit 120 is high), the control unit 120 may correct the eye movement data based on the amount of deviation of the inner corner of the eye in step S110.

The control unit 120 determines whether or not an operation to end the examination has been received from the operator 1 (step S111). The operation to end the examination by the operator 1 includes, for example, an operation of clicking an examination end button displayed on the display 300 with the mouse 502 . If it is determined that an operation to end the examination has not been received (NO in step S111), the control unit 120 updates the N value to N=N+1 (step S112). After step S112, the control unit 120 returns the process to step S106 and continues the inspection. On the other hand, when determining that an operation to end the examination has been received (YES in step S111), the control unit 120 ends the examination.

When determining that the amount of deviation of the inner corner of the eye is greater than the threshold (NO in step S109), the controller 120 notifies the operator 1 of the deviation of the goggles (imaging device 400) (step S113). Specifically, a warning sound indicating that the goggles have deviated from the subject 2 is output from the speaker, or a warning display is displayed on the display 300 . The control unit 120 determines that the goggles are out of alignment with the subject 2 and that correct examination cannot be performed, and after notifying the operator 1 of the goggle deviation, the process ends. Note that the control unit 120 may automatically delete the eye movement data in which the goggles are displaced from the storage device 700 when the examination ends. Conversely, if control unit 120 determines that the derived deviation amount is larger than the threshold value (NO in step S109), control unit 120 stores data indicating that in association with the eye movement data derived at that timing. may be stored in

The imaging device 400 has two imaging units, a first imaging unit 411 and a second imaging unit 412, in a housing 401, and images each eye of the subject 2 individually. Therefore, the control unit 120 derives the displacement amount between the head of the subject 2 and the housing 401 based on the individual eye images captured by the first imaging unit 411 and the second imaging unit 412. can be done. In the above description, it was explained that the control unit 120 derives the displacement amount between the head of the subject 2 and the housing 401 based on the image data B of the left eye of the subject 2. As for the image data A of the right eye of the person 2, it is possible to similarly derive the amount of deviation between the head of the subject 2 and the housing 401 by specifying the inner corner of the right eye as a singular point. As a result, the control unit 120 corrects the eyeball position and head angular velocity data in the image data A based on the amount of deviation in the image data A, and corrects the image data It is possible to correct the eye position in B and the head angular velocity data.

Of course, the control unit 120 identifies the inner corner of the eye from the image of one of the right eye or the left eye of the subject 2 as a singular point, derives the amount of deviation between the head of the subject 2 and the housing 401, The position of the eyeballs in the image data of both eyes of 2 and the data of the head angular velocity may be corrected.

(Embodiment 2)
In the eye movement data processing system 10 according to Embodiment 1, the data processing device 100 searches for the inner corner region of the subject 2 from the image data C by image pattern matching, and specifies the region as a feature point. In the present embodiment, data analysis/learning using AI (Artificial Intelligence) is utilized in order to specify the inner corner region of subject 2, which is a feature point.

The eye movement data processing system according to Embodiment 1 has the same configuration as the configuration of the eye movement data processing system 10 shown in FIGS. do not have. The data processing device 100 has an estimation unit for estimating feature points in an image based on image data captured by the imaging device 400 . Note that the estimation unit can be realized by a program executed by the control unit 120. FIG.

FIG. 9 is a block diagram for explaining processing for estimating feature points using an estimation model in the data processing device according to the second embodiment. Based on the image data C captured by the imaging device 400 and the estimation model 171 including the neural network 172, the estimating unit 140 searches for the inner corner region of the subject 2 and estimates the region as a feature point. The estimation unit 140 outputs image data Ca on which the frame of the estimated inner corner region R is superimposed. Note that the image data C is image data including the eye of the subject 2 . Also, the region R is a rectangular region, and may be rectangular as shown in FIG.

Here, the estimation model 141 includes a neural network 142 and parameters 143 used by the neural network 142 . The parameter 143 includes at least one of a weighting factor used for calculation by the neural network 142 and a judgment value used for estimation judgment.

The neural network 142 is a convolution neural network (CNN: Convolution Neural Network), a recurrent neural network (RNN: Recurrent Neural Network), or an LSTM network (Long Short Term Memory Network) for image recognition by deep learning. A well-known neural network used in processing is applied.

In the learning stage, the estimation model 141 is optimized (adjusted) through learning based on image data including the eye of the subject 2 and the result of identifying the inner corner region. Specifically, FIG. 10 is a block diagram for explaining the learning stage of the estimation model in data processing apparatus 100 according to the second embodiment. As shown in FIG. 10, the estimation model 141, when the image data Cb specifying the inner corner region R1 is input as teacher data, the neural network 142 based on the image data Cb extracts from the image of the eye of the subject 2 Estimate the area of the inner corner of the eye. Then, the estimation model 141 determines whether or not its own estimation result matches the inner corner region R1, which is the correct data associated with the input image data Cb. The parameter 143 is optimized by updating the parameter 143 so that the two match if they do not match while not updating. Note that learning of the estimation model 141 is not limited to the learning stage, and may be performed in the operation stage as well.

The estimation unit 140 estimates the inner corner region R of the subject 2 based on the image data C captured by the imaging device 400 and the estimation model 171 including the neural network 172 . The control unit 120 outputs the identified inner corner region R, that is, the upper left coordinate r of the rectangular region R as the position data of the inner corner (feature point) of the left eye of the subject 2 . As described in Embodiment 1, the control unit 120 detects the head and body of the subject 2 from the change from the feature point of the previous frame (N−1 frame) to the feature point of the current frame (N frame). 401 is derived, and the position of the eyeball in the image data C and data of the head angular velocity are corrected from the derived deviation amount between the head of the subject 2 and the housing 401 .

As a result, the estimating unit 140 can accurately and easily estimate the feature points (the inner corner region R) of the subject 2 based on the image data using so-called AI technology such as a neural network. It is possible to realize effective dental treatment.

The estimating unit 140 is not limited to being provided in the data processing device 100, and may exist in a form of cloud computing such as a server installed outside the space in which the data processing device 100 is installed. In this case, the estimating unit 140 is connected to the data processing device 100 and is also connected to a plurality of data processing devices 100 installed at other locations. Based on this, feature points of the subject 2 may be estimated. By doing so, the frequency of machine learning by the estimating unit 140 is further increased, and the estimating unit 140 can estimate the feature points of the subject 2 with higher accuracy.

[About modification]
In the eye movement data processing system 10 according to Embodiments 1 and 2, the data processing device 100 derives the displacement amount between the head of the subject 2 and the housing 401 from the change in the feature points, and the derived subject 2, the position of the eyeball in the image data C and the angular velocity data of the head are corrected based on the amount of deviation between the head and the housing 401 . However, if the processing capability of the arithmetic processing unit 420 provided in the imaging device 400 is high, the arithmetic processing unit 420 derives the amount of deviation between the head of the subject 2 and the housing 401 from the change in the feature points, and derives The position of the eyeballs in the image data C and the head angular velocity data may be corrected based on the amount of deviation between the head of the subject 2 and the housing 401 .

As a result, the image data C output from the imaging device 400 becomes image data in consideration of the displacement between the subject 2 and the housing 401, and the processing load on the data processing device 100 is reduced.

In the eye movement data processing system 10 according to Embodiments 1 and 2, it has been described that the position data of the inner corner of the eye (feature point) is specified by searching for the rectangular region R including the shape of the inner corner of the eye. , and the position of the end may be specified as the position data of the feature point of the inner corner or the outer corner of the eye. Further, in the eye movement data processing system 10 according to Embodiments 1 and 2, the region R including the inner corner or the outer corner of the eye may be specified not by a rectangle but by another shape (for example, a triangle, a circle, etc.). . For example, if the region R including the inner corner or the outer corner of the eye is specified by a triangle, the barycentric coordinates of the triangle are specified, and if the region R that includes the inner corner or the outer corner of the eye is specified by a circle, the center coordinates of the circle are the positions of the feature points of the inner corner or the outer corner of the eye, respectively. Identify as data.

It should be considered that the embodiments disclosed this time are illustrative in all respects and not restrictive. The scope of the present disclosure is indicated by the scope of claims rather than the above description, and is intended to include all modifications within the scope and meaning of equivalents of the scope of claims.

1 operator, 2 subject, 10 eye movement data processing system, 100 data processing device, 110 I/O interface, 120 control unit, 130 image processing unit, 140 estimation unit, 300 display, 400 imaging device, 401, 401a , 401b housing, 401c eyepiece, 401d opening, 402 light shielding cover, 403 wiring, 410 glass plate, 411 first imaging unit, 412 second imaging unit, 420 arithmetic processing unit, 450 head sensor, 501 keyboard, 502 mouse, 600 stimulus signal processor.

Claims (14)

A data processing device for processing data from an imaging device that images an eye in medical care,
an input unit for inputting image data of the eye of the subject captured by an imaging unit held in a housing of the imaging device mounted on the head of the subject;
and a derivation unit that derives a deviation amount between the subject's head and the housing based on the image data input by the input unit. The derivation unit detects feature points of the subject from image data captured by the imaging unit, and derives a displacement amount between the subject's head and the housing from a movement amount of the feature points. 2. A data processing apparatus according to claim 1. 3. The data processing apparatus according to claim 2, wherein the feature point includes at least one of the inner and outer corners of the eye of the subject. 4. The derivation unit estimates the position of at least one of the inner and outer corners of the eye of the subject based on at least image data captured by the imaging unit and an estimation model generated by machine learning. The data processing device according to . 5. The estimating model according to claim 4, which is machine-learned in advance based on image data including at least one of the inner and outer corners of the eye and position information of at least one of the inner and outer corners of the eye in the image data. Data processing equipment. The two imaging units are held in the housing and individually photograph the eyes of the subject,
6. The derivation unit according to any one of claims 1 to 5, wherein the derivation unit derives a displacement amount between the subject's head and the housing based on image data of individual eyes captured by the imaging unit. 1. The data processing device according to claim 1. The data processing device according to any one of claims 1 to 6, wherein the deriving section corrects the position of the eyeball in the image data captured by the imaging section based on the derived displacement amount. The derivation unit corrects detection data from a detection unit that is held in the housing and detects movement of the subject's head, based on the derived deviation amount. A data processing apparatus according to any one of claims 1 to 3. The derivation unit includes eye movement data including at least one motion data out of a horizontal direction, a vertical direction, and a rotational direction of the eyeball of the subject from a plurality of image data captured in time series by the imaging unit. calculated as data,
The data processing device according to any one of claims 1 to 8, wherein the eye movement data is corrected based on the derived deviation amount. The data processing apparatus according to any one of claims 1 to 9, further comprising a notification unit that notifies the deviation of the housing when the derived deviation amount is greater than or equal to a predetermined threshold value. 10. The method according to claim 9, wherein when the derived amount of deviation becomes equal to or greater than a predetermined threshold value, the derivation unit stores data indicating that effect in the storage unit in association with the eye movement data derived at that timing. Data processing apparatus as described. An eye movement data processing system for processing eye movement data in a balance function test,
an imaging device for imaging an eyeball of a subject;
a data processing device that receives data from the imaging device and processes the data;
The imaging device is
a housing mounted on the subject's head;
an imaging unit that is held by the housing and captures an image of the subject's eye,
The data processing device is
an input unit into which image data of the subject's eye captured by the imaging unit is input;
an eye movement data processing system, comprising: a derivation unit for deriving a deviation amount between the subject's head and the housing based on the image data input by the input unit. A data processing method for processing data from an imaging device for imaging an eye in medicine, comprising:
a step of inputting image data of the eye of the subject captured by an imaging unit held in a housing of the imaging device mounted on the head of the subject;
A data processing method comprising: deriving an amount of deviation between the subject's head and the housing based on input image data. A program executed by a data processing device that processes data from an imaging device that images an eye in medical care,
a step of inputting image data of the eye of the subject captured by an imaging unit held in a housing of the imaging device mounted on the head of the subject;
and deriving an amount of displacement between the subject's head and the housing based on input image data.

Images