JP2000301962A - Eye condition detection device, drowsy driving alarm device - Google Patents

Abstract

(57) [Summary] [Problem] To appropriately judge erroneous recognition of an eye and prevent erroneous alarm due to tracking error. SOLUTION: Image input means CL1 for inputting a face image
And an eye opening detecting means CL7 for detecting an eye opening from image data of the face input by the image input means CL1.
When the eye opening value detected by the eye opening detecting means CL7 does not change for a predetermined time T1, it is determined that the eye position is erroneously recognized.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an eye condition detecting device and a dozing operation warning device for detecting and alerting a dozing condition of a driver of a vehicle, a boat operator, an operator of a plant or the like.

[0002]

2. Description of the Related Art An apparatus of this type using conventional image processing is disclosed, for example, in Japanese Patent Application Laid-Open No. 10-143669. In this configuration, the position of the eye is detected with respect to the grayscale image, and the eye is tracked while performing the open / closed eye determination based on the eye opening value.

[0003]

However, in the conventional apparatus, it is determined whether the eye position is erroneously detected by detecting and holding the maximum value and the minimum value as the eye opening value data. Judgment was made based on whether an out-of-range value was output. Therefore, if an eye whose opening degree value is substantially constant other than the eye is recognized as an eye, it cannot be determined that the eye position detection is erroneous recognition. If a value other than the opening value data is output due to the influence of a shadow or the like, it is determined that the eye is erroneously recognized.

Further, since the eye to be detected is specified only by a change in the density of the image in the tracking area, noises around the eyes when the face is moved greatly and fast or when glasses are worn may occur. In many cases, when the detection target is switched from an eye to an eyebrow or eyeglasses frame, if the eyebrows are thin or the eyeglasses frame is thin, the opening value becomes small and a false alarm is generated.

Accordingly, an object of the present invention is to provide an eye condition detection device and a drowsy driving alarm device that can appropriately judge eye misidentification. It is another object of the present invention to provide a drowsy driving alarm device that reduces a false alarm due to a tracking error or the like by determining a movement state of an eye position and limiting an alarm output.

[0006]

According to a first aspect of the present invention, there is provided an image input means for inputting a face image, and an eye opening for detecting an eye open degree from face image data input by the image input means. Detecting means for erroneously recognizing the position of the eye when the eye opening value detected by the eye opening detecting means does not change for a predetermined time T1. .

According to a second aspect of the present invention, there is provided the eye state detecting apparatus according to the first aspect, wherein the open / closed state of the eye is determined based on a change in the eye opening value output from the eye opening detecting means. And a wakefulness determining means for determining the wakefulness from the change of the open / closed state.

[0008] The invention of claim 3 is claim 1 or claim 2.
An eye position detecting device for detecting eye position coordinates from face image data input by the image input device, wherein the eye position coordinates move by a predetermined value or more. Is characterized in that the predetermined time T1 is shortened.

According to a fourth aspect of the present invention, there is provided the eye condition detecting apparatus according to the second aspect, wherein the eye position detecting means for detecting the position coordinates of the eye from the face image data inputted by the image input means; A warning means for issuing a warning when the arousal level is determined to be low by the arousal level determination means; and a drowsiness warning for a predetermined time T2 when the position coordinates of the eyes move by a predetermined value or more. Alarm output inhibiting means for inhibiting output.

According to a fifth aspect of the present invention, there is provided the eye condition detecting apparatus according to the third aspect, wherein an alarm is issued when the arousal level determination means determines that the arousal level is decreasing. An alarm output prohibiting unit that prohibits a drowsiness alarm output for a predetermined time T2 when the eye position coordinates move by a predetermined value or more is provided.

The invention of claim 6 is the invention of claim 4 or claim 5.
The drowsiness driving warning device according to the above, wherein the predetermined time T2 for prohibiting the output of the drowsiness warning is longer than the predetermined time T1 for determining erroneous recognition of eyes.

A seventh aspect of the present invention is the drowsy driving alarm device according to any one of the fourth to sixth aspects, wherein the alarm output prohibiting means sets the predetermined time T2 after the position coordinate of the eye moves by a predetermined value or more. If an eye is detected within an allowable area based on the position before the movement within the period, the prohibition of the drowsiness alarm output is released.

According to an eighth aspect of the present invention, there is provided the drowsy driving warning device according to any one of the third to seventh aspects, wherein when the eye position detecting means detects the position coordinates of the eye, The drowsiness alarm output by the alarm output inhibition means is inhibited only for a predetermined time T3.

[0014]

According to the first aspect of the present invention, if the data to be detected is an eye, there is a change in the eye opening value within a predetermined time T1. Since there is no change in the opening value of the eye, it is possible to appropriately judge the erroneous recognition of the eye and prevent erroneous detection of the eye position.

According to the second aspect of the invention, in addition to the effect of the first aspect, the degree of arousal can be determined by the eyes to be detected.

According to a third aspect of the present invention, in addition to the effects of the first or second aspect of the present invention, when the position coordinate of the eye moves by a predetermined value or more, the detection target is switched from the eye to something other than the eye. There is a possibility that the time T1 for checking a state in which there is no change in the degree of opening of the eyes can be shortened by setting a short time T1 to keep tracking the wrong target.

According to a fourth aspect of the present invention, in addition to the effect of the second aspect, the position coordinates of the eye move by a predetermined value or more, and the detection target is switched from the eye to something other than the eye. If there is only a vertical width equivalent to the closed eyes, the drowsiness alarm output is prohibited during the predetermined time T2, and this time is a grace period for making an appropriate determination, and an error caused by recognizing that the eyes are closed is generated. Alarms can be prevented.

According to a fifth aspect of the present invention, in addition to the effect of the third aspect, the position coordinates of the eye move by a predetermined value or more, and the detection target is switched from the eye to something other than the eye. If there is only a vertical width equivalent to the closed eyes, the drowsiness alarm output is prohibited during the predetermined time T2, and this time is a grace period for making an appropriate determination, and an error caused by recognizing that the eyes are closed is generated. Warnings can be reduced.

According to the invention of claim 6, in addition to the effect of the invention of claim 4 or 5, the position coordinates of the eye move by a predetermined value or more, and the detection target is switched from the eye to something other than the eye. If an object other than the eye has only a vertical width equivalent to the closed eye, the drowsy alarm output is prohibited for a predetermined time T2 longer than the predetermined time T1 for determining erroneous recognition of the eye. False alarms caused by recognition can be reliably prevented.

According to the seventh aspect of the present invention, the fourth to sixth aspects are provided.
In addition to the effects of the invention, in a vehicle in which a face moves up and down violently like a truck, the position coordinates of the eye may move beyond a predetermined value even if the eye is accurately tracked. However, in such a case, by detecting that the detection target has returned to the allowable region within the predetermined time T2, a reliable alarm can be issued.

According to the invention of claim 8, claims 3 to 7 are provided.
In addition to the effects of the present invention, it is possible to reduce false alarms that occur when the eye position detecting means erroneously detects an eye and the target of the erroneous detection is only a vertical width equivalent to closing the eye.

[0022]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present apparatus can be used as a drowsy driving alarm for operators of railway vehicles, ships, plants and the like in addition to automobiles, but this embodiment will be described as applied to automobiles.

FIG. 1 is a functional block diagram of an eye condition detecting device and a drowsy driving alarm device to which one embodiment of the present invention is applied. The device includes an image input means CL1, a density detecting means CL2, Point extracting means CL3, continuous data extracting means CL4, eye position detecting means CL5,
Intra-region point extraction means CL6 and eye opening detection means C
L7, backup means CL8, and alertness determination means C
L9, alarm output prohibiting means CL10, and alarm means CL1
1 is provided.

The image input means CL1 inputs a face image. The density detecting means CL2 is provided with the image input means C.
L1 detects a local increase in density in a vertical pixel row of the input face image data. The intra-image point extracting means CL3 constitutes a point extracting means in the present embodiment, and determines a pixel as an extraction point based on the increase in the density detected by the density detecting means CL2 and its change state. The continuous data extracting means CL4 extracts continuous data extending in the horizontal direction of the face when the extraction points extracted by the intra-image point extracting means CL3 approach each other in adjacent pixel columns.

The eye position detecting means CL5 selects (detects) an eye from the continuous data extracted by the continuous data extracting means. The area point extracting means CL6
Specifies a predetermined area including the eye based on the position of the eye detected by the eye position detection means CL5, sets this area as a tracking area, and increases the density in the vertical direction in the tracking area and A pixel is determined from the change state and is set as an extraction point. The eye opening detection means CL7 extracts the continuous data extending in the horizontal direction of the face when the extraction points extracted by the intra-area point extraction means CL6 approach each other in adjacent pixel rows, thereby obtaining details of the eyes. Then, a range in which the density value is read out is determined by detecting a proper position, and the opening degree of the eye is detected from a density change state within the range. The eye opening detecting means C
L7 can cope with a change in the position of the eye including the outside of the tracking region by setting a processing region in the image to be captured next based on the detailed position of the continuous data appearing in the predetermined region. Is configured to be detected.
Therefore, the intra-region point extracting means CL6 and the eye opening detecting means CL7 constitute an eye opening detecting means in the present embodiment.

The continuous data detected by the in-region point extracting means CL6 is based on the in-image point extracting means CL3.
Although it is finer and more accurate than the continuous data detected by (1) and (2), the latter can be detected with the same roughness as the former, so that the processing can be accelerated.

The backup means CL8 checks the change of the eye opening value output from the eye opening detecting means CL7, and the time series change of the eye opening value exceeds the predetermined time T1. If this is not the case, the process returns to the eye position detection processing by the eye position detection means CL5 again.
That is, the eye position detection is performed again.

The wakefulness determination means CL9 determines the wakefulness from the change in the open / closed state of the eyes by the eye openness detection means CL7. The alarm output prohibiting unit CL10 limits the alarm output for a predetermined time T2 by determining the moving state of the eye. The alarm means CL11 issues an alarm such as a buzzer when the alertness determination means CL9 determines that the alertness is decreasing.

FIG. 2 is a configuration block diagram according to an embodiment of the present invention. In FIG. 2, the image input means CL
The TV camera 21 as 1 is installed in the instrument of the automobile, and photographs the driver's face from the front. In the present embodiment, the input image of the TV camera 21 is composed of 512 pixels in the horizontal direction (X) and 480 pixels in the vertical direction (Y) as shown in FIG.

The input image taken by the TV camera 21 is stored in the image memory 23 as digital image data via the A / D converter 22. The output of the image memory 23 is input to the image data operation circuit 24.

The image data calculation circuit 24 detects the density of the vertical pixel row of the face based on the input image data.
A pixel is defined as an extraction point based on the detected increase in the density of the pixel row and its change state, and the extracted points adjacent to each other in the pixel row direction of the adjacent pixel row are successively extracted. Extract continuous data extending horizontally. That is, the image data calculation circuit 24 includes the density detecting means CL2 and the in-image point extracting means CL.
3 and the continuous data extracting means CL4.
The output of the image data calculation circuit 24 is the eye position detection circuit 2
5 is input.

The eye position detecting circuit 25 detects an eye position by selecting an eye from the continuous data, and constitutes the eye position detecting means CL5 in the present embodiment. The output of the eye position detection circuit 25 is the open / closed eye detection circuit 2
6 is input.

The open / closed eye detection circuit 26 specifies a predetermined area including the eye based on the position of the eye detected by the eye position detection circuit 25, and sets this area as a tracking area.
A pixel is determined as an extraction point from the increase in the density in the vertical direction and its change state in the tracking area, and the extraction points adjacent to each other in the pixel row direction of the adjacent pixel row are consecutively connected. By extracting continuous data extending in the lateral direction of the face, a detailed position of the eyes is detected, a range from which a density value is read out is specified, and an opening degree of the eye is detected from a density change state within the range. Also, the open / closed eye detection circuit 26
Can respond to changes in the eye position, including outside the tracking area, by setting the processing area in the next captured image based on the detailed position of the continuous data appearing in the predetermined area. It is configured to be able to detect. That is, the open / closed eye detection circuit 26 constitutes the intra-region point extraction means CL6 and the eye opening degree detection means CL7 in the present embodiment. The output of the open / closed eye detection circuit 26 is input to the backup circuit 27.

The backup circuit 27 checks the change in the eye opening value output from the open / closed eye detection circuit 26, and if the change in the time series of the eye opening value exceeds a predetermined time T1, If there is not, the process returns to the eye position detection processing by the eye position detection circuit 25 again, and the backup means CL8 is configured in the present embodiment.

The arousal level determination circuit 28 determines the arousal level from the change in the open / closed state of the eye output from the open / closed eye detection circuit 26, and in the present embodiment, the arousal level determination means CL9
Is composed.

The alarm output limiting circuit 29 limits the alarm output for a predetermined time T2 by judging the state of movement of the eyes.
Is composed.

The alarm device 30 is provided with the wakefulness determination circuit 28
When it is determined that the arousal level is decreasing, a warning to alert the driver is issued, and in the present embodiment, the warning means CL11 is configured.

Next, the flow of operation in the above configuration will be described with reference to the flowcharts of FIGS. First,
In step 301, the face portion is photographed by the TV camera 21 (shown in FIG. 2), and in step 302, the input image for one frame is converted into the A / D converter 22 (shown in FIG. 2).
Are converted into digital signals and stored in the image memory 23 (shown in FIG. 2).

Next, in step 303, it is checked whether or not an eye tracking area has been set. The eye tracking area refers to a predetermined area including the eye, and by detecting the position of continuous data appearing in this area, the detailed position of the eye is recognized, and the density data at the specified position is read. To detect the degree of eye opening. Further, by setting a processing area in the next image to be captured based on the detailed position of the continuous data appearing in the predetermined area, it is possible to cope with a change in the eye position including outside the tracking area.

If the eye tracking area has not been set,
In steps 304 and 305, the position of the eye is detected, and based on this, the width in the horizontal direction (X direction) and the width in the vertical direction (Y direction) to be the eye tracking areas are set. Details of the eye position detection are shown in the flowchart of FIG.
This will be described later with reference to the explanatory diagrams shown in FIGS. Step 3
If it is determined that the tracking area of the eye is set in 03, the opening degree of the eye and the detailed position detection are performed in step 306. Details of this processing are shown in the flowchart of FIG.
This will be described later with reference to the explanatory diagrams shown in FIGS.

Thereafter, in step 307, by comparing the change in the eye opening value detected in the previous step with the value in the previous frame, it is determined whether or not a state corresponding to the change in eye opening has occurred. Is determined. When the process enters step 307 for the first time immediately after the start of the program, since the value in the previous image frame is not stored, it is determined that there is no change, and the process proceeds to step 308. In step 308, it is determined whether or not a state in which the opening degree value has not changed has continued for a predetermined time T1 or more. If the state has not exceeded the predetermined time T1, the process proceeds to step 401 in FIG. The predetermined time T1 is set at a timing at which blinking is considered to occur once.

If it is determined in step 308 that the time is equal to or longer than the predetermined time T1, since there is no change in the opening degree due to blinking, it is determined that the data to be detected is not the eye, and in step 309, the eye tracking area is determined. Clear step 30
The processing returns to 1 and the eye position detection processing is performed again. Here, if the data to be detected is an eye, there is a chronological change in the eye opening value within a predetermined time T1, but if the data to be detected is not an eye, the change is within a predetermined time T1. Because there is no change in the time series of the eye opening value, it is possible to properly determine the erroneous recognition of the eye, and if it is determined that the erroneous recognition of the eye position is performed, Since the eye position detection process is performed again, erroneous detection of the eye position can be prevented.

If it is determined in step 307 that there is a change in the opening degree of the eye, or if there is no change in the opening degree value, step 308 is executed.
If it is determined that the predetermined time T1 has not been exceeded, the process proceeds to step 401 in FIG. In step 401,
An open / closed eye is determined by using a reference value that can be set by learning a change range of the opening degree value. In step 402, the movement amount of the tracking (detection) target is determined using the eye position coordinates calculated in step 306 of FIG. 3 together with the eye opening value. At this time, it is also confirmed that the alarm restriction flag is OFF in order to control the timer for restricting the alarm. If it is not determined in step 402 that the position coordinates of the eye to be tracked have moved by a predetermined value or more, the process proceeds to step 404.

If it is determined in step 402 that the position coordinates of the eye to be tracked have moved by a predetermined value or more, the process proceeds to step 403, where the alarm limit timer is started and the alarm limit timer is turned on. After that, the process proceeds to step 404. In step 404, it is determined whether or not the alarm limit timer for controlling the alarm limit time has exceeded the predetermined time T2. Only when the alarm limit timer is in the activated state and has exceeded the predetermined time T2, the process proceeds to step 405. Then, the alarm restriction flag is turned off. Details of this processing will be described later with reference to the explanatory diagrams shown in FIGS.

In step 406, the tracking area of the eye is updated, and in step 407, the arousal level is determined based on the appearance pattern of the closed-eye output. In step 408, the ON / OFF state of the alarm restriction flag is confirmed. The result of the determination that the alarm restriction flag is OFF and the eye closed in step 401 is continuously output, and it is determined that a long eye closed has occurred. In the case, a warning to alert the driver is issued in step 409.
Fire at. Thereafter, the process returns to step 301 in FIG. 3 to input the next image, and the same processing is continued.

Next, details of the eye position detection will be described.

The flow of the eye position detecting process will be described with reference to the flowchart of FIG. First, in step 501, point extraction processing is performed on data on a line in the Y-axis direction as shown in FIG. 6, and after one line is completed, processing is shifted to the next adjacent line, and all lines in a predetermined direction are processed. It is determined whether or not point extraction has been completed. Step 5
If it is determined in 01 that point extraction has not been performed on all lines, the process proceeds to step 502. In this step 502, arithmetic mean calculation of density values of one line in a predetermined direction is performed. This processing is intended to eliminate small variations in the change in density value at the time of capturing image data, and to capture a global change in density value. FIG.
FIG. 6A shows a processing result of the arithmetic averaging operation of the line data of Xa in FIG.

In step 503 of FIG.
The differential operation is performed on the arithmetic average value that is the operation result of Step 2. Point extraction is performed using the differential value that is the result of this processing. The method for extracting the points is as follows: points (p1 to p5) at which the differential value changes from negative to positive;
Extract points where the graph is convex downward. next,
Changes in density value until reaching that point (q1-q
It is determined whether or not 5) is equal to or less than a predetermined value, and whether or not it falls within a gray portion in FIG. 7B. A3) is extracted.

After this processing is completed for one line, step 50
At 5, the processing is switched to the processing of the next line. During the repetition of this processing, for example, in the case of a line like Xb shown in FIG. 6, there is a case where there is no extraction point as can be seen from FIGS. 8 (a) and 8 (b).

When it is determined in step 501 that point extraction for all lines has been completed, points as shown in FIG. 9 are extracted. That is, on the Xc line in FIG.
Two points 1, 1 and A2 are extracted, and four points A1, A2, A3, and A4 are extracted on the Xd line.

Thereafter, the flow shifts to step 506, where Y of the extraction points (A1, A2, A3...) Of each adjacent line is determined.
The coordinate values are compared, and if the Y coordinate value is within a predetermined value, the continuous data group number, continuous start line number, and continuous data number are stored as continuous data. This specific processing will be described with reference to FIG. On line 1,
There are two extraction points whose Y coordinate values are 192 and 229. At the point where the Y coordinate value of line 1 is 192, there is no line on the left side, and there is no continuous data at this stage, so the group number of the continuous data is "1".
In addition, since there is no continuous data at this stage at the point where the Y coordinate value is 229 for the same reason, the group number of the continuous data is set to “2”.

Next, the Y coordinate value 191 of the line 2 on the right
Are the Y coordinate values 192 and 1 of the line 1 on the left
Since the point is within 0, the group number of the continuous data is set to “1”. At this time, the number of continuous data is two. When the same determination is performed at the point of the Y coordinate value 224 of the line 2, the group number of the continuous data is “2” and the number of continuous data is 2.

At the point of the Y coordinate value 360 of the next line 3, there is no point within 360 and 10 on the line 2 on the left, so the group number of the continuous data is "3" and the number of continuous data is 1 Becomes

The continuous start line number in step 506 is a line number having a point at which the number of continuous data is determined to be one.

At step 506, the continuity of the points on each line is determined until the processing is completed for all the lines, and the process proceeds to step 507.

In step 507, the average value of the Y coordinate values of the points having the same continuous data group number is stored in the average value of the continuous points. This value can be used as a representative Y coordinate value of the group. Further, a continuous end line is obtained from the continuous start line and the number of continuous data, and an average value of the continuous start line and the continuous end line is stored. This value can be used as the representative X coordinate value of the group.

At step 508, each continuous group data obtained in this way is used to calculate the length of each continuous group,
By determining based on (X, Y) coordinate values, the position of the eye can be specified (detected).

Here, a specific method of detecting the position of the eye will be described with reference to FIG.

First, considering the characteristic amount of the eye, it can be defined as a horizontally long and upwardly convex arcuate shape. Based on this definition, if the continuous data is narrowed down, the eye becomes horizontal. , The difference between the Y coordinate values of the continuous start point and the continuous end point can be narrowed down to small continuous data, based on the condition that the number of continuous points continues for 5 or more points, and the condition of an arc shape. When narrowing down the continuous data based on this determination, FIG.
Groups G1 to G6 as shown in FIG.

Next, considering the positions of the representative coordinate values of X and Y of each group described above, as shown in FIG.
From the proximity in the X coordinate direction, ZONE: L, ZON
It can be classified into E: C and ZONE: R. This is because the left eye and the left eyebrow do not greatly separate in the X coordinate direction, and the right eye and the right eyebrow do not greatly separate in the X coordinate direction. Also, continuous data due to the shadow under the nose,
Mouth continuous data is located near the center.

As described above, the data is further classified based on the degree of approach in the X coordinate direction, and the position of the eye can be easily detected by narrowing down the data. The continuous data included in ZONE: L is the left eye and the left eyebrow, and ZONE: R
When the continuous data included in the image data is determined to be the right eye and the right eyebrow, the positions of the eyes are G3 and G4, and the coordinate values thereof can be specified. In this way, eye position detection can be performed for both eyes, but for the purpose of dozing detection, it is assumed that no driver will sleep with only one eye closed. Eye detection is limited to one (left eye). The reason for limiting to one eye (left eye) of this driver is not only to save the calculation processing time, but also in the case of a right-hand drive vehicle, the right eye, which may be exposed to direct light, has strong light. This is because the shape of the eye may be difficult to catch.

Next, the details of the eye opening detection in step 306 in FIG. 3 will be described.

FIG. 1 shows a method for detecting the eye opening value.
As shown in FIG. 4, the point Q where the density change from the white part of the skin to the black part of the eye is maximum, and the point R where the density change from the black part of the eye to the white part of the skin is maximum. , And a method of setting a binarization threshold, which will be described below with reference to the flowchart of FIG. 12, to obtain the vertical width of the black part of the eye.

Now, the details of the eye opening detection process by setting the binarization threshold will be described.

First, the flow of a binarization threshold setting method for converting into a binarized image for which the opening is detected will be described with reference to the flowchart of FIG. First, step 12
In FIG. 13A, point extraction processing is performed on the data on the line in the Y-axis direction as shown in FIG. 13A, and after one line is completed, the processing shifts to the processing of the next adjacent line. It is determined whether or not the point extraction has been completed. If it is determined in step 1201 that point extraction has not been performed on all lines, step 120
Move to 2.

In this step 1202, 1 in the predetermined direction
The arithmetic operation of the density value of the line is performed. This processing is intended to eliminate small variations in the change in density value at the time of capturing image data, and to capture a global change in density value. FIG. 13B shows the processing result of the arithmetic averaging operation of the line data of Xa in FIG. FIG.
In step 1203, a differential operation is performed on the arithmetic mean value that is the operation result of step 1202. This processing result is shown in FIG. In step 1204 of FIG. 12, point extraction is performed using the differential value that is the result of the operation in step 1203. In this point extraction method, a point P at which the differential value changes from negative to positive, that is, a point at which the graph becomes convex to the left in FIG. 13B is extracted. Next, the change in the density value before and after the point is equal to or less than a predetermined value,
It is determined whether or not this is the case, and whether or not it is within the gray portion in FIG. 13C, and a point P having a change in density value that satisfies the above condition is extracted.

If it is determined in step 1205 that the extraction point P exists in the Xa column as shown in FIG. 13A, the process proceeds to step 1206, where the differential values before and after the point P are the maximum and minimum values. The density value of the Y coordinate of the Q point and the R point is N
al (density value of Y coordinate at which differential value becomes minimum) and Nah
The density value of the Y coordinate at which the differential value becomes the maximum is stored, and the process proceeds to step 1207.

At step 1207, the processing is switched to the processing for the next line, and the same processing is repeated until it is confirmed at step 1201 that the processing for all the lines has been completed. That is, FIG.
As shown in FIG. 4A, in the Xb column after the Xa column, the density values of the Y coordinate of the Q point and the R point which are the maximum and minimum values of the differential value before and after the extraction point P in the Xb column are represented by Nbl and Nbh. Go to memory as.

The minimum differential value N_l of the density change before the extraction point stored in each line is a portion where the density value changes most from a bright portion to a dark portion, and a portion darker than this density value, that is, It can be said that it is the density value of the portion that hits the eye toward the extraction point P. Also,
The maximum differential value of the density change after the extraction point stored in each line, N_h, is a portion where the density value changes most from a dark portion to a bright portion, and a portion where the brightness from the extraction point P to this density value becomes brighter. It can be said that the density value hits the eyes. Therefore, the values of N_l and N_h are used as threshold setting information for reliably converting the eye part to a black pixel (0) and the skin part around the eye to a white pixel (1) in the binarization processing. Can be.

If it is confirmed in step 1201 that the processing of all the lines has been completed, the process proceeds to step 1208, and a binarization threshold is set from the information on the density values (N_l, N_h) of the extraction points of each line. The following method can be considered for setting the binarization threshold.

First, the density N_ of the minimum differential value of each line
The minimum value of 1 is updated and set based on that value.

Second, the density N_ of the maximum differential value of each line
The maximum value of h is updated and set based on that value.

Third, the density N_ of the minimum differential value of each line
Set based on the average value of l.

Fourth, the density N_ of the maximum differential value of each line
h is set based on the average value.

Fifth, the average value of all N_l and N_h of each line is set as a reference.

Sixth, the gradation up amount is corrected and set using the level of the differential value with reference to the continuous data of the eyes and the brightest density value at point P of each line to be extracted. (If the level of the differential value is large, the tone added to the brightest density value at point P is set large, and if the level of the differential value is small, the tone added to the brightest density value at point P is set small. Next, a method of detecting the eye opening will be described with reference to FIG.

Using the binarization threshold value obtained by the above-described method, the binarization process is performed by further limiting the area in the range where continuous data has appeared. The binarized image is as shown in FIG. 15 when the detection target person is normal and when he or she is dozing.
At this time, when the maximum number of consecutive continuations of the eye portion converted into black pixels in the vertical direction is counted, the value increases in a normal state (opened eyes) and decreases in a dozing state (closed eyes). In this way, the eye opening is detected.

Next, step 4 in the flowchart of FIG.
Details of the eye tracking method in 06 will be described with reference to FIG. In the initial state of this processing, that is, in the first frame,
Naturally, since the eye tracking area E is not set, the eye tracking area E is set in steps 304 and 305 in FIG. At this time, the center coordinates of the eye and the center coordinates of the tracking area E of the eye match as shown in FIG. After the end of the series of processes, the process proceeds to the process of the second frame, and step 303
When the process proceeds to step 306, since the eye tracking region E has already been set, the process proceeds to step 306, where the eye opening and the position are detected. At this time, the detected positions of the eyes are as shown in FIG. The tracking area E of the eye in FIG.
Since the current eye position is the image data captured in the second frame while the position is set in the frame, the center point of the eye is located at the center of the eye tracking area E due to the movement of the face or the like. The gap is coming. However, as long as the eye is within the eye tracking area E, the eye opening and the detailed eye position detection can be performed.

At step 406, the tracking area E of the eye is updated based on the center coordinates of the eye calculated from the continuous data of FIG. Therefore, unless the eye moves extremely fast,
The tracking area E of the eye can be made to follow the movement. FIGS. 16C and 16D show the positional relationship between the eye position and the eye tracking area E in the face image data captured in the third and fourth frames.

Next, the function of limiting the alarm output by determining the movement state of the eye position and reducing false alarms will be described with reference to steps 402 to 405 in FIG.

An example in which the tracking coordinates (position coordinates) of the eye move by a predetermined value or more and the detection target is switched from the eye to the frame of glasses will be described with reference to FIG. When the person to be imaged wears glasses as in FIG. 17A, FIG.
As shown in FIG. 7, since there are many data such as eyeglass frames that become noise around the eyes, when the face moves quickly upward, the continuous data 1 of the eyes is included in the eye tracking area E.
Goes out of the tracking area E, and the continuous data 2 of the frame of the glasses enters instead. At this time, the tracking area E of the eye may be pulled by the continuous data 2 of the frame of the eyeglasses and the target data may be switched. In such a state, the frame of the eyeglasses has a vertical width equivalent to that of the closed eyes, so that it is recognized that the eyes are closed and a false alarm is issued.

In order to solve this, in step 402, it is detected whether or not the tracking coordinates (position coordinates) of the eye have moved by a predetermined value or more.
At 3, the alarm limit timer is started and the alarm limit flag is turned on. Then, at step 404, it is determined whether or not the alarm limit timer has exceeded a predetermined time T2.
If the number does not exceed 2, the alarm is limited in step 408. When the time exceeds the predetermined time T2, the alarm restriction flag is turned off at step 405 to return to the alarm enabled mode.

That is, if the position coordinates of the eye move by a predetermined value or more and the object to be detected is switched from the eye to an object other than the eye, and the object other than the replaced eye has a vertical width equivalent to the closed eye, the predetermined time T2 Since the falling asleep alarm output is limited during this period, this time becomes a grace period for proper tracking, and it is possible to reduce false alarms caused by erroneously recognizing that the eyes are closed due to an eye tracking error. Further, if the predetermined time T2 for restricting the alarm is set longer than the predetermined time T1 for determining the erroneous recognition of the eye, as shown in FIG. 18, the predetermined time T2 is longer than the predetermined time T1 for determining the erroneous recognition of the eye. , The output of the drowsiness alarm is limited, so that it is possible to reliably prevent an erroneous alarm caused by erroneously recognizing that the eyes are closed due to a mistake in tracking the eyes.

Further, when the position coordinates of the eye output from the open / closed eye detection circuit 26 move by a predetermined value or more, misrecognition of the eyes is determined as compared with the case where the position coordinates of the eye do not move by a predetermined value or more. If the predetermined time T1 to be set is set to be a short time T1 ', as shown in FIG. 19, it is possible to reduce the time for keeping track of an erroneous target.

Next, as in a vehicle in which the face frequently moves up and down like a truck frequently, even if the eyes are tracked correctly, the amount of movement of the eye position coordinates becomes a predetermined value or more. Cases occur. In this case, if the alarm is restricted in the same manner as described above, a situation may occur in which the alarm for preventing falling asleep is not issued even though the eye is correctly tracked.

In order to solve this problem, the alarm output limiting circuit 29 determines whether the position of the eye output from the open / closed eye detection circuit 26 has moved by a predetermined value or more and the position before the movement by the predetermined value or more. Set the reference allowable area. And
If the position coordinates of the eye move by a predetermined value or more, the alarm restriction flag is turned on and the alarm restriction timer is started. However, if the detection target returns to the allowable range during the predetermined time T2, the dozing alarm output Is configured to remove the restriction.

That is, as shown in FIG. 20 (a), when the first moving point A in which the eye position coordinate exceeds a predetermined value occurs, the eye movement is not recognized based on the point O before the movement. An area E1 is set. Thereafter, when the tracking target enters the allowable area E1 as indicated by C while counting up to the predetermined time T2 of the alarm restriction, the alarm restriction flag is immediately turned off. FIG. 20 shows that the tracking target has really been replaced by doing so.
It can be distinguished from the case of (b).

Finally, a method for preventing an erroneous alarm from being erroneously detected when the position of the eye is detected and an opening value of the erroneously detected image data is small will be described with reference to the flowcharts of FIGS. 21 and 22. explain. In the flowcharts of FIGS. 21 and 22, description of the same configuration steps as those in the flowcharts of FIGS. 3 and 4 will be omitted, and only different configuration steps will be described.

That is, after the position of the eye is detected in step 2104 of FIG.
N and the alarm limit timer 2 is started. Step 2
At 107, it is determined whether or not the alarm limit timer 2 has exceeded a predetermined time T3. During the predetermined time T3, no alarm is issued in step 2208 of FIG. If the predetermined time T3 has been exceeded, the alarm restriction flag 2 is set to O in step 2108.
Return to the alarm enabled state as FF. In other words, since the alarm restriction is imposed immediately after the detection of the eye position, the eye is erroneously detected,
It is possible to prevent an erroneous alarm that is issued when the opening value of the erroneously detected image data is small.

[Brief description of the drawings]

FIG. 1 is a functional block diagram according to an embodiment of the present invention.

FIG. 2 is a configuration block diagram according to an embodiment of the present invention.

FIG. 3 is a flowchart showing an overall operation.

FIG. 4 is a flowchart showing an overall operation.

FIG. 5 is a flowchart illustrating an operation of detecting an eye position.

FIG. 6 is an explanatory diagram related to eye position detection.

FIGS. 7A and 7B are explanatory diagrams relating to eye position detection, respectively.

FIGS. 8A and 8B are explanatory diagrams relating to eye position detection, respectively.

FIG. 9 is an explanatory diagram related to eye position detection.

FIG. 10 is an explanatory diagram relating to eye position detection.

FIGS. 11A and 11B are explanatory diagrams relating to eye position detection.

FIG. 12 is a flowchart illustrating a method for calculating an eye binarization threshold value for obtaining an eye opening value.

FIGS. 13A, 13B, and 13C are explanatory diagrams each showing a method of obtaining a binarization threshold.

FIGS. 14A, 14B, and 14C are explanatory diagrams each showing a method of obtaining a binarization threshold.

FIG. 15 is a diagram illustrating a method of outputting an eye opening value.

FIGS. 16 (a), (b), (c) and (d) are explanatory diagrams relating to an eye tracking method.

FIGS. 17A and 17B are explanatory diagrams of an example in which a false alarm is generated due to an eye tracking error. FIG.

FIG. 18 is an explanatory diagram relating to an alarm limiting method.

FIG. 19 is an explanatory diagram relating to an alarm limiting method.

FIGS. 20 (a) and (b) are explanatory diagrams relating to the alarm limiting method, respectively.

FIG. 21 is a flowchart illustrating a method for preventing a false alarm due to a false detection of an eye position.

[Explanation of symbols]

CL1 Image input means CL2 Density detecting means CL3 Point extracting means in image (point extracting means) CL4 Continuous data extracting means CL5 Eye position detecting means CL6 Point extracting means in area (eye opening detecting means) CL7 Eye opening detecting Means CL8 Backup Means CL9 Arousal Determining Means CL10 Warning Output Prohibiting Means CL11 Warning Means 21 TV Camera (Image Input Means) 24 Image Data Calculation Circuit (Density Detection Means, Point Extraction Means, Continuous Data Extraction Means) 25 Eye Position Detection Circuit (Eye position detection means) 26 Open / closed eye detection circuit (Eye opening degree detection means) 27 Backup circuit (Backup means) 28 Arousal level determination circuit (Arousal level determination means) 29 Alarm output restriction circuit (Alarm output inhibition means) 30 Alarm circuit (alarm means)

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G06T 7/20 G06F 15/62 380 9A001 G08B 21/06 15/70 405 (72) Inventor Takeshi Fukuda Ibaraki 31 Ohira, Tone-cho, Kitasoma-gun Niles Department Co., Ltd. (72) Inventor Masayuki Kanada 2 Takara-cho, Kanagawa-ku, Yokohama-shi, Kanagawa Nissan Motor Co., Ltd. Address F-term in Nissan Motor Co., Ltd. (Reference) 2F065 AA03 AA14 AA56 BB18 CC16 DD04 FF01 JJ03 JJ26 KK02 QQ03 QQ07 QQ13 QQ25 QQ29 QQ36 RR07 SS09 UU05 3D037 FA05 FB09 4C038 PP05 PQ04 DB07 DA08 DC14 DC36 5L096 DA03 FA19 FA32 FA69 GA02 GA17 GA34 GA36 HA02 HA05 JA11 9A001 BB02 BB03 BB04 EE02 EE05 GG 05 HH24 HH25 JJ46 JJ77 KK37 KK42 KK56 LL03

Claims (8)

[Claims] 1. An image input means for inputting a face image, an eye opening detection means for detecting an eye opening degree from face image data input by the image input means, and an eye opening detection means A determination means for determining that the position of the eye is erroneously recognized when the eye opening value detected in step (a) does not change for a predetermined time T1. 2. The eye state detection device according to claim 1, wherein the eye open / closed state is determined from a change in the eye openness value output from the eye openness detection means, and the open / closed state of the eye is determined. An eye state detection device comprising: an arousal level determination unit configured to determine an arousal level from a change. 3. The eye condition detecting device according to claim 1, wherein the eye position detecting device detects position coordinates of the eye from image data of the face input by the image input device. The eye state detecting device, wherein when the position coordinates of the eye move by a predetermined value or more, the predetermined time T1 is shortened. 4. The eye condition detection device according to claim 2, wherein the eye position detection device detects position coordinates of the eye from image data of the face input by the image input device, and the arousal level determination. Alarm means for issuing an alarm when it is determined that the arousal level has decreased, and an alarm output for prohibiting the dozing alarm output for a predetermined time T2 when the position coordinates of the eye have moved by a predetermined value or more. A drowsy driving alarm device comprising a prohibition unit. 5. The eye state detection device according to claim 3, wherein: an alarm unit that issues an alarm when the arousal level is determined to be low by the arousal level determination unit; and the position coordinates of the eye. An alarm output prohibiting means for prohibiting a drowsiness alarm output for a predetermined time T2 when the vehicle moves by a predetermined value or more. 6. The drowsy driving alarm device according to claim 4, wherein the predetermined time T2 for prohibiting the output of the drowsiness warning is longer than the predetermined time T1 for determining erroneous recognition of the eyes. Dozing driving alarm device. 7. The drowsy driving alarm device according to claim 4, wherein the alarm output prohibiting means moves within a predetermined time T2 after the position coordinates of the eye have moved by a predetermined value or more. A drowsiness driving alarm device that releases prohibition of a drowsiness alarm output when an eye is detected within an allowable area based on a previous position. 8. The drowsy driving alarm device according to claim 4, wherein the alarm output prohibiting unit is provided when the eye position detecting unit detects the position coordinates of the eye. The drowsiness alarm output is prohibited only for a predetermined time T3.