KR102849692B1 - Drowsiness risk assessment method and system tailored to the driver in the vehicle - Google Patents

Abstract

A method and system for monitoring the driver's condition in real time and providing safety features such as drowsiness prevention are provided. A method for assessing the risk of driver drowsiness in a vehicle, according to an embodiment of the present invention, comprises the steps of: acquiring data through a sensor unit including a camera installed in the vehicle, an ECG (electrocardiogram) sensor and a fingerprint sensor provided on the steering wheel of the vehicle, and a differential pressure sensor installed on the driver's seat; performing driver authentication based on the acquired data by a processor; if driver authentication is successfully performed, estimating the driver's real-time heart rate, respiration rate, and HRV (heart rate variability) based on the acquired data by the processor; and if driver authentication is successfully performed, determining the driver's risk of drowsiness based on the acquired data by the processor. Accordingly, from the personal authentication process to the biosignal acquisition process, factors such as skin moisture content and interior vehicle brightness are estimated and weighted differently to maintain a high level of personal authentication and biosignal acquisition accuracy tailored to each situation. Furthermore, by detecting the driver's drowsiness, the system can prevent drowsy driving by providing binaural beats instead of an alarm sound when drowsy driving is detected.

Description

Drowsiness risk assessment method and system tailored to the driver in the vehicle

The present invention relates to a method and system for determining the risk of driver drowsiness, and more particularly, to a method and system for monitoring the condition of a vehicle driver in real time and providing safety functions such as drowsiness prevention.

Technologies for monitoring a driver's condition in real time by utilizing the driver's bio-signals have been developed in various ways.

Specifically, Korean Patent Publication No. 10-2022-0000771 relates to a technology for monitoring the driver's condition by installing a theta wave (sleepiness wave) detection sensor (SENSE), one of the brain waves, on the steering wheel.

These conventional technologies do not take into account changes due to various factors (brightness, measurement site, measurement area, skin humidity) in an actual environment, and therefore have the disadvantage of poor accuracy due to changes in biometric signals and fingerprint images caused by changes in skin-to-electrode impedance and inability to recognize faces due to brightness when performing actual authentication.

Therefore, it is necessary to find a way to maintain high accuracy of personal authentication and bio-signal acquisition according to each situation by estimating skin humidity level, brightness inside the vehicle, etc. from the personal authentication process to the bio-signal acquisition process and applying different weights.

Korean Patent Publication No. 10-2022-0000771 (Title of the invention: A sensor that detects theta waves (sleepiness waves), one of the brain waves, by wrapping a sensor (SENSE) in a circular shape around the steering wheel to prevent drowsiness while driving, detects theta waves when the driver becomes drowsy, and generates an alarm sound to prevent drowsy driving)

The present invention has been devised to solve the above problems, and the purpose of the present invention is to provide a method and system for determining the risk of driver drowsiness customized for a driver in a vehicle, which can maintain a high level of accuracy in personal authentication and bio-signal acquisition by estimating the level of skin humidity, brightness inside a vehicle, etc. and applying different weights from the personal authentication process to the bio-signal acquisition process.

In addition, another object of the present invention is to provide a method and system for determining the risk of driver drowsiness customized for a driver in a vehicle, which detects the driver's drowsiness state and provides binaural beats instead of an alarm sound when drowsy driving is detected, thereby preventing drowsy driving.

According to one embodiment of the present invention for achieving the above object, a method for determining a driver's risk of drowsiness in a vehicle, the method includes: a step of obtaining data through a sensor unit including a camera installed in the vehicle, an ECG (electrocardiogram) sensor and a fingerprint sensor provided on a steering wheel of the vehicle, and a differential pressure sensor installed on a driver's seat; a step of a processor performing driver authentication based on the obtained data; a step of the processor estimating the driver's real-time heart rate, respiration rate, and HRV (heart rate variability) based on the obtained data when the driver authentication is performed normally; and a step of the processor determining the driver's risk of drowsiness based on the obtained data when the driver authentication is performed normally.

And, when the processor acquires a face image of the driver through a camera installed in the vehicle, it can correct the brightness of the face image by extracting brightness information through the camera's light sensor.

Additionally, the processor can extract brightness information and, when the brightness of the face image is corrected, extract individual facial feature points from the brightness-corrected face image.

And, when the processor extracts the driver's ECG signal and fingerprint image through the ECG sensor and fingerprint sensor provided on the steering wheel of the vehicle, the processor assigns a weight for driver authentication to the ECG signal and fingerprint image respectively based on the driver's skin (palm) humidity measured based on the skin impedance of the driver's (passenger's seat passenger's) skin (palm), so that when the driver's skin humidity measurement value is higher than a preset threshold, a relatively higher weight for driver authentication is assigned to the ECG signal, and when the driver's skin humidity measurement value is lower than the preset threshold, a relatively higher weight for driver authentication is assigned to the fingerprint image.

In addition, the processor can normalize the data acquired in the data acquisition step, train a CNN algorithm to extract local features by utilizing the normalized data as learning data, perform a verification process for the performance of the trained CNN algorithm on the newly normalized data, and select only the CNN algorithms with proven performance to be used in the driver authentication process.

And the step of estimating the driver's heart rate, respiration rate, and HRV is, after driver authentication is normally performed, if it is determined that the driver's both hands are in contact with the steering wheel through the ECG sensor provided on the steering wheel, the processor can estimate the driver's heart rate, respiration rate, and HRV based on the driver's ECG signal extracted through the ECG sensor, the facial image acquired through the camera, and the differential pressure signal extracted through the differential pressure sensor installed on the driver's seat.

In addition, the step of determining the driver's drowsiness risk may include extracting a first determination factor of the driver's drowsiness risk through a camera, and extracting a second determination factor of the driver's drowsiness risk based on the estimated HRV in the estimation step, thereby determining the driver's drowsiness risk. The first determination factor of the driver's drowsiness risk may be the driver's eyelid state (open/closed) analyzed based on a facial image, and the second determination factor of the driver's drowsiness risk may be parasympathetic nerve activity calculated through frequency analysis of HRV.

And in the step of judging the driver's drowsiness risk, when extracting the second judgment factor of the driver's drowsiness risk based on the estimated HRV, the HRV data measured in the time domain is converted to the frequency domain and the intensity of the signal is analyzed by frequency band to calculate the parasympathetic nerve activity.

In addition, the step of determining the driver's drowsiness risk level can be performed by using a pressure sensor installed on the driver's seat to generate different frequencies (beta wave output) on the left and right sides of the driver to transmit binaural beats to the driver in order to wake up the driver if the drowsiness risk level is higher than a preset threshold.

Meanwhile, according to another embodiment of the present invention, a system for determining a driver's risk of drowsiness in a vehicle includes a sensor unit including a camera installed in the vehicle, an ECG (electrocardiogram) sensor and a fingerprint sensor provided on a steering wheel of the vehicle, and a differential pressure sensor installed on a driver's seat; and a processor for performing driver authentication based on acquired data, estimating the driver's real-time heart rate, respiration rate, and HRV (heart rate variability) based on the acquired data when the driver authentication is performed normally, and determining the driver's risk of drowsiness based on the acquired data.

And according to another embodiment of the present invention, a method for determining a driver's risk of drowsiness in a vehicle according to a personalized method comprises: a step of obtaining data through a sensor unit including a camera installed in the vehicle, an ECG sensor and a fingerprint sensor provided on a steering wheel of the vehicle, and a differential pressure sensor installed on a driver's seat; a step of a processor performing driver authentication based on the obtained data; and a step of the processor determining the driver's risk of drowsiness based on the obtained data if the driver authentication is performed normally.

In addition, according to another embodiment of the present invention, a system for determining the risk of driver drowsiness in a vehicle according to a personalized system includes a sensor unit including a camera installed in the vehicle, an ECG (electrocardiogram) sensor and a fingerprint sensor provided on a steering wheel of the vehicle, and a differential pressure sensor installed on a driver's seat; and a processor for performing driver authentication based on the acquired data, and determining the risk of driver drowsiness based on the acquired data if the driver authentication is performed normally.

As described above, according to embodiments of the present invention, from the personal authentication process to the bio-signal acquisition process, the skin humidity level, the brightness inside the vehicle, etc. are estimated and weights are applied differently to maintain a high level of personal authentication and bio-signal acquisition accuracy according to each situation.

Additionally, it can detect the driver's drowsiness and prevent drowsy driving by providing binaural beats instead of an alarm sound when drowsy driving is detected.

FIG. 1 is a drawing provided for the description of a vehicle driver personalized drowsiness risk assessment system according to one embodiment of the present invention;
FIG. 2 is a drawing illustrating a sensor unit installed inside a vehicle according to one embodiment of the present invention.
FIG. 3 is a drawing illustrating an ECG (electrocardiogram) sensor and a fingerprint sensor installed on a steering wheel according to one embodiment of the present invention.
Figure 4 is a flowchart provided to explain a method for determining the risk of driver drowsiness in a vehicle according to one embodiment of the present invention.
FIG. 5 is a drawing provided for a more detailed description of the steps for performing driver authentication according to one embodiment of the present invention;
FIG. 6 is a drawing provided for a more detailed description of the steps for estimating a driver's real-time heart rate, respiration rate, and HRV (heart rate variability) according to one embodiment of the present invention; and
FIG. 7 is a drawing provided for a more detailed description of a step for determining a driver's drowsiness risk according to one embodiment of the present invention.

Hereinafter, the present invention will be described in more detail with reference to the drawings.

FIG. 1 is a drawing provided for the description of a driver's personalized drowsiness risk assessment system in a vehicle according to one embodiment of the present invention, FIG. 2 is a drawing illustrating a sensor unit installed inside a vehicle according to one embodiment of the present invention, and FIG. 3 is a drawing illustrating an ECG (electrocardiogram) sensor (112) and a fingerprint sensor (113) installed on a steering wheel according to one embodiment of the present invention.

The in-vehicle driver drowsiness risk assessment system (hereinafter collectively referred to as the “system”) according to the present embodiment estimates skin humidity level, brightness inside the vehicle, etc. from the personal authentication process to the bio-signal acquisition process, and applies different weights to maintain high accuracy of personal authentication and bio-signal acquisition according to each situation, and detects the driver’s drowsiness state, and provides binaural beats instead of an alarm sound when drowsy driving is detected.

For this purpose, the system may include a sensor unit (110), a processor (120), and a storage unit (130).

The sensor unit (110) is equipped with a camera (111) installed in the vehicle, an ECG (electrocardiogram) sensor (112) and a fingerprint sensor (113) installed on the steering wheel of the vehicle, and a differential pressure sensor (114) installed on the driver's seat, so that the processor (120) can obtain data necessary for operation.

That is, the sensor unit (110) can obtain a face image of the driver through a camera (111) installed in the vehicle, extract an ECG signal of the driver through an ECG sensor (112) installed on the steering wheel of the vehicle, extract a fingerprint image of the driver through a fingerprint sensor (113) installed on the steering wheel of the vehicle, and extract a differential pressure signal through a differential pressure sensor (114) installed on the driver's seat.

Here, the ECG sensor (112) may be composed of a plurality of plates made of a metal material having electrical conductivity.

And, when the driver's bio-signal is extracted through the bio-signal electrode, the ECG sensor (112) performs an analog filtering operation on the extracted driver's bio-signal and can extract individual unique features from the filtered signal.

In addition, the ECG sensor (112) can obtain the skin impedance of the driver's skin (palm) by utilizing a biosignal electrode, and measure (estimate) the driver's skin (palm) humidity based on this.

The fingerprint sensor (113) may include a fingerprint recognition module (not shown) that extracts, in digital form, individual unique features within the extracted fingerprint image when the driver's fingerprint image is extracted. Here, the individual unique features within the extracted fingerprint image may include individual features such as the ridge ends and branch points of the fingerprint.

The differential pressure sensor (114) installed on the driver's seat acquires the driver's heart rate data, and the processor (120) can extract BCG signal data by performing signal filtering and signal analysis processes on the heart rate data acquired through the differential pressure sensor (114).

The storage unit (130) is a storage medium that can store programs and data necessary for the processor (120) to operate.

The processor (120) is provided to process all matters of the system.

For example, the processor (120) performs driver authentication based on the acquired data, and if the driver authentication is performed normally, the processor estimates the driver's real-time heart rate, respiration rate, and HRV (heart rate variability) based on the acquired data, and can determine the driver's drowsiness risk based on the acquired data.

In addition, in the case of an electrocardiogram measured from a steering wheel, the signal quality (SNR) is high, but there is a disadvantage that the driver must place both hands on the steering wheel. Therefore, the processor (120) analyzes the electrocardiogram measured from the steering wheel, estimates the driver's heart rate and respiration through peak detection and frequency analysis, labels these values, and matches the facial image acquired from the camera (111) and the differential pressure signal acquired from the differential pressure sensor (114) to analyze the driver's condition using deep learning technology.

FIG. 4 is a flowchart provided to explain a method for determining a driver's personalized drowsiness risk in a vehicle according to one embodiment of the present invention.

The method for determining the risk of driver drowsiness in a vehicle according to the present embodiment can be executed by the system described above with reference to FIGS. 1 to 3.

A method for determining the risk of driver drowsiness in a vehicle can obtain data through a sensor unit (110) (S410) and perform driver authentication based on the obtained data through a processor (120) (S420).

Here, the data acquisition task (S410) through the sensor unit (110) may be performed again at the request of the processor (120) to proceed with step S430 or step S440 even after driver authentication is performed normally.

And, the method for determining the driver's risk of drowsiness in a vehicle can estimate the driver's real-time heart rate, respiration rate, and HRV (heart rate variability) based on the acquired data through the processor (120) when driver authentication is performed normally (S430), and determine the driver's risk of drowsiness based on the acquired data (S440).

For example, if the processor determines that the driver's both hands are in contact with the steering wheel through the ECG sensor (112) provided on the steering wheel after driver authentication is normally performed, the processor can estimate the driver's heart rate, respiration rate, and HRV based on the driver's ECG signal extracted through the ECG sensor (112), the facial image acquired through the camera (111), and the differential pressure signal extracted through the differential pressure sensor (114) installed on the driver's seat.

And, when determining the driver's drowsiness risk, the processor extracts a first determination factor of the driver's drowsiness risk through the camera (111), and extracts a second determination factor of the driver's drowsiness risk based on the estimated HRV in the estimation step, thereby determining the driver's drowsiness risk.

At this time, the first factor in determining the driver's drowsiness risk may be the driver's eyelid state (open/closed) analyzed based on the facial image, and the second factor in determining the driver's drowsiness risk may be the parasympathetic nerve activity calculated through frequency analysis of HRV.

FIG. 5 is a drawing provided for a more detailed description of steps for performing driver authentication according to one embodiment of the present invention.

The system can perform an image extraction task (A) through a camera (111), a bio-signal (BCG) extraction task (B) through a differential pressure sensor (114), and a bio-signal (ECG) and fingerprint image extraction task (C) through a steering wheel to perform driver authentication.

Specifically, when the system performs an image extraction task (A) through a camera (111), when acquiring a driver's face image through a camera (111) installed in a vehicle, the system can correct the brightness of the face image by extracting brightness information through a light sensor of the camera (111), and when the brightness of the face image is corrected by extracting the brightness information, the system can extract individual facial feature points (e.g., contour, eyes, nose, mouth, etc.) from the brightness-corrected face image.

And, when the system performs the bio-signal (BCG) extraction task (B) through the differential pressure sensor (114), the system can obtain the driver's heart rate data through the differential pressure sensor (114) installed on the driver's seat, and extract BCG signal data by performing signal filtering (preprocessing) and signal analysis processes on the heart rate data using the processor (120).

In addition, when the system performs a bio-signal (ECG) and fingerprint image extraction task (C) through a steering wheel, the system extracts an ECG signal through an ECG sensor (112) composed of a plurality of electrically conductive metal plates, extracts individual unique features through a filtering process, obtains skin impedance of the driver's skin (palm) using a bio-signal electrode, and measures (estimates) the driver's skin (palm) humidity based on this.

And, when the system performs a bio-signal (ECG) and fingerprint image extraction task (C) through a steering wheel, the system can weight the ECG signal and fingerprint image for driver authentication based on the driver's skin (palm) humidity measured based on the skin impedance of the driver's skin (palm).

For example, the system may assign a relatively higher weight for driver authentication to an ECG signal if the driver's skin moisture measurement is above a preset threshold, and may assign a relatively higher weight for driver authentication to a fingerprint image if the driver's skin moisture measurement is below the preset threshold.

And the system can perform a driver authentication task (D) by applying data obtained through an image extraction task (A), a biosignal (BCG) extraction task (B), and a fingerprint image extraction task (C) to a CNN algorithm (deep learning model).

Specifically, when the system performs a driver authentication task (D), it divides the acquired data into learning data and verification data, then creates and trains a model using the learning data, and acquires real-time data to perform personal authentication using a classification model.

That is, the system normalizes data acquired through image extraction task (A), biosignal (BCG) extraction task (B), and fingerprint image extraction task (C), trains a CNN algorithm to extract minor features using the normalized data as learning data, performs a performance verification process for the trained CNN algorithm on the newly normalized data, and selects only the CNN algorithms with proven performance to be used in the driver authentication process.

FIG. 6 is a diagram provided for a more detailed description of a step of estimating a driver's real-time heart rate, respiration rate, and HRV (heart rate variability) according to one embodiment of the present invention.

The system can perform a Lead-off Detection task (1), a bio-signal (ECG) extraction task (2) through a steering wheel, an image and bio-signal (ECG) extraction task (3) through a camera (111) and a vehicle seat, and a model learning and application task (4) through a CNN algorithm to estimate the driver's real-time heart rate, respiration rate, and HRV (heart rate variability).

Specifically, when the system performs the Lead-off Detection task (1), it can determine whether the driver's both hands are in contact with the steering wheel through the ECG sensor (112) provided on the steering wheel.

And, when the system performs the bio-signal (ECG) extraction task (2) through the steering wheel, when the bio-signal (ECG) is extracted through the ECG sensor (112) provided on the steering wheel, the system extracts BCG signal data through signal filtering and signal analysis processes, and performs heart rate, respiration, and R-Peak analysis through peak detection (R-peak detection through processes such as low-frequency filtering, differentiation, moving average, and threshold detection) and frequency analysis (FFT).

In addition, the system can perform a labeling task (preprocessing task) on the data acquired during the process of extracting a bio-signal (ECG) through a steering wheel (2), and match it with learning data of a CNN algorithm for estimating the driver's real-time heart rate, respiration rate, and HRV (heart rate variability).

For example, since the system does not always have both hands on the steering wheel, it can label each time it has both hands on the steering wheel for a certain amount of time and use that as training data.

And, when the system performs the image and bio-signal (ECG) extraction task (3) through the camera (111) and the vehicle seat, the result of the image extraction task (A) through the camera (111) performed for driver authentication and the result of the bio-signal (BCG) extraction task (B) through the differential pressure sensor (114) can be utilized.

In addition, when the system performs model learning and application tasks (4) through a CNN algorithm, the result data of the image and bio-signal (ECG) extraction task (3) and the result data (labeling data) of the bio-signal (ECG) extraction task (2) are used as learning data to generate and train a model of a CNN algorithm for extracting local features, and by applying real-time data to the model of the trained CNN algorithm, a verification procedure for the model performance of the CNN algorithm is performed, and only models of the CNN algorithm with proven performance are selected to estimate the driver's real-time heart rate, respiration rate, and HRV (heart rate variability).

FIG. 7 is a drawing provided for a more detailed description of a step for determining a driver's drowsiness risk according to one embodiment of the present invention.

The system can perform a task of estimating heart rate variability (HRV) (a), a task of extracting a first judgment factor for driver drowsiness risk through a camera (111) (b), a task of extracting a second judgment factor for driver drowsiness risk through heart rate variability (c), and a task of judging drowsiness risk through the first and second judgment factors (d) to determine the driver's drowsiness risk.

Specifically, when the system performs a heart rate variability (HRV) estimation task (a), the system can utilize HRV among the estimation results of the driver's real-time heart rate, respiration rate, and HRV (heart rate variability).

And, when the system performs the first judgment factor extraction task (b) of the driver drowsiness risk level through the camera (111), if facial image data is extracted using the camera (111), the facial image data can be analyzed by utilizing a model of a CNN algorithm that can classify the state of the driver's eyelids (degree of opening/closing).

At this time, a CNN algorithm model capable of classifying the driver's eyelid state (open/closed degree) can be trained using facial image data as learning data, and the trained CNN algorithm model performs a performance verification procedure, and only models of the CNN algorithm with proven performance are selected to classify the driver's eyelid state (open/closed degree).

In addition, when the system performs the second factor extraction task (d) of driver drowsiness risk assessment through heart rate variability, the higher the parasympathetic nerve activity, the easier it is to fall into drowsiness, so the system can calculate the parasympathetic nerve activity and assess the drowsiness risk.

That is, when the system performs the task of extracting the second judgment factor of the driver's drowsiness risk level through heart rate variability (D), it can calculate parasympathetic nerve activity by converting HRV data measured in the time domain into the frequency domain and analyzing the intensity of the signal by frequency band. Here, the system can calculate parasympathetic nerve activity by utilizing the property that as the intensity of the high-frequency band of the signal increases, sympathetic nerve activity increases, and as the intensity of the low-frequency band increases, parasympathetic nerve activity increases.

And, when the system performs the task of judging the risk of drowsiness (D) through the first and second judgment factors, if the risk of drowsiness is higher than a preset threshold, the system can prevent drowsy driving by generating different frequencies (beta wave output) on the left and right sides of the driver's seat using a differential pressure sensor (114) installed on the driver's seat to transmit binaural beats to the driver in order to wake up the driver.

Meanwhile, it goes without saying that the technical idea of the present invention can also be applied to a computer-readable recording medium containing a computer program that performs the functions of the device and method according to the present embodiment. In addition, the technical idea according to various embodiments of the present invention can be implemented in the form of computer-readable code recorded on a computer-readable recording medium. The computer-readable recording medium can be any data storage device that can be read by a computer and store data. For example, the computer-readable recording medium can be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, etc. In addition, the computer-readable code or program stored on the computer-readable recording medium can be transmitted through a network connected between computers.

In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications can be made by a person having ordinary skill in the art to which the present invention pertains without departing from the gist of the present invention as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present invention.

110: Sensor section
111: Camera
112: ECG (electrocardiogram) sensor
113: Fingerprint sensor
114: Differential pressure (Peizo) sensor
120: Processor
130: Storage

Claims (12)

A step of acquiring data through a sensor unit including a camera installed in a vehicle, an ECG (electrocardiogram) sensor and fingerprint sensor installed on the steering wheel of the vehicle, and a differential pressure sensor installed on the driver's seat;
A step in which the processor performs driver authentication based on the acquired data;
If driver authentication is performed normally, the processor estimates the driver's real-time heart rate, respiration rate, and HRV (heart rate variability) based on the acquired data; and
If driver authentication is performed normally, the processor includes a step of determining the driver's drowsiness risk based on the acquired data;
The processor is,
When acquiring a driver's face image through a camera installed in the vehicle, the brightness information is extracted through the camera's light sensor to correct the brightness of the face image.
The processor is,
When the brightness of the face image is corrected by extracting brightness information, individual facial feature points including the face contour, eyes, nose, and mouth are extracted from the brightness-corrected face image.
The processor is,
When extracting the driver's ECG signal and fingerprint image through the ECG sensor and fingerprint sensor installed on the steering wheel of the vehicle, the ECG signal and fingerprint image are each given a weight for driver authentication based on the driver's (passenger's) skin (palm) humidity measured based on the skin impedance of the driver's (passenger's) skin (palm).
If the driver's skin moisture measurement value is above a preset threshold, the ECG signal is given a relatively higher weight for driver authentication,
If the driver's skin moisture measurement value is below a preset threshold, the fingerprint image is given a relatively higher weight for driver authentication.
The steps to estimate the driver's heart rate, breathing rate and HRV are:
After driver authentication is normally performed, if it is determined that both of the driver's hands are in contact with the steering wheel through the ECG sensor installed on the steering wheel, the processor estimates the driver's heart rate, respiration rate, and HRV based on the driver's ECG signal extracted through the ECG sensor, the facial image acquired through the camera, and the differential pressure signal extracted through the differential pressure sensor installed on the driver's seat.
The steps to determine the driver's risk of drowsiness are:
The first factor for determining the driver's drowsiness risk is extracted through the camera, and the second factor for determining the driver's drowsiness risk is extracted based on the estimated HRV in the estimation step, thereby determining the driver's drowsiness risk.
The first factor in determining the risk of driver drowsiness is
The driver's eyelid status (open/closed) is analyzed based on the facial image,
The second factor in determining the risk of driver drowsiness is
It is the parasympathetic nerve activity calculated through frequency analysis of HRV.
The steps to determine the driver's risk of drowsiness are:
When extracting the second judgment factor of driver drowsiness risk based on the estimated HRV, the HRV data measured in the time domain is converted to the frequency domain, and the signal intensity is analyzed by frequency band to calculate the parasympathetic nerve activity.
The steps to acquire data are:
Whenever the driver's hands touch the steering wheel, the driver's ECG signal is extracted through the ECG sensor.
The processor is,
A method for assessing the risk of driver drowsiness in a vehicle, characterized in that when a driver's ECG signal is extracted, a labeling task is performed on the extracted ECG signal and the extracted ECG signal is used as learning data for a CNN algorithm for estimating the driver's real-time heart rate, respiration rate, and HRV (heart rate variability).
delete delete delete In claim 1,
The processor is,
A method for assessing the risk of driver drowsiness in a vehicle, characterized by normalizing the acquired data in the data acquisition stage, training a CNN algorithm to extract local features using the normalized data as learning data, performing a verification process for the performance of the trained CNN algorithm using the newly normalized data, and selecting only the CNN algorithms with proven performance to use in the driver authentication process.
delete delete delete In claim 1,
The steps to determine the driver's risk of drowsiness are:
A method for determining the risk of drowsiness of a driver in a vehicle, characterized in that when the risk of drowsiness exceeds a preset threshold, a pressure sensor installed on the driver's seat is used to generate different frequencies (beta wave output) on the left and right sides of the driver to transmit binaural beats to the driver in order to wake the driver up.
A sensor unit including a camera installed in the vehicle, an ECG (electrocardiogram) sensor and a fingerprint sensor installed on the steering wheel of the vehicle, and a differential pressure sensor installed on the driver's seat; and
A processor that performs driver authentication based on the acquired data, estimates the driver's real-time heart rate, respiration rate, and HRV (heart rate variability) based on the acquired data when the driver authentication is performed normally, and determines the driver's drowsiness risk based on the acquired data;
The processor is,
When acquiring a driver's face image through a camera installed in the vehicle, the brightness information is extracted through the camera's light sensor to correct the brightness of the face image.
The processor is,
When the brightness of the face image is corrected by extracting brightness information, individual facial feature points including the face contour, eyes, nose, and mouth are extracted from the brightness-corrected face image.
The processor is,
When extracting the driver's ECG signal and fingerprint image through the ECG sensor and fingerprint sensor installed on the steering wheel of the vehicle, the ECG signal and fingerprint image are each given a weight for driver authentication based on the driver's (passenger's) skin (palm) humidity measured based on the skin impedance of the driver's (passenger's) skin (palm).
If the driver's skin moisture measurement value is above a preset threshold, the ECG signal is given a relatively higher weight for driver authentication,
If the driver's skin moisture measurement value is below a preset threshold, the fingerprint image is given a relatively higher weight for driver authentication.
The processor is,
After driver authentication is normally performed, if it is determined that the driver's both hands are in contact with the steering wheel through the ECG sensor installed on the steering wheel, the driver's heart rate, breathing rate, and HRV are estimated based on the driver's ECG signal extracted through the ECG sensor, the facial image acquired through the camera, and the differential pressure signal extracted through the differential pressure sensor installed on the driver's seat.
The processor is,
When the first judgment factor of the driver's drowsiness risk is extracted through the camera, the second judgment factor of the driver's drowsiness risk is extracted based on the estimated HRV, and the driver's drowsiness risk is determined.
The first factor in determining the risk of driver drowsiness is
The driver's eyelid status (open/closed) is analyzed based on the facial image,
The second factor in determining the risk of driver drowsiness is
It is the parasympathetic nerve activity calculated through frequency analysis of HRV.
The processor is,
When extracting the second judgment factor of driver drowsiness risk based on the estimated HRV, the HRV data measured in the time domain is converted to the frequency domain, and the signal intensity is analyzed by frequency band to calculate the parasympathetic nerve activity.
The ECG sensor is
Every time the driver's hands touch the steering wheel, the driver's ECG signal is extracted,
The processor is,
A vehicle driver drowsiness risk assessment system tailored to the driver's individuality, characterized in that when a driver's ECG signal is extracted, a labeling task is performed on the extracted ECG signal and the extracted ECG signal is used as learning data for a CNN algorithm for estimating the driver's real-time heart rate, respiration rate, and HRV (heart rate variability).
A step of acquiring data through a sensor unit including a camera installed in a vehicle, an ECG sensor and fingerprint sensor installed on the steering wheel of the vehicle, and a differential pressure sensor installed on the driver's seat;
A step in which the processor performs driver authentication based on the acquired data; and
If driver authentication is performed normally, the processor includes a step of determining the driver's drowsiness risk based on the acquired data;
The processor is,
When acquiring a driver's face image through a camera installed in the vehicle, the brightness information is extracted through the camera's light sensor to correct the brightness of the face image.
The processor is,
When the brightness of the face image is corrected by extracting brightness information, individual facial feature points including the face contour, eyes, nose, and mouth are extracted from the brightness-corrected face image.
The processor is,
When extracting the driver's ECG signal and fingerprint image through the ECG sensor and fingerprint sensor installed on the steering wheel of the vehicle, the ECG signal and fingerprint image are each given a weight for driver authentication based on the driver's (passenger's) skin (palm) humidity measured based on the skin impedance of the driver's (passenger's) skin (palm).
If the driver's skin moisture measurement value is above a preset threshold, the ECG signal is given a relatively higher weight for driver authentication,
If the driver's skin moisture measurement value is below a preset threshold, the fingerprint image is given a relatively higher weight for driver authentication.
The steps to determine the risk of drowsiness are:
After driver authentication is normally performed, if it is determined that both of the driver's hands are in contact with the steering wheel through the ECG sensor installed on the steering wheel, the processor estimates the driver's heart rate, respiration rate, and HRV based on the driver's ECG signal extracted through the ECG sensor, the facial image acquired through the camera, and the differential pressure signal extracted through the differential pressure sensor installed on the driver's seat.
The steps to determine the risk of drowsiness are:
The first factor for determining the driver's drowsiness risk is extracted through the camera, and the second factor for determining the driver's drowsiness risk is extracted based on the estimated HRV in the estimation step, thereby determining the driver's drowsiness risk.
The first factor in determining the risk of driver drowsiness is
The driver's eyelid status (open/closed) is analyzed based on the facial image,
The second factor in determining the risk of driver drowsiness is
It is the parasympathetic nerve activity calculated through frequency analysis of HRV.
The steps to determine the risk of drowsiness are:
When extracting the second judgment factor of driver drowsiness risk based on the estimated HRV, the HRV data measured in the time domain is converted to the frequency domain, and the signal intensity is analyzed by frequency band to calculate the parasympathetic nerve activity.
The steps to acquire data are:
Whenever the driver's hands touch the steering wheel, the driver's ECG signal is extracted through the ECG sensor.
The processor is,
A method for assessing the risk of driver drowsiness in a vehicle, characterized in that when a driver's ECG signal is extracted, a labeling task is performed on the extracted ECG signal and the extracted ECG signal is used as learning data for a CNN algorithm for estimating the driver's real-time heart rate, respiration rate, and HRV (heart rate variability).
A sensor unit including a camera installed in the vehicle, an ECG (electrocardiogram) sensor and a fingerprint sensor installed on the steering wheel of the vehicle, and a differential pressure sensor installed on the driver's seat; and
A processor that performs driver authentication based on the acquired data, and if the driver authentication is performed normally, determines the driver's drowsiness risk based on the acquired data;
The processor is,
When acquiring a driver's face image through a camera installed in the vehicle, the brightness information is extracted through the camera's light sensor to correct the brightness of the face image.
The processor is,
When the brightness of the face image is corrected by extracting brightness information, individual facial feature points including the face contour, eyes, nose, and mouth are extracted from the brightness-corrected face image.
The processor is,
When extracting the driver's ECG signal and fingerprint image through the ECG sensor and fingerprint sensor installed on the steering wheel of the vehicle, the ECG signal and fingerprint image are each given a weight for driver authentication based on the driver's (passenger's) skin (palm) humidity measured based on the skin impedance of the driver's (passenger's) skin (palm).
If the driver's skin moisture measurement value is above a preset threshold, the ECG signal is given a relatively higher weight for driver authentication,
If the driver's skin moisture measurement value is below a preset threshold, the fingerprint image is given a relatively higher weight for driver authentication.
The processor is,
After driver authentication is normally performed, if it is determined that the driver's both hands are in contact with the steering wheel through the ECG sensor installed on the steering wheel, the driver's heart rate, breathing rate, and HRV are estimated based on the driver's ECG signal extracted through the ECG sensor, the facial image acquired through the camera, and the differential pressure signal extracted through the differential pressure sensor installed on the driver's seat.
The processor is,
When the first judgment factor of the driver's drowsiness risk is extracted through the camera, the second judgment factor of the driver's drowsiness risk is extracted based on the estimated HRV, and the driver's drowsiness risk is determined.
The first factor in determining the risk of driver drowsiness is
The driver's eyelid status (open/closed) is analyzed based on the facial image,
The second factor in determining the risk of driver drowsiness is
It is the parasympathetic nerve activity calculated through frequency analysis of HRV.
The processor is,
When extracting the second judgment factor of driver drowsiness risk based on the estimated HRV, the HRV data measured in the time domain is converted to the frequency domain, and the signal intensity is analyzed by frequency band to calculate the parasympathetic nerve activity.
The ECG sensor is
Every time the driver's hands touch the steering wheel, the driver's ECG signal is extracted,
The processor is,
A vehicle driver drowsiness risk assessment system tailored to the driver's individuality, characterized in that when a driver's ECG signal is extracted, a labeling task is performed on the extracted ECG signal and the extracted ECG signal is used as learning data for a CNN algorithm for estimating the driver's real-time heart rate, respiration rate, and HRV (heart rate variability).