JP7154071B2 - Driving state monitoring support system, driving state monitoring support method and program - Google Patents

Description

The present invention relates to a driving state monitoring support system, a driving state monitoring support method, and a program for supporting monitoring of a driving state such as whether or not a driver is driving a vehicle.

2. Description of the Related Art In recent years, with the spread of mobile terminals such as mobile phones and smart phones, many people can easily and easily transmit and receive information such as conversations and exchanges of e-mails at any place.
In addition, since base stations are installed everywhere, the mobile terminal can be used even in a moving vehicle, and the driver can operate the mobile terminal other than driving.

As a result, the number of drivers who operate a mobile terminal while driving a vehicle is increasing, and most of the driver's attention is devoted to operating the mobile terminal, resulting in a significant decrease in the driver's attention to driving the vehicle. Resulting in.
As a result, the driver's distracted driving may put the vehicle in a dangerous state, and the risk of a traffic accident due to distracted driving is increasing.

As described above, in order to deter driving, a device that monitors a moving vehicle that is transmitting mobile phone radio waves and detects whether the driver is carrying out illegal driving by holding a mobile phone in hand and talking. There is (for example, see Patent Document 1).
In order to confirm the running state of the vehicle, this device has an acceleration device mounted on the vehicle, and determines whether or not the vehicle is running by means of the acceleration device.

In addition, it detects whether or not the driver is operating the mobile phone from the radio waves emitted from the mobile phone, identifies the vehicle in which the mobile phone radio wave is being emitted while the vehicle is running, and If the driver is operating the mobile phone from the imaged image, it is determined that the driver is driving illegally.

In recent years, an environment has been developed in which human motions can be easily analyzed by image analysis using AI (artificial intelligence).
For this reason, AI is used to analyze captured images of drive recorders to determine whether or not the driver is driving while operating a mobile terminal (for example, Non-Patent Document 1). reference).

JP-A-2005-56000

``You don't operate your smartphone while driving, do you? ISP's transportation IT solution that monitors and prevents driving while using AI'' (https://news.mynavi.jp/article/20180531-isp/, 2018 (searched on August 20, 2018)

However, the apparatus for detecting illegal driving of Patent Document 1 requires that the vehicle be newly provided with an acceleration sensor for detecting the running state of the vehicle and a receiver for detecting radio waves transmitted by the mobile phone.
Further, the device for detecting illegal driving disclosed in Patent Document 1 uses an imaging device installed in an imaging area to determine whether the operation of a mobile phone is performed by a driver who is driving a vehicle.

Therefore, if the operation of the mobile phone is finished before the position of the imaging device, it becomes impossible to determine whether the person operating the mobile phone is the driver or the passenger. They should be placed at short distances.
In addition, it is necessary to analyze the data of each of the acceleration sensor, the receiving device, and the imaging device at the same timing. System development is required.

In addition, AI, for example, CNN (convolutional neural network) and other deep learning technology, such as CNN (convolutional neural network), in the analysis of whether or not you are driving, the objects and human facial expressions captured in the captured image are precisely analyzed. can be specified.
However, since it is not possible to detect whether the vehicle is running or not, it is possible to detect that the human condition is unnatural, but it is difficult to determine whether the vehicle is driving while driving. is.

The present invention has been made in view of such circumstances, and does not need to be mounted on the vehicle, such as an acceleration device and a receiver, and uses a fixed-point imaging device such as a drive recorder that is generally mounted on the vehicle. It is therefore an object of the present invention to provide a driving state monitoring support system, a driving state monitoring support method, and a program for easily detecting driving while the driver is driving.

The present invention has been made to solve the above-described problems. A driving condition monitoring support system according to the present invention has a fixed imaging range, and a fixed-point imaging device that captures images of the inside of a vehicle, including the driver's seat, in time series. a changed region extraction unit that compares each of the obtained image data with the immediately preceding image data and the immediately subsequent image data, respectively, and generates a masking image showing a changed region composed of pixels having a change; and the changed region in the image data. and a motion classification model, which is a machine learning model for classifying the skin color region into one of a plurality of classes indicating the motion state of the driver. A driver state determining unit receives the skin color region and classifies the operating state of the driver into classes; The masking image is input to a travel classification model, which is a machine learning model for classifying into one of the stop classes of the state, and the vehicle travel determination unit performs class classification processing, and the driver state determination unit includes the skin color region. is classified into the mobile terminal operation class in which the driver is operating the mobile terminal, and the vehicle driving determination unit classifies the masking image into the driving class, the corresponding image data is classified into the driving class when the driver operates the vehicle. and a result integration unit for extracting image candidates in which the portable terminal is being operated while driving .

In the driving state monitoring support system of the present invention, the changed area extracting unit converts each of the captured images into a grayscale image, compares the grayscale images immediately before and after the grayscale image, and performs a corresponding operation. It is characterized in that, when a difference in gradation of each pixel at a position exceeds a preset threshold value, the changed area is extracted as the pixel having a change.

In the driving state monitoring support system of the present invention, the changed region extracting unit compares the grayscale image with a grayscale image immediately preceding the grayscale image, and the region of pixels having a change obtained by comparing the grayscale image with and a region of pixels having a change obtained by comparing the grayscale image immediately after the grayscale image is used as the masking image.

In the driving state monitoring support system of the present invention, the driving classification model divides the vehicle state into a driving class or a stop class depending on whether or not the image portion corresponding to the car window of the vehicle in the masking image has changed. It is characterized by classifying into

In the driving state monitoring support system of the present invention, the changing region extracting unit divides the image data into a predetermined number of segments, and compares the existence probability of the segment overlapping the skin color region with the segment not overlapping the skin color region. and increasing the certainty of the mobile terminal operation class operating the mobile terminal in the action classification model in accordance with the existence probability.

In the driving state monitoring support method of the present invention, the changing region extracting unit extracts each image data captured in time series by a fixed-point imaging device that captures an image of the inside of the vehicle including the driver's seat in a fixed imaging range. a changing area extracting step of comparing the image data with the immediately succeeding image data and generating a masking image showing a changing area made up of pixels having a change; a skin-color region extraction process for extracting a skin-color region including skin-color pixels from the vehicle; A driver state determination process in which the skin color region is input to the classification model and class classification of the driver's motion state is performed, and the vehicle/pillow s determination unit determines the state in which the vehicle is running on the masking image. a vehicle travel determination process of inputting the masking image to a travel classification model, which is a machine learning model for classifying into one of a travel class and a stop class in which the vehicle is stopped, and performing class classification processing; a result integration unit for classifying the flesh-colored area into a portable terminal operation class in which the driver is operating a portable terminal by the driver state determination process, and classifying the masking image into the driving class by the vehicle driving determination process; and a result integrating step of extracting the corresponding image data, if classified, as an image candidate of the driver operating the portable terminal while driving the vehicle .

In the driving state monitoring support method of the present invention, the changing region extracting unit captures an image of the inside of the vehicle including the driver's seat in time series by a fixed-point imaging device that captures an image of the inside of the vehicle including the driver's seat. a changing region extracting step of comparing each with immediately preceding image data and succeeding image data, respectively, to generate a masking image showing a changing region made up of pixels having a change; A skin-color region extraction process that extracts a skin-color region containing skin-color pixels from the region corresponding to the , and a machine-learned motion classification model in which the driver state determination unit classifies the driver's motion state into one of a plurality of classes. A driver state determination process for inputting the skin color region to and performing class classification processing of the driver state, a driving class in which the vehicle driving determination unit indicates the driving state of the vehicle, and the vehicle and a vehicle travel determination process of inputting the masking image to a machine-learned travel classification model that classifies the vehicle into one of the stop classes that indicate a state in which the vehicle is stopped, and performing class classification processing. Characterized by

The program of the present invention causes the computer to convert each of the image data captured in time series by a fixed-point imaging device that captures an image of the inside of the vehicle including the driver's seat with a fixed imaging range into immediately preceding image data and immediately following image data. changed region extracting means for generating a masking image showing a changed region composed of pixels having a change in comparison with each other; means for inputting the flesh-colored area into an action classification model, which is a machine learning model for classifying the flesh-colored area into one of a plurality of classes indicating the action state of the driver, and classifying the action state of the driver. driver state determination means, and a driving classification model that is a machine learning model that classifies the masking image into either a driving class in which the vehicle is running or a stop class in which the vehicle is stopped. , the vehicle travel determining means for inputting the masking image and performing class classification processing ; When the vehicle driving determination means classifies the masking image into the driving class, the corresponding image data is functioned as a result integrating means for extracting image candidates of the driver operating the mobile terminal while driving the vehicle . It is a program for

According to the present invention, there is no need to mount an acceleration device or a receiver on the vehicle, and a fixed-point imaging device such as a drive recorder that is generally mounted on the vehicle can be used to drive the vehicle while the driver is driving. It is possible to provide a driving state monitoring support system, a driving state monitoring support method, and a program that support the detection of easily.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure which shows the structural example of the driving|running state monitoring assistance system by one Embodiment of this invention. FIG. 7 is a conceptual diagram illustrating generation of a clipped image from a framed image by clipping processing based on a second masking image by the rectangular area generation unit 15 according to the present embodiment. 4 is a diagram showing a configuration example of a driver state determination table in a storage unit 20 in this embodiment; FIG. It is a figure which shows the structural example of the driving state determination table of the memory|storage part 20 in this embodiment. FIG. 5 is a conceptual diagram showing the process of extracting a driving scene by the result integration unit 18 of the present embodiment; 3 is a diagram showing a configuration example of a scene table of a storage unit 20 in this embodiment; FIG. 4 is a flowchart showing an operation example of a process for supporting monitoring of driving in the driving state monitoring support system of the present embodiment; FIG. 2 is a conceptual diagram explaining each image data generated by processing each configuration in the driving state monitoring support system; FIG. 10 is a conceptual diagram for explaining a process in which the driver's state determination unit 16 in the present embodiment uses the presence probability map to improve the accuracy of classification into the mobile terminal operation class in the class classification of the clipped image.

The present invention is used to determine dangerous illegal driving, so-called driving while operating a mobile terminal such as a smartphone or a mobile phone while driving a vehicle, and it is estimated that driving is being performed while driving with a high probability. The present invention relates to a driving state monitoring support system that supports extraction of image data from image data of a captured image.
Hereinafter, a driving state monitoring support system according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing a configuration example of a driving state monitoring support system according to one embodiment of the present invention.
In FIG. 1, the driving state monitoring support system 1 in this embodiment includes a captured image input unit 11, a framed image generation unit 12, a change region extraction unit 13, a skin color region extraction unit 14, a rectangular region generation unit 15, a driver state A determination unit 16 , a vehicle travel determination unit 17 , a result integration unit 18 , an image presentation control unit 19 and a storage unit 20 are provided.

The captured image input unit 11 inputs image data of a captured image captured by the imaging device 2 (for example, a moving image in this embodiment). Then, the captured image input unit 11 adds user identification information for identifying the user of the imaging device 2 to the input image data, and writes and stores it in the storage unit 20 . In this embodiment, for example, a moving image name is given as user identification information and used as the file name of image data ((moving image name).mpg).
Here, the imaging device 2 is a fixed-point imaging device (fixed-point camera), such as a drive recorder for imaging the inside of the vehicle including the driver in the driver's seat and the vehicle window. Image data of the captured image is read by the captured image input unit 11 from a memory card, which is a storage medium of the drive recorder.

The framed image generation unit 12 converts the image data stored in the storage unit 20 into a predetermined number of framed images per unit time (for example, one second).
Here, the framed image generation unit converts, for example, an MPEG (moving picture experts group)-4 format captured image of the imaging device 2 into a JPEG (joint photographic experts group) format framed image, which is a still image. do.

Then, the framed image generation unit 12 adds the chronological order of the framed images to the end of the moving image name for each of the image data converted into the JPEG format, and writes and stores them in the storage unit 20. .
The file name of the framed image is (video name)_1 (frame number). jpg, and given as a moving image name and a frame number indicating the chronological order in the image data of the moving image.
As described above, in the present embodiment, a moving image is supplied as a captured image from the drive recorder. image data corresponding to an image) may be supplied in time series.

The changed area extraction unit 13 extracts a moving pixel area in each of the framed images, that is, an area composed of pixels having pixel values that change before and after the framed image in time series.
Here, the changing region extracting unit 13 sequentially reads each of the framed images from the storage unit 20 based on the order of time series.
Then, the changing region extracting unit 13 converts each of the framed images into monochrome and converts them into grayscale images, and converts each pixel of the grayscale image and other grayscale images immediately before and after each grayscale image. A difference in gradation from the corresponding pixel is obtained.

At this time, the changing region extracting unit 13 obtains the difference in gradation from the corresponding pixel for each of the framed images T1, T2, and T3 that are continuous in time series.
Here, for example, the framed image T1 is the framed image (video name)_1. jpg, and the framed image T2 is a framed image (video name)_2. jpg, and the framed image T3 is a framed image (video name)_3. jpg.

The changing area extracting unit 13 obtains the difference in gradation of corresponding pixels between the grayscale image T2 and the immediately preceding grayscale image T1, and scales the absolute value of the difference in gradation (amount of change in gradation) of each pixel. A difference image T2-T1 is generated.
Similarly, the changing area extracting unit 13 obtains the difference in the gradation of corresponding pixels between the grayscale image T2 and the immediately following grayscale image T3, and uses the absolute value of the difference in the gradation of the pixel as the gradation. A difference image T2-T3 is obtained.

Then, for example, in each of the difference image T2-T1 and the difference image T2-T3, the changing region extracting unit 13 determines the gradation of pixels whose gradation (difference) exceeds a preset difference threshold as " 1”, and binarization is performed by setting the gradation of pixels equal to or lower than the difference threshold to “0”.
That is, the changed region extracting unit 13 extracts pixels that have changed from immediately before and after.

In each of the binarized difference image T2-T1 and the difference image T2-T3, the changing region extracting unit 13 defines a region of pixels in which pixels having a gradation of "1" overlap as a changing region, and extracts pixels of the changing region. A first masking image is generated in which the gradation is set to "1" and the gradation of pixels other than the change area is set to "0".
That is, the changing area extracting unit 13 obtains the logical product of the gradation levels of the pixels corresponding to the binarized difference image T2-T1 and the differential image T2-T3, and uses the result of the logical product as the gradation level. Generate a masking image.

Here, for each of the generated first masking images, the changing region extracting unit 13 adds information indicating the chronological order of the framed images and the first masking image to the end of the moving image name. is written in the storage unit 20 and stored.
The file name of the first masking image is framed image (video name)_2. For images corresponding to jpg, (movie name)_2_m1. jpg, and in addition to the moving image name and the frame number indicating the chronological order in the image data of the moving image, m1 indicating that it is the first masking image is added.

The skin-color region extraction unit 14 extracts a skin-color region, which is a region containing skin color, from among the changed regions in the first masking image, using each of the corresponding framed image and the first masking image in chronological order. .
Here, the skin color region extraction unit 14 extracts the first masking image (video name)_n_m1. A framed image (video name)_n.jpg corresponding to the changed area in jpg. Determine if the image area in the jpg contains skin tones.

At this time, the skin color region extraction unit converts the RGB (red, green, blue) values of the gradation of each pixel of the framed image into coordinate values in the HSV (hue, saturation, value) color space, and extracts the predetermined skin color. Pixels corresponding to coordinate values included in the color space area to be the area are extracted as pixels of the skin color area.
Compared to the RGB color space, the HSV color space is more robust to changes in "vividity" and "brightness", so it extracts the skin color area (human skin color) in the framed image where the ambient light changes. It is possible to make it easier.

As a result, the skin color region extraction unit 14 extracts the framed image (video name)_n. A first masking image (video name)_n_m1 corresponding to an image area containing skin color in jpg. The change area in jpg is extracted as a skin color area.
At this time, in each of the extracted skin color regions, the skin color region extraction unit 14 extracts the pixel value of each pixel from a plurality of neighboring pixels surrounding each pixel, for example, 48 pixels (7×7 pixels centered on the target pixel). In the pixel, the pixel value is converted to the median value of the pixel values of 48 pixels excluding the center pixel. That is, smoothing is performed using a median filter.

Then, the skin color region extraction unit 14 extracts the first masking image (video name)_n_m1. A second masking image is generated by converting the gradation of pixels in the skin-colored changing area in jpg to "1" and converting the gradation of pixels in the area including other changing areas to "0". The second masking image may have one skin color area or a plurality of skin color areas depending on the operating conditions of the driver and the passenger.
The file name of the second masking image is framed image (video name)_n. For images corresponding to jpg, (movie name)_n_m2. jpg, and in addition to the moving image name and the frame number indicating the chronological order in the image data of the moving image, m2 indicating that it is the second masking image is added.

The rectangular area generation unit 15 generates a framed image (video name)_n. jpg, the second masking image (video name)_n_m2. Cut out the area corresponding to each skin color area in jpg.
Then, the rectangular area generation unit 15 generates a framed image (video name)_n. Generate a clipped image consisting of jpg regions.

At this time, as already described, the second masking image (video name)_n_m2. jpg, if there are multiple skin-colored areas, the framed image (movie name)_n. A plurality of clipped images are clipped from jpg.
For this reason, in order to simplify subsequent processing, the rectangular area generation unit 15 removes skin color areas that exceed a preset number of pixels or skin color areas that are less than the number of pixels from the second masking image.

Then, the rectangular area generation unit 15 generates a rectangular area surrounding the image area of the framed image corresponding to the skin color area of the second masking image, converts the rectangular area into a square (area enlargement), and produces a framed image. An image area surrounded by this rectangular area is cut out from the image to generate a cut-out image.
The rectangular area generation unit 15 assigns a file name to each of the generated rectangular images, and writes and stores them in the storage unit 20 .
The file name of the clipped image is framed image (video name)_n. For images corresponding to jpg, (movie name)_n_r. jpg, and in addition to the name of the moving image and the frame number indicating the chronological order in the image data of the moving image, the clipped image number r indicating the number of the clipped image in the framed image (which of the plurality) is given.

FIG. 2 is a conceptual diagram illustrating generation of a clipped image from a framed image by clipping processing based on the second masking image by the rectangular area generation unit 15 according to the present embodiment.
FIG. 2(a) is an image area showing a hand 201 of a driver operating a smart phone 200 (an example of a mobile terminal) in a framed image.

FIG. 2(b) shows a rectangular area 100S obtained by enclosing an area 220 in the framed image with a rectangular frame 100, which corresponds to the skin color area of the second masking image. A region 150 is a flesh-colored image region, but is removed from the second masking image because it has less than a predetermined number of pixels, and is not extracted as a rectangular region.
In FIG. 2B, the rectangular area 100S has a side (right side and left side) in the x-axis direction with the number of pixels a and a side (upper side and lower side) in the y-axis direction with the number of pixels b.

2C, the upper side of the rectangular frame 100 is extended by b/2 pixels in the positive direction of the y-axis, the lower side is extended by b/2 pixels in the negative direction of the y-axis, and the right side of the rectangular frame 100 is extended in the x-axis direction. It shows that the cropped image 300, which is a square image, is generated by extending the number of pixels by a/2 in the positive direction and extending the left side by the number of pixels by a/2 in the negative direction of the x-axis.
A machine learning model for inputting a general image and classifying the input image, which will be described later, requires that the input image have an aspect ratio of "1". For this reason, in the present embodiment, processing is performed to make the clipped image a square. This eliminates the process of adjusting the aspect ratio of the clipped image when inputting it to the machine learning model, which will be described later. can be raised.

Returning to FIG. 1, the driver state determination unit 16 inputs each cutout image generated by the rectangular area generation unit 15 to the motion classification model, which is the machine learning model, and performs class classification processing for each cutout image. do The motion classification model classifies, for example, a hand operating a smartphone (class #1), a hand operating an object (steering wheel, shift lever, etc.) that does not need to be detected (class #2), and a human faces (class #3), human arms (class #4), and other objects (flesh colored objects, class #5). Also, the motion classification model inputs each of the images of the above classes, and trains the images corresponding to each class (class #1 to class #5) so as to increase the confidence that the image is classified into the class indicated by the image. It is machine-learned in advance using it as data (data obtained by visually checking images and classifying them into classes).

Further, the driver state determination unit 16 writes and stores the framed image obtained by cutting out the cutout image and the classification class of the cutout image in the framed image in the driver state determination table in the storage unit 20 . Here, the driver state determination unit 16 receives the clipped image and selects the class with the highest degree of certainty among the classes output as class classification by the action classification model as the classified class. .

FIG. 3 is a diagram showing a configuration example of the driver state determination table of the storage unit 20 in this embodiment. The driver state determination table of FIG. 3 has columns of file name of framed image, classified class, and degree of certainty of classified class for each record.
The file name of the framed image is (video name)_n. jpg file name. In addition, the file name of the framed image for each record indicates the file name of the framed image corresponding to the chronological order when the moving image is framed as the framed image.

The classified classes are class #1 (portable terminal operation class showing the driver's hand operating a smartphone) and class #2 (objects that do not need to be detected (steering wheels, shift levers, etc.). class #3 (human face), class #4 (human arm), class #5 (other objects), and so on. Also, this classified class is the class with the highest degree of certainty among classes #1 to #5 in the class classification of the clipped image to which the action classification model is input. The maximum certainty factor indicates the certainty factor of the class described in the classified class column.

Returning to FIG. 1, the vehicle travel determination unit 17 inputs each of the first masking images to a travel classification model, which is a machine learning model, and determines whether the first masking image is classified into the class in which the vehicle is traveling. make a judgment as to whether The driving classification model classifies, for example, a driving class (a class in which the vehicle is running), a stop class (a class in which the vehicle is in a running state), and the like. Also, the driving classification model inputs each of the images of the above classes, and corresponds to each of the driving class and the stop class in the same manner as the motion classification model so as to increase the certainty of being classified into the class indicated by the image. The images are machine-learned in advance using teacher data as learning data.

Further, the vehicle travel determination unit 17 writes and stores the framed image corresponding to the first masking image and the classification class of the first masking image in the travel state determination table in the storage unit 20 . Here, the vehicle travel determination unit 17 receives the first masking image and selects the class with the highest degree of certainty among the classes output as class classification by the travel classification model as the classified class. .
FIG. 4 is a diagram showing a configuration example of the running state determination table of the storage unit 20 in this embodiment. The driving state determination table in FIG. 4 has columns for each record, name of the file name of the framed image, classified class, and degree of certainty of the classified class.

The file name of the framed image is (video name)_n. jpg file name. In addition, the file name of the framed image for each record indicates the file name of the framed image corresponding to the chronological order when the moving image is framed as the framed image. Classified classes include the above-described driving class (a class in which the vehicle is in a running state), a stop class (a class in which the vehicle is in a stopped state), and the like. Also, this classified class is the class with the highest degree of certainty among the driving class to the stop class in class classification of the first masking image to which the driving classification model is input. The maximum certainty factor indicates the certainty factor of the class described in the classified class column.

Returning to FIG. 1, the result integration unit 18 refers to each of the driver state determination table and the driving state determination table in the storage unit 20, and each of the classification classes is class #1 (portable terminal operation class) and the driving class Search for framed images where . Here, the result integration unit 18 extracts the framed images whose classification classes are the mobile terminal operation class and the driving class as driving candidate images. The driving candidate image indicates a framed image in which the driver is estimated to be driving while driving with a high degree of certainty.

Then, the result integration unit 18 writes data to the driving state flag of the scene table in the storage unit 20 to indicate whether each of the time-series consecutive framed images is a driving candidate image.
Further, the result integration unit 18 extracts a driving determination scene after the determination as to whether or not all the framed images in the storage unit 20 are driving candidate images. This driving determination scene is extracted as a driving determination scene from framed images of distracted driving candidate images to framed images of normal driving candidate images in which distracted driving is not performed, from the framed images in time series.

FIG. 5 is a conceptual diagram showing the process of extracting a driving scene by the result integration unit 18 of this embodiment. In FIG. 5, scene #S1 ends with a framed image (video name)_n-8.jpg that is not a driving candidate image. Scene #S2 starts with a framed image (movie name)_n.jpg that is a driving candidate image and ends with a framed image (movie name)_n-8.jpg that is not a driving candidate image. Scene #S3 starts with a framed image (animation name)_n+q.jpg, which is a driving candidate image.

Image areas 501_1 and 501_2 in the framed image (movie name)_n.jpg of scene #S2 respectively correspond to the changed area of the car window and the changed area of the driver's hand.
Then, the changed area corresponding to the image area 501_1 in the first masking image is classified into the driving class by the driving state classification model, and the clipped image corresponding to the changed area (skin color area) of the image area 501_2 is classified into the portable terminal by the action classification model. classified into operational classes.
As a result, the result integrating unit 18 classifies the first masking image (moving image name)_n_m1.jpg into the driving class, and the clipped image (moving image name)_n_r.jpg into the mobile terminal operation class. (movie name)_n.jpg is determined to be a candidate image for driving while driving.

Similarly, the result integration unit 18 classifies the image area 502_1 into the driving class and the image area 502_2 into the mobile terminal operation class. Determined to be an image.
On the other hand, the changed area corresponding to the image area 501_1 in the first masking image is classified into the driving class by the driving state classification model, and the clipped image corresponding to the changed area (flesh color area) of the image area 501_2 is class ## by the action classification model. classified as 5.
As a result, the result integrating unit 18 determines that the framed image (video name)_n+p.jpg is a normal driving candidate image that is not a driving candidate image.
Then, the result integrating unit 18 sets the time-series framed image group from the framed image (moving image name)_n.jpg to the framed image (moving image name)_n+p.jpg as scene #S2.

Returning to FIG. 1, the image presentation control unit 19 refers to the scene table in the storage unit 20, and extracts a framed image having the highest degree of certainty in the mobile terminal operation class for each driving determination scene. , is displayed on a display unit (not shown).
In this process, a user, for example, a driver who manages driving selects an arbitrary captured image (movie name)_n.mpg input from the drive recorder to the driving condition monitoring support system 1, This is performed by performing control to display a driving candidate image, which is an image showing the driving state of the driver in this imaged image (movie name)_n.mpg.

Further, in the present embodiment, the first masking image (moving image name)_n_m1.jpg, the second masking image (moving image name)_n_m2.jpg, and the clipped image ( Movie name)_n_r.jpg Each frame number is associated with the frame number in the file name.

FIG. 6 is a diagram showing a configuration example of a scene table of the storage unit 20 in this embodiment. The scene table in FIG. 6 has columns for each record, namely framed image file name, driving state flag, and scene number.
The framed image file name is a file name indicated by (moving image name)_n.jpg including information (n) indicating the chronological order when a moving image is framed as a framed image.

The driving state flag is a flag indicating whether or not the framed image is determined to be the driving candidate image. For example, this driving state flag is set to "1" in the case of the image candidate for the distracted driving image, and is set to "0" in the case of the image candidate for the normal driving image, which is driving without the distracted driving. The scene number indicates the chronological number of the scene to which the framed image belongs. Here, the scene is a unit from the framed image with the driving flag of "1" to the framed image with the driving flag of "0". For this reason, continuous normal operation image candidates may not belong to a scene.

Next, using FIGS. 7 and 8, the flow of processing for supporting monitoring of driving by the driving state monitoring support system of the present embodiment will be described. FIG. 7 is a flowchart showing an operation example of a process for supporting monitoring of driving in the driving state monitoring support system of the present embodiment. FIG. 8 is a conceptual diagram for explaining image data generated by processing by each configuration in the driving state monitoring support system.

Step S101: The captured image input unit 11 inputs image data of a moving image captured by the imaging device 2, assigns a moving image name as a file name, and converts the image data of the moving image as shown in FIG. is written as a file name ((moving image name).mpg) in the storage unit 20 and stored.

Step S102: The framed image generation unit 12 generates the image data (movie name) of the moving image stored in the storage unit 20. mpg is read out and converted into a predetermined number of framed images per unit time (for example, one second) as shown in FIG. 8(b).
Then, the framed image generation unit 12 adds (moving image name)_n. jpg, and write and store them in the storage unit 20 in chronological order according to the order of the frame numbers.

Step S103: The changing area extracting unit 13 sequentially reads the framed images (moving image name)_n.jpg from the storage unit 20, and converts each of them to monochrome to generate a grayscale image.
Then, the changing area extracting unit 13 temporarily writes and stores the grayscale image in the storage unit 20 in association with the framed image (moving image name)_n.jpg.

Step S104: The changing region extracting unit 13 corresponds to the framed image (moving image name)_n.jpg indicating the changing region composed of changing pixels in the framed image (moving image name)_n.jpg in chronological order. generating a first masking image for
At this time, the changing area extracting unit 13 obtains the difference in gradation between the grayscale image and corresponding pixels of other grayscale images immediately before and after the grayscale image.
Then, the changed region extracting unit 13 sets the gradient of each pixel whose difference in gradient between the target grayscale image and the immediately preceding grayscale image exceeds the difference threshold to “1”, A first difference image (difference image T2-T1 described above) is generated by performing binarization with pixel gradation set to "0".

In addition, the changed region extracting unit 13 sets the gradient of each pixel whose difference in gradient between the target grayscale image and the immediately following grayscale image exceeds the difference threshold to "1", and A second difference image (the above-described difference image T2-T3) is generated by performing binarization with the gradation of pixels set to "0".
Next, the changing region extracting unit 13 extracts a framed image (video name)_1.jpg in each of the binarized first and second difference images of the gradation, as shown in FIG. 8(c). , the region of pixels where pixels with a gradation of "1" overlap is defined as a changing region, the gradation of pixels in the changing region is set to "1", and the gradation of pixels other than the changing region is set to "0". The first masking image (video name)_n_m1. generate jpg.

Step S105: The skin color region extraction unit 14 extracts the first masking image (video name)_n_m1 . A skin color area, which is an area including skin color, is extracted from the change area in jpg.
Here, the skin color region extraction unit 14 extracts the first masking image (video name)_n_m1. A framed image (video name)_n.jpg corresponding to the changed area in jpg. Determine if the image area in the jpg contains skin tones.
As a result, the skin color region extraction unit 14 extracts the framed image (video name)_n. A first masking image (video name)_n_m1 corresponding to an image area containing skin color in jpg. The change area in jpg is extracted as a skin color area.

Step S106: The skin color region extraction unit 14 extracts the first masking image (video name)_n_m1. A second masking image (moving image name)_n_m2.jpg in which the gradation of the pixels in the skin-colored changing area is set to "1" and the gradation of the pixels in the area including the other changing area is converted to "0". jpg as a framed image (video name)_n. Generated corresponding to jpg.

Step S107: As shown in FIG. 8(e), the rectangular area generation unit 15 generates framed images (video name)_n. jpg, regions corresponding to each of the skin color regions in the second masking image (movie name)_n_m2.jpg are cut out, and the cutout image (movie name)_n_r.jpg is obtained. generate jpg.

Step S108: The driver's state determination unit 16 extracts the clipped image (movie name)_n_r. jpg is input to an action classification model, which is a machine learning model, and extracted images (movie name)_n_r. Classify each jpg.
Then, the driver state determination unit 16 selects the clipped image (video name)_n_r. As a result of class classification of jpg, the degree of confidence for classification into each class is obtained.
As a result, the driver state determination unit 16 selects the class with the highest degree of certainty as the classified class, and extracts the clipped image (video name)_n_r. jpg is associated with the clipped framed image (movie name)_n.jpg, and is written and stored in the driver state determination table of the storage unit 20 .

Step S109: The vehicle travel determination unit 17 sequentially reads each of the first masking images from the storage unit 20, and inputs them to the travel classification model, which is a machine learning model.
As a result, the vehicle travel determination unit 17 classifies each of the first masking images (video name)_n_m1.jpg into the travel class or the stop class.
Then, the vehicle running determination unit 17 stores the class (running class or stop class) as a result of class classification by the running classification model in each of the columns of the classified classes and the columns of the degree of certainty in the running state determination table of the storage unit 20 . class) and confidence are written and stored. At this time, the vehicle travel determination unit 17 determines that the traveling class or the stopped class, whichever has the higher degree of certainty, is the class classified by the class classification.

Step S<b>110 : The result integration unit 18 refers to each of the driver state determination table and the driving state determination table in the storage unit 20 .
Then, the result integration unit 18 compares the classes classified by the motion classification model and the classes classified by the driving classification model. Here, the result integration unit 18 extracts the framed images whose classification classes are the mobile terminal operation class and the driving class as driving candidate images. Meanwhile, the driving candidate image indicates a framed image (movie name)_n.jpg that is estimated with a high degree of certainty that the driver is driving while driving.

On the other hand, if the class classified by the action classification model is not class #1, or if the class classified by the driving classification model is not the driving class, the result integration unit 18 converts the framed image (video name)_n.jpg to The image is a normal driving candidate image showing that the driver is not driving while distracted.
Then, the result integrating unit 18 stores the driving state flag column of the record corresponding to each of the framed images in the scene table of the storage unit 20 as a flag when the framed image is determined to be a driving candidate image. Enter "1". In addition, the result integrating unit 18 sets a flag in the driving state flag column of the record corresponding to each framed image in the scene table of the storage unit 20 when the framed image is determined to be a normal driving candidate image. Enter "0".

Step S111: The result integration unit 18 refers to the scene table in the storage unit 20, and confirms each driving state flag in each record of the framed image.
Then, the result integration unit 18 refers to the scene table in the storage unit 20, groups the framed images (movie name)_n.jpg with the driving state flags from “1” to “0” in time series, and groups them. A scene number is assigned to the group of framed images, and the scene is defined as a driving determination scene.
In addition, the result integrating unit 18 stores the scene number column corresponding to the grouped framed image (video name)_n.jpg in the scene table of the storage unit 20 for the driving determination scene to which the framed image belongs. Write the scene number. On the other hand, the result integrating unit 18 writes a character indicating that the framed image (video name)_n.jpg, which is not included in the driving determination scene, does not belong to the driving determination scene, for example, "NO", in the scene number column. .

With the above-described configuration, according to the present embodiment, there is no need to mount an acceleration device, a receiver, etc. on a vehicle as in the conventional art, and a fixed-point imaging device such as a drive recorder generally mounted on a vehicle can be used. In order to extract a driving candidate image while the driver operates the mobile terminal and the vehicle is likely to be running using the captured image, the person in charge of managing the driver is asked to It is possible to easily present a driving determination scene that is highly likely to be performed, and to assist the driver in facilitating the detection of driving.

Next, the clipped image (video name)_n_r. In the jpg class classification, an operation for improving classification accuracy for the mobile terminal operation class (class #1) will be described.
FIG. 9 shows a clipped image (movie name)_n_r. FIG. 10 is a conceptual diagram for explaining a process for improving the accuracy of classification into mobile terminal operation classes in class classification of jpg. The driver state determination unit 16 divides the framed image (movie name)_n.jpg (300) into a predetermined plurality of (h×i) segments (image regions) 300S.
Then, the driver's state determination unit 16 sets the initial value of the coefficient α to be given to the positions of all the divided segments to 1.0.
In addition, the driver state determination unit 16 selects the clipped image (movie name)_n_r. The existence probability α of the position of the segment 300S overlapping jpg is multiplied by a predetermined coefficient β (for example, 1.1), and the existence probability α of the position of the segment 300S other than that is multiplied by a predetermined coefficient γ (for example, 0.9).

As described above, the driver state determination unit 16 extracts the clipped image (video name)_n_r. Every time jpg class classification is performed, the existence probability α at the position of each segment is multiplied by coefficients β and γ, respectively.
As a result, in each of the time-series framed images, the presence probability α of the segment overlapping the clipped image classified into the mobile terminal operation class increases sequentially. On the other hand, the existence probability α of the segment at the position not divided into the mobile terminal operation class gradually decreases.

Then, when the action classification model classifies the cut-out image (movie name)_n_r.jpg cut out from the framed image (movie name)_n.jpg, the driver state determination unit 16 determines that the cut-out image The geometric mean of the existence probability α of the segment overlapping (movie name)_n_r.jpg is multiplied by the certainty factor of class #1 of the clipped image.
In each of the segments 300S corresponding to the position of the framed image (video name)_n.jpg (300) in FIG. 9, the clipped image (video name)_n_r.jpg classified into class #1 by the action classification model overlaps. Since it exists at the position, the existence probability α of the segment in the region 350 is greater than the initial value "1".

For example, after the class classification of the clipped image clipped from the framed image (movie name)_n.jpg is completed, the driver state determination unit 16 determines that the immediately preceding framed image (movie name)_n-1.jpg Using the set existence probability α of each segment, the certainty factor of class #1 output as a result of classification is corrected.
That is, the driver state determination unit 16 calculates the geometric mean of the existence probability α of the segment 300S in the region 360 overlapping the clipped image clipped from the framed image (movie name)_n.jpg.

For example, if segment 300S_1 in region 360 has α=0.8, 300S_2 has α=0.9, 300S_3 has α=0.9, 300S_4 has α=1.1, .
Geometric mean Σ=(0.8×0.9×0.9×1.1×…×1.2) ^1/9
calculated as

Then, the driver state determination unit 16 multiplies the confidence of class #1 by the geometric mean Σ in the class classification of the clipped image clipped from the framed image (movie name)_n.jpg.
Further, the driver state determination unit 16 multiplies the certainty of class #1 (portable terminal operation class) by the geometric mean Σ, compares the certainty of each classified class, and finds the highest certainty. class as the result of classification.

As a result, if the certainty of the mobile terminal operation class is slightly smaller than the other classes immediately after the class classification by the action classification model, from the existence probability α of the segment overlapping the clipped image (movie name)_n_r.jpg When the obtained geometric mean Σ exceeds 1, the degree of certainty is reversed, and the probability that the mobile terminal operation class is selected as the class classification increases.
The same person has a habit of operating the smartphone and often operates the smartphone in the same position. Classification accuracy can be improved.

In addition, the confidence map described above is sequentially updated by clipped images in the framed image generated from the input captured image, or is reset for each driving determination scene and sequentially updated by the clipped image in the driving determination scene. Alternatively, the similarity between the preceding and succeeding framed images is compared, and the driving candidate image is reset at the timing when a scene change below a predetermined similarity threshold occurs in the image, and the subsequent framed images are sequentially configured. is appropriately set for each target to be extracted.

In addition, using video data captured by the fixed-point imaging device of the driving state monitoring support system shown in FIG. A program for realizing the function to extract and present is recorded in a computer-readable recording medium, and the program recorded in this recording medium is read into a computer system and executed. A process of extracting and presenting driving candidate images from the moving image data may be performed. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices.

The "computer system" also includes the home page providing environment (or display environment) if the WWW system is used.
The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. Furthermore, "computer-readable recording medium" means a medium that dynamically retains a program for a short period of time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It also includes those that hold programs for a certain period of time, such as volatile memories inside computer systems that serve as servers and clients in that case. Further, the program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system.

Although the embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and designs and the like are included within the scope of the gist of the present invention.

REFERENCE SIGNS LIST 1 Driving state monitoring support system 11 Captured image input unit 12 Framed image generation unit 13 Change region extraction unit 14 Skin color region extraction unit 15 Rectangular region generation unit 16 Driver state determination unit 17 Vehicle running determination unit 18... Result integration unit 19... Image presentation control unit 20... Storage unit

Claims (7)

Each of the image data captured in time series by a fixed-point imaging device that captures an image of the interior of the vehicle, including the driver's seat, with a fixed imaging range is compared with each of the previous image data and the immediately following image data, and pixels having changes are detected. a changing region extraction unit that generates a masking image showing a changing region consisting of
a skin color region extraction unit for extracting a skin color region including skin color pixels from a region corresponding to the changed region in the image data;
The skin-colored region is input to an action classification model, which is a machine learning model for classifying the skin-colored region into one of a plurality of classes indicating the action state of the driver, and classifying the action state of the driver. a person state determination unit;
The masking image is input to a driving classification model, which is a machine learning model that classifies the masking image into either a driving class in which the vehicle is running or a stop class in which the vehicle is stopped. , a vehicle travel determination unit that performs class classification processing ;
When the driver state determination unit classifies the skin color region into the portable terminal operation class in which the driver is operating the portable terminal, and the vehicle driving determination unit classifies the masking image into the driving class, the corresponding a result integration unit that extracts the image data as an image candidate of the driver operating the portable terminal while driving the vehicle;
A driving state monitoring support system comprising: The changed region extraction unit
Each of the image data is converted into a grayscale image, and the grayscale images immediately before and after the grayscale image are compared with each other, and a difference in gradation between pixels at corresponding positions exceeds a preset threshold. 2. The driving state monitoring support system according to claim 1 , wherein the changed area is extracted as the pixel having a change, if the change occurs. The changed region extraction unit
A region of pixels having a change obtained by comparing said grayscale image and a grayscale image immediately preceding said grayscale image with a region of pixels having a change obtained by comparing said grayscale image and said grayscale image immediately following said grayscale image. 3. The driving state monitoring support system according to claim 2 , wherein a region that overlaps with a pixel region having a change is used as the masking image. The driving classification model is
According to whether or not an image portion corresponding to a car window of the vehicle is changed in the masking image, the state of the vehicle is classified into a driving class or a stop class , respectively. The driving state monitoring support system according to any one of 1. The changing region extracting unit divides the image data into a predetermined number of segments, increases the existence probability of the segments that overlap the skin color region compared to the segments that do not overlap the skin color region, 5. The driving state monitoring support system according to any one of claims 1 to 4 , wherein the reliability of a mobile terminal operation class operating the mobile terminal is increased in accordance with the existence probability. . A changing region extracting unit compares each image data captured in time series by a fixed-point imaging device that captures an image of the interior of the vehicle including the driver's seat in a fixed imaging range with the immediately preceding image data and the immediately following image data. a changing region extraction process for generating a masking image showing a changing region composed of pixels having a change;
a skin color region extraction step in which a skin color region extraction unit extracts a skin color region including skin color pixels from a region corresponding to the changed region in the image data;
A driver state determination unit inputs the skin color region to an action classification model, which is a machine learning model for classifying the skin color region into one of a plurality of classes indicating the action state of the driver, and determines the action of the driver. a driver state determination process for classifying states;
A driving classification model, which is a machine learning model in which the vehicle driving determination unit classifies the masked image into either a driving class in which the vehicle is driving or a stop class in which the vehicle is stopped, a vehicle travel determination process of inputting the masking image and performing class classification processing ;
A result integration unit classifies the flesh-colored area into a mobile terminal operation class in which a driver operates a mobile terminal by the driver state determination process, and classifies the masking image into the driving class by the vehicle travel determination process. a result integration step of extracting the corresponding image data as an image candidate of the driver operating the portable terminal while driving the vehicle,
A driving state monitoring support method, comprising: the computer,
Each of the image data captured in time series by a fixed-point imaging device that captures an image of the interior of the vehicle, including the driver's seat, with a fixed imaging range is compared with each of the previous image data and the immediately following image data, and pixels having changes are detected. changing region extracting means for generating a masking image showing a changing region consisting of
skin-color region extracting means for extracting a skin-color region including skin-color pixels from a region corresponding to the changed region in the image data;
The skin-colored region is input to an action classification model, which is a machine learning model for classifying the skin-colored region into one of a plurality of classes indicating the action state of the driver, and classifying the action state of the driver. person state determination means,
The masking image is input to a driving classification model, which is a machine learning model that classifies the masking image into either a driving class in which the vehicle is running or a stop class in which the vehicle is stopped. , vehicle travel determination means for performing class classification processing,
When the driver state determination means classifies the skin color area into the mobile terminal operation class in which the driver is operating the mobile terminal, and the vehicle travel determination means classifies the masking image into the travel class, the corresponding Result integration means for extracting the image data as an image candidate of the driver operating the portable terminal while driving the vehicle.
A program to function as

Images