JP2009294753A - Image processor and image processing method - Google Patents

Abstract

An image processing apparatus and an image processing method are provided that can set a threshold value as a reference for a driver's looking aside according to the running state of a vehicle and make a more accurate aside look determination.
An image processing ECU calculates a driver's face orientation angle and outputs it to a vehicle control system. Further, the image processing ECU 3 changes the threshold angle Θ, which serves as a reference for determining whether the driver's face orientation is normal, based on the vehicle information of the vehicle, and controls the changed threshold Θ ′. Output to system 4. The vehicle control system 4 determines whether or not the driver's face orientation is normal based on the face orientation angle θ and the threshold value Θ ′ output from the image processing ECU 3.
[Selection] Figure 7

Description

  The present invention relates to an image processing apparatus and an image processing method, and more specifically, image processing for calculating the driver's face orientation from an image of the driver's face and determining the driver's look-aside state. The present invention relates to an apparatus and an image processing method.

2. Description of the Related Art Conventionally, there has been known an apparatus that images a driver's face of a vehicle by a camera arranged on an instrument panel of the vehicle and detects the driver's eyes and mouth from the captured image. As an example of such an apparatus, there is an apparatus disclosed in Patent Document 1, for example. The device disclosed in Patent Document 1 captures a driver's face with a camera and processes the captured image to detect the driver's eyes and mouth. Further, when the actual vehicle speed is 10 km / h or less, the detection processing is prohibited when the steering angle or the yaw rate is a predetermined value or more.
JP-A-6-293228

  By the way, the driver's face is imaged with a camera or the like installed in the vehicle, and the driver's eyes and mouth (and the driver's face orientation angle) are detected from the captured image. A device that determines the state has a scene where it is necessary to determine the state of the driver even when the vehicle is turning. As an example, the direction of the driver's face is detected from the driver's eyes and mouth, and whether or not the driver is looking aside is determined from the viewpoint of so-called preventive safety in order to prevent accidents. Sometimes it is also necessary, and the driver's condition determination may be more important in situations where the vehicle is traveling on a curve. However, the device described in Patent Document 1 prohibits the process of detecting the driver's eyes and mouth when the actual vehicle speed is 10 km / h or less and the steering angle or yaw rate is a predetermined value or more. ing. For this reason, the driver's state may not be determined when the vehicle turns.

  Further, in an apparatus that images a driver's face of the vehicle by a camera installed in the vehicle and determines the driver's side-view state from the captured image, the device is generally directed to the driver. When the driver's face angle exceeds the threshold set to the left and right with respect to the camera based on the state of facing the front of the camera, it is determined that the driver is looking aside. When it is determined whether or not the driver is looking aside in this way, for example, when the vehicle turns, the face angle exceeds the threshold even though the driver is looking in the direction in which the vehicle will travel. , There is a possibility that it will be determined to be in a state of looking aside. On the other hand, even when the driver is distracted by the scenery in front of the vehicle when turning the vehicle and is not looking at the direction in which the vehicle is traveling, the face angle is within the threshold and the driver is not looking aside. There is a possibility that it will be judged.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to set a threshold value as a reference for a driver's side-arming determination according to the traveling state of the vehicle, and to make a more accurate side-arming determination. An object is to provide an image processing apparatus and an image processing method.

  In order to achieve the above object, the present invention employs the following configuration. In other words, the first invention uses the image captured by the imaging unit that captures the face of the driver seated in the driver's seat of the vehicle, and determines the driver's face orientation angle with respect to the imaging direction of the imaging unit. A face direction angle calculating means for calculating; a vehicle information acquiring means for acquiring vehicle information of the vehicle; a threshold angle setting means for setting a threshold angle on at least one of left and right with respect to the imaging direction; and the vehicle information acquisition. The vehicle information acquired by the means is used to calculate a predicted gazing direction that is predicted to be watched by the driver when the vehicle is turning, and at least one of the threshold angles set by the threshold angle setting means is calculated as the predicted caution. Threshold angle moving means for moving in a direction in which the viewpoint direction changes, and determination means for determining that the driver is in an aside state when the driver's face orientation angle deviates from the threshold angle. An image processing apparatus. Note that the threshold angle setting means of the present invention may provide two threshold angles on the left and right with respect to the imaging direction of the imaging means. In this case, the threshold angle moving means may move the two threshold angles in the direction in which the predicted gazing point direction changes, or move only one of the threshold angles in the direction in which the predicted gazing point direction changes. It ’s okay. Further, the threshold angle setting means of the present invention may provide one threshold angle only in the left direction with respect to the imaging direction of the imaging means. In this case, the threshold angle moving means moves one threshold angle provided in the left direction in a direction in which the predicted gazing direction changes. The threshold angle setting means of the present invention may provide one threshold angle only in the right direction with respect to the imaging direction of the imaging means. In this case, the threshold angle moving means moves one threshold angle provided in the right direction in a direction in which the predicted gazing direction changes.

  In a second aspect based on the first aspect, the threshold angle setting means sets the threshold angle on both the left and right sides with respect to the imaging direction, and the threshold angle moving means sets the threshold angle. An image processing apparatus that moves the predicted gazing point direction in a changing direction.

  According to a third invention, in the first or second invention, the threshold angle moving means predicts an arrival point of the vehicle after elapse of a predetermined time using the vehicle information, and the arrival from the vehicle. The image processing apparatus calculates a direction to a point as the predicted gaze direction.

  In a fourth aspect based on the third aspect, the threshold angle moving means moves the threshold angle by a first angle formed by a speed direction indicating the moving direction of the vehicle at the current time and the predicted gaze direction. An image processing apparatus.

  According to a fifth invention, in the fourth invention, vehicle specification storage means for storing the vehicle specifications of the vehicle is further provided. Further, the threshold angle moving means calculates a slip angle indicating a side-slip angle of the center of gravity of the vehicle as a second angle by using the vehicle information and the vehicle specifications, and the first angle and the second angle. Is the image processing apparatus that moves the threshold angle by the angle difference between

  In a sixth aspect based on the fourth aspect, the threshold angle moving means calculates, as a third angle, an angle formed by the speed direction and the imaging direction, and the first angle and the third angle are calculated. The image processing apparatus moves the threshold angle by an angle difference.

  In a seventh aspect based on the third aspect, the vehicle information acquired by the vehicle information acquisition means includes a yaw rate of the vehicle. The threshold angle moving means is an image processing device that predicts the arrival point using the yaw rate.

  In an eighth aspect based on the third aspect, further comprising vehicle specification storage means for storing the vehicle specifications of the vehicle. The vehicle information acquired by the vehicle information acquisition means includes the yaw rate of the vehicle and the vehicle speed of the vehicle, and the threshold angle moving means has predetermined the yaw rate, the vehicle speed, and the vehicle specifications. An image processing apparatus that calculates a direction to the arrival point by substituting into a mathematical expression.

  In a ninth aspect based on the third aspect, the vehicle further includes vehicle specification storage means for storing the vehicle specifications of the vehicle. The vehicle information acquired by the vehicle information acquisition unit includes the lateral acceleration of the vehicle and the vehicle speed of the vehicle, and the threshold angle moving unit determines the lateral acceleration, the vehicle speed, and the vehicle specifications in advance. This is an image processing apparatus that calculates the direction to the arrival point by substituting into the obtained mathematical expression.

  In a tenth aspect based on the third aspect, the apparatus further comprises vehicle specification storage means for storing the vehicle specifications of the vehicle. The vehicle information acquired by the vehicle information acquisition means includes the steering angle of the vehicle and the vehicle speed of the vehicle, and the threshold angle moving means determines the steering angle, the vehicle speed, and the vehicle specifications in advance. This is an image processing apparatus that calculates the direction to the arrival point by substituting into the obtained mathematical expression.

  According to an eleventh aspect of the invention, in the first or second aspect of the invention, when the driver's face orientation calculated by the face orientation angle calculating means deviates from the threshold angle, the vehicle is controlled to perform the driving. It is an image processing apparatus further provided with the control means which alert | reports to a person.

  In a twelfth aspect of the present invention, using the image captured by the imaging unit that captures the face of the driver seated in the driver's seat of the vehicle, the driver's face angle is calculated based on the imaging direction of the imaging unit. A face angle calculation step, a vehicle information acquisition step for acquiring vehicle information of the vehicle, a threshold angle setting step for setting a threshold angle on at least one of left and right with respect to the imaging direction, and the vehicle information acquisition step. A predicted gaze direction predicted to be watched by the driver when the vehicle turns is calculated using the acquired vehicle information, and at least one of the threshold angles set in the threshold angle setting step is calculated as the predicted gaze direction. A threshold angle moving step for moving in a direction in which the driver changes, and a determination that the driver is in a look-aside state when the face angle of the driver deviates from the threshold angle. And a step, an image processing method.

  According to the first aspect of the invention, for example, when the vehicle is turning, the driver of the vehicle determines that the driver is looking aside despite looking at the turning direction. Or when the driver is looking at the front of the vehicle when the vehicle is turning (ie, the driver is not looking in the direction the vehicle is turning) Can be prevented. In other words, by setting a threshold value that serves as a reference for a driver's side-by-side determination according to the traveling state of the vehicle, a more accurate side-by-side determination can be made.

  According to the second aspect, threshold angles are provided on the left and right with respect to the imaging direction, respectively, and the threshold angles are moved in directions in which the predicted gaze direction changes. For example, if the driver is not looking at the direction that must be watched for safe driving, it is not only determined that the driver is looking aside. Can be prevented.

  According to the third aspect, the predicted gazing direction can be easily calculated. Generally, when a vehicle is turning, a driver who intends to drive safely watches the direction in which the vehicle travels. In other words, the driver predicts to some extent a point that the vehicle will reach after a certain period of time, and pays attention to the point for safe driving. Therefore, if the elapsed time that is the point at which the driver is gazing is set in advance, the predicted gazing direction can be easily calculated using the elapsed time.

  According to the fourth aspect of the invention, the angle formed by the speed direction of the vehicle and the predicted gazing point direction is calculated, and the threshold angle is moved by the angle. In other words, since the threshold angle is moved in consideration of the angle formed by the speed direction of the vehicle and the predicted gazing direction, the direction that the driver who changes depending on the driving state of the vehicle should see is the center. Aside-look judgment can be made.

  According to the fifth aspect of the invention, the threshold angle is moved by an angle that considers not only the angle formed by the speed direction of the vehicle and the predicted gazing point direction but also the slip angle of the vehicle. Therefore, a more accurate threshold angle can be set according to the current behavior of the vehicle.

  According to the sixth aspect, the threshold angle is moved by an angle that also takes into account the slip angle of the vehicle and the angle difference between the imaging direction of the imaging means and the front direction of the vehicle. Therefore, a more accurate threshold angle can be set according to the current behavior of the vehicle and the imaging direction error of the imaging means.

  According to the seventh aspect, by providing, for example, a yaw rate sensor that detects the yaw rate in the vehicle, the predicted gazing direction can be easily calculated.

  According to the eighth aspect of the invention, for example, by providing the vehicle with a vehicle speed sensor and a yaw rate sensor that detect the vehicle speed of the vehicle, the predicted gazing point direction can be easily calculated.

  According to the ninth aspect, the predicted gazing direction can be easily calculated by providing the vehicle with, for example, a lateral acceleration sensor and a vehicle speed sensor that detect the lateral acceleration of the vehicle.

  According to the tenth aspect of the invention, the predicted gaze direction can be easily calculated by providing the vehicle with a steering angle sensor and a vehicle speed sensor that detect the steering angle of the vehicle, for example.

  According to the eleventh aspect of the present invention, when the driver is looking aside or looking away, the driver can be alerted by voice or display using the equipment of the vehicle. Therefore, it is possible to prevent accidents caused by the driver's sideways and looking away, and so-called preventive safety is possible. In addition, since the determination means determines that the driver is looking aside when it deviates from the threshold angle moved by the threshold angle movement means, for example, when the driver is looking aside, If the driver is not alerted, or if the driver is not looking aside, the driver may not be alerted.

  According to the image processing method of the present invention, the same effects as those of the above-described image processing apparatus of the present invention can be obtained.

(First embodiment)
The configuration and operation of the image processing apparatus according to the first embodiment of the present invention will be described below with reference to the drawings. In the present embodiment, description will be made assuming that the image processing apparatus is installed in a vehicle (for example, a passenger car, a commercial vehicle, etc.).

  FIG. 1 is a block diagram illustrating an example of a configuration of a side-by-side determination system that includes an image processing apparatus according to the first embodiment of the present invention. In FIG. 1, the aside look determination system includes a camera 1, a vehicle information detection unit 2, an image processing ECU (Electronic Control Unit) 3, and a vehicle control system 4.

  The camera 1 is, for example, a CCD camera, an infrared camera, or the like that images a driver's face. The camera 1 is installed at a position where the face of the driver seated in the driver's seat can be imaged from the front. FIG. 2 is a diagram illustrating an example of the installation position of the camera 1. As shown in FIG. 2, the camera 1 is installed on, for example, the steering column of the vehicle so that the face of the driver seated in the driver's seat can be imaged from the front. The installation position of the camera 1 shown in FIG. 2 is an example. For example, the camera 1 may be installed in the meter hood or the meter panel of the vehicle. Further, the camera 1 images the driver's face from the front at predetermined time intervals. Then, the camera 1 outputs the captured image to the image processing ECU 3.

  The vehicle information detection unit 2 detects vehicle information of the vehicle. Specifically, the vehicle information detection unit 2 includes a speed sensor that detects the speed of the vehicle, a yaw rate sensor that detects the yaw rate ω of the vehicle, and an acceleration in the vehicle width direction (hereinafter referred to as a lateral direction) that acts on the position of the center of gravity of the vehicle. Various sensors such as a lateral acceleration sensor for detecting the acceleration α) and a steering angle sensor for detecting the steering angle δ of the vehicle. Note that the armpit determination system including the image processing apparatus according to the first embodiment may include a vehicle speed sensor and a yaw rate sensor.

  The image processing ECU 3 is an information processing apparatus including a face orientation detection microcomputer (hereinafter referred to as a face orientation detection microcomputer) 31, a face information storage unit 32, a vehicle specification value storage unit 33, an interface circuit, and the like. Further, the camera 1, the vehicle information detection unit 2, and the vehicle control system 4 are connected to the image processing ECU 3.

  The face orientation detection microcomputer 31 includes a CPU (Central Processing Unit) and a memory. The face orientation detection microcomputer 31 uses the image captured by the camera 1 to generate information related to the driver's face orientation (specifically, the driver's face orientation angle). The face orientation detection microcomputer 31 corresponds to an example of a face orientation angle calculation unit, vehicle information acquisition unit, threshold angle setting unit, and threshold angle movement unit of the present invention.

  Here, the face orientation angle of the driver will be described with reference to FIG. FIG. 3 is a top view of the image captured by the camera 1 and the head of the driver. In this embodiment, in order to simplify the description, it is assumed that the head of the driver has a cylindrical shape with the top of the head as the upper surface. The driver's face rotates around the center line of the cylindrical head. Here, it is assumed that the driver's face orientation angle is θ, and the state facing the front with respect to the camera 1 is θ = 0 ° (see FIG. 3A). The face orientation detection microcomputer 31 calculates the face orientation angle θR when the driver turns to the right with respect to the camera 1 and calculates the face orientation angle θL when the driver turns to the left. Specifically, the face orientation angles θR and θL are set with reference to the optical axis of the camera 1 (see FIG. 2). In the state where the driver is seated in the driver's seat of the vehicle, the face-facing angle θR increases as it faces the front right side of the vehicle, and the face-facing angle θL increases as it faces the left side. FIG. 3B is a diagram showing a state where the driver is directed to the right by 20 ° with respect to the camera 1 as an example. In this case, the face orientation detection microcomputer 31 calculates the face orientation angle as θR = 20 °. Similarly, when the driver turns to the left by 20 ° with respect to the camera 1, for example, the face orientation angle is calculated as θL = 20 °. Note that the above-described method for calculating the face direction angle is an example, and the method for calculating the face direction angle is not limited to the above example, and a conventionally known method may be used.

  Returning to the description of FIG. 1, the face information storage unit 32 includes a readable / writable storage medium, and stores face information of various persons. Specifically, face information refers to each piece of information necessary for estimating the face size, eye position and size, mouth position and size, nose position and size, etc. from the image captured by the camera. This is information indicating the position and size of the facial part. The face information includes template images (for example, mouth template images) of various human face parts. Note that the information stored in the face information storage unit 32 may be updated as appropriate by, for example, the rewriting operation of the user of the vehicle.

  The vehicle specification value storage unit 33 is a storage medium that stores the specification values of the vehicle. The vehicle specification values include, for example, the weight M of the vehicle, the length L of the wheel base of the vehicle, the length Lf from the center of gravity of the vehicle to the front wheel shaft of the vehicle, and the center of gravity of the vehicle to the vehicle. It is a unique value of the vehicle such as the length Lr to the rear wheel axle. The vehicle specification value storage unit 33 corresponds to an example of specification storage means of the present invention.

  Although details will be described later, the vehicle control system 4 determines whether or not the driver is looking aside based on the driver's face direction angle output from the face direction detection microcomputer 31 and the threshold angle that is a reference for the look-ahead determination. Is an information processing apparatus for determining Further, if the vehicle control system 4 determines that “the driver is looking aside” as a result of the determination, the vehicle control system 4 controls the vehicle to perform preventive safety. Specifically, the preventive safety mentioned above means that if the vehicle control system 4 determines that the driver is looking aside (side-viewing state), the vehicle's equipment is controlled and a warning or display is used to control the driver. Is to call attention. Furthermore, the preventive safety includes so-called pre-crash control such as assisting a brake operation performed by the driver to avoid a collision with an obstacle. The vehicle control system 4 corresponds to an example of a determination unit and a control unit of the present invention.

  As described above, the face orientation detection microcomputer 31 calculates the driver's face orientation angles θR and θL based on the position of the camera 1 (specifically, the camera optical axis), and the vehicle control system 4 calculates the face orientation angles θR and θL. Outputs the angle. Further, although details will be described later, the face direction detection microcomputer 31 appropriately changes the threshold angle serving as a reference for the look-aside determination according to the traveling state of the vehicle and outputs the calculated angle to the vehicle control system 4. .

  On the other hand, the vehicle control system 4 acquires the driver's face orientation angle θR or θL output from the face orientation detection microcomputer 31. Furthermore, the vehicle control system 4 obtains a threshold angle that is a reference for the look-ahead judgment output from the face orientation detection microcomputer 31, and when the face orientation angle deviates from the threshold angle, “the driver is looking aside. It is judged. Hereinafter, the threshold angle that the vehicle control system 4 uses as a reference for the driver's side-look determination will be described in detail with reference to FIG. 4.

  FIG. 4A is a diagram when the vehicle is traveling straight (hereinafter referred to as a straight traveling state). Hereinafter, with reference to FIG. 4A, the reference for the driver's side-look determination and the threshold angle will be described. As shown in FIG. 4 (a), the threshold angle is the threshold angle ΘR on the right side when viewed from the driver in a state where the driver is seated on the driver's seat with respect to the center line of the vehicle. The left side is ΘL. Then, the vehicle control system 4 acquires the face orientation angles θR and θL and the threshold angles ΘR and ΘL from the face orientation detection microcomputer 31, and the driver's face orientation angles θR and θL deviate from the respective threshold angles. If, in other words, θR> ΘL or θL> ΘL, it is determined that “the driver is looking aside”.

  By the way, for example, when the vehicle is traveling on a curve (hereinafter referred to as a turning state), the driver of the vehicle is not in front of the vehicle, but in the direction that the vehicle will travel from now on (hereinafter, the estimated arrival direction). It is common to look at the planned direction of arrival of the vehicle with the face facing. FIG. 4B is a diagram showing an example of the turning state of the vehicle. As shown in FIG. 4B, assuming that the vehicle is traveling on a right curve as an example, the driver watches the planned arrival direction of the vehicle (specifically, the front right side of the vehicle). Since the threshold angles ΘR and ΘL are applied as they are, inconvenience may occur. Specifically, in FIG. 4A, since the vehicle is traveling straight, the driver is in front (with respect to the camera 1). However, in the case of FIG. 4B, since the driver is facing the right side with respect to the camera 1, when the threshold angle ΘR is applied as it is to determine whether the driver is in the sideways state, the driver watches. The driver's face angle θR may exceed the threshold angle ΘR in spite of looking at the direction that must be present. Therefore, the face orientation detection microcomputer 31 changes at least one of the threshold angles ΘR and ΘL according to the traveling state of the vehicle, and calculates the changed threshold angle Θ′R or Θ′L.

  In this way, the vehicle control system 4 deviates from the threshold angle ΘR even though the driver of the vehicle is looking at the expected direction of arrival (the vehicle front right side in FIG. 4B). It is possible to prevent the driver from judging that the driver is looking aside. In addition, if the vehicle control system 4 determines the driver's sideways state using the threshold angle Θ′R instead of the threshold ΘR, the vehicle control system 4 is based on the determination even if the driver faces his / her face in the direction that the driver must watch. The driver will not be alarmed and the driver will not feel bothered. On the other hand, if the vehicle control system 4 uses the threshold angle Θ′L to determine the driver's look-aside state, the driver is gazing at the front left side of the vehicle (that is, the side opposite to the direction in which the vehicle travels). Nevertheless, the vehicle control system 4 no longer determines that the driver ’s face is normal, and alerts the driver when the driver is in a dangerous driving state looking aside. be able to. In FIG. 4B, the description has been made assuming that the vehicle travels on the right curve. However, the analogy can be applied even when the vehicle travels on the left curve.

  Hereinafter, with reference to FIGS. 5 to 9, an example of operations performed by each unit of the image processing ECU 3 and the vehicle control system 4 according to the present embodiment will be described.

  FIG. 5 is a flowchart showing an example of processing performed in the image processing ECU 3 according to the present embodiment. The process shown in the flowchart shown in FIG. 5 is performed by the face orientation detection microcomputer 31 provided in the image processing ECU 3 executing a predetermined program. Further, the processing shown in the flowchart shown in FIG. 5 is started when the power of the image processing apparatus is turned on (for example, the ignition switch of the vehicle on which the image processing ECU 3 is mounted is turned on). In addition, the processing ends when the power of the image processing ECU 3 is turned off (for example, the ignition switch is turned off). The ignition switch is hereinafter referred to as IG.

  In step S51 of FIG. 5, the face orientation detection microcomputer 31 acquires an image captured by the camera 1, and advances the process to the next step S52.

  In step S52, the face orientation detection microcomputer 31 detects the driver's face orientation. Hereinafter, an example of the operation for calculating the face orientation of the driver performed by the face orientation detection microcomputer 31 will be described with reference to FIG. FIG. 6 is a flowchart showing an example of the operation for calculating the driver's face orientation angle in step S52 of FIG.

  In step S61 of FIG. 6, the face orientation detection microcomputer 31 performs a Sobel filter process in a predetermined direction (for example, the vertical direction) of the image acquired in step S51. Then, the face orientation detection microcomputer 31 uses the difference in luminance in the processed image (Sobel image) to extract a vertical edge point from the processed image, and the process proceeds to the next step S62. Proceed.

  In step S62, the face orientation detection microcomputer 31 performs a process of calculating the width W (see FIG. 3) of the driver's face in the sobel image using the vertical edge points extracted from the sobel image. Here, an example of specific processing performed by the face orientation detection microcomputer 31 will be described. The face orientation detection microcomputer 31 generates a vertical histogram for the extracted vertical edge points, and extracts the right edge and the left edge of the face by detecting the contours of the left and right ends of the face. Then, the face direction detection microcomputer 31 calculates a length obtained by subtracting the left end position from the extracted face right end position as the face width W of the driver, and proceeds to the next step S63.

  In step S63, the face orientation detection microcomputer 31 determines whether or not the driver's face width W has been calculated in step S62. If the face orientation detection microcomputer 31 can calculate the face width W of the driver (YES), the process proceeds to the next step S64. On the other hand, if, for example, the vertical edge point cannot be extracted from the image captured by the camera 1 or the driver's face right edge and face left edge cannot be calculated, the face orientation detection microcomputer 31 returns to step S51 and performs processing. repeat.

  In step S64, the face orientation detection microcomputer 31 uses the face width W calculated in step S62 to estimate the positions of the driver's eyes, nose, and mouth (hereinafter referred to as face parts) in the image captured by the camera 1. To do. For example, the face orientation detection microcomputer 31 uses the data indicating the face width W calculated in step S62 and the face information stored in the face information storage unit 32, so that each face part exists in the image. Set the expected range. Then, the face direction detection microcomputer 31 performs processing for extracting an edge line from the image within the set range, and uses the template image of each face part stored in the face information storage unit 32 to perform the above edge processing. Pattern matching is performed on the image from which the line has been extracted, the position of each face part is estimated, and the process proceeds to the next step S65.

  In step S65, the face orientation detection microcomputer 31 determines whether each face part has been estimated. If the face orientation detection microcomputer 31 can estimate the position of each facial part (for example, the degree of coincidence with the mouth template image is equal to or greater than a predetermined value), the process proceeds to step S66. On the other hand, if the face orientation detection microcomputer 31 cannot estimate the position of each face part, the process returns to step S51 and repeats the process.

  In step S <b> 66, the face orientation detection microcomputer 31 obtains the face center line. As an example of processing, the face orientation detection microcomputer 31 obtains the face centerline in the face image based on the eye, nose, and mouth positions estimated in step S64. Then, the process proceeds to the next step S67.

  In step S67, it is determined whether or not the center line of the face has been obtained. If the face orientation detection microcomputer 31 can obtain the center line of the face (YES), the process proceeds to step S68, and if the face center line cannot be obtained (NO), the process proceeds to step S51. Return processing.

  In step S68, the face orientation detection microcomputer 31 calculates the driver's face orientation angle θR or θL, outputs the calculation result to the vehicle control system 4, and ends the processing of the flowchart shown in FIG.

  Returning to the description of FIG. 5, the face direction detection microcomputer 31 calculates threshold angles Θ′R and Θ′R in step S <b> 53 and outputs them to the vehicle control system 4. Hereinafter, an example of an operation for calculating the threshold angles Θ′R and Θ′L performed by the face orientation detection microcomputer 31 will be described with reference to FIGS. 7 and 8.

  First, a case where the vehicle is in a turning state, for example, a case where the vehicle is traveling on a right curve will be described with reference to FIG. FIG. 7 is a diagram showing a state when the vehicle is making a right turn as an example. As shown in FIG. 7, in general, an instantaneous vehicle movement direction (hereinafter referred to as a speed direction) during turning is relative to a turning circle drawn by the center of gravity of the vehicle when the vehicle is turning. The tangential direction of the turning circle where the center of gravity of the vehicle at the present time is located. And the speed direction of a vehicle can be represented by the vector which has a magnitude | size according to the vehicle speed of the said vehicle. That is, the speed direction indicates the speed and moving direction of the vehicle at the current time. The direction of the vehicle is the vehicle front direction along the center line of the vehicle. In the present embodiment, since the camera 1 is installed so as to capture images from the front with respect to the driver, the optical axis of the camera 1 and the orientation (center line) of the vehicle when viewed from above the vehicle. Therefore, the center line of the vehicle can be considered parallel to the optical axis of the camera 1 (hereinafter referred to as the camera optical axis) and the horizontal direction.

  The angle β is a so-called slip angle that is a side slip angle of the center of gravity of the vehicle. That is, the angle β is an angle formed by the direction of the vehicle and the speed direction. If the camera 1 is installed so that the optical axis of the camera 1 and the direction of the vehicle (center line) are parallel to the horizontal direction, the angle β is determined by the camera optical axis and the speed direction. It can be regarded as an angle formed in the horizontal plane.

  The angle γ is an angle formed by the planned arrival direction and the speed direction, and can be obtained by a product of time T and yaw rate ω. Here, as shown in FIG. 7, when the vehicle is making a right turn, the driver of the vehicle does not drive the vehicle in front of the vehicle in order to drive safely. Specifically, gaze at the vehicle front right side). In other words, generally, when turning the vehicle, a driver who wants to drive safely watches the direction in which the vehicle travels. In other words, the driver predicts to some extent a point that the vehicle will reach after a certain period of time, and pays attention to the point for safe driving. Therefore, if the elapsed time that is the point at which the driver is gazing is set in advance, when the driver turns the vehicle by performing an accelerator or brake operation, the product of the time T and the yaw rate ω can be easily used. A predicted gaze point can be calculated.

  The face orientation detection microcomputer 31 calculates the threshold angles Θ′R and Θ′L in consideration of the angle γ and the angles γ and β as described above, and outputs the calculation results to the vehicle control system. When calculating the threshold angle Θ ′, the face direction detection microcomputer 31 calculates only one of the threshold angles Θ′R and Θ′L, and outputs the calculation result to the vehicle control system 4. To do.

  Then, the vehicle control system 4 makes a driver's side-look determination using the threshold angle output from the face orientation detection microcomputer 31.

Hereinafter, an example of the operation for calculating the threshold angle Θ ′ performed by the face orientation detection microcomputer 31 will be described with reference to FIG. 8. FIG. 8 is a flowchart showing an example of the operation for calculating the threshold angle Θ ′ in step S53 of FIG.

  In step S <b> 81 of FIG. 8, the face orientation detection microcomputer 31 acquires the yaw rate ω of the vehicle detected by the vehicle information detection unit 2. Then, the process proceeds to the next step S82.

  In step S82, the face orientation detection microcomputer 31 uses the yaw rate ω acquired in step S81 to calculate the angle γ based on the time T, the yaw rate ω, and (T × ω), and proceeds to the next step S83. .

  In step S83, the face orientation detection microcomputer 31 acquires the vehicle speed V of the vehicle detected by the vehicle information detection unit 2, and proceeds to the next step S84.

  In step S84, the face orientation detection microcomputer 31 calculates the angle β. Specifically, it can be calculated by Equation (1) based on the yaw rate ω acquired in step S81, the vehicle speed V acquired in step S83, and the specification value of the vehicle.

  In Equation (1), M is the weight of the vehicle, L is the wheel base of the vehicle, Lf is the distance from the front wheel axis of the vehicle to the center of gravity of the vehicle, and Lr is the center of gravity of the vehicle from the rear wheel axis of the vehicle. The distance to the point, Kr is the cornering power of the vehicle rear wheel, V is the vehicle speed of the vehicle, and ω is the yaw rate of the vehicle. Note that the above values of M, L, Lf, and Lr are values determined according to vehicle specifications, and can be acquired from the vehicle specification value storage unit 33. The cornering power Kr is a value determined by the tire characteristics and the weight applied to the tire, and information such as the characteristics of the tires attached to the vehicle may be stored in the vehicle specification value storage unit 33 in advance. It can be calculated with reference to the information from both specification value storage units 33. Then, the face orientation detection microcomputer 31 proceeds to the next step S85.

  In step S85, the face orientation detection microcomputer 31 calculates an angle ΔΘ. The angle ΔΘ is an angle that can be obtained by ΔΘ = | γ | − | β | as shown in FIG. Then, the face orientation detection microcomputer 31 proceeds to the next step S86.

  In step S86, the face orientation detection microcomputer 31 calculates a threshold angle Θ 'based on the angle ΔΘ calculated in step S85. Here, an example of the process of calculating the threshold angle Θ ′ performed by the face orientation detection microcomputer 31 in step S86 will be described using specific numerical values.

  Here, as an example, it is assumed that the threshold angles ΘR and ΘL are set to ΘR = 20 ° and ΘL = 20 °, respectively. In this case, the face orientation detection microcomputer 31 changes the threshold value Θ to the turning direction side of the vehicle. The turning direction of the vehicle can be known by a yaw rate sensor provided in the vehicle. For example, assume that the angle ΔΘ calculated in step S85 is ΔΘ = 10 °. In the example shown in FIG. 7, the face orientation detection microcomputer 31 calculates Θ′R = 30 ° by adding ΘR = 20 ° to the right by 10 °. Further, the face orientation detection microcomputer 31 calculates Θ′L = 10 ° by subtracting ΘL = 20 ° by 10 ° to the right. That is, the face orientation detection microcomputer 31 changes the threshold angle Θ toward the predicted gaze point of the vehicle.

  The face orientation detection microcomputer 31 may calculate the threshold angles Θ′R and Θ ′ for both the threshold angles ΘR and ΘL, or only one of the threshold angles Θ′R or Θ′L. (For example, in the case of FIG. 7, the changed threshold value Θ′R may be calculated only for the threshold value ΘR). Then, the face orientation detection microcomputer 31 outputs at least one of the threshold angle Θ′R and the threshold angle Θ′L to the vehicle control system 4 and ends the processing of the flowchart shown in FIG. 8.

  In the above-described example, the angle β is calculated after calculating the angle γ, but in another example, the angle γ may be calculated after calculating the angle β. That is, in the flowchart shown in FIG. 8, after step S83 and step S84 are performed first, step S81 and step S82 may be performed.

  In the above-described example, the threshold Θ is changed using the angle γ and the angle β, so that the threshold Θ can be changed in consideration of the current behavior (slip angle) of the vehicle. However, when such an effect is not expected, the threshold Θ may be changed using only the angle γ. In general, the angle β is a relatively small value compared to the angle γ when the vehicle is running normally. Therefore, even if the value of the angle β is ignored and only the angle γ is used to change the threshold Θ, the object of the present application can be achieved.

  Furthermore, in the above-described example, the camera 1 is installed so that the vehicle is in front of the driver when the vehicle is traveling straight, so that when viewed in the vehicle horizontal direction, the optical axis of the camera 1 and the center of the vehicle The direction of the center line of the vehicle is regarded as the optical axis of the camera 1 because it is the same as the line, but there may be a case where the direction of the center line and the direction of the optical axis are different. In that case, the center line may be regarded as the optical axis of the camera 1 in consideration of the angle difference between the direction of the center line and the direction of the optical axis.

  Returning to the description of FIG. 5, in step S54, the image processing ECU 3 determines whether or not the ignition switch of the vehicle is turned off. Then, when the ignition switch is turned off, the image processing ECU 3 ends the process according to the flowchart. On the other hand, when the ignition switch is not turned off, the image processing ECU 3 returns to step S51 and repeats the process.

  As described above, the image processing ECU 3 calculates the driver's face orientation angle θ and outputs it to the vehicle control system 4. Further, the image processing ECU 3 changes the threshold angle Θ, which is a criterion for determining whether the driver's face is normal, based on the yaw rate and the vehicle speed of the vehicle, and sets the changed threshold Θ ′ to the vehicle Output to the control system 4.

  Next, referring to FIG. 9, whether the driver's face orientation is normal based on the processing performed by the vehicle control system 4 (specifically, based on the face orientation angle θ and the threshold value Θ ′ output from the image processing ECU 3). An example of the process for determining whether or not will be described. FIG. 9 is a flowchart showing an example of processing performed in the vehicle control system 4.

  In step S91 of FIG. 9, the vehicle control system 4 acquires the driver's face orientation angle (specifically θR or θL) from the face orientation detection microcomputer 31, and advances the processing to the next step S92.

  In step S92, the vehicle control system 4 acquires the threshold angle from the face direction detection microcomputer 31, and proceeds to the next step S93.

  In step S93, the vehicle control system 4 determines whether or not the face orientation angle and the threshold angle have been acquired. If the vehicle control system 4 determines that the face orientation angle and the threshold angle have been acquired (YES), the process proceeds to the next step S94. If it is determined that acquisition has failed (NO), the process proceeds to step S91. To return.

  In step S94, the vehicle control system 4 determines whether or not the driver's face orientation is normal. Specifically, the vehicle control system 4 determines whether or not the driver's face orientation is normal based on the face orientation angle obtained in step S91 and the threshold angle obtained in step S92. For example, when the face orientation angle θR = 25 ° and the threshold angle Θ′R = 20 ° are output from the face orientation detection microcomputer 31, the vehicle control system 4 indicates that the driver's face orientation is not normal. Judged to be in a state of looking aside. Then, the process proceeds to step S95. On the other hand, if it is determined that the driver's face orientation is normal and the driver is not looking aside, the process proceeds to step S96.

  In step S95, the vehicle control system 4 controls the equipment on the vehicle and alerts the driver. At this time, the collision control system 4 may perform pre-crash control as necessary.

  In step S96, the vehicle control system 4 determines whether or not the ignition switch of the vehicle is turned off. And the vehicle control system 4 complete | finishes the process by the said flowchart, when an ignition switch is turned off. On the other hand, if the ignition switch is not turned off, the vehicle control system 4 returns to step S91 and repeats the process.

  As described above, according to the first embodiment, the threshold angle that serves as the determination criterion of the driver's side-by-side state is calculated according to the driving state of the vehicle, and the side-by-side determination is performed according to the calculation result. You can judge aside. Furthermore, it is possible to more accurately carry out so-called preventive safety, which prevents accidents caused by a driver's looking aside or looking away.

(Second Embodiment)
Next, an image processing apparatus according to the second embodiment of the present invention will be described. In the aside look determination system including the image processing apparatus according to the first embodiment described above, the angle β is calculated based on the equation (1) using the vehicle speed V of the vehicle and the yaw rate ω of the vehicle. In the image processing apparatus according to the second embodiment, the angle β is calculated from the lateral acceleration α of the vehicle and the vehicle speed V of the vehicle. In the second embodiment, the vehicle may be provided with a vehicle speed sensor and a lateral acceleration sensor. Further, the operation is similar to the operation and configuration of the armpit determination system including the image processing procedure according to the first embodiment except that the method for obtaining the angle γ and the angle β (that is, the processing in step S53 in FIG. 5) is different. A description of the same operation and configuration will be omitted. Therefore, in the following description, only the process for calculating the threshold value Θ ′ performed by the face orientation detection microcomputer 31 of the image processing apparatus ECU 3 will be described.

  FIG. 10 is a flowchart illustrating an example of processing for calculating the threshold angle Θ ′ performed by the face orientation detection microcomputer 31 of the image processing apparatus according to the second embodiment.

  In step S101 of FIG. 10, the face orientation detection microcomputer 31 acquires the vehicle lateral acceleration α and the vehicle speed V from the vehicle information detection unit 2, and advances the processing to the next step S102.

  In step S102, the face orientation detection microcomputer 31 calculates the angle β and proceeds to the next step S103. Specifically, the angle β can be calculated by the equation (2) using the lateral acceleration α of the vehicle, the vehicle speed V, and the specification value of the vehicle acquired in step S101.

  In Equation (2), M is the weight of the vehicle, L is the wheel base of the vehicle, Lf is the distance from the front wheel axis of the vehicle to the center of gravity of the vehicle, and Lr is the center of gravity of the vehicle from the rear wheel axis of the vehicle. The distance to the point, Kr is the cornering power of the vehicle rear wheel, V is the vehicle speed of the vehicle, and α is the lateral acceleration of the vehicle. Note that the above values of the weight M, the wheel base L, the distance Lf, and the distance Lr are values determined by the specifications of the vehicle and can be acquired from the vehicle specification value storage unit 33. The cornering power Kr is a value determined by the tire characteristics and the weight applied to the tire, and information such as the characteristics of the tires attached to the vehicle may be stored in the vehicle specification value storage unit 33 in advance. It can be calculated with reference to the information from both specification value storage units 33. Then, the face orientation detection microcomputer 31 proceeds to the next step S103.

  In step S <b> 103, the face orientation detection microcomputer 31 derives the yaw rate ω by modifying the equation (1) described above to obtain the equation of yaw rate ω (specifically, ω = β · V · (omitted)). . Specifically, the face orientation detection microcomputer 31 calculates the angle β in step S102. Accordingly, since the weight M, wheel base L, distance Lf, distance Lr, cornering power Kr, vehicle speed V, and angle β are already known in the equation of yaw rate ω obtained by transforming the above formula (1), the yaw rate ω is derived. be able to. Then, the face orientation detection microcomputer 31 proceeds to the next step S104.

  In step S104, the face orientation detection microcomputer 31 calculates an angle γ. The angle γ can be obtained by the product of T and ω. That is, as described in the first embodiment, generally, when the vehicle is turning, a driver who wants to drive safely watches the direction in which the vehicle travels. That is, the driver predicts to some extent a point that the vehicle will reach after a certain period of time, and pays attention to the point for safe driving. Therefore, if the elapsed time that is the point at which the driver is gazing is set in advance, when the driver turns the vehicle by performing an accelerator or brake operation, the product of the time T and the yaw rate ω can be easily used. A predicted gaze point can be calculated. Then, the face orientation detection microcomputer 31 proceeds to the next step S105.

  In step S105, the face orientation detection microcomputer 31 calculates the angle ΔΘ, and proceeds to the next step S106. Note that the method of calculating the angle ΔΘ performed by the face orientation detection microcomputer 31 in step S105 is the same as that in step S85 of FIG. 8 in the first embodiment, and thus detailed description thereof is omitted.

  In step S <b> 106, the face orientation detection microcomputer 31 calculates the threshold angle Θ ′ and ends the process of the flowchart shown in FIG. 10. Note that the method for calculating the threshold Θ ′ performed by the face orientation detection microcomputer 31 in step S106 is the same as that in step S86 of FIG. 8 in the first embodiment, and thus detailed description thereof is omitted.

(Third embodiment)
Next, a third embodiment will be described. In the aside look determination system including the image processing apparatus according to the second embodiment described above, the angle β is calculated based on Equation (2) using the vehicle speed V of the vehicle and the lateral acceleration α of the vehicle. In the third embodiment, a mode in which the angle β is calculated from the steering angle δ and the vehicle speed V will be described. In the third embodiment, the vehicle may be provided with a speed sensor and a steering angle sensor. Also, the operation of the side-by-side determination system including the image processing procedure according to the first embodiment is different except that the method for obtaining the angle γ and the angle β is different, that is, the processing in step S53 of the flowchart shown in FIG. 5 is different. Since it is the same as that of a structure, description is abbreviate | omitted. Therefore, only the process of calculating the threshold angle Θ ′ performed by the face orientation detection microcomputer 31 of the image processing ECU 3 will be described.

  FIG. 11 is a flowchart showing an example of processing for calculating the threshold angle Θ ′ performed by the face orientation detection microcomputer 31 of the image processing apparatus according to the third embodiment.

  In step S111 of FIG. 11, the face direction detection microcomputer 31 acquires the steering angle δ and the vehicle speed V of the vehicle from the vehicle information detection unit 2, and proceeds to the next step S112.

In step S112, the face orientation detection microcomputer 31 calculates the angle β. Specifically, the angle β is calculated by the mathematical formula (3) using the steering angle δ, the vehicle speed V, and the specification values of the vehicle acquired in step S111.

  In Equation (3), M is the weight of the vehicle, L is the wheel base of the vehicle, Lf is the distance from the front wheel axis of the vehicle to the center of gravity of the vehicle, and Lr is the center of gravity of the vehicle from the rear wheel axis of the vehicle. The distance to the point, Kf is the cornering power of the vehicle front wheel, Kr is the cornering power of the vehicle rear wheel, V is the vehicle speed of the vehicle, and δ is the steering angle of the vehicle. Note that the above values of the weight M, the wheel base L, the distance Lf, and the distance Lr are values determined by the specifications of the vehicle and can be acquired from the vehicle specification value storage unit 33. The cornering power Kf (Kr) is a value determined by the tire characteristics and the weight applied to the tire, and the vehicle specification value storage unit 33 stores information such as the characteristics of the tires mounted on the vehicle in advance. In this case, the information can be calculated by referring to the information from both specification value storage units 33. Then, the face orientation detection microcomputer 31 proceeds to the next step S113.

  In step S113, the face orientation detection microcomputer 31 derives the yaw rate ω by modifying the above-described equation (1) to obtain the equation of yaw rate ω (specifically, ω = β · V · (omitted)). . Specifically, the face orientation detection microcomputer 31 calculates the angle β in step S112. Accordingly, since the weight M, wheel base L, distance Lf, distance Lr, cornering power Kr, vehicle speed V, and angle β are already known in the equation of yaw rate ω obtained by transforming the above formula (1), the yaw rate ω is derived. be able to. Then, the face orientation detection microcomputer 31 proceeds to the next step S114.

  In step S114, the face orientation detection microcomputer 31 calculates the angle γ. The angle γ can be obtained by the product of T and ω. That is, as described in the first embodiment, generally, when the vehicle is turning, a driver who wants to drive safely watches the direction in which the vehicle travels. In other words, the driver predicts to some extent a point that the vehicle will reach after a certain period of time, and pays attention to the point for safe driving. Therefore, if the elapsed time is set in advance as a point where the driver is expected to watch, when the driver turns the vehicle by performing an accelerator or brake operation, the product of time T and yaw rate ω is easy. A gaze point can be calculated. Then, the face orientation detection microcomputer 31 proceeds to the next step S115.

  In step S115, the face orientation detection microcomputer 31 calculates the angle ΔΘ, and proceeds to the next step S116. Note that the method of calculating the angle ΔΘ performed by the face orientation detection microcomputer 31 in step S115 is the same as that in step S85 of FIG. 8 in the first embodiment, and thus detailed description thereof is omitted.

  In step S116, the face orientation detection microcomputer 31 calculates the threshold value Θ 'and ends the process of the flowchart shown in FIG. Note that the method of calculating the threshold angle Θ ′ performed by the face orientation detection microcomputer 31 in step S116 is the same as that in step S86 of FIG. 8 in the first embodiment, and thus detailed description thereof is omitted.

  In the first embodiment, the second embodiment, and the third embodiment described above, the case where the vehicle is turning right has been described as an example. Omitted.

  Further, in each of the first, second, and third embodiments described above, the image processing ECU 3 calculates the driver's face orientation angle and the threshold angle, and the vehicle control system 4 is in an aside state based on the calculation results. Judged. In another example, the image processing ECU 3 may calculate the driver's face orientation angle and the threshold angle, and further determine the look-ahead state. In this case, the image processing ECU 3 may determine whether or not the user is in the side-by-side state and transmit the determination result to the vehicle control system 4.

  Furthermore, the face orientation detection microcomputer 31 of each embodiment may have two threshold angles on the left and right sides with respect to the imaging direction of the camera 1. In this case, the face direction detection microcomputer 31 may move the two threshold angles in the direction in which the predicted gaze direction changes, or only one of the threshold angles in the direction in which the predicted gaze direction changes. You can move it to

  Further, the face orientation detection microcomputer 31 of each embodiment may provide one threshold angle only in the left direction with respect to the imaging direction of the camera 1. In this case, the face orientation detection microcomputer 31 moves one threshold angle provided in the left direction in a direction in which the predicted gazing point direction changes. Further, the face direction detection microcomputer 31 of each embodiment may provide one threshold angle only in the right direction with respect to the imaging direction of the camera 1. In this case, the face orientation detection microcomputer 31 moves one threshold angle provided in the right direction in a direction in which the predicted gazing point direction changes.

  The aspect described in the above embodiment is merely a specific example, and does not limit the technical scope of the present invention. Therefore, it is possible to employ any configuration within a range where the effects of the present application are achieved.

  The image processing device and the image processing method according to the present invention can make a more accurate look-aside determination by setting a threshold value that is a reference for a look-aside determination of the driver according to the driving state of the vehicle. The present invention is useful as an image processing apparatus, an image processing method, and the like that calculate the driver's face orientation from an image of a face and determine the driver's look-aside state.

1 is a block diagram illustrating an example of a configuration of an aside look determination system including an image processing apparatus according to a first embodiment. The figure which shows an example of the installation position of the camera 1 Image taken by camera 1 and top view of driver's head The figure which shows an example of the state of the vehicle which is drive | working The flowchart which showed an example of the process performed in image treatment ECU3 which concerns on 1st this embodiment. The flowchart which showed an example of the operation | movement which calculates the driver's face direction angle in step S52 of FIG. A diagram showing the situation when the vehicle is turning right The flowchart which showed an example of the operation | movement which calculates threshold value (theta) 'of FIG.5 S53. The flowchart which showed an example of the process performed in the vehicle control system 4 The flowchart which showed an example of the process which calculates angle | corner (theta) 'of the threshold value which the face direction detection microcomputer 31 of the image processing apparatus which concerns on 2nd Embodiment performs. The flowchart which showed an example of the process which calculates angle | corner (theta) 'of the threshold value which the face direction detection microcomputer 31 of the image processing apparatus which concerns on 3rd Embodiment performs.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera 2 Vehicle information detection part 3 Image processing ECU
31 Face orientation detection microcomputer 32 Face information storage unit 33 Vehicle specification value storage unit 4 Vehicle control system

Claims (12)

A face direction angle calculating unit that calculates an angle of the driver's face relative to the imaging direction of the imaging unit using an image captured by the imaging unit that captures the face of the driver seated in the driver's seat of the vehicle; ,
Vehicle information acquisition means for acquiring vehicle information of the vehicle;
Threshold angle setting means for setting a threshold angle on at least one of left and right with respect to the imaging direction;
At least one of the threshold angles set by the threshold angle setting unit is calculated by calculating a predicted gaze point direction predicted to be watched by the driver when the vehicle is turning using the vehicle information acquired by the vehicle information acquisition unit. Threshold angle moving means for moving the predicted gazing point direction in the direction in which
An image processing apparatus comprising: determination means for determining that the driver is looking aside when the driver's face orientation angle deviates from the threshold angle. The threshold angle setting means sets the threshold angle on both the left and right sides with respect to the imaging direction,
The image processing apparatus according to claim 1, wherein the threshold angle moving unit moves the threshold angle in a direction in which the predicted gazing point direction changes.   The threshold angle moving means predicts the arrival point of the vehicle after elapse of a predetermined time using the vehicle information, and calculates the direction from the vehicle to the arrival point as the predicted gaze direction. Item 3. The image processing apparatus according to Item 1 or 2.   The image processing apparatus according to claim 3, wherein the threshold angle moving unit moves the threshold angle by a first angle formed by a speed direction indicating the moving direction of the vehicle at the current time and the predicted gazing point direction. Vehicle specification storage means for storing vehicle specifications of the vehicle,
The threshold angle moving means calculates a slip angle indicating a lateral slip angle of the center of gravity of the vehicle as a second angle using the vehicle information and the vehicle specifications, and calculates the first angle and the second angle. The image processing apparatus according to claim 4, wherein the threshold angle is moved by an angle difference.   The threshold angle moving means calculates an angle formed by the speed direction and the imaging direction as a third angle, and moves the threshold angle by an angle difference between the first angle and the third angle. An image processing apparatus according to 1. The vehicle information acquired by the vehicle information acquisition means includes the yaw rate of the vehicle,
The image processing apparatus according to claim 3, wherein the threshold angle moving unit predicts the arrival point using the yaw rate. Vehicle specification storage means for storing vehicle specifications of the vehicle,
The vehicle information acquired by the vehicle information acquisition means includes the yaw rate of the vehicle and the vehicle speed of the vehicle,
The image processing apparatus according to claim 3, wherein the threshold angle moving unit calculates a direction to the arrival point by substituting the yaw rate, the vehicle speed, and the vehicle specifications into a predetermined mathematical formula. Vehicle specification storage means for storing vehicle specifications of the vehicle,
The vehicle information acquired by the vehicle information acquisition means includes a lateral acceleration of the vehicle and a vehicle speed of the vehicle,
The image processing apparatus according to claim 3, wherein the threshold angle moving unit calculates a direction to the arrival point by substituting the lateral acceleration, the vehicle speed, and the vehicle specifications into a predetermined mathematical expression. Vehicle specification storage means for storing vehicle specifications of the vehicle,
The vehicle information acquired by the vehicle information acquisition means includes a steering angle of the vehicle and a vehicle speed of the vehicle,
The image processing apparatus according to claim 3, wherein the threshold angle moving unit calculates a direction to the destination by substituting the steering angle, the vehicle speed, and the vehicle specifications into a predetermined mathematical formula.   2. The control device according to claim 1, further comprising a control unit configured to control the vehicle and notify the driver when the driver's face direction calculated by the face direction angle calculation unit deviates from the threshold angle. 2. The image processing apparatus according to 2. A face angle calculation step for calculating a face angle of the driver based on the imaging direction of the imaging unit using an image captured by the imaging unit that captures the face of the driver seated in the driver's seat of the vehicle; ,
Vehicle information acquisition step of acquiring vehicle information of the vehicle;
A threshold angle setting step for setting a threshold angle on at least one of left and right with respect to the imaging direction;
At least one of the threshold angles set in the threshold angle setting step is calculated by using the vehicle information acquired in the vehicle information acquisition step to calculate a predicted gaze point direction predicted to be watched by the driver when the vehicle turns. Threshold angle movement step for moving the direction of the point of interest to change,
An image processing method comprising: a determination step of determining that the driver is looking aside when the driver's face orientation angle deviates from the threshold angle.

Images