JP2017211200A - Camera calibrating device, and camera calibrating method - Google Patents

Abstract

An object of the present invention is to easily perform a camera calibration process. In step 2, an image processing unit performs a process of generating a target image by performing distortion correction and viewpoint conversion on a captured image acquired from a camera installed in a vehicle. In step 3, the image processing unit performs a process of determining whether a traffic light, a road sign, and the like exist from the generated target image. If an affirmative determination is made in step 3, the image processing unit extracts vertical edges such as a traffic light and a road sign in step 4, and calculates the inclination angle of the extracted vertical edges. The image processing unit calculates a deviation between a current mounting angle of the camera installed in the vehicle and an initial mounting angle of the camera installed in the vehicle based on the angle. [Selection diagram] FIG.

Description

  The present disclosure relates to a technique for calibrating a camera.

  When a camera installed in a vehicle is being used, the camera mounting angle may shift. A technique for calibrating such a camera shift is known. For example, a technique for calibrating a camera by installing a calibration board on a road surface and imaging the calibration board with a camera is known. However, when a camera is calibrated using this technique, it is necessary to calibrate the camera by placing a calibration board on a road surface or the like, and there is a problem that the camera calibration operation cannot be automated. Therefore, in order to solve this problem, a technique for calibrating the camera by imaging the lane markings and road markings drawn on the road surface with the camera is disclosed (see Patent Document 1).

JP 2003-329411 A

  In the example described in Patent Document 1, in a captured image acquired from a camera installed in a vehicle, a vanishing point that is an intersection of extending two white lines that divide a lane is calculated, and the vanishing point, The camera is calibrated together with information from an angle sensor that detects the traveling angle of the vehicle. That is, white line information and information from the angle sensor are required. For this reason, the camera calibration processing performed based on these pieces of information is complicated.

  One aspect of the present disclosure has been made in view of these problems, and an object thereof is to easily perform a camera calibration process.

  One aspect of the present disclosure is a camera calibration device (6), which includes an extraction unit (6, S1 to S4, S21 to S24, S41 to S46), and an inclination calculation unit (6, S4, S24, S47 to S49). And a deviation calculation unit (6, S9, S10, S29, S30). An extraction part extracts the vertical edge of the object which has the linear part extended in a perpendicular direction in the captured image acquired from the camera (2, 3) installed in the vehicle. The inclination calculation unit calculates the inclination of the vertical edge extracted by the extraction unit. The deviation calculation unit calculates the deviation between the current installation angle of the camera installed in the vehicle and the initial installation angle of the camera installed in the vehicle based on the inclination of the vertical edge calculated by the inclination calculation unit. A deviation calculation process is performed.

  According to such a camera calibration device, the camera can be calibrated based on the inclination of the vertical edge. Therefore, the camera calibration device of the present disclosure does not require information from the angle sensor, unlike, for example, a camera calibration device that requires information on the lane boundary line and information obtained from the angle sensor. Therefore, if the camera calibration apparatus according to the present disclosure is used, the camera calibration process can be easily performed.

  Another aspect of the present disclosure is a camera calibration method, which is a vertical edge of an object having a linear portion extending in a vertical direction in a captured image obtained from a camera (2, 3) installed in a vehicle. Are extracted (6, S1-S4, S21-S24, S41-S46), the inclination of the extracted vertical edge is calculated (6, S4, S24, S47-S49), and based on the calculated inclination of the vertical edge Then, the difference between the current mounting angle of the camera installed in the vehicle and the initial mounting angle of the camera installed in the vehicle is calculated (6, S9, S10, S29, S30).

  According to such a camera calibration method, since it has the same characteristics as the camera calibration device described above, the same function and effect as the camera calibration device can be obtained, and the camera calibration process can be easily performed. it can.

  Note that the reference numerals in parentheses described in this column and in the claims indicate the correspondence with the specific means described in the embodiment described later as one aspect, and the technical scope of the present disclosure It is not limited.

It is a block diagram showing the structure of a camera calibration system. It is a flowchart (the 1) showing a 1st camera calibration process. It is a flowchart (the 2) showing a 1st camera calibration process. (A) is a figure which shows the characteristic of the vertical edge when the attachment position is not shifted | deviated. (B) And (C) is a figure which shows the characteristic of the vertical edge in case the pitch angle has shifted | deviated. (D) And (E) is a figure which shows the characteristic of the vertical edge in case the roll angle has shifted | deviated. (F), (G), (H), and (I) are diagrams showing characteristics of vertical edges when the pitch angle and the roll angle are shifted. It is a flowchart (the 1) showing a 2nd camera calibration process. It is a flowchart (the 2) showing a 2nd camera calibration process. It is a flowchart showing a tracking process. (A) is obtained by correcting the distortion of the captured image. (B) is a target image.

Hereinafter, embodiments for carrying out the present disclosure will be described with reference to the drawings.
[1. First Embodiment]
[1-1. Configuration of camera calibration system 1]
The configuration of the camera calibration system 1 will be described with reference to FIG. The camera calibration system 1 is an in-vehicle system mounted on a vehicle. Hereinafter, a vehicle equipped with the camera calibration system 1 is referred to as a host vehicle. The camera calibration system 1 includes a front camera 2, a rear camera 3, an ECU 4, and a display device 5. The ECU 4 is an abbreviation for “Electronic Control Unit”, that is, an abbreviation for an electronic control unit.

  The front camera 2 is installed at the front part of the host vehicle so that the optical axis of the front camera 2 is tilted downward and captures an image of the lower front of the host vehicle. The rear camera 3 is installed at the rear part of the host vehicle so that the optical axis of the rear camera 3 is tilted downward, and captures an image of the lower rear of the host vehicle. The front camera 2 and the rear camera 3 perform imaging at a predetermined time interval (for example, 1/15 second interval). Further, the front camera 2 and the rear camera 3 include wide-angle lenses. For this reason, the captured images captured by the front camera 2 and the rear camera 3 have distortions such that the degree of reduction of the subject becomes larger in the peripheral portion than in the central portion of the captured image.

  The ECU 4 includes an image processing unit 6, an image signal processing unit 7, an in-vehicle signal processing unit 8, a memory 9, and a power supply circuit 10. The image processing unit 6 is configured around a known microcomputer having a CPU, a RAM, a ROM, and the like (not shown). Various functions of the image processing unit 6 are realized by the CPU executing a program stored in a non-transitional physical recording medium. In this example, the memory 9 corresponds to a non-transitional tangible recording medium that stores a program. Further, by executing this program, a method corresponding to the program is executed. Note that the number of microcomputers constituting the image processing unit 6 may be one or plural.

  The image signal processing unit 7 is an interface for outputting signals input from the front camera 2 and the rear camera 3 to the image processing unit 6. The in-vehicle signal processing unit 8 is an interface for outputting a vehicle speed signal input from a vehicle speed sensor that detects the vehicle speed of the host vehicle to the image processing unit 6.

  The memory 9 is a recording medium that records programs for realizing various functions. In the memory 9, initial mounting angles of the front camera 2 and the rear camera 3 installed in the host vehicle are recorded as design values.

  The power supply circuit 10 converts a voltage input from a battery (not shown) of the host vehicle into an appropriate voltage and supplies it to each part of the ECU 4. The display device 5 is a known device that displays an image, and is installed at a position where a driver or the like in the vehicle can visually recognize the displayed image. The display device 5 is, for example, a liquid crystal display.

[1-2. First camera calibration process of camera calibration system 1]
Next, the first camera calibration process (in other words, the camera calibration method) executed by the image processing unit 6 will be described with reference to the flowcharts of FIGS. The first camera calibration process is started when the ignition switch of the host vehicle is turned on, and is repeatedly executed every predetermined time (for example, 1/15 seconds). In addition, although the 1st camera calibration process demonstrated below shows the example processed based on the captured image acquired from the front camera 2, it may be processed based on the captured image acquired from the rear camera 3. Similar processing is performed.

  In step 1, the image processing unit 6 performs processing for acquiring a captured image from the front camera 2. In the subsequent step 2, the image processing unit 6 performs a process of generating the target image 52 from the acquired captured image. An example of the target image 52 is shown in FIG. In step 2, specifically, the image processing unit 6 performs known distortion correction on the captured image. An example of the captured image 51 subjected to distortion correction is shown in FIG. Then, the image processing unit 6 performs distortion conversion of the captured image 51 subjected to distortion correction, which converts the distortion-corrected captured image 51 into an image of a viewpoint having a pitch angle (in other words, a dip angle) of the front camera 2 of 0 degrees. To generate the target image 52. The viewpoint conversion process is performed based on an initial design value of the front camera 2 installed in the host vehicle.

  In subsequent step 3, the image processing unit 6 includes a left area (hereinafter referred to as a left area) and a right area (hereinafter referred to as a left area) obtained by dividing the area above the horizontal line set in the generated target image 52 approximately equally. The right region is referred to as a right region.) A process for determining whether a traffic light, a road sign, or the like exists is performed. Specifically, the image processing unit 6 performs pattern recognition for analyzing the target image 52 using a template such as a traffic light and a road sign recorded in advance in the memory 9, and whether or not a traffic light and a road sign exist. The process which determines is performed. The horizontal line is set in the target image 52 based on the initial design value of the front camera 2 installed in the host vehicle. If a negative determination is made in step 3, the process returns to step 1 again. On the other hand, if an affirmative determination is made in step 3, the process proceeds to step 4.

  In step 4, the image processing unit 6 extracts a vertical edge such as a traffic signal and a road sign that have been pattern-recognized, calculates an inclination angle of the vertical edge, and further accumulates data of the angle. The process in step 4 is performed for each of the left region and the right region. Here, the vertical edge means the linear portion of a stationary object having a linear portion extending in the vertical direction. Examples of stationary objects include traffic lights, road signs, and buildings. The vertical direction is not limited to the vertical direction in a strict sense, and may not be strictly in the vertical direction as long as the desired effect is achieved. Specifically, in step 4, the image processing unit 6 extracts, as vertical edges, a support column that supports a lamp of a traffic light, a support column that supports a display board on which an arrow indicating one-way traffic, or the like is displayed. This extraction is performed by a known edge detection process or the like. The image processing unit 6 calculates the coordinates of the start point and end point on the target image 52 with the two end points of the extracted vertical edge as the start point and end point, respectively. The image processing unit 6 calculates the angle of inclination of the vertical edge based on the coordinate information and accumulates the data.

  In step 5, the image processing unit 6 determines whether or not the number of target images 52 in the left area and the number of target images 52 in the right area including the vertical edges extracted in step 4 is greater than a predetermined number. Perform the process. If a negative determination is made here, the process returns to step 1. On the other hand, if an affirmative determination is made in step 5, the process proceeds to step 6. Specifically, for example, if the predetermined number is set to 100, the number of target images in the left area including the vertical edge exceeds 100, and the number of target images in the right area including the vertical edge does not reach 100. The processing unit 6 once stops the processing of the left area. For the right region, the process returns to step 1, and the image processing unit 6 performs the processing from step 1 to step 5 until the number of target images 52 in the right region including the vertical edge exceeds 100. If the number of target images 52 in the right area including the vertical edge exceeds 100, the left area processing and right area processing proceeds to step 6.

  In subsequent step 6, the image processing unit 6 performs a process of calculating an average value of the angles accumulated in step 4. This process is performed for each angle of inclination of the vertical edge extracted from the left and right areas.

In the subsequent step 7, the image processing unit 6 subtracts the average value of the vertical edge angle of the left region from the right angle (90 degrees) and the average value of the vertical edge angle of the right region from the right angle (90 degrees). It was calculated and a value, performs the processing for judging whether or not it is greater than at least one of the threshold TH ₁ for the absolute value of the calculated value. For example, when the current mounting angle of the front camera 2 is not deviated from the initial mounting angle of the front camera 2, the inclination angle of the vertical edge after the viewpoint conversion processing is ideally 90 degrees, and the calculated value The absolute value of is 0. As the deviation between the current mounting angle of the front camera 2 and the initial mounting angle of the front camera 2 increases, the absolute value increases.

The threshold TH ₁ is calibrated or needed whether the criterion value of the front camera 2. If a negative determination is made in step 7, i.e., when the absolute value of the threshold value TH ₁ the following values for the deviation between the current mounting angle and the initial mounting angle of the front camera 2 is small, after the process of step 8, again Steps 1 to 7 are performed. In step 8, the image processing unit 6 performs processing for discarding data in the order of old data. If an affirmative determination is made in step 7, i.e., the absolute value of the case of a value greater than the threshold value TH _1, since the deviation between the current mounting angle and the initial mounting angle of the front camera 2 is large, the calibration of the front camera 2 It progresses to the process after step 9 required.

In step 9, the image processing unit 6 calculates the absolute value of the difference between the average value of the inclination angle of the vertical edge extracted from the left region and the average value of the inclination angle of the vertical edge extracted from the right region. Then, a process of determining whether or not the absolute value is larger than the threshold value TH ₂ is performed. Threshold TH ₂ is a criterion value for whether or not it is necessary to calibrate the roll and pitch of both the front camera 2. If an affirmative determination is made in step 9, the process proceeds to step 13. In step 13, the image processing unit 6 performs a process of displaying an alarm on the display device that calibration of both the roll and pitch front cameras 2 is necessary. After step 13, the first camera calibration process ends.

  On the other hand, if a negative determination is made in step 9, the process proceeds to step 10. In step 10, the image processing unit 6 performs a process of determining whether or not the signs of the values of the left region and the right region calculated in step 7 are the same. Here, the sign means the sign of the value calculated in step 7, and means minus (−) and plus (+). That is, it is positive when the angle of inclination of the vertical edge is not less than 0 degrees and less than 90 degrees, and is negative when it is greater than 90 degrees and less than 180 degrees.

  If a negative determination is made in step 10, the process proceeds to step 11. In step 11, the image processing unit 6 performs a process of displaying an alarm on the display device that the front camera 2 needs to be calibrated only for the pitch. After step 11, the first camera calibration process ends.

  If an affirmative determination is made in step 10, the process proceeds to step 12. In step 12, the image processing unit 6 performs a process of displaying an alarm on the display device that the front camera 2 only with a roll needs to be calibrated. After step 12, the first camera calibration process ends.

Next, with respect to specific examples of the calibration process of the front camera 2, (A), (B), (C), (D), (E), (F), (G), (H), FIG. And (I).
The average vertical edges extracted from the left side area and the right side area are shown as (A), (B), (C), (D), (E), (F), (G), (H), and FIG. This is indicated by two lines (I). The left line of the two lines indicates an average vertical edge extracted from the left area, and the right line of the two lines indicates an average vertical edge extracted from the right area.

  As shown in FIG. 4A, if the average vertical edges extracted from the left and right regions are not so inclined, that is, if a negative determination is made in step 7, the front camera 2 Do not display an alarm on the display device that calibration is required.

  On the other hand, as shown in FIGS. 4B to 4I, when at least one of the average vertical edge inclinations extracted from the left area and the right area is inclined larger than a certain value, If the determination in step 7 is affirmative, an alarm that the front camera 2 needs to be calibrated is displayed on the display device.

As shown in FIGS. 4B and 4C, the difference between the average value of the inclination angle of the vertical edge extracted from the left region and the average value of the inclination angle of the vertical edge extracted from the right region is calculated. An alarm that the front camera 2 needs to be calibrated only for the pitch when the absolute value is equal to or less than the threshold TH ₂ and the sign of the average value of the inclination angles of the vertical edges extracted from the left and right regions is different. Is displayed on the display device.

As shown in FIGS. 4D and 4E, the difference between the average value of the inclination angle of the vertical edge extracted from the left area and the average value of the inclination angle of the vertical edge extracted from the right area. absolute value is the threshold value TH ₂ the following values, alarm that the sign of the average value of the inclination angle of the vertical edges extracted from the left area and right area are the same case, it is necessary to calibrate the front camera 2 rolls only Is displayed on the display device.

As shown in FIGS. 4F to 4I, the absolute value of the difference between the average value of the inclination of the vertical edge extracted from the left area and the average value of the inclination of the vertical edge extracted from the right area is a threshold value. If it is greater than TH _2, it performs processing for displaying a warning that rolls and calibration of the front camera 2 both pitch is required for the display device.

[1-3. effect]
According to the first embodiment described in detail above, the following effects can be obtained.
(1a) The image processing unit 6 of the first embodiment calibrates the front camera 2 based on the angle of inclination of the vertical edge. Therefore, the image processing unit 6 does not require information from the angle sensor, unlike an image processing unit that requires information on the lane boundary line and information obtained from the angle sensor, for example. Further, the image processing unit 6 does not require map information, for example, unlike an image processing unit that requires map information to specify a road to be measured. Therefore, if the image processing unit 6 is used, the calibration process of the front camera 2 can be easily performed.

  (1b) The image processing unit 6 according to the first embodiment extracts a vertical edge from the left region and the right region, and performs a process of comparing the inclinations of the extracted vertical edges. Therefore, it can be seen whether the required calibration of the front camera 2 is a roll or a pitch.

  (1c) In the first embodiment, the processing is performed on the left region and the right region of the target image 52, respectively. The left area and the right area are areas where there is a high probability that traffic lights, road signs, and the like exist. Therefore, for example, when the processing is performed on the left region and the right region as compared with the case where the processing is performed on the entire target image 52, the processing time is reduced, and erroneous recognition can be further suppressed.

  (1d) The image processing unit 6 of the first embodiment performs viewpoint conversion processing, and performs processing for extracting vertical edges from the target image 52 after the viewpoint conversion processing. Therefore, even when the front camera 2 is installed in the host vehicle so that the optical axis of the front camera 2 is obliquely downward, the front camera 2 is installed in the same manner as when the front camera 2 is installed with a pitch angle of 0 degrees. Can be calibrated.

  (1e) In the first embodiment, the horizontal line is set in the target image 52 based on the initial design value of the front camera 2 installed in the host vehicle. Therefore, for example, the image processing unit 6 does not need information from the angle sensor, unlike the image processing unit that requires information on the lane boundary line and information obtained from the angle sensor. The front camera 2 can be calibrated.

  (1f) The image processing unit 6 of the first embodiment accumulates the data of the inclination angle of the vertical edge, and performs the calibration process of the front camera 2 based on the average value of the accumulated data. Therefore, the image processing unit 6 can perform the calibration process of the front camera 2 based on highly accurate information.

(1 g) an image processing unit 6 of the first embodiment, when the absolute value of the threshold value TH ₁ the following values for the deviation between the current mounting angle and the initial mounting angle of the front camera 2 is small, the front camera 2 Do not perform processing that requires calibration. Therefore, the processing load can be reduced.

  (1h) The image processing unit 6 according to the first embodiment performs a process of displaying an alarm on the display device that the camera posture of the roll and pitch needs to be calibrated. Therefore, the image processing unit 6 informs the driver that the current mounting angle of the front camera 2 installed in the host vehicle is different from the initial mounting angle of the front camera 2 installed in the host vehicle. It can be grasped.

  In the first embodiment, the image processing unit 6 corresponds to a camera calibration device, and the average value corresponds to a statistical value. Further, the inclination of the vertical edge extracted from the left area corresponds to the first inclination, and the inclination of the vertical edge extracted from the right area corresponds to the second inclination. S1 to S4 correspond to processing as an extraction unit, S4 corresponds to processing as an inclination calculation unit, S9 and S10 correspond to processing as a deviation calculation unit, and S4 corresponds to processing as an accumulation unit. S6 corresponds to processing as a statistical unit, and S11 to S13 correspond to processing as a display processing unit.

[2. Second Embodiment]
[2-1. Difference from the first embodiment]
Since the basic configuration of the second embodiment is the same as that of the first embodiment, the description of the common configuration will be omitted, and the description will focus on the differences. Note that the same reference numerals as those in the first embodiment indicate the same configuration, and the preceding description is referred to.

[2-2. Second camera calibration process of camera calibration system 1]
Next, a second camera calibration process (in other words, a camera calibration method) executed by the image processing unit 6 of the second embodiment instead of the first camera calibration process (FIGS. 2 and 3) of the first embodiment. Will be described with reference to the flowcharts of FIGS. 5, 6, and 7. The second camera calibration process is started when the ignition switch of the host vehicle is turned on, and is repeatedly executed every predetermined time (for example, 1/15 second). The second camera calibration process described below is an example of processing based on a captured image acquired from the front camera 2, but may be processed based on a captured image acquired from the rear camera 3. Similar processing is performed.

In step 21, the image processing unit 6 executes the same process as in step 1 described above. In subsequent step 22, the image processing unit 6 executes the same process as in step 2 described above.
In subsequent step 23, the image processing unit 6 performs a process of determining whether or not a visual target exists for each of the left search area 53 and the right search area 54 of the target image 52. Specifically, in the second embodiment, for example, a vertical edge is extracted from a visually recognized object such as a traffic light or a road sign, and a stationary object including a support column that supports the visually recognized object. When the stationary object is a traffic light, the lamp portion of the traffic light corresponds to the visual recognition target. When the stationary object is a road sign, the display board portion on which an arrow indicating one-way is displayed corresponds to the visual recognition target. An example of the left search area 53 and the right search area 54 is shown in FIG. A left area and a right area excluding a predetermined area that equally divides the target image 52 left and right and a predetermined area at the top of the target image 52 from an upper area where the target image 52 is divided roughly equally up and down, These are set as a left search area 53 and a right search area 54, respectively. Specifically, in step 23, the image processing unit 6 performs pattern recognition for analyzing a captured image using a template for a visual target of a road sign recorded in advance in the memory 9, and determines whether or not a visual target exists. The process which determines is performed.

In step 24, the image processing unit 6 performs tracking processing on the visual recognition target. This process is performed for the left search area 53 and the right search area 54, respectively.
Specifically, when the visual recognition target is continuously recognized in the previous target image 52 and the latest target image 52, the past visual recognition target and the latest target image recognized from the previous target image 52 are displayed. The latest visual recognition target recognized from 52 is compared to determine whether or not they are the same visual recognition target. Here, the same visual target means the same object, and visual targets installed at different points are not included even if the outer shape is the same. Different visual recognition objects mean visual recognition objects other than the same visual recognition object. When it is determined that the latest visual target is the same visual target as the past visual target, the vertical edge of the support pillar is extracted for the latest visual target, and the coordinate information of the vertical edge is recorded. . This process is continued while the same visual target is recognized in the target image 52. That is, the same visual target is tracked. Meanwhile, the coordinate information of the visual recognition target is accumulated. Thereafter, data of coordinate information that maximizes the length of the line segment of the vertical edge is selected from the accumulated coordinate information, and processing for calculating the inclination angle of the line segment is performed. Note that when a plurality of visual targets exist in one target image 52, tracking is performed in units of visual targets. A specific flow of the tracking process will be described with reference to the flowchart of FIG.

  First, in step 41, the image processing unit 6 performs a process of determining whether or not the tracking flag is on. The tracking flag is an on / off flag indicating whether or not a visual target to be tracked exists in the target image 52. At the start of the tracking process, the tracking flag is set to off.

If a negative determination is made in step 41, the process proceeds to step 42. Immediately after the tracking process is started, the tracking flag is off, and the process proceeds to step 42.
In step 42, the image processing unit 6 performs processing for turning on the tracking flag. After step 42, the process proceeds to step 43.

  In step 43, the image processing unit 6 records the position information of the visual target existing in the target image 52 in the memory 9. After step 43, the process returns to step 41. In this tracking process, a series of processes until returning to step 41, that is, one cycle of processing, is performed on the same target image 52 as a processing target, and returning to step 41 allows the target image 52 to be processed to be processed. Next, the target image 52 is switched in order. That is, when the imaging timing by the front camera 2 is expressed as T1, T2, T3,..., The processing up to step 43 described above is performed on the target image 52 at the imaging timing T1, and processing returns to step 41. The target is switched to the target image 52 at the imaging timing T2. Thereafter, when the process returns to step 41 again, the processing target is switched to the target image 52 at the imaging timing T3.

  If an affirmative determination is made in step 41, the process proceeds to step 44. In step 44, the image processing unit 6 records the position information of the visual target existing in the target image 52 in the memory 9. After step 44, the process proceeds to step 45.

  In step 45, the image processing unit 6 performs a process of determining whether or not the past visual target one cycle before and the latest visual target are the same visual target. Whether or not they are the same visual target is compared with the position information of the past visual target recorded in step 43 or 44 in the previous cycle and the position information of the visual target recorded in step 44 of the current cycle. It is determined by doing. That is, since it can be estimated how the same visual target moves in the target image 52 based on the vehicle speed of the host vehicle, the latest visual target is located at a position estimated based on the past visual target. It can be determined whether it is the same visual recognition object by determining whether it exists. If an affirmative determination is made in step 45, the process proceeds to step 46.

  In step 46, the image processing unit 6 performs a process of extracting the latest supporting pillar to be visually recognized as a vertical edge and recording data of coordinate information of the extracted vertical edge. Specifically, the image processing unit 6 extracts a vertical edge from a portion in which it is estimated that the latest viewing target support pillar exists in the target image 52 by a known edge detection process or the like. The image processing unit 6 performs processing for calculating the coordinates of the start point and the end point on the captured image, and recording the coordinate information in the memory 9, using both end points of the extracted vertical edge as the start point and end point, respectively. After step 46, the process proceeds to step 48.

  In step 48, the image processing unit 6 performs a process of determining whether or not the tracking flag is on. If an affirmative determination is made in step 48, the process returns to step 41. If a negative determination is made in step 48, the operation proceeds to step 49 described later.

  On the other hand, if a negative determination is made in step 45, the process proceeds to step 47. In step 47, the image processing unit 6 performs processing for turning off the tracking flag. After step 47, the process proceeds to step 48, a negative determination is made at step 48, and the process proceeds to step 49.

  In step 49, the image processing unit 6 selects a vertical edge having the maximum length of the line segment based on the coordinate information of the vertical edge accumulated in step 46, and calculates the inclination angle of the vertical edge. I do. The calculated angle is accumulated and used for the processing after step 49. After step 49, the process proceeds to step 25.

  In step 25, the image processing unit 6 determines whether both the number of target images 52 in the left search area 53 and the number of target images 52 in the right search area 54 including the vertical edges extracted in step 24 are greater than a predetermined number. Processing to determine whether or not.

  In step 26, the image processing unit 6 performs a process of calculating an average value of the most frequent value segments of the angles accumulated in step 49. This process is performed for the left search area 53 and the right search area 54, respectively. Specifically, the image processing unit 6 classifies the accumulated angle distribution by a predetermined unit of, for example, 0.5 degrees, and mode value classification that is a maximum frequency classification in the classified frequency distribution. The process which calculates the average value of is performed.

  In step 27 to step 33, the image processing unit 6 executes the same processing as in the above-described step 7 to step 13 using the average value of the mode value section instead of the average value of the first embodiment.

[2-3. effect]
According to the second embodiment described in detail above, the following effects are obtained in addition to the effects of the first embodiment described above.

  (2a) The image processing unit 6 of the second embodiment selects coordinate information that maximizes the length of the line segment of the vertical edge, and uses the inclination angle of the line segment for subsequent processing. That is, reliable data is used for subsequent processing. Therefore, the accuracy of subsequent processing is increased.

In addition, the image processing unit 6 of the second embodiment performs tracking processing in units of visual recognition targets. Therefore, one angle of inclination of the vertical edge can be obtained from the same viewing target.
In the second embodiment, the average value of the mode value section corresponds to the statistical value. Further, the inclination of the vertical edge extracted from the left search area 53 corresponds to the first inclination, and the inclination of the vertical edge extracted from the right search area 54 corresponds to the second inclination. S21 to S24 and S41 to S46 correspond to processing as an extraction unit, S24 and S47 to S49 correspond to processing as an inclination calculation unit, S29 and S30 correspond to processing as a deviation calculation unit, and S24. , S41 to S46 correspond to processing as an accumulation unit, S26 corresponds to processing as a statistical value, and S31 to S33 correspond to processing as a display processing unit.

[3. Other Embodiments]
As mentioned above, although the form for implementing this indication was demonstrated, this indication is not limited to the above-mentioned embodiment, and can carry out various modifications.

  (3a) In the first embodiment described above, an example is shown in which the region above the horizontal line set in the target image 52 is divided approximately equally into the left and right as the processing region, but is not limited thereto. For example, a region obtained by dividing an upper region of an image whose viewpoint is converted by a predetermined dip angle substantially equally to the left and right may be used as a processing region. The predetermined dip angle may be, for example, about half of the dip angle of the design value of the on-vehicle camera. Moreover, the area | region divided into the right and left substantially equally from the whole object image irrespective of a horizontal line may be sufficient. In the first embodiment, as an example of the camera calibration process, the process is performed on each of the two areas, the left area and the right area. However, the present invention is not limited to this. The camera calibration process may be performed on the entire target image and a part of the target image. Examples of the target image include a region above the horizontal line without dividing the target image into left and right.

  Similarly, in the second embodiment, examples of the left search area 53 and the right search area 54 are shown as processing areas, but the present invention is not limited to this. In the second embodiment, as an example of the camera calibration process, processing is performed on each of the two areas of the left search area 53 and the right search area 54. However, the present invention is not limited to this. Absent.

  (3b) In the first embodiment and the second embodiment, the memory 9 shows an example in which the initial mounting angles of the front camera 2 and the rear camera 3 installed in the host vehicle are recorded as design values. However, the present invention is not limited to this. For example, when the initial mounting angles of the front camera and the rear camera installed in the host vehicle are not used as the design values for the camera calibration process, the initial mounting angles may not be recorded in the memory.

  (3c) In the second embodiment, the coordinate information that maximizes the length of the line segment of the vertical edge is selected as the coordinate information of the vertical edge. However, the present invention is not limited to this. For the selection of the coordinate information of the vertical edge, for example, coordinate information in which the length of the line segment of the vertical edge becomes a certain length or more may be selected.

  (3d) In the first embodiment, the average value is exemplified as the statistical value, but the present invention is not limited to this. Examples of the statistical value include an average value and a median value of the mode value category. Moreover, in the said 2nd Embodiment, although the average value of the mode value division was illustrated as a statistical value, it is not limited to this. Examples of the statistical value include an average value and a median value.

  (3e) In the first embodiment and the second embodiment, as the camera calibration process, the angle of inclination of the vertical edge is accumulated, and the front camera 2 is based on a value statistically processed from the accumulated data. Although an example in which the calibration process is performed is shown, the present invention is not limited to this. As the camera calibration process, for example, the camera calibration process may be performed every time the angle is calculated without accumulating the angle of inclination of the vertical edge.

In (3f) of the first and second embodiments, the calibration process of a camera, performs the processing for judging whether at least one of the absolute value of the calculated value is larger than the threshold TH ₁ An example is shown, but the present invention is not limited to this. Calibration process cameras, for example, at least one of the absolute value without processing to determine whether it is larger than the threshold TH ₁ of the calculated value by performing a process that requires a calibration of the camera Also good.

  (3g) The image processing unit 6 accumulates data of the angle of inclination of the vertical edge, and when the distribution of the accumulated plurality of data varies from a predetermined degree of variation, a process that requires camera calibration is performed. It does not have to be done. According to such a configuration, the case where the detection of the angle of inclination of the vertical edge is not stable can be eliminated.

  (3h) In the first embodiment and the second embodiment, as an example of the camera calibration process, an alarm is displayed on the display device that the front camera 2 needs to be calibrated. However, the present invention is not limited to this. is not. The camera calibration process may not display an alarm on the display device that the camera needs to be calibrated.

  (3i) In the first embodiment and the second embodiment, as the process of calculating the inclination of the vertical edge, the vertical edge is extracted from the target image 52 after the viewpoint conversion process, and the extracted inclination angle of the vertical edge is determined. Although the example which performs the process to calculate was shown, it is not limited to this. The vertical edge inclination is calculated by extracting the vertical edge before the viewpoint conversion process, calculating the coordinate information of the extracted vertical edge inclination, and correcting the coordinate information based on the viewpoint conversion process. The angle of inclination may be calculated.

  (3j) In the second embodiment, the example in which the traffic light is the ramp portion of the traffic light and the display portion of the road sign is shown as the visual recognition target. However, the present invention is not limited to this. Examples of the visual recognition target include an object having a specific shape.

  (3k) In the first embodiment and the second embodiment described above, examples of the front camera 2 and the rear camera 3 including wide-angle lenses are shown as the camera, but the present invention is not limited to this. If the camera is not equipped with a wide-angle lens, the distortion correction process need not be performed.

  (3l) In the first embodiment, the image processing unit 6 may proceed to the process of step 7 when the frequency near the average value of the angle of the vertical edge is equal to or greater than a certain ratio of the accumulated data. In the second embodiment, the image processing unit 6 may proceed to the process of step 27 when the frequency near the mode value of the angle of the vertical edge becomes equal to or greater than a certain ratio of the accumulated data.

  (3m) In the first embodiment, as an example of the camera calibration process, when the number of target images 52 including vertical edges is greater than a certain number, the process proceeds to step 6. However, the present invention is not limited to this. . As camera calibration processing, for example, instead of step 5, processing for determining whether or not the time for accumulating the angle of inclination of the vertical edge has passed for a certain period of time may be performed. In this case, if an affirmative determination is made, the process proceeds to step 6. If a negative determination is made, the process returns to step 1. Further, as the camera calibration process, for example, the process may proceed to step 6 without performing the process of step 5.

  Similarly, in the second embodiment, as an example of the camera calibration process, when the number of target images 52 including vertical edges is greater than a certain number, the process proceeds to step 26. However, the present invention is not limited to this. .

  (3n) After finishing the first camera calibration process and the second camera calibration process, the image processing unit 6 uses the feature points on the road surface of the common area of the left and right cameras and the front camera 2 to calibrate the left and right cameras. Necessity determination processing may be performed. Similarly, after completing the first camera calibration process and the second camera calibration process, the image processing unit 6 uses the feature points on the road surface of the common area of the left and right cameras and the rear camera 3 to calibrate the left and right cameras. Necessity determination processing may be performed.

  (3o) A plurality of functions of one constituent element in the above embodiment may be realized by a plurality of constituent elements, or a single function of one constituent element may be realized by a plurality of constituent elements. . Further, a plurality of functions possessed by a plurality of constituent elements may be realized by one constituent element, or one function realized by a plurality of constituent elements may be realized by one constituent element. Moreover, you may abbreviate | omit a part of structure of the said embodiment. In addition, at least a part of the configuration of the above embodiment may be added to or replaced with the configuration of the other embodiment. In addition, all the aspects included in the technical idea specified only by the wording described in the claim are embodiment of this invention.

  (3p) In addition to the image processing unit 6 described above, a system including the image processing unit 6 as a constituent element, a program for causing a computer to function as the image processing unit 6, and a non-transitive semiconductor memory storing the program The present disclosure can also be realized in various forms such as a substantial recording medium.

  (3q) The image processing unit executes the process of the flowchart as a function configuration realized by the CPU executing the program. The method for realizing these processes constituting the image processing unit is not limited to software, and some or all of the elements may be realized using hardware that combines a logic circuit, an analog circuit, and the like.

DESCRIPTION OF SYMBOLS 1 ... Camera calibration system, 2 ... Front camera, 3 ... Rear camera, 4 ... ECU, 5 ... Display apparatus, 6 ... Image processing part, 7 ... Image signal processing part, 8 ... In-vehicle signal processing part, 9 ... Memory, 10 ... power supply circuit, 51 ... captured image with distortion corrected, 52 ... target image, 53 ... left search area, 54 ... right search area.

Claims (14)

Extraction unit (6, S1 to S4, S21 to S24, S41) that extracts vertical edges of an object having a linear portion extending in the vertical direction in a captured image acquired from a camera (2, 3) installed in a vehicle. To S46),
An inclination calculating section (6, S4, S24, S47 to S49) for calculating the inclination of the vertical edge extracted by the extracting section;
Based on the inclination of the vertical edge calculated by the inclination calculation unit, a deviation between a current attachment angle of the camera installed in the vehicle and an initial attachment angle of the camera installed in the vehicle is calculated. A deviation calculation unit (6, S9, S10, S29, S30) for performing a deviation calculation process to be calculated;
A camera calibration device (6). The camera calibration device according to claim 1,
The extraction unit extracts the vertical edge from each of a left region and a right region obtained by dividing the captured image into left and right,
The inclination calculation unit calculates a first inclination that is an inclination of the vertical edge extracted from the left area and a second inclination that is an inclination of the vertical edge extracted from the right area;
The deviation calculation unit is a camera calibration apparatus that performs the deviation calculation process by comparing the first inclination and the second inclination calculated by the inclination calculation unit. The camera calibration device according to claim 1 or 2,
The extraction unit extracts the vertical edge from an upper region set in the captured image,
The tilt calculation unit is a camera calibration device that calculates the tilt of the vertical edge extracted from the upper region. The camera calibration device according to any one of claims 1 to 3,
The extraction unit performs a viewpoint conversion process (6, S2, S22) for converting the captured image into an image of a viewpoint with a pitch angle of the camera of 0 degrees, and the vertical edge is extracted from the image after the viewpoint conversion process. Camera calibration device to extract. The camera calibration device according to any one of claims 1 to 4, wherein
The camera calibration device further comprises a memory (9),
A camera calibration apparatus in which an initial design value of the camera installed in the vehicle is recorded in a memory. A camera calibration device according to any one of claims 1 to 5,
The inclination calculation unit calculates the inclination of the vertical edge having the maximum length of the line segment of the vertical edge from the vertical edges extracted by the extraction unit (6, S24, S49), and camera calibration. apparatus. The camera calibration device according to any one of claims 1 to 6,
An accumulation unit (6, S4, S24, S41 to S46) for accumulating the data of the inclination of the vertical edge calculated by the inclination calculation unit;
A statistical unit (6, S6, S26) for calculating a statistical value that is a statistically processed value from the plurality of data accumulated by the accumulation unit;
Further comprising
The said deviation calculation part is a camera calibration apparatus which performs the said deviation calculation process based on the said statistical value calculated by the said statistics part. The camera calibration device according to claim 7,
The deviation calculating unit performs the deviation calculating process when the statistical value calculated by the statistical unit is determined to be larger than a predetermined slope value (YES in S6, YES in S27). Perform a camera calibration device. The camera calibration device according to claim 7 or claim 8,
The deviation calculation unit is a camera calibration apparatus that does not perform the deviation calculation process when the distribution of the plurality of data accumulated by the accumulation unit varies from a predetermined variation degree. The camera calibration device according to any one of claims 1 to 9,
Display processing for displaying on the display device (5) that the current mounting angle of the camera installed in the vehicle is deviated from the initial mounting angle of the camera installed in the vehicle A camera calibration device further comprising a unit (6, S11 to S13, S31 to S33). A camera calibration device according to any one of claims 1 to 10,
The said extraction part is the camera calibration apparatus which extracts the said vertical edge of the said object after recognizing the said object in the said captured image (6, S1-S4). A camera calibration device according to any one of claims 1 to 11,
The object includes a visual recognition target to be visually recognized and a support column that supports the visual recognition target,
The said extraction part is a camera calibration apparatus which extracts the said vertical edge of the said support pillar with which the recognized said visual recognition object is provided after recognizing the said visual recognition target in the said captured image (6, S21-S24, S41-S46). A camera calibration device according to any one of claims 1 to 12,
The camera calibration apparatus, wherein the object is at least one of a traffic light and a road sign. In a captured image acquired from a camera (2, 3) installed in a vehicle, vertical edges of an object having a linear portion extending in the vertical direction are extracted (6, S1-S4, S21-S24, S41-S46). ),
Calculating the inclination of the extracted vertical edge (6, S4, S24, S47 to S49);
Based on the calculated inclination of the vertical edge, a deviation between the current mounting angle of the camera installed in the vehicle and the initial mounting angle of the camera installed in the vehicle is calculated (6, S9, S10, S29, S30), camera calibration method.

Images