DE102024203931A1 - Vehicle camera calibration on the test bench using video - Google Patents

Abstract

The invention relates to a method for calibrating a camera arranged in a vehicle, comprising the steps of: providing a video recorded with a first camera (3) of a first vehicle (1) and assigning it a pitch angle and a roll angle; setting up a second camera (7) in a test rig; playing back the provided video and transmitting the image information of the video to the second camera (7) in the second vehicle (5); and calibrating the second camera (7) with respect to an orientation taking into account the temporal profiles of the pitch angle and the roll angle of the first camera (3) assigned to the played video;

Description

The invention relates to a method for calibrating a camera for a vehicle.

Modern vehicles typically feature a variety of driver assistance systems or are designed for automated operation, where conventional manual driving is replaced by an automated system. In all cases, camera systems typically play a crucial role in environmental perception.

• Die DE 11 2016 001 150 T5 betrifft hierzu ein Verfahren zum Kalibrieren von wenigstens zwei Kameras eines Fahrzeugs, wobei das Verfahren umfasst: Aufnehmen eines Bildes einer Szene durch jede Kamera, Bestimmen einer Bodenebene des Fahrzeugs anhand von Merkmalen des Bildes, Definieren eines Ursprungspunkts eines Bezugsrahmens des Fahrzeugs als auf der Bodenebene befindlich und Bestimmen einer Translation eines Bezugsrahmens der Kamera zum In-Übereinstimmung-Bringen einer Position des Kamera-Bezugsrahmens mit einer entsprechenden Position des Fahrzeug-Bezugsrahmens, wobei die Bestimmung der Translation des Kamera-Bezugsrahmens anhand eines Bildes eines Kalibrierungsziels von der Kamera und eines Bildes des Kalibrierungsziels von wenigstens einer in der Nähe der Kamera vorgesehenen anderen Kamera durchgeführt wird.

A camera that points in the vehicle's main direction of travel, a so-called front camera, typically requires calibration with reference to the horizon or vanishing point. Various methods for calibrating vehicle cameras are known in the art:

• The DE 11 2016 001 150 T5 This relates to a method for calibrating at least two cameras of a vehicle, wherein the method comprises: capturing an image of a scene by each camera, determining a ground plane of the vehicle based on features of the image, defining an origin point of a reference frame of the vehicle as being located on the ground plane, and determining a translation of a reference frame of the camera to align a position of the camera reference frame with a corresponding position of the vehicle reference frame, wherein the determination of the translation of the camera reference frame is carried out based on an image of a calibration target from the camera and an image of the calibration target from at least one other camera provided in the vicinity of the camera.

The EP 2 665 037 A1 The device further relates to an automatic calibration device with an onboard camera, comprising: a camera mounted on a vehicle to obtain multiple images of the vehicle's surroundings; a motion vector calculation unit configured to determine motion vectors among the multiple images obtained from the camera; a road surface point selection unit configured to select points that are no fewer than three on a road surface in a predetermined image among the multiple images, and to select multiple combinations of two points from the points that are no fewer than three on the road surface; a Z-axis rotation calculation unit configured to determine, in a camera coordinate system with an optical axis as the Z-axis, a rotation about the Z-axis such that the road surface is parallel to an X-axis, based on the multiple combinations of the two point selection units and motion vectors corresponding to each point of the multiple combinations of the two points; and a coordinate rotation correction unit configured to perform a rotation correction about the Z-axis for the images received from the camera based on the amount of rotation about the Z-axis determined by the calculation unit for the amount of rotation about the Z-axis.

The DE 10 2008 004 370 A1 The document also relates to an image adjustment method for a video image, which is continuously captured by means of an image sensor of a first camera of a motor vehicle by means of pixels arranged in image lines from the environment of the motor vehicle and displayed on a display device of the motor vehicle, with respect to a rotation relative to at least one reference object, wherein in a first step the video image is measured using the at least one reference object, whereby a required rotation angle of the video image relative to the reference object is determined as the measurement result; wherein in a second step a corresponding shear factor for the image lines is determined from the determined required rotation angle, in particular taking into account the line width; and wherein in a third step an image processing device stores the shear factor and uses it for the continuous shearing of the image lines.

For the design and/or validation of an automated driving control system, be it a driver assistance system or a control system for autonomous vehicle operation, it is often advantageous to place a vehicle component in a stationary test rig and simulate a traffic situation in software using artificial sensor signals and/or control signals. Conducting a real test drive to calibrate a camera is complex and naturally requires the availability of a vehicle.

The object of the invention is therefore to calibrate a front camera of a vehicle on such a HiL test bench.

The invention is defined by the features of the independent claims. Advantageous further developments and embodiments are the subject of the dependent claims.

• - Bereitstellen eines während einer realen Fahrt mit einer ersten Kamera eines ersten Fahrzeugs aufgenommenen Videos der Umgebung des ersten Fahrzeugs;
• - Bereitstellen eines während derselben Fahrt des ersten Fahrzeugs mit einer inertialen Messeinheit erzeugten Datensatzes über einen zeitlichen Verlauf eines Nickwinkels und eines zeitlichen Verlaufs eines Rollwinkels des ersten Fahrzeugs gegenüber der Erde, wobei der jeweilige zeitliche Verlauf zeitlich Bildinformationen des Videos zugeordnet ist;
• - Aufstellen einer zweiten Kamera in einem Versuchsstand und Verbinden einer Videoschnittstelle an einem Bildsensor der zweiten Kamera mit einer Datenquelle, sodass an der zweiten Kamera anstatt der Erzeugung eines von einem aktuellen Lichteinfall abhängigen Signals ein künstliches Signal aus der Datenquelle an der Videoschnittstelle empfangen wird;
• - Abspielen des bereitgestellten Videos aus der Fahrt des ersten Fahrzeugs und Übermitteln der Bildinformationen des Videos über die Datenquelle an die zweite Kamera;
• - Kalibrieren der zweiten Kamera bezüglich einer Ausrichtung unter Berücksichtigung der zum abgespielten Video zugeordneten zeitlichen Verläufe des Nickwinkels und des Rollwinkels der ersten Kamera;

A first aspect of the invention relates to a method for calibrating a camera arranged in a vehicle, comprising the steps of:

• - Providing a video of the surroundings of the first vehicle, recorded during an actual journey with a first camera of a first vehicle;
• - Providing a data set generated during the same journey of the first vehicle with an inertial measurement unit about a time course of a pitch angle and a time course of a roll angle of the first vehicle relative to the earth, wherein the respective time course is assigned to time-related image information of the video;
• - Setting up a second camera in a test setup and connecting a video interface on an image sensor of the second camera to a data source, so that instead of generating a signal dependent on an actual light incidence, an artificial signal from the data source is received at the video interface of the second camera;
• - Playing the provided video from the journey of the first vehicle and transmitting the image information of the video via the data source to the second camera;
• - Calibrating the second camera with respect to its alignment, taking into account the temporal profiles of the pitch angle and roll angle of the first camera associated with the played video;

As a prerequisite for the successful execution of the procedure, an actual test drive with a first vehicle must be carried out at a later date. A first camera is installed in the first vehicle and directed towards the vehicle's surroundings. Preferably, the first camera is aligned with the vehicle's main direction of travel. Calibration is necessary for object recognition and for using the optical flow during the vehicle's journey; this calibration establishes the vanishing point or a horizon line.

The vanishing point, or horizon line, depends in particular on the vehicle's current pitch and roll angles. To compensate for these angles, a vehicle equipped with a calibrated camera typically continuously records the pitch and roll angles to enable image analysis based on camera data. Pitch and roll angles are preferably measured using an inertial measurement unit while the vehicle is in motion. For example, gyroscopes can be used, whose stability is exploited to determine changes in the vehicle's longitude.

While the information about the current pitch angle and roll angle can already be used to compensate for the rotational movement of the vehicle when the first vehicle is traveling with the first camera, this information is also recorded and temporally assigned to the sequence of a video generated by the stream of image information from the first camera for the purpose of carrying out the procedure.

For the second camera on a test stand, a video from a real journey is thus available, and for each video frame of the video at a specific time, information about the current pitch angle and roll angle of the first camera, with which the video was recorded, is available.

A second camera, sharing essential characteristics with the first, is placed in a test rig, preferably by mounting it in a second vehicle located within the test rig or driving simulator itself. The second camera is prepared or modified so that its signal is not generated by incident light, but rather electronically via a video interface on the second camera, the video from the actual movement of the first vehicle is fed in as if the second camera were currently located in the first vehicle and recording the image information during its journey.

During the playback of the video from the actual journey at the video interface of the second camera, a corresponding data stream of the recorded pitch and roll angles is also played back to represent the vehicle's rotational movement associated with the video. The temporal profiles of the pitch and roll angles serve to provide the relevant states for calibrating the second camera, enabling precise calibration.

The advantage of this approach is that the same video from the first camera in the first vehicle can be used to repeatedly calibrate a second camera in a test rig, without the latter being used in an actual test drive. This also allows for the advantageous calibration of multiple second cameras using the same video from the first camera.

According to an advantageous embodiment, a time-based speed profile of the first vehicle during its journey is provided and used to calibrate the second camera.

According to a further advantageous embodiment, the second camera is located in a second The vehicle is positioned in a test rig. This allows, particularly during vehicle assembly, the second camera to be mounted and calibrated in the second vehicle without conducting a test drive with the second vehicle.

According to a further advantageous embodiment, the test rig is a hardware-in-the-loop test rig, and the second vehicle has a driver assistance system or an automatic driving control system which is operated with played-back, previously recorded sensor data and/or synthetic sensor data.

According to a further advantageous embodiment, the respective time course of the pitch angle and the roll angle of the first camera replaces a data stream of a real inertial measuring unit of the second vehicle when calibrating the second camera.

While in regular operation of the second vehicle such an inertial measuring unit would continuously provide the current pitch and roll angles of the vehicle relative to the Earth, and thus also the pitch and roll angles of the second camera relative to the Earth, in the test setup this data stream can be replaced by the artificial one previously recorded in the first vehicle, and the temporal profiles of these angles from the real test phase can be used to compensate for an apparent movement of the environment of the second vehicle according to the image information from the video.

According to a further advantageous embodiment, the first camera and the second camera are identical in construction. Preferably, the first camera and the second camera are of the same type, so that a video recorded with the first camera can be played back at the video interface of the second camera in a compatible manner, thus creating the effect at the second camera that the played-back video at the video interface is equivalent to actual image information obtained during a real journey.

During the test drive, the first camera in the first vehicle has a certain mounting height above the ground. This mounting height is reflected in the image information in the video from the first camera. If the mounting heights of the first and second cameras in their respective vehicles differ too much, the calibration of the second camera can become inaccurate or faulty.

According to a further advantageous embodiment, the installation height of the first camera in the first vehicle above the ground and the installation height of the second camera in the test rig is within predetermined limits with regard to a deviation between the installation heights.

According to a further advantageous embodiment, a deviation in the installation height of the first camera in the first vehicle above ground to the installation height of the second camera in the test rig is compensated by image transformation.

According to a further advantageous embodiment, a plurality of second cameras are calibrated with one and the same video from the first vehicle and the time series of the roll angle and pitch angle of the first vehicle associated with the video.

According to a further advantageous embodiment, the first camera and the second camera are front cameras. Front cameras are characterized by the fact that they are directed in a main direction of travel of the vehicle.

Further advantages, features, and details will become apparent from the following description, in which – possibly with reference to the drawing – at least one embodiment is described in detail. Identical, similar, and/or functionally equivalent parts are identified by the same reference numerals.

• 1: Ein Verfahren zum Kalibrieren einer in einem Fahrzeug angeordneten Kamera gemäß einem Ausführungsbeispiel der Erfindung.

It shows:

• 1 : A method for calibrating a camera arranged in a vehicle according to an embodiment of the invention.

1 Figure 1 shows a method for calibrating a second camera 7 located in a second vehicle 5. Prior to the calibration process of the second camera 7 in the second vehicle 5, a test drive of the first vehicle 1 with a first camera 3 takes place. The first camera 3 is a front camera and thus directed in the main direction of travel of the first vehicle 1. During the drive of the first vehicle 1, a video is recorded using the first camera 3. The first vehicle 1 also has an inertial measurement unit, whereby both the video and a pitch angle and a roll angle of the first vehicle 1 are stored in their respective temporal relationships, so that a specific point in time in the video from the first camera 3 can be assigned to a pitch angle and roll angle of the first vehicle 1. By determining a pitch angle and a roll angle using the inertial measurement unit of the first vehicle 1, a pitch angle and a roll angle of the first camera 3 can also be determined. The orientation relative to Earth is known, as the first camera 3 on the first vehicle 1 is assumed to be stationary. For the calibration process of the second camera 7, the video, along with the respective time-dependent pitch and roll angles as recorded by the first vehicle 1, is fed into a video interface of the second camera 7. The fed-in video behaves as if the second vehicle 5 were moving and the second camera 7 were capturing the surrounding environment. The second camera 7 is located on the second vehicle 5, which, however, remains stationary during the calibration process and is positioned in a test rig. In a processing unit of the second vehicle 5, the second camera 7 is calibrated according to its orientation by playing back the video from the first vehicle 1. The pitch and roll angle information in the video is compensated for by the pitch and roll angle information from the associated file. This calibrates a vanishing point and a horizon line for the second camera 7. The video can be used as a further source for further calibration for another second vehicle 5, without the need for further real test drives.

Although the invention has been further illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations can be derived from them by a person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned as examples are truly only examples and are not to be understood in any way as limiting, for instance, the scope of protection, the possible applications, or the configuration of the invention. Rather, the preceding description and the description of the figures enable a person skilled in the art to implement the exemplary embodiments in concrete terms. With knowledge of the disclosed inventive concept, a person skilled in the art can make numerous modifications, for example, regarding the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

List of reference symbols

1

first vehicle

3

first camera

5

second vehicle

7

second camera

QUOTES CONTAINED IN THE DESCRIPTION

This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited patent literature

• DE 11 2016 001 150 T5 [0003]
• EP 2 665 037 A1 [0004]
• DE 10 2008 004 370 A1 [0005]

Claims (10)

A method for calibrating a camera installed in a vehicle, comprising the steps of: - Providing a video of the surroundings of the first vehicle (1) recorded during an actual journey with a first camera (3) of the first vehicle (1); - Providing a data set generated during the same journey of the first vehicle (1) with an inertial measurement unit, showing the temporal evolution of a pitch angle and a temporal evolution of a roll angle of the first vehicle (1) relative to the ground, wherein the respective temporal evolution is associated with temporal image information from the video; - Setting up a second camera (7) in a test rig and connecting a video interface on an image sensor of the second camera (7) to a data source, such that instead of generating a signal dependent on the current light incidence, an artificial signal from the data source is received at the video interface of the second camera (7); - Playing back the provided video from the journey of the first vehicle (1) and transmitting the image information of the video via the data source to the second camera (7); - Calibrating the second camera (7) with respect to an alignment taking into account the temporal profiles of the pitch angle and roll angle of the first camera (3) associated with the played video; Procedure according to Claim 1 , wherein a time series of the speed of the first vehicle (1) during its journey is provided and is used to calibrate the second camera (7). Method according to one of the preceding claims, wherein the second camera (7) is arranged in a second vehicle (5) which is placed in a test rig. Procedure according to Claim 3 , wherein the test setup is a hardware-in-the-loop test setup or driving simulator, and the second vehicle (5) has a driver assistance system or an automatic driving control system which is operated using played-back previously recorded sensor data and/or synthetic sensor data. Procedure according to Claim 4 , wherein the respective temporal profile of the pitch angle and the roll angle of the first camera (3) replaces a data stream of a real inertial measuring unit of the second vehicle (5) during the calibration of the second camera (7). Method according to one of the preceding claims, wherein the first camera (3) and the second camera (7) are identical in construction. Procedure according to one of the Claims 1 until 6 , wherein the installation height of the first camera (3) in the first vehicle (1) above ground and the installation height of the second camera (7) in the test rig is within specified limits with respect to a deviation between the installation heights. Procedure according to one of the Claims 1 until 6 , wherein a deviation in the installation height of the first camera (3) in the first vehicle (1) above ground to the installation height of the second camera (7) in the test rig is compensated by image transformation. Method according to one of the preceding claims, wherein a plurality of second cameras (7) are calibrated with one and the same video from the first vehicle (1) and the time profiles of the roll angle and pitch angle of the first vehicle (1) associated with the video. Method according to one of the preceding claims, wherein the first camera (3) and the second camera (7) are front cameras.